> For the complete documentation index, see [llms.txt](https://docs.origintrail.io/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.origintrail.io/operate/swm-public-snapshot-gc.md). # SWM Public Snapshot Garbage Collection The file-backed shared-memory (SWM) public snapshot store enables a pressure-based garbage collector by default. GC v1 is an incident guardrail: it bounds disk growth using file age and free-space watermarks. It does not inspect RDF references or replication proofs. Set `gc.enabled` to `false` for an explicit per-node opt-out. An additional **opt-in finalized-publish collector** can remove unreferenced graph-scoped KA snapshot files after publication, even when disk space is ample. It is separate from v1's emergency age-based eviction; see below. ## Recommended 75 GiB node configuration ```json { "sharedMemoryPublicSnapshotStorage": { "enabled": true, "directory": "/var/lib/dkg/swm-public-snapshots", "gc": { "enabled": true, "intervalMs": 300000, "triggerFreeBytes": 16106127360, "targetFreeBytes": 26843545600, "hardReserveBytes": 5368709120, "minAgeMs": 604800000, "staleTempAgeMs": 3600000 } } } ``` GC is enabled unless `gc.enabled` is `false`. When enabled, the store: 1. Checks the snapshot filesystem every `intervalMs` and before a write that could cross a watermark. 2. Removes abandoned atomic-write `.tmp` files older than `staleTempAgeMs`. 3. Starts snapshot eviction below `triggerFreeBytes` and deletes eligible `.nq` or legacy `.json` files oldest first. 4. Stops when `targetFreeBytes` is available or no eligible files remain. 5. Never age-evicts a file newer than `minAgeMs`, or a file in use by this process. 6. Rejects a new snapshot with error code `SNAPSHOT_STORAGE_CAPACITY` when the write would consume `hardReserveBytes` and GC cannot recover enough space. The hard reserve protects the triple store and other node state from an `ENOSPC` cascade. It should be lower than the trigger; the target should be at least as high as the trigger. On filesystems of 75 GiB or more, the automatic defaults are a 5 GiB reserve, 15 GiB trigger, and 25 GiB target. On smaller filesystems, all three automatic watermarks scale together to 1/15, 1/5, and 1/3 of total filesystem capacity. Supplying any watermark opts into fixed byte values, with omitted watermarks retaining their unscaled defaults. ## V1 safety boundary V1 treats sufficiently old snapshots as recoverable cache. Metadata may still refer to an evicted digest. Normal SWM synchronization detects a missing or invalid local blob, fetches it from a peer, verifies its digest and count, and writes it back. Some direct local resolution paths do not yet refetch on demand, so nodes that may hold the only copy of locally produced data should use a longer minimum age or leave v1 disabled until replication is established. GC v2 will replace age as the deletion proof with recorded provenance, replication/finality evidence, leases, and cache-miss rehydration. See [`../active-now/swm-public-snapshot-gc-v2-spec.md`](https://github.com/OriginTrail/dkg/tree/main/docs/active-now/swm-public-snapshot-gc-v2-spec.md). ## Automatic cleanup after confirmed publication Merge this into the existing node configuration to enable the new collector: ```json { "sharedMemoryPublicSnapshotStorage": { "gc": { "enabled": true, "finalizedCleanupEnabled": true, "finalizedRetentionMs": 86400000 } } } ``` `finalizedCleanupEnabled` defaults to **false**. The retention interval defaults to **24 hours**, measured from the latest retirement request for that digest, not from the file's modification time. Configuration takes effect at startup. The existing confirmed/durable graph-scoped KA cleanup path records the digest in a small `.retired` sidecar before removing its SWM graph and current operation metadata. The background collector, on its existing five-minute interval: 1. Waits for the retirement grace period. 2. Checks all RDF graphs for explicit snapshot references or implicit digest references without a graph-backed snapshot. Another operation/context graph referencing the same bytes prevents deletion. 3. Excludes active readers, serving pages, and file-plus-metadata writes. With finalized cleanup enabled, sync public and private recovery hold each touched file's lease through the final metadata commit. Reused refs take the same lease and the store checks that the file still exists without decoding the payload again; if collection already removed it, normal recovery fetches it again before committing metadata. Leases are shared by physical directory identity, including symlink/junction and case aliases. A busy digest does not prevent collection of unrelated digests. With cleanup disabled these operation-long leases are not taken, so pressure GC treats a file only as in use while an individual read or write is running, as before this feature. Reuse still asks the store whether the file exists (`snapshotExists`, a check whose short lease stays inside the store): a file that pressure GC removed between rounds is fetched again rather than reported as present. The probe answers "absent" only for a file that does not exist. A file that is present but cannot be opened (a permission or I/O error, something that is not a regular file) makes the probe fail, and that failure ends the operation instead of being read as a missing file: fetching a copy would not repair the file, because a write skips a path that already exists. A failure inside a sync pass still reports the progress the pass had made. 4. Deletes the unreferenced `.nq`/legacy `.json` file, its cached validation/page index, and its persisted page-index row where the adapter supports deletion. 5. Removes the retirement record last. An interrupted deletion can finish on the next pass after restart. Reusing a digest with `putSnapshot` cancels retirement; another confirmed publication starts a new grace period. A publication's cleanup is bounded by the assertion version that publication confirmed. It runs under the per-asset SWM write lock and first resolves the asset's SWM head. When the head is already at a later version (one shared or staged while the publication waited for confirmation), or cannot be resolved, the cleanup records no retirement and removes nothing: that version's SWM graph, operation rows, StorageACK copies and snapshot stay in place until its own publication confirms or the SWM TTL expires. A core that signed a StorageACK for the asset also holds that copy's operation row (id prefix `storage-ack-`), which carries the same snapshot digest and would otherwise keep the file referenced until the SWM TTL. The same cleanup boundary removes those rows once the SWM graph is dropped, only for the same asset and only at or below the newest version being cleaned up. It does this before the asset's own operation rows are deleted, because that version boundary is read from them; with those rows gone the update finds no boundary and removes nothing. The rows go in one conditional store update: the store itself joins the copies to the cleaned operations of the same asset, so neither the number of copies (there is no cap) nor the order a store would return them in changes which are removed. Versions are compared as canonical decimal strings (by length, then digit by digit), so the boundary is exact for any positive assertion version: beyond 2^53 a JavaScript number is inexact, and beyond 2^63 - 1 a numeric comparison in the store is no longer supported on Oxigraph. A version is usable when the store reads it as a canonical positive integer (`xsd:integer` with only digits and no leading zero, which is what every writer emits); a literal that is not one is neither a boundary nor a discharged copy. Which other spellings a store still folds to an integer (a datatype such as `xsd:int`, a leading `+` or zero) depends on the store, and none of them is ever treated as a larger version than it is. A later version's copy, another asset's copy, any other kind of operation and the cleaned operations themselves keep counting as references. Without a usable version among the cleaned operations there is no boundary, and nothing is removed. The update is bounded on the client side (10 seconds, with no abort signal handed to the store, as for the other cleanup queries). An update that fails, is refused or times out, or a store that cannot run an update, leaves the rows (and so the file) in place and logs one warning. Removing the rows is the store's cost: measured, 700 copies take tens of milliseconds on Oxigraph and seconds on Blazegraph, and 4,000 copies on the order of ten seconds or more on Blazegraph (machine dependent, and about the same as removing them one page at a time), so an unusually large set on a Blazegraph node can outlast the bound. Then the warning is logged, the cleanup carries on, and the copies stay unless the store still completes the update. Failures to record retirement are logged and do not turn a successful publish into an error. A missing checker, unreadable record, failed reference query, or failed unlink retains/retries the candidate. Pressure GC never age-evicts a marked candidate. While free space is above the hard reserve, a candidate waits out its grace period. When free space (less the size of a pending write) is below `hardReserveBytes`, marked candidates become eligible before their grace period ends, until the reclaim target is met. They still go through the same reference check, so an unavailable checker, a malformed record or a failed query retains the file, and a referenced file is kept. Each pass examines at least 32 records and a pass keeps going past that while candidates keep clearing, within a five-second scheduling budget (an in-flight reference query may run to its two-second deadline). It stops at the first retained candidate after the first 32, rotating past busy, retained and failed candidates to avoid starving later files. Records are visited in digest order (code-unit comparison, independent of the process locale), starting after the last one examined. That position is saved in `finalized-collection-cursor.json` in the snapshot directory, so a restart resumes instead of re-scanning from the start of a large backlog; a missing or unreadable file restarts from the beginning. The retirement lookup and the reference check bound their wait on the client side. They do not pass an abort signal to the triple store, whose own deadline governs the query. `[SWM-SNAPSHOT-GC]` logs distinguish `finalized`, `referenced`, and `failed` counts. The collector visits only the expected two hexadecimal directory levels. It does not enter unrelated directories such as a mounted volume's `lost+found`. ### Scope and rollout This is a narrow lifecycle addition, **not the complete proposed GC v2**. It does not inventory or independently prove finality for historical files, scan external job queues, obtain replication commitments, or add cache-miss fetching. Its retirement authority is the existing confirmed/durable SWM cleanup boundary; remaining RDF operation references prevent collection. It does not change the RDF cleanup policy or make arbitrary operations eligible for retirement. Use one daemon per snapshot directory. Cross-process maintenance/CLI access is not protected by the process-local lease mechanism. Do not enable it on a directory concurrently used by another process. Do not retarget a directory symlink/junction or replace its underlying directory while the process runs. A custom snapshot store without the optional `lifecycle` capability keeps its own retention behavior. A store that has one implements `snapshotExists(ref)` (the existence probe: it resolves `true` for a present file and `false` only for an absent one, and rejects for anything else, such as an unreadable file or a failing lease gate; any short lease it needs stays inside the store), `markPublished` (retirement scheduling), and reports `finalizedCleanupEnabled`. The lease itself is one optional operation, `operationLease(ref)`: a store that offers it makes every publication and recovery hold one lease per touched digest until its final metadata commit, and a store that omits it takes none. The file store offers it exactly when finalized cleanup is enabled. A lifecycle that reports finalized cleanup enabled must offer `operationLease`, and the `WorkspaceSnapshotLifecycle` type requires it. Plain JavaScript, a cast, or an object written for an earlier development shape (`acquire` / `acquireExisting`, which never shipped in a release) gets past the type, so the same rule is enforced at run time, where the lifecycle is first used (when an operation scope starts, and when the publisher is constructed): a lifecycle that reports finalized cleanup enabled without an `operationLease` function, or that has no `snapshotExists` function, makes the scope or the publisher throw an error that names the current contract. It does not fall back to another method, and nothing runs without the operation-long leases that cleanup relies on. A cleanup-disabled lifecycle may omit `operationLease`; that is its retention policy, and it raises no error. A `snapshotExists` that resolves anything other than a boolean is refused too, rather than read as "absent". `WorkspaceSnapshotScope` owns a complete operation's I/O and `retainExisting` leases. Publication and recovery use `snapshotOperation`, whose implementation receives a scope rather than a raw store; leases close after the final metadata commit, on either success or failure. Missing reused bytes follow the existing fetch path. The scope must account for every I/O method, including optional ones. The file store accepts `isSnapshotReferenced` once in its constructor options. The agent's optional `publicSnapshotStoreFactory` receives the constructed RDF store. The daemon uses this factory to share one indexed snapshot store across agent and publisher, without a late-bound checker. CLI construction uses named options (`store`, `pageIndexStore`, `log`). Direct SDK users creating a file store must pass `snapshotReferenceCheck(theTripleStore)` to enable collection. Without a checker, marked candidates are retained. `PublishedSnapshotRetirement` owns RDF discovery and canonical literal decoding at the durable cleanup boundary. `FinalizedSnapshotCollector` owns marker persistence, bounded scheduling and reference checks; the file store supplies payload and derived-index removal. Marker changes are queued per digest across all directory aliases: a later retirement or reuse cannot be overwritten by an older pending rename. Physical-directory lookup/enqueue preserves request order, including cold aliases; marker I/O for different digests remains concurrent. All cleanup, including stale temporary files, uses the same shared lifecycle gate. Existing unmarked files, legacy entity/root-scoped publishes, and candidates whose retirement record could not be saved remain under the original pressure policy. This feature does not immediately reclaim the pre-upgrade backlog. Start with one node and new graph-scoped publishes; verify VM reads/durable sync, reference retention, restart recovery and GC logs before enabling it elsewhere. Disabling `finalizedCleanupEnabled` returns to the existing pressure policy; `gc.enabled=false` disables both collectors. --- # Agent Instructions This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com. ## Querying This Documentation If you need additional information that is not directly available in this page, you can query the documentation by asking a question. Perform an HTTP GET request on the following URL with the `ask` and `goal` query parameters: ``` GET https://docs.origintrail.io/operate/swm-public-snapshot-gc.md?ask=&goal= ``` `ask` is the immediate question: it should be specific, self-contained, and written in natural language. `goal` is what the user is ultimately trying to achieve, the reason they need the answer. Sharing it helps GitBook give you a better, more relevant answer. A goal is most helpful when it describes the outcome the user wants rather than restating the question. For example, with `ask=how do I create an API token`, a goal like `automate deployments from our CI pipeline` lets GitBook tailor the answer to that use case. The response will contain a direct answer to the question and relevant excerpts and sources from the documentation. Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.