> For the complete documentation index, see [llms.txt](https://docs.origintrail.io/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.origintrail.io/use-dkg/backpressure-observability.md).
# Backpressure Observability
DKG exposes one common pressure model for its in-memory schedulers. It answers four questions without requiring scheduler-specific log archaeology:
1. Which scheduler and lane are under pressure?
2. Is work waiting, rejected, or admitted but possibly stuck?
3. Which bounded operation classes account for that work?
4. Did the scheduler recover?
This is an observability layer only. It does not change queue ordering, concurrency, reservation, displacement, timeout, or retry behavior.
## Architecture
```mermaid
flowchart LR
subgraph Producers["Work producers"]
SWM["SWM and store callers"]
Sync["Sync requester stages"]
end
subgraph Schedulers["Existing scheduling policies"]
Store["StorePriorityScheduler ack / health / normal / background"]
Global["PriorityAdmissionQueue global sync admission"]
end
subgraph Core["Shared core observability"]
Lifecycle["ObservableScheduler lifecycle enqueue / start / reject / finish"]
Tracker["SchedulerPressureTracker ages, counts, bounded operation summaries"]
Registry["BackpressureRegistry"]
Monitor["BackpressureMonitor 5 s samples, transition and recovery logs"]
OTel["OpenTelemetry metrics"]
end
Status["Public /api/status state only"]
Admin["Node-admin /api/diagnostics/backpressure"]
Logs["Daemon log"]
SWM --> Store
Sync --> Global
Store --> Lifecycle
Global --> Lifecycle
Lifecycle --> Tracker
Tracker --> OTel
Tracker --> Registry
Registry --> Monitor
Monitor --> OTel
Monitor --> Logs
Registry --> Status
Registry --> Admin
```
The tracker records references to bounded labels and timestamps, not work closures or payloads. A snapshot can contain a maximum of eight queued and eight active operation summaries per lane. Labels are sanitized and truncated before they reach metrics, logs, or the diagnostics response.
The first registered sources are:
* `store`: the external-store priority scheduler and its `ack`, `health`, `normal`, and `background` lanes;
* `sync-global`: the process-wide sync admission queue and its sync lanes.
The two divide capacity differently, and every lane row names the model it is reported under in its `capacityModel` field:
| `capacityModel` | Meaning | Source |
| --------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------- |
| `partitioned` | The lane holds a private queue allocation and fills independently of its neighbours. Nothing validates the scheduler-level limit against the lane allocations — a scheduler may publish their sum (`store` does), or cap its total lower — so read a lane's own limit for lane pressure and the scheduler's for the rollup, and derive neither from the other. | `store` |
| `shared` | Every lane draws on one queue and one concurrency pool. The lane's `queueLimit`/`inflightLimit` **are** that pool's ceilings — the same number on every lane, never to be summed. | `sync-global` |
Three counts sit on every lane row beside the two ceilings, and they differ only under `shared`:
| Field | Meaning |
| ------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `queued` | This lane's own backlog — *who* is waiting. The attribution signal. |
| `pressureQueued` | The depth the lane's `state` was classified against, and the numerator that belongs with `queueLimit`. Equal to `queued` for a `partitioned` lane, and the whole pool's depth for a `shared` lane **that has work waiting against a ceiling**. Where no depth applies to a shared lane — an empty backlog, or a pool with no `queueLimit` — nothing was classified on depth and the lane reports its own `queued`, so it never reads as utilized. |
| `pressureInflight` | The count that belongs with `inflightLimit`: this lane's own under a `partitioned` allocation, and on a `shared` row the pool's occupancy, **always** — it has none of `pressureQueued`'s fallbacks, because no state is ever classified on concurrency, so a shared row's ratio is a pool ratio by construction and cannot contradict the row's own state. |
So compute utilization from **`pressureQueued / queueLimit`**, never from `queued`, and no consumer has to special-case the model. All three are optional on the type, so a scheduler written against an older `dkg-core` still satisfies it: absent `capacityModel` means `partitioned`, absent `pressureQueued` means `queued`, and absent `pressureInflight` means `inflight`.
### Attributing `sync-global` pressure to a trigger
The `lane` of a `sync-global` entry says *what kind of work* is queued (`durable`, `changelog`, `shared_memory`, `swm_recovery`), but every trigger funnels into the same few lanes. Its `operation` label therefore pairs the collapsed work class with the **admission source** — the trigger that enqueued it — as `:`:
| Source | Trigger |
| -------------------- | --------------------------------------------------------------------- |
| `catchup-foreground` | explicit Context Graph catch-up (`POST /api/context-graph/subscribe`) |
| `catchup-background` | automatic post-approval / reconcile catch-up |
| `on-connect` | sync-on-connect after a peer dial |
| `reconcile` | the periodic sync reconciler |
| `vm-recovery` | foreground repair of specific missing Knowledge Assets |
| `swm-recovery` | curator-targeted shared-memory recovery |
| `unspecified` | a caller that did not declare an origin |
### Tuning foreground catch-up
Two knobs govern the foreground Context Graph catch-up that most often shows up as `catchup-foreground` pressure. Both are read once at daemon start.
| Variable | Default | Effect |
| -------------------------------------- | -------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `DKG_CATCHUP_STOP_ON_PROOF` | on | The catch-up walks peers in escalating waves and stops once the resolved curator has settled every requested plane. Set to `0`, `false`, `no`, or `off` to restore the previous behaviour: every sync-capable peer, both requested planes, no early stop. Use this if a graph ever lands short — foreground catch-up optimises for one authoritative payload, while breadth remains the background reconcile lane's job. |
| `DKG_CATCHUP_BACKPRESSURE_MAX_WAIT_MS` | `180000` | Wall-clock budget one foreground plane may spend being **refused** by local `sync-global` admission before the job reports a retryable `deferred`. Measured from before the first attempt, so an attempt's own queue time counts against it. It does not cancel a round the scheduler has already accepted — that one is doing real work and is bounded by `SYNC_TOTAL_TIMEOUT_MS`. The default sits above both a full head-of-line round (120 s) and the queue waits that motivated it. An explicit `0` disables retries; a blank value is treated as unset. |
| `DKG_CATCHUP_MAX_CONCURRENT_PEERS` | `4` | Caps in-flight per-peer sync rounds, and therefore the widest escalation wave. Raising it above the `sync-global` queue depth lets a single catch-up saturate the scheduler against itself. |
So `{"operation":"durable:catchup-foreground","count":4,"oldestAgeMs":109000}` in a `queuedOperations` summary reads as "four explicit catch-up durable admissions are queued, the oldest for 109 seconds", and the matching `activeOperations` entry gives the same view for admitted work. Both halves are closed sets, so the label space stays bounded (5 × 8) and, as before, no Context Graph id or peer id ever reaches a metric, log line, or diagnostics response — an unrecognized source is clamped to `unspecified`.
The sync **responder** limiter is a separate queue and is not instrumented, so its `pre_authorization` and `responder` lanes appear in no snapshot, metric, or log line. `sync-global` covers requester-side admission only.
Other schedulers can extend `ObservableScheduler` and call its protected lifecycle methods at their existing admission boundaries. They keep complete ownership of policy, and declare how their lanes divide capacity through `capacityModel` — a scheduler that declares nothing is `partitioned`, and a lane of a partitioned scheduler that publishes no limit of its own is simply not classified on depth.
## Pressure states
| State | Meaning |
| ----------- | ----------------------------------------------------------------------------------- |
| `healthy` | No age, utilization, rejection, or active-duration threshold is crossed. |
| `degraded` | A queue is old or at least 75% utilized, but is not full. |
| `saturated` | A queue is full or an admission rejection occurred in the recent visibility window. |
| `stalled` | The oldest admitted operation crossed the scheduler's active-duration threshold. |
State precedence is `stalled` > `saturated` > `degraded` > `healthy`. A recent rejection remains visible for 60 seconds so a short full-queue event is not missed between monitor samples.
"A queue is full" means the queue that lane's work is actually waiting behind. For a `partitioned` lane that is its own allocation. For a `shared` lane it is the pool, so **every lane holding queued work reports the pool's pressure** — one lane at 1 of a full pool of 4 is `saturated`, because the next admission in that lane is the one that gets rejected. A lane with nothing queued stays `healthy` however full the pool is: it is not being held back, and per-lane `queued` and `queuedOperations` remain the attribution signal for who is. A log line whose `pressureQueued` differs from its `queued` carries both for this reason, and because the scheduler's own `lane: "all"` line is emitted only while the rollup outranks every lane — which, on an all-shared scheduler, it never does.
These states describe evidence, not root cause. For example, a stalled store operation can be caused by Blazegraph, Oxigraph, disk, or a caller that never settles. Use the operation summary and surrounding store logs to continue the investigation.
## Read current pressure
`GET /api/status` is public and includes only the aggregate state:
```json
{
"backpressure": {
"state": "degraded",
"schedulers": [
{ "scheduler": "store", "state": "degraded" },
{ "scheduler": "sync-global", "state": "healthy" }
],
"diagnosticsAvailable": "/api/diagnostics/backpressure"
}
}
```
The detailed route requires the node-level admin token. Agent-scoped tokens are rejected because the response describes node-wide work.
```bash
TOKEN=$(dkg auth show)
curl -sS \
-H "Authorization: Bearer $TOKEN" \
http://127.0.0.1:9200/api/diagnostics/backpressure | jq
```
The response reports current queue/inflight counts and limits, oldest ages, cumulative lifecycle/rejection counts, and bounded operation summaries. It does not expose request bodies, SPARQL text, graph or peer identifiers, work closures, or durable queue payloads.
## Log behavior
The daemon samples registered sources every five seconds. It emits:
* a warning immediately when a lane enters or changes a non-healthy state;
* one warning summary per minute while that state persists;
* an info message when the lane recovers.
All messages start with `[backpressure]` and carry a JSON object:
```
[warn] [backpressure] {"event":"transition","scheduler":"store","lane":"normal","state":"degraded","previousState":"healthy","queued":3,"queueLimit":4,"inflight":4,"inflightLimit":4,"oldestQueuedAgeMs":15234,"oldestActiveAgeMs":19310,"rejectedTotal":0,"queuedOperations":[{"operation":"blazegraph.query","count":3,"oldestAgeMs":15234}],"activeOperations":[{"operation":"publisher.swm.graphScopedReplace","count":4,"oldestAgeMs":19310}]}
```
Per-item enqueue/start logs are deliberately avoided. Transition and periodic summary logging make sustained pressure visible without creating a log storm that competes with the overloaded scheduler.
## Metrics
The common OpenTelemetry instruments use bounded `scheduler` and `lane` attributes:
| Metric | Type | Purpose |
| --------------------------------------- | --------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `dkg.backpressure.queue_depth` | gauge | Current waiting work in this lane — the attribution signal |
| `dkg.backpressure.pressure_depth` | gauge | The depth this lane's state was classified against. **Divide this by `queue_limit` for utilization**, not `queue_depth`: on a `shared` lane the limit is the pool's, so pairing it with the lane's own backlog underreports. Equal to `queue_depth` on a `partitioned` lane |
| `dkg.backpressure.queue_limit` | gauge | Configured queue capacity |
| `dkg.backpressure.inflight` | gauge | Current admitted work in this lane — attribution |
| `dkg.backpressure.pressure_inflight` | gauge | The admitted count `inflight_limit` bounds — this lane's own under `partitioned`, the pool's under `shared` |
| `dkg.backpressure.inflight_limit` | gauge | Configured concurrency |
| `dkg.backpressure.oldest_queued_age_ms` | gauge | Head-of-line age |
| `dkg.backpressure.oldest_active_age_ms` | gauge | Oldest admitted duration |
| `dkg.backpressure.events_total` | counter | Lifecycle and rejection events |
| `dkg.backpressure.queue_wait_ms` | histogram | Completed queue waits |
| `dkg.backpressure.active_duration_ms` | histogram | Completed admitted durations |
Like the lane `queueLimit`/`inflightLimit` they pair with, `pressure_depth` and `pressure_inflight` **must never be summed across lanes** on a `shared` scheduler: every lane reports the same pool figure, and the `lane="all"` row carries the scheduler's own totals under the same instrument names, so a `sum()` without a lane filter double-counts on every scheduler.
Operation names are intentionally excluded from the common current-value gauges. They remain available in bounded diagnostic/log summaries, while metrics retain predictable cardinality.
## Failure containment
Instrumentation is fail-open:
* metric recording failures do not alter admission or completion;
* one broken source is reported in the registry's `failures` array without hiding healthy sources;
* log callback failures do not stop the monitor;
* the monitor timer is unreferenced and is stopped during daemon shutdown.
---
# Agent Instructions
This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com.
## Querying This Documentation
If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question.
Perform an HTTP GET request on the current page URL with the `ask` query parameter, and the optional `goal` query parameter:
```
GET https://docs.origintrail.io/use-dkg/backpressure-observability.md?ask=&goal=
```
`ask` is the immediate question: it should be specific, self-contained, and written in natural language.
`goal` is optional and describes the broader end goal you are ultimately trying to accomplish on behalf of the user. GitBook uses it to tailor the answer towards what is most useful for that goal.
The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.
Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.