Skip to content
For the complete documentation index, see llms.txt. This page is also available as Markdown.

Async Publisher Wallets

Async publishing moves Knowledge Assets from Shared Working Memory to Verifiable Memory in the background. Each publish is a job in the node's publish queue. Publisher wallets sign the jobs' on-chain transactions, and each wallet runs one publisher worker.

The wallets live in publisher-wallets.json under the DKG data directory. Manage them with:

dkg publisher wallet add <privateKey>
dkg publisher wallet list
dkg publisher wallet remove <address>
dkg publisher enable
dkg publisher publish-async <context-graph-id> <name>

Queue settings are on Async Queue Operations. What each answer means for a client is on Async Share and Publish Jobs.

Receipt, finality, and local completion

A job enters broadcast when its signed transaction is durably recorded before send. Receipt confirmation can trigger reconciliation while the executor finishes local work. The reconciler independently checks the transaction at the configured confirmation depth and binds canonical publish evidence to that job before writing included and releasing its wallet. finalized also requires the executor to settle, local lifecycle repair/private staging to complete, and the final job record to be persisted. An included job can therefore have already reached chain finality while local work remains.

GET /api/publisher/job?id=<id> and GET /api/publisher/jobs expose timestamps.receiptObservedAt and timestamps.finalityObservedAt (Unix milliseconds). The first records the receipt-confirmation callback, or the inclusion itself when nothing was observed earlier (it then equals includedAt). The second is recorded only when a reconciliation or recovery pass proves the transaction at the configured confirmation depth. A job whose executor completes before any such pass has no finalityObservedAt: reaching finalized is local completion, and it is not recorded as a chain observation. Both are distinct from includedAt and finalizedAt, which describe job transitions. Receipt observations are diagnostics and never release a wallet or authorize a publish. Observations are matched to the job's transaction hash. A daemon that runs the publisher shows them on these endpoints as soon as they are made, and the next state transition persists them. A restart before that transition loses the observation, and a job view read by another process shows it only once it is persisted. Older jobs and local-only finalizations omit both fields.

The default scheduler has four total slots (capped by available logical CPUs), with one slot each reserved for ACK and health work. This leaves two shared normal/background slots on a four-CPU host. Background work protects one normal slot and uses at most one slot by default. These are intentional backend admission limits, rather than a count of publisher wallets; six wallets do not imply six concurrent store operations. DKG_STORE_MAX_CONCURRENT and the lane reservation settings control these limits. Post-confirmation store work uses this scheduler, and the daemon's main event loop also gates the completion of job-state reads and writes.

Pause and resume the running publisher

Node administrators can pause all new claims, including jobs already accepted in the queue:

curl -H "Authorization: Bearer $DKG_API_TOKEN" -X POST -d '{}' http://127.0.0.1:9200/api/publisher/pause
curl -H "Authorization: Bearer $DKG_API_TOKEN" http://127.0.0.1:9200/api/publisher/control
curl -H "Authorization: Bearer $DKG_API_TOKEN" -X POST -d '{}' http://127.0.0.1:9200/api/publisher/resume

Use the daemon's configured API port. Agent-scoped tokens cannot call these node-wide controls. Responses contain desiredState, effectiveState, generation, acknowledgedGeneration, updatedAt, and each wallet's claim gate and generation. A successful pause waits for claims already inside the claim transaction to finish and fences every wallet before acknowledging. A job claimed before that boundary may continue execution, broadcasting, receipt handling, and finalization. Recovery remains active while paused; pause does not cancel or clear job evidence, and admission can continue accepting jobs.

The live runtime persists desired state in publisher-control.json in its data directory, and restores it before startup recovery or wallet loops begin. Repeating the current desired state is idempotent; successful transitions increment the durable generation. Resume persists intent before opening gates. Shutdown closes gates before draining a concurrent control request, so a persisted resume may take effect on the next startup while the closing runtime remains fenced. DKG_PUBLISHER_START_PAUSED=1 overrides persisted running intent at startup; a subsequent live resume can reopen the runtime.

A missing runtime returns HTTP 503. A persistence/control failure also returns 503, fences the runtime, and requires restart to reload durable intent; a failed request has no acknowledgement. A status read during a transition may show the previous acknowledged desired state alongside gates already closing. File contents are synced before atomic rename, and the parent directory is synced on supported POSIX hosts. Windows preserves process-restart state without claiming the same directory-fsync power-loss guarantee.

Admission retries

All publisher wallets share the daemon's RPC request budget. For rate sizing, the restart requirement and diagnosing local wait versus provider latency, see RPC Request Budget.

Before vm/publish-async creates a job, it checks the shared snapshot. Store pressure gets a retryable 503. Resend the same request, without sharing again, only if it says jobCreated: false; otherwise a job may exist, so find it first. For 409 PUBLISH_INTENT_STALE, see Stale update intents.

Clearing finished jobs

Finished jobs stay until they are cleared. dkg publisher clear finalized|failed, or POST /api/publisher/clear with the node-admin token, clears them in bulk and returns the count removed.

A re-submit of a Knowledge Asset is matched to its newest job, by context graph, name, sub-graph and agent lane. A bulk clear never lets an older failed job become the newest again. So it keeps a finalized job while an older failed job of the same asset remains. It also never removes a held job. It can remove fewer jobs than are listed, and it is slow on a large queue.

dkg publisher clear-job <job-id> removes one finished job, even the newest, through the running daemon (API: POST /api/publisher/clear-job { "jobId": "…" }). It never opens the store offline; cleared and already_absent exit 0, while a refusal exits 1 with its reason. After you clear the newest job, a re-submit can restart an older failed job of the same asset. A held job is removed only with --allow-pending-transaction (API: "allowPendingTransaction": true): check its transaction first, or a re-submit can publish again.

If the daemon does not answer, dkg publisher cancel, retry and clear refuse to write the store directly while the node home's daemon.pid names a running process. They print Refusing to run "dkg publisher <command>" offline and exit 1. Try again once the daemon answers, or run dkg stop first. jobs, job and stats refuse in the same way; with no daemon running they open the store read-only.

Old finalized jobs can also be removed on a timer. This cleanup is off by default; see Async Queue Operations.

Stale update intents

An update of an already-published KA can only be published as one above the KA's latest confirmed version; the publisher, the ACK quorum, peers and the chain all require exactly that number, and wm/finalize returns the number it sealed the draft with (assertionVersion). A draft finalized under any other number — sealed by an older node, or because the published version moved on meanwhile (for example another update of the same KA was still being published when the draft was finalized and landed first) — cannot be published, so the node refuses it before anything is staged or sent. At vm/publish-async admission nothing is created at all: 409 PUBLISH_INTENT_STALE with jobCreated: false; vm/publish answers the same 409. A job that was already queued passes the preflight (which does not compare the number), signs its local update attestation, and then fails terminally as publish_intent_stale from validated when update() refuses it (never the retryable rpc_unavailable). The message names the sealed number, the required number and the confirmed version.

Recovery, per KA:

  1. If you sent this publish before, read that job first (Retry safely). If it is finalized and carries your seal, the update is published: stop. If it is still running or held, wait until it finalizes or fails and is no longer held, then check again.

  2. If a failed job still holds the KA (a corrected publish gets 409 with its existingJobId), read it with GET /api/publisher/job?id=<id>. A retryState.waitingReason of pending_chain_proof or recovery means it may have sent a transaction, perhaps the one that moved the version on: follow Held jobs and do not clear it. Otherwise clear it by id: dkg publisher clear-job <job-id>.

  3. POST /api/knowledge-assets/{name}/wm/pull-from { "layer": "swm" } re-opens the shared content as a draft (add "onConflict": "replace" if a WM draft already exists). If the message says the published version moved on (the sealed number is below the required one), use "layer": "vm" instead and re-apply your edits.

  4. wm/finalize — check that the returned assertionVersion equals the number the message asked for.

  5. swm/share, then vm/publish-async (or vm/publish).

A core that already ACK-signed the abandoned draft declines its replacement transiently until the pending window ends (DKG_STORAGE_ACK_PENDING_TX_WINDOW_MS, 5 minutes by default), so retry the publish after that. Peers that already hold an earlier shared draft of the KA may keep showing it in Shared Memory until the update is published: a re-share carrying the same or a lower number than a draft a peer already holds is not applied there, and with swmAwaitCuratorAck on a curator that holds the earlier draft rejects the re-share before it commits. Finalizing a replacement draft also replaces the private payload sealed under that number, so an earlier shared draft that carried private content can no longer be re-opened from Shared Memory once its replacement is finalized.

Held jobs

A job is held when it failed after a transaction may have been sent (it persisted a transaction hash, or failed from included) and nothing has accounted for that transaction yet. "Held" means the publication outcome is not yet known — not that the chain can never answer. A held job is never retried, keeps its Knowledge Asset's lifecycle, and survives bulk cleanup, because running it again could publish the same asset twice.

By contrast, a transient RPC failure raised while the publish transaction was still being prepared (every endpoint failed during estimate/sign, the local request governor was full, or a bounded read timed out) is not held: nothing was signed or sent for the publication, so the node records it as a retryable failure and re-runs the same job on its bounded backoff, re-validating authority and intent each time. A TRAC approval or a context-graph registration may already have been sent by that attempt; neither is the publication. The identical error after the pre-send write-ahead recorded a transaction hash — a broadcast that exhausted every endpoint, a receipt wait that timed out — names a transaction and is held.

What recovery is waiting for

GET /api/publisher/job?id=<id> and dkg publisher job <job-id> return retryState. For a job that is not moving it carries blocker: { code, summary, missing? }: derived on read, additive, fixed text (never an RPC URL, hash or payload). The set of codes is open; read summary for any you do not know. retryState is derived by the running daemon; with the daemon stopped, dkg publisher job prints the job alone. The 503 LIFT_JOB_PENDING_CHAIN_PROOF returned to a re-submit carries the same blocker.

blocker.code

What it says

Can an operator supply it?

chain_recheck_pending

The record is complete and this node can settle it. While the publisher runtime is running and not paused, recovery re-checks on a bounded backoff, finalizes the same job if the transaction mined, and releases a CREATE for a re-run only once the transaction is proven never sent (an UPDATE whose transaction never landed stays held). What the latest re-check found is reported as blocker.lastCheck when the running publisher holds an observation of it (below); without one it is not named.

Nothing to supply. Wait; restore RPC access if the provider is the problem.

recovery_not_configured

The record is complete, but this node has no chain-recovery capability for the job's wallet and operation (no resolver, the chain adapter cannot answer, or the publisher runtime is not running).

Restore chain access, the adapter capability or the runtime. A retry is refused while a transaction may exist.

no_transaction_hash, no_signer_wallet

Nothing can be asked about the job at all.

No supported control writes evidence into a job. The best-effort journal (GET /api/publisher/journal, named-KA jobs on the daemon) may still hold the hash; inspect the transaction on a block explorer.

claim_or_validation_missing, publish_identity_unpinned

The chain can be asked, but a mined transaction cannot be turned into this job's finalization or recognized as its effect.

No. Inspect the transaction yourself.

operation_unmarked

The record does not say whether the signed transaction was a create or an update, so no operation-specific proof applies and none is guessed.

No.

nonce_missing (CREATE)

Recovery still finalizes the job if the transaction mined, but can never prove it unsent, so it can never release it for a re-run.

No.

intended_root_missing (UPDATE)

Recovery cannot recognize the effect. Absence is never proof for an UPDATE: a later third-party update would make a replay write a stale root over newer state, so recovery never releases one.

No.

blocker.missing lists every gap of an incomplete record, so you are not sent to fix one and then meet the next.

What the latest chain re-check found (blocker.lastCheck)

For a chain_recheck_pending job, the job-detail routes (GET /api/publisher/job, job-payload and the legacy jobs/<id> shapes) add blocker.lastCheck: { outcome, at } when the running publisher holds an observation of the latest re-check of that exact job. at is epoch milliseconds on the daemon's clock, taken when the re-check finished. outcome is a closed set of codes (never provider text, which can carry RPC URLs or keys):

outcome

What the latest re-check found

pending-mempool

The chain has the transaction in its mempool.

pending-awaiting-confirmation

The transaction is mined but not yet confirmed to the configured depth.

rpc-unavailable

The chain RPC could not answer: every endpoint failed, a bounded request timed out, or the local request governor was full. It does not by itself mean the provider is down.

absence-unproven

The chain has no record of the transaction, but the proof that would let this node release the job is not established, so nothing is released. For an UPDATE absence is never proof; for a CREATE it is any of: the signed nonce is not provably spent, the pinned identity is already minted (a replacement transaction may have published) or its state could not be read, no identity is pinned, or the pinned snapshot could not be read. Do not assume it is only an RPC problem.

inconclusive

Nothing was established and the cause is not classified (an adapter that cannot answer, a confirmation this node cannot map to evidence, another failure). It never means the provider is fine.

unrecognized

A mined CREATE transaction carries no publish this node can parse (an UPDATE is verified against its intended root instead).

recovered

The chain confirmed the transaction but this node did not apply it (yet): finalization declined, or the pass ran out of time between the answer and applying it.

reverted

A job holding an earlier attempt's hash on an UPDATE: a revert proves that transaction had no effect, but an UPDATE is never re-run from it (a replay could write a stale root over newer state), so it stays held.

not-found

Only a third-party resolver reports it for an UPDATE (the built-in resolver reports absence-unproven instead); an UPDATE is never released by absence.

deadline

The pass's time budget ended before the lookup answered.

error

The re-check threw: the lookup, the claim transaction or applying its answer.

It is observability only and changes nothing about when a job is re-checked or what is done with the answer. It lives in the running daemon's memory, so there is none before the first re-check and after a restart; a paused dispatcher (DKG_PUBLISHER_START_PAUSED), a pass that did not reach the job (each pass asks at most 25 jobs within 15 seconds) and the idle 60-second cadence all leave an older one in place, so read at against the clock. The 503 LIFT_JOB_PENDING_CHAIN_PROOF body carries the same blocker without lastCheck.

Retryable jobs that carry no transaction evidence get a blocker too. not_auto_retryable: the failure is not one the publisher retries by itself, and POST /api/publisher/retry or re-submitting the identical request re-runs it as the same job. auto_retry_disabled and retry_not_scheduled: a retry that was already scheduled fires again once autoRetryEnabled is switched back on, but a job recorded while automatic retry was off was never scheduled and is not released by switching it on, so re-run it by hand. retry_budget_spent: re-submitting the identical vm/publish-async request re-arms a full budget on the same job. While the daemon reports the publisher runtime unavailable, waitingReason reads operator and no blocker is added: GET /api/status names that reason.

Operator decision path for a held job

  1. Inspect the exact job with dkg publisher job <job-id>: the retained transaction hash (broadcast.txHash, or recovery.txHashChecked after a reset), the signing wallet, the nonce, the operation kind and retryState.blocker.

  2. Tell the three problems apart. A provider or access problem leaves the record complete (chain_recheck_pending): recovery keeps asking and resumes by itself once the lookup works. Missing local evidence shows up as the missing gaps above. An operation-specific limit (an UPDATE has no absence proof; a CREATE needs its nonce and pinned asset identity) is why a complete-looking record can still be held.

  3. Restore the lookup or the capability, or inspect the transaction yourself, then request only the control that applies. POST /api/publisher/retry and a re-submit refuse a held job by design; recovery is the only automatic exit.

  4. If no adequate proof can be obtained, an operator cannot manufacture a safe retry either. Keep the hold and its evidence. Elapsed time is not proof of non-effect, and neither is a missing receipt; escalating or deliberately abandoning the job is a decision, not a proof.

  5. What clearing does. dkg publisher clear-job <id> --allow-pending-transaction (API: POST /api/publisher/clear-job { "jobId": "<id>", "allowPendingTransaction": true }) removes that one record. A held job needs the flag (node-operator token, or the agent that enqueued the job), as does a job recovery owns or one still in flight. Without it the clear answers 409 with reason: "pending_transaction" for a held job or one recovery owns, and "nonterminal" for one still in flight. Clearing abandons this node's tracking of the job. It does not cancel a transaction that was already sent, does not prove that none was, and does not authorize a duplicate-safe replacement — a re-submit afterwards mints a new job for the Knowledge Asset, which may publish it a second time. A clear never touches the journal, so whatever it recorded about the job (best-effort, named-KA jobs) stays readable. POST /api/publisher/cancel removes only a job that is still accepted and cannot act on a held one.

Publication latency and store pressure

Measure queueing separately from chain time using the job's timestamps: acceptedAt to claimedAt is admission wait; rpcAcceptedAt to receiptObservedAt is receipt wait; finalityObservedAt to finalizedAt is the remaining local completion interval. Missing timestamps mean that stage was not observed, rather than a zero-duration stage.

Wallet count alone does not determine throughput. Claims serialize durable job and wallet ownership writes, and those writes share store capacity with other node work. Durable admission and resume wake parked wallet loops; repeated notifications coalesce, and ownership checks and error backoff still apply. The polling interval remains a fallback, rather than a required delay for new work. Idle wallets are offered new work least recently used first: a wake reaches them in that order, and their polls fall in one shared window in the same order, so sparse work rotates across the wallets instead of always going to the first one listed. A busy wallet is never waited for. Automatic retry selection uses the persisted retry deadline index, and recovery inventories exclude accepted and finalized history. Recovery still retains every failed job needed to preserve its proof backoff.

A detached named-KA executor can finish its own queue record after confirmed chain execution and successful local finalization. Failed local bookkeeping, incomplete results, or lost claim ownership retain the existing recovery path. Private catalog placement remains asynchronous after its durable repair marker is stored; publication does not wait for placement.

The five-minute async_admission_usage log line shows publisher pick-up, claim-to-finish time and the accepted backlog; Monitor Store and Queue Pressure says which fields to read.

Use GET /api/diagnostics/backpressure to inspect store lane occupancy. The dkg.store.scheduler_queue_wait_ms and dkg.store.scheduler_operation_duration_ms histograms separate admission wait from admitted execution time by priority and static operation label. The duration histogram also labels outcome as ok or error; its sample count shows completed operation volume. Increasing store concurrency is not a substitute for diagnosing the operations occupying the normal lane.

For a focused investigation, start the daemon with DKG_ASYNC_PUBLISH_TRACE=1. The opt-in async_publish_claim_trace log records monotonic claim-lock wait/hold boundaries, native wallet-ownership decisions, durable lock changes and worker outcomes. It adds no store reads and is disabled by default. Wallet identifiers are hashed; job IDs are retained for correlation, so treat these logs as operational diagnostics. A free wallet can still wait for the claim lock or store capacity, and a worker may still be finishing an earlier execution after its wallet is released. Do not infer wallet availability from job finalization.

Funding

Every async publisher wallet needs the chain's native gas token because it submits on-chain transactions.

For TRAC payment, choose one path:

Payment path
Wallet requirement

PCA-funded

Register the wallet address as a Publishing Conviction Account agent.

Direct spend

Fund the wallet with enough TRAC and native gas.

If neither path is ready, the daemon can still start, but publish attempts fail when the transaction reaches the chain.

A refusal for funds (insufficient_funds) happens before the transaction enters the mempool, so it proves the transaction never took effect. The publisher therefore retries the job by itself on its bounded retry backoff, and keeps the refused wallet out of claims meanwhile:

  • The wallet waits 30 seconds, doubling up to 10 minutes while it stays short. After each wait the publisher reads the wallet's balance. The wallet claims again once the balance covers the cost the chain stated for the refused transaction or, when the chain stated none, once its native balance, its TRAC balance or what its Publishing Conviction Account can still cover has grown since the refusal. Registering the wallet as an agent of a funded account counts, with no transfer to the wallet. A balance the publisher cannot read never keeps a wallet out.

  • Only the refused wallet is kept out. Publication costs vary, so a wallet holding less than the stated cost still claims, and may pay for a cheaper job; if it cannot, the chain refuses it too and it is kept out the same way.

  • While no wallet can pay, jobs wait as accepted instead of failing. Top up a wallet and they continue.

  • A refused wallet that is not topped up gets one more attempt after an hour, since a stated cost goes stale when gas prices fall.

The exclusion is kept in memory, so after a restart a wallet that is still short is refused once more before it is excluded again. A job that may have sent a transaction is not affected: it stays held until chain proof settles it. A refusal settles only the refused transaction. It does not release a job while an earlier transaction of that job is still unsettled, or when another endpoint may have accepted the same transaction and lost its answer. A job that failed with insufficient_funds before this behaviour existed stays failed; re-submitting its request creates a new job.

Identity And Attribution

An async publisher wallet does not need an on-chain node identity or profile to claim jobs.

The daemon still reads each wallet's identity at startup:

Resolved identity
Behavior

0

Publish in no-attribution mode.

N > 0

Use N as publisher-node attribution by default.

publisherNodeIdentityId is attribution metadata. It is not PCA eligibility, direct-spend eligibility, author identity, or ACK quorum identity.

Use --publisher-node-identity-id 0 when you want explicit no-attribution for one publish:

Use a non-zero override only when an operator intentionally wants the publish to carry a specific publisher-node attribution claim.

Optional Node Attribution Setup

Core node profile setup creates an identity for the node's primary operational wallet. Separate async publisher wallets are not automatically attached to that identity.

To make a separate publisher wallet resolve to a non-zero node identity, authorize it as an operational wallet for that node identity with the current POST /api/operational-wallets API route or an equivalent future CLI wrapper. If you do not need publisher-node attribution, leave the wallet identityless.

ACK quorum still depends on core receiver identities and operational keys. Changing async publisher wallet attribution does not relax ACK requirements.

Last updated

Was this helpful?