> For the complete documentation index, see [llms.txt](https://docs.origintrail.io/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.origintrail.io/how-dkg-works/p2p-resilience.md). # P2P Resilience Status: implemented in the current V10 code. The remaining-work section tracks follow-ups that are not required for the shipped hooks. ## Context Two common DKG deployments sit behind home/office NAT: * Developer laptops running `dkg start` on a residential connection. * Hobby/demo nodes on cloud providers where the container is reachable from the public internet but the *application* doesn't know its public IP. These nodes depend on the testnet **circuit relay** peers to be dialable by anyone else. The relay path is fragile by construction: 1. A reservation must be held on the relay. Two distinct knobs from libp2p's `circuitRelayServer({ reservations: ... })`, both set explicitly in `packages/core/src/node.ts`: * `reservationTtl` (libp2p `ServerReservationStoreInit.reservationTtl`, `RELAY_RESERVATION_TTL_MS = 2h`) — how long a reservation lasts before the reservee must renew. This is the actual reservation expiry. * `defaultDurationLimit` (`ServerReservationStoreInit.defaultDurationLimit`, `RELAY_DEFAULT_DURATION_LIMIT_MS = 30 min`) — the maximum lifetime of a single relayed *circuit* (per-stream cap). Bumped from libp2p's effectively-5-minute default so chat-style intermittent traffic doesn't tear individual circuits down underneath the application during quiet windows. 2. The TCP connection to the relay must stay alive. 3. Both sides must have a relay in common with live reservations, or one side must be directly dialable. In practice we were seeing one or more of the following failure modes: * Relay reservations silently expiring even while the underlying TCP connection to the relay stayed healthy (so existing watchdog said "all good"). This made the node unreachable from the outside without anyone noticing. * A direct peer connection between two edge nodes surviving just long enough for the initial sync to fail, then dropping before any retry. * An edge node coming online and only syncing on its first successful `peer:connect`; if that attempt failed over a lossy relay, no further attempt was made until the daemon was restarted. ## Implemented hooks ### 1. Relay reservation renewal (primary fix) `packages/core/src/node.ts` — the `watchdogTick` now detects **lapsed reservations** in addition to **dropped TCP connections**. On each tick, the watchdog snapshots `libp2p.getMultiaddrs()` and treats the full set of circuit-relay reservations as a **pool**, not as per-relay slots. This matches how js-libp2p's circuit-relay-v2 transport behaves by default: it holds a single reservation at a time, rotating among the configured relays as needed. A per-relay "reservation missing" check would spuriously drop+redial idle relays forever; a pool-level check targets the condition that actually kills reachability. For every configured relay peer: * **Happy path.** TCP connection up AND (this relay holds our reservation OR *any* relay does) → nothing to do. * **Reservation pool empty.** TCP connection up but the set of `/p2p-circuit` self-addresses is empty → we are unreachable. The watchdog closes the existing connection and redials the relay; the circuit-relay transport re-listens on `/p2p-circuit`, which re-requests a fresh reservation. A per-relay grace window (`RELAY_RESERVATION_GRACE_MS`, 15 s) suppresses false-positive re-fires while a freshly-negotiated reservation is still on the wire. * **Transport down.** No TCP connection → classic redial path (pre-existing behaviour, unchanged). **Why this is the primary fix.** If our reservation on relay R has lapsed, no peer anywhere on the network can reach us via R until we re-reserve. No amount of application-level reconnection logic fixes this — the circuit simply won't open. Every other hook in this document is a complement that helps *after* the underlying reachability has been restored. ### 2. Reconnect-on-gossip `packages/agent/src/dkg-agent.ts` — `maybeDialGossipSender()`. When any gossipsub message is delivered to us, the sender peer is both a proof-of-life signal *and* a useful dial target: * GossipSub messages are signed by the original publisher, so `from` is the author regardless of how many mesh hops the message took. * If we're not currently connected to that peer, we best-effort dial them: first via peerStore (which may already know their multiaddrs from a previous direct exchange), then by constructing explicit `/p2p-circuit` multiaddrs through each configured relay. * A 30-second per-peer cooldown (`GOSSIP_DIAL_COOLDOWN_MS`) prevents dial storms from chatty topics. This catches the case where two edge nodes briefly lose their direct circuit without either noticing, but gossipsub still routes their messages to each other via the mesh. The arriving message becomes the trigger to rebuild the direct link so subsequent sync requests have a path. ### 3. Catchup-on-connection:open `packages/agent/src/dkg-agent.ts` — the agent now listens on `connection:open` in addition to `peer:connect`. `peer:connect` fires once, the first time libp2p sees a peer. If that peer temporarily disconnects and later reconnects, `peer:connect` does **not** fire again. Without this second listener, a first-contact sync that failed over a flaky relay would never be retried. A 60-second per-peer cooldown (`CATCHUP_ON_CONNECT_COOLDOWN_MS`) dedupes overlapping direct + relayed connections for the same peer (each of which fires its own `connection:open`). ## Remaining work These are known follow-ups that are not required for the current shipped hooks. ### DCUtR upgrade verification `dcutr()` is enabled in the libp2p service stack, which should attempt to upgrade relayed connections to direct NAT-holepunched ones. We do track upgrades via `setupConnectionObservability` but we don't *verify* they actually happen in the wild for our test peers. Need: * A metric/log line per day counting how many relayed connections successfully upgrade to direct vs. stay relayed. * An alert when that ratio trends to zero (indicates DCUtR is failing silently — e.g., because a firewall blocks the holepunch packets). ### Multi-relay fanout for reservations Currently each node tries to hold a reservation on every configured relay in `network.relays`. That's fine for a 4-relay testnet but doesn't scale to a larger relay set. We should: * Hold reservations on *N* diverse relays (e.g., 2–3), picked by network latency / geographic diversity rather than config order. * Rotate which relays we hold reservations on if a subset becomes unhealthy. ### Rendezvous / peer exchange discovery Today, discovering a peer for the first time requires either: * They were in our `bootstrapPeers` (unlikely for edge peers), or * We learned their addresses from on-chain agent profile publication (which only works if they've published a profile and we've synced the agents CG). For ephemeral CGs like local tic-tac-toe games the second path is overkill. A libp2p rendezvous service on the relays would let peers announce "I'm in context graph X" and let other members discover them directly. Worth prototyping once v2 of the demo flow is stable. ### Persistent per-peer routing hints When `maybeDialGossipSender` successfully dials a peer via a specific relay, we should persist that relay choice in peerStore. Right now each gossip event re-runs the relay fanout from scratch. A simple `tag.value` update per successful dial would let libp2p prefer known-good relays on subsequent dials. ### Public dial-back / reachability signalling An edge node can currently only be reached *through* a relay. Even when AutoNAT knows the node is reachable (e.g., a cloud VM with a mapped port), the node still treats itself as unreachable and hides behind the relay. We should emit an `announce`-multiaddr when AutoNAT confirms a direct address works, so other nodes can skip the relay entirely. ### Smoke test for reservation renewal The reservation-renewal watchdog doesn't have a unit test because simulating a lapsed reservation cleanly requires a real circuit-relay server we can instruct to expire a reservation. An e2e smoke test using two `DKGNode` instances + one relay instance would close this gap. ## Config knobs All timing constants are in the source files above. Defaults: | Knob | Default | Where | | --------------------------------- | ------- | --------------------------------- | | `RELAY_WATCHDOG_BASE_INTERVAL_MS` | 10 s | `packages/core/src/node.ts` | | `RELAY_WATCHDOG_MAX_INTERVAL_MS` | 5 min | `packages/core/src/node.ts` | | `RELAY_RESERVATION_GRACE_MS` | 15 s | `packages/core/src/node.ts` | | `GOSSIP_DIAL_COOLDOWN_MS` | 30 s | `packages/agent/src/dkg-agent.ts` | | `GOSSIP_DIAL_TIMEOUT_MS` | 10 s | `packages/agent/src/dkg-agent.ts` | | `CATCHUP_ON_CONNECT_COOLDOWN_MS` | 60 s | `packages/agent/src/dkg-agent.ts` | ## Testing Unit tests for the two application-level hooks live in `packages/agent/test/p2p-resilience.test.ts`: * Reconnect-on-gossip dials on first message from an unconnected peer * Reconnect-on-gossip skips peers already connected * Reconnect-on-gossip ignores messages from our own peer id * Reconnect-on-gossip throttles repeat attempts within the cooldown * Catchup-on-connection:open fires `trySyncFromPeer` on each new conn * Catchup-on-connection:open dedupes within cooldown * Catchup-on-connection:open ignores our own peer id The relay reservation renewal path is verified end-to-end only (two laptops behind NAT, testnet relays) — see the "Smoke test for reservation renewal" remaining-work item. --- # Agent Instructions This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com. ## Querying This Documentation If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question. Perform an HTTP GET request on the current page URL with the `ask` query parameter, and the optional `goal` query parameter: ``` GET https://docs.origintrail.io/how-dkg-works/p2p-resilience.md?ask=&goal= ``` `ask` is the immediate question: it should be specific, self-contained, and written in natural language. `goal` is optional and describes the broader end goal you are ultimately trying to accomplish on behalf of the user. GitBook uses it to tailor the answer towards what is most useful for that goal. The response will contain a direct answer to the question and relevant excerpts and sources from the documentation. Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.