> For the complete documentation index, see [llms.txt](https://docs.origintrail.io/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.origintrail.io/connect-agents-and-systems/local-llm.md).

# Run a Local LLM

Use this guide to connect a local model to the DKG MCP tools through an OpenAI-compatible `llama.cpp` or Ollama server. The model does not connect to the DKG daemon directly: `dkg llm` starts an MCP client, discovers the available tools with `tools/list`, validates every tool call, and writes a readable interaction trace.

The runtime is read-only by default. Write tools are exposed only with `--allow-write`, and the user must still explicitly request the mutation.

If you are delegating setup to a coding agent, give it the dedicated [`Local LLM Agent Runbook`](https://docs.origintrail.io/for-agents/local-llm-agent-runbook.md). It contains a copy-paste instruction, model decision table, Query Catalog gate, exact launch commands, and completion checks.

## Supported local model servers

Use [`llama.cpp`](https://github.com/ggml-org/llama.cpp) and its `llama-server` executable as the default, documented local inference server for DKG. [`Ollama`](https://docs.ollama.com/) is also supported through the same OpenAI-compatible API boundary. Choose one server per DKG local-LLM session; adding Ollama support does not remove or replace llama.cpp support.

This selects the inference runtime, not the model family: `llama-server` can serve Qwen, Bonsai, Llama, and other compatible GGUF models. Qwen3-8B Q4\_K\_M remains the recommended model below.

The boundary should remain explicit:

* The local model server is operator-managed. DKG does not silently install, upgrade, start, or stop llama.cpp or Ollama.
* `dkg llm` owns the MCP client, tool discovery, metadata-driven relevance routing, DKG system context, schema validation, retry policy, bounded chat history, and text trace.
* Do not bypass that harness by attaching the DKG MCP server directly through a model server's own tool or MCP configuration. That would skip DKG's tested system context, tool-budget routing, validation, retry, and readable interaction log.
* The default endpoint is `http://127.0.0.1:8080/v1/chat/completions`; `--llama-url` and `DKG_LLM_URL` remain escape hatches for another compatible server.
* A supported server must provide `POST /v1/chat/completions`, non-streaming OpenAI-compatible responses, and OpenAI-style tool calls. `GET /v1/models` is the provider-neutral readiness contract; llama.cpp's `GET /health` remains a compatibility fallback if `/v1/models` is unavailable.
* Keep the server bound to `127.0.0.1` by default. Remote or LAN exposure needs an explicit authentication and network-security decision.

This gives DKG a reproducible llama.cpp reference path across macOS, Linux, and Windows while keeping the client provider-neutral. Readiness succeeds through `/v1/models` on either backend, or through llama.cpp's `/health` fallback.

## Install llama.cpp and llama-server

Prefer an operating-system package for normal use. Build from source when a specific acceleration backend or an unreleased upstream fix is required. The canonical upstream references are the [`llama.cpp` installation guide](https://github.com/ggml-org/llama.cpp/blob/master/docs/install.md), [`build guide`](https://github.com/ggml-org/llama.cpp/blob/master/docs/build.md), and [`llama-server` guide](https://github.com/ggml-org/llama.cpp/blob/master/tools/server/README.md).

### macOS

Homebrew is the recommended installation:

```bash
brew install llama.cpp
command -v llama-server
llama-server --version
```

The Homebrew formula supports both Apple Silicon and Intel macOS. Apple Metal acceleration is enabled by default in upstream macOS builds.

To build the server from source instead:

```bash
xcode-select --install
brew install cmake git
git clone https://github.com/ggml-org/llama.cpp.git
cd llama.cpp
cmake -S . -B build -DCMAKE_BUILD_TYPE=Release
cmake --build build --config Release --target llama-server --parallel
./build/bin/llama-server --version
```

Use the resulting absolute path, normally `<llama.cpp>/build/bin/llama-server`, in the launch commands below.

### Linux

If Homebrew is already installed, the shortest supported path is:

```bash
brew install llama.cpp
command -v llama-server
llama-server --version
```

Conda is an official cross-platform alternative:

```bash
conda install -c conda-forge llama.cpp
command -v llama-server
llama-server --version
```

For an Ubuntu/Debian CPU build from source:

```bash
sudo apt-get update
sudo apt-get install -y build-essential cmake git
git clone https://github.com/ggml-org/llama.cpp.git
cd llama.cpp
cmake -S . -B build -DCMAKE_BUILD_TYPE=Release
cmake --build build --config Release --target llama-server --parallel
./build/bin/llama-server --version
```

For NVIDIA acceleration, install a compatible NVIDIA driver and CUDA toolkit, then configure the same source checkout with the CUDA backend:

```bash
cmake -S . -B build -DGGML_CUDA=ON -DCMAKE_BUILD_TYPE=Release
cmake --build build --config Release --target llama-server --parallel
./build/bin/llama-server --version
```

Use upstream prebuilt release assets or the backend-specific build guide for Vulkan, ROCm/HIP, SYCL, or OpenVINO rather than guessing backend flags.

### Windows

Open PowerShell and install the upstream Winget package:

```powershell
winget install llama.cpp
```

Open a new PowerShell after installation, then verify the executable:

```powershell
Get-Command llama-server.exe
llama-server.exe --version
```

For a source build, install Visual Studio 2022 with the **Desktop development with C++** workload, including CMake and Git. In a **Developer PowerShell for VS 2022** run:

```powershell
git clone https://github.com/ggml-org/llama.cpp.git
Set-Location llama.cpp
cmake -S . -B build
cmake --build build --config Release --target llama-server
& .\build\bin\Release\llama-server.exe --version
```

If the selected CMake generator places the executable directly under `build\bin`, use that discovered path. For NVIDIA acceleration, prefer the matching CUDA prebuilt asset from the official [`llama.cpp` releases](https://github.com/ggml-org/llama.cpp/releases), or build with `-DGGML_CUDA=ON` after installing the CUDA toolkit.

### Optional portable Docker server

On a machine with Docker, the official CPU server image avoids a host build:

```bash
docker run --rm \
  -p 127.0.0.1:8080:8080 \
  ghcr.io/ggml-org/llama.cpp:server \
  -hf Qwen/Qwen3-8B-GGUF:Q4_K_M \
  -c 8192 \
  --jinja \
  --host 0.0.0.0 \
  --port 8080
```

Use the upstream `server-cuda` image and `--gpus all` only on a correctly configured NVIDIA Docker host.

### Optional Hugging Face CLI

Public models used with `llama-server -hf` do not require a separate `hf` download step. Install the current `hf` CLI when a model is gated, private, or must be resolved to an explicit local GGUF path.

macOS and Linux:

```bash
curl -LsSf https://hf.co/cli/install.sh | bash
hf auth login
hf auth whoami
```

Windows PowerShell:

```powershell
powershell -ExecutionPolicy ByPass -c "irm https://hf.co/cli/install.ps1 | iex"
hf auth login
hf auth whoami
```

Do not put a Hugging Face token in a command line, repository file, or DKG interaction log. Use `hf auth login` or the `HF_TOKEN` environment variable.

### Verify the llama.cpp server contract

Start one of the models below, wait for loading to finish, and verify both readiness and the OpenAI-compatible surface:

```bash
curl -sS http://127.0.0.1:8080/health
curl -sS http://127.0.0.1:8080/v1/models
```

Windows PowerShell can use `curl.exe` with the same URLs. Do not start `dkg llm` until `/v1/models` returns HTTP 200. `/health` should also return HTTP 200 and `{"status":"ok"}` for the reference llama.cpp server.

## Install and run Ollama

Install Ollama using its official [`Quickstart`](https://docs.ollama.com/quickstart), then pull and start a model:

```bash
ollama --version
ollama pull qwen3:8b
OLLAMA_CONTEXT_LENGTH=8192 ollama serve
```

The desktop application may already have the server running. If `ollama serve` reports that port `11434` is already in use, keep the existing Ollama server and do not start a second one. DKG's prompt, history, and tool budget require an Ollama context of at least 8192 tokens. For the desktop application, set the **Context length** slider in Settings to at least `8192` and restart the app. For a Linux system service, run `sudo systemctl edit ollama`, add this override, then reload and restart the service:

```ini
[Service]
Environment="OLLAMA_CONTEXT_LENGTH=8192"
```

```bash
sudo systemctl daemon-reload
sudo systemctl restart ollama
```

The commands follow Ollama's official [`Context length`](https://docs.ollama.com/context-length) and [`server configuration`](https://docs.ollama.com/faq#how-do-i-configure-ollama-server) guidance. Load the selected model, then verify both its allocation and the shared OpenAI-compatible contract:

```bash
ollama run qwen3:8b "Reply with OK."
ollama ps
curl -sS http://127.0.0.1:11434/v1/models
```

Require `ollama ps` to report `CONTEXT` of at least `8192` for `qwen3:8b`. `/v1/models` returning HTTP 200 is not proof of the allocated context length, and its `data` array must list the configured `qwen3:8b` model (the implicit `qwen3:latest` alias is also accepted). A different non-empty model list is not ready for this configuration.

Use these DKG settings for Ollama:

```bash
export DKG_LLM_URL=http://127.0.0.1:11434/v1/chat/completions
export DKG_LLM_MODEL=qwen3:8b
export DKG_LLM_BACKEND=ollama
```

Ollama documents its OpenAI-compatible chat-completions, model-list, streaming, and tools support in the [`OpenAI compatibility guide`](https://docs.ollama.com/api/openai-compatibility). DKG uses non-streaming requests. Some Ollama versions or models may return prose even when a tool is required; the DKG runtime retries once with an explicit tool-only instruction and then fails closed instead of accepting an ungrounded DKG answer.

## Recommended model

Use **Qwen3-8B Q4\_K\_M** as the default local model. It provided the strongest tool-calling result in the real-DKG benchmark on the reference 16 GB Apple Silicon machine.

| Model and quantization | Real-DKG score | Relative resource use                       | Recommendation                                                                  |
| ---------------------- | -------------: | ------------------------------------------- | ------------------------------------------------------------------------------- |
| Qwen3-8B Q4\_K\_M      |          13/13 | Medium                                      | Recommended default for chat, catalog use, reads, and bounded writes            |
| Bonsai-8B Q1\_0        |           8/13 | Low                                         | Use only with a pre-built Query Catalog and a narrowly routed tool set          |
| Qwen3.8-27B UD-IQ1\_M  |           9/13 | High memory pressure and very slow on 16 GB | Not recommended on 16 GB; the 1-bit quantization did not outperform Qwen3-8B Q4 |

These scores compare the same 13 scenarios against a real DKG daemon and MCP server using the documented llama.cpp reference path. They are not Ollama parity claims or universal hardware results. A new model and server combination should pass the bundled real-DKG benchmark before it becomes a recommended default.

## Small-model rule: build the Query Catalog first

For Q1, aggressively quantized, or otherwise tool-weak models, prepare the domain Query Catalog **before** starting the end-user chat.

Do not rely on a small model to design arbitrary SPARQL at runtime. Use this onboarding workflow instead:

1. A domain engineer, deterministic generator, or stronger model drafts the reusable SPARQL queries.
2. Make each query parameterized with typed `{{parameter}}` placeholders.
3. Pin its Context Graph, sub-graph, and memory view.
4. Execute the query against fixture or live data and verify its expected result columns.
5. Save it with `dkg_query_catalog_save` only after it passes.
6. Expose `dkg_query_catalog_list` and `dkg_query_catalog_run` to the small model. Avoid the generic `dkg_query` fallback unless the model has passed a raw-SPARQL evaluation.
7. If no catalog entry matches a request, return that limitation instead of inventing a selector or SPARQL query.

Example MCP payload for a reviewed catalog entry:

```json
{
  "name": "dkg_query_catalog_save",
  "arguments": {
    "projectId": "manufacturing",
    "name": "Products by category",
    "description": "Return products for one reviewed category.",
    "subGraph": "products",
    "catalogSlug": "product-search",
    "view": "verifiable-memory",
    "sparql": "SELECT ?product ?name WHERE { ?product <schema:category> {{category}} ; <schema:name> ?name } ORDER BY ?name",
    "parameters": [
      {
        "name": "category",
        "type": "string",
        "label": "Category",
        "required": true
      }
    ],
    "resultColumn": "product"
  }
}
```

Always run the saved selector with a representative parameter before making it available to the small model.

## Prerequisites

* A configured DKG node and Context Graph.
* A built or installed `dkg` CLI containing the `dkg llm` command.
* A recent llama.cpp `llama-server` or Ollama installation.
* Enough memory for the selected local model and its context window.

Verify an installed node:

```bash
dkg status
```

For a source checkout, build the required packages once:

```bash
pnpm install
pnpm --filter @origintrail-official/dkg-local-llm build
pnpm --filter @origintrail-official/dkg-mcp build
pnpm --filter @origintrail-official/dkg build
```

The remaining steps use three terminals.

## Terminal 1: start DKG

For a globally installed node:

```bash
dkg start
dkg status
```

For a source checkout:

```bash
export DKG_HOME=/absolute/path/to/dkg-local

DKG_HOME="$DKG_HOME" \
  node packages/cli/dist/cli.js daemon-foreground-worker
```

Leave the daemon running. Its default API is `http://127.0.0.1:9200`.

## Terminal 2: start one local model server

Choose either llama.cpp or Ollama. Do not run two servers on the same port.

### Option A: llama.cpp reference server

Run only one `llama-server` on port 8080 at a time.

#### Recommended: Qwen3-8B Q4\_K\_M

```bash
/absolute/path/to/llama.cpp/build/bin/llama-server \
  -hf Qwen/Qwen3-8B-GGUF:Q4_K_M \
  -ngl 999 \
  -c 8192 \
  --flash-attn on \
  --jinja \
  --temp 0.15 \
  --top-p 0.9 \
  --repeat-penalty 1.05 \
  --host 127.0.0.1 \
  --port 8080
```

#### Low-memory experiment: Bonsai-8B Q1\_0

Use this model only after completing the Query-Catalog-first workflow above.

```bash
/absolute/path/to/llama.cpp/build/bin/llama-server \
  -hf prism-ml/Bonsai-8B-gguf:Q1_0 \
  -ngl 999 \
  -c 8192 \
  --flash-attn on \
  --jinja \
  --temp 0.15 \
  --top-p 0.9 \
  --repeat-penalty 1.05 \
  --host 127.0.0.1 \
  --port 8080
```

#### 27B 1-bit experiment

This configuration is not recommended on a 16 GB machine. If testing it, use one slot and disable the unused vision projector:

```bash
MODEL_PATH="$(hf download unsloth/Qwen3.8-27B-GGUF Qwen3.8-27B-UD-IQ1_M.gguf --format quiet)"

/absolute/path/to/llama.cpp/build/bin/llama-server \
  -m "$MODEL_PATH" \
  -ngl 999 \
  -c 8192 \
  -np 1 \
  --flash-attn on \
  --jinja \
  --no-mmproj \
  --temp 0.15 \
  --top-p 0.9 \
  --repeat-penalty 1.05 \
  --host 127.0.0.1 \
  --port 8080
```

Wait for `model loaded`, then check the endpoint:

```bash
curl -sS http://127.0.0.1:8080/health
```

Expected result:

```json
{"status":"ok"}
```

Also verify the provider-neutral readiness route:

```bash
curl -sS http://127.0.0.1:8080/v1/models
```

### Option B: Ollama

Complete the canonical [Install and run Ollama](#install-and-run-ollama) procedure above. It owns the installation, server-ownership, minimum-context, model-list, and readiness policy; this terminal walkthrough intentionally does not repeat it. Leave the verified Ollama server running, then continue with the DKG client below.

## Terminal 3: start DKG chat

For LLM chat, Context Graph scope is explicit. `--project` pins this session; the command does not treat an inherited `DKG_PROJECT` or the MCP configuration's first graph as an invisible fallback. Every scoped MCP call is logged with a materialized `projectId`. A query-catalog selector returned by `list` keeps that evidence graph for a subsequent `run`, even if the session pin is different.

With a globally installed CLI:

```bash
export DKG_PROJECT=my-context-graph

dkg llm \
  --interactive \
  --project "$DKG_PROJECT" \
  --model qwen3-8b-q4-k-m
```

From a source checkout:

```bash
export DKG_HOME=/absolute/path/to/dkg-local
export DKG_PROJECT=my-context-graph

DKG_HOME="$DKG_HOME" \
  node packages/cli/dist/cli.js llm \
  --interactive \
  --project "$DKG_PROJECT" \
  --model qwen3-8b-q4-k-m
```

Omit `--interactive` and append a prompt for a one-shot request:

```bash
dkg llm --project "$DKG_PROJECT" \
  "Which saved DKG Query Catalog queries are available?"
```

The default endpoint is `http://127.0.0.1:8080/v1/chat/completions`. Override it with `--llama-url` or `DKG_LLM_URL`.

For Ollama, use its endpoint and exact model tag:

```bash
dkg llm \
  --interactive \
  --project "$DKG_PROJECT" \
  --llama-url http://127.0.0.1:11434/v1/chat/completions \
  --model qwen3:8b
```

`--llama-url` is the existing compatibility option name; it accepts any supported OpenAI-compatible local endpoint and does not select llama.cpp by itself.

## Use the local LLM from Node UI

Node UI exposes the same tested local-LLM runtime in the **Agents** panel. It does not put a second router in the browser and it does not connect the model directly to DKG. The browser sends a bounded request to the daemon; the daemon owns the session, MCP child process, DKG tool discovery, schema validation, read-only policy, and text trace.

Set the model endpoint and model name in the environment that starts the DKG daemon. For llama.cpp:

```bash
export DKG_LLM_URL=http://127.0.0.1:8080/v1/chat/completions
export DKG_LLM_MODEL=qwen3-8b-q4-k-m
export DKG_LLM_BACKEND=llama.cpp
dkg start
```

For Ollama:

```bash
export DKG_LLM_URL=http://127.0.0.1:11434/v1/chat/completions
export DKG_LLM_MODEL=qwen3:8b
export DKG_LLM_BACKEND=ollama
dkg start
```

If the daemon is already running, restart it after changing these variables. Keep the selected local model server running, then open Node UI and select **DKG Local LLM** in the Agents panel. There is no separate provider selector in the browser: Node UI uses `DKG_LLM_URL`, `DKG_LLM_MODEL`, and `DKG_LLM_BACKEND` from the daemon environment. `DKG_LLM_BACKEND` accepts `ollama`, `llama.cpp`, or the backward-compatible default `auto`. Node UI hides the integration when none of those environment overrides is set and the default local endpoint does not match a supported llama.cpp or Ollama health contract. A different HTTP service listening on the same port is not treated as a Local LLM. An explicitly configured server remains visible while unavailable so the panel can report its error. The integration remains read-only: the daemon always creates this UI runtime with writes disabled. The HTTP surface is also node-admin-only. Agent-scoped bearer tokens receive `403` and cannot start, continue, or clear the daemon-owned operator session.

The selected Context Graph is sent with the first chat turn and becomes the session lock. Selecting another graph does not silently retarget the active conversation. Use the integration menu to clear the session first, then select the new graph and send the next message. In this UI mode the lock is enforced at the tool-execution boundary: only an explicit allowlist of implementations known to be single-graph scoped is exposed, and their model-supplied graph arguments are pinned to the selected graph. Unscoped cross-graph discovery and multi-graph tools such as `dkg_memory_search` are not exposed. Browser disconnect and daemon shutdown abort and drain the active model/MCP turn before the private MCP session is closed, so cancelled work cannot outlive teardown or enter hidden conversation history.

### Startup and readiness

The daemon creates only the lightweight service during startup. MCP tool discovery and the model runtime are initialized lazily on the first chat turn. Daemon UI initialization is bounded and uses the same cancellation signal as the turn, so shutdown can terminate a stalled MCP connect or `tools/list` and dispose the partial child. Chat, clear, and close are exclusive lifecycle operations; a clear keeps the service busy until session cleanup and Context Graph lock reset are both complete.

```mermaid
sequenceDiagram
  actor Operator
  participant Model as Local model server
  participant Daemon as DKG daemon
  participant Service as LocalLlmService
  participant UI as Node UI

  Operator->>Model: Start llama.cpp or Ollama
  Operator->>Daemon: Start DKG with DKG_LLM_URL and DKG_LLM_MODEL
  Daemon->>Service: Create read-only service
  Note over Service: No MCP child and no model session yet
  UI->>Daemon: GET /api/local-llm/health
  Daemon->>Service: health()
  Service->>Model: GET /v1/models
  alt OpenAI-compatible models route is ready
    Model-->>Service: 200 model list
  else Models route unavailable
    Service->>Model: GET /health (llama.cpp fallback)
    Model-->>Service: 200 {status: "ok"}
  end
  Service-->>Daemon: ready, reachable, busy, initialized
  Daemon-->>UI: Render DKG Local LLM as available
```

### Grounded chat and Query Catalog tool loop

The runtime discovers MCP tools and their schemas with `tools/list`. Each model tool call is selected from that metadata, validated, executed through MCP, and returned to the model as DKG evidence. The UI receives the final answer plus the tool names and trace path; it never executes a DKG tool itself.

```mermaid
sequenceDiagram
  actor User
  participant UI as Node UI
  participant API as DKG daemon API
  participant Service as LocalLlmService
  participant Runtime as DkgLocalLlmRuntime
  participant Model as Local model server
  participant MCP as dkg mcp serve
  participant DKG as DKG node and store
  participant Trace as Text trace

  User->>UI: Ask a question for the selected Context Graph
  UI->>API: POST /api/local-llm/chat<br/>{message, sessionId, contextGraphId}
  API->>API: Validate body and Context Graph ID
  API->>Service: chat(message, contextGraphId)
  Service->>Model: GET /v1/models (or llama.cpp /health fallback)
  alt First turn in the daemon-owned session
    Service->>MCP: Start private stdio MCP child
    Service->>Runtime: Create read-only bounded runtime
    Runtime->>MCP: tools/list
    MCP-->>Runtime: Tool names and JSON schemas
    Service->>Service: Lock session to Context Graph
  end
  Runtime->>Model: System context, history, and routed tool schemas
  Model-->>Runtime: dkg_query_catalog_list or dkg_query_catalog_run
  Runtime->>Runtime: Validate schema, policy, and tool budget
  Runtime->>MCP: tools/call with explicit projectId
  MCP->>DKG: Execute catalog read
  DKG-->>MCP: DKG evidence
  MCP-->>Runtime: Structured tool result
  Runtime->>Model: DKG evidence for final answer
  Model-->>Runtime: Grounded response
  Runtime->>Trace: Append requests, tool calls, evidence, and answer
  Runtime-->>Service: Answer, tool calls, and trace path
  Service-->>API: Read-only response envelope
  API-->>UI: Render answer and tool metadata
```

The default UI session allows at most four tool calls per turn, exposes at most eight routed tools, retains six turns and 8,000 history characters, and caps tool/evidence payloads before they return to the model.

### Clear the session and switch Context Graphs

```mermaid
sequenceDiagram
  actor User
  participant UI as Node UI
  participant API as DKG daemon API
  participant Service as LocalLlmService
  participant Runtime as DkgLocalLlmRuntime
  participant MCP as dkg mcp serve

  User->>UI: Clear DKG Local LLM session
  UI->>API: POST /api/local-llm/session/clear {}
  API->>Service: clear()
  alt A turn is still running
    Service-->>API: 409 LOCAL_LLM_BUSY
    API-->>UI: Wait for the active turn
  else Session is idle
    Service->>Runtime: Clear bounded history
    Service->>MCP: Close private child
    Service->>Service: Release Context Graph lock
    Service-->>API: {ok: true, readOnly: true}
    API-->>UI: Clear rendered conversation
  end
  Note over UI,Service: The next turn may lock a different Context Graph
```

### Example: run a parameterized saved query

The screenshots below show a real local Qwen/llama.cpp session against a DKG Query Catalog. The Context Graph and selector are demonstration data, not hardcoded routing rules.

1. Select **DKG Local LLM** and the evidence Context Graph.

   ![DKG Local LLM agent and selected Context Graph](https://4166702221-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F3Xvop5eIlAwwtLtmyreq%2Fuploads%2Fgit-blob-ccf59cc819503f10919224d48a736cd195ceda2a%2F01-local-llm-agent-and-context-graph.jpg?alt=media)
2. Ask the model to list the saved queries. The final response reports `dkg_query_catalog_list` and renders the catalog metadata.

   ![Query Catalog list prompt](https://4166702221-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F3Xvop5eIlAwwtLtmyreq%2Fuploads%2Fgit-blob-255f1ab8cee6b61bb2af1b69a708c6a6c326246c%2F02-query-catalog-prompt.jpg?alt=media)

   ![Query Catalog list result](https://4166702221-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F3Xvop5eIlAwwtLtmyreq%2Fuploads%2Fgit-blob-4816bf5fa00f73a2ff7b7c4e0443187ab20de141%2F03-query-catalog-result.jpg?alt=media)
3. Run the returned selector with a typed parameter. The result reports `dkg_query_catalog_run` and contains only the DKG-backed model and label.

   ![Parameterized Query Catalog prompt](https://4166702221-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F3Xvop5eIlAwwtLtmyreq%2Fuploads%2Fgit-blob-720bb84f4e6720dc839c2ebeda6c1fd70e3b7014%2F04-parametric-query-prompt.jpg?alt=media)

   ![Parameterized Query Catalog result](https://4166702221-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F3Xvop5eIlAwwtLtmyreq%2Fuploads%2Fgit-blob-b9ec09bc9623c1600ec096e4891a57e12a87753f%2F05-parametric-query-result.jpg?alt=media)

## Agent smoke test

Run these prompts in order:

1. `hello` — should answer without a DKG tool call.
2. `What is the status of this DKG node?` — should call `dkg_status`.
3. `Which saved queries are available in this DKG Query Catalog?` — should call `dkg_query_catalog_list`.
4. Ask it to run one exact selector returned by step 3 with declared parameter values — should call `dkg_query_catalog_run`.
5. Ask a question that the catalog does not cover — a catalog-first small model should report the gap, not invent a selector.

Useful interactive commands:

| Command    | Purpose                                                  |
| ---------- | -------------------------------------------------------- |
| `/tools`   | Show the complete MCP-compatible tool surface            |
| `/history` | Show retained bounded chat turns and evidence tool names |
| `/log`     | Print the current plain-text interaction log path        |
| `/clear`   | Clear bounded session history                            |
| `/help`    | Show commands and session budget                         |
| `/exit`    | End the session                                          |

Logs are owner-only text files under `<DKG_HOME>/logs/local-llm` by default. They include system context, model requests, tool calls, DKG results, retries, and final answers with secrets redacted.

## Domain adapters

Use a domain profile to add literal routing keywords, adapter tool ranking hints, and a domain-specific context addendum without patching the generic router:

```bash
dkg llm \
  --interactive \
  --project manufacturing \
  --adapter /absolute/path/to/domain-adapter.js \
  --domain-profile /absolute/path/to/domain-profile.json \
  --model qwen3-8b-q4-k-m
```

Keep business IDs, expected answers, and benchmark fixtures out of the generic system context. Domain facts must come from the DKG tool results.

## Writes

Read-only is the safe default. Enable writes only for an operator-approved session:

```bash
dkg llm --interactive --project my-context-graph --allow-write
```

`--allow-write` does not automatically mutate DKG. The prompt must explicitly request a routed mutation. Do not expose share, publish, registration, or destructive tools to a small model unless the workflow has an additional operator approval gate.

## Troubleshooting

### `connection refused` on port 8080 or 11434

The selected model server is stopped or the configured URL uses the wrong port. Check its provider-neutral readiness route:

```bash
curl -sS http://127.0.0.1:8080/v1/models   # llama.cpp default
curl -sS http://127.0.0.1:11434/v1/models  # Ollama default
```

For llama.cpp, `curl -sS http://127.0.0.1:8080/health` is also supported as a readiness fallback. Confirm that the daemon's `DKG_LLM_URL` points to the same server and port, then restart the daemon if you changed the environment.

### The model invents selectors or SPARQL

Use the Query-Catalog-first workflow. Verify that the selected Context Graph contains reviewed catalog entries, and route the small model only to `dkg_query_catalog_list` and `dkg_query_catalog_run`.

### The catalog is empty

Catalog definitions are Context-Graph-specific. Generate, validate, and save the domain queries before the chat demo. Selecting another Context Graph does not copy its catalog.

### Answers are slow or time out

Use Qwen3-8B Q4\_K\_M, keep one model server running, reduce competing memory pressure, and preserve the 8192-token server context. Increase `--request-timeout-ms` only after confirming the model is still generating.

### The model rejects the context size

An explicit `exceed_context_size_error` response allows a bounded retry. The runtime first omits earlier conversation history, preserving the current question, selected Context Graph and completed tool results. It then tries a concise answer from those results without sending tool definitions. Completed tools are not run again. If the question depends on an identifier from the omitted history, the model is told to ask for it. Timeouts and other failures are not retried through this recovery path.

A domain profile can declare `requiredTools`. Session initialization fails before inference when a required tool is unavailable in the permitted scope. Catalog reads with a `catalogSlug` filter select that catalog at the store, avoiding retrieval of unrelated historical catalog entries.

### Inspect the exact interaction

Use `/log`, then open the printed file with `less`. The trace distinguishes model generation failures from MCP, DKG, and query-result failures.

## Validate another model

Before recommending another model, run the production real-DKG benchmark. It uses the same runtime, a real MCP child process, a real DKG daemon/store, and independent state verification. See [`packages/local-llm/README.md`](https://github.com/OriginTrail/dkg/tree/main/packages/local-llm/README.md#real-dkg-benchmark).


---

# Agent Instructions
This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com.

## Querying This Documentation
If you need additional information that is not directly available in this page, you can query the documentation by asking a question.

Perform an HTTP GET request on the following URL with the `ask` and `goal` query parameters:

```
GET https://docs.origintrail.io/connect-agents-and-systems/local-llm.md?ask=<question>&goal=<user_goal>
```

`ask` is the immediate question: it should be specific, self-contained, and written in natural language.
`goal` is what the user is ultimately trying to achieve, the reason they need the answer. Sharing it helps GitBook give you a better, more relevant answer. A goal is most helpful when it describes the outcome the user wants rather than restating the question. For example, with `ask=how do I create an API token`, a goal like `automate deployments from our CI pipeline` lets GitBook tailor the answer to that use case.

The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.
