OriginTrail
  • Get started with OriginTrail
  • OriginTrail Ecosystem — Call for Papers (Coming Soon)
  • Key Concepts
    • Decentralized Knowledge Graph (DKG)
    • DKG key concepts
  • Build with DKG
    • Quickstart (test drive the DKG in 5 mins)
      • Quickstart with Node.js
      • Quickstart with Python
    • ChatDKG builder toolkit
      • DKG SDK
        • Development environment setup
        • DKG Javascript SDK (dkg.js)
          • Interact with DKG paranets
          • Knowledge submission & curation
          • Paranet's incentives pool implementation
        • DKG Python SDK (dkg.py)
      • DKG paranets
        • Deploying a DKG paranet
        • Building with DKG paranets
        • Syncing a DKG Paranet
        • Initial Paranet Offerings (IPOs)
          • IPO specification
          • Launching your IPO
          • Paranet's incentives pool
          • IPO voting
      • AI agents
        • ElizaOS DKG agent
        • Custom DKG Python agent
        • Custom DKG JavaScript agent
    • DKG Edge Node
      • DKG Edge Node architecture
      • Get started with the Edge Node boilerplate
        • Automated setup with the installer
        • Manual setup
        • Usage example
      • Customize & build with the Edge Node
      • Knowledge Mining and dRAG examples
      • Deploy your Edge Node based project
        • Automated deployment with installer
      • DKG Edge Node inception program
      • DKG Edge Node API documentation
    • DKG Core Node
      • Run a V8 Core Node on testnet
        • Preparation for V8 DKG Core Node deployment
        • V8 DKG Core Node installation
      • Run a V8 Core Node on mainnet
        • Preparation for V8 DKG Core Node deployment
        • V8 DKG Core Node installation
  • Delegated staking
    • Delegated staking—Introduction
      • Step-by-step staking
      • Redelegating stake
  • Integrated Blockchains
    • Base blockchain
      • Connect to Base
    • Gnosis chain
      • Connect to Gnosis
    • NeuroWeb
    • Teleport instructions - NeuroWeb
    • Bridging to Moonbeam
    • Deployed smart contracts
  • Bounties & rewards
    • General bug bounty
    • Code contributions & V8 bug bounty
  • Whitepapers & RFCs
    • OriginTrail whitepaper
    • OriginTrail RFCs
  • Useful Resources
    • What's new with OriginTrail V8
    • DKG V8 guidebook
      • Protocol updates
      • Feature roadmap
      • How to upgrade to V8?
    • Public nodes
    • Tutorials
    • Test token faucet
    • Development principles
    • Community created resources
    • Linked data & knowledge graphs
    • Available networks, network details and RPCs
    • OT Node Engine implementation details
      • Modules
      • Command Executor
    • Contribution guidelines
      • Guidelines for automated test contributions
    • Explore the OriginTrail ecosystem
Powered by GitBook
On this page
  • Setup with a local DKG network [Recommended]
  • System Requirements
  • Software Dependencies
  • Setup process
  • 1 - Setting up local DKG network:
  • 2 - Cloning DKG Edge Node services:
  • 3 - Configuring DKG Edge Node services
  • Setup with pre-deployed V8 DKG Runtime Testnet Node
  • System Requirements
  • Software Dependencies
  • Setup process:
  • 1 - Cloning DKG Edge Node services:
  • 2 - Whitelisting your local IP on the pre-deployed V8 DKG Runtime Node:
  • 3 - Configuring DKG Edge Node services

Was this helpful?

Edit on GitHub
  1. Build with DKG
  2. DKG Edge Node
  3. Get started with the Edge Node boilerplate

Manual setup

Setting up an Edge Node project to explore its capabilities and build your own projects. As an open-source solution, the Edge Node offers complete customization to fit your needs.

PreviousAutomated setup with the installerNextUsage example

Last updated 3 months ago

Was this helpful?

To manually set up the DKG Edge Node in your local environment, you can choose between the following two options:

  1. [Recommended]

Setup with a local DKG network [Recommended]

This option allows you to configure the Edge Node services within your local development environment, utilizing a locally deployed DKG network. This setup provides full control over the environment and is ideal for developing custom processing pipelines and testing in a self-contained network.

System Requirements

  • Operating System: macOS, Linux

  • RAM: At least 8 GB

  • CPU: 4

  • Storage: At least 20 GB of available space

  • Network: Stable internet connection

Software Dependencies

Make sure the following services are installed and properly configured:

  • Git: Version control system

    • Windows:

    • Linux: (via a package manager such assudo apt install git for Ubuntu)

    • macOS:

  • MySQL 8: Database service

    • Windows:

    • Linux: (via sudo apt install mysql-server for Ubuntu)

    • macOS:

  • Redis: In-memory data structure store ()

    • Windows:

    • Linux:

    sudo apt update
    sudo apt install redis-server
    • macOS:

    brew install redis
  • Node.js: JavaScript runtime environment

    • v20.04: Used for the local network/ot-node setup

    • v22.4.0: Used for Edge Node services

  • v11: for running Blazegraph triple store

  • Python v3.11.7

We recommend using NVM and Pyenv in order to be able to switch Node.js versions easily and install Python.

Setup process

1 - Setting up local DKG network:

Edge Node services require a V8 DKG Runtime Node endpoint in order to be configured and initialized so for this setup, running a local DKG network is essential.

In order to deploy a local DKG network, do the following:

  • Clone the OriginTrail node engine: git clone https://github.com/OriginTrail/ot-node

  • Enter ot-node directory and checkout to v8/develop branch: cd ot-node && git checkout v8/develop

  • Switch to Node.js v20 and install node modules: npm install

  • Run blazegraph.jar file with the following command: java -server -Xmx4g -jar blazegraph.jar in order to start the triple store database (from the directory where Blazegraph is downloaded)

  • Make sure that the MySQL service is running

  • In ot-node directory, create .env file in ot-node directory and populate it with the following parameters (If your MySQL is password protected, add the password to REPOSITORY_PASSWORD):

NODE_ENV=development
RPC_ENDPOINT_BC1=http://127.0.0.1:8545
RPC_ENDPOINT_BC2=http://127.0.0.1:9545
REPOSITORY_PASSWORD=
  • Start the local DKG network by executing the following command from the ot-node directory: bash tools/local-network-setup/setup-macos-environment.sh --nodes=5

  • Once the network deployment is initiated, you will be provided with the local hardhat blockchain and the amount of local DKG nodes that you defined via --nodes flag

Once the network is up and running, you will select one of the local node endpoints to configure the Edge Node services, which will be explained in the continuation of these instructions.

2 - Cloning DKG Edge Node services:

In order to kick off the installation process, you need to clone all Edge Node services to your local environment using git clone command.

3 - Configuring DKG Edge Node services

The instructions for configuring DKG Edge Node services are also available in the README file of each service's GitHub repository, where you can follow the steps provided.

3.1 Setup Edge Node Authentication Service:

  • Create database 'edge-node-auth-service'

  • Create .env file with cp .env.example .env in the dir.

  • Generate random strings for the following .env variables:

    • JWT_SECRET and

    • SECRET (you can use openssl rand -hex 64 for example)

  • Install node modules with npm install (use Node.js v22.4.0)

  • Setup your database with npx sequelize-cli db:migrate and npx sequelize-cli db:seed:all - This will generate a demo user, a wallet with funds for the local network, and configure your local Edge Node Authentication Service to connect to the first node from your local network.

seed command will create an example user with the following credentials:

username admin and password admin123

  • Initiate Edge Node Authentication service with: npm run start

3.2 - Configuring Edge Node API service:

  1. Create .env file with cp .env.example .env in the service directory

  2. Create the database mentioned in .env

  3. Install node modules with npm install (use Node.js v22.4.0)

  4. Execute migrations: npx sequelize-cli db:migrate

  5. Setup Runtime node MySQL operational db connection by populating the following values in the .env file:

RUNTIME_NODE_OPERATIONAL_DB_USERNAME=root
RUNTIME_NODE_OPERATIONAL_DB_PASSWORD=
RUNTIME_NODE_OPERATIONAL_DB_DATABASE=operationaldb0
RUNTIME_NODE_OPERATIONAL_DB_HOST=127.0.0.1
RUNTIME_NODE_OPERATIONAL_DB_DIALECT=mysql

When your local DKG network is deployed, each of the local nodes will create its own operational database named operationaldb0, operationaldb1, operationaldb2, and so on, depending on the size of your local network.

  1. Initialize Redis and make sure that it's running on its default port 6379

  2. Start the service: npm run start

3.3 - Configuring Edge Node UI:

In order to set up the Edge Node UI locally, please run the following commands:

  • Create .env file: cp .env.example .env

  • Install ionic/cli: npm install -g @ionic/cli

  • Install electron: npm install -g electron

  • Install node modules: npm install (use Node.js v22.4.0)

  • To run the application in your browser: ionic serve

  • To run the application in your desktop app: npm run start-electron

Once the Edge Node UI is ready, the only accessible screen will be /login. Use the credentials created during the Auth service setup previously.

3.4 - Configuring Edge Node Knowledge Mining:

In order to setup Edge Node UI locally, please follow the steps below:

  • Create .env file with cp .env.example .env in the dir.

  • Add the absolute path to your DAGs folder into the DAG_FOLDER_NAME variable in the .env file

  • Setup Python environment: pyenv local 3.11.7

It's recommended to use pyenv and install Python 3.11 locally within the app's directory to avoid conflicts with other Python versions on your machine.

  • A virtual environment should be set up to install the required dependencies. You can do this by running the following command: python -m venv .venv && source .venv/bin/activate

  • Install Python requirements: pip install -r requirements.txt

  • Setup Apache Airflow service:

Airflow pipelines are an integral part of the Knowledge Mining Service, used to create automated data processing workflows. The primary purpose of these pipelines is to generate content for Knowledge Assets based on input files.

  • Generate default Airflow config: airflow config list --defaults

  • Open the Airflow configuration file located at "~/airflow/airflow.cfg" file and update the following parameters as presented below:

load_examples = False
dags_folder = YOUR_PATH_TO/edge-node-knowledge-mining/dags
parallelism = 32
max_active_tasks_per_dag = 16
max_active_runs_per_dag = 16
enable_xcom_pickling = True
  • Initiate Airflow database and create an admin user with:

airflow db init
airflow users  create --role Admin --username admin --email admin --firstname admin --lastname admin --password admin
  • Initiate Airflow scheduler: airflow scheduler

  • Pick up new jobs and start them:

airflow dags unpause exampleDAG
airflow dags unpause pdf_to_jsonld
airflow dags unpause simple_json_to_jsonld
  • Initiate Airflow webserver: airflow webserver --port 8080

  • Initiate Edge Node Knowledge Mining: python app.py

airflow scheduler, airflow webserver and python app.py should be initiated in parallel inside the virtual environment (use separated terminal windows to run them).

  • Create MySQL logging: CREATE DATABASE `ka-mining-api-logging` CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci;

3.5 - Configuring Edge Node dRAG:

In order to set up the Edge Node dRAG locally, please follow the steps below:

  • Install node modules: npm install (use Node.js v22.4.0)

  • Create .env file: cp .env.example .env

  • Add your LLM_API_KEY into the .env (dRAG uses LLM to formulate the answer)

  • Create a database for dRAG logging: CREATE DATABASE drag_logging;

  • Run migrations: npx sequelize-cli db:migrate

  • Initiate the service: npm run start

(Optional) If you want to use the vectorization controller, you have to set up the following services: HuggingFace, Zilliz, and Cohere. If you want to customize the experience further, you can also modify the code to use any other service for embedding/vector search and reranking.

Once you've finalized configuring all DKG Edge Node services, please make sure that they are exposed on the following ports:

Setup with pre-deployed V8 DKG Runtime Testnet Node

System Requirements

  • Operating System: macOS, Linux, Windows with WSL

  • RAM: At least 8 GB

  • CPU: 4

  • Storage: At least 20 GB available space

  • Network: Stable internet connection

Software Dependencies

Make sure the following services are installed and properly configured:

  • Git: Version control system

  • MySQL 8: Database service

  • Node.js v22.4.0: JavaScript runtime environment

  • Python v3.11.7

We recommend using NVM and Pyenv in order to be able to switch Node.js versions easily and install Python.

Setup process:

1 - Cloning DKG Edge Node services:

In order to kick off the installation process, you need to clone all Edge Node services to your local environment using git clone command.

2 - Whitelisting your local IP on the pre-deployed V8 DKG Runtime Node:

  • SSH to the server where you have your V8 DKG Core Node up and running

  • Edit .origintrail_noderc configuration file with nano or any other editor

  • Locate the auth section in the configuration file and add your local IP as presented below:

    "auth": {
        "ipWhitelist": [
            "::1",
            "127.0.0.1",
            "<your_local_ip_address>"
        ]
    }
  • Restart your node with otnode-restart command in order for changes to configuration to be applied

3 - Configuring DKG Edge Node services

The instructions for configuring DKG Edge Node services are also available in the README file of each service's GitHub repository, where you can follow the steps provided.

3.1 Setup Edge Node Authentication Service:

  • Create database 'edge-node-auth-service'

  • Create .env file with cp .env.example .env in the dir.

  • Generate random strings for the following .env variables:

    • JWT_SECRET and

    • SECRET (you can use openssl rand -hex 64 for example)

  • Install node modules with npm install (use Node.js v22.4.0)

  • Setup your database with npx sequelize-cli db:migrate and npx sequelize-cli db:seed:all - This will generate a demo user, a wallet with funds for the local network, and configure your local Edge Node Authentication Service to connect to the first node from your local network.

seed command will create an example user with the following credentials:

username admin and password admin123

  • Since the node is by default configured to automatically work with your local network, the following variables inside of the UserConfigs table (database: edge-node-auth-service) should be updated to match your pre-deployed V8 DKG Runtime Node information:

    • "run_time_node_endpoint": http://<your_node_endpoint_or_ip>

    • "run_time_node_port": 8900

    • "edge_node_environment": testnet

    • "blockchain": base:84532

  • Replace placeholder wallet - The following table "user_wallets" will be populated with a pre-defined wallet address (local network related) which should be replaced by your Base Sepolia wallet with funds (ETH and TRAC)

  • Initiate Edge Node Authentication Service with: npm run start

3.2 - Configuring Edge Node API Service:

  1. Create .env file with cp .env.example .env in the service directory

  2. Create the database mentioned in .env

  3. Install node modules with npm install (use Node.js v23)

  4. Execute migrations: npx sequelize-cli db:migrate

  5. Expose V8 DKG Core node operational database (MySQL) to a local Edge Node API service:

    • SSH to your Runtime Node server

    • Expose port 3306 on your server to your local IP address (firewall configuration)

    • Enable MySQL remote connection by changing the bind-address from 127.0.0.0 to 0.0.0.0 in /etc/mysql/mysql.conf.d/mysqld.cnf

    • Restart mysql.service: systemctl restart mysql.service

    • Create new MySQL user for Edge Node API service to use:

      • mysql -u root -p

      • When asked for the password, use the password you created during the node setup process

      • Create a user for remote access: CREATE USER'username'@'%' IDENTIFIED BY'your_password';

      • GRANT ALL PRIVILEGES ON*.*TO'username'@'%'WITH GRANT OPTION;

      • FLUSH PRIVILEGES;

    • Setup Runtime Node MySQL operational db connection by populating the following values in the .env file:

      RUNTIME_NODE_OPERATIONAL_DB_USERNAME=root
      RUNTIME_NODE_OPERATIONAL_DB_PASSWORD=<runtime_node_operationaldb_password>
      RUNTIME_NODE_OPERATIONAL_DB_DATABASE=operationaldb
      RUNTIME_NODE_OPERATIONAL_DB_HOST=<endpoint/ip_of_your_runtime_node>
      RUNTIME_NODE_OPERATIONAL_DB_DIALECT=mysql
  6. Initialize Redis and make sure that it's running on its default port 6379

  7. Start the service: npm run start

3.3 - Configuring Edge Node UI:

In order to set up the Edge Node UI locally, please run the following commands:

  • Create .env file: cp .env.example .env

  • Install ionic/cli: npm install -g @ionic/cli

  • Install electron: npm install -g electron

  • Install node modules: npm install (use Node.js v23)

  • To run the application in your browser: ionic serve

  • To run the application in your desktop app: npm run start-electron

Once the Edge Node UI is ready, the only accessible screen will be /login. Use the credentials created during the Auth Service setup previously.

3.4 - Configuring Edge Node Knowledge Mining:

In order to setup Edge Node UI locally, please follow the steps below:

  • Create .env file with cp .env.example .env in the dir.

  • Add the absolute path to your DAGs folder into the DAG_FOLDER_NAME variable in the .env file

  • Setup Python environment: pyenv local 3.11.7

It's recommended to use pyenv and install Python 3.11 locally within the app's directory to avoid conflicts with other Python versions on your machine.

  • A virtual environment should be set up to install the required dependencies. You can do this by running the following command: python -m venv .venv && source .venv/bin/activate

  • Install Python requirements: pip install -r requirements.txt

  • Setup Apache Airflow service:

Airflow pipelines are an integral part of the Knowledge Mining service, used to create automated data processing workflows. The primary purpose of these pipelines is to generate content for Knowledge Assets based on input files.

  • Generate default Airflow config: airflow config list --defaults

  • Open the Airflow configuration file located at "~/airflow/airflow.cfg" file and update the following parameters as presented below:

load_examples = False
dags_folder = YOUR_PATH_TO/edge-node-knowledge-mining/dags
parallelism = 32
max_active_tasks_per_dag = 16
max_active_runs_per_dag = 16
enable_xcom_pickling = True
  • Initiate Airflow database and create an admin user with:

airflow db init
airflow users  create --role Admin --username admin --email admin --firstname admin --lastname admin --password admin
  • Initiate Airflow scheduler: airflow scheduler

  • Pick up new jobs and start them:

airflow dags unpause exampleDAG
airflow dags unpause pdf_to_jsonld
airflow dags unpause simple_json_to_jsonld
  • Initiate Airflow webserver: airflow webserver --port 8080

  • Initiate Edge Node Knowledge Mining: python app.py

airflow scheduler, airflow webserver and python app.py should be initiated in parallel inside the virtual environment (use separated terminal windows to run them).

  • Create MySQL logging: CREATE DATABASE `ka-mining-api-logging` CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci;

3.5 - Configuring Edge Node dRAG:

In order to set up Edge Node dRAG locally, please follow the steps below:

  • Install node modules: npm install (use Node.js v22.4.0)

  • Create .env file: cp .env.example .env

  • Add your LLM_API_KEY into the .env (dRAG uses LLM to formulate the answer)

  • Create a database for dRAG logging: CREATE DATABASE drag_logging;

  • Run migrations: npx sequelize-cli db:migrate

  • Initiate the service: npm run start

(Optional) In case you want to use the vectorization controller, you have to set up the following services: HuggingFace, Zilliz, and Cohere. If you want to customize the experience further, you can also modify the code to use any other service for embedding/vector search and reranking.

Once you've finalized configuring all DKG Edge Node services, please make sure that they are exposed on the following ports:

Download blazegraph.jar (Triple store db) from the following and save it to the location of your choice locally

Run the following MySQL query on edge-node-auth-service database:

Once the Airflow webserver is initiated, your pipelines should be available on

Edge Node Knowledge Mining README is available .

HuggingFace - used for vectorization embedding model

Zilliz - used for hosting the vector database

Cohere ReRanker - used for improving retrieval results accuracy

Edge Node Authentication Service:

Edge Node API:

Edge Node UI:

Edge Node Knowledge Mining:

Edge Node dRAG:

Prior to proceeding with this setup option, it is essential to have a V8 DKG Core Node operational on the testnet, as this is a critical requirement. If you have not yet deployed a Testnet Core Node, please consult the installation guide available on the following for detailed instructions.

Redis: In-memory data structure store ()

Run the following MySQL query on edge-node-auth-service database:

Instructions on how to use the TRAC faucet can be found .

Once the Airflow webserver is initiated, your pipelines should be available on

HuggingFace - used for vectorization embedding model

Zilliz - used for hosting the vector database

Cohere ReRanker - used for improving retrieval results accuracy

Edge Node Authentication Service:

Edge Node API:

Edge Node UI:

Edge Node Knowledge Mining:

Edge Node dRAG:

link
Authentication service
Edge Node API
Edge Node interface
Knowledge mining API
dRAG
UserConfig.sql
http://localhost:8080/home
here
https://huggingface.co/
https://cloud.zilliz.com/
https://dashboard.cohere.com/
http://localhost:3001
http://localhost:3002
http://localhost:5173
http://localhost:5005
http://localhost:5002
page
documentation
Authentication service
Edge Node API
Edge Node interface
Knowledge mining API
dRAG
UserConfig.sql
here
http://localhost:8080/home
https://huggingface.co/
https://cloud.zilliz.com/
https://dashboard.cohere.com/
http://localhost:3001
http://localhost:3002
http://localhost:5173
http://localhost:5005
http://localhost:5002
Download Git for Windows
Install Git on Linux
Download Git for macOS
Download MySQL Installer for Windows
Install MySQL on Linux
Download MySQL for macOS
documentation
Install Redis on Windows
Set up local Edge Node services with a local DKG network
Configure local Edge Node services with a pre-deployed V8 DKG Core Testnet Node