Running a Node and Operator Manual

Deployment

Node setup in four commands

Clone the repository into your preferred deployment directory (the default service unit targets /opt/fleetmesh). Python 3.12 and pip install cryptography are the only prerequisites.

# 1. Initialize node directory and identity
python3 -m node.meshnode init /var/lib/meshnode --class edge \
    --operator <operator-pubkey> --relay wss://relay.example.org --face-port 7777

# 2. Add an initial peering configuration
python3 -m node.meshnode peer /var/lib/meshnode <peer-pubkey> \
    --shard bristol --filter '{"kinds":[1],"#t":["bristol"]}'

# 3. Install the systemd service unit
install -m 0644 node/meshnode.service /etc/systemd/system/meshnode.service

# 4. Enable and start the daemon
systemctl enable --now meshnode

During init, the node generates identity keys and mines a kind 11801 descriptor with 20 bits of proof-of-work (typically 1–15 seconds on a standard cloud vCPU). The descriptor is cached on disk and reused across restarts unless endpoints or relays change.

The node directory contains:

• keys.json: Node private keys (permissions 0600).
• config.json: Local configuration (relays, peering targets, capacity limits).
• descriptor.json: The published kind 11801 descriptor with PoW.
• events.sqlite: Deduplicated nostr events, tag indices, and Negentropy trees.
• grants.sqlite: Active, pending, and historical peering grants.
• accounts.sqlite: Window meter counters, receipt hash chains, and strain logs.
• control.sock: Active UNIX domain socket for CLI IPC.

Network Architecture

Public relay face and TLS reverse proxy

A node without public inbound ports remains fully reachable over the Nostr transport via gift-wrapped DIDComm messages routed through mutual public relays. To serve external clients over WebSocket, bind the relay face to localhost and place a TLS reverse proxy (such as Caddy) in front:

# /etc/caddy/Caddyfile
relay.example.org {
    reverse_proxy 127.0.0.1:7777
}

Declare the public URL in your node config so peers can dial the relay endpoint directly:

"endpoints": [{"transport": "wss", "address": "wss://relay.example.org", "priority": 40}]

When endpoints change, the node automatically mines an updated descriptor. Peers discover new endpoints by comparing the endpoints digest in periodic heartbeats.

Operator Interface

Command reference (the 40 subcommands)

meshnode provides forty subcommands designed for automation, lifecycle management, bilateral peering, compute routing, transport diagnostics, and settlement. Commands reading state operate directly against local SQLite files (and query the live daemon when running). Configuration commands write to config.json, which the daemon reloads on its next tick. Action commands communicate with the running daemon over control.sock. The table is held to the parser by scripts/check-cli.py, which fails the build when a subcommand is added without a row here.

Declarative Manifests

Kubernetes-style manifests and non-destructive reconciliation

For fleet operations, CI/CD pipelines, and cloud-native deployments, Fleetmesh provides a declarative manifest suite modeled after Kubernetes resources under apiVersion: fleetmesh.org/v1alpha1. Operators declare desired state in version-controlled YAML or JSON files:

Fleetmesh provides two complementary software products: the meshnode sovereign daemon and the standalone meshctl declarative fleet management CLI (built in Rust).

Fleet Management with meshctl

meshctl provides declarative, non-destructive hot-reconciliation against running meshnode daemons via the meshnode-control/1 Unix domain socket. Applying a manifest dynamically updates peering filters, rate limits, and budget allocations without dropping live QUIC/TLS sessions or resetting receipt hash chains.

# Apply manifests with non-destructive live hot-reconciliation:
meshctl apply -f manifests/peering-bristol.yaml

# Dry-run validation to preview reconciliation deltas:
meshctl apply -f manifests/ --dry-run

# Diff manifest against live daemon state:
meshctl diff -f manifests/peering-bristol.yaml

# Inspect node and fleet health via the control socket:
meshctl status --dir /var/lib/fleetmesh

# Compile manifests to production deployment targets:
meshctl export -f manifests/meshnode-anchor.yaml --target k8s > k8s-node.yaml
meshctl export -f manifests/meshnode-anchor.yaml --target quadlet > ~/.config/containers/systemd/node.container
meshctl export -f manifests/meshnode-anchor.yaml --target systemd > /etc/systemd/system/meshnode.service
meshctl export -f manifests/meshnode-anchor.yaml --target compose > docker-compose.yml

# Convert legacy fleetmesh.org/v1alpha1 manifests to fleetmesh.org/v1alpha1:
meshctl convert -f legacy-manifest.yaml -o fleetmesh.yaml

# Node-local fallback init:
meshnode init --manifest manifests/meshnode-anchor.yaml --peering manifests/peering-bristol.yaml
meshnode run

Automated Monitoring

The watchdog daemon (meshwatch)

The watcher daemon (node/watch.py) runs on a systemd timer (meshwatch.timer) to execute recurring health audits against the running node. It uses clean process exit codes for seamless integration with Prometheus, Datadog, or nagios:

• Exit 0 (Healthy): All relay subscriptions are connected, databases pass integrity checks, and no unexpected drop rates or severe throttles exist.
• Exit 1 (Wrong): An issue requiring operator attention: relay down, active rate throttling, S1/S2/S3 pricing divergence, rejected config, or protocol digest mismatch.
• Exit 2 (Unreachable): A node directory or remote SSH host could not be reached, keys could not be loaded, or connectivity failed entirely.

# Inspect health status via the CLI
python3 -m node.meshnode watch /var/lib/meshnode

Accounting In Practice

Reading window evaluations

Bilateral accountability centers on the window evaluation log. Inspect it using meshnode windows <peer>:

windows with f22c163a211c50e3  (measured/claimed, in the issuer frame)
window  closed UTC        severity  req  events_in  bytes_in      receipts in  sent
496817  2026-09-04 18:00  S0        0/-  37/37      45275/45275   2            2
496818  2026-09-04 19:00  S0        0/-  12/12      14820/14820   1            1

Key metrics to observe:

• Issuer frame: All metrics measure peer resource usage under the granted budget.
• measured vs. claimed: measured is the traffic received at your door; claimed is what the signed peer receipt reported. A dash indicates no claim was received.
• Severity: S0 (Clean, AIMD grant increase applies), S1 (Minor deviation, grant halved), S2 (Substantial breach, grant reduced to zero), S3 (Signed proof of fraud, immediate peering termination).

Reliability Principles

Three architectural invariants

Diagnostics

Troubleshooting checklist

• Peer is quiet (no traffic flowing): Run meshnode health. If the peer is on a probation grant and has hit its sync-range limit, silence is expected until the next window cycle. Verify by checking active throttles.
• Relay connectivity: Check the relay status line in health. Send a test kind 21059 gift wrap through the public relay to confirm end-to-end receipt.
• Oversized envelopes: Nostr relays enforce maximum event sizes (often 64–128 KB). Ensure synchronization batches fit within configured fragment size limits.
• Window mismatch: Run meshnode windows <peer> on both sides. A divergence between measured and claimed indicates dropped packets or an unacknowledged delivery.
• Stale descriptor: If peering negotiations fail, confirm that the peer holds the latest descriptor by checking the endpoints digest in the heartbeat.