Running a node

Privacy rules

Privacy Guarantees & Threat Modeling

Publishing receipts and reports makes traffic volumes public, and publishing grant envelopes makes the existence and timing of peerings public. Those are the deliberate trades: auditability of enforcement in exchange for a coarse view of how much data moved and when relationships changed.

The cost of a public grant table is not, as it is easy to assume, mere "traffic volume leakage". Such a table exposes full peering topologies, bilateral subjective ratings and exact downgrade timestamps. Blinded commitments remove that surveillance surface entirely.

• Counters, never addresses. Receipts and reports carry aggregate counts only. Client IP addresses, user agents, and connection fingerprints stay on the node that observed them. A relay behind a proxy resolves client addresses to enforce local rate limits. That address resolution MUST NOT reach any federated mesh surface. Exposing resolved client IP addresses across the mesh creates an unacceptable deanonymization vector.
• Counters are bucketed. Published counts round to two significant figures. Exact byte counts across short windows fingerprint individual conversations; the reputation arithmetic does not need that resolution and MUST NOT be given it.
• Gift wraps are not a shard. The replication rule is defined in § Replication. Refusals are strictly silent. Distinguishing between nonexistent events and forbidden events leaks private communication metadata. A federated puller requesting unauthorized #p-scoped Kind 1059 events receives the exact response given to an unauthenticated stranger.
• Interests are coarse. An interest filter is a standing declaration of what a node cares about, readable by whoever serves it. Nodes SHOULD declare interests at kind and tag granularity rather than author granularity; a node that wants one person's events should pull the shard containing them. An anchor MAY take the full firehose within a scope, which is the least revealing option available to it.
• Leaves hide behind mediators. Hole punching exposes IP addresses to peers and rendezvous servers. The v1 leaf profile disables direct peer connections entirely. A leaf maintains a single wss connection to its mediator. Only the mediator observes its network address. Future WebRTC extensions MUST NOT make direct peer connections a default.
• Two mediators, not one. A mediator observes message sizes, arrival times, and connection schedules. Under the Three-Layer Operator Independence Model, leaves SHOULD maintain dual-mediation with two distinct anchors. The leaf operator selects anchors with distinct operator pubkeys and separate BGP ASNs. This distributes message metadata so no single intermediary observes complete traffic patterns.

Attack surface

Security Analysis & Failure Modes

Alpha contract

Alpha Evaluation Scope & Guarantees

The mesh ships as alpha and will stay there for some time. That word is doing real work: it licenses breaking changes to nearly everything specified here, on the understanding that a short list of promises holds anyway. Operators are volunteering hardware, and volunteers who get burned do not come back, so that list is the whole basis on which anyone should be asked to run a node.

Promises that hold during alpha

What alpha explicitly permits breaking

• Kind numbers, and every tag name inside them.
• Grant, receipt and report schemas, including which limits exist.
• Every constant in the window arithmetic and the trust weighting.
• DIDComm protocol URIs, message names and state machines.
• Transport ALPNs, gossipsub topic names, and the negotiation ladder's ordering.

Breaking changes are not announced, because nothing announces anything here. A node declares the version it speaks, peers compare digests on connect, and a mismatch declines politely rather than half-speaking — see § Governance. During alpha the declared version is fleetmesh/0.1-draft, whose whole meaning is that its digest may change on any day and nobody is owed a migration. Running alpha means accepting a re-sync, and possibly several.

What alpha is for measuring

Alpha is not only a warning label; it is the only chance to collect the numbers that several deferred decisions are waiting on. Nodes SHOULD record and publish, in aggregate:

• Whether grants bind at all, and which limit binds first. This is the measurement that answers § Grants & receipts' settlement question with data instead of a guess. If capacity is never the scarce thing, the settlement field gets deleted rather than filled in.
  A first answer, from a reference sync rather than from a network. Reconciling a plain text shard, events_in is exhausted long before any byte limit: those events average 290 bytes, so a member-tier events_in ceiling of 3,000 is reached with the bytes_in ceiling 39× away, and the same holds at probation and trusted because both scale together. For text traffic the byte ceilings are decoration and events_in is the grant. Either it is too tight or the byte ceilings are far too loose; one of the two should move before anyone calls these calibrated. Blob traffic is the case that would invert it, and blob_bytes defaults to zero.
• The distribution of divergence between claimed and measured counters among peers nobody suspects. Every constant in the ladder is calibrated against that distribution or against nothing.
• How often the transport ladder falls through to wss, and at which rung. If hole punching succeeds rarely enough, the edge class needs rethinking rather than tuning.
• Which retention mode operators actually choose, and whether peers leaves them unable to evaluate a candidate before peering.
• Whether disk is ever the binding constraint. Coverage declined for capacity, shards dropped for capacity rather than age, and whether any operator raises blob_bytes above its default at all. This is the measurement that answers § Grants & receipts' storage question. If nobody ever declines a shard for want of disk, storage scarcity was imaginary, and the counters are deleted rather than acted on.

{
  "kind": 21802,
  "tags": [
    ["metric", "grant_binding", "events_in:12", "bytes_out:45", "sync_ranges:3"],
    ["metric", "divergence_hist", "s0:984", "s1:14", "s2:2", "s3:0"],
    ["metric", "transport_success", "iroh:820", "libp2p:120", "wss:60", "unix:0"],
    ["metric", "retention_mode", "peers"],
    ["metric", "shard_bytes_held", "p50:8388608", "p95:134217728"],
    ["metric", "coverage_declined", "capacity:0", "policy:3"],
    ["metric", "shard_dropped", "age:2", "capacity:0"],
    ["metric", "blob_ceiling_raised", "false"],
    ["sample_window", "86400"],
    ["expiration", "1787090000"]
  ],
  "content": ""
}

Diagnostic Collection Path: Kind 21802 is an ephemeral event (NIP-01, 20000 ≤ kind < 30000) broadcast over gossipsub or published to opted-in diagnostic monitors. To preserve absolute privacy, all counters are coarse-binned and aggregated across the whole node over a 24-hour window, with no per-peer attribution or identifying metadata.

Provenance during alpha

Data received solely through the mesh carries provisional status. A peer's claim justifies verification rather than assertion. Concretely: a node MUST NOT publish completeness: asserted for shards held only through mesh replication. The gateway MUST record mesh origin on all quarantined events. This prevents third-party replication from laundering unverified peer data into authoritative records.

Implementation

Reference Implementation Components

Very little here is new machinery. A conformant node is an ordinary NIP-01 relay with three additions: a pluggable event store, a write-policy seam, and an embedded-database backend for the small targets. Good relay software already has all three. did:nostr is specified and implementable in a few hundred lines. Blossom already addresses blobs by content hash. What has to be built is range reconciliation, the mesh layer itself, and the gateway that keeps it at arm's length from whatever else an operator runs.

New crates

This is the shape a production implementation should take, and none of it exists. The one implementation that does is Python, built around the executable reference in ref/ rather than in this layout, and its only claim is that it runs the whole of this document once over a wire and found several defects doing so. Read the list below as a plan.

What an existing relay has to add

Every item below is scoped to a capability rather than a file, because it has to be implementable against any relay codebase. Each is also independently useful: a relay that adds them all and never joins the mesh has still gained range sync, better rate limiting and a correctness fix.

Rollout

• W1
  NegentropyNIP-77 in the relay, both stores, conformance tests. Useful on its own — every client with range sync benefits immediately, and nothing else here works without it.
• W2
  BoundaryThe gateway skeleton, its own namespace, keys and quota, and the unplug test wired into CI. Built before there is anything to gate, because a boundary retrofitted after traffic exists is a boundary that already leaked.
• W3
  IdentityNode descriptors, the DID node profile, revocation. Publishable and resolvable before anything peers, so identity can be validated in isolation.
• W4
  Control planeDIDComm v2 over the wss fallback only. The slowest transport, chosen first: it exercises every protocol without a p2p stack in the way.
• W5
  GrantsIssuance, receipts, window evaluation, the ladder — shadow mode only. At least four weeks against real traffic before any constant is treated as settled.
• W6
  Transportsiroh and libp2p, the negotiation ladder, mediation. The first point at which an edge behind NAT is a real participant.
• W7
  The binaryPackaged fleetmesh-node, ARM images, a one-command install, a published bootstrap.toml. The first release an outsider can actually run — and the point at which alpha starts meaning something to someone other than us.
• W8
  LeavesMobile library and QR pairing, at the scoped profile only: one wss connection to a mediator, foreground sync, push to wake. Last, because a leaf depends on every other layer being real — and small, because the operating systems have already decided how small.

Operator surface

Declarative Manifests & State Reconciliation

An operator surface that takes thirty minutes to read or relies on custom imperative scripts fails before it has served a single packet. A node's whole operational surface is authored as YAML or JSON manifests, following the cloud-native pattern (apiVersion: fleetmesh.org/v1alpha1). The standalone meshctl CLI (built in Rust) manages them and reconciles them live against running meshnode daemons via the meshnode-control/1 Unix domain socket, executing non-destructive hot-reconciliation and multi-target compilation. The Python reference node consumes the same manifests once, at meshnode init --manifest.

The declarative manifest remains the single source of truth across all deployment environments. It deploys identically on Kubernetes, rootless Podman Quadlets, headless Raspberry Pis, or embedded mobile leaf runtimes.

Core Resource Kinds

Declarative Manifest Examples

apiVersion: fleetmesh.org/v1alpha1
kind: MeshNode
metadata:
  name: bristol-anchor-01
  labels:
    region: eu-west
    env: production
spec:
  class: anchor
  identity:
    did: "did:nostr:d4e287a91176b6a0ff2ff2384a6c8e5473f309a47ef0e854d92305574581eb08"
    keyAgreement: "fec01c7d23a4b9180fa49c30f40d99ef87b3a98c0b2efd1487ea0b240398f498c"
  operator:
    # Distinct from the node DID above. Layer 1 of the independence model detects
    # shadow nodes by comparing operator pubkeys; reusing the node key defeats it.
    pubkey: "9b21c0f4e8a17d3562bc04ea7f19d8c05e3a6b7419fd28ce03b5a6142d7e08f3"
    policyUrl: "https://relay.example.org/policy.txt"
  endpoints:
    - transport: iroh
      ticket: "iroh_ticket_node_7f8a9b"
      priority: 10
    - transport: libp2p
      address: "/dnsaddr/relay.example.org/p2p/12D3KooWDpJ7As7BWAwRMfu1VU2WCqNjvq387JEYKDBj4kx6nXTN"
      priority: 20
    - transport: wss
      address: "wss://relay.example.org"
      priority: 40
  storage:
    engine: lmdb
    path: "/var/lib/fleetmesh/data"
    maxBytes: 107374182400  # 100 GiB
  capabilities:
    - relay
    - blossom
    - mediator
    - negentropy

apiVersion: fleetmesh.org/v1alpha1
kind: Peering
metadata:
  name: bristol-community-sync
spec:
  peer: "did:nostr:e5f398b02287c7b1003003495b7d9f65840410b58f0f1965e03416685692fc19"
  transport: iroh
  ingress:
    filter:
      kinds: [0, 1, 3, 7, 10002]
      "#t": ["bristol"]
    plugin: "/usr/local/bin/mesh-filter-in"
    maxEventsPerSec: 100
    maxBytesPerSec: 2097152
  egress:
    filter:
      kinds: [0, 1, 3, 7, 10002]
      "#t": ["bristol"]
    plugin: "/usr/local/bin/mesh-filter-out"
    maxEventsPerSec: 100
    maxBytesPerSec: 2097152
  budget:
    window: 3600
    events_in: 3000
    bytes_in: 33554432
    bytes_out: 268435456

apiVersion: fleetmesh.org/v1alpha1
kind: InferenceRouter
metadata:
  name: routstr-gpu-bridge
spec:
  provider: routstr
  endpoint: "http://127.0.0.1:8000/v1"
  models:
    - id: "llama-3.3-70b-instruct"
      rateMsatPerToken: 12
      contextWindow: 131072
    - id: "deepseek-r1"
      rateMsatPerToken: 18
      contextWindow: 65536
  payment:
    cashuMint: "https://mint.minibits.cash/Bitcoin"
    bolt11: true

Hot-Reconciliation & Non-Destructive Semantics

A central flaw in traditional relay configurations is that changing a filter drops all active connections. meshctl enforces non-destructive hot-reconciliation: