1.13. Sidecars & Pluggable Bridges

1.13.1. Sidecars: git, OCI and media hosting on a mesh node

A fleetmesh node can front three kinds of hosting without reimplementing any of the protocols: git repositories (GRASP, NIP-34), OCI container images (ncontainer) and a tagged media repository (magpie). All follow the same shape - a node that wants to offer this work runs the relevant upstream project as a sidecar, advertises it on its own descriptor, and optionally prices access through the same Cashu/NUT-24 machinery docs/paid-fetch.md already describes for Blossom blobs. This also makes a fleetmesh node a plausible place to host fleetmesh’s own releases and other mesh-hosted software.

1.13.1.1. Why a bridge, not a reimplementation

Three shapes were on the table:

  • Reimplement git-smart-http or the OCI distribution protocol inside the node. Rejected: it is a large, security-sensitive surface (git’s own CVE history is long) that upstream projects already maintain, test and patch. A reimplementation would mean carrying that burden forever for no functional gain over running the real thing.

  • A deep integration where the node’s own HTTP face terminates git/OCI requests directly. Rejected for the same reason, plus it would couple the node’s release cadence to two external wire protocols it does not control.

  • A thin bridge (what this is): run the off-the-shelf projects (ngit-grasp, an ncontainer-compatible OCI gateway, magpie) as sidecars, each behind its own port, and have the node do three things only: declare the capability, publish where the sidecar lives, and gate access when the operator has priced it. This is the chosen design.

fleetmesh relays and serves the GRASP/ncontainer Nostr kinds (30617, 30618, 1617-1633, 30624) and magpie’s hoardstr sidecar notes (kind 1 with a hoardstr-sidecar tag) the same way it does any other; it never reinterprets their content. schema/kinds.json’s note on these kinds calls this “fronted, not minted.”

1.13.1.2. What the node declares

A node with an enabled entry in the sidecars block of config.json

"sidecars": {
  "grasp":      {"enabled": true, "endpoint": "http://127.0.0.1:7861"},
  "ncontainer": {"enabled": true, "endpoint": "http://127.0.0.1:7862"},
  "magpie":     {"enabled": true, "endpoint": "http://127.0.0.1:7863"}
}

adds the matching capability to its kind 11801 descriptor’s c tags:

["c", "https://fleetmesh.org/grasp/1.0"]
["c", "https://fleetmesh.org/ncontainer/1.0"]
["c", "https://fleetmesh.org/magpie/1.0"]

and a pointer to where the sidecar can be reached:

["grasp-endpoint", "https://node.example.org/git"]
["ncontainer-endpoint", "https://node.example.org/oci"]
["magpie-endpoint", "https://node.example.org/magpie"]

All are optional protocols (schema/constants.json’s optional_protocols), never required of any node class the way peering/1.0 and budget/1.0 are. A node that declares none of them carries no such tag and runs no sidecar - this is strictly additive to the existing descriptor.

(Before the block existed these were top-level grasp:, ncontainer: and magpie: keys. The node still reads that shape for one release; meshnode check names the move.)

fleetmesh_node/planes/services/sidecars/compartment.py is the compartment that owns this, but what a sidecar IS is not in it. Each one is a manifest - a small JSON document naming the capability, the endpoint tag and the self-tests that prove the sidecar speaks its protocol (fleetmesh_node/planes/services/sidecars/manifest.py is the reader). The node ships three in fleetmesh_node/planes/services/sidecars/manifests/; an operator adds one that is not shipped with the same entry plus a manifest:

"sidecars": {
  "myhost": {"enabled": true, "endpoint": "http://127.0.0.1:9300",
             "manifest": "@/etc/fleetmesh/myhost.sidecar.json"}
}

and a plugin may declare one in code (Sidecars.declare). The rules are the same for all three but one: only a shipped manifest may put its capability under https://fleetmesh.org/, because those identifiers are governed by the schema digest. Everyone else uses their own origin (https://their.org/myhost/1.0), exactly as a dialect does. A manifest with no checks, one accepting a 5xx, or one whose name is not the key it is configured under is refused: the entry shows up in trouble() and declares nothing. Adding a fourth shipped sidecar is a JSON file in fleetmesh_node/planes/services/sidecars/manifests/ and the schema places schema/kinds.json’s kind 11801 note lists - not a change to fleetmesh_node/planes/services/sidecars/compartment.py.

1.13.1.2.1. Declared only while the self-tests pass

Setting sidecars.grasp.enabled is a wish, not a fact: the node has no authority over a process it did not start. So the compartment never declares a capability from config alone. Every row ships self-tests that exercise the protocol’s own routes - the manifest’s checks - not merely “something answers HTTP”:

sidecar

checks

grasp

HEAD /; GET /<probe>.git/info/refs?service=git-upload-pack answering 200 with application/x-git-upload-pack-advertisement, or 404 (git handler: no such repo)

ncontainer

HEAD /; GET /v2/ answering 200 or 401 (the one route the OCI distribution spec mandates)

magpie

HEAD /; HEAD /api/v1/files/<sha256>/content answering 200/402/404 from the files handler

A 5xx, a refused connection, a timeout, or the wrong content type fails the check. The tests run:

  • at start, before the compartments’ first tick, so a node never comes up advertising a sidecar it has not seen answer;

  • on cadence (every 300 s) from tick();

  • with no node at all: meshnode sidecar-manifest FILE [--endpoint URL] lints a manifest and, given an endpoint, runs its checks from the shell - what a sidecar’s author does before any operator configures it (--init NAME writes a starting manifest).

  • on demand: meshnode sidecars DIR test [--name NAME] (control verb sidecars.sidecar_test) runs them now and prints each check.

The c tag and <name>-endpoint tag appear on the kind 11801 only while the last run passed. When the verified set changes - a sidecar starts answering, stops, or is removed from config - the compartment rebuilds and republishes the descriptor there and then (NodeContext.redescribe), so a peer never dials a protocol nothing behind this node answers. A configured sidecar whose tests fail is reported through trouble()/refusals with the failing check and why, and meshnode sidecars DIR shows verified, last_checked and the per-check results. On restart, a saved descriptor is reused regardless of which optional protocols it carried; the startup run then corrects it if the sidecar’s state differs.

1.13.1.3. Payment: the manifest says how a sidecar can be metered

The protocols meter very differently at the HTTP layer a reverse proxy actually sees, so a manifest declares which of three shapes applies, and the proxy is rendered from that - nothing else in the deployment names a sidecar:

"mount": "/git",                       # public path prefix (default /<name>)
"pricing": {"mode": "session"},        # none | session | blob
"run": {"image": "...", "port": 7861}  # for the deployer; see "Deploying the sidecars"
  • none - the sidecar’s objects are already fetched through this node’s Blossom face (or are free); no gate in front of it. ncontainer.

  • session - one 402 in front of the whole session, before the first request goes through. The price is fetch_pricing.sidecar_session_msat.<name> (grasp’s older spelling grasp_session_msat still wins for grasp) and the descriptor carries ["price", "<name>-session", <msat>, "msat"]. grasp.

  • blob - per content-addressed object: blob_route is an anchored regex whose first group is the sha256 in the URL, and the node’s own per-blob 402 applies, so nothing new is priced. magpie.

The linter refuses a blob mode without a route, a route without a group, or a route on any other mode. The three shipped ones:

  • GRASP is priced per clone/fetch session, not per git object. A git-http negotiation is a back-and-forth of many small requests (info/refs, several upload-pack rounds); a proxy sitting in front of it cannot cleanly meter individual objects the way it can a single Blossom blob fetch. PaidFetch.quote_session(subject, fetcher) issues one quote for an opaque session subject (grasp:<repo-identifier>), and grasp_price_set/grasp_price_get (operator CLI / control-socket commands) set a flat grasp_session_msat. The descriptor carries ["price", "grasp-session", <msat>, "msat"] when this is configured.

  • ncontainer is priced per blob, unchanged from how any Blossom fetch is priced today: every OCI layer and config blob the gateway serves resolves to a content-addressed sha256, and an ncontainer gateway built to fetch its blobs from a Blossom-compatible backend goes through the existing PaidFetch.quote()/settle() path with no new code. There is no ncontainer-session price - the manifest (kind 30624) names a digest, and the resolved digest is priced exactly like any other blob.

  • magpie is priced per file, and only for the bytes. Its API is many independent GETs; the metadata and search surface (/api/v1/tags, /api/v1/search, file records) is free, and only GET /api/v1/files/<sha256>/content is gated. The path already carries the sha256, so Caddy’s forward_auth hands it to HEAD /<sha256> on the node’s face - the exact route a Blossom fetch is quoted on - and the existing PaidFetch.quote()/settle() apply unchanged. The gate is opt-in (sidecars.magpie.priced in the inventory) because the node can only quote a blob it can stat: it holds when magpie’s CAS is the node’s blob store (see below) or the files are mirrored, and would refuse a file only magpie has.

1.13.1.4. How the gate actually works

The sidecars have no knowledge of Cashu or this node’s pricing config - that is deliberate; they stay off-the-shelf. Gating happens one layer up, in Caddy, in front of each sidecar:

node.example.org {
    handle_path /git/* {
        forward_auth 127.0.0.1:7777 {
            uri /sidecar-session/grasp/{path.0}
            copy_headers Authorization X-Cashu
        }
        reverse_proxy 127.0.0.1:7861   # ngit-grasp
    }
    handle_path /oci/* {
        reverse_proxy 127.0.0.1:7862   # ncontainer gateway
    }
    handle_path /magpie/* {
        @content path_regexp content ^/api/v1/files/([0-9a-f]{64})/content$
        forward_auth @content 127.0.0.1:7777 {   # only when magpie priced
            uri /{re.content.1}
            method HEAD
            copy_headers Authorization X-Cashu
        }
        reverse_proxy 127.0.0.1:7863   # magpie
    }
    reverse_proxy 127.0.0.1:7777        # the node's own face
}

forward_auth calls back into the node’s own HTTP face at GET /sidecar-session/<name>/<subject> (fleetmesh_node/protocols/nostr/blossom.py, alongside the existing Blossom blob routes; /grasp-session/<subject> is the older spelling for grasp - same quote/settle/402 machinery, same NUT-24 wire). It answers 200 (free, a partner, or already paid) or 402 with a quote, exactly like a blob fetch; Caddy only forwards the request to ngit-grasp on a 2xx. /oci/* carries no forward_auth because the gateway’s own blob fetches already go through this node’s priced Blossom path when configured to use it as a backend. /magpie/* gates only the content route, and only when the instance sets magpie priced.

1.13.1.5. magpie and the node’s blob store

magpie keeps its own content-addressed store: <data>/objects/<ab>/<cd>/<sha256> (fanout depth 2 by default, --fanout 0..8), with main.db and activity.db beside it. The node’s BlobStore is <node>/blobs/data/<ab>/<sha256> (depth 1) with blobs.sqlite beside it. The two are separate by default: a file magpie should serve that lives on the mesh is claimed by Blossom URL (POST /api/v1/files/claim), which fetches it through the node’s own priced face like any other client.

Sharing one tree is an opt-in. BlobStore takes its layout from the blobs config block:

"blobs": {"fanout": 2, "data_dir": "/var/lib/meshnode-magpie/<instance>/objects"}

fanout is the number of two-hex-character directory levels (0..8, default 1); data_dir is where the blobs live (default <node>/blobs/data). The index (blobs.sqlite) always stays beside the node. Both values are recorded on first open and a later open with different ones is refused (blobstore.LayoutMismatch) - a silent depth change would 404 every blob the store holds. What sharing does and does not give you:

  • Shared disk is not a shared catalogue. Each side indexes only the files it wrote; a blob magpie ingested is on disk but unknown to HEAD /<sha> until something tells the node about it, and vice versa. The honest shape is one writer per tree, the other reading.

  • magpie writes 0444 files under 0755 directories; the node must run as the same user (or group with read/unlink) to serve and evict them.

  • The node’s retention eviction unlinks files; magpie’s main.db would then point at nothing. Pin what magpie owns, or cap the node’s store so it never evicts in a shared tree.

  • Recovery sweeps differ (magpie quarantines, the node re-stats). Neither repairs the other’s index.

So: separate stores by default, shared root when the operator understands the catalogue split and runs one writer. Not the default and not a goal of the bridge.

magpie’s own Nostr side (NIP-98 request signing, hoardstr sidecar notes for federated tags) needs only a relay: point its relay client at this node’s face (ws://127.0.0.1:<face_port>) and the notes travel with everything else the node relays. Nothing in fleetmesh_node/ reads them.

Considered and set aside: tying blob placement or fanout to the node’s kind 30803 shard names. A shard is a filter over events plus a time range, reconciled by fingerprint; a sha256 prefix is neither, and blob/1.0 already has its own placement path (have/offer/pull). The two should not share a coordinate.

1.13.1.6. Deploying the sidecars

roles/meshnode in the ops repository provisions each as a Docker container under systemd, fleetmesh-<name>@<instance>.service, keyed by the manifest name - the same key as the node’s sidecars.<name> block and the Caddy route:

meshnode_instances:
  - name: "meshnode"
    enable_caddy: true
    domain: "node.example.org"
    sidecars:
      grasp:      {enabled: true, session_msat: 2000}
      ncontainer: {enabled: true}
      magpie:     {enabled: true, priced: false}

(The pre-table keys grasp_enabled, grasp_session_msat, ncontainer_enabled, magpie_enabled, magpie_priced are read for one release more.) How each one runs is the run block of its manifest - image, port, data path, env/cmd - so the manifest is the whole contract and there is no second table to keep in step: the role reads the manifests the controller can see (the shipped ones in fleetmesh_node/planes/services/sidecars/manifests/, then fleetmesh_sidecar_manifest_dirs) and renders the unit and the Caddy route from them. A third-party sidecar is one manifest file. Per host, fleetmesh_sidecar_overrides.<name>: {image, port, ...} lays a pin over the manifest’s defaults; a manifest with no run is a sidecar the operator runs by hand (route, no unit). The node itself never reads run. Docker is only installed on a host that enables at least one of them - a node running none carries no container runtime. magpie is a single static Go binary (CGO_ENABLED=0 go build ./cmd/magpie); there is no upstream image, so its manifest says magpie:latest and the override names whatever the operator built from a reviewed commit. It binds loopback unless told --allow-public, which the manifest’s cmd passes because the container’s loopback is not the host’s - the published port is still bound to 127.0.0.1 on the host. Image tags in the shipped manifests are :latest, which is the explicit “override before production” value, not a recommendation: pin a reviewed release tag per host the same way this project pins its FIPS/NPACK dependency versions (a manually reviewed commit, never automated).

A node operator who does not want Docker on the host at all can still run either sidecar by hand (or under its own systemd unit) and simply set grasp.endpoint/ncontainer.endpoint/magpie.endpoint directly in config.json - the Ansible role is a convenience, not a requirement; fleetmesh_node/planes/services/sidecars/compartment.py only cares that the configured endpoint answers.

For the step-by-step a sidecar author follows, see docs/agents/SIDECAR.md.

1.13.1.7. Rhio Cluster Sync Sidecar (fleetmesh-rhio)

The Rhio sidecar daemon (meshnode/fleetmesh_node/sidecar/fleetmesh_rhio.py) coordinates micro-datacenter clustering and blob replication:

  • Execution: Run as a decoupled standalone daemon or systemd service:

    fleetmesh-rhio /var/lib/fleetmesh/node-1 --nats localhost:4222 --s3 http://minio.local:9000 --network-id cluster-alpha
    
  • IPC Protocol (fleetmesh-rhio/1): Communicates with meshnode over a local Unix domain socket (rhio.sock) using newline-delimited JSON.

  • Supported IPC Commands:

    • status: Returns sidecar health, connected status, NATS/S3 endpoints, and blob/message totals.

    • publish: Dispatches message to NATS JetStream subject (fleetmesh.<pubkey>).

    • store_blob: Stores base64 binary payload and calculates SHA-256 digest.

    • get_blob: Retrieves stored binary blob by SHA-256 digest.

    • sync_state: Reports multi-node cluster synchronization state.

  • Fault Isolation & Recovery: RhioSidecarClient provides automatic socket cleanup and reconnect recovery on unexpected disconnects or broker resets.

1.13.1.8. Tests

  • fleetmesh_node/tests/test_paid_fetch.py - the GRASP session-pricing test group: default-free, grasp_price_set/get, quote_session shape, partner-free fetches, and settlement against a fake mint.

  • fleetmesh_node/tests/test_sidecars.py - the compartment alone: nothing declared from config, declared only after a passing run at start, cadence and on-demand runs, redeclare on gain/loss/removal, and the real checks against a stdlib HTTP stub (git advertisement content type, /v2/, magpie files route, non-http schemes never dialled).

  • fleetmesh_node/tests/test_rhio_sidecar.py - the Rhio sidecar daemon and IPC client: status, publish, store_blob, get_blob, sync_state, and automatic socket reconnection recovery.

  • fleetmesh_node/tests/test_rhio_package.py - live HTTP ingress, Bao Merkle tree verification, INSERT OR IGNORE sync state retention, and TransportRouter rank 12 integration.

  • fleetmesh_node/tests/test_extensions.py - a live node with a dead grasp endpoint publishes no grasp/1.0; sidecars.sidecar_test over the control socket verifies a stub git server and republishes; restart reuses or corrects.

  • core/tests/test_unplug.py - confirms grasp/1.0, ncontainer/1.0 and magpie/1.0 are declared identifiers (schema/messages.json’s reused_protocols), not undeclared hosts a node would have to reach unprompted.