1.13. Sidecars & Pluggable Bridges
1.13.1. Sidecars: git, OCI and media hosting on a mesh node
A fleetmesh node can front three kinds of hosting without reimplementing any
of the protocols: git repositories (GRASP, NIP-34), OCI container images
(ncontainer) and a tagged media repository (magpie). All follow the same
shape - a node that wants to offer this work runs the relevant upstream
project as a sidecar, advertises it on its own descriptor, and optionally
prices access through the same Cashu/NUT-24 machinery docs/paid-fetch.md
already describes for Blossom blobs. This also makes a fleetmesh node a
plausible place to host fleetmesh’s own releases and other mesh-hosted
software.
1.13.1.1. Why a bridge, not a reimplementation
Three shapes were on the table:
Reimplement git-smart-http or the OCI distribution protocol inside the node. Rejected: it is a large, security-sensitive surface (git’s own CVE history is long) that upstream projects already maintain, test and patch. A reimplementation would mean carrying that burden forever for no functional gain over running the real thing.
A deep integration where the node’s own HTTP face terminates git/OCI requests directly. Rejected for the same reason, plus it would couple the node’s release cadence to two external wire protocols it does not control.
A thin bridge (what this is): run the off-the-shelf projects (
ngit-grasp, an ncontainer-compatible OCI gateway,magpie) as sidecars, each behind its own port, and have the node do three things only: declare the capability, publish where the sidecar lives, and gate access when the operator has priced it. This is the chosen design.
fleetmesh relays and serves the GRASP/ncontainer Nostr kinds (30617, 30618,
1617-1633, 30624) and magpie’s hoardstr sidecar notes (kind 1 with a
hoardstr-sidecar tag) the same way it does any other; it never
reinterprets their content. schema/kinds.json’s note on these kinds calls
this “fronted, not minted.”
1.13.1.2. What the node declares
A node with an enabled entry in the sidecars block of config.json
"sidecars": {
"grasp": {"enabled": true, "endpoint": "http://127.0.0.1:7861"},
"ncontainer": {"enabled": true, "endpoint": "http://127.0.0.1:7862"},
"magpie": {"enabled": true, "endpoint": "http://127.0.0.1:7863"}
}
adds the matching capability to its kind 11801 descriptor’s c tags:
["c", "https://fleetmesh.org/grasp/1.0"]
["c", "https://fleetmesh.org/ncontainer/1.0"]
["c", "https://fleetmesh.org/magpie/1.0"]
and a pointer to where the sidecar can be reached:
["grasp-endpoint", "https://node.example.org/git"]
["ncontainer-endpoint", "https://node.example.org/oci"]
["magpie-endpoint", "https://node.example.org/magpie"]
All are optional protocols (schema/constants.json’s optional_protocols),
never required of any node class the way peering/1.0 and budget/1.0 are.
A node that declares none of them carries no such tag and runs no sidecar -
this is strictly additive to the existing descriptor.
(Before the block existed these were top-level grasp:, ncontainer: and
magpie: keys. The node still reads that shape for one release; meshnode check names the move.)
fleetmesh_node/planes/services/sidecars/compartment.py is the compartment that owns this, but what a sidecar IS
is not in it. Each one is a manifest - a small JSON document naming the
capability, the endpoint tag and the self-tests that prove the sidecar
speaks its protocol (fleetmesh_node/planes/services/sidecars/manifest.py is the reader). The node
ships three in fleetmesh_node/planes/services/sidecars/manifests/; an operator adds one that is not
shipped with the same entry plus a manifest:
"sidecars": {
"myhost": {"enabled": true, "endpoint": "http://127.0.0.1:9300",
"manifest": "@/etc/fleetmesh/myhost.sidecar.json"}
}
and a plugin may declare one in code (Sidecars.declare). The rules are
the same for all three but one: only a shipped manifest may put its
capability under https://fleetmesh.org/, because those identifiers are
governed by the schema digest. Everyone else uses their own origin
(https://their.org/myhost/1.0), exactly as a dialect does. A manifest
with no checks, one accepting a 5xx, or one whose name is not the key it
is configured under is refused: the entry shows up in trouble() and
declares nothing. Adding a fourth shipped sidecar is a JSON file in
fleetmesh_node/planes/services/sidecars/manifests/ and the schema places schema/kinds.json’s kind
11801 note lists - not a change to fleetmesh_node/planes/services/sidecars/compartment.py.
1.13.1.2.1. Declared only while the self-tests pass
Setting sidecars.grasp.enabled is a wish, not a fact: the node has no authority over
a process it did not start. So the compartment never declares a capability
from config alone. Every row ships self-tests that exercise the protocol’s
own routes - the manifest’s checks - not merely “something answers HTTP”:
sidecar |
checks |
|---|---|
grasp |
|
ncontainer |
|
magpie |
|
A 5xx, a refused connection, a timeout, or the wrong content type fails the check. The tests run:
at start, before the compartments’ first tick, so a node never comes up advertising a sidecar it has not seen answer;
on cadence (every 300 s) from
tick();with no node at all:
meshnode sidecar-manifest FILE [--endpoint URL]lints a manifest and, given an endpoint, runs its checks from the shell - what a sidecar’s author does before any operator configures it (--init NAMEwrites a starting manifest).on demand:
meshnode sidecars DIR test [--name NAME](control verbsidecars.sidecar_test) runs them now and prints each check.
The c tag and <name>-endpoint tag appear on the kind 11801 only while
the last run passed. When the verified set changes - a sidecar starts
answering, stops, or is removed from config - the compartment rebuilds and
republishes the descriptor there and then (NodeContext.redescribe), so a
peer never dials a protocol nothing behind this node answers. A configured
sidecar whose tests fail is reported through trouble()/refusals with the
failing check and why, and meshnode sidecars DIR shows verified,
last_checked and the per-check results. On restart, a saved descriptor is
reused regardless of which optional protocols it carried; the startup run
then corrects it if the sidecar’s state differs.
1.13.1.3. Payment: the manifest says how a sidecar can be metered
The protocols meter very differently at the HTTP layer a reverse proxy actually sees, so a manifest declares which of three shapes applies, and the proxy is rendered from that - nothing else in the deployment names a sidecar:
"mount": "/git", # public path prefix (default /<name>)
"pricing": {"mode": "session"}, # none | session | blob
"run": {"image": "...", "port": 7861} # for the deployer; see "Deploying the sidecars"
none- the sidecar’s objects are already fetched through this node’s Blossom face (or are free); no gate in front of it. ncontainer.session- one 402 in front of the whole session, before the first request goes through. The price isfetch_pricing.sidecar_session_msat.<name>(grasp’s older spellinggrasp_session_msatstill wins for grasp) and the descriptor carries["price", "<name>-session", <msat>, "msat"]. grasp.blob- per content-addressed object:blob_routeis an anchored regex whose first group is the sha256 in the URL, and the node’s own per-blob 402 applies, so nothing new is priced. magpie.
The linter refuses a blob mode without a route, a route without a group, or a route on any other mode. The three shipped ones:
GRASP is priced per clone/fetch session, not per git object. A git-http negotiation is a back-and-forth of many small requests (
info/refs, severalupload-packrounds); a proxy sitting in front of it cannot cleanly meter individual objects the way it can a single Blossom blob fetch.PaidFetch.quote_session(subject, fetcher)issues one quote for an opaque session subject (grasp:<repo-identifier>), andgrasp_price_set/grasp_price_get(operator CLI / control-socket commands) set a flatgrasp_session_msat. The descriptor carries["price", "grasp-session", <msat>, "msat"]when this is configured.ncontainer is priced per blob, unchanged from how any Blossom fetch is priced today: every OCI layer and config blob the gateway serves resolves to a content-addressed sha256, and an ncontainer gateway built to fetch its blobs from a Blossom-compatible backend goes through the existing
PaidFetch.quote()/settle()path with no new code. There is noncontainer-sessionprice - the manifest (kind 30624) names a digest, and the resolved digest is priced exactly like any other blob.magpie is priced per file, and only for the bytes. Its API is many independent GETs; the metadata and search surface (
/api/v1/tags,/api/v1/search, file records) is free, and onlyGET /api/v1/files/<sha256>/contentis gated. The path already carries the sha256, so Caddy’sforward_authhands it toHEAD /<sha256>on the node’s face - the exact route a Blossom fetch is quoted on - and the existingPaidFetch.quote()/settle()apply unchanged. The gate is opt-in (sidecars.magpie.pricedin the inventory) because the node can only quote a blob it can stat: it holds when magpie’s CAS is the node’s blob store (see below) or the files are mirrored, and would refuse a file only magpie has.
1.13.1.4. How the gate actually works
The sidecars have no knowledge of Cashu or this node’s pricing config - that is deliberate; they stay off-the-shelf. Gating happens one layer up, in Caddy, in front of each sidecar:
node.example.org {
handle_path /git/* {
forward_auth 127.0.0.1:7777 {
uri /sidecar-session/grasp/{path.0}
copy_headers Authorization X-Cashu
}
reverse_proxy 127.0.0.1:7861 # ngit-grasp
}
handle_path /oci/* {
reverse_proxy 127.0.0.1:7862 # ncontainer gateway
}
handle_path /magpie/* {
@content path_regexp content ^/api/v1/files/([0-9a-f]{64})/content$
forward_auth @content 127.0.0.1:7777 { # only when magpie priced
uri /{re.content.1}
method HEAD
copy_headers Authorization X-Cashu
}
reverse_proxy 127.0.0.1:7863 # magpie
}
reverse_proxy 127.0.0.1:7777 # the node's own face
}
forward_auth calls back into the node’s own HTTP face at
GET /sidecar-session/<name>/<subject> (fleetmesh_node/protocols/nostr/blossom.py, alongside the
existing Blossom blob routes; /grasp-session/<subject> is the older
spelling for grasp - same quote/settle/402 machinery, same NUT-24
wire). It answers 200 (free, a partner, or already paid) or 402 with a
quote, exactly like a blob fetch; Caddy only forwards the request to
ngit-grasp on a 2xx. /oci/* carries no forward_auth because the
gateway’s own blob fetches already go through this node’s priced Blossom
path when configured to use it as a backend. /magpie/* gates only the
content route, and only when the instance sets magpie priced.
1.13.1.5. magpie and the node’s blob store
magpie keeps its own content-addressed store: <data>/objects/<ab>/<cd>/<sha256>
(fanout depth 2 by default, --fanout 0..8), with main.db and
activity.db beside it. The node’s BlobStore is
<node>/blobs/data/<ab>/<sha256> (depth 1) with blobs.sqlite beside it.
The two are separate by default: a file magpie should serve that lives on
the mesh is claimed by Blossom URL (POST /api/v1/files/claim), which
fetches it through the node’s own priced face like any other client.
Sharing one tree is an opt-in. BlobStore takes its layout from the blobs
config block:
"blobs": {"fanout": 2, "data_dir": "/var/lib/meshnode-magpie/<instance>/objects"}
fanout is the number of two-hex-character directory levels (0..8, default 1);
data_dir is where the blobs live (default <node>/blobs/data). The index
(blobs.sqlite) always stays beside the node. Both values are recorded on
first open and a later open with different ones is refused
(blobstore.LayoutMismatch) - a silent depth change would 404 every blob the
store holds. What sharing does and does not give you:
Shared disk is not a shared catalogue. Each side indexes only the files it wrote; a blob magpie ingested is on disk but unknown to
HEAD /<sha>until something tells the node about it, and vice versa. The honest shape is one writer per tree, the other reading.magpie writes 0444 files under 0755 directories; the node must run as the same user (or group with read/unlink) to serve and evict them.
The node’s retention eviction unlinks files; magpie’s
main.dbwould then point at nothing. Pin what magpie owns, or cap the node’s store so it never evicts in a shared tree.Recovery sweeps differ (magpie quarantines, the node re-stats). Neither repairs the other’s index.
So: separate stores by default, shared root when the operator understands the catalogue split and runs one writer. Not the default and not a goal of the bridge.
magpie’s own Nostr side (NIP-98 request signing, hoardstr sidecar notes
for federated tags) needs only a relay: point its relay client at this
node’s face (ws://127.0.0.1:<face_port>) and the notes travel with
everything else the node relays. Nothing in fleetmesh_node/ reads them.
Considered and set aside: tying blob placement or fanout to the node’s kind 30803 shard names. A shard is a filter over events plus a time range, reconciled by fingerprint; a sha256 prefix is neither, and blob/1.0 already has its own placement path (have/offer/pull). The two should not share a coordinate.
1.13.1.6. Deploying the sidecars
roles/meshnode in the ops repository provisions each as a Docker container under systemd,
fleetmesh-<name>@<instance>.service, keyed by the manifest name - the same
key as the node’s sidecars.<name> block and the Caddy route:
meshnode_instances:
- name: "meshnode"
enable_caddy: true
domain: "node.example.org"
sidecars:
grasp: {enabled: true, session_msat: 2000}
ncontainer: {enabled: true}
magpie: {enabled: true, priced: false}
(The pre-table keys grasp_enabled, grasp_session_msat,
ncontainer_enabled, magpie_enabled, magpie_priced are read for one
release more.) How each one runs is the run block of its manifest -
image, port, data path, env/cmd - so the manifest is the whole contract and
there is no second table to keep in step: the role reads the manifests the
controller can see (the shipped ones in fleetmesh_node/planes/services/sidecars/manifests/, then
fleetmesh_sidecar_manifest_dirs) and renders the unit and the Caddy route
from them. A third-party sidecar is one manifest file. Per host,
fleetmesh_sidecar_overrides.<name>: {image, port, ...} lays a pin over the
manifest’s defaults; a manifest with no run is a sidecar the operator
runs by hand (route, no unit). The node itself never reads run. Docker is
only installed on a host that enables at least one of them - a node running
none carries no container runtime. magpie is a single static Go binary
(CGO_ENABLED=0 go build ./cmd/magpie); there is no upstream image, so its
manifest says magpie:latest and the override names whatever the operator
built from a reviewed commit. It binds loopback unless told
--allow-public, which the manifest’s cmd passes because the container’s
loopback is not the host’s - the published port is still bound to
127.0.0.1 on the host. Image tags in the shipped manifests are :latest,
which is the explicit “override before production” value, not a
recommendation: pin a reviewed release tag per host the same way this
project pins its FIPS/NPACK dependency versions (a manually reviewed
commit, never automated).
A node operator who does not want Docker on the host at all can still run
either sidecar by hand (or under its own systemd unit) and simply set
grasp.endpoint/ncontainer.endpoint/magpie.endpoint directly in
config.json - the Ansible role is a convenience, not a requirement;
fleetmesh_node/planes/services/sidecars/compartment.py only cares that the configured endpoint answers.
For the step-by-step a sidecar author follows, see docs/agents/SIDECAR.md.
1.13.1.7. Rhio Cluster Sync Sidecar (fleetmesh-rhio)
The Rhio sidecar daemon (meshnode/fleetmesh_node/sidecar/fleetmesh_rhio.py) coordinates micro-datacenter clustering and blob replication:
Execution: Run as a decoupled standalone daemon or systemd service:
fleetmesh-rhio /var/lib/fleetmesh/node-1 --nats localhost:4222 --s3 http://minio.local:9000 --network-id cluster-alpha
IPC Protocol (
fleetmesh-rhio/1): Communicates withmeshnodeover a local Unix domain socket (rhio.sock) using newline-delimited JSON.Supported IPC Commands:
status: Returns sidecar health, connected status, NATS/S3 endpoints, and blob/message totals.publish: Dispatches message to NATS JetStream subject (fleetmesh.<pubkey>).store_blob: Stores base64 binary payload and calculates SHA-256 digest.get_blob: Retrieves stored binary blob by SHA-256 digest.sync_state: Reports multi-node cluster synchronization state.
Fault Isolation & Recovery:
RhioSidecarClientprovides automatic socket cleanup and reconnect recovery on unexpected disconnects or broker resets.
1.13.1.8. Tests
fleetmesh_node/tests/test_paid_fetch.py- the GRASP session-pricing test group: default-free,grasp_price_set/get,quote_sessionshape, partner-free fetches, and settlement against a fake mint.fleetmesh_node/tests/test_sidecars.py- the compartment alone: nothing declared from config, declared only after a passing run at start, cadence and on-demand runs, redeclare on gain/loss/removal, and the real checks against a stdlib HTTP stub (git advertisement content type,/v2/, magpie files route, non-http schemes never dialled).fleetmesh_node/tests/test_rhio_sidecar.py- the Rhio sidecar daemon and IPC client: status, publish, store_blob, get_blob, sync_state, and automatic socket reconnection recovery.fleetmesh_node/tests/test_rhio_package.py- live HTTP ingress, Bao Merkle tree verification,INSERT OR IGNOREsync state retention, and TransportRouter rank 12 integration.fleetmesh_node/tests/test_extensions.py- a live node with a dead grasp endpoint publishes nograsp/1.0;sidecars.sidecar_testover the control socket verifies a stub git server and republishes; restart reuses or corrects.core/tests/test_unplug.py- confirmsgrasp/1.0,ncontainer/1.0andmagpie/1.0are declared identifiers (schema/messages.json’sreused_protocols), not undeclared hosts a node would have to reach unprompted.