1.8. Updates & Release Verification

1.8.1. Updates: what a node does about versions it did not write

Operators asked for a node they do not have to watch: set it to run the latest compatible release, or pin one, and stop reading changelogs. This document says how far that goes, where it stops, and why.

1.8.1.1. The stance

Check and report, apply only within a policy the operator set, restart rather than hot-swap, pinned by default.

Three things this project already does rule out the naive version (“the node fetches the latest and runs it”):

  1. Every dependency pin in this tree - the FIPS and NPACK versions, the GRASP and ncontainer sidecar images, the schema digest - is a manually reviewed commit. scripts/dep-version-check.py detects a newer upstream and says so; it never bumps. A node that pulls code at runtime is the auto-bump this project refused everywhere else.

  2. A compartment runs in-process with the node’s keys. Fetching code over the network and executing it is remote code execution by design. The only question is who holds the key that signs what is fetched, and the answer has to be “a key the operator chose”, not “whoever published last”.

  3. A node that changes behaviour without the operator’s say-so is one the operator no longer controls. A fleetmesh node’s whole point is that its operator does control it.

So the design gives the operator what they asked for - not having to watch - by letting them approve a policy and a key once, instead of each version.

1.8.1.2. Tiers

1.8.1.2.1. Tier 1 - compatibility declaration (shipped)

Every compartment declares api = 'subsystem/1' and, if it reads a schema, protocol = <digest>. Registry.assemble() refuses a compartment whose API major differs or whose protocol digest is not the one this node speaks, with both values in the message. See docs/compartments.md.

This is the precondition for anything else here. Without it, “compatible” means “happened to import”; with it, a node can say exactly why it will not load something, and a compartment built outside the tree can say exactly what it expects.

1.8.1.2.2. Tier 2 - signed release channel, operator policy (check-and-report shipped)

A release is published on fleetmesh: a kind 30617 repository announcement and kind 1617/30618 state under the project’s GRASP identity, with the artefact (a wheel, a tarball, or an OCI manifest for the sidecars) as a Blossom blob - the same GRASP/ncontainer path docs/sidecars.md describes, used for the node’s own releases. The release event is signed by a release key.

The operator pins that key and chooses a policy per compartment, or for the whole node:

"updates": {
  "release_key": "<64-hex pubkey of the release signer>",
  "channel": ["wss://relay.example.org"],
  "check_every": 21600,
  "policy": { "*": "pinned", "reporter": "track:patch", "dvm": "track:minor" },
  "pinned": "0.9.3"
}

"updates": true is the short form: the key that made this node, the node’s own relays as the channel, a check every six hours, pinned everywhere. Without the block the node checks nothing - an operator opts into being told.

“The key that made me”: scripts/release.sh stamps the publisher’s pubkey into fleetmesh_node/PUBLISHER of every packaged tree (a checkout reads it from packages/fleetmesh-node/.npack/manifest.json), and updates.release_key defaults to it. A fork’s build therefore listens to the fork’s signer with nobody editing anything; a build with no stamp falls back to the project key (PROJECT_RELEASE_KEY). It is a plain config value: write release_key to listen to someone else, and updates.check / state() always show which key is in effect. What the tree cannot do is change it - a release event carries no key for the node to adopt.

What the node knows about itself lives in fleetmesh_node/release.py: fleetmesh_node/RELEASE is the semver (edited by a reviewed release commit, like every other pin here), fleetmesh_node/COMMIT is stamped by scripts/release.sh into a packaged tree, and a tree with .git is source: checkout - a developer’s tree, which the node reports on and never installs into. meshnode api and status.json carry the release block.

The check: ask the channel for kind 9900 release events by the pinned key under the name fleetmesh-node; verify each signature; refuse (and count) anything by another key, with no artefact hash, or declaring no provides fleetmesh/0.1-draft@<digest>; among what is left, the newest semver above what this node runs (pinned if the operator said, else RELEASE) that speaks this node’s digest is newest_compatible; the newest that speaks another digest is incompatible_newer - reported, never an update, because the schema is a reviewed commitment (docs/versioning.md). What was seen and acknowledged sits in updates.sqlite so a restart does not re-announce. The commands go over the control socket - updates.check, updates.show, updates.ack with version - and meshnode health carries the trouble line.

The query is kind 9900 by author only. Relays index single-letter tags (NIP-01), so a #name filter returns nothing on every public relay; the node asks for the key’s releases and keeps the ones named fleetmesh-node. The key’s other packages (fleetmesh-schema) are skipped, not counted as refusals.

Soaked on a 1 GB / 1 vCPU droplet (debian 13, python 3.13) running the packaged tree: the node reports fleetmesh-node 0.1.0 (release, <commit>), sees the 0.1.0 release event on nos.lol and relay.primal.net, and when a 0.1.1 was published said fleetmesh-node 0.1.1 available (patch); you run 0.1.0; policy pinned in health until acked. 32 MB resident.

The release itself: packages/fleetmesh-node/.npack/manifest.json is the package (payload fleetmesh_node/ and fleetmesh_core/, with fleetmesh_node/COMMIT and fleetmesh_node/PROTOCOL stamped by scripts/release.sh); fleetmesh_node/RELEASE and the manifest version must agree or release.sh refuses - one reviewed commit moves both. The sign job builds it on every protected push; release publishes on a tag.

Cutting a release:

  1. One reviewed commit bumps fleetmesh_node/RELEASE and packages/fleetmesh-node/.npack/manifest.json version together.

  2. Tag that commit v<version> and push the tag. release.sh refuses a tag that does not name the version the tree says it is.

  3. Run the manual release job on the tag’s pipeline. It publishes both packages, but asks the relays first (scripts/release_live.py) and skips a name/version that is already live and not revoked - so re-running the job, or tagging a node release while the schema did not move, never puts a second kind 9900 for a live version on the relays.

Pulling one back: scripts/revoke_release.py <release event id> "<reason>" --publish signs a kind 9901 naming the release; the node asks for 9900 and 9901 together and a revoked release is not an update, whichever order the relays return them. Revocations are kept in updates.sqlite and shown in updates.show. The throwaway 0.1.1 from the soak was pulled this way.

  • pinned - the default, and in this release the only policy. The node reports a newer compatible patch or minor through trouble() (“fleetmesh-node 0.9.4 available (patch); you run 0.9.3; policy pinned - meshnode updates.show for the release”) and does nothing else. A major is in state() but not in trouble(): an operator who chose pinned is told what they could move to without a protocol or API change, not shouted at about a jump they would have to opt into. updates.ack silences one version; a newer one speaks again.

  • track:patch / track:minor - not yet. The validator refuses the line by name (updates.policy: 'track:patch' is not available yet; this release checks and reports) so an operator who writes it learns it does nothing, instead of believing it does. When it arrives, the node applies a release within that range when: the release event verifies against the pinned key; the artefact’s hash matches the event; the new compartment’s api and protocol pass the Tier 1 check; and it is at a restart the node schedules. It never applies a major, and never applies anything the operator did not put a track on.

Applying means: fetch the artefact to the node directory, verify, write the new version into the manifest the node loads from, request a restart through systemd (systemctl restart fleetmesh-node@<instance> via the control socket’s existing privileges), and say so in the journal. If the restarted node refuses the compartment at assembly, the previous version is still on disk and the unit’s restart policy brings it back; the refusal is in trouble().

What this does not do: run anything it just downloaded without a restart; touch a compartment whose policy is pinned; trust a release event signed by any key but the pinned one; apply a release whose protocol differs from the schema this node speaks.

1.8.1.2.3. Tier 3 - hot reload (not planned)

Swapping a compartment in a running node means handing its state across versions: sqlite schemas it owns, in-memory meters, open DIDComm threads, half-published findings. The hardest bugs budget and reporter have had were state-continuity bugs within one version. A restart is a clean boundary, the systemd units exist, and a node that is down for the two seconds a restart takes is not a node anyone will notice. Not worth it.

1.8.1.3. What this does for independent development

The real enabler for a compartment developed outside the tree is Tier 1 plus the typed NodeContext: a stable constructor contract, a declared API version, a refusal with a reason when it does not match, and a test harness (fleetmesh_node/tests/test_assemble.py) that exercises the assembler without a NodeApp. Someone can write a compartment, test it against the registry, and hand it to an operator who adds it to their manifest.

Tier 2 makes that compartment updatable on the operator’s terms, and dogfoods the hosting this project ships: a fleetmesh node’s releases live on a fleetmesh node.

The risk is the version matrix. Twenty-four compartments from one tree have one version; twenty-four from several authors have many. Tier 1’s refusal-with-reason is what keeps “compatible” honest - a node never ends up running “whatever resolved”. It will still be the operator’s job to decide whether a third-party compartment is worth running, and nothing here should pretend otherwise.

1.8.1.4. Status

tier

state

1 - compatibility declaration

shipped (Compartment.api, Compartment.protocol, Registry.assemble)

2 - check-and-report

shipped (fleetmesh_node/updates/, fleetmesh_node/release.py, UpdatesConfig)

2 - apply within track

after check-and-report has run on real nodes for a while

3 - hot reload

not planned