5. Fleet Monitoring & Supervision (meshctl watch)
Continuous operational monitoring is vital for maintaining mesh health and preventing silent failures. While unit tests verify code against known edge cases, live fleet operations can uncover subtle hardware and networking issues—such as mid-window receipt chain breaks, stale bilateral grants, or relay payload size cap rejections.
meshctl watch provides operators with a single-screen real-time health dashboard that inspects one or many Fleetmesh nodes simultaneously.
5.1. Operational Philosophy
Zero Remote Attack Surface: A Fleetmesh node does not expose an unauthenticated HTTP or TCP administrative management port. Its administrative interface is strictly a local Unix domain socket (
meshnode-control/1). To monitor remote nodes, meshctl watch connects over standard SSH (e.g.ssh root@node-a.fleet meshctl watch --json /var/lib/meshnode). One static Rust binary on each machine is all that is required.Exit Status Drives Automation: meshctl watch is engineered to integrate cleanly into standard Unix scheduling systems (e.g. systemd timers, cron, Prometheus node-exporter): - Exit 0: The mesh is completely healthy. Zero issues to report. - Exit 1: Something is wrong that requires operator attention (broken receipt chain, compartment in trouble, unpayable budget, or protocol digest mismatch). - Exit 2: A target node could not be reached over network or SSH.
5.2. Systemd Timer Integration
Fleetmesh includes systemd service and timer units in contrib/meshwatch.service and contrib/meshwatch.timer. Because exit status indicates health, a failed run triggers a failed systemd unit, which existing monitoring infrastructure (Prometheus, Nagios, Datadog) already observes:
# /etc/systemd/system/meshwatch.service
[Unit]
Description=Fleetmesh Health Watchdog
After=network.target
[Service]
Type=oneshot
ExecStart=/usr/local/bin/meshctl watch --quiet root@node-1:/var/lib/fleetmesh root@node-2:/var/lib/fleetmesh
# /etc/systemd/system/meshwatch.timer
[Unit]
Description=Run meshctl watch every 10 minutes
[Timer]
OnBootSec=5min
OnUnitActiveSec=10min
[Install]
WantedBy=timers.target
5.3. What meshctl watch Checks
Every invocation inspects the following invariants across all target nodes:
Compartment Health: Verifies that all active compartments (transport, eventstore, keystore, peering, settlement) report healthy status with zero unhandled exceptions.
Relay Subscriptions: Ensures inbound and outbound WebSocket relay connections are active and draining.
Bilateral Grants: Validates that peering grants are live, budgets are not exhausted, and window pricing remains within nominal bounds (
S0ladder).Throttling Detection: Reports throttled peers as diagnostic informational notes (not fatal faults, as throttling is normal behavior when bandwidth grants bind).
Protocol Digest Agreement: The most critical check: Compares the cryptographic protocol digest (
schema/DIGEST) reported by every node. If two nodes do not agree on the protocol digest, meshctl watch flags an immediate critical fault. Protocol divergence costs peering tiers and prevents consensus.
5.4. CI Pipeline & Published Spec Validation
meshctl watch can also monitor upstream Gitlab CI pipelines and published protocol specifications, consolidating software delivery status with live mesh state:
export FLEETMESH_CI=https://gitlab.example.com/api/v4/projects/141
export FLEETMESH_CI_TOKEN_FILE=~/.gitlab-token
export FLEETMESH_SITE=https://fleetmesh.org/spec
export FLEETMESH_PUBLISHER=npub1...
meshctl watch root@node-a:/var/lib/fleetmesh root@node-b:/var/lib/fleetmesh
When configured:
- If a CI build breaks, it is surfaced directly in the watch output.
- If the node’s running code digest differs from the published spec on FLEETMESH_SITE, meshctl watch names the required remediation (deploy, republish, or roll).
- If the CI API is unreachable, it reports unknown rather than crying wolf, preventing false alarms.
5.5. JSON Output for Ingestion
With the --json flag, meshctl watch emits structured data:
{
"at": 1791835000,
"nodes": [
{
"node": "node-a",
"pubkey": "z6MkhaXgBZDvotDkL5257faiztiGiC2QtKLGpbnnEGta2doK",
"digest": "81d909663c01e4280f6699723ceb396349fa8fd480b343efdbbf063e34b9c481",
"relays": 3,
"peers": 12,
"wrong": [],
"note": []
}
],
"ci": "passing",
"published": "match"
}