Fleet Monitoring & Supervision (meshctl watch) ================================================ Continuous operational monitoring is vital for maintaining mesh health and preventing silent failures. While unit tests verify code against known edge cases, live fleet operations can uncover subtle hardware and networking issues—such as mid-window receipt chain breaks, stale bilateral grants, or relay payload size cap rejections. `meshctl watch` provides operators with a single-screen real-time health dashboard that inspects one or many Fleetmesh nodes simultaneously. Operational Philosophy ---------------------- 1. **Zero Remote Attack Surface**: A Fleetmesh node does not expose an unauthenticated HTTP or TCP administrative management port. Its administrative interface is strictly a local Unix domain socket (``meshnode-control/1``). To monitor remote nodes, `meshctl watch` connects over standard **SSH** (e.g. ``ssh root@node-a.fleet meshctl watch --json /var/lib/meshnode``). One static Rust binary on each machine is all that is required. 2. **Exit Status Drives Automation**: `meshctl watch` is engineered to integrate cleanly into standard Unix scheduling systems (e.g. systemd timers, cron, Prometheus node-exporter): - **Exit 0**: The mesh is completely healthy. Zero issues to report. - **Exit 1**: Something is wrong that requires operator attention (broken receipt chain, compartment in trouble, unpayable budget, or protocol digest mismatch). - **Exit 2**: A target node could not be reached over network or SSH. Systemd Timer Integration ------------------------- Fleetmesh includes systemd service and timer units in ``contrib/meshwatch.service`` and ``contrib/meshwatch.timer``. Because exit status indicates health, a failed run triggers a failed systemd unit, which existing monitoring infrastructure (Prometheus, Nagios, Datadog) already observes: .. code-block:: ini # /etc/systemd/system/meshwatch.service [Unit] Description=Fleetmesh Health Watchdog After=network.target [Service] Type=oneshot ExecStart=/usr/local/bin/meshctl watch --quiet root@node-1:/var/lib/fleetmesh root@node-2:/var/lib/fleetmesh # /etc/systemd/system/meshwatch.timer [Unit] Description=Run meshctl watch every 10 minutes [Timer] OnBootSec=5min OnUnitActiveSec=10min [Install] WantedBy=timers.target What meshctl watch Checks ------------------------- Every invocation inspects the following invariants across all target nodes: - **Compartment Health**: Verifies that all active compartments (transport, eventstore, keystore, peering, settlement) report healthy status with zero unhandled exceptions. - **Relay Subscriptions**: Ensures inbound and outbound WebSocket relay connections are active and draining. - **Bilateral Grants**: Validates that peering grants are live, budgets are not exhausted, and window pricing remains within nominal bounds (``S0`` ladder). - **Throttling Detection**: Reports throttled peers as diagnostic informational notes (not fatal faults, as throttling is normal behavior when bandwidth grants bind). - **Protocol Digest Agreement**: **The most critical check**: Compares the cryptographic protocol digest (``schema/DIGEST``) reported by every node. If two nodes do not agree on the protocol digest, `meshctl watch` flags an immediate critical fault. Protocol divergence costs peering tiers and prevents consensus. CI Pipeline & Published Spec Validation ---------------------------------------- `meshctl watch` can also monitor upstream Gitlab CI pipelines and published protocol specifications, consolidating software delivery status with live mesh state: .. code-block:: sh export FLEETMESH_CI=https://gitlab.example.com/api/v4/projects/141 export FLEETMESH_CI_TOKEN_FILE=~/.gitlab-token export FLEETMESH_SITE=https://fleetmesh.org/spec export FLEETMESH_PUBLISHER=npub1... meshctl watch root@node-a:/var/lib/fleetmesh root@node-b:/var/lib/fleetmesh When configured: - If a CI build breaks, it is surfaced directly in the watch output. - If the node's running code digest differs from the published spec on ``FLEETMESH_SITE``, `meshctl watch` names the required remediation (deploy, republish, or roll). - If the CI API is unreachable, it reports ``unknown`` rather than crying wolf, preventing false alarms. JSON Output for Ingestion ------------------------- With the ``--json`` flag, `meshctl watch` emits structured data: .. code-block:: json { "at": 1791835000, "nodes": [ { "node": "node-a", "pubkey": "z6MkhaXgBZDvotDkL5257faiztiGiC2QtKLGpbnnEGta2doK", "digest": "81d909663c01e4280f6699723ceb396349fa8fd480b343efdbbf063e34b9c481", "relays": 3, "peers": 12, "wrong": [], "note": [] } ], "ci": "passing", "published": "match" }