1.17. FIPS Native Transport Architecture

Experimental

In the code and tested, but not supported in this alpha: it may change or go away. See Experimental features.

1.17.1. FIPS Native API Transport Architecture for Fleetmesh

Document Version: 1.0-draft
Status: Architecture Specification
Subject: Identity-Native Datagram Transport & Control Plane over the Free Internetworking Peering System (FIPS)
Related Specs: discovery-libp2p-and-fips.md, running-over-fips.md, spec.html


1.17.1.1. 1. Executive Summary

Fleetmesh currently treats FIPS as an emulated IPv6 network adapter (fips0), running standard WebSocket (wss://) servers over .fips synthetic domain names resolved to fd00::/8 addresses. While functional, that model relies on root privileges (CAP_NET_ADMIN to create TUN interfaces), full TCP/IP overhead, and synthetic DNS resolution.

This document specifies the FIPS Native API Transport (fips-native) for Fleetmesh. By interfacing directly with FIPS’s unprivileged Unix control socket via file descriptor passing (SCM_RIGHTS), Fleetmesh can:

  1. Address peers directly by Nostr identity (npub / secp256k1 public key) rather than synthetic IPv6 addresses.

  2. Eliminate root/TUN requirements, operating cleanly in user-space, rootless containers, and locked-down Kubernetes pods.

  3. Route control-plane and application traffic directly over FSP (FIPS Session Protocol), leveraging native end-to-end Noise XK session encryption.

  4. Multiplex Fleetmesh compartments across 16-bit FSP Service Ports (peering, budget receipts, compute market matching, streaming tokens).

  5. Establish a dual-lane architecture: ultra-low-latency identity-native datagrams for control and app messages, with automated fallback to TCP/WSS or Iroh QUIC for bulk CAS blobs.


1.17.1.2. 2. Architectural Comparison

+---------------------------------------------------------------------------------------+
| Traditional Mode: FIPS as IPv6 TUN Adapter                                           |
|                                                                                       |
|   Fleetmesh Node -> WSS Engine -> DNS (.fips) -> OS Kernel IPv6 Stack -> TUN Adapter   |
|   - Requires root/CAP_NET_ADMIN to create fips0                                       |
|   - High latency, TCP-over-mesh encapsulation                                         |
|   - Binds to synthetic fd00::/8 IP addresses                                          |
+---------------------------------------------------------------------------------------+
                                           vs
+---------------------------------------------------------------------------------------+
| Mesh-Native Mode: FIPS Native API (fips-native)                                       |
|                                                                                       |
|   Fleetmesh Node -> Unix Domain Socket -> SCM_RIGHTS (FD) -> FSP Native Datagram Flows  |
|   - Runs completely in user-space (fips group permissions)                           |
|   - Direct npub:port addressing with zero IP headers or DNS lookups                  |
|   - Hop-by-hop Noise IK and end-to-end Noise XK mesh routing                         |
|   - Preserves Fleetmesh protocol framing and bilateral accounting semantics            |
+---------------------------------------------------------------------------------------+

1.17.1.2.1. 2.1 Feature & Constraint Matrix

Dimension

FIPS IPv6 Adapter (fips0)

FIPS Native API (fips-native)

Permissions

Root / CAP_NET_ADMIN (TUN device)

Unprivileged user (member of fips group)

Addressing

IPv6 (fd00::/8) via synthetic DNS

Direct npub / 32-byte hex public key

Transport

TCP over IP over FMP/FSP

Direct FSP datagrams

Header Overhead

40B IPv6 + 20B TCP + 106B FIPS

4B FSP Port Header + 106B FIPS

Payload Ceiling

MTU clamped by OS / MSS

Path MTU (~1170 bytes on UDP paths)

Reliability

TCP in-kernel retransmission

Best-effort datagrams (application framed)

Multiplexing

OS TCP/UDP ports

FSP Service Ports (0..65535)

Best For

Bulk NIP-77 replication, large blobs

Control plane, peering, compute negotiation, streams


1.17.1.3. 3. FSP Service Port Allocation Table

The FIPS Session Protocol (FSP) provides 16-bit application service ports. Fleetmesh reserves the following ports for its protocol compartments:

FSP Port

Compartment

Protocol URI / Message Types

Typical Payload Size

11801

Peering & Discovery

peering/1.0/* (propose, offer, accept, decline)

350 – 550 bytes

21801

Telemetry & Liveness

Kind 21801 Heartbeats, link quality probes

120 – 200 bytes

30801

Accounting & Grants

budget/1.0/receipt, signed window settlements

400 – 800 bytes

30810

Compute Negotiation

compute/1.0/* (quote, accept, sealed, settled, refute)

300 – 650 bytes

30811

Real-Time Streaming

compute/1.1/* (stream token deltas, progress heartbeats)

90 – 250 bytes

30812

Session Cancellation

compute/1.1/cancel, dispute notices

150 – 300 bytes

7777

Control & Interop

Local node diagnostic socket, health probes

Variable

All control and compute negotiation messages comfortably fit within the single-datagram path MTU (~1170 bytes), avoiding fragmentation entirely.


1.17.1.4. 4. FIPS Native API Protocol & Control Mechanics

The native API operates over a standard Unix domain stream socket:

  • Default Path: /var/run/fips/control.sock (system daemon) or $HOME/.fips/control.sock (user daemon).

  • Format: Newline-delimited JSON request/response strings.

  • Data Channel: Active flows and listeners are returned to the client as open file descriptors using Unix ancillary data (sendmsg / recvmsg with SCM_RIGHTS).

1.17.1.4.1. 4.1 Message Exchange Protocol

sequenceDiagram
    autonumber
    participant M as Fleetmesh Node (meshnode)
    participant F as FIPS Daemon (fipsd)
    participant P as Remote Peer (npub2...)

    Note over M,F: Step 1: Bind Inbound Listener
    M->>F: {"cmd": "listen", "port": 30810}\n
    F-->>M: SCM_RIGHTS [listener_fd] + {"status": "ok", "port": 30810}\n
    Note over M: Fleetmesh polls listener_fd for incoming client flows

    Note over M,F: Step 2: Open Outbound Flow to Peer
    M->>F: {"cmd": "connect", "peer": "npub2...", "port": 30810}\n
    F-->>M: SCM_RIGHTS [flow_fd] + {"status": "connected", "peer": "npub2...", "port": 30810}\n

    Note over M,P: Step 3: Bilateral Datagram Exchange
    M->>P: sendmsg(flow_fd, compute/1.0/quote)
    P-->>M: recvmsg(flow_fd, compute/1.0/accept)

    Note over M: Step 4: Teardown
    M->>M: close(flow_fd)

1.17.1.4.2. 4.2 Control Commands

1.17.1.4.2.1. 1. listen (Register Inbound Port)
{"cmd": "listen", "port": 30810}

Response: File descriptor passed via SCM_RIGHTS representing the bound listener socket.

{"status": "ok", "port": 30810, "state": "listening"}
1.17.1.4.2.2. 2. connect (Establish Outbound Flow)
{"cmd": "connect", "peer": "npub180cvv07tjdrrgpa0j7j7tmnyl2yr6yr7l8j4s3evf6u64th6gkwsyjh6w6", "port": 30810}

Response: File descriptor passed via SCM_RIGHTS representing the connected flow. Datagrams written to this descriptor are encrypted via Noise XK and routed over FMP to the destination peer.

{"status": "connected", "peer": "npub180cvv07...", "port": 30810, "max_payload": 1170}
1.17.1.4.2.3. 3. stats (Query Flow Telemetry)
{"cmd": "stats", "peer": "npub180cvv07...", "port": 30810}

Response:

{"rtt_ms": 24, "loss_rate": 0.0, "bytes_sent": 1420, "bytes_recv": 890, "max_payload": 1170}

1.17.1.5. 5. Security Architecture & Key Isolation

A critical finding in §3.5 of discovery-libp2p-and-fips.md is the Key-Sharing Risk:

The Fleetmesh node key (node_sk) must never be shared with an external daemon or exposed to a shared Unix socket.

1.17.1.5.1. 5.1 Key Isolation Architecture

+-------------------------------------------------------------------------------+
| Fleetmesh Security Perimeter                                                    |
|                                                                               |
|   keys.json (0600)                                                            |
|   └── node_sk (secp256k1) ──────────> Signs Grants, Receipts, Descriptors     |
|                                                                               |
|   Kind 11801 Node Descriptor                                                  |
|   └── ["key", "fips", "<fips_pubkey_hex>"] ───+ Binding verified by peers     |
+------------------------------------------------│------------------------------+
                                                 │
                                                 v
+-------------------------------------------------------------------------------+
| FIPS Security Perimeter (Isolated)                                            |
|                                                                               |
|   fips.key (0600)                                                             |
|   └── fips_sk (secp256k1) ──────────> Encrypts FMP links & FSP Noise XK flows|
|   /var/run/fips/control.sock                                                  |
+-------------------------------------------------------------------------------+
  1. Independent Keys:

    • meshnode maintains its master identity in keys.json (node_sk). It never rotates; rotating it creates a new node identity.

    • fipsd runs under its own separate secp256k1 key (fips.key).

  2. Cryptographic Binding via Descriptor:

    • Fleetmesh publishes a Kind 11801 descriptor carrying a dedicated FIPS key binding tag:

      ["key", "fips", "<fips_pubkey_hex>"]
      
    • Remote peers verify this tag against the signed descriptor. Traffic received on FSP flows signed by <fips_pubkey_hex> is attributed directly to the Fleetmesh node.

  3. Blast Radius Mitigation:

    • If fips.key or the fips Unix socket is compromised, the Fleetmesh master identity remains uncompromised.

    • Recovery requires only generating a new fips.key and republishing a new Kind 11801 descriptor with the updated ["key", "fips", ...] tag.


1.17.1.6. 6. Datagram Framing Specification

When sending DIDComm or Fleetmesh messages over raw FSP datagrams, a compact 4-byte binary framing header precedes the payload:

 0                   1                   2                   3
 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|  Magic (0x4D) | Version (0x01)|       Stream / Frag Seq       |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Flags (1 Byte)|  Reserved(1B) |         Payload Length        |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|                        Payload Bytes ...                      |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

1.17.1.6.1. Header Fields

  1. Magic (8 bits): Constant 0x4D ('M').

  2. Version (8 bits): Protocol framing version 0x01.

  3. Seq (16 bits): Sequence index (for streaming tokens or fragmented envelopes). Matches seq in compute/1.1/stream.

  4. Flags (8 bits):

    • 0x01 (FLAG_FINAL): Terminal frame / chunk in a multi-packet sequence.

    • 0x02 (FLAG_COMPRESSED): Payload compressed with Zstandard.

    • 0x04 (FLAG_DIDCOMM): Payload is a DIDComm v2 JWE (didcomm.encrypt, ECDH-1PU authcrypt from the sending node’s agreement key). Plaintext DIDComm is not accepted on this rung: the receiver opens every frame with didcomm_inbound.open_inbound, which refuses anything that is not an envelope and marks the message authenticated only when the envelope’s skid names the same node as from. The FIPS flow gives confidentiality and peer authentication at the network layer, but the node identity a message claims is proved by the envelope alone - the same rule as the relay and iroh rungs - so a process that can write to the flow cannot speak as another node.

    • 0x08 (FLAG_ACK_REQUESTED): Request explicit FSP application ACK.

  5. Reserved (8 bits): Reserved for future extensions (0x00).

  6. Payload Length (16 bits): Size of the payload in bytes (maximum 1162 bytes per datagram).


1.17.1.7. 7. Node Integration & Transport Ladder

In fleetmesh_node/transport/, FIPS Native API is incorporated as a first-class transport rung:

TRANSPORT_RUNG_FIPS_NATIVE: int = 10  # Highest priority when available
TRANSPORT_RUNG_IROH: int        = 20  # Direct QUIC
TRANSPORT_RUNG_WSS_FIPS: int    = 30  # WebSocket over fips0 TUN
TRANSPORT_RUNG_WSS_CLEAR: int   = 40  # Clearnet WebSocket
TRANSPORT_RUNG_RELAY_WRAP: int  = 50  # NIP-59 gift wrap over Nostr relays (fallback)

1.17.1.7.1. 7.1 Dynamic Ladder Selection Logic

When Fleetmesh dispatches a message to peer P:

  1. Probe FIPS Native: If local control.sock exists and peer P advertised a FIPS key in its Kind 11801 descriptor:

    • Attempt connect(peer_fips_key, port).

    • If flow established within 150ms, transmit framed datagram.

  2. Fall Back to Iroh QUIC: If native FIPS flow is rejected or times out, attempt direct Iroh QUIC connection using the peer’s Ed25519 transport key.

  3. Fall Back to WSS / TUN: If Iroh is unavailable, attempt WebSocket dial over wss://<npub>.fips:port or clearnet.

  4. Universal Fallback (NIP-59 Wrap): If all direct links fail, wrap the message in a Kind 21059 gift wrap and publish to the peer’s Kind 10050 inbox relays.


1.17.1.8. 8. Implementation Components

1.17.1.8.1. 8.1 Python Native Client Interface (fleetmesh_node/transport/fips_rung/core.py)

"""fleetmesh_node/transport/fips_rung/core.py: Native Unix socket client for FIPS control API."""
import array
import json
import os
import socket
from typing import Any, Optional, Tuple

class FipsTransportError(Exception):
    pass

class FipsNativeTransport:
    def __init__(self, sock_path: str = "/var/run/fips/control.sock"):
        self.sock_path = sock_path
        self.ctrl_sock: Optional[socket.socket] = None

    def is_available(self) -> bool:
        return os.path.exists(self.sock_path) and os.access(self.sock_path, os.R_OK | os.W_OK)

    def connect(self) -> None:
        if not self.is_available():
            raise FipsTransportError(f"FIPS control socket not accessible at {self.sock_path}")
        self.ctrl_sock = socket.socket(socket.AF_UNIX, socket.SOCK_STREAM)
        self.ctrl_sock.connect(self.sock_path)

    def open_flow(self, peer_npub: str, port: int) -> socket.socket:
        """Connect to peer npub on FSP port, returning connected datagram socket."""
        req = json.dumps({"cmd": "connect", "peer": peer_npub, "port": port}) + "\n"
        self.ctrl_sock.sendall(req.encode("utf-8"))

        # Receive file descriptor passed via SCM_RIGHTS
        fds = array.array("i")
        msg, ancdata, flags, addr = self.ctrl_sock.recvmsg(
            1024, socket.CMSG_LEN(fds.itemsize)
        )
        for cmsg_level, cmsg_type, cmsg_data in ancdata:
            if cmsg_level == socket.SOL_SOCKET and cmsg_type == socket.SCM_RIGHTS:
                fds.frombytes(cmsg_data[:len(cmsg_data) - (len(cmsg_data) % fds.itemsize)])
                if fds:
                    flow_fd = fds[0]
                    return socket.fromfd(flow_fd, socket.AF_UNIX, socket.SOCK_DGRAM)

        resp = json.loads(msg.decode("utf-8")) if msg else {}
        raise FipsTransportError(f"FIPS connect failed: {resp.get('error', 'no fd passed')}")

    def listen_port(self, port: int) -> socket.socket:
        """Listen on FSP service port, returning listener datagram socket."""
        req = json.dumps({"cmd": "listen", "port": port}) + "\n"
        self.ctrl_sock.sendall(req.encode("utf-8"))

        fds = array.array("i")
        msg, ancdata, flags, addr = self.ctrl_sock.recvmsg(
            1024, socket.CMSG_LEN(fds.itemsize)
        )
        for cmsg_level, cmsg_type, cmsg_data in ancdata:
            if cmsg_level == socket.SOL_SOCKET and cmsg_type == socket.SCM_RIGHTS:
                fds.frombytes(cmsg_data[:len(cmsg_data) - (len(cmsg_data) % fds.itemsize)])
                if fds:
                    return socket.fromfd(fds[0], socket.AF_UNIX, socket.SOCK_DGRAM)

        raise FipsTransportError(f"FIPS listen failed on port {port}")

1.17.1.9. 9. Rollout & Phasing Plan

  1. Phase 1: Transport Client Core (fleetmesh_node/transport/fips_rung/core.py)

    • Implement FipsNativeTransport with SCM_RIGHTS fd passing.

    • Loopback unit tests mocking FIPS control socket and file descriptor transfer.

  2. Phase 2: Port Allocation & Framing Integration

    • Bind FSP Port 30810 for compute negotiation and 30811 for real-time streaming.

    • Wire ComputeSession.stream_over_quic() / stream_over_fips() to native FSP datagrams.

  3. Phase 3: Transport Ladder Integration (fleetmesh_node/transport/)

    • Add fips-native as the top-priority rung when fips.sock is active and peer advertises FIPS capability.

    • Add descriptor advertising tag ["endpoint", "fips-native", "<fips_pubkey>:30810"].

  4. Phase 4: Multi-Node Integration Testing

    • Run end-to-end compute negotiation across two local nodes connected over live FIPS native flows.