1.16. Non-Deterministic Compute & Workload Routing

Experimental

In the code and tested, but not supported in this alpha: it may change or go away. See Experimental features.

1.16.1. Non-Deterministic Compute: Bacalhau, LLM Inference, and Agentic Workflows

A design study and working note on extending fleetmesh’s compute market to non-deterministic runtimes. Not part of the published site, not built by build.sh, and not a second specification.


1.16.1.1. 1. The Core Tension

Fleetmesh’s deterministic compute layer (compute/1.0, fleetmesh_node/planes/services/compute/executors/wasm.py, fleetmesh_core/crypto/zk.py) rests on a single invariant: two honest nodes evaluating the same job produce identical bytes.

module + input ──> execute() ──> result_hash = SHA256(result)

Because execution is bitwise reproducible, dispute adjudication under the severity ladder is purely mathematical:

  • An independent re-run that produces a different result_hash is an S2 misreport (rerun-divergence).

  • A Groth16 proof failing verification against the public inputs is an S2 invalid proof (proof-invalid).

  • A seller failing to provide a preimage that opens the sealed ciphertext is an S3 delivery fraud (sealed-key-does-not-open).

Non-deterministic compute shatters this equality.

  1. LLM Inference: Floating-point variations across GPU architectures (NVIDIA vs. AMD vs. Apple Silicon), CUDA kernel race conditions (non-deterministic atomicAdd), variable decoding temperatures ($T > 0$), top-$p$, top-$k$, and quantization profiles (FP16 vs. INT4 vs. AWQ vs. GGUF) mean that two honest runs of the same prompt will almost never produce the same SHA-256 hash.

  2. Bacalhau Containerized Batch Jobs: “Compute Over Data” running arbitrary Docker or WASM containers over local datasets. Real-world data science pipelines depend on system clocks, thread scheduling, multithreading, pseudo-random seeds, and external data fetches.

  3. Autonomous Agent Workflows: Multi-step ReAct loops, tool calls, and branching decision trees where each intermediate observation shapes subsequent prompts and actions.

If fleetmesh evaluated non-deterministic jobs using bitwise equality, every honest provider would be priced as a liar ($S2/S3$) on their first re-run.

Yet fleetmesh’s prime directive remains binding:

“The mesh judges conformance, never content.” A node cannot evaluate whether an LLM’s summary was “perceptive” or an agent’s code was “elegant”. The mesh judges only whether the peer conformed to its signed, objective contractual obligations.

This document works out how non-deterministic compute can be integrated into fleetmesh without violating that principle.


1.16.1.2. 2. Conformance Without Bitwise Equality

In place of byte-level equality, non-deterministic compute defines four distinct Verification Modes declared in the Kind 30810 Ask before execution begins.

                      Kind 30810 Compute Ask
                                │
                        [verify tag]
                                │
       ┌────────────────────────┼────────────────────────┬────────────────────────┐
       ▼                        ▼                        ▼                        ▼
    "delivery"          "schema-bounds"          "tee-attestation"      "quasi-deterministic"
  (Zero subjectivity;  (Syntactic JSON Schema /   (Hardware enclave       (Greedy decoding T=0;
   preimage opens       CFG grammar constraint    signature over model    fixed seed & pinned
   valid payload)       evaluation)               & I/O digests)          weight digests)

1.16.1.2.1. 2.1 Mode delivery (Accountable Sealing Baseline)

The buyer explicitly accepts non-determinism and purchases execution on the provider’s reputation.

  • The seller executes the job, hashes the raw plaintext result $h = \text{SHA256}(R)$, encrypts $R$ with key $k$, and commits to payment hash $p = \text{SHA256}(k)$ in compute/1.0/sealed.

  • The buyer settles the invoice or escrow, releasing preimage $k$ in compute/1.0/settled.

  • Admissible Evidence: Only delivery fraud is admissible:

    • sealed-key-does-not-open: Preimage $k$ does not decrypt the ciphertext.

    • result-hash-mismatch-after-decrypt: Decrypted plaintext does not hash to $h$.

    • empty-payload: Plaintext is zero-length or corrupted.

  • Content evaluation is entirely out-of-band: the buyer may adjust the seller’s tier via meshnode trust or decline to peer again, but no adverse Kind 30802 report can be filed for subjective dissatisfaction.

1.16.1.2.2. 2.2 Mode schema-bounds (Syntactic & Grammar Conformance)

Crucial for LLM tool calling, structured extraction, and agent steps. The Ask binds a formal JSON Schema or CFG grammar (e.g. llama.cpp BNF grammar or outlines regex/schema) that the output must satisfy.

  • The Ask carries ["schema_cid", "<blossom-sha256>"].

  • The seller is contractually bound to deliver output that strictly conforms to the schema.

  • Admissible Evidence:

    • schema-validation-failure: The decrypted plaintext fails Draft 2020-12 validation against the committed schema.

    • An S2 misreport is admissible if and only if a third-party validator confirms the schema failure.

  • If the output conforms syntactically, the job is conformant regardless of semantic variation.

1.16.1.2.3. 2.3 Mode tee-attestation (Hardware Confidential Computing)

Status: Not built. fleetmesh_core/accounting/market.py knows the mode name and its admissible evidence kinds, and fleetmesh_node/keystore/enclave.py parses/verifies attestation documents in isolation, but no executor produces a TEE attestation and compute.wire.evaluate_job_evidence has no branch for this mode. An Ask that requests it will be quoted but cannot be adjudicated.

For privacy-critical LLM inference, proprietary models, and enterprise Bacalhau jobs.

  • The executor runs inside a certified Trusted Execution Environment:

    • NVIDIA Confidential Computing (Hopper H100/H200 / Blackwell B200 CC).

    • AMD SEV-SNP / Intel TDX / Apple Secure Enclave.

  • The enclave generates an attestation report signing: $$\text{Sign}_{\text{TEE}}(\text{SHA256}(\text{model_weights}) \parallel \text{SHA256}(\text{input}) \parallel \text{SHA256}(\text{output}) \parallel \text{params})$$

  • Admissible Evidence:

    • attestation-invalid: The hardware signature fails validation against the manufacturer’s root CA (NVIDIA/AMD/Intel).

    • measurement-mismatch: The model or container digest executed does not match the Ask’s commitment.

1.16.1.2.4. 2.4 Mode quasi-deterministic (Reproducible Subsets)

Status: Not built. Only the admissibility rule in fleetmesh_core/accounting/market.py exists; there is no multi-executor sampling, no peer re-run, and no divergence tolerance logic anywhere in fleetmesh_node/.

When temperature is strictly $0.0$ (greedy decoding), seed is pinned, batch size is $1$, and exact weight digests are specified.

  • Adjudication admits multi-executor sampling: 2-of-3 executor consensus.

  • Divergence is priced only if a supermajority of peer executors contradict the seller’s token trajectory beyond a defined tolerance threshold.


1.16.1.3. 3. The Runtime Architecture

The compute engine in fleetmesh_node/planes/services/compute/executors/ (base.py and registry.py) uses an abstract driver model (ComputeExecutor and ExecutorRegistry). Non-deterministic runtimes plug directly into this registry:

                            ExecutorRegistry
                                    │
       ┌────────────────────────────┼────────────────────────────┐
       ▼                            ▼                            ▼
  WasmExecutor              BacalhauExecutor              LlmExecutor
(Wasmtime / WASI)        (Docker / Compute Over Data)  (vLLM / Ollama / llama.cpp)
                                                                 │
                                                                 ▼
                                                           AgentExecutor
                                                        (ReAct / State Machine)

1.16.1.3.1. 3.1 Bacalhau Distributed Data Executor (BacalhauExecutor)

Status: Built & Operational. Implemented in [bacalhau.py](file:///Users/j/Code/fleetmesh/meshnode/fleetmesh_node/planes/services/compute/executors/bacalhau.py) and [nondet.py](file:///Users/j/Code/fleetmesh/meshnode/fleetmesh_node/planes/services/compute/executors/nondet.py#L197). Supports live Bacalhau v1 REST API (BacalhauRestClient), Blossom CAS volume staging and output packaging (BlossomMountBridge), 5-state lifecycle progress tracking (queued $\to$ downloading_inputs $\to$ running $\to$ uploading_outputs $\to$ completed), cancellation token handling, and test double MockBacalhauClient. Outside mock_mode or active live client, the honesty gate refuses unconfigured execution and omits the engine from advertised offers.

Bacalhau provides Docker and WASM execution across distributed datasets without moving data across WAN boundaries.

  • Blossom CAS Integration: Fleetmesh provides Blossom Content-Addressable Storage (fleetmesh_node/protocols/nostr/blossom.py).

    • Inputs: Mounted directly from Blossom SHA-256 CAS endpoints or IPFS CIDs into container /inputs.

    • Outputs: Output directory /outputs is tarred, hashed, uploaded to Blossom CAS, and referenced by SHA-256 digest in the Kind 30811 Job Receipt.

  • Execution Contract:

    • Engine ID: bacalhau/docker/1.0 or bacalhau/wasm/1.0.

    • Asynchronous batch polling via Bacalhau client API (bacalhau job run).

    • Metering units: msat_per_cpu_minute, msat_per_gpu_minute, msat_per_gib_ram.

1.16.1.3.2. 3.2 LLM Inference Executor (LlmExecutor)

Status: Built & Operational. Implemented in [nondet.py](file:///Users/j/Code/fleetmesh/meshnode/fleetmesh_node/planes/services/compute/executors/nondet.py#L26). Supports live OpenAI-compatible HTTP endpoints (vLLM, Ollama, llama-server) via netfetch (16 MiB response cap, configurable timeout, private network allowance). Draft 2020-12 JSON Schema and regex grammar bounds verification in [evaluate_job_evidence](file:///Users/j/Code/fleetmesh/meshnode/fleetmesh_node/planes/services/compute_wire/verify.py#L145). Per-frame authenticated ChaCha20Poly1305 stream framing in [encrypt_stream_frame](file:///Users/j/Code/fleetmesh/meshnode/fleetmesh_node/planes/services/compute_wire/sealing.py#L90).

Provides native integration with high-throughput local inference engines:

  • Backends: vLLM (OpenAI-compatible HTTP/gRPC), Ollama, llama-server (llama.cpp), or Aphrodite Engine.

  • Model Advertising: Node descriptors (Kind 11801) and compute offers (Kind 11803) advertise loaded models, context windows, quantization, and hardware accelerator specs.

  • Grammar Enforcement: When verify: "schema-bounds" is requested, the buyer verifies output against Draft 2020-12 JSON Schemas and regular expression grammars.

  • Metering Units: msat_per_1k_prompt and msat_per_1k_completion.

1.16.1.3.3. 3.3 Autonomous Agent Workflow Executor (AgentExecutor)

Status: Built & Operational. Implemented in [agent.py](file:///Users/j/Code/fleetmesh/meshnode/fleetmesh_node/planes/services/compute/executors/agent.py) and [nondet.py](file:///Users/j/Code/fleetmesh/meshnode/fleetmesh_node/planes/services/compute/executors/nondet.py#L385). Full multi-step ReAct (Reasoning + Action) execution loop bounded by max_steps and fuel_cap. Includes standard tool harness: AST-safe calc, sha256, regex_search, fetch_blob (Blossom CAS), sub_ask (recursive child Kind 30810 Asks with budget ceilings and recursion depth control $\le 3$), and finish. Pluggable reasoning backends: ScriptedAgentBackend and live LlmAgentBackend. Honesty gate suppresses unconfigured agent engines from published offers.

An orchestration layer running multi-step reasoning loops:

  • Standard-Library ReAct Harness: Maintains a step-wise state machine (Thought $\to$ Action $\to$ Observation $\to$ Reflection) without heavyweight external dependencies.

  • Recursive Mesh Delegation: When an agent needs a sub-computation (e.g. OCR an image, query a vector database, run a Python analysis sandbox), it can act as a buyer on the fleetmesh order book, publishing a child Kind 30810 Ask to peer nodes and aggregating the receipts.


1.16.1.4. 4. Wire Protocol & Streaming Extensions (compute/1.1)

The atomic request/response flow of compute/1.0 (Quote $\to$ Accept $\to$ Sealed $\to$ Settled $\to$ Receipt) cannot support real-time token generation or multi-hour data jobs.

1.16.1.4.1. 4.1 Real-Time Streaming Over Native Iroh QUIC

Status: Built & Operational. Implemented in [ComputeSession](file:///Users/j/Code/fleetmesh/meshnode/fleetmesh_node/planes/services/compute_wire/session.py#L85) and [sealing.py](file:///Users/j/Code/fleetmesh/meshnode/fleetmesh_node/planes/services/compute_wire/sealing.py#L90). Per-frame ChaCha20Poly1305 AEAD streaming encryption where each sequence chunk binds sequence number as Additional Authenticated Data (AAD) into a 12-byte nonce, transmitting authenticated base64 ciphertext frames over Iroh QUIC or native FIPS sockets.

Rather than flooding Nostr relays with thousands of ephemeral gift wraps, interactive token streaming leverages fleetmesh’s native Iroh QUIC transport (fleetmesh_node/transport/iroh_rung/quic.py, ALPN fleetmesh/didcomm/1):

  1. Handshake: Bilateral negotiation (quote and accept) occurs over standard DIDComm.

  2. Direct QUIC Stream: The seller opens a multiplexed bidirectional QUIC stream directly to the buyer’s ticketed endpoint.

  3. Encrypted Framing: Tokens stream as chunked, sequence-numbered frames encrypted under the session key: $$\text{Frame}i = \text{ChaCha20Poly1305}(k{\text{session}}, \text{seq}_i \parallel \text{delta_text})$$

  4. Final Sealing: On completion, the seller transmits the final compute/1.1/sealed message with the aggregate result hash and token counters.

1.16.1.4.2. 4.2 Long-Running Job Progress & Heartbeats

For multi-minute Bacalhau or agent workflows:

  • compute/1.1/progress:

    {
      "type": "https://fleetmesh.org/compute/1.1/progress",
      "thid": "<ask-session-id>",
      "body": {
        "state": "running",           // queued | downloading_inputs | running | uploading_outputs
        "progress": 0.65,
        "elapsed_seconds": 142,
        "logs_tail": "Processing epoch 3/5..."
      }
    }
    
  • compute/1.1/cancel: Permits either party to abort a job under defined cancellation penalty terms.

1.16.1.4.3. 4.3 Metering & Escrow Settlement Models

Status: hold-invoice escrow Stub (in-memory ledger only). fleetmesh_node/settlement/ tracks escrow state (CREATED → HELD → SETTLED/CANCELLED) and the preimage/payment_hash binding, and test_compute_wire exercises the delta-refund path. But payment.LightningHoldPaymentRail mints a synthetic bolt11 string and never talks to a Lightning node, and gateways.NWCLightningPaymentRail issues an ordinary NWC make_invoice, not an HTLC hold invoice: funds are not actually locked until the seller reveals the preimage. The Cashu rail (model 2) is Built against a live mint via gateways.LiveCashuPaymentRail.

Two settlement models accommodate variable-length generations:

  1. Pre-Funded Escrow with Delta Refund (Lightning Hold-Invoice):

    • Buyer locks budget_ceiling_msat via an HTLC hold invoice tied to payment_hash = SHA256(preimage).

    • On completion, the seller claims only the actual metered cost ($N_{\text{tokens}} \times \text{rate}$), canceling or refunding the remainder.

  2. Pay-As-You-Stream (Cashu Nutzap Micropayments):

    • For continuous streaming or autonomous agent steps, the buyer delivers blinded Cashu tokens (fleetmesh_core/accounting/cashu.py, Kind 9321) every $N$ tokens or every completed tool step.

    • Zero settlement counterparty risk for either node.


1.16.1.5. 5. Schema & Manifest Specifications

1.16.1.5.1. 5.1 Kind 11803: Compute Offer (Extended)

A selling node advertises LLM and container capabilities:

{
  "kind": 11803,
  "pubkey": "<seller-pubkey>",
  "tags": [
    ["executor", "llm"],
    ["model", "meta-llama/Llama-3.3-70B-Instruct"],
    ["context_window", "131072"],
    ["quantization", "fp16"],
    ["hardware", "gpu:2x-h100-80gb"],
    ["feature", "streaming"],
    ["feature", "tool-calling"],
    ["feature", "grammar-json"],
    ["rate", "40", "msat_per_1k_prompt"],
    ["rate", "120", "msat_per_1k_completion"],
    ["expiration", "1727600000"]
  ],
  "content": ""
}

1.16.1.5.2. 5.2 Kind 30810: Compute Ask (Order Book Entry)

Confidential ask committing to the job specification:

{
  "kind": 30810,
  "pubkey": "<buyer-pubkey>",
  "tags": [
    ["d", "<blinded-ask-id>"],
    ["commit", "<sha256-of-canonical-job-spec>"],
    ["executor", "llm"],
    ["model", "meta-llama/Llama-3.3-70B-Instruct"],
    ["verify", "schema-bounds"],
    ["schema_cid", "bafkreia..."],              // Blossom/IPFS CID of required output schema
    ["bid", "25000"],                           // msat maximum ceiling
    ["deadline", "1727500500"],
    ["expiration", "1727600000"]
  ],
  "content": ""
}

1.16.1.5.3. 5.3 Kind 30811: Compute Job Receipt (Encrypted Payload)

Carried encrypted via NIP-44 under the blinded coordinate:

{
  "ask_id": "<ask-event-id>",
  "executor": "llm",
  "model": "meta-llama/Llama-3.3-70B-Instruct",
  "result_hash": "<sha256-of-output>",
  "counters": {
    "prompt_tokens": 1284,
    "completion_tokens": 342,
    "total_tokens": 1626,
    "gpu_seconds": 1.48
  },
  "paid_msat": 9242,
  "payment_hash": "<32-byte-hex>",
  "settlement_rail": "lightning",
  "completed_at": 1727500480
}

1.16.1.5.4. 5.4 NIP-90 DVM Gateway Mapping (fleetmesh_node/dvm/)

Standard client-facing NIP-90 jobs map cleanly into the non-deterministic mesh:

  • Kind 5050 / 6050: Text generation $\to$ LlmExecutor.

  • Kind 5100 / 6100: Image generation (Flux / SDXL) $\to$ BacalhauExecutor or GPU diffusion driver.

  • Kind 5300 / 6300: Text embeddings $\to$ LlmExecutor.

  • Kind 5900–5999: Arbitrary container pipelines $\to$ BacalhauExecutor.


1.16.1.6. 6. Implementation Plan & Open Decision Ledger

When this track is resumed, implementation will proceed in four phases:

1.16.1.6.1. Phase 1: Conformance & Market Core

  • Add schema-bounds, delivery, and tee-attestation verify modes to fleetmesh_core/accounting/market.py and schema/constants.json.

  • Update core/tests/test_mechanisms.py and conformance vectors for non-deterministic dispute admissibility.

  • Add formal Draft 2020-12 JSON schemas in schemas/compute/ for LLM and Bacalhau job descriptors.

  • Add formal Draft 2020-12 WorkloadRouter manifest schema (fleetmesh.org/v1alpha1) and reference manifests.

1.16.1.6.2. Phase 2: Runtimes & Execution Drivers

  • Implement LlmExecutor in fleetmesh_node/planes/services/compute/executors/nondet.py with OpenAI-compatible HTTP connector (vLLM / Ollama) and grammar sampling.

  • Implement BacalhauExecutor in fleetmesh_node/planes/services/compute/executors/nondet.py with Blossom CAS input/output volume mounting.

  • Implement AgentExecutor with a lightweight, zero-dependency ReAct state loop.

1.16.1.6.3. Phase 3: Streaming Wire & QUIC Multiplexing

  • Extend fleetmesh_node/planes/services/compute_wire/ to support compute/1.1/stream and compute/1.1/progress.

  • Wire streaming token delivery directly into fleetmesh_node/transport/iroh_rung/quic.py QUIC bidirectional streams.

  • Implement variable token escrow reconciliation in fleetmesh_node/planes/services/settle/.

1.16.1.6.4. Phase 4: Integration & Testnet

  • Wire NIP-90 Kind 5050/6050 text generation into fleetmesh_node/planes/services/dvm/translate.py.

  • Add unit and end-to-end integration tests in fleetmesh_node/tests/test_compute_nondet.py.

  • Extend meshnode compute CLI subcommands (meshnode compute models, meshnode compute stream).


1.16.1.6.5. Open Decision Ledger (To Revisit)

  1. Attestation Trust Roots: Should hardware TEE root certificates (NVIDIA/AMD/Intel) be embedded directly in fleetmesh_node/keystore/enclave.py or resolved via an external PKI authority?

  2. Referee Consensus Thresholds: If multi-executor sampling is used for LLM disputes, what is the minimum stake and node diversity required to form an admissible referee committee?

  3. Container Network Isolation: For Bacalhau jobs, should containers be strictly air-gapped (network=none), or should outbound egress be permitted to authorized Blossom/IPFS endpoints?