6. Storage & Synchronization (fleetmesh_core.storage)

The fleetmesh_core.storage package provides abstract store contracts, Negentropy range-based set reconciliation, Blossom media authorization, and replication algorithms.

6.1. Store Interfaces

MeshStore: what a storage backend must agree with every other one about.

The specification requires that every backend produce byte-identical range fingerprints for identical event sets, because two nodes whose fingerprints disagree cannot reconcile and cannot tell that from having different data.

The fingerprint algorithm is NIP-77’s and is pinned in fleetmesh_core/storage/replication.py and vectors/fingerprint.json. That is only half of what agreement needs. The other half is which events are in the set, and writing this module is what found that the prose never said. Four rules decide it, all now in constants.json under store, and all four are places where two honest implementations would otherwise diverge:

  1. ephemeral kinds are never stored, so they never enter a fingerprint

  2. replaceable and addressable kinds keep only the winner

  3. a valid NIP-09 deletion removes its target, permanently - and a deletion that arrives before its target still binds, which is why a tombstone records who asked rather than merely that someone did: authority depends on the target’s author, and that is not known yet

  4. an expired event is not in the set

Standard library only. Two backends, one behavioural suite: core/tests/test_store.py runs the same assertions against both, which is the store conformance suite the specification’s Conformance section asks for.

fleetmesh_core.storage.store.in_clause(column, values)[source]

(sql, args) for column IN <values> with a bounded variable count.

Parameters:
Return type:

tuple[str, list[Any]]

fleetmesh_core.storage.store.chunked(values, size=500)[source]

Yield values in runs of at most size, for statements run per chunk.

Parameters:
fleetmesh_core.storage.store.is_ephemeral(kind)[source]
Parameters:

kind (int)

Return type:

bool

fleetmesh_core.storage.store.is_replaceable(kind)[source]
Parameters:

kind (int)

Return type:

bool

fleetmesh_core.storage.store.is_addressable(kind)[source]
Parameters:

kind (int)

Return type:

bool

fleetmesh_core.storage.store.ascii_int(value)[source]

The integer a wire string spells, or None if it is not one.

A wire integer is one or more ASCII digits and nothing else: no sign, no whitespace, no underscores, no other script’s digits. Python’s int() and str.isdigit() accept all of those, and a second implementation’s parser accepts a different subset, so a value one side read as a number the other read as garbage. This is the one spelling both agree on.

Parameters:

value (Any)

Return type:

int | None

fleetmesh_core.storage.store.is_hex64(value)[source]

Exactly 64 lowercase hex digits: the spelling of an id or a pubkey.

Parameters:

value (Any)

Return type:

bool

fleetmesh_core.storage.store.is_int(value)[source]

A JSON integer: an int, and not a bool (which is one to Python).

Parameters:

value (Any)

Return type:

bool

fleetmesh_core.storage.store.tags(event, name)[source]

Every tag of one name, in order. Three modules had written this for themselves before it was worth having once. A tag whose elements are not all strings is skipped: NIP-01 tags are arrays of strings, and a reader that indexed into one of another shape crashed instead of reporting.

Parameters:
Return type:

list[list[str]]

fleetmesh_core.storage.store.first_tag(event, name)[source]

The first tag of one name, or None. A reader wanting the value takes t[1] and checks the length first: a tag with no value is well-formed NIP-01 and means nothing here.

Parameters:
Return type:

list[str] | None

fleetmesh_core.storage.store.d_tag(event)[source]
Parameters:

event (Mapping[str, Any])

Return type:

str

fleetmesh_core.storage.store.coordinate(event)[source]

The key supersession is resolved within, or None for a regular event.

This is the NIP-01 coordinate, kind:pubkey:d - the same string a NIP-09 a tag carries - so a deletion by coordinate can be matched against it byte for byte. A replaceable kind has an empty d: kind:pubkey:.

Parameters:

event (Mapping[str, Any])

Return type:

str | None

fleetmesh_core.storage.store.expiration(event)[source]

NIP-40, or None. The first well-formed expiration tag wins; a value that is not ASCII digits is not an expiration tag at all, so it is skipped rather than read as expiry now: a peer cannot be allowed to evict an event by sending garbage, and two implementations with different int parsers cannot be allowed to disagree about whether it was garbage.

Parameters:

event (Mapping[str, Any])

Return type:

int | None

fleetmesh_core.storage.store.supersedes(new, old)[source]

NIP-01: later created_at wins; on a tie the lexicographically lower id.

The tie-break matters more than it looks. Two nodes that saw both versions must pick the same one or their fingerprints differ forever, and a tie is exactly what a replaceable event republished within one second produces.

Parameters:
Return type:

bool

fleetmesh_core.storage.store.matches(event, filt)[source]

NIP-01 filter matching, including single-letter #tag filters.

Every condition is an AND across fields and an OR within a field, which is NIP-01’s rule and the one thing a backend cannot get creative about.

Parameters:
Return type:

bool

fleetmesh_core.storage.store.compiled(filt)[source]

filt without since/until/limit, its lists as sets: the form matches answers fastest when it is asked about many events.

_in_range and newest_matching built that reduced filter again for every event, and matches tested ids and authors as lists - O(n) per event, with up to validate.MAX_IDS ids - and rebuilt each tag set per event per tag (audit 2026-10-15, E2). Same answers: membership is membership. A list holding anything but scalars is left a list, and answered as before.

Parameters:

filt (Mapping[str, Any])

Return type:

Mapping[str, frozenset[str | int | float | bool | None] | Sequence[Any] | int | str | None]

class fleetmesh_core.storage.store.MeshStore[source]

Bases: object

The interface, and every rule that is the same whatever the backend.

A backend supplies raw storage: _put, _drop, _all, _get, _tombstone, _is_tombstoned. Everything that decides set membership lives here, so a new backend cannot accidentally invent its own answer to the four questions above - which is the whole point of a shared suite.

get_event(event_id)[source]
Parameters:

event_id (str)

Return type:

Mapping[str, Any] | None

has_event(event_id)[source]
Parameters:

event_id (str)

Return type:

bool

insert_event(event, now)[source]

Returns one of: stored, duplicate, ephemeral, expired, superseded, deleted, or a deletion outcome. now is explicit rather than read from the clock so that two backends fed the same history agree exactly.

All or nothing, through _atomic: every write one insertion makes lands together or not at all.

Parameters:
Return type:

str

enumerate_range(filt, since, until, now)[source]

(created_at ASC, id ASC). The order is observable, so it is normative even though the fingerprint does not need it.

Parameters:
Return type:

list[str]

items_range(filt, since, until, now)[source]

(created_at, id) pairs in the same order, which is the record shape negentropy reconciles over - timestamp first, id second, ascending.

Parameters:
Return type:

list[tuple[int, str]]

fingerprint_range(filt, since, until, now)[source]

NIP-77’s fingerprint over that set. 16 bytes, as 32 hex characters.

Parameters:
Return type:

str

newest_matching(filt, since, until, now, cap)[source]

At most cap matching events, newest first, ties broken by id ascending.

Every other range query answers “the whole set”, which is the only answer a fingerprint or a reconciliation can use. A REQ is the one caller that knows up front how many it will send, and on an open face the set can be four orders of magnitude larger than the limit. Asking for the set and then slicing it is what a backend cannot afford, so the question is asked in the shape the answer is wanted in.

Parameters:
Return type:

list[Mapping[str, Any]]

class fleetmesh_core.storage.store.MemoryStore[source]

Bases: MeshStore

A dict. The naive backend the specification says is conformant.

__init__()[source]
Return type:

None

class fleetmesh_core.storage.store.SqliteStore[source]

Bases: MeshStore

sqlite3 from the standard library, storing the event as canonical JSON.

Deliberately naive: it scans. The specification is explicit that how a backend computes a range is unspecified and only the value on the wire is normative, so a backend that scans and one with a prefix-sum index must agree - and this one exists to be the slow, obviously-correct side of that comparison.

__init__(path=':memory:')[source]
Parameters:

path (str)

Return type:

None

newest_matching(filt, since, until, now, cap)[source]

The bounded answer, read newest first and stopped at cap.

The inherited version narrows in SQL and then holds every match as a parsed event so it can be sorted. That is the whole store on an open face - which is how a 1 GB box serving 8,923 events died on a REQ for the newest twenty. Here sqlite does the ordering on the index the events table already has, and rows are parsed one at a time until enough of them have matched.

Parameters:
Return type:

list[Mapping[str, Any]]

close()[source]

Safely close the underlying database connection.

Return type:

None

fleetmesh_core.storage.store.event_id(pubkey, created_at, kind, tags=None, content='')[source]

NIP-01 event id: sha256 over the canonical serialisation.

Parameters:
Return type:

str

fleetmesh_core.storage.store.make_event(pubkey, created_at, kind, tags=None, content='')[source]

A well-formed event without a signature.

The store never checks one: the interface takes “an independently authenticated event”, so verification happens before insertion and belongs to whatever did the authenticating. That makes this cheap enough to build corpora of thousands, which the tests need.

It is not a complete nostr event - NIP-01 requires a sig, and anything that puts an event on the wire needs make_signed_event instead.

Parameters:
Return type:

UnsignedNostrEvent

fleetmesh_core.storage.store.make_signed_event(sk_hex, created_at, kind, tags=None, content='')[source]

A complete NIP-01 event, signature and all.

Slow - the curve arithmetic here is pure Python - so it is for the places that need a real event rather than a corpus: anything validated against schemas/common/nostr-event.schema.json, or put in a sync/1.0/deliver.

Parameters:
Return type:

NostrEvent

fleetmesh_core.storage.store.verify_event(ev)[source]

NIP-01: the id is the hash of the serialised event and the signature is BIP-340 over that id by the event’s pubkey. False for anything malformed, which includes a missing tags or content: NIP-01 requires both, and an event without them is not one that another implementation would hash the same way.

Parameters:

ev (Mapping[str, Any])

Return type:

bool

6.2. PostgreSQL Storage Layer

The event store on PostgreSQL: jsonb, and a GIN index instead of a tag table.

MeshStore (store.py) holds every rule that decides set membership, and a backend supplies six primitives: _put, _drop, _all, _get, _tombstone, _tombstoned_by. Narrowing is optional - the base can read everything and filter in Python and still be correct - so what is here beyond the six is index work, and the shared vectors prove it gives the same answers as the backend that scans.

Not a port of SqliteStore. That one keeps a tags table because SQLite has no indexable way into a JSON array, and pays for it with an EXISTS subquery per tag name, an _index_tags write per event and a _backfill_tags sweep on every open. A jsonb containment test subsumes all three:

json -> ‘tags’ @> ‘[[“t”,”bristol”]]’

is one indexable predicate, with NIP-01’s own semantics falling out of the shape - values within a name OR together, different names AND. So there is no tag table, no secondary write per event, and no backfill.

The fingerprint is safe over jsonb. fingerprint_range is taken over event IDS (enumerate_range), never over stored bytes, and an id is a hex string and created_at an integer - both round-trip exactly. Key order does not matter either: crypto.canonical sorts keys itself, so a dict read back from jsonb re-canonicalises to the same bytes it was written from.

The driver is optional and imported lazily. fleetmesh_core depends on nothing, and nothing here changes that for a node on SQLite.

fleetmesh_core.storage.store_postgres.driver()[source]

psycopg, or a refusal naming the remedy rather than an ImportError.

Return type:

Any

fleetmesh_core.storage.store_postgres.available()[source]
Return type:

bool

class fleetmesh_core.storage.store_postgres.PostgresStore[source]

Bases: MeshStore

The event store on PostgreSQL. One connection, as the node has one writer.

__init__(url, schema='public')[source]

schema namespaces the two tables inside the node’s database.

The default puts them where an unqualified name already looks, which is what a node with nothing else in its database wants. Naming one keeps the event tier’s two tables clear of the state tier’s, which is the arrangement when a node gives both tiers one database.

Parameters:
Return type:

None

close()[source]
Return type:

None

newest_matching(filt, since, until, now, cap)[source]

The bounded answer, ordered by the index and stopped at cap.

The inherited version holds every match as a parsed event so it can sort - the whole store, on an open face. Here the database orders on the index events_created_at already provides and rows are parsed one at a time until enough have matched.

Parameters:
Return type:

list[Mapping[str, Any]]

6.3. Negentropy Reconciliation

Negentropy V1: the set reconciliation NIP-77 standardises, over a MeshStore.

The specification makes NIP-77 a hard prerequisite for peering and defines no reconciliation of its own, so this is an implementation of somebody else’s protocol rather than a definition of ours. It follows negentropy/docs/negentropy-protocol-v1.md.

Two halves, and only one of them is normative:

the wire format protocol byte 0x61, ranges of (upper bound, mode, payload),

delta-encoded timestamps, minimal id prefixes. Two implementations MUST agree here byte for byte.

the split how a differing range is divided, and when to stop

splitting and just list the ids. The document does not specify it, deliberately: any split interoperates as long as the ranges stay adjacent and cover the whole space.

The constants below are therefore this implementation’s choices, not the protocol’s, and are marked as such. Getting them wrong costs round trips and bytes; it cannot cost correctness.

Standard library only.

Verified against hoytech’s reference implementation. scripts/check-negentropy.py drives fleetmesh_core/conformance/harness.py and the reference’s own test/js/harness.js through the negentropy project’s conformance protocol, in both directions, over the record distributions its fuzzer uses. Also verified here: the encoder and decoder round trip, the frames match hand-computed bytes in vectors/negentropy.json, and two stores holding different sets converge.

class fleetmesh_core.storage.negentropy.ReconcileReport[source]

Bases: TypedDict

have: list[str]
need: list[str]
rounds: int
bytes_sent: int
bytes_received: int
bytes: int
exception fleetmesh_core.storage.negentropy.Malformed[source]

Bases: FleetmeshError, ValueError

A message that is not a negentropy message, or is truncated.

exception fleetmesh_core.storage.negentropy.UnsupportedVersion[source]

Bases: FleetmeshError, ValueError

A negentropy message in a version this code does not speak.

Not an error on the wire. A server answers it with the highest version it does speak, which is the only way a client can find out to downgrade.

__init__(version)[source]
Parameters:

version (int)

Return type:

None

fleetmesh_core.storage.negentropy.encode_varint(n)[source]
Parameters:

n (int)

Return type:

bytes

fleetmesh_core.storage.negentropy.decode_varint(buf, i)[source]
Parameters:
Return type:

tuple[int, int]

class fleetmesh_core.storage.negentropy.Bound[source]

Bases: object

An exclusive upper bound: a timestamp and a disambiguating id prefix.

Omitted trailing prefix bytes are implicitly zero, so a bound compares as (timestamp, prefix padded to 32 bytes) against an item’s (timestamp, id).

__init__(timestamp, prefix=b'')[source]
Parameters:
Return type:

None

timestamp
prefix
key()[source]
Return type:

tuple[int, bytes]

fleetmesh_core.storage.negentropy.minimal_bound(prev_item, item)[source]

The shortest bound separating two adjacent items.

Per the protocol: if the timestamps differ the prefix is empty, otherwise it is their common id prefix plus one byte. Sending more than that is legal and wasteful; sending less is wrong, because the bound would not separate them.

Parameters:
Return type:

Bound

fleetmesh_core.storage.negentropy.encode_message(ranges)[source]

ranges: [(upper Bound, mode, payload)] in ascending order.

Parameters:

ranges (Sequence[tuple[Bound, int, Any]])

Return type:

bytes

fleetmesh_core.storage.negentropy.decode_message(buf)[source]

Bytes to ranges, or an exception. Never a partial parse.

A message arrives from a peer that has proved nothing yet, so this rejects rather than repairs. The reference implementation makes the same choices, and where it normalises instead of rejecting - a bound after infinity - so does this, because two implementations that read the same bytes differently is the failure this whole module exists to avoid.

Parameters:

buf (bytes)

Return type:

list[tuple[Bound, int, Any]]

fleetmesh_core.storage.negentropy.fingerprint(items)[source]

NIP-77’s, over the ids in order-independent fashion. fleetmesh_core/storage/replication.py owns the algorithm; this exists so the caller reads in terms of items.

Parameters:

items (Sequence[tuple[int, str]])

Return type:

bytes

class fleetmesh_core.storage.negentropy.Negentropy[source]

Bases: object

One side of a reconciliation.

items are (timestamp, id-hex), sorted ascending by (timestamp, id) - which is the order MeshStore.enumerate_range already returns, deliberately.

Only the client learns anything: reconcile() returns the ids it holds that the server does not, and the ids the server holds that it does not. The server answers and learns nothing, which is why a leaf can reconcile against an anchor without telling it what a phone contains.

__init__(items, is_client=True)[source]
Parameters:
Return type:

None

items: list[tuple[int, str]]
is_client: bool
initiate()[source]

The client’s opening message: the whole space, split.

Return type:

bytes

reconcile(message)[source]

Process one message. Returns (reply or None, have, need).

have and need are only ever populated for the client. A reply of None means this side has nothing further to say, which is how the protocol terminates.

Parameters:

message (bytes)

Return type:

tuple[bytes | None, list[str], list[str]]

fleetmesh_core.storage.negentropy.reconcile(client_items, server_items, max_rounds=32)[source]

Run a whole reconciliation and report what it cost.

Returns a dict with the client’s have/need, the number of round trips and the bytes on the wire - which is the measurement the specification’s claim about catching up ‘in kilobytes of fingerprints’ has never been held to.

Parameters:
Return type:

ReconcileReport

6.4. Sync Protocol

sync/1.0: carrying reconciliation between two nodes under a grant.

This is where the pieces meet. fleetmesh_core/storage/negentropy.py finds which events two nodes differ on; fleetmesh_core/storage/store.py says which events are in the set; a grant says how much capacity the peer has and which events it is allowed to send at all. A sync session is the only place all three are true at once, and the only place the counters a receipt reports actually come from.

Three rules are enforced here rather than described:

the interest contract an event delivered outside the interest filter is an

S3 finding, not an inefficiency. It is how a peer would make its traffic somebody else’s problem.

the budget sync_ranges, events_in and bytes_in are grant limits.

They are counted as the session runs, so the receipt is a record of what happened rather than an estimate.

duplicates an event delivered that the receiver already held is

counted. A peer with working reconciliation sits near zero; dup_ratio is what tells a broken one from a replaying one.

Messages are DIDComm plaintext bodies as schemas/didcomm/sync/ defines them. Negentropy frames travel hex-encoded, which is what NIP-77 specifies and what messages.json now pins - the schema had said “hex or base64”, which is two wire formats and therefore none.

Standard library only.

fleetmesh_core.storage.sync.delivered(message)[source]

How many events a deliver actually carries.

The envelopes are the carrier, so they are what is counted. body.count is the sender’s claim about its own message and a meter measures what arrived, not what was claimed; body.events is the in-process form, before the node layer moves them into the attachment. Counting the body alone is how delivered events briefly went unmetered the moment they stopped riding there.

Parameters:

message (Mapping[str, Any])

Return type:

int

fleetmesh_core.storage.sync.wire_bytes(message)[source]

What the body costs on the wire, before DIDComm enveloping. The envelope adds a constant per message; what a grant meters is the payload.

Parameters:

message (Mapping[str, Any])

Return type:

int

class fleetmesh_core.storage.sync.Grant[source]

Bases: object

The part of a decrypted kind 30801 a sync session needs.

scope is {direction: filter}. A grant with no push scope forbids the subject from sending anything, which is what a pull-only interest means.

__init__(event_id, tier, limits, scope=None, window=3600)[source]
Parameters:
Return type:

None

classmethod from_payload(payload, event_id)[source]

Build from a decrypted kind 30801 payload, which is what fleetmesh_core/peering/peering.py produces. The two modules meet here and nowhere else.

Parameters:
Return type:

Grant

permits(event, direction='push')[source]
Parameters:
Return type:

bool

exceeded(counters)[source]

Which limits this session’s counters have gone past, and by how much. Reported as overrun so fleetmesh_core/accounting/window.py can classify it unchanged.

A limit of zero is a limit, not an absence: the window prices any use over it as an infinite overrun, and this said the opposite by skipping it, so a grant with blob_bytes 0 - the specification’s own example - was priced S3 by the ladder and refused by nobody.

Parameters:

counters (Mapping[str, Any])

Return type:

dict[str, float]

class fleetmesh_core.storage.sync.Session[source]

Bases: object

One side of one sync/1.0 session over one shard.

The initiator reconciles, asks for what it lacks, and inserts what arrives. The responder answers frames and serves events its grant permits it to send. Neither trusts the other: the initiator checks every delivered event against its own interest filter, and the responder refuses to serve outside the scope its own grant declared.

__init__(my_store, grant, shard, interest, since=0, until=2147483648, now=None, is_initiator=True, verify=None)[source]
Parameters:
Return type:

None

property now: int

The instant this session runs at, supplied by whoever built it.

Raised rather than defaulted: a session that ran at epoch 0 would enumerate against an expiration nobody meant and write a receipt for window 0, and both would look valid. A missing clock is a programming error, so it is reported as one at the point of use.

open()[source]

sync/1.0/open, carrying the first negentropy frame.

Return type:

Mapping[str, Any]

on_open(message)[source]

Responder. Checks the interest against its own grant before answering.

An interest wider than the scope the grant declared is refused rather than trimmed: quietly serving a narrower set than was asked for looks to the asker like missing data, which is the failure this protocol exists to make impossible to confuse with anything else.

Parameters:

message (Mapping[str, Any])

Return type:

Mapping[str, Any]

on_range(message)[source]

Either side. Reconciles one frame and says what follows from it.

The responder only ever answers frames. The initiator also accumulates the difference, and when reconciliation converges it asks for the half it is missing - which is the point at which this stops being a conversation about fingerprints and starts costing bandwidth.

Parameters:

message (Mapping[str, Any])

Return type:

Mapping[str, Any]

on_need(message)[source]

Responder. Serves only what its grant permits it to send.

An event it holds but may not push is simply not sent. Sending it would be the S3 the receiver would then file, so refusing is the responder protecting its own standing.

Parameters:

message (Mapping[str, Any])

Return type:

Mapping[str, Any]

on_deliver(message)[source]

Initiator. Inserts what arrived, and prices what should not have.

Every event is checked against the interest this session declared. One outside it is an S3: the interest contract is the whole basis on which a peer accepted the traffic.

Parameters:

message (Mapping[str, Any])

Return type:

Mapping[str, Any]

dup_ratio()[source]
Return type:

float

receipt(seq=0, prev=None, at=None)[source]

The budget/1.0/receipt body this session’s traffic accounts for.

The counters are what happened, not a summary of it, which is the whole claim the accounting rests on. at defaults to the session clock; an explicit 0 is an explicit 0, not a request for the default.

Parameters:
  • seq (int)

  • prev (str | None)

  • at (int | None)

Return type:

ReceiptBody

class fleetmesh_core.storage.sync.SyncRunReport[source]

Bases: TypedDict

transcript: list[Mapping[str, Any]]
messages: int
initiator: dict[str, int]
responder: dict[str, int]
findings: list[tuple[str, str]]
need: list[str]
dup_ratio: float
bytes: int
status: str | None
fleetmesh_core.storage.sync.run(initiator, responder, max_messages=64)[source]

Drive a whole session and report it.

Turn-taking is strict alternation: whoever did not send the last message handles it. Returns the transcript, both sides’ counters, the findings either side would file, and what it cost. A session that has not finished inside max_messages is itself the finding.

Parameters:
Return type:

SyncRunReport

6.5. Blob Replication

Delivery envelopes, range fingerprints and transport selection.

fleetmesh_core.storage.replication.hop_message_hex(event_id, prev_hop, at)[source]
Parameters:
  • event_id (str)

  • prev_hop (str | None)

  • at (int)

Return type:

str

fleetmesh_core.storage.replication.append_hop(envelope, node_did, sk_hex, at)[source]

Sign over (event_id, prev_hop, at) and append. This is what makes a forwarder accountable for traffic it did not originate.

Parameters:
Return type:

DeliveryEnvelope

fleetmesh_core.storage.replication.verify_envelope(envelope)[source]

Check every hop signature and the hop limit. Returns failures.

Parameters:

envelope (DeliveryEnvelope)

Return type:

list[str]

fleetmesh_core.storage.replication.signed_by(envelope, pubkey)[source]

True when this key appears in the path with a hop signature that verifies.

What makes a delivery envelope citable evidence: the forwarder signed the hop even when somebody else authored the event, so it is the subject’s own statement that it sent this, at this moment.

Parameters:
Return type:

bool

fleetmesh_core.storage.replication.chargeable_hops(envelope, my_grants)[source]

Which nodes in the path this receiver holds a grant with.

A forwarder outside the grant table is uncharged - and also unknown, so probation-limited. This is what closes grant-evasion routing.

Parameters:
Return type:

list[str]

fleetmesh_core.storage.replication.varint(n)[source]

Base-128, most significant digit first, high bit set on all but the last.

Parameters:

n (int)

Return type:

bytes

fleetmesh_core.storage.replication.fingerprint(event_ids)[source]

NIP-77 Negentropy V1 fingerprint: 16 bytes.

acc = sum(id as 32-byte little-endian uint) mod 2**256 out = SHA256(acc_le_32 || varint(count))[:16]

The count is what stops the additive structure colliding trivially - two different sets summing to the same accumulator still differ in cardinality almost always, and always for the sets a shard actually contains.

Parameters:

event_ids (Iterable[str])

Return type:

str

fleetmesh_core.storage.replication.fingerprint_union(ids_a, ids_b)[source]

Demonstrates the property the sum buys: disjoint sub-ranges compose.

Not a wire operation - it exists so the tests can assert that splitting a range and fingerprinting the halves is equivalent to fingerprinting the whole, which is the reason this construction was chosen over a chained hash.

Parameters:
Return type:

str

fleetmesh_core.storage.replication.sort_for_enumeration(events)[source]

(created_at ASC, id ASC).

Required for enumerate_range, whose output order is observable. The fingerprint does not need it: a sum is order-independent, which is a property worth having rather than a rule worth stating.

Parameters:

events (Iterable[NostrEvent | UnsignedNostrEvent | Mapping[str, Any]])

Return type:

list[str]

fleetmesh_core.storage.replication.negotiate(mine, theirs)[source]

Rank the mutually supported transports, most preferred first.

Each side advertises {scheme: preference}, low first. The ranking is by the sum of both preferences, so neither side dictates: a transport one peer hates cannot win just because the other loves it. Equal sums rank by scheme name, ascending bytewise - constants.json transport.ladder_tie_break - so two implementations pick the same rung for the same descriptor pair.

Parameters:
Return type:

list[str]

fleetmesh_core.storage.replication.ladder_exhausted(attempted, mine, theirs)[source]

A peer may only be reported unreachable once every shared rung failed.

Parameters:
Return type:

bool

6.6. Blossom Media Authorization

Blossom authorization and the mirror exchange (profile blossom-mirror).

BUD-01 names the kind 24242 authorization event: tags t (the verb), x (the blob sha256), expiration; the content is free text. It travels as Authorization: Nostr <base64(JSON event)>.

BUD-04 is PUT /mirror with body {“url”: <source>}; the server fetches, hashes and refuses with 401 unless the hash is the one the x tag committed to. This node’s profile narrows it for replication: the verb is mirror when a node asks a partner to copy, the source is the owner’s public Blossom URL <base>/<sha256>, and the 200 body is the standard blob descriptor {url, sha256, size, type, uploaded} plus cid.

Verification (check_auth) is pure and takes now so a second node and a vector can both hold the same event to the same verdict.

exception fleetmesh_core.storage.blossom_auth.AuthError[source]

Bases: FleetmeshError, ValueError

class fleetmesh_core.storage.blossom_auth.MirrorRequest[source]

Bases: TypedDict

url: str
class fleetmesh_core.storage.blossom_auth.BlobDescriptor[source]

Bases: TypedDict

url: str
sha256: str
size: int
type: str
uploaded: int
cid: str
fleetmesh_core.storage.blossom_auth.auth_event(signer_sk, verb, sha256_hex, now, expiration_seconds=300, content='')[source]

A signed kind 24242 for one verb over one blob.

Parameters:
  • signer_sk (str)

  • verb (str)

  • sha256_hex (str)

  • now (int)

  • expiration_seconds (int)

  • content (str)

Return type:

NostrEvent

fleetmesh_core.storage.blossom_auth.auth_header(ev)[source]
Parameters:

ev (NostrEvent | UnsignedNostrEvent | Mapping[str, Any])

Return type:

str

fleetmesh_core.storage.blossom_auth.parse_auth_header(value)[source]

The event the header carried, unverified; None if it is not one.

Parameters:

value (str)

Return type:

NostrEvent | UnsignedNostrEvent | Mapping[str, Any] | None

fleetmesh_core.storage.blossom_auth.check_auth(ev, verb, now, sha256_hex=None)[source]

The signer’s pubkey, or AuthError naming the first rule the event breaks: signature, kind, verb, hash, expiry - in that order.

Parameters:
Return type:

str

fleetmesh_core.storage.blossom_auth.x_tag(ev)[source]

The lowercase sha256 the event commits to, if any.

Parameters:

ev (NostrEvent | UnsignedNostrEvent | Mapping[str, Any])

Return type:

str | None

fleetmesh_core.storage.blossom_auth.mirror_request(source_base, sha256_hex)[source]

The body a replicating node PUTs to a partner’s /mirror.

Parameters:
  • source_base (str)

  • sha256_hex (str)

Return type:

MirrorRequest

fleetmesh_core.storage.blossom_auth.mirror_response(sha256_hex, size, content_type, uploaded, cid)[source]

The 200 body: the blob descriptor, url relative to the serving base.

Parameters:
Return type:

BlobDescriptor

6.7. Linked Data & DAGs

Linked data: schema.org projections of the documents a node publishes for others to find.

A release, a paid-fetch quote, a blob descriptor and a node descriptor are each already a typed structure with a wire form that some other standard fixes (npack kind 9900, the paid-fetch-402 body, Blossom BUD-02, kind 11801). Those stay the source of truth: signed, byte-exact, unchanged here.

What this module adds is a second reading of each, in the vocabulary at https://schema.org, as JSON-LD - plain dicts with an @context. A consumer that has never read fleetmesh’s documents but knows what an Offer with a priceSpecification is, or a SoftwareApplication with a softwareVersion, can read what a node says without us in between. That is the whole point; nothing in the protocol depends on these.

Rules, so the two readings never drift:

  • Projections only for public descriptive documents. Operator state, configuration and protocol wire bodies are not projected; schema.org has no words for them and nobody outside the node reads them.

  • Every term emitted exists in the pinned vocabulary (schema/schemaorg-terms.json, version; the pin also names the one @context emitted, which the unplug scan reads from there - a node reaches nothing). terms() lists what a projection used; the suite and fleetmesh.org’s checker hold it against the pin, so a typo or a deprecated term never ships.

  • Nothing is invented. A projection carries only facts the source carries; where schema.org has no exact slot the fact goes under additionalProperty as a PropertyValue whose propertyID is the fleetmesh field name, so the mapping is visible in the output itself.

  • Money is in millisatoshi and schema.org prices are in a currency. The price is in BTC as a decimal string (schema.org names the ticker form for cryptocurrencies) and the exact msat rides beside it in a PropertyValue with propertyID price_msat. A reader that honours the convention loses nothing; one that does not still gets the right order of magnitude in a unit it knows.

  • Times are Unix seconds in the sources and ISO 8601 UTC in schema.org; iso() is the one conversion.

Standard library only. Nothing here reads the network or the wall clock.

fleetmesh_core.storage.linked_data.iso(unix)[source]

Unix seconds as ISO 8601 UTC, the form schema.org’s Date/DateTime take.

Parameters:

unix (int)

Return type:

str

fleetmesh_core.storage.linked_data.btc(msat)[source]

msat as a decimal BTC string with no float on the way: 1 msat is ‘0.00000000001’.

Parameters:

msat (int)

Return type:

str

fleetmesh_core.storage.linked_data.software_application(release)[source]

A release.ReleaseDict as a schema:SoftwareApplication.

name/version/os/arch have exact slots. The artifact digest is the sha256 of a MediaObject in associatedMedia (a release names the artifact’s hash, not a URL). The signer is the author, as a did:nostr. protocols and artifact_event have no slot and are kept by name.

Parameters:

release (ReleaseDict | Mapping[str, Any])

Return type:

MutableMapping[str, Any]

fleetmesh_core.storage.linked_data.offer(quote, url=None)[source]

A paidfetch.QuoteBody as a schema:Offer for one blob.

The item offered is the blob as a MediaObject (sha256, contentSize in bytes). The price is a CompoundPriceSpecification: one component per items[] entry, each naming who it is for (to, payee) and carrying its msat exactly. offers[] rails become acceptedPaymentMethod texts (cashu, …); their requests are rail-specific and not projected.

Parameters:
Return type:

MutableMapping[str, Any]

fleetmesh_core.storage.linked_data.media_object(blob)[source]

A blossom_auth.BlobDescriptor as a schema:MediaObject.

Every BUD-02 field has an exact slot: url -> contentUrl, sha256 -> sha256, size -> contentSize (bytes), type -> encodingFormat, uploaded -> uploadDate. An IPFS cid, when present, is a second identifier.

Parameters:

blob (BlobDescriptor | Mapping[str, Any])

Return type:

MutableMapping[str, Any]

fleetmesh_core.storage.linked_data.rung_of(scheme)[source]
Parameters:

scheme (str)

Return type:

str

fleetmesh_core.storage.linked_data.service(desc, inbox=None)[source]

A signed kind 11801 - and the node’s kind 10050, when the caller has it - as a schema:Service.

The node is the Service, identified by its did:nostr; the operator is the provider. availableChannel is the transport ladder: one ServiceChannel per endpoint tag, in the order the router walks them (preference, low first), each naming its scheme, its rung and its preference; then, last, the floor: one ServiceChannel per inbox relay the node’s kind 10050 names (relaylist.dm_relays), rung nostr, floor: true - the store-and-forward channel every node has and tries last. The floor is read from the 10050 and not from the descriptor because that is where the protocol keeps it; an endpoint wss on a descriptor says the node hosts a relay, a different fact, and is projected as such (rung nostr, no floor bit). serviceType is the node class (leaf/edge/anchor), which declares connectivity, never capability; c tags (protocol identifiers and capabilities such as hosting a relay) are kept by name, as are the policy digest and software. expiration -> expires.

Parameters:
Return type:

MutableMapping[str, Any]

fleetmesh_core.storage.linked_data.standing(pubkey, tell)[source]

A standing.StandingTellBody about pubkey, as the thing it is about.

Not a schema:Review. A tier is the outcome of this node’s own windows (S0..S4 with meaning: S1 halves a limit, S4 revokes), replaceable and withdrawable; a reviewRating is an ordinal any aggregator would average. So the subject is an Organization (a node, operated, not a person) and the tell rides as PropertyValues named by fleetmesh field. The teller is tell[‘target’]’s evaluator - the node serving this - and is not repeated; the HTTP origin says who is speaking.

Parameters:
Return type:

MutableMapping[str, Any]

class fleetmesh_core.storage.linked_data.Vocabulary[source]

Bases: TypedDict

origin: str
context: str
version: str
types: list[str]
properties: list[str]
fleetmesh_core.storage.linked_data.vocabulary()[source]

The pinned schema.org vocabulary, schema/schemaorg-terms.json.

Return type:

Vocabulary

fleetmesh_core.storage.linked_data.terms(doc)[source]

(types, properties) a projection used, walked recursively. Keys with an @ are JSON-LD keywords, not vocabulary.

Parameters:

doc (Any)

Return type:

tuple[set[str], set[str]]

fleetmesh_core.storage.linked_data.unknown_terms(doc, vocab=None)[source]

Terms in doc the pinned vocabulary does not define. Empty is the only acceptable answer for anything a node emits.

Parameters:
Return type:

list[str]

6.8. Content Addressable aRchives (CAR v1)

CAR (Content Addressable aRchive) v1 reader and writer.

Standard library only. Format per IPLD CAR v1 specification: - Header: DAG-CBOR map {“version”: 1, “roots”: [CIDLink, …]} prefixed with unsigned varint length. - Blocks: Sequence of varint(len(cid) + len(data)) || binary_cid || data.

exception fleetmesh_core.storage.car.CarError[source]

Bases: FleetmeshError

Base error for CAR file processing.

fleetmesh_core.storage.car.read_car(car_bytes)[source]

Parse CAR v1 byte string into (roots, {cid: block_bytes}).

Parameters:

car_bytes (bytes)

Return type:

tuple[list[CIDLink], dict[CIDLink, bytes]]

fleetmesh_core.storage.car.write_car(roots, blocks)[source]

Serialize roots and blocks into a CAR v1 byte string.

Parameters:
Return type:

bytes

class fleetmesh_core.storage.car.CarReader[source]

Bases: object

Sequential reader for CAR v1 archive streams.

__init__(stream)[source]
Parameters:

stream (BinaryIO)

class fleetmesh_core.storage.car.CarWriter[source]

Bases: object

Sequential writer for CAR v1 archive streams.

__init__(stream, roots)[source]
Parameters:
write_block(cid, data)[source]

Write a single block to the CAR stream.

Parameters:
Return type:

None

6.9. ATProto Merkle Search Tree (MST)

Merkle Search Tree (MST) for ATProto repositories.

Standard library only. Implements: - 16-fanout Merkle Search Tree with deterministic layer assignment

based on leading zero hex nibbles of SHA-256(key).

  • Node DAG-CBOR serialization with common prefix compaction (p, k, v, t, l).

  • Deterministic tree operations: put, get, delete, list, diff.

  • Empty tree root CID bafyreie5737gdxlw5i64vzichcalba3z2v5n6icifvx5xytvske7mr3hpm.

exception fleetmesh_core.storage.mst.MSTError[source]

Bases: FleetmeshError

Base error for MST operations.

fleetmesh_core.storage.mst.leading_zeros(key)[source]

Calculate the MST layer of a key by counting leading zeros in SHA-256 hex digest.

Parameters:

key (str)

Return type:

int

class fleetmesh_core.storage.mst.TreeEntry[source]

Bases: object

An entry within an MST node pointing to a record value and an optional right subtree.

__init__(key, val, tree=None)[source]
Parameters:
key
val
tree
class fleetmesh_core.storage.mst.MSTNode[source]

Bases: object

An MST node holding a left subtree pointer and an ordered list of TreeEntries.

__init__(left=None, entries=None)[source]
Parameters:
left
entries
to_cbor_dict()[source]

Convert node to DAG-CBOR dict with prefix compaction per ATProto spec.

Return type:

Mapping[str, Any]

to_cbor()[source]

Serialize node to canonical DAG-CBOR bytes and compute its CID.

Return type:

tuple[CIDLink, bytes]

classmethod from_cbor(data)[source]

Deserialize an MST node from DAG-CBOR bytes.

Parameters:

data (bytes)

Return type:

MSTNode

class fleetmesh_core.storage.mst.MST[source]

Bases: object

Merkle Search Tree backed by an in-memory or pluggable blockstore.

__init__(root_cid=None, blocks=None, blockstore=None)[source]
Parameters:
property root_cid: CIDLink
property blocks: dict[CIDLink, bytes]
get(key)[source]

Retrieve the CIDLink value stored at key, or None if not found.

Parameters:

key (str)

Return type:

CIDLink | None

put(key, val)[source]

Insert or update a key/value mapping and return the new root CID.

Parameters:
Return type:

CIDLink

delete(key)[source]

Remove a key and return the new root CID.

Parameters:

key (str)

Return type:

CIDLink

items(prefix='')[source]

Return all (key, val) pairs in alphabetical order, optionally filtered by prefix.

Parameters:

prefix (str)

Return type:

list[tuple[str, CIDLink]]

keys(prefix='')[source]
Parameters:

prefix (str)

Return type:

list[str]

diff(other)[source]

Compute difference between this MST and another MST.

Returns (created, updated, deleted): - created: keys in other not in self [(key, val), …] - updated: keys in both with different vals [(key, val), …] - deleted: keys in self not in other [key, …]

Parameters:

other (MST)

Return type:

tuple[list[tuple[str, CIDLink]], list[tuple[str, CIDLink]], list[str]]