4. Bulk Data Synchronization (StrfryPipe)

When seeding a new Fleetmesh node or migrating historical data between repositories, transferring millions of events over individual WebSocket frames introduces significant latency and protocol overhead.

StrfryPipe provides a high-throughput, bulk streaming pipeline connecting the node’s SQLite eventstore with strfry’s native command-line data migration utilities (strfry export and strfry import).

4.1. Architecture & Operations

The pipeline operates by streaming line-delimited JSON (JSONL) records over standard Unix pipes:

Export:
┌───────────────────────┐                  ┌────────────────────────┐
│ Fleetmesh EventStore  │ ──(Streaming)──► │ strfry import          │
│ (store.sqlite)        │                  │ (Writes to LMDB)       │
└───────────────────────┘                  └────────────────────────┘

Import:
┌───────────────────────┐                  ┌────────────────────────┐
│ strfry export         │ ──(Streaming)──► │ Fleetmesh EventStore   │
│ (Reads from LMDB)     │                  │ (Batched SQLite Write) │
└───────────────────────┘                  └────────────────────────┘

4.2. Exporting Node Events to strfry

To dump events from a Fleetmesh node’s local SQLite store and ingest them into strfry’s LMDB database:

# Export via meshnode CLI
meshnode strfry /var/lib/fleetmesh export --out events.jsonl

# Pipe directly into strfry import
meshnode strfry /var/lib/fleetmesh export | strfry import

# Alternatively, using the standalone module
python3 -m fleetmesh_strfry export --dir /var/lib/fleetmesh | strfry import

Key behaviors: - Emits events in canonical NIP-01 JSON format, one per line. - Respects store index ordering to ensure linear disk writes in LMDB. - Skips ephemeral events (kinds 20000–29999) that should not be permanently stored.

4.3. Importing strfry Events into the Node

To populate a Fleetmesh node’s SQLite store from an existing strfry archive or backup:

# Export from strfry and import into node
strfry export | meshnode strfry /var/lib/fleetmesh import

# Import from a saved JSONL file
python3 -m fleetmesh_strfry import --dir /var/lib/fleetmesh --in backup.jsonl

Performance Optimizations: - Chunked Batching: Events are read and committed in transactions of 1,000–5,000 events, maximizing SQLite write speed. - Duplicate Suppression: Uses SQLite INSERT OR IGNORE semantics indexed by event ID, allowing imports to resume seamlessly if interrupted. - Replaceable Event Pruning: Automatically resolves parameterized replaceable (kinds 30000–39999) and replaceable (kinds 10000–19999) events, retaining only the newest record per coordinate.