4. Bulk Data Synchronization (StrfryPipe)
When seeding a new Fleetmesh node or migrating historical data between repositories, transferring millions of events over individual WebSocket frames introduces significant latency and protocol overhead.
StrfryPipe provides a high-throughput, bulk streaming pipeline connecting the node’s SQLite eventstore with strfry’s native command-line data migration utilities (strfry export and strfry import).
4.1. Architecture & Operations
The pipeline operates by streaming line-delimited JSON (JSONL) records over standard Unix pipes:
Export:
┌───────────────────────┐ ┌────────────────────────┐
│ Fleetmesh EventStore │ ──(Streaming)──► │ strfry import │
│ (store.sqlite) │ │ (Writes to LMDB) │
└───────────────────────┘ └────────────────────────┘
Import:
┌───────────────────────┐ ┌────────────────────────┐
│ strfry export │ ──(Streaming)──► │ Fleetmesh EventStore │
│ (Reads from LMDB) │ │ (Batched SQLite Write) │
└───────────────────────┘ └────────────────────────┘
4.2. Exporting Node Events to strfry
To dump events from a Fleetmesh node’s local SQLite store and ingest them into strfry’s LMDB database:
# Export via meshnode CLI
meshnode strfry /var/lib/fleetmesh export --out events.jsonl
# Pipe directly into strfry import
meshnode strfry /var/lib/fleetmesh export | strfry import
# Alternatively, using the standalone module
python3 -m fleetmesh_strfry export --dir /var/lib/fleetmesh | strfry import
Key behaviors: - Emits events in canonical NIP-01 JSON format, one per line. - Respects store index ordering to ensure linear disk writes in LMDB. - Skips ephemeral events (kinds 20000–29999) that should not be permanently stored.
4.3. Importing strfry Events into the Node
To populate a Fleetmesh node’s SQLite store from an existing strfry archive or backup:
# Export from strfry and import into node
strfry export | meshnode strfry /var/lib/fleetmesh import
# Import from a saved JSONL file
python3 -m fleetmesh_strfry import --dir /var/lib/fleetmesh --in backup.jsonl
Performance Optimizations: - Chunked Batching: Events are read and committed in transactions of 1,000–5,000 events, maximizing SQLite write speed. - Duplicate Suppression: Uses SQLite INSERT OR IGNORE semantics indexed by event ID, allowing imports to resume seamlessly if interrupted. - Replaceable Event Pruning: Automatically resolves parameterized replaceable (kinds 30000–39999) and replaceable (kinds 10000–19999) events, retaining only the newest record per coordinate.