Bulk Data Synchronization (StrfryPipe) ======================================== When seeding a new Fleetmesh node or migrating historical data between repositories, transferring millions of events over individual WebSocket frames introduces significant latency and protocol overhead. `StrfryPipe` provides a high-throughput, bulk streaming pipeline connecting the node's SQLite eventstore with strfry's native command-line data migration utilities (`strfry export` and `strfry import`). Architecture & Operations ------------------------- The pipeline operates by streaming line-delimited JSON (JSONL) records over standard Unix pipes: .. code-block:: text Export: ┌───────────────────────┐ ┌────────────────────────┐ │ Fleetmesh EventStore │ ──(Streaming)──► │ strfry import │ │ (store.sqlite) │ │ (Writes to LMDB) │ └───────────────────────┘ └────────────────────────┘ Import: ┌───────────────────────┐ ┌────────────────────────┐ │ strfry export │ ──(Streaming)──► │ Fleetmesh EventStore │ │ (Reads from LMDB) │ │ (Batched SQLite Write) │ └───────────────────────┘ └────────────────────────┘ Exporting Node Events to strfry ------------------------------- To dump events from a Fleetmesh node's local SQLite store and ingest them into strfry's LMDB database: .. code-block:: sh # Export via meshnode CLI meshnode strfry /var/lib/fleetmesh export --out events.jsonl # Pipe directly into strfry import meshnode strfry /var/lib/fleetmesh export | strfry import # Alternatively, using the standalone module python3 -m fleetmesh_strfry export --dir /var/lib/fleetmesh | strfry import Key behaviors: - Emits events in canonical NIP-01 JSON format, one per line. - Respects store index ordering to ensure linear disk writes in LMDB. - Skips ephemeral events (kinds 20000–29999) that should not be permanently stored. Importing strfry Events into the Node ------------------------------------- To populate a Fleetmesh node's SQLite store from an existing strfry archive or backup: .. code-block:: sh # Export from strfry and import into node strfry export | meshnode strfry /var/lib/fleetmesh import # Import from a saved JSONL file python3 -m fleetmesh_strfry import --dir /var/lib/fleetmesh --in backup.jsonl Performance Optimizations: - **Chunked Batching**: Events are read and committed in transactions of 1,000–5,000 events, maximizing SQLite write speed. - **Duplicate Suppression**: Uses SQLite `INSERT OR IGNORE` semantics indexed by event ID, allowing imports to resume seamlessly if interrupted. - **Replaceable Event Pruning**: Automatically resolves parameterized replaceable (kinds 30000–39999) and replaceable (kinds 10000–19999) events, retaining only the newest record per coordinate.