1.20. Record-Then-Answer Accounting

1.20.1. Record then answer

A working note on one class of bug. Not part of the published site and not a specification. It exists because five races surfaced in one day, four of them in shipped code, and three of those were the same mistake in three different files:

state written after the message that announces it

The fixes were going in one at a time as CI tripped over them. This is the deliberate pass instead: every place this node sends something, read once and asked the same question.

Can a peer - or an operator, or a test - observe the state this line writes before it is written?

1.20.1.1. The rule

Write it, then say it. A message is the last thing a handler does, and everything the message entitles a reader to know is already on record when it leaves.

The reason it has to be a rule rather than a judgement is the failure mode. Getting it wrong costs nothing on a fast machine, because the sender usually wins a race measured in bytecodes. It costs a 1-in-n failure on a loaded runner, and what the loser is told is not “later” but “no”. A reader that asks whether a session happened and is told nothing cannot tell that answer from the session never having happened. That is why these arrive as flakes in other people’s tests and not as bugs where they live.

Two consequences, and they pull in opposite directions, so both are written down:

One-shot artefacts record first and undo the record if the send fails. Otherwise the choice is between a record a peer can contradict and a statement this node believes it made and will never make again. Neither is acceptable, and neither has to be accepted: the record goes first, and the failure takes it back. What is left is doing the thing twice, which is the direction that is noise rather than damage whenever the artefact is signed and idempotent.

Repeating announcements stamp first and do not undo it. An advertisement that goes out every sixty seconds needs no retry - the next one is the retry - and a stamp written only on success turns a link that refuses the send into a packet storm. lan._announce is this case, presence’s heartbeat is this case, and the difference from the paragraph above is whether missing one is recoverable by doing nothing.

1.20.1.2. What the pass found

Three defects, all in shipped code, all of them load-dependent and none of them visible in a fast local run.

The relay face registered a subscription after the EOSE that announced it. relayserver._on_req replayed the stored events, sent the EOSE, and only then put the filters where _fanout looks. relay.subscribe(ready=True) documents waiting for that EOSE as the way for a caller to know its subscription exists, so a client that did exactly what it is told - wait, then cause an event - could still have the event fanned out to nobody. For an ephemeral kind that is final: no relay stores one, so no timeout on the reader recovers it. This is the whole liveness path for heartbeats, for gift-wrapped transport messages and for the stored-message acks. It is also, almost certainly, the test_identity flake that had been recorded as pre-existing and unrelated.

The cost of the new order is that an event arriving during the replay can go out live and appear in the replay too. NIP-01 requires a client to tolerate a duplicate id; nobody can recover a lost ephemeral event.

The peering acceptor sent its offer before recording the grant behind it. peering_wire.handle sent the offer, then called grants.put. The accept handler refuses an accept from a node it holds no live issued grant for, which is what stops a stranger’s accept from making this node hold a grant it never offered. Sent first, that guard fires on the node’s own offer: the proposer answers an offer the instant it reads one, that accept lands on another connection thread, and if it arrived first it was ignored as coming from a node this node did not offer to. The peering then stood half-made - the proposer pulling under a grant the issuer had recorded, the issuer holding nothing back

  • and nothing anywhere said so.

The handover memo was kept over a send that never left the node. _send does not raise: it records what could not be sent and returns. So marking the offer after it meant marking it whether or not anything left, and the same set is offered to the same peer once. A relay that took nothing therefore produced a replica that should have been able to assert a shard and silently could not. Now the memo is written first and taken back when _send reports nothing left, in _offer_handover and in hand_over_all both.

Two more changes for consistency rather than for a bug: transport._ack counts before it sends and uncounts if it raised, the way the relay face counts what it refuses; and lan’s sends are guarded, because a joined multicast socket is not a link that forwards and an OSError from one send was ending the whole compartment’s tick.

1.20.1.3. The other side of it

There is a version of this that no ordering inside the sender can fix. pull returns the moment the initiator hands its own done to the transport. The responder’s record of that session is written when the responder receives the done, which is strictly later, and nothing on the wire tells the initiator when the far side has finished writing. So a reader of the other node’s record has to wait for it.

This is what test_sync_wire’s split-delivery assertion was doing wrong, and it is why the ordering fix in sync_wire did not close it: the session that holds events back for the next one ends with the initiator’s done, not the responder’s. test_budget and test_reports_wire had already grown a wait_for helper for exactly this; test_sync_wire now has one too.

The invariant a responder can keep, and does, is the one for sessions it ends itself: at the moment its done leaves, the record is already there.

1.20.1.4. The call sites, and the verdict on each

Every _send, send, send_text and publish outside the tests, read once. Recorded so the next person does not have to sweep it again.

Where

Order

Verdict

sync_wire._send

records unsent on failure

the one place that decides nothing left the node

sync_wire.handle open

session registered, then the reply

correct

sync_wire.handle refusals

refusal recorded, then the done

correct

sync_wire.handle done

_finish, then the done

correct, fixed 2026-09-26 (a6f1cfa)

sync_wire.pull

listener registered, then the open

correct

sync_wire._offer_handover

was send then memo

fixed here

sync_wire.hand_over_all

was send then memo

fixed here

peering_wire.amend

grant written and published, then the amend

correct

peering_wire.propose

grants.put, then the accept

correct

peering_wire.handle propose

was offer then grants.put

fixed here

peering_wire.handle re-offer, decline

write nothing

not applicable

budget.send_receipt

chain persisted, then the receipt

correct, and says so in a comment

budget.handle

reply built from state already written

correct

transport.send

awaiting held before the first relay is asked

correct, and says so in a comment

transport._ack

was send then counter

fixed here, cosmetic

relayserver._on_event

stored and counted, then the OK

correct, fixed 2026-09-26 (9ce0600)

relayserver._on_auth

_authed written, then the OK

correct

relayserver._on_req

was replay, EOSE, then register

fixed here

relayserver._neg_*

reconciler state written, then the frame

correct

relay client query, subscribe

_subs written, then the REQ

correct

relay client close

CLOSE, then a local pop

local only

app.terminate

grants zeroed, then the peer is told

correct

presence descriptor, heartbeat

publish, then the due stamp

repeating; a raise retries

discovery_wire discovery, coverage

publish, then the fingerprint memo

repeating; a raise retries

announce._publish

sidecar call, then the id memo

repeating; a raise retries

lan._announce, _serve, stop

was unguarded sends

fixed here

reporter.publish

publish, then persist

left; see below

1.20.1.5. What this pass deliberately left

reporter.publish persists after it publishes. The report is on the relays and the node’s own record of having published it follows. This is not a thread race - no peer can ask this node about it - it is a crash-durability question: a process that dies in between leaves a public report that withdraw will say was never published. Closing it properly means persisting an intent before the publish and reconciling on restart, which is a state machine and not a reordering. Worth doing. Not worth pretending it is this.

discovery_wire sets _claims before it knows a relay took the claim. publish_all’s result is discarded there, so published_claim can name a claim no relay holds, and a handover can then be offered over it. The artefact stands on its own and the replica’s own claim carries the chain, so nothing false is asserted - but a reader cannot see that the signer ever claimed the shard. The fix is reading the publish result, which is a change with its own questions about what to do with the answer.

Neither is a race. Both are written here so that the next sweep does not have to rediscover them.