1.20. Record-Then-Answer Accounting
1.20.1. Record then answer
A working note on one class of bug. Not part of the published site and not a specification. It exists because five races surfaced in one day, four of them in shipped code, and three of those were the same mistake in three different files:
state written after the message that announces it
The fixes were going in one at a time as CI tripped over them. This is the deliberate pass instead: every place this node sends something, read once and asked the same question.
Can a peer - or an operator, or a test - observe the state this line writes before it is written?
1.20.1.1. The rule
Write it, then say it. A message is the last thing a handler does, and everything the message entitles a reader to know is already on record when it leaves.
The reason it has to be a rule rather than a judgement is the failure mode. Getting it wrong costs nothing on a fast machine, because the sender usually wins a race measured in bytecodes. It costs a 1-in-n failure on a loaded runner, and what the loser is told is not “later” but “no”. A reader that asks whether a session happened and is told nothing cannot tell that answer from the session never having happened. That is why these arrive as flakes in other people’s tests and not as bugs where they live.
Two consequences, and they pull in opposite directions, so both are written down:
One-shot artefacts record first and undo the record if the send fails. Otherwise the choice is between a record a peer can contradict and a statement this node believes it made and will never make again. Neither is acceptable, and neither has to be accepted: the record goes first, and the failure takes it back. What is left is doing the thing twice, which is the direction that is noise rather than damage whenever the artefact is signed and idempotent.
Repeating announcements stamp first and do not undo it. An advertisement
that goes out every sixty seconds needs no retry - the next one is the retry -
and a stamp written only on success turns a link that refuses the send into a
packet storm. lan._announce is this case, presence’s heartbeat is this
case, and the difference from the paragraph above is whether missing one is
recoverable by doing nothing.
1.20.1.2. What the pass found
Three defects, all in shipped code, all of them load-dependent and none of them visible in a fast local run.
The relay face registered a subscription after the EOSE that announced it.
relayserver._on_req replayed the stored events, sent the EOSE, and only then
put the filters where _fanout looks. relay.subscribe(ready=True) documents
waiting for that EOSE as the way for a caller to know its subscription
exists, so a client that did exactly what it is told - wait, then cause an
event - could still have the event fanned out to nobody. For an ephemeral kind
that is final: no relay stores one, so no timeout on the reader recovers it.
This is the whole liveness path for heartbeats, for gift-wrapped transport
messages and for the stored-message acks. It is also, almost certainly, the
test_identity flake that had been recorded as pre-existing and unrelated.
The cost of the new order is that an event arriving during the replay can go out live and appear in the replay too. NIP-01 requires a client to tolerate a duplicate id; nobody can recover a lost ephemeral event.
The peering acceptor sent its offer before recording the grant behind it.
peering_wire.handle sent the offer, then called grants.put. The accept
handler refuses an accept from a node it holds no live issued grant for, which
is what stops a stranger’s accept from making this node hold a grant it never
offered. Sent first, that guard fires on the node’s own offer: the proposer
answers an offer the instant it reads one, that accept lands on another
connection thread, and if it arrived first it was ignored as coming from a node
this node did not offer to. The peering then stood half-made - the proposer
pulling under a grant the issuer had recorded, the issuer holding nothing back
and nothing anywhere said so.
The handover memo was kept over a send that never left the node. _send
does not raise: it records what could not be sent and returns. So marking the
offer after it meant marking it whether or not anything left, and the same set
is offered to the same peer once. A relay that took nothing therefore produced
a replica that should have been able to assert a shard and silently could not.
Now the memo is written first and taken back when _send reports nothing
left, in _offer_handover and in hand_over_all both.
Two more changes for consistency rather than for a bug: transport._ack
counts before it sends and uncounts if it raised, the way the relay face counts
what it refuses; and lan’s sends are guarded, because a joined multicast
socket is not a link that forwards and an OSError from one send was ending the
whole compartment’s tick.
1.20.1.3. The other side of it
There is a version of this that no ordering inside the sender can fix. pull
returns the moment the initiator hands its own done to the transport. The
responder’s record of that session is written when the responder receives the
done, which is strictly later, and nothing on the wire tells the initiator when
the far side has finished writing. So a reader of the other node’s record has
to wait for it.
This is what test_sync_wire’s split-delivery assertion was doing wrong, and
it is why the ordering fix in sync_wire did not close it: the session that
holds events back for the next one ends with the initiator’s done, not the
responder’s. test_budget and test_reports_wire had already grown a
wait_for helper for exactly this; test_sync_wire now has one too.
The invariant a responder can keep, and does, is the one for sessions it ends itself: at the moment its done leaves, the record is already there.
1.20.1.4. The call sites, and the verdict on each
Every _send, send, send_text and publish outside the tests, read once.
Recorded so the next person does not have to sweep it again.
Where |
Order |
Verdict |
|---|---|---|
|
records |
the one place that decides nothing left the node |
|
session registered, then the reply |
correct |
|
refusal recorded, then the done |
correct |
|
|
correct, fixed 2026-09-26 (a6f1cfa) |
|
listener registered, then the open |
correct |
|
was send then memo |
fixed here |
|
was send then memo |
fixed here |
|
grant written and published, then the amend |
correct |
|
|
correct |
|
was offer then |
fixed here |
|
write nothing |
not applicable |
|
chain persisted, then the receipt |
correct, and says so in a comment |
|
reply built from state already written |
correct |
|
|
correct, and says so in a comment |
|
was send then counter |
fixed here, cosmetic |
|
stored and counted, then the OK |
correct, fixed 2026-09-26 (9ce0600) |
|
|
correct |
|
was replay, EOSE, then register |
fixed here |
|
reconciler state written, then the frame |
correct |
|
|
correct |
|
CLOSE, then a local pop |
local only |
|
grants zeroed, then the peer is told |
correct |
|
publish, then the due stamp |
repeating; a raise retries |
|
publish, then the fingerprint memo |
repeating; a raise retries |
|
sidecar call, then the id memo |
repeating; a raise retries |
|
was unguarded sends |
fixed here |
|
publish, then persist |
left; see below |
1.20.1.5. What this pass deliberately left
reporter.publish persists after it publishes. The report is on the
relays and the node’s own record of having published it follows. This is not a
thread race - no peer can ask this node about it - it is a crash-durability
question: a process that dies in between leaves a public report that withdraw
will say was never published. Closing it properly means persisting an intent
before the publish and reconciling on restart, which is a state machine and not
a reordering. Worth doing. Not worth pretending it is this.
discovery_wire sets _claims before it knows a relay took the claim.
publish_all’s result is discarded there, so published_claim can name a
claim no relay holds, and a handover can then be offered over it. The artefact
stands on its own and the replica’s own claim carries the chain, so nothing
false is asserted - but a reader cannot see that the signer ever claimed the
shard. The fix is reading the publish result, which is a change with its own
questions about what to do with the answer.
Neither is a race. Both are written here so that the next sweep does not have to rediscover them.