3.1.3. Operations Plane (fleetmesh_node.planes.operations)

The Operations Plane manages autonomous node tasks, timer ticks, and signed software update lifecycles.

3.1.3.1. Scheduler Compartment

The pulls this node is configured to make, and every reason one did not happen.

This was seventy lines in the middle of the node’s tick, and it was the worst place in the node to be a stranger. Five things can stop a scheduled pull: the peer has no peering yet and the proposal raised; the proposal came back declined; this node is over its own ceiling and has stopped pulling as the last shed rung; the grant this node holds covers no session this window; or the session itself failed. Every one of them was appended to the tick’s return value as a tuple, printed once, and then gone. meshnode refusals could not show any of them, health counted none of them, and an operator watching a peer go quiet had to read the journal back far enough to find the line - if the journal still had it.

A refusal that is not recorded is indistinguishable from silence. This compartment is the five of them given a channel.

class fleetmesh_node.planes.operations.scheduler.compartment.SchedulerConfig[source]

Bases: ParamBlock

SchedulerConfig(DERIVED: ‘bool’ = False, default_pull_every: ‘int’ = 900, decline_retry: ‘int’ = 3600)

key: ClassVar[str] = 'scheduler'
name = 'scheduler'
DERIVED: bool = False
PARAMS: tuple[Parameter[Any], ...] = (Parameter(key='default_pull_every', name='Default Pull Cadence', description='Seconds between pulls when peer grant has no rate', type=<class 'int'>, default=900, must='', tags=('scheduler', 'pull'), required=False, choices=(), verify_regex='', live=False, item=None, field='', secret=False, requires='', sources=(), minimum=None, maximum=None, env='', validate=None, coerce=None, short='', cli_name='', positional=False, nargs='', cli=True), Parameter(key='decline_retry', name='Decline Retry Horizon', description='Seconds before re-proposing to declined peer', type=<class 'int'>, default=3600, must='', tags=('scheduler', 'pull'), required=False, choices=(), verify_regex='', live=False, item=None, field='', secret=False, requires='', sources=(), minimum=None, maximum=None, env='', validate=None, coerce=None, short='', cli_name='', positional=False, nargs='', cli=True))
default_pull_every: int = 900
decline_retry: int = 3600
__init__(DERIVED=False, default_pull_every=900, decline_retry=3600)
Parameters:
  • DERIVED (bool)

  • default_pull_every (int)

  • decline_retry (int)

Return type:

None

class fleetmesh_node.planes.operations.scheduler.compartment.Scheduler[source]

Bases: Compartment

requires: tuple[str, ...] = ('budget', 'capacity', 'strain')
name: str = 'scheduler'
reads: tuple[str, ...] = ('peers',)
live: tuple[str, ...] = ('peers',)
config_cls

alias of SchedulerConfig

CHANNELS = (Channel(name='pull_scheduled', description='Pull scheduled for peer and shard', level='note', wire=None, say='scheduled pull for peer {peer} on shard {shard}', tags=('scheduler', 'pull')), Channel(name='pull_refused', description='Scheduled pull refused', level='note', wire=None, say='pull refused for peer {peer} on shard {shard}: {reason}', tags=('scheduler', 'pull')))
METRICS = (Metric(name='scheduler_pulls_scheduled', description='Times pulls were scheduled.', kind='counter', unit='', labels=(), counts='pull_scheduled', sample=None, buckets=(), tags=('scheduler', 'pull')), Metric(name='scheduler_pulls_refused', description='Times scheduled pulls were refused.', kind='counter', unit='', labels=(), counts='pull_refused', sample=None, buckets=(), tags=('scheduler', 'pull')))
NEEDS = (Need(key='grant_store', source='stores.grants', block=None, late=False, getter=False, path=None, field=None), Need(key='budget', source='budget', block=None, late=False, getter=False, path=None, field=None), Need(key='capacity', source='capacity', block=None, late=False, getter=False, path=None, field=None), Need(key='propose', source='propose', block=None, late=False, getter=False, path=None, field=None), Need(key='pull', source='pull', block=None, late=False, getter=False, path=None, field=None), Need(key='strain', source='strain', block=None, late=False, getter=False, path=None, field=None))
STORES = (Store(key='_last_pull', description='when a session last ran for (peer, shard)', kind='table', shape=None, bound=1024, evict='lru', keyed_by='pull', persist='memory', sensitive=False, tags=()), Store(key='_declined', description='when a declined proposal may be tried again for (peer, shard)', kind='table', shape=None, bound=1024, evict='lru', keyed_by='pull', persist='memory', sensitive=False, tags=()), Store(key='_throttled', description='(epoch, why) last recorded for (peer, shard)', kind='table', shape=None, bound=1024, evict='lru', keyed_by='pull', persist='memory', sensitive=False, tags=()), Store(key='_shed', description='whether the pull is held under the ceiling for (peer, shard)', kind='table', shape=None, bound=1024, evict='lru', keyed_by='pull', persist='memory', sensitive=False, tags=()))
COMMANDS = (Command(key='show', name='Scheduler', description='every scheduled pull, when it last ran, and what is holding it back now', tags=('scheduler', 'status'), parameters=(), permission='scheduler/show', process=(), run='state', reply='{pulls, throttled, held, declined_until}', reads=(), writes=()),)
__init__(grant_store, budget, capacity, propose, pull, strain=None, clock=None, node=None)[source]
Parameters:
  • grant_store (grants_mod.GrantStore)

  • budget (Budget)

  • capacity (Optional[Capacity])

  • propose (Callable[..., tuple[str, Any]])

  • pull (Callable[..., SyncSessionResult])

  • strain (Optional[StrainLedger])

  • clock (Optional[Callable[[], int]])

  • node (Optional[Any])

Return type:

None

grants: grants_mod.GrantStore
budget: Budget
capacity: Capacity | None
propose: Callable[..., tuple[str, Any]]
pull: Callable[..., SyncSessionResult]
strain: StrainLedger | None
tick(now)[source]

Periodic work. Runs due EVERY jobs when not overridden.

Parameters:

now (int)

Return type:

list[tuple[Any, …]]

throttled()[source]

The pulls this node is holding back this window, and until when.

A throttled peer goes quiet in both directions: no sessions, so no usage, so no receipts due either way. The relay line then says “no event yet” and an operator reads a dead node. Four nodes on three machines produced exactly that reading for twenty minutes. Nothing was wrong; the grant’s ceiling had been reached. Say so, and say when it turns.

Return type:

list[ThrottledRow]

state()[source]

What this compartment is, as plain data. Never raises. A compartment with a shape worth naming returns its TypedDict.

Return type:

SchedulerStateReport

trouble()[source]

What an operator should be told without having to ask for it.

Everything here is WRONG: a watcher reds on it and a deploy fails on it. Something worth reading that is not a fault goes in notes(). May return a list of strings or typed Trouble instances.

Return type:

list[str]

abstract = False
commands = {'show': {'args': {}, 'does': 'every scheduled pull, when it last ran, and what is holding it back now', 'reply': '{pulls, throttled, held, declined_until}', 'subsystem': 'scheduler'}}

3.1.3.2. Updates & Verification Compartment

The compartment. Optional; “updates”: true in config.json turns it on.

class fleetmesh_node.planes.operations.updates.core.Updates[source]

Bases: Compartment

Check-and-report: the newest release this node could run, said once.

trouble() speaks when a newer release is within what the operator’s policy would apply - which in this tier is never, so it speaks about any newer compatible patch or minor and stays quiet about a major: an operator who chose pinned is told what they could move to without a protocol or API change, not shouted at about a jump they would have to opt into. updates.ack <version> silences one version; a newer one speaks again.

name: str | None = 'updates'
optional: bool = True
enabled_by: str | None = 'updates'
reads: tuple[str, ...] = ('updates', 'relays')
live: tuple[str, ...] = ('updates',)
config_cls

alias of UpdatesConfig

SCHEMA: tuple[Any, ...] = (Table(name='kv', columns=(Column(name='k', type='TEXT', nullable=False, primary_key=True, default=None, unique=False), Column(name='v', type='TEXT', nullable=False, primary_key=False, default=None, unique=False)), primary_key=(), indices=(), without_rowid=False), Table(name='seen', columns=(Column(name='version', type='TEXT', nullable=False, primary_key=True, default=None, unique=False), Column(name='event_id', type='TEXT', nullable=False, primary_key=False, default=None, unique=False), Column(name='sha256', type='TEXT', nullable=False, primary_key=False, default=None, unique=False), Column(name='first_seen', type='INTEGER', nullable=False, primary_key=False, default=None, unique=False), Column(name='acked', type='INTEGER', nullable=False, primary_key=False, default=0, unique=False)), primary_key=(), indices=(), without_rowid=False), Table(name='revoked', columns=(Column(name='event_id', type='TEXT', nullable=False, primary_key=True, default=None, unique=False), Column(name='reason', type='TEXT', nullable=False, primary_key=False, default=None, unique=False), Column(name='seen', type='INTEGER', nullable=False, primary_key=False, default=None, unique=False)), primary_key=(), indices=(), without_rowid=False))
STORE_CLS

alias of UpdateStore

NEEDS: tuple[Any, ...] = (Need(key='db_path', source=None, block=None, late=False, getter=False, path='updates.sqlite', field=None), Need(key='cfg', source=None, block=<class 'fleetmesh_node.config.compartments.UpdatesConfig'>, late=False, getter=True, path=None, field=None), Need(key='relays', source='relays', block=None, late=False, getter=False, path=None, field=None))
STORES: tuple[Store, ...] = (Store(key='releases', description='recent release announcements seen', kind='ring', shape=<class 'dict'>, bound=256, evict='oldest', keyed_by='', persist='memory', sensitive=False, tags=()),)
METRICS: tuple[Any, ...] = (Metric(name='release_available', description='Times a newer compatible release was announced by this node.', kind='counter', unit='', labels=(), counts='release.available', sample=None, buckets=(), tags=('updates', 'release')), Metric(name='release_revoked', description='Times a release this node could run was found revoked.', kind='counter', unit='', labels=(), counts='release.revoked', sample=None, buckets=(), tags=('updates', 'release')), Metric(name='updates_check', description='How long a release check took, end to end.', kind='histogram', unit='seconds', labels=(), counts='', sample=None, buckets=(0.005, 0.025, 0.1, 0.5, 1.0, 5.0, 30.0), tags=('updates', 'latency')))
RELEASE_AVAILABLE: str = 'release.available'
RELEASE_REVOKED: str = 'release.revoked'
CHANNELS: tuple[Any, ...] = (Channel(name='release.available', description='A release newer than the one this node runs is published and compatible with the operator policy. Said once per version; `updates.ack <version>` silences one.', level='note', wire=<function Updates.<lambda>>, say='release {version} available ({kind}) - running {running}', tags=('updates', 'release')), Channel(name='release.revoked', description='A release this node could run was revoked by the signing key. Trouble rather than a note: a revoked release is one nobody should move to.', level='trouble', wire=<function Updates.<lambda>>, say='release {version} revoked: {reason}', tags=('updates', 'release', 'trouble')))
COMMANDS: tuple[Any, ...] = (Command(key='check', name='Check Updates', description='ask the channel now; the state after', tags=('updates', 'check'), parameters=(), permission='updates/check', process=(), run='check', reply='dict', reads=(), writes=()), Command(key='show', name='Updates Status', description='the newest compatible release as parsed, and every one seen', tags=('updates', 'status'), parameters=(), permission='updates/show', process=(), run='show_cmd', reply='dict', reads=(), writes=()), Command(key='ack', name='Acknowledge Update', description='stop mentioning this version in trouble', tags=('updates', 'ack'), parameters=(Parameter(key='version', name='version', description='semver version to acknowledge', type=<class 'str'>, default='', must='', tags=(), required=True, choices=(), verify_regex='', live=False, item=None, field='', secret=False, requires='', sources=(), minimum=None, maximum=None, env='', validate=None, coerce=None, short='', cli_name='', positional=False, nargs='', cli=True),), permission='updates/ack', process=(), run='ack_cmd', reply='dict', reads=(), writes=()))
EVERY: tuple[Any, ...] = (Every(seconds=<function Updates.<lambda>>, run='periodic_check', name='check_updates', report=None),)
__init__(db_path=':memory:', cfg=None, relays=None, clock=None, fetch=<function _relay_fetch>, running=None, node=None, db=None)[source]
Parameters:
Return type:

None

settings()[source]

This compartment’s own config block, typed, as the file says now.

config_cls names the Block; its key is read through config() so the read is counted against reads like any other. Parsed on each call, so a compartment that lists its key in live sees an edit without a restart; the others call it once at construction.

Return type:

UpdatesConfig

channel()[source]
Return type:

list[str]

version()[source]

What this node runs, by the operator’s word if they gave one.

Return type:

str

policy_for(compartment='*')[source]
Parameters:

compartment (str)

Return type:

str

check(now=None)[source]
Parameters:

now (int | None)

Return type:

UpdateStateReport

periodic_check(now)[source]
Parameters:

now (int)

Return type:

list[tuple[Any, …]]

state()[source]

What this compartment is, as plain data. Never raises. A compartment with a shape worth naming returns its TypedDict.

Return type:

UpdateStateReport

trouble()[source]

What an operator should be told without having to ask for it.

Everything here is WRONG: a watcher reds on it and a deploy fails on it. Something worth reading that is not a fault goes in notes(). May return a list of strings or typed Trouble instances.

Return type:

list[str]

show_cmd()[source]
Return type:

Mapping[str, Any]

ack_cmd(args)[source]
Parameters:

args (Mapping[str, Any])

Return type:

Mapping[str, Any]

stop()[source]

Anything that must be said before the node goes. Must not raise.

Return type:

Any

abstract: ClassVar[bool] = False
commands: Mapping[str, CommandSpec] = {'ack': {'args': {'version': 'required'}, 'does': 'stop mentioning this version in trouble', 'reply': 'dict', 'subsystem': 'updates'}, 'check': {'args': {}, 'does': 'ask the channel now; the state after', 'reply': 'dict', 'subsystem': 'updates'}, 'show': {'args': {}, 'does': 'the newest compatible release as parsed, and every one seen', 'reply': 'dict', 'subsystem': 'updates'}}