3.1.3. Operations Plane (fleetmesh_node.planes.operations)
The Operations Plane manages autonomous node tasks, timer ticks, and signed software update lifecycles.
3.1.3.1. Scheduler Compartment
The pulls this node is configured to make, and every reason one did not happen.
This was seventy lines in the middle of the node’s tick, and it was the worst place in the node to be a stranger. Five things can stop a scheduled pull: the peer has no peering yet and the proposal raised; the proposal came back declined; this node is over its own ceiling and has stopped pulling as the last shed rung; the grant this node holds covers no session this window; or the session itself failed. Every one of them was appended to the tick’s return value as a tuple, printed once, and then gone. meshnode refusals could not show any of them, health counted none of them, and an operator watching a peer go quiet had to read the journal back far enough to find the line - if the journal still had it.
A refusal that is not recorded is indistinguishable from silence. This compartment is the five of them given a channel.
- class fleetmesh_node.planes.operations.scheduler.compartment.SchedulerConfig[source]
Bases:
ParamBlockSchedulerConfig(DERIVED: ‘bool’ = False, default_pull_every: ‘int’ = 900, decline_retry: ‘int’ = 3600)
- name = 'scheduler'
- PARAMS: tuple[Parameter[Any], ...] = (Parameter(key='default_pull_every', name='Default Pull Cadence', description='Seconds between pulls when peer grant has no rate', type=<class 'int'>, default=900, must='', tags=('scheduler', 'pull'), required=False, choices=(), verify_regex='', live=False, item=None, field='', secret=False, requires='', sources=(), minimum=None, maximum=None, env='', validate=None, coerce=None, short='', cli_name='', positional=False, nargs='', cli=True), Parameter(key='decline_retry', name='Decline Retry Horizon', description='Seconds before re-proposing to declined peer', type=<class 'int'>, default=3600, must='', tags=('scheduler', 'pull'), required=False, choices=(), verify_regex='', live=False, item=None, field='', secret=False, requires='', sources=(), minimum=None, maximum=None, env='', validate=None, coerce=None, short='', cli_name='', positional=False, nargs='', cli=True))
- class fleetmesh_node.planes.operations.scheduler.compartment.Scheduler[source]
Bases:
Compartment- config_cls
alias of
SchedulerConfig
- CHANNELS = (Channel(name='pull_scheduled', description='Pull scheduled for peer and shard', level='note', wire=None, say='scheduled pull for peer {peer} on shard {shard}', tags=('scheduler', 'pull')), Channel(name='pull_refused', description='Scheduled pull refused', level='note', wire=None, say='pull refused for peer {peer} on shard {shard}: {reason}', tags=('scheduler', 'pull')))
- METRICS = (Metric(name='scheduler_pulls_scheduled', description='Times pulls were scheduled.', kind='counter', unit='', labels=(), counts='pull_scheduled', sample=None, buckets=(), tags=('scheduler', 'pull')), Metric(name='scheduler_pulls_refused', description='Times scheduled pulls were refused.', kind='counter', unit='', labels=(), counts='pull_refused', sample=None, buckets=(), tags=('scheduler', 'pull')))
- NEEDS = (Need(key='grant_store', source='stores.grants', block=None, late=False, getter=False, path=None, field=None), Need(key='budget', source='budget', block=None, late=False, getter=False, path=None, field=None), Need(key='capacity', source='capacity', block=None, late=False, getter=False, path=None, field=None), Need(key='propose', source='propose', block=None, late=False, getter=False, path=None, field=None), Need(key='pull', source='pull', block=None, late=False, getter=False, path=None, field=None), Need(key='strain', source='strain', block=None, late=False, getter=False, path=None, field=None))
- STORES = (Store(key='_last_pull', description='when a session last ran for (peer, shard)', kind='table', shape=None, bound=1024, evict='lru', keyed_by='pull', persist='memory', sensitive=False, tags=()), Store(key='_declined', description='when a declined proposal may be tried again for (peer, shard)', kind='table', shape=None, bound=1024, evict='lru', keyed_by='pull', persist='memory', sensitive=False, tags=()), Store(key='_throttled', description='(epoch, why) last recorded for (peer, shard)', kind='table', shape=None, bound=1024, evict='lru', keyed_by='pull', persist='memory', sensitive=False, tags=()), Store(key='_shed', description='whether the pull is held under the ceiling for (peer, shard)', kind='table', shape=None, bound=1024, evict='lru', keyed_by='pull', persist='memory', sensitive=False, tags=()))
- COMMANDS = (Command(key='show', name='Scheduler', description='every scheduled pull, when it last ran, and what is holding it back now', tags=('scheduler', 'status'), parameters=(), permission='scheduler/show', process=(), run='state', reply='{pulls, throttled, held, declined_until}', reads=(), writes=()),)
- grants: grants_mod.GrantStore
- pull: Callable[..., SyncSessionResult]
- strain: StrainLedger | None
- throttled()[source]
The pulls this node is holding back this window, and until when.
A throttled peer goes quiet in both directions: no sessions, so no usage, so no receipts due either way. The relay line then says “no event yet” and an operator reads a dead node. Four nodes on three machines produced exactly that reading for twenty minutes. Nothing was wrong; the grant’s ceiling had been reached. Say so, and say when it turns.
- Return type:
- state()[source]
What this compartment is, as plain data. Never raises. A compartment with a shape worth naming returns its TypedDict.
- Return type:
SchedulerStateReport
- trouble()[source]
What an operator should be told without having to ask for it.
Everything here is WRONG: a watcher reds on it and a deploy fails on it. Something worth reading that is not a fault goes in notes(). May return a list of strings or typed Trouble instances.
- abstract = False
- commands = {'show': {'args': {}, 'does': 'every scheduled pull, when it last ran, and what is holding it back now', 'reply': '{pulls, throttled, held, declined_until}', 'subsystem': 'scheduler'}}
3.1.3.2. Updates & Verification Compartment
The compartment. Optional; “updates”: true in config.json turns it on.
- class fleetmesh_node.planes.operations.updates.core.Updates[source]
Bases:
CompartmentCheck-and-report: the newest release this node could run, said once.
trouble() speaks when a newer release is within what the operator’s policy would apply - which in this tier is never, so it speaks about any newer compatible patch or minor and stays quiet about a major: an operator who chose pinned is told what they could move to without a protocol or API change, not shouted at about a jump they would have to opt into. updates.ack <version> silences one version; a newer one speaks again.
- config_cls
alias of
UpdatesConfig
- SCHEMA: tuple[Any, ...] = (Table(name='kv', columns=(Column(name='k', type='TEXT', nullable=False, primary_key=True, default=None, unique=False), Column(name='v', type='TEXT', nullable=False, primary_key=False, default=None, unique=False)), primary_key=(), indices=(), without_rowid=False), Table(name='seen', columns=(Column(name='version', type='TEXT', nullable=False, primary_key=True, default=None, unique=False), Column(name='event_id', type='TEXT', nullable=False, primary_key=False, default=None, unique=False), Column(name='sha256', type='TEXT', nullable=False, primary_key=False, default=None, unique=False), Column(name='first_seen', type='INTEGER', nullable=False, primary_key=False, default=None, unique=False), Column(name='acked', type='INTEGER', nullable=False, primary_key=False, default=0, unique=False)), primary_key=(), indices=(), without_rowid=False), Table(name='revoked', columns=(Column(name='event_id', type='TEXT', nullable=False, primary_key=True, default=None, unique=False), Column(name='reason', type='TEXT', nullable=False, primary_key=False, default=None, unique=False), Column(name='seen', type='INTEGER', nullable=False, primary_key=False, default=None, unique=False)), primary_key=(), indices=(), without_rowid=False))
- STORE_CLS
alias of
UpdateStore
- NEEDS: tuple[Any, ...] = (Need(key='db_path', source=None, block=None, late=False, getter=False, path='updates.sqlite', field=None), Need(key='cfg', source=None, block=<class 'fleetmesh_node.config.compartments.UpdatesConfig'>, late=False, getter=True, path=None, field=None), Need(key='relays', source='relays', block=None, late=False, getter=False, path=None, field=None))
- STORES: tuple[Store, ...] = (Store(key='releases', description='recent release announcements seen', kind='ring', shape=<class 'dict'>, bound=256, evict='oldest', keyed_by='', persist='memory', sensitive=False, tags=()),)
- METRICS: tuple[Any, ...] = (Metric(name='release_available', description='Times a newer compatible release was announced by this node.', kind='counter', unit='', labels=(), counts='release.available', sample=None, buckets=(), tags=('updates', 'release')), Metric(name='release_revoked', description='Times a release this node could run was found revoked.', kind='counter', unit='', labels=(), counts='release.revoked', sample=None, buckets=(), tags=('updates', 'release')), Metric(name='updates_check', description='How long a release check took, end to end.', kind='histogram', unit='seconds', labels=(), counts='', sample=None, buckets=(0.005, 0.025, 0.1, 0.5, 1.0, 5.0, 30.0), tags=('updates', 'latency')))
- CHANNELS: tuple[Any, ...] = (Channel(name='release.available', description='A release newer than the one this node runs is published and compatible with the operator policy. Said once per version; `updates.ack <version>` silences one.', level='note', wire=<function Updates.<lambda>>, say='release {version} available ({kind}) - running {running}', tags=('updates', 'release')), Channel(name='release.revoked', description='A release this node could run was revoked by the signing key. Trouble rather than a note: a revoked release is one nobody should move to.', level='trouble', wire=<function Updates.<lambda>>, say='release {version} revoked: {reason}', tags=('updates', 'release', 'trouble')))
- COMMANDS: tuple[Any, ...] = (Command(key='check', name='Check Updates', description='ask the channel now; the state after', tags=('updates', 'check'), parameters=(), permission='updates/check', process=(), run='check', reply='dict', reads=(), writes=()), Command(key='show', name='Updates Status', description='the newest compatible release as parsed, and every one seen', tags=('updates', 'status'), parameters=(), permission='updates/show', process=(), run='show_cmd', reply='dict', reads=(), writes=()), Command(key='ack', name='Acknowledge Update', description='stop mentioning this version in trouble', tags=('updates', 'ack'), parameters=(Parameter(key='version', name='version', description='semver version to acknowledge', type=<class 'str'>, default='', must='', tags=(), required=True, choices=(), verify_regex='', live=False, item=None, field='', secret=False, requires='', sources=(), minimum=None, maximum=None, env='', validate=None, coerce=None, short='', cli_name='', positional=False, nargs='', cli=True),), permission='updates/ack', process=(), run='ack_cmd', reply='dict', reads=(), writes=()))
- EVERY: tuple[Any, ...] = (Every(seconds=<function Updates.<lambda>>, run='periodic_check', name='check_updates', report=None),)
- __init__(db_path=':memory:', cfg=None, relays=None, clock=None, fetch=<function _relay_fetch>, running=None, node=None, db=None)[source]
- settings()[source]
This compartment’s own config block, typed, as the file says now.
config_cls names the Block; its key is read through config() so the read is counted against reads like any other. Parsed on each call, so a compartment that lists its key in live sees an edit without a restart; the others call it once at construction.
- Return type:
- state()[source]
What this compartment is, as plain data. Never raises. A compartment with a shape worth naming returns its TypedDict.
- Return type:
UpdateStateReport
- trouble()[source]
What an operator should be told without having to ask for it.
Everything here is WRONG: a watcher reds on it and a deploy fails on it. Something worth reading that is not a fault goes in notes(). May return a list of strings or typed Trouble instances.
- commands: Mapping[str, CommandSpec] = {'ack': {'args': {'version': 'required'}, 'does': 'stop mentioning this version in trouble', 'reply': 'dict', 'subsystem': 'updates'}, 'check': {'args': {}, 'does': 'ask the channel now; the state after', 'reply': 'dict', 'subsystem': 'updates'}, 'show': {'args': {}, 'does': 'the newest compatible release as parsed, and every one seen', 'reply': 'dict', 'subsystem': 'updates'}}