Workflow and state machines¶
This page is for anyone who needs to know why a round, a stage or a research task is in the state it is in, and what can move it. It shows the three state machines, says who may trigger each transition, and explains the rules that keep repeated and late events from doing harm.
Status. The transitions below are the ones in the code and its tests, run against the simulated AC5 only. Some have no entry point in the running daemon yet; the text says which. See Status and known limitations.
Three machines, one owner¶
A round is one pass from a goal to a report. A stage is one execution segment of a round, governed by one run sheet. A research task is one piece of work for a model role. They are modelled separately because they move for different reasons: a lost link can pause a stage while a task is still analysing last week's data.
Only the workflow service changes them: WorkflowService in workflow/service.py for rounds and stages, TaskBoard in workflow/tasks.py for tasks. The state names are in workflow/states.py, and their Chinese labels are in the Glossary.
Round¶
stateDiagram-v2
[*] --> designing: round created
designing --> stage_review: first run sheet submitted
stage_review --> stage_running: stage started
stage_running --> stage_review: stage ended, more stages left
stage_running --> data_verification: last stage ended
stage_running --> stage_review: stage stopped, operator chooses redo or next
stage_running --> data_verification: stage stopped, operator chooses verify
stage_review --> data_verification: stage stopped before it ran, operator chooses verify
stage_running --> stage_review: stage sent back to review after a recovery number change
data_verification --> analysis: snapshot frozen
analysis --> delivered: report published
delivered --> [*]
aborted --> [*]
note right of aborted : Reached from every state that is not final, by abort_round
| Transition | Trigger | Who |
|---|---|---|
new → designing |
create_round |
operator or workflow identity |
designing → stage_review |
the first submit_run_sheet without blocking findings |
operator, workflow identity or model role |
stage_review → stage_running |
start_stage succeeds |
operator or workflow identity |
stage_running → stage_review or data_verification |
a stage reaches ended; the protocol's stage list decides which |
executor event |
after a stage is stopped |
choose_after_stop with redo, next, verify or abort |
operator |
stage_running → stage_review |
a stage leaves recovery_check after a recovery number change |
runtime |
data_verification → analysis |
mark_snapshot_frozen, called by freeze_dataset when the snapshot is complete and verifies |
researcher |
analysis → delivered |
mark_report_published, called by publish_research_report when the report is final |
researcher |
every state that is not final → aborted |
abort_round |
operator |
A stopped stage does not move the round by itself. It only marks the round as waiting for the operator's choice, and no run sheet can be submitted until the choice is made (round_effect in workflow/store.py). The reason: after a stop, only a person knows whether the data so far is still worth analysing.
Stage¶
stateDiagram-v2
[*] --> pending_review: run sheet submitted
pending_review --> running: started with a valid approval
running --> paused: operator pauses, or a pause event
paused --> running: operator resumes, device matches
running --> manual: operator takes over
paused --> manual: operator takes over
manual --> running: operator hands back
running --> recovery_check: old writes, pump-off or end actions unconfirmed
paused --> recovery_check: old writes or pump-off unconfirmed
manual --> recovery_check: old writes or pump-off unconfirmed
pending_review --> recovery_check: restart or recovery number change
recovery_check --> paused: checked, was running or paused
recovery_check --> manual: checked, was manual
recovery_check --> pending_review: checked, was pending review or recovery number changed
running --> ended: end condition met, end actions confirmed
paused --> stopped: longest wait expired, run sheet says stop
ended --> [*]
stopped --> [*]
note right of stopped : Also reached from every state that is not final, by stop_stage or abort_round
A restart or a recovery number change also sends running, paused and manual stages to recovery_check; the diagram shows that edge only once.
| Transition | Trigger | Who |
|---|---|---|
new → pending_review |
submit_run_sheet creates the stage |
operator, workflow identity or model role |
stays pending_review |
approve records an approval; a newer run sheet version voids the old one |
approve: operator with recent re-authentication |
pending_review → running |
start_stage: valid approval, no safety conflict, no pump interlock, no other live stage on the device |
operator or workflow identity |
running → paused |
pause_stage |
operator or observer identity |
running → paused |
DISCONNECT_AFFECTS, READINGS_STALE, EXTERNAL_CHANGE, CLOCK_ANOMALY, RESULT_UNKNOWN, RULE_PAUSE, ALERT_PAUSE |
executor event; ALERT_PAUSE comes from the observer |
paused → running |
resume_stage: valid approval, no interlock, no unsettled action, device matches the run sheet |
operator |
running, paused → manual |
takeover |
operator |
manual → running |
handback; the differences to the run sheet are listed and can be recorded as an intervention |
operator |
running, paused, manual → recovery_check |
TRANSPORT_UNCONFIRMED, PUMP_OFF_UNCONFIRMED; from running also END_ACTIONS not confirmed |
executor event |
every state that is not final → recovery_check |
process start, recovery number change | runtime |
recovery_check → paused, manual or pending_review |
the exit conditions hold | runtime, every tick for every stage in recovery_check; operator with record_onsite_check |
running → ended |
END_CONDITION_MET, then END_ACTIONS confirmed |
executor event |
paused → stopped, or stays paused |
MAX_WAIT_EXPIRED; the run sheet's on_expiry decides |
executor event |
every state that is not final → stopped |
stop_stage, abort_round; from pending_review also revoke_approval |
operator |
running, paused keep their state |
switch_run_sheet: a new version with new write rights |
operator with recent re-authentication |
Write rights move with the state. Entering running grants a new stage right. paused revokes it and keeps the care right. manual revokes both and grants a manual right. recovery_check revokes all three. ended revokes the stage right and keeps the care right until the next stage starts. Each change is a Plan: hand the old rights over first, grant the new ones, then save the state and its history row in one transaction (_apply, _commit). What a right is, is on Control and safety model.
Ending takes two events on purpose. END_CONDITION_MET only marks the stage as end_pending; the executor then carries out the run sheet's end actions under the stage right it still holds. Only END_ACTIONS with confirmed true ends the stage. If the end actions cannot be confirmed, the device is in an unknown state, so the stage goes to recovery_check instead.
Research task¶
stateDiagram-v2
[*] --> in_progress: task created
in_progress --> remote_queued: remote job submitted
remote_queued --> computing: job running
computing --> fetching: job finished
fetching --> in_progress: artifacts fetched
remote_queued --> reconcile: submission uncertain
computing --> reconcile: receipt uncertain
reconcile --> in_progress: rechecked and resubmitted
reconcile --> ended: abandoned, with a reason
in_progress --> needs_review: confirmatory report with open questions
needs_review --> done: reviewed
in_progress --> done: run finished
in_progress --> needs_attention: model failed, budget or deadline reached
needs_attention --> in_progress: scope widened
needs_attention --> ended: ended, with a reason
done --> [*]
ended --> [*]
note right of needs_attention : Also reached from remote_queued, computing, fetching, reconcile and needs_review
The diagram is the table _ALLOWED in workflow/tasks.py. Every other transition is Denied("state"), and ending a task from reconcile or needs_attention needs a reason.
Only part of it is triggered by code today:
in_progress→done,needs_revieworneeds_attention: the dispatcher, when a model run ends (agents/dispatch.py).in_progress,remote_queued,computing,fetching→needs_attention:TaskBoard.expire, once per deadline, called from the tick.- The remote-job cycle and the ways out of
reconcile,needs_attentionandneeds_revieware allowed by the board, but nothing insrc/calls them. The compute service keeps its own attempt phase, and there is no operator route for resolving a task yet.
Who may trigger a transition¶
There are three kinds of trigger, and they differ in how much they are trusted.
- An operator command. The caller's
Actorcomes from a workbench session.require_operatorinworkflow/permissions.pyrefuses every other kind; a model role getsmodel_role.approveandswitch_run_sheetalso need a re-authentication within 300 seconds, and the workbench routes ask the same for resume and takeover. Reason: these are the decisions that put the device under a plan. - An event. The executor and the observer report a
StageEventthroughStagePort.report. An event carries no identity. It is a statement of fact ("the read-back differs"), and the workflow service decides what it means for the current state. An event can pause, end or stop a stage, or hold it for checking. It can never start or resume one. - A model role. It can file a run-sheet draft, which creates a stage in
pending_review. The researcher's tools can move a round fromdata_verificationtoanalysisand fromanalysistodelivered, after checking that the round belongs to the task's project. A model role can do nothing to a stage that has started.
Two more identities exist for services. A workflow identity may create projects and rounds and may start a stage that has a valid approval. An observer identity may call pause_stage. No code in src/ uses either yet: the observer pauses through the ALERT_PAUSE event.
Entry points in the daemon: /wb has routes for create_project, save_protocol and create_round; for approve, start, pause, resume, stop, takeover, handback, on-site check and filing a draft; and for choose_after_stop and abort_round. revoke_approval and switch_run_sheet are service methods with tests but without a route or command. See HTTP, workbench and MCP interfaces.
The same event advances a state once¶
Events are delivered at least once. The executor may report the same fact on the next tick, a tool call may be retried, a process may restart in between. So every event has an event_id, and the id is stored in the same transaction as the change it caused.
- Stage events.
reportignores an id it has seen. Otherwise the id is inserted intowf_eventstogether with the new state. If two reports race, the second insert fails and its whole transaction rolls back. An event that does not apply to the current state is stored asdropped. - Round events.
mark_roundadvances only if the round is in the expected state, once per id. The ids aresnapshot_frozen:<round>:<snapshot>andreport_published:<round>:<artifact>. - Task events.
TaskBoard.transitionreturns the task unchanged when the id was already used. A task is created once pertrigger_event_id.
An example from tests/workflow/test_workflow_events.py: a RULE_PAUSE event pauses the stage. The operator resumes. The same event arrives again. The stage stays running, and the history holds one rule_pause row.
Operator commands are safe to repeat in a simpler way: starting a running stage, pausing a paused one or taking over a manual one returns the current state. Behind both sits a revision number on the stage row. A change computed from an older revision is refused with Denied("state"), so a slow caller cannot overwrite a newer state.
Late events are dropped by right number, not by time¶
A late event is one that was true when it was observed and is no longer relevant. The executor sees "the light differs" and reports EXTERNAL_CHANGE. Before the report is handled, the operator pauses, fixes the light and resumes. If the event were applied now, it would pause a stage that was just checked.
The rule: when the executor reports about a running stage, it adds the stage write right it observed under as detail.right_number. The workflow service drops the event if that number is not the stage's current one (_plan_event).
Why the right number and not a timestamp:
- It marks exactly the right boundary. A new stage right is granted at every start, resume, handback and run-sheet switch. Those are the moments after which an older observation no longer counts.
- The wall clock can jump. The executor itself treats wall time as untrusted when it steps (
ClockTrustinexecutor/outage.py). An event stamped just before a clock correction could look newer than the resume that followed it. - It is exact. A pause and a resume can fall within the same second. Two integers are either equal or not.
Events without a number, such as the observer's ALERT_PAUSE, are judged by the state alone: a pause event only ever applies to a running stage.
Recovery check: what it blocks and what it allows¶
recovery_check means: the device state, or the result of a write, is not known. The stage remembers the state it came from (prior_state) and loses its stage, manual and care rights.
| Blocked | Still allowed |
|---|---|
| Timed actions, feedback rules and end actions | Protection rules, while the link is up |
| Manual writes and irrigation | The pump-off interlock path |
| Pause, resume, takeover, handback, run-sheet switch | Care rules, once the old write path is confirmed stopped and the interlock is off (not if the stage was manual) |
| Starting another stage on the same device | Sampling, alerts and the daily report |
stop_stage, abort_round, record_onsite_check |
The stage leaves when one of two things is true (_may_exit):
- the old write path is confirmed stopped, the interlock is off, no action is unsettled after a read-back, and, if the stage was
runningorpaused, the device matches the run sheet; or - an operator recorded an on-site check.
The runtime tries this on every tick for every stage in recovery_check (_follow_up in runtime.py). That includes a stage that was still waiting for review when the process restarted, although it has never started and so is not the active stage.
Where the stage goes depends on where it came from: running or paused becomes paused; manual stays manual with a new manual right; pending_review stays pending_review. It never returns straight to running, because a person has to confirm the resume. If the recovery number changed in the meantime, the safety configuration must be approved again first, and the stage then goes to pending_review for a new approval.
Where the code differs from Appendix B¶
The state tables in Appendix B of the architecture document are the design. The code differs in these points:
- The round moves to
stage_runningwhen the stage starts, not when the run sheet is approved. Approving and starting are two calls. - A stage ends in two events, and unconfirmed end actions lead to
recovery_check. switch_run_sheetalso works on apausedstage. There is no option to keep the old version running during the review.revoke_approvalonly applies to a stage inpending_review. A started stage is stopped withstop_stage.- Leaving
recovery_checkalso requires the pump interlock to be off. - Longest-wait expiry changes the state but raises no alert of its own. When it stops the stage, the
stage_waitingalert of the pause is resolved, because the stage is no longer paused. - For research tasks, the remote-job cycle and the user's resolutions are allowed transitions without a caller.
See also¶
- Architecture overview
- Control and safety model
- The agents
- Run a research round
- Troubleshooting runbook
- Denial codes and stage events
- Tests:
tests/workflow/,tests/tasks/
Pages were last checked against the code on 2026-11-11. DiAGRI mode has been tested on a simulated AC5 only: see Status and known limitations. To change a page, read how this wiki is edited first.