Notifications, daily reports and backup¶
This page is for an operator who sets up what the daemon sends out and what it saves: the daily report time, the notification channels and their credentials, the heartbeat to an outside monitor, and the database backups, including how to restore one.
Not verified. Delivery through Telegram and email, and the heartbeat to an outside service, have been tested with fake transports only. No message has been sent to a real service. See Status and known limitations.
What exists today¶
| Feature | Configured with | State |
|---|---|---|
| Daily report | report_time, report_timezone in the configuration file |
Made by serve while a stage governs the device. |
| Telegram channel | Environment variables | Built by serve when its variables are set. |
| Email channel | Environment variables | Built by serve when its variables are set. |
| Heartbeat to an outside monitor | Environment variables | Built by serve when both URLs are set; one alone stops the start. |
| Database backups | backup_dir, backup_interval_seconds |
Run by serve when backup_dir is set. |
Every change on this page takes effect when serve starts. Configuration fields are described in the Configuration reference.
Set the daily report time¶
- Stop the daemon.
- Set the two fields in the configuration file:
report_time is HH:MM in 24-hour form. report_timezone is an IANA zone name. The defaults are 08:00 and UTC.
3. Start serve.
What happens then (runtime.py, _daily_report, and observer/report.py):
- Once the local time has passed
report_time, the report for yesterday's local date is made. It is made only while a stage governs the device: from the first stage start of a round until that round is delivered or aborted. - After downtime, every missed day is reported, oldest first, up to the seven most recent days.
- The report has the time-weighted mean of each reading, the data coverage with its gaps, the protective actions and the manual interventions of that day, and a "Frozen readings" section that lists each period of that day in which a sensor kept reporting the same value (for example
temp stuck at 24 from 08:00 to 10:05, or(still frozen)if it has not ended;(no longer checked)is added when the period ended because the sensor was taken off the check, not because the reading moved;(close reason unrecorded)marks a period closed before the reason was kept, which must not be read as a recovery), so you know which stretches of data are suspect. At most five periods are listed, periods that are still frozen first and then by start time; a last lineand N more (temp, co2)names the sensors of the periods left out. Times are in the report timezone; a time outside that day also shows its date, and a time that cannot be shown reads "unknown time". No model writes any part of it. The runtime passes no ranges, crop observations or to-do items, so those parts are empty. - It is saved once per project and date in the table
observer_reportsofobserver.sqlite3, and queued on every notification channel.
An unknown zone name is not caught when the file is loaded. It shows up later as an alert of kind tick_step with the key report.
There is no route to read a saved report. You see it only through a notification channel.
Set up a notification channel¶
The channels are in observer/channels.py. Their settings come only from the environment of serve, never from the configuration file. runtime.py (default_channels) builds a channel only when both of its required variables are set at start.
| Channel | Required | Optional |
|---|---|---|
| Telegram | GROWBRIDGE_TELEGRAM_TOKEN (the bot token), GROWBRIDGE_TELEGRAM_CHAT_IDS (chat ids, separated by commas) |
none |
GROWBRIDGE_SMTP_HOST, GROWBRIDGE_EMAIL_TO (addresses, separated by commas) |
GROWBRIDGE_SMTP_PORT (default 587), GROWBRIDGE_SMTP_USER, GROWBRIDGE_SMTP_PASSWORD, GROWBRIDGE_SMTP_FROM |
- Stop the daemon.
- Set the variables in the environment that
servestarts in:
export GROWBRIDGE_TELEGRAM_TOKEN='YOUR_BOT_TOKEN'
export GROWBRIDGE_TELEGRAM_CHAT_IDS='YOUR_CHAT_ID'
The installers in scripts/ write no environment into the background service, so for a service you have to add the variables to the service definition yourself. Not verified.
3. Start serve.
What the channels do:
- Telegram posts to the registered chat ids and has no code that reads incoming messages. Email uses STARTTLS and logs in only when
GROWBRIDGE_SMTP_USERis set. The sender address isGROWBRIDGE_SMTP_FROM, or the user, orgrowbridge@localhost. - Two kinds of message are sent: each alert, once, as
[severity] message, and each daily report as its full text. Resolving an alert sends nothing. - A channel reports
accepted,unknownorfailed. Anything not accepted is tried again after 30 seconds, then with a doubling delay up to 30 minutes, for at most 20 attempts in all (observer/outbox.py). These numbers are constants in the code, not configuration. Anunknownresult can therefore give you the same message twice.
There is no command that sends a test message. To check delivery, read GET /wb/alerts: the alert run_health names oldest_undelivered_alert when a notification on a configured channel has stayed undelivered for 900 seconds. Alerts themselves can always be read there, with or without a channel.
With no channel set¶
This is what serve does when neither channel is configured, checked on the simulated device:
- No alert and no daily report is queued for sending. Alerts are still recorded, and daily reports are still saved.
- The warning alert
notify_unconfiguredis raised at start. It is resolved the next timeservestarts with a channel. - The run health is not affected:
oldest_undelivered_alertcounts only notifications queued for a channel that has a sender. - When you configure a channel later and restart, every alert that is still open is queued and sent on it. Alerts that were resolved in the meantime, and daily reports made in the meantime, are not sent.
The same holds when you remove a channel: a notification already queued for it stays pending and no longer counts for the run health, and the alerts that are still open are queued on the channels that remain.
There is no command that clears the queue. A notification that used up its 20 attempts on a configured channel keeps oldest_undelivered_alert failing.
The heartbeat to an outside monitor¶
What is implemented, in observer/health.py and observer/channels.py:
- Two signals: a host heartbeat sent with every beat, and a run heartbeat sent only while the run health is good. When the run health is bad, a failure signal is sent instead and the run heartbeat stops.
- The runtime beats once every 60 seconds (
HEARTBEAT_SECONDSinruntime.py), and at once when the run health turns good or bad. An outside monitor needs a grace period longer than 60 seconds. HeartbeatChannelrequests a URL read from an environment variable for a heartbeat, and the same URL followed by/failfor a failure.- A signal that cannot be delivered raises the alert
external_monitorwith the keyhostorrun.
How to configure it¶
serve reads two environment variables (runtime_heartbeat.py, called from _serve_auto_research in cli.py). Both hold a URL from your monitoring service, for example a Healthchecks.io ping URL:
| Variable | Signal |
|---|---|
GROWBRIDGE_HEARTBEAT_HOST_URL |
The host heartbeat. |
GROWBRIDGE_HEARTBEAT_RUN_URL |
The run heartbeat. A failure goes to the same URL followed by /fail. |
- Both or neither. With both set,
servesends the heartbeats. With only one set,serverefuses to start and says which variable is missing: watching half of the failures would look like monitoring while it is not. A blank value counts as not set. - Neither set. No request is made. The runtime raises the standing warning alert
monitor_unconfiguredwith the keyheartbeats, and resolves it the next timeservestarts with both variables. - The URLs are secrets. They are read only from the environment, when a signal is sent. They appear in no log line, alert text, error message or
repr. Request URLs are never logged:serveruns at INFO, and the HTTP library would log every request URL at that level, sologging_setup.pyholds thehttpxandhttpcoreloggers at WARNING. This also protects the Telegram bot token, which is part of the Bot API request path. - Give the monitor a grace period of more than 60 seconds; the monitor's own phone alert is what tells you when a heartbeat stops.
Not verified: the heartbeat has only been tested against a fake HTTP client. No signal has been sent to a real monitoring service.
The run health itself is computed on every tick and is visible as the alert run_health. Its probes and limits:
| Probe | Fails when |
|---|---|
growbridge_response |
No packet from the device for 60 seconds. |
executor_evaluation |
No executor tick for 30 seconds. |
capture |
No sample for stale_seconds, or for the run sheet's sampling.max_gap_seconds while a stage governs the device. |
archive_write |
No write to the project archive for 600 seconds. |
oldest_undelivered_alert |
A notification on a configured channel has been undelivered for 900 seconds. |
offsite_backup |
The last backup is older than two backup intervals. Checked only when backup_dir is set. |
Set up backups¶
- Stop the daemon.
- Set the fields:
Use a directory on other media than the data directory. The code does not check that. The interval is at least 60 seconds; the default is 21600 (6 hours).
3. Start serve. The first backup starts on the first tick. After it, the directory holds twelve files: control, workflow, archive, observer, compute and agents, each as <name>.sqlite3 and <name>.sqlite3.manifest.json.
How it works (runtime_backup.py, archive/backup.py):
- How often. A backup starts when the oldest of the six last-success times is older than the interval. It runs in the background and does not hold up the executor.
- Consistency. Each database is copied through a separate read-only connection inside one read transaction, so the copy is one consistent state and writers are not blocked.
- Integrity. The copy is checked with
PRAGMA integrity_checkbefore it replaces the previous backup. A failed copy or check leaves the previous backup in place. The manifest holdssha256,sizeandcreated_at(UTC seconds) of the file. - One generation. Each backup replaces the one before. Keeping older copies is up to you.
- Failure. The alert
backupis raised with the error, and the next attempt waits 300 seconds. With nobackup_dir, the warningbackup_unconfiguredstays open.
Not in the backup: the bridge sample history growbridge.sqlite3, the files under archive/ in the data directory (data snapshot files), the files under agent_work/ (research report files), the host_id file and the configuration file. The databases keep the records and digests of those files, not the files.
To check a backup by hand, compare the file with its manifest, for example with shasum -a 256 /absolute/path/to/backup/control.sqlite3. This uses a tool outside the product.
Restore a backup¶
There is no restore command. The steps below are not covered by a test, apart from one that opens a backup file and finds the same records (tests/archive/test_archive_storage_backup.py).
- Stop the daemon.
- Check each backup file against its manifest.
- Move the old
*.sqlite3files of the six databases, and the-waland-shmfiles next to them, out of the data directory. Copy the six backup files in. - Raise the recovery number:
- Start
serveand continue with "Restore from backup or a new write host" in the Troubleshooting runbook.
What a restore does to the recovery number. Copying files back changes nothing: the restored control database carries the recovery number it had when the backup was made, and nothing detects that a restore happened. Step 4 is what makes the restore safe. The bump raises the number by one, revokes every write right, voids queued writes and every approval, moves each unfinished stage to recovery_check, and makes every earlier session and the earlier safety approval invalid. Why this is required is on Control and safety model.
See also¶
- Enable DiAGRI
- Troubleshooting runbook
- Configuration reference
- Data, archive and traceability
- Status and known limitations
Pages were last checked against the code on 2026-11-11. DiAGRI mode has been tested on a simulated AC5 only: see Status and known limitations. To change a page, read how this wiki is edited first.