orcaLog in

Alerts

What orca pushed to the ntfy topic (not shown: anyone who knows it can read it), newest first; the same alert is sent at most once every 30 minutes. Watchdog has the rules that pause pipelines; orca notify test sends a test message.

WhenWhen the alert was raised, or resolved; the full time on hover.KindWhat raised the alert: a watchdog trip, a run that failed with no retry left, a stalled fill, the control disk filling, a worker gone quiet, memory pressure holding runs, a failed backup or another script's orca notify; 'resolved' is the all-clear sent once it ended. Its priority is on hover.AlertThe title and the message, as they were sent to the phone.LinksWhat the alert is about: its run, the pipelines it paused or names, the watchdog's rule, the worker, or Control.DeliveredWhether ntfy took it: sent; waiting for its first try; failed, retrying (the error, tried again after a wait that doubles up to 15 minutes); or not sent, when a newer one or its resolution replaced it first.StateFor alerts that end (a watchdog trip, a stalled fill, the disk, memory pressure): open while it still holds, resolved once it ended, or raised again when the same alert was sent anew before that. Empty for one that doesn't end, such as a run failed.
run failedorca: mojo_versions run 2685 failedmojo_versions rofaidaezzat/clustring- every repository in this run failed (1)run 2685sent
run failedorca: gharchive run 4013 failedgharchive 2025-06-12 2025-06-12 hour 17: GET https://data.gharchive.org/2025-06-12-17.json.gz: HTTP 404run 4013sent
watchdog resolvedorca watchdog: failing resolvedAcknowledged, or its pipelines are unpaused.rule failingsent
run failedorca: mojo_activity run 1120 failedmojo_activity 2023-05-24 interrupted: its process stopped, and its retries are spentrun 1120sent
run failedorca: mojo_activity run 1117 failedmojo_activity 2023-05-23 interrupted: its process stopped, and its retries are spentrun 1117sent