Skip to content

feat: read the cause of a jubilant wait timeout from the status it prints - #19

Open
tonyandrewmeyer wants to merge 1 commit into
canonical:mainfrom
tonyandrewmeyer:ai-failure-notifier-wait-status
Open

tonyandrewmeyer wants to merge 1 commit into
canonical:mainfrom
tonyandrewmeyer:ai-failure-notifier-wait-status

Conversation

@tonyandrewmeyer

@tonyandrewmeyer tonyandrewmeyer commented Oct 20, 2026 •

Copy link
Copy Markdown
Collaborator

When jubilant.wait() times out in an integration test, the enricher now picks the apps and units that were in error or blocked out of the Status(...) that jubilant prints with the timeout, and the prompt says that an app in error there supports a likely root cause. Before, pytest's summary line only said "wait timed out after 900s", the rest of the message was dropped, and the 40-line tail ended up being the storage part of the status dump. So canonical/operator#2834 had no likely root cause, although the log showed tempo in error (hook failed: "start") in one job and "upgrade-charm" in the other. Run on those two job logs, the parser now gives tempo: error (hook failed: "start") and the same for tempo/0.

  • Only the copy of the status in pytest's summary is read, because with --tb native the same dump is also in the traceback, and reading both would list everything twice.
  • The new field isn't part of the signature hash, so existing :sig= markers still match.
  • The enricher also now logs each envelope entry's action, target, confidence and dedup reason. None of that is posted, so for #2834 there's no way to tell why the model decided what it did.

…ints

When `jubilant.wait()` times out, pytest's summary line only says "wait timed out after 900s". The reason is in the `Status(...)` that makes up the rest of the message, which the parser dropped, so the model had nothing to base a likely root cause on. The signature now lists the apps and units that were in error or blocked in that dump (for example `tempo/0: error (hook failed: "start")`), and the prompt says when that supports a cause.

The enricher also now logs the envelope's routing decision, confidence and dedup reason. Those aren't posted anywhere, so until now there was no way to tell afterwards why a run was routed where it was.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@tonyandrewmeyer

Copy link
Copy Markdown
Collaborator Author

I ran this on my fork with a canary that prints the real summary and Status(...) from operator run 39219345374, with the enricher pinned to this branch. When there was no existing issue to match, it opened tonyandrewmeyer/operator#67 with "The tempo app was in error state with a failed start hook, causing the wait to time out." under Likely root cause, which is the section #2834 didn't have. When there was an existing issue, the comment it posted said that tempo and tempo/0 were in error. It paraphrases the status rather than quoting hook failed: "start", and doesn't put tempo in the title, but I think that's fine as it is. The new log line showed up on every run that got an answer.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant