T-1091 [CI] Fail the E2E test when Vector logs errors or warnings

[PetrHeinz]

The E2E workflow only checked that every pod was Running. Vector can be running while its transforms fail on every event (for example the to_float conversion error from the metrics pods parser that a customer hit), so the test now reads the Vector logs of both the agent DaemonSet and the aggregator Deployment and fails when any line is logged at WARN or ERROR level during the minute the stack is observed.

Why the existing "Show logs from Vector pod" step was not enough:

• It selected app.kubernetes.io/name=vector, which matches the agent pod only. The aggregator (where the Kubernetes metrics parsers live since Extract k8s metrics to a separate standalone deployment #26) is labelled app.kubernetes.io/name=vector-aggregator, so the step never showed its logs at all.
• kubectl logs with a label selector defaults to the last 10 lines per pod, so even for the agent only the tail was printed. Every recent run shows exactly 10 "Found new file to watch" lines and nothing else.

What changes:

• Wait for a while with the running pods records the time the 60 s observation starts in OBSERVED_SINCE. At that point metrics-server is available, both Vector pods are running and the hello-world pod has produced its logs.
• Show logs from Vector pods selects app.kubernetes.io/name in (vector, vector-aggregator), uses --tail=-1 for the full log and --prefix so each line names its pod. This is the debugging view and stays unfiltered.
• Check that Vector logged no errors or warnings while the pods were running reads the same pods with --since-time=$OBSERVED_SINCE and fails the job, printing the offending lines, when any line's level column is WARN or ERROR. The pattern is anchored on the [pod/...] prefix and the timestamp, so a log message that merely contains the word ERROR does not match.

What the first full look at the logs showed (run 38409435353, the first commit of this PR with no window and no exclusion), and why the check is scoped the way it is:

• The aggregator logs four WARN ... verify_certificate option is DISABLED lines at startup. They come from the tls overrides in the E2E values template, which are the same overrides the Kubernetes docs give customers, so every real install prints them too. They happen once at startup, before the observation window.
• The aggregator's http_client sources start scraping before metrics-server listens, so the first two scrapes log WARN ... Connection refused plus ERROR ... HTTP request processing error and then recover. Both Deployments start together, so this happens in every install and is also before the window. A scrape that still fails inside the window (wrong service name, metrics-server down) keeps failing every 15 s and is caught.
• The metrics sink logged one WARN ... connection closed before message completed followed by WARN ... Retrying after error inside the window. This is the keep-alive race between a pooled client connection and the server closing it, the retry succeeded, and it would recur at random. The check therefore drops sink{...} lines of those two kinds. A rejected request (4xx) is logged by Vector as a non-retriable ERROR and still fails the test.

The workflow installs the last released chart, so chart-side changes (for example an init container that makes the aggregator wait for metrics-server, which would remove the startup errors customers see) are separate work and would only show up here after a release.

🤖 Generated with Claude Code