fix(SDK-7061): resilient TRA build-stop — retry + success-gated flag + detached watchdog

[pri-gadhiya]

What is this about?

Fixes SDK-7061: WebdriverIO runs where the Test Observability / TRA build appears "stuck" and only closes via the ~60-min server-side inactivity timeout, even though the run finished.

Root cause: the build-stop (PUT /api/v1/builds/<uuid>/stop) was sent once, best-effort — no retry, and it did not check the HTTP status. If that request failed/timed out, or the WDIO launcher was terminated (CI cancel / kill -9 / OOM) before or while it was sent, the build was never marked complete.

This PR makes the build-stop resilient (verified on WebdriverIO-cucumber, before/after, including a transient stop-failure and a launcher kill -9):

1. stopBuildUpstream now retries (3× with backoff) and treats a non-2xx response as a failure (previously a non-2xx was logged as success).
2. buildStopped is set only when the stop actually succeeds, so a failed stop stays recoverable by the exit-cleanup path.
3. A detached shutdown-watchdog (own session; survives a kill -9 of the launcher's process group) sends the build-stop if the launcher dies without a clean shutdown. The JWT is read from the inherited environment (never written to disk); the coordination marker is owner-only (0o600).

Known boundary (out of scope here): a full container / process-group teardown that also reaps the watchdog still needs a server-side inactivity/heartbeat safeguard — tracked separately (the shared browserstack-binary stopBuild path has the same retry/orphan gap for binary-flow SDKs).

Related Jira task/s

• https://browserstack.atlassian.net/browse/SDK-7061
• Related (server-side appearance analysis): https://browserstack.atlassian.net/browse/TRAP-4046

Release (mandatory for every PR — required for the ready-for-review label)

Version bump: (required — tick exactly one)

• minor (backwards-compatible feature)
• patch (bug fix or other small change)

Release notes type: (optional)

• New Feature
• Bug Fix
• Other Improvement

Release notes (customer-facing): (optional but encouraged)

• Fixed WebdriverIO test results sometimes not appearing (builds staying "in progress") when the build-completion signal failed or the test runner was interrupted mid-shutdown.

Release notes (internal): (required — engineer-facing; what actually changed / why)

• stopBuildUpstream hardened: 3× retry + backoff + response.ok check (was a single best-effort PUT; non-2xx was logged as success).
• buildStopped gated on stop success (was set unconditionally), so the exit-cleanup path can re-attempt a failed stop.
• New detached shutdown-watchdog (buildWatchdog.ts / buildWatchdogMarker.ts) spawned at build start; survives a launcher kill -9, sends the stop from a separate session. JWT read from inherited env (not persisted to disk); marker file 0o600, log dir 0o700.

Checklist

• Ready to review
• Has it been tested locally?

PR Validations

Run Tests: Comment RUN_TESTS to trigger sanity tests.