test: add Android full emulator coverage - #1484
Conversation
bfca2fd to
d8de04d
Compare
94e47ce to
8538b7b
Compare
|
PR #1484 is not ready at exact head
No |
19a1f55 to
d835e27
Compare
|
Exact-head review found two blockers:
The manifest/evidence machinery is otherwise structurally meaningful, but the expanded live-scenario scope goes beyond #1426’s original guardrail and needs explicit coordination/approval after the stack is repaired. |
c6a379a to
30b8aba
Compare
|
Exact head |
258f65b to
54e3d10
Compare
|
Dependency update: #1482 has now merged, while this draft remains on unchanged head |
8538b7b to
eacf244
Compare
Size Report
Startup median (7 runs, lower is better):
Top changed chunks: no changes in the largest emitted chunks. |
|
Re-review at
The previously reported stopped-trace MCP artifact-schema issue is resolved. Static, Coverage, Integration, type/layering, Linux, macOS, and CodeQL are green; Android Release is a confirmed PR-owned failure and device smoke is not complete. |
eacf244 to
84b1416
Compare
|
Addressed the current review findings in 84b1416:\n\n- Registered the Android Expo module as |
thymikee
left a comment
thymikee
left a comment
There was a problem hiding this comment.
The full-tier coverage is valuable and the production trace-artifact fix is sound, but exact head 84b1416cf is not ready yet.
- P2 — keep timing diagnostic-only.
test/integration/android-emulator-e2e/live-coverage-report.ts:17-34reintroduces a ten-minute wall-clock assertion into functional coverage. #1482 explicitly removed this because shared emulator load can make a correct journey exceed an arbitrary threshold. The job already has an outer timeout and the report records timings; delete the assertion/import/constant. - P2 — do not rediscover the coverage report by traversal order.
.github/workflows/android.yml:78-79and.github/workflows/replays-nightly.yml:142-143usefind ... | tail -n 1. With multiple artifact directories this can select a stale report—the same defect corrected during #1482. The runner already prints the exact report path and timing summary, so remove both lookups and inline Node programs. - P2 — reuse completed build infrastructure.
.github/workflows/android.yml:75-76repeats the dedicated CLI build and daemon reset immediately above the emulator action. In the nightly workflow,setup-android-replay-hostalready restores or packages and verifies both Android helpers, but line 137 repackages them withpnpm build:android. Keep only work that the emulator genuinely requires.
Exact-head Android/iOS smoke and native Release checks are still running. No ready-for-human label until these regressions are removed and the full Android emulator suite passes.
|
One additional P2 from the tightening pass: |
|
Addressed the latest tightening-pass feedback in 29ab77f:\n\n- Moved event pagination and replay-evidence logic to shared |
|
Re-review at Not ready: both exact-head device smokes reach the fixture but fail on deterministic selector state—iOS cannot find No |
|
Addressed the exact-head device-smoke failures in 9ef5bf1:\n\n- iOS now reveals |
|
Addressed the requested tightening in fe123c0: moved platform-neutral assertNonEmptyFile and assertMp4File into test/integration/live-device-e2e/assertions.ts, with Android and iOS assertion modules importing and re-exporting the same public helpers. No generic runner, .ad templating, or iOS scenario split added.\n\nValidation: pnpm check:affected --run; focused Android/iOS coverage tests (22 passed); pnpm typecheck. |
|
Re-reviewed |
thymikee
added
the
ready-for-human
|
Addressed the exact-head device failures in ef1dbaa.\n\n- Android: preserve the native-alert removal assertion by revealing the alert canary before taking the automation diff baseline after Home/Recents restoration.\n- iOS: reveal and assert each canary immediately before its interaction; the bounded reveal for Press intentionally does not claim Long press is simultaneously visible.\n\nNo retries or assertion weakening added. Focused coverage tests: 22/22; format/typecheck passed. Fresh exact-head Android and iOS smoke jobs are queued. |
|
Re-reviewed |
|
android-emulator-runner runs the script with /usr/bin/sh, which is dash on
ubuntu runners and rejects `-o pipefail`:
/usr/bin/sh: 1: set: Illegal option -o pipefail
The Android full emulator suite has therefore failed on every nightly since
it landed in #1484, aborting before the first command. The sibling emulator
scripts in android.yml and perf-nightly.yml already use `set -eu`/`set -e`;
match them. The serial assignment stays guarded by `test -n".
) `assertElementTextAfterScrolling` waited 1000ms per attempt. The wait's whole budget goes to its first capture, so that budget has to cover one snapshot of the current surface; on a loaded simulator it does not, and every attempt fails with `wait_capture_stalled` instead of reporting the element as off-screen. This broke the iOS Smoke Tests lane on main. #1484 moved the smoke automation scenario onto this helper for `automation-press` and `automation-longpress`; before that it was only reached from the full lane, so the tight budget went unnoticed. Every sibling wait in the same scenario already budgets 2500ms or more (`label="Settings"` 10000, `alert wait` 5000, `wait text` 2500). Two changes: - Raise the per-attempt budget to 2500ms, matching the nearest sibling. - Stop spending a scroll attempt on a stalled capture. A stall means the snapshot never came back, so the surface was never read — it is not evidence the element is off-screen, and scrolling on it moves the surface for an unrelated reason. Two stall retries absorb a slow runner; a genuine absence still consumes attempts and still fails. The final assertion now carries the last wait's JSON, so a future failure says whether it stalled or genuinely never found the element. Claude-Session: https://claude.ai/code/session_01RXQLYV7etZx3gcXsUsrQJ8 Co-authored-by: Claude <noreply@anthropic.com>
The full:observability-artifacts scenario (#1482/#1484) had never executed end-to-end: both nightly Android Full Emulator Suite runs since merge died on adb infra before the suite ran, and a live run fails deterministically at its first perf assertion. Fixing that revealed four more latent failures, each written against iOS or remote-daemon behavior the Android live run does not have. Validated with two consecutive green full-tier runs on a dedicated Pixel 9 Pro XL emulator. - perf metrics: assert totalPssKb (Android's required meminfo field) instead of the Apple-only residentMemoryKb. - presses: reveal the Quick-actions card with scroll steps before pressing home-open-catalog/home-open-settings — Android snapshots only contain on-screen nodes — and restore scroll top before waiting on the home title, since scroll position persists across tab switches. - batch get: target id="dismiss-notice" (a node that owns its text); Android resolves the home-title container to a child's text (the subtitle), unlike iOS's container label. - events: run an explicit snapshot so the timeline assertion holds when the scenario runs standalone under AGENT_DEVICE_ANDROID_E2E_SCENARIOS. - artifacts: assert the local-client contract — trace-log tracked, downloadable, consumed; screen-recording inventory entries only exist for remote clients (artifacts without a client localPath are never tracked). Also stop consuming the download response body in the assert message before arrayBuffer() reads it.
Three Android fixture defects from the #1482/#1484 full-tier suite, none of which ever executed in CI (both nightlies since failed on adb infra before the suite ran). All three verified live on a fresh API 36 emulator with a pixel_7-geometry AVD and a Release fixture APK: - 01-navigation-scroll.ad clicked label=Catalog, but the expo-router NativeTabs cart badge leaks '0 new notifications' into the tab's content description even while hidden, and unselected native tabs expose no child text node - exact match can never hit. Target the composed label the device actually exposes (deterministic at fixture start: cart is 0 after --relaunch). The badge does not leak on iOS, so the iOS twin keeps label="Catalog". - checkout-form-android.ad opened by iOS display name 'Agent Device Tester'; Android open resolves packages (the APK label is 'Agentdevicelab'), so APP_NOT_INSTALLED was guaranteed. Use the package id, matching gesture-lab-android.ad. - gesture-lab-android.ad aimed every gesture at y=700, above the gesture card (its targets span y754-1329 on pixel_7 geometry; the home screen gained content above the card since authoring). Re-aim pans inside the exact-two-pointer zone, flings on the image clear of that zone, and pinch/rotate/transform at the card center. Verified: full suite passes 2/2 via the public test command (20 + 32 steps replayed). Refs #1478
…#1538) * fix(test): repair Android replay fixtures against live device reality Three Android fixture defects from the #1482/#1484 full-tier suite, none of which ever executed in CI (both nightlies since failed on adb infra before the suite ran). All three verified live on a fresh API 36 emulator with a pixel_7-geometry AVD and a Release fixture APK: - 01-navigation-scroll.ad clicked label=Catalog, but the expo-router NativeTabs cart badge leaks '0 new notifications' into the tab's content description even while hidden, and unselected native tabs expose no child text node - exact match can never hit. Target the composed label the device actually exposes (deterministic at fixture start: cart is 0 after --relaunch). The badge does not leak on iOS, so the iOS twin keeps label="Catalog". - checkout-form-android.ad opened by iOS display name 'Agent Device Tester'; Android open resolves packages (the APK label is 'Agentdevicelab'), so APP_NOT_INSTALLED was guaranteed. Use the package id, matching gesture-lab-android.ad. - gesture-lab-android.ad aimed every gesture at y=700, above the gesture card (its targets span y754-1329 on pixel_7 geometry; the home screen gained content above the card since authoring). Re-aim pans inside the exact-two-pointer zone, flings on the image clear of that zone, and pinch/rotate/transform at the card center. Verified: full suite passes 2/2 via the public test command (20 + 32 steps replayed). Refs #1478 * docs(test): pin the Android gesture fixture's validated emulator geometry The re-aimed coordinates are validated on CI's profile (pixel_7 1080x2400 @420); any booted emulator can receive them via test-app:replay:android, so the fixture and README now say which geometry the numbers mean and what a mismatch failure looks like. The checkout twin is selector-driven and unconstrained. Co-Authored-By: Claude <noreply@anthropic.com> --------- Co-authored-by: Claude <noreply@anthropic.com>
1 participant
Summary
Extends the Android emulator coverage introduced in #1482 with a full nightly tier while keeping the required smoke tier focused. The full suite adds fixture-observed coverage for:
The coverage report now records the executed tier and a manifest-derived classification rollup, and the nightly lane requires the trusted native fixture produced for the exact head.
Bugs exposed and fixed
trace stopwrote a file but did not publish it as a daemon-managedtrace-log, so it was absent from artifact inventory and could not be downloaded through the artifact endpoint.Builds on merged PR #1482. Scope spans Android E2E scenarios, the Android test fixture, trace artifact publication, and CI orchestration (31 files).
Validation