Conversation
mmitche
changed the title
|
@mmitche I wanted to look into this, but I think the "APK Test" lanes might be suitable for Helix to prototype first. The NUnit (ancient!) MSBuild integration tests we have, would require installing .NET SDK w/ the currently built Android workload on Helix -- and I'm not sure if Helix can do that or not. Those NUnit tests create projects, build them, run them on eumulators, and assert the output. @simonrozsival enabled a simple Android arm32 job here, so Helix is being used & working in one place so far: |
Good to know, thanks. |
Build the existing Android device-test flavors in Azure Pipelines, submit one deterministic work item per flavor to the proven Android Helix queue, and centrally publish pulled TRX and device diagnostics. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
188932a to
350fab9
Compare
mmitche
changed the title
Resolve the prepared SDK once, pass SDK/JDK paths to every clean/build invocation, and create the diagnostics root before any build can fail. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Match the proven MAUI R2R Helix payload layout by shipping platform-tools beside each APK instead of relying on correlation payload extraction layout. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Use the Windows public build agent so each work item receives a compatible adb.exe, while starting after the existing macOS artifact build only. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Include the USERPROFILE android-toolchain location used by the Windows setup template. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Shut down dotnet build servers after staging each flavor so shared TestRunner.Core outputs are not locked by the next APK build. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Use the pipeline variable established before APK builds when staging Windows platform-tools for Helix. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Reset LASTEXITCODE after accepted robocopy success codes so APK submission preparation does not fail after generating all work items. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Use the same explicit artifact paths as the existing APK lanes instead of selecting a potentially stale recursive match. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Use the same dotnet test/MTP build path as the existing APK lanes, tolerate the expected no-device launch failure on the preparation agent, and require the instrumented APK output. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Use aapt badging output to stage the exact instrumentation component from each built APK and include installed instrumentation in device diagnostics. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Remove downloaded package artifacts before SignAndroidPackage, then require and inspect the newly built APK so instrumentation diagnostics cannot be based on stale outputs. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Allow the optional label and other aapt badging fields between instrumentation name and target package, and print badging on future parse failures. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Build each Helix APK with a minimal explicit instrumentation entry and the actual application package, then verify the resulting component with aapt before submission. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Use project-relative obj/Helix manifest paths because the Android build treats AndroidManifest as project-relative even when given an absolute path. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Use a conditional MSBuild import so only the APK test project receives the generated manifest while project references retain their normal manifests. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Apply the app-only AndroidManifest override through CustomAfterMicrosoftCommonTargets and log the resolved value before Android package validation. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Let the two device-test projects consume a private Helix-only AndroidManifest path directly, avoiding global property propagation to project references. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Temporarily enable the guarded stage by default on this draft branch so the public PR pipeline can provide a same-commit dual-path run while Azure CLI authentication is unavailable. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Use the AndroidManifestOverlay item intended for additive manifest entries so the normal generated test manifest and explicit instrumentation are both packaged. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Inject a Helix-only assembly-level Instrumentation attribute into each test application so normal manifest generation emits the runner component without replacing the app manifest. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Add explicit assembly-level Instrumentation attributes to both device-test applications so normal SignAndroidPackage builds retain their runners without pipeline manifest overrides. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
cce79ef to
188932a
Compare
mmitche
changed the title
|
Thanks — I paused the host-only iteration and built the APK-lane prototype to test this suggestion. The APK lanes are definitely critical-path relevant (they ended the longest path in 3/13 P80 builds), and their eight device invocations are already natural sub-15-minute work items. The Android Helix payload itself is also much smaller. The blocker I found is the current MTP artifact boundary:
I preserved that experiment on The original SDK/workload concern is now demonstrated as solved: build 1568901 ran the old and Helix host paths from the same commit and succeeded with exact parity — Windows 1,504/1,504 and Linux 167/167, with identical pass/skip outcomes. Helix reduced the Windows execution tail from ~100m to ~44m and Linux BuildTest from 74m to ~21m after submission. The tradeoff is still-large shared payloads (9.04 GiB Windows / 6.11 GiB Linux), now documented in the PR. My recommendation is to keep the proven host path here, then make “export the exact runnable MTP package/split set” a prerequisite for a follow-up APK Helix migration. |
Temporarily default this draft branch to the APK-only validation path so public CI can verify the build-server shutdown fix from build 1569600 without unrelated test stages. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
188932a to
9fe8d92
Compare
mmitche
changed the title
|
@mmitche ok, I see probably two issues:
So, I think an initial path forward could be;
|
Use a project-local opt-in property so only the test application consumes the generated instrumentation manifest. Preserve build binlogs and merged manifest diagnostics for failed payload preparation. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
aapt badging omits the instrumentation record even when the packaged binary manifest contains it. Inspect AndroidManifest.xml directly and retain both aapt views for diagnostics. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Inject instrumentation into the normal generated manifest, embed Debug assemblies for direct installation, grant Android 16/17 local-network permissions, retain device diagnostics, and retry unhealthy device assignments once. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Wait for delayed Android device attachment, retain per-attempt TRX output, retry a failed instrumentation run once, and keep first-cause diagnostics from being masked during cleanup. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
PowerShell 5.1 promoted adb daemon startup stderr into a terminating error before device discovery. Capture native stderr without masking the process exit code. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Use the public Ubuntu Android 29 emulator queue that matches the existing APK lanes, generate a bash work-item runner, and use queue-provided adb instead of Windows platform-tools. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Normalize generated bash scripts before Helix upload so the Ubuntu Android queue can execute them without CRLF parsing failures. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Resolve adb from Android SDK environment variables and known queue paths, with a bounded filesystem fallback when the queue does not add platform-tools to PATH. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Wait for an online API 29 emulator, select its serial explicitly, and scope every adb command so the queue's second offline emulator cannot make commands ambiguous. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Select the queue's x86_64 API 29 emulator when both x86 and x86_64 devices are online so the staged APK native libraries match the device ABI. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Wait for the API 29 package service, retry transient installs, add exact test-name instrumentation filters, and run the JNI count checks tracked by #12031 in fresh processes before merging their TRX results. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Re-enable the existing test stages alongside the now-passing Helix prototype so the macOS APK lanes and all eight Helix flavors execute from the same commit at the 15-minute target. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Register the TRX namespace as the default before merging isolated and remaining results so Azure Pipelines parses and publishes every Helix test run. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Keep the ElementTree merge command at Python top level while registering the default TRX namespace, avoiding the indentation failure seen in parity run 1575040. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Do not fall back to the API 29 x86 emulator while x86_64 is still booting; the staged APKs contain x64 native libraries and cannot install on the 32-bit device. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Apply explicit time limits to boot, package-service, install, instrumentation, pull, and cleanup commands so an unresponsive API 29 emulator fails causally instead of consuming the entire Helix work-item timeout. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Return the public pipeline to opt-in behavior after the exact same-commit parity run and remove an unrelated formatting-only change from the prototype history. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
mmitche
changed the title
|
@jonathanpeppers I ran the all-eight prototype far enough to answer the lane-split question with same-commit evidence. Build 1575692 ran both existing macOS APK jobs and Helix correlation 5dccff50-899e-4a07-b4b3-a4ef3d5116a6: all eight work items passed, with zero fully-qualified inventory or pass/fail/skip differences for every flavor. TRX publication also succeeded. The operational evidence supports your proposed split: Debug had to embed assemblies, so Helix does not replace fast-deployment/sideload coverage; AAB flavors were converted to APK, so Helix does not replace Play-style split coverage. NoAab and JcwGen are the safest first production candidates. The correlation itself took 8.94m with 267.5 MiB total payload, but sequential package preparation made the prototype job 42.3m, so package production/reuse is the next optimization before moving a lane. The PR body has the full parity and timing table. |
|
I think we could split this up as:
We might currently have multiple I'd be OK taking out of draft and merging, if it's working in this way. |
|
/review |
|
✅ Android PR Reviewer completed successfully!
|
![github-actions[bot]](https://avatars.githubusercontent.com/in/15368?s=60&v=4){kind=small}
There was a problem hiding this comment.
The Helix payload construction, bounded device operations, manifest preservation, and final TRX publication are thoughtfully implemented. CI is green across all 44 reported checks; the prototype stage is default-off in that run, while the PR documents a separate successful opt-in parity run.
Before merge, align apkTestsHelixOnly with its advertised standalone behavior and keep caller-supplied exclude-test filtering active when only built-in exclusions are disabled. The target-minutes parameter should also be removed or given an observable effect.
Generated by Android PR Reviewer for #12479 · gpt56 · 139.1 AIC · ⌖ 9.04 AIC · ⊞ 25.7K
Comment /review to run again
| displayName: APK Tests Helix Prototype | ||
| dependsOn: | ||
| - mac_build | ||
| condition: and(succeeded(), or(ne(variables['SkipTestStages'], 'true'), eq('${{ parameters.apkTestsHelixOnly }}', 'true'))) |
There was a problem hiding this comment.
🤖 apkTestsHelixOnly does not actually make this a prototype-only run. It only allows this stage to run when SkipTestStages was separately set by skipTests=true; selecting apkTestsHelixOnly=true by itself still leaves the package, Linux, MSBuild, emulator, Java.Interop, Android tools, and MAUI stages enabled. Please either derive SkipTestStages from this flag (guarded by the prototype being enabled) or rename/document this as a companion override that requires both flags.
(Rule: Pipeline conditions must match advertised behavior)
| Log.Info (LogTag, $"Excluding test: {name}"); | ||
| } | ||
| } | ||
| foreach (var name in GetListExtra ("exclude-test")) { |
There was a problem hiding this comment.
🤖 exclude-test extra under noexclusions, even though the log message says that switch skips only built-in exclusions and the existing caller-supplied exclude extra below is intentionally applied outside this block. Consequently, -e noexclusions true -e exclude-test Foo silently runs Foo. Please move this loop alongside the other command-line exclusion handling.
(Rule: Test option semantics)
| [string] $WorkItemOS = 'windows', | ||
|
|
||
| [ValidateRange(1, 1440)] | ||
| [int] $TargetMinutes = 15 |
There was a problem hiding this comment.
🤖 💡 YAGNI — TargetMinutes is accepted and exposed as a pipeline knob, but its only use is writing the value to work-item-generation.json; it does not affect partitioning, scheduling, or timeout behavior. That makes changing apkTestsHelixTargetMinutes operationally inert. Since the current design deliberately keeps each flavor indivisible, consider removing this parameter until it controls behavior (or wire it into the relevant policy).
(Rule: Remove unused configuration)
I still need to look into this a bit further. Not convined it's where we want to be yet. |
This branch has not been deployed
3 participants
Summary
Guarded APK-lane Helix prototype for AB#12334. The proven host-test prototype remains preserved at
prototype/host-build-tests-helix-preserved(188932a).The public pipeline opt-in is
enableApkTestsHelixPrototype=true; the final default isfalse.apkTestsHelixTargetMinutesdefaults to15. Existing macOS APK jobs remain enabled unlessapkTestsHelixOnly=trueis selected for prototype-only diagnosis.Architecture
A Windows Azure Pipelines job builds and stages the eight measured APK flavors once: Debug, Release, NoAab, CoreCLR, Mono, CoreCLRTrimmable, NativeAOT, and JcwGen. Each deterministic work item contains one signed APK plus metadata and scripts; the API 29 Ubuntu Android queue supplies the x64 emulator and
adb, so platform-tools are not duplicated.The build preserves the normal generated application manifest, injects only instrumentation and local-network permissions, validates the packaged binary manifest with
aapt, and retains per-flavor binlogs/manifests. The Linux runner waits for the x64 emulator and package service, bounds every adb operation, installs the APK, runsam instrument, captures console/logcat/device/package state, pulls TRX, and propagates failures. Process-global JNI count tests tracked by #12031 run in a fresh process; their TRX is merged with the remaining inventory and published centrally.At the 15-minute target, each flavor is already an indivisible batch of hundreds of tests below the target, so the prototype emits eight flavor work items rather than one test per item.
Exact same-commit parity
Run: dnceng-public build 1575692 at source head
5dbbb63.Helix correlation:
5dccff50-899e-4a07-b4b3-a4ef3d5116a6.Both existing macOS APK jobs and all eight Helix work items succeeded. Comparison used fully-qualified test identity plus outcome multisets:
*The explicit CoreCLR lane was recently removed as a duplicate; its Helix result was compared with Release, whose exact inventory/outcomes are equivalent.Helix TRX publication produced eight Azure test runs successfully. NativeAOT's isolated JNI check failed its first process-global count attempt and passed the retained retry, consistent with #12031.
Timing and payload evidence
Final correlation had zero initial queue delay, completed in 8.94 minutes, and used one available emulator worker serially. Work-item durations:
Compressed work-item payloads were 14.8-86.2 MiB, 267.5 MiB total. The only shared correlation payload was a 2.3 KiB Helix command bundle; no SDK/workload/platform-tools payload is repeated on Helix.
End-to-end preparation remains the main limitation: the Helix AzDO job was 42.3m versus 40.5m/32.9m for the two existing macOS APK jobs. The device correlation itself is fast, but building eight packages sequentially offsets that gain. Queue capacity also varied in earlier experiments (0-33m queue wait).
Jonathan's lane-split concern
The experiment confirms the proposed split is safest:
Validation
build-tools/automation/apk-tests-helix/ApkTestsHelix.Tests.ps1passes (Windows/Linux generation, bash/PowerShell syntax, manifest preservation, deterministic metadata).git diff --checkpasses.e63e99dpassed build 1575806; the APK prototype stage was absent at its default-off setting.Tracking