Skip to content

[CI] Parallel runtime build races host_runtime_contract.h generation before API diff #26745

Closed
Closed
[CI] Parallel runtime build races host_runtime_contract.h generation before API diff#26745
Labels
ci-postmortemCI post-mortem analysis: flaky tests, infrastructure failures, shared regressionscopilot

Description

@dalexsoto
dalexsoto
opened on Sep 25, 2026

Failure signature

make all -j 4 IGNORE_SIMULATORS=1 can compile .libs/iossimulator-x64/runtime-dotnet-coreclr.o from runtime/runtime.m before host_runtime_contract.h is generated. The compiler then fails at runtime.m:28:10 with fatal error: 'host_runtime_contract.h' file not found.

Classification: source-related native Makefile prerequisite/build-order race, not a failed API comparison or a genuinely flaky test. Impact: two distinct monitored PRs each fail xamarin-macios-pr-apidiff (definition 22268) before either the current-PR breaking-diff or vs-stable comparison executes.

Confirmed occurrences

  1. #26727 / head ff7a8dd2bd32fb7cc16fbc2cc4659593eaf5bc0c (app/dotnet-maestro): Azure run 15456544 / Detect API changes / Build, pipeline 22268, stage/job/task attempt 1, task log 83. Failure at 2026-09-25T02:14:45.996818Z while compiling the iOS simulator x64 runtime object. The later publisher job could not find change-detection.zip (log 123) because the build never reached API detection; this is a cascade.
  2. #26737 / head c4a67d7e9d7aff24385027093bbe252ff238f951 (rolfbjarne): Azure run 15462899 / Detect API changes / Build, same definition, attempt 1, task log 83. Exact same object/header error at 2026-09-25T15:04:59.946192Z. This PR changes only .config/dotnet-tools.json; the updated dotnet-ilrepack 2.0.48 restored successfully before the native build failed. Missing change-detection.zip in publisher log 123 is again downstream.

First observed: 2026-09-25T02:14:45.996818Z. Latest observed: 2026-09-25T15:04:59.946192Z. Totals: 2 confirmed build failures across 2 distinct monitored PRs; multiple failed GitHub child checks/publisher tasks are not additional occurrences.

Root-cause investigation and reproduction pattern

The PR heads have identical runtime/Makefile (Git blob e887763bb67e6a472793588ebb53c9843fc906cc) and mk/rules.mk (blob b41dfec03ff631a7546ddde32877af2d30905867). The Makefile has a rule to generate host_runtime_contract.h, but lists it as a prerequisite of coreclr-bridge.m, not of the runtime.m object that includes it; the object pattern rule compiles from %.m and EXTRA_DEPENDENCIES, which omit this header. In both failed parallel builds the OBJC [iossimulator-x64] runtime-dotnet-coreclr.o step precedes the missing-header error without an earlier header-download line. Other same-head #26727 package/simulator builds downloaded the header before compilation and succeeded, consistent with ordering-sensitive behavior.

Separate searches over all open and closed issues covered the header name, object/task, fatal error, runtime Makefile component, and parallel native build ordering. No exact match was found. The broad matches #11799 (unrelated NSUrlSession null-reference crash) and #26221 (unrelated in-test HTTP redirect stall) were inspected and excluded; this is their first canonical issue for this root.

Workaround, availability, and next step

No workaround has been verified or applied. A source fix must make header generation an explicit prerequisite before compiling runtime.m (or otherwise guarantee that ordering); a rerun merely to hope for a green result is not authorized. The affected bot and Rolf branches were not edited, committed to, or pushed by this monitor.

Azure Build task logs 83 and publisher logs 123 are linked above and are retention-limited. No primary native binlog, raw current-PR/vs-stable API-diff artifact, or Helix test work item was produced; neither API comparison ran. No bindings or suppression files were changed. Worker identities, machine IPs, credentials, and feed URLs are omitted.

AI-assisted CI monitoring issue.

Activity

  1. dalexsoto
    added
    copilot
    ci-postmortemCI post-mortem analysis: flaky tests, infrastructure failures, shared regressions
    on Sep 25, 2026

  2. dalexsoto commented on Sep 25, 2026

    @dalexsoto
    dalexsoto
    on Sep 25, 2026
    MemberAuthor

    Third distinct PR hit the same native generated-header Makefile race

    #26683 / head 2fbdc937e9bf5d9e9d52ccd3631b33bf41c870b8 (app/dotnet-maestro) failed Azure API-diff run 15463858 / Detect API changes / Build (pipeline 22268, stage/job/task attempt 1, task log 83). Under make all -j 4 IGNORE_SIMULATORS=1, compilation of .libs/iossimulator-x64/runtime-dotnet-coreclr.o began before host_runtime_contract.h existed and failed at runtime/runtime.m:28:10 at 2026-09-25T16:44:51.092502Z. The PR head has the same runtime/Makefile and mk/rules.mk Git blobs as the previously confirmed #26727 and #26737 runs; this is their exact missing-prerequisite build-order root, not an inferred API-binding regression.

    Detect API changes was skipped; the later publisher's missing change-detection.zip (task log 123) is a cascade. Neither the informational current-PR breaking API diff nor the separate vs-stable comparison ran or produced a raw artifact. Classification: source-related native Makefile race. Counts: 3 confirmed failures across 3 distinct monitored PRs; extra GitHub child checks and the publisher job are not separate occurrences. No monitor retry, source edit, binding change, suppression edit, commit, or push was made to this bot branch.

    Issue-specific diagnostics: The refreshed issue body has no Additional info to collect section. Exact PR/head/run/attempt/job/task, native object/header/parallel command, identical rule blobs, comparison skip and publisher cascade were collected from task logs 83/123. Baseline-specific API diff results, a primary native binlog and a Helix item are unavailable because the build stopped before comparison or tests. Worker identity/IP and infrastructure credentials are withheld; artifacts are retention-limited. No retry merely to hope the race passes; accepted-request counts remain uncertified.

    AI-assisted CI monitoring update.

  3. dalexsoto commented on Sep 26, 2026

    @dalexsoto
    dalexsoto
    on Sep 26, 2026
    MemberAuthor

    Fourth distinct PR: the same native generated-header build-order race

    #26752 / head 15ba488b334adeca274b89aa273e3e65d93500ef (target main) failed Azure API-diff run 15476877 / Detect API changes / Build (pipeline definition 22268; run, stage, job and task attempt 1; job 276491e5-a85e-51e3-6c38-98e261218d1a, task 76afb9de-7feb-5cd3-1173-f84cb7d12f5e, log 83). At 2026-09-26T15:07:44.916Z, make all -j 4 IGNORE_SIMULATORS=1 compiled .libs/iossimulator-x64/runtime-dotnet-coreclr.o before host_runtime_contract.h existed; runtime/runtime.m:28:10 failed with the same missing-header error. This PR changes only dotnet/Workloads/SignList.xml; its runtime/Makefile and mk/rules.mk Git blobs match the three earlier occurrences exactly.

    Classification: source-related missing Makefile prerequisite/build-order race; 4 confirmed occurrences across 4 distinct monitored PRs (#26727, #26737, #26683, #26752). The sibling #26753 API-diff run 15476880 completed its Build task successfully while its other checks were still running; it is not counted as a failure. No workaround is verified. No monitor retry was submitted: this source-related failure is ineligible, and all-actor retry-budget totals are not certified. No source edit or push was made.

    Issue-specific diagnostics for #26752: The refreshed issue body has no Additional info to collect section (not applicable). Detect API changes was skipped, so neither the informational current-PR breaking diff nor the separate vs-stable comparison ran. The publisher's change-detection.zip ENOENT (task log 123) is a downstream cascade, not another occurrence or a baseline finding. Only build-configuration and agent-log artifacts were published; raw comparison results, a primary native binlog, and a Helix work item are unavailable. Task logs 83/123 are available subject to Azure retention. Worker identifiers, IPs, and credentials are omitted; no binding or suppression-file change was made.

    AI-assisted CI monitoring update.

  4. dalexsoto commented on Sep 26, 2026

    @dalexsoto
    dalexsoto
    on Sep 26, 2026
    MemberAuthor

    #26752 external API-diff attempt 2 reproduced the same header race

    The attempt-1 failure on #26752 / unchanged head 15ba488b334adeca274b89aa273e3e65d93500ef remains confirmed. An externally initiated reexecution of its existing Azure API-diff run 15476877, attempt 2 / Detect API changes / Build again failed at 2026-09-26T16:56:27.548Z: make all -j 4 IGNORE_SIMULATORS=1 compiled .libs/iossimulator-x64/runtime-dotnet-coreclr.o before host_runtime_contract.h existed (runtime/runtime.m:28:10, task log 183). Pipeline 22268, stage/job attempt 2, job 023efb3e-d10c-52da-87d2-e51a6fae706a, Build task 2ee5ba36-f97c-51c0-5f40-8dfaf3c2d983 (task-record attempt 1). The publisher job 74162e51-23ec-5152-5f9a-eb0fad5f6eae also ran on attempt 2; its task log 223 failed because change-detection-2.zip did not exist. The parent run completed failed at 17:00:19Z.

    Classification: the same source-related Makefile prerequisite race as attempt 1, not a new PR or an API-diff result. #26745 totals: 5 confirmed failure occurrences across 4 distinct monitored PRs. Detect API changes was skipped again: neither the informational current-PR breaking comparison nor the separate vs-stable comparison ran, and no raw comparison artifact was produced. No binding or suppression-file edit is justified. The separate #26752 simulator stress-test timeout is unresolved and not counted here.

    Retry accounting: The monitor submitted no retry. Azure records stage/job attempt 2 and both failed jobs, establishing at least one external accepted reexecution, but the accepted request payload, actor, and exact all-actor request grouping were not retained. Exact PR-lifetime/head accepted-request counts therefore remain uncertified; no further retry is authorized, and a source-related race is ineligible in any case. No source edit, commit or push was made.

    Issue-specific diagnostics: The refreshed issue body has no Additional info to collect section (not applicable to this occurrence). Exact PR/head/run/stage/job/task attempt, native object/header error, failed publisher cascade, and terminal result came from the Azure timeline and task logs 183/223. Baseline-specific API results, a primary native binlog, and a Helix item are unavailable because the build stopped before comparison/tests; artifacts and task logs are retention-limited. Worker identifiers, machine IPs and credentials are omitted.

    AI-assisted CI monitoring update.

  5. dalexsoto commented on Sep 27, 2026

    @dalexsoto
    dalexsoto
    on Sep 27, 2026
    MemberAuthor

    #26737: API-diff reexecution passed; missing prerequisite is not fixed

    #26737, unchanged head c4a67d7e9d7aff24385027093bbe252ff238f951: Azure API-diff run 15462899 finished succeeded at 2026-09-27 20:55 UTC. On externally started API diff stage attempt 2, Detect API changes job 023efb3e-d10c-52da-87d2-e51a6fae706a succeeded; its Build task (log 184) and change-detection task (log 198) succeeded. Attempt 1 had failed while parallel runtime compilation read host_runtime_contract.h before the generation prerequisite ran.

    Signature/classification: source-related|runtime-Makefile-missing-host_runtime_contract.h-prerequisite|parallel-make-j4|iossimulator-x64-runtime-dotnet-coreclr.o|fatal-header-not-found - source-related missing Makefile prerequisite/build-order race, not an API comparison-baseline finding. This attempt's success is an observed reexecution outcome, not a source fix or permission for the monitor to retry source-related failures. The #26737 simulator stage is being tracked separately. Totals: 4 distinct monitored PRs, 5 confirmed failure occurrences; this is a retry outcome, not a sixth failure. No source edit or monitor retry was made, and exact all-actor retry counts remain uncertified. The missing explicit header dependency remains the diagnosis.

    Additional info to collect: The refreshed issue body requests none for the original #26737 occurrence or this retry outcome; no items are unavailable or not permitted. Diagnostic source: Azure run/attempt/job/task timeline and logs (retention limited).

  6. dalexsoto commented on Sep 28, 2026

    @dalexsoto
    dalexsoto
    on Sep 28, 2026
    MemberAuthor

    #26576: fifth distinct PR reproduces the native header prerequisite race

    #26576, new head 28427e7844e56f788f9eb6d0e645906d6077daa8, failed Azure API-diff pipeline run 15482691 / Detect API changes / Build task on stage/job/task attempt 1, task log 83, at 2026-09-28 00:44:42 UTC. The build started OBJC [iossimulator-x64] runtime-dotnet-coreclr.o, then runtime/runtime.m:28:10: fatal error: 'host_runtime_contract.h' file not found; make reported the failing .libs/iossimulator-x64/runtime-dotnet-coreclr.o target. This is the exact missing generated-header prerequisite/build-order root from this issue, not an API binding delta.

    The downstream Upload / Publish change detection results job then failed Decompress change detection artifacts (task ddab1acc-f64a-509c-ea96-e2080ffe06f6, log 123) because change-detection.zip did not exist after the native build failure. No API comparison executed and no API-diff result or binlog artifact was published, so no conclusion is drawn about either current-PR or vs-stable API baselines. Only build-configuration and agent-log artifacts are listed.

    Signature/classification: source-related|runtime-Makefile-missing-host_runtime_contract.h-prerequisite|parallel-make-j4|iossimulator-x64-runtime-dotnet-coreclr.o|fatal-header-not-found - source-related parallel Makefile prerequisite race. Totals: 6 confirmed occurrences across 5 distinct monitored PRs, including this one new PR/head/run/task record. The external branch is Rolf-authored; the monitor did not edit it or request a source-related retry. Exact all-actor retry budgets remain uncertified; the other new-head CI runs are still active. The required explicit header dependency remains the diagnosis, with no verified workaround.

    Additional info to collect: The refreshed issue body requests none for this occurrence; no checklist items are unavailable or not permitted. The absent API-diff ZIP/binlog is a build-stage consequence, not a checklist omission. Azure logs are retention-limited.

  7. dalexsoto commented on Sep 28, 2026

    @dalexsoto
    dalexsoto
    on Sep 28, 2026
    MemberAuthor

    #26576: external API-diff reexecution passed, header prerequisite unchanged

    #26576, unchanged head 28427e7844e56f788f9eb6d0e645906d6077daa8: Azure API-diff run 15482691 completed succeeded at 2026-09-28 01:35:49 UTC on an externally initiated API diff stage attempt 2. Detect API changes job 023efb3e-d10c-52da-87d2-e51a6fae706a succeeded, as did its Build task (log 184) and Detect API changes task (log 198). The original attempt-1 failure at the missing host_runtime_contract.h prerequisite, plus its downstream missing artifact, remains a confirmed historical occurrence. Attempt 1 failed before either API comparison; the successful reexecution is not an API-binding or suppression-file source fix.

    Signature/classification: source-related|runtime-Makefile-missing-host_runtime_contract.h-prerequisite|parallel-make-j4|iossimulator-x64-runtime-dotnet-coreclr.o|fatal-header-not-found - source-related parallel runtime build-order race. Totals remain six confirmed failure records across five distinct monitored PRs. This is a retry outcome, not a seventh failure or proof the missing prerequisite was repaired. The separate #26576 simulator run 15482690 is still being monitored; its new tvOS MonoTouch.Dialog file-lock failure is tracked under #26688. The monitor submitted no retry and made no Rolf-branch source edit; all-actor accepted-request counts remain uncertified.

    Additional info to collect: The refreshed issue body requests none for the original failure or this successful external attempt; no requested items are unavailable or not permitted. Azure run/task logs have limited retention.

  8. dalexsoto commented on Sep 28, 2026

    @dalexsoto
    dalexsoto
    on Sep 28, 2026
    MemberAuthor

    #26727: external API-diff attempt 2 passed; header dependency remains missing

    #26727, head ff7a8dd2bd32fb7cc16fbc2cc4659593eaf5bc0c: Azure API-diff run 15456544 completed succeeded at 2026-09-28 11:02:31 UTC. Its externally initiated API diff stage attempt 2 succeeded: Detect API changes job 023efb3e-d10c-52da-87d2-e51a6fae706a / Build task 2ee5ba36-f97c-51c0-5f40-8dfaf3c2d983 passed (task log 184), and publisher job 74162e51-23ec-5152-5f9a-eb0fad5f6eae passed. The stage finished succeeded at 11:02:31 UTC.

    Signature/classification: source-related|runtime-Makefile-missing-host_runtime_contract.h-prerequisite|parallel-make-j4|iossimulator-x64-runtime-dotnet-coreclr.o|fatal-header-not-found — the original attempt-1 runtime.m:28:10 missing-header failure is still a confirmed source-related Makefile prerequisite/build-order race. This attempt's success is an observed retry outcome, not proof that the missing dependency was repaired; the branch was not edited by this monitor. Attempt 1 stopped before either API comparison. Attempt 2's baseline-specific raw diff entries were not independently examined, so no claim is made about the informational current-PR breaking diff or the separate vs-stable comparison. No bindings or suppression files were changed. Totals remain 6 confirmed failures across 5 distinct monitored PRs; this success is not a seventh failure. No workaround is verified.

    The monitor submitted no retry. The accepted request payload/actor and exact all-actor PR-lifetime/head request counts remain uncertified; an observed stage attempt is not an accepted-request count. #26727's separate simulator reexecution remains in progress and does not establish a result for that test. Additional info to collect: the refreshed issue body requests none for this occurrence or outcome; no issue-specific checklist items are unavailable or not permitted. Azure task logs have limited retention; no Helix item applies to this direct native build task.

  9. dalexsoto commented on Sep 28, 2026

    @dalexsoto
    dalexsoto
    on Sep 28, 2026
    MemberAuthor

    #26733: same generated-header prerequisite fails before API comparison

    #26733, head a428b3062de3bb5da4477166eab048eb79659925 (rolfbjarne): initial Azure API-diff run 15462222 (definition 22268, stage/job/task attempt 1) failed. Detect API changes / Build job 276491e5-a85e-51e3-6c38-98e261218d1a / task 76afb9de-7feb-5cd3-1173-f84cb7d12f5e (log 83) failed 2026-09-25 13:46:31 UTC while compiling iossimulator-x64/runtime-dotnet-coreclr.o: runtime/runtime.m:28:10: fatal error: 'host_runtime_contract.h' file not found; make was waiting for unfinished jobs.

    The separate Upload / Publish change detection results / Decompress change detection artifacts job 443a5880-1958-5052-6859-a80de6521633 / task ddab1acc-f64a-509c-ea96-e2080ffe06f6 failed downstream (log 123) because change-detection.zip was never produced (ENOENT). Attempt 1 published only build configuration and agent logs: neither its current-PR breaking comparison nor its separate vs-stable comparison ran or produced a raw API-diff artifact. No API binding or suppression-file edit is justified by this result.

    Signature/classification: source-related|runtime-Makefile-missing-host_runtime_contract.h-prerequisite|parallel-make-j4|iossimulator-x64-runtime-dotnet-coreclr.o|fatal-header-not-found — source-related native Makefile build-order/prerequisite race, matching the same object/header/compiler error on five prior monitored PRs. The ledger now has 7 confirmed occurrences across 6 distinct monitored PRs. An externally initiated API-diff stage attempt 2 is currently rerunning Detect API changes; no comparison or fix outcome is inferred while it is active. No monitor retry or Rolf branch source edit.

    Issue-specific diagnostics: Refreshed body has no Additional info to collect section. Build task, target/header, downstream archive error, stage/job/attempt and published-artifact inventory were checked in the exact Azure run. A raw comparison artifact and Helix item are unavailable because build failed before comparison; Azure logs are retention-limited.

  10. rolfbjarne commented on Sep 28, 2026

    @rolfbjarne
    rolfbjarne
    on Sep 28, 2026
    Member

    Fixed in e9efee9.

  11. rolfbjarne
    closed this as completedon Sep 28, 2026

  12. dalexsoto commented on Sep 28, 2026

    @dalexsoto
    dalexsoto
    on Sep 28, 2026
    MemberAuthor

    #26733: same-head external API-diff retry passed before the merged fix

    #26733, unchanged head a428b3062de3bb5da4477166eab048eb79659925: the previously reported initial Azure API-diff run 15462222 failed on host_runtime_contract.h while compiling iossimulator-x64/runtime-dotnet-coreclr.o, before either API comparison. On an externally initiated API diff stage attempt 2, Detect API changes job 023efb3e-d10c-52da-87d2-e51a6fae706a / Build task 2ee5ba36-f97c-51c0-5f40-8dfaf3c2d983 (log 184) generated the header at 2026-09-28 16:54:39 UTC, before compiling the same simulator-x64 object at 16:55:43 UTC; the Build task succeeded. Both Detect API changes and the reexecuted Upload job succeeded; the stage finished 17:16:29 UTC and the parent run succeeded 17:16:32 UTC.

    The published change-detection-2 artifact was inspected by baseline: previous-api-comparison/api-diff.md (current PR/previous-commit comparison) reports no breaking changes for iOS, tvOS, Mac Catalyst and macOS. The separate stable-api-comparison/api-diff.md also reports no breaking changes on those four platforms. This confirms a same-head external reexecution success; it does not turn the attempt-1 header race into an API-binding failure, authorize suppression edits, or prove a source fix on this PR.

    The PR merged at 17:17:25 UTC, after that retry. This issue is now closed (COMPLETED) at 17:23:52 UTC with a subsequent explicit fix commit e9efee9170863145a44244477d3b33f771f73b64. The failed initial PR run and its successful retry predate that closure; they do not invalidate it. Do not reopen: the merged fix closed this issue less than two weeks ago, and no post-fix failing build is established. Original ledger totals remain 7 confirmed header failures across 6 distinct monitored PRs; this retry success is an outcome, not another occurrence.

    Issue-specific diagnostics: The refreshed issue body has no Additional info to collect section. Exact PR/head/run/stage/job/task, native header/target ordering, raw previous-vs-stable summaries, stage/parent result and closure reason/time are recorded. No Helix item applies; artifact retention is limited. No monitor retry or Rolf branch source edit was made, and accepted-request budgets remain uncertified.

  13. dalexsoto commented on Sep 28, 2026

    @dalexsoto
    dalexsoto
    on Sep 28, 2026
    MemberAuthor

    #26742: three Build-stage occurrences of the header prerequisite race

    #26742, head e3dfaa231df909142496c99a324ee543325cbc0f (rolfbjarne): Azure API-diff run 15463335 (definition 22268) queued 2026-09-25 15:46:16 UTC. Its Detect API changes Build job failed in three separate stage/job attempts on the same exact source prerequisite: runtime/runtime.m:28:10: fatal error: 'host_runtime_contract.h' file not found when compiling iossimulator-x64/runtime-dotnet-coreclr.o under parallel make -j4.

    Stage attempt Failing Azure job / Build task Build / Upload log IDs Build failure
    1 Build job 276491e5-a85e-51e3-6c38-98e261218d1a / task 76afb9de-7feb-5cd3-1173-f84cb7d12f5e 83 / 123 2026-09-25 15:59:44 UTC
    2 Build job 023efb3e-d10c-52da-87d2-e51a6fae706a / task 2ee5ba36-f97c-51c0-5f40-8dfaf3c2d983 183 / 223 2026-09-28 16:03:24 UTC
    3 Build job ebc15484-a46c-5b92-dad3-28071c5698c0 / task 3991de11-94d4-59b4-e7d5-b73f5bf78128 283 / 323 2026-09-28 18:06:39 UTC

    Every corresponding Upload / Publish change detection results failure (ENOENT missing output archive) is downstream of its Build failure. Neither comparison baseline was reached in any attempt; no raw current-PR/previous-commit comparison and no separate API diff vs stable artifact were published. This is a source-related missing generated-header prerequisite, not an API-binding change or a nonempty current-PR breaking diff. The parent run finished failed at 2026-09-28 18:10:18 UTC. The three stage attempts are three occurrences on one new PR, not three PRs and not proof of three accepted retry requests.

    Signature/classification: source-related|runtime-Makefile-missing-host_runtime_contract.h-prerequisite|parallel-make-j4|iossimulator-x64-runtime-dotnet-coreclr.o|fatal-header-not-found — source-related parallel-build Makefile race; 10 confirmed occurrences across 7 distinct monitored PRs. This issue is CLOSED/COMPLETED since 2026-09-28 17:23:52 UTC with the separately reported fix commit e9efee9170863145a44244477d3b33f771f73b64. This run was queued before that closure, and inclusion of the fix in its head is not established. Under the default-branch postmortem rule, a fix-merged issue closed less than two weeks ago is not reopened; recurrence evidence belongs on this closed canonical issue. No suppression or binding edits, monitor retries, or Rolf-branch source edits were made.

    Issue-specific diagnostics: The refreshed issue body has no Additional info to collect heading. Each PR/head/run/stage attempt/Build and Upload job/task/log was checked; raw current-PR and vs-stable API comparisons are unavailable because Build never reached either baseline, not because the baselines were equal. No Helix work item applies; logs/artifacts have limited retention. Exact all-actor accepted-request counts remain uncertified.

  14. dalexsoto commented on Sep 29, 2026

    @dalexsoto
    dalexsoto
    on Sep 29, 2026
    MemberAuthor

    #26683: header race recurs in tested merge ref containing the closed fix

    #26683, new head 61bab7b98949bc624b5b50ec0b7342bb165d8da9 (app/dotnet-maestro), queued its initial API-diff run 15497184 (definition 22268, stage/job/task attempt 1) at 2026-09-29 02:03:10 UTC. Detect API changes job 276491e5-a85e-51e3-6c38-98e261218d1a / Build task 76afb9de-7feb-5cd3-1173-f84cb7d12f5e (task log 83) failed at 02:32:37 UTC while compiling .libs/iossimulator-x64/runtime-dotnet-coreclr.o: runtime/runtime.m:28:10: fatal error: 'host_runtime_contract.h' file not found; the native make task exited 2. The subsequent Upload / Publish change detection results job 443a5880-1958-5052-6859-a80de6521633 / Decompress change detection artifacts task ddab1acc-f64a-509c-ea96-e2080ffe06f6 (log 123) failed downstream on ENOENT for the absent change-detection-1/change-detection.zip. The API stage and parent run ended failed at 02:36:31 UTC.

    Neither API-comparison baseline was reached. No raw current-PR/previous-commit comparison or separate API diff vs stable artifact was published. This is the generated-header Build failure—not evidence of a binding discrepancy, a nonempty informational beta-Xcode current-PR diff, or a reason to change tools/apidiff/suppression-files/*.xml.

    This is the exact source-related|runtime-Makefile-missing-host_runtime_contract.h-prerequisite|parallel-make-j4|iossimulator-x64-runtime-dotnet-coreclr.o|fatal-header-not-found source-related build-graph race already counted on #26683's earlier head 2fbdc937e9bf5d9e9d52ccd3631b33bf41c870b8, run 15463858. New totals: 11 confirmed occurrences across 7 distinct monitored PRs, not eight PRs.

    Fix inclusion checked against the build actually tested: the Azure sourceVersion is PR merge ref 22ec70bc175456f2a446c5856357c84c5086e75e, not the PR head. The merged fix commit e9efee9170863145a44244477d3b33f771f73b64 is an ancestor of that merge ref (ahead 29, behind 0), and its runtime/Makefile in the tested ref contains runtime.m: host_runtime_contract.h. Nonetheless the same compiler error occurred. The precise remaining native dependency path is not established by the task log; this post-fix merge-ref result suggests the rule was insufficient in this execution, not that the source fix was absent.

    The canonical issue remains CLOSED/COMPLETED since 2026-09-28 17:23:52 UTC. Do not reopen: the repository postmortem rule forbids reopening a fix-merged issue closed less than two weeks ago, even while this early post-fix evidence is recorded. No monitor retry or bot-branch source edit was made; accepted-request counts for 25/PR-lifetime and 10/head remain uncertified, and the new head's other CI runs are still active.

    Issue-specific diagnostics: The refreshed issue body has no Additional info to collect section. Exact PR/head/merge ref, run/stage/job/task/log, header/target, fix ancestry and source rule, Upload cascade and both unavailable comparison baselines were checked. The missing compiler prerequisite's precise dependency-chain cause remains unavailable; no Helix item applies to this native Azure Build. Artifacts have limited retention.

  15. dalexsoto commented on Sep 29, 2026

    @dalexsoto
    dalexsoto
    on Sep 29, 2026
    MemberAuthor

    #26742: fourth external API-diff attempt repeats the native missing-header failure

    #26742, exact head e3dfaa231df909142496c99a324ee543325cbc0f: Azure API-diff run 15463335, stage/job attempt 4, Detect API changes job fefc53de-50f4-5f83-8e4d-7c16fc51688e / Build task 5e655a77-1fbf-583e-9f42-2348d3f4c237 (task log 383) failed at 2026-09-29 06:11:26 UTC. runtime/runtime.m:28:10: fatal error: 'host_runtime_contract.h' file not found while compiling .libs/iossimulator-x64/runtime-dotnet-coreclr.o; native make exited 2. The Upload job 504297c4-393d-5e94-e457-38a255e4cfe4 / Decompress task ef44a766-b9e9-50e9-c21f-d28728e0bd7c (log 423) subsequently failed downstream at 06:14:59 UTC: ENOENT for change-detection-4/change-detection.zip. The parent run ended failed at 06:15:17 UTC.

    Neither API baseline was reached. There is no raw current-PR/commit breaking diff or separate API diff vs stable artifact in this attempt; the informational beta-Xcode current-PR comparison cannot be inferred or silenced by binding/suppression changes. The tested Azure PR merge ref e2403ebe8a1f4c78c014a51e1cbceaf257990cbb does not contain merged fix commit e9efee9170863145a44244477d3b33f771f73b64 (GitHub commit comparison: diverged, 7 commits behind that fix). This attempt is therefore not evidence that the fix regressed on a ref containing it.

    This is the exact source-related|runtime-Makefile-missing-host_runtime_contract.h-prerequisite|parallel-make-j4|iossimulator-x64-runtime-dotnet-coreclr.o|fatal-header-not-found already recorded on three earlier attempts of the same PR. Canonical totals: 12 confirmed occurrences across 7 distinct monitored PRs, not eight PRs. The externally initiated stage-4 reexecution failed; its Azure accepted-request payload and grouping with earlier attempts are unavailable, so stage attempt 4 is not asserted as four accepted retries. No monitor retry was made: this is a source-related Build failure, exact all-actor 25-PR-lifetime / 10-head request totals are uncertified, and Rolf's branch is source-read-only. There is no verified source workaround for this tested ref.

    The issue remains CLOSED/COMPLETED since 2026-09-28 17:23:52 UTC. Not reopening: the tested ref lacks the merged fix, and the repository's less-than-two-weeks rule independently forbids reopening a recently fix-merged issue. Issue-specific diagnostics: the refreshed issue body has no Additional info to collect section. Exact PR/head/tested merge ref, fix ancestry, run/attempt/job/task/logs, native compiler error, Upload cascade and artifact inventory were checked; no Helix work item applies, and the raw API comparisons are unavailable because Build stopped before them. Azure artifacts are retention-limited.

  16. dalexsoto commented on Sep 29, 2026

    @dalexsoto
    dalexsoto
    on Sep 29, 2026
    MemberAuthor

    #26739: native generated-header Build fails on both initial and external attempts

    #26739, exact head 7c71e2302de07d7c543816b0814a6e7e178e7fec (rolfbjarne): API-diff run 15463051, definition 22268, tested PR merge ref 9935e361c16e21999cebe1b48e91db89267ed2a6. On initial stage attempt 1, Detect API changes / Build task log 83 failed 2026-09-25 15:27:47 UTC: runtime/runtime.m:28:10: fatal error: 'host_runtime_contract.h' file not found compiling .libs/iossimulator-x64/runtime-dotnet-coreclr.o; make exited 2. The subsequently failed Upload job 443a5880-1958-5052-6859-a80de6521633 (Decompress log 123) reported ENOENT for change-detection-1/change-detection.zip, a downstream cascade. Azure replaced the initial Detect job/task record IDs when it started attempt 2; their names, task log and run/attempt are retained, but those original record IDs are unavailable.

    An externally initiated API-diff stage/job attempt 2 on the same merge ref again failed Detect API changes job 023efb3e-d10c-52da-87d2-e51a6fae706a / Build task 2ee5ba36-f97c-51c0-5f40-8dfaf3c2d983 (log 183) at 2026-09-29 06:51:05 UTC with the same header, object and native make error. Upload job 74162e51-23ec-5152-5f9a-eb0fad5f6eae / Decompress task 2d23ecaa-219a-52d7-f057-2a90cec312b0 (log 223) failed on the missing change-detection-2/change-detection.zip; the parent finished failed at 06:54:57 UTC. This is two task occurrences on one newly counted PR: 14 confirmed occurrences across 8 distinct monitored PRs for source-related|runtime-Makefile-missing-host_runtime_contract.h-prerequisite|parallel-make-j4|iossimulator-x64-runtime-dotnet-coreclr.o|fatal-header-not-found, not nine PRs.

    Neither attempt reached either API baseline. No raw current-PR/commit breaking diff or separate API diff vs stable was published; no binding or suppression-file edit is justified by either missing artifact. GitHub commit comparison shows the tested PR merge ref diverges from and is seven commits behind merged fix e9efee9170863145a44244477d3b33f771f73b64: these are not post-fix executions. The canonical issue is CLOSED/COMPLETED since 2026-09-28 17:23:52 UTC; do not reopen because this ref lacks the fix and the merged-fix closure is less than two weeks old.

    Issue-specific diagnostics: the refreshed issue body has no Additional info to collect section. Exact PR/head, tested merge ref/fix ancestry, Azure run/attempt/jobs/tasks/logs, compiler error, both Upload cascades and absent comparisons were checked. No native primary binlog or Helix test work item is available for these failed Build steps; Azure artifacts are retention-limited. No monitor retry: repeated source-related Build failures are ineligible, complete all-actor 25-PR-lifetime / 10-head accepted-request counts are uncertified, and the PR branch is source-read-only. The external attempt number is not an accepted-request count.

  17. dalexsoto commented on Sep 29, 2026

    @dalexsoto
    dalexsoto
    on Sep 29, 2026
    MemberAuthor

    #26742: fifth external attempt repeats the native missing-header Build failure

    #26742, exact head e3dfaa231df909142496c99a324ee543325cbc0f: externally initiated API-diff run 15463335, stage/job attempt 5, Detect API changes job e42f49e1-69be-5f93-1756-4c76ecc84f89 / Build task 04bd809a-afd6-5b92-d320-79a860a5ad74 (task log 483) failed 2026-09-29 09:45:43 UTC on runtime/runtime.m:28:10: fatal error: 'host_runtime_contract.h' file not found while compiling .libs/iossimulator-x64/runtime-dotnet-coreclr.o; native make exited 2. The Upload job 06b6cd6a-b5bb-5114-a24f-bf2a135c891b / Decompress task 6531a8cc-577d-541b-1d5d-b6e4e46b9fd9 (log 523) failed downstream at 09:49:08 UTC on ENOENT for the absent change-detection-5/change-detection.zip. The parent ended failed at 09:49:29.

    This is the exact source-related|runtime-Makefile-missing-host_runtime_contract.h-prerequisite|parallel-make-j4|iossimulator-x64-runtime-dotnet-coreclr.o|fatal-header-not-found already documented on four prior attempts of the same PR: 15 confirmed occurrences across 8 distinct monitored PRs, not nine PRs. Azure again tested PR merge ref e2403ebe8a1f4c78c014a51e1cbceaf257990cbb, which GitHub commit comparison shows diverged from and behind fix e9efee9170863145a44244477d3b33f771f73b64. This is not a post-fix failure. The canonical issue remains CLOSED/COMPLETED since 2026-09-28 17:23:52 UTC; do not reopen because this tested ref lacks the fix and the merged-fix closure is under two weeks old.

    API baselines: the native Build failed before either the expected current-PR/commit breaking diff or the distinct API diff vs stable could run. No raw comparison artifact was published; the missing Upload ZIP is a cascade, not a binding compatibility result and not grounds to change API bindings or suppression XML.

    Issue-specific diagnostics: the refreshed issue body has no Additional info to collect section. Exact PR/head/tested ref, stage/job/task attempt and logs, object/header, fix ancestry, Upload cascade and absent artifacts were checked. A raw API comparison and Helix test item are unavailable because Build failed first; artifacts are retention-limited. This was an external stage attempt: attempt number 5 does not certify five accepted Azure retry requests. No monitor retry was submitted for a source-related failure, all-actor 25-PR-lifetime / 10-head request totals remain uncertified, and Rolf's branch was not edited.

  18. dalexsoto commented on Sep 29, 2026

    @dalexsoto
    dalexsoto
    on Sep 29, 2026
    MemberAuthor

    #26739: external API-diff attempt 3 repeats the native missing-header Build failure

    #26739, exact head 7c71e2302de07d7c543816b0814a6e7e178e7fec: externally initiated API-diff run 15463051, definition 22268, stage/job attempt 3, Detect API changes job ebc15484-a46c-5b92-dad3-28071c5698c0 / Build task 3991de11-94d4-59b4-e7d5-b73f5bf78128 (log 283) failed at 2026-09-29 17:53:08 UTC: runtime/runtime.m:28:10: fatal error: 'host_runtime_contract.h' file not found while compiling .libs/iossimulator-x64/runtime-dotnet-coreclr.o. The Upload job 91f32d2c-0c96-5229-773f-8932455b7e52 / Decompress task b4a34ced-26ac-5edd-8c72-e003f5f8c95c (log 323) then failed downstream at 17:56:33 on ENOENT for absent change-detection-3/change-detection.zip. The parent finished failed at 17:56:48 UTC.

    This is the exact source-related|runtime-Makefile-missing-host_runtime_contract.h-prerequisite|parallel-make-j4|iossimulator-x64-runtime-dotnet-coreclr.o|fatal-header-not-found already recorded for this PR's attempts 1 and 2. Totals: 16 confirmed task occurrences across 8 distinct monitored PRs, not nine. The tested merge ref 9935e361c16e21999cebe1b48e91db89267ed2a6 still diverges from and is behind fix e9efee9170863145a44244477d3b33f771f73b64; this is not evidence of a post-fix failure. Canonical #26745 remains CLOSED/COMPLETED since 2026-09-28 17:23:52 UTC and is not reopened under the recent merged-fix rule.

    API baselines: the native Build failed before both the informational current-PR/commit breaking diff and the separate vs-stable comparison. Neither raw comparison was published. The Upload ENOENT is a cascade, not an API binding or suppression-file gate.

    Issue-specific diagnostics: refreshed issue body has no Additional info to collect section. Exact PR/head/merge ref, Azure attempt/job/task IDs and logs, object/header, fix ancestry, Upload cascade and artifact inventory were checked. Both raw API comparisons and a Helix test item are unavailable because Build stopped first; Azure artifacts are retention-limited. No monitor retry or source edit: this is a source-related prerequisite race on a Rolf-authored branch, and all-actor accepted-request 25-PR-lifetime / 10-head counts remain uncertified.

  19. dalexsoto commented on Sep 29, 2026

    @dalexsoto
    dalexsoto
    on Sep 29, 2026
    MemberAuthor

    #26742: external API-diff attempt 6 passed on the unchanged pre-fix ref

    #26742, exact head e3dfaa231df909142496c99a324ee543325cbc0f: externally initiated API-diff run 15463335, definition 22268, stage/job attempt 6, Detect API changes job dda58d72-662d-5314-8745-d533dd19fa83 / Build task e10942e5-772e-5395-8ec8-d3be39b50a01 (Build log 584) and Detect API changes task 64660360-34e7-5d44-ef5d-b891dc3c3f04 (log 598) succeeded at 2026-09-29 18:01:54 UTC. Upload job d849f4d6-6b13-56cd-2ba8-a6c79f22b94b succeeded by 18:05:44, and the parent completed succeeded at 18:05:51; raw artifact change-detection-6 was published.

    Root and counts: this attempt did not reproduce source-related|runtime-Makefile-missing-host_runtime_contract.h-prerequisite|parallel-make-j4|iossimulator-x64-runtime-dotnet-coreclr.o|fatal-header-not-found, which failed on the preceding five attempts of this PR. It does not add a failure occurrence: the signature remains 16 confirmed occurrences across 8 distinct monitored PRs. Azure tested the same merge ref e2403ebe8a1f4c78c014a51e1cbceaf257990cbb, which still diverges from and is behind merged fix e9efee9170863145a44244477d3b33f771f73b64. A passing reexecution of this ordering-sensitive source race is not proof of a source fix or post-fix validation. The PR still has merge conflicts; no monitor retry, conflict resolution or source edit was performed. Canonical #26745 remains CLOSED/COMPLETED since 2026-09-28 17:23:52 UTC and is not reopened.

    Separate API baselines: the raw current-PR/commit comparison against b3c1f8e1381a9b7ccab17c4399432c6ded74ca02 shows no breaking changes on iOS, tvOS, Mac Catalyst or macOS; its nonempty raw macOS text reports unnecessary suppressions CP0001 (3) and CP0002 (2). The independent vs-stable comparison also shows no breaking changes on those four platforms, but its raw macOS text reports CP0001 (3) and no CP0002. The two baselines were checked separately; these notes do not establish an API gate violation or authorize editing bindings/suppression XML.

    Issue-specific diagnostics: the refreshed issue body has no Additional info to collect section. Exact PR/head/unchanged tested ref, run/stage/job/task IDs, terminal outcome and both raw comparisons were collected. There is no Helix work item for this direct Azure Build, and the accepted external request payload/actor and exact 25-PR-lifetime / 10-head counts are unavailable from the stage attempt number. Azure logs and artifacts are retention-limited.

  20. dalexsoto commented on Sep 29, 2026

    @dalexsoto
    dalexsoto
    on Sep 29, 2026
    MemberAuthor

    #26739: external API-diff attempt 4 succeeds on the unchanged pre-fix merge ref

    #26739, head 7c71e2302de07d7c543816b0814a6e7e178e7fec: externally initiated Azure API-diff run 15463051, API diff stage attempt 4. Detect API changes job fefc53de-50f4-5f83-8e4d-7c16fc51688e / Build task 5e655a77-1fbf-583e-9f42-2348d3f4c237 (log 384) fetched host_runtime_contract.h and completed succeeded at 2026-09-29 21:37:32 UTC. Detect API changes task 14c4b2b5-8273-5274-f73e-77b723954acf (log 398), Upload job 504297c4-393d-5e94-e457-38a255e4cfe4, stage and parent all succeeded, the parent at 21:51:24 UTC. The PR merged at 21:52:04 UTC on the same head.

    The raw change-detection-4 artifact and task output have two separate comparisons: current-PR/previous commit versus b3c1f8e1381a9b7ccab17c4399432c6ded74ca02 and versus stable. Both report no breaking changes for iOS, tvOS, Mac Catalyst and macOS. Their nonempty raw macOS notes differ: current-PR has three CP0001 and two CP0002 unnecessary suppressions, whereas vs-stable has three CP0001 and zero CP0002. These notes are not a proven failing gate; no binding or suppression XML was changed.

    Signature/classification: source-related|runtime-Makefile-missing-host_runtime_contract.h-prerequisite|parallel-make-j4|iossimulator-x64-runtime-dotnet-coreclr.o|fatal-header-not-found remains 16 confirmed source-related Build-task occurrences across 8 distinct monitored PRs. Attempts 1–3 on this same PR failed the missing-header prerequisite race before a meaningful API comparison; attempt 4's pass on the unchanged tested merge ref is a retry outcome, not a source fix or proof the race is eliminated. The issue is closed/completed after a merged fix; this successful pre-fix rerun does not invalidate closure, so it is not reopened. The refreshed issue body has no Additional info to collect section. Exact PR/head, run/stage/job/task attempts, Build log and separate raw comparison artifacts were checked; no Helix item applies to this Azure Build. Artifacts are retention-limited. No monitor retry, source edit or Rolf-branch push; all-actor accepted-request 25-PR-lifetime / 10-head counts remain uncertified.

  21. dalexsoto commented on Oct 1, 2026

    @dalexsoto
    dalexsoto
    on Oct 1, 2026
    MemberAuthor

    #26727 new dependency head: generated-header Build failure before either API comparison

    #26727, new head ec1cb82226ab283817888e6cf3448796ea671d39: xamarin-macios-pr-apidiff run 15529211 / Detect API changes / Build, definition 22268, initial stage/job/task attempt 1, log 83, failed 2026-10-01 02:11:42.603823 UTC under make all -j 4 IGNORE_SIMULATORS=1: .libs/iossimulator-x64/runtime-dotnet-coreclr.o, runtime/runtime.m:28:10: fatal error: 'host_runtime_contract.h' file not found; native make exited 2. The .NET 11 SDK download fallback installed successfully, so the earlier primary download URL misses are not the cause.

    The publisher / Decompress task, log 123, then failed downstream with ENOENT for absent change-detection-1/change-detection.zip. It adds no independent root occurrence. Parent finished failed at 2026-10-01T02:15:22.330391+00:00.

    Tested revision and closing fix: Azure checkout log 44 confirms merge ref f777fc559f943c7b81f9fab78cb2084825253ed5, which GitHub comparison places 31 commits ahead / zero behind fix e9efee9170863145a44244477d3b33f771f73b64. The PR head itself diverges from that fix (2 ahead / 11 behind). The tested tree's Makefile blob 26feee7bcd938c7cda4e22827516ad628febbeb0 includes runtime.m: host_runtime_contract.h; mk/rules.mk blob b41dfec03ff631a7546ddde32877af2d30905867 still compiles object patterns from %.m and EXTRA_DEPENDENCIES, which do not directly include this generated header. The task logged no preceding header-download step. The fix-containing tested tree still reproduced the missing-header build signature; a primary native Make execution graph/binlog was not published, so a permanent correction is not claimed.

    API-baseline guardrail: neither current-PR/commit breaking-diff nor the separate vs-stable comparison was reached. No raw diff artifact was produced. This is not an informational beta-Xcode API diff, binding discrepancy, or reason to alter suppressions.

    Signature/classification: source-related|runtime-Makefile-missing-host_runtime_contract.h-prerequisite|parallel-make-j4|iossimulator-x64-runtime-dotnet-coreclr.o|fatal-header-not-found, source-related native build prerequisite/order failure. 17 confirmed occurrences across 8 distinct monitored PRs; this adds one new head/run occurrence, not a new distinct PR.

    Canonical state/reopening: issue remains CLOSED / COMPLETED, closed 2026-09-28T17:23:52Z for the recorded merged fix. Not reopening: the default-branch postmortem policy forbids fix-merged reopening within two weeks. The new fix-containing evidence is retained on this canonical issue, not a duplicate.

    Issue-specific checklist: the refreshed body has no Additional info to collect section. Exact PR/head/tested merge, run/attempt/job/task/log, compiler error, source rules, fix ancestry, publisher cascade and terminal parent were collected. Primary native binlog/Make execution graph and both raw API-diff artifacts are unavailable: not published because the upstream Build failed. No Helix work item applies. Worker/IP/credential details are omitted; artifacts are retention-limited.

    No monitor retry, source edit, commit or push. This source-related failure does not authorize a retry; all-actor accepted-request 25-lifetime / 10-head accounting remains uncertified.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    ci-postmortemCI post-mortem analysis: flaky tests, infrastructure failures, shared regressionscopilot

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions