September release candidate (DO NOT MERGE) - #1112
Draft
ChristianPavilonis wants to merge 405 commits into
Draft
Conversation
Resolve #744 (legacy entry-point removal + streaming publisher) against the server-side auction: keep the buffered auction path on all adapters, supersede the streaming publisher for publisher navigation. Verified: fastly/core/spin/ axum/cloudflare tests + clippy + fmt clean.
- Escape page-controlled slot fields and validate the gampad network id before splicing scraped values into trusted-server.toml - Add navigation and teardown timeouts to the verify browser collector - Snapshot evidence before the scroll pass so load-time entries keep phase initial_load; add a Chrome-gated regression fixture - Cap collector evidence lists in the injected script and after decode - Preserve CRLF line endings and render non-finite floor_price as valid TOML when updating configs in place - Cover all 128 gate combinations in the core ad-stack mirror test - Extract slot TOML rendering/merging/splicing into slot_toml.rs
…nto feature/ts-cli-ad-templates
- Recognize [creative_opportunities] table headers carrying inline comments in the in-place splice and replace_key_in_section, so a valid operator config is updated instead of gaining a duplicate section - Default generated page_patterns from the recorded post-redirect final URL instead of the requested URL, falling back to the requested URL when the recorded final URL is invalid - Escape DEL (U+007F) in toml_string, which TOML basic strings reject alongside chars below U+0020 Each fix carries a parse-backed regression test.
Main squash-merged the same server-side ad-templates feature (#680), then added Tinybird auction telemetry (#818), edgezero v0.0.4 pins (#862), and external first-party Prebid bundle loading (#743). Resolutions: - Files where this branch matched the impl branch tip take main's version (telemetry/prebid-bundle additions on top of identical feature code): orchestrator, endpoints, openrtb, prebid integration, fastly app, .gitignore, prebid docs - creative_opportunities.rs keeps this branch's version (adds the shared ad-stack gate and pattern machinery used by the ts CLI; main's copy was identical minus those additions) - publisher.rs takes main's version with this branch's should_run_server_side_ad_stack delegation to evaluate_ad_stack_gate re-applied - run.rs keeps both sides' CLI tests (ad-templates/audit + prebid bundle)
Harden the request-scoped EC KV snapshot work (#851) with the test coverage the spec's test strategy called for, plus small cleanups the plan requested. Tests: - ec::kv: snapshot upsert (write-without-read, unchanged-preserves-gen, refresh-once, CAS re-merge, tombstone-rejects, store-fail) and conditional tombstone (CAS conflict, store failure, disappear-on-retry) - ec::mod: generate_if_needed collision retry + exhaustion; default and read-path recovery-ineligibility (non-Fastly adapter contract) - ec::prebid_eids: collect_eid_cookie_updates merge and empty-registry - ec::finalize: NotRead/Failed/tombstone/subresource no-rotate paths and two-ID existing-only withdrawal - ec::pull_sync: request-wide aggregation into one bulk write across concurrency batches, plus no-dispatch for non-present snapshots - publisher: concurrent-vs-eager origin scheduling order and origin-start failure, using recording HTTP/KV collaborators Cleanups: - Extract rewrite_origin_request to remove the duplicated origin-rewrite logic across the concurrent and eager paths - Bind the orphaned EC ID once in recover_orphaned_ec Docs: - Note the fastly-ssl vendor-header layering wart in core scheme detection and the origin-forwarding strip (comments only; behavior unchanged)
A Failed request-scoped snapshot no longer short-circuits tombstone_existing_from_snapshot. A transient read error earlier in the request must not silently drop a consent withdrawal, so a non-authoritative snapshot is re-read (bounded by MAX_CAS_RETRIES) and the row is tombstoned when present. An authoritative Missing snapshot stays a no-op.
Address PR review findings: - Gate orphan-recovery eligibility to the publisher fallback after a successful origin start. Named routes, integration proxies, and request-filter short circuits no longer reach EC finalization with recovery authorized, so a blocked or non-publisher response cannot rotate an identity. - Never downgrade an in-request Add-confirmed Present snapshot on a preload refresh miss, and confirm an authoritative miss with a second read (after the origin round trip) before rotating. A single eventually-consistent edge miss can no longer rotate a valid identity; a now-visible row is adopted instead. - Preload the origin-overlapped snapshot with the unfiltered active EC ID while keeping the consent-filtered ID for auction identity, so consent-withdrawn navigations keep the withdrawal CAS off the post-origin latency path. - Resolve the initial usable snapshot outside the CAS retry counter in both partner upsert and conditional tombstoning, so a generation-unavailable or refreshed snapshot keeps all five write attempts. - Defer /auction and page-bids identity-graph reads until a live auction actually runs with a partner registry, avoiding billable KV reads that cannot be consumed. Add tests for four-conflicts-then-fifth-write CAS, transient Add->Missing->Present confirmation, and recovery-eligibility lifecycle across named routes, filter short circuits, and origin-start failures.
Ad-heavy publisher pages (video players, continuous ad refresh, anti-bot scripts) may never fire the `load` event, so `page.goto` would block until the navigation timeout and the audit failed before scraping any slots. Article pages consistently timed out this way while lighter listing pages succeeded. Navigate without hard-failing on the load wait: a load-wait or main-document-response timeout is downgraded to a "results may be partial" warning, and the existing settle loop is the real readiness signal. The settle loop now also accepts `interactive` readyState, since these pages define their GPT slots before (or without ever) reaching `complete`. Load wait is bounded separately at 12s and the settle cap is raised to 12s so lazily-defined slots are captured.
`render_slots` prepends a `# Slots managed by ...` header, but the in-place splice preserved the previous copy in the scalar block and inserted a fresh one, so each `ts audit ad-templates generate` run against an already-managed config appended another duplicate comment block. Extract the two header lines to constants and strip any prior copy (and the blank lines it leaves) from the preserved head before re-inserting the rendered slots, so repeated runs keep exactly one header. Add a regression test that splices three times and asserts a single header.
…o feature/ts-cli-ad-templates
CI's rustfmt wraps the single method-chain argument of this assert! onto its own lines; the compact form the merge brought in passed locally but failed the format gate. Match CI's canonical form.
…apshot-ec-ttl # Conflicts: # crates/trusted-server-adapter-fastly/src/app.rs # crates/trusted-server-core/src/auction/endpoints.rs # crates/trusted-server-core/src/ec/finalize.rs # crates/trusted-server-core/src/ec/prebid_eids.rs # crates/trusted-server-core/src/ec/pull_sync.rs # crates/trusted-server-core/src/http_util.rs
The CLI still called `resolved_gam_unit_path`, which core replaced with the
path-aware `render_gam_unit_path` when `{section}` templating landed, so the
crate no longer compiled. Both call sites now derive the section through
`CreativeOpportunitiesConfig::section_for_path` and render the template, and
`ExpectedSlot`/`ConfiguredJson` carry an optional unit path so an over-limit
dynamic render is reported rather than silently matched against the wrong unit.
Also resolves the outstanding review findings on these paths:
- Write the operator config through a same-directory temp file, fsync, and
rename, so a failed write cannot truncate `trusted-server.toml`.
- Validate TLS certificates in both audit browser sessions; opting out now
requires `--danger-accept-invalid-certs`.
- Refuse a redirect that leaves the requested origin during verify unless
`--allow-cross-origin-redirect` is passed, so another origin's evidence
cannot satisfy `--strict`.
- Reject page patterns the runtime cannot compile before they reach the file,
through a new shared `compile_page_pattern` in core.
- Reject `creative_opportunities` declared in a form the line-based splice
cannot edit, instead of appending a duplicate table.
- Drop non-integer GPT sizes in the collector so one fluid size cannot fail
deserialization of the whole evidence payload.
- Escape control characters in page-controlled text written to the terminal.
`ts audit ad-templates generate` derived everything it wrote from a live,
page-controlled ad stack and never checked the result, so several reachable
inputs produced a config that cannot load. An unloadable `trusted-server.toml`
is not a degraded ad stack: `build_state` fails and the adapter answers every
route from the startup error router, so the whole site returns 500 once pushed.
Add a write-side gate that runs the candidate through `Settings::from_toml`,
the same `finalize_deserialized` chain the runtime uses at startup. It runs on
the `--dry-run` path too, so a clean preview is now evidence the config loads.
When the target config was already unloadable before the run, the gate reports
that as a warning instead of blaming this run, so a freshly bootstrapped file
carrying placeholder secrets can still be updated.
Close the three reachable paths at their source as well:
- Skip a scraped slot whose ad-unit path contains `{` or `}`. The path is a
template and there is no escape syntax, so a literal brace either fails
config load or is silently reinterpreted as a placeholder.
- Skip a slot whose div id normalizes to nothing (a wholly ephemeral id such
as a React SSR marker). An empty `div_id` fails config load, and as a runtime
prefix it would bind the slot to the first id-bearing element on the page.
- Refuse to create a `[creative_opportunities]` section with no GAM network id
rather than writing one that omits the required key. This is reachable
because the network id is only recovered from an all-digit leading segment,
which an MCM child-network path does not have.
Groundwork for discovering ad slots across a site's sections rather than from a single page. Nothing calls this yet; `run_update_slots` is unchanged. `AuditCollector` gains a defaulted `collect_pages` that streams each page to a sink, so every existing implementor keeps working and the caller can fold a page into its evidence and drop the DOM immediately instead of holding every serialization at once. The browser collector overrides it to launch Chrome once for the whole crawl: a cold start plus a fresh profile dominates the cost of a multi-page run, and the shared profile carries a bot-protection clearance cookie earned on the first page across the rest of the walk. Page discovery reads the hydrated DOM rather than the served markup, because an app-router page keeps its link graph in the framework payload — parsing raw HTML finds only a fraction of a site's sections. Sitemaps are fetched from inside the open page via `fetch` plus `DOMParser`, which inherits the session's cookies and Chrome's TLS fingerprint, gets transparent gzip and XML parsing, and so needs no new Rust dependency. `crawl_plan` turns links and sitemap entries into a bounded page set: one landing page and one article per section, ranked by whether navigation and the sitemap corroborate each other, capped by section and page budgets. Sections dropped for budget are reported rather than silently omitted. Same-origin is enforced on links and on sitemap entries alike, since a `Sitemap:` directive can name any host and the crawl carries operator cookies.
The Cloudflare integration harness writes wrangler.integration.generated.toml at test time; it was swept into the previous commit by accident. Ignore it so local CI=1 runs cannot commit it again.
# Conflicts: # crates/trusted-server-js/lib/test/integrations/prebid/index.test.ts
# Conflicts: # crates/trusted-server-js/lib/src/integrations/gpt/index.ts # crates/trusted-server-js/lib/test/integrations/gpt/ad_init.test.ts
# Conflicts: # crates/trusted-server-core/src/publisher.rs
A static configuration choice should not emit a warning during every request-time settings load. Removing the log keeps runtime output focused without changing validation or creative processing behavior. Resolves: #1089
# Conflicts: # crates/trusted-server-adapter-fastly/src/app.rs
# Conflicts: # crates/trusted-server-adapter-fastly/src/app.rs # crates/trusted-server-core/src/config.rs # scripts/template-cache-local-test.sh # trusted-server.example.toml
# Conflicts: # docs/superpowers/plans/2026-08-26-auction-timeline-offsets.md
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
September release candidate
This PR is an integration branch for release-candidate validation. Do not merge it into
main.Already included through main:
Included PRs