Skip to content

Python: combined DCA validation on rebased shared-SSA baseline - #154

Draft
yoff wants to merge 18 commits into
yoff-python-rebased-ssa-baseline-dcafrom
yoff-python-rebased-combined-dca
Draft

Python: combined DCA validation on rebased shared-SSA baseline#154
yoff wants to merge 18 commits into
yoff-python-rebased-ssa-baseline-dcafrom
yoff-python-rebased-combined-dca

Conversation

@yoff

@yoff yoff commented Sep 3, 2026 •
edited
Loading

Copy link
Copy Markdown
Owner

DCA-only combined validation

This draft is DCA-only and not a merge proposal. It is layer 2 of a fresh two-PR validation stack. No DCA was launched and no native stack metadata was registered.

Immutable base

Included sources

Each selected source's unique effective patch is included exactly once, in dependency order:

  1. Python: switch dataflow library to new (shared) CFG + SSA github/codeql#21925 production flip and current fixes, base f49429a4c4fa9c1676ecb70e4ee47bb0d446823a, head f2a6889d6765c2f4b2f3e6351cee2e4b8ecd3af4: ec8c30a2df365dafad35c9bdf33e46dfc3f230a8, 432c2d3c0917b073387d81b8c3144d8af83ed815, 1a8e317b4a328bea1059453a2ab3ba6eada09e3f, feee854ca86f8a5d0321a95c99c89c499868967b, and f2a6889d6765c2f4b2f3e6351cee2e4b8ecd3af4.
  2. Python: unpin legacy CFG/ESSA from the AST cached stage (DCA measurement) github/codeql#22116 unique unpin commit 2cedc52ad950e632b0c5663cf05385616430bfa8.
  3. Python: bind captured type-tracking jumps from source github/codeql#22407 diagnostic 1cf7b0efcd32baa195c79461da3a19b6b042ddd1 and captured-jump binding rewrite d7066b766b04c8c8d1be1ac6941ec0da0d3f2c61.
  4. Python: cache exposed shared SSA adjacency relations github/codeql#22421 adjacency diagnostic 2c980d76628e10e99bc106f4f203cf7a18d11d45 and exposed cache 0933871654a8e70c6b4de8c46fd38f917755036f.
  5. Python: cache shared SSA liveness across staged consumers (DCA) github/codeql#22465 unique liveness-cache commits only: c191e3cf8414425e6e88096ec0c971c275e14b00 and head 8791147a0bc72325c35004ab741442cd7de3122d.
  6. Python: cache shared SSA definition reachability #148 unique definition-reachability commits only: e9bb6ff417c7a5574ef3f1ff84637b029c10e9ca and head fc71c20369d455a2ec5fb9969cafeabc62a03ccb.
  7. Python: cache shared CFG scope for DCA validation #149 CFG scope cache d828b0e2d6de619f1c095b2c82a28a4ee138a2a5.
  8. Cache Python shared-CFG node mapping for DCA #151 canonical CFG getNode cache 133845e41556803743fbc1f76e7fce54941f20a1.
  9. Python shared CFG: optimize explicit-step binding order #152 consumer-local explicitStep late-binding rewrite 3dfcc4f14e1e5832cd964d3828b4c5638e365d47.

Explicit exclusions

Current-main adaptations

Patch and tree verification

  • Candidate head: d75f96aca016100826c64a5e8185ea63fdae8edf
  • Candidate tree: a6db1f3911a26fda054594641afb26d37abca60a
  • Baseline-to-candidate: 18 commits, 260 files, 6,840 insertions, 6,303 deletions.
  • Stable patch IDs and git range-diff match 15 source commits byte-for-byte exactly once. The only source commits with intentionally changed patch IDs are the Python: switch dataflow library to new (shared) CFG + SSA github/codeql#21925 flip and Python shared CFG: optimize explicit-step binding order #152 rewrite described above; the final generated-expectation adaptation is isolated in its own commit.
  • The candidate file set exactly equals the union of selected source-commit file sets; there are no unrelated paths.
  • Every candidate commit retains source authorship and a Copilot co-author trailer.

Validation

  • codeql query format --check-only passed for every changed .ql and .qll file.
  • 111 targeted tests pass across 55 directories, covering:
    • shared-CFG exception reachability and evaluation order;
    • dominance, store/load, true/false outcomes, and CFG/SSA integration;
    • SsaTest and legacy CmpTest comparison;
    • the exposed AdjacentUses contract;
    • captured-call-target diagnostics and captured-jump/type-tracking behavior;
    • liveness and definition-reachability cache consumers;
    • CFG scope and canonical node mapping;
    • the explicit-step rewrite;
    • the complete directly affected Python dataflow library-test subtree;
    • modification-of-default-value and direct guard specialization.

Exact shared-CFG relation evidence

A read-only replay reused the preserved deterministic relation harness and exact databases for Airflow a9da0f7fb48dc7526b2745be3e8fe64e1c775da2, Nova 8f3976d4cc5390fe649f2ff94afc971c5d6f7bc0, and Salt d036b1177efeec175164571e9cc07b52cddf7844.

Compared with the prior aggregate/effective github#22380 patch, merged github#22380 changes the exact Python shared-CFG relations:

Corpus Reachable nodes old -> merged Successor edges old -> merged
Airflow 4,045,061 -> 4,045,051 (-10) 4,134,749 -> 4,129,744 (-5,005)
Nova 3,363,505 -> 3,363,505 (0) 3,410,579 -> 3,409,506 (-1,073)
Salt 5,990,271 -> 5,990,113 (-158) 6,284,457 -> 6,268,978 (-15,479)

Every old-versus-merged relation hash changes, including Nova reachable nodes at unchanged cardinality. A second replay shows the final candidate has exactly the merged control's node/edge counts and hashes on all three corpora, confirming the selected caching and explicit-step changes preserve these relations.

yoff and others added 18 commits September 2, 2026 21:10
@yoff
Python: switch dataflow library to new (shared) CFG + SSA
afb6a99 
Flips the Python dataflow trunk from the legacy CFG (semmle/python/Flow.qll)
and legacy ESSA SSA (semmle/python/essa/*) to the new shared CFG facade
(semmle.python.controlflow.internal.Cfg) and the new SSA adapter
(semmle.python.dataflow.new.internal.SsaImpl), both introduced
additively in the preceding PRs in this stack.

This is the trunk-flip equivalent of the original draft PR github#21894 (kept
around as documentation), rebased on top of the four preparatory PRs:

  P1: Remove AstNode.getAFlowNode() and rewrite callers (github#21919).
  P2: Qualify Flow.qll's AST references with Py:: prefix (github#21920).
  P3: Add new shared-CFG-backed control flow graph (github#21921).
  P4: Add new shared-SSA-backed SSA adapter (github#21923).

The Python dataflow library (semmle/python/dataflow/new/) now imports
the new CFG facade and SSA adapter. All CFG-typed predicates
(ControlFlowNode, CallNode, BasicBlock, NameNode, AttrNode, ...) are
qualified with the Cfg:: prefix; SSA references switch from
EssaVariable/EssaDefinition to SsaImpl::Definition/SourceVariable.

GuardNode is redesigned to use the new CFG's outcome-node model
(isAfterTrue / isAfterFalse) instead of the legacy ConditionBlock +
flipped indirection. Only BarrierGuard<...> is preserved as public
API.

Framework files (Bottle, FastApi, Django, Tornado, Pyramid, Stdlib,
...) are updated to take CFG nodes from the new facade.

A handful of dataflow consistency tweaks for the new CFG:
- Augmented-assignment targets are treated as both load and store.
- 'from X import *' produces uncertain SSA writes for unknown names.
- CFG nodes are canonicalised so dataflow does not see equivalent
  pre/post-order pairs as distinct nodes.

Two AST tweaks for the new CFG:
- AstNodeImpl: omit PEP 695 type-parameter names from
  FunctionDefExpr / ClassDefExpr children.
- ImportResolution: drop the legacy essa import.

Test churn (~175 files): reblessed library- and query-test .expected
files reflect slightly different CFG granularity, different toString
output, and a handful of true alert deltas in security queries.

Verification: all 367 lib + src + consistency-queries compile clean.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@yoff
Python: preserve bindingset on new-CFG node dominance wrappers
99da2dd 
The `Cfg::ControlFlowNode` facade re-exports the shared CFG library's
`dominates`/`strictlyDominates` predicates, which are declared
`bindingset[this, that]` + `pragma[inline_late]` and are meant to be used
as bound-pair membership checks. The facade wrappers dropped these
annotations (using plain `pragma[inline]`), so even though the only
callers — the `with` / `async with` taint steps in DataFlowPrivate.qll
and TaintTrackingPrivate.qll — bind both endpoints, the optimizer was
free to materialise `Cfg::ControlFlowNode.strictlyDominates/1` as a full
O(nodes^2) relation over the (larger) shared-CFG node set.

On some projects this dominated analysis time entirely (DCA showed e.g.
ICTU/quality-time and biosimulations regressing ~75-160x). Restoring
`bindingset[this, other]` + `pragma[inline_late]` on the wrappers turns
the predicate back into a bound-pair check and is result-preserving (only
binding annotations change, the predicate body is unchanged).

Reproduced on ICTU/quality-time: full python-security-extended suite went
from stalling >20min on `strictlyDominates` to completing in ~6min; all
ControlFlow and dataflow/coverage library tests pass.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@yoff
Python: fix shared CFG facade CI checks
62b45bb 
Document the public expression adapter and apply the canonical QL annotation ordering required by the formatter.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Copilot-Session: 529363f5-bc7d-4f0b-9f47-e03ba9aa0cdf
@yoff
Python: test direct guard specialization semantics
00f24b8 
Prove that the query-shaped shared-SSA relation for direct truthiness guards is equivalent to the generic BarrierGuard abstraction on the modification-of-default-value regression corpus. This guards the performance specialization against semantic drift before changing production code.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Copilot-Session: 07c775e7-cd7c-4e1c-8d97-5194ffd43e1a
@yoff
Python: specialize direct truthiness guards
7bf61bc 
Keep the modification-of-default-value query on shared SSA while expressing its two direct truthiness checks in a query-shaped predicate. The generic BarrierGuard abstraction causes the evaluator to materialize and rescan a 1,579,772,664-row def-use pair relation before applying branch control. Binding both concrete NameNode uses in one predicate lets the optimizer fuse the same joins with controlsBlock and persist only the 4,992 guarded uses.

On FreeCAD@0def330, three prewarmed evaluator runs improve from 309.159-329.621s to 77.781-81.349s with byte-identical query results. A direct symmetric-difference evaluation returns zero rows, and the CommandInjection path-query control retains identical results, work, and plan hashes.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Copilot-Session: 07c775e7-cd7c-4e1c-8d97-5194ffd43e1a
@yoff
Python: unpin legacy CFG/ESSA from the AST cached stage
df7127a 
The legacy CFG (`Flow.qll`) and legacy ESSA (`Essa`/`SsaCompute`/
`SsaDefinitions`) were pinned into the always-on `Stages::AST` cached stage
via `Stages::AST::ref()` and the matching `backref()` disjuncts. Because a
cached stage is materialized as a unit once any of its predicates is demanded
(and every query demands e.g. `Expr.toString()`), this forced the legacy
CFG/ESSA to be computed for *every* query -- including the security/dataflow
queries, which after the shared-CFG dataflow flip no longer depend on the
legacy CFG at all.

Since `Stages::AST::ref()` is `1 = 1`, removing it is result-preserving; it
only changes stage scheduling. After this change the legacy CFG/ESSA is no
longer materialised for queries that do not genuinely reference it. Verified
on the full `python-security-extended` suite and on django: legacy CFG/ESSA
families materialised drop from ~165 to 0 with byte-identical results.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@yoff
Add captured call target diagnostic
e271e3c 
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
@yoff
Bind captured type-tracking jumps from their source
393408e 
The previous capturedJumpStep shape introduced an independent
Cfg::DefinitionNode and related it to the captured variable before nodeTo
bound the relevant scope-entry definition. On substantial databases, the
evaluator chose a plan that materialized a high-duplication store/variable
join before applying the target entry and source-node constraints.

Bind the scope-entry definition from nodeTo first, derive its source variable,
and then match nodeFrom's DefinitionNode directly to that variable's store.
This is relation-equivalent existential elimination: the old
nodeFrom.asCfgNode() = def and def.getNode() = store constraints become
nodeFrom.asCfgNode().(Cfg::DefinitionNode).getNode() = store, preserving the
DefinitionNode type restriction and the unchanged enclosing-scope condition.

On Airflow a9da0f7 with CodeQL 2.26.2, against exact github#21925 head
1a8e317:

* The call-target diagnostic retains the identical 59,732-edge set while
  tuples joined fall from 1,032,403,207 to 66,969,953 (-93.5%), maximum
  duplication falls from 1,473,981 to 4,053, and evaluator wall time falls
  from 59.9s to 6.2s.
* py/clear-text-logging-sensitive-data retains every result tuple (130 alerts,
  256,708 path edges, 102,385 path nodes, and 116,976 subpaths) while tuples
  joined fall from 1,329,594,809 to 242,111,554 (-81.8%) and evaluator wall
  time falls from 94s to 14.4s.

The preceding commit is intentionally a semantic call-target invariant
diagnostic that passes on the unoptimized relation and after this rewrite. An
inline MISSING/SPURIOUS red-state would assert an artificial semantic delta;
this optimization must preserve every valid captured call target.

These measurements cover one exact substantial Airflow database and two query
shapes. No DCA was run, the fix does not reduce legitimate call-graph or path
growth, and broader fleet performance remains to be confirmed separately.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 5dfda5f5-08c8-481b-9ecb-299018701497
@yoff
Python: test exposed shared SSA adjacency relations
391330e 
Exercise the three public AdjacentUses relations and compare them with their internal projection or recursive expansion contracts. The cache-placement defect changes evaluator specialization and work rather than semantic results, so a semantic contract snapshot is the stable red-state equivalent; wall-time assertions would be machine-dependent and flaky.

This intentionally records no MISSING or SPURIOUS rows: the expected invariant is exact relation equality before and after cache placement changes.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Copilot-Session: 21ab8585-861f-42c9-a834-451604646c6b
@yoff
Python: cache exposed shared SSA adjacency relations
7bef6d2 
The shared SSA module requires language adapters to cache predicates that they expose. The Python adapter exposed firstUse, adjacentUseUse, and useOfDef without restoring that cache boundary, unlike the legacy AdjacentUses implementation.

On exact historical Salt, the missing boundary caused the same 6,313,793-row liveAtExit fixed point to be evaluated twice. The equivalent plans received distinct RA hashes (c6bc8xgji0uv6seurbhesjqd315 versus fabf1xs3jb6t67a2buq2iv8iof4 for unsafe deserialization, and c6bc8xgji0uv6seurbhesjqd315 versus 8270excv27ldlfrtk19ou81d206 for modification-of-default-value) because one inherited an unrelated cached-empty sentinel while the other used a literal empty base.

Cache the three Python adapter relations rather than generic liveness. This restores the documented shared-SSA contract at the narrow language boundary and avoids imposing a 6.31M-row generic cache on every language instantiation.

On current head 1a8e317 with exact saltstack/salt@d036b117, three matched prewarmed -j1 repeats reduced median evaluator time from 51.294s to 45.103s for unsafe deserialization and from 42.377s to 34.238s for modification-of-default-value. Median paired reductions were 6.428s and 8.247s. Joined tuples fell by 58,255,670 and 85,664,313; recursive pipeline runs fell by 1,999 and 3,015. Both queries retained the identical empty endpoint hash 2a514e093aae140a14f6bf77beebe1ad in every repeat.

Historical exact controls also retained 483,922 definitions, 169,921 phi inputs, 390,548 first uses, 475,226 adjacent uses, and 123,231 semantic call edges with zero left-only or right-only rows. Historical Salt evaluator recovery was 16.5% and 19.9%. Cold prewarm evaluator time was neutral (106.609s to 106.620s), so this is a warm-query optimization rather than a claimed cold-cache speedup.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Copilot-Session: 21ab8585-861f-42c9-a834-451604646c6b
@yoff
SSA: add opt-in cached liveness construction
f58c3ca 
Allow language adapters that expose one SSA instantiation through multiple
cached API stages to persist the complete liveness fixed point once. Keep
the existing demand-specialized factory as the default.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 07c775e7-cd7c-4e1c-8d97-5194ffd43e1a
@yoff
Python: cache shared SSA liveness across stages
f711417 
Use the opt-in shared SSA factory so Python reuses the same complete
liveness relation across its staged public SSA and data-flow consumers.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 07c775e7-cd7c-4e1c-8d97-5194ffd43e1a
@yoff
SSA: add opt-in definition reachability caching
8dc8995 
Allow language-specific SSA instantiations that cross multiple cached API stages to cache the complete definition-rank and end-of-block reachability fixed points. Keep the default factory demand-specialized so unrelated instantiations retain their existing evaluator behavior.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Copilot-Session: 07c775e7-cd7c-4e1c-8d97-5194ffd43e1a
@yoff
Python: cache shared SSA definition reachability
c9a90f9 
Select the opt-in cached definition-reachability factory for Python main SSA. The independent capture-SSA instantiation continues to use the default uncached factory.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Copilot-Session: 07c775e7-cd7c-4e1c-8d97-5194ffd43e1a
@yoff
Python: cache shared CFG scope
8dff23d 
Cache the Python-local shared-CFG scope relation after semantic inputs and before staged SSA and dataflow consumers.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Copilot-Session: 07c775e7-cd7c-4e1c-8d97-5194ffd43e1a
@yoff
Cache Python shared CFG node mapping
342ed91 
The canonical injects-to-Python-AST mapping is evaluated repeatedly across DCA prewarm and target stages. Cache getNode so this truthful mapping forms a reusable evaluation boundary without changing its semantics.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Copilot-Session: 857236eb-3350-48ba-9bf1-6bf5a387191b
@yoff
Bind explicit CFG successors before expansion
8a34a9e 
Late-inline the explicit-step after-value wrapper so AST and successor demand are bound before expanding control-flow nodes, without changing CFG semantics.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
@yoff
Python: rebless implicit import SSA definitions
d75f96a 
Regenerate the typetracking import diagnostic expectation after rebasing the production flip onto the current-main shared-SSA baseline. The two exit-use definitions are now labeled as implicit; query behavior is otherwise unchanged.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
@github-actions github-actions Bot added documentation Improvements or additions to documentation Python labels Sep 3, 2026
@yoff yoff mentioned this pull request Sep 3, 2026
Draft
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

documentation Improvements or additions to documentation Python

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@yoff