feat(detect): enumerate large trees via git, repo, and submodules - #3287
feat(detect): enumerate large trees via git, repo, and submodules#3287albertbu wants to merge 2 commits into
Conversation
Use git ls-files, Google repo manifests, and nested worktrees so ignored trees are pruned at directory granularity instead of a Python walk. Co-authored-by: Cursor <cursoragent@cursor.com>
![graphify-labs[bot]](https://avatars.githubusercontent.com/in/4356270?s=60&v=4){kind=small}
There was a problem hiding this comment.
Graphify reviewed this change.
Worth a look — the grounded gate found no coupling regressions or blocking issues, but 5 advisory finding(s) below merit a look before merge.
Formal verification. No changes could be formally verified in this run.
Graphify review — findings
Adds Git-aware enumeration to detect() so large trees are listed via git ls-files (--cached plus --others --exclude-standard), Google repo project.list worktrees, and nested git submodules/worktrees rather than walked, keeping gitignored and submodule-internal paths out of the scan. Prunes Buildroot output//output_<board> sysroots and .repo metadata via _is_buildroot_output and expanded _SKIP_DIRS, and drops non-classifiable names under --code-only through _maybe_code_filename. Falls back to the existing os.walk when git ls-files fails or times out (300s), parallelises word counting and regular-file checks via the thread pool, and emits rate-limited --verbose heartbeats during long walks.
Worth a look
- Symlink outside root now scanned when not a symlink but resolves outside root —
graphify/detect.py· Escalate · high- agreed by 2 of 2 members but NOT verified (no proof, no reproducing execution) — consensus is not a verdict; needs human review
- Files reached through an out-of-root symlinked directory are no longer rejected —
graphify/detect.py· Escalate · high- agreed by 2 of 2 members but NOT verified (no proof, no reproducing execution) — consensus is not a verdict; needs human review
- Memory scan follows out-of-root symlink directories —
graphify/detect.py:2391· Escalate · high- agreed by 2 of 2 members but NOT verified (no proof, no reproducing execution) — consensus is not a verdict; needs human review
- Files under out-of-root symlinked directories can be admitted —
graphify/detect.py:2562· Escalate · high- agreed by 2 of 2 members but NOT verified (no proof, no reproducing execution) — consensus is not a verdict; needs human review
- Symlink files with targets outside root are no longer skipped for non-symlinks; but non-symlink out-of-root resolution check dropped —
graphify/detect.py· Escalate · medium- agreed by 2 of 2 members but NOT verified (no proof, no reproducing execution) — consensus is not a verdict; needs human review
Analysis details — impact, health, verification
Impact & health
Graphify review
Impact — 2140 functions depend on the 807 functions this change touches.
Health — this change adds coupling hotspots:
- new:
extract()— 520 callers, 43 callees - new:
_rebuild_code()— 113 callers, 50 callees - new:
detect()— 116 callers, 17 callees - new:
save_manifest()— 40 callers, 11 callees - new:
extract_files_direct()— 17 callers, 20 callees - new:
extract_xaml()— 19 callers, 17 callees - new:
extract_corpus_parallel()— 26 callers, 11 callees - new:
dispatch_command()— 2 callers, 123 callees - …and 33 more — each is listed as a finding
Verification — 2140 functions in the blast radius were not formally verified this run (proofs are advisory here).
Gate & verification
graphify gate
PASS — objectively clean (no health regressions, tests not run — proofs not run this pass (advisory)). Grounded, not self-assessed.
Advisory (not blocking):
- verification_scope: 1165 function(s) in the blast radius were not formally verified this run
Formal verification
Could not verify: Could not verify detect.
The verifier did not have enough to check detect, so it is saying so rather than guessing. No false assurance is the whole point.
Guarantee: No guarantee either way, this is an honest abstention, not a pass.
Note: Reason: parameter `root` is annotated `Path` — outside the synthesizable primitive/collection set
Could not verify: Could not verify detect\_incremental.
The verifier did not have enough to check detect\_incremental, so it is saying so rather than guessing. No false assurance is the whole point.
Guarantee: No guarantee either way, this is an honest abstention, not a pass.
Note: Reason: parameter `root` is annotated `Path` — outside the synthesizable primitive/collection set
Could not verify: Could not verify \_git\_tracked\_path\_keys.
The verifier did not have enough to check \_git\_tracked\_path\_keys, so it is saying so rather than guessing. No false assurance is the whole point.
Guarantee: No guarantee either way, this is an honest abstention, not a pass.
Note: Reason: parameter `root` is annotated `Path` — outside the synthesizable primitive/collection set
Could not verify: Could not verify \_is\_noise\_dir.
The verifier did not have enough to check \_is\_noise\_dir, so it is saying so rather than guessing. No false assurance is the whole point.
Guarantee: No guarantee either way, this is an honest abstention, not a pass.
Note: Reason: parameter `parent` is annotated `'Path | None'` — outside the synthesizable primitive/collection set
· 6 grounded finding(s) anchored inline below; 35 more finding(s) on lines outside this diff (see the check run).
| return name[dot:].lower() in _CORPUS_EXT_LOWER | ||
|
|
||
|
|
||
| def _is_noise_dir(part: str, parent: "Path | None" = None) -> bool: |
There was a problem hiding this comment.
_is_noise_dir()
10 callers depend on it (afferent coupling).
Grounded coupling-delta finding (deterministic), not an LLM guess.
There was a problem hiding this comment.
Health flags are expected: git enumeration reuses _is_noise_dir so walk and ls-files prune the same trees. detect()’s 116 callers are pre-existing. Not splitting those helpers in this PR.
| return _path_is_under(worktree, root) or _path_is_under(root, worktree) | ||
|
|
||
|
|
||
| def _git_enumerate_named_worktrees( |
There was a problem hiding this comment.
_git_enumerate_named_worktrees()
fans out to 7 callees (efferent coupling).
Grounded coupling-delta finding (deterministic), not an LLM guess.
| ) | ||
|
|
||
|
|
||
| def _git_enumerate_files( |
There was a problem hiding this comment.
_git_enumerate_files()
fans out to 8 callees (efferent coupling).
Grounded coupling-delta finding (deterministic), not an LLM guess.
|
|
||
|
|
||
| def detect(root: Path, *, follow_symlinks: bool | None = None, google_workspace: bool | None = None, extra_excludes: list[str] | None = None, cache_root: Path | None = None, gitignore: bool = True) -> dict: | ||
| def detect(root: Path, *, follow_symlinks: bool | None = None, google_workspace: bool | None = None, extra_excludes: list[str] | None = None, cache_root: Path | None = None, gitignore: bool = True, code_only: bool = False) -> dict: |
There was a problem hiding this comment.
detect()
fans out to 17 callees (efferent coupling); 116 callers depend on it (afferent coupling).
Grounded coupling-delta finding (deterministic), not an LLM guess.
| vprint(f"scanning {root} ...") | ||
| heartbeat = _Heartbeat("scanning") | ||
|
|
||
| def _walk_from( |
There was a problem hiding this comment.
_walk_from()
fans out to 8 callees (efferent coupling).
Grounded coupling-delta finding (deterministic), not an LLM guess.
|
|
||
| for p in all_files: | ||
| # For memory dir files, skip hidden/noise filtering | ||
| def _admit(p: Path) -> tuple: |
There was a problem hiding this comment.
_admit()
fans out to 8 callees (efferent coupling).
Grounded coupling-delta finding (deterministic), not an LLM guess.
Gating the out-of-root check on islink() admitted regular files reached by following a symlink directory, including graphify-out/memory/ which skips ignore pruning. Restore the v8 resolve()-under-root admit guard and prune those dirs during the memory walk too. Co-authored-by: Cursor <cursoragent@cursor.com>
|
Out-of-root resolve() guard is restored |
There was a problem hiding this comment.
Graphify reviewed this change.
Worth a look — the grounded gate found no coupling regressions or blocking issues, but 5 advisory finding(s) below merit a look before merge.
Formal verification. No changes could be formally verified in this run.
Graphify review — findings
Adds a git-aware enumeration path to detect() that lists large trees via git ls-files (cached + untracked, excluding standard ignores), Google repo .repo/project.list project sets, and nested git worktrees/submodules, so gitignored and submodule directories are enumerated from their own indexes instead of being walked. Prunes Buildroot output//output_<board>/ sysroots (detected by host/target/images/staging layout) and .repo metadata, drops non-classifiable filenames under --code-only, and parallelizes word counting via a thread pool with a serial fallback. Emits rate-limited stderr heartbeats during long walks under --verbose, with a 300s git ls-files timeout that falls back to the Python os.walk scan when Git is unavailable or times out.
Worth a look
- memory_dir files may be lost when git enumeration replaces all_files —
graphify/detect.py· Escalate · high- agreed by 2 of 2 members but NOT verified (no proof, no reproducing execution) — consensus is not a verdict; needs human review
- code_only path skips Google Workspace/Office conversion but stores original binary as readable file —
graphify/detect.py· Escalate · medium- agreed by 2 of 2 members but NOT verified (no proof, no reproducing execution) — consensus is not a verdict; needs human review
- non-descend walk clears dirnames only when not in_memory, leaking subtree files into top-level bucket —
graphify/detect.py· Escalate · medium- agreed by 2 of 2 members but NOT verified (no proof, no reproducing execution) — consensus is not a verdict; needs human review
- Escaping .gitmodules path can make scanner run Git outside the scan root —
graphify/detect.py:1542· Escalate · medium- agreed by 2 of 2 members but NOT verified (no proof, no reproducing execution) — consensus is not a verdict; needs human review
- Concurrent subtree walkers mutate shared ignored_dirs without synchronization —
graphify/detect.py:2287· Escalate · medium- agreed by 2 of 2 members but NOT verified (no proof, no reproducing execution) — consensus is not a verdict; needs human review
Analysis details — impact, health, verification
Impact & health
Graphify review
Impact — 2142 functions depend on the 809 functions this change touches.
Health — this change adds coupling hotspots:
- new:
extract()— 520 callers, 43 callees - new:
_rebuild_code()— 113 callers, 50 callees - new:
detect()— 117 callers, 17 callees - new:
save_manifest()— 40 callers, 11 callees - new:
extract_files_direct()— 17 callers, 20 callees - new:
extract_xaml()— 19 callers, 17 callees - new:
extract_corpus_parallel()— 26 callers, 11 callees - new:
dispatch_command()— 2 callers, 123 callees - …and 33 more — each is listed as a finding
Verification — 2142 functions in the blast radius were not formally verified this run (proofs are advisory here).
Gate & verification
graphify gate
PASS — objectively clean (no health regressions, tests not run — proofs not run this pass (advisory)). Grounded, not self-assessed.
Advisory (not blocking):
- verification_scope: 1167 function(s) in the blast radius were not formally verified this run
Formal verification
Could not verify: Could not verify detect.
The verifier did not have enough to check detect, so it is saying so rather than guessing. No false assurance is the whole point.
Guarantee: No guarantee either way, this is an honest abstention, not a pass.
Note: Reason: parameter `root` is annotated `Path` — outside the synthesizable primitive/collection set
Could not verify: Could not verify detect\_incremental.
The verifier did not have enough to check detect\_incremental, so it is saying so rather than guessing. No false assurance is the whole point.
Guarantee: No guarantee either way, this is an honest abstention, not a pass.
Note: Reason: parameter `root` is annotated `Path` — outside the synthesizable primitive/collection set
Could not verify: Could not verify \_git\_tracked\_path\_keys.
The verifier did not have enough to check \_git\_tracked\_path\_keys, so it is saying so rather than guessing. No false assurance is the whole point.
Guarantee: No guarantee either way, this is an honest abstention, not a pass.
Note: Reason: parameter `root` is annotated `Path` — outside the synthesizable primitive/collection set
Could not verify: Could not verify \_is\_noise\_dir.
The verifier did not have enough to check \_is\_noise\_dir, so it is saying so rather than guessing. No false assurance is the whole point.
Guarantee: No guarantee either way, this is an honest abstention, not a pass.
Note: Reason: parameter `parent` is annotated `'Path | None'` — outside the synthesizable primitive/collection set
· 6 grounded finding(s) anchored inline below; 35 more finding(s) on lines outside this diff (see the check run).
| return name[dot:].lower() in _CORPUS_EXT_LOWER | ||
|
|
||
|
|
||
| def _is_noise_dir(part: str, parent: "Path | None" = None) -> bool: |
There was a problem hiding this comment.
_is_noise_dir()
10 callers depend on it (afferent coupling).
Grounded coupling-delta finding (deterministic), not an LLM guess.
| return _path_is_under(worktree, root) or _path_is_under(root, worktree) | ||
|
|
||
|
|
||
| def _git_enumerate_named_worktrees( |
There was a problem hiding this comment.
_git_enumerate_named_worktrees()
fans out to 7 callees (efferent coupling).
Grounded coupling-delta finding (deterministic), not an LLM guess.
| ) | ||
|
|
||
|
|
||
| def _git_enumerate_files( |
There was a problem hiding this comment.
_git_enumerate_files()
fans out to 8 callees (efferent coupling).
Grounded coupling-delta finding (deterministic), not an LLM guess.
|
|
||
|
|
||
| def detect(root: Path, *, follow_symlinks: bool | None = None, google_workspace: bool | None = None, extra_excludes: list[str] | None = None, cache_root: Path | None = None, gitignore: bool = True) -> dict: | ||
| def detect(root: Path, *, follow_symlinks: bool | None = None, google_workspace: bool | None = None, extra_excludes: list[str] | None = None, cache_root: Path | None = None, gitignore: bool = True, code_only: bool = False) -> dict: |
There was a problem hiding this comment.
detect()
fans out to 17 callees (efferent coupling); 117 callers depend on it (afferent coupling).
Grounded coupling-delta finding (deterministic), not an LLM guess.
| vprint(f"scanning {root} ...") | ||
| heartbeat = _Heartbeat("scanning") | ||
|
|
||
| def _walk_from( |
There was a problem hiding this comment.
_walk_from()
fans out to 8 callees (efferent coupling).
Grounded coupling-delta finding (deterministic), not an LLM guess.
|
|
||
| for p in all_files: | ||
| # For memory dir files, skip hidden/noise filtering | ||
| def _admit(p: Path) -> tuple: |
There was a problem hiding this comment.
_admit()
fans out to 8 callees (efferent coupling).
Grounded coupling-delta finding (deterministic), not an LLM guess.
1 participant
Use git ls-files, Google repo manifests, and nested worktrees so ignored trees are pruned at directory granularity instead of a Python walk.
slice review: v8...albertbu:graphify:albertb/detect-git-enum