AIStoreHub Package Vault: Codegraph
Package Slug: codegraph
Archetype: agent
Upstream Repository: https://github.com/colbymchenry/codegraph
Workspace Files & Specifications (67 files)
- .claude/skills/add-lang/SKILL.md (11070 bytes)
--- name: add-lang description: Add tree-sitter language support to codegraph end-to-end — wire the grammar + extractor, write tests, then benchmark extraction quality and retrieval value on 3 popular real-world repos. Use when the user runs /add-lang <language> or asks to add/support a new language (e.g. Lua, Elixir, Zig, OCaml) in codegraph. --- # Add a language to CodeGraph Wire a new tree-sitter language into codegraph's extraction pipeline, prove it extracts real symbols on popular repos, and prove it beats no-codegraph for an agent. Runs **fully autonomously** — pick repos, benchmark, update docs, then report. **Never commit, push, publish, or tag** (house rule); leave all changes for the user to review. The argument is the language token used throughout the `Language` union, e.g. `lua`, `elixir`, `zig`. If none was given, ask which language. Use the lowercase single-token form everywhere (`csharp`, not `c#`). ## Prerequisites - Run from the codegraph repo root. `node`, `git`, `gh`, and a logged-in `claude` CLI (the benchmark spawns real `claude -p` runs). - The benchmark uses the local dev build — Step 8 builds + links it on PATH. ## Workflow Copy this checklist and work through it in order: ``` - [ ] 1. Resolve language; bail early if already supported (just benchmark) - [ ] 2. Find a grammar + health-check it (ABI / heap corruption) - [ ] 3. Discover the grammar's AST node types (dump-ast.mjs) - [ ] 4. Wire the language (4 files; sometimes a 5th core touch) - [ ] 5. Build + verify-extraction loop until PASS - [ ] 6. Add extraction tests; make them green - [ ] 7. Auto-pick 3 popular repos by size tier; add to corpus.json - [ ] 8. Benchmark all 3: extraction + with/without A/B - [ ] 9. Update README + CHANGELOG - [ ] 10. Report; do NOT commit ``` ### Step 1 — Resolve + short-circuit Check whether the language is already wired: look for the token in the `LANGUAGES` const (`src/types.ts`) and the `EXTRACTORS` map (`src/extraction/languages/index.ts`). If it is already supported (e.g. `typescript`, `rust`), **skip Steps 2–6** and go straight to benchmarking (Steps 7–8) to validate/measure it — note in the report that no code changed. ### Step 2 — Find a grammar, then health-check it ```bash ls node_modules/tree-sitter-wasms/out/ | grep -i <lang> # csharp -> c_sharp ``` - **Present** → likely off-the-shelf; `grammars.ts` resolves it from `tree-sitter-wasms` automatically. (Many languages: elixir, zig, ocaml, solidity, toml, yaml, …) - **Absent** → vendor a `.wasm` into `src/extraction/wasm/` (like `pascal` / `scala` / `lua`) and add the token to the vendored branch in Step 4. **Always health-check before writing an extractor — a *present* grammar can still be unusable:** ```bash node scripts/add-lang/check-grammar.mjs <lang> path/to/valid-sample.<ext> ``` It prints the grammar's ABI version and runs it the way indexing does: in a child process with indexing's V8 flags, beside the vendored grammars, parsing a valid sample many times and once at the 1 MiB file-size limit. If it **FAILs** (ERROR trees, or later parses giving a different tree, on valid code — an old ABI corrupting the shared WASM heap, which silently drops nested calls/imports on every file after the first; e.g. the tree-sitter-wasms **Lua** grammar is ABI 13 and fails — or the process running it crashing or hanging, even after every parse finished), do NOT use that wasm. **Vendor a newer (ABI 14/15) build instead:** ```bash npm pack @tree-sitter-grammars/tree-sitter-<lang> # often ships a prebuilt *.wasm # or build one: npx tree-sitter build --wasm (needs Docker/emscripten) cp <the>.wasm src/extraction/wasm/tree-sitter-<lang>.wasm ``` then add the token to the vendored branch in Step 4 and re-run check-grammar on the vendored path until it PASSes. **If you cannot obtain a healthy wasm, STOP and tell the user.** ### Step 3 — Discover AST node types Get a representative source file (write a small sample covering functions, classes/structs, imports, enums; or `curl` a raw file from a known repo), then: ```bash node scripts/add-lang/dump-ast.mjs <lang> path/to/sample.<ext> # vendored grammar: pass the wasm path instead of the token node scripts/add-lang/dump-ast.mjs src/extraction/wasm/tree-sitter-<lang>.wasm sample.<ext> ``` The frequency table + field names (`name:`, `parameters:`, `body:`, `return_type:`) tell you what to map. Open the existing extractor closest to the language's paradigm as a model: `rust.ts`/`scala.ts` (functional, traits), `java.ts`/`csharp.ts` (OO), `python.ts`/`ruby.ts` (scripting), `go.ts` (top-level methods + receivers). ### Step 4 — Wire the language (4 files) These are exact, fragile wiring — match the existing style precisely: 1. **`src/types.ts`** — TWO edits: - add `'<lang>',` to the `LANGUAGES` const (before `'unknown'`); - add `'**/*.<ext>',` to `DEFAULT_CONFIG.include`. **Don't skip this** — it's the file-scan allowlist; without the glob, `codegraph init` finds **0 files** even though detection/extraction are wired. 2. **`src/extraction/grammars.ts`** — three maps: - `WASM_GRAMMAR_FILES`: `<lang>: 'tree-sitter-<lang>.wasm',` - `EXTENSION_MAP`: each file extension → `'<lang>'` (e.g. `'.lua': 'lua',`) - `getLanguageDisplayName`: `<lang>: '<Display Name>',` - **vendored only**: add `<lang>` to the `(lang === 'pascal' || lang === 'scala' || …)` wasm-path branch. 3. **`src/extraction/languages/<lang>.ts`** — new file exporting `export const <lang>Extractor: LanguageExtractor = { … }`. Map the node types from Step 3. Required fields: `functionTypes`, `classTypes`, `methodTypes`, `interfaceTypes`, `structTypes`, `enumTypes`, `typeAliasTypes`, `importTypes`, `callTypes`, `variableTypes`, `nameField`, `bodyField`, `paramsField`. Add hooks as the grammar needs them (`getSignature`, `getVisibility`, `isExported`, `extractImport`, `visitNode`, `getReceiverType`, `interfaceKind`, `enumMemberTypes`, etc. — see `src/extraction/tree-sitter-types.ts`). 4. **`src/extraction/languages/index.ts`** — `import { <lang>Extractor } from './<lang>';` and add `<lang>: <lang>Extractor,` to `EXTRACTORS`. **Sometimes a 5th, core touch in `src/extraction/tree-sitter.ts`** — variable extraction has per-language branches in `extractVariable` (the generic fallback only finds direct `identifier`/`variable_declarator` children). If the grammar nests declared names (e.g. Lua's `variable_declaration → variable_list`), add a `} else if (this.language === '<lang>')` branch there, mirroring the existing ts/python/go ones. Import forms that aren't a distinct node (Lua/Ruby `require` is a *call*) are handled in the extractor's `visitNode` hook instead. ### Step 5 — Build + verify loop ```bash npm run build # tsc + copy-assets (copies any vendored *.wasm into dist/) ``` Index a small sample repo and check extraction: ```bash ( cd <sample-repo> && codegraph init ) node scripts/add-lang/verify-extraction.mjs <sample-repo> <lang> ``` `verify-extraction.mjs` fails (exit 1) if the language isn't detected or only `file`/`import` nodes were produced — the classic symptom of wrong node-type names. On FAIL or a thin WARN: re-run `dump-ast.mjs` on a richer file, fix the mappings in `<lang>.ts`, `npm run build`, re-index, re-verify. **Repeat until PASS.** ### Step 6 — Tests Add to `__tests__/extraction.test.ts`, modeled on the `Rust Extraction` block: - a `detectLanguage` assertion in `describe('Language Detection')` - a `describe('<Lang> Extraction')` block asserting functions/classes/imports are extracted from an inline source string. ```bash npx vitest run __tests__/extraction.test.ts ``` Green before continuing. ### Step 7 — Auto-pick 3 repos + corpus Pick **without asking**. Find candidates, then curate 3 that are genuinely `<lang>`-dominant, one per size tier: ```bash gh search repos --language=<lang> --sort=stars --limit 40 \ --json fullName,stargazerCount,description ``` Tiers (match `corpus.json`): **Small** <~150 files · **Medium** ~150–1500 · **Large** >~1500. Skip repos that are tagged `<lang>` but mostly another language. Write one cross-file architecture **question** per repo (the kind that needs tracing across files). Add a `"<Language>"` block to `.claude/skills/agent-eval/corpus.json` (fields: `name`, `repo`, `size`, `files`, `question`) so `/agent-eval` can reuse them. ### Step 8 — Benchmark all 3 (extraction + A/B) Make the dev build the codegraph on PATH **once**, then loop: ```bash npm run build && ./scripts/local-install.sh scripts/add-lang/bench.sh <lang> <name> <url> "<question>" headless # ×3 ``` `bench.sh` clones (shared `/tmp/codegraph-corpus`), wipes + indexes, runs `verify-extraction.mjs`, then the with/without retrieval A/B via `scripts/agent-eval/run-all.sh` (skips the paid A/B if extraction is broken). Read each `parse-run.mjs` summary printed by `run-all.sh`: tool calls, file `Read`s, Grep/Bash, codegraph-tool calls, duration, and **cost** — for both the `with` and `without` arms. After the loop, restore the dev link if needed: `./scripts/local-install.sh`. ### Step 9 — Docs + CHANGELOG - **README.md**: add `<Lang>` to the "19+ Languages" feature bullet, and add a row to the **Supported Languages** table: `| <Lang> | \`.ext\` | Full support (classes, methods, …) |`. - **CHANGELOG.md**: add an `## [Unreleased]` section at the top (above the latest version) with `### Added` → a user-perspective bullet, e.g. *"CodeGraph now indexes **<Lang>** (`.ext`) — functions, classes, imports, and call edges."* If `## [Unreleased]` already exists, append under it. (It's folded into the next versioned block at release time.) ### Step 10 — Report (do NOT commit) Summarize for review: - **Files changed**: the 4 wiring edits + new extractor + tests + README + CHANGELOG + corpus.json (+ any vendored `.wasm`). - **Extraction** per repo: files / nodes / edges / `verify-extraction` result. - **A/B** per repo: `with` vs `without` (tool calls, file Reads, cost) and a one-line verdict — did codegraph reduce effort, and did both arms reach a correct answer? - **Gaps / follow-ups** (node types not yet mapped, resolution edges missing, framework routes, etc.). Hand the changes to the user. **Do not** run `git commit`/`push` or publish — releases go through the GitHub Actions Release workflow. ## Notes - The A/B spawns real **paid** `claude -p` runs (opus, `--max-budget-usd`), 2 arms × 3 repos. The corpus dir `/tmp/codegraph-corpus` is shared with `/agent-eval`, so clones are reused across runs. - Any new `*.wasm` must live in `src/extraction/wasm/` — `copy-assets` (run by `npm run build`) ships it; otherwise it won't be in `dist/`. - An index must be served by the **same** binary that built it. Step 8 builds + links the dev build first, so this holds. - If a grammar can't be obtained, or extraction can't reach PASS, **STOP and report** — don't ship a half-wired language. - .claude/skills/agent-eval/corpus.json (22255 bytes)
{ "_comment": "Test corpus for /agent-eval. Add entries freely. size: Small (<~150 files), Medium (~150-1500), Large (>~1500). 'question' is a representative architectural question that exercises cross-file understanding.", "TypeScript": [ { "name": "ky", "repo": "https://github.com/sindresorhus/ky", "size": "Small", "files": "~25", "question": "How does ky implement request retries and timeouts?" }, { "name": "excalidraw", "repo": "https://github.com/excalidraw/excalidraw", "size": "Medium", "files": "~600", "question": "How does Excalidraw render and update canvas elements?" }, { "name": "vscode", "repo": "https://github.com/microsoft/vscode", "size": "Large", "files": "~10000", "question": "How does the extension host communicate with the main process?" } ], "JavaScript": [ { "name": "express", "repo": "https://github.com/expressjs/express", "size": "Small", "files": "~50", "question": "How does Express route a request through its middleware stack?" } ], "Go": [ { "name": "cobra", "repo": "https://github.com/spf13/cobra", "size": "Small", "files": "~50", "question": "How does cobra parse commands and flags?" }, { "name": "gin", "repo": "https://github.com/gin-gonic/gin", "size": "Medium", "files": "~150", "question": "How does gin route requests through its middleware chain?" }, { "name": "terraform", "repo": "https://github.com/hashicorp/terraform", "size": "Large", "files": "~4000", "question": "How does Terraform build and walk the resource dependency graph?" }, { "name": "cosmos-sdk", "repo": "https://github.com/cosmos/cosmos-sdk", "size": "Large", "files": "~5000", "question": "How does a bank module MsgSend message reach the account balance update? Trace the cross-module call path from the bank keeper's Send handler through to the account/balance store update." } ], "Python": [ { "name": "click", "repo": "https://github.com/pallets/click", "size": "Small", "files": "~60", "question": "How does click parse command-line arguments into commands?" }, { "name": "flask", "repo": "https://github.com/pallets/flask", "size": "Medium", "files": "~90", "question": "How does Flask dispatch a request to a view function?" }, { "name": "django", "repo": "https://github.com/django/django", "size": "Large", "files": "~2700", "question": "How does Django's ORM build and execute a query from a QuerySet?" } ], "Rust": [ { "name": "clap", "repo": "https://github.com/clap-rs/clap", "size": "Medium", "files": "~200", "question": "How does clap parse arguments against a derived command definition?" }, { "name": "tokio", "repo": "https://github.com/tokio-rs/tokio", "size": "Large", "files": "~700", "question": "How does tokio schedule and run async tasks on its runtime?" }, { "name": "deno", "repo": "https://github.com/denoland/deno", "size": "Large", "files": "~1500", "question": "How does Deno load and execute a TypeScript module?" } ], "Java": [ { "name": "gson", "repo": "https://github.com/google/gson", "size": "Medium", "files": "~200", "question": "How does Gson serialize an object to JSON?" }, { "name": "okhttp", "repo": "https://github.com/square/okhttp", "size": "Medium", "files": "~640", "question": "How does OkHttp process a request through its interceptor chain?" }, { "name": "guava", "repo": "https://github.com/google/guava", "size": "Large", "files": "~3000", "question": "How does Guava's CacheBuilder build and configure a cache?" } ], "Kotlin": [ { "name": "koin", "repo": "https://github.com/InsertKoinIO/koin", "size": "Medium", "files": "~300", "question": "How does Koin resolve and inject dependencies?" }, { "name": "leakcanary", "repo": "https://github.com/square/leakcanary", "size": "Medium", "files": "~250", "question": "How does LeakCanary detect and analyze a memory leak?" } ], "Swift": [ { "name": "alamofire", "repo": "https://github.com/Alamofire/Alamofire", "size": "Small", "files": "~100", "question": "How does Alamofire build, send, and validate a request?" } ], "C#": [ { "name": "serilog", "repo": "https://github.com/serilog/serilog", "size": "Medium", "files": "~250", "question": "How does Serilog route a log event to its sinks?" }, { "name": "jellyfin", "repo": "https://github.com/jellyfin/jellyfin", "size": "Large", "files": "~2500", "question": "How does Jellyfin scan and identify items in a media library?" } ], "Ruby": [ { "name": "sinatra", "repo": "https://github.com/sinatra/sinatra", "size": "Small", "files": "~60", "question": "How does Sinatra match a request to a route handler?" }, { "name": "discourse", "repo": "https://github.com/discourse/discourse", "size": "Large", "files": "~3000", "question": "How does Discourse create and render a new post?" } ], "PHP": [ { "name": "slim", "repo": "https://github.com/slimphp/Slim", "size": "Small", "files": "~80", "question": "How does Slim handle a request through its middleware?" }, { "name": "laravel", "repo": "https://github.com/laravel/framework", "size": "Large", "files": "~3000", "question": "How does Laravel resolve and dispatch a route to a controller?" } ], "C": [ { "name": "redis", "repo": "https://github.com/redis/redis", "size": "Large", "files": "~600", "question": "How does Redis parse and dispatch a client command?" } ], "C++": [ { "name": "json", "repo": "https://github.com/nlohmann/json", "size": "Small", "files": "~100", "question": "How does nlohmann::json parse a JSON string into a value?" }, { "name": "grpc", "repo": "https://github.com/grpc/grpc", "size": "Large", "files": "~3000", "question": "How does gRPC dispatch an incoming RPC to its handler?" } ], "Dart": [ { "name": "flutter", "repo": "https://github.com/flutter/flutter", "size": "Large", "files": "~6000", "question": "How does Flutter build and lay out a widget tree?" } ], "Svelte": [ { "name": "shadcn-svelte", "repo": "https://github.com/huntabyte/shadcn-svelte", "size": "Medium", "files": "~600", "question": "How do shadcn-svelte components compose and apply their styling?" } ], "Lua": [ { "name": "lualine.nvim", "repo": "https://github.com/nvim-lualine/lualine.nvim", "size": "Small", "files": "~120", "question": "How does lualine assemble and render its statusline sections and components?" }, { "name": "telescope.nvim", "repo": "https://github.com/nvim-telescope/telescope.nvim", "size": "Medium", "files": "~80", "question": "How does Telescope wire a picker to its finder, sorter, and previewer?" }, { "name": "kong", "repo": "https://github.com/Kong/kong", "size": "Large", "files": "~1330", "question": "How does Kong execute plugins across a request's lifecycle phases?" } ], "Luau": [ { "name": "Knit", "repo": "https://github.com/Sleitnick/Knit", "size": "Small", "files": "~10", "question": "How does Knit register services and expose them to clients?" }, { "name": "vide", "repo": "https://github.com/centau/vide", "size": "Small", "files": "~40", "question": "How does vide track reactive sources and re-run effects when state changes?" }, { "name": "Fusion", "repo": "https://github.com/dphfox/Fusion", "size": "Medium", "files": "~115", "question": "How does Fusion build and update its reactive UI graph from state objects?" } ], "Objective-C": [ { "name": "Masonry", "repo": "https://github.com/SnapKit/Masonry", "size": "Small", "files": "~50", "question": "How does Masonry build and activate Auto Layout constraints from its block DSL?" }, { "name": "FMDB", "repo": "https://github.com/ccgus/fmdb", "size": "Medium", "files": "~80", "question": "How does FMDB execute a prepared SQL statement and bind parameters?" }, { "name": "SDWebImage", "repo": "https://github.com/SDWebImage/SDWebImage", "size": "Large", "files": "~400", "question": "How does SDWebImage download, cache, and decode an image for a UIImageView?" } ], "Mixed iOS (Swift+ObjC)": [ { "name": "Charts", "repo": "https://github.com/danielgindi/Charts", "size": "Small", "files": "~270", "question": "How does the ChartsDemo ObjC demo controller drive the Swift Charts library to animate and notify a data update?" }, { "name": "realm-swift", "repo": "https://github.com/realm/realm-swift", "size": "Medium", "files": "~370", "question": "How does a Swift `Realm.write { realm.add(obj) }` reach the Objective-C persistence layer?" }, { "name": "wikipedia-ios", "repo": "https://github.com/wikimedia/wikipedia-ios", "size": "Large", "files": "~1700", "question": "How does tapping a search result reach the article-fetch network call across the Swift / ObjC boundary?" } ], "React Native (legacy bridge + TurboModule)": [ { "name": "@react-native-async-storage", "repo": "https://github.com/react-native-async-storage/async-storage", "size": "Small", "files": "~60", "question": "How does `setItem` in JS reach the native `legacy_multiSet` implementation?" }, { "name": "react-native-svg", "repo": "https://github.com/software-mansion/react-native-svg", "size": "Medium", "files": "~700", "question": "How does a JS `Svg.getTotalLength(...)` reach the iOS / Android native implementation via TurboModule?" }, { "name": "react-native-firebase", "repo": "https://github.com/invertase/react-native-firebase", "size": "Large", "files": "~1100", "question": "How does a native iOS push notification reach the JS `messaging().onMessage(...)` listener?" } ], "Expo Modules": [ { "name": "expo-haptics", "repo": "https://github.com/expo/expo/tree/main/packages/expo-haptics", "size": "Small", "files": "~15", "question": "How does `Haptics.notificationAsync(...)` in JS reach `UINotificationFeedbackGenerator` in the Swift Module?" }, { "name": "expo-camera", "repo": "https://github.com/expo/expo/tree/main/packages/expo-camera", "size": "Medium", "files": "~70", "question": "How does a JS `CameraView.takePictureAsync(options)` reach the native AVCaptureSession / CameraDevice call?" } ], "React Native Fabric (view components)": [ { "name": "react-native-segmented-control", "repo": "https://github.com/react-native-segmented-control/segmented-control", "size": "Small", "files": "~25", "question": "How does JSX `<SegmentedControl onChange={cb}/>` reach the native onChange handler on iOS/Android?" }, { "name": "react-native-screens", "repo": "https://github.com/software-mansion/react-native-screens", "size": "Medium", "files": "~1200", "question": "How does JSX `<ScreenStack>` reach the native RNSScreenStackView component?" }, { "name": "react-native-skia", "repo": "https://github.com/Shopify/react-native-skia", "size": "Large", "files": "~1000", "question": "How does a `<SkiaPictureView/>` JSX usage reach the iOS / Android native renderer?" } ], "R": [ { "name": "AnomalyDetection", "repo": "https://github.com/twitter/AnomalyDetection", "size": "Small", "files": "~24", "question": "How does AnomalyDetectionTs go from the exported entry function to the underlying S-H-ESD statistical test? Name the functions on the path in order." }, { "name": "dplyr", "repo": "https://github.com/tidyverse/dplyr", "size": "Medium", "files": "~450", "question": "When mutate() is called on a grouped data frame, which functions handle the grouping and expression evaluation, in order, from mutate() down?" }, { "name": "ggplot2", "repo": "https://github.com/tidyverse/ggplot2", "size": "Large", "files": "~1150", "question": "When a ggplot object is printed, how does the plot actually get built and drawn — trace the path from print/plot to where geoms render. Name the key functions in order." } ], "COBOL": [ { "name": "cics-genapp", "repo": "https://github.com/cicsdev/cics-genapp", "size": "Small", "files": "~60", "question": "When a new insurance policy is added through the LGTESTP1 CICS test harness, which programs handle the request on the way to the DB2 insert? Trace the path and name the programs in order." }, { "name": "carddemo", "repo": "https://github.com/aws-samples/aws-mainframe-modernization-carddemo", "size": "Medium", "files": "~75", "question": "How does the online bill-payment flow work — from the COBIL00C screen program to the transaction record being written? Name the paragraphs performed in order and the copybooks that define the records involved." }, { "name": "CobolCraft", "repo": "https://github.com/meyfa/CobolCraft", "size": "Medium", "files": "~270", "question": "How does an incoming player chat message travel from packet handling to being broadcast to the other connected players? Name the programs on the path in order." } ], "VB.NET": [ { "name": "policyplus", "repo": "https://github.com/Fleex255/PolicyPlus", "size": "Small", "files": "~94", "question": "When the user toggles a policy to Enabled in the policy-setting editor and clicks OK, how does the new state end up written into the loaded policy source (POL file or registry)? Trace the path from the EditSetting dialog to the concrete write." }, - .claude/skills/agent-eval/SKILL.md (3840 bytes)
--- name: agent-eval description: Benchmark CodeGraph retrieval quality on a real codebase by comparing agent behavior with vs without CodeGraph. Use when the user runs /agent-eval or asks to test, benchmark, audit, or validate a codegraph version (the local dev build or a published npm version) against a language's repo. --- # CodeGraph Quality Audit Measures how much CodeGraph helps an agent versus plain grep/read, for a chosen codegraph version on a chosen real-world repo. Drives the harness in `scripts/agent-eval/`. ## Prerequisites - `tmux` 3+, a logged-in `claude` CLI, `node`, `git` (macOS/Linux). - Run from the codegraph repo root. ## Workflow Copy this checklist: ``` - [ ] 1. Pick version (local or npm) - [ ] 2. Pick language - [ ] 3. Pick repo by size - [ ] 4. Pick harness (headless / tmux / both) - [ ] 5. Run audit.sh in the background - [ ] 6. Report results ``` **Step 1 — version.** Ask with `AskUserQuestion`: which codegraph version to test. Offer "Local dev build" and "Latest published"; the free-text "Other" lets the user type a specific version (e.g. `0.7.10`). Map the answer to a VERSION token: - "Local dev build" → `local` - "Latest published" → `latest` - a typed version → that string (e.g. `0.7.10`) **Step 2 — language.** Read `.claude/skills/agent-eval/corpus.json`. Ask with `AskUserQuestion` which language to test, listing the languages that have entries. **Step 3 — repo.** From the chosen language's entries, ask which repo. Label each option with its size and file count, e.g. `excalidraw — Medium (~600 files)`. Each entry carries the `repo` URL and a representative `question`. **Step 4 — harness.** Ask with `AskUserQuestion` which harness to run, and map the answer to a MODE token: - "Headless" → `headless` — `claude -p` with stream-json: exact tokens/cost and a clean tool sequence (2 runs, fast, no TTY). - "Interactive (tmux)" → `tmux` — drives the real Claude TUI in tmux: faithful Explore-subagent behavior, metrics from session logs (2 runs, slower). - "Both" → `all` — headless + interactive (4 runs). **Step 5 — run.** Launch in the background (sets the version, clones if missing, wipes + re-indexes, runs the chosen arms — several minutes): ```bash scripts/agent-eval/audit.sh <VERSION> <repo-name> <repo-url> "<question>" <MODE> ``` **Step 6 — report.** When the job finishes, read the log and report per arm: - Headless (`parse-run.mjs`): total tool calls, file `Read`s, Grep/Bash, codegraph-tool calls, duration, **total cost**. - Interactive (`parse-session.mjs`): the `VERDICT: codegraph_explore used Nx | Read N | Grep/Bash N` and `TOKENS:` lines. - Both paths also print the three feedback metrics — residual context occupancy, explore sufficiency, allocation efficiency — and a headless A/B ends with a side-by-side `ARM COMPARISON` table. Report that table, and check its contamination row first: `CLI calls that RETURNED output` > 0 means the arm reached codegraph through Bash and its numbers are void. How to read the rest: `docs/benchmarks/agent-eval-feedback-metrics.md`. Lead with cost + tool/Read counts — they are the reliable signals; raw token in/out are confounded by subagent delegation and prompt caching. State whether codegraph reduced effort and whether both arms reached a correct answer. ## Notes - The index is rebuilt every run (`audit.sh` wipes `.codegraph`) — different versions extract differently, so an index must be served by the same binary that built it. - `audit.sh` temporarily mutates the global `codegraph` install for the test, then restores your dev link via `local-install.sh`. - Corpus repos are cloned to `/tmp/codegraph-corpus` (reused if already present). - Add or edit repos in `corpus.json` (fields: `name`, `repo`, `size`, `files`, `question`).
- AGENTS.md (42728 bytes)
# AGENTS.md Canonical project guidance for coding agents working in this repository (Codex/Astra, Claude Code via `@AGENTS.md`, Cursor, etc.). **Codex size note:** root `AGENTS.md` is intentionally kept near ~35 KiB (critical build/test/arch/retrieval rules retained). Longer validation methodology + the Excalidraw worked example live in `docs/AGENTS.md`. This environment sets `project_doc_max_bytes = 49152` so root (and root+`docs/` when cwd is under `docs/`) are not silently truncated at the 32 KiB default. ## Project Overview CodeGraph is a local-first code intelligence library + CLI + MCP server. It parses any supported codebase with tree-sitter, stores symbols/edges/files in SQLite (FTS5), and exposes a knowledge graph to AI agents (Claude Code, Cursor, Codex CLI, opencode) over MCP. Per-project data lives in `.codegraph/`. Extraction is deterministic — derived from AST, not LLM-summarized. Distributed as `@colbymchenry/codegraph` on npm; same binary serves as installer, indexer, and MCP server. ## Build, Test, Run ```bash npm run build # tsc + copy schema.sql and *.wasm + build the viewer into dist/; chmods dist/bin/codegraph.js npm run build:lib # the viewer's components as @colbymchenry/codegraph-ui (ui/dist) — NOT part of `build` npm run dev # tsc --watch npm run clean # rm -rf dist npm test # vitest run (all) npm run test:watch npm run test:eval # only __tests__/evaluation/ npm run eval # build then run __tests__/evaluation/runner.ts via tsx npm run cli # build then run the local dist binary # Single test file / pattern npx vitest run __tests__/installer-targets.test.ts npx vitest run __tests__/extraction.test.ts -t "TypeScript" ``` `copy-assets` (called from `build`) copies `src/db/schema.sql` and all `src/extraction/wasm/*.wasm` files into `dist/`. **Any new SQL or grammar wasm must be copied or it won't ship.** One other build step writes into `dist/` and is subject to the same rule: `build:ui` builds the browser viewer into `dist/viewer/` (never `dist/ui/` — that's the terminal ui). `scripts/check-ui-build.mjs` asserts both `dist/viewer/` and the copied grammars in `dist/extraction/wasm/` after every build and inside every release archive — the viewer's syntax highlighting reads a file with the same grammar the engine indexed it with, so a missing wasm is an unhighlighted screen as well as an extraction gap. `npm run build:lib` is separate and does NOT run as part of `npm run build`: it compiles the same `ui/src` tree a second way, with `svelte-package`, into `ui/dist` — the `@colbymchenry/codegraph-ui` component library the Pro app imports (task CG-61). `scripts/check-ui-package.mjs` then prunes the standalone app's shell out of it, resolves the extensionless import specifiers `svelte-package` leaves behind, and asserts the seam: nothing outside `lib/adapter.js` may reach the network. The package is **prepared, not published** — `ui/package.json` carries `"private": true` deliberately, and `scripts/pack-npm.sh` only packs a tarball when `CODEGRAPH_PACK_UI=1`. Tests run as **two Vitest projects** (`test.projects` in `vitest.config.mts`): `engine` (node) and `ui` (jsdom, the Svelte plugin, `resolve.conditions: ['browser']`) for the single `__tests__/ui-package.test.ts`. `npm test` still runs both. The split is not cosmetic — `browser` is a package-resolution condition, and applied globally it hands the engine's suites the browser builds of `web-tree-sitter` and friends. The root config (`vitest.config.mts`, `.mts` because the plugin is ESM-only and the repo is CJS) is the shared base; note that a project with `extends: true` **concatenates** the base's `include` with its own, which is why the `ui` project sets `extends: false`. Node engines: `>=20.0.0 <25.0.0`. There is a hard exit on Node 25.x and below 20 (see `src/bin/node-version-check.ts`). That range is what users run. Working in this repo needs Node 22.12 or newer: `devEngines` makes `npm ci` and `npm run` stop on anything older, for Vite 7 and Vitest 4. ## Architecture ### Layered pipeline ``` files → ExtractionOrchestrator (tree-sitter) → DB (nodes/edges/files) ↓ ReferenceResolver (imports, name-matching, framework patterns) ↓ GraphQueryManager / GraphTraverser (callers, callees, impact) ↓ ContextBuilder (markdown/JSON for AI consumption) ``` The public API surface is `src/index.ts` — the `CodeGraph` class wires all the layers and re-exports types. Library users only touch this file; the MCP server and CLI also drive it. ### Module layout - `src/index.ts` — `CodeGraph` class: `init`/`open`/`close`, `indexAll`, `sync`, `searchNodes`, `getCallers`/`getCallees`, `getImpactRadius`, `buildContext`, `watch`/`unwatch`. **It must stay light to load:** a read command, a query worker and the prompt hook all load it, and never index. What only indexing needs (extraction, resolution, the watcher) goes through `engine()` (`src/lazy-engine.ts`, listed in `src/engine.ts`), never an import of its own; `__tests__/lazy-engine.test.ts` fails otherwise. - `src/db/` — `DatabaseConnection`, `QueryBuilder` (prepared statements), `schema.sql`, `sqlite-adapter.ts`. Backed by Node's built-in **`node:sqlite`** (`DatabaseSync`) — real SQLite with WAL + FTS5, exposed through a thin better-sqlite3-shaped adapter. The bundled runtime always ships Node ≥22.5, so `node:sqlite` is always available: **no native build step and no wasm fallback**. (Running from source needs Node ≥22.5.) `codegraph status` reports the live backend (`node-sqlite`, the sole backend). - `src/extraction/` — `ExtractionOrchestrator`, tree-sitter wrappers, per-language extractors under `languages/` (one file per language), plus standalone extractors for non-tree-sitter formats (`svelte-extractor.ts`, `vue-extractor.ts`, `liquid-extractor.ts`, `dfm-extractor.ts` for Delphi). `parse-worker.ts` runs heavy parsing off the main thread. - `src/resolution/` — `ReferenceResolver` orchestrates `import-resolver.ts` (with `path-aliases.ts` for tsconfig path aliases + cargo workspace member globs), `name-matcher.ts`, and `frameworks/` (Express, Elysia, Laravel, CodeIgniter 3, Rails, FastAPI, Django, Flask, Spring, Gin, Axum, ASP.NET, Vapor, React Router, Next.js — `nextjs.ts`: pages and `route.ts` handlers from files, `router.push` / `redirect` / `NextResponse.redirect` as `navigates` edges, with `next-router-synthesizer.ts` for `<Link href>` and for the routes a proxy's `config.matcher` accepts, read by `next-proxy.ts` — Expo Router, SvelteKit, Vue/Nuxt, Cargo workspaces). Frameworks emit `route` nodes and `references` edges. `parameter-flow.ts` (the engine, its SQL in `db/parameter-flow-queries.ts`) and `ruby-parameter-flow.ts` (one language's adapter) move a call on a parameter to the method of the one class every call site passes, after the calls edges persist. A sync looks only as far as what it changed reaches, from a record the last pass left in one metadata key, and whatever it can't settle from that falls back to a pass over the project. `callback-synthesizer.ts` holds the whole-graph synthesis passes (`SYNTH_PASSES`, merged in registry order — first-seen wins a duplicate pair) with the language gates; `cpp-template-instantiations.ts` is the pass for a C++ call through a template parameter, onto each class the project writes as the argument (read from `Name<Foo>`, `f<Foo>()`, a deduced call or a gtest type list), capped per parameter and by how widely a template is written; `tier-synthesizer.ts` is the cross-tier pass (a client's literal `fetch`/`axios` path onto its own route, a queue job onto its consumer, a bus / socket event onto its handler — `channel`, `tier`, `registeredAt` on every edge; registered before the in-process emitter pass so its more specific edge wins); `synth-utils.ts` has the helpers they share (`enclosingFn`, `enclosingValue`, `makeLineAt`). Express's `postExtract` composes `app.use('/prefix', router)` mounts onto a mounted file's route names, idempotently (the original path stays in `qualifiedName`). Elysia's (`elysia.ts`) does the same for `.use(plugin)` per exported instance, and Express reads no route in a file that imports Elysia. - `src/graph/` — `GraphTraverser` (BFS/DFS, impact radius, path finding) and `GraphQueryManager` (high-level queries), plus the shared query-time derivations more than one surface renders: `named-symbol-flow.ts` (the one path finder, behind `codegraph_explore`'s Flow section and the viewer's Flow strip), `dynamic-boundary-report.ts` (where the graph stops), `type-hierarchy.ts` (ancestors/subtypes and the implementation count explore prints and the viewer draws), `dead-code.ts` (unreferenced symbols, and every reason a candidate is NOT claimed), `synthesized-hop.ts` (how a synthesized hop reads, in explore's Flow, the `codegraph_node` trail and ContextBuilder's call paths — a new synthesizer's wording goes here), `file-cycles.ts` (circular imports, in `codegraph cycles` and the viewer's map). A derivation that two callers render must live here, not in `ToolHandler` — two derivations eventually disagree. - `src/context/` — `ContextBuilder` + formatter for markdown/JSON output. - `src/search/` — full-text query parser and helpers for FTS5. - `src/sync/` — `FileWatcher` (native FSEvents/inotify/RDCW) with debounce + filter, and git-hook helpers. - `src/mcp/` — MCP server (`MCPServer`, `tools.ts`, `transport.ts`). `server-instructions.ts` is what the server returns in the MCP `initialize` response — keep it in sync with the user-facing tool guidance. - `src/installer/` — see below. - `src/bin/codegraph.ts` — CLI (commander). Subcommands: `install`, `uninstall`, `init`, `uninit`, `index`, `sync`, `status`, `unlock`, `query`, `explore`, `context`, `node`, `files`, `callers`, `callees`, `impact`, `affected`, `cycles`, `daemon`, `telemetry`, `upgrade`, `version`. Hidden from `--help`: `serve --mcp`, `prompt-hook`, and `ui`/`web` unless `CODEGRAPH_UI=1`. - `src/ui/` — terminal UI (shimmer progress, worker). - `src/ui-server/` -- read-only JSON API for the `codegraph ui` browser viewer (`api/`: `node`, `flow`, `map`, `screens`, `steps`, `deadcode`, `trails`, `program`, ...) plus static server; Svelte viewer lives in `ui/` (see `docs/design/codegraph-ui-design-spec.md`). `screens`/`steps`/`program` share one fold (`via`/`when` via `graph/branch-guards.ts`); `api/effects.ts` curates calls that leave the index; `api/route-roots.ts` names where a route's code starts. Derivations rendered by more than one surface belong in `src/graph/`, not `ToolHandler`. **Not released yet:** `codegraph ui` / `web` are refused (and hidden from `--help`) unless `CODEGRAPH_UI=1` is set — `src/bin/viewer-gate.ts`. Viewer changelog entries wait in `docs/viewer-launch-changelog.md`, not under `[Unreleased]`. At launch: delete the gate, move those entries back, drop the "not released yet" notes from `site/`. ### NodeKind / EdgeKind Defined in `src/types.ts`. Both extractors and resolvers must use these exact strings. - **NodeKind**: `file`, `module`, `class`, `struct`, `interface`, `trait`, `protocol`, `function`, `method`, `property`, `field`, `variable`, `constant`, `enum`, `enum_member`, `type_alias`, `namespace`, `parameter`, `import`, `export`, `route`, `component`, `union`. - **EdgeKind**: `contains`, `calls`, `imports`, `exports`, `extends`, `implements`, `references`, `type_of`, `returns`, `instantiates`, `overrides`, `decorates`. ### Multi-agent installer `src/installer/` is the entry point for `codegraph install` (and the bare `codegraph`/`npx @colbymchenry/codegraph` invocation). Architecture: - `targets/registry.ts` lists every supported agent. - `targets/types.ts` defines the `AgentTarget` interface — adding another agent (Continue, Zed…) is **one new file in `targets/` + one entry in `registry.ts`**. Each target owns its config-file location and MCP-server JSON/TOML/JSONC writing. (Claude, Codex, Gemini, opencode, Droid, Qwen, DSH, Grok, Kimi and Pi also write a short instructions-file block — see below.) - Current targets: `claude.ts`, `cursor.ts`, `codex.ts`, `opencode.ts`, `hermes.ts`, `gemini.ts`, `antigravity.ts`, `kiro.ts`, `copilot-vscode.ts`, `copilot-cli.ts`, `copilot-jetbrains.ts`, `bob.ts`, `droid.ts`, `windsurf.ts`, `qwen.ts`, `dsh.ts`, `grok.ts`, `kimi.ts`, `qoder.ts`, `pi.ts`. - `targets/toml.ts` is a hand-rolled TOML serializer scoped to `[mcp_servers.codegraph]` (used by Codex and Grok, through `writeTomlTable` / `deleteTomlTable`). Sibling tables and `[[array_of_tables]]` are preserved verbatim. No new dependency. - `targets/json-config.ts` edits `mcpServers.codegraph` in place through `jsonc-parser`, for Bob, Droid, Windsurf, Qwen, Kimi, Qoder and Pi. An entry already in the file is the user's: agents write switches into it and users pin a `command` or `cwd`, so a re-install or `--refresh` only fills in missing keys. DSH's and Grok's writers keep to the same rule. Use it for a new agent with that file shape; a file with nothing left in it is deleted on uninstall, through `removeFileUnlessLinked`. - `*_HOME`-style overrides (`QWEN_HOME`, `KIMI_CODE_HOME`, `GROK_HOME`, `DSH_HOME`, `QODER_CONFIG_DIR`, `PI_CODING_AGENT_DIR`) go through `configDirFromEnv` in `targets/shared.ts`; each must also be cleared in `__tests__/setup-home-sandbox.ts` and in the suite's `setHome`. - opencode reads `opencode.jsonc` by default; the installer prefers existing `.jsonc`, falls back to `.json`, and creates `.jsonc` for greenfield installs. Edits are surgical via `jsonc-parser` so user comments and formatting survive install/re-install/uninstall round-trips. The MCP entry is OpenCode 2's native `mcp.servers.codegraph` with `disabled: false` and `codemode: false` (so `codegraph_explore` stays on the native tool list); a pre-#1698 `mcp.codegraph` + `enabled` entry is migrated on re-install and removed by uninstall. - `instructions-template.ts` holds `CODEGRAPH_INSTRUCTIONS_BLOCK`, a deliberately short marker-fenced (`<!-- CODEGRAPH_START -->`/`<!-- CODEGRAPH_END -->`) `## CodeGraph` block. The Claude, Codex, Gemini, opencode, Droid, Qwen, DSH, Grok, Kimi and Pi targets write it into their instructions file (`CLAUDE.md` / `~/.codex/AGENTS.md` / `~/.gemini/GEMINI.md` / `~/.config/opencode/AGENTS.md` / `~/.factory/AGENTS.md` / `~/.qwen/QWEN.md` / `~/.dsh/AGENTS.md` / `~/.grok/rules/codegraph.md` / `~/.kimi-code/AGENTS.md` / `~/.pi/agent/AGENTS.md`) via `upsertInstructionsEntry`, and `uninstall` removes it. History: #529 removed the old full playbook because it duplicated the MCP `initialize` instructions; #704 brought back this small block because the instructions file reaches two audiences `initialize` cannot — Task-tool subagents and non-MCP harnesses (the `codegraph explore` CLI). Cursor and Kiro write no block; their `install` only strips one a previous install left behind. Bob, Windsurf and Qoder are MC - aistorehub.json (578 bytes)
{ "name": "Codegraph", "slug": "codegraph", "version": "1.0.0", "type": "agent", "description": "Pre-indexed code knowledge graph, auto syncs on code changes, for Claude Code, Codex, Gemini, Cursor, OpenCode, AntiGravity, Kiro, CoPilot, and Hermes Agent — fewer tokens, fewer tool calls, 100% local", "owner": "community", "repository": "catalog", "github_repo_url": "https://github.com/colbymchenry/codegraph", "tags": [ "trending", "typescript", "colbymchenry", "github", "open-source" ], "imported_at": "2026-11-21T23:56:27.595Z" } - BUNDLING.md (3975 bytes)
# Distribution: self-contained bundles CodeGraph ships a **vendored Node runtime** alongside the app. Because Node 22.5+ has a built-in real SQLite (`node:sqlite`, with WAL + FTS5), bundling Node means: - **No native build** — `better-sqlite3` is gone, so there are zero native addons to compile or rebuild. - **No wasm fallback** — and therefore no more `database is locked` (issue #238). - **No Node-version dependence** — the app always runs on the bundled Node, whatever the user has (or doesn't have) installed. ## What's in a bundle Built by [`scripts/build-bundle.sh`](scripts/build-bundle.sh) — one archive per platform, identical recipe (only the Node download differs): ``` codegraph-<target>/ node | node.exe # official Node runtime for <target> lib/ dist/ # compiled app (+ tree-sitter .wasm grammars, schema.sql) node_modules/ # production deps only (pure JS / wasm — portable) bin/ codegraph | codegraph.cmd # launcher → runs the bundled Node with the app ``` Targets: `darwin-arm64`, `darwin-x64`, `linux-x64`, `linux-arm64`, `win32-x64`, `win32-arm64`. Unix targets produce `.tar.gz` (shell launcher); Windows produces `.zip` (`node.exe` + a `.cmd` launcher). ```bash scripts/build-bundle.sh linux-x64 # -> release/codegraph-linux-x64.tar.gz scripts/build-bundle.sh win32-x64 # -> release/codegraph-win32-x64.zip ``` The Node runtime is the official nodejs.org release named by the script's default `NODE_VERSION` (or its second argument). The download is refused unless it matches the SHA-256 listed in that release's `SHASUMS256.txt`. Because dropping better-sqlite3 left **zero native addons**, building a bundle is pure file-packaging — **any** target builds on **any** OS (the whole matrix builds on one Linux runner). Cross-compilation isn't a concern; only *run-testing* a bundle needs the target platform (or emulation, e.g. `docker run --platform linux/amd64`). ## Install channels (all deliver the same bundle) 1. **`curl | sh`** ([`install.sh`](install.sh)) — no Node required; ideal for a fresh Linux VPS over SSH. Detects os/arch, pulls the archive from GitHub Releases, symlinks `codegraph` onto PATH. Re-run to upgrade; `--uninstall` to remove. 2. **npm** ([`scripts/npm-shim.js`](scripts/npm-shim.js)) — preserves `npm i -g @colbymchenry/codegraph`. The main package is a tiny shim; the bundles ship as per-platform `optionalDependencies` (`@colbymchenry/codegraph-<target>` with `os`/`cpu`), so npm installs only the matching one. The shim — run by the user's Node — execs the bundle, so the real work runs on the bundled Node 24. Works even on old Node. On Windows it invokes the bundled `node.exe` against the app entry directly (not the `.cmd` launcher) — modern Node throws `EINVAL` when asked to spawn a `.cmd`/`.bat`. 3. **Windows** ([`install.ps1`](install.ps1)) — `irm … | iex`; same flow as install.sh (detect arch, pull the `.zip` from Releases, add to PATH). 4. **Homebrew / Scoop** — TODO (tap + cask pointing at the Release archives). ## Release pipeline [`.github/workflows/release.yml`](.github/workflows/release.yml) — manually triggered. Reads the version from `package.json`, builds every platform bundle on one runner, creates the GitHub Release (notes from `CHANGELOG.md`), and publishes the npm shim + per-platform packages. Requires the `NPM_TOKEN` repo secret. Still TODO: - **Code signing** — the main gap for "download & run": macOS Gatekeeper needs a Developer ID + notarization; Windows needs Authenticode. Homebrew softens the macOS case (handles quarantine). - Retire the now-vestigial Node-version gate in `src/bin/codegraph.ts` — the bundle always runs Node 24, and the npm shim does no tree-sitter work. - Re-wire `npm uninstall` cleanup (the agent-config `preuninstall`) through the shim — the generated main package doesn't carry it. - CLAUDE.md (735 bytes)
# CLAUDE.md Claude Code project guidance for this repository. Primary instructions live in the canonical agent guide — import it: @AGENTS.md ## Claude-only notes - Prefer the repo-root `AGENTS.md` as the source of truth for build/test/architecture/house rules. Edit that file (not this wrapper) when guidance changes. - Claude Code can `@`-import nested guides too (e.g. `@docs/AGENTS.md`) when working on design/eval docs; Codex loads nested `AGENTS.md` automatically when the session cwd is under that directory. - Do not reintroduce a duplicated `## CodeGraph` MCP tool-guidance block here — `src/mcp/server-instructions.ts` is the single source of truth (issue #529); the installer strips legacy marker blocks on upgrade.
- CONTRIBUTING.md (5056 bytes)
# Contributing to CodeGraph Thanks for helping out. Small, focused changes are the easiest to review and land. **Reporting a bug?** Open an [issue](https://github.com/colbymchenry/codegraph/issues) with your CodeGraph version (`codegraph --version`), Node.js version, operating system, the steps to reproduce it, what you expected, and what happened instead. A small public repository or snippet that shows the problem helps most. **Asking for a feature?** Describe the use case and why what's there today doesn't cover it. ## Build from source You need Git, npm, and Node.js 22.12 or later, below 25. `npm ci` stops on anything older, because the build and test tools need it and running from source uses Node's built-in `node:sqlite`; the published installers bundle their own Node, so this only matters here. ```bash # Fork the repository on GitHub, then: git clone https://github.com/<your-username>/codegraph.git cd codegraph git remote add upstream https://github.com/colbymchenry/codegraph.git npm ci npm run build ``` A globally installed `codegraph` doesn't run your changes, so run your build directly: ```bash node dist/bin/codegraph.js --help # the build as it stands npm run cli -- status # rebuild first, then run a command npm run dev # recompile TypeScript on every save ``` `npm run build` covers the CLI, the SQL schema, the parser grammars and the browser viewer. Two builds are separate: - `npm run build:kernel` builds the optional native extraction kernel, and needs a Rust toolchain. Tests that need the kernel are skipped without it. - `npm run build:lib` builds the viewer's component library. ## Test your change ```bash npx vitest run __tests__/extraction.test.ts # one file npx vitest run __tests__/extraction.test.ts -t "TypeScript" # tests matching a name npm run test:watch # rerun as you edit npm test # everything ``` Run the focused tests while you work, then `npm test` before opening a pull request. Tests that run the compiled CLI use `dist/`, and the test setup rebuilds it first when it's stale. - **Tests live in `__tests__/`** and use real files and real SQLite databases, with no database mocking. Create temporary folders with `fs.mkdtempSync` and remove them in `afterEach`. - **A bug fix needs a test that fails without the fix.** - **Behavior that differs by platform** gets a gated test, such as `it.runIf(process.platform === 'win32')(...)`, and needs checking on the platforms it affects. See [cross-platform validation](AGENTS.md#cross-platform-validation). - **New language or framework support** needs validating on real repositories. Start with the `/add-lang` skill in [`.claude/skills/add-lang/SKILL.md`](.claude/skills/add-lang/SKILL.md), then follow the [validation methodology](docs/AGENTS.md#validation-methodology-required-for-every-new-languageframework) and the [coverage playbook](docs/design/dynamic-dispatch-coverage-playbook.md). Before extending a router or web framework, read [framework coverage](docs/design/framework-coverage.md). - **A documentation-only change** doesn't need the test suite. Check its examples and links instead. ## Prepare a pull request - Branch from the latest `main`, and keep to one concern. Unrelated formatting, CI or feature work belongs in its own pull request. - Name commits `type(scope): what now works`, such as `fix(go): a constant in a const ( … ) group keeps its own doc comment`. Types in use are `feat`, `fix`, `docs`, `test`, `perf`, `chore` and `ci`. - In the description, explain the problem, the resulting behavior, and what you ran, including any failures and any platform you couldn't test. - Keep existing CLI commands and the default MCP tools working, unless the pull request exists to change them and says why. - Add a line to `CHANGELOG.md` under `## [Unreleased]`, in `### New Features` or `### Fixes`, for anything a user would notice. Write one plain sentence about what changed for them, without internal file names or benchmark numbers. Changes to the `codegraph ui` viewer, which isn't released yet, go in [`docs/viewer-launch-changelog.md`](docs/viewer-launch-changelog.md) instead. - Don't bump the version, and change `package-lock.json` only when you add or remove a dependency. - Use public or made-up examples. Keep local paths, private code and machine-specific settings out of fixtures. ## What happens next A maintainer reads every change closely before anything lands, and may ask for changes. Keep your branch up to date by rebasing on `main`. A maintainer may also land your contribution as their own commit, rewritten where needed, instead of merging your branch. You're credited in the changelog either way. ## Find your way around - [Documentation index](docs/README.md): guides, design notes and benchmarks. - [AGENTS.md](AGENTS.md): the architecture and the development rules, which [CLAUDE.md](CLAUDE.md) imports for Claude Code.
- docs/AGENTS.md (6885 bytes)
# AGENTS.md (docs/) Nested Codex guidance under `docs/`. Loaded with the repo-root `AGENTS.md` when cwd is under `docs/` (Codex walks root to cwd; shared `project_doc_max_bytes` budget). Root `AGENTS.md` already carries the non-negotiable retrieval principles (adapt-the-tool, explore budgets, end-to-end synthesis). This file holds the longer validation methodology and Excalidraw worked example that were moved out of root to fit the budget. ### Validation methodology (REQUIRED for every new language/framework) For each **language × framework**, validate on **small, medium, and large** real repos with **≥3 different flow prompts** each: 1. **Pick the canonical flow** for the framework ("how does X reach Y": state→render, request→handler→view, query→SQL, action→reducer→store…). 2. **Deterministic probes** (`scripts/agent-eval/probe-{node,explore}.mjs` against the built `dist/`): `codegraph_explore` with the flow's symbol names connects from→to end-to-end with no break (its Flow section shows the path); **no node explosion** (`select count(*) from nodes` stable before/after re-index); synthesized-edge **precision** spot-check (`select … where provenance='heuristic'`). 3. **Agent A/B** (`scripts/agent-eval/run-all.sh <repo> "<Q>"`): with vs without codegraph, **≥2 runs/arm** (run-to-run variance is large — never conclude from n=1). Record **duration, total tool calls, Read, Grep**. Optional forced-Read-0 sufficiency proof via the block-read hook (`scripts/agent-eval/hook-settings.json`). - **Every run also reports three feedback metrics** — residual context occupancy, explore sufficiency (what the agent did NEXT after each explore), and allocation efficiency (share of returned bytes the answer cited) — under each run, plus a side-by-side arm table (`compare-arms.mjs`). Entry point: `docs/benchmarks/agent-eval-feedback-metrics.md`. Reading them: `Read a file we returned` is an allocation miss, `Read a file we did NOT return`/`Grep` is recall; allocation efficiency is **relative** (attribution is by citation) so it is only valid between builds on the same question; occupancy *shares* are Claude Code / 200k and don't transfer to another host — the arm ratio does. - **The `codegraph` CLI is blocked in every arm** (`no-cli-shim.sh`: sanitized PATH + a PreToolUse hook, shared by both harnesses). Without it 14 of 15 without-arm runs in one 7-repo pass reached codegraph through Bash. Check the contamination row before believing any number: `CLI calls that RETURNED output` > 0 invalidates the run (in a new-vs-baseline A/B it silently drops calls from all three metrics, since a CLI explore is not a tool call). - **Model policy — every A/B arm runs Claude with `--model sonnet --effort high`. Always. Never Opus/Fable.** All `scripts/agent-eval/*.sh` default to this (`MODEL`/`EFFORT` env override exists — don't raise it without an explicit reason from the maintainer). Two reasons, and the second matters more than cost: (a) Sonnet doesn't burn tokens; (b) **Sonnet is the deliberate floor model** — codegraph's real users attach it to whatever agent they already run (Cursor Composer, Gemini, etc.), so we validate on a "dumber" model on purpose: a stronger model's tool-use covers up the salience/sufficiency problems a weaker one exposes. An affordance that lands on Sonnet generalizes up to every host; one that only works on Opus/Fable doesn't generalize down to the agents most users actually have. Both arms always use the same model. - **MCP attach is a startup-latency issue, not a hard block.** On a multi-step task the agent dives into Read/grep before codegraph finishes its ~2-3s startup (worse when the eval is itself run nested inside a Claude session, under CPU contention), so it runs with no codegraph. Fix: **pre-warm a persistent daemon** for the target (`CODEGRAPH_DAEMON_IDLE_TIMEOUT_MS` high; spawn `serve --mcp --path <target> </dev/null &`; wait for `.codegraph/daemon.sock`) **and skip the startup re-exec** (`CODEGRAPH_WASM_RELAUNCHED=1`) so claude connects before the agent's first turn. Don't trust claude's `init` snapshot — it can read `status:"pending"` / 0 tools even when it then connects; judge by actual codegraph usage in `parse-run.mjs`'s `by type`. To isolate a change — **new-build vs baseline-build, both codegraph-on** (vs run-all.sh's with-vs-without) — use `scripts/agent-eval/ab-new-vs-baseline.sh <indexed-repo> "<task>" [baseline-ref]` (it bakes in the pre-warm). 4. **Pass bar:** a normal flow question reaches **~0 Read/Grep within the repo's explore-call budget**, runs **faster** than without-codegraph, and shows **no regression on a control repo**. Record the numbers in `docs/design/dynamic-dispatch-coverage-playbook.md` (the coverage matrix). Full playbook + per-mechanism design: `docs/design/dynamic-dispatch-coverage-playbook.md` and `docs/design/callback-edge-synthesis.md`. ### Worked example — Excalidraw (TS/React, medium, 643 files) The template to replicate per language/framework. Question: *"how does updating an element re-render the canvas on screen?"* (the full flow crosses three React boundaries: observer callback, `setState`→`render`, and JSX child). | Stage | duration | Read | Grep | codegraph | |---|---|---|---|---| | Without codegraph | 115–139s | 9–10 | 10–11 | 0 | | Broken (explore-budget regression) | 131–139s | 5–10 | 3–5 | 6–14 | | Fixed (budget + msgs + synthesis) | 64–112s | 0–2 | 2–4 | 3–**10** | | + trace-first steering | **51–74s** | **0–2** | 0–4 | **3–4** | n=4 unhooked runs/stage, same prompt. After steering flow questions to `codegraph_trace` first: **best run 0 Read / 0 Grep / 3 codegraph / 51s**; **2 of 4 fully clean** (0 Read, 0 Grep). Steering eliminated the over-drill variance — call count tightened from 3–10 to 3–4, trace adoption went 3/4 → 4/4, and the `search`+`callers` path-reconstruction floundering dropped to 0. Run-to-run variance is still real; report the range, never a single run. **Residual reads/greps are all the nonce data-flow** (`canvasNonce` — a local prop with no graph edges); that's the def-use/data-flow frontier, left deliberately uncovered (tracking every local would explode the graph). Validated: `trace(mutateElement, renderStaticScene)` connects in **6 hops** across all three boundaries (`mutateElement → triggerUpdate → [callback] triggerRender → [react-render] render → [jsx] StaticCanvas → renderStaticScene`), each hop showing inline source + the wiring site; node count stable at 9,289; 1 callback + 46 react-render + 280 jsx-render synthesized edges (no explosion, precision-checked). Also see: `docs/design/dynamic-dispatch-coverage-playbook.md`, `docs/design/callback-edge-synthesis.md`, `docs/benchmarks/call-sequence-analysis.md`, `docs/benchmarks/agent-eval-feedback-metrics.md`. - install.sh (5917 bytes)
#!/bin/sh # # CodeGraph standalone installer. # # Downloads a self-contained bundle (a vendored Node runtime + the app) from # GitHub Releases. No Node.js, no build tools, no npm required — ideal for a # fresh Linux VPS over SSH. # # curl -fsSL https://raw.githubusercontent.com/colbymchenry/codegraph/main/install.sh | sh # # Upgrade: run `codegraph upgrade` (or just re-run the same command). # Uninstall: curl -fsSL .../install.sh | sh -s -- --uninstall # # Environment: # CODEGRAPH_VERSION release tag to install (default: latest) # CODEGRAPH_INSTALL_DIR bundle location (default: ~/.codegraph) # CODEGRAPH_BIN_DIR symlink location (default: ~/.local/bin) set -eu REPO="colbymchenry/codegraph" INSTALL_DIR="${CODEGRAPH_INSTALL_DIR:-$HOME/.codegraph}" BIN_DIR="${CODEGRAPH_BIN_DIR:-$HOME/.local/bin}" if [ "${1:-}" = "--uninstall" ]; then rm -f "$BIN_DIR/codegraph" rm -rf "$INSTALL_DIR" echo "CodeGraph uninstalled (removed $INSTALL_DIR and $BIN_DIR/codegraph)." exit 0 fi # 1. Detect platform → target triple matching the release archives. os="$(uname -s)" arch="$(uname -m)" case "$os" in Darwin) os="darwin" ;; Linux) os="linux" ;; MINGW*|MSYS*|CYGWIN*) echo "codegraph: on Windows, open PowerShell and run:" >&2 echo " irm https://raw.githubusercontent.com/colbymchenry/codegraph/main/install.ps1 | iex" >&2 exit 1 ;; *) echo "codegraph: unsupported OS '$os'." >&2; exit 1 ;; esac case "$arch" in arm64|aarch64) arch="arm64" ;; x86_64|amd64) arch="x64" ;; *) echo "codegraph: unsupported architecture '$arch'." >&2; exit 1 ;; esac target="${os}-${arch}" # 2. Resolve the version (latest release unless pinned). # # Resolve "latest" from the releases/latest *web* redirect, not the GitHub API: # the unauthenticated API is rate-limited to 60 requests/hour per IP and returns # 403 once exhausted — routine on shared/cloud hosts and CI (issue #325). The # redirect (github.com/<repo>/releases/latest -> .../releases/tag/vX.Y.Z) has no # such limit. Fall back to the API if the redirect can't be read. version="${CODEGRAPH_VERSION:-}" if [ -z "$version" ]; then version="$(curl -fsSLI -o /dev/null -w '%{url_effective}' "https://github.com/$REPO/releases/latest" \ | sed -n 's#.*/releases/tag/##p')" fi if [ -z "$version" ]; then version="$(curl -fsSL "https://api.github.com/repos/$REPO/releases/latest" \ | sed -n 's/.*"tag_name": *"\([^"]*\)".*/\1/p' | head -n1)" fi [ -n "$version" ] || { echo "codegraph: could not resolve latest version; set CODEGRAPH_VERSION (e.g. CODEGRAPH_VERSION=v0.9.4)." >&2; exit 1; } # Release tags are vX.Y.Z; accept a bare X.Y.Z in CODEGRAPH_VERSION too. case "$version" in v*) ;; *) version="v$version" ;; esac # 3. Download + extract the bundle. url="https://github.com/$REPO/releases/download/$version/codegraph-${target}.tar.gz" echo "Installing CodeGraph $version ($target)..." tmp="$(mktemp -d)" trap 'rm -rf "$tmp"' EXIT curl -fsSL "$url" -o "$tmp/cg.tar.gz" || { echo "codegraph: download failed: $url" >&2; exit 1; } dest="$INSTALL_DIR/versions/$version" rm -rf "$dest" mkdir -p "$dest" # Archives contain a top-level codegraph-<target>/ dir; strip it. tar -xzf "$tmp/cg.tar.gz" -C "$dest" --strip-components=1 # 4. Symlink the launcher onto PATH and mark the current version. mkdir -p "$BIN_DIR" ln -sf "$dest/bin/codegraph" "$BIN_DIR/codegraph" ln -sfn "$dest" "$INSTALL_DIR/current" echo "Installed to $dest" echo "Linked $BIN_DIR/codegraph" # 5. Prune older bundles so they don't pile up across upgrades (issue #1074). # Each release lives in its own versions/<v> dir (~50 MB with the vendored Node # runtime). `codegraph upgrade` re-runs this script, which drops in a new dir # and re-points `current` + the launcher — but it never removed the old dirs, so # they accumulated indefinitely. Keep only what we just installed ($dest) and # delete the rest. Safe even if a daemon is still executing an older bundle: on # POSIX the inode stays alive until that process exits, so removing the dir can't # break a running process. (Windows installs overwrite a single dir in place and # never reach this.) The markers below let a unit test run this exact block. # >>> CODEGRAPH_PRUNE_OLD_VERSIONS pruned=0 if [ -d "$INSTALL_DIR/versions" ]; then for d in "$INSTALL_DIR/versions"/*; do [ -d "$d" ] || continue if [ "$d" != "$dest" ]; then if rm -rf "$d"; then pruned=$((pruned + 1)) fi fi done fi if [ "$pruned" -gt 0 ]; then echo "Removed $pruned older version(s)" fi # <<< CODEGRAPH_PRUNE_OLD_VERSIONS # 6. PATH sanity. Two ways this install can fail to be the codegraph that runs: # 1. $BIN_DIR isn't on PATH at all. # 2. A *different* codegraph sits earlier on PATH and shadows ours — most # often a stale `npm i -g @colbymchenry/codegraph`, whose launcher keeps # running its own version-pinned bundle, so `codegraph --version` disagrees # with what we just installed (issue #1071). # Walk PATH once: note whether $BIN_DIR is present and which codegraph wins. on_path=0 winner="" oldifs="$IFS"; IFS=: for dir in $PATH; do [ -n "$dir" ] || continue if [ "$dir" = "$BIN_DIR" ]; then on_path=1; fi if [ -z "$winner" ] && [ -x "$dir/codegraph" ] && [ ! -d "$dir/codegraph" ]; then winner="$dir/codegraph" fi done IFS="$oldifs" if [ "$on_path" -eq 0 ]; then echo "" echo "$BIN_DIR is not on your PATH. Add it:" echo " export PATH=\"$BIN_DIR:\$PATH\"" elif [ -n "$winner" ] && [ "$winner" != "$BIN_DIR/codegraph" ]; then echo "" echo "Warning: another codegraph is earlier on your PATH and will run instead:" echo " $winner" echo " (this install: $BIN_DIR/codegraph)" echo "If 'codegraph --version' shows an unexpected version, remove the other copy" echo "(e.g. 'npm rm -g @colbymchenry/codegraph') or put $BIN_DIR first on PATH." fi echo "" echo "Done. Run: codegraph --help" - INSTRUCTIONS.md (3840 bytes)
--- name: agent-eval description: Benchmark CodeGraph retrieval quality on a real codebase by comparing agent behavior with vs without CodeGraph. Use when the user runs /agent-eval or asks to test, benchmark, audit, or validate a codegraph version (the local dev build or a published npm version) against a language's repo. --- # CodeGraph Quality Audit Measures how much CodeGraph helps an agent versus plain grep/read, for a chosen codegraph version on a chosen real-world repo. Drives the harness in `scripts/agent-eval/`. ## Prerequisites - `tmux` 3+, a logged-in `claude` CLI, `node`, `git` (macOS/Linux). - Run from the codegraph repo root. ## Workflow Copy this checklist: ``` - [ ] 1. Pick version (local or npm) - [ ] 2. Pick language - [ ] 3. Pick repo by size - [ ] 4. Pick harness (headless / tmux / both) - [ ] 5. Run audit.sh in the background - [ ] 6. Report results ``` **Step 1 — version.** Ask with `AskUserQuestion`: which codegraph version to test. Offer "Local dev build" and "Latest published"; the free-text "Other" lets the user type a specific version (e.g. `0.7.10`). Map the answer to a VERSION token: - "Local dev build" → `local` - "Latest published" → `latest` - a typed version → that string (e.g. `0.7.10`) **Step 2 — language.** Read `.claude/skills/agent-eval/corpus.json`. Ask with `AskUserQuestion` which language to test, listing the languages that have entries. **Step 3 — repo.** From the chosen language's entries, ask which repo. Label each option with its size and file count, e.g. `excalidraw — Medium (~600 files)`. Each entry carries the `repo` URL and a representative `question`. **Step 4 — harness.** Ask with `AskUserQuestion` which harness to run, and map the answer to a MODE token: - "Headless" → `headless` — `claude -p` with stream-json: exact tokens/cost and a clean tool sequence (2 runs, fast, no TTY). - "Interactive (tmux)" → `tmux` — drives the real Claude TUI in tmux: faithful Explore-subagent behavior, metrics from session logs (2 runs, slower). - "Both" → `all` — headless + interactive (4 runs). **Step 5 — run.** Launch in the background (sets the version, clones if missing, wipes + re-indexes, runs the chosen arms — several minutes): ```bash scripts/agent-eval/audit.sh <VERSION> <repo-name> <repo-url> "<question>" <MODE> ``` **Step 6 — report.** When the job finishes, read the log and report per arm: - Headless (`parse-run.mjs`): total tool calls, file `Read`s, Grep/Bash, codegraph-tool calls, duration, **total cost**. - Interactive (`parse-session.mjs`): the `VERDICT: codegraph_explore used Nx | Read N | Grep/Bash N` and `TOKENS:` lines. - Both paths also print the three feedback metrics — residual context occupancy, explore sufficiency, allocation efficiency — and a headless A/B ends with a side-by-side `ARM COMPARISON` table. Report that table, and check its contamination row first: `CLI calls that RETURNED output` > 0 means the arm reached codegraph through Bash and its numbers are void. How to read the rest: `docs/benchmarks/agent-eval-feedback-metrics.md`. Lead with cost + tool/Read counts — they are the reliable signals; raw token in/out are confounded by subagent delegation and prompt caching. State whether codegraph reduced effort and whether both arms reached a correct answer. ## Notes - The index is rebuilt every run (`audit.sh` wipes `.codegraph`) — different versions extract differently, so an index must be served by the same binary that built it. - `audit.sh` temporarily mutates the global `codegraph` install for the test, then restores your dev link via `local-install.sh`. - Corpus repos are cloned to `/tmp/codegraph-corpus` (reused if already present). - Add or edit repos in `corpus.json` (fields: `name`, `repo`, `size`, `files`, `question`).
- LICENSE (1070 bytes)
MIT License Copyright (c) 2026 Colby Mchenry Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions: The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software. THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
- package.json (2540 bytes)
{ "name": "@colbymchenry/codegraph", "version": "1.6.4", "description": "Supercharge AI coding agents with semantic code intelligence — surgical context, fewer tool calls, faster answers. 100% local.", "repository": { "type": "git", "url": "git+https://github.com/colbymchenry/codegraph.git" }, "main": "dist/index.js", "types": "dist/index.d.ts", "bin": { "codegraph": "./dist/bin/codegraph.js" }, "files": [ "dist", "scripts", "README.md" ], "workspaces": [ "ui" ], "devEngines": { "runtime": { "name": "node", "version": ">=22.12.0", "onFail": "error" } }, "scripts": { "build": "tsc && tsc --emitDeclarationOnly --noCheck --removeComments false && npm run copy-assets && npm run build:ui && node -e \"require('fs').chmodSync('dist/bin/codegraph.js', 0o755)\"", "build:ui": "cd ui && npm run build && cd .. && node scripts/check-ui-build.mjs", "build:lib": "cd ui && npm run build:lib", "preuninstall": "node dist/bin/uninstall.js", "copy-assets": "node -e \"const fs=require('fs');fs.mkdirSync('dist/db',{recursive:true});fs.copyFileSync('src/db/schema.sql','dist/db/schema.sql');fs.mkdirSync('dist/extraction/wasm',{recursive:true});fs.readdirSync('src/extraction/wasm').filter(f=>f.endsWith('.wasm')).forEach(f=>fs.copyFileSync('src/extraction/wasm/'+f,'dist/extraction/wasm/'+f))\"", "dev": "tsc --watch", "build:kernel": "bash scripts/build-kernel.sh", "cli": "npm run build && node dist/bin/codegraph.js", "test": "vitest run", "test:watch": "vitest", "test:eval": "vitest run __tests__/evaluation/", "eval": "npm run build && npx tsx __tests__/evaluation/runner.ts", "clean": "node -e \"const fs=require('fs');fs.rmSync('dist',{recursive:true,force:true})\"" }, "keywords": [ "code-intelligence", "knowledge-graph", "static-analysis" ], "author": "", "license": "MIT", "dependencies": { "@clack/prompts": "^1.3.0", "commander": "^14.0.2", "fast-string-width": "^3.0.2", "fast-wrap-ansi": "^0.2.0", "ignore": "^7.0.14", "jsonc-parser": "^3.3.1", "picomatch": "^4.0.3", "sisteransi": "^1.0.5", "tree-sitter-wasms": "^0.1.11", "web-tree-sitter": "^0.25.3" }, "devDependencies": { "@sveltejs/vite-plugin-svelte": "^6.2.4", "@types/node": "^20.19.30", "@types/picomatch": "^4.0.2", "jsdom": "^25.0.1", "typescript": "^5.0.0", "vitest": "^4.1.11" }, "engines": { "node": ">=20.0.0 <25.0.0" } } - README.md (79958 bytes)
<div align="center"> # CodeGraph Already installed? Run `codegraph upgrade` Follow [@getcodegraph](https://x.com/getcodegraph) on X for updates. ### Supercharge Claude Code, Cursor, Codex, OpenCode, Hermes Agent, Gemini, Antigravity, Kiro, GitHub Copilot, and more with Semantic Code Intelligence **The fastest complete code graph · surgical context · built for how agents actually work · 100% local** <picture> <source media="(prefers-color-scheme: dark)" srcset="https://raw.githubusercontent.com/colbymchenry/codegraph/main/assets/rust-logo-dark.svg?v=1"> <img src="https://raw.githubusercontent.com/colbymchenry/codegraph/main/assets/rust-logo.svg?v=1" height="30" alt="Rust" align="center"> </picture> **Kernel powered by Rust** ### [Documentation & Website →](https://colbymchenry.github.io/codegraph/) [](https://www.npmjs.com/package/@colbymchenry/codegraph) [](https://opensource.org/licenses/MIT) [](https://nodejs.org/) [](#verified-releases) [](#verified-releases) [](#supported-platforms) [](#supported-platforms) [](#supported-platforms) [](#supported-agents) [](#supported-agents) [](#supported-agents) [](#supported-agents) [](#supported-agents) [](#supported-agents) [](#supported-agents) [](#supported-agents) [](#supported-agents) [](#supported-agents) [](#supported-agents) [](#supported-agents) [](#supported-agents) [](#supported-agents) [](#supported-agents) [](#supported-agents) [](#supported-agents) [](#supported-agents) <br> **The CodeGraph platform is coming** — for every PR, know exactly what to test, what could break, which flows are affected, and whether business logic is compromised. <a href="https://getcodegraph.com"><img alt="Join the waitlist for early beta access" src="https://raw.githubusercontent.com/colbymchenry/codegraph/main/assets/waitlist.svg?v=2" height="52"></a> <sub>Get <b>early beta access</b> to the hosted product · <a href="https://getcodegraph.com">getcodegraph.com</a></sub> </div> ## Contents - [Get Started](#get-started) - [Language Support](#language-support) - [Why CodeGraph?](#why-codegraph) - [Key Features](#key-features) - [Framework-aware Routes](#framework-aware-routes) - [Mixed iOS / React Native / Expo bridging](#mixed-ios--react-native--expo-bridging) - [Quick Start](#quick-start) - [How It Works](#how-it-works) - [CLI Reference](#cli-reference) - [MCP Tools](#mcp-tools) - [Library Usage](#library-usage) - [Configuration](#configuration) - [Telemetry](#telemetry) - [Verified releases](#verified-releases) - [Supported Platforms](#supported-platforms) - [Supported Agents](#supported-agents) - [Supported Languages](#supported-languages) - [Measured cross-file coverage](#measured-cross-file-coverage) - [Troubleshooting](#troubleshooting) - [Contributing](#contributing) - [License](#license) ## Get Started ### 1. Install the CLI **No Node.js required** — one command grabs the right build for your OS: ```bash # macOS / Linux curl -fsSL https://raw.githubusercontent.com/colbymchenry/codegraph/main/install.sh | sh # Windows (PowerShell) irm https://raw.githubusercontent.com/colbymchenry/codegraph/main/install.ps1 | iex ``` <details> <summary><b>Already have Node? Use npm instead (works on any version)</b></summary> ```bash npm i -g @colbymchenry/codegraph ``` <sub>CodeGraph bundles its own runtime — nothing to compile, no native build, works the same everywhere. The installer puts `codegraph` on your PATH but **doesn't change your current shell** — open a new terminal before the next step so the command resolves.</sub> <sub>**Upgrade any time** with `codegraph upgrade` — it detects how you installed (bundle, npm, or npx) and updates in place. Add `--check` to see if an update is available, or `codegraph upgrade <version>` to pin one. After an upgrade, `codegraph status` and your agent tell you when a project's index was built by an older version: run `codegraph index` there to rebuild it.</sub> </details> ### 2. Wire up your agent(s) In a **new terminal**, run the installer to connect CodeGraph to the agents you use: ```bash codegraph install ``` <sub>Detects and auto-configures Claude Code, Cursor, Codex CLI, opencode, Hermes Agent, Gemini CLI, Antigravity IDE, Kiro, GitHub Copilot (VS Code, Copilot CLI, JetBrains IDEs), IBM Bob, Factory Droid, Windsurf, Qwen Code, DeepSeek Harness, Grok Build, Kimi Code, Qoder CLI, and Pi — wiring the CodeGraph MCP server into each. **This is the step that connects CodeGraph to your agent;** installing the CLI in step 1 does not do it on its own. It only wires up your agent — it does **not** index any code; building each project's graph is the separate `codegraph init` in step 3. (Shortcut: `npx @colbymchenry/codegraph` downloads and runs this in one go.)</sub> ### 3. Initialize each project ```bash cd your-project codegraph init ``` <sub>`codegraph init` creates the local `.codegraph/` directory and builds the full graph in the same step — one command, done.</sub> <div align="center">  </div> ### 4. No more syncing! Auto-sync is enabled by default. CodeGraph watches the project and updates the graph on every file change — while your agent edits code, or you add, modify, or delete files. **The index is never stale, and there is nothing to re-run.** ### Uninstall Changed your mind? One command removes CodeGraph from every agent it configured **and** the CLI itself — every install it finds (standalone bundle, npm global package, launcher link), shown to you before anything is deleted: ```bash codegraph uninstall ``` Pass `--keep-cli` to remove only the agent configurations and keep the CLI installed. <sub>Reverses the installer — strips CodeGraph's MCP server config, instructions, and permissions from each configured agent. Your project indexes (`.codegraph/`) are left untouched; remove those per-project with `codegraph uninit`. Use `--target` to remove from specific agents, or `--yes` to run non-interactively.</sub> --- ## Language Support Every language below gets the same treatment — full structural extraction and cross-file resolution into one graph, no per-language setup: <p align="center"> <img src="https://raw.githubusercontent.com/colbymchenry/codegraph/main/assets/languages/typescript.svg?v=1" width="104" height="104" alt="TypeScript" /> <img src="https://raw.githubusercontent.com/colbymchenry/codegraph/main/assets/languages/javascript.svg?v=1" width="104" height="104" alt="JavaScript" /> <img src="https://raw.githubusercontent.com/colbymchenry/codegraph/main/assets/languages/arkts.svg?v=1" width="104" height="104" alt="ArkTS" /> <img src="https://raw.githubusercontent.com/colbymchenry/codegraph/main/assets/languages/python.svg?v=1" width="104" height="104" alt="Python" /> <img src="https://raw.githubusercontent.com/colbymchenry/codegraph/main/assets/languages/go.svg?v=1" width="104" height="104" alt="Go" /> <img src="https://raw.githubusercontent.com/colbymchenry/codegraph/main/assets/languages/rust.svg?v=1" width="104" height="104" alt="Rust" /> <img src="https://raw.githubusercontent.com/colbymchenry/codegraph/main/assets/languages/java.svg?v=1" width="104" height="104" alt="Java" /> <img src="https://raw.githubusercontent.com/colbymchenry/codegraph/main/assets/languages/csharp.svg?v=1" width="104" height="104" alt="C#" /> <img src="https://raw.githubusercontent.com/colbymchenry/codegraph/main/assets/languages/php.svg?v=1" width="104" height="104" alt="PHP" /> <img src="https://raw.githubusercontent.com/colbymchenry/codegraph/main/assets/languages/ruby.svg?v=1" width="104" height="104" alt="Ruby" /> <img src="https://raw.githubusercontent.com/colbymchenry/codegraph/main/assets/languages/c.svg?v=1" width="104" height="104" alt="C" /> <img src="https://raw.githubusercontent.com/colbymchenry/codegraph/main/assets/languages/cpp.svg?v=1" width="104" height="104" alt="C++" /> <img src="https://raw.githubusercontent.com/colbymchenry/codegraph/main/assets/languages/objective-c.svg?v=1" width="104" height="104" alt="Objective-C" /> <img src="https://raw.githubusercontent.com/colbymchenry/codegraph/main/assets/languages/metal.svg?v=1" width="104" height="104" alt="Metal" /> <img src="https://raw.githubusercontent.com/colbymchenry/codegraph/main/assets/languages/cuda.svg?v=1" width="104" height="104" alt="CUDA" /> <img src="https://raw.githubusercontent.com/colbymchenry/codegraph/main/assets/languages/swift.svg?v=1" width="104" height="104" alt="Swift" /> <img src="https://raw.githubusercontent.com/colbymchenry/codegraph/main/assets/languages/kotlin.svg?v=1" width="104" height="104" alt="Kotlin" /> <img src="https://raw.githubusercontent.com/colbymchenry/codegraph/main/assets/languages/scala.svg?v=1" width="104" height="104" alt="Scala" /> <img src="https://raw.githubusercontent.com/colbymchenry/codegraph/main/assets/languages/dart.svg?v=1" width="104" height="104" alt="Dart" /> <img src="https://raw.githubusercontent.com/colbymchenry/codegraph/main/assets/languages/svelte.svg?v=1" width="104" height="104" alt="Svelte" /> <img src="https://raw.githubusercontent.com/colbymchenry/codegraph/main/assets/languages/vue.svg?v=1" width="104" height="104" alt="Vue" /> <img src="https://raw.githubusercontent.com/colbymchenry/codegraph/main/assets/languages/astro.svg?v=1" width="104" height="104" alt="Astro" /> <img src="https://raw.githubusercontent.com/colbymchenry/codegraph/main/assets/languages/liquid.svg?v=1" width="104" height="104" alt="Liquid" /> <img src="https://raw.githubusercontent.com/colbymchenry/codegraph/main/assets/languages/delphi.svg?v=1" width="104" height="104" alt="Pascal / Delphi" /> <img src="https://raw.githubusercontent.com/colbymchenry/codegraph/main/assets/languages/lua.svg?v=1" width="104" height="104" alt="Lua" /> <img src="https://raw.githubusercontent.com/colbymchenry/codegraph/main/assets/languages/r.svg?v=1" width="104" height="104" alt="R" /> <img src="https://raw.githubusercontent.com/colbymchenry/codegraph/main/assets/languages/luau.svg?v=1" width="104" height="104" alt="Luau" /> <img src="https://raw.githubusercontent.com/colbymchenry/codegraph/main/assets/languages/cfml.svg?v=1" width="104" height="104" alt="CFML" /> <img src="https://raw.githubusercontent.com/colbymchenry/codegraph/main/assets/languages/cobol.svg?v=1" width="104" height="104" alt="COBOL" /> <img src="https://raw.githubusercontent.com/colbymchenry/codegraph/main/assets/languages/vbnet.svg?v=1" width="104" height="104" alt="Visual Basic .NET" /> <img src="https://raw.githubusercontent.com/colbymchenry/codegraph/main/assets/languages/erlang.svg?v=1" width="104" height="104" alt="Erlang" /> <img src="https://raw.githubusercontent.com/colbymchenry/codegraph/main/assets/languages/solidity.svg?v=1" width="104" height="104" alt="Solidity" /> <img src="https://raw.githubusercontent.com/colbymchenry/codegraph/main/assets/languages/terraform.svg?v=1" width="104" height="104" alt="Terraform / OpenTofu" /> <img src="https://raw.githubusercontent.com/colbymchenry/codegraph/main/assets/languages/nix.svg?v=1" width="104" height="104" alt="Nix" /> </p> <sub>Per-language details — extensions, frameworks, and what exactly gets extracted — in [Supported Languages](#supported-languages).</sub> --- ## Why CodeGraph? When an AI agent needs to understand code — to answer a question or make a change — it discovers structure the slow way: grep, glob, and Read, one file at a time, rebuilding call paths and dependencies by hand. That's a pile of tool calls and round-trips before it even starts the real work. **CodeGraph hands the agent the exact code it needs in one call.** It's a pre-built knowledge graph of every symbol, call edge, and dependency in your codebase — so instead of crawling files, the agent asks one question and gets back the relevant source, the call paths between those symbols (including dynamic-dispatch hops grep can't follow), and the blast radius of a change. **Surgical context, not a file-by-file search** — which means fewer tool calls and faster answers on every codebase, large or small. <img width="1536" height="1024" alt="token-cost-savings-scale" src="https://github.com/user-attachments/assets/eb74a11a-a3ab-4b01-80a6-19f78352ae8e" /> > **A note on cost:** CodeGraph's win on *every* codebase is precision — the agent stops crawling files and answers from the graph. On current models that precision is also a large direct saving: the 2026-08 re-measurement, on a harness that blocks the CLI in both arms, put it at **44% lower cost and 62% fewer tokens on average** across the seven benchmark repos, because a strong model *without* the graph burns its budget re-deriving structure. Cost tracks how much *discovery* a question demands more than raw repo size: 57–78% on questions the file-reading agent needed 28–43 tool calls to answer, near-even where it got there in 7. > **A
- SKILL.md (11070 bytes)
--- name: add-lang description: Add tree-sitter language support to codegraph end-to-end — wire the grammar + extractor, write tests, then benchmark extraction quality and retrieval value on 3 popular real-world repos. Use when the user runs /add-lang <language> or asks to add/support a new language (e.g. Lua, Elixir, Zig, OCaml) in codegraph. --- # Add a language to CodeGraph Wire a new tree-sitter language into codegraph's extraction pipeline, prove it extracts real symbols on popular repos, and prove it beats no-codegraph for an agent. Runs **fully autonomously** — pick repos, benchmark, update docs, then report. **Never commit, push, publish, or tag** (house rule); leave all changes for the user to review. The argument is the language token used throughout the `Language` union, e.g. `lua`, `elixir`, `zig`. If none was given, ask which language. Use the lowercase single-token form everywhere (`csharp`, not `c#`). ## Prerequisites - Run from the codegraph repo root. `node`, `git`, `gh`, and a logged-in `claude` CLI (the benchmark spawns real `claude -p` runs). - The benchmark uses the local dev build — Step 8 builds + links it on PATH. ## Workflow Copy this checklist and work through it in order: ``` - [ ] 1. Resolve language; bail early if already supported (just benchmark) - [ ] 2. Find a grammar + health-check it (ABI / heap corruption) - [ ] 3. Discover the grammar's AST node types (dump-ast.mjs) - [ ] 4. Wire the language (4 files; sometimes a 5th core touch) - [ ] 5. Build + verify-extraction loop until PASS - [ ] 6. Add extraction tests; make them green - [ ] 7. Auto-pick 3 popular repos by size tier; add to corpus.json - [ ] 8. Benchmark all 3: extraction + with/without A/B - [ ] 9. Update README + CHANGELOG - [ ] 10. Report; do NOT commit ``` ### Step 1 — Resolve + short-circuit Check whether the language is already wired: look for the token in the `LANGUAGES` const (`src/types.ts`) and the `EXTRACTORS` map (`src/extraction/languages/index.ts`). If it is already supported (e.g. `typescript`, `rust`), **skip Steps 2–6** and go straight to benchmarking (Steps 7–8) to validate/measure it — note in the report that no code changed. ### Step 2 — Find a grammar, then health-check it ```bash ls node_modules/tree-sitter-wasms/out/ | grep -i <lang> # csharp -> c_sharp ``` - **Present** → likely off-the-shelf; `grammars.ts` resolves it from `tree-sitter-wasms` automatically. (Many languages: elixir, zig, ocaml, solidity, toml, yaml, …) - **Absent** → vendor a `.wasm` into `src/extraction/wasm/` (like `pascal` / `scala` / `lua`) and add the token to the vendored branch in Step 4. **Always health-check before writing an extractor — a *present* grammar can still be unusable:** ```bash node scripts/add-lang/check-grammar.mjs <lang> path/to/valid-sample.<ext> ``` It prints the grammar's ABI version and runs it the way indexing does: in a child process with indexing's V8 flags, beside the vendored grammars, parsing a valid sample many times and once at the 1 MiB file-size limit. If it **FAILs** (ERROR trees, or later parses giving a different tree, on valid code — an old ABI corrupting the shared WASM heap, which silently drops nested calls/imports on every file after the first; e.g. the tree-sitter-wasms **Lua** grammar is ABI 13 and fails — or the process running it crashing or hanging, even after every parse finished), do NOT use that wasm. **Vendor a newer (ABI 14/15) build instead:** ```bash npm pack @tree-sitter-grammars/tree-sitter-<lang> # often ships a prebuilt *.wasm # or build one: npx tree-sitter build --wasm (needs Docker/emscripten) cp <the>.wasm src/extraction/wasm/tree-sitter-<lang>.wasm ``` then add the token to the vendored branch in Step 4 and re-run check-grammar on the vendored path until it PASSes. **If you cannot obtain a healthy wasm, STOP and tell the user.** ### Step 3 — Discover AST node types Get a representative source file (write a small sample covering functions, classes/structs, imports, enums; or `curl` a raw file from a known repo), then: ```bash node scripts/add-lang/dump-ast.mjs <lang> path/to/sample.<ext> # vendored grammar: pass the wasm path instead of the token node scripts/add-lang/dump-ast.mjs src/extraction/wasm/tree-sitter-<lang>.wasm sample.<ext> ``` The frequency table + field names (`name:`, `parameters:`, `body:`, `return_type:`) tell you what to map. Open the existing extractor closest to the language's paradigm as a model: `rust.ts`/`scala.ts` (functional, traits), `java.ts`/`csharp.ts` (OO), `python.ts`/`ruby.ts` (scripting), `go.ts` (top-level methods + receivers). ### Step 4 — Wire the language (4 files) These are exact, fragile wiring — match the existing style precisely: 1. **`src/types.ts`** — TWO edits: - add `'<lang>',` to the `LANGUAGES` const (before `'unknown'`); - add `'**/*.<ext>',` to `DEFAULT_CONFIG.include`. **Don't skip this** — it's the file-scan allowlist; without the glob, `codegraph init` finds **0 files** even though detection/extraction are wired. 2. **`src/extraction/grammars.ts`** — three maps: - `WASM_GRAMMAR_FILES`: `<lang>: 'tree-sitter-<lang>.wasm',` - `EXTENSION_MAP`: each file extension → `'<lang>'` (e.g. `'.lua': 'lua',`) - `getLanguageDisplayName`: `<lang>: '<Display Name>',` - **vendored only**: add `<lang>` to the `(lang === 'pascal' || lang === 'scala' || …)` wasm-path branch. 3. **`src/extraction/languages/<lang>.ts`** — new file exporting `export const <lang>Extractor: LanguageExtractor = { … }`. Map the node types from Step 3. Required fields: `functionTypes`, `classTypes`, `methodTypes`, `interfaceTypes`, `structTypes`, `enumTypes`, `typeAliasTypes`, `importTypes`, `callTypes`, `variableTypes`, `nameField`, `bodyField`, `paramsField`. Add hooks as the grammar needs them (`getSignature`, `getVisibility`, `isExported`, `extractImport`, `visitNode`, `getReceiverType`, `interfaceKind`, `enumMemberTypes`, etc. — see `src/extraction/tree-sitter-types.ts`). 4. **`src/extraction/languages/index.ts`** — `import { <lang>Extractor } from './<lang>';` and add `<lang>: <lang>Extractor,` to `EXTRACTORS`. **Sometimes a 5th, core touch in `src/extraction/tree-sitter.ts`** — variable extraction has per-language branches in `extractVariable` (the generic fallback only finds direct `identifier`/`variable_declarator` children). If the grammar nests declared names (e.g. Lua's `variable_declaration → variable_list`), add a `} else if (this.language === '<lang>')` branch there, mirroring the existing ts/python/go ones. Import forms that aren't a distinct node (Lua/Ruby `require` is a *call*) are handled in the extractor's `visitNode` hook instead. ### Step 5 — Build + verify loop ```bash npm run build # tsc + copy-assets (copies any vendored *.wasm into dist/) ``` Index a small sample repo and check extraction: ```bash ( cd <sample-repo> && codegraph init ) node scripts/add-lang/verify-extraction.mjs <sample-repo> <lang> ``` `verify-extraction.mjs` fails (exit 1) if the language isn't detected or only `file`/`import` nodes were produced — the classic symptom of wrong node-type names. On FAIL or a thin WARN: re-run `dump-ast.mjs` on a richer file, fix the mappings in `<lang>.ts`, `npm run build`, re-index, re-verify. **Repeat until PASS.** ### Step 6 — Tests Add to `__tests__/extraction.test.ts`, modeled on the `Rust Extraction` block: - a `detectLanguage` assertion in `describe('Language Detection')` - a `describe('<Lang> Extraction')` block asserting functions/classes/imports are extracted from an inline source string. ```bash npx vitest run __tests__/extraction.test.ts ``` Green before continuing. ### Step 7 — Auto-pick 3 repos + corpus Pick **without asking**. Find candidates, then curate 3 that are genuinely `<lang>`-dominant, one per size tier: ```bash gh search repos --language=<lang> --sort=stars --limit 40 \ --json fullName,stargazerCount,description ``` Tiers (match `corpus.json`): **Small** <~150 files · **Medium** ~150–1500 · **Large** >~1500. Skip repos that are tagged `<lang>` but mostly another language. Write one cross-file architecture **question** per repo (the kind that needs tracing across files). Add a `"<Language>"` block to `.claude/skills/agent-eval/corpus.json` (fields: `name`, `repo`, `size`, `files`, `question`) so `/agent-eval` can reuse them. ### Step 8 — Benchmark all 3 (extraction + A/B) Make the dev build the codegraph on PATH **once**, then loop: ```bash npm run build && ./scripts/local-install.sh scripts/add-lang/bench.sh <lang> <name> <url> "<question>" headless # ×3 ``` `bench.sh` clones (shared `/tmp/codegraph-corpus`), wipes + indexes, runs `verify-extraction.mjs`, then the with/without retrieval A/B via `scripts/agent-eval/run-all.sh` (skips the paid A/B if extraction is broken). Read each `parse-run.mjs` summary printed by `run-all.sh`: tool calls, file `Read`s, Grep/Bash, codegraph-tool calls, duration, and **cost** — for both the `with` and `without` arms. After the loop, restore the dev link if needed: `./scripts/local-install.sh`. ### Step 9 — Docs + CHANGELOG - **README.md**: add `<Lang>` to the "19+ Languages" feature bullet, and add a row to the **Supported Languages** table: `| <Lang> | \`.ext\` | Full support (classes, methods, …) |`. - **CHANGELOG.md**: add an `## [Unreleased]` section at the top (above the latest version) with `### Added` → a user-perspective bullet, e.g. *"CodeGraph now indexes **<Lang>** (`.ext`) — functions, classes, imports, and call edges."* If `## [Unreleased]` already exists, append under it. (It's folded into the next versioned block at release time.) ### Step 10 — Report (do NOT commit) Summarize for review: - **Files changed**: the 4 wiring edits + new extractor + tests + README + CHANGELOG + corpus.json (+ any vendored `.wasm`). - **Extraction** per repo: files / nodes / edges / `verify-extraction` result. - **A/B** per repo: `with` vs `without` (tool calls, file Reads, cost) and a one-line verdict — did codegraph reduce effort, and did both arms reach a correct answer? - **Gaps / follow-ups** (node types not yet mapped, resolution edges missing, framework routes, etc.). Hand the changes to the user. **Do not** run `git commit`/`push` or publish — releases go through the GitHub Actions Release workflow. ## Notes - The A/B spawns real **paid** `claude -p` runs (opus, `--max-budget-usd`), 2 arms × 3 repos. The corpus dir `/tmp/codegraph-corpus` is shared with `/agent-eval`, so clones are reused across runs. - Any new `*.wasm` must live in `src/extraction/wasm/` — `copy-assets` (run by `npm run build`) ships it; otherwise it won't be in `dist/`. - An index must be served by the **same** binary that built it. Step 8 builds + links the dev build first, so this holds. - If a grammar can't be obtained, or extraction can't reach PASS, **STOP and report** — don't ship a half-wired language. - src/bin/codegraph.ts (154908 bytes)
#!/usr/bin/env node /** * CodeGraph CLI * * Command-line interface for CodeGraph code intelligence. * * Usage: * codegraph Run interactive installer (when no args) * codegraph install Run interactive installer * codegraph uninstall Remove CodeGraph from your agents * codegraph init [path] Initialize CodeGraph in a project * codegraph uninit [path] Remove CodeGraph from a project * codegraph index [path] Index all files in the project * codegraph sync [path] Sync changes since last index * codegraph status [path] Show index status * codegraph query <search> Search for symbols * codegraph files [options] Show project file structure * codegraph context <task> Build context for a task * codegraph callers <symbol> Find what calls a function/method * codegraph callees <symbol> Find what a function/method calls * codegraph impact <symbol> Analyze what code is affected by changing a symbol * codegraph implementations <symbol> List the types that implement or extend a trait, interface or class * codegraph affected [files] Find test files affected by changes * codegraph cycles [path] List circular imports between files * codegraph ui [path] Open the browser viewer (alias: web; not released yet — needs CODEGRAPH_UI=1) * codegraph upgrade [version] Update CodeGraph to the latest release */ // FIRST import, before anything else loads: capture process.ppid while our // launcher is (almost certainly) still alive. A launcher killed mid-startup // otherwise blinds the PPID watchdog forever (#1185) — see early-ppid.ts. import '../mcp/early-ppid'; // The browser viewer is not part of a release yet (see viewer-gate). Refuse // `ui` / `web` — also as `help ui` or `ui --help` — before any startup work, // unless CODEGRAPH_UI=1 opts in. import { requestedViewerCommand, viewerEnabled } from './viewer-gate'; { const viewerCommand = requestedViewerCommand(process.argv.slice(2)); if (viewerCommand && !viewerEnabled()) { process.stderr.write(`error: 'codegraph ${viewerCommand}' is not in this release yet. The browser viewer is coming in an upcoming release.\n`); process.exit(1); } } // Persist V8 compile artifacts across runs. Every invocation — and every worker // thread, which re-requires the whole extraction module graph — skips // recompiling unchanged sources. import '../compile-cache'; import { Command } from 'commander'; import * as path from 'path'; import * as fs from 'fs'; import { getCodeGraphDir, makeGitignoreLocalOnly, isInitialized, hasSchemalessDb, hasForeignDbFile, unsafeIndexRootReason, findNearestCodeGraphRoot, planFrontload, isTaskNotification, isAgentMessage, isCrossSessionMessage, stripSystemReminders, hasStructuralKeyword, extractCodeTokens, capPromptHookInjection, codeGraphDirName, DEFAULT_CODEGRAPH_DIR } from '../directory'; import { extractProseCandidates } from '../search/identifier-segments'; import { detectWorktreeIndexMismatch, worktreeMismatchWarning } from '../sync/worktree'; import { createShimmerProgress } from '../ui/shimmer-progress'; import { getGlyphs } from '../ui/glyphs'; import { ansiColorsEnabled } from '../ui/color'; import { buildNode25BlockBanner, buildNodeTooOldBanner, MIN_NODE_MAJOR } from './node-version-check'; import { installFatalHandlers } from './fatal-handler'; import { relaunchWithWasmRuntimeFlagsIfNeeded } from '../extraction/wasm-runtime-flags'; import { installCommandSupervision, watchParent } from './command-supervision'; import { EXTRACTION_VERSION } from '../extraction/extraction-version'; import { getTelemetry, TELEMETRY_DOCS, recordIndexEvent } from '../telemetry'; // Value import, but dependency-free by design so `--help` text can name the // default port without dragging node:http into every other subcommand; the // server itself is loaded lazily inside the `ui` action. See ui-server/constants. import { BROWSER_ENV, DEFAULT_UI_PORT } from '../ui-server/constants'; import type { UiServerHandle } from '../ui-server'; import { lookupSymbolNodes, describeSymbolNode, groupDefinitions } from '../graph/symbol-lookup'; import { callSiteLines, formatCallSiteLines } from '../graph/call-sites'; import { goPackageReaderFor } from '../graph/go-package'; import { groupFilesByPackage, packageGroupTitle, packageListingTitle } from '../graph/owning-package'; import { buildTypeHierarchy, canHaveHierarchy, MAX_DESCENDANTS, MAX_DESCENDANT_DEPTH } from '../graph/type-hierarchy'; import { isTestCode } from '../graph/test-code'; import type { Node, Edge, Language } from '../types'; import { isTestPath } from '../search/query-utils'; // Decided once, before `--color`/`--no-color` are stripped from argv below // (#1281). Piped/redirected stdout, NO_COLOR, or --no-color -> plain output. const COLORS_ENABLED = ansiColorsEnabled(); // Lazy-load heavy modules (CodeGraph, runInstaller) to keep CLI startup fast. async function loadCodeGraph(): Promise<typeof import('../index')> { try { return await import('../index'); } catch (err) { const msg = err instanceof Error ? err.message : String(err); const [red, reset] = COLORS_ENABLED ? ['\x1b[31m', '\x1b[0m'] : ['', '']; console.error(`${red}${getGlyphs().err}${reset} Failed to load CodeGraph modules.`); console.error(`\n Node: ${process.version} Platform: ${process.platform} ${process.arch}`); console.error(`\n Error: ${msg}`); console.error('\n Try reinstalling with: npm install -g @colbymchenry/codegraph\n'); process.exit(1); } } // Dynamic import helper — tsc compiles import() to require() in CJS mode, // which fails for ESM-only packages. This bypasses the transformation. // eslint-disable-next-line @typescript-eslint/no-implied-eval const importESM = new Function('specifier', 'return import(specifier)') as (specifier: string) => Promise<typeof import('@clack/prompts')>; // Block CodeGraph on Node.js 25.x — V8's turboshaft WASM JIT has a Zone // allocator bug that reliably crashes when compiling tree-sitter // grammars (see #54, #81, #140). The previous behaviour was a soft // console.warn that scrolls off-screen before the OOM crash 30 seconds // later, leading to a steady stream of "what is this OOM" reports. // Hard-exit before any WASM work; allow override via env var for users // who patched V8 themselves or want to test a future fix. const nodeVersion = process.versions.node; const nodeMajor = parseInt(nodeVersion.split('.')[0] ?? '0', 10); if (nodeMajor >= 25) { process.stderr.write(buildNode25BlockBanner(nodeVersion) + '\n'); if (!process.env.CODEGRAPH_ALLOW_UNSAFE_NODE) { process.exit(1); } // Override active — banner shown for visibility, continuing. } // Enforce the supported Node floor. `engines` in package.json only *warns* on // install (unless engine-strict), so hard-block here to actually keep users off // unsupported versions. Mirrors the 25+ block above. See package.json `engines`. if (nodeMajor < MIN_NODE_MAJOR) { process.stderr.write(buildNodeTooOldBanner(nodeVersion) + '\n'); if (!process.env.CODEGRAPH_ALLOW_UNSAFE_NODE) { process.exit(1); } // Override active — banner shown for visibility, continuing. } // Re-exec with V8's `--liftoff-only` if it isn't already set, so tree-sitter's // large WASM grammars never hit the turboshaft Zone OOM (`Fatal process out of // memory: Zone`) on Node >= 22. No-op under the bundled launcher, which already // passes the flag. Must run before any grammar (in the parse worker, which // inherits this process's flags) is compiled. See ../extraction/wasm-runtime-flags. relaunchWithWasmRuntimeFlagsIfNeeded(__filename); // Last-resort fatal handlers: log a bounded line and exit non-zero. A fault // that reaches here escaped every boundary, so the process is in an undefined // state — keeping it alive is what let the detached MCP daemon orphan and pin a // CPU core with no recovery (#799, #850). Installed before the command branch // so it also covers a synchronous throw during startup. See ./fatal-handler. installFatalHandlers(); // Check if running with no arguments - run installer if (process.argv.length === 2) { import('../installer').then(({ runInstaller }) => runInstaller() ).catch((err) => { console.error('Installation failed:', err instanceof Error ? err.message : String(err)); process.exit(1); }); } else { // Normal CLI flow main(); } function main() { const program = new Command(); // Version from package.json const packageJson = JSON.parse( fs.readFileSync(path.join(__dirname, '..', '..', 'package.json'), 'utf-8') ); // Make the version trivial to reach. commander's `.version()` (below) wires up // `--version` and `-V`; intercept the spellings it can't — lowercase `-v` and // single-dash `-version` — before any parsing. (commander's version short flag // is the capital `-V`, and its parser rejects a multi-character single-dash // flag.) The bare `codegraph version` subcommand is registered further down so // the affordance also shows up in `codegraph --help`. const firstArg = process.argv[2]; if (firstArg === '-v' || firstArg === '-version') { console.log(packageJson.version); return; } // ============================================================================= // ANSI Color Helpers (avoid chalk ESM issues) // ============================================================================= // `--color` / `--no-color` are global and position-independent — they were // already read by ansiColorsEnabled() at module load, so strip them before // commander parses (a subcommand would otherwise reject the unknown flag). process.argv = process.argv.filter((a) => a !== '--color' && a !== '--no-color'); const colors = COLORS_ENABLED ? { reset: '\x1b[0m', bold: '\x1b[1m', dim: '\x1b[2m', red: '\x1b[31m', green: '\x1b[32m', yellow: '\x1b[33m', blue: '\x1b[34m', cyan: '\x1b[36m', white: '\x1b[37m', gray: '\x1b[90m', } : { reset: '', bold: '', dim: '', red: '', green: '', yellow: '', blue: '', cyan: '', white: '', gray: '', }; const chalk = { bold: (s: string) => `${colors.bold}${s}${colors.reset}`, dim: (s: string) => `${colors.dim}${s}${colors.reset}`, red: (s: string) => `${colors.red}${s}${colors.reset}`, green: (s: string) => `${colors.green}${s}${colors.reset}`, yellow: (s: string) => `${colors.yellow}${s}${colors.reset}`, blue: (s: string) => `${colors.blue}${s}${colors.reset}`, cyan: (s: string) => `${colors.cyan}${s}${colors.reset}`, white: (s: string) => `${colors.white}${s}${colors.reset}`, gray: (s: string) => `${colors.gray}${s}${colors.reset}`, }; program .name('codegraph') .description('Code intelligence and knowledge graph for any codebase') .version(packageJson.version) // Parsed manually before commander runs (any argv position works); declared // here so they show up in --help. NO_COLOR / FORCE_COLOR env vars are also // honored, and piped output defaults to no color (#1281). .option('--color', 'force ANSI colors even when stdout is not a TTY') .option('--no-color', 'disable ANSI colors (NO_COLOR env is also honored)'); // Anonymous usage telemetry (see TELEMETRY.md): record the invoked subcommand // NAME only — never arguments or paths. Counts buffer locally; network sends // piggyback on commands that run long anyway (quick commands only append to // the local buffer at exit, costing nothing). // install/uninstall are absent on purpose: the installer flushes at its own // end, AFTER its consent prompt — a flush here would fire the first-run // notice before the user ever sees the toggle. const TELEMETRY_FLUSH_COMMANDS = new Set(['init', 'uninit', 'index', 'sync', 'upgrade']); program.hook('preAction', (_thisCommand, actionCommand) => { try { // The detached daemon re-invokes `serve --mcp` internally — not a user action. if (process.env.CODEGRAPH_DAEMON_INTERNAL) return; const name = actionCommand.name(); if (name === 'telemetry') return; // managing telemetry is not usage getTelemetry().recordUsage('cli_command', name, true); if (TELEMETRY_FLUSH_COMMANDS.has(name)) getTelemetry().maybeFlush(); } catch { /* telemetry must never break the CLI */ } }); // ============================================================================= // Helper Functions // ============================================================================= /** * Resolve project path from argument or current directory * Walks up parent directories to find nearest initialized CodeGraph project * (must have .codegraph/codegraph.db, not just .codegraph/lessons.db) */ function resolveProjectPath(pathArg?: string): string { const absolutePath = path.resolve(pathArg || process.cwd()); // If exact path is initialized (has codegraph.db), use it if (isInitialized(absolutePath)) { return absolutePath; } // Walk up to find nearest parent with CodeGraph initialized // Note: findNearestCodeGraphRoot finds any .codegraph folder, but we need one with codegraph.db let current = absolutePath; const root = path.parse(current).root; while (current !== root) { const parent = path.dirname(current); if (parent === current) break; current = parent; if (isInitialized(current)) { return current; } } // Not found - return original path (will fail later with helpful error) return absolutePath; } /** * Format a number with commas */ function formatNumber(n: number): string { return n.toLocaleString(); } /** * Format duration in milliseconds to human readable */ function formatDuration(ms: number): string { if (ms < 1000) { return `${ms}ms`; } const seconds = ms / 1000; if (seconds < 60) { return `${seconds.toFixed(1)}s`; } const minutes = Math.floor(seconds / 60); const remainingSeconds = seconds % 60; return `${minutes}m ${remainingSeconds.toFixed(0)}s`; } // Shimmer progress renderer (runs in a worker thread for smooth animation) // Imported at top of file from '../ui/shimmer-progress' /** * Create a plain-text progress callback for --verbose mode. * No animations, no ANSI tricks — just timestamped lines to stdout. */ function createVerboseProgress(): (progress: { phase: string; current: number; total: number; currentFile?: string }) => void { let lastPhase = ''; let lastPct = -1; const startTime = Date.now(); return (progress) => { const elapsed = ((Date.now() - startTime) / 1000).toFixed(1); if (progress.phase !== lastPhase) { lastPhase = progress.phase; lastPct = -1; console.log(`[${elapsed}s] Phase: ${progress.phase}`); } if (progress.total > 0) { const pct = Math.floor((progress.current / progress.total) * 100); // Log every 5% to keep output manageable if (pct >= lastPct + 5 || progres - src/bin/command-supervision.ts (4996 bytes)
/** * Process supervision for long-running CLI commands (`index` / `init --index`). * * Indexing a large repo can run for a while on the main thread, and #999 * surfaced two ways that goes wrong when nothing is watching it: * * 1. **Orphaned worker.** `index` runs in a child re-exec'd with * `--liftoff-only` (the WASM-flag relaunch). Its parent blocks in * `spawnSync`, so when the parent shim is killed it cannot forward the * signal — the child keeps running, now orphaned, pinning a core. The PPID * watchdog (#277) notices the parent/host went away and exits the child. * 2. **Wedged indexer.** The `#850` main-thread liveness watchdog — which * SIGKILLs a process whose event loop stops turning — was wired only into * the MCP `serve` path, so a wedged `index`/`init` was never auto-killed. * * Both reuse the exact mechanisms `serve` already uses; this just makes them * available to a one-shot command. Best-effort and self-disabling: a missing * watchdog never blocks the command from running. Both honour the same env * switches as `serve` (`CODEGRAPH_NO_WATCHDOG`, `CODEGRAPH_PPID_POLL_MS=0`). * * Unlike the daemon — whose main thread only does fast, bounded work — the * `index`/`init` path runs reference resolution and dynamic-edge synthesis * SYNCHRONOUSLY on this thread, and on a large repo that is legitimately many * seconds of work. So those spans yield cooperatively to the event loop * (`src/resolution/cooperative-yield.ts`) to keep the heartbeat alive; without * that the watchdog would SIGKILL a valid, in-progress index (#1091). The * distinction it must preserve — kill a TRUE wedge, spare slow-but-progressing * work — is exactly what cooperative yielding buys: a genuinely stuck span never * reaches its next yield, so it still trips the timeout. */ import { installMainThreadWatchdog, WatchdogOptions } from '../mcp/liveness-watchdog'; import { supervisionLostReason, parsePpidPollMs, parseHostPpid } from '../mcp/ppid-watchdog'; import { isProcessAlive } from '../mcp/daemon-registry'; import { EARLY_PPID } from '../mcp/early-ppid'; import { HOST_PPID_ENV } from '../extraction/wasm-runtime-flags'; export interface CommandSupervision { /** Tear down both watchdogs. Idempotent; call when the command finishes. */ stop(): void; } /** * Install the liveness + PPID watchdogs for the duration of a CLI command. * `label` is used in the shutdown notice (e.g. `"index"`). Returns a handle * whose `stop()` must be called when the command completes so neither watchdog * outlives it. * * Pass `watchdog.progressPaths` (the project's SQLite DB + `-wal`) so the * liveness watchdog can tell a slow-but-progressing store on degraded storage * (files advancing) from a true wedge (they aren't) — one long synchronous * SQLite statement on a 150-IOPS disk otherwise gets a healthy index * SIGKILLed (#1231). */ export function installCommandSupervision(label: string, watchdog: WatchdogOptions = {}): CommandSupervision { // Liveness watchdog: a separate process that SIGKILLs us if our event loop // stops turning for too long (a wedged synchronous loop). Self-disables on // CODEGRAPH_NO_WATCHDOG. const liveness = installMainThreadWatchdog(watchdog); // PPID watchdog: detect that the parent (or the host threaded past the // relaunch shim) died and we've been orphaned, then exit instead of leaking. const stopParentWatch = watchParent((reason) => { try { process.stderr.write(`[CodeGraph ${label}] Parent process exited (${reason}); aborting.\n`); } catch { /* stderr gone with the parent — exit anyway */ } process.exit(1); }); let stopped = false; return { stop(): void { if (stopped) return; stopped = true; stopParentWatch(); liveness?.stop(); }, }; } /** * Call `onLost` once, when the parent — or the host threaded past the * relaunch shim — goes away. The shim blocks in `spawnSync` and cannot forward * a signal, so this is how a re-exec'd child learns it was orphaned. Returns a * stop function. `CODEGRAPH_PPID_POLL_MS=0` disables it. */ export function watchParent(onLost: (reason: string) => void): () => void { // Baseline from the CLI entry's earliest-possible capture — reading // process.ppid here would miss a launcher killed during startup (#1185). const originalPpid = EARLY_PPID; const hostPpid = parseHostPpid(process.env[HOST_PPID_ENV]); const pollMs = parsePpidPollMs(process.env.CODEGRAPH_PPID_POLL_MS); if (pollMs <= 0) return () => {}; const timer = setInterval(() => { const reason = supervisionLostReason({ originalPpid, currentPpid: process.ppid, hostPpid, isAlive: isProcessAlive, }); if (reason) { clearInterval(timer); onLost(reason); } }, pollMs); // Never let the watchdog itself keep the process alive past its real work. timer.unref(); return () => clearInterval(timer); } - src/bin/fatal-handler.ts (4119 bytes)
/** * Last-resort handlers for uncaught exceptions and unhandled rejections. * * Reaching one of these means a fault escaped every boundary (per-request * try/catch in the MCP transport, the file watcher's own `'error'` handlers, * telemetry's fail-silent contract) — i.e. the process is in an undefined * state. Node's default in that case is to print and exit non-zero. The CLI * previously OVERRODE that to "log the error and keep running", which is the * bug behind two production incidents: * * - #799 — a stdin socket `'error'` escalated here; the server logged it and * kept running, orphaning the detached MCP daemon and (on Linux) spinning a * POLLHUP fd at 100% CPU. Fixed for that one trigger by treating stdin * failure as shutdown (`src/mcp/stdin-teardown.ts`). * - #850 — a *different* uncaught exception hit the same handler. Logging it * forced V8 to lazily format the Error's `.stack`, which entered a * non-terminating source-position walk and pinned a core. Because the * handler kept the process alive, the detached daemon was left wedged: its * PPID watchdog and idle-timer (both `setInterval`s) could no longer fire, * and nothing respawned it — unrecoverable without a manual `kill`. * * The fix restores the safe default: log a BOUNDED, hang-proof line, then exit * non-zero so a fresh daemon starts on the next connection. * * Two properties are load-bearing and covered by tests: * 1. {@link describeFatal} never reads `error.stack` and never hands the raw * Error to `console.*`. The lazy stack getter is exactly the step that can * wedge (#850); since it would run *inside* this handler, touching it could * block the very `exit()` below. Name + message are plain string * properties and are always safe. * 2. We write synchronously to fd 2 and then exit, so the message is flushed * even though `process.exit()` doesn't drain async streams. */ import * as fs from 'fs'; /** * Render an uncaught value for the last-resort log WITHOUT triggering stack * formatting. Pure and total — never throws, never touches `.stack`. */ export function describeFatal(value: unknown): string { if (value instanceof Error) { const name = typeof value.name === 'string' && value.name ? value.name : 'Error'; // `message` is a plain own/proto string property — reading it does NOT // format the stack (which is what can loop forever, #850). const message = typeof value.message === 'string' ? value.message : ''; return message ? `${name}: ${message}` : name; } try { return String(value); } catch { // e.g. an object with a throwing `toString` / `Symbol.toPrimitive`. return '<unstringifiable value>'; } } /** Best-effort synchronous stderr write that can never keep a doomed process alive. */ function writeStderr(line: string): void { try { fs.writeSync(2, line); } catch { /* stderr closed/gone — nothing more we can safely do */ } } /** Injectable seams so the wiring is testable without registering real handlers. */ export interface FatalHandlerDeps { /** Event target to attach to. Defaults to `process`. */ target?: NodeJS.EventEmitter; /** How to terminate. Defaults to `process.exit`. */ exit?: (code: number) => void; /** How to emit the bounded line. Defaults to a synchronous fd-2 write. */ write?: (line: string) => void; } /** * Install the uncaught-exception / unhandled-rejection handlers. Both log a * bounded line and then exit non-zero (Node's default fatal semantics). */ export function installFatalHandlers(deps: FatalHandlerDeps = {}): void { const target = deps.target ?? process; const exit = deps.exit ?? ((code: number) => process.exit(code)); const write = deps.write ?? writeStderr; target.on('uncaughtException', (error: unknown) => { write(`[CodeGraph] Uncaught exception: ${describeFatal(error)}\n`); exit(1); }); target.on('unhandledRejection', (reason: unknown) => { write(`[CodeGraph] Unhandled rejection: ${describeFatal(reason)}\n`); exit(1); }); } - src/bin/node-version-check.ts (3067 bytes)
/** * Node.js version compatibility check. * * Node 25.x has a V8 turboshaft WASM JIT Zone allocator bug that * reliably crashes CodeGraph with `Fatal process out of memory: Zone` * during tree-sitter grammar compilation. This module owns the * user-facing banner shown before exit. Kept side-effect-free so it's * safe to import from tests without triggering CLI bootstrap. */ /** * Build the bordered banner shown when CodeGraph detects an * unsupported Node.js major version (currently 25+). Pinned via unit * test so the recovery commands and override instructions can't be * silently stripped by future edits. * * Uses ASCII glyphs to stay readable on Windows OEM-codepage consoles * (see ../ui/glyphs.ts for the rationale). */ export function buildNode25BlockBanner(nodeVersion: string): string { const sep = '-'.repeat(72); return [ sep, `[CodeGraph] Unsupported Node.js version: ${nodeVersion}`, sep, 'Node.js 25.x has a V8 WASM JIT (turboshaft) Zone allocator bug that', 'crashes with `Fatal process out of memory: Zone` when CodeGraph', 'compiles tree-sitter grammars. CodeGraph WILL crash on this Node', 'version mid-indexing. See https://github.com/colbymchenry/codegraph/issues/81', '', 'Fix: install Node.js 22 LTS:', ' nvm install 22 && nvm use 22 # nvm', ' brew install node@22 && brew link --overwrite --force node@22 # Homebrew', '', 'To override (NOT recommended - you will likely OOM):', ' CODEGRAPH_ALLOW_UNSAFE_NODE=1 codegraph ...', sep, ].join('\n'); } /** * Lowest supported Node.js major version. Matches the `engines` floor in * package.json. Below this, CodeGraph relies on language features / native APIs * that aren't present, and the combination is untested. `engines` alone only * *warns* on install (unless the user set `engine-strict`), so the CLI bootstrap * also hard-blocks here to actually enforce the floor. */ export const MIN_NODE_MAJOR = 20; /** * Build the bordered banner shown when CodeGraph detects a Node.js major below * {@link MIN_NODE_MAJOR}. Pinned via unit test so the recovery commands and the * override env var can't be silently stripped by future edits. * * Uses ASCII glyphs to stay readable on Windows OEM-codepage consoles * (see ../ui/glyphs.ts for the rationale). */ export function buildNodeTooOldBanner(nodeVersion: string): string { const sep = '-'.repeat(72); return [ sep, `[CodeGraph] Unsupported Node.js version: ${nodeVersion}`, sep, `CodeGraph requires Node.js ${MIN_NODE_MAJOR} or newer. Older versions lack`, 'language features and native APIs CodeGraph depends on, and are not', 'tested or supported.', '', 'Fix: install Node.js 22 LTS:', ' nvm install 22 && nvm use 22 # nvm', ' brew install node@22 && brew link --overwrite --force node@22 # Homebrew', '', 'To override (NOT recommended - unsupported):', ' CODEGRAPH_ALLOW_UNSAFE_NODE=1 codegraph ...', sep, ].join('\n'); } - src/bin/uninstall.ts (1153 bytes)
#!/usr/bin/env node /** * CodeGraph preuninstall cleanup script * * Runs automatically when `npm uninstall -g @colbymchenry/codegraph` * is called. Loops over every known agent target's `uninstall(loc)` * for the global location only — local-location entries live inside * project working trees and aren't ours to nuke at npm-uninstall * time. * * This script must never throw — a failed cleanup must not block * uninstall. */ try { // Lazy require so any module-level error in the registry can't // bubble out and abort the npm uninstall. // eslint-disable-next-line @typescript-eslint/no-require-imports const { ALL_TARGETS } = require('../installer/targets/registry') as typeof import('../installer/targets/registry'); for (const target of ALL_TARGETS) { if (!target.supportsLocation('global')) continue; try { target.uninstall('global'); } catch { // Each target is independently safe-to-skip; per-target failure // must not stop the loop. } } } catch { // If the registry itself can't be loaded (e.g. partial install), // we silently skip cleanup. Uninstall still completes. } - src/bin/viewer-gate.ts (1327 bytes)
/** * The browser viewer (`codegraph ui`, alias `web`) ships in the package but is * not part of a release yet: it has been exercised on Expo apps, not on API, * Laravel, Spring, Angular or Swift projects. Until it launches, both spellings * are refused before any startup work unless `CODEGRAPH_UI=1` is set, which * keeps it usable for the people testing it. Dependency-free on purpose: the * entry point checks this before loading anything else. */ /** Setting this to `1` enables the viewer commands. */ export const VIEWER_ENV = 'CODEGRAPH_UI'; const VIEWER_COMMANDS = new Set(['ui', 'web']); /** The program-level flags, which take no value and may precede the command. */ const PROGRAM_FLAGS = new Set(['--color', '--no-color']); export function viewerEnabled(env: NodeJS.ProcessEnv = process.env): boolean { return env[VIEWER_ENV] === '1'; } /** * The viewer command these arguments ask for (`ui` or `web`), whether run * directly or through `help`, or null when they ask for something else. */ export function requestedViewerCommand(args: readonly string[]): string | null { const positional = args.filter((arg) => !PROGRAM_FLAGS.has(arg)); const command = positional[0] === 'help' ? positional[1] : positional[0]; return command !== undefined && VIEWER_COMMANDS.has(command) ? command : null; } - src/compile-cache.ts (923 bytes)
/** * Keeps V8's compiled code between runs (Node 22.1 and later), so a process and * each of its worker threads load their modules without recompiling them. * Import this first: the cache covers only what is compiled after it is on. The * cache is best-effort, and a folder it can't write costs nothing. * * A folder on a drive that doesn't exist does cost: on Windows Node's directory * creation then spins forever, so a temp folder on an unplugged disk or an * unmounted RAM drive hung every command at start, at 100% of a core. */ import * as fs from 'fs'; import * as os from 'os'; import * as path from 'path'; try { if (process.platform !== 'win32' || fs.existsSync(path.parse(os.tmpdir()).root)) { // eslint-disable-next-line @typescript-eslint/no-require-imports (require('node:module') as { enableCompileCache?: () => void }).enableCompileCache?.(); } } catch { /* cache is best-effort */ } - src/context/formatter.ts (9265 bytes)
/** * Context Formatter * * Formats TaskContext as markdown or JSON for consumption by Claude. */ import { Node, Edge, TaskContext, Subgraph } from '../types'; import { isGeneratedFile } from '../extraction/generated-detection'; /** * Format context as markdown * * Creates a compact markdown document optimized for Claude with minimal context usage: * - Brief summary * - Entry points with locations * - Code blocks only for key symbols */ export function formatContextAsMarkdown( context: TaskContext, /** * Generated-file test. Defaults to the filename convention alone; the * ContextBuilder passes a DB-backed predicate so files flagged by their * HEADER at index time (#1500) demote here too. */ isGenerated: (filePath: string) => boolean = isGeneratedFile ): string { const lines: string[] = []; // Header with query lines.push('## Code Context\n'); lines.push(`**Query:** ${context.query}\n`); // Entry points - compact format. Re-sort so generated files (.pb.go, // .pulsar.go, mocks, …) rank LAST — a flow query should lead with the // hand-written implementation, not protobuf scaffolding. const orderedEntries = [...context.entryPoints].sort((a, b) => { const aGen = isGenerated(a.filePath) ? 1 : 0; const bGen = isGenerated(b.filePath) ? 1 : 0; return aGen - bGen; }); if (orderedEntries.length > 0) { lines.push('### Entry Points\n'); for (const node of orderedEntries) { const location = node.startLine ? `:${node.startLine}` : ''; lines.push(`- **${node.name}** (${node.kind}) - ${node.filePath}${location}`); if (node.signature) { lines.push(` \`${node.signature}\``); } } lines.push(''); } // Related symbols - compact list (skip verbose structure tree). Drop nodes // in generated source files (`.pb.go` / `.pulsar.go` / mocks / …) — agents // chasing a flow never want to land on protobuf scaffolding (cosmos-Q3 used // to list `gov.pulsar.go::GetExpeditedThreshold` and `1.pulsar.go::Get` in // Related Symbols, pure noise that displaced real-flow entries). const otherSymbols = Array.from(context.subgraph.nodes.values()) .filter(n => !context.entryPoints.some(e => e.id === n.id)) .filter(n => !isGenerated(n.filePath)) .slice(0, 10); // Limit to 10 related symbols if (otherSymbols.length > 0) { lines.push('### Related Symbols\n'); const byFile = new Map<string, Node[]>(); for (const node of otherSymbols) { const existing = byFile.get(node.filePath) || []; existing.push(node); byFile.set(node.filePath, existing); } for (const [file, nodes] of byFile) { const nodeList = nodes.map(n => `${n.name}:${n.startLine}`).join(', '); lines.push(`- ${file}: ${nodeList}`); } lines.push(''); } // Code blocks - only for key entry points. Re-sort so non-generated blocks // show first (consistent with Entry Points reordering above). if (context.codeBlocks.length > 0) { const orderedBlocks = [...context.codeBlocks].sort((a, b) => { const aGen = isGenerated(a.filePath) ? 1 : 0; const bGen = isGenerated(b.filePath) ? 1 : 0; return aGen - bGen; }); lines.push('### Code\n'); for (const block of orderedBlocks) { const nodeName = block.node?.name ?? 'Unknown'; lines.push(`#### ${nodeName} (${block.filePath}:${block.startLine})\n`); lines.push('```' + block.language); lines.push(block.content); lines.push('```\n'); } } return lines.join('\n'); } /** * Format context as JSON * * Returns a structured JSON representation suitable for programmatic use. */ export function formatContextAsJson(context: TaskContext): string { // Convert Map to array for JSON serialization const serializable = { query: context.query, summary: context.summary, entryPoints: context.entryPoints.map(serializeNode), nodes: Array.from(context.subgraph.nodes.values()).map(serializeNode), edges: context.subgraph.edges.map(serializeEdge), codeBlocks: context.codeBlocks.map((block) => ({ filePath: block.filePath, startLine: block.startLine, endLine: block.endLine, language: block.language, content: block.content, nodeName: block.node?.name, nodeKind: block.node?.kind, })), relatedFiles: context.relatedFiles, stats: context.stats, }; return JSON.stringify(serializable, null, 2); } /** * Format a subgraph as an ASCII tree structure */ export function formatSubgraphTree(subgraph: Subgraph, entryPoints: Node[]): string { const lines: string[] = []; const printed = new Set<string>(); // Build adjacency list for outgoing edges const outgoing = new Map<string, Edge[]>(); for (const edge of subgraph.edges) { const existing = outgoing.get(edge.source) ?? []; existing.push(edge); outgoing.set(edge.source, existing); } // Print each entry point as a tree root for (const entry of entryPoints) { formatNodeTree(entry, subgraph, outgoing, printed, lines, 0, ''); lines.push(''); // Blank line between trees } // Print any remaining nodes not reached from entry points const remaining: Node[] = []; for (const node of subgraph.nodes.values()) { if (!printed.has(node.id)) { remaining.push(node); } } if (remaining.length > 0 && remaining.length <= 10) { lines.push('Other relevant symbols:'); for (const node of remaining) { const location = node.startLine ? `:${node.startLine}` : ''; lines.push(` ${node.kind}: ${node.name} (${node.filePath}${location})`); } } else if (remaining.length > 10) { lines.push(`... and ${remaining.length} more related symbols`); } return lines.join('\n').trim(); } /** * Format a single node and its relationships */ function formatNodeTree( node: Node, subgraph: Subgraph, outgoing: Map<string, Edge[]>, printed: Set<string>, lines: string[], depth: number, prefix: string ): void { if (printed.has(node.id)) { return; } printed.add(node.id); // Node header const location = node.startLine ? `:${node.startLine}` : ''; const signature = node.signature ? ` - ${truncate(node.signature, 50)}` : ''; lines.push(`${prefix}${node.kind}: ${node.name} (${node.filePath}${location})${signature}`); // Outgoing edges const edges = outgoing.get(node.id) ?? []; const significantEdges = edges.filter((e) => ['calls', 'extends', 'implements', 'imports', 'references'].includes(e.kind) ); // Group by kind const edgesByKind = new Map<string, Edge[]>(); for (const edge of significantEdges) { const existing = edgesByKind.get(edge.kind) ?? []; existing.push(edge); edgesByKind.set(edge.kind, existing); } // Print edges grouped by kind const newPrefix = prefix + ' '; for (const [kind, kindEdges] of edgesByKind) { if (kindEdges.length > 3) { // Summarize if too many const names = kindEdges .slice(0, 3) .map((e) => { const target = subgraph.nodes.get(e.target); return target?.name ?? 'unknown'; }) .join(', '); lines.push(`${newPrefix}├── ${kind}: ${names} and ${kindEdges.length - 3} more`); } else { for (let i = 0; i < kindEdges.length; i++) { const edge = kindEdges[i]!; const target = subgraph.nodes.get(edge.target); const targetName = target?.name ?? 'unknown'; const connector = i === kindEdges.length - 1 ? '└──' : '├──'; lines.push(`${newPrefix}${connector} ${kind} → ${targetName}`); } } } // Recurse for directly connected nodes (limited depth) if (depth < 1) { for (const edge of significantEdges.slice(0, 3)) { const target = subgraph.nodes.get(edge.target); if (target && !printed.has(target.id)) { formatNodeTree(target, subgraph, outgoing, printed, lines, depth + 1, newPrefix); } } } } /** * Serialize a node for JSON output */ function serializeNode(node: Node): Record<string, unknown> { return { id: node.id, kind: node.kind, name: node.name, qualifiedName: node.qualifiedName, filePath: node.filePath, language: node.language, startLine: node.startLine, endLine: node.endLine, signature: node.signature, docstring: node.docstring, visibility: node.visibility, isExported: node.isExported, isAsync: node.isAsync, isStatic: node.isStatic, }; } /** * Serialize an edge for JSON output */ function serializeEdge(edge: Edge): Record<string, unknown> { return { source: edge.source, target: edge.target, kind: edge.kind, line: edge.line, column: edge.column, }; } /** * Truncate a string with ellipsis */ function truncate(str: string, maxLength: number): string { if (str.length <= maxLength) { return str; } return str.slice(0, maxLength - 3) + '...'; } /** * Format bytes as human-readable string */ export function formatBytes(bytes: number): string { if (bytes < 1024) { return `${bytes} bytes`; } else if (bytes < 1024 * 1024) { return `${(bytes / 1024).toFixed(1)} KB`; } else { return `${(bytes / (1024 * 1024)).toFixed(1)} MB`; } } - src/context/index.ts (69606 bytes)
/** * Context Builder * * Builds rich context for tasks by combining FTS search with graph traversal. * Outputs structured context ready to inject into Claude. */ import * as fs from 'fs'; import * as path from 'path'; import { Node, Edge, NodeKind, EdgeKind, Subgraph, CodeBlock, TaskContext, TaskInput, BuildContextOptions, FindRelevantContextOptions, SearchResult, } from '../types'; import { QueryBuilder } from '../db/queries'; import { GraphTraverser } from '../graph'; import { formatContextAsMarkdown, formatContextAsJson } from './formatter'; import { logDebug } from '../errors'; import { validatePathWithinRoot, isConfigLeafNode } from '../utils'; import { isTestFile, extractSearchTerms, scorePathRelevance, getStemVariants, isDistinctiveIdentifier, blankNonAsciiWords, splitAtCjkSwitches, meetsQueryWordFloor, } from '../search/query-utils'; import { LOW_CONFIDENCE_MARKER } from './markers'; import { findNamedCopybooks, isCopybookInclude } from '../graph/cobol-copybooks'; import { describeSynthesizedHop } from '../graph/synthesized-hop'; /** * Most top-level declarations of one copybook that become entry points — its * record layouts, or the first paragraphs of a procedure copybook. The rest of * the copybook is one hop away (explore pins the whole file). */ const MAX_COPYBOOK_DECLARATION_ENTRIES = 3; /** * The most symbol names, and search terms, of one query that * `findRelevantContext` searches the index for. A question or a bag of symbol * names stays well below both: a name like `RefreshTokenService` is one * symbol but several terms (`refreshtokenservice`, `refresh`, `token`, * `service`, stems). A pasted log, document or source file runs to hundreds * of words, and every word costs several searches of the index, so such a * query took minutes on a large project and the prompt hook gave up on it * (#2620). Both lists put code-shaped words (camelCase, snake_case, dotted * names) first and keep the order they appear in, so a long query is searched * by its identifiers and its opening words. */ export const MAX_QUERY_SYMBOLS = 64; export const MAX_QUERY_TERMS = 128; /** * Extract likely symbol names from a natural language query * * Identifies potential code symbols using patterns: * - CamelCase: UserService, signInWithGoogle * - snake_case: user_service, sign_in * - SCREAMING_SNAKE: MAX_RETRIES * - dot.notation: app.isPackaged (extracts both sides) * - Single words that look like identifiers (no spaces, not common English words) * - Names in other scripts, whole: 로그인, 寻路服务, état (#516, #1262) * * @param query - Natural language query * @returns Array of potential symbol names */ export function extractSymbolsFromQuery(query: string): string[] { const symbols = new Set<string>(); // The ASCII patterns below would read `état` as `tat`; they see such words // blanked out, and the Unicode pass at the end takes them whole. const asciiView = blankNonAsciiWords(query); // Extract CamelCase identifiers (2+ chars, starts with letter) const camelCasePattern = /\b([A-Z][a-z]+(?:[A-Z][a-z]*)*|[a-z]+(?:[A-Z][a-z]*)+)\b/g; let match; while ((match = camelCasePattern.exec(asciiView)) !== null) { if (match[1] && match[1].length >= 2) { symbols.add(match[1]); } } // Extract snake_case identifiers const snakeCasePattern = /\b([a-z][a-z0-9]*(?:_[a-z0-9]+)+)\b/gi; while ((match = snakeCasePattern.exec(asciiView)) !== null) { if (match[1] && match[1].length >= 3) { symbols.add(match[1]); } } // Extract SCREAMING_SNAKE_CASE const screamingPattern = /\b([A-Z][A-Z0-9]*(?:_[A-Z0-9]+)+)\b/g; while ((match = screamingPattern.exec(asciiView)) !== null) { if (match[1]) { symbols.add(match[1]); } } // Extract ALL_CAPS acronyms (2+ chars, e.g., REST, HTTP, LRU, API) const acronymPattern = /\b([A-Z]{2,})\b/g; while ((match = acronymPattern.exec(asciiView)) !== null) { if (match[1]) { symbols.add(match[1]); } } // Extract dot.notation and split into parts (e.g., "app.isPackaged" -> ["app", "isPackaged"]) const dotPattern = /\b([a-zA-Z][a-zA-Z0-9]*(?:\.[a-zA-Z][a-zA-Z0-9]*)+)\b/g; while ((match = dotPattern.exec(asciiView)) !== null) { if (match[1]) { // Add both the full path and individual parts symbols.add(match[1]); const parts = match[1].split('.'); for (const part of parts) { if (part.length >= 2) { symbols.add(part); } } } } // Extract plain lowercase identifiers (3+ chars, not already matched) // Catches symbol names like "undo", "redo", "history", "render", "parse" const lowercasePattern = /\b([a-z][a-z0-9]{2,})\b/g; while ((match = lowercasePattern.exec(asciiView)) !== null) { if (match[1]) { symbols.add(match[1]); } } // Names with a letter outside ASCII, whole runs only (#2005 keeps partial // Chinese words out of search): `로그인`, `寻路服务`, `état`. A name that // switches between Han, kana or Hangul and another script, or joins words // with `_`, is tried as written (`get用户信息`) and by its words, so // `NavMeshAgent类型` also names `类型`; its ASCII side is the patterns' above. for (const run of query.match(/[\p{L}\p{N}\p{M}_$]+/gu) ?? []) { if (!/[^\x00-\x7F]/.test(run)) continue; const words = run.split(/[_$]+/).flatMap(splitAtCjkSwitches); if (words.length > 1 && meetsQueryWordFloor(run)) symbols.add(run); for (const word of words) { if (/[^\x00-\x7F]/.test(word) && meetsQueryWordFloor(word)) symbols.add(word); } } // Filter out common English words that aren't likely symbol names const commonWords = new Set([ 'the', 'and', 'for', 'with', 'from', 'this', 'that', 'have', 'been', 'will', 'would', 'could', 'should', 'does', 'done', 'make', 'made', 'use', 'used', 'using', 'work', 'works', 'find', 'found', 'show', 'call', 'called', 'calling', 'get', 'set', 'add', 'all', 'any', 'how', 'what', 'when', 'where', 'which', 'who', 'why', 'not', 'but', 'are', 'was', 'were', 'has', 'had', 'its', 'can', 'did', 'may', 'also', 'into', 'than', 'then', 'them', 'each', 'other', 'some', 'such', 'only', 'same', 'about', 'after', 'before', 'between', 'through', 'during', 'without', 'again', 'further', 'once', 'here', 'there', 'both', 'just', 'more', 'most', 'very', 'being', 'having', 'doing', 'system', 'need', 'needs', 'want', 'wants', 'like', 'look', 'change', 'changes', 'changed', 'changing', // Common English nouns/verbs that match thousands of unrelated code symbols 'layer', 'handle', 'handles', 'handling', 'incoming', 'outgoing', 'data', 'flow', 'flows', 'level', 'levels', 'request', 'requests', 'response', 'responses', 'implement', 'implements', 'implementation', 'interface', 'interfaces', 'class', 'classes', 'method', 'methods', 'trigger', 'triggers', 'affected', 'affect', 'affects', 'else', 'code', 'failing', 'failed', 'silently', 'decide', 'decides', 'return', 'returns', 'returned', 'take', 'takes', 'taken', 'check', 'checks', 'checked', 'create', 'creates', 'created', 'read', 'reads', 'write', 'writes', 'written', 'start', 'starts', 'stop', 'stops', 'run', 'runs', 'running', ]); return Array.from(symbols).filter(s => !commonWords.has(s.toLowerCase())); } /** * Default options for context building * * Tuned for minimal context usage while still providing useful results: * - Fewer nodes and code blocks by default * - Smaller code block size limit * - Shallower traversal */ const DEFAULT_BUILD_OPTIONS: Required<BuildContextOptions> = { maxNodes: 20, // Reduced from 50 - most tasks don't need 50 symbols maxCodeBlocks: 5, // Reduced from 10 - only show most relevant code maxCodeBlockSize: 1500, // Reduced from 2000 includeCode: true, format: 'markdown', searchLimit: 3, // Reduced from 5 - fewer entry points traversalDepth: 1, // Reduced from 2 - shallower graph expansion minScore: 0.3, }; /** * Node kinds that provide high information value in context results. * Imports/exports are excluded because they have near-zero information density - * they tell you something exists, not how it works. */ const HIGH_VALUE_NODE_KINDS: NodeKind[] = [ 'function', 'method', 'class', 'interface', 'type_alias', 'struct', 'union', 'trait', 'component', 'route', 'variable', 'constant', 'enum', 'module', 'namespace', ]; /** * Default options for finding relevant context */ const DEFAULT_FIND_OPTIONS: Required<FindRelevantContextOptions> = { searchLimit: 3, // Reduced from 5 traversalDepth: 1, // Reduced from 2 maxNodes: 20, // Reduced from 50 minScore: 0.3, edgeKinds: [], nodeKinds: HIGH_VALUE_NODE_KINDS, // Filter out imports/exports by default seedNames: [], // Segment-vocab supplement — filled by the facade }; // Re-export the low-confidence sentinel (defined in a dependency-free leaf so // the MCP layer can import it without pulling this module's deps onto the // cold-start path). Builder code below uses the imported binding directly. export { LOW_CONFIDENCE_MARKER } from './markers'; /** * Context Builder * * Coordinates semantic search and graph traversal to build * comprehensive context for tasks. */ export class ContextBuilder { private projectRoot: string; private queries: QueryBuilder; private traverser: GraphTraverser; constructor( projectRoot: string, queries: QueryBuilder, traverser: GraphTraverser ) { this.projectRoot = projectRoot; this.queries = queries; this.traverser = traverser; } /** * Whether the project's `codegraph.json` `deprioritize` patterns cover this * path (#982). Explore ranks through its own path scorer as well as through * `searchNodes`, so the lever has to be applied here too or the setting would * only half-work — and explore is the surface #982 actually reports on. * * Only the -15 relevance penalty is shared. Explore's hard `continue` filters * and its non-production budget cap are deliberately NOT joined: those REMOVE * content, and `deprioritize` is a ranking lever by definition — `exclude` is * the lever for taking things out of reach. * * Returned as a predicate with the patterns read once, for the candidate * loops of one search: asking the live matcher per candidate re-checked * `codegraph.json` hundreds of thousands of times on a long query (#2620). */ private deprioritizedPaths(): (filePath: string) => boolean { const matcher = this.queries.getDeprioritizedPathSnapshot?.() ?? this.queries.getDeprioritizedPathMatcher?.(); return (filePath) => matcher?.(filePath) ?? false; } /** * Build context for a task * * Pipeline: * 1. Parse task input (string or {title, description}) * 2. Run semantic search to find entry points * 3. Expand graph around entry points * 4. Extract code blocks for key nodes * 5. Format output for Claude * * @param input - Task description or object with title/description * @param options - Build options * @returns TaskContext (structured) or formatted string */ async buildContext( input: TaskInput, options: BuildContextOptions = {} ): Promise<TaskContext | string> { const opts = { ...DEFAULT_BUILD_OPTIONS, ...options }; // Parse input const query = typeof input === 'string' ? input : `${input.title}${input.description ? `: ${input.description}` : ''}`; // Find relevant context (semantic search + graph expansion) const subgraph = await this.findRelevantContext(query, { searchLimit: opts.searchLimit, traversalDepth: opts.traversalDepth, maxNodes: opts.maxNodes, minScore: opts.minScore, }); // Get entry points (nodes from semantic search) const entryPoints = this.getEntryPoints(subgraph); // Extract code blocks for key nodes const codeBlocks = opts.includeCode ? await this.extractCodeBlocks(subgraph, opts.maxCodeBlocks, opts.maxCodeBlockSize) : []; // Get related files const relatedFiles = this.getRelatedFiles(subgraph); // Generate summary const summary = this.generateSummary(query, subgraph, entryPoints); // Calculate stats const stats = { nodeCount: subgraph.nodes.size, edgeCount: subgraph.edges.length, fileCount: relatedFiles.length, codeBlockCount: codeBlocks.length, totalCodeSize: codeBlocks.reduce((sum, block) => sum + block.content.length, 0), }; const context: TaskContext = { query, subgraph, entryPoints, codeBlocks, relatedFiles, summary, stats, }; // Return formatted output or raw context if (opts.format === 'markdown') { // Bounded candidate set (entry points + subgraph + code blocks), so the // DB-backed generated check is one probe, not a per-comparison query. const isGenerated = this.queries.generatedPredicateFor([ ...entryPoints.map((n) => n.filePath), ...Array.from(subgraph.nodes.values(), (n) => n.filePath), ...codeBlocks.map((b) => b.filePath), ]); return formatContextAsMarkdown(context, isGenerated) + this.buildCallPathsSection(subgraph) + (subgraph.confidence === 'low' ? this.buildLowConfidenceNote(entryPoints) : ''); } else if (opts.format === 'json') { return formatContextAsJson(context); } return context; } /** * Honest handoff appended when retrieval confidence is low (the query matched * mostly common words). Instead of the usual "this covers the surface" framing * — which, when wrong, sends the agent off to Read/Grep — it admits the * uncertainty and routes the agent to the precise tools (explore with real * symbol names, search, or files to browse the closest areas we *did* surface). */ private buildLowConfidenceNote(entryPoints: Node[]): string { const dirs: string[] = []; const seen = new Set<string>(); for (const n of entryPoints) { const slash = n.filePath.lastIndexOf('/'); const dir = slash > 0 ? n.filePath.slice(0, slash) : n.filePath; if (!seen.has(dir)) { seen.add(dir); dirs.push(dir); } if (dirs.length >= 4) break; } const dirLine = dirs.length ? `\n- \`codegraph_files\` a likely area: ${dirs.map(d => `\`${d}\``).join(', ')}` : ''; return `\n\n${LOW_CONFIDENCE_MARKER}\n\n` + 'This query matched mostly on common words, so the entry points above may ' + 'be off-target — treat them as a starting point, not a complete answer. ' + 'For a reliable result:\n' + '- `codegraph_explore` with the **exact symbol names** you are after ' + '(class / function / method names), or\n' + '- `codegraph_search <name>` for one specific symbol' + dirLine + '\n\nDo not assume the list above is comprehensive.'; } /** * - src/context/markers.ts (986 bytes)
/** * Stable sentinel strings shared between the context builder (which emits them * into its markdown) and the MCP layer (which detects them to adjust framing). * * Intentionally a dependency-free leaf module: the MCP tool layer imports this * to recognise a low-confidence response, and routing that recognition through * the full context module would drag its dependencies onto the cold-start path. * Keep this file import-free. */ /** * Heading that leads the honest low-confidence handoff appended to a context * response when the query resolved only to weak/isolated matches. The MCP layer * checks for it to suppress the contradictory "this is comprehensive, don't call * explore" small-repo footer. Changing the text is a breaking sentinel change — * both the emitter (`ContextBuilder`) and the detector (`src/mcp/tools.ts`) * import this constant, so they stay in sync automatically. */ export const LOW_CONFIDENCE_MARKER = '### ⚠️ Low-confidence match';
- src/db/index.ts (45241 bytes)
/** * Database Layer * * Handles SQLite database initialization and connection management. */ import { SqliteDatabase, SqliteBackend, createDatabase } from './sqlite-adapter'; import * as fs from 'fs'; import * as path from 'path'; import { SchemaVersion } from '../types'; import { runMigrations, getCurrentVersion, CURRENT_SCHEMA_VERSION } from './migrations'; import { getCodeGraphDir, statInode } from '../directory'; import { toDatabaseOpenError } from './open-error'; export { SqliteDatabase, SqliteBackend } from './sqlite-adapter'; export { DatabaseOpenError } from './open-error'; export { NodeSqliteUnavailableError } from './node-sqlite'; /** SQLite's memory-mapped window on the database file: 256 MB. */ const MMAP_SIZE_BYTES = 268435456; /** * Apply connection-level PRAGMAs. Shared by `initialize` and `open` so the two * paths can't drift. * * `busy_timeout` is set FIRST, before any pragma that might touch the database * file (notably `journal_mode`). If another process holds a write lock at open * time, the later pragmas — and the connection's first query — then wait out * the lock instead of throwing "database is locked" immediately. See issue #238. * * The 5s window (was 120s) rides out a normal incremental sync; the old * 2-minute wait presented as a frozen, hung agent. With WAL, reads never block * on a writer, so this timeout only governs cross-process write contention * (e.g. the git-hook `codegraph sync` running while the MCP server writes). */ function configureConnection(db: SqliteDatabase, readOnly = false): void { db.pragma('busy_timeout = 5000'); // MUST be first — see above db.pragma('foreign_keys = ON'); if (!readOnly) db.pragma('journal_mode = WAL'); // node:sqlite supports WAL on every platform db.pragma('synchronous = NORMAL'); // safe with WAL mode db.pragma('cache_size = -64000'); // 64 MB page cache db.pragma('temp_store = MEMORY'); // temp tables in memory db.pragma(`mmap_size = ${MMAP_SIZE_BYTES}`); // memory-mapped I/O // Without a journal_size_limit the -wal file never shrinks below its // high-water mark while a connection lives: checkpoints fold frames back but // leave the file at full size, so one giant deferred-sync WAL stays giant // forever. With the limit set, any checkpoint that resets the WAL truncates // the file back down. Killed-process leftovers are handled separately by // healOversizedWal() at open. (#1431) db.pragma(`journal_size_limit = ${WAL_HEAL_THRESHOLD_BYTES}`); } /** * Run the rest of an open on the connection `db`. Its first statements read * the file, so a database SQLite cannot use (locked, damaged, not a database, * an I/O error) fails here as well as when the connection is made. Such a * failure names SQLite's code and the path ({@link DatabaseOpenError}), and * leaves no connection open. */ function withOpenFailures<T>(db: SqliteDatabase, dbPath: string, finish: () => T): T { try { return finish(); } catch (err) { try { db.close(); } catch { /* the open's failure is the one to report */ } throw toDatabaseOpenError(err, dbPath); } } /** * WAL size past which `healOversizedWal` (run at every `open`) checkpoints and * truncates the file, and to which `journal_size_limit` clips the WAL after any * resetting checkpoint. A SIGKILL'd process (the #850 liveness watchdog, OOM, * crash) can leave an arbitrarily large WAL behind — a whole deferred-sync * run's worth (#1248) — and before #1431 no later session ever shrank it: the * file just grew, killed session after killed session, until the disk filled * (25.6 GB observed). 64 MB is far above anything a healthy open ever sees * (a clean close deletes the WAL) yet small enough to cap the leak. * Override with `CODEGRAPH_WAL_HEAL_MB` (also feeds `journal_size_limit`). */ export const WAL_HEAL_THRESHOLD_BYTES = resolveWalHealBytes(process.env.CODEGRAPH_WAL_HEAL_MB); /** Resolve the heal threshold from the env override (MB); invalid ⇒ 64 MB. */ export function resolveWalHealBytes(envVal: string | undefined): number { if (envVal !== undefined && envVal !== '') { const n = Number(envVal); if (Number.isFinite(n) && n > 0) return Math.floor(n * 1024 * 1024); } return 64 * 1024 * 1024; } /** * Database connection wrapper with lifecycle management */ export class DatabaseConnection { private db: SqliteDatabase; private dbPath: string; private backend: SqliteBackend; /** * `dev:ino` of the DB file at the moment we opened it (or null when the * platform/filesystem reports no usable inode). Lets us notice when the file * we hold open has been unlinked and REPLACED by a new file at the same path * — a git worktree removed and re-added, or `.codegraph/` deleted and * re-`init`ed under a long-lived server — at which point our fd reads a now * dead inode forever (#925). See `isReplacedOnDisk`. */ private openedInode: string | null; /** * Whether FTS5 is available in this Node.js build. When false, search * falls back to LIKE + fuzzy matching (#1532). */ readonly fts5Available: boolean; private constructor(db: SqliteDatabase, dbPath: string, backend: SqliteBackend, fts5Available: boolean, readonly readOnly = false) { this.db = db; this.dbPath = dbPath; this.backend = backend; this.fts5Available = fts5Available; this.openedInode = statInode(dbPath); } /** * Initialize a new database at the given path */ static initialize(dbPath: string): DatabaseConnection { // Ensure parent directory exists const dir = path.dirname(dbPath); if (!fs.existsSync(dir)) { fs.mkdirSync(dir, { recursive: true }); } const { db, backend } = createDatabase(dbPath); return withOpenFailures(db, dbPath, () => DatabaseConnection.initializeOn(db, backend, dbPath)); } /** The rest of {@link initialize}: configure the connection and create the schema. */ private static initializeOn(db: SqliteDatabase, backend: SqliteBackend, dbPath: string): DatabaseConnection { configureConnection(db); // Run schema initialization, splitting FTS5 from the rest so // codegraph still works when Node.js was built without FTS5 (#1532). const schemaPath = path.join(__dirname, 'schema.sql'); const schema = fs.readFileSync(schemaPath, 'utf-8'); const FTS5_MARKER = '-- Full-text search index on node names, docstrings, and signatures'; const ftsIdx = schema.indexOf(FTS5_MARKER); let fts5Available = true; if (ftsIdx >= 0) { const preFts = schema.slice(0, ftsIdx); // FTS ends after the update trigger; required tables and indexes follow // it in schema.sql and must still be created when FTS5 is unavailable. const ftsSection = schema.slice(ftsIdx).match( /^[\s\S]*?CREATE TRIGGER IF NOT EXISTS nodes_au\b[\s\S]*?END;/ )?.[0]; if (!ftsSection) throw new Error('schema.sql: FTS5 update trigger not found'); // Execute everything before FTS5 first db.exec(preFts); // Try FTS5; if it fails, skip it and continue with LIKE-only search try { db.exec(ftsSection); } catch (err: any) { fts5Available = false; const msg = err?.message ?? String(err); console.warn( `[codegraph] FTS5 not available in this Node.js build (${msg}). ` + `Search will fall back to LIKE + fuzzy matching. ` + `For full-text search, use a Node.js build with FTS5 enabled.` ); } db.exec(schema.slice(ftsIdx + ftsSection.length)); } else { db.exec(schema); } // Record current schema version so migrations aren't re-applied on open const currentVersion = getCurrentVersion(db); if (currentVersion < CURRENT_SCHEMA_VERSION) { db.prepare( 'INSERT OR IGNORE INTO schema_versions (version, applied_at, description) VALUES (?, ?, ?)' ).run(CURRENT_SCHEMA_VERSION, Date.now(), 'Initial schema includes all migrations'); } return new DatabaseConnection(db, dbPath, backend, fts5Available); } /** * Open an existing database. * * `deferIndexRecovery` leaves the secondary indexes an interrupted bulk load * dropped for {@link recoverIndexesOffThread} instead of rebuilding them * here. Rebuilding one scans and sorts its whole table: on a large index * that is minutes of synchronous work, and on a watched event loop the * liveness watchdog kills the process before the index commits, every time * it starts (#1887). Reads stay correct meanwhile, only slower. */ static open(dbPath: string, options: { readOnly?: boolean; deferIndexRecovery?: boolean } = {}): DatabaseConnection { if (!fs.existsSync(dbPath)) { throw new Error(`Database not found: ${dbPath}`); } const { db, backend } = createDatabase(dbPath, options); return withOpenFailures(db, dbPath, () => DatabaseConnection.openOn(db, backend, dbPath, options)); } /** The rest of {@link open}: configure the connection, then migrate and repair a writable one. */ private static openOn( db: SqliteDatabase, backend: SqliteBackend, dbPath: string, options: { readOnly?: boolean; deferIndexRecovery?: boolean }, ): DatabaseConnection { configureConnection(db, options.readOnly); // Detect FTS5 availability for search fallback (#1532) let fts5Available = true; try { db.exec("SELECT * FROM nodes_fts LIMIT 0"); } catch { fts5Available = false; } // Check and run migrations if needed const conn = new DatabaseConnection(db, dbPath, backend, fts5Available, options.readOnly); // A concurrent reader must leave migrations, bulk-load repair, and WAL // maintenance to the writer, including when versions differ (#1963). if (options.readOnly) return conn; const currentVersion = getCurrentVersion(db); if (currentVersion < CURRENT_SCHEMA_VERSION) { runMigrations(db, currentVersion); } // Self-heal a bulk-load window that never closed (crash between // beginBulkNodeLoad and endBulkNodeLoad): the FTS triggers are missing and // nodes_fts is stale. Rebuild + recreate so search stays in sync. conn.healBulkNodeLoad(); if (options.deferIndexRecovery) conn.deferredIndexDdls = conn.missingBulkSecondaryIndexDdls(); else conn.healBulkSecondaryIndexes(); // Self-heal a killed session's leftover oversized WAL (#1431) — one // statSync when healthy, off-thread checkpoint+truncate when not. void conn.healOversizedWal(); return conn; } /** * FTS maintenance triggers dropped/recreated around a bulk load. * Names must match schema.sql. */ private static readonly FTS_TRIGGER_NAMES = ['nodes_ai', 'nodes_ad', 'nodes_au'] as const; /** * Enter bulk-load mode: drop the per-row FTS sync triggers so mass node * inserts skip per-row tokenization. MUST be paired with endBulkNodeLoad() * (use try/finally); a crash inside the window is healed on the next open(). * The window is DB-wide (triggers are schema objects), which is safe because * endBulkNodeLoad() rebuilds nodes_fts from the nodes table wholesale — any * row written by anyone during the window is captured by the rebuild. */ beginBulkNodeLoad(): void { if (!this.fts5Available) return; for (const t of DatabaseConnection.FTS_TRIGGER_NAMES) { this.db.exec(`DROP TRIGGER IF EXISTS ${t}`); } } /** * Leave bulk-load mode: rebuild the whole FTS index from the nodes table in * one pass (far cheaper than per-row trigger firings), then recreate the * triggers by re-running schema.sql (idempotent — everything in it is * IF NOT EXISTS). */ endBulkNodeLoad(): void { if (!this.fts5Available) return; this.db.exec(`INSERT INTO nodes_fts(nodes_fts) VALUES('rebuild')`); this.recreateFtsTriggers(); } /** * NON-UNIQUE secondary indexes maintained per-row during the parse phase's * bulk inserts — the store-architecture arc's first lever (plan §4d: dubbo's * parse-loop wall is 94% store-writer busy, and the #1320 post-mortem showed * statement batching and sorted inserts are ~zero on this path because * B-TREE MAINTENANCE is the floor). A fresh init writes every row of * nodes/unresolved_refs/files exactly once and reads none of them until * resolution, so the parse window can drop all of these and rebuild each in * one table scan afterwards — the same measured trade as the resolution * phase's edge-index window (2.8s → 1.1s inserting, ~0.3s recreating). * Primary keys and UNIQUE constraints stay (upserts and OR-IGNORE dedup * conflict on them). */ private static readonly BULK_PARSE_INDEX_NAMES = [ 'idx_nodes_kind', 'idx_nodes_name', 'idx_nodes_qualified_name', 'idx_nodes_file_path', 'idx_nodes_language', 'idx_nodes_file_line', 'idx_nodes_lower_name', 'idx_unresolved_from_node', 'idx_unresolved_name', 'idx_unresolved_file_path', 'idx_unresolved_from_name', 'idx_unresolved_status', 'idx_unresolved_failed_tail', 'idx_files_language', 'idx_files_modified_at', ] as const; /** * Enter bulk-parse-load mode (FRESH-INIT ONLY — the caller gates on a fresh * DB, because an incremental index deletes per-file rows mid-phase and needs * the file_path indexes): drop every parse-lane secondary index, including * the four non-unique edge indexes (parse inserts contains-edges too; the * UNIQUE identity index stays for INSERT OR IGNORE dedup, and its `source` * prefix keeps source-keyed reads indexed, as in the edge window). MUST be * paired with endBulkParseLoad(); a crash inside the window is healed on the * next DatabaseConnection open (schema.sql re-applies CREATE INDEX IF NOT * EXISTS). */ beginBulkParseLoad(): void { for (const idx of DatabaseConnection.BULK_PARSE_INDEX_NAMES) { this.db.exec(`DROP INDEX IF EXISTS ${idx}`); } this.beginBulkEdgeLoad(); } /** * Leave bulk-parse-load mode: recreate everything the window dropped, one * table scan per index, with a yield between statements (same * liveness-watchdog rationale as endBulkEdgeLoad — at kernel scale each * build is a long synchronous scan). The edge indexes are rebuilt here too, * so paths that never enter the resolution phase's own bulk-edge window * (small runs) are left with a complete schema; the batched resolver's * beginBulkEdgeLoad simply re-drops them (DROP IF EXISTS — idempotent). */ async endBulkParseLoad(): Promise<void> { const schemaPath = path.join(__dirname, 'schema.sql'); const schema = fs.readFileSync(schemaPath, 'utf-8'); for (const idx of DatabaseConnection.BULK_PARSE_INDEX_NAMES) { const m = schema.match(new RegExp(`CREATE INDEX IF NOT EXISTS ${idx}\\b[^;]*;`)); if (!m) throw new Error(`schema.sql: parse index ${idx} not found for bulk-load recreation`); this.db.exec(m[0 - src/db/migrations.ts (18001 bytes)
/** * Database Migrations * * Schema versioning and migration support. */ import { SqliteDatabase } from './sqlite-adapter'; import { referenceNameTail, MEMBER_TAIL_SQL, MODULE_PARKED_SQL } from './reference-tail'; /** * Current schema version */ export const CURRENT_SCHEMA_VERSION = 16; /** * Migration definition */ interface Migration { version: number; description: string; up: (db: SqliteDatabase) => void; } /** * All migrations in order * * Note: Version 1 is the initial schema, handled by schema.sql * Future migrations go here. */ const migrations: Migration[] = [ { version: 2, description: 'Add project metadata, provenance tracking, and unresolved ref context', up: (db) => { db.exec(` CREATE TABLE IF NOT EXISTS project_metadata ( key TEXT PRIMARY KEY, value TEXT NOT NULL, updated_at INTEGER NOT NULL ); ALTER TABLE unresolved_refs ADD COLUMN file_path TEXT NOT NULL DEFAULT ''; ALTER TABLE unresolved_refs ADD COLUMN language TEXT NOT NULL DEFAULT 'unknown'; ALTER TABLE edges ADD COLUMN provenance TEXT DEFAULT NULL; CREATE INDEX IF NOT EXISTS idx_unresolved_file_path ON unresolved_refs(file_path); CREATE INDEX IF NOT EXISTS idx_edges_provenance ON edges(provenance); `); }, }, { version: 3, description: 'Add lower(name) expression index for memory-efficient case-insensitive lookups', up: (db) => { db.exec(` CREATE INDEX IF NOT EXISTS idx_nodes_lower_name ON nodes(lower(name)); `); }, }, { version: 4, description: 'Drop redundant idx_edges_source / idx_edges_target (covered by source_kind / target_kind composites)', up: (db) => { db.exec(` DROP INDEX IF EXISTS idx_edges_source; DROP INDEX IF EXISTS idx_edges_target; `); }, }, { version: 5, description: 'Add nodes.return_type — normalized return/result type for receiver-type inference (C++ singletons/factories, #645)', up: (db) => { db.exec(` ALTER TABLE nodes ADD COLUMN return_type TEXT; `); }, }, { version: 6, description: 'Dedup duplicate edge rows and add a UNIQUE identity index so INSERT OR IGNORE actually dedups (#1034)', up: (db) => { // `insertEdge` has always used `INSERT OR IGNORE`, but the edges table had // no UNIQUE constraint, so nothing conflicted and byte-identical rows // accumulated whenever two passes emitted the same edge. Collapse each // identity group to its lowest id, then add the constraint that makes // `OR IGNORE` keep its promise. IFNULL folds nullable line/col so // coordinate-less edges dedup too (SQLite treats each NULL as distinct) — // and it MUST match the GROUP BY exactly, or the index creation would // fail on a pair the DELETE left behind. Idempotent: the index is // `IF NOT EXISTS` and the DELETE is a no-op once the table is unique. db.exec(` DELETE FROM edges WHERE id NOT IN ( SELECT MIN(id) FROM edges GROUP BY source, target, kind, IFNULL(line, -1), IFNULL(col, -1) ); CREATE UNIQUE INDEX IF NOT EXISTS idx_edges_identity ON edges(source, target, kind, IFNULL(line, -1), IFNULL(col, -1)); `); }, }, { version: 7, description: 'Add name_segment_vocab — prose-word → symbol-name lookup for the prompt hook’s graph-derived gate', up: (db) => { // DDL only — instant on any size database (the row-churn hazards of #1067 // don't apply). The table starts EMPTY on migrated databases; `sync` // detects that over a populated graph and backfills batched+yielding // (CodeGraph.rebuildNameSegmentVocab), and a full index into a fresh // database fills it from scratch. Keep the definition in lockstep with // schema.sql. db.exec(` CREATE TABLE IF NOT EXISTS name_segment_vocab ( segment TEXT NOT NULL, name TEXT NOT NULL, PRIMARY KEY (segment, name) ) WITHOUT ROWID; `); }, }, { version: 8, description: 'Track attempted-but-unresolvable refs as status=failed so sync can retry them when a changed file adds a matching symbol (#1240)', up: (db) => { // DDL only — instant on any size database. No backfill needed: rows are // only ever queried by name_tail once they carry status='failed', and // both fields are written together by markReferencesFailed. Legacy rows // (all 'pending' after this migration) are orphans from interrupted runs // that the #1187 sweep grinds down on the next sync, marking survivors // failed with their tails as it goes. The tail index is partial: on a // healthy index the pending set is empty and the failed set is the only // population worth indexing. Keep the definitions in lockstep with // schema.sql. ALTER TABLE has no IF NOT EXISTS, so guard each column for // idempotency — a database created from current schema.sql already has // both (matters when migrations are re-run from an older recorded // version, as the v6 regression test does). const cols = db.prepare('PRAGMA table_info(unresolved_refs)').all() as Array<{ name: string }>; const hasColumn = (name: string) => cols.some((c) => c.name === name); if (!hasColumn('status')) { db.exec("ALTER TABLE unresolved_refs ADD COLUMN status TEXT NOT NULL DEFAULT 'pending'"); } if (!hasColumn('name_tail')) { db.exec("ALTER TABLE unresolved_refs ADD COLUMN name_tail TEXT NOT NULL DEFAULT ''"); } db.exec(` CREATE INDEX IF NOT EXISTS idx_unresolved_status ON unresolved_refs(status); CREATE INDEX IF NOT EXISTS idx_unresolved_failed_tail ON unresolved_refs(name_tail) WHERE status = 'failed'; `); }, }, { version: 9, description: 'Add files.generated — index-time content-header generated-file detection for ranking (#1500)', up: (db) => { // DDL only — instant on any size database, and NO backfill: the flag is // derived from file CONTENT, which this migration has no access to (the // files table stores a hash, not the bytes). Migrated rows therefore stay // 0 until the next full index re-extracts them, and every reader unions // the flag with the path-only check, so an un-backfilled database keeps // exactly the pre-#1500 behavior instead of regressing. `sync` heals it // file-by-file as files change. This is why the CHANGELOG entry says a // re-index is required to pick up the new detection. // // ALTER TABLE has no IF NOT EXISTS, so guard for idempotency — a database // created from current schema.sql already has the column (matters when // migrations are re-run from an older recorded version, as the v6 // regression test does). Keep in lockstep with schema.sql. const cols = db.prepare('PRAGMA table_info(files)').all() as Array<{ name: string }>; if (!cols.some((c) => c.name === 'generated')) { db.exec('ALTER TABLE files ADD COLUMN generated INTEGER NOT NULL DEFAULT 0'); } db.exec( 'CREATE INDEX IF NOT EXISTS idx_files_generated ON files(path) WHERE generated = 1' ); }, }, { version: 10, description: 'Track synthesis inputs and stabilize synthesis traversal for incremental refresh (#1988)', up: (db) => { db.exec(` DROP INDEX IF EXISTS idx_nodes_kind; CREATE INDEX idx_nodes_kind ON nodes(kind, file_path, start_line, id); CREATE TABLE IF NOT EXISTS synthesis_inputs ( file_path TEXT PRIMARY KEY REFERENCES files(path) ON DELETE CASCADE ); CREATE INDEX IF NOT EXISTS idx_edges_synthesis_site ON edges(CASE WHEN json_valid(metadata) THEN json_extract(metadata, '$.registeredAt') END) WHERE CASE WHEN json_valid(metadata) THEN json_extract(metadata, '$.synthesizedBy') END IS NOT NULL; UPDATE edges SET metadata = json_set(CASE WHEN json_valid(metadata) THEN metadata ELSE '{}' END, '$.synthesizedBy', 'go-method-contains') WHERE kind = 'contains' AND provenance IS NULL AND EXISTS ( SELECT 1 FROM nodes s JOIN nodes t ON t.id = edges.target WHERE s.id = edges.source AND s.language = 'go' AND t.language = 'go' AND s.kind IN ('struct', 'class', 'interface', 'enum', 'type_alias') AND t.kind = 'method' AND s.file_path != t.file_path ); INSERT OR REPLACE INTO project_metadata(key, value, updated_at) VALUES ('synthesis_pending', '1', 0); `); }, }, { version: 11, description: 'Guard synthesis metadata lookups against malformed JSON', up: (db) => { // Existing v10 indexes keep their old expression under IF NOT EXISTS. // Rebuild transactionally; the guarded v10 definition also lets older // databases containing malformed metadata reach this migration safely. db.exec(` DROP INDEX IF EXISTS idx_edges_synthesis_site; CREATE INDEX idx_edges_synthesis_site ON edges(CASE WHEN json_valid(metadata) THEN json_extract(metadata, '$.registeredAt') END) WHERE CASE WHEN json_valid(metadata) THEN json_extract(metadata, '$.synthesizedBy') END IS NOT NULL; `); }, }, { version: 12, description: 'Retry a failed import when sync adds the file it names: path tails and failed-import indexes', up: (db) => { // Partial over failed imports, so both stay small. Keep the definitions // in lockstep with schema.sql. db.exec(` CREATE INDEX IF NOT EXISTS idx_unresolved_failed_import_tail ON unresolved_refs(reference_kind, name_tail) WHERE status = 'failed' AND reference_kind = 'imports'; CREATE INDEX IF NOT EXISTS idx_unresolved_failed_import_name ON unresolved_refs(reference_kind, reference_name) WHERE status = 'failed' AND reference_kind = 'imports'; `); // A path import parked before this version carries its dotted tail — // the extension or a path fragment — which no file's keys match. // Rewrite it to the path tail a failed import is parked under now. // Idempotent: a rewritten tail rewrites to itself. const update = db.prepare('UPDATE unresolved_refs SET name_tail = ? WHERE id = ?'); const rows = db .prepare("SELECT id, reference_name FROM unresolved_refs WHERE status = 'failed' AND reference_kind = 'imports' AND reference_name LIKE '%/%'") .all() as Array<{ id: number; reference_name: string }>; for (const row of rows) update.run(referenceNameTail(row.reference_name, 'imports'), row.id); }, }, { version: 13, description: 'Retry a failed path reference when sync adds the file it names: file-name tails', up: (db) => { // A path reference parked before this version — a Liquid // `snippets/price.liquid` — carries its extension as its tail, which no // node is named. Rewrite it to the file name a failed path reference is // parked under now. Idempotent: a rewritten tail rewrites to itself. const update = db.prepare('UPDATE unresolved_refs SET name_tail = ? WHERE id = ?'); const rows = db .prepare("SELECT id, reference_name, name_tail FROM unresolved_refs WHERE status = 'failed' AND reference_kind = 'references' AND reference_name LIKE '%/%.%'") .all() as Array<{ id: number; reference_name: string; name_tail: string }>; for (const row of rows) { const tail = referenceNameTail(row.reference_name, 'references'); if (tail !== row.name_tail) update.run(tail, row.id); } }, }, { version: 14, description: 'Retry a failed route module reference when sync adds or changes its file: module tails', up: (db) => { // A route's reference to the module it lazily loads, parked before this // version — React Router's `lazy-import:./pages/Team`, Vue Router's and // Angular's `import:./home/home.component#HomeComponent`, each also // behind `layout:` — carries a fragment of its path as its tail // ('/pages/Team', 'component#HomeComponent'), which no file's keys // match. Rewrite it to the module tail it is parked under now. // Idempotent: a rewritten tail rewrites to itself. // // Each prefix is a range of idx_unresolved_name. The `+` keeps the // planner off idx_unresolved_status, which it otherwise picks although // 'failed' is nearly every row: on vscode's index that read 830K rows, // about a second, to select none. const update = db.prepare('UPDATE unresolved_refs SET name_tail = ? WHERE id = ?'); const rows = db .prepare(`SELECT id, reference_name, reference_kind, name_tail FROM unresolved_refs WHERE (reference_name GLOB 'lazy-import:*' OR reference_name GLOB 'import:*' OR reference_name GLOB 'layout:*') AND +status = 'failed' AND +reference_kind IN ('references', 'calls')`) .all() as Array<{ id: number; reference_name: string; reference_kind: string; name_tail: string }>; for (const row of rows) { const tail = referenceNameTail(row.reference_name, row.reference_kind); if (tail !== row.name_tail) update.run(tail, row.id); } }, }, { version: 15, description: 'Retry a failed reference through an import binding by its whole name: module-tail name index', up: (db) => { // A failed reference through an import binding the module declares // under another name is parked under the module's key from this // version on, and still looked up by its whole name. Rows parked before // keep the tail their name gives, which the name lookup already finds; // a re-index parks them under their modules. Keep the definition in // lockstep with schema.sql. db.exec(` CREATE INDEX IF NOT EXISTS idx_unresolved_failed_module_name ON unresolved_refs(status, reference_name) WHERE status = 'failed' AND name_tail GLOB 'module:*'; `); }, }, { version: 16, description: 'Retry a failed member reference through an import binding by its member: module-tail member index', up: (db) => { // A failed member reference through an import binding (`store.get`) is // parked under the module's key from this version on, and still looked // up by its member, the tail its name gives. Rows parked before keep // that tail, which the tail lookup already finds; a re-index parks // them under their modules. Keep the definition in lockstep with // schema.sql: the expression is MEMBER_TAIL_SQL. db.exec(` CREATE INDEX IF NOT EXISTS idx_unresolved_failed_module_member ON unresolved_refs(status, ${MEMBER_TAIL_SQL}) WHERE ${MODULE_PARKED_SQL}; `); - src/db/name-substring-index.ts (8378 bytes)
/** * An in-memory answer to the substring lookup `codegraph_explore` runs for * every word of its query (#2620): * * SELECT … FROM nodes WHERE name LIKE '%word%' [AND name NOT LIKE 'word%'] * AND kind IN (…) ORDER BY length(name) LIMIT n * * A leading `%` can't use an index, so SQLite reads every node of those kinds * for each call: about 80 ms on a 560k-node index, four calls a word, and a * pasted log or document has hundreds of words. This index reads the names * once and answers each call with one string search. * * It returns exactly what SQLite would. LIKE folds case for ASCII letters only, * `_` matches any one character, and `length()` counts characters, not UTF-16 * units, so all three are done that way here. SQLite leaves rows of equal * length in the order its plan read them: through `idx_nodes_kind` (kind, * file_path, start_line, id) on a large index, by rowid when it scans a small * one. The index keeps both orders, and the caller says which plan SQLite * chose. A pattern with a `%` is left to SQLite. */ /** * The kinds the index holds: the definitions and callables explore's * camel-case and compound steps ask about, and the variables and constants its * named-symbol seeding adds. */ export const NAME_SUBSTRING_KINDS: readonly string[] = [ 'class', 'component', 'constant', 'enum', 'function', 'interface', 'method', 'protocol', 'struct', 'trait', 'type_alias', 'union', 'variable', ]; /** Separates the names in {@link NameSubstringIndex.hay}; no name may contain it. */ const SEPARATOR = '\u0000'; export interface NameSubstringIndex { /** Kind codes: `kinds[entryKind[e]]` is entry `e`'s kind. */ readonly kinds: readonly string[]; /** Every distinct ASCII-lowercased name, each followed by {@link SEPARATOR}. */ readonly hay: string; /** Where each distinct name starts in `hay`, plus one past the end. */ readonly starts: Int32Array; /** The entries of distinct name `i` are `nameEntries[nameEntryStart[i] … nameEntryStart[i+1])`, ascending. */ readonly nameEntryStart: Int32Array; readonly nameEntries: Int32Array; /** Per entry, in index order. */ readonly entryRowid: Float64Array; readonly entryKind: Uint8Array; readonly entryLength: Int32Array; } /** Lowercase ASCII letters only, as SQLite's LIKE compares them. */ function asciiLower(text: string): string { return /[A-Z]/.test(text) ? text.replace(/[A-Z]+/g, (run) => run.toLowerCase()) : text; } /** `length(text)` as SQLite counts it: characters, so a surrogate pair is one. */ function sqliteLength(text: string): number { let length = text.length; for (let i = 0; i < text.length; i++) { const code = text.charCodeAt(i); if (code >= 0xd800 && code <= 0xdbff && i + 1 < text.length) { const next = text.charCodeAt(i + 1); if (next >= 0xdc00 && next <= 0xdfff) { length--; i++; } } } return length; } /** * Build the index from rows of {@link NAME_SUBSTRING_KINDS} read in * `idx_nodes_kind` order, which becomes the entries' order. Returns null when * a name holds the separator, so the caller keeps asking SQLite. */ export function buildNameSubstringIndex( rows: Iterable<{ rowid: number; kind: string; name: string }>, ): NameSubstringIndex | null { const kindCodes = new Map(NAME_SUBSTRING_KINDS.map((kind, i) => [kind, i])); const nameIds = new Map<string, number>(); const names: string[] = []; const rowids: number[] = []; const kinds: number[] = []; const lengths: number[] = []; const nameOf: number[] = []; for (const row of rows) { const code = kindCodes.get(row.kind); if (code === undefined) continue; const lowered = asciiLower(row.name); if (lowered.includes(SEPARATOR)) return null; let id = nameIds.get(lowered); if (id === undefined) { id = names.length; nameIds.set(lowered, id); names.push(lowered); } rowids.push(row.rowid); kinds.push(code); lengths.push(sqliteLength(row.name)); nameOf.push(id); } const starts = new Int32Array(names.length + 1); let offset = 0; for (let i = 0; i < names.length; i++) { starts[i] = offset; offset += names[i]!.length + SEPARATOR.length; } starts[names.length] = offset; // Group entries by name, keeping index order inside each group. const nameEntryStart = new Int32Array(names.length + 1); for (const id of nameOf) nameEntryStart[id + 1]!++; for (let i = 0; i < names.length; i++) nameEntryStart[i + 1]! += nameEntryStart[i]!; const fill = nameEntryStart.slice(0, names.length); const nameEntries = new Int32Array(nameOf.length); for (let entry = 0; entry < nameOf.length; entry++) nameEntries[fill[nameOf[entry]!]!++] = entry; return { kinds: NAME_SUBSTRING_KINDS, hay: names.join(SEPARATOR) + SEPARATOR, starts, nameEntryStart, nameEntries, entryRowid: Float64Array.from(rowids), entryKind: Uint8Array.from(kinds), entryLength: Int32Array.from(lengths), }; } /** The distinct name whose span in `hay` holds `offset`. */ function nameAt(starts: Int32Array, offset: number): number { let lo = 0; let hi = starts.length - 2; while (lo < hi) { const mid = (lo + hi + 1) >> 1; if (starts[mid]! <= offset) lo = mid; else hi = mid - 1; } return lo; } /** * A matcher for a LIKE pattern without `%`: where the next match at or after * `from` starts, and whether one starts exactly at `at`. Without `_` this is a * plain string search; each `_` matches any one character but the separator. */ function patternMatcher(needle: string): { next(hay: string, from: number): number; startsAt(hay: string, at: number): boolean } { if (!needle.includes('_')) { return { next: (hay, from) => hay.indexOf(needle, from), startsAt: (hay, at) => hay.startsWith(needle, at), }; } // `u`, so a wildcard takes a whole character, as SQLite's `_` does. const source = needle .split('_') .map((part) => part.replace(/[\^$\\.*+?()[\]{}|/]/g, '\\$&')) .join('[^\\u0000]'); const anywhere = new RegExp(source, 'gu'); const anchored = new RegExp(source, 'uy'); return { next(hay, from) { anywhere.lastIndex = from; const match = anywhere.exec(hay); return match ? match.index : -1; }, startsAt(hay, at) { anchored.lastIndex = at; return anchored.test(hay); }, }; } /** * The order SQLite's plan reads candidate rows in, which its ORDER BY keeps * for rows of equal length: the kind index's, or the table's by rowid. */ export type ScanOrder = 'index' | 'rowid'; /** * The rowids SQLite's substring lookup returns, in its order, or null when the * index can't answer exactly (a `%` in the pattern, an empty pattern, or a * kind it doesn't hold). `order` is the order the plan reads rows in. */ export function searchNameSubstring( index: NameSubstringIndex, substring: string, kinds: readonly string[], excludePrefix: boolean, limit: number, order: ScanOrder, ): number[] | null { const needle = asciiLower(substring); if (needle.length === 0 || needle.includes('%') || needle.includes(SEPARATOR)) return null; const wanted = new Uint8Array(index.kinds.length); for (const kind of kinds) { const code = index.kinds.indexOf(kind); if (code < 0) return null; wanted[code] = 1; } const { hay, starts, nameEntryStart, nameEntries, entryKind, entryLength } = index; const matcher = patternMatcher(needle); const hits: number[] = []; for (let at = matcher.next(hay, 0); at >= 0; ) { const name = nameAt(starts, at); if (!(excludePrefix && matcher.startsAt(hay, starts[name]!))) { for (let k = nameEntryStart[name]!; k < nameEntryStart[name + 1]!; k++) { const entry = nameEntries[k]!; if (wanted[entryKind[entry]!]) hits.push(entry); } } at = matcher.next(hay, starts[name + 1]!); } // Entry numbers follow the kind index, rowids the table: either breaks // length ties the way SQLite's plan does. const rowid = index.entryRowid; hits.sort(order === 'index' ? (a, b) => (entryLength[a]! - entryLength[b]!) || (a - b) : (a, b) => (entryLength[a]! - entryLength[b]!) || (rowid[a]! - rowid[b]!)); // A negative LIMIT is no limit in SQLite. if (limit >= 0 && hits.length > limit) hits.length = limit; return hits.map((entry) => index.entryRowid[entry]!); } - src/db/node-sqlite.ts (1462 bytes)
/** * Node's built-in `node:sqlite`, loaded in one place, so that a runtime * without it fails with the one explanation that fits it: CodeGraph needs * Node.js 22.5+, or its own bundled runtime. Any other failure to open a * database is SQLite's own, and is reported as such (see ./open-error.ts). */ /** The parts of `node:sqlite` CodeGraph uses. */ export interface NodeSqliteModule { // eslint-disable-next-line @typescript-eslint/no-explicit-any DatabaseSync: new (path: string, options?: { readOnly?: boolean }) => any; } /** `node:sqlite` is missing or could not be loaded in this Node.js runtime. */ export class NodeSqliteUnavailableError extends Error { constructor(cause: unknown) { const reason = cause instanceof Error ? cause.message : String(cause); super( `Node.js ${process.version} could not load its built-in SQLite module (${reason}).\n` + 'CodeGraph requires node:sqlite (Node.js 22.5+). Install the self-contained CodeGraph ' + 'release (it bundles a compatible Node), or run on Node 22.5+.', { cause } ); this.name = 'NodeSqliteUnavailableError'; } } /** Load `node:sqlite`, or throw {@link NodeSqliteUnavailableError}. */ export function loadNodeSqlite(): NodeSqliteModule { try { // eslint-disable-next-line @typescript-eslint/no-require-imports return require('node:sqlite') as NodeSqliteModule; } catch (err) { throw new NodeSqliteUnavailableError(err); } } - src/db/open-error.ts (4669 bytes)
/** * Opening a CodeGraph database failed in SQLite itself: report what SQLite * said, with the database's path and a short hint at the usual cause. Before, * every such failure came out as "CodeGraph requires node:sqlite (Node.js * 22.5+)", which sent users after their Node version when the file was locked, * unreadable, or on a network share. That hint is now only for a runtime that * cannot load `node:sqlite` at all (./node-sqlite.ts); an index Windows and WSL * share keeps its own guidance (./wsl-shared-index.ts). */ import { NodeSqliteUnavailableError } from './node-sqlite'; import { toWslSharedIndexError, WslSharedIndexError } from './wsl-shared-index'; /** SQLite's primary result codes, by number. node:sqlite reports the extended code; its low byte is the primary one. */ const SQLITE_CODE_NAMES: readonly string[] = [ 'SQLITE_OK', 'SQLITE_ERROR', 'SQLITE_INTERNAL', 'SQLITE_PERM', 'SQLITE_ABORT', 'SQLITE_BUSY', 'SQLITE_LOCKED', 'SQLITE_NOMEM', 'SQLITE_READONLY', 'SQLITE_INTERRUPT', 'SQLITE_IOERR', 'SQLITE_CORRUPT', 'SQLITE_NOTFOUND', 'SQLITE_FULL', 'SQLITE_CANTOPEN', 'SQLITE_PROTOCOL', 'SQLITE_EMPTY', 'SQLITE_SCHEMA', 'SQLITE_TOOBIG', 'SQLITE_CONSTRAINT', 'SQLITE_MISMATCH', 'SQLITE_MISUSE', 'SQLITE_NOLFS', 'SQLITE_AUTH', 'SQLITE_FORMAT', 'SQLITE_RANGE', 'SQLITE_NOTADB', ]; /** What to check, for the failures an open meets in practice. */ const HINTS: Readonly<Record<string, string>> = { SQLITE_BUSY: 'Another process has the database locked; retry when it is done.', SQLITE_PERM: 'Check that you can read and write the file and its folder.', SQLITE_READONLY: 'The database or its folder is read-only, and CodeGraph has to write its index there.', SQLITE_IOERR: 'SQLite could not read or write the file. On a network share, or another filesystem without ' + 'reliable file locking, keep the project and its index on a local disk.', SQLITE_CORRUPT: 'The database is damaged; run "codegraph index" to rebuild it.', SQLITE_FULL: 'The disk is full.', SQLITE_CANTOPEN: 'Check that the folder exists and that you can read and write it, and that no other program ' + '(a backup or antivirus scan, say) holds the file open.', SQLITE_NOTADB: 'The file is not a SQLite database; move it aside and run "codegraph init" to build a new index.', }; /** A failure SQLite reported while opening the database at `dbPath`. */ export class DatabaseOpenError extends Error { /** node:sqlite's error code ("ERR_SQLITE_ERROR"), kept from the original. */ readonly code: unknown; /** SQLite's extended result code, kept from the original. */ readonly errcode: unknown; readonly errstr: unknown; /** SQLite's name for the primary result code, e.g. "SQLITE_CANTOPEN". */ readonly sqliteCode: string; readonly dbPath: string; constructor(original: Error, dbPath: string) { const codes = original as Error & { code?: unknown; errcode?: unknown; errstr?: unknown }; const primary = typeof codes.errcode === 'number' ? codes.errcode & 0xff : -1; const sqliteCode = SQLITE_CODE_NAMES[primary] ?? `SQLite error ${String(codes.errcode)}`; const hint = HINTS[sqliteCode]; super( `Could not open the CodeGraph database at ${dbPath}: ${original.message} (${sqliteCode}).${hint ? ` ${hint}` : ''}`, { cause: original } ); this.name = 'DatabaseOpenError'; this.code = codes.code; this.errcode = codes.errcode; this.errstr = codes.errstr; this.sqliteCode = sqliteCode; this.dbPath = dbPath; } } /** * The error to throw for `err`, raised while opening the database at * `dbPath`: the Windows/WSL guidance for an index the two share, a * {@link DatabaseOpenError} for any other failure SQLite reported (it carries * a result code), and anything else unchanged, the missing-`node:sqlite` * explanation included. */ export function toDatabaseOpenError(err: unknown, dbPath: string): unknown { if (err instanceof DatabaseOpenError || err instanceof WslSharedIndexError || err instanceof NodeSqliteUnavailableError) { return err; } const shared = toWslSharedIndexError(err, dbPath); if (shared) return shared; if (err instanceof Error && typeof (err as { errcode?: unknown }).errcode === 'number') { return new DatabaseOpenError(err, dbPath); } return err; } /** * A project's index exists but could not be opened: what a caller reports as * that, not as "no index here". The Windows/WSL case has its own guidance. */ export function isDatabaseOpenFailure(err: unknown): err is DatabaseOpenError | NodeSqliteUnavailableError { return err instanceof DatabaseOpenError || err instanceof NodeSqliteUnavailableError; } - src/db/parameter-flow-queries.ts (13341 bytes)
/** * What the parameter-flow pass reads and writes (see * ../resolution/parameter-flow). A pass makes these reads by the thousand, so * each shape is prepared once, and the ones whose planner choice matters name * their index: a full index runs the pass before it has planner stats, and * left to itself SQLite scans a language's nodes, or every failed row, for each. */ import type { Edge, Language, Node, UnresolvedReference } from '../types'; import type { QueryBuilder } from './queries'; import { referenceNameTail } from './reference-tail'; import type { SqliteDatabase, SqliteStatement } from './sqlite-adapter'; /** One call a language's code makes, as the pass reads it. */ export interface CallSiteRow { callerId: string; filePath: string; line: number; column: number; refName: string; /** What it resolved to; null for a call that failed. */ targetId: string | null; confidence: number; /** The edge that holds it, or the failed row. */ edgeId?: number; rowId?: number; /** A parameter flow answered it: its edge says so, or its failed row was once a guess a flow ruled out. */ viaFlow: boolean; } /** What a sync compares of a file before and after it changes it. */ export interface FlowFacts { /** Its classes, modules and constants, with what they inherit and mix in, as written. */ classes: string[]; /** The names of the methods it declares. */ methods: string[]; /** The references it calls. */ calls: string[]; } /** What the pass reads of a method, a function or a class. */ export type NodeOutline = Pick<Node, 'id' | 'kind' | 'name' | 'qualifiedName' | 'filePath' | 'startLine' | 'startColumn' | 'endLine'> & { /** 1 for a static method. */ isStatic: number; }; /** Where each caller a read of call sites takes is; a caller it doesn't know isn't one. */ type CallerFiles = { get(id: string): { filePath: string } | undefined }; type CallEdgeRow = { id: number; source: string; target: string; line: number | null; col: number | null; metadata: string | null }; type FailedCallRow = { id: number; from_node_id: string; line: number; col: number; reference_name: string; file_path: string; candidates: string | null }; /** How a failed row says a parameter flow ruled its guess out. */ const FLOW_MARK = '["parameter-flow"]'; /** Set once a pass has written to the index. */ const FLOW_MARK_KEY = 'parameter_flows'; /** The node columns of an outline. */ const NODE_OUTLINE = `id, kind, name, qualified_name AS qualifiedName, file_path AS filePath, start_line AS startLine, start_column AS startColumn, end_line AS endLine, is_static AS isStatic`; /** What the pass reads as a method. */ const METHOD_KINDS = `kind IN ('method', 'function')`; const REACHED_METHODS = `SELECT ${NODE_OUTLINE} FROM (SELECT DISTINCT target FROM edges WHERE kind = 'calls') CROSS JOIN nodes ON id = target WHERE ${METHOD_KINDS} AND language = ?`; const SCOPES_IN_FILE = `SELECT ${NODE_OUTLINE} FROM nodes WHERE file_path = ? AND kind IN ('class', 'module')`; const OUTLINE = `SELECT ${NODE_OUTLINE}, language FROM nodes WHERE id = ?`; const METHOD_NAMED = `SELECT 1 FROM nodes INDEXED BY idx_nodes_name WHERE name = ? AND ${METHOD_KINDS} AND language = ? LIMIT 1`; const CALL_INTO = `SELECT 1 FROM edges WHERE target = ? AND kind = 'calls' LIMIT 1`; const EDGE_COLUMNS = 'id, source, target, line, col, metadata'; const CALLS_INTO = `SELECT ${EDGE_COLUMNS} FROM edges WHERE target = ? AND kind = 'calls'`; const CALLS_OUT_OF = `SELECT ${EDGE_COLUMNS} FROM edges WHERE source = ? AND kind = 'calls'`; const CALLS_OF_NAME = `SELECT e.id, e.source, e.target, e.line, e.col, e.metadata FROM nodes n CROSS JOIN edges e ON e.target = n.id WHERE n.name = ? AND e.kind = 'calls'`; const FAILED_CALL_COLUMNS = 'SELECT id, from_node_id, line, col, reference_name, file_path, candidates FROM unresolved_refs'; const FAILED_CALL_SITE = `${FAILED_CALL_COLUMNS} WHERE status = 'failed' AND reference_kind = 'calls' AND language = ?`; const FAILED_CALLS_OF_NAME = `${FAILED_CALL_SITE} AND name_tail = ?`; const FAILED_CALLS_BY = `${FAILED_CALL_COLUMNS} INDEXED BY idx_unresolved_from_node WHERE from_node_id = ? AND status = 'failed' AND reference_kind = 'calls' AND language = ?`; const FACT_NODES = `SELECT kind, name, qualified_name AS owner, start_line AS line, start_column AS col FROM nodes WHERE file_path = ? AND kind IN ('class', 'module', 'variable', 'constant', 'method', 'function')`; const FACT_EDGES = `SELECT e.kind, json_extract(e.metadata, '$.refName') AS ref, e.line, e.col, n.qualified_name AS owner FROM nodes n CROSS JOIN edges e ON e.source = n.id WHERE n.file_path = ? AND e.kind IN ('calls', 'extends', 'implements') ORDER BY e.id`; const FACT_REFS = `SELECT r.reference_kind AS kind, r.reference_name AS ref, r.line, r.col, n.qualified_name AS owner FROM unresolved_refs r LEFT JOIN nodes n ON n.id = r.from_node_id WHERE r.file_path = ? AND r.reference_kind IN ('calls', 'extends', 'implements') ORDER BY r.id`; const NAMES_PER_READ = 500; function toFailedCallSite(row: FailedCallRow): CallSiteRow { return { callerId: row.from_node_id, filePath: row.file_path, line: row.line, column: row.col, refName: row.reference_name, targetId: null, confidence: 0, rowId: row.id, viaFlow: row.candidates === FLOW_MARK, }; } export class ParameterFlowQueries { private readonly statements = new Map<string, SqliteStatement>(); constructor(private readonly db: SqliteDatabase, private readonly queries: QueryBuilder) {} private statement(sql: string): SqliteStatement { let statement = this.statements.get(sql); if (!statement) this.statements.set(sql, (statement = this.db.prepare(sql))); return statement; } /** The indexed files of a language. */ getFilePathsByLanguage(language: Language): string[] { const rows = this.statement('SELECT path FROM files WHERE language = ?').all(language) as Array<{ path: string }>; return rows.map((row) => row.path); } /** A language's methods that a call reaches, as outlines. */ getReachedMethods(language: Language): NodeOutline[] { return this.statement(REACHED_METHODS).all(language) as NodeOutline[]; } /** A language's methods of these names, as outlines. */ getMethodsNamed(names: readonly string[], language: Language): NodeOutline[] { const outlines: NodeOutline[] = []; for (let i = 0; i < names.length; i += NAMES_PER_READ) { const chunk = names.slice(i, i + NAMES_PER_READ); outlines.push(...this.db.prepare( `SELECT ${NODE_OUTLINE} FROM nodes INDEXED BY idx_nodes_name WHERE name IN (${chunk.map(() => '?').join(', ')}) AND ${METHOD_KINDS} AND language = ?` ).all(...chunk, language) as NodeOutline[]); } return outlines; } /** The classes and modules a file declares, as outlines. */ getScopesInFile(filePath: string): NodeOutline[] { return this.statement(SCOPES_IN_FILE).all(filePath) as NodeOutline[]; } /** One node's outline, with its language. */ getNodeOutline(id: string): (NodeOutline & { language: Language }) | undefined { return this.statement(OUTLINE).get(id) as (NodeOutline & { language: Language }) | undefined; } /** Whether a language has a method of this name. */ hasMethodNamed(name: string, language: Language): boolean { return this.statement(METHOD_NAMED).get(name, language) !== undefined; } /** Whether any call reaches a node. */ hasCallInto(id: string): boolean { return this.statement(CALL_INTO).get(id) !== undefined; } /** * What a file holds that a flow reads: see FlowFacts. A call that failed counts * as much as one that resolved, so it reads the same whichever way a sync's * resolution goes. The order is the file's, which an ancestor's depends on. */ getFlowFacts(filePath: string): FlowFacts { type Fact = { kind: string; line: number | null; col: number | null; owner: string | null; ref?: string | null; name?: string }; const facts = [FACT_NODES, FACT_EDGES, FACT_REFS].flatMap((sql) => this.statement(sql).all(filePath)) as Fact[]; facts.sort((a, b) => (a.line ?? 0) - (b.line ?? 0) || (a.col ?? 0) - (b.col ?? 0)); const found: FlowFacts = { classes: [], methods: [], calls: [] }; for (const fact of facts) { if (fact.kind === 'method' || fact.kind === 'function') found.methods.push(fact.name!); else if (fact.kind === 'calls') { if (fact.ref) found.calls.push(fact.ref); } else if (fact.ref !== null) found.classes.push(`${fact.kind} ${fact.owner} ${fact.ref ?? ''}`); } return found; } /** * The calls edges into a node, out of one, or into any node of a name, from * `callers`, with what they reached and how sure that was. Synthesized edges * carry no reference. */ getCallEdges(of: { target: string } | { source: string } | { targetName: string }, callers: CallerFiles): CallSiteRow[] { const rows = ('target' in of ? this.statement(CALLS_INTO).all(of.target) : 'source' in of ? this.statement(CALLS_OUT_OF).all(of.source) : this.statement(CALLS_OF_NAME).all(of.targetName)) as CallEdgeRow[]; return this.toCallSites(rows, callers); } private toCallSites(rows: CallEdgeRow[], callers: CallerFiles): CallSiteRow[] { const sites: CallSiteRow[] = []; for (const row of rows) { const caller = callers.get(row.source); if (!caller || row.line === null || row.metadata === null) continue; const metadata = JSON.parse(row.metadata) as { refName?: string; confidence?: number; parameterFlow?: string }; if (metadata.refName === undefined) continue; sites.push({ callerId: row.source, filePath: caller.filePath, line: row.line, column: row.col ?? 0, refName: metadata.refName, targetId: row.target, confidence: metadata.confidence ?? 0, edgeId: row.id, viaFlow: metadata.parameterFlow !== undefined, }); } return sites; } /** A language's failed calls of a method name, or made by one caller. */ getFailedCallSites(language: Language, of: { tail: string } | { callerId: string }): CallSiteRow[] { const rows = ('tail' in of ? this.statement(FAILED_CALLS_OF_NAME).all(language, of.tail) : this.statement(FAILED_CALLS_BY).all(of.callerId, language)) as FailedCallRow[]; return rows.map(toFailedCallSite); } /** * What an earlier pass did and left in the index: its edges, and the failed * rows of the guesses it ruled out. Nothing until a pass has written one: on a * fresh index, looking is a scan of every edge. */ getFlowCallSites(language: Language, callers: CallerFiles): CallSiteRow[] { if (this.queries.getMetadata(FLOW_MARK_KEY) === null) return []; const edges = this.db.prepare( `SELECT ${EDGE_COLUMNS} FROM edges WHERE kind = 'calls' AND metadata LIKE '%"parameterFlow"%'` ).all() as CallEdgeRow[]; const rows = this.db.prepare(`${FAILED_CALL_SITE} AND candidates = ?`).all(language, FLOW_MARK) as FailedCallRow[]; return [...this.toCallSites(edges, callers), ...rows.map(toFailedCallSite)]; } /** Remember that a pass wrote to this index. */ noteParameterFlows(): void { this.queries.setMetadata(FLOW_MARK_KEY, '1'); } /** * Swap what held each call a flow answers, an edge or a failed row, for its new * edge, as one transaction: a call left with neither has nothing to bring it back. */ retargetCalls(edgeIds: number[], rowIds: number[], edges: Edge[]): void { this.db.transaction(() => { this.queries.deleteEdgesByIds(edgeIds); this.queries.deleteReferencesByRowIds(rowIds); this.queries.insertEdges(edges); })(); } /** Put the calls no flow answers now back as references to resolve, as one transaction, for the same reason. */ restoreCalls(edgeIds: number[], rowIds: number[], refs: UnresolvedReference[]): void { this.db.transaction(() => { this.queries.deleteReferencesByRowIds(rowIds); this.queries.replaceResolutionEdgesWithUnresolvedRefs(edgeIds, refs); })(); } /** * Replace guesses a flow ruled out with failed rows that say so, as one * transaction: the edge is gone, and a later pass whose flow is gone puts the * reference back through resolution. */ replaceEdgesWithFlowMarkedRefs(edgeIds: number[], refs: UnresolvedReference[]): void { const insert = this.db.prepare( `INSERT INTO unresolved_refs (from_node_id, reference_name, reference_kind, line, col, candidates, file_path, language, status, name_tail) VALUES (?, ?, ?, ?, ?, ?, ?, ?, 'failed', ?)` ); if (refs.length === 0) return; this.db.transaction(() => { this.queries.deleteEdgesByIds(edgeIds); for (const ref of refs) { insert.run(ref.fromNodeId, ref.referenceName, ref.referenceKind, ref.line, ref.column, FLOW_MARK, ref.filePath ?? '', ref.language ?? 'unknown', referenceNameTail(ref.referenceName, ref.referenceKind)); } })(); } /** What the last pass over a language left for the next sync: null or blank for nothing to go on. */ getRecord(language: Language): string | null { return this.queries.getMetadata(`parameter_flow_record_${language}`); } setRecord(language: Language, record: string): void { this.queries.setMetadata(`parameter_flow_record_${language}`, record); } } - src/db/queries.ts (264607 bytes)
/** * Database Queries * * Prepared statements for CRUD operations on the knowledge graph. */ import { SqliteDatabase, SqliteStatement } from './sqlite-adapter'; import { Node, Edge, FileRecord, UnresolvedReference, NodeKind, EdgeKind, Language, GraphStats, SearchOptions, SearchResult, } from '../types'; import { safeJsonParse } from '../utils'; import { isTestFile, kindBonus, nameMatchBonus, scorePathRelevance } from '../search/query-utils'; import { parseQuery, boundedEditDistance, type ParsedQuery } from '../search/query-parser'; import { isGeneratedFile } from '../extraction/generated-detection'; import { splitIdentifierSegments } from '../search/identifier-segments'; import { isModuleKey, referenceNameTail, MEMBER_TAIL_SQL, MODULE_PARKED_SQL } from './reference-tail'; import { ParameterFlowQueries } from './parameter-flow-queries'; import { NAME_SUBSTRING_KINDS, buildNameSubstringIndex, searchNameSubstring, type NameSubstringIndex, type ScanOrder } from './name-substring-index'; /** * Files that should not be candidates for "dominant file" detection: tests, * examples and tool-generated files. Generated files (`*.pb.go`, `*.pulsar.go`, * mock outputs, …) often have huge in-file edge counts that dwarf the real * source — etcd's `rpc.pb.go` has 4× the in-file edges of `server.go`. * * Path patterns plus, when the caller passes the indexed set, files whose * HEADER declares them generated — a `payroll.go` full of generated CRUD has * exactly the same edge-density problem as `rpc.pb.go` and nothing in its name * to catch it (#1500). */ function isLowValueFile(filePath: string, generated?: ReadonlySet<string>): boolean { if (generated?.has(filePath)) return true; // Tests, examples, fixtures and benchmarks by the engine's own reading — // LLVM's `unittests/`, Google's `foo_unittest.cc`, a C++ `FooTest.cpp` — // plus the singular `__test__/` some projects use. return isTestFile(filePath) || /(?:^|\/)__test__\//i.test(filePath) || isGeneratedFile(filePath); } const SQLITE_PARAM_CHUNK_SIZE = 500; /** * How an import names a folder only through where it is written: `.`, or a * folder up to twelve levels above (`..`, `../..`, with or without the * trailing `/`). */ const FOLDER_SPECIFIERS: readonly string[] = (() => { const names = ['.', './']; for (let levels = 1; levels <= 12; levels++) { const up = Array.from({ length: levels }, () => '..').join('/'); names.push(up, `${up}/`); } return names; })(); /** A name a member reference's receiver can be: not a file's name, nor a `module:` key. */ const RECEIVER_NAME = /^[A-Za-z_$][\w$]*$/; /** * A SQL predicate: is the node aliased `alias` a member an INTERFACE declares? * * `method_signature` / `property_signature` enter the graph as `method` / * `property` nodes hung off their interface by a `contains` edge (#1638). They * have no body and originate no behaviour, so for a structural judgement about * a FILE they are the interface restated, not an extra thing the file declares. * See {@link QueryBuilder.getAmbientDeclarationPathsAmong}, the one caller, for * why treating them as opaque would break that rule in three places at once. * * Seeks `idx_edges_target_kind`, so it costs a key lookup per row rather than a * join over the whole edge table. */ const IS_INTERFACE_MEMBER = (alias: string): string => `EXISTS ( SELECT 1 FROM edges ce JOIN nodes owner ON owner.id = ce.source WHERE ce.target = ${alias}.id AND ce.kind = 'contains' AND owner.kind = 'interface' )`; /** * A SQL predicate: is the reference name in `column` written through one of * the names in the JSON array bound to the next two parameters, as the name * itself or as a member read through it (`Order.objects` through `Order`)? */ const THROUGH_NAMES = (column: string): string => `( ${column} IN (SELECT value FROM json_each(?)) OR (instr(${column}, '.') > 1 AND substr(${column}, 1, instr(${column}, '.') - 1) IN (SELECT value FROM json_each(?))) )`; /** * How much of the exact-name bonus a `deprioritize`d path keeps (#982). Damped * rather than zeroed: a query that genuinely targets that tree must still rank * it, the same "discount, don't erase" rule the path penalty follows. * * Derived rather than picked. `nameMatchBonus`'s prefix arm tops out below * `10 + 30 = 40`, and a de-prioritized node also takes the -15 path penalty, so * `80 * SCALE - 15 > 40` is what stops a damped WHOLE-QUERY exact match from * losing to a mere prefix match. 0.75 clears it (45). Measured on a 62k-node * django index: at 0.25 that invariant breaks in practice — `child`, `parent` * and `method` lose rank 1 to `children`, `all_parents` and `method_decorator` * — while crowd-out removal is almost flat between 0.75 and 0.5 (39 vs 40 of 88 * peripheral top-10 slots cleared), so a deeper discount buys little and costs * the invariant. Pinned by a test. */ export const DEPRIORITIZED_NAME_BONUS_SCALE = 0.75; /** * A `deprioritize` path predicate (#982) that can also hand out a copy with * the project's patterns read once, for loops over many paths (#2620). */ export interface DeprioritizedPathMatcher { (filePath: string): boolean; snapshot?: () => (filePath: string) => boolean; } /** * The fields that tell one node of a file from another across a re-index of * that file, whose node ids change with every line shift (#2276). */ export type NodeIdentity = Pick< Node, 'id' | 'kind' | 'name' | 'qualifiedName' | 'signature' | 'startLine' | 'startColumn' >; /** * Database row types (snake_case from SQLite) */ interface NodeRow { id: string; kind: string; name: string; qualified_name: string; file_path: string; language: string; start_line: number; end_line: number; start_column: number; end_column: number; docstring: string | null; signature: string | null; visibility: string | null; is_exported: number; is_async: number; is_static: number; is_abstract: number; decorators: string | null; type_parameters: string | null; return_type: string | null; updated_at: number; } interface EdgeRow { id: number; source: string; target: string; kind: string; metadata: string | null; line: number | null; col: number | null; provenance: string | null; } interface FileRow { path: string; content_hash: string; language: string; size: number; modified_at: number; indexed_at: number; node_count: number; errors: string | null; /** Absent on pre-v9 rows read through a stale prepared statement. */ generated?: number | null; } interface UnresolvedRefRow { id: number; from_node_id: string; reference_name: string; reference_kind: string; line: number; col: number; candidates: string | null; file_path: string; language: string; status: string; name_tail: string; } /** * Convert database row to Node object */ function rowToNode(row: NodeRow): Node { return { id: row.id, kind: row.kind as NodeKind, name: row.name, qualifiedName: row.qualified_name, filePath: row.file_path, language: row.language as Language, startLine: row.start_line, endLine: row.end_line, startColumn: row.start_column, endColumn: row.end_column, docstring: row.docstring ?? undefined, signature: row.signature ?? undefined, visibility: row.visibility as Node['visibility'], isExported: row.is_exported === 1, isAsync: row.is_async === 1, isStatic: row.is_static === 1, isAbstract: row.is_abstract === 1, decorators: row.decorators ? safeJsonParse(row.decorators, undefined) : undefined, typeParameters: row.type_parameters ? safeJsonParse(row.type_parameters, undefined) : undefined, returnType: row.return_type ?? undefined, updatedAt: row.updated_at, }; } /** * rowToNode for rows that arrive as arrays, whose columns are in `columns` order. * Name and id lookups are most of what resolving a reference spends in SQLite, * and a row as an array costs about a third less to read than one as an object. */ function arrayToNode(columns: string[]): (row: unknown[]) => Node { const at = (column: keyof NodeRow): number => columns.indexOf(column); const id = at('id'); const kind = at('kind'); const name = at('name'); const qualifiedName = at('qualified_name'); const filePath = at('file_path'); const language = at('language'); const startLine = at('start_line'); const endLine = at('end_line'); const startColumn = at('start_column'); const endColumn = at('end_column'); const docstring = at('docstring'); const signature = at('signature'); const visibility = at('visibility'); const isExported = at('is_exported'); const isAsync = at('is_async'); const isStatic = at('is_static'); const isAbstract = at('is_abstract'); const decorators = at('decorators'); const typeParameters = at('type_parameters'); const returnType = at('return_type'); const updatedAt = at('updated_at'); return (row) => ({ id: row[id] as string, kind: row[kind] as NodeKind, name: row[name] as string, qualifiedName: row[qualifiedName] as string, filePath: row[filePath] as string, language: row[language] as Language, startLine: row[startLine] as number, endLine: row[endLine] as number, startColumn: row[startColumn] as number, endColumn: row[endColumn] as number, docstring: (row[docstring] ?? undefined) as string | undefined, signature: (row[signature] ?? undefined) as string | undefined, visibility: row[visibility] as Node['visibility'], isExported: row[isExported] === 1, isAsync: row[isAsync] === 1, isStatic: row[isStatic] === 1, isAbstract: row[isAbstract] === 1, decorators: row[decorators] ? safeJsonParse(row[decorators] as string, undefined) : undefined, typeParameters: row[typeParameters] ? safeJsonParse(row[typeParameters] as string, undefined) : undefined, returnType: (row[returnType] ?? undefined) as string | undefined, updatedAt: row[updatedAt] as number, }); } /** A prepared `SELECT * FROM nodes …`, whose rows come back as nodes. */ interface NodeStatement { all(...params: unknown[]): Node[]; get(...params: unknown[]): Node | undefined; iterate(...params: unknown[]): IterableIterator<Node>; } function prepareNodeStatement(db: SqliteDatabase, sql: string): NodeStatement { const stmt = db.prepare(sql); const columns = stmt.rowsAsArrays(); const toNode = (columns ? arrayToNode(columns) : rowToNode) as (row: unknown) => Node; return { all: (...params) => (stmt.all(...params) as unknown[]).map(toNode), get: (...params) => { const row = stmt.get(...params); return row === undefined ? undefined : toNode(row); }, *iterate(...params) { for (const row of stmt.iterate(...params)) yield toNode(row); }, }; } /** * A search's `path:` and `name:` filters, lowercased once. The values of one * family are OR'd and the two families AND'd, and each value is a literal, * case-insensitive substring of the file path or the symbol's name, as the * query parser documents. JavaScript decides each row a statement returns, * since `toLowerCase` folds Unicode, where SQLite's `lower()` and LIKE fold * ASCII only, and LIKE would read a `%` or `_` in the value as a wildcard; the * statement's own SQL only narrows the rows (see fieldFilterSql). */ interface FieldFilters { paths: string[]; names: string[]; } /** The query's `path:`/`name:` filters, or undefined when it has none. */ function fieldFiltersOf(parsed: ParsedQuery): FieldFilters | undefined { if (parsed.pathFilters.length === 0 && parsed.nameFilters.length === 0) return undefined; return { paths: parsed.pathFilters.map((p) => p.toLowerCase()), names: parsed.nameFilters.map((n) => n.toLowerCase()), }; } function passesNameFilters(name: string, filters: FieldFilters): boolean { if (filters.names.length === 0) return true; const lowered = name.toLowerCase(); return filters.names.some((n) => lowered.includes(n)); } function passesFieldFilters(name: string, filePath: string, filters: FieldFilters): boolean { if (filters.paths.length > 0) { const lowered = filePath.toLowerCase(); if (!filters.paths.some((p) => lowered.includes(p))) return false; } return passesNameFilters(name, filters); } const PRINTABLE_ASCII = /^[ -~]*$/; /** * WHERE terms (each starting with ` AND `) that keep every `nodes` row the * filters could pass, so that SQLite drops most of the others before they * reach JavaScript, which still decides (see passesFieldFilters). On text of * printable ASCII alone, SQLite's `lower()` and `instr()` agree exactly with * `toLowerCase()` and `includes()`, so a printable ASCII value is tested in * SQL. Text holding any other character goes on to JavaScript whatever it * holds, and that is the only text a value with another character can match. */ function fieldFilterSql(filters: FieldFilters, params: (string | number)[]): string { let sql = ''; for (const [column, values] of [['nodes.file_path', filters.paths], ['nodes.name', filters.names]] as const) { if (values.length === 0) continue; const terms: string[] = []; for (const value of values) { if (!PRINTABLE_ASCII.test(value)) continue; terms.push(`instr(lower(${column}), ?) > 0`); params.push(value); } terms.push(`${column} GLOB '*[^ -~]*'`); sql += ` AND (${terms.join(' OR ')})`; } return sql; } /** * Convert database row to Edge object */ /** A failed python ref through one attribute (`self.c.send`) or a call receiver (`make().send`). */ const FAILED_PY_ATTR_REF = `status = 'failed' AND language = 'python' AND reference_name NOT GLOB '*.*.*.*' AND (reference_name GLOB '*.*.*' OR reference_name GLOB '*().*')`; function rowToEdge(row: EdgeRow): Edge { return { source: row.source, target: row.target, kind: row.kind as EdgeKind, metadata: row.metadata ? safeJsonParse(row.metadata, undefined) : undefined, line: row.line ?? undefined, column: row.col ?? undefined, provenance: row.provenance as Edge['provenance'], }; } /** * Convert database row to FileRecord object */ function rowToFileRecord(row: FileRow): FileRecord { return { path: row.path, contentHash: row.content_hash, language: row.language as Language, size: row.size, modifiedAt: row.modified_at, indexedAt: row.indexed_at, nodeCount: row.node_count, errors: row.errors ? safeJsonParse(row.errors, undefined) : undefined, generated: row.generated === 1, }; } /** * Files ranked by edges between two of their own symbols — the input to * `getDominantFile`. CROSS JOIN pins the loop order to edges first, then a * lookup by id for each end: one pass over the edges whatever the statistics * say. With a plain JOIN the planner may start from nodes and, for each node, * walk every node in its file and probe edges by (source, target) — q - src/db/reference-tail.ts (8175 bytes)
/** * The keys sync's failed-ref retry (#1240) matches on: the tail a failed ref * is parked under, and the names a newly added file can be imported as. * Shared by the query layer, which writes and reads them, and the migrations * that rewrote the tails parked before path imports, path references and * module references had ones of their own. */ /** * Last segment of a (possibly dotted/qualified) reference name — the part a * new symbol's plain node name could match: 'util.greet' → 'greet', * 'mod::fn' → 'fn', 'greet' → 'greet'. Written to unresolved_refs.name_tail * when a ref is marked failed, so the #1240 retry lookup can match dotted * refs against newly-added node names. * * An import written as a path names a FILE, so its tail is the path's last * segment cut at its first dot — `package:app/b.dart` → 'b', `./foo` → 'foo', * `inc/db.php` → 'db' — the stem {@link importPathKeys} takes from a file * that could satisfy it. The dotted tail was the extension ('dart', 'php') or * a path fragment ('/foo'), which no file ever matched. * * A route's reference to the module it lazily loads names a file the same * way, by the module's path with or without its extension: React Router's * `lazy-import:./pages/Team`, Vue Router's and Angular's * `import:./home/home.component#HomeComponent`. It is parked under that * path's stem behind 'module:' — 'module:Team', 'module:home' — the key * {@link moduleReferenceKeys} gives a file that could be the module. A bare * stem would be any symbol's name, which a module reference does not wait * for: 'index' and 'types' are the tails of thousands of failed calls. Its * dotted tail was a path fragment ('/pages/Team') or what follows the * module's own dot ('component#HomeComponent'), which no file ever matched. * * Any other reference written as a path to a file names it by the file's own * name: Liquid's `{% render 'price' %}` is `snippets/price.liquid`, parked as * 'price.liquid' — the name path matching looks the file up by, and the name * of the node a file that appears later is given. Its dotted tail was the * extension, 'liquid'. A call's slashes are in its arguments, a comment or a * division, so only a `references` ref is read as a path. */ export function referenceNameTail(referenceName: string, referenceKind?: string): string { if (referenceKind === 'imports' && referenceName.replace(/\/+$/, '').includes('/')) { const stem = pathStem(referenceName); if (stem) return stem; } // Vue Router's route names the component it renders as a call. if (referenceKind === 'references' || referenceKind === 'calls') { const route = MODULE_REFERENCE.exec(referenceName); const tail = route ? moduleTail(route[1] ?? route[2]!) : ''; if (tail) return tail; } if (referenceKind === 'references') { const fileName = referenceName.slice(referenceName.lastIndexOf('/') + 1); if (fileName !== referenceName && FILE_NAME.test(fileName)) return fileName; } // Erlang refs carry a written arity (`f/1`, `mod::fn/2` — #1610); the tail a // new symbol's plain name could match is the arity-less function name. const base = referenceName.replace(/\/\d{1,3}$/, '') || referenceName; const idx = Math.max(base.lastIndexOf('.'), base.lastIndexOf(':')); return idx >= 0 ? base.slice(idx + 1) : base; } /** A file name with an extension: 'price.liquid', 'icon.logo.liquid', 'about.tsx' — not 'x.component#Name'. */ const FILE_NAME = /^[\w$@+~.-]+\.[A-Za-z][A-Za-z0-9]*$/; /** * A route's reference to the module it lazily loads, capturing the module's * path: React Router's `lazy-import:<path>`, Vue Router's and Angular's * `import:<path>#<export>` — a path may hold a `#` itself, as vben's `#/views/…` * alias does — each also behind `layout:`, for a layout route's module. */ const MODULE_REFERENCE = /^(?:layout:)?(?:lazy-import:(.+)|import:(.+)#[^#]+)$/; /** What a module reference's tail starts with. No symbol's name does, so a lookup by symbol names never finds one. */ const MODULE_KEY = 'module:'; /** Whether a tail or lookup key waits for a module (`module:Team`) rather than for a name. */ export function isModuleKey(key: string): boolean { return key.startsWith(MODULE_KEY); } /** * SQL for the tail a reference parked under its module (`module:*`) has by its * own name: what follows the name's last dot — `store.get` → 'get' — or the * whole name. A member reference through an import binding still waits for * its member (see * `importBindingTail`), so the retry, the navigation retry and the dead-code * check look it up by that. idx_unresolved_failed_module_member indexes the * expression, and SQLite uses an expression index only for a query that * spells the expression exactly so: schema.sql and the v16 migration do. */ export const MEMBER_TAIL_SQL = "substr(reference_name, length(rtrim(reference_name, replace(reference_name, '.', ''))) + 1)"; /** The rows parked under a module: the partial WHERE of the indexes that find them. */ export const MODULE_PARKED_SQL = "status = 'failed' AND name_tail GLOB 'module:*'"; /** * The tail of a reference that waits for a module rather than for a name: the * module path's stem behind 'module:' — `./pages/Team` → 'module:Team', * `@/views/error/404.vue` → 'module:404' — the key * {@link moduleReferenceKeys} gives each file that could be the module, or '' * when the path's last segment is only dots. */ export function moduleTail(modulePath: string): string { const stem = pathStem(modulePath); return stem ? MODULE_KEY + stem : ''; } /** A path's last segment up to its first dot — `./pages/Team` → 'Team', `a/b.dart` → 'b' — or '' when that is only dots. */ function pathStem(path: string): string { const trimmed = path.replace(/\/+$/, ''); const stem = fileNameStem(trimmed.slice(trimmed.lastIndexOf('/') + 1)); // `..` and `.` name a folder only through the referring file's location. return /^\.*$/.test(stem) ? '' : stem; } /** A file or folder name up to its first dot: 'b.dart' → 'b', 'types.d.ts' → 'types', '.eslintrc.js' → '.eslintrc'. */ function fileNameStem(name: string): string { const dot = name.indexOf('.', 1); return dot > 0 ? name.slice(0, dot) : name; } /** * What an import of the file at `filePath` can be written as, for the failed * imports a sync that adds the file retries: its name (`b.h`, a bare include) * and stem (`b`, the tail of `a/b.h`, `./b`, `pkg.b`), and the same two for * its folder, which an import names for its index file (`./ui` → * `ui/index.ts`, `pkg` → `pkg/__init__.py`) or as a Go package. */ export function importPathKeys(filePath: string): string[] { const parts = filePath.split('/'); const keys = new Set<string>(); for (const name of parts.slice(-2)) { keys.add(name); keys.add(fileNameStem(name)); } return [...keys]; } /** * The keys a module reference the file at `filePath` could satisfy is parked * under: its {@link importPathKeys} behind 'module:' — `src/pages/Team.tsx` * is the module of `lazy-import:./pages/Team`, `pages/Team/index.tsx` of the * same path through its folder. A sync looks them up for the files it adds * AND the ones it changes: a route renders the component its module exports, * so an edit that gives the module one is what the route waited for. So does * a reference through an import binding of the module, and a direct member * reference through any binding (`importBindingTail`), which are parked under * the same keys. */ export function moduleReferenceKeys(filePath: string): string[] { const keys = importPathKeys(filePath); // A Python package's `__init__.py` is a submodule of the package above it, // as a module file beside it is through its folder: `app/pkg/sub/__init__.py` // is what `from app.pkg import sub as s`, parked as 'module:pkg', imports. const parts = filePath.split('/'); const above = parts.length >= 3 && parts[parts.length - 1] === '__init__.py' ? parts[parts.length - 3]! : ''; if (above && !keys.includes(above)) keys.push(above); return keys.map((key) => MODULE_KEY + key); } - src/db/schema.sql (11730 bytes)
-- CodeGraph SQLite Schema -- Version 1 -- Schema version tracking CREATE TABLE IF NOT EXISTS schema_versions ( version INTEGER PRIMARY KEY, applied_at INTEGER NOT NULL, description TEXT ); -- Insert initial version INSERT INTO schema_versions (version, applied_at, description) VALUES (1, strftime('%s', 'now') * 1000, 'Initial schema'); -- ============================================================================= -- Core Tables -- ============================================================================= -- Nodes: Code symbols (functions, classes, variables, etc.) CREATE TABLE IF NOT EXISTS nodes ( id TEXT PRIMARY KEY, kind TEXT NOT NULL, name TEXT NOT NULL, qualified_name TEXT NOT NULL, file_path TEXT NOT NULL, language TEXT NOT NULL, start_line INTEGER NOT NULL, end_line INTEGER NOT NULL, start_column INTEGER NOT NULL, end_column INTEGER NOT NULL, docstring TEXT, signature TEXT, visibility TEXT, is_exported INTEGER DEFAULT 0, is_async INTEGER DEFAULT 0, is_static INTEGER DEFAULT 0, is_abstract INTEGER DEFAULT 0, decorators TEXT, -- JSON array type_parameters TEXT, -- JSON array return_type TEXT, -- normalized return/result type name (e.g. C++ method return, for receiver-type inference) updated_at INTEGER NOT NULL ); -- Edges: Relationships between nodes CREATE TABLE IF NOT EXISTS edges ( id INTEGER PRIMARY KEY AUTOINCREMENT, source TEXT NOT NULL, target TEXT NOT NULL, kind TEXT NOT NULL, metadata TEXT, -- JSON object line INTEGER, col INTEGER, provenance TEXT DEFAULT NULL, FOREIGN KEY (source) REFERENCES nodes(id) ON DELETE CASCADE, FOREIGN KEY (target) REFERENCES nodes(id) ON DELETE CASCADE ); -- Files: Tracked source files. -- `generated` is the index-time verdict from extraction/generated-detection.ts: -- the filename convention (*.pb.go, *.g.dart, …) OR a generation banner in the -- file's header. Go's convention is a CONTENT marker, so a generated -- `payroll.go` beside hand-written use-cases is invisible to the path check -- alone (#1500) — deciding it here means ranking never reads file headers per -- request. Migration v9 adds the column to existing databases; rows keep the -- 0 default until the next full index, so readers treat it as a hint that -- only ever ADDS to the path signal, never overrides it. CREATE TABLE IF NOT EXISTS files ( path TEXT PRIMARY KEY, content_hash TEXT NOT NULL, language TEXT NOT NULL, size INTEGER NOT NULL, modified_at INTEGER NOT NULL, indexed_at INTEGER NOT NULL, node_count INTEGER DEFAULT 0, errors TEXT, -- JSON array generated INTEGER NOT NULL DEFAULT 0 ); -- Unresolved References: References that need resolution after full indexing. -- status lifecycle: rows are inserted 'pending' by extraction; a completed -- resolution pass either deletes a row (resolved) or marks it 'failed' -- (attempted, no match — kept so a later sync can retry it when a changed -- file introduces a symbol that could satisfy it, #1240). name_tail is the -- last segment of reference_name ('util.greet' → 'greet'), written when a -- row is marked failed, so the retry lookup matches new node names against -- dotted refs too. Rows follow their from_node via ON DELETE CASCADE, so -- re-extracting or deleting a file clears its stale rows in any status. CREATE TABLE IF NOT EXISTS unresolved_refs ( id INTEGER PRIMARY KEY AUTOINCREMENT, from_node_id TEXT NOT NULL, reference_name TEXT NOT NULL, reference_kind TEXT NOT NULL, line INTEGER NOT NULL, col INTEGER NOT NULL, candidates TEXT, -- JSON array file_path TEXT NOT NULL DEFAULT '', language TEXT NOT NULL DEFAULT 'unknown', status TEXT NOT NULL DEFAULT 'pending', name_tail TEXT NOT NULL DEFAULT '', FOREIGN KEY (from_node_id) REFERENCES nodes(id) ON DELETE CASCADE ); -- ============================================================================= -- Indexes for Query Performance -- ============================================================================= -- Node indexes CREATE INDEX IF NOT EXISTS idx_nodes_kind ON nodes(kind, file_path, start_line, id); CREATE INDEX IF NOT EXISTS idx_nodes_name ON nodes(name); CREATE INDEX IF NOT EXISTS idx_nodes_qualified_name ON nodes(qualified_name); CREATE INDEX IF NOT EXISTS idx_nodes_file_path ON nodes(file_path); CREATE INDEX IF NOT EXISTS idx_nodes_language ON nodes(language); CREATE INDEX IF NOT EXISTS idx_nodes_file_line ON nodes(file_path, start_line); CREATE INDEX IF NOT EXISTS idx_nodes_lower_name ON nodes(lower(name)); -- Full-text search index on node names, docstrings, and signatures CREATE VIRTUAL TABLE IF NOT EXISTS nodes_fts USING fts5( id, name, qualified_name, docstring, signature, content='nodes', content_rowid='rowid' ); -- Triggers to keep FTS index in sync CREATE TRIGGER IF NOT EXISTS nodes_ai AFTER INSERT ON nodes BEGIN INSERT INTO nodes_fts(rowid, id, name, qualified_name, docstring, signature) VALUES (NEW.rowid, NEW.id, NEW.name, NEW.qualified_name, NEW.docstring, NEW.signature); END; CREATE TRIGGER IF NOT EXISTS nodes_ad AFTER DELETE ON nodes BEGIN INSERT INTO nodes_fts(nodes_fts, rowid, id, name, qualified_name, docstring, signature) VALUES ('delete', OLD.rowid, OLD.id, OLD.name, OLD.qualified_name, OLD.docstring, OLD.signature); END; CREATE TRIGGER IF NOT EXISTS nodes_au AFTER UPDATE ON nodes BEGIN INSERT INTO nodes_fts(nodes_fts, rowid, id, name, qualified_name, docstring, signature) VALUES ('delete', OLD.rowid, OLD.id, OLD.name, OLD.qualified_name, OLD.docstring, OLD.signature); INSERT INTO nodes_fts(rowid, id, name, qualified_name, docstring, signature) VALUES (NEW.rowid, NEW.id, NEW.name, NEW.qualified_name, NEW.docstring, NEW.signature); END; -- Prose-word → symbol-name lookup for the prompt hook's graph-derived gate. -- One row per (segment, name): segment is a lowercased word of a symbol name -- ("OrderStateMachine" → order, state, machine — see identifier-segments.ts), -- which lets natural-language prompt words be verified against the graph in -- any language whose technical nouns are Latin script. File nodes are -- excluded — a file's basename duplicates the symbols inside it and skews the -- singleton-vs-cluster rarity statistics. FTS can't serve this lookup (its -- tokenizer keeps camelCase names as single tokens), so segments are -- materialized on the node write path. -- Deletions leave orphan rows ON PURPOSE: rows are PROPOSALS, always -- re-verified against nodes before being surfaced (CodeGraph.getSegmentMatches), -- and a full index prunes them once it completes (a fresh one starts the table -- empty). Populated lazily on old databases (empty until the next index/sync -- heals it). CREATE TABLE IF NOT EXISTS name_segment_vocab ( segment TEXT NOT NULL, name TEXT NOT NULL, PRIMARY KEY (segment, name) ) WITHOUT ROWID; -- Edge indexes. -- idx_edges_source / idx_edges_target are intentionally omitted — -- the (source, kind) and (target, kind) composites below cover the -- corresponding source-only / target-only lookups via SQLite's -- left-prefix scan, so the narrow indexes are dead weight on writes. -- Migration v4 drops them on existing databases. CREATE INDEX IF NOT EXISTS idx_edges_kind ON edges(kind); CREATE INDEX IF NOT EXISTS idx_edges_source_kind ON edges(source, kind); CREATE INDEX IF NOT EXISTS idx_edges_target_kind ON edges(target, kind); -- Edge identity uniqueness. An edge IS uniquely (source, target, kind, line, -- col); insertEdge uses `INSERT OR IGNORE`, but without something UNIQUE to -- conflict on it behaved like a plain INSERT, so two passes emitting the same -- edge produced byte-identical duplicate rows that inflated counts and flowed -- into callers/impact (#1034). IFNULL folds the nullable line/col so -- coordinate-less edges (synthesized / file-level) dedup too — SQLite treats -- each NULL as distinct otherwise. Migration v6 dedups existing rows + adds -- this on older databases. CREATE UNIQUE INDEX IF NOT EXISTS idx_edges_identity ON edges(source, target, kind, IFNULL(line, -1), IFNULL(col, -1)); -- File indexes. -- idx_files_generated is PARTIAL: the generated set is a small minority of any -- repo, so a lookup that intersects a bounded candidate list with it stays -- proportional to the generated files, not to the repo. CREATE INDEX IF NOT EXISTS idx_files_language ON files(language); CREATE INDEX IF NOT EXISTS idx_files_modified_at ON files(modified_at); CREATE INDEX IF NOT EXISTS idx_files_generated ON files(path) WHERE generated = 1; -- Unresolved refs indexes CREATE INDEX IF NOT EXISTS idx_unresolved_from_node ON unresolved_refs(from_node_id); CREATE INDEX IF NOT EXISTS idx_unresolved_name ON unresolved_refs(reference_name); CREATE INDEX IF NOT EXISTS idx_unresolved_file_path ON unresolved_refs(file_path); CREATE INDEX IF NOT EXISTS idx_unresolved_from_name ON unresolved_refs(from_node_id, reference_name); CREATE INDEX IF NOT EXISTS idx_unresolved_status ON unresolved_refs(status); CREATE INDEX IF NOT EXISTS idx_unresolved_failed_tail ON unresolved_refs(name_tail) WHERE status = 'failed'; -- Sync's failed-import retry looks failed imports up by tail and by whole -- name. The leading reference_kind is what makes the planner pick these over -- idx_unresolved_failed_tail / idx_unresolved_name, whose per-key averages -- hide how many failed calls a common tail (`index`, `types`) has. CREATE INDEX IF NOT EXISTS idx_unresolved_failed_import_tail ON unresolved_refs(reference_kind, name_tail) WHERE status = 'failed' AND reference_kind = 'imports'; CREATE INDEX IF NOT EXISTS idx_unresolved_failed_import_name ON unresolved_refs(reference_kind, reference_name) WHERE status = 'failed' AND reference_kind = 'imports'; -- A failed ref through an import binding is parked under its module -- (`module:tag`) and looked up by its whole name as well. The leading status -- is the equality term that keeps the planner on this small index. CREATE INDEX IF NOT EXISTS idx_unresolved_failed_module_name ON unresolved_refs(status, reference_name) WHERE status = 'failed' AND name_tail GLOB 'module:*'; -- A member reference through an import binding (`store.get`) is parked under -- its module too, and still looked up by its member: the tail its name gives, -- what follows its last dot. Spell the expression exactly as MEMBER_TAIL_SQL -- (reference-tail.ts) does, or SQLite will not use this index. CREATE INDEX IF NOT EXISTS idx_unresolved_failed_module_member ON unresolved_refs(status, substr(reference_name, length(rtrim(reference_name, replace(reference_name, '.', ''))) + 1)) WHERE status = 'failed' AND name_tail GLOB 'module:*'; CREATE INDEX IF NOT EXISTS idx_edges_provenance ON edges(provenance); -- Sync's third-file wiring lookup must not scan every synthesized edge. -- CASE short-circuits malformed metadata; keep these expressions identical -- in migrations and synthesis queries so SQLite can use the partial index. CREATE INDEX IF NOT EXISTS idx_edges_synthesis_site ON edges(CASE WHEN json_valid(metadata) THEN json_extract(metadata, '$.registeredAt') END) WHERE CASE WHEN json_valid(metadata) THEN json_extract(metadata, '$.synthesizedBy') END IS NOT NULL; -- Retain the cheap source-gate verdict when a later sync deletes the source. CREATE TABLE IF NOT EXISTS synthesis_inputs ( file_path TEXT PRIMARY KEY REFERENCES files(path) ON DELETE CASCADE ); -- Project metadata for version/provenance tracking CREATE TABLE IF NOT EXISTS project_metadata ( key TEXT PRIMARY KEY, value TEXT NOT NULL, updated_at INTEGER NOT NULL ); - src/db/sqlite-adapter.ts (8226 bytes)
/** * SQLite Adapter * * Thin wrapper over Node's built-in `node:sqlite` (`DatabaseSync`), exposed * through a small better-sqlite3-shaped interface so the rest of the codebase * is storage-agnostic. * * CodeGraph ships with a bundled Node runtime, so `node:sqlite` (real SQLite, * with WAL + FTS5) is always available — there is no native build step and no * wasm fallback. When run from source instead, it requires Node >= 22.5. */ import { loadNodeSqlite } from './node-sqlite'; import { toDatabaseOpenError } from './open-error'; import { toWslSharedIndexError } from './wsl-shared-index'; export interface SqliteStatement { run(...params: any[]): { changes: number; lastInsertRowid: number | bigint }; get(...params: any[]): any; all(...params: any[]): any[]; /** * Lazily yield result rows one at a time instead of materializing the whole * set with `all()`. Use for unbounded scans (e.g. every function/method node) * so memory stays O(1) in the row count rather than O(rows) — see #610, where * `all()`-ing every symbol on a dense project spiked the heap into an OOM. */ iterate(...params: any[]): IterableIterator<any>; /** * From now on, hand each row over as an array of its values rather than an * object keyed by column, which is about a third cheaper to read. Returns the * column names, in array order, or undefined on a runtime that can't, whose * rows stay objects (Node before 22.16). */ rowsAsArrays(): string[] | undefined; } export interface TransactionOptions { /** * Begin with `BEGIN IMMEDIATE`, taking the write lock up front, instead of a * deferred `BEGIN`. A connection that finds the lock taken then waits it out * (busy_timeout); a deferred transaction that has already read gets * SQLITE_BUSY at once when it comes to write, since SQLite will not wait for * a lock there. No effect when nested in an open transaction. */ immediate?: boolean; } export interface SqliteDatabase { prepare(sql: string): SqliteStatement; exec(sql: string): void; pragma(str: string, options?: { simple?: boolean }): any; transaction<T>(fn: (...args: any[]) => T, options?: TransactionOptions): (...args: any[]) => T; close(): void; readonly open: boolean; /** Undefined on a runtime without `isTransaction`; callers must then not memoize. */ readonly inTransaction?: boolean; } /** * The active SQLite backend. Only one now (`node:sqlite`); kept as a named type * so `codegraph status` and the per-instance reporting have a stable shape. */ export type SqliteBackend = 'node-sqlite'; /** * Wraps Node's built-in `node:sqlite` (`DatabaseSync`) to match the * better-sqlite3 interface the rest of the code expects. * * node:sqlite is real SQLite compiled into Node, so it supports WAL, FTS5, * mmap, and `@named` params natively — the only shims needed are the * better-sqlite3 conveniences node:sqlite omits: a `.pragma()` helper, a * `.transaction()` helper, and `open` (node:sqlite exposes `isOpen`). */ class NodeSqliteAdapter implements SqliteDatabase { private _db: any; private _txDepth = 0; private readonly _dbPath: string; constructor(dbPath: string, opts?: { readOnly?: boolean }) { const { DatabaseSync } = loadNodeSqlite(); this._dbPath = dbPath; this._db = opts?.readOnly ? new DatabaseSync(dbPath, { readOnly: true }) : new DatabaseSync(dbPath); } /** * What a failed call throws: the error itself, or — for a "disk I/O error" * on a WSL index that Windows CodeGraph shares — the actionable rewrite * (#995). Every open runs its PRAGMAs and first reads through the methods * below, and so does every later query. `iterate()` is left raw: a * row-by-row wrapper would tax the unbounded scans it exists for, and a * session reads through `get`/`all` long before it reaches one. */ private failure(err: unknown): unknown { return toWslSharedIndexError(err, this._dbPath) ?? err; } get inTransaction(): boolean | undefined { return this._db.isTransaction; } get open(): boolean { return this._db.isOpen; } prepare(sql: string): SqliteStatement { // node:sqlite matches better-sqlite3's calling convention (variadic // positional args, or a single object for @named params), so params forward // through unchanged. let stmt: any; try { stmt = this._db.prepare(sql); } catch (err) { throw this.failure(err); } const failure = (err: unknown): unknown => this.failure(err); return { run(...params: any[]) { let r: any; try { r = stmt.run(...params); } catch (err) { throw failure(err); } return { changes: Number(r?.changes ?? 0), lastInsertRowid: r?.lastInsertRowid ?? 0, }; }, get(...params: any[]) { try { return stmt.get(...params); } catch (err) { throw failure(err); } }, all(...params: any[]) { try { return stmt.all(...params); } catch (err) { throw failure(err); } }, iterate(...params: any[]) { return stmt.iterate(...params); }, rowsAsArrays() { if (typeof stmt.setReturnArrays !== 'function' || typeof stmt.columns !== 'function') return undefined; stmt.setReturnArrays(true); return stmt.columns().map((column: { name: string }) => column.name); }, }; } exec(sql: string): void { try { this._db.exec(sql); } catch (err) { throw this.failure(err); } } pragma(str: string, options?: { simple?: boolean }): any { const trimmed = str.trim(); // Write pragma ("key = value"): node:sqlite is real SQLite, so every pragma // (WAL, mmap, synchronous, …) applies as-is. if (trimmed.includes('=')) { this.exec(`PRAGMA ${trimmed}`); return; } // Read pragma. Default: the row object (e.g. { journal_mode: 'wal' }). // `{ simple: true }` returns just the single column value, like better-sqlite3. const row = this.prepare(`PRAGMA ${trimmed}`).get(); if (options?.simple) { return row && typeof row === 'object' ? Object.values(row)[0] : row; } return row; } transaction<T>(fn: (...args: any[]) => T, options?: TransactionOptions): (...args: any[]) => T { return (...args: any[]) => { // Nested call (a transaction()-wrapped helper invoked from inside another // transaction): run the body directly inside the enclosing transaction. // BEGIN would throw "cannot start a transaction within a transaction", // so no existing caller ever relied on nested rollback granularity — // flattening is behavior-preserving and free. if (this._txDepth > 0) { this._txDepth++; try { return fn(...args); } finally { this._txDepth--; } } this._db.exec(options?.immediate ? 'BEGIN IMMEDIATE' : 'BEGIN'); this._txDepth = 1; try { const result = fn(...args); this._db.exec('COMMIT'); this._txDepth = 0; return result; } catch (error) { this._db.exec('ROLLBACK'); this._txDepth = 0; throw error; } }; } close(): void { // node:sqlite's DatabaseSync.close() throws if already closed; make it // idempotent to match better-sqlite3 (callers may close more than once). if (this._db.isOpen) this._db.close(); } } /** * Create a database connection backed by `node:sqlite`. * * Returns the active backend alongside the db so each `DatabaseConnection` can * report it per-instance — MCP can open multiple project DBs in one process, so * a process-global would race. */ export function createDatabase(dbPath: string, opts?: { readOnly?: boolean }): { db: SqliteDatabase; backend: SqliteBackend } { try { return { db: new NodeSqliteAdapter(dbPath, opts), backend: 'node-sqlite' }; } catch (error) { // A runtime without node:sqlite says so (NodeSqliteUnavailableError); // anything SQLite itself reported names its code and the path. throw toDatabaseOpenError(error, dbPath); } } - src/db/synthesis-stage.ts (4203 bytes)
import { createDatabase, type SqliteDatabase } from './sqlite-adapter'; import { QueryBuilder } from './queries'; import { createYielder } from '../resolution/cooperative-yield'; // Ownership is independent of provenance: Go method containment is structural. export const SYNTHESIZED_EDGE = "CASE WHEN json_valid(metadata) THEN json_extract(metadata, '$.synthesizedBy') END IS NOT NULL"; /** A private edge overlay: passes see base edges plus their new Go prerequisites. */ export class SynthesisStage { readonly db: SqliteDatabase; readonly queries: QueryBuilder; constructor(dbPath: string) { this.db = createDatabase(dbPath).db; try { this.db.pragma('busy_timeout = 5000'); this.db.pragma('foreign_keys = ON'); this.db.pragma('synchronous = NORMAL'); this.db.pragma('wal_autocheckpoint = 0'); this.db.exec(` CREATE TEMP TABLE synthesis_inputs (file_path TEXT PRIMARY KEY); CREATE TEMP TABLE synthesis_edges ( id INTEGER PRIMARY KEY, source TEXT, target TEXT, kind TEXT, metadata TEXT, line INTEGER, col INTEGER, provenance TEXT ); CREATE UNIQUE INDEX temp.synthesis_identity ON synthesis_edges (source, target, kind, IFNULL(line, -1), IFNULL(col, -1)); CREATE INDEX temp.synthesis_source ON synthesis_edges(source, kind); CREATE INDEX temp.synthesis_target ON synthesis_edges(target, kind); CREATE TEMP VIEW edges AS SELECT * FROM main.edges WHERE NOT COALESCE((${SYNTHESIZED_EDGE}), 0) UNION ALL SELECT * FROM synthesis_edges; CREATE TEMP TRIGGER synthesis_insert INSTEAD OF INSERT ON edges BEGIN INSERT OR IGNORE INTO synthesis_edges (source, target, kind, metadata, line, col, provenance) SELECT NEW.source, NEW.target, NEW.kind, NEW.metadata, NEW.line, NEW.col, NEW.provenance WHERE NOT EXISTS ( SELECT 1 FROM main.edges WHERE source = NEW.source AND target = NEW.target AND kind = NEW.kind AND IFNULL(line, -1) = IFNULL(NEW.line, -1) AND IFNULL(col, -1) = IFNULL(NEW.col, -1) AND NOT COALESCE((${SYNTHESIZED_EDGE}), 0) ); END; `); this.queries = new QueryBuilder(this.db); } catch (error) { this.db.close(); throw error; } } async publish(backpressure?: () => Promise<void> | null): Promise<void> { await backpressure?.(); const yieldToLoop = createYielder(); // Keep the replacement atomic for other connections. Yield between bounded // writes, but checkpoint only OUTSIDE this transaction (uncommitted frames // cannot be folded). Pass execution/staging above never writes the main WAL. this.db.exec('BEGIN IMMEDIATE'); try { const remove = this.db.prepare(`DELETE FROM main.edges WHERE id IN ( SELECT id FROM main.edges WHERE ${SYNTHESIZED_EDGE} LIMIT 2000 )`); while (remove.run().changes > 0) await yieldToLoop(); const insert = this.db.prepare(`INSERT OR IGNORE INTO main.edges (source, target, kind, metadata, line, col, provenance) SELECT source, target, kind, metadata, line, col, provenance FROM synthesis_edges WHERE id > ? AND id <= ?`); const max = this.db.prepare('SELECT MAX(id) AS id FROM synthesis_edges').get()?.id ?? 0; for (let i = 0; i < max; i += 2000) { insert.run(i, i + 2000); await yieldToLoop(); } // Only indexed files: an input names a file the passes read, but a lazy // value's module can resolve to a workspace package's directory // (`import('icons')`), and a file can leave the index while the passes // run. Either broke the foreign key into files(path) and rolled the whole // sync back (#2565). this.db.exec(`DELETE FROM main.synthesis_inputs; INSERT INTO main.synthesis_inputs (file_path) SELECT file_path FROM temp.synthesis_inputs WHERE file_path IN (SELECT path FROM main.files)`); this.db.exec('COMMIT'); } catch (error) { this.db.exec('ROLLBACK'); throw error; } await backpressure?.(); } close(): void { this.db.close(); } } - src/db/wal-valve.ts (17535 bytes)
/** * WAL checkpoint valve — bounds WAL growth while auto-checkpointing is * deferred during a bulk index (#1231). * * Why deferral: SQLite's default `wal_autocheckpoint` (1000 pages) re-writes * hot B-tree/FTS pages into the main DB file over and over during a bulk * index — measured at ~95% of ALL disk I/O, and the difference between 45s * and 19+ minutes on HDD-class storage (150 random IOPS). Deferring * checkpoints turns the store into pure sequential WAL appends; each backfill * pass writes distinct pages once, in page order (≈ sequential). * * Why a valve: unbounded deferral is its own failure mode, both measured in * the #1231 repro. The WAL duplicates hot pages per COMMIT, so it grows far * faster than the DB (5.9GB WAL for a ~340MB DB on a 3.3k-file index) — * filling the disk, and poisoning every subsequent read that must page * through it (the first resolution-phase read blocked the main thread >60s * and the #850 liveness watchdog killed the healthy index). The valve * watches WAL growth on a timer and, past a soft threshold, backfills with * `PRAGMA wal_checkpoint(PASSIVE)` on a worker-thread connection — PASSIVE * never blocks the writer, and off-thread means the main thread (and the * watchdog heartbeat) keep turning regardless of how long a backfill takes. * * The load-bearing subtlety: a WAL file's SIZE never shrinks. After a full * backfill, the writer's next commit RESTARTS the WAL from the top and the * frames recycle inside the same file — so raw size says nothing about the * un-backfilled backlog, and a size-triggered valve degenerates into firing * (and pausing the writer) forever once the file passes its threshold * (measured: guava crawled at ~9min per 160 files). Instead the valve * tracks `sizeAtLastFullBackfill` — refreshed whenever a checkpoint reports * `log === checkpointed` (everything backfilled) — and triggers on GROWTH * beyond that baseline, which only happens when genuinely un-backfilled * frames push past the file's high-water mark. * * Backpressure: if the writer outruns the checkpointer past a hard cap of * growth (2× soft), {@link backpressure} pauses the writer (at a safe, * between-transactions boundary) until a FULL backfill lands. One in-flight * pass is not enough: on a disk saturated by the writer, every concurrent * PASSIVE pass is already stale by the time it finishes (the writer appended * past its snapshot), so neither SQLite's WAL wrap nor the baseline ever * trigger and the WAL grows without bound (measured: 5.9GB on guava at 150 * IOPS, then a >60s read stall and a watchdog kill). With the writer parked, * the next pass covers everything, the WAL wraps on the following commit, * and the pause is the disk's honest catch-up cost — the correct terminal * mode when hardware genuinely can't keep up with the append rate. * * Fail-closed (#1539): if parked backfills cannot progress (a reader pinning * frames) while the WAL is past the hard/file caps, the valve throws * {@link WalValveAbortError} instead of releasing the writer. The previous * "futility latch" disabled parking for 60s after consecutive give-ups so a * pinned reader would not churn checkpoint workers — but that also let the * WAL grow without a bound (observed 64 GiB on a kernel-scale daemon catch-up * with the query pool holding read marks). Aborting with a clear error is the * safe terminal mode; the caller closes readers / retries once the pin clears. */ import type { DatabaseConnection } from './index'; /** * Thrown when the valve cannot checkpoint past its documented caps while a * reader pins WAL frames (#1539). Callers (index/sync) should surface this and * stop writing rather than risk unbounded disk growth. */ export class WalValveAbortError extends Error { readonly code = 'WAL_VALVE_ABORT' as const; readonly walBytes: number; readonly fileCapBytes: number; readonly hardBytes: number; constructor(message: string, sizes: { walBytes: number; fileCapBytes: number; hardBytes: number }) { super(message); this.name = 'WalValveAbortError'; this.walBytes = sizes.walBytes; this.fileCapBytes = sizes.fileCapBytes; this.hardBytes = sizes.hardBytes; } } /** Soft WAL-growth threshold (MB) that triggers an off-thread passive checkpoint. */ const DEFAULT_WAL_VALVE_MB = 256; /** Hard cap = this × soft threshold; past it the writer pauses for a full backfill. */ const HARD_CAP_MULTIPLIER = 2; /** File cap = this × soft threshold; past it the barrier also TRUNCATEs the file. */ const FILE_CAP_MULTIPLIER = 4; /** Passes attempted per writer pause before giving up (a pinned reader could stall forever). */ const MAX_PAUSED_BACKFILL_PASSES = 20; /** How often the timer looks at the WAL file size. */ const CHECK_INTERVAL_MS = 2000; /** * Resolve the valve's soft threshold from the `CODEGRAPH_WAL_VALVE_MB` * override; non-numeric / non-positive values fall back to the default. */ export function resolveWalValveMb(envVal: string | undefined, dbSizeBytes?: number): number { if (envVal !== undefined && envVal !== '') { const n = Number(envVal); if (Number.isFinite(n) && n > 0) return Math.floor(n); } // Scale with the project when the caller knows the DB size: every fold // re-writes hot B-tree pages into the main file (the #1231 pathology in // bounded form — 111s of a kernel-scale batch loop at the flat 256MB cap, // §7a.2), so a big project affords a proportionally bigger transient WAL // (~dbSize/4 soft ⇒ file cap ≈ dbSize) in exchange for ~4× fewer folds. if (dbSizeBytes !== undefined && dbSizeBytes > 0) { return Math.min(2048, Math.max(DEFAULT_WAL_VALVE_MB, Math.floor(dbSizeBytes / 4 / (1024 * 1024)))); } return DEFAULT_WAL_VALVE_MB; } export class WalCheckpointValve { private timer: ReturnType<typeof setInterval> | null = null; private inflight: Promise<void> | null = null; /** Writer pause in progress (hard cap breached): passes loop until a full backfill. */ private pause: Promise<void> | null = null; /** * WAL file size observed when a checkpoint last reported the ENTIRE WAL * backfilled. Growth is measured against this baseline — see the header * comment for why absolute size cannot be used. */ private sizeAtLastFullBackfill = 0; private readonly softBytes: number; private readonly hardBytes: number; private readonly fileCapBytes: number; /** * Consecutive parked-backfill give-ups. Used only for diagnostics in the * abort message — parking is never disabled (#1539 fail-closed). */ private consecutiveGiveUps = 0; constructor( private readonly db: DatabaseConnection, softMb: number = resolveWalValveMb(process.env.CODEGRAPH_WAL_VALVE_MB), private readonly intervalMs: number = CHECK_INTERVAL_MS, log: (msg: string) => void = () => {} ) { this.softBytes = softMb * 1024 * 1024; this.hardBytes = this.softBytes * HARD_CAP_MULTIPLIER; this.fileCapBytes = this.softBytes * FILE_CAP_MULTIPLIER; // CODEGRAPH_WAL_VALVE_DEBUG=1 surfaces valve decisions to stderr without // needing the caller's verbose plumbing — the observability gap that let // §7a.1 run 1 fail silently (give-ups were verbose-gated and invisible). this.log = process.env.CODEGRAPH_WAL_VALVE_DEBUG ? (m) => console.error(`[wal-valve] ${m}`) : log; } private readonly log: (msg: string) => void; private mb(n: number): string { return `${Math.round(n / 1024 / 1024)}MB`; } /** Un-backfilled growth estimate: bytes the WAL has grown past the last full backfill. */ private growthBytes(): number { return this.db.getWalSizeBytes() - this.sizeAtLastFullBackfill; } /** Begin watching the WAL. Idempotent; the timer never holds the loop open. */ start(): void { if (this.timer) return; // One armed line per run under either diagnostics env: §7a.1's failed // kernel-scale runs burned three 25-minute cycles before "is the valve // even alive?" could be answered. if (process.env.CODEGRAPH_SYNTH_TIMINGS || process.env.CODEGRAPH_WAL_VALVE_DEBUG) { console.error(`[wal-valve] armed soft=${this.mb(this.softBytes)} hard=${this.mb(this.hardBytes)} wal=${this.mb(this.db.getWalSizeBytes())}`); } let ticks = 0; this.timer = setInterval(() => { if ((++ticks % 15) === 0) { this.log(`alive: wal=${this.mb(this.db.getWalSizeBytes())} baseline=${this.mb(this.sizeAtLastFullBackfill)} inflight=${this.inflight ? 'y' : 'n'} paused=${this.pause ? 'y' : 'n'}`); } this.check(); }, this.intervalMs); this.timer.unref?.(); } /** Stop watching. Any in-flight checkpoint keeps running — await drain(). */ stop(): void { if (this.timer) { clearInterval(this.timer); this.timer = null; } } /** One poll: fire an off-thread passive checkpoint when growth passes the soft threshold. */ check(): void { if (!this.pause && !this.inflight && this.growthBytes() > this.softBytes) this.fire(); } /** * Writer-side backstop, called at a between-transactions boundary. Returns * null (no wait) while growth is under the hard cap; past it, returns a * promise that resolves only once a FULL backfill has landed — see the * header comment for why a single pass is not enough on a saturated disk. */ backpressure(): Promise<void> | null { if (this.pause) return this.pause; // Two independent triggers: // - growth: un-backfilled BACKLOG past the hard cap (the original valve). // - file size: a WAL can stay fully backfilled and still grow without // bound — the writer only restarts at frame 0 if a commit finds no // reader marks, which §7a.1's instrumented run showed never happens // in practice (file marched 361→721MB through two COMPLETE // backfills). Past the file cap, park and TRUNCATE at the barrier — // the backfill part is instant when the backlog is already folded. if (this.growthBytes() <= this.hardBytes && this.db.getWalSizeBytes() <= this.fileCapBytes) return null; this.log(`backpressure: wal=${this.mb(this.db.getWalSizeBytes())} baseline=${this.mb(this.sizeAtLastFullBackfill)} — pausing writer for full backfill`); const t0 = Date.now(); this.pause = this.backfillFully().finally(() => { this.pause = null; this.log(`backpressure released after ${Date.now() - t0}ms: wal=${this.mb(this.db.getWalSizeBytes())} baseline=${this.mb(this.sizeAtLastFullBackfill)}`); }); return this.pause; } /** Await any in-flight checkpoint and writer pause. */ async drain(): Promise<void> { while (this.pause || this.inflight) { if (this.pause) await this.pause; if (this.inflight) await this.inflight; } } /** * Phase-boundary fold: backfill the ENTIRE WAL now (off-thread, awaited). * Called between bulk phases — e.g. after parsing, before resolution's * first reads — so the next phase never pages a bulk-write-sized WAL on * the main thread (the post-parse read against a multi-GB WAL is what * blew the #850 watchdog's 60s window in the #1231 repro). The await * keeps the event loop (and the watchdog heartbeat) turning. */ async foldNow(): Promise<void> { await this.drain(); if (this.growthBytes() <= 0) return; this.log(`foldNow: wal=${this.mb(this.db.getWalSizeBytes())} baseline=${this.mb(this.sizeAtLastFullBackfill)}`); this.pause = this.backfillFully().finally(() => { this.pause = null; }); await this.pause; } /** * With the writer parked on the returned promise, loop passive passes until * one reports the entire WAL backfilled (typically the second: the first * drains the pass that was already running against a stale snapshot). After * a bounded number of passes without a full backfill — e.g. a reader * pinning the WAL — throws {@link WalValveAbortError} when still past the * hard/file caps (#1539 fail-closed). Soft give-up under those caps is * reserved for foldNow on a modest backlog that could not complete. */ private async backfillFully(): Promise<void> { for (let i = 0; i < MAX_PAUSED_BACKFILL_PASSES; i++) { if (this.inflight) await this.inflight; // fold in the stale in-flight pass first const res = await this.db.checkpointWalPassive(); if (!res) { // Machinery unavailable: fail closed past the documented caps (#1539), // otherwise soft-return so a non-WAL / closing connection does not abort. const walBytes = this.db.getWalSizeBytes(); const growth = this.growthBytes(); if (walBytes > this.fileCapBytes || growth > this.hardBytes) { throw new WalValveAbortError( `WAL checkpoint machinery unavailable while over the documented cap ` + `(wal=${this.mb(walBytes)}, fileCap=${this.mb(this.fileCapBytes)}). ` + `Aborting to avoid unbounded disk growth.`, { walBytes, fileCapBytes: this.fileCapBytes, hardBytes: this.hardBytes } ); } return; } this.log(`backfill pass ${i + 1}: busy=${res.busy} log=${res.log} checkpointed=${res.checkpointed} wal=${this.mb(this.db.getWalSizeBytes())}`); if (res.busy === 0 && res.log === res.checkpointed) { // Backfill complete AND we are at a parked barrier (backfillFully only // runs under a writer pause): the no-reader window is guaranteed, so // chop the FILE too — a fully-backfilled WAL otherwise keeps growing // whenever commits land while pool readers hold marks (§7a.1: 22GB // on-disk at kernel scale despite backfills). A racing reader turns // this into a no-op (busy=1); the passive result above still stands. const trunc = await this.db.checkpointWalTruncate(); if (trunc) this.log(`truncate: busy=${trunc.busy} wal=${this.mb(this.db.getWalSizeBytes())}`); this.sizeAtLastFullBackfill = this.db.getWalSizeBytes(); this.consecutiveGiveUps = 0; return; } } this.consecutiveGiveUps++; const walBytes = this.db.getWalSizeBytes(); const growth = this.growthBytes(); const msg = `backfill gave up after ${MAX_PAUSED_BACKFILL_PASSES} passes ` + `(streak ${this.consecutiveGiveUps}) — a reader is pinning the WAL ` + `(wal=${this.mb(walBytes)} growth=${this.mb(growth)} ` + `hard=${this.mb(this.hardBytes)} fileCap=${this.mb(this.fileCapBytes)})`; this.log(msg); // Give-ups are rare and load-bearing for §7a.1-class diagnosis — surface // them on any timing-instrumented run, not just valve-debug ones. if (process.env.CODEGRAPH_SYNTH_TIMINGS && !process.env.CODEGRAPH_WAL_VALVE_DEBUG) { console.error(`[wal-valve] ${msg}`); } // Fail closed (#1539): never release the writer past the documented caps // when checkpoints cannot progress. The old futility latch disabled // parking for 60s and allowed unbounded growth (6 - src/db/wsl-shared-index.ts (4509 bytes)
/** * Windows + WSL sharing one index on a Windows drive (issue #995). * * A repo under `/mnt/<drive>/` can be used from both sides: Windows-native * CodeGraph and WSL CodeGraph then open the same `.codegraph/codegraph.db` * across the 9p/DrvFs bridge, where SQLite's file locking and its `-shm` * shared memory don't hold between the two OSes. WSL's connection fails with a * bare "disk I/O error" (SQLITE_IOERR). A private WSL index on the same drive * works fine, so a fresh WSL project there already gets its own * `.codegraph-wsl` (`codeGraphDirNameFor`). An index already in `.codegraph` — * built by Windows first, or by WSL before that default — is kept as it is, * and when it fails this way the SQLite adapter rewrites the error into the * `CODEGRAPH_DIR=.codegraph-wsl` instruction instead of letting it reach the * user as an unexplained I/O failure. */ import * as path from 'path'; import { DEFAULT_CODEGRAPH_DIR, WSL_CODEGRAPH_DIR } from '../directory'; import { isWindowsDriveMount, isWslWindowsDrive } from '../sync/watch-policy'; /** SQLite's primary result code for an I/O failure; extended codes keep it in the low byte. */ const SQLITE_IOERR = 10; /** * Is `err` a SQLite I/O error? node:sqlite reports the EXTENDED result code on * `errcode` (SQLITE_IOERR_LOCK = 3850, SQLITE_IOERR_SHMMAP = 5130, ...), whose * low byte is the primary code. An error without one (re-wrapped somewhere * along the way) is matched by SQLite's message for the whole family. */ export function isSqliteIoError(err: unknown): boolean { if (!err || typeof err !== 'object') return false; const { errcode, message } = err as { errcode?: unknown; message?: unknown }; if (typeof errcode === 'number') return (errcode & 0xff) === SQLITE_IOERR; return typeof message === 'string' && /disk I\/O error/i.test(message); } /** * A SQLite I/O error on an index that Windows CodeGraph most likely shares. * Keeps the original as `cause` and carries its SQLite codes, so a caller that * branches on them sees the same values as before the rewrite. */ export class WslSharedIndexError extends Error { readonly code: unknown; readonly errcode: unknown; readonly errstr: unknown; constructor(original: Error, dataDir: string) { super( `${original.message} on the CodeGraph index at ${dataDir}\n` + "This project is on a Windows drive, where SQLite's file locking doesn't work across " + 'the Windows/WSL boundary, so this usually means CodeGraph on Windows is using the same ' + "index. Windows and WSL can't share one index on a Windows drive; give WSL its own:\n" + ` 1. Set CODEGRAPH_DIR=${WSL_CODEGRAPH_DIR} in the WSL environment (your shell profile, ` + 'or the env of the MCP server your agent starts).\n' + ` 2. Run "codegraph init" in WSL to build that index. Windows keeps using ${DEFAULT_CODEGRAPH_DIR}.`, { cause: original } ); this.name = 'WslSharedIndexError'; const codes = original as Error & { code?: unknown; errcode?: unknown; errstr?: unknown }; this.code = codes.code; this.errcode = codes.errcode; this.errstr = codes.errstr; } } /** * Inputs that can be overridden in tests so the decision is deterministic * without touching `/proc/version` (see `WatchProbe`). */ export interface WslSharedIndexProbe { /** Defaults to `detectWsl()` (through `isWslWindowsDrive`). */ isWsl?: boolean; } /** * The error to throw in place of `err`: a {@link WslSharedIndexError} when a * SQLite I/O error hit the default-named index of a project on a Windows drive * under WSL, else `null` (throw `err` unchanged). * * Only the default `.codegraph` counts: it is the directory Windows CodeGraph * opens too. A WSL that already has its own directory (`.codegraph-wsl`, or * any `CODEGRAPH_DIR`) hit some other I/O failure, and pointing it at * `CODEGRAPH_DIR` would mislead. */ export function toWslSharedIndexError( err: unknown, dbPath: string, probe: WslSharedIndexProbe = {} ): WslSharedIndexError | null { if (err instanceof WslSharedIndexError) return err; if (!(err instanceof Error) || !isSqliteIoError(err)) return null; const dataDir = path.dirname(dbPath); if (path.basename(dataDir) !== DEFAULT_CODEGRAPH_DIR) return null; const onWslWindowsDrive = probe.isWsl === undefined ? isWslWindowsDrive(dbPath) : probe.isWsl && isWindowsDriveMount(dbPath); return onWslWindowsDrive ? new WslSharedIndexError(err, dataDir) : null; } - src/directory.ts (59500 bytes)
/** * Directory Management * * Manages the .codegraph/ directory structure for CodeGraph data. */ import { execFileSync } from 'child_process'; import * as fs from 'fs'; import * as os from 'os'; import * as path from 'path'; import { isWslWindowsDrive } from './sync/watch-policy'; import { loadNodeSqlite } from './db/node-sqlite'; import { sensitiveDirectoryRefusal } from './utils'; /** The default per-project data directory name. */ export const DEFAULT_CODEGRAPH_DIR = '.codegraph'; /** * The data directory name WSL gives a fresh project on a Windows drive, so it * never shares one index with CodeGraph on Windows (issue #995). Indexing and * watching skip every `.codegraph-*` sibling on both sides (#636). */ export const WSL_CODEGRAPH_DIR = '.codegraph-wsl'; let warnedBadDirName = false; /** * Resolve the per-project data directory name, honoring the `CODEGRAPH_DIR` * environment override (default `.codegraph`). The override is a single path * segment that lives in the project root. * * Why this exists: two environments that share one working tree must NOT share * one `.codegraph/` — most concretely Windows-native and WSL (issue #636). The * daemon lockfile (`.codegraph/daemon.pid`) records a platform-specific pid and * socket path (a Windows named pipe vs a WSL Unix socket), and SQLite file * locking across the WSL2 ↔ Windows filesystem boundary is unreliable, so two * daemons sharing one index risks corruption. Setting `CODEGRAPH_DIR=.codegraph-win` * on one side gives each environment its own index in the same tree. * * Read live (not captured at load) so it is both process-accurate and testable. * An override that isn't a plain directory name — empty, containing a path * separator, `.`, `..`/traversal, or absolute — is ignored (we keep the * default) rather than risk writing the index outside the project or into the * project root itself; we warn once to stderr so the misconfiguration is seen. */ export function codeGraphDirName(): string { const raw = process.env.CODEGRAPH_DIR?.trim(); if (!raw) return DEFAULT_CODEGRAPH_DIR; const invalid = raw === '.' || raw.includes('..') || raw.includes('/') || raw.includes('\\') || path.isAbsolute(raw); if (invalid) { if (!warnedBadDirName) { warnedBadDirName = true; // stderr only — stdout is the MCP protocol channel. console.warn( `[codegraph] Ignoring invalid CODEGRAPH_DIR="${raw}" — it must be a plain ` + `directory name (no path separators, no "..", not absolute). Using "${DEFAULT_CODEGRAPH_DIR}".` ); } return DEFAULT_CODEGRAPH_DIR; } return raw; } /** * CodeGraph directory name — a load-time snapshot of {@link codeGraphDirName}. * A running process's environment is fixed, so this equals the live value; * it's kept as a stable string export for backward compatibility. Internal code * resolves the name through {@link codeGraphDirName} / {@link getCodeGraphDir} * so the `CODEGRAPH_DIR` override always applies. */ export const CODEGRAPH_DIR = codeGraphDirName(); /** * Is `name` (a single path segment) a CodeGraph data directory? Matches the * default `.codegraph`, the active `CODEGRAPH_DIR` override, and any * `.codegraph-*` sibling. File-watching and the indexer skip ALL of these, so * when two environments share one working tree (Windows + WSL, issue #636) * neither indexes or watches the other's index directory. */ export function isCodeGraphDataDir(name: string): boolean { return ( name === DEFAULT_CODEGRAPH_DIR || name === codeGraphDirName() || name.startsWith(DEFAULT_CODEGRAPH_DIR + '-') ); } /** * The data directory name for one project: {@link codeGraphDirName}, except * for a project on a Windows drive under WSL (`/mnt/c/...`) with no * `CODEGRAPH_DIR` set. Windows-native CodeGraph opens `.codegraph` in that * same tree, and SQLite's locking doesn't hold across the 9p/DrvFs bridge, so * the two sharing one index fails with "disk I/O error" (issue #995). There: * * 1. `.codegraph-wsl/` exists → it. Once WSL has its own index it keeps it, * even after Windows builds a `.codegraph` beside it. * 2. `.codegraph/codegraph.db` exists → `.codegraph`. An index built before * this default is kept rather than silently rebuilt somewhere else. * 3. neither → `.codegraph-wsl`, so a fresh WSL index never shares. * * Every other host keeps the plain name without a stat: the WSL check is * cached per process. */ export function codeGraphDirNameFor(projectRoot: string): string { if (process.env.CODEGRAPH_DIR?.trim() || !isWslWindowsDrive(projectRoot)) return codeGraphDirName(); try { if (fs.statSync(path.join(projectRoot, WSL_CODEGRAPH_DIR)).isDirectory()) return WSL_CODEGRAPH_DIR; } catch { // absent — fall through } if (fs.existsSync(path.join(projectRoot, DEFAULT_CODEGRAPH_DIR, 'codegraph.db'))) return DEFAULT_CODEGRAPH_DIR; return WSL_CODEGRAPH_DIR; } /** * Get the .codegraph directory path for a project */ export function getCodeGraphDir(projectRoot: string): string { return path.join(projectRoot, codeGraphDirNameFor(projectRoot)); } /** * Check if a project has been initialized with CodeGraph. * * Requires `.codegraph/codegraph.db` to exist AND to carry the codegraph * schema. A file that merely exists — empty, or a SQLite database with no * tables, as an interrupted `init` or a stray `touch` leaves behind — used to * count as initialized, so one such file in an ANCESTOR directory (worst * case: `$HOME`) captured the upward resolution of every project beneath it * and made their real indexes unreachable (#1895). * * The probe is cheap and gated so hot callers (the prompt hook, MCP root * resolution on every call) pay one `stat`: file size, then a read-only * SQLite open with a single `sqlite_master` lookup — memoized per path + * mtime + size so an unchanged db is never reopened. A database that cannot * be inspected counts as initialized (see probeSchema) — only a proven-absent * schema, or a file SQLite refuses as not a database, says no. */ export function isInitialized(projectRoot: string): boolean { const codegraphDir = getCodeGraphDir(projectRoot); if (!fs.existsSync(codegraphDir) || !fs.statSync(codegraphDir).isDirectory()) { return false; } // Must have codegraph.db, not just .codegraph folder const dbPath = path.join(codegraphDir, 'codegraph.db'); let st: fs.Stats; try { st = fs.statSync(dbPath); } catch { return false; } return hasCodeGraphSchema(dbPath, st); } /** * `codegraph.db` exists at `projectRoot` but does not carry the schema, and * `init` can add it in place: an empty file, or a SQLite database without the * codegraph tables (#1895). A file that is not SQLite at all is NOT this case — * see {@link hasForeignDbFile}. */ export function hasSchemalessDb(projectRoot: string): boolean { const dbPath = path.join(getCodeGraphDir(projectRoot), 'codegraph.db'); let st: fs.Stats; try { st = fs.statSync(dbPath); } catch { return false; } if (!st.isFile() || isInitialized(projectRoot)) return false; return st.size === 0 || probeSchema(dbPath) === 'no-schema'; } /** * `codegraph.db` exists at `projectRoot` and is not a SQLite database (no * header magic): SQLite refuses to open it, so `init` cannot rebuild it in * place. The caller must say so rather than promise a repair; nothing here * deletes the file. */ export function hasForeignDbFile(projectRoot: string): boolean { const dbPath = path.join(getCodeGraphDir(projectRoot), 'codegraph.db'); let st: fs.Stats; try { st = fs.statSync(dbPath); } catch { return false; } return st.isFile() && st.size > 0 && probeSchema(dbPath) === 'not-sqlite'; } /** A SQLite file header is 100 bytes; anything shorter cannot hold a schema. */ const SQLITE_HEADER_SIZE = 100; /** SQLITE_NOTADB: SQLite read the file and it is not a database. */ const SQLITE_NOTADB = 26; const schemaProbeCache = new Map<string, { mtimeMs: number; size: number; ok: boolean }>(); function hasCodeGraphSchema(dbPath: string, st: fs.Stats): boolean { if (!st.isFile() || st.size < SQLITE_HEADER_SIZE) return false; const cached = schemaProbeCache.get(dbPath); if (cached && cached.mtimeMs === st.mtimeMs && cached.size === st.size) return cached.ok; const probe = probeSchema(dbPath); const ok = probe === 'schema' || probe === 'unknown'; schemaProbeCache.set(dbPath, { mtimeMs: st.mtimeMs, size: st.size, ok }); return ok; } /** * What `codegraph.db` holds, asked of SQLite itself through a read-only * connection: the codegraph schema, a database without it, not a database at * all (SQLITE_NOTADB), or `unknown` — locked, busy, a WAL db in a directory we * cannot create `-shm` in (read-only checkout, mount, another user's tree), * disk I/O. Callers treat `unknown` as initialized: the pre-existing behaviour * for a database we cannot inspect. * * The file is never read through a descriptor of our own, not even for its * 16-byte header. Closing ANY descriptor on a database file drops every POSIX * lock this process holds on it, including those of a connection it already * has open (sqlite.org/howtocorrupt.html §2.2.1) — and the MCP server resolves * projects through isInitialized on every call while it holds the index as * its writer. SQLite's own connections share one lock table per file, so a * second connection opened and closed here leaves the first one's locks alone. */ function probeSchema(dbPath: string): 'schema' | 'no-schema' | 'not-sqlite' | 'unknown' { let db: any = null; try { // A runtime without node:sqlite can't inspect the file: `unknown`, so the // command goes on to open it and reports why that is impossible. const { DatabaseSync } = loadNodeSqlite(); db = new DatabaseSync(dbPath, { readOnly: true }); const row = db.prepare("SELECT 1 AS ok FROM sqlite_master WHERE type = 'table' AND name = 'nodes'").get(); return row !== undefined ? 'schema' : 'no-schema'; } catch (error) { return (error as { errcode?: number })?.errcode === SQLITE_NOTADB ? 'not-sqlite' : 'unknown'; } finally { // Never hold the handle: Windows file locking would block the owner. try { db?.close(); } catch { /* already closed */ } } } /** * Find the nearest parent directory containing .codegraph/ * * Walks up from the given path to find a CodeGraph-initialized project, * similar to how git finds .git/ directories. * * @param startPath - Directory to start searching from * @returns The project root containing .codegraph/, or null if not found */ /** * Reason a directory is unsafe to use as an index ROOT, or null when it's fine. * * Indexing your home directory or a filesystem root drags in caches, `Library`, * every other project, etc. — a multi-GB index, constant file-watcher churn, and * (pre-1.0 on macOS) a file-descriptor blowup that exhausted `kern.maxfiles` and * took unrelated apps / the whole machine down (#845). The classic trigger: * running the installer or `codegraph init` from `$HOME`, which auto-indexes the * current directory. These are never intended project roots, so the installer * and `init`/`index` refuse them (overridable with `--force`). * * Pure-ish (reads only `os.homedir()` + realpath) so it's easy to unit-test. * The returned string is a human phrase that slots into "… looks like {reason}". */ export function unsafeIndexRootReason(projectRoot: string): string | null { const resolve = (p: string): string => { try { return fs.realpathSync(path.resolve(p)); } catch { return path.resolve(p); } }; const resolved = resolve(projectRoot); // Filesystem root: `/` on POSIX, a drive root like `C:\` on Windows. if (path.parse(resolved).root === resolved) { return 'the filesystem root'; } const home = resolve(os.homedir()); // Case-insensitive on macOS/Windows (case-preserving but case-insensitive FS). const norm = (p: string): string => process.platform === 'darwin' || process.platform === 'win32' ? p.toLowerCase() : p; const r = norm(resolved); const h = norm(home); if (r === h) { return 'your home directory'; } // An ancestor of home (e.g. `/Users`, `/home`) — even broader than home. if (h.startsWith(r + path.sep)) { return 'a parent of your home directory'; } return null; } /** * `dev:ino` for a path, or null if it can't be stat'd or the platform doesn't * report a usable inode. Read as bigints: WSL DrvFs (`/mnt/c`) reports inodes * above 2^53, where a plain number rounds nearby inodes onto one value. Windows * st_ino is unreliable across handle reopens, so we deliberately return null * there — the deleted-but-open-inode hazard this guards (#925) is a POSIX * file-semantics issue that doesn't arise on Windows (an open file can't be * unlinked). */ export function statInode(p: string): string | null { if (process.platform === 'win32') return null; try { const s = fs.statSync(p, { bigint: true }); return `${s.dev}:${s.ino}`; } catch { return null; } } /** * Canonicalize a project root for IDENTITY: hashing it into a rendezvous key * (the daemon's named pipe / tmpdir socket, a registry record). Not for display * — the caller-visible root keeps its own spelling. * * The contract is that two spellings of ONE directory yield ONE string, because * a client that derives a different key silently fails to meet the daemon that * is already running: it probes a socket nobody bound, spawns a redundant * daemon, and that daemon dies on the lock the first one holds, so the session * degrades to a single-process engine. `path.resolve` alone does NOT satisfy * that contract — NTFS is case-insensitive, so `d:\work\codegraph` and * `D:\work\codegraph` name one directory but two strings, and non-native * `fs.realpathSync` keeps whichever casing the caller passed (only `.native` * asks the filesystem for the on-disk name — the same reason * {@link isSameIndexRoot} uses it). Both spellings really do occur: a * cwd-derived root is the on-disk case, while a client-supplied * `rootUri`/`workspaceFolders` path arrives as `file:///d%3A/…`. * * The Windows lowercase is belt-and-braces on top of the native realpath — NTFS * can't hold two directories differing only by case, and it also absorbs a * `\\?\`-prefixed native result diverging from a plain one. */ export function canonicalProjectRoot(projectRoot: string): string { const resolved = path.resolve(projectRoot); let canonical = resolved; try { canonical = fs.realpathSync.native(resolved); } catch { // ENOENT/EACCES/ELOOP — the root is normally there (`.codegraph/` lives in // it), so this is a fallback rather than a path we expect to take. } return process.platform === 'win32' ? canonical.toLowerCase() : canonical; } /** * Whether two resolved index roots are one index spelled two ways — a symlinked * checkout, or - src/engine.ts (1236 bytes)
/** * The half of CodeGraph that builds and updates an index: extraction, * resolution and the file watcher. `src/index.ts` reaches all of it through * `engine()` in `./lazy-engine`, never with an import of its own, so a command * that only reads an index doesn't load it. See that file for why. */ export { ExtractionOrchestrator, extractFromSource } from './extraction'; export { changedModuleConfigs, changedRoutes, createResolver, moduleConfigLanguages, parseModuleConfigSnapshot } from './resolution'; export { hasSynthesisPattern, isJsxChildCandidate } from './resolution/callback-synthesizer'; export { aliasMayName } from './resolution/const-alias-synthesizer'; export { ESM_IMPORT_LANGUAGES } from './resolution/import-resolver'; export { createImportReader, reopenNameMatchedImports, reopenReferencesThroughAddedFiles, reopenReferencesThroughBarrels, reopenReferencesThroughChangedDefaults, reopenReferencesThroughDeclaredPackages, reopenReferencesThroughObjectMembers, reopenReferencesThroughRetargetedModules, } from './resolution/module-importers'; export { manifestStamp, parseManifestStamp, trustedManifestList } from './resolution/workspace-packages'; export { FileWatcher } from './sync/watcher'; - src/errors.ts (7152 bytes)
/** * CodeGraph Error Classes * * Custom error types for better error handling and debugging. * * @module errors * * @example * ```typescript * import { FileError, ParseError, setLogger, silentLogger } from 'codegraph'; * * // Catch specific error types * try { * await cg.indexAll(); * } catch (error) { * if (error instanceof FileError) { * console.log(`File error at ${error.filePath}: ${error.message}`); * } else if (error instanceof ParseError) { * console.log(`Parse error at ${error.filePath}:${error.line}`); * } * } * * // Disable logging for tests * setLogger(silentLogger); * ``` */ /** * Base error class for all CodeGraph errors. * * All CodeGraph-specific errors extend this class, allowing you to catch * all CodeGraph errors with a single catch block. * * @example * ```typescript * try { * await cg.indexAll(); * } catch (error) { * if (error instanceof CodeGraphError) { * console.log(`CodeGraph error [${error.code}]: ${error.message}`); * } * } * ``` */ export class CodeGraphError extends Error { /** Error code for categorization (e.g., 'FILE_ERROR', 'PARSE_ERROR') */ readonly code: string; /** Additional context about the error */ readonly context?: Record<string, unknown>; constructor(message: string, code: string, context?: Record<string, unknown>) { super(message); this.name = 'CodeGraphError'; this.code = code; this.context = context; // Maintain proper stack trace for V8 if (Error.captureStackTrace) { Error.captureStackTrace(this, this.constructor); } } } /** * Error reading or accessing files */ export class FileError extends CodeGraphError { readonly filePath: string; constructor(message: string, filePath: string, cause?: Error) { super(message, 'FILE_ERROR', { filePath, cause: cause?.message }); this.name = 'FileError'; this.filePath = filePath; if (cause) { this.cause = cause; } } } /** * Error parsing source code */ export class ParseError extends CodeGraphError { readonly filePath: string; readonly line?: number; readonly column?: number; constructor( message: string, filePath: string, options?: { line?: number; column?: number; cause?: Error } ) { super(message, 'PARSE_ERROR', { filePath, line: options?.line, column: options?.column, cause: options?.cause?.message, }); this.name = 'ParseError'; this.filePath = filePath; this.line = options?.line; this.column = options?.column; if (options?.cause) { this.cause = options.cause; } } } /** * Error with database operations */ export class DatabaseError extends CodeGraphError { readonly operation: string; constructor(message: string, operation: string, cause?: Error) { super(message, 'DATABASE_ERROR', { operation, cause: cause?.message }); this.name = 'DatabaseError'; this.operation = operation; if (cause) { this.cause = cause; } } } /** * Error with search operations */ export class SearchError extends CodeGraphError { readonly query: string; constructor(message: string, query: string, cause?: Error) { super(message, 'SEARCH_ERROR', { query, cause: cause?.message }); this.name = 'SearchError'; this.query = query; if (cause) { this.cause = cause; } } } /** * Error with vector/embedding operations */ export class VectorError extends CodeGraphError { constructor(message: string, operation: string, cause?: Error) { super(message, 'VECTOR_ERROR', { operation, cause: cause?.message }); this.name = 'VectorError'; if (cause) { this.cause = cause; } } } /** * Error with configuration */ export class ConfigError extends CodeGraphError { constructor(message: string, details?: Record<string, unknown>) { super(message, 'CONFIG_ERROR', details); this.name = 'ConfigError'; } } /** * A refused path — the caller asked for something outside the project root, or * for a sensitive system directory. Deliberately a plain `Error` and NOT a * {@link CodeGraphError}: it is a security marker every read sink tests with * `instanceof`, not a categorized operational failure, and the MCP layer treats * it as one of the only two "stop trying" conditions (see `mcp/tools.ts`). * * It lives here — in the dependency-free error module — rather than next to its * first caller so that a consumer can enforce the refusal WITHOUT importing the * MCP tool graph. `mcp/tools.ts` re-exports it, so the class identity stays * single and every existing `instanceof` check keeps working. */ export class PathRefusalError extends Error {} /** * Simple logger for CodeGraph operations * * By default, logs to console.warn for warnings and console.error for errors. * Can be configured to use custom logging. */ export interface Logger { debug(message: string, context?: Record<string, unknown>): void; warn(message: string, context?: Record<string, unknown>): void; error(message: string, context?: Record<string, unknown>): void; } /** * Default console-based logger */ export const defaultLogger: Logger = { debug(message: string, context?: Record<string, unknown>): void { if (process.env.CODEGRAPH_DEBUG) { console.debug(`[CodeGraph] ${message}`, context ?? ''); } }, warn(message: string, context?: Record<string, unknown>): void { console.warn(`[CodeGraph] ${message}`, context ?? ''); }, error(message: string, context?: Record<string, unknown>): void { console.error(`[CodeGraph] ${message}`, context ?? ''); }, }; /** * Silent logger (no output) - useful for tests */ export const silentLogger: Logger = { debug(): void {}, warn(): void {}, error(): void {}, }; /** * Current logger instance (can be replaced) */ let currentLogger: Logger = defaultLogger; /** * Set the global logger */ export function setLogger(logger: Logger): void { currentLogger = logger; } /** * Thrown by a `syncFn` to signal that the underlying sync couldn't acquire * the cross-process write lock (#449). The watcher treats this as "no * progress" — preserves `pendingFiles`, skips `onSyncComplete`, and the * `finally` block reschedules. Quiet (debug-only) because a long-running * external indexer can hit this every debounce cycle. */ export class LockUnavailableError extends Error { constructor(message = 'CodeGraph file lock unavailable; another process is writing') { super(message); this.name = 'LockUnavailableError'; } } /** * Get the current logger */ export function getLogger(): Logger { return currentLogger; } /** * Log a debug message */ export function logDebug(message: string, context?: Record<string, unknown>): void { currentLogger.debug(message, context); } /** * Log a warning message */ export function logWarn(message: string, context?: Record<string, unknown>): void { currentLogger.warn(message, context); } /** * Log an error message */ export function logError(message: string, context?: Record<string, unknown>): void { currentLogger.error(message, context); } - src/extraction/astro-extractor.ts (12416 bytes)
import { Node, Edge, ExtractionResult, ExtractionError, UnresolvedReference } from '../types'; import { generateNodeId } from './tree-sitter-helpers'; import { TreeSitterExtractor } from './tree-sitter'; import { isLanguageSupported } from './grammars'; import { foldScriptResult, sfcFileNode, SFC_SCRIPT_ELEMENT, sfcBlockLineRanges } from './sfc-script'; /** * Astro built-in components — compiler-provided (`<Fragment>`) or shipped by * `astro:components` (`<Code>`, `<Debug>`), not user code. */ const ASTRO_BUILTIN_COMPONENTS = new Set(['Fragment', 'Code', 'Debug']); /** * AstroExtractor - Extracts code relationships from Astro component files * * Astro files are multi-language: a TypeScript frontmatter block fenced by * `---` lines, a JSX-like HTML template, and optional <script>/<style> blocks. * Rather than parsing a full Astro grammar, we extract the frontmatter and * <script> contents and delegate them to the TypeScript TreeSitterExtractor * (Astro processes both as TypeScript by default — no `lang` attr needed). * * Also extracts function calls from template expressions (`{fn(...)}`) and * component usages (`<PascalCase>`) so cross-file edges are captured even * when the only reference lives in markup. * * Every .astro file produces a component node (Astro components are always * importable). */ export class AstroExtractor { private filePath: string; private source: string; private nodes: Node[] = []; private edges: Edge[] = []; private unresolvedReferences: UnresolvedReference[] = []; private errors: ExtractionError[] = []; constructor(filePath: string, source: string) { this.filePath = filePath; this.source = source; } /** * Extract from Astro source */ extract(): ExtractionResult { const startTime = Date.now(); try { // The file, holding the component the .astro file is this.nodes.push(sfcFileNode(this.filePath, this.source, 'astro')); const componentNode = this.createComponentNode(); this.edges.push({ source: `file:${this.filePath}`, target: componentNode.id, kind: 'contains' }); // Extract and process the frontmatter block (--- fenced, TypeScript) const frontmatter = this.extractFrontmatter(); if (frontmatter) { this.processScriptContent(frontmatter, componentNode.id, 'frontmatter'); } // Extract and process <script> blocks (client-side, TypeScript-capable) for (const block of this.extractScriptBlocks()) { this.processScriptContent(block, componentNode.id, 'script'); } // Ranges the template scans must skip: frontmatter + <script>/<style> const coveredRanges = this.getCoveredRanges(frontmatter); // Extract function calls from template expressions ({fn(...)}) this.extractTemplateCalls(componentNode.id, coveredRanges); // Extract component usages from template (<ComponentName>) this.extractTemplateComponents(componentNode.id, coveredRanges); } catch (error) { this.errors.push({ message: `Astro extraction error: ${error instanceof Error ? error.message : String(error)}`, severity: 'error', code: 'parse_error', }); } return { nodes: this.nodes, edges: this.edges, unresolvedReferences: this.unresolvedReferences, errors: this.errors, durationMs: Date.now() - startTime, }; } /** * Create a component node for the .astro file */ private createComponentNode(): Node { const lines = this.source.split('\n'); const fileName = this.filePath.split(/[/\\]/).pop() || this.filePath; const componentName = fileName.replace(/\.astro$/, ''); const id = generateNodeId(this.filePath, 'component', componentName, 1); const node: Node = { id, kind: 'component', name: componentName, qualifiedName: `${this.filePath}::${componentName}`, filePath: this.filePath, language: 'astro', startLine: 1, endLine: lines.length, startColumn: 0, endColumn: lines[lines.length - 1]?.length || 0, isExported: true, // Astro components are always importable updatedAt: Date.now(), }; this.nodes.push(node); return node; } /** * Extract the frontmatter block: the content between the opening `---` * fence (first non-blank line of the file) and the closing `---` fence. * An unclosed fence is treated as "no frontmatter" rather than swallowing * the whole template as TypeScript. * * Returns the content plus its 0-indexed start line, or null. */ private extractFrontmatter(): { content: string; startLine: number; endLine: number } | null { const lines = this.source.split('\n'); // Opening fence must be the first non-blank line let openIdx = -1; for (let i = 0; i < lines.length; i++) { const trimmed = lines[i]!.trim(); if (trimmed === '') continue; if (trimmed === '---') openIdx = i; break; } if (openIdx === -1) return null; // Closing fence let closeIdx = -1; for (let i = openIdx + 1; i < lines.length; i++) { if (lines[i]!.trim() === '---') { closeIdx = i; break; } } if (closeIdx === -1) return null; return { content: lines.slice(openIdx + 1, closeIdx).join('\n'), startLine: openIdx + 1, // 0-indexed line where content starts endLine: closeIdx, // 0-indexed line of the closing fence }; } /** * Extract <script> blocks from the template portion */ private extractScriptBlocks(): Array<{ content: string; startLine: number }> { const blocks: Array<{ content: string; startLine: number }> = []; // A self-closing `<script ... />` (e.g. `is:inline set:html={...}`) has no // body; match it on its own so it cannot run on to the next `</script>`. const scriptRegex = new RegExp(SFC_SCRIPT_ELEMENT); let match; while ((match = scriptRegex.exec(this.source)) !== null) { if (match.groups?.content === undefined) continue; const content = match.groups.content; // Calculate the 0-indexed line where the content begins. The content // starts right after the opening tag's `>` — its leading `\n` is part // of the content, so relative line 1 sits ON the tag's closing line // (do not add 1 here; that double-counts the embedded newline). const beforeScript = this.source.substring(0, match.index); const scriptTagLine = (beforeScript.match(/\n/g) || []).length; const openingTag = match[0].substring(0, match[0].indexOf('>') + 1); const openingTagLines = (openingTag.match(/\n/g) || []).length; const contentStartLine = scriptTagLine + openingTagLines; // 0-indexed blocks.push({ content, startLine: contentStartLine }); } return blocks; } /** * Process frontmatter / script content by delegating to TreeSitterExtractor. * Astro treats both as TypeScript by default. */ private processScriptContent( block: { content: string; startLine: number }, componentNodeId: string, label: 'frontmatter' | 'script' ): void { if (!isLanguageSupported('typescript')) { this.errors.push({ message: `Parser for typescript not available, cannot parse Astro ${label} block`, severity: 'warning', }); return; } // Delegate to TreeSitterExtractor const extractor = new TreeSitterExtractor(this.filePath, block.content, 'typescript'); const result = extractor.extract(); // Frontmatter runs on every render of the component, and a <script> is // the page's own client code: both are the component's doing. foldScriptResult( result, { filePath: this.filePath, componentNodeId, lineOffset: block.startLine, language: 'astro', perInstance: true }, { nodes: this.nodes, edges: this.edges, unresolvedReferences: this.unresolvedReferences, errors: this.errors } ); } /** * Line ranges (0-indexed, inclusive) the template scans must skip: * the frontmatter block and <script>/<style> blocks. */ private getCoveredRanges( frontmatter: { startLine: number; endLine: number } | null ): Array<[number, number]> { const coveredRanges: Array<[number, number]> = []; if (frontmatter) { // Cover from the opening fence line through the closing fence line coveredRanges.push([frontmatter.startLine - 1, frontmatter.endLine]); } // A self-closing tag covers nothing: its attributes // (`set:html={serializeJsonLd(schema)}`) are template expressions. coveredRanges.push(...sfcBlockLineRanges(this.source)); return coveredRanges; } /** * Extract function calls from Astro template expressions. * * Astro templates embed JSX-like expressions (`{formatDate(post.date)}`, * `class:list={cn(...)}`), so calls frequently live in markup rather than * the frontmatter. We scan template lines for `{expression}` groups and * extract call patterns from them. A `{` group left open at end-of-line * (the pervasive `{posts.map((post) => (` pattern) contributes the calls * on its opening line. */ private extractTemplateCalls( componentNodeId: string, coveredRanges: Array<[number, number]> ): void { const lines = this.source.split('\n'); // Complete groups: {...} — excluding JSX comments ({/* ... */}) const exprRegex = /\{([^}/][^}]*)\}/g; // A group opened but not closed on this line const openExprRegex = /\{([^}/][^}]*)$/; for (let lineIdx = 0; lineIdx < lines.length; lineIdx++) { if (coveredRanges.some(([start, end]) => lineIdx >= start && lineIdx <= end)) continue; const line = lines[lineIdx]!; const exprs: Array<{ text: string; offset: number }> = []; let exprMatch; while ((exprMatch = exprRegex.exec(line)) !== null) { exprs.push({ text: exprMatch[1]!, offset: exprMatch.index }); } const openMatch = openExprRegex.exec(line.replace(exprRegex, '')); if (openMatch) { exprs.push({ text: openMatch[1]!, offset: line.lastIndexOf('{') }); } for (const expr of exprs) { // Extract function calls: identifiers followed by ( // Matches: cn(...), formatDate(...), obj.method(...) const callRegex = /\b([a-zA-Z_$][\w$.]*)\s*\(/g; let callMatch; while ((callMatch = callRegex.exec(expr.text)) !== null) { const calleeName = callMatch[1]!; // Skip control-flow keywords valid inside expressions if (calleeName === 'if' || calleeName === 'await' || calleeName === 'function') continue; this.unresolvedReferences.push({ fromNodeId: componentNodeId, referenceName: calleeName, referenceKind: 'calls', line: lineIdx + 1, // 1-indexed column: expr.offset + callMatch.index, filePath: this.filePath, language: 'astro', }); } } } } /** * Extract component usages from the Astro template. * * PascalCase tags like <Layout>, <PostCard /> represent component * instantiations — analogous to function calls in imperative code. * Lowercase tags are native HTML (Astro does not register kebab-case * components the way Vue does, so those are real custom elements and * are skipped). */ private extractTemplateComponents( componentNodeId: string, coveredRanges: Array<[number, number]> ): void { const lines = this.source.split('\n'); // Opening/self-closing tags (closing tags </Foo> start with </ so won't match) const componentTagRegex = /<([A-Z][a-zA-Z0-9_$]*)\b/g; for (let lineIdx = 0; lineIdx < lines.length; lineIdx++) { if (coveredRanges.some(([start, end]) => lineIdx >= start && lineIdx <= end)) continue; const line = lines[lineIdx]!; let match; while ((match = componentTagRegex.exec(line)) !== null) { const componentName = match[1]!; if (ASTRO_BUILTIN_COMPONENTS.has(componentName)) continue; this.unresolvedReferences.push({ fromNodeId: componentNodeId, referenceName: componentName, referenceKind: 'references', line: lineIdx + 1, // 1-indexed column: match.index + 1, filePath: this.filePath, language: 'astro', }); } } } } - src/extraction/cfml-extractor.ts (36938 bytes)
import type { Node as SyntaxNode } from 'web-tree-sitter'; import { Node, Edge, ExtractionResult, ExtractionError, UnresolvedReference, Language } from '../types'; import { generateNodeId, NodeIdAllocator } from './tree-sitter-helpers'; import { TreeSitterExtractor } from './tree-sitter'; import { getParser } from './grammars'; import { indexParseBudgetMs, parseTimeoutMessage, parseWithinBudget } from './parse-budget'; /** Tags whose own children include an expression: `<cfset …>`, `<cfif …>`, `<cfelseif …>`, `<cfreturn …>`, and `#…#`. */ const TAG_EXPRESSION_PARENTS: ReadonlySet<string> = new Set([ 'cf_set_tag', 'cf_if_tag', 'cf_elseif_tag', 'cf_return_tag', 'hash_expression', ]); /** * The expression node types (shared with the cfscript grammar) that can * contain a call. A positive list, so a tag's markup — body content, its * `var` keyword, an ERROR — is never mistaken for an expression; a bare * identifier or literal can't call anything and is skipped. */ const TAG_EXPRESSION_TYPES: ReadonlySet<string> = new Set([ 'call_expression', 'member_expression', 'subscript_expression', 'new_expression', 'assignment_expression', 'augmented_assignment_expression', 'binary_expression', 'unary_expression', 'update_expression', 'ternary_expression', 'elvis_expression', 'parenthesized_expression', 'sequence_expression', 'string', 'array', 'object', 'ordered_struct', 'function_expression', 'arrow_function', ]); /** An expression in tag markup and the scope (function, component or file node) that owns its calls. */ interface TagExpression { startIndex: number; endIndex: number; /** Where it starts in the file — the same line in the source extractTagExpressions synthesizes. */ row: number; column: number; /** Its start column in the synthesized source. */ textColumn: number; scopeId: string; } /** * CfmlExtractor - Extracts code relationships from CFML source (.cfc/.cfm). * * tree-sitter-cfml splits CFML into two related grammars: `cfml` (tag-based — * `<cfcomponent>`/`<cffunction>`/HTML) and `cfscript` (modern bare-script * `component { ... }` syntax). The `cfml` grammar's own injections.scm treats * bare-script content as an opaque blob meant to be re-parsed by `cfscript` — * that re-parsing only happens at the editor/highlighting layer, not in the * raw AST, so this extractor replicates it: a file whose first real token * isn't `<` is delegated wholesale to the cfscript grammar (the dominant * modern style); otherwise the file is walked tag-by-tag with the cfml * grammar, delegating any `<cfscript>` tag bodies the same way — and the * expressions written in tags themselves (`<cfset>`, `<cfif>`, `#hash#`, …) * through the same cfscript extraction (see extractTagExpressions). */ export class CfmlExtractor { private filePath: string; private source: string; private language: Language; private nodes: Node[] = []; private nodeIds = new NodeIdAllocator(); private edges: Edge[] = []; private unresolvedReferences: UnresolvedReference[] = []; private errors: ExtractionError[] = []; private tagExpressions: TagExpression[] = []; /** `language` is the file's detected language — `'cfml'` for `.cfc`/`.cfm`, `'cfscript'` for `.cfs`. Both dialect-switch internally; this only controls the language tag stamped onto emitted nodes/refs. */ constructor(filePath: string, source: string, language: Language = 'cfml') { this.filePath = filePath; this.source = source; this.language = language; } extract(): ExtractionResult { const startTime = Date.now(); try { if (isBareScriptCfml(this.source)) { this.extractBareScript(); } else { this.extractTagBased(); } } catch (error) { this.errors.push({ message: `CFML extraction error: ${error instanceof Error ? error.message : String(error)}`, severity: 'error', code: 'parse_error', }); } return { nodes: this.nodes, edges: this.edges, unresolvedReferences: this.unresolvedReferences, errors: this.errors, durationMs: Date.now() - startTime, }; } /** Modern bare-script `.cfc`/`.cfm`: delegate the whole file to the cfscript grammar. */ private extractBareScript(): void { const extractor = new TreeSitterExtractor(this.filePath, this.source, 'cfscript'); const result = extractor.extract(); // cfscript's `component`/`interface` node has no `name` field — a CFC's // component name is always implicit from its file name, never declared // in source — so the generic extractor names it '<anonymous>'. const componentName = this.componentNameFromPath(); for (const node of result.nodes) { node.language = this.language; if (node.name === '<anonymous>' && (node.kind === 'class' || node.kind === 'interface')) { node.name = componentName; node.qualifiedName = `${this.filePath}::${componentName}`; } else if (node.qualifiedName === '<anonymous>' || node.qualifiedName.startsWith('<anonymous>::')) { // Members were scoped under the anonymous component (`<anonymous>::save`) // — carry the rename into their scope chains so type-validated method // resolution (which wants `UserService::save`, see resolveMethodOnType) // can match them. Inner genuinely-anonymous segments are untouched. node.qualifiedName = componentName + node.qualifiedName.slice('<anonymous>'.length); } this.nodes.push(node); } this.edges.push(...result.edges); for (const ref of result.unresolvedReferences) { ref.language = this.language; this.unresolvedReferences.push(ref); } this.errors.push(...result.errors); } /** Legacy tag-based CFML: walk `<cfcomponent>`/`<cffunction>`, delegating `<cfscript>` bodies. */ private extractTagBased(): void { const parser = getParser('cfml'); if (!parser) { // Same code as TreeSitterExtractor's missing parser, so a grammar that // failed to load is never stored as this file's index data (#2335). this.errors.push({ message: 'cfml grammar not loaded', severity: 'error', code: 'parser_error', }); return; } // Within the file's budget, as every indexing parse (#3274). const budgetMs = indexParseBudgetMs(this.source.length); const tree = parseWithinBudget(parser, this.source, budgetMs); if (!tree) { this.errors.push({ message: parseTimeoutMessage(budgetMs), severity: 'error', code: 'parse_error', }); return; } try { const fileNode = this.createFileNode(); this.walkProgram(tree.rootNode, fileNode.id); } finally { // Trees hold wasm heap memory V8's GC never sees — free it per file. tree.delete(); } this.extractTagExpressions(); } /** Build the file's own `kind:'file'` node, spanning the whole source. Tag-based files need this explicitly — unlike `extractBareScript` (which delegates the whole file to `TreeSitterExtractor` and inherits its file node), `extractTagBased` walks the tree itself and has no other source of one. */ private createFileNode(): Node { const lines = this.source.split('\n'); const id = generateNodeId(this.filePath, 'file', this.filePath, 1); const fileNode: Node = { id, kind: 'file', name: this.filePath.split(/[/\\]/).pop() || this.filePath, qualifiedName: this.filePath, filePath: this.filePath, language: this.language, startLine: 1, endLine: lines.length, startColumn: 0, endColumn: lines[lines.length - 1]?.length || 0, updatedAt: Date.now(), }; this.nodes.push(fileNode); return fileNode; } /** * Walks `program`'s named children with a single forward cursor (not an * index loop) — `extractComponent` consumes a variable run of FOLLOWING * siblings as the component body (see its doc comment), so this must * resume from whatever it last consumed rather than revisiting those same * cffunction/cfscript siblings a second time as bogus top-level symbols. */ private walkProgram(root: SyntaxNode, fileNodeId: string): void { let child: SyntaxNode | null = root.namedChild(0); while (child) { if (child.type === 'cf_component_open_tag') { child = this.extractComponent(child, fileNodeId).nextSibling; continue; } // Template scope: a cffunction outside any cfcomponent wrapper (rare, // but legal in a .cfm template) is a top-level function contained by // the file, and template code's calls are the file's. this.visitTag(child, root.type, fileNodeId, undefined, true); child = child.nextSibling; } } /** * `<cfcomponent extends="Base" implements="IFoo,IBar">...</cfcomponent>`. * The grammar's implicit-end-tag scanner means component body content * (cffunction tags, cfscript tags, etc.) appears as the open tag's FOLLOWING * siblings in `program`, not nested children — walk forward to the matching * cf_component_close_tag. */ private extractComponent(openTag: SyntaxNode, containerId: string | undefined): SyntaxNode { const name = this.tagAttr(openTag, 'name') ?? this.componentNameFromPath(); const id = this.nodeIds.generate(this.filePath, 'class', name, openTag.startPosition.row + 1, openTag.startPosition.column); const classNode: Node = { id, kind: 'class', name, qualifiedName: `${this.filePath}::${name}`, filePath: this.filePath, language: this.language, startLine: openTag.startPosition.row + 1, endLine: openTag.startPosition.row + 1, // extended below once the close tag is found startColumn: openTag.startPosition.column, endColumn: openTag.endPosition.column, isExported: true, updatedAt: Date.now(), }; this.nodes.push(classNode); if (containerId) { this.edges.push({ source: containerId, target: classNode.id, kind: 'contains' }); } const extendsName = this.tagAttr(openTag, 'extends'); if (extendsName) { this.unresolvedReferences.push({ fromNodeId: classNode.id, referenceName: extendsName, referenceKind: 'extends', filePath: this.filePath, line: openTag.startPosition.row + 1, column: openTag.startPosition.column, language: this.language, }); } this.pushInheritanceRefs(classNode.id, this.tagAttr(openTag, 'implements'), 'implements', openTag); // Walk siblings between the open tag and its close tag. let sibling = openTag.nextSibling; let lastNode: SyntaxNode = openTag; while (sibling) { if (sibling.type === 'cf_component_close_tag') { lastNode = sibling; break; } // Component scope: cffunctions are its methods, pseudo-constructor // code's calls (a component-level `<cfset init()>`) are the component's. this.visitTag(sibling, openTag.parent?.type ?? '', classNode.id, classNode.name, true); lastNode = sibling; sibling = sibling.nextSibling; } classNode.endLine = lastNode.endPosition.row + 1; return lastNode; } /** * `<cfinterface extends="IBase,IOther">...</cfinterface>` (#2091). Unlike * `<cfcomponent>` (see the implicit-end-tag note on `extractComponent`) the * grammar has no dedicated rule for it: it is a generic `cf_tag` whose body * — the `<cffunction>` signatures — is ordinary nested children. Named like * a component (the file name, unless a `name` attribute says otherwise), so * `<cfcomponent implements="ISearchable">` resolves to it. An interface may * extend several interfaces, comma-separated. */ private extractInterfaceTag(tag: SyntaxNode, containerId: string): void { const startTag = this.cfStartTag(tag) ?? tag; const name = this.tagAttr(startTag, 'name') ?? this.componentNameFromPath(); const id = this.nodeIds.generate(this.filePath, 'interface', name, tag.startPosition.row + 1, tag.startPosition.column); this.nodes.push({ id, kind: 'interface', name, qualifiedName: `${this.filePath}::${name}`, filePath: this.filePath, language: this.language, startLine: tag.startPosition.row + 1, endLine: tag.endPosition.row + 1, startColumn: tag.startPosition.column, endColumn: tag.endPosition.column, isExported: true, updatedAt: Date.now(), }); this.edges.push({ source: containerId, target: id, kind: 'contains' }); this.pushInheritanceRefs(id, this.tagAttr(startTag, 'extends'), 'extends', startTag); this.delegateNestedTags(tag, id, name, true); } /** One unresolved `extends`/`implements` ref per name in a comma-separated tag attribute. */ private pushInheritanceRefs(fromNodeId: string, list: string | undefined, kind: 'extends' | 'implements', tag: SyntaxNode): void { if (!list) return; for (const name of list.split(',').map((s) => s.trim()).filter(Boolean)) { this.unresolvedReferences.push({ fromNodeId, referenceName: name, referenceKind: kind, filePath: this.filePath, line: tag.startPosition.row + 1, column: tag.startPosition.column, language: this.language, }); } } /** * `<cffunction name="..." access="..." returntype="...">...</cffunction>`. * `parentClassId` decides `method` vs top-level `function`; `containerId` is * the `contains`-edge target (the class when inside one, otherwise the file * node for a bare top-level cffunction) — kept separate so a top-level * function still gets a containment edge without being misclassified as a * method of the file. A method's qualifiedName is scoped under * `parentClassName` (`TagService::save`, the same `Class::member` shape the * generic extractor produces) so type-validated method resolution can match. */ private extractFunctionTag(tag: SyntaxNode, parentClassId: string | undefined, containerId: string | undefined, parentClassName?: string): void { const name = this.tagAttr(tag, 'name'); if (!name) return; const kind = parentClassId ? 'method' : 'function'; const id = this.nodeIds.generate(this.filePath, kind, name, tag.startPosition.row + 1, tag.startPosition.column); const access = this.tagAttr(tag, 'access'); const visibility = access === 'private' ? 'private' : access === 'package' ? 'internal' : access ? 'public' : undefined; const fnNode: Node = { id, kind, name, qualifiedName: parentClassName ? `${parentClassName}::${name}` : `${this.filePath}::${name}`, filePath: this.filePath, language: this.language, startLine: tag.startPosition.row + 1, endLine: tag.endPosition.row + 1, startColumn: tag.startPosition.column, endColumn: tag.endPosition.column, visibility, r - src/extraction/commonjs-requires.ts (3202 bytes)
/** * CommonJS `require('…')` as an import. * * An ESM `import … from './application'` is an `import_statement` the * extractor turns into an `imports` reference from the file, which the import * resolver lands on the module's file. A `require('./application')` is just a * call, so a CommonJS file depended on nothing at file level: Express's * `lib/express.js` requires `./application`, `./request` and `./response` and * the index said it imports nothing — an empty Imports rail in the viewer, no * link on the Map, and no file dependency for impact. A binding used only as a * value (`mixin(app, proto)`) never made a call edge to stand in for it either. * * Read from the source after extraction rather than in a walker, so the wasm * extractor and the native kernel get it alike — including the kernel's * deferred-decode transport, which carries these beside its buffers. Only the * module's own file reference is emitted — the names a binding is called * through already resolve through the require binding (import-resolver's * CommonJS reader). */ import type { Language, UnresolvedReference } from '../types'; import { stripCommentsForRegex } from '../resolution/strip-comments'; const COMMONJS_LANGUAGES: ReadonlySet<Language> = new Set(['javascript', 'jsx', 'typescript', 'tsx']); /** * `require('x')` with a literal specifier, not a member's (`foo.require(…)`) * or a property (`require.resolve(…)`). A template literal counts only when * it has no `${…}` hole. */ const REQUIRE_CALL = /(?<![\w$.])require\s*\(\s*(?:'([^'\n]+)'|"([^"\n]+)"|`([^`$\n]+)`)\s*\)/g; /** * A module of this project: a relative or rooted path, or a project alias * (`@/lib/x`, `~/x`, `#internal`). A package (`express`, `lodash/fp`) is * outside the index — as a name it would only match some same-named symbol. */ const LOCAL_SPECIFIER = /^(?:\.{1,2}(?:\/|$)|\/|[@~]\/|#)/; /** `import x = require('y')` is TypeScript's import-equals: an `import_statement` already. */ const IMPORT_EQUALS = /\bimport\s+(?:type\s+)?[\w$]+\s*=\s*$/; /** The file node both extractors mint: `file:<path>`. */ export function commonJsRequireRefs(filePath: string, source: string, language: Language): UnresolvedReference[] { if (!COMMONJS_LANGUAGES.has(language) || !source.includes('require')) return []; const safe = stripCommentsForRegex(source, language === 'javascript' || language === 'jsx' ? 'javascript' : 'typescript'); const out: UnresolvedReference[] = []; const seen = new Set<string>(); REQUIRE_CALL.lastIndex = 0; let m: RegExpExecArray | null; while ((m = REQUIRE_CALL.exec(safe)) !== null) { const specifier = (m[1] ?? m[2] ?? m[3] ?? '').trim(); if (!LOCAL_SPECIFIER.test(specifier) || seen.has(specifier)) continue; const lineStart = safe.lastIndexOf('\n', m.index - 1) + 1; if (IMPORT_EQUALS.test(safe.slice(lineStart, m.index))) continue; seen.add(specifier); out.push({ fromNodeId: `file:${filePath}`, referenceName: specifier, referenceKind: 'imports', line: safe.slice(0, m.index).split('\n').length, column: m.index - lineStart, filePath, language, }); } return out; } - src/extraction/dfm-extractor.ts (4825 bytes)
import { Node, Edge, ExtractionResult, ExtractionError, UnresolvedReference } from '../types'; import { generateNodeId } from './tree-sitter-helpers'; /** * Custom extractor for Delphi DFM/FMX form files. * * DFM/FMX files describe the visual component hierarchy and event handler * bindings. They use a simple text format (object/end blocks) that we parse * with regex — no tree-sitter grammar exists for this format. * * Extracted information: * - Components as NodeKind `component` * - Nesting as EdgeKind `contains` * - Event handlers (OnClick = MethodName) as UnresolvedReference → EdgeKind `references` */ export class DfmExtractor { private filePath: string; private source: string; private nodes: Node[] = []; private edges: Edge[] = []; private unresolvedReferences: UnresolvedReference[] = []; private errors: ExtractionError[] = []; constructor(filePath: string, source: string) { this.filePath = filePath; this.source = source; } /** * Extract components and event handler references from DFM/FMX source */ extract(): ExtractionResult { const startTime = Date.now(); try { const fileNode = this.createFileNode(); this.parseComponents(fileNode); } catch (error) { this.errors.push({ message: `DFM extraction error: ${error instanceof Error ? error.message : String(error)}`, severity: 'error', code: 'parse_error', }); } return { nodes: this.nodes, edges: this.edges, unresolvedReferences: this.unresolvedReferences, errors: this.errors, durationMs: Date.now() - startTime, }; } /** Create a file node for the DFM form file */ private createFileNode(): Node { const lines = this.source.split('\n'); const id = generateNodeId(this.filePath, 'file', this.filePath, 1); const fileNode: Node = { id, kind: 'file', name: this.filePath.split('/').pop() || this.filePath, qualifiedName: this.filePath, filePath: this.filePath, language: 'pascal', startLine: 1, endLine: lines.length, startColumn: 0, endColumn: lines[lines.length - 1]?.length || 0, updatedAt: Date.now(), }; this.nodes.push(fileNode); return fileNode; } /** Parse object/end blocks and extract components + event handlers */ private parseComponents(fileNode: Node): void { const lines = this.source.split('\n'); const stack: Node[] = [fileNode]; const objectPattern = /^\s*(object|inherited|inline)\s+(\w+)\s*:\s*(\w+)/; const eventPattern = /^\s*(On\w+)\s*=\s*(\w+)\s*$/; const endPattern = /^\s*end\s*$/; const multiLineStart = /=\s*\(\s*$/; const multiLineItemStart = /=\s*<\s*$/; let inMultiLine = false; let multiLineEndChar = ')'; for (let i = 0; i < lines.length; i++) { const line = lines[i]!; const lineNum = i + 1; // Skip multi-line properties if (inMultiLine) { if (line.trimEnd().endsWith(multiLineEndChar)) inMultiLine = false; continue; } if (multiLineStart.test(line)) { inMultiLine = true; multiLineEndChar = ')'; continue; } if (multiLineItemStart.test(line)) { inMultiLine = true; multiLineEndChar = '>'; continue; } // Component declaration const objMatch = line.match(objectPattern); if (objMatch) { const [, , name, typeName] = objMatch; const nodeId = generateNodeId(this.filePath, 'component', name!, lineNum); const node: Node = { id: nodeId, kind: 'component', name: name!, qualifiedName: `${this.filePath}#${name}`, filePath: this.filePath, language: 'pascal', startLine: lineNum, endLine: lineNum, startColumn: 0, endColumn: line.length, signature: typeName, updatedAt: Date.now(), }; this.nodes.push(node); this.edges.push({ source: stack[stack.length - 1]!.id, target: nodeId, kind: 'contains', }); stack.push(node); continue; } // Event handler const eventMatch = line.match(eventPattern); if (eventMatch) { const [, , methodName] = eventMatch; this.unresolvedReferences.push({ fromNodeId: stack[stack.length - 1]!.id, referenceName: methodName!, referenceKind: 'references', line: lineNum, column: 0, }); continue; } // Block end if (endPattern.test(line)) { if (stack.length > 1) { const node = stack.pop()!; node.endLine = lineNum; node.endColumn = line.length; } } } } } - src/extraction/extraction-version.ts (1329 bytes)
/** * Extraction version * * A monotonically-increasing integer that identifies the *shape and depth* of * what the extractor writes into the graph. Unlike `CURRENT_SCHEMA_VERSION` * (which tracks the SQLite table layout and is migrated in place), this tracks * the EXTRACTED CONTENT — node kinds, edges, synthesizers, resolver coverage. * * When an index was built by an older engine whose `EXTRACTION_VERSION` is * below the running engine's, the data on disk is structurally fine but * *stale*: it's missing whatever a newer extractor would now produce. A schema * migration can't backfill that — only a re-index can. So this is the signal * `codegraph status` uses to recommend a re-index, and the reason `codegraph * upgrade` reminds users to refresh their projects. * * BUMP THIS when a release changes extraction output enough that existing * indexes should be rebuilt to benefit — e.g. a new language/framework * extractor, a new dynamic-dispatch synthesizer, a new node/edge kind, or a * resolver fix that materially changes which edges exist. Do NOT bump for * pure bug fixes, CLI/UX changes, or schema-only migrations. Over-bumping * turns the re-index hint into noise — keep it honest (see CLAUDE.md, "Honesty * in the product is load-bearing"). */ export const EXTRACTION_VERSION = 29;
- src/extraction/function-ref.ts (45151 bytes)
/** * Function-as-value capture (#756) — registration-linking for callbacks. * * A function name used as a VALUE — passed as a call argument * (`register_handler(target_cb)`, `signal(SIGINT, handler)`), assigned to a * field or function pointer (`o->cb = target_cb`, `OnFire := TargetCb`), * placed in a struct/object initializer (`{ .recv_cb = my_cb }`, * `{ recv: targetCb }`, `Ops{Cb: targetCb}`), or listed in a function table * (`static cb_t table[] = { cb_a, cb_b }`) — is a real dependency that static * call extraction misses entirely: `callers(target_cb)` showed nothing but * direct calls, so every callback looked dead and its registration sites were * invisible to impact analysis. * * This module captures those value positions during the AST walk as * `function_ref` candidates. Capture is table-driven per language (the value * positions and wrapper forms differ per grammar — `&fn` in C, `Main::fn` in * Java, `::fn` in Kotlin, `#selector(fn)` in Swift, `@TargetCb` in Pascal, * `method(:fn)` in Ruby). Candidates are GATED at end-of-file extraction * (see `TreeSitterExtractor.flushFnRefCandidates`): only names matching a * same-file function/method or an imported binding survive, which bounds * volume and keeps precision high. Resolution then matches survivors against * function/method nodes ONLY (`matchFunctionRef` in * `src/resolution/name-matcher.ts`) and persists them as `references` edges, * which `callers`/`impact` already traverse. * * Deliberately NOT covered (resolving the *dispatch* — `o->cb(x)` → the * registered function — needs data-flow through struct fields; a wrong edge * is worse than none): indirect-call resolution. Member values where the * receiver isn't `this`/`self` (`pool.submit(obj.method)`, `Submit(c.store.Fetch)`) * retain their receiver (#1820), so resolution can use type/import scope * before considering a unique method name. */ import type { Node as SyntaxNode } from 'web-tree-sitter'; import { getNodeText, getChildByField } from './tree-sitter-helpers'; export interface FnRefCandidate { name: string; line: number; column: number; /** Which capture position produced this candidate (gate policy keys on it). */ mode: CaptureMode; /** * True when the value was an explicit reference form (`&fn`, `&Cls::m`, * `::fn`, `#selector`, `method(:sym)`) rather than a bare identifier — * C++'s flush policy keys on it. */ explicitRef: boolean; /** * Skip the same-file/import name gate for this candidate. Set for PHP * string callables in known HOF positions: PHP global functions are * referenced cross-file WITHOUT imports (global namespace), so the gate * can't see them — the strong positional prior (a string argument to * `usort`/`array_map`/…) plus resolution's unique-or-drop rule carry the * precision instead. Python/Go member values also skip the name gate, * retaining their receiver for scoped resolution. */ skipGate?: boolean; } /** How to pull candidate value nodes out of a dispatched container node. */ type CaptureMode = | 'args' // every named child is a potential value (call argument lists) | 'rhs' // the assignment right-hand side (named field, else last named child) | 'value' // the `value` field of a keyed pair (object/struct/table initializers) | 'list' // every named child (array / initializer-list / table positional elements) | 'varinit'; // a variable declarator's initializer value interface CaptureRule { mode: CaptureMode; /** Field holding the value for rhs/value/varinit (defaults per mode). */ field?: string; } export interface FnRefSpec { /** Bare identifier node types that can act as a function value. */ idTypes: Set<string>; /** Container node type → how to extract candidate values from it. */ dispatch: Map<string, CaptureRule>; /** * Transparent wrapper layers between a container and its values * (`argument`, `value_argument`, `literal_element`, `expression_list`…). * Value: the field to descend into, or null for "named children". * `expression_list` fans out to ALL named children (Go multi-assign). */ layers?: Map<string, string | null>; /** * Unary wrappers whose operand is the function value — C/C++ `&fn` * (pointer_expression), Pascal `@Fn` (exprUnary), Scala eta `fn _` * (postfix_expression). Value: operand field, or null for first named child. */ unwrap?: Map<string, string | null>; /** * Whole-node reference forms needing bespoke name extraction — * `method_reference` (Java), `callable_reference` / `navigation_expression` * (Kotlin), `selector_expression` (Swift `#selector` / ObjC `@selector`), * Ruby `method(:sym)` calls, and `this.method` member forms. */ special?: Set<string>; /** * Capture modes whose candidates skip the same-file/import gate and rely on * resolution's unique-or-drop rule instead. C-family only: an initializer * value, function-pointer assignment RHS, or table element is a * function-pointer position by construction, and C has no symbol imports — * the dominant repo-scale pattern (`server.c`'s command table naming * handlers defined across files) would otherwise be invisible. Call * arguments stay gated everywhere (locals passed as args dwarf callbacks). */ ungatedModes?: Set<CaptureMode>; /** * C++ only: in args/rhs/varinit positions, accept ONLY explicit reference * forms (`&fn`, `&Cls::method`) — never bare identifiers. C++ codebases are * dense with generic free-function/accessor names (`begin`, `end`, `out`, * `size`, `data`) that collide with parameters and locals, and out-of-line * member definitions extract as function-kind nodes — bare-id matching on * fmt was mostly wrong edges. File-scope initializer tables (value/list) * still accept bare identifiers, same as C. */ addressOfOnly?: boolean; } /** Names that are never function references even when grammars call them identifiers. */ const NAME_STOPLIST = new Set([ 'this', 'self', 'super', 'null', 'nil', 'true', 'false', 'undefined', 'new', 'NULL', 'nullptr', 'None', ]); // --------------------------------------------------------------------------- // Per-language specs. Node types verified against each grammar (probe fixtures // in the #756 investigation; see docs/design/function-ref-capture.md). // --------------------------------------------------------------------------- /** C / C++ / Objective-C share the C-family initializer & assignment shapes. */ function cFamilySpec(extra?: { special?: string[]; addressOfOnly?: boolean }): FnRefSpec { return { idTypes: new Set(['identifier']), dispatch: new Map<string, CaptureRule>([ ['argument_list', { mode: 'args' }], ['assignment_expression', { mode: 'rhs', field: 'right' }], ['init_declarator', { mode: 'varinit', field: 'value' }], ['initializer_list', { mode: 'list' }], ['initializer_pair', { mode: 'value', field: 'value' }], ]), unwrap: new Map([['pointer_expression', 'argument']]), special: new Set(extra?.special ?? []), // C has no symbol imports, and callbacks are registered cross-file at repo // scale (redis: server.c's command table names handlers from t_*.c) — so // initializer positions bypass the gate and lean on resolution's // unique-or-drop rule. ONLY 'value'/'list' (struct/array initializers), // and the flush additionally requires FILE scope: a C file-scope // initializer is a constant-expression context, so a bare identifier // there can only be a function address (or enum/macro, which the // function-kind filter drops) — never a variable. 'rhs'/'varinit' were // tried and produced false edges (`prev = next`, `*str = field` — data // assignments matching a unique same-named function elsewhere), so // assignments stay gated to same-file/import. ungatedModes: new Set<CaptureMode>(['value', 'list']), addressOfOnly: extra?.addressOfOnly, }; } // `this.handleClick` capture (member_expression) emits a `this.`-PREFIXED // candidate name: resolution scopes it to the enclosing symbol's class // (qualified-name prefix), so `this.fonts` (a property, post-#808) and // inherited/unknown members yield no edge, while same-class methods — // `btn.on('click', this.handleClick)`, the observer-registration idiom — // resolve precisely. Bare identifiers stay function-kind-only (a bare id can // never be a method value in JS). const TS_JS_SPEC: FnRefSpec = { // `shorthand_property_identifier`: `{ handleSubmit }` — the object a hook // returns its handlers in, and a namespace object's members. idTypes: new Set(['identifier', 'shorthand_property_identifier']), dispatch: new Map<string, CaptureRule>([ ['arguments', { mode: 'args' }], ['assignment_expression', { mode: 'rhs', field: 'right' }], ['variable_declarator', { mode: 'varinit', field: 'value' }], ['pair', { mode: 'value', field: 'value' }], ['array', { mode: 'list' }], // A JSX attribute value or child: `onPress={handleSubmit}`, `renderItem={renderRow}`, // `<Route component={Home}/>`. The expression's one named child is the value; a // spread or a call normalizes to nothing. This is THE handler-binding idiom of // React, and without it a tap's handler had no edge from the component that // renders it — the Screens and Steps views could not see what a tap does. ['jsx_expression', { mode: 'list' }], // An object literal's shorthand members — `return { handleApprove, // handleRetake }` from a hook, `const Api = { upload, createFolder }`. // Every named child is offered; only a shorthand identifier normalizes // (a `pair` is its own container above, a spread or a method is nothing). ['object', { mode: 'list' }], ]), special: new Set(['member_expression']), }; const PYTHON_SPEC: FnRefSpec = { idTypes: new Set(['identifier']), dispatch: new Map<string, CaptureRule>([ ['argument_list', { mode: 'args' }], ['assignment', { mode: 'rhs', field: 'right' }], ['keyword_argument', { mode: 'value', field: 'value' }], // Thread(target=worker) ['pair', { mode: 'value', field: 'value' }], ['list', { mode: 'list' }], // `return SomeClass` / `return handler` — factory returns are how DRF // wires views to serializers (get_serializer_class) and how Python // factories hand back callables (#1478). A single returned expression is // a direct named child, so 'list' covers it; tuple returns (`return A, B`) // sit under an expression_list child and are deliberately not descended. ['return_statement', { mode: 'list' }], ]), special: new Set(['attribute']), }; const GO_SPEC: FnRefSpec = { idTypes: new Set(['identifier']), dispatch: new Map<string, CaptureRule>([ ['argument_list', { mode: 'args' }], ['assignment_statement', { mode: 'rhs', field: 'right' }], ['short_var_declaration', { mode: 'rhs', field: 'right' }], ['var_spec', { mode: 'varinit', field: 'value' }], ['keyed_element', { mode: 'value' }], // value = last literal_element child ['literal_value', { mode: 'list' }], // positional composite literals ]), layers: new Map<string, string | null>([ ['literal_element', null], ['expression_list', null], ]), special: new Set(['selector_expression']), }; const RUST_SPEC: FnRefSpec = { idTypes: new Set(['identifier']), dispatch: new Map<string, CaptureRule>([ ['arguments', { mode: 'args' }], ['assignment_expression', { mode: 'rhs', field: 'right' }], ['field_initializer', { mode: 'value', field: 'value' }], ['array_expression', { mode: 'list' }], ['static_item', { mode: 'varinit', field: 'value' }], ['let_declaration', { mode: 'varinit', field: 'value' }], ]), // `g.map(Graph::counts)`, `map(Self::helper)`, `or_else(fs::default_sep)`: // a function passed through its path (#3254), resolved as that path's call. special: new Set(['scoped_identifier']), }; const JAVA_SPEC: FnRefSpec = { // No bare-identifier function values in Java — only method references. idTypes: new Set<string>(), dispatch: new Map<string, CaptureRule>([ ['argument_list', { mode: 'args' }], ['assignment_expression', { mode: 'rhs', field: 'right' }], ['variable_declarator', { mode: 'varinit', field: 'value' }], ]), special: new Set(['method_reference']), }; const KOTLIN_SPEC: FnRefSpec = { idTypes: new Set<string>(), dispatch: new Map<string, CaptureRule>([ ['value_arguments', { mode: 'args' }], ['assignment', { mode: 'rhs' }], // RHS = last named child (no field in grammar) ]), layers: new Map<string, string | null>([['value_argument', null]]), special: new Set(['callable_reference', 'navigation_expression']), }; const CSHARP_SPEC: FnRefSpec = { idTypes: new Set(['identifier']), dispatch: new Map<string, CaptureRule>([ ['argument_list', { mode: 'args' }], ['assignment_expression', { mode: 'rhs', field: 'right' }], // covers `+=` event subscription ['initializer_expression', { mode: 'list' }], ['variable_declarator', { mode: 'varinit' }], ]), layers: new Map<string, string | null>([['argument', null]]), special: new Set(['member_access_expression']), }; const RUBY_SPEC: FnRefSpec = { // Bare identifiers in Ruby args are method CALLS or locals, never function // values — only the `method(:name)` idiom (and `&method(:name)`) plus // hook-DSL symbols (`before_action :authenticate`) qualify. idTypes: new Set<string>(), dispatch: new Map<string, CaptureRule>([ ['argument_list', { mode: 'args' }], ['pair', { mode: 'value', field: 'value' }], ]), layers: new Map<string, string | null>([['block_argument', null]]), special: new Set(['call', 'simple_symbol']), }; /** * Rails/ActiveSupport-style hook DSLs whose symbol arguments name a method of * the enclosing class: lifecycle callbacks (`before_action`, `after_save`, * `around_create`, `skip_before_action`…), `validate :method`, `set_callback`, * `helper_method`, and `rescue_from(..., with: :handler)`. NOT `validates` * (plural) — its symbols name ATTRIBUTES, not methods. */ const RUBY_HOOK_RE = /^(skip_)?(before|after|around)_[a-z_]+$/; const RUBY_HOOK_NAMES = new Set(['validate', 'set_callback', 'helper_method', 'rescue_from']); function isRubyHookCall(name: string): boolean { return RUBY_HOOK_RE.test(name) || RUBY_HOOK_NAMES.has(name); } const SWIFT_SPEC: FnRefSpec = { idTypes: new Set(['simple_identifier']), dispatch: new Map<string, CaptureRule>([ ['value_arguments', { mode: 'args' }], ['assignment', { mode: 'rhs', field: 'result' }], ['array_literal', { mode: 'list' }], ['property_declaration', { mode: 'varinit', field: 'value' }], ]), layers: new Map<string, string | null>([['value_argument', 'value']]), special: new Set(['selector_expression']), }; const SCALA_SPEC: FnRefSpec = { idTypes: new Set(['identifier']), di - src/extraction/generated-detection.ts (13206 bytes)
/** * Generated-file detection for symbol-disambiguation down-ranking. * * When a query like "Send" matches 17 symbols across protobuf scaffolding, * test mocks, and the hand-written implementation, the FTS ranker often * surfaces the generated stubs first because their names are identical * to the implementation's name (validated empirically on cosmos-sdk — * see project_go_multi_module_audit memory). Generated stubs frequently * have no body to trace from, so the agent ends up reading source anyway. * * This is a relevance hint consulted at disambiguation time (findSymbol / * findAllSymbols / explore ranking / codegraph_search formatting), NOT a * hard filter — generated nodes are still in the graph and remain * reachable; they just rank LAST when there's a real implementation with * the same name. * * Two signals, deliberately separate: * * 1. {@link isGeneratedFile} — PATH only, pure and synchronous. Most * generated files follow the `<basename>.<tool>.<ext>` convention * (`.pb.go`, `_grpc.pb.go`, `.g.dart`, `_pb2.py`). Free to call * anywhere, including in a sort comparator. * * 2. {@link hasGeneratedHeader} — CONTENT banner in the file's head. Go's * own convention is a content marker, not a filename one, so a * generated `payroll.go` sitting beside hand-written use-cases is * invisible to (1) — that is issue #1500. Evaluated ONCE at index time * (the file's content is already in memory for parsing) and persisted * on the file record as `files.generated`; readers get it from the DB * rather than re-reading headers per request. See * GENERATED_CONTENT_PATTERNS below for the banners recognized. * * Consumers that have a bounded candidate list should use the DB-backed * union (`QueryBuilder.getGeneratedPathsAmong` / * `CodeGraph.getGeneratedFilePaths`) so both signals apply; the path-only * check remains the fallback for callers with no database in hand and for * indexes built before the flag existed. * * NOTE for future editors: the banner literals quoted in this file sit * BELOW the header window this detector scans, so the module does not * classify itself. `generated-detection.test.ts` pins that — if you move * the pattern table upward, the test fails rather than the repo silently * demoting its own file. */ const GENERATED_PATTERNS: ReadonlyArray<RegExp> = [ // Go — protobuf / gRPC / pulsar /\.pb\.go$/, /\.pulsar\.go$/, /_grpc\.pb\.go$/, // Go — mockgen output. Default emits `mock_<src>.go`; many projects // (cosmos-sdk uses `expected_*_mocks.go`) rename to `*_mock.go` / // `*_mocks.go`. Matching either suffix catches both conventions // without false-positive risk on hand-written sources. /_mock\.go$/, /_mocks\.go$/, /^mock_[^/]+\.go$/, // TypeScript / JavaScript — common codegen suffixes (Apollo / GraphQL // codegen, Prisma, Hasura, ts-proto, gRPC-web, swagger-codegen). /\.generated\.[jt]sx?$/, /\.gen\.[jt]sx?$/, /\.pb\.[jt]s$/, /_pb\.[jt]s$/, /_grpc_pb\.[jt]s$/, // Minified bundles vendored into a repo (docs sites, examples). Their // single-letter symbols make name-based edges pure noise. /[.-]min\.m?js$/, // Python — protobuf / gRPC / openapi-codegen /_pb2(_grpc)?\.py$/, /_pb2\.pyi$/, // C++ — protobuf /\.pb\.(cc|h)$/, // C# — protobuf / gRPC (protoc-gen-csharp puts output under obj/ but // many projects also commit *.g.cs and *Grpc.cs siblings) /\.g\.cs$/, /Grpc\.cs$/, // Java — protobuf / gRPC: protoc-gen-java emits `*OuterClass.java`, // protoc-gen-grpc-java emits `*Grpc.java`. The XxxImplBase abstract // class lives inside Xxx*Grpc.java. /OuterClass\.java$/, /Grpc\.java$/, // Swift — protobuf /\.pb\.swift$/, // Dart — build_runner / freezed / json_serializable / chopper /\.g\.dart$/, /\.freezed\.dart$/, /\.pb\.dart$/, /\.pbgrpc\.dart$/, /\.chopper\.dart$/, // Rust — common build.rs OUT_DIR outputs are usually outside the source // tree, but in-tree generated files often use `*.generated.rs`. /\.generated\.rs$/, ]; /** * Whether `filePath` looks like a tool-generated source file based on * its filename. Path-only — does not read content. The result is a * relevance hint for disambiguation, not a hard claim. */ export function isGeneratedFile(filePath: string): boolean { return GENERATED_PATTERNS.some((p) => p.test(filePath)); } // ============================================================================= // Content-header detection (#1500) // ============================================================================= /** * How much of a file's head to consider "the header". Generous enough for a * build-tag block + an Apache-2.0 license preamble (~15 lines) sitting above * the banner, tight enough that a `"// Code generated ... DO NOT EDIT."` * string constant in the *body* of a code generator's own source can't * masquerade as a banner. */ const HEADER_SCAN_CHARS = 8192; const HEADER_SCAN_LINES = 60; /** * Cheap pre-filter run on the header of EVERY indexed file. Every marker * below contains the stem "generat", so one unanchored scan rejects ~all * hand-written source before any line splitting happens — this is what keeps * content detection off the index-time cost budget. */ const GENERATED_STEM = /generat/i; /** * Line-comment leaders across the languages we index. A banner must sit on a * comment line (or inside an open block comment, tracked below): generators * always emit theirs as a comment, and requiring it rules out string literals * and identifiers that merely contain the words. * * `--` covers SQL/Haskell/Lua, `%` LaTeX/Erlang/Prolog, `;` Lisp/asm/ini, * `'` VB, `!` Fortran, `*` a continuation line inside a `/* … *\/` block. */ const COMMENT_LEADER = /^\s*(?:\/\/|\/\*+|\*+\/?|#+|--+|<!--|%+|;+|'|!|\(\*|\{-|"""|'''|=begin|<#|@rem\b|rem\b)/i; /** * Openers/closers for block comments, so a banner on an unprefixed line * inside `/* … *\/` (or `<!-- … -->`, or a Python module docstring) still * counts. Deliberately naive — it only runs over a file's first few dozen * lines, where a `/*` inside a string literal is vanishingly rare, and the * worst case of a mis-tracked state is a ranking hint, not a wrong answer. */ const BLOCK_DELIMS: ReadonlyArray<{ open: string; close: string }> = [ { open: '/*', close: '*/' }, { open: '<!--', close: '-->' }, { open: '"""', close: '"""' }, { open: "'''", close: "'''" }, { open: '=begin', close: '=end' }, { open: '<#', close: '#>' }, ]; /** * The banners themselves. Each is a real convention emitted by a widely-used * generator; the list is precision-first, because a false positive silently * demotes hand-written code in every ranking path. */ const GENERATED_CONTENT_PATTERNS: ReadonlyArray<RegExp> = [ // Go's codified convention — `^// Code generated .* DO NOT EDIT\.$`, defined // by `go generate` and honored by gofmt, golangci-lint and GitHub linguist. // Emitted verbatim by protoc-gen-go, mockgen, sqlc, ent, wire, stringer, and // by in-house generators like the FKIT CRUD in #1500 — where the file is // named `payroll.go` and nothing in the PATH gives it away. /\bcode generated\b.{0,200}?\bdo not edit\b/i, // protoc's Java/C#/Python banner ("Generated by the protocol buffer // compiler. DO NOT EDIT!"), ANTLR, Dagger, FlatBuffers, rust-bindgen, // Xcode asset catalogs, Bazel rules. /\b(?:automatically |auto[- ]?)?generated (?:by|from|with)\b.{0,200}?\bdo not (?:edit|modify|change)\b/i, // The `@generated` marker: the JS/TS ecosystem's convention (Relay, GraphQL // codegen, protobuf-es/Buf, Meta's `@generated SignedSource<<…>>`), also // what linguist and `git diff` collapse on. Guarded against `foo@generated` // and `@@generated` so only a standalone tag matches. /(?:^|[^\p{L}\p{N}_@])@generated\b/u, // .NET's `<auto-generated>` / `<auto-generated />` doc tag: Roslyn, the // WinForms designer, T4 templates, protoc-gen-csharp, EF scaffolding. /<auto-?generated\s*\/?>/i, // swagger-codegen / OpenAPI Generator ("NOTE: This class is auto generated // by OpenAPI Generator"), Thrift ("Autogenerated by Thrift Compiler"), // FlatBuffers ("automatically generated by the FlatBuffers compiler"). // "by" is required — bare "automatically generated" appears in hand-written // prose ("the table below is automatically generated at runtime"). /\b(?:automatically generated|auto[- ]?generated|autogenerated) by\b/i, // The "run this command to regenerate" shape: Cloudflare Wrangler // ("Generated by Wrangler by running `wrangler types` (hash: …)"), and the // same phrasing used by other CLI-driven emitters. Bare "generated by" is // deliberately NOT enough — it is ordinary prose — so the reproduction // instruction is the discriminator: the banner must name a tool AND then // say `by running`, i.e. TWO separate "by" clauses. That rules out // "the report is generated by running the nightly job", which has only one. /\bgenerated by\s+\S.{0,80}?\bby running\b/i, // Self-declaring in-house banners that name no tool. /\bthis (?:file|class|code|module) (?:is|was) (?:auto[- ]?)?generated\b/i, // The reverse ordering: "DO NOT EDIT — this is a generated file". /\bdo not (?:edit|modify)\b.{0,120}?\b(?:auto[- ]?generated|generated file|generated code)\b/i, ]; /** * Whether the head of `content` carries a recognized machine-generation * banner. Bounded to {@link HEADER_SCAN_CHARS} / {@link HEADER_SCAN_LINES}, * and the marker must sit on a comment line — a generator's own source, which * holds the banner as a string constant in its body, is not flagged. * * Called once per file during extraction (content is already in memory), NOT * per query: the verdict is persisted on the file record. */ export function hasGeneratedHeader(content: string): boolean { if (!content) return false; const head = content.length > HEADER_SCAN_CHARS ? content.slice(0, HEADER_SCAN_CHARS) : content; // Fast reject for ~every hand-written file: no line splitting, no allocation // (V8 keeps `head` as a sliced view of `content`). if (!GENERATED_STEM.test(head)) return false; const lines = head.split('\n'); const limit = Math.min(lines.length, HEADER_SCAN_LINES); let openBlock: (typeof BLOCK_DELIMS)[number] | null = null; for (let i = 0; i < limit; i++) { const line = lines[i]!; const inBlock = openBlock !== null; if (inBlock || COMMENT_LEADER.test(line)) { for (const pattern of GENERATED_CONTENT_PATTERNS) { if (pattern.test(line)) return true; } } // Advance the block-comment state AFTER testing, so the opening line of a // `/* Code generated … */` block is itself matched by the leader rule. if (openBlock) { if (line.includes(openBlock.close)) openBlock = null; continue; } for (const delim of BLOCK_DELIMS) { const at = line.indexOf(delim.open); if (at < 0) continue; // Same-line close (`/* … */`, a one-line docstring) leaves no open block. if (line.indexOf(delim.close, at + delim.open.length) < 0) openBlock = delim; break; } } return false; } /** * The union signal: path convention OR content banner. This is what the * indexer persists to `files.generated`. */ export function detectGeneratedFile(filePath: string, content: string): boolean { return isGeneratedFile(filePath) || hasGeneratedHeader(content) || isMinifiedContent(filePath, content); } /** Scripts a bundler or minifier writes as a few enormous lines. */ const MINIFIABLE_SCRIPT = /\.(?:m?js|cjs)$/i; /** The loader every webpack bundle defines for its modules. */ const WEBPACK_RUNTIME = /\bfunction __webpack_require__\s*\(/; /** A line this long is not written by hand. */ const MINIFIED_LINE_CHARS = 1000; /** * A minified or bundled script that is not NAMED so — a docs site's * `bundle.js`, a sample's vendored `app.js`: most of its text sits on lines of * a thousand characters or more, and that text is code, dense with `;{}(),`, * not one long string literal (a base64 fixture) in an ordinary file. Its * one-letter functions (`n`, `t`, `e`) otherwise topped the most-depended-on * lists of the projects that vendor it. */ export function isMinifiedContent(filePath: string, content: string): boolean { if (!MINIFIABLE_SCRIPT.test(filePath) || content.length < 4 * MINIFIED_LINE_CHARS) return false; // A webpack build carries its module loader, however readable its lines. if (WEBPACK_RUNTIME.test(content)) return true; let long = 0; let punctuation = 0; for (let start = 0; start < content.length; ) { let end = content.indexOf('\n', start); if (end < 0) end = content.length; if (end - start >= MINIFIED_LINE_CHARS) { long += end - start; for (let i = start; i < end; i++) { const c = content.charCodeAt(i); // ; { } ( ) , if (c === 59 || c === 123 || c === 125 || c === 40 || c === 41 || c === 44) punctuation++; } } start = end + 1; } return long >= content.length * 0.5 && punctuation >= long * 0.03; } - src/extraction/grammars.ts (54225 bytes)
/** * Grammar Loading and Caching * * Uses web-tree-sitter (WASM) for universal cross-platform support. * Grammars are loaded lazily — only languages actually present in the project * are compiled, keeping V8 WASM memory pressure low on large codebases. */ import * as path from 'path'; import * as fs from 'fs'; import * as fsp from 'fs/promises'; import { Parser, Language as WasmLanguage } from 'web-tree-sitter'; import { ExtractionError, Language } from '../types'; export type GrammarLanguage = Exclude<Language, 'svelte' | 'vue' | 'astro' | 'liquid' | 'razor' | 'yaml' | 'twig' | 'xml' | 'properties' | 'unknown'>; /** * WASM filename map — maps each language to its .wasm grammar file * in the tree-sitter-wasms package. */ const WASM_GRAMMAR_FILES: Record<GrammarLanguage, string> = { typescript: 'tree-sitter-typescript.wasm', tsx: 'tree-sitter-tsx.wasm', javascript: 'tree-sitter-javascript.wasm', jsx: 'tree-sitter-javascript.wasm', python: 'tree-sitter-python.wasm', go: 'tree-sitter-go.wasm', rust: 'tree-sitter-rust.wasm', java: 'tree-sitter-java.wasm', c: 'tree-sitter-c.wasm', cpp: 'tree-sitter-cpp.wasm', csharp: 'tree-sitter-c_sharp.wasm', php: 'tree-sitter-php.wasm', ruby: 'tree-sitter-ruby.wasm', swift: 'tree-sitter-swift.wasm', kotlin: 'tree-sitter-kotlin.wasm', dart: 'tree-sitter-dart.wasm', pascal: 'tree-sitter-pascal.wasm', scala: 'tree-sitter-scala.wasm', lua: 'tree-sitter-lua.wasm', r: 'tree-sitter-r.wasm', luau: 'tree-sitter-luau.wasm', objc: 'tree-sitter-objc.wasm', cfml: 'tree-sitter-cfml.wasm', cfscript: 'tree-sitter-cfscript.wasm', cfquery: 'tree-sitter-cfquery.wasm', cobol: 'tree-sitter-cobol.wasm', vbnet: 'tree-sitter-vbnet.wasm', erlang: 'tree-sitter-erlang.wasm', solidity: 'tree-sitter-solidity.wasm', terraform: 'tree-sitter-terraform.wasm', arkts: 'tree-sitter-arkts.wasm', nix: 'tree-sitter-nix.wasm', }; /** * File extension to Language mapping */ export const EXTENSION_MAP: Record<string, Language> = { '.ts': 'typescript', '.tsx': 'tsx', // ESM/CJS TypeScript module extensions — parsed as TS (no JSX). (#366) '.mts': 'typescript', '.cts': 'typescript', // ArkTS (HarmonyOS / OpenHarmony) — a TypeScript superset with declarative // UI (`@Component struct` + `build()`). Own grammar (a tree-sitter-typescript // -style fork); plain `.ts` in an ArkTS project stays TypeScript. (#648) '.ets': 'arkts', '.js': 'javascript', '.mjs': 'javascript', '.cjs': 'javascript', // SAP HANA XS Classic server-side JavaScript. (#556) '.xsjs': 'javascript', '.xsjslib': 'javascript', '.jsx': 'jsx', '.py': 'python', '.pyw': 'python', '.go': 'go', '.rs': 'rust', '.java': 'java', '.c': 'c', '.h': 'c', // Could also be C++, defaulting to C '.cpp': 'cpp', '.cc': 'cpp', '.cxx': 'cpp', '.hpp': 'cpp', '.hxx': 'cpp', '.cs': 'csharp', // ASP.NET Razor / Blazor markup — custom RazorExtractor (links @model/@inject/ // component tags to their C# types; markup isn't a tree-sitter grammar). '.cshtml': 'razor', '.razor': 'razor', '.php': 'php', // Drupal-specific PHP file extensions '.module': 'php', '.install': 'php', '.theme': 'php', '.inc': 'php', // YAML (used for Drupal routing files; no symbol extraction, file-level tracking only) '.yml': 'yaml', '.yaml': 'yaml', // Twig templates (file-level tracking only, no symbol extraction) '.twig': 'twig', '.rb': 'ruby', '.rake': 'ruby', '.swift': 'swift', '.kt': 'kotlin', '.kts': 'kotlin', '.dart': 'dart', '.liquid': 'liquid', '.svelte': 'svelte', '.vue': 'vue', '.astro': 'astro', '.r': 'r', '.pas': 'pascal', '.dpr': 'pascal', '.dpk': 'pascal', '.lpr': 'pascal', '.dfm': 'pascal', '.fmx': 'pascal', '.scala': 'scala', '.sc': 'scala', '.lua': 'lua', '.luau': 'luau', '.m': 'objc', '.mm': 'objc', '.sol': 'solidity', // CFML: .cfc/.cfm parse with the tag-aware `cfml` grammar (custom CfmlExtractor // dialect-switches to cfscript for bare-script content); .cfs is pure CFScript. '.cfc': 'cfml', '.cfm': 'cfml', '.cfs': 'cfscript', // Metal Shading Language ≈ C++14: the C++ grammar extracts its functions, // structs, and calls. MSL-specific `[[attribute]]` annotations are blanked // pre-parse for `.metal` files (see blankMetalAttributes in c-cpp.ts). (#1121) '.metal': 'cpp', // CUDA ≈ C++ plus execution-space specifiers (`__global__` …) and // `<<<grid, block>>>` kernel-launch syntax: the C++ grammar extracts its // functions/structs/classes/calls once blankCudaConstructs (pre-parse; gated // by these extensions OR by content for CUDA living in `.h`/`.hpp` headers — // see c-cpp.ts) blanks the CUDA-only tokens. (#387) '.cu': 'cpp', '.cuh': 'cpp', '.nix': 'nix', // XML: file-level tracking; the MyBatis extractor matches `<mapper namespace="...">` // shape and emits SQL-statement nodes (other XML returns empty). '.xml': 'xml', // COBOL: programs (.cbl/.cob) and copybooks (.cpy). Vendored grammar // (patched yutaro-sakamoto/tree-sitter-cobol) handles fixed-format column // rules, EXEC CICS/SQL blocks, and standalone copybook fragments. '.cbl': 'cobol', '.cob': 'cobol', '.cobol': 'cobol', '.cpy': 'cobol', // VB.NET: vendored grammar (patched govindbanura/tree-sitter-vbnet) — classes, // modules, interfaces, structures, properties, events, Handles clauses, LINQ. '.vb': 'vbnet', // Erlang: modules (.erl) and header files (.hrl). Vendored WhatsApp/ // tree-sitter-erlang grammar (the ELP grammar). '.erl': 'erlang', '.hrl': 'erlang', // escripts parse natively — the grammar has a first-class `shebang` node. // (`.app`/`.app.src` resource files route via isErlangAppFile below: their // last-dot extension is too generic for this map.) '.escript': 'erlang', // Spring config: `application.properties` / `application-*.properties`. Same // shape as the `.yml` variants — the YAML/properties extractor emits one node // per leaf key, and the Spring resolver links `@Value("${k}")` references. '.properties': 'properties', // Terraform / OpenTofu / HCL config — tree-sitter-terraform dialect of HCL. '.tf': 'terraform', '.tfvars': 'terraform', '.tofu': 'terraform', }; /** MPEG transport stream: fixed 188-byte packets, each opening with 0x47. */ const MPEG_TS_PACKET_SIZE = 188; const MPEG_TS_SYNC_BYTE = 0x47; /** * Consecutive packets whose sync byte must line up before a file counts as * video — 3 KB of head. A stream shorter than that is cheap to parse anyway; * the cost #1910 is about comes from clips hundreds of KB long. */ const MPEG_TS_MIN_PACKETS = 16; /** * Share of the head that must be control bytes (below 0x20, other than the * whitespace ones) for it to count as binary. Compressed audio and video put * about one byte in eight there; source text puts none. */ const MPEG_TS_MIN_CONTROL_SHARE = 1 / 64; /** * How many bytes of a file's head `isMpegTransportStream` needs — enough to * see `MPEG_TS_MIN_PACKETS` sync bytes plus the packets between them. */ export const MPEG_TS_SNIFF_BYTES = MPEG_TS_PACKET_SIZE * MPEG_TS_MIN_PACKETS; /** * Whether these leading bytes are an MPEG transport stream — the OTHER thing a * `.ts` file can be. Golden video fixtures (`testdata/*.ts`, e2e clips) share * TypeScript's extension, and tree-sitter takes ~28 s to chew through a 900 KB * clip for zero symbols (#1910), so the decision has to be made from the head * of the file, before any parse. * * Two conditions, both required: * 1. the sync byte 0x47 sits at every 188-byte packet boundary of the first * `MPEG_TS_MIN_PACKETS` packets — every packet of a transport stream * opens with it, and nothing else pads to 188; * 2. the head is binary: at least `MPEG_TS_MIN_CONTROL_SHARE` of it is * control bytes, as any compressed payload is. * 0x47 is the letter `G`, so (1) alone could match source whose lines happen * to put a `G` at every 188-byte stride. Checking for a single NUL was not * enough to close that: one NUL in a comment is still TypeScript. (2) asks for * dozens of control bytes, which no source file carries. * * `head` is the first `MPEG_TS_SNIFF_BYTES` (or fewer) bytes of the file. */ export function isMpegTransportStream(head: Uint8Array): boolean { const lastSync = MPEG_TS_PACKET_SIZE * (MPEG_TS_MIN_PACKETS - 1); if (head.length <= lastSync) return false; for (let off = 0; off <= lastSync; off += MPEG_TS_PACKET_SIZE) { if (head[off] !== MPEG_TS_SYNC_BYTE) return false; } let control = 0; for (let i = 0; i < head.length; i++) { const b = head[i]!; // Tab, newline, vertical tab, form feed and carriage return are text. if (b < 0x20 && (b < 0x09 || b > 0x0d)) control++; } return control >= head.length * MPEG_TS_MIN_CONTROL_SHARE; } /** Whether `filePath` carries the one extension MPEG-TS shares with a language. */ export function hasMpegTsExtension(filePath: string): boolean { return filePath.length > 3 && filePath.slice(-3).toLowerCase() === '.ts'; } /** * Whether a file is one CodeGraph can parse, based purely on its extension. * This is the single source of truth for "should we index this file" — derived * from EXTENSION_MAP so parser support and indexing selection never drift. * * `overrides` is the project's validated custom extension → language map (from * `codegraph.json`); when present its extensions count as indexable in addition * to the built-ins. Omitting it is byte-identical to the zero-config behavior. * * `rootDir` is the project root `filePath` is relative to. A Shopify theme's * JSON templates count only when it is given, since telling a theme apart takes * a look at the files beside them (see `isShopifyLiquidJson`). */ export function isSourceFile(filePath: string, overrides?: Record<string, Language>, rootDir?: string): boolean { if (isPlayRoutesFile(filePath)) return true; // Play `conf/routes` is extensionless if (isShopifyLiquidJson(filePath, rootDir)) return true; // Shopify OS 2.0 JSON templates / section groups if (isErlangAppFile(filePath)) return true; // OTP `.app`/`.app.src` resource files const dot = filePath.lastIndexOf('.'); if (dot < 0) return false; const ext = filePath.slice(dot).toLowerCase(); return ext in EXTENSION_MAP || (!!overrides && ext in overrides); } /** * What marks a Shopify theme's root, beside its `templates/` and `sections/`: * the `layout/theme.liquid` every theme must have, or its settings schema. */ const SHOPIFY_THEME_MARKERS = ['layout/theme.liquid', 'config/settings_schema.json']; /** The folders a Shopify theme is made of. */ const SHOPIFY_THEME_FOLDERS = /^(assets|blocks|config|layout|locales|sections|snippets|templates)$/i; /** * The root of the Shopify theme `filePath` is in: the directory holding the * theme folder (one `folders` matches) the file sits in, when that directory * also holds one of `SHOPIFY_THEME_MARKERS`. Project-relative and * `/`-separated, '' for the project root; undefined when the file is in no * theme. `exists` says whether a project-relative path exists. The folder * nearest the file is tried first. */ export function shopifyThemeRoot( filePath: string, exists: (relativePath: string) => boolean, folders: RegExp = SHOPIFY_THEME_FOLDERS, ): string | undefined { const segments = filePath.split('/'); for (let i = segments.length - 2; i >= 0; i--) { if (!folders.test(segments[i]!)) continue; const themeDir = segments.slice(0, i).join('/'); if (SHOPIFY_THEME_MARKERS.some((marker) => exists(themeDir ? `${themeDir}/${marker}` : marker))) return themeDir; } return undefined; } /** * Shopify OS 2.0 JSON template (`templates/*.json`) or section group * (`sections/*.json`) — these reference sections by `"type"`, so the Liquid * extractor links them. (config/ + locales/ JSON have no section refs.) * * Only inside a Shopify theme: the directory holding that `templates/` or * `sections/` must also hold one of `SHOPIFY_THEME_MARKERS`. Plenty of other * projects keep JSON in folders with those names — a .NET project template's * `templates/<name>/.template.config/template.json`, schematics, CMS content — * and it is not Liquid. The theme is looked for under `rootDir`, the project * root `filePath` is relative to; without one there is no theme to find. */ export function isShopifyLiquidJson(filePath: string, rootDir?: string): boolean { // Allow nested template dirs (`templates/customers/login.json`), not just // top-level (`templates/product.json`). if (rootDir === undefined || !/(^|\/)(templates|sections)\/.+\.json$/i.test(filePath)) return false; const exists = (relativePath: string): boolean => fs.existsSync(path.join(rootDir, relativePath)); return shopifyThemeRoot(filePath, exists, /^(templates|sections)$/i) !== undefined; } /** * OTP application resource file: `<app>.app.src` (checked into every rebar3/ * erlang.mk app) or its compiled `<app>.app`. Erlang TERMS, not forms — the * grammar parses them as top-level expressions, and the Erlang extractor's * application-tuple handler turns `{mod, {Mod, _}}` and `{applications, […]}` * into entry-module and dependency edges. Routed by full suffix because the * last-dot extension (`.src`) is far too generic for EXTENSION_MAP. */ export function isErlangAppFile(filePath: string): boolean { return /\.app(?:\.src)?$/i.test(filePath); } /** * Play Framework routes file: the extensionless `conf/routes` (and included * `conf/*.routes`). No grammar — route extraction is done by the Play framework * resolver, so it's processed through the no-grammar (`yaml`-style) path. */ export function isPlayRoutesFile(filePath: string): boolean { return ( filePath === 'conf/routes' || filePath.endsWith('/conf/routes') || filePath.endsWith('.routes') ); } /** * Caches for loaded grammars and parsers */ const parserCache = new Map<Language, Parser>(); const languageCache = new Map<Language, WasmLanguage>(); const unavailableGrammarErrors = new Map<Language, string>(); let parserInitialized = false; /** * Initialize the tree-sitter WASM runtime. Must be called before loading grammars. * Does NOT load any grammar WASM files — use loadGrammarsForLanguages() for that. * Idempotent — safe to call multiple times. */ export async function initGrammars(): Promise<void> { if (parserInitialized) return; await Parser.init(); parserInitialized = true; } /** * Grammars that ship their own vendored WASMs under `dist/extraction/wasm/` * (not in tree-sitter-wasms, or the tree-sitter-wasms build is too old). * Lua: tree-sitter-wasms ships an ABI-13 build that corrupts the shared WASM * heap under web-tree-sitter 0.25 (drops nested calls/imports on every file * after the first); we vendor the upstream ABI-15 wasm instead. C#: the * tree-sitter-wasms build (ABI 13) has no primary-constructor support and * parses `class Foo(...)` - src/extraction/index.ts (202992 bytes)
/** * Extraction Orchestrator * * Coordinates file scanning, parsing, and database storage. */ import * as fs from 'fs'; import * as fsp from 'fs/promises'; import * as path from 'path'; import * as os from 'os'; import * as crypto from 'crypto'; import { execFileSync } from 'child_process'; import { Language, FileRecord, ExtractionResult, ExtractionError, Edge, UnresolvedReference, } from '../types'; import { QueryBuilder, NodeIdentity } from '../db/queries'; import { extractFromSource } from './tree-sitter'; import { ParseWorkerPool, parseWindowSize, resolveParsePoolSize, resolveParseTimeoutMs } from './parse-pool'; import { StoreWriter, StoreBundle, finalizeStoreBundle } from './store-writer'; import { materializeKernelResult } from './kernel'; import { detectGeneratedFile } from './generated-detection'; import { detectLanguage, isSourceFile, isLanguageSupported, isFileLevelOnlyLanguage, initGrammars, loadGrammarsForLanguages, readGrammarWasmBytes, isMpegTransportStream, hasMpegTsExtension, MPEG_TS_SNIFF_BYTES, hasGrammarLoadFailure } from './grammars'; import { loadExtensionOverrides, loadIncludeIgnoredPatterns, loadExcludePatterns, loadIncludePatterns, PROJECT_CONFIG_FILENAME } from '../project-config'; import { isCodeGraphDataDir } from '../directory'; import { logDebug, logWarn } from '../errors'; import { validatePathWithinRoot, normalizePath, isMissingPathError, escapesRoot } from '../utils'; import ignore, { Ignore } from 'ignore'; import { detectFrameworks, getFrameworkResolver } from '../resolution/frameworks'; import { declaredDependencies } from '../resolution/frameworks/package-deps'; import type { ResolutionContext } from '../resolution/types'; import { createYielder, type MaybeYield } from '../resolution/cooperative-yield'; import { resurrectRefFromDroppedEdge } from '../resolution/resurrect-ref'; import { createImportReader, reopenReferencesThroughRemovedFiles } from '../resolution/module-importers'; import { manifestEvidence, type ManifestEvidence } from '../resolution/workspace-packages'; import { MAX_SOURCE_FILE_SIZE_BYTES, oversizeStamp, readBoundedSource, readBoundedSourceSync } from '../file-limits'; export { oversizeStamp }; /** * Number of files to read in parallel during indexing. * File reads are I/O-bound; batching overlaps I/O wait with CPU parse work. */ const FILE_IO_BATCH_SIZE = 10; /** * How many files the `sync()` reconcile processes between cooperative yields to * the event loop. The reconcile runs two O(files) loops of synchronous `fs` * calls (existsSync for removals, statSync for adds/mods); on a very large repo * (~100k files) an un-yielded run wedges the main thread for minutes, which both * trips the liveness watchdog (it SIGKILLs a process whose loop stops turning) * and blocks the first MCP tool call behind the catch-up gate (issue #905). * Yielding every N files keeps the socket, the watchdog heartbeat, and any * concurrent read query responsive while the reconcile runs. */ const SYNC_RECONCILE_YIELD_INTERVAL = 1000; // PARSER_RESET_INTERVAL moved to parse-worker.ts (runs in worker thread) /** * Maximum time (ms) to wait for a single file to parse in the worker thread. * If tree-sitter hangs or WASM runs out of memory, this prevents the entire * indexing run from freezing. The worker is restarted after a (hard) timeout. * Env-overridable via CODEGRAPH_PARSE_TIMEOUT_MS for slow storage (#1231). */ const PARSE_TIMEOUT_MS = resolveParseTimeoutMs(process.env.CODEGRAPH_PARSE_TIMEOUT_MS); /** * Number of files to parse before recycling the worker thread. * WASM linear memory can grow but NEVER shrink (WebAssembly spec limitation). * The only way to reclaim tree-sitter's WASM heap is to destroy the entire * V8 isolate by terminating the worker thread and spawning a fresh one. * This interval balances memory usage against the cost of reloading grammars. */ const WORKER_RECYCLE_INTERVAL = 250; /** * Progress callback for indexing operations */ export interface IndexProgress { phase: 'scanning' | 'parsing' | 'storing' | 'resolving' | 'linking'; current: number; total: number; currentFile?: string; } /** * Result of an indexing operation */ export interface IndexResult { success: boolean; filesIndexed: number; filesSkipped: number; filesErrored: number; /** * How many indexable files the scan discovered — the ground truth the * indexed/skipped/errored tallies must add up to. A shortfall means files * were silently dropped mid-pipeline (e.g. a killed worker under load) and * the index is PARTIAL; callers surface that rather than trusting the * counts. Only set by full-index runs (indexAll), not indexFiles/sync. */ filesDiscovered?: number; /** * Files the scan saw but has no grammar for, tallied by extension. Only the * degenerate case needs it: a project of unsupported files otherwise looks * exactly like an empty one (0 files, state `complete`), so nothing tells the * user — or an agent — that there was code here CodeGraph could not read * (#1502). Counted during the scan's existing walk. */ filesSkippedUnsupported?: number; /** The most common unsupported extensions, biggest first. */ topUnsupportedExtensions?: { ext: string; count: number }[]; nodesCreated: number; edgesCreated: number; errors: ExtractionError[]; /** * Wall time in milliseconds. `CodeGraph.indexAll` reports the whole run — * scanning, parsing and storing, then resolving references and linking — * as it does the node and edge totals; the orchestrator's own result * covers only the files' extraction. */ durationMs: number; } /** The languages whose modules pass bindings on with `export … from` (#2978). */ const RE_EXPORTING_LANGUAGES = new Set<string>(['typescript', 'tsx', 'javascript', 'jsx']); /** * Files a resolution answer's receiver type can be read from (`typeFrom`): * python modules and stubs, and TS/JS modules, whose signatures type the * callbacks other files pass them. */ const TYPE_FROM_FILE = /\.(?:pyi?|[mc]?[jt]sx?)$/; /** * Result of a sync operation */ export interface SyncResult { /** References attempted by the pending-reference recovery sweep, if run. */ pendingRefsProcessed?: number; /** Pending references successfully resolved by the recovery sweep. */ pendingRefsResolved?: number; /** Pending references the recovery sweep could not resolve. */ pendingRefsUnresolved?: number; filesChecked: number; filesAdded: number; filesModified: number; filesRemoved: number; nodesUpdated: number; /** * Wall time in milliseconds. `CodeGraph.sync` reports the whole sync, * resolution and linking included; the orchestrator's own result covers * only reconciling and re-extracting the files. */ durationMs: number; changedFilePaths?: string[]; /** * The part of `changedFilePaths` that had nothing indexed before this sync: * new files, and tracked ones that had no nodes (over the size limit, or a * parse failure) and now do. The latter are still counted as modified. */ addedFilePaths?: string[]; /** Paths not absorbed because reading or extraction failed; retain for status/retry. */ failedFilePaths?: string[]; /** * Symbol names whose set of definitions this sync CHANGED — names the synced * files gained or lost, as the symmetric difference of their `file\0name` * definition pairs before and after the store phase (per file, so a name * moving between two changed files does not cancel itself out). * Resolution picks among all same-named definitions project-wide, * so these are exactly the names whose already-resolved edges — in files this * sync never touched — may now bind elsewhere and must be re-resolved for the * index to stay convergent with a full rebuild (CG-33). * * A body-only edit leaves this empty, which is the common case and costs * nothing downstream. */ definitionDelta?: string[]; /** * Modified ES modules whose `export … from` statements changed: a re-export * that points at another module now (`export { foo } from './a'` → * `'./b'`), one taken away or added, or a clause that passes other names on. * Their importers' bindings through them must be re-resolved: nothing those * files declare changed, and the importers never did (#2978). An edit to a * plain `import` leaves this empty. */ reExportsChangedFilePaths?: string[]; /** Tracked files this sync found gone and deleted from the index. */ removedFilePaths?: string[]; /** * The frameworks extraction detected, when this sync had to detect them * again because what detection read had changed (#2516). Absent when the * cached detection still held. */ detectedFrameworks?: string[]; } /** * Calculate SHA256 hash of file contents */ export function hashContent(content: string): string { return crypto.createHash('sha256').update(content).digest('hex'); } /** * What change detection hashes for a file: its text when it is under the size * limit, the size stamp when it is over — an oversize file is never decoded. * `null` when the bytes show a `.ts` file is an MPEG transport stream, not * TypeScript (#1910): decided from the bytes already read, so it costs no I/O * and runs only for a file that is new or changed — never once per `.ts` file * at discovery, which on a slow disk is a random read per file per scan. */ function readSourceOrStamp(fullPath: string): string | null { const { stats, bytes } = readBoundedSourceSync(fullPath); if (bytes === null) return oversizeStamp(stats.size); return isMpegTsBytes(fullPath, bytes) ? null : bytes.toString('utf8'); } /** * What framework detection found at a path, to tell later whether it changed: * a file's size and modification time, a folder's sub-folders, or '-' when * nothing is there. */ function diskSignature(fullPath: string, isFolder = false): string { try { if (isFolder) { return fs.readdirSync(fullPath, { withFileTypes: true }) .filter((entry) => entry.isDirectory()) .map((entry) => entry.name) .sort() .join('/'); } const stats = fs.statSync(fullPath, { throwIfNoEntry: false }); return stats ? `${stats.size}:${stats.mtimeMs}` : '-'; } catch { return '-'; } } /** Whether these bytes, read from `filePath`, are a `.ts` MPEG transport stream (#1910). */ function isMpegTsBytes(filePath: string, bytes: Buffer): boolean { if (!hasMpegTsExtension(filePath) || !isMpegTransportStream(bytes.subarray(0, MPEG_TS_SNIFF_BYTES))) return false; logDebug('Skipping MPEG transport stream named .ts — not TypeScript', { filePath }); return true; } /** * Directory names that are dependency, build, cache, or tooling output across the * languages/frameworks CodeGraph supports — curated from the canonical * github/gitignore templates. Excluded by default so the graph reflects your code, * not third-party noise, without requiring a `.gitignore` (issue #407). The * exclusion applies uniformly (git or not, tracked or not); the only opt-in is an * explicit `.gitignore` negation (e.g. `!vendor/`). First-party-prone or generic * names (`packages`, `lib`, `app`, `bin`, `src`, `deps`, `env`, `tmp`, `storage`, * `Library`) are deliberately NOT listed, to avoid ever hiding real source. * * Only dirs that actually contain *indexable source* (or are enormous) earn a slot * — IDE/state dirs like `.idea`/`.vs` are omitted because CodeGraph indexes only * recognized source extensions, so they produce no symbols regardless. */ const DEFAULT_IGNORE_DIRS: ReadonlySet<string> = new Set([ // JS / TS — dependency directories 'node_modules', 'bower_components', 'jspm_packages', 'web_modules', '.yarn', '.pnpm-store', // JS / TS — framework & bundler build / cache / deploy output '.next', '.nuxt', '.svelte-kit', '.turbo', '.vite', '.parcel-cache', '.angular', '.docusaurus', 'storybook-static', '.vinxi', '.nitro', 'out-tsc', '.vercel', '.netlify', '.wrangler', // Build output (common across ecosystems) 'dist', 'build', 'out', '.output', // Test / coverage 'coverage', '.nyc_output', // Python '__pycache__', '__pypackages__', '.venv', 'venv', '.pixi', '.pdm-build', '.mypy_cache', '.pytest_cache', '.ruff_cache', '.tox', '.nox', '.hypothesis', '.ipynb_checkpoints', '.eggs', // Rust / JVM (Maven, Gradle, Scala) 'target', '.gradle', // .NET 'obj', // Vendored deps (Go, PHP/Composer, Ruby/Bundler) 'vendor', // Swift / iOS '.build', 'Pods', 'Carthage', 'DerivedData', '.swiftpm', // Dart / Flutter '.dart_tool', '.pub-cache', // Native (Android NDK, C/C++ deps) '.cxx', '.externalNativeBuild', 'vcpkg_installed', // Scala tooling '.bloop', '.metals', // Lua / Luau (LuaRocks) 'lua_modules', '.luarocks', // Delphi / RAD Studio IDE backups (duplicate .pas source — would double-count) '__history', '__recovery', // Generic cache '.cache', ]); /** * Android resource directory types. A `res/` tree holds ONLY non-code resources — * layouts, drawables, value bags (strings/colors/styles), menus, navigation * graphs — split into one typed subdirectory per kind, optionally density/locale/ * version-qualified (`values-es`, `drawable-hdpi`, `layout-v21`, …). None of it * yields an extractable code symbol, yet on an Android app it DOMINATES the tree * (one report: 26k XML files = 97% of the project, 0 symbols), bloating the DB, * slowing indexing, and skewing both the file count and `codegraph_explore` * results (#1047). So these are excluded by default. The structure is * self-identifying — a non-Android project has no `res/layout/` etc., so it's * untouched — and the only XML that DOES produce symbols (MyBatis mappers) lives * under `src/main/resources/`, never `res/`, so nothing useful is dropped. * `res/raw/` is deliberately NOT here: it holds arbitrary bundled assets that can * be code-ish (a `.sql` schema, a `.js`), so we leave it indexed. Override any of * these with a `.gitignore` negation (e.g. `!res/values/`). */ const ANDROID_RES_TYPES: readonly string[] = [ 'anim', 'animator', 'color', 'drawable', 'font', 'layout', 'menu', 'mipmap', 'navigation', 'transition', 'values', 'xml', ]; /** * `build` is also a legal JVM package segment: directories named `build` under * a conventional source root (any source set: main, test, androidTest, ...) are * packages, not module/build output (#1642). */ const JVM_BUILD_PACKAGE_DIRS: readonly string[] = [ '**/src/*/java/**/build/', '**/src/*/kotlin/**/build/', '**/src/*/scala/**/build/', ]; /** Gitignore-style patterns for the `ignore` matcher: the dirs above plus a few globs. */ const DEFAULT_IGNORE_PATTERNS: string[] = [ ...Array.from(DEFAULT_IGNORE_DIRS, (d) => `${d}/`), '*.egg-info/', // Python packaging metadata 'cmake-build-*/', // CLion / CMake build trees 'bazel-*/', // Bazel output symlink trees // Android res - src/extraction/kernel/decode.ts (7369 bytes)
/** * Decode the kernel's flat buffers into an ExtractionResult — the single * JS-side pass over the per-file tables. See layout.ts for the byte layout * and codegraph-kernel/src/buffers.rs for the writer. */ import type { Edge, EdgeKind, ExtractionError, ExtractionResult, Language, Node, NodeKind, ReferenceKind, UnresolvedReference, } from '../../types'; import { NODE_KINDS, EDGE_KINDS } from '../../types'; import type { KernelBuffers } from './loader'; import { EDGE, EDGE_ROW_SIZE, FLAG, FUNCTION_REF_CODE, KERNEL_ABI_VERSION, META, META_SIZE, NODE, NODE_ROW_SIZE, NONE, PROVENANCES, REF, REF_FLAG_FILE_PATH, REF_ROW_SIZE, VISIBILITIES, } from './layout'; /** Read an (offset, len) arena string; undefined when absent. */ function str(arena: Buffer, row: Buffer, at: number): string | undefined { const off = row.readUInt32LE(at); if (off === NONE) return undefined; const len = row.readUInt32LE(at + 4); return arena.toString('utf8', off, off + len); } /** NUL-joined list field; undefined when absent. */ function strList(arena: Buffer, row: Buffer, at: number): string[] | undefined { const joined = str(arena, row, at); return joined === undefined ? undefined : joined.split('\0'); } /** Tri-state boolean from a (present, value) bit pair. */ function flag(flags: number, pair: number): boolean | undefined { if ((flags & (1 << (pair * 2))) === 0) return undefined; return (flags & (1 << (pair * 2 + 1))) !== 0; } function u32opt(row: Buffer, at: number): number | undefined { const v = row.readUInt32LE(at); return v === NONE ? undefined : v; } export function decodeExtractBuffers( buffers: KernelBuffers, filePath: string, language: Language ): ExtractionResult { const { meta, arena } = buffers; if (meta.length < META_SIZE) throw new Error(`kernel meta too short: ${meta.length}`); const version = meta.readUInt8(META.version); if (version !== KERNEL_ABI_VERSION) { throw new Error(`kernel buffer ABI ${version} != expected ${KERNEL_ABI_VERSION}`); } const nodeCount = meta.readUInt32LE(META.nodeCount); const edgeCount = meta.readUInt32LE(META.edgeCount); const refCount = meta.readUInt32LE(META.refCount); const now = Date.now(); const nodes: Node[] = new Array(nodeCount); // Node-table row index → node id, for edge/ref endpoint resolution. const idByRow: string[] = new Array(nodeCount); for (let i = 0; i < nodeCount; i++) { const row = buffers.nodes.subarray(i * NODE_ROW_SIZE, (i + 1) * NODE_ROW_SIZE); const id = str(arena, row, NODE.id)!; idByRow[i] = id; const flags = row.readUInt16LE(NODE.flags); const node: Node = { id, kind: NODE_KINDS[row.readUInt8(NODE.kind)] as NodeKind, name: str(arena, row, NODE.name)!, qualifiedName: str(arena, row, NODE.qualifiedName)!, filePath, language, startLine: row.readUInt32LE(NODE.startLine), endLine: row.readUInt32LE(NODE.endLine), startColumn: row.readUInt32LE(NODE.startColumn), endColumn: row.readUInt32LE(NODE.endColumn), updatedAt: now, }; const docstring = str(arena, row, NODE.docstring); if (docstring !== undefined) node.docstring = docstring; const signature = str(arena, row, NODE.signature); if (signature !== undefined) node.signature = signature; const visibility = VISIBILITIES[row.readUInt8(NODE.visibility)]; if (visibility !== undefined) node.visibility = visibility; const isExported = flag(flags, FLAG.isExported); if (isExported !== undefined) node.isExported = isExported; const isAsync = flag(flags, FLAG.isAsync); if (isAsync !== undefined) node.isAsync = isAsync; const isStatic = flag(flags, FLAG.isStatic); if (isStatic !== undefined) node.isStatic = isStatic; const isAbstract = flag(flags, FLAG.isAbstract); if (isAbstract !== undefined) node.isAbstract = isAbstract; const decorators = strList(arena, row, NODE.decorators); if (decorators !== undefined) node.decorators = decorators; const typeParameters = strList(arena, row, NODE.typeParameters); if (typeParameters !== undefined) node.typeParameters = typeParameters; const returnType = str(arena, row, NODE.returnType); if (returnType !== undefined) node.returnType = returnType; const extraJson = str(arena, row, NODE.extraJson); if (extraJson !== undefined) Object.assign(node, JSON.parse(extraJson) as Partial<Node>); nodes[i] = node; } const edges: Edge[] = new Array(edgeCount); for (let i = 0; i < edgeCount; i++) { const row = buffers.edges.subarray(i * EDGE_ROW_SIZE, (i + 1) * EDGE_ROW_SIZE); const sourceIdx = row.readUInt32LE(EDGE.sourceIdx); const targetIdx = row.readUInt32LE(EDGE.targetIdx); const edge: Edge = { source: sourceIdx === NONE ? str(arena, row, EDGE.sourceIdStr)! : idByRow[sourceIdx]!, target: targetIdx === NONE ? str(arena, row, EDGE.targetIdStr)! : idByRow[targetIdx]!, kind: EDGE_KINDS[row.readUInt8(EDGE.kind)] as EdgeKind, }; const line = u32opt(row, EDGE.line); if (line !== undefined) edge.line = line; const column = u32opt(row, EDGE.column); if (column !== undefined) edge.column = column; const provenance = PROVENANCES[row.readUInt8(EDGE.provenance)]; if (provenance !== undefined) edge.provenance = provenance; const metadataJson = str(arena, row, EDGE.metadataJson); if (metadataJson !== undefined) edge.metadata = JSON.parse(metadataJson) as Record<string, unknown>; edges[i] = edge; } const unresolvedReferences: UnresolvedReference[] = new Array(refCount); for (let i = 0; i < refCount; i++) { const row = buffers.refs.subarray(i * REF_ROW_SIZE, (i + 1) * REF_ROW_SIZE); const fromIdx = row.readUInt32LE(REF.fromIdx); const kindByte = row.readUInt8(REF.kind); // No filePath/language on ordinary refs: the wasm extractors emit them // WITHOUT the denormalized fields (the store fills `ref.filePath ?? // filePath`), and the kernel must match the extractFromSource seam // exactly. The ONE exception is flagged (REF_FLAG_FILE_PATH): the // ruby/php visitNode hooks set `filePath: ctx.filePath` on their // mixin/trait `implements` refs — re-attach the decode call's own // filePath, which is that exact value. const ref: UnresolvedReference = { fromNodeId: fromIdx === NONE ? str(arena, row, REF.fromIdStr)! : idByRow[fromIdx]!, referenceName: str(arena, row, REF.referenceName)!, referenceKind: kindByte === FUNCTION_REF_CODE ? 'function_ref' : (EDGE_KINDS[kindByte] as ReferenceKind), line: row.readUInt32LE(REF.line), column: row.readUInt32LE(REF.column), }; if ((row.readUInt8(REF.flags) & REF_FLAG_FILE_PATH) !== 0) ref.filePath = filePath; const candidates = strList(arena, row, REF.candidates); if (candidates !== undefined) ref.candidates = candidates; unresolvedReferences[i] = ref; } let errors: ExtractionError[] = []; const errorsOff = meta.readUInt32LE(META.errorsOff); if (errorsOff !== NONE) { const errorsLen = meta.readUInt32LE(META.errorsLen); errors = JSON.parse(arena.toString('utf8', errorsOff, errorsOff + errorsLen)) as ExtractionError[]; } return { nodes, edges, unresolvedReferences, errors, durationMs: 0 }; } - src/extraction/kernel/index.ts (17679 bytes)
/** * Kernel routing — which languages go through the native kernel, and the * single entry point the extraction path calls. * * Routing policy is deliberately TS-side and per-language (migration plan §2): * a language routes to the kernel only after its equivalence gate passes; * everything else stays on the wasm path forever if need be. Rollback per * language = removing it from DEFAULT_ROUTED (or CODEGRAPH_KERNEL=0 for all). * * Routing status: TypeScript/TSX/JavaScript/JSX are default-routed (R3 gate * passed 2026-07-16 — full-index dumps byte-identical on express/excalidraw/ * vscode, control repo unchanged; see the migration plan §4a). Override with * CODEGRAPH_KERNEL_LANGS=<langs|all> (replaces the default set), or * CODEGRAPH_KERNEL=0 (kill switch, everything → wasm). */ import type { ExtractionResult, Language } from '../../types'; import { commonJsRequireRefs } from '../commonjs-requires'; import { indexParseBudgetMs, parseTimeoutResult } from '../parse-budget'; import { EXTRACTORS } from '../languages'; import { getKernel, kernelSupports } from './loader'; import { decodeExtractBuffers } from './decode'; import { KERNEL_ABI_VERSION as LAYOUT_ABI, META as LAYOUT_META, NONE as LAYOUT_NONE, } from './layout'; export { getKernel, kernelSupports, resetKernelForTests } from './loader'; export { decodeExtractBuffers } from './decode'; /** * Languages routed to the kernel by default (gate-passed only — see the * per-language tracker in docs/design/rust-kernel-migration-plan.md §4). * Per-file safety valve regardless of routing: a file whose parse tree * contains ERRORS defers to the wasm extractor (error recovery differs * between UTF-8 and UTF-16 parsing — wasm's recovery is canonical). */ const DEFAULT_ROUTED: ReadonlySet<Language> = new Set<Language>([ 'typescript', 'tsx', 'javascript', 'jsx', 'java', 'python', 'go', // R7a (2026-07-17): parity swept 0-diff on redis/git/fmt/protobuf/ALS // (2,389 files compared) + full-init dump-diffs byte-identical; erroring // files defer per-file to wasm (routine for macro-heavy C/C++ — see // scripts/kernel-parity.mjs --max-deferral). 'c', 'cpp', // R7b (2026-07-20): parity swept 0-diff on ripgrep/tokio/rust-analyzer // (2,108 files byte-parity) + full-init dump-diffs byte-identical. Rust // deferral is ~0% on normal repos; token-macro-table sources (rust-analyzer's // parser crates, 18%) error on BOTH arms — grammar-inherent, not a walker // signal. 'rust', // R7b (2026-07-20): parity swept 0-diff on serilog/Newtonsoft.Json/jellyfin // (3,229 files byte-parity) + full-init dump-diffs byte-identical ×3. // Deferral 0.05–3.3% — both-branches-kept `#if` damage that errors on BOTH // arms (the preParse blanking hoist keeps the kernel's input identical). 'csharp', // R7b (2026-07-20): parity swept 0-diff on sinatra/jekyll/rails (3,763 // files byte-parity, 0 deferrals — ruby error incidence is 0.00%) + // full-init dump-diffs byte-identical ×3. Any deferral on a ruby sweep is // a walker-bug signal, not grammar reality. 'ruby', // R7b (2026-07-20): parity swept 0-diff on monolog/laravel-framework/ // symfony (13,950 files byte-parity) + full-init dump-diffs byte-identical // ×3. Deferral ≈0–0.1% (genuinely-broken fixtures) — default sweep guard. 'php', // R7b (2026-07-20): parity swept 0-diff on Alamofire/vapor/swift-nio (720 // clean files byte-parity; Alamofire's 348 #1020 property nodes reproduced // exactly) + full-init dump-diffs byte-identical ×3. Swift error incidence // is structurally 9–27% on BOTH arms (heavy #if conditionalization) — // sweeps run --max-deferral 0.3; a deferral-rate JUMP is the bug signal. 'swift', // R7b (2026-07-20): parity swept 0-diff on okio/okhttp/kotlinx.coroutines // (1,861 clean files byte-parity; KMP expect/actual synthesis identical — // 412 edges both arms on kotlinx.coroutines) + full-init dump-diffs // byte-identical ×3. Deferral 4.7–8.5% both-arm grammar reality // (fun-interface misparses + PHANTOM hasError files) — default sweep guard // holds; a JUMP past ~10% is the bug signal. 'kotlin', // R7b batch 4 (2026-07-20): parity swept 0-diff on dplyr/ggplot2/shiny + // AnomalyDetection + full-init dump-diffs byte-identical ×3. R error // incidence is ~0% (deferrals 0/0/0/1 across the gate repos — the 1 is a // moustache-template pseudo-R file, both-arm): any deferral on an R sweep // is a walker-bug signal, not grammar reality. 'r', // R7b batch 4 (2026-07-20): one walker, two dialects. Parity swept 0-diff // on kong/lazy.nvim/lua-resty-core (lua) + lune/Fusion (luau) + full-init // dump-diffs byte-identical. Lua error incidence ~0% (any deferral is a // walker-bug signal); luau 1.4–7.1% both-arm grammar reality (generic type // packs, default type params) — a JUMP past ~10% is the bug signal. 'lua', 'luau', // R7b batch 4 (2026-07-20): parity swept 0-diff on os-lib/cats + scala3 // compiler/src + library/src + full-init dump-diffs byte-identical. // Error incidence is bimodal: mainstream Scala-2-style repos ~0-2% // (os-lib 0%, cats 1.8%) but scala-3-frontier code runs 10-18% with // PHANTOM-dominated error sets (hasError=true, zero ERROR nodes — // capture-checking `^` types; defer on the FLAG). Sweeps over // scala3-style repos use --max-deferral 0.3 (swift precedent); a // deferral JUMP on cats/os-lib-style code is the bug signal. 'scala', // R7b batch 4 (2026-07-20): parity swept 0-diff on shelf/bloc/flutter + // full-init dump-diffs byte-identical. Error incidence 3.4-20.7% both-arm // grammar reality (empty object patterns — the sealed-class idiom — and // unnamed `library;` dominate; flutter sits at ~21%) — sweeps use // --max-deferral 0.3; the RATE, not the count, is the signal (flutter // HEAD drifts). 'dart', ]); /** * Per-language TS post-pass over the decoded result — the escape hatch for * logic `.scm` queries can't express (macro salvage, dialect sniffing, * wrapper-based component recognition). Runs synchronously after decode, * before the framework extract() hooks the caller applies. Keep these SMALL: * anything heavy belongs in the Rust emitter. */ export type KernelPostPass = (result: ExtractionResult, source: string) => void; const POST_PASSES: Partial<Record<Language, KernelPostPass>> = { // (none yet — R2+) }; /** * The preParse hoist (checklist §arch-1): languages with an offset-preserving * `preParse` hook (c/cpp macro blanking, csharp #237, metal #1121, cuda #1172) * apply it HERE, before the kernel call, so both arms parse identical blanked * bytes and none of the blanking logic needs a Rust port. The wasm fallback * path is untouched — TreeSitterExtractor applies the same hook itself on the * RAW source it receives, so a kernel error/defer still extracts identically. * Every blank is an equal-length-space replacement, so offsets, lines, and * columns survive; `filePath` rides along for the extension-gated dialect * blanks (`.metal` attributes; `.cu`/`.cuh` + content-gated CUDA). */ function preParsedSource(filePath: string, source: string, language: Language): string { const pre = EXTRACTORS[language]?.preParse; return pre ? pre(source, filePath) : source; } /** * Whether the file's re-exports keep their statements as written while the * kernel reads the preParsed bytes (`reExportAsWritten`: TypeScript's * `export type *`, #2474). The decoded result then takes those import nodes' * signatures from `source`, as TreeSitterExtractor does, and the undecoded * fast path steps aside for it. */ function keepsReExportsAsWritten(source: string, pre: string, language: Language): boolean { return pre !== source && EXTRACTORS[language]?.reExportAsWritten?.(source) === true; } /** Each re-export import node's signature, read from `source` by the node's span. */ function restoreReExportSignatures(result: ExtractionResult, source: string): void { let lineStarts: number[] | null = null; for (const node of result.nodes) { if (node.kind !== 'import' || !node.signature?.startsWith('export')) continue; if (!lineStarts) { lineStarts = [0]; for (let i = source.indexOf('\n'); i !== -1; i = source.indexOf('\n', i + 1)) lineStarts.push(i + 1); } const start = lineStarts[node.startLine - 1]! + node.startColumn; const end = lineStarts[node.endLine - 1]! + node.endColumn; node.signature = source.slice(start, end).trim(); } } function isRouted(language: Language): boolean { const env = process.env.CODEGRAPH_KERNEL_LANGS; if (env === undefined || env === '') return DEFAULT_ROUTED.has(language); if (env === 'all') return true; return env .split(',') .map((s) => s.trim()) .includes(language); } /** True when `language` would be extracted by the kernel right now. */ export function kernelRoutes(language: Language): boolean { return isRouted(language) && kernelSupports(language); } /** Warned-once registry so a broken language logs a single line, not one per file. */ const warned = new Set<string>(); /** * One-slot defer memo. A file the kernel defers (parse errors → wasm) used to * pay the full pipeline again at every seam: the worker's raw try blanked + * native-parsed it, extractFromSource's kernel try blanked + native-parsed it * AGAIN, and the wasm extractor then re-applied preParse a third time. On a * high-deferral tree (the Linux kernel defers ~79% of files) that waste * dominated the arm's parse phase. The slot remembers the LAST deferred * (file, source, language) so (a) a repeat kernel attempt for the same file * short-circuits to null, and (b) the wasm fallback can reuse the * already-blanked source instead of re-running preParse. Source is matched by * string identity — the worker passes the same string through every seam. */ let deferSlot: { filePath: string; source: string; language: Language; pre: string } | null = null; /** * One-slot timeout memo, beside deferSlot: the LAST file whose native parse * ran past its budget (`timeout:`, #3274). Its decoded-path retry answers * from here instead of parsing again, and no file that times out takes the * wasm path, which runs the same tree-sitter and would only spend the budget * a second time. */ let timeoutSlot: { filePath: string; source: string; language: Language; budgetMs: number } | null = null; /** The budget a just-timed-out file ran past, if it matches. */ function takeTimedOut(filePath: string, source: string, language: Language): number | null { return timeoutSlot && timeoutSlot.filePath === filePath && timeoutSlot.source === source && timeoutSlot.language === language ? timeoutSlot.budgetMs : null; } /** The hoisted preParse output for a just-deferred file, if it matches. */ export function takeDeferredPreParse( filePath: string, source: string, language: Language ): string | null { if ( deferSlot && deferSlot.filePath === filePath && deferSlot.source === source && deferSlot.language === language ) { return deferSlot.pre; } return null; } /** The raw table buffers + the cheap facts the orchestrator needs pre-decode. */ export interface KernelRawResult { buffers: NonNullable<ExtractionResult['kernelBuffers']>; counts: { nodes: number; edges: number; refs: number }; errors: ExtractionResult['errors']; } /** * Extract via the kernel WITHOUT decoding — the bulk-index fast path. The * tables ride to the store boundary as buffers (decoded on the store worker), * so the main thread never materializes per-node objects. Returns null under * exactly the conditions tryKernelExtract does, PLUS when the language has a * registered post() pass (post passes operate on decoded results, so those * languages keep the decoded path), and for a file whose re-exports keep their * statements as written (keepsReExportsAsWritten), which the decoded path * restores. */ export function tryKernelExtractRaw( filePath: string, source: string, language: Language ): KernelRawResult | null { if (!kernelRoutes(language) || POST_PASSES[language]) return null; const kernel = getKernel(); if (!kernel) return null; if (takeDeferredPreParse(filePath, source, language) !== null) return null; // already deferred if (takeTimedOut(filePath, source, language) !== null) return null; // already timed out const pre = preParsedSource(filePath, source, language); // Its signatures come from the source as written, which needs the decoded path. if (keepsReExportsAsWritten(source, pre, language)) return null; const budgetMs = indexParseBudgetMs(pre.length); try { const buffers = kernel.extractFile(filePath, pre, language, budgetMs); const meta = buffers.meta; if (meta.readUInt8(LAYOUT_META.version) !== LAYOUT_ABI) { throw new Error(`kernel buffer ABI ${meta.readUInt8(0)} != expected ${LAYOUT_ABI}`); } const counts = { nodes: meta.readUInt32LE(LAYOUT_META.nodeCount), edges: meta.readUInt32LE(LAYOUT_META.edgeCount), refs: meta.readUInt32LE(LAYOUT_META.refCount), }; let errors: ExtractionResult['errors'] = []; const errorsOff = meta.readUInt32LE(LAYOUT_META.errorsOff); if (errorsOff !== LAYOUT_NONE) { const errorsLen = meta.readUInt32LE(LAYOUT_META.errorsLen); errors = JSON.parse( buffers.arena.toString('utf8', errorsOff, errorsOff + errorsLen) ) as ExtractionResult['errors']; } return { buffers, counts, errors }; } catch (err) { const message = err instanceof Error ? err.message : String(err); if (message.includes('defer:')) { deferSlot = { filePath, source, language, pre }; return null; } // The decoded path (extractFromSource) records the timeout from the slot. if (message.startsWith('timeout:')) { timeoutSlot = { filePath, source, language, budgetMs }; return null; } if (!warned.has(language)) { warned.add(language); process.stderr.write( `[codegraph-kernel] ${language} extraction failed (${message}) — falling back to the wasm path\n` ); } return null; } } /** * Decode a buffer-carrying result (see ExtractionResult.kernelBuffers) into a * plain, fully-materialized ExtractionResult — the fallback for store paths * that need objects (main-thread store, tests). */ export function materializeKernelResult( result: ExtractionResult, filePath: string, language: Language ): ExtractionResult { if (!result.kernelBuffers) return result; const b = result.kernelBuffers; const asBuf = (u: Uint8Array) => Buffer.from(u.buffer, u.byteOffset, u.byteLength); const decoded = decodeExtractBuffers( { meta: asBuf(b.meta), nodes: asBuf(b.nodes), edges: asBuf(b.edges), refs: asBuf(b.refs), arena: asBuf(b.arena) }, filePath, language ); decoded.durationMs = result.durationMs; // References read beside the tables (a CommonJS `re - src/extraction/kernel/layout.ts (3899 bytes)
/** * Native-kernel buffer layout — TS mirror of codegraph-kernel/src/buffers.rs. * * The kernel returns five Buffers per file: meta, nodes, edges, refs, arena. * Rows are fixed-width little-endian; strings are (offset, len) pairs into * the UTF-8 arena; `offset === NONE` means "field absent". * * THIS FILE AND buffers.rs MUST MATCH BYTE FOR BYTE. Any layout change bumps * KERNEL_ABI_VERSION on both sides — the loader refuses a version it doesn't * know and the extraction path falls back to wasm. * * NodeKind / EdgeKind / provenance / visibility cross the boundary as indexes * into NODE_KINDS / EDGE_KINDS (src/types.ts) and the small tables below, so * those array orders are part of the contract (append, never reorder). The * loader additionally verifies the kernel's own kind tables against * NODE_KINDS/EDGE_KINDS at load time, so a stale .node degrades to the wasm * path instead of mis-decoding. */ export const KERNEL_ABI_VERSION = 2; /** Sentinel for "absent" in u32 slots and string-ref offsets. */ export const NONE = 0xffffffff; export const META_SIZE = 36; export const NODE_ROW_SIZE = 96; export const EDGE_ROW_SIZE = 44; export const REF_ROW_SIZE = 40; /** meta byte offsets */ export const META = { version: 0, // u8 nodeCount: 4, // u32 edgeCount: 8, // u32 refCount: 12, // u32 arenaLen: 16, // u32 errorsOff: 20, // u32 (NONE = no errors) errorsLen: 24, // u32 durationMs: 28, // f64 (kernel-side wall; introspection only) } as const; /** node row byte offsets */ export const NODE = { kind: 0, // u8 — NODE_KINDS index visibility: 1, // u8 — VISIBILITIES index (0 = absent) flags: 2, // u16 — (present, value) bit pairs, see FLAG startLine: 4, // u32 endLine: 8, // u32 startColumn: 12, // u32 endColumn: 16, // u32 name: 20, // str qualifiedName: 28, // str id: 36, // str — kernel-computed node id docstring: 44, // str signature: 52, // str decorators: 60, // str — NUL-joined list typeParameters: 68, // str — NUL-joined list returnType: 76, // str extraJson: 84, // str — JSON of any extra Node props (escape hatch) metrics: 92, // u32 — reserved (Arc 3.2 per-node code metrics) } as const; /** edge row byte offsets */ export const EDGE = { sourceIdx: 0, // u32 (NONE → sourceIdStr) targetIdx: 4, // u32 (NONE → targetIdStr) kind: 8, // u8 — EDGE_KINDS index provenance: 9, // u8 — PROVENANCES index (0 = absent) line: 12, // u32 (NONE = absent) column: 16, // u32 (NONE = absent) metadataJson: 20, // str sourceIdStr: 28, // str targetIdStr: 36, // str } as const; /** ref row byte offsets */ export const REF = { fromIdx: 0, // u32 (NONE → fromIdStr) kind: 4, // u8 — EDGE_KINDS index, or FUNCTION_REF_CODE flags: 5, // u8 — REF_FLAGS bits (v2) line: 8, // u32 column: 12, // u32 referenceName: 16, // str candidates: 24, // str — NUL-joined list fromIdStr: 32, // str } as const; /** ReferenceKind wire code for the internal-only `function_ref` (#756). */ export const FUNCTION_REF_CODE = 200; /** * Ref-row flag bits (v2). FILE_PATH: the ref carries `filePath` = the * extracted file — the ruby/php visitNode hooks set `filePath: ctx.filePath` * on their mixin/trait `implements` refs (unlike every other extraction ref, * which the store denormalizes); decode re-attaches its own filePath * parameter, which is byte-identical. */ export const REF_FLAG_FILE_PATH = 1; /** Node bool-flag bit pairs: bit(2n) = present, bit(2n+1) = value. */ export const FLAG = { isExported: 0, isAsync: 1, isStatic: 2, isAbstract: 3, } as const; /** visibility byte values (0 = absent). */ export const VISIBILITIES = [undefined, 'public', 'private', 'protected', 'internal'] as const; /** provenance byte values (0 = absent). */ export const PROVENANCES = [undefined, 'tree-sitter', 'scip', 'heuristic'] as const; - src/extraction/kernel/loader.ts (6917 bytes)
/** * Native-kernel loader — finds, loads, and contract-verifies the * codegraph-kernel .node addon. * * The kernel is OPTIONAL everywhere. Every failure mode here (no binary for * this platform, dlopen error, ABI/kind-table mismatch) resolves to `null` * and the extraction path silently keeps using the wasm pipeline — a missing * or stale kernel must never break indexing, only skip the speedup. Set * CODEGRAPH_KERNEL_DEBUG=1 to see why a kernel didn't load. * * Kill switch: CODEGRAPH_KERNEL=0 disables the kernel entirely (checked per * call so tests and embedders can flip it at runtime). * * Search order: * 1. CODEGRAPH_KERNEL_PATH — explicit .node path (dev/testing override) * 2. <up3>/kernel/codegraph-kernel.node — the release bundle layout * (lib/dist/** next to lib/kernel/; see scripts/build-bundle.sh) * 3. <up3>/codegraph-kernel/prebuilds/<platform>-<arch>/codegraph-kernel.node * — from-source runs and tests (staged by scripts/build-kernel.sh) * * "up3" = three directories above this file, which is the package root both * from src/extraction/kernel/ and from dist/extraction/kernel/. */ import * as fs from 'fs'; import * as path from 'path'; import { createRequire } from 'module'; import { NODE_KINDS, EDGE_KINDS } from '../../types'; import { KERNEL_ABI_VERSION } from './layout'; /** Raw buffer tables for one file — see layout.ts for the byte layout. */ export interface KernelBuffers { meta: Buffer; nodes: Buffer; edges: Buffer; refs: Buffer; arena: Buffer; } export interface KernelContractInfo { abiVersion: number; kernelVersion: string; nodeKinds: string[]; edgeKinds: string[]; languages: string[]; } export interface KernelGrammarInfo { abiVersion: number; nodeKindCount: number; fieldCount: number; nodeKinds: string[]; fieldNames: string[]; } /** Input to the cFnPtr extraction sweep: one file's raw text + its struct * node extents (`endLine ?? startLine` applied by the caller). */ export interface CfnptrFileIn { text: string; structs: { id: string; startLine: number; endLine: number }[]; } /** Per-file facts from the native cFnPtr extraction sweep — mirror of the * Rust `CfnptrFacts` (see codegraph-kernel/src/cfnptr.rs); semantics match * the JS sweep in src/resolution/c-fnptr-synthesizer.ts. */ export interface CfnptrFactsOut { fnPtrTypedefs: string[]; fnTypeTypedefs: string[]; structs: { id: string; parsed: boolean; fields: { name: string; index: number; ptr: boolean; type: string }[] }[]; inlinePtr: boolean; inlineTypes: string[]; inlineTags: string[]; initTokens: string[]; arrayElems: string[]; aliasNames: string[]; dPairs: string[]; dispatchFields: string[]; arrayDispatchNames: string[]; includes: string[]; } export interface KernelModule { /** * `parseBudgetMs`: how long the native parse may run before it is cancelled, * with a `timeout:` error (#3274). An older binary ignores it. */ extractFile(filePath: string, content: string, language: string, parseBudgetMs?: number): KernelBuffers; contractInfo(): KernelContractInfo; grammarInfo(language: string): KernelGrammarInfo | null; /** Batched cFnPtr extraction sweep (task #5 step 2). OPTIONAL: absent on * older binaries — callers feature-detect and keep their JS path. */ cfnptrScanFiles?(files: CfnptrFileIn[]): CfnptrFactsOut[]; /** Native `stripCommentsForRegex(text, 'c')` — differential-oracle hook. */ cfnptrStripC?(text: string): string; } const debugEnabled = () => process.env.CODEGRAPH_KERNEL_DEBUG === '1'; function debug(msg: string): void { if (debugEnabled()) process.stderr.write(`[codegraph-kernel] ${msg}\n`); } /** Languages the loaded binary supports (contract-verified). Empty when no kernel. */ let kernelLanguages: ReadonlySet<string> = new Set(); /** undefined = not attempted yet; null = attempted and unavailable. */ let cached: KernelModule | null | undefined; function candidatePaths(): string[] { const candidates: string[] = []; if (process.env.CODEGRAPH_KERNEL_PATH) candidates.push(process.env.CODEGRAPH_KERNEL_PATH); const packageRoot = path.resolve(__dirname, '..', '..', '..'); candidates.push(path.join(packageRoot, 'kernel', 'codegraph-kernel.node')); candidates.push( path.join( packageRoot, 'codegraph-kernel', 'prebuilds', `${process.platform}-${process.arch}`, 'codegraph-kernel.node' ) ); return candidates; } /** * Verify the binary speaks our wire contract: same ABI version and byte-equal * NodeKind/EdgeKind tables (kinds cross the boundary as indexes into these). */ function verifyContract(mod: KernelModule, from: string): boolean { const info = mod.contractInfo(); if (info.abiVersion !== KERNEL_ABI_VERSION) { debug(`${from}: ABI ${info.abiVersion} != expected ${KERNEL_ABI_VERSION} — ignoring kernel`); return false; } const sameTable = (a: readonly string[], b: readonly string[]) => a.length === b.length && a.every((v, i) => v === b[i]); if (!sameTable(info.nodeKinds, NODE_KINDS) || !sameTable(info.edgeKinds, EDGE_KINDS)) { debug(`${from}: NodeKind/EdgeKind tables differ from src/types.ts — ignoring kernel`); return false; } return true; } /** * Load (once per process) and return the kernel module, or null when * unavailable. The kill switch is NOT checked here — callers route through * `kernelAvailable()` / `tryKernelExtract()` which check it per call. */ export function getKernel(): KernelModule | null { if (cached !== undefined) return cached; cached = null; for (const candidate of candidatePaths()) { try { if (!fs.existsSync(candidate)) continue; // createRequire: works identically from CJS output and future ESM. const req = createRequire(__filename); const mod = req(candidate) as KernelModule; if (typeof mod.extractFile !== 'function' || typeof mod.contractInfo !== 'function') { debug(`${candidate}: missing expected exports — ignoring`); continue; } if (!verifyContract(mod, candidate)) continue; kernelLanguages = new Set(mod.contractInfo().languages); debug(`loaded ${candidate} (languages: ${[...kernelLanguages].join(', ')})`); cached = mod; break; } catch (err) { debug(`${candidate}: failed to load — ${err instanceof Error ? err.message : String(err)}`); } } return cached; } /** True when the kill switch is off, a verified binary is loaded, and it supports `language`. */ export function kernelSupports(language: string): boolean { if (process.env.CODEGRAPH_KERNEL === '0') return false; return getKernel() !== null && kernelLanguages.has(language); } /** Test hook: forget the loaded module so a changed env is re-evaluated. */ export function resetKernelForTests(): void { cached = undefined; kernelLanguages = new Set(); } - src/extraction/languages/arkts.ts (5827 bytes)
import type { LanguageExtractor } from '../tree-sitter-types'; import { typescriptExtractor } from './typescript'; import type { Node as SyntaxNode } from 'web-tree-sitter'; /** * ArkTS (HarmonyOS / OpenHarmony, `.ets`) — a TypeScript superset whose * headline feature is declarative UI: an `@Component struct` with a `build()` * method describing the view tree, `@State`/`@Prop`/`@Link` reactive * properties, and global `@Builder`/`@Extend`/`@Styles` functions. * * The vendored grammar (harmony-contrib/tree-sitter-arkts) extends * tree-sitter-javascript exactly the way tree-sitter-typescript does, so every * TS node type — and therefore the whole typescriptExtractor — applies * verbatim. ArkTS-specific shapes it adds: * * - `struct_declaration` / `struct_body` — the `@Component struct`. Same * `name:`/`body:` fields as class_declaration; members are ordinary * `method_definition` / `public_field_definition` nodes, so struct members * extract through the standard class-member paths. * - `arkui_component_expression` — a build()-DSL component instantiation * (`Column() { … }`). Carries a `function:` field (the component), an * optional `children:` block, and — unlike TS — the CHAINED ATTRIBUTES as * repeated `property:`/`arguments:` field pairs on the SAME node * (`Text(x).fontSize(16).opacity(0.6)` is ONE node, not nested calls). * Handled by the arkts branch in extractCall (tree-sitter.ts). * - Decorators on functions (`@Builder function F() {}`) — invalid in TS, * first-class here (a `decorator:` field on function_declaration), so the * core's existing extractDecoratorsFor path captures them. */ /** Reactive/state decorators that make a member worth flagging (searchable). */ const DECORATED_MEMBER_TYPES = new Set([ 'struct_declaration', 'public_field_definition', 'method_definition', 'function_declaration', ]); /** * Collect decorator names for a declaration from BOTH positions the grammar * produces: direct `decorator` children (`@Entry @Component struct X`, * `@State count` on a field) and preceding `decorator` siblings (`@Builder` * before a method_definition inside struct_body; `@Component` on the * export_statement wrapping `export struct X`). The backwards sibling walk * stops at the first non-decorator so an earlier declaration's decorators * never leak in (mirrors extractDecoratorsFor's sibling pass). */ function collectDecoratorNames(node: SyntaxNode): string[] | undefined { const names: string[] = []; const nameOf = (dec: SyntaxNode): string | undefined => { for (let i = 0; i < dec.namedChildCount; i++) { const child = dec.namedChild(i); if (!child) continue; if (child.type === 'identifier') return child.text; if (child.type === 'call_expression') { // `@StorageLink('theme')` / `@Extend(Text)` — the decorator name is // the callee. const fn = child.childForFieldName('function'); if (fn?.type === 'identifier') return fn.text; } } return undefined; }; for (let i = 0; i < node.namedChildCount; i++) { const child = node.namedChild(i); if (child?.type === 'decorator') { const n = nameOf(child); if (n) names.push(n); } } const parent = node.parent; if (parent) { // Find this node among the parent's named children by start offset // (wrapper identity is not stable across navigation), then walk backwards. const start = node.startIndex; let idx = -1; for (let i = 0; i < parent.namedChildCount; i++) { const sib = parent.namedChild(i); if (sib && sib.startIndex === start) { idx = i; break; } } for (let i = idx - 1; i >= 0; i--) { const sib = parent.namedChild(i); if (!sib || sib.type !== 'decorator') break; const n = nameOf(sib); if (n) names.unshift(n); } } return names.length > 0 ? names : undefined; } export const arktsExtractor: LanguageExtractor = { ...typescriptExtractor, // `@Component struct X { … }` — extractStruct handles it (kind `struct`, // members extracted like class members, `this.m()` resolution and the // class/struct containment gates in the name-matcher all apply as-is). The // component-ness is preserved on the node's decorators (`Component`, // `Entry`, `CustomDialog`, `Reusable`), captured by extractModifiers below. structTypes: ['struct_declaration'], // build()-DSL component instantiations are call sites: `TodoRow({...})` // inside a parent's build() is the parent→child component edge, resolved by // the ordinary call pipeline against the child's struct node. The arkts // branch in extractCall also lifts each chained `.attr(...)` (emitted // dot-prefixed so it can ONLY resolve to `@Extend`/`@Styles`/`@Builder` // attribute helpers — see matchReference) and `.onXxx(this.handler)` // method-reference bindings. `leading_dot_expression` is the detached-chain // shape the grammar produces when a nested component's chain starts on the // line after its closing `}` inside arkui_children. callTypes: ['call_expression', 'arkui_component_expression', 'leading_dot_expression'], // Surface ArkTS decorators on the node's `decorators` list (searchable, and // the hook a future ArkUI state→build synthesizer keys off). Core paths // already emit `decorates` REFERENCES for classes/methods/properties/ // functions; this hook is what puts the names on struct nodes too — // extractStruct has no extractDecoratorsFor call, and node.decorators is // only populated via extractModifiers (see createNode). extractModifiers: (node) => { if (!DECORATED_MEMBER_TYPES.has(node.type)) return undefined; return collectDecoratorNames(node); }, }; - src/extraction/languages/c-cpp.ts (154863 bytes)
import type { Node as SyntaxNode } from 'web-tree-sitter'; import { getChildByField, getNodeText } from '../tree-sitter-helpers'; import type { LanguageExtractor } from '../tree-sitter-types'; /** * Resolve the declared identifier inside a C declarator. A `declaration`'s * `declarator` field nests the name through `init_declarator` (with value), * `pointer_declarator`/`array_declarator`/`parenthesized_declarator` * wrappers (each via their own `declarator` field) down to an `identifier`. * A `function_declarator` means the declaration is a function prototype (or a * function-pointer var) — return null so it isn't extracted as a variable. */ export function cDeclaratorIdentifier(node: SyntaxNode | null): SyntaxNode | null { let cur: SyntaxNode | null = node; let guard = 0; while (cur && guard++ < 12) { switch (cur.type) { case 'identifier': return cur; case 'function_declarator': return null; case 'init_declarator': case 'pointer_declarator': case 'array_declarator': case 'parenthesized_declarator': cur = getChildByField(cur, 'declarator'); break; default: return null; } } return null; } /** * Whether a C or Objective-C declaration is a constant: a `const` or * `static const` file-scope declaration carries a `type_qualifier` child * reading "const". Extracted as `constant`, plain globals as `variable`. A * `const` inside a pointer declarator (`char *const p`) qualifies the * pointer, and the declaration stays a variable. */ export function isConstDeclaration(node: SyntaxNode): boolean { return node.namedChildren.some((c: SyntaxNode) => c.type === 'type_qualifier' && c.text === 'const'); } /** C and C++ specifiers that define a class-like type, or an enum, when they carry a body (C has no `class_specifier`). */ const CLASS_LIKE_SPECIFIERS = new Set(['class_specifier', 'struct_specifier', 'union_specifier', 'enum_specifier']); /** Conditional compilation a class body can hold members in. */ const MEMBER_CONDITIONALS = new Set(['preproc_if', 'preproc_ifdef', 'preproc_else', 'preproc_elif', 'preproc_elifdef']); /** A type the program computes (`decltype(x)`, `auto`), which names no declaration. */ const COMPUTED_TYPE_SPECIFIERS = new Set(['decltype', 'placeholder_type_specifier']); /** * A name of the C++ standard library, or a type C itself gives a name the * grammars read like any other. A project's copy of one is a compiler shim * (jemalloc's msvc_compat/C99/stdbool.h), which code elsewhere never means. */ const C_STANDARD_TYPE_NAME = /^(?:(?:::\s*)?std\s*::|(?:_Bool|_Complex|wchar_t|char(?:8|16|32)_t|u?intmax_t|u?int_(?:least|fast)\d+_t|max_align_t|nullptr_t)$)/; /** What a declarator's next part is: a field where the grammar gives one, the first named child where it doesn't. */ function innerDeclarator(declarator: SyntaxNode): SyntaxNode | null { return getChildByField(declarator, 'declarator') ?? declarator.namedChild(0); } /** * Whether a member declarator declares a member function (`Status Get(int);`, * `Foo* Make();`, `Slice operator[](int);`), rather than data, which may be a * pointer to a function (`void (*cb)(int);`), whose function declarator wraps * a parenthesized one. */ function declaresMemberFunction(declarator: SyntaxNode): boolean { let d: SyntaxNode | null = declarator; while (d && (d.type === 'pointer_declarator' || d.type === 'reference_declarator')) d = innerDeclarator(d); return d?.type === 'function_declarator' && getChildByField(d, 'declarator')?.type !== 'parenthesized_declarator'; } /** Is this class-like specifier an unnamed member of another (`struct S { union { … } u; };`)? */ export function isUnnamedMemberType(specifier: SyntaxNode): boolean { return !getChildByField(specifier, 'name') && specifier.parent?.type === 'field_declaration'; } /** A type name a declaration writes: the name as references carry it, and the node it is read at. */ export interface CTypeName { name: string; node: SyntaxNode; } /** * The type names a C or C++ class, struct or union's members are declared * with, in source order: `pause_event` in `pause_event by_value;`, `*p`, `a[3]` * or `(*cb)(pause_event *)`, the `struct` or `enum` a member names by its tag * (`struct client *c`), and whatever a template's arguments name, in * `std::vector<Foo>` too. `specifier` is the class's own, carrying the body; * `ownName` is what the index calls it, which a typedef may decide. * * A name inside an unnamed member (`union { Foo a; }`, `struct { Foo a; } s;`) * is the holder's: its members are members of the holder, and nothing can name * the unnamed type (isUnnamedMemberType). A member function's return type * counts, since a member a macro trails reads the same (`Version* v GUARDED_BY(mu);`), * but not its parameters. Not read: builtins; a type of `std`, though its * template arguments are; the class's own name, which a list's `next` writes; * a type the program computes; a nested definition that declares no member * (`class Rep;`, `struct Node { … };`). * * Mirrored in the kernel (ccpp/mod.rs). */ export function cMemberTypeNames(specifier: SyntaxNode, ownName: string, source: string): CTypeName[] { const names: CTypeName[] = []; const own = new Set([ownName]); const tag = getChildByField(specifier, 'name'); if (tag) own.add(stripCppTemplateArgs(getNodeText(tag, source))); const body = getChildByField(specifier, 'body'); if (body) collectMemberTypeNames(body, own, source, names); return names; } /** * The type names a C or C++ alias is declared with: `redisObject` in `typedef * struct redisObject robj;`, `Foo` in `using Foos = std::vector<Foo>;`. Code * names the alias, so this is how a change to the type reaches what holds one. * Skipped as in cMemberTypeNames: `typedef struct dict dict;` names no other type. */ export function cAliasTypeNames(alias: SyntaxNode, ownName: string, source: string): CTypeName[] { const names: CTypeName[] = []; const own = new Set([ownName]); const type = getChildByField(alias, 'type'); if (type) collectCTypeNames(type, own, source, names); for (const declarator of alias.childrenForFieldName('declarator')) { if (declarator) collectDeclaratorTypeNames(declarator, own, source, names); } return names; } function collectMemberTypeNames(list: SyntaxNode, own: ReadonlySet<string>, source: string, names: CTypeName[]): void { for (const child of list.namedChildren) { if (MEMBER_CONDITIONALS.has(child.type)) { collectMemberTypeNames(child, own, source, names); continue; } if (child.type !== 'field_declaration') continue; const type = getChildByField(child, 'type'); if (!type) continue; const declarators = child.childrenForFieldName('declarator').filter((d: SyntaxNode | null): d is SyntaxNode => !!d); const unnamedBody = CLASS_LIKE_SPECIFIERS.has(type.type) && !getChildByField(type, 'name') ? getChildByField(type, 'body') : null; if (unnamedBody) collectMemberTypeNames(unnamedBody, own, source, names); else if (declarators.length > 0) collectCTypeNames(type, own, source, names); for (const declarator of declarators) { if (!declaresMemberFunction(declarator)) collectDeclaratorTypeNames(declarator, own, source, names); } } } /** The types a declarator names, past the type written before it: a function pointer's parameters. */ function collectDeclaratorTypeNames(declarator: SyntaxNode, own: ReadonlySet<string>, source: string, names: CTypeName[]): void { if (declarator.type === 'function_declarator') { const parameters = getChildByField(declarator, 'parameters'); if (parameters) collectCTypeNames(parameters, own, source, names); } if (!declarator.type.endsWith('declarator')) return; const inner = innerDeclarator(declarator); if (inner) collectDeclaratorTypeNames(inner, own, source, names); } /** Collects the type names under `node` onto `names`, in source order. */ function collectCTypeNames(node: SyntaxNode, own: ReadonlySet<string>, source: string, names: CTypeName[]): void { if (COMPUTED_TYPE_SPECIFIERS.has(node.type)) return; if (node.type === 'type_identifier' || node.type === 'qualified_identifier' || node.type === 'template_type') { const name = stripCppTemplateArgs(getNodeText(node, source)).trim(); if (name && !own.has(name) && !C_STANDARD_TYPE_NAME.test(name)) names.push({ name, node }); collectTemplateArguments(node, own, source, names); return; } const tag = CLASS_LIKE_SPECIFIERS.has(node.type) ? getChildByField(node, 'name') : null; for (const c of tag ? [tag] : node.namedChildren) collectCTypeNames(c, own, source, names); } /** The types the template arguments of a written name name, wherever in the name they are. */ function collectTemplateArguments(node: SyntaxNode, own: ReadonlySet<string>, source: string, names: CTypeName[]): void { for (const c of node.namedChildren) { if (c.type === 'template_argument_list') collectCTypeNames(c, own, source, names); else if (c.type === 'qualified_identifier' || c.type === 'template_type') collectTemplateArguments(c, own, source, names); } } /** * Whether a C/C++ type specifier defines a class, struct, union or enum, as * opposed to naming one (`struct Foo *p`, the forward declaration `class Foo;`). */ export function isClassLikeDefinition(node: SyntaxNode): boolean { return CLASS_LIKE_SPECIFIERS.has(node.type) && getChildByField(node, 'body') !== null; } /** * The `declaration` a class, struct, union or enum is defined in the type of — * `struct Foo { … } foo;` — or null for one written on its own, in a typedef or * in a member's type. Objective-C shares it (objc.ts). */ export function definingDeclaration(node: SyntaxNode): SyntaxNode | null { if (!isClassLikeDefinition(node)) return null; const parent = node.parent; return parent?.type === 'declaration' && getChildByField(parent, 'type')?.id === node.id ? parent : null; } /** * The name of an unnamed class, struct, union or enum that a declaration * defines: the first variable the declaration declares. No code can name the * type of `static struct { … } SPT;` (only `SPT`), so the struct is indexed as * `SPT`, the way `typedef struct { … } Name;` is indexed as `Name`. Undefined * for a named type, and when the declaration declares no variable (a * function's return type). Mirrored in the kernel (ccpp/mod.rs). Objective-C * shares it (objc.ts), and has no kernel walker. */ export function unnamedTypeVariableName(node: SyntaxNode, source: string): string | undefined { if (getChildByField(node, 'name')) return undefined; const declaration = definingDeclaration(node); if (!declaration) return undefined; for (const declarator of declaration.childrenForFieldName('declarator')) { const identifier = cDeclaratorIdentifier(declarator); if (identifier) return getNodeText(identifier, source); } return undefined; } /** * Find the function NAME's `qualified_identifier` (`Foo::bar`) inside a * declarator, skipping the `parameter_list` — a parameter with a qualified type * (`const std::string& x`) must NOT be mistaken for the method name. Without the * skip, a plain free function `std::string TableFileName(const std::string&...)` * was named `string` (from the parameter type), so calls to it never resolved * and its file looked like nothing depended on it. */ function findDeclaratorQualifiedId(declarator: SyntaxNode): SyntaxNode | undefined { const queue: SyntaxNode[] = [declarator]; while (queue.length > 0) { const current = queue.shift()!; if (current.type === 'qualified_identifier') return current; for (let i = 0; i < current.namedChildCount; i++) { const child = current.namedChild(i); // Don't descend into parameters, the trailing return type or the // clauses after them — `const std::string&`, `-> std::string`, // nlohmann/json's `noexcept(noexcept(JSONSerializer<U>::to_json(…)))`, // `throw(std::bad_alloc)`, `requires std::integral<T>` name no function. if (child && !DECLARATOR_NON_NAME_PARTS.has(child.type)) { queue.push(child); } } } return undefined; } /** The parts of a function declarator that hold types or expressions, never its name. */ const DECLARATOR_NON_NAME_PARTS: ReadonlySet<string> = new Set([ 'parameter_list', 'trailing_return_type', 'noexcept', 'throw_specifier', 'requires_clause', ]); /** * Single-argument macros are ambiguous without their definition (#1373). * Require a preceding local #define whose replacement uses its sole parameter * as the function declarator; registration, token-pasting, and K&R forms stay * untouched. Walk enclosing scopes, but never guess across conditional macros. */ function recoverSingleArgMacroDefinedName(node: SyntaxNode, source: string): string | undefined { if (node.type !== 'function_definition') return undefined; const declarator = getChildByField(node, 'declarator'); let macro: SyntaxNode | null = null; let argument: SyntaxNode | null = null; if (declarator?.type === 'parenthesized_declarator' && declarator.namedChildCount === 1) { // C: NATIVE_FN is parsed as the return type, (name) as the declarator. macro = getChildByField(node, 'type'); argument = declarator.namedChild(0); if (macro?.type !== 'type_identifier' || argument?.type !== 'identifier') return undefined; } else if (declarator?.type === 'function_declarator' && !getChildByField(node, 'type')) { // C++: NATIVE_FN(name) is parsed as an implicit-return-type function. macro = getChildByField(declarator, 'declarator'); const params = getChildByField(declarator, 'parameters'); const param = params?.namedChild(0); if (macro?.type !== 'identifier' || params?.namedChildCount !== 1 || param?.type !== 'parameter_declaration' || param.namedChildCount !== 1) return undefined; argument = param.namedChild(0); if (argument?.type !== 'type_identifier') return undefined; } if (!macro || !argument) return undefined; const macroName = getNodeText(macro, source); for (let scope: SyntaxNode | null = node; scope; scope = scope.parent) { if (scope.type === 'preproc_else' || scope.type.startsWith('preproc_elif')) return undefined; for (let prev = scope.previousNamedSibling; prev; prev = prev.previousNamedSibling) { if (prev.type.startsWith('preproc_if')) return undefined; if (prev.type === 'preproc_call' && getChildByField(prev, 'directive')?.text === '#undef' && getChildByField(prev, 'argument')?.text.trim() === macroName) return undefined; if (prev.type !== 'preproc_function_def' && prev.type !== 'preproc_def') continue; if (getChildByField(prev, 'name')?.text !== macroName) continue; const params = getChildByField(prev, 'parameters'); const param = params?.namedChild(0); const value = getChildByField(prev, 'value' - src/extraction/languages/cfquery.ts (918 bytes)
import type { LanguageExtractor } from '../tree-sitter-types'; /** * `<cfquery>` SQL bodies: `#hash#` expressions inside the SQL text are real * CFML expressions (tree-sitter-cfml's `cfquery` grammar parses them * structurally — `call_expression`/`member_expression`, same shape as * cfscript's), so a call like `#getCurrentUser().getId()#` embedded in a * WHERE clause is a genuine call edge. The surrounding SQL keywords/ * identifiers aren't symbols CodeGraph models — only `call_expression` is * mapped, so extraction yields call references and nothing else. */ export const cfqueryExtractor: LanguageExtractor = { functionTypes: [], classTypes: [], methodTypes: [], interfaceTypes: [], structTypes: [], enumTypes: [], typeAliasTypes: [], importTypes: [], callTypes: ['call_expression'], variableTypes: [], nameField: 'name', bodyField: 'body', paramsField: 'parameters', }; - src/extraction/languages/cfscript.ts (7705 bytes)
import type { Node as SyntaxNode } from 'web-tree-sitter'; import { getNodeText, getChildByField } from '../tree-sitter-helpers'; import type { LanguageExtractor } from '../tree-sitter-types'; /** CFML access modifiers (`public`/`private`/`package`/`remote`) on a function_declaration. */ function cfmlVisibility(node: SyntaxNode): 'public' | 'private' | 'protected' | 'internal' | undefined { for (let i = 0; i < node.childCount; i++) { const child = node.child(i); if (child?.type === 'access_type') { const text = child.text; if (text === 'public') return 'public'; if (text === 'private') return 'private'; if (text === 'package') return 'internal'; if (text === 'remote') return 'public'; } } return undefined; } /** * A string literal's text when it is written out in full, or null when a * `#…#` expression builds it at runtime (`"#dir#/x.cfm"`, `"#dir#.Batch"`): * such a path names no file, and such a name no component, the index can * link. */ function literalString(node: SyntaxNode | null | undefined, source: string): string | null { if (node?.type !== 'string' || node.namedChildren.some((c: SyntaxNode) => c.type !== 'string_fragment')) return null; return getNodeText(node, source).replace(/^["']|["']$/g, '').trim() || null; } /** The value of a `template=…` argument or attribute, when it is a literal string. */ function templateArgument(node: SyntaxNode, source: string): string | null { if (node.type !== 'assignment_expression') return null; const left = getChildByField(node, 'left'); if (left?.type !== 'identifier' || getNodeText(left, source).toLowerCase() !== 'template') return null; return literalString(getChildByField(node, 'right'), source); } /** * Whether a call is `cfinclude(…)` — the function form of `include` and of * `<cfinclude>` (ACF 11+, Lucee): an include of a template, not a call (#2651). */ export function isCfIncludeCall(node: SyntaxNode, source: string): boolean { const fn = getChildByField(node, 'function'); return fn?.type === 'identifier' && getNodeText(fn, source).toLowerCase() === 'cfinclude'; } /** The literal template `cfinclude(template="x.cfm")` or `cfinclude("x.cfm")` names. */ function cfIncludeCallTemplate(node: SyntaxNode, source: string): string | null { const args = getChildByField(node, 'arguments'); if (!args) return null; for (const arg of args.namedChildren) { const named = templateArgument(arg, source); if (named) return named; } return literalString(args.namedChildren[0], source); } /** A component name or dotted component path (`Batch`, `contenscms.core.queue.Batch`). */ const COMPONENT_PATH = /^[A-Za-z_][\w$-]*(?:\.[A-Za-z_][\w$-]*)*$/; /** * The component `createObject("component", "a.b.Batch")` makes — or the * one-argument `createObject("a.b.Batch")` (Lucee, ACF), or with named * arguments, `createObject(type="component", component="a.b.Batch")` — as * written (#2652). Null for any other call, for another type of object * (`createObject("java", …)`), and for a name built at runtime. */ export function cfmlCreatedComponent(call: SyntaxNode, source: string): string | null { if (call.type !== 'call_expression') return null; const fn = getChildByField(call, 'function'); if (fn?.type !== 'identifier' || getNodeText(fn, source).toLowerCase() !== 'createobject') return null; const args = getChildByField(call, 'arguments')?.namedChildren ?? []; let type: string | null | undefined; let name: string | null | undefined; if (args.length > 0 && args.every((a: SyntaxNode) => a.type === 'assignment_expression')) { for (const arg of args) { const key = getNodeText(getChildByField(arg, 'left') ?? arg, source).toLowerCase(); if (key === 'type') type = literalString(getChildByField(arg, 'right'), source); else if (key === 'component') name = literalString(getChildByField(arg, 'right'), source); } } else if (args.length === 1) { name = literalString(args[0], source); } else if (args.length >= 2) { type = literalString(args[0], source); name = literalString(args[1], source); } if (type !== undefined && type?.toLowerCase() !== 'component') return null; return name && COMPONENT_PATH.test(name) ? name : null; } /** * The component an expression constructs, when the expression is a * construction a call is chained onto: `new a.b.Batch()` or a component * `createObject(…)`, parenthesized or not (#2652). */ export function cfmlConstructedComponent(node: SyntaxNode, source: string): string | null { let expr: SyntaxNode | null = node; while (expr?.type === 'parenthesized_expression') expr = expr.namedChild(0); if (!expr) return null; if (expr.type === 'new_expression') { const ctor = getChildByField(expr, 'constructor'); const name = ctor ? getNodeText(ctor, source).replace(/\s+/g, '') : ''; return COMPONENT_PATH.test(name) ? name : null; } return cfmlCreatedComponent(expr, source); } export const cfscriptExtractor: LanguageExtractor = { functionTypes: ['function_declaration', 'function_expression', 'arrow_function'], classTypes: ['component'], // `component` is reused for both `component { ... }` and `interface { ... }` — // the only difference is the literal first token (verified via the grammar's // native binding: child(0).type is 'component' or 'interface', both unnamed). classifyClassNode: (node) => (node.child(0)?.type === 'interface' ? 'interface' : 'class'), methodTypes: ['function_declaration', 'method_definition'], interfaceTypes: [], structTypes: [], enumTypes: [], typeAliasTypes: [], importTypes: ['import_statement', 'include_statement'], callTypes: ['call_expression'], variableTypes: ['variable_declaration'], propertyTypes: ['property_declaration'], nameField: 'name', bodyField: 'body', paramsField: 'parameters', getVisibility: cfmlVisibility, getSignature: (node, source) => { const params = getChildByField(node, 'parameters'); return params ? getNodeText(params, source) : undefined; }, extractImport: (node, source) => { const importText = source.substring(node.startIndex, node.endIndex).trim(); if (node.type === 'include_statement') { // `include "path/to/file.cfm";` — the included template path — or the // attribute form, `include template="path/to/file.cfm";` (#2651). A // path built at runtime (`include "#view#";`) is no import. const expr = node.namedChildren.find((c: SyntaxNode) => c.type === 'string'); const attr = expr ? undefined : node.namedChildren.find((c: SyntaxNode) => c.type === 'assignment_expression'); const moduleName = expr ? literalString(expr, source) : attr ? templateArgument(attr, source) : null; return moduleName ? { moduleName, signature: importText } : null; } if (node.type === 'call_expression') { // `cfinclude(template="path/to/file.cfm");` (#2651). const moduleName = isCfIncludeCall(node, source) ? cfIncludeCallTemplate(node, source) : null; return moduleName ? { moduleName, signature: importText } : null; } // `import com.foo.Bar;` (dotted path) or `import "java:java.util.ArrayList";` (string form) const sourceNode = getChildByField(node, 'source'); if (!sourceNode) return null; let moduleName: string; if (sourceNode.type === 'import_path') { moduleName = sourceNode.namedChildren .map((c: SyntaxNode) => getNodeText(c, source)) .join('.'); } else { moduleName = getNodeText(sourceNode, source).replace(/^["']|["']$/g, ''); } return moduleName ? { moduleName, signature: importText } : null; }, }; - src/extraction/languages/cobol.ts (19999 bytes)
/** * COBOL Language Extractor * * COBOL's AST (vendored, patched build of yutaro-sakamoto/tree-sitter-cobol) * is fundamentally different from block-structured languages, so extraction * runs almost entirely through the custom visitNode hook (the Pascal pattern): * * - A program (PROGRAM-ID) becomes a `module` node. * - PROCEDURE DIVISION sections and paragraphs become `function` nodes. The * grammar emits them FLAT — a section_header/paragraph_header followed by * sibling statements — so extents are reconstructed here: a paragraph runs * from its header to the next header, a section to the next section header. * - PERFORM (including THRU ranges), GO TO, and CALL 'literal' become `calls` * references. A dynamic CALL through a data name is skipped — announce, * don't guess. EXEC CICS LINK/XCTL PROGRAM('X') with a literal target also * becomes a `calls` reference; EXEC SQL INCLUDE X becomes an `imports` * reference (DB2's COPY). * - COPY statements become `import` nodes + `imports` references. * - DATA DIVISION entries become `variable` (01/77 levels), `field` (nested * levels, contained in their group item), or `constant` (88-level condition * names) nodes, so impact queries on working-storage names work. * - Standalone copybooks (.cpy) parse via the grammar's copybook_fragment * entry point: data copybooks yield their record structure, procedure * copybooks yield paragraphs. * * The grammar is fixed-format (code area columns 8-72). preParse detects a * free-format file (division header or level number starting before column 8) * and indents every line by 7 spaces: line numbers are preserved, columns * drift by 7 — acceptable for line-oriented consumers. */ import type { Node as SyntaxNode } from 'web-tree-sitter'; import { getNodeText, getChildByField } from '../tree-sitter-helpers'; import type { ExtractorContext, LanguageExtractor } from '../tree-sitter-types'; /** EXEC CICS LINK/XCTL — program target, quoted literal or data name. */ const EXEC_CICS_PROGRAM_RE = /\b(?:LINK|XCTL)\b[\s\S]*?\bPROGRAM\s*\(\s*(?:['"]([A-Za-z0-9$#@-]+)['"]|([A-Za-z0-9-]+))\s*\)/i; /** EXEC CICS RETURN/START — next transaction, quoted literal or data name. */ const EXEC_CICS_TRANSID_RE = /\b(?:RETURN|START)\b[\s\S]*?\bTRANSID\s*\(\s*(?:['"]([A-Za-z0-9$#@]{1,4})['"]|([A-Za-z0-9-]+))\s*\)/i; /** EXEC SQL INCLUDE <member> — the SQL flavor of COPY. */ const EXEC_SQL_INCLUDE_RE = /\bSQL\b[\s\S]*?\bINCLUDE\s+([A-Za-z0-9$#@-]+)/i; /** The VALUE literal in a data item's signature ("PIC X(04) VALUE 'CB00'"). */ const VALUE_LITERAL_RE = /\bVALUE\s+['"]([A-Za-z0-9$#@-]+)['"]/i; function line(node: SyntaxNode): number { return node.startPosition.row + 1; } function endLineOf(node: SyntaxNode): number { return node.endPosition.row + 1; } /** Collapse whitespace runs so multi-line clauses read as one signature. */ function collapse(text: string, cap = 120): string { const flat = text.replace(/\s+/g, ' ').trim(); return flat.length > cap ? flat.slice(0, cap - 1) + '…' : flat; } function currentScope(ctx: ExtractorContext): string | undefined { return ctx.nodeStack[ctx.nodeStack.length - 1]; } /** * DB2 convention writes `EXEC SQL INCLUDE member END-EXEC` with no sentence * period, sitting between paragraphs like a COPY — which derails the * grammar's sentence machinery. Terminate the single-line form by writing a * period into the character right after END-EXEC (a space), or appending one * at end of line. Both are offset-preserving for every other character. */ function terminateSqlIncludes(source: string): string { const lineRe = /^([ \t]*(?:[0-9]{6})?[ \t]+EXEC\s+SQL\s+INCLUDE\s+[A-Za-z0-9$#@-]+\s+END-EXEC)([ \t]|$)/i; return source .split('\n') .map((l) => { if (!/END-EXEC/i.test(l) || /END-EXEC\s*\./i.test(l)) return l; const m = lineRe.exec(l); if (!m) return l; const head = m[1]!; return m[2] === '' ? head + '.' : head + '.' + l.slice(head.length + 1); }) .join('\n'); } function addRef( ctx: ExtractorContext, fromNodeId: string | undefined, referenceName: string, referenceKind: 'calls' | 'imports' | 'references', at: SyntaxNode ): void { if (!fromNodeId || !referenceName) return; ctx.addUnresolvedReference({ fromNodeId, referenceName, referenceKind, line: line(at), column: at.startPosition.column, }); } /** COPY <book> [IN/OF lib] — import node + imports reference. */ function handleCopy(node: SyntaxNode, ctx: ExtractorContext): void { const book = getChildByField(node, 'book'); if (!book) return; const name = getNodeText(book, ctx.source) .trim() .replace(/^['"]|['"]$/g, ''); if (!name) return; ctx.createNode('import', name, node, { signature: collapse(getNodeText(node, ctx.source)), }); addRef(ctx, currentScope(ctx), name, 'imports', node); } /** * A CICS option can name its target through a data item instead of a * literal (`TRANSID(WS-TRANID)` with `05 WS-TRANID ... VALUE 'CB00'`). * Dereference it against the SAME FILE's already-extracted data items — * the DATA DIVISION precedes the PROCEDURE DIVISION, so they are in * ctx.nodes by the time EXEC blocks are mined. Anything else (copybook * arrays, computed names) is dynamic dispatch: skipped, not guessed. */ function derefSameFileValue(name: string, ctx: ExtractorContext): string | undefined { const upper = name.toUpperCase(); for (const node of ctx.nodes) { if (node.filePath !== ctx.filePath) continue; if (node.kind !== 'variable' && node.kind !== 'field' && node.kind !== 'constant') continue; if (node.name.toUpperCase() !== upper) continue; const value = node.signature ? VALUE_LITERAL_RE.exec(node.signature) : null; return value?.[1]; } return undefined; } /** * EXEC ... END-EXEC blocks are opaque single nodes. Mine the text for the * statically-resolvable shapes: CICS LINK/XCTL to a program (cross-program * call), CICS RETURN/START TRANSID (the pseudo-conversational hop — emitted * as a `cics-transid:XXXX` reference the CICS framework resolver maps to * the owning program), and SQL INCLUDE (a copybook import). */ function handleExec( node: SyntaxNode, ctx: ExtractorContext, fromNodeId: string | undefined ): void { const text = getNodeText(node, ctx.source); const cics = EXEC_CICS_PROGRAM_RE.exec(text); if (cics) { const program = cics[1] ?? (cics[2] ? derefSameFileValue(cics[2], ctx) : undefined); if (program) addRef(ctx, fromNodeId, program, 'calls', node); } const transid = EXEC_CICS_TRANSID_RE.exec(text); if (transid) { const tx = transid[1] ?? (transid[2] ? derefSameFileValue(transid[2], ctx) : undefined); if (tx && /^[A-Za-z0-9$#@]{1,4}$/.test(tx)) { addRef(ctx, fromNodeId, `cics-transid:${tx.toUpperCase()}`, 'calls', node); } } const include = EXEC_SQL_INCLUDE_RE.exec(text); if (include?.[1]) { ctx.createNode('import', include[1], node, { signature: collapse(text), }); addRef(ctx, fromNodeId ?? currentScope(ctx), include[1], 'imports', node); } } /** * Walk a run of DATA DIVISION entries (working-storage section children or a * record_description_list). Level numbers drive nesting: an entry closes all * open entries with level >= its own. 88-level condition names attach to the * open item as constants and never open a scope. */ function walkDataEntries(entries: SyntaxNode[], ctx: ExtractorContext): void { interface Item { node: SyntaxNode; level: number; name: string | null; } const items: (Item | null)[] = entries.map((e) => { if (e.type !== 'data_description') return null; const levelNode = e.namedChildren.find((c: SyntaxNode) => c?.type === 'level_number'); const nameNode = e.namedChildren.find((c: SyntaxNode) => c?.type === 'entry_name'); const level = levelNode ? parseInt(getNodeText(levelNode, ctx.source), 10) : NaN; const name = nameNode ? getNodeText(nameNode, ctx.source).trim() : null; return { node: e, level: Number.isFinite(level) ? level : 1, name }; }); /** Levels 01, 66, and 77 always open at top level, whatever came before. */ const isTopLevel = (level: number): boolean => level === 1 || level === 66 || level === 77; /** A group item extends to the last entry before the next level <= its own. */ const groupEnd = (i: number): number => { const self = items[i]!; let end = endLineOf(self.node); for (let j = i + 1; j < entries.length; j++) { const it = items[j]; if (!it) continue; if (it.level !== 88 && (it.level <= self.level || isTopLevel(it.level))) break; end = Math.max(end, endLineOf(it.node)); } return end; }; const open: { level: number }[] = []; let pushed = 0; const closeTo = (level: number) => { while (open.length > 0 && open[open.length - 1]!.level >= level) { open.pop(); ctx.popScope(); pushed--; } }; for (let i = 0; i < entries.length; i++) { const entry = entries[i]!; if (entry.type === 'copy_statement') { handleCopy(entry, ctx); continue; } if (entry.type === 'exec_statement') { handleExec(entry, ctx, currentScope(ctx)); continue; } const item = items[i]; if (!item) continue; const isCondition = item.level === 88; if (!isCondition) closeTo(isTopLevel(item.level) ? 0 : item.level); // FILLER / unnamed entries carry no symbol; they only occupy layout space. if (!item.name || /^FILLER$/i.test(item.name)) continue; const kind = isCondition ? 'constant' : open.length === 0 ? 'variable' : 'field'; const created = ctx.createNode(kind, item.name, item.node, { endLine: isCondition ? endLineOf(item.node) : groupEnd(i), signature: collapse(getNodeText(item.node, ctx.source)), }); if (created && !isCondition) { ctx.pushScope(created.id); open.push({ level: item.level }); pushed++; } } while (pushed > 0) { ctx.popScope(); pushed--; } } /** * Special registers and CICS/SQL communication areas — writes to these are * runtime plumbing, not project data flow; referencing them would only mint * dangling refs (they have no declaration node). */ const SPECIAL_REGISTER_RE = /^(RETURN-CODE|SQLCODE|SQLSTATE|TALLY|EIB[A-Z-]+|DFH[A-Z-]+|WHEN-COMPILED|LENGTH|ADDRESS)$/i; /** The base data name of an assignment target (first WORD, ignoring subscripts/OF-qualifiers). */ function targetBaseName(target: SyntaxNode, source: string): SyntaxNode | null { if (target.type === 'WORD') return target; for (const child of target.namedChildren) { if (!child) continue; const found = targetBaseName(child, source); if (found) return found; } return null; } /** * Emit a `references` ref for every data item a statement WRITES (MOVE TO, * ADD TO/GIVING, SUBTRACT FROM/GIVING, COMPUTE =). Write-sites are what an * impact query on a working-storage or copybook field needs: "what changes * WS-TOTAL" is the COBOL analogue of value-reference edges. Read operands * are deliberately not tracked — they would multiply edge volume for little * impact value. */ function emitWriteRefs( statement: SyntaxNode, fields: string[], fromNodeId: string | undefined, ctx: ExtractorContext ): void { for (const field of fields) { for (const target of statement.childrenForFieldName(field)) { if (!target) continue; const word = targetBaseName(target, ctx.source); if (!word) continue; const name = getNodeText(word, ctx.source).trim(); if (!name || SPECIAL_REGISTER_RE.test(name)) continue; addRef(ctx, fromNodeId, name, 'references', word); } } } /** * Collect call/import references from a statement subtree, attributed to the * enclosing paragraph/section (or the program when no paragraph is open). */ function collectRefs( node: SyntaxNode, fromNodeId: string | undefined, ctx: ExtractorContext ): void { switch (node.type) { case 'move_statement': emitWriteRefs(node, ['dst'], fromNodeId, ctx); return; case 'add_statement': emitWriteRefs(node, ['to', 'giving'], fromNodeId, ctx); return; case 'compute_statement': emitWriteRefs(node, ['left'], fromNodeId, ctx); return; case 'subtract_statement': { // SUBTRACT x FROM t — `from` is the target; with GIVING, `giving` is. const hasGiving = node.childrenForFieldName('giving').length > 0; emitWriteRefs(node, [hasGiving ? 'giving' : 'from'], fromNodeId, ctx); return; } case 'perform_statement_call_proc': { // PERFORM A [THRU B] — every label is a paragraph/section call target. const proc = getChildByField(node, 'procedure'); if (proc) { for (const label of proc.namedChildren) { if (label?.type !== 'label') continue; const name = getNodeText(label, ctx.source).trim(); addRef(ctx, fromNodeId, name, 'calls', label); } } return; } case 'call_statement': { // CALL 'PROG' — static cross-program call. CALL data-name is dynamic // dispatch through a variable: skipped (announce, don't guess). const x = getChildByField(node, 'x'); if (x?.type === 'string') { const name = getNodeText(x, ctx.source).replace(/^['"]|['"]$/g, '').trim(); addRef(ctx, fromNodeId, name, 'calls', x); } return; } case 'goto_statement': { const to = getChildByField(node, 'to'); if (to) { addRef(ctx, fromNodeId, getNodeText(to, ctx.source).trim(), 'calls', to); } return; } case 'exec_statement': handleExec(node, ctx, fromNodeId); return; case 'copy_statement': handleCopy(node, ctx); return; default: for (const child of node.namedChildren) { if (child) collectRefs(child, fromNodeId, ctx); } } } /** "PARA-NAME." → "PARA-NAME"; "SEC-NAME SECTION." → "SEC-NAME". */ function headerName(header: SyntaxNode, source: string): string { const text = getNodeText(header, source).trim().replace(/\.$/, '').trim(); return text.split(/\s+/)[0] ?? text; } /** * Walk the flat PROCEDURE DIVISION (or a procedure copybook fragment): * reconstruct section/paragraph extents from header positions and attribute * the sibling statements between headers to the open paragraph. */ function walkProcedureChildren( children: SyntaxNode[], divisionEndLine: number, ctx: ExtractorContext ): void { /** 1-based last line of the region started at children[i]. */ const regionEnd = (i: number, sectionsOnly: boolean): number => { for (let j = i + 1; j < children.length; j++) { const c = children[j]!; if (c.type === 'section_header' || (!sectionsOnly && c.type === 'paragraph_header')) { // Header starts at 0-based row R → the region ends on 1-based line R. return Math.max(c.startPosition.row, line(children[i]!)); } } - src/extraction/languages/cpp-brace-scopes.ts (14496 bytes)
/** * C++ scopes read from the source's braces, for a file whose parse tree has * errors. * * tree-sitter recovers from a construct it can't parse by inserting the token * it expected, and inside a class body that token is often a `}`: an unknown * macro in front of a member (`PROTOBUF_FUTURE_ADD_EARLY_NODISCARD * absl::string_view name() const`, `struct ALIGN_AS(64U) HandleImpl`) closes * the class there. Every `}` after it then closes the scope one level out, so * the rest of the class, and of each namespace around it, parses as * declarations of an outer scope: protobuf's `FieldDescriptor` indexed as * `FieldDescriptor` instead of `google::protobuf::FieldDescriptor`, rocksdb's * `struct Opts` outside the class it is declared in. Recovery also runs the * other way and keeps a scope open past its own `}`. * * The braces are rarely what the parser misread, so the walker takes each * declaration's namespaces and enclosing classes from them instead * (TreeSitterExtractor.visitInCppBraceScope; mirrored in the kernel, * ccpp/mod.rs). The scan skips comments, string and character literals (raw * strings and digit separators too) and preprocessor lines. Each branch of an * `#if` is read from the braces open at the `#if`, and the scan continues * after `#endif` with the first branch's — so `#if A` `struct X : B {` * `#else` `struct X {` `#endif` opens one scope, not two. A file whose braces * don't balance gets no scopes and keeps the tree's. */ /** Intervals that nest properly, added in order of their opening offset. */ export class NestedIntervals<T> { private readonly opens: number[] = []; private readonly closes: number[] = []; private readonly parents: number[] = []; private readonly values: T[] = []; /** * Adds the interval `(open, close)`. One that opens before the last one * added, or crosses one that holds it, is refused (false). */ add(open: number, close: number, value: T): boolean { const count = this.opens.length; if (close <= open || (count > 0 && open <= this.opens[count - 1]!)) return false; const parent = this.innermost(open); if (parent >= 0 && close >= this.closes[parent]!) return false; this.opens.push(open); this.closes.push(close); this.parents.push(parent); this.values.push(value); return true; } /** The values of the intervals holding `offset` (open < offset < close), outermost first. */ at(offset: number): T[] { const out: T[] = []; for (let k = this.innermost(offset); k >= 0; k = this.parents[k]!) out.push(this.values[k]!); return out.reverse(); } /** * The innermost interval holding `offset`, or -1. It is the last interval * opening before `offset` or one of the intervals around that one. */ private innermost(offset: number): number { let lo = 0; let hi = this.opens.length - 1; let k = -1; while (lo <= hi) { const mid = (lo + hi) >> 1; if (this.opens[mid]! < offset) { k = mid; lo = mid + 1; } else { hi = mid - 1; } } while (k >= 0 && this.closes[k]! <= offset) k = this.parents[k]!; return k; } } export interface CppBraceScopes { /** The offset of the `}` that closes the `{` at `open`. */ closeOf(open: number): number | undefined; /** The named namespaces whose braces hold `offset`, outermost first, as written (`a::b`). */ namespacesAt(offset: number): string[]; /** * Whether `offset` is at namespace scope: every brace around it opens a * namespace, named or not, and none a class, a function or a linkage block. */ atNamespaceScope(offset: number): boolean; /** The 1-based line and the column of `offset`. */ positionOf(offset: number): { line: number; column: number }; } const NEWLINE = 10; const isBlank = (c: number): boolean => c === 32 || c === 9 || c === 13 || c === 12 || c === 11; const isWordStart = (c: number): boolean => (c >= 97 && c <= 122) || (c >= 65 && c <= 90) || c === 95 || c === 36 || c >= 128; const isDigit = (c: number): boolean => c >= 48 && c <= 57; const isWordChar = (c: number): boolean => isWordStart(c) || isDigit(c); const RAW_STRING_PREFIX = /^(?:u8|[uUL])?R$/; // A raw string's delimiter: up to 16 printable ASCII characters but `"`, `(`, `)` and `\`. const RAW_STRING_DELIMITER = /^[!#-'*-[\]-~]{0,16}$/; /** The end of a `//` comment starting at `i`: its newline (a `\` before it continues the comment). */ function lineCommentEnd(source: string, i: number): number { for (;;) { const nl = source.indexOf('\n', i); if (nl < 0) return source.length; let k = nl - 1; if (source.charCodeAt(k) === 13) k--; if (source.charCodeAt(k) !== 92) return nl; i = nl + 1; } } /** Past whitespace and comments from `i`. */ function skipBlank(source: string, i: number): number { const n = source.length; while (i < n) { const c = source.charCodeAt(i); if (isBlank(c) || c === NEWLINE) i++; else if (c === 47 && source.charCodeAt(i + 1) === 47) i = lineCommentEnd(source, i + 2); else if (c === 47 && source.charCodeAt(i + 1) === 42) { const end = source.indexOf('*/', i + 2); i = end < 0 ? n : end + 2; } else break; } return i; } /** The end of the identifier at `i`, or -1 when none starts there. */ function wordEnd(source: string, i: number): number { if (!isWordStart(source.charCodeAt(i))) return -1; let j = i + 1; while (j < source.length && isWordChar(source.charCodeAt(j))) j++; return j; } /** Past the `(…)` group at `i` (strings inside skipped), or -1. */ function parenGroupEnd(source: string, i: number): number { let depth = 0; for (let j = i; j < source.length; j++) { const c = source.charCodeAt(j); if (c === 34 || c === 39) j = quotedEnd(source, j) - 1; else if (c === 40) depth++; else if (c === 41 && --depth === 0) return j + 1; } return -1; } /** Past the string or character literal whose quote is at `i` (it stops at a newline). */ function quotedEnd(source: string, i: number): number { const quote = source.charCodeAt(i); let j = i + 1; while (j < source.length) { const c = source.charCodeAt(j); if (c === 92) j += source.charCodeAt(j + 1) === 13 && source.charCodeAt(j + 2) === NEWLINE ? 3 : 2; else if (c === quote) return j + 1; else if (c === NEWLINE) return j; else j++; } return source.length; } /** * After the `namespace` keyword ending at `from`: the name as written and the * offset of the `{` that opens the body. Null for anything else, such as * `using namespace std;` or `namespace fs = std::filesystem;`. Takes C++17 * attributes (`namespace [[deprecated]] old {`), nested and inline names * (`namespace a::inline b {`), and an attribute macro after the name * (`namespace std _GLIBCXX_VISIBILITY(default) {`). */ function namespaceHead(source: string, from: number): { name: string; brace: number } | null { let i = skipBlank(source, from); while (source.startsWith('[[', i)) { const end = source.indexOf(']]', i + 2); if (end < 0) return null; i = skipBlank(source, end + 2); } let nameStart = -1; let nameEnd = -1; for (let end = wordEnd(source, i); end >= 0; end = wordEnd(source, i)) { if (nameStart < 0) nameStart = i; nameEnd = end; i = skipBlank(source, end); if (!source.startsWith('::', i)) break; i = skipBlank(source, i + 2); const inline = wordEnd(source, i); if (inline >= 0 && source.slice(i, inline) === 'inline') i = skipBlank(source, inline); } for (let end = wordEnd(source, i); end >= 0 && nameStart >= 0; end = wordEnd(source, i)) { const paren = skipBlank(source, end); const after = source.charCodeAt(paren) === 40 ? parenGroupEnd(source, paren) : -1; if (after < 0) return null; i = skipBlank(source, after); } if (source.charCodeAt(i) !== 123) return null; return { name: nameStart < 0 ? '' : source.slice(nameStart, nameEnd), brace: i }; } /** * Scan `source` (a C++ file as the parser saw it, after preParse) for its * braces and namespaces. Null when the braces don't balance. */ export function scanCppBraceScopes(source: string): CppBraceScopes | null { const n = source.length; const closes = new Map<number, number>(); let open: number[] = []; // Per `#if` group: the braces open at the `#if`, and those open after its // first branch (null until a second branch starts). const branches: Array<{ atIf: number[]; afterFirst: number[] | null }> = []; const namespaceAt = new Map<number, string>(); // `{` offset → name, for named namespaces const namespaceBraces = new Set<number>(); // `{` offsets of every namespace, named or not let pendingNamespace: { name: string; brace: number } | null = null; let balanced = true; let lineStart = true; let wordStart = -1; // the last identifier or number read let wordEndAt = -1; let i = source.charCodeAt(0) === 0xfeff ? 1 : 0; // past a byte-order mark, so line 1 can be a directive while (i < n) { const c = source.charCodeAt(i); if (c === NEWLINE) { lineStart = true; i++; continue; } if (isBlank(c)) { i++; continue; } if (c === 47 && source.charCodeAt(i + 1) === 47) { i = lineCommentEnd(source, i + 2); continue; } if (c === 47 && source.charCodeAt(i + 1) === 42) { const end = source.indexOf('*/', i + 2); i = end < 0 ? n : end + 2; continue; } if (c === 35 && lineStart) { // A directive runs to the end of its line, through `\` continuations // and comments. Only `"` strings are skipped in it: an apostrophe in // `#error Don't …` is no character literal. let j = i + 1; while (j < n && isBlank(source.charCodeAt(j))) j++; const nameEnd = wordEnd(source, j); const directive = nameEnd < 0 ? '' : source.slice(j, nameEnd); while (j < n) { const d = source.charCodeAt(j); if (d === NEWLINE) { let k = j - 1; if (source.charCodeAt(k) === 13) k--; if (source.charCodeAt(k) !== 92) break; j++; } else if (d === 47 && source.charCodeAt(j + 1) === 47) { j = lineCommentEnd(source, j + 2); } else if (d === 47 && source.charCodeAt(j + 1) === 42) { const end = source.indexOf('*/', j + 2); j = end < 0 ? n : end + 2; } else if (d === 34) { j = quotedEnd(source, j); } else { j++; } } if (directive === 'if' || directive === 'ifdef' || directive === 'ifndef') { branches.push({ atIf: open.slice(), afterFirst: null }); } else if ( (directive === 'else' || directive === 'elif' || directive === 'elifdef' || directive === 'elifndef') && branches.length > 0 ) { const group = branches[branches.length - 1]!; if (group.afterFirst === null) group.afterFirst = open; open = group.atIf.slice(); } else if (directive === 'endif' && branches.length > 0) { const group = branches.pop()!; if (group.afterFirst !== null) open = group.afterFirst; } i = j; continue; } lineStart = false; if (c === 34) { // A raw string (`R"x(…)x"`) can hold any brace and quote; its prefix is // the word that ends right at the quote. if (wordEndAt === i && i - wordStart <= 3 && RAW_STRING_PREFIX.test(source.slice(wordStart, i))) { const paren = source.indexOf('(', i + 1); const delimiter = paren < 0 ? '' : source.slice(i + 1, paren); if (paren >= 0 && RAW_STRING_DELIMITER.test(delimiter)) { const end = source.indexOf(`)${delimiter}"`, paren + 1); i = end < 0 ? n : end + delimiter.length + 2; continue; } } i = quotedEnd(source, i); continue; } if (c === 39) { i = quotedEnd(source, i); continue; } if (isDigit(c) || (c === 46 && isDigit(source.charCodeAt(i + 1)))) { // A number, digit separators (`1'000'000`) and exponent signs included. let j = i + 1; while (j < n) { const d = source.charCodeAt(j); if (isWordChar(d) || d === 46) j++; else if (d === 39 && isWordChar(source.charCodeAt(j + 1))) j += 2; else if ((d === 43 || d === 45) && /[eEpP]/.test(source[j - 1]!)) j++; else break; } wordStart = i; wordEndAt = j; i = j; continue; } if (isWordStart(c)) { const end = wordEnd(source, i); wordStart = i; wordEndAt = end; if (end - i === 9 && source.startsWith('namespace', i)) { const head = namespaceHead(source, end); if (head) pendingNamespace = head; } i = end; continue; } if (c === 123) { open.push(i); if (pendingNamespace && pendingNamespace.brace === i) { if (pendingNamespace.name) namespaceAt.set(i, pendingNamespace.name); namespaceBraces.add(i); pendingNamespace = null; } } else if (c === 125) { const from = open.pop(); if (from === undefined) balanced = false; else if (!closes.has(from)) closes.set(from, i); } i++; } if (!balanced || open.length > 0) return null; const namespaces = new NestedIntervals<string>(); for (const [brace, name] of [...namespaceAt].sort((a, b) => a[0] - b[0])) { const close = closes.get(brace); if (close !== undefined) namespaces.add(brace, close, name); } let lineStarts: number[] | null = null; let scopes: NestedIntervals<boolean> | null = null; return { closeOf: (at) => closes.get(at), namespacesAt: (offset) => namespaces.at(offset), atNamespaceScope: (offset) => { if (!scopes) { scopes = new NestedIntervals<boolean>(); for (const [brace, close] of [...closes].sort((a, b) => a[0] - b[0])) scopes.add(brace, close, namespaceBraces.has(brace)); } return scopes.at(offset).every((isNamespace) => isNamespace); }, positionOf: (offset) => { if (!lineStarts) { lineStarts = [0]; for (let k = source.indexOf('\n'); k >= 0; k = source.indexOf('\n', k + 1)) lineStarts.push(k + 1); } let lo = 0; let hi = lineStarts.length - 1; while (lo < hi) { const mid = (lo + hi + 1) >> 1; if (lineStarts[mid]! <= offset) lo = mid; else hi = mid - 1; } return { line: lo + 1, column: offset - lineStarts[lo]! }; }, }; } - src/extraction/languages/cpp-macro-methods.ts (22865 bytes)
/** * C++ methods a function-like macro defines, minted where the file invokes * the macro. * * protocolbuffers/protobuf defines most of its descriptor accessors this way. * `FieldDescriptor` declares `int number() const;`, and descriptor.h defines * it further down with `PROTOBUF_DEFINE_ACCESSOR(FieldDescriptor, number, * int)`, the macro being `inline TYPE CLASS::FIELD() const { return FIELD##_; }`. * The parser never sees that definition (the preParse blanks the invocation, * an unknown statement without a semicolon), and a declaration without a body * mints no node, so `FieldDescriptor::number` had no node at all and none of * its callers linked. * * This reads the file's own `#define`s whose body defines `Class::method(…) … {` * at its top level, expands each invocation at namespace scope (its * parameters, and `##` pasting), and mints a method node per definition, as the * walker mints an out-of-line definition: named and qualified like one (the * enclosing namespaces composed with the written receiver), and contained by * the file and by the same-file class of the receiver's name. The node spans * the invocation. Its signature is the expanded head (`inline const * FileDescriptor* Descriptor::file() const`), which the resolver reads where it * would read a definition's source: the return type a chained call goes on * through, and the parameters an overload is picked by. The expanded body is * not read, so the method has no calls of its own. * * It runs on the original source after either extractor (the kernel or the * wasm walker), so both mint the same nodes. Only macros the file defines * itself are expanded: an extraction sees one file. Constructors and * destructors are left out: constructor resolution reads a constructor's * parameter list as its signature, and a destructor is not called by name. */ import { Edge, ExtractionResult, Node } from '../../types'; import { generateNodeId } from '../tree-sitter-helpers'; import { normalizeCppReturnType, stripCppTemplateArgs } from './c-cpp'; import { scanCppBraceScopes } from './cpp-brace-scopes'; /** A function-like macro a file defines (not a variadic one: none defines a method in practice). */ interface CppMacro { name: string; params: string[]; /** Its replacement text: continuations spliced, comments blanked. */ body: string; /** Where its `#` is. */ at: number; } /** A method definition at the top level of a macro's expansion. */ interface CppDefinitionHead { /** The receiver as written, template arguments dropped (`Descriptor`, `internal::ExtensionSet`). */ receiver: string; name: string; /** The text before the receiver: specifiers and the return type (`inline const FileDescriptor*`). */ returnText: string; /** Everything up to the body's `{`, whitespace collapsed. */ head: string; } const isWordStart = (c: number): boolean => (c >= 97 && c <= 122) || (c >= 65 && c <= 90) || c === 95; const isWordChar = (c: number): boolean => isWordStart(c) || (c >= 48 && c <= 57); /** A function-like `#define` whose replacement, on any of its lines, writes `::` and a `{`. */ const METHOD_MACRO_CANDIDATE = /^[ \t]*#[ \t]*define[ \t]+[A-Za-z_]\w*\([^)\n]*\)((?:[^\n]*\\\r?\n)*[^\n]*)/gm; /** * Whether the C++ `source` may define a method with a macro of its own: a * cheap test of the raw text, so the bulk-index fast path, which never sees * this pass, is left only for a file that can't have one. */ export function mayDefineCppMacroMethods(source: string): boolean { if (!source.includes('define')) return false; METHOD_MACRO_CANDIDATE.lastIndex = 0; for (let m = METHOD_MACRO_CANDIDATE.exec(source); m; m = METHOD_MACRO_CANDIDATE.exec(source)) { if (m[1]!.includes('::') && m[1]!.includes('{')) return true; } return false; } /** Add the methods the file's own macros define (see the module comment) to an extraction result. */ export function appendCppMacroMethods(result: ExtractionResult, filePath: string, source: string, preParse: (source: string) => string): void { if (!mayDefineCppMacroMethods(source)) return; const minted = cppMacroMethods(filePath, source, preParse, result.nodes); result.nodes.push(...minted.nodes); result.edges.push(...minted.edges); } /** * The method nodes, and their `contains` edges, for every definition the * file's own method-defining macros expand to where the file invokes them at * namespace scope. `nodes` are the file's extracted nodes: the class a * method's receiver names, and the ids already taken. */ function cppMacroMethods( filePath: string, source: string, preParse: (source: string) => string, nodes: readonly Node[], ): { nodes: Node[]; edges: Edge[] } { const out = { nodes: [] as Node[], edges: [] as Edge[] }; const { macros, undefs } = cppMacroDirectives(source); const defining = macros.filter((m) => cppDefinitionHeads(m.body.replace(/\s*##\s*/g, '')).length > 0); if (defining.length === 0) return out; // Scopes as the parser saw them (after preParse, offsets unchanged), like the walker's. const scopes = scanCppBraceScopes(preParse(source)); if (!scopes) return out; const names = new Set(defining.map((m) => m.name)); const taken = new Set(nodes.map((n) => n.id)); const fileId = `file:${filePath}`; for (const call of cppMacroInvocations(source, names)) { if (!scopes.atNamespaceScope(call.at)) continue; const macro = liveMacro(defining, undefs, call.name, call.at); const expansion = macro && expandCppMacro(macro, call.args); if (!expansion) continue; const start = scopes.positionOf(call.at); const end = scopes.positionOf(call.end); const prefix = scopes.namespacesAt(call.at); for (const def of cppDefinitionHeads(expansion)) { const receiverLast = def.receiver.split('::').pop()!; // A constructor's or destructor's. if (def.name === receiverLast || def.name.startsWith('~')) continue; let id = generateNodeId(filePath, 'method', def.name, start.line); if (taken.has(id)) id = `${id}:${start.column}`; for (let k = 1; taken.has(id); k++) id = `${generateNodeId(filePath, 'method', def.name, start.line)}:${start.column}:${k}`; taken.add(id); const returnType = normalizeCppReturnType(cppTypeText(def.returnText)); out.nodes.push({ id, kind: 'method', name: def.name, qualifiedName: composeQualifiedName(prefix, def.receiver, def.name), filePath, language: 'cpp', startLine: start.line, endLine: end.line, startColumn: start.column, endColumn: end.column, signature: def.head, ...(returnType ? { returnType } : {}), updatedAt: Date.now(), }); out.edges.push({ source: fileId, target: id, kind: 'contains' }); // As for an out-of-line definition, the class of its receiver's name in the same file holds it. const owner = nodes.find((n) => n.name === def.receiver && n.filePath === filePath && (n.kind === 'struct' || n.kind === 'union' || n.kind === 'class' || n.kind === 'enum' || n.kind === 'trait')); if (owner) out.edges.push({ source: owner.id, target: id, kind: 'contains' }); } } return out; } /** * The qualified name of a method written `receiver::name` inside the * namespaces `prefix`, as the walker composes an out-of-line definition's * (TreeSitterExtractor.composeReceiverQualifiedName): the receiver is anchored * at the first namespace it re-spells. */ function composeQualifiedName(prefix: readonly string[], receiver: string, name: string): string { const base = `${receiver}::${name}`; if (prefix.length === 0) return base; const anchor = prefix.indexOf(receiver.split('::')[0]!); const kept = anchor === -1 ? prefix : prefix.slice(0, anchor); return kept.length > 0 ? `${kept.join('::')}::${base}` : base; } /** The type a definition's leading text writes, without its specifiers, template header and attributes. */ function cppTypeText(returnText: string): string { let text = returnText.trim(); for (;;) { const before = text; if (text.startsWith('template')) { const open = text.indexOf('<'); const close = open < 0 ? -1 : matching(text, open, '<', '>'); if (close < 0) return ''; text = text.slice(close + 1).trim(); } text = text.replace(/^\[\[[^\]]*\]\]\s*/, '') .replace(/^(?:inline|static|virtual|constexpr|consteval|explicit|extern|friend|__forceinline)\b\s*/, '') // An attribute macro in front (`PROTOBUF_NOINLINE`, `ABSL_MUST_USE_RESULT`). .replace(/^[A-Z][A-Z0-9]*_[A-Z0-9_]*(?:\s*\([^()]*\))?\s+/, ''); if (text === before) return text; } } /** Where the `close` matching the `open` at `text[from]` is, or -1. */ function matching(text: string, from: number, open: string, close: string): number { let depth = 0; for (let i = from; i < text.length; i++) { const c = text[i]!; if (c === '"' || c === "'") i = quotedEnd(text, i) - 1; else if (c === open) depth++; else if (c === close && --depth === 0) return i; } return -1; } /** Past the string or character literal whose quote is at `i`. */ function quotedEnd(text: string, i: number): number { const quote = text[i]; let j = i + 1; while (j < text.length && text[j] !== quote && text[j] !== '\n') j += text[j] === '\\' ? 2 : 1; return Math.min(text.length, j + 1); } /** Past the blanks at `i`. */ function skipBlank(text: string, i: number): number { while (i < text.length && /\s/.test(text[i]!)) i++; return i; } /** The end of the identifier at `i`, or -1. */ function wordEnd(text: string, i: number): number { if (!isWordStart(text.charCodeAt(i))) return -1; let j = i + 1; while (j < text.length && isWordChar(text.charCodeAt(j))) j++; return j; } /** * The qualified name starting at `i` that an argument list follows * (`Descriptor::file(`, `internal::ExtensionSet::Has(`, `Box<T>::get(`, * `Matrix::operator()(`): its segments and the offset of the `(`. Null when no * such name starts there, or it has no `::`. */ function qualifiedCallableAt(text: string, i: number): { segments: string[]; open: number } | null { const segments: string[] = []; let j = i; if (text.startsWith('::', j)) j = skipBlank(text, j + 2); for (;;) { if (text.startsWith('operator', j) && !isWordChar(text.charCodeAt(j + 8))) { const op = /^operator\s*(\(\s*\)|\[\s*\]|[^\s\w(]+|\w+)\s*\(/.exec(text.slice(j)); if (!op || segments.length === 0) return null; segments.push(`operator${op[1]!.replace(/\s+/g, '')}`); return { segments, open: j + op[0].length - 1 }; } const tilde = text[j] === '~'; const end = wordEnd(text, tilde ? skipBlank(text, j + 1) : j); if (end < 0) return null; const word = text.slice(tilde ? skipBlank(text, j + 1) : j, end); segments.push(tilde ? `~${word}` : word); let k = skipBlank(text, end); if (text[k] === '<' && !tilde) { const close = matching(text, k, '<', '>'); if (close < 0) return null; segments[segments.length - 1] += text.slice(k, close + 1); k = skipBlank(text, close + 1); } if (text.startsWith('::', k)) { if (tilde) return null; j = skipBlank(text, k + 2); continue; } return text[k] === '(' && segments.length > 1 ? { segments, open: k } : null; } } /** * After a definition's parameter list (from `i`): the offset of its body's * `{`, past the qualifiers it may write (`const`, `noexcept(…)`, `override`, * `&&`, an attribute, an attribute macro, a trailing return type). -1 for * anything else: a declaration (`;`), `= default`, a constructor's * initializer list. */ function bodyOpenAfter(text: string, i: number): number { let j = skipBlank(text, i); for (;;) { const c = text[j]; if (c === '{') return j; if (c === '&') { j = skipBlank(text, j + (text[j + 1] === '&' ? 2 : 1)); continue; } if (text.startsWith('[[', j)) { const close = text.indexOf(']]', j + 2); if (close < 0) return -1; j = skipBlank(text, close + 2); continue; } if (text.startsWith('->', j)) { // A trailing return type runs to the body. let depth = 0; for (let k = j + 2; k < text.length; k++) { const d = text[k]; if (d === '(' || d === '<' || d === '[') depth++; else if (d === ')' || d === '>' || d === ']') depth--; else if (d === '{' && depth <= 0) return k; else if (d === ';' && depth <= 0) return -1; } return -1; } const end = wordEnd(text, j); if (end < 0) return -1; const word = text.slice(j, end); if (!/^(?:const|volatile|override|final|noexcept|throw|[A-Z][A-Z0-9]*_[A-Z0-9_]*)$/.test(word)) return -1; j = skipBlank(text, end); if (text[j] === '(' && word !== 'const' && word !== 'volatile' && word !== 'override' && word !== 'final') { const close = matching(text, j, '(', ')'); if (close < 0) return -1; j = skipBlank(text, close + 1); } } } /** The method definitions at the top level of a macro's expansion `text`, in order. */ function cppDefinitionHeads(text: string): CppDefinitionHead[] { const heads: CppDefinitionHead[] = []; let segment = 0; // where the current top-level declaration began for (let i = 0; i < text.length; ) { const c = text[i]!; if (c === '"' || c === "'") { i = quotedEnd(text, i); continue; } if (c === ';' || c === '}') { segment = ++i; continue; } if (c === '{' || c === '(') { // A body or group that is no definition read here (a namespace, a class, a statement). const close = matching(text, i, c, c === '{' ? '}' : ')'); if (close < 0) return heads; if (c === '{') segment = close + 1; i = close + 1; continue; } const callable = isWordStart(text.charCodeAt(i)) || (c === ':' && text[i + 1] === ':') ? qualifiedCallableAt(text, i) : null; if (!callable) { const end = wordEnd(text, i); i = end < 0 ? i + 1 : end; continue; } const close = matching(text, callable.open, '(', ')'); if (close < 0) return heads; const open = bodyOpenAfter(text, close + 1); if (open < 0) { i = close + 1; continue; } // An explicit specialization's arguments (`Read<int32_t, TYPE_INT32>`) are no part of its name. const last = callable.segments[callable.segments.length - 1]!; const name = last.startsWith('operator') ? last : stripCppTemplateArgs(last); const receiver = stripCppTemplateArgs(callable.segments.slice(0, -1).join('::')); if (receiver) { heads.push({ receiver, name, returnText: text.slice(segment, i).trim(), head: text.slice(segment, open).replace(/\s+/g, ' ').trim(), }); } const bodyClose = matching(text, open, '{', '}'); if (bodyClose < 0) return heads; segment = i = bodyClose + 1; } return heads; } - src/extraction/languages/csharp.ts (7855 bytes)
import type { Node as SyntaxNode } from 'web-tree-sitter'; import { getNodeText } from '../tree-sitter-helpers'; import type { LanguageExtractor } from '../tree-sitter-types'; /** * Blank C# conditional-compilation directive lines (`#if` / `#elif` / `#else` / * `#endif`) before parsing. The vendored tree-sitter-c-sharp grammar mis-parses * a `#if` that appears *inside an enum member list* — the canonical * multi-targeting shape: * * enum ReadType { * #if HAVE_DATE_TIME_OFFSET * ReadAsDateTimeOffset, * #endif * ReadAsDouble, * } * * It emits an ERROR that, for a nested enum, detaches the *enclosing class's* * member list, so most of the class's methods drop out of the index. Removing * the directive lines (keeping the guarded code) sidesteps it. Both branches of * an `#if/#else` are kept — the same behaviour the previous grammar produced, * and the right default for a code graph (index every symbol regardless of * build flags). Replacement preserves byte offsets (directive text → spaces, * newlines kept) so every symbol's line/column stays exact. (#237) */ export function blankCsharpPreprocessorDirectives(source: string): string { if (source.indexOf('#') === -1) return source; // Conditional-compilation directives only. `#region`/`#pragma`/`#nullable` // parse fine and are left alone. A directive must be the first non-space token // on its line (C# requirement), so anchor to line start. const re = /^([ \t]*)#[ \t]*(if|elif|else|endif)\b[^\n]*/gm; return source.replace(re, (m, indent) => indent + ' '.repeat(m.length - indent.length)); } /** * A C# method's declared return type, normalized to the bare class name a chained * `Foo.Create().Bar()` could be called on (the #645/#608 mechanism). The return * type lives in the `returns` field (`static Foo Create()` → `Foo`); built-in * `predefined_type` (void/int/string/…) and arrays yield undefined, generics are * unwrapped to the base type, nullable `Foo?` is stripped, and a dotted namespace * is reduced to the simple name. Constructors have no `returns` field → undefined. */ function extractCsharpReturnType(node: SyntaxNode, source: string): string | undefined { const typeNode = node.childForFieldName('returns'); if (!typeNode) return undefined; if (typeNode.type === 'predefined_type' || typeNode.type === 'array_type') return undefined; let t = getNodeText(typeNode, source).trim(); t = t.replace(/\?+$/, ''); // nullable `Foo?` t = t.replace(/<[^>]*>/g, ''); // generics `List<Foo>` → `List` const last = t.split('.').pop()?.trim(); // namespace `Ns.Foo` → `Foo` if (!last || !/^[A-Za-z_]\w*$/.test(last)) return undefined; return last; } /** Where a C# declaration with no access modifier defaults to `internal`. */ const CSHARP_TYPE_SCOPES = new Set(['compilation_unit', 'namespace_declaration', 'file_scoped_namespace_declaration']); export const csharpExtractor: LanguageExtractor = { preParse: blankCsharpPreprocessorDirectives, functionTypes: [], // Records are first-class type declarations in modern C# (DTOs, value objects, // MediatR/CQRS messages). Without these, references to a record never resolve // (#237). The shipped grammar parses EVERY record form as record_declaration — // `record struct` / `readonly record struct` included (it has no // record_struct_declaration node; that structTypes entry is forward-compat // only) — so classifyClassNode tells the value-type form apart by its // `struct` keyword child. (#831 follow-up) classTypes: ['class_declaration', 'record_declaration'], methodTypes: ['method_declaration', 'constructor_declaration'], interfaceTypes: ['interface_declaration'], structTypes: ['struct_declaration', 'record_struct_declaration'], classifyClassNode: (node) => { if (node.type === 'record_declaration') { for (let i = 0; i < node.childCount; i++) { if (node.child(i)?.type === 'struct') return 'struct'; } } return 'class'; }, enumTypes: ['enum_declaration'], enumMemberTypes: ['enum_member_declaration'], typeAliasTypes: [], // Namespaces qualify type names so same-named types in different namespaces are // distinguishable (e.g. `ApplicationCore.Entities.CatalogBrand` vs // `BlazorShared.Models.CatalogBrand`). A file-scoped `namespace Foo;` covers the // whole file (extractFilePackage); a block `namespace Foo { … }` only its body // (the visitor scopes it). packageTypes: ['file_scoped_namespace_declaration'], extractPackage: (node: SyntaxNode, source: string) => { const name = node.childForFieldName('name') ?? node.namedChildren.find((c: SyntaxNode) => c.type === 'qualified_name' || c.type === 'identifier'); return name ? getNodeText(name, source) : null; }, importTypes: ['using_directive'], callTypes: ['invocation_expression'], variableTypes: ['local_declaration_statement'], fieldTypes: ['field_declaration'], propertyTypes: ['property_declaration'], nameField: 'name', bodyField: 'body', paramsField: 'parameters', returnField: 'type', getReturnType: extractCsharpReturnType, getVisibility: (node) => { for (let i = 0; i < node.childCount; i++) { const child = node.child(i); if (child?.type === 'modifier') { const text = child.text; if (text === 'public') return 'public'; if (text === 'private') return 'private'; if (text === 'protected') return 'protected'; if (text === 'internal') return 'internal'; } } // No modifier: an interface's members are public, a type declared in a // namespace (or the file) is internal, and a class or struct member is private. const container = node.parent?.type === 'declaration_list' ? node.parent.parent : node.parent; if (container?.type === 'interface_declaration') return 'public'; if (!container || CSHARP_TYPE_SCOPES.has(container.type)) return 'internal'; return 'private'; }, isStatic: (node) => { for (let i = 0; i < node.childCount; i++) { const child = node.child(i); if (child?.type === 'modifier' && child.text === 'static') { return true; } } return false; }, // `const` and `static readonly` fields are C# constants (`MaxItems`, lookup // tables, shared config). Drives `constant` kind so value-reference edges // target them; instance `readonly` / plain `static` fields stay `field`s. isConst: (node) => { let hasStatic = false; let hasReadonly = false; for (let i = 0; i < node.childCount; i++) { const child = node.child(i); if (child?.type !== 'modifier') continue; const t = child.text; if (t === 'const') return true; if (t === 'static') hasStatic = true; else if (t === 'readonly') hasReadonly = true; } return hasStatic && hasReadonly; }, isAsync: (node) => { for (let i = 0; i < node.childCount; i++) { const child = node.child(i); if (child?.type === 'modifier' && child.text === 'async') { return true; } } return false; }, extractImport: (node, source) => { const importText = source.substring(node.startIndex, node.endIndex).trim(); // C# using directives: using System, using System.Collections.Generic, using static X, using Alias = X const qualifiedName = node.namedChildren.find((c: SyntaxNode) => c.type === 'qualified_name'); if (qualifiedName) { return { moduleName: getNodeText(qualifiedName, source), signature: importText }; } // Simple namespace like "using System;" - get the first identifier const identifier = node.namedChildren.find((c: SyntaxNode) => c.type === 'identifier'); if (identifier) { return { moduleName: getNodeText(identifier, source), signature: importText }; } return null; }, }; - src/extraction/languages/dart.ts (53449 bytes)
import type { Node as SyntaxNode } from 'web-tree-sitter'; import type { UnresolvedReference } from '../../types'; import { getNodeText, BUILTIN_TYPE_NAMES } from '../tree-sitter-helpers'; import type { LanguageExtractor } from '../tree-sitter-types'; /** * Whether a Dart `type_identifier` names a type a reference should point at. * Dart writes types in UpperCamelCase (`_Private`, `$Generated` too), so a * lowercase one is a built-in like `num` or `dynamic` — or no type at all: * error recovery around syntax the grammar predates (dot shorthands like * `.leading`) turns argument names into `type_identifier`s. Two more are not * types of their own: the import prefix of `p.Foo` (it names a library), and * the class a `new` / `const` constructor call names (the call links it). */ export function isDartTypeName(node: SyntaxNode): boolean { if (!/^[_$]*[A-Z]/.test(node.text) || BUILTIN_TYPE_NAMES.has(node.text)) return false; if (node.nextSibling?.type === '.') return false; const parent = node.parent; return parent?.type !== 'new_expression' && parent?.type !== 'const_object_expression'; } /** * A `references` ref from `fromNodeId` for every type named inside `node` — a * declared type, a generic argument, a cast or a type test (#2327). */ export function pushDartTypeRefs( node: SyntaxNode, fromNodeId: string, push: (ref: UnresolvedReference) => void, ): void { if (node.type === 'type_identifier') { if (isDartTypeName(node)) { push({ fromNodeId, referenceName: node.text, referenceKind: 'references', line: node.startPosition.row + 1, column: node.startPosition.column, }); } return; } for (const child of node.namedChildren) pushDartTypeRefs(child, fromNodeId, push); } /** * The name a typedef declares. The grammar gives `type_alias` no fields, and * the first type name is not always it: the legacy form writes its return * type first (`typedef Report LegacyMaker(int n);`). The name is the last type * name before the `=`, the type parameters or the parameter list. */ export function dartTypeAliasName(node: SyntaxNode): SyntaxNode | undefined { let name: SyntaxNode | undefined; for (let i = 0; i < node.childCount; i++) { const child = node.child(i); if (!child || child.type === '=' || child.type === 'type_parameters' || child.type === 'formal_parameter_list') break; if (child.type === 'type_identifier') name = child; } return name; } /** `scope` and the type parameters `node` declares, when it declares any (`Mapper<T>`, `T Function<T>(T a)`). */ function dartTypeParameterScope(node: SyntaxNode, scope: ReadonlySet<string>): ReadonlySet<string> { const parameters = node.namedChildren.find((c: SyntaxNode) => c.type === 'type_parameters'); if (!parameters) return scope; const inner = new Set(scope); for (const parameter of parameters.namedChildren) { const name = parameter.type === 'type_parameter' ? parameter.namedChildren.find((c: SyntaxNode) => c.type === 'type_identifier') : undefined; if (name) inner.add(name.text); } return inner; } /** pushDartTypeRefs for a typedef's aliased type: a type parameter in `scope` is no type of its own. */ function pushDartAliasedTypeRefs( node: SyntaxNode, fromNodeId: string, scope: ReadonlySet<string>, push: (ref: UnresolvedReference) => void, ): void { if (node.type === 'type_identifier') { // `p.T` is the type an import names, whatever is declared as `T` here. const parameter = scope.has(node.text) && node.previousSibling?.type !== '.'; if (!parameter && isDartTypeName(node)) { push({ fromNodeId, referenceName: node.text, referenceKind: 'references', line: node.startPosition.row + 1, column: node.startPosition.column, }); } return; } const inner = dartTypeParameterScope(node, scope); for (const child of node.namedChildren) pushDartAliasedTypeRefs(child, fromNodeId, inner, push); } /** * A `references` ref from the typedef `node` for every type its aliased type * names (#2327): a function type's return and parameter types, generic * arguments, record fields, a type parameter's bound, and the legacy form's * return and parameter types. A type parameter, the typedef's own or a * generic function type's (`T Function<T>(T a)`), means that parameter and no * type of the project's. The typedef's annotations are not its type. */ export function pushDartTypeAliasRefs( node: SyntaxNode, fromNodeId: string, push: (ref: UnresolvedReference) => void, ): void { const name = dartTypeAliasName(node); const scope = dartTypeParameterScope(node, new Set()); for (const child of node.namedChildren) { if (child.type === 'annotation' || (name && child.equals(name))) continue; pushDartAliasedTypeRefs(child, fromNodeId, scope, push); } } /** The bodies a Dart field `declaration` can sit in: a class's or mixin's, an extension's, an enum's. */ const DART_MEMBER_BODIES: ReadonlySet<string> = new Set(['class_body', 'extension_body', 'enum_body']); /** The named nodes a top-level variable's declared type is written as (`Report? x;`, `List<Report> xs = [];`). */ const DART_TOP_LEVEL_TYPES: ReadonlySet<string> = new Set(['type_identifier', 'type_arguments', 'function_type', 'record_type']); /** * Whether an `initialized_identifier` declares a field or a top-level * variable — `var cache = load();`, `final _ctl = TextEditingController();` * in a class — rather than the second variable of a local declaration * (`for (var i = 0, j = n(); …)`), which a function body's walk covers. */ function isDartFieldOrTopLevelEntry(node: SyntaxNode): boolean { const list = node.parent; if (list?.type !== 'initialized_identifier_list') return false; const owner = list.parent; if (owner?.type === 'program') return true; return owner?.type === 'declaration' && owner.parent !== null && DART_MEMBER_BODIES.has(owner.parent.type); } /** The comments tree-sitter-dart keeps as named nodes: line and block comments, and dartdoc. */ const DART_COMMENTS: ReadonlySet<string> = new Set(['comment', 'documentation_comment']); /** * The named sibling before `node`, past any comments. A Dart member chain is * an identifier followed by sibling selectors, and the grammar keeps a comment * as a sibling wherever it is written, so in `tester //` + newline + * `.state(…)` (the empty `//` keeps dart format from joining the lines) the * comment, not `tester`, is right before the `.state` selector. */ function dartPrevNamed(node: SyntaxNode): SyntaxNode | null { let prev = node.previousNamedSibling; while (prev && DART_COMMENTS.has(prev.type)) prev = prev.previousNamedSibling; return prev; } /** The named sibling after `node`, past any comments (see dartPrevNamed). */ function dartNextNamed(node: SyntaxNode): SyntaxNode | null { let next = node.nextNamedSibling; while (next && DART_COMMENTS.has(next.type)) next = next.nextNamedSibling; return next; } /** * The receiver of the member access `selector`: the node in front of it, past * any comments, or the end of an arrow function's body when the grammar ended * the arrow in front of a generic call — `(ref) => ref.watch<int>(p)` comes * out as `((ref) => ref).watch<int>(p)`. Dart writes a member access on the * arrow itself in parentheses, so a receiver in that place is never the * arrow. It names the call (`ref.watch`) and a static member's class; a call * chained on that call keeps its bare name (see dartMisparsedGenericCall). */ export function dartReceiverOf(selector: SyntaxNode): SyntaxNode | null { const prev = dartPrevNamed(selector); const body = prev?.type === 'function_expression' ? prev.lastNamedChild : null; if (body?.type !== 'function_expression_body') return prev; // The arrow can end on an operator's operand, which is the receiver: `(s) => // s.id == context.read<Cubit>().id` comes out as `((s) => s.id == // context).read<Cubit>()…`, and `() => !context.watch<Bloc>().busy` as // `(() => !context).watch<Bloc>()…`. let end = body.lastNamedChild; while (end && DART_OPERATOR_EXPRESSIONS.has(end.type)) end = end.lastNamedChild; return end; } /** The expressions an arrow function's body can end in with the operand on the right last. */ const DART_OPERATOR_EXPRESSIONS: ReadonlySet<string> = new Set([ 'equality_expression', 'relational_expression', 'logical_and_expression', 'logical_or_expression', 'if_null_expression', 'additive_expression', 'multiplicative_expression', 'bitwise_and_expression', 'bitwise_or_expression', 'bitwise_xor_expression', 'shift_expression', 'unary_expression', 'await_expression', ]); /** Dart's lowercase built-in types, which a type argument can name: `<int>`, `<void>`. */ const DART_LOWERCASE_TYPES: ReadonlySet<string> = new Set(['int', 'double', 'num', 'bool', 'dynamic', 'void']); /** What a callee can sit under in front of a generic call: `await`, `-`, `!`. */ const DART_PREFIXED: ReadonlySet<string> = new Set(['unary_expression', 'await_expression']); /** A generic call tree-sitter-dart parsed as two comparisons (dartMisparsedGenericCall). */ interface DartMisparsedCall { /** The node the callee ends with: what a parsed call's argument part follows. */ callee: SyntaxNode; /** The identifier naming the type argument (`Repo` in `<Repo>` and `<p.Repo>`). */ typeName: SyntaxNode; } /** * A generic call tree-sitter-dart parsed as two comparisons, found from its * `<`. `ref.read<Repo>(repoProvider)`, `Provider<int>((ref) => 0)` and * `DropdownButton<String>(items: items)` come out as `(ref.read < Repo) > * (repoProvider)`: a `relational_expression` inside another, with the * arguments as a `parenthesized_expression`, or a `record_literal` when they * are named. With tree-sitter-dart d4d8f3e that happened about as often as * the call parsed as one, because the grammar broke the tie by the code * around it; 0038280 parses them as calls, but for a type argument list split * over lines in a few places (#3270). Dart reads it as a call, * and so does this when the type argument names a type (`Repo`, `p.Repo`, * `Map<K, V>`, `int`, `void`) and the code is laid out as a call: `<` against * the callee and `(` against the `>`. A comparison is written `a < b`. * * A call chained on it keeps the bare name it had (`increment` in * `BlocProvider.of<CounterCubit>(context).increment()`), where a parsed * chain gets `BlocProvider.of().increment`. The resolver gives both the same * method: it types a bare link from the chain written before it, and an * encoded chain the same way when `of` declares no type of its own to return * (`static T of<T>(…)`, or a factory outside the project) (#750). */ export function dartMisparsedGenericCall(lt: SyntaxNode): DartMisparsedCall | undefined { if (lt.type !== 'relational_operator' || lt.firstChild?.type !== '<') return undefined; const inner = lt.parent; const outer = inner?.parent; if (inner?.type !== 'relational_expression' || outer?.type !== 'relational_expression') return undefined; const left = outer.namedChild(0); if (left?.startIndex !== inner.startIndex || left.endIndex !== inner.endIndex) return undefined; const gt = outer.namedChild(1); if (gt?.type !== 'relational_operator' || gt.firstChild?.type !== '>') return undefined; const args = outer.namedChild(2); if (args?.type !== 'parenthesized_expression' && args?.type !== 'record_literal') return undefined; if (args.startIndex !== gt.endIndex) return undefined; // A comment against the `<` (`ref.read /* c */<Repo>(p)`) is a sibling of // its own, so the callee is not against it: not laid out as a call. const before = lt.previousNamedSibling; if (!before || DART_COMMENTS.has(before.type) || before.endIndex !== lt.startIndex) return undefined; // The type argument: `T`, `p.T`, `T<…>` or `p.T<…>`, and nothing else. const head = lt.nextNamedSibling; if (head?.type !== 'identifier') return undefined; let typeName = head; let rest = head.nextNamedSibling; if (rest?.type === 'selector' && rest.firstNamedChild?.type === 'unconditional_assignable_selector') { const name = rest.firstNamedChild.namedChildren.find((c: SyntaxNode) => c.type === 'identifier'); if (!name) return undefined; typeName = name; rest = rest.nextNamedSibling; } if (rest?.type === 'selector' && rest.namedChildCount === 1 && rest.firstNamedChild?.type === 'type_arguments') { rest = rest.nextNamedSibling; } if (rest) return undefined; if (!/^[_$]*[A-Z]/.test(typeName.text) && !DART_LOWERCASE_TYPES.has(typeName.text)) return undefined; // `await repo.load<User>(id)` and `-x.size<int>(y)` put the callee under the prefix. let callee: SyntaxNode = before; while (DART_PREFIXED.has(callee.type)) { const last: SyntaxNode | null = callee.lastNamedChild; if (!last) return undefined; callee = last; } return { callee, typeName }; } /** * Whether a member access — `.member` on `receiver`, followed by `next` under * `parent` — belongs to a misparsed generic call instead of reading a member: * its callee (`ref.read` in `ref.read<Repo>(p)`, after an `await` too) is * called, and the prefixed type argument (`p.Repo` in `x.read<p.Repo>(y)`) * names a type. So do the import prefix and a prefixed type argument of a * generic constructor call the grammar misparsed (dartMisparsedCtorIn). */ function dartInMisparsedGenericCall( receiver: SyntaxNode, next: SyntaxNode | null, parent: SyntaxNode | null, ): boolean { let after = next; let up = parent; while (!after && up && DART_PREFIXED.has(up.type)) { after = up.nextNamedSibling; up = after ? null : up.parent; } if (after?.type === 'relational_operator') { if (dartMisparsedGenericCall(after)) return true; // `p.Box<int>.named(…)`, under an `await` too: the `<` after the prefix. const rel = after.parent; if (rel && dartMisparsedCtorIn(rel, receiver)) return true; } if (parent?.type !== 'relational_expression') return false; const before = receiver.previousNamedSibling; if (before?.type === 'relational_operator' && dartMisparsedGenericCall(before) !== undefined) return true; return dartMisparsedCtorIn(parent, receiver); } /** A generic constructor call tree-sitter-dart parsed as comparisons and a dot shorthand (dartMisparsedCtor). */ export interface DartMisparsedCtor { /** Where the call starts, as a parsed one does: the type's name, or its import prefix. */ head: SyntaxNode; /** The identifier naming the type (`Future` in `Future<void>.delayed(d)`). */ type: SyntaxNode; /** The constructor's name (`delayed`). */ name: SyntaxNode; /** The identifiers naming a type argument that the grammar read as an expression. */ typeArgs: SyntaxNode[]; /** The `>` closing the type arguments, inside the relational expression holding the call. */ gt: SyntaxNode; /** The relational expres - src/extraction/languages/erlang.ts (16188 bytes)
import type { Node as SyntaxNode } from 'web-tree-sitter'; import { getNodeText, getChildByField, getPrecedingDocstring, namedArgCount, } from '../tree-sitter-helpers'; import type { LanguageExtractor, ExtractorContext } from '../tree-sitter-types'; // Node names follow the vendored WhatsApp/tree-sitter-erlang grammar (0.19, // ABI 14) — the grammar behind the Erlang Language Platform (ELP). // // Erlang is form-based, and three of its shapes don't fit the generic // extractor, so every symbol-bearing top-level form is dispatched through the // visitNode hook below instead: // - a function's name lives on its CLAUSE, not the fun_decl, and the grammar // emits one fun_decl PER CLAUSE — consecutive same-name same-ARITY // fun_decl forms (clauses of one function) are merged into a single // function node here. Arity is part of an Erlang function's identity // (`f/1` and `f/2` are unrelated definitions — #1610), so each arity gets // its own node, qualified `mod::f/1` / `mod::f/2`; // - type-position expressions (-spec/-type/-callback bodies, record field // types) parse as `call` nodes, so descending into them would mint bogus // call refs to type names (`pid()`, `term()`); the hook consumes those // subtrees; // - record_decl carries its fields as direct children (no body field), which // the generic extractStruct would skip as a forward declaration. // Calls (local `f(X)`, remote `mod:f(X)`, `fun f/1` references, and record // usages) are handled by the erlang branch in extractCall — remote calls are // emitted as `mod::f/2` (arity counted at the call site), byte-identical to // the qualifiedName above, so cross-module resolution rides the standard // qualified-name matcher; local calls are emitted `f/2` and resolved by the // erlang arity step in matchReference. /** Text of an atom with quoted-atom quotes stripped (`'EXIT'` → `EXIT`). */ function atomText(node: SyntaxNode, source: string): string { return getNodeText(node, source).replace(/^'([\s\S]*)'$/, '$1'); } function collapseWs(text: string): string { return text.replace(/\s+/g, ' ').trim(); } // --- Per-file memos. Extraction is file-sequential within a worker, so a // single-entry memo keyed by filePath is safe (and resets naturally). --- /** * Exported `name/arity` keys for the current file ('all' for * -compile(export_all)). Keyed by arity because `-export([f/1])` exports * exactly f/1 — f/2 in the same module stays private (#1610). A malformed * `fa` with no arity node falls back to the bare name key. */ let exportsFile = ''; let exportsMemo: Set<string> | 'all' = new Set(); /** * Clause-merge state: the previous fun_decl's name, arity, and node id. A * fun_decl whose clause repeats that (name, arity) is a continuation clause of * the SAME function and attaches to the existing node instead of creating a * duplicate. A same-name DIFFERENT-arity fun_decl is an unrelated function * (Erlang identity is `name/arity`) and gets its own node (#1610). Keying on * adjacency stays safe: clauses of one function must be adjacent in Erlang — * a non-adjacent redefinition of the same name/arity is a compile error. */ let lastFnFile = ''; let lastFnName = ''; let lastFnArity = -1; let lastFnId = ''; function moduleExports(node: SyntaxNode, source: string, filePath: string): Set<string> | 'all' { if (filePath === exportsFile) return exportsMemo; let root: SyntaxNode = node; while (root.parent) root = root.parent; let result: Set<string> | 'all' = new Set<string>(); for (let i = 0; i < root.namedChildCount; i++) { const form = root.namedChild(i); if (!form) continue; if ( form.type === 'compile_options_attribute' && getNodeText(form, source).includes('export_all') ) { result = 'all'; break; } if (form.type === 'export_attribute') { for (const fa of form.namedChildren) { if (fa.type !== 'fa') continue; const fun = getChildByField(fa, 'fun'); if (!fun) continue; const name = atomText(fun, source); const arityNode = getChildByField(fa, 'arity'); const arityValue = arityNode ? getChildByField(arityNode, 'value') : null; const arity = arityValue ? getNodeText(arityValue, source) : null; result.add(arity !== null ? `${name}/${arity}` : name); } } } exportsFile = filePath; exportsMemo = result; return result; } /** Argument count of a clause/sig: the `args` (expr_args) field's arguments. */ function nodeArity(withArgs: SyntaxNode): number { return namedArgCount(getChildByField(withArgs, 'args')); } /** * The -spec directly above a function (comments may sit between), if it names * it AND matches its arity — the spec for `header/3` sitting between the * `header/2` and `header/3` definitions must attach to /3 only (#1610). A * spec whose sigs can't be read (defensive) is accepted on the name alone. */ function precedingSpec(node: SyntaxNode, name: string, arity: number, source: string): SyntaxNode | null { let prev = node.previousNamedSibling; while (prev && prev.type === 'comment') prev = prev.previousNamedSibling; if (prev?.type === 'spec') { const specFun = getChildByField(prev, 'fun'); if (specFun && atomText(specFun, source) === name) { const sigs = prev.namedChildren.filter((c) => c.type === 'type_sig'); if (sigs.length === 0 || sigs.some((sig) => nodeArity(sig) === arity)) return prev; } } return null; } /** `name(Args) when Guard` — the clause text up to the `->`. */ function clauseHeader(clause: SyntaxNode, source: string): string | undefined { const body = getChildByField(clause, 'body'); const end = body ? body.startIndex : clause.endIndex; return collapseWs(source.substring(clause.startIndex, end)) || undefined; } function handleFunDecl(node: SyntaxNode, ctx: ExtractorContext): boolean { const clauses = node.namedChildren.filter((c) => c.type === 'function_clause'); const first = clauses[0]; if (!first) return true; // macro-templated clause (`?M(...) -> ...`) — no static name const nameNode = getChildByField(first, 'name'); if (!nameNode) return true; const name = atomText(nameNode, ctx.source); if (!name) return true; const arity = nodeArity(first); // Continuation clause of the SAME function (same name AND arity): extend the // existing node's span and attribute this clause's calls to it. if (ctx.filePath === lastFnFile && name === lastFnName && arity === lastFnArity && lastFnId) { for (let i = ctx.nodes.length - 1; i >= 0; i--) { const n = ctx.nodes[i]; if (n && n.id === lastFnId) { if (node.endPosition.row + 1 > n.endLine) n.endLine = node.endPosition.row + 1; break; } } ctx.pushScope(lastFnId); for (const clause of clauses) ctx.visitFunctionBody(clause, lastFnId); ctx.popScope(); return true; } const spec = precedingSpec(node, name, arity, ctx.source); const exports = moduleExports(node, ctx.source, ctx.filePath); const fn = ctx.createNode('function', name, node, { docstring: getPrecedingDocstring(spec ?? node, ctx.source), signature: spec ? collapseWs(getNodeText(spec, ctx.source)).slice(0, 300) : clauseHeader(first, ctx.source), isExported: exports === 'all' || exports.has(`${name}/${arity}`) || exports.has(name), }); if (!fn) return true; // Arity is part of the function's identity — carry it on the qualified name // (`mod::f/2`), the canonical Erlang spelling and the only persisted slot. // The node NAME stays bare so name search and bare-name matching still work. fn.qualifiedName = `${fn.qualifiedName}/${arity}`; ctx.pushScope(fn.id); // The whole clause is walked (not just the body) so record patterns in the // arguments and guard calls contribute references too. for (const clause of clauses) ctx.visitFunctionBody(clause, fn.id); ctx.popScope(); lastFnFile = ctx.filePath; lastFnName = name; lastFnArity = arity; lastFnId = fn.id; return true; } function handleRecordDecl(node: SyntaxNode, ctx: ExtractorContext): boolean { const nameNode = getChildByField(node, 'name'); if (!nameNode) return true; const rec = ctx.createNode('struct', atomText(nameNode, ctx.source), node, { docstring: getPrecedingDocstring(node, ctx.source), signature: collapseWs(getNodeText(node, ctx.source)).slice(0, 300), }); if (rec) { ctx.pushScope(rec.id); for (const field of node.namedChildren) { if (field.type !== 'record_field') continue; const fieldName = getChildByField(field, 'name'); if (fieldName) ctx.createNode('field', atomText(fieldName, ctx.source), field); } ctx.popScope(); } return true; // field types/defaults are type-position exprs — don't descend } function handleTypeAlias(node: SyntaxNode, ctx: ExtractorContext): boolean { const typeName = getChildByField(node, 'name'); // type_name wrapper const nameNode = typeName ? getChildByField(typeName, 'name') : null; if (nameNode) { ctx.createNode('type_alias', atomText(nameNode, ctx.source), node, { signature: collapseWs(getNodeText(node, ctx.source)).slice(0, 200), }); } return true; } function handlePpDefine(node: SyntaxNode, ctx: ExtractorContext): boolean { const lhs = getChildByField(node, 'lhs'); const nameNode = lhs ? getChildByField(lhs, 'name') : null; if (!nameNode) return true; const macro = ctx.createNode('constant', getNodeText(nameNode, ctx.source), node, { signature: collapseWs(getNodeText(node, ctx.source)).slice(0, 200), }); // The replacement's calls execute at expansion sites, but attributing them // to the MACRO node keeps them true exactly once: `-define(LOG_AUDIT(E), // audit_logger:log(E))` gives the LOG_AUDIT constant a `calls` edge to the // logger, and each `?LOG_AUDIT(...)` use site links to the constant (see the // macro_call_expr case in extractCall) — so the chain // `caller → LOG_AUDIT → audit_logger:log` traverses without minting a // per-use duplicate of the body's calls. const replacement = getChildByField(node, 'replacement'); if (macro && replacement) { ctx.pushScope(macro.id); ctx.visitFunctionBody(replacement, macro.id); ctx.popScope(); } return true; } function handleBehaviour(node: SyntaxNode, ctx: ExtractorContext): boolean { const nameNode = getChildByField(node, 'name'); const parentId = ctx.nodeStack[ctx.nodeStack.length - 1]; if (nameNode && parentId) { // `-behaviour(x)` implements x's callback contract. Resolves when the // behaviour module is in the repo; OTP behaviours (gen_server, …) simply // stay unresolved. ctx.addUnresolvedReference({ fromNodeId: parentId, referenceName: atomText(nameNode, ctx.source), referenceKind: 'implements', line: node.startPosition.row + 1, column: node.startPosition.column, }); } return true; } /** * OTP application resource file (`<app>.app.src` / `<app>.app`): a single * `{application, Name, Props}.` term the grammar parses as a top-level * expression. Two properties carry graph structure — `{mod, {Mod, _Args}}` * names the application-callback module (the app's entry point), and * `{applications, [...]}` / `{included_applications, [...]}` declare the apps * this one depends on. In an umbrella repo those resolve to the sibling app's * module of the same name (the OTP convention); kernel/stdlib and other * out-of-repo apps stay unresolved. */ function handleAppResourceTuple(node: SyntaxNode, ctx: ExtractorContext): boolean { const parentId = ctx.nodeStack[ctx.nodeStack.length - 1]; const props = node.namedChildren[2]; if (!parentId || props?.type !== 'list') return true; const ref = (nameNode: SyntaxNode, kind: 'references' | 'imports'): void => { const name = atomText(nameNode, ctx.source); if (!name) return; ctx.addUnresolvedReference({ fromNodeId: parentId, referenceName: name, referenceKind: kind, line: nameNode.startPosition.row + 1, column: nameNode.startPosition.column, }); }; for (const prop of props.namedChildren) { if (prop.type !== 'tuple' || prop.namedChildren.length < 2) continue; const key = prop.namedChildren[0]; const value = prop.namedChildren[1]; if (!key || key.type !== 'atom' || !value) continue; const keyName = atomText(key, ctx.source); if (keyName === 'mod' && value.type === 'tuple') { const mod = value.namedChildren[0]; if (mod?.type === 'atom') ref(mod, 'references'); } else if ( (keyName === 'applications' || keyName === 'included_applications') && value.type === 'list' ) { for (const app of value.namedChildren) { if (app.type === 'atom') ref(app, 'imports'); } } } return true; // nothing else in an app term carries graph structure } export const erlangExtractor: LanguageExtractor = { functionTypes: ['fun_decl'], // dispatched via visitNode (name lives on the clause) classTypes: [], methodTypes: [], interfaceTypes: [], structTypes: ['record_decl'], // dispatched via visitNode (fields are direct children) enumTypes: [], typeAliasTypes: ['type_alias', 'opaque'], // dispatched via visitNode importTypes: ['import_attribute', 'pp_include', 'pp_include_lib'], callTypes: [ 'call', 'internal_fun', // fun f/1 'external_fun', // fun mod:f/1 'record_expr', // #rec{...} construction 'record_update_expr', // X#rec{...} 'record_index_expr', // #rec.field 'record_field_expr', // X#rec.field 'macro_call_expr', // ?MACRO / ?MACRO(...) — links use sites to the -define constant ], variableTypes: [], nameField: 'name', bodyField: 'body', paramsField: 'args', // `-module(m)` wraps the file's declarations in a namespace so every // function's qualifiedName is `m::f` — which is exactly the reference shape // the extractCall erlang branch emits for remote calls, so `mod:f(...)` // resolves through matchByQualifiedName with no resolver changes. packageTypes: ['module_attribute'], extractPackage: (node, source) => { const name = getChildByField(node, 'name'); return name ? atomText(name, source) : null; }, extractImport: (node, source) => { if (node.type === 'import_attribute') { const mod = getChildByField(node, 'module'); if (!mod) return null; return { moduleName: atomText(mod, source), signature: collapseWs(getNodeText(node, source)).slice(0, 200), }; } // pp_include / pp_include_lib — a C-include-style file dependency on a .hrl. const file = getChildByField(node, 'file'); if (!file) return null; const headerPath = getNodeText(file, source).replace(/^"/, '').replace(/"$/, ''); if (!headerPath) return null; return { moduleName: headerPath, signature: getNodeText(node, source).trim() }; }, visitNode: (node, ctx) => { switch (node.t - src/extraction/languages/go.ts (13015 bytes)
import type { Node as SyntaxNode } from 'web-tree-sitter'; import { getNodeText, getChildByField } from '../tree-sitter-helpers'; import type { LanguageExtractor } from '../tree-sitter-types'; /** * A Go function's declared return type, normalized to the bare type a chained * `New().Method()` could be called on (the #645/#608 mechanism). Reads the * `result` field: a pointer `*Foo` is unwrapped to `Foo`, a multi-return * `(*Foo, error)` takes the first result (the idiomatic value-or-error shape), * a qualified `pkg.Foo` reduces to its last segment, and generics to the base. * Built-ins / unnamed results simply fail the later existence check. */ function extractGoReturnType(node: SyntaxNode, source: string): string | undefined { let result = getChildByField(node, 'result'); if (!result) return undefined; // Multi-return `(T, error)` → the first result's type. if (result.type === 'parameter_list') { const first = result.namedChildren.find((c: SyntaxNode) => c.type === 'parameter_declaration'); if (!first) return undefined; result = getChildByField(first, 'type') ?? first; } // Unwrap a pointer `*Foo` → `Foo`. if (result?.type === 'pointer_type') { result = result.namedChildren.find( (c: SyntaxNode) => c.type === 'type_identifier' || c.type === 'qualified_type' || c.type === 'generic_type', ) ?? result; } if (!result) return undefined; const text = getNodeText(result, source) .trim() .replace(/^\*/, '') .replace(/<[^>]*>/g, '') .replace(/\[[^\]]*\]/g, ''); // strip generic args `Foo[T]` const last = text.split('.').pop()?.trim(); // qualified `pkg.Foo` → `Foo` if (!last || !/^[A-Za-z_]\w*$/.test(last)) return undefined; return last; } /** * Go's predeclared types. None is ever a node in the graph, and as the lone * term of an interface (`interface{ int64 }`) a basic type is a type-set * constraint, not an embedded interface. */ const GO_PREDECLARED_TYPES: ReadonlySet<string> = new Set([ 'any', 'bool', 'byte', 'comparable', 'complex64', 'complex128', 'error', 'float32', 'float64', 'int', 'int8', 'int16', 'int32', 'int64', 'rune', 'string', 'uint', 'uint8', 'uint16', 'uint32', 'uint64', 'uintptr', ]); /** * The name node of the type a Go embedding names, in a struct or an interface: * `Base`, `*Base` (the `*` is a sibling token), `pkg.Base` and `Base[T]` all * embed `Base`. A qualified type yields its name, not its package: the package * stays in the source text, and resolution reads it back from the reference's * position (#2322). Undefined for any other type (a literal, `~T`, a union's * term) and for a predeclared one but `error`: an interface with a method, * which an embedding brings in like any type's from outside the index. */ export function goEmbeddedTypeName(type: SyntaxNode | null | undefined, source: string): SyntaxNode | undefined { if (type?.type === 'generic_type') type = getChildByField(type, 'type'); if (type?.type === 'qualified_type') return getChildByField(type, 'name') ?? undefined; if (type?.type !== 'type_identifier') return undefined; const name = getNodeText(type, source); return name === 'error' || !GO_PREDECLARED_TYPES.has(name) ? type : undefined; } /** * The names a type name written bare inside a Go type declaration can mean * without naming another type: the declaration's own type parameters, and * its own name, which in a recursive type is the declaration itself. */ function goDeclaredNames(decl: SyntaxNode, source: string): Set<string> { const names = goTypeParameterNames(decl, source); const nameNode = getChildByField(decl, 'name'); if (nameNode) names.add(getNodeText(nameNode, source)); return names; } /** The type parameters a Go type declaration declares: `T` and `K` in `type Store[T any, K comparable] …`. */ function goTypeParameterNames(decl: SyntaxNode, source: string): Set<string> { const names = new Set<string>(); for (const param of getChildByField(decl, 'type_parameters')?.namedChildren ?? []) { for (const c of param.namedChildren) { if (c.type === 'identifier') names.add(getNodeText(c, source)); } } return names; } /** * Collects the name nodes of the types `type` names onto `names`, in source * order, but Go's predeclared types and, written bare, the `declared` names * of the declaration it is written in. */ function collectGoTypeNames(type: SyntaxNode, declared: ReadonlySet<string>, source: string, names: SyntaxNode[]): void { // `qualified`: the name of a `pkg.Name`, which is that package's whatever // this one declares. const walk = (n: SyntaxNode, qualified: boolean): void => { if (n.type === 'type_identifier') { const text = getNodeText(n, source); if (!GO_PREDECLARED_TYPES.has(text) && (qualified || !declared.has(text))) names.push(n); return; } for (const c of n.namedChildren) walk(c, n.type === 'qualified_type'); }; walk(type, false); } /** * The name nodes of the types a Go `type_alias` node names on its right-hand * side, in source order: an alias's (`Event` in `type Event = mvccpb.Event`, * `List` and `Event` in `type Page = List[Event]`) and a defined type's * (`WatchResponse` in `type WatchChan <-chan WatchResponse`, `Context` in * `type HandlerFunc func(*Context)`), each where it is written, so resolution * reads a package qualifier back. Not types it names: the declaration's own * type parameters (`T` in `type Items[T any] = List[T]`), Go's predeclared * types, and its own name written bare, which in a recursive type (`type * stateFn func(*Lexer) stateFn`) is the declaration itself. A generic alias * has no rule in tree-sitter-go 0.23: it parses as a `type_spec` whose `=` is * an error, and reads as any other declaration here. */ export function goAliasTypeNames(node: SyntaxNode, source: string): SyntaxNode[] { const type = getChildByField(node, 'type'); if (!type) return []; const names: SyntaxNode[] = []; collectGoTypeNames(type, goDeclaredNames(node, source), source, names); return names; } /** * The name nodes of the types a Go struct's fields are declared with, in * source order: `ResponseHeader` in etcd's `Header *ResponseHeader`, * `HandlersChain` in gin's `handlers HandlersChain`, and every type a field's * function, map, channel or anonymous struct type names. `node` declares the * struct (a `type_spec`, or a `type_alias` of a struct literal). An embedded * field's type is a supertype (goEmbeddedTypeName), but the types it is * instantiated with are not: `Pod` in client-go's * `listers.ResourceIndexer[*v1.Pod]`. Skipped as in goAliasTypeNames: the * struct's own type parameters, the predeclared types, and its own name * written bare, which a linked list's `next *node` names. */ export function goStructFieldTypeNames(node: SyntaxNode, source: string): SyntaxNode[] { const fields = getChildByField(node, 'type')?.namedChildren.find((c: SyntaxNode) => c.type === 'field_declaration_list'); const names: SyntaxNode[] = []; if (!fields) return names; const declared = goDeclaredNames(node, source); for (const field of fields.namedChildren) { if (field.type !== 'field_declaration') continue; let type = getChildByField(field, 'type'); // Embedded: no field name (extractInheritance's test). if (type && !field.namedChildren.some((c: SyntaxNode) => c.type === 'field_identifier')) { type = type.type === 'generic_type' ? getChildByField(type, 'type_arguments') : null; } if (type) collectGoTypeNames(type, declared, source, names); } return names; } /** * The name nodes of the types a Go interface method's signature names, in * source order: `OpOption` and `PutResponse` in etcd's `Put(ctx * context.Context, key, val string, opts ...OpOption) (*PutResponse, error)`, * and `Context`, whose package stays in the source. `decl` declares the * interface. Skipped: Go's predeclared types and, written bare, the * interface's own type parameters (`T` in `type Store[T any] interface`). Its * own name is a type the method names (`Children() []Node`). */ export function goInterfaceMethodTypeNames(method: SyntaxNode, decl: SyntaxNode, source: string): SyntaxNode[] { const params = goTypeParameterNames(decl, source); const names: SyntaxNode[] = []; for (const field of ['parameters', 'result']) { const part = getChildByField(method, field); if (part) collectGoTypeNames(part, params, source, names); } return names; } export const goExtractor: LanguageExtractor = { functionTypes: ['function_declaration'], classTypes: [], // Go doesn't have classes methodTypes: ['method_declaration'], interfaceTypes: [], // Handled via type_spec → resolveTypeAliasKind structTypes: [], // Handled via type_spec → resolveTypeAliasKind enumTypes: [], typeAliasTypes: ['type_spec', 'type_alias'], // `type A B` and `type A = B` importTypes: ['import_declaration'], callTypes: ['call_expression'], variableTypes: ['var_declaration', 'short_var_declaration', 'const_declaration'], methodsAreTopLevel: true, nameField: 'name', bodyField: 'body', paramsField: 'parameters', returnField: 'result', getReturnType: extractGoReturnType, getSignature: (node, source) => { const params = getChildByField(node, 'parameters'); const result = getChildByField(node, 'result'); if (!params) return undefined; let sig = getNodeText(params, source); if (result) { sig += ' ' + getNodeText(result, source); } return sig; }, resolveTypeAliasKind: (node, _source) => { // Go type_spec: `type Foo struct { ... }` or `type Bar interface { ... }`, // and an alias of a literal (`type Foo = struct { ... }`) alike. // The inner type is in the 'type' field of the type_spec node const typeChild = getChildByField(node, 'type'); if (!typeChild) return undefined; if (typeChild.type === 'struct_type') return 'struct'; if (typeChild.type === 'interface_type') return 'interface'; return undefined; }, isExported: (node, source) => { // Go: a symbol is exported when its identifier starts with an uppercase letter. // Look at the `name` field directly: function_declaration, // method_declaration, type_spec, and a var_spec / const_spec, whose first // name is the one the extractor gives a node (never the declaration // around a spec, which has no name). const nameNode = getChildByField(node, 'name'); if (nameNode) { const text = getNodeText(nameNode, source); const first = text.charCodeAt(0); return first >= 65 && first <= 90; // A-Z } return false; }, getReceiverType: (node, source) => { // Go method_declaration has a "receiver" field: func (sl *scrapeLoop) run(...) // The receiver is a parameter_list containing a parameter_declaration // with a type that may be a pointer_type (*scrapeLoop) or plain type (scrapeLoop) const receiver = getChildByField(node, 'receiver'); if (!receiver) return undefined; // Find the type identifier inside the receiver const text = getNodeText(receiver, source); // Extract type name from "(sl *Type)", "(sl Type)", "(*Type)", "(Type)" and // generic receivers "(s *Stack[T])". Anchor on the opening "(" and skip an // optional receiver var name; the old `name)`-anchored pattern never matched // the `[T])` suffix, so generic-type methods were orphaned from their type // (no struct→method `contains` edge). (#583) const match = text.match(/\(\s*(?:[A-Za-z_]\w*\s+)?\*?\s*([A-Za-z_]\w*)/); return match?.[1]; }, // A type declared on its own, `type Foo struct{…}`, is a `type_declaration` // holding one spec, and its doc comment comes before the declaration, outside // the spec. A member of a `type ( … )` group has its comment beside it in the // parentheses, and the comment above the group is the group's. A group of one // is that type's declaration, as go doc reads it, unless its member has a // comment of its own. getDeclarationWrapper: (node) => { const parent = node.parent; if (parent?.type !== 'type_declaration' || !parent.firstNamedChild?.equals(node)) return undefined; const specs = parent.namedChildren.filter( (c: SyntaxNode) => c.type === 'type_spec' || c.type === 'type_alias' ); return specs.length === 1 ? parent : undefined; }, // A comment after code on its line (`const sides = 4 // sides of a square.`) // is that line's, never the doc of the declaration below it. docstringSkipsTrailingComments: true, // A doc comment ends on the line right above its declaration, and a blank // line ends it, as go doc reads it: a banner or a note above that blank line // documents nothing. docstringSkipsDetachedComments: true, // A directive Go tools read, like `//go:embed static` or // `//nolint:errcheck`, is not part of the doc comment go doc shows. docstringSkipsDirectives: true, }; - TELEMETRY.md (7634 bytes)
# Telemetry CodeGraph collects a small set of **anonymous usage statistics** — which commands and tools get used, which languages get indexed, which agents drive usage — so we can tell which of the 20+ languages and 8 agent integrations deserve the most work. This page is the complete list of what is collected. If a field isn't on this page, it isn't collected; the ingest endpoint enforces this list as an allowlist and is itself [public, auditable code](telemetry-worker/) in this repository. ## Turning it off Any of these works, permanently: ```bash codegraph telemetry off # stores your choice (and deletes any unsent data) ``` ```bash export CODEGRAPH_TELEMETRY=0 # per-shell / per-CI override export DO_NOT_TRACK=1 # the cross-tool standard — always honored ``` For one MCP server entry, add `--no-telemetry` to its arguments (`"args": ["serve", "--mcp", "--no-telemetry"]`). It works like `CODEGRAPH_TELEMETRY=0` for that server and for the background daemon it starts. A daemon that is already running for the project, started by a server without the flag, keeps its own setting until it exits; `codegraph telemetry off` covers every process. `codegraph telemetry status` shows the current state, what decided it, and your machine ID. The interactive installer (`codegraph install`) asks up front with a visible default-on toggle and never re-asks. If you never saw the installer (e.g. `npx` straight into `init`), a one-line notice is printed to stderr before the first time anything is sent. Off means off: when disabled, CodeGraph records nothing, opens no connection to the telemetry endpoint, and sends no "opted out" ping. Running processes recheck the stored choice before recording, persisting, and each send. Turning it off removes the local identity and unsent queues (including claimed queues); turning it back on creates a new identity. An HTTP request already started cannot be recalled, but opt-out prevents later request chunks and prevents its unsent data from being requeued. Environment overrides still apply: `CODEGRAPH_TELEMETRY=1` explicitly forces telemetry on for that process even when the stored choice is off; `DO_NOT_TRACK=1` takes precedence. Separately from telemetry, the MCP server checks GitHub for a newer release in the background (at most once a day) so it can tell you an update exists — it fetches a version number and sends nothing about you or your machine. `DO_NOT_TRACK=1` disables this check too; to turn off only the update check, use `CODEGRAPH_NO_UPDATE_CHECK=1`. ## What is collected Every payload carries this envelope: | field | example | notes | |---|---|---| | `machine_id` | `b3a8c1…` | random UUID minted on first send — derived from nothing | | `codegraph_version` | `0.9.9` | | | `os` / `arch` | `darwin` / `arm64` | platform identifiers only | | `node_major` | `22` | major version only | | `ci` | `false` | whether the `CI` env var was set | | `schema_version` | `2` | bumped when this page changes (v2 dropped the `index` event's `sqlite_backend` field) | And one of four events: - **`install`** — when `codegraph install` configures agents: which agents (`["claude","cursor",…]`), global vs project-local, and whether it was a fresh install, an upgrade, or a re-run. - **`index`** — when a full index completes: the **language names** present (e.g. `["typescript","go"]`), the file count as a **coarse bucket** (`<100`, `100-1k`, `1k-10k`, `10k+`), and the duration as a bucket (`<10s`, `10-60s`, `1-5m`, `5m+`). - **`usage_rollup`** — one line per day per tool: the tool or CLI command **name** (e.g. `codegraph_explore`, `init`), how many times it ran, how many errored, and — for MCP tools — the connecting agent's name and version from the MCP handshake (e.g. `Claude Code 2.1`). The Claude Code prompt hook also counts its **gate decision** (fired fully, fired as a hint, or did nothing — fixed counter names like `prompt-hook-gate-medium-segment`); the prompt itself is never read, stored, or sent. - **`uninstall`** — when `codegraph uninstall`/`uninit` runs: which agents were removed. Usage is **aggregated locally into daily totals** before anything is sent — there is no per-call event stream, and nothing is sent in real time. ### The browser viewer sends nothing `codegraph ui` (the local viewer) has no telemetry of its own. The server it starts makes no outbound connections at all, and the page in your browser talks only to that server on `127.0.0.1`: nothing about the symbols you open, the searches you type, or the path you walk leaves your machine, and none of it is recorded anywhere. The only thing telemetry ever learns about the viewer is what it learns about every command: that a command named `ui` was run, once, on a day, in the daily `usage_rollup` above. The command never triggers a send of its own, and `codegraph telemetry off`, `CODEGRAPH_TELEMETRY=0`, or `DO_NOT_TRACK=1` switches off even that count, as it does everything else on this page. ## What is never collected - **No source code.** No file paths, file names, directory names, repository names or URLs, symbol names, search queries, or anything else derived from the contents of an indexed project. - **No IP addresses.** The ingest endpoint never reads, logs, or stores the client IP — and there is no analytics vendor downstream that could. No geolocation. - **No fingerprinting.** The machine ID is a random UUID stored in `~/.codegraph/telemetry.json` — delete that file (or run `codegraph telemetry off`, then `on`) and the old ID is gone forever, with no way to reconnect it. - **No personal data.** No usernames, hostnames, emails, or environment variables. ## How it travels Events POST to `telemetry.getcodegraph.com` — a first-party endpoint whose complete source lives in [`telemetry-worker/`](telemetry-worker/) in this repository. It validates every event and property against the allowlist above (anything else is dropped), never reads the client IP, and rate-limits per machine ID. Sends are fire-and-forget with a short timeout: offline or air-gapped machines buffer a bounded local file (256 KB cap) and never retry-loop, log errors, or slow a command down. Recording refreshes the small local consent file, then increments an in-memory counter; MCP tool calls never wait for telemetry network requests or queue writes. ## Where it is stored Accepted events are written to **our own database on Cloudflare** (D1) and go nowhere else. **No third-party analytics vendor receives any of this data**, because the ingest endpoint makes no outbound requests at all — its source is the entire path your events take, and there is nothing after it. This is a stronger guarantee than a promise not to share: there is no second party to share with. What is kept is checkable rather than asserted. The storage schema — [`telemetry-worker/migrations/`](telemetry-worker/migrations/), checked in beside the endpoint that writes it — is the complete list of what a row can hold, with a comment on every column. Individual events are **deleted after 90 days**. What outlives them is anonymous daily totals: counts per day of things like operating system, version, and language, plus which days each machine ID was active so returning-user numbers survive. No event details, and still nothing that identifies a person or a codebase. The engineering contract behind all of this — including the rule that schema changes must update this page, the client, and the public endpoint in one PR — is in [`docs/design/telemetry.md`](docs/design/telemetry.md).
- tsconfig.json (952 bytes)
{ "compilerOptions": { "target": "ES2022", "module": "commonjs", "lib": ["ES2022"], "declaration": true, "declarationMap": true, "sourceMap": true, "removeComments": true, "outDir": "./dist", "rootDir": "./src", "strict": true, "noImplicitAny": true, "strictNullChecks": true, "strictFunctionTypes": true, "strictBindCallApply": true, "strictPropertyInitialization": true, "noImplicitThis": true, "alwaysStrict": true, "noUnusedLocals": true, "noUnusedParameters": true, "noImplicitReturns": true, "noFallthroughCasesInSwitch": true, "noUncheckedIndexedAccess": true, "esModuleInterop": true, "skipLibCheck": true, "forceConsistentCasingInFileNames": true, "resolveJsonModule": true, "paths": { "web-tree-sitter": ["./src/web-tree-sitter.d.ts"] } }, "include": ["src/**/*"], "exclude": ["node_modules", "dist", "__tests__"] }