AIStoreHub Package Vault: Vllm Ascend
Package Slug: vllm-ascend
Archetype: skill
Upstream Repository: https://github.com/vllm-project/vllm-ascend
Workspace Files & Specifications (68 files)
- .agents/README.md (4897 bytes)
# vLLM Ascend skills This directory contains the skills for vLLM Ascend. Note: Please copy the skills directory `.agents/skills` to `.claude/skills` if you want to use the skills in this repo with Claude code. ## Table of Contents - [vLLM Ascend skills](#vllm-ascend-skills) - [Table of Contents](#table-of-contents) - [vLLM Ascend Model Adapter Skill](#vllm-ascend-model-adapter-skill) - [What it does](#what-it-does) - [File layout](#file-layout) - [Quick start](#quick-start) - [Key constraints](#key-constraints) - [Two-stage validation](#two-stage-validation) - [vLLM Ascend Release Skill](#vllm-ascend-release-skill) - [What it does](#what-it-does-1) - [File layout](#file-layout-1) - [Getting started](#getting-started) - [Key guidelines](#key-guidelines) ## vLLM Ascend Model Adapter Skill Adapt and debug models for vLLM on Ascend NPU — covering both already-supported architectures and new models not yet registered in vLLM. ### What it does This skill guides an AI agent through a deterministic workflow to: 1. Triage a model checkpoint (architecture, quant type, multimodal capability). 2. Implement minimal code changes in `/vllm-workspace/vllm` and `/vllm-workspace/vllm-ascend`. 3. Validate via a two-stage gate (dummy fast gate + real-weight mandatory gate). 4. Deliver one signed commit with code, test config, and tutorial doc. ### File layout | File | Purpose | | ---- | ------- | | `SKILL.md` | Skill definition, constraints, and execution playbook | | `references/workflow-checklist.md` | Step-by-step commands and templates | | `references/troubleshooting.md` | Symptom-action pairs for common failures | | `references/fp8-on-npu-lessons.md` | FP8 checkpoint handling on Ascend | | `references/multimodal-ep-aclgraph-lessons.md` | VL, EP, and ACLGraph patterns | | `references/deliverables.md` | Required outputs and commit discipline | ### Quick start 1. Open a conversation with the AI agent inside the vllm-ascend dev container. 2. Invoke the skill (e.g. `/vllm-ascend-model-adapter`). 3. Provide the model path (default `/models/<model-name>`) and the originating issue number. 4. The agent follows the playbook in `SKILL.md` and produces a ready-to-merge commit. ### Key constraints - Never upgrade `transformers`. - Start `vllm serve` from `/workspace` (direct command, port 8000). - Dummy-only evidence is not sufficient — real-weight validation is mandatory. - Final delivery is exactly one signed commit in the current repo. ### Two-stage validation - **Stage A (dummy)**: fast architecture / operator / API path check with `--load-format dummy`. - **Stage B (real)**: real-weight loading, fp8/quant path, KV sharding, runtime stability. Both stages require request-level verification (`/v1/models` + at least one chat request), not just startup success. ## vLLM Ascend Release Skill Manage the complete release process for vLLM Ascend, from the release checklist issue to the post-release announcement. ### What it does This skill guides an AI agent through the end-to-end release workflow: 1. Create the release feedback issue and the release checklist issue. 2. Triage release-blocking bugs and identify must-merge PRs. 3. Check test coverage for new features and models, and scan Nightly CI status. 4. Assemble release notes from `release_notes/fragments/` and audit commits without fragments. 5. Update version references across the documentation. 6. Guide release execution and post-release steps, including the WeChat article. ### File layout | File | Purpose | | ---- | ------- | | `SKILL.md` | Phases, commands, and human decision points for the full release process | | `scripts/` | Release automation: checklist creation, bug/PR/test/nightly scans, commit fetch, release note aggregation, version updates, announcements | | `templates/` | Release checklist and feedback issue templates | | `references/version-files.yaml` | Files that need version updates and their update patterns | | `references/ref-past-release-notes-highlight.md` | Style and category reference for past release notes | ### Getting started 1. Open a conversation with the AI agent. 2. Invoke the skill (e.g. `/vllm-ascend-release`). 3. Provide the release version, branch, target date, and release manager. 4. Work through the phases in `SKILL.md`; the skill pauses for human review at the key decision points (bug triage, functional testing, release execution). ### Key guidelines - Release notes are assembled from `release_notes/fragments/`; the audit step lists commits without fragments so nothing user-facing is missed. The fragment principles live in `release_notes/README.md`. - Human oversight is required for bug prioritization, test results, and the final release decision. - Scripts are idempotent and safe to re-run; checklist updates replace a section instead of appending duplicates. - .agents/skills/vllm-ascend-model-adapter/agents/openai.yaml (43 bytes)
policy: allow_implicit_invocation: false
- .agents/skills/vllm-ascend-model-adapter/references/deliverables.md (2918 bytes)
# Deliverables ## Required outputs in current repo 1. One final signed commit (`git commit -sm ...`) containing the adaptation changes. 2. Chinese analysis report(精简但完整): - model architecture summary - incompatibility root causes - code changes and rationale - startup and inference verification evidence - feature status matrix(supported / unsupported / checkpoint-missing / not-applicable) - max model len: config theoretical vs runtime practical - dummy-vs-real validation matrix(what dummy proved / what only real proved) - false-ready cases and final resolution path(if any) - fallback ladder evidence(which fallback was tried, what changed) 3. Chinese compact runbook: - how to start server in `/workspace` (direct command, default `:8000`) - how to run OpenAI-compatible validation - optional eager fallback command - optional `TORCHDYNAMO_DISABLE=1` fallback command (if relevant) 4. Test config YAML at `tests/e2e/models/configs/<ModelName>.yaml` — must include `model_name`, `hardware`, `tasks` with accuracy metrics (name + value), and `num_fewshot`. Use accuracy results from evaluation to populate metric values. Follow the schema of existing configs (e.g. `Qwen3-8B.yaml`). 5. Tutorial doc at `docs/source/tutorials/models/<ModelName>.md` — must follow the standard template: Introduction, Supported Features, Environment Preparation (with docker tabs for A2/A3), Deployment (with serve script), Functional Verification (with curl example), Accuracy Evaluation, Performance. Fill in model-specific details (HF path, hardware requirements, TP size, max-model-len, served-model-name, sample curl, accuracy table). 6. Post SKILL.md content or AI-assisted workflow summary as a comment on the originating GitHub issue. ## Commit discipline - Keep one signed commit for code changes in the current working repo. - If implementation occurred in `/vllm-workspace/*`, backport minimal final diff to current repo before commit. - Keep diff scoped to target model adaptation. ## Validation discipline - Always provide log file paths for key claims. - Keep docs synchronized with latest successful test mode (do not leave stale command variants as default). - Final report must include pass/fail reason for each key feature attempt: ACLGraph / EP / flashcomm1 / MTP / multimodal. - EP and flashcomm1 are MoE-only checks; for non-MoE models mark as not-applicable with evidence. - Final report should include baseline capacity result (`128k + bs16`) or explicit reason if not feasible. - Dummy-first can be used to speed up iterations, but real-weight gate is mandatory before final sign-off. - Startup-only evidence is insufficient; include first-request smoke results. ## Suggested final response structure - What changed - What went well / what went wrong - Validation performed - Commit hash and changed files - Optional next step
- .agents/skills/vllm-ascend-model-adapter/references/fp8-on-npu-lessons.md (1757 bytes)
# FP8-on-NPU Lessons ## 1) Recommended debug order 1. Start with `--load-format dummy` to quickly verify architecture path. 2. Run with real weights to validate weight mapping and load-time stability. 3. If blocked by fp8 execution limits on NPU, use fp8->bf16 dequantization loading path. 4. Validate `/v1/models`, then one text request, then one VL request (if multimodal). ## 2) FP8 checkpoint on NPU Common symptom: - `fp8 quantization is currently not supported in npu`. Recommended pattern: - do not force fp8 execution kernels on NPU; - dequantize fp8 weights to bf16 during loading using paired tensors: - `*.weight` - `*.weight_scale_inv` - keep strict unpaired scale/weight checks to avoid silent corruption. ## 3) Typical real-only risks (dummy may not expose) - missing fp8 scale keys during real shard loading; - wrong weight remap path only triggered by real checkpoints; - KV/QK norm sharding mismatch under TP + replicated KV heads. ## 4) KV replication + TP pitfalls Typical symptom: - shape mismatch like `128 vs 64` when `tp_size > num_key_value_heads`. Recommended pattern: - detect KV-head replication explicitly; - use local norm/shard loader path for replicated KV heads; - avoid assuming uniform divisibility for all head dimensions. ## 5) ACLGraph stability for fp8-origin checkpoints Recommended pattern: - prefer `HCCL_OP_EXPANSION_MODE=AIV` when using graph mode; - keep practical capture sizes and re-test from small, stable shapes; - use `--enforce-eager` only as temporary isolation fallback. ## 6) Reporting discipline Always report both: - what dummy validated (fast gate), and - what only real weights validated (mandatory gate). Do not sign off fp8-on-NPU adaptation with dummy-only evidence. - .agents/skills/vllm-ascend-model-adapter/references/multimodal-ep-aclgraph-lessons.md (2141 bytes)
# Multimodal + EP + ACLGraph Lessons This note captures practical patterns that repeatedly matter for VL checkpoints on Ascend. ## 1) Out-of-box feature expectation Try best to validate key features by default: - ACLGraph - MTP - multimodal (if model supports VL) - EP (MoE models only) - flashcomm1 (MoE models only) If any feature fails, keep logs and explain the reason in the final report. For non-MoE models, EP/flashcomm1 should be marked not-applicable. ## 2) Validate in this order 1. Single text request success (`/v1/models` + `/v1/chat/completions`). 2. Single text+image request success. 3. Graph evidence (`Replaying aclgraph`) when graph mode is expected. 4. Capacity baseline: `128k + bs16`. 5. Concurrency expansion if needed (`32/64` suggested). ## 3) EP + graph startup expectations - Startup latency is much higher than eager due to: - compile warmup - graph capture rounds - multimodal encoder profiling - Do not treat slow startup as failure unless logs show hard errors. ## 4) Always distinguish two max lengths - **Theoretical max**: from model config (`max_position_embeddings`). - **Practical max**: largest value that actually starts and serves on current hardware + TP/EP settings. Report both values explicitly. ## 5) Multimodal testing with temporary layer reduction - Reducing `num_hidden_layers` can speed smoke tests. - This does **not** remove ViT structure itself. - Still require one full-layer validation before final sign-off. ## 6) Feature-status semantics Use four categories: - ✅ supported and verified - ❌ framework-level unsupported - ⚠️ checkpoint missing (weights/config do not provide feature) - N/A not-applicable (for example EP/flashcomm1 on non-MoE models) Typical examples: - flashcomm1 on non-MoE VL models is often N/A or ❌ depending on framework gate. - MTP may be ⚠️ checkpoint missing even if framework has code paths. ## 7) Keep docs and defaults aligned with latest success path - If EP+graph is validated and requested/expected, it should be the default runbook path. - Eager mode should be documented as fallback/troubleshooting only. - .agents/skills/vllm-ascend-model-adapter/references/troubleshooting.md (6673 bytes)
# Troubleshooting ## Direct run doesn't pick your code changes Symptoms: - `vllm serve` behavior still old after code edits. Actions: 1. Check runtime import path: ```bash python - <<'PY' import vllm print(vllm.__file__) PY ``` 2. Ensure edits were made under `/vllm-workspace/vllm` and/or `/vllm-workspace/vllm-ascend`. 3. Avoid PYTHONPATH-overlay workflow unless as temporary debugging fallback. ## Server fails to bind on `:8000` or fails with HCCL bind errors Symptoms: - Port bind fail on startup. - HCCL error like `Communication_Error_Bind_IP_Port(EJ0003)`. Actions: 1. Kill stale `vllm serve` processes. 2. Ensure `:8000` is free. 3. Retry clean startup before changing code. ## Startup appears "stuck" in graph mode Symptoms: - Process alive, but `curl /v1/models` not ready yet. - Logs show compile/graph capture messages for a long time. Actions: 1. Keep waiting until graph capture completes. 2. Look for `Capturing CUDA graphs ...` and `Graph capturing finished`. 3. Only declare failure after an explicit error or timeout window. ## False-ready: startup succeeds but first request crashes Symptoms: - `Application startup complete` exists. - `GET /v1/models` may return 200. - First text or VL request crashes workers/engine. Actions: 1. Always run at least one text smoke request immediately after ready. 2. For VL models, always run one text+image smoke request as well. 3. Treat first-request crash as runtime failure (do not mark as success). 4. Capture first runtime error signature and branch to targeted fallback. ## Architecture not recognized Symptoms: - `ValueError` or log shows unresolved architecture. Actions: 1. Verify `architectures` in model `config.json`. 2. Add mapping to `vllm/model_executor/models/registry.py`. 3. Ensure module and class names exactly match. ## Remote code import fails on transformers symbols Symptoms: - Missing class/function in current `transformers`. Actions: 1. Do not upgrade `transformers`. 2. Prefer native vLLM implementation. 3. If unavoidable, copy required modeling files from sibling transformers source. ## Weight loading key mismatch Symptoms: - Missing/unexpected key warnings during load. Actions: 1. Inspect checkpoint key prefixes. 2. Add explicit mapping logic. 3. Keep mapping minimal and auditable. 4. Re-test with full shards, not only tiny-layer smoke runs. ## FP8 checkpoint on Ascend A2/A3 (must dequant to bf16) Symptoms: - fp8 kernels unsupported or unstable on Ascend A2/A3. Actions: 1. Do not force fp8 quantization kernels on Ascend. 2. Use load-time fp8->bf16 dequantization path (weight + scale pairing). 3. Add strict unpaired scale/weight checks to avoid silent corruption. ## QK norm mismatch (KV heads / TP / head divisibility) Symptoms: - Shape mismatch like `128 vs 64` when `tp_size > num_key_value_heads`. - Similar mismatch when head topology is not cleanly divisible. Actions: 1. Detect KV-head replication case. 2. Use local `k_norm` shard path for replicated KV heads. 3. Avoid assumptions that all head dimensions split evenly under current TP. 4. Validate both normal and edge topology cases explicitly. ## MLA attention runtime failures after ready Symptoms: - First request fails with signatures like `AtbRingMLAGetWorkspaceSize` / `AtbRingMLA`. - May also show `aclnnFusedInferAttentionScoreV3 ... error code 561002`. Actions: 1. Reproduce with one minimal text request (deterministic payload). 2. Try eager isolation (`--enforce-eager`) once to verify whether issue is graph-only. 3. If eager still fails, prioritize model/backend code fix path (not runtime flags only). 4. Check `vllm-ascend` MLA/rope/platform implementation used by known-good runs. ## VL + TorchDynamo interpolate contiguous failure Symptoms: - `torch._dynamo.exc.TorchRuntimeError`. - Stack contains `torch.nn.functional.interpolate`. - Error contains `NPU contiguous operator only supported contiguous memory format`. Actions: 1. Add `TORCHDYNAMO_DISABLE=1` and retry with same serve args. 2. Validate both text and text+image after startup. 3. If this stabilizes startup and inference, record it as current fallback path. 4. Keep code-level fix exploration as next step, but do not block delivery if fallback is accepted. ## Multimodal processor signature mismatch (`skip_tensor_conversion`) Symptoms: - Early failure before engine ready. - `convert_to_tensors() got an unexpected keyword argument 'skip_tensor_conversion'`. Actions: 1. Identify processor compatibility mismatch (HF remote processor vs current transformers API). 2. Use text-only isolation (`--limit-mm-per-prompt '{"image":0,"video":0,"audio":0}'`) only to separate layers, not as final fix. 3. Expect potential follow-up core failures after bypassing processor path; keep logs for both layers. 4. Align to known-good model dispatch and processor compatibility implementation. ## Text-only isolation triggers meta tensor load errors Symptoms: - `NotImplementedError: Cannot copy out of meta tensor; no data!` - May occur after disabling multimodal prompt items. Actions: 1. Treat as secondary failure signature (after bypassing earlier MM-processor failure). 2. Do not assume text-only isolation is universally safe for all VL models. 3. Return to model-specific code-fix path with captured signatures. ## Config max length works on paper but not in runtime Symptoms: - `max_position_embeddings` is large, but service fails or OOM with that value. Actions: 1. Record config max (theoretical). 2. Find practical max by successful startup + serving under target TP/EP setup. 3. Report both values explicitly in docs. ## flashcomm1 / MTP confusion on VL checkpoints Symptoms: - flashcomm1 enabled but startup fails. - MTP expected but no effect. Actions: 1. Only validate flashcomm1 for MoE models; non-MoE mark as not-applicable. 2. Verify MTP from both config and weight index (`mtp/nextn` keys). 3. Mark unsupported vs checkpoint-missing clearly. ## ACL graph capture fails (507903) Symptoms: - `AclmdlRICaptureEnd ... 507903` - `rtStreamEndCapture ... invalidated stream capture sequence` Actions: 1. Prefer `HCCL_OP_EXPANSION_MODE=AIV` for graph capture stability. 2. Reduce shape pressure (`--max-model-len`) and retry. 3. Temporarily fallback `--enforce-eager` for isolation. ## API reachable but output quality odd Symptoms: - `/v1/models` works but output has template artifacts. Actions: 1. Use deterministic request (`temperature=0`, bounded `max_tokens`). 2. Verify endpoint (`/v1/chat/completions` vs `/v1/completions`) matches model template. 3. Confirm non-empty output and HTTP 200 before success declaration. - .agents/skills/vllm-ascend-model-adapter/references/workflow-checklist.md (8334 bytes)
# Workflow Checklist ## 0) Environment prerequisites Set these once per session. Defaults match the official vllm-ascend Docker image. ```bash # --- configurable paths (adjust if your layout differs) --- VLLM_SRC=/vllm-workspace/vllm # vLLM source root VLLM_ASCEND_SRC=/vllm-workspace/vllm-ascend # vllm-ascend source root WORK_DIR=/workspace # directory to run vllm serve from MODEL_ROOT=/models # parent directory of model checkpoints ``` Expected environment: - Hardware: Ascend A2 or A3 server - Software: official vllm-ascend Docker image (see `./Dockerfile` for full contents) - TP=16 typical for A3 (16-NPU), TP=8 typical for A2 (8-NPU) ## 1) Fast triage commands ```bash MODEL_PATH=${MODEL_ROOT}/<model-name> echo "MODEL_PATH=$MODEL_PATH" # model inventory ls -la "$MODEL_PATH" # architecture + quant hints rg -n "architectures|model_type|quantization_config|torch_dtype|max_position_embeddings|num_nextn_predict_layers|version|num_attention_heads|num_key_value_heads|num_experts" "$MODEL_PATH/config.json" # state-dict key layout hints (if index exists) ls -la "$MODEL_PATH"/*index*.json 2>/dev/null || true # model custom code (if exists) ls -la "$MODEL_PATH"/*.py 2>/dev/null || true ``` ## 2) Confirm implementation and delivery roots ```bash # implementation roots (fixed by Dockerfile) cd "$VLLM_SRC" && git status -s cd "$VLLM_ASCEND_SRC" && git status -s # runtime import source check (expect vllm-workspace path) python - <<'PY' import vllm print(vllm.__file__) PY # direct-run working directory cd "$WORK_DIR" && pwd # delivery root (current repo) cd <current-repo> git status -s ``` ## 3) Session hygiene (before rerun) ```bash # stop stale servers pkill -f "vllm serve|api_server|EngineCore" || true # confirm port 8000 is free netstat -ltnp 2>/dev/null | rg ':8000' || true ``` When user explicitly requests reset: ```bash cd "$VLLM_SRC" && git reset --hard && git clean -fd cd "$VLLM_ASCEND_SRC" && git reset --hard && git clean -fd ``` ## 4) New model onboarding checklist ```bash # architecture mapping check in vLLM rg -n "<ArchitectureClass>|registry" "$VLLM_SRC"/vllm/model_executor/models/registry.py # optional: inspect model config and weight index quickly cat "$MODEL_PATH/config.json" cat "$MODEL_PATH"/*index*.json 2>/dev/null || true ``` If architecture is missing/incompatible, minimally do: 1. Add model adapter under `$VLLM_SRC/vllm/model_executor/models/<new_model>.py`. 2. Add processor under `$VLLM_SRC/vllm/transformers_utils/processors/<new_model>.py` when needed. 3. Register architecture in `$VLLM_SRC/vllm/model_executor/models/registry.py`. 4. Add explicit loader/remap rules for checkpoint key patterns (qkv/norm/rope/fp8 scales). 5. Touch `$VLLM_ASCEND_SRC` only when backend-specific errors are confirmed. ## 5) Typical implementation touch points - `$VLLM_SRC/vllm/model_executor/models/<new_model>.py` - `$VLLM_SRC/vllm/transformers_utils/processors/<new_model>.py` - `$VLLM_SRC/vllm/model_executor/models/registry.py` - `$VLLM_ASCEND_SRC/vllm_ascend/...` (only if backend behavior requires it) ## 6) Syntax sanity checks ```bash python -m py_compile \ "$VLLM_SRC"/vllm/model_executor/models/<new_model>.py python -m py_compile \ "$VLLM_SRC"/vllm/transformers_utils/processors/<new_model>.py 2>/dev/null || true ``` ## 7) Two-stage serve templates (direct run, default `:8000`) ### Stage A: dummy fast gate (first try) ```bash cd "$WORK_DIR" MODEL_PATH=${MODEL_ROOT}/<model-name> HCCL_OP_EXPANSION_MODE=AIV \ VLLM_ASCEND_ENABLE_FLASHCOMM1=0 \ vllm serve "$MODEL_PATH" \ --served-model-name <served-name> \ --trust-remote-code \ --dtype bfloat16 \ --max-model-len <practical-max-len-or-131072> \ --tensor-parallel-size <TP-size> \ --max-num-seqs 16 \ --load-format dummy \ --port 8000 ``` ### Stage B: real-weight mandatory gate ```bash # remove this from Stage A: --load-format dummy ``` > Note: dummy is not equivalent to real weights. Real gate is mandatory before sign-off. ### EP + ACLGraph (feature-first, MoE only) ```bash # add to Stage B when model is MoE and validating EP: --enable-expert-parallel ``` ### flashcomm1 check (MoE only) ```bash # only evaluate flashcomm1 when model is MoE VLLM_ASCEND_ENABLE_FLASHCOMM1=1 ``` ### Eager fallback (isolation) ```bash # add to command for isolation only: --enforce-eager ``` ### TorchDynamo fallback (for VL interpolate-contiguous failures) ```bash # add env var when logs contain: # torch._dynamo.exc.TorchRuntimeError + interpolate + # "NPU contiguous operator only supported contiguous memory format" TORCHDYNAMO_DISABLE=1 ``` ## 8) Readiness + smoke checks (must verify true-ready) ```bash # readiness for i in $(seq 1 200); do curl -sf http://127.0.0.1:8000/v1/models >/tmp/models.json && break sleep 3 done # text smoke (required) curl -s http://127.0.0.1:8000/v1/chat/completions \ -H 'Content-Type: application/json' \ -d '{"model":"<served-name>","messages":[{"role":"user","content":"say hi"}],"temperature":0,"max_tokens":16}' # VL smoke (required for multimodal models) # send one text+image OpenAI-compatible request and require non-empty choices. ``` > `Application startup complete` alone is not success. If first request crashes, treat as runtime failure (false-ready). ## 9) Feature validation checklist (default out-of-box) 1. `GET /v1/models` returns 200. 2. Text request returns 200 and non-empty output. 3. If VL model: text+image request returns 200. 4. ACLGraph evidence exists (`Replaying aclgraph`) where expected. 5. EP path is validated only for MoE models; non-MoE must be marked not-applicable. 6. flashcomm1 is validated only for MoE models; non-MoE must be marked not-applicable. 7. MTP status verified from config + weight index (enabled vs checkpoint-missing). 8. Dummy-vs-real differences are explicitly reported (if any). 9. Any false-ready case is explicitly marked as failure (with log signature). ## 10) Fallback ladder (recommended order) 1. Keep same params and reproduce once to ensure deterministic failure signature. 2. Add `--enforce-eager` to isolate graph-capture influence. 3. For VL + dynamo/interpolate/contiguous failures, add `TORCHDYNAMO_DISABLE=1`. 4. For multimodal-processor suspicion, isolate text-only by: - `--limit-mm-per-prompt '{"image":0,"video":0,"audio":0}'` - then check whether failure moves from processor layer to model core. 5. If issue persists, map failure signature to known-good implementation and patch minimal code. ## 11) Capacity baseline + sweep - Baseline (single machine): **`max-model-len=128k` + `max-num-seqs=16`**. - If baseline passes, expand to `max-num-seqs=32/64` when requested. - If baseline cannot pass due hardware/runtime limits, report explicit root cause. ## 12) Delivery checklist ```bash # in current working repo (delivery root) git add <changed-files> git commit -sm "<message>" ``` Confirm: - one signed commit only - Chinese analysis + Chinese runbook present - feature status matrix included with pass/fail reason - dummy stage and real stage validation evidence included - false-ready cases (if any) documented with final fallback status ### Test config generation - Generate `tests/e2e/models/configs/<ModelName>.yaml` using accuracy results from evaluation. - Must include: `model_name` (HF path), `hardware` (e.g. "Atlas A2 Series"), `tasks` (list with `name` and `metrics` containing `name` + `value`), `num_fewshot`. - Follow the schema of existing configs (e.g. `Qwen3-8B.yaml`). ### Tutorial doc generation - Generate `docs/source/tutorials/models/<ModelName>.md` from the standard template. - Fill in model-specific details: HF path, hardware requirements, TP size, max-model-len, served-model-name, sample curl request, accuracy table. - Must include sections: Introduction, Supported Features, Environment Preparation (with docker tabs for A2/A3), Deployment (with serve script), Functional Verification (with curl example), Accuracy Evaluation, Performance. - Update `docs/source/tutorials/models/index.md` to include the new tutorial entry. ### GitHub issue comment - Post SKILL.md content or AI-assisted workflow summary as a comment on the originating GitHub issue. Confirm both test config YAML and tutorial doc are included in the signed commit. - .agents/skills/vllm-ascend-model-adapter/SKILL.md (8748 bytes)
--- name: vllm-ascend-model-adapter description: "Adapt and debug existing or new models for vLLM on Ascend NPU. Implement in /vllm-workspace/vllm and /vllm-workspace/vllm-ascend, validate via direct vllm serve from /workspace, and deliver one signed commit in the current repo." disable-model-invocation: true --- # vLLM Ascend Model Adapter ## Overview Adapt Hugging Face or local models to run on `vllm-ascend` with minimal changes, deterministic validation, and single-commit delivery. This skill is for both already-supported models and new architectures not yet registered in vLLM. ## Read order 1. Start with `references/workflow-checklist.md`. 2. Read `references/multimodal-ep-aclgraph-lessons.md` (feature-first checklist). 3. If startup/inference fails, read `references/troubleshooting.md`. 4. If checkpoint is fp8-on-NPU, read `references/fp8-on-npu-lessons.md`. 5. Before handoff, read `references/deliverables.md`. ## Hard constraints - Never upgrade `transformers`. - Primary implementation roots are fixed by Dockerfile: - `/vllm-workspace/vllm` - `/vllm-workspace/vllm-ascend` - Start `vllm serve` from `/workspace` with direct command by default. - Default API port is `8000` unless user explicitly asks otherwise. - Feature-first default: try best to validate ACLGraph / EP / flashcomm1 / MTP / multimodal out-of-box. - `--enable-expert-parallel` and flashcomm1 checks are MoE-only; for non-MoE models mark as not-applicable with evidence. - If any feature cannot be enabled, keep evidence and explain reason in final report. - Do not rely on `PYTHONPATH=<modified-src>:$PYTHONPATH` unless debugging fallback is strictly needed. - Keep code changes minimal and focused on the target model. - Final deliverable commit must be one single signed commit in the current working repo (`git commit -sm ...`). - Keep final docs in Chinese and compact. - **Dummy-first is encouraged for speed, but dummy is NOT fully equivalent to real weights.** - **Never sign off adaptation using dummy-only evidence; real-weight gate is mandatory.** ## Execution playbook ### 1) Collect context - Confirm model path (default `/models/<model-name>`; if environment differs, confirm with user explicitly). - Confirm implementation roots (`/vllm-workspace/vllm`, `/vllm-workspace/vllm-ascend`). - Confirm delivery root (the current git repo where the final commit is expected). - Confirm runtime import path points to `/vllm-workspace/*` install. - Use default expected feature set: ACLGraph + EP + flashcomm1 + MTP + multimodal (if model has VL capability). - User requirements extend this baseline, not replace it. ### 2) Analyze model first - Inspect `config.json`, processor files, modeling files, tokenizer files. - Identify architecture class, attention variant, quantization type, and multimodal requirements. - Check state-dict key prefixes (and safetensors index) to infer mapping needs. - Decide whether support already exists in `vllm/model_executor/models/registry.py`. ### 3) Choose adaptation strategy (new-model capable) - Reuse existing vLLM architecture if compatible. - If architecture is missing or incompatible, implement native support: - add model adapter under `vllm/model_executor/models/`; - add processor under `vllm/transformers_utils/processors/` when needed; - register architecture in `vllm/model_executor/models/registry.py`; - implement explicit weight loading/remap rules (including fp8 scale pairing, KV/QK norm sharding, rope variants). - If remote code needs newer transformers symbols, do not upgrade dependency. - If unavoidable, copy required modeling files from sibling transformers source and keep scope explicit. - If failure is backend-specific (kernel/op/platform), patch minimal required code in `/vllm-workspace/vllm-ascend`. ### 4) Implement minimal code changes (in implementation roots) - Touch only files required for this model adaptation. - Keep weight mapping explicit and auditable. - Avoid unrelated refactors. ### 5) Two-stage validation on Ascend (direct run) #### Stage A: dummy fast gate (recommended first) - Run from `/workspace` with `--load-format dummy`. - Goal: fast validate architecture path / operator path / API path. - Do not treat `Application startup complete` as pass by itself; request smoke is mandatory. - Require at least: - startup readiness (`/v1/models` 200), - one text request 200, - if VL model, one text+image request 200, - ACLGraph evidence where expected. #### Stage B: real-weight mandatory gate (must pass before sign-off) - Remove `--load-format dummy` and validate with real checkpoint. - Goal: validate real-only risks: - weight key mapping, - fp8/fp4 dequantization path, - KV/QK norm sharding with real tensor shapes, - load-time/runtime stability. - Require HTTP 200 and non-empty output before declaring success. - Do not pass Stage B on startup-only evidence. ### 6) Validate inference and features - Send `GET /v1/models` first. - Send at least one OpenAI-compatible text request. - For multimodal models, require at least one text+image request. - Validate architecture registration and loader path with logs (no unresolved architecture, no fatal missing-key errors). - Try feature-first validation: EP + ACLGraph path first; eager path as fallback/isolation. - If startup succeeds but first request crashes (false-ready), treat as runtime failure and continue root-cause isolation. - For `torch._dynamo` + `interpolate` + `NPU contiguous` failures on VL paths, try `TORCHDYNAMO_DISABLE=1` as diagnostic/stability fallback. - For multimodal processor API mismatch (for example `skip_tensor_conversion` signature mismatch), use text-only isolation (`--limit-mm-per-prompt` set image/video/audio to 0) to separate processor issues from core weight loading issues. - Capacity baseline by default (single machine): `max-model-len=128k` + `max-num-seqs=16`. - Then expand concurrency (e.g., 32/64) if requested or feasible. ### 7) Backport, generate artifacts, and commit in delivery repo - If implementation happened in `/vllm-workspace/*`, backport minimal final diff to current working repo. - Generate test config YAML at `tests/e2e/models/configs/<ModelName>.yaml` following the schema of existing configs (must include `model_name`, `hardware`, `tasks` with accuracy metrics, and `num_fewshot`). Use accuracy results from evaluation to populate metric values. - Generate tutorial markdown at `docs/source/tutorials/models/<ModelName>.md` following the standard template (Introduction, Supported Features, Environment Preparation with docker tabs, Deployment with serve script, Functional Verification with curl example, Accuracy Evaluation, Performance). Fill in model-specific details: HF path, hardware requirements, TP size, max-model-len, served-model-name, sample curl, and accuracy table. - Update `docs/source/tutorials/models/index.md` to include the new tutorial. - Confirm test config YAML and tutorial doc are included in the staged files. - Commit code changes once (single signed commit). ### 8) Prepare handoff artifacts - Write comprehensive Chinese analysis report. - Write compact Chinese runbook for server startup and validation commands. - Include feature status matrix (supported / unsupported / checkpoint-missing / not-applicable). - Include dummy-vs-real validation matrix and explicit non-equivalence notes. - Include changed-file list, key logs, and final commit hash. - Post the SKILL.md content (or a link to it) as a comment on the originating GitHub issue to document the AI-assisted workflow. ## Quality gate before final answer - Service starts successfully from `/workspace` with direct command. - OpenAI-compatible inference request succeeds (not startup-only). - Key feature set is attempted and reported: ACLGraph / EP / flashcomm1 / MTP / multimodal. - Capacity baseline (`128k + bs16`) result is reported, or explicit reason why not feasible. - **Dummy stage evidence is present (if used), and real-weight stage evidence is present (mandatory).** - Test config YAML exists at `tests/e2e/models/configs/<ModelName>.yaml` and follows the established schema (`model_name`, `hardware`, `tasks`, `num_fewshot`). - Tutorial doc exists at `docs/source/tutorials/models/<ModelName>.md` and follows the standard template (Introduction, Supported Features, Environment Preparation, Deployment, Functional Verification, Accuracy Evaluation, Performance). - Tutorial index at `docs/source/tutorials/models/index.md` includes the new model entry. - Exactly one signed commit contains all code changes in current working repo. - Final response includes commit hash, file paths, key commands, known limits, and failure reasons where applicable. - .agents/skills/vllm-ascend-release/agents/openai.yaml (43 bytes)
policy: allow_implicit_invocation: false
- .agents/skills/vllm-ascend-release/references/ref-past-release-notes-highlight.md (16282 bytes)
## v0.14.0rc1 - 2026.01.26 This is the first release candidate of v0.14.0 for vLLM Ascend. Please follow the [official doc](https://docs.vllm.ai/projects/ascend/en/latest) to get started. This release includes all the changes in v0.13.0rc2. So We just list the differences from v0.13.0rc2. If you are upgrading from v0.13.0rc1, please read both v0.14.0rc1 and v0.13.0rc2 release notes. ### Highlights - 310P support is back now. In this release, only basic dense and vl models are supported with eager mode. We'll keep improving and maintaining the support for 310P. [#5776](https://github.com/vllm-project/vllm-ascend/pull/5776) - Support compressed tensors moe w8a8-int8 quantization. [#5718](https://github.com/vllm-project/vllm-ascend/pull/5718) - Support Medusa speculative decoding. [#5668](https://github.com/vllm-project/vllm-ascend/pull/5668) - Support Eagle3 speculative decoding for Qwen3vl. [#4848](https://github.com/vllm-project/vllm-ascend/pull/4848) ### Features - Xlite Backend supports Qwen3 MoE now. [#5951](https://github.com/vllm-project/vllm-ascend/pull/5951) - Support DSA-CP for PD-mix deployment case. [#5702](https://github.com/vllm-project/vllm-ascend/pull/5702) - Add support of new W4A4_LAOS_DYNAMIC quantization method. [#5143](https://github.com/vllm-project/vllm-ascend/pull/5143) ### Performance - The performance of Qwen3-next has been improved. [#5664](https://github.com/vllm-project/vllm-ascend/pull/5664) [#5984](https://github.com/vllm-project/vllm-ascend/pull/5984) [#5765](https://github.com/vllm-project/vllm-ascend/pull/5765) - The CPU bind logic and performance has been improved. [#5555](https://github.com/vllm-project/vllm-ascend/pull/5555) - Merge Q/K split to simplify AscendApplyRotaryEmb for better performance. [#5799](https://github.com/vllm-project/vllm-ascend/pull/5799) - Add Matmul Allreduce Rmsnorm fusion Pass. It's disabled by default. Set `fuse_allreduce_rms=True` in `--additional_config` to enable it. [#5034](https://github.com/vllm-project/vllm-ascend/pull/5034) - Optimize rope embedding with triton kernel for huge performance gain. [#5918](https://github.com/vllm-project/vllm-ascend/pull/5918) - support advanced apply_top_k_top_p without top_k constraint. [#6098](https://github.com/vllm-project/vllm-ascend/pull/6098) - Parallelize Q/K/V padding in AscendMMEncoderAttention for better performance. [#6204](https://github.com/vllm-project/vllm-ascend/pull/6204) ### Others - model runner v2 support triton of penalty. [#5854](https://github.com/vllm-project/vllm-ascend/pull/5854) - model runner v2 support eagle spec decoding. [#5840](https://github.com/vllm-project/vllm-ascend/pull/5840) - Fix multi-modal inference OOM issues by setting `expandable_segments:True` by default. [#5855](https://github.com/vllm-project/vllm-ascend/pull/5855) - `VLLM_ASCEND_ENABLE_MLAPO` is set to `True` by default. It's enabled automatically on decode node in PD deployment case. Please note that this feature will cost more memory. If you are memory sensitive, please set it to False. [#5952](https://github.com/vllm-project/vllm-ascend/pull/5952) - SSL config can be set to kv_extra_config for PD deployment with mooncake layerwise connector. [#5875](https://github.com/vllm-project/vllm-ascend/pull/5875) - support `--max_model_len=auto`. [#6193](https://github.com/vllm-project/vllm-ascend/pull/6193) ### Dependencies - TorchNPU is upgraded to 2.9.0 [#6112](https://github.com/vllm-project/vllm-ascend/pull/6112) ### Deprecation & Breaking Changes - EPLB config options is moved to `eplb_config` in [additional config](https://docs.vllm.ai/projects/ascend/en/latest/user_guide/configuration/additional_config.html). The old ones are removed in this release. - The profiler envs, such as `VLLM_TORCH_PROFILER_DIR` and `VLLM_TORCH_PROFILER_WITH_PROFILE_MEMORY` do not work with vLLM Ascend now. Please use vLLM `--profiler-config` parameters instead. [#5928](https://github.com/vllm-project/vllm-ascend/pull/5928) ### Known Issues - If you hit the pickle error from `EngineCore` process sometimes, please cherry-pick the [PR](https://github.com/vllm-project/vllm/pull/32022) into your local vLLM code. This known issue will be fixed in vLLM in the next release. ## v0.13.0rc2 - 2026.01.24 This is the second release candidate of v0.13.0 for vLLM Ascend. In this rc release, we fixed lots of bugs and improved the performance of many models. Please follow the [official doc](https://docs.vllm.ai/projects/ascend/en/v0.13.0/) to get started. Any feedback is welcome to help us to improve the final version of v0.13.0. ### Highlights We mainly focus on quality and performance improvement in this release. The spec decode, graph mode, context parallel and EPLB have been improved significantly. A lot of bugs have been fixed and the performance has been improved for DeepSeek3.1/3.2, Qwen3 Dense/MOE models. ### Features - implement basic framework for batch invariant [#5517](https://github.com/vllm-project/vllm-ascend/pull/5517) - Eagle spec decode feature now works with full graph mode. [#5118](https://github.com/vllm-project/vllm-ascend/pull/5118) - Context Parallel(PCP&DCP) feature is more stable now. And it works for most case. Please try it out. - MTP and eagle spec decode feature now works in most cases. And it's suggested to use them in most cases. - EPLB feature more stable now. Many bugs have been fixed. Mix placement works now [#6086](https://github.com/vllm-project/vllm-ascend/pull/6086) - Support kv nz feature for DeepSeek decode node in disagg-prefill scenario [#3072](https://github.com/vllm-project/vllm-ascend/pull/3072) ### Model Support - LongCat-Flash is supported now.[#3833](https://github.com/vllm-project/vllm-ascend/pull/3833) - minimax_m2 is supported now. [#5624](https://github.com/vllm-project/vllm-ascend/pull/5624) - Support for cross-attention and whisper models [#5592](https://github.com/vllm-project/vllm-ascend/pull/5592) ### Performance - Many custom ops and triton kernels are added in this release to speed up the performance of models. Such as `RejectSampler`, `MoeInitRoutingCustom`, `DispatchFFNCombine` and so on. - Improved the performance of Layerwise Connector [#5303](https://github.com/vllm-project/vllm-ascend/pull/5303) ### Others - Basic support Model Runner v2. Model Runner V2 is the next generation of vLLM. It will be used by default in the future release. [#5210](https://github.com/vllm-project/vllm-ascend/pull/5210) - Fixed a bug that the zmq send/receive may failed [#5503](https://github.com/vllm-project/vllm-ascend/pull/5503) - Supported to use full-graph with Qwen3-Next-MTP [#5477](https://github.com/vllm-project/vllm-ascend/pull/5477) - Fix weight transpose in RL scenarios [#5567](https://github.com/vllm-project/vllm-ascend/pull/5567) - Adapted SP to eagle3 [#5562](https://github.com/vllm-project/vllm-ascend/pull/5562) - Context Parallel(PCP&DCP) support mlapo [#5672](https://github.com/vllm-project/vllm-ascend/pull/5672) - GLM4.6 support mtp with fullgraph [#5460](https://github.com/vllm-project/vllm-ascend/pull/5460) - Support setting tp=1 for the Eagle draft model [#5804](https://github.com/vllm-project/vllm-ascend/pull/5804) - Flashcomm1 feature now works with qwen3-vl [#5848](https://github.com/vllm-project/vllm-ascend/pull/5848) - Support fine-grained shared expert overlap [#5962](https://github.com/vllm-project/vllm-ascend/pull/5962) ### Dependencies - CANN is upgraded to 8.5.0 - TorchNPU is upgraded to 2.8.0.post1. Please note that the post version will not be installed by default. Please install it by hand from [pypi mirror](https://mirrors.huaweicloud.com/ascend/repos/pypi/torch-npu/). - triton-ascend is upgraded to 3.2.0 ### Deprecation & Breaking Changes - `CPUOffloadingConnector` is deprecated. We'll remove it in the next release. It'll be replaced by CPUOffload feature from vLLM in the future. - eplb config options is moved to `eplb_config` in [additional config](https://docs.vllm.ai/projects/ascend/en/latest/user_guide/configuration/additional_config.html). The old ones will be removed in the next release. - `ProfileExecuteDuration` [feature](https://docs.vllm.ai/projects/ascend/en/latest/developer_guide/performance_and_debug/profile_execute_duration.html) is deprecated. It's replaced by `ObservabilityConfig` from vLLM. - The value of `VLLM_ASCEND_ENABLE_MLAPO` env will be set to True by default in the next release. It'll be enabled in decode node by default. Please note that this feature will cost more memory. If you are memory sensitive, please set it to False. ## v0.13.0rc1 - 2025.12.27 This is the first release candidate of v0.13.0 for vLLM Ascend. We landed lots of bug fix, performance improvement and feature support in this release. Any feedback is welcome to help us to improve vLLM Ascend. Please follow the [official doc](https://docs.vllm.ai/projects/ascend/en/latest) to get started. ### Highlights - Improved the performance of DeepSeek V3.2, please refer to [tutorials](https://docs.vllm.ai/projects/ascend/en/latest/tutorials/DeepSeek-V3.2.html) - Qwen3-Next MTP with chunked prefill is supported now [#4770](https://github.com/vllm-project/vllm-ascend/pull/4770), please refer to [tutorials](https://docs.vllm.ai/projects/ascend/en/latest/tutorials/Qwen3-Next.html) - [Experimental] Prefill Context Parallel and Decode Context Parallel are supported, but notice that it is an experimental feature now, welcome any feedback. please refer to [context parallel feature guide](https://docs.vllm.ai/projects/ascend/en/latest/user_guide/feature_guide/context_parallel.html) ### Features - Support openPangu Ultra MoE [4615](https://github.com/vllm-project/vllm-ascend/pull/4615) - A new quantization method W8A16 is supported now. [#4541](https://github.com/vllm-project/vllm-ascend/pull/4541) - Cross-machine Disaggregated Prefill is supported now. [#5008](https://github.com/vllm-project/vllm-ascend/pull/5008) - Add UCMConnector for KV Cache Offloading. [#4411](https://github.com/vllm-project/vllm-ascend/pull/4411) - Support async_scheduler and disable_padded_drafter_batch in eagle. [#4893](https://github.com/vllm-project/vllm-ascend/pull/4893) - Support pcp + mtp in full graph mode. [#4572](https://github.com/vllm-project/vllm-ascend/pull/4572) - Enhance all-reduce skipping logic for MoE models in NPUModelRunner [#5329](https://github.com/vllm-project/vllm-ascend/pull/5329) ### Performance Some general performance improvement: - Add l2norm triton kernel [#4595](https://github.com/vllm-project/vllm-ascend/pull/4595) - Add new pattern for AddRmsnormQuant with SP, which could only take effect in graph mode. [#5077](https://github.com/vllm-project/vllm-ascend/pull/5077) - Add async exponential while model executing. [#4501](https://github.com/vllm-project/vllm-ascend/pull/4501) - Remove the transpose step after attention and switch to transpose_batchmatmul [#5390](https://github.com/vllm-project/vllm-ascend/pull/5390) - To optimize the performance in small batch size scenario, an attention operator with flash decoding function is offered, please refer to item 22 in [Public FAQs](https://docs.vllm.ai/projects/ascend/en/latest/faqs.html) to enable it. ### Other - OOM error on VL models is fixed now. We're keeping observing it, if you hit OOM problem again, please submit an issue. [#5136](https://github.com/vllm-project/vllm-ascend/pull/5136) - Fixed an accuracy bug of Qwen3-Next-MTP when batched inferring. [#4932](https://github.com/vllm-project/vllm-ascend/pull/4932) - Fix npu-cpu offloading interface change bug. [#5290](https://github.com/vllm-project/vllm-ascend/pull/5290) - Fix MHA model runtime error in aclgraph mode [#5397](https://github.com/vllm-project/vllm-ascend/pull/5397) - Fix unsuitable moe_comm_type under ep=1 scenario [#5388](https://github.com/vllm-project/vllm-ascend/pull/5388) ### Deprecation & Breaking Changes - `VLLM_ASCEND_ENABLE_DENSE_OPTIMIZE` is removed and `VLLM_ASCEND_ENABLE_PREFETCH_MLP` is recommend to replace as they always be enabled together. [#5272](https://github.com/vllm-project/vllm-ascend/pull/5272) - `VLLM_ENABLE_FUSED_EXPERTS_ALLGATHER_EP` is dropped now. [#5270](https://github.com/vllm-project/vllm-ascend/pull/5270) - `VLLM_ASCEND_ENABLE_NZ` is disabled for float weight case, since we notice that the performance is not good in some float case. Feel free to set it to 2 if you make sure it works for your case. [#4878](https://github.com/vllm-project/vllm-ascend/pull/4878) - `chunked_prefill_for_mla` in `additional_config` is dropped now. [#5296](https://github.com/vllm-project/vllm-ascend/pull/5296) - `dump_config` in `additional_config` is renamed to `dump_config_path` and the type is change from `dict` to `string`. [#5296](https://github.com/vllm-project/vllm-ascend/pull/5296) ### Dependencies - vLLM version has been upgraded to 0.13.0 and drop 0.12.0 support. [#5146](https://github.com/vllm-project/vllm-ascend/pull/5146) - Transformer version has been upgraded >= 4.57.3 [#5250](https://github.com/vllm-project/vllm-ascend/pull/5250) ### Known Issues - Qwen3-Next doesn't support long sequence scenario, and we should limit `gpu-memory-utilization` according to the doc to run Qwen3-Next. We'll improve it in the next release - The functional break on Qwen3-Next when the input/output is around 3.5k/1.5k is fixed, but it introduces a regression on performance. We'll fix it in next release. [#5357](https://github.com/vllm-project/vllm-ascend/issues/5357) - There is a precision issue with curl on ultra-short sequences in DeepSeek-V3.2. We'll fix it in next release. [#5370](https://github.com/vllm-project/vllm-ascend/issues/5370) ## v0.11.0 - 2025.12.16 We're excited to announce the release of v0.11.0 for vLLM Ascend. This is the official release for v0.11.0. Please follow the [official doc](https://docs.vllm.ai/projects/ascend/en/v0.11.0) to get started. We'll consider to release post version in the future if needed. This release note will only contain the important change and note from v0.11.0rc3. ### Highlights - Improved the performance for deepseek 3/3.1. [#3995](https://github.com/vllm-project/vllm-ascend/pull/3995) - Fixed the accuracy bug for qwen3-vl. [#4811](https://github.com/vllm-project/vllm-ascend/pull/4811) - Improved the performance of sample. [#4153](https://github.com/vllm-project/vllm-ascend/pull/4153) - Eagle3 is back now. [#4721](https://github.com/vllm-project/vllm-ascend/pull/4721) ### Other - Improved the performance for kimi-k2. [#4555](https://github.com/vllm-project/vllm-ascend/pull/4555) - Fixed a quantization bug for deepseek3.2-exp. [#4797](https://github.com/vllm-project/vllm-ascend/pull/4797) - Fixed qwen3-vl-moe bug under high concurrency. [#4658](https://github.com/vllm-project/vllm-ascend/pull/4658) - Fixed an accuracy bug for Prefill Decode disaggregation case. [#4437](https://github.com/vllm-project/vllm-ascend/pull/4437) - Fixed some bugs for EPLB [#4576](https://github.com/vllm-proj
- .agents/skills/vllm-ascend-release/references/version-files.yaml (3824 bytes)
# Version Files Reference # # This file defines all files that need version updates during a release. # The script update_version_references.py loads this configuration. # # Pattern syntax: # - Use X.X.X as placeholder for version numbers (will match \d+.\d+.\d+ and rc versions) # - Use XXXX as placeholder for issue numbers (will match \d+) # - 'pattern' entries are auto-updated with regex replacement # - 'variable' entries update variable definitions (YAML, shell, Docker) # - 'action' entries require manual intervention files: - path: README.md description: Main README with getting started instructions updates: - pattern: "pip install vllm-ascend==X.X.X" replacement: "pip install vllm-ascend=={version_num}" - pattern: "vllm-ascend:vX.X.X" replacement: "vllm-ascend:{version}" - path: README.zh.md description: Chinese README updates: - pattern: "pip install vllm-ascend==X.X.X" replacement: "pip install vllm-ascend=={version_num}" - pattern: "vllm-ascend:vX.X.X" replacement: "vllm-ascend:{version}" - path: mkdocs.yml description: MkDocs configuration; version variables live in the `extra:` block updates: - pattern: 'vllm_version: vX.X.X' replacement: 'vllm_version: {version}' - pattern: 'vllm_ascend_version: vX.X.X' replacement: 'vllm_ascend_version: {version}' - pattern: 'pip_vllm_version: X.X.X' replacement: 'pip_vllm_version: {version_num}' - pattern: 'pip_vllm_ascend_version: X.X.X' replacement: 'pip_vllm_ascend_version: {version_num}' - path: docs/source/faqs.md description: FAQs page with feedback issue link updates: - pattern: "https://github.com/vllm-project/vllm-ascend/issues/XXXX" replacement: "{feedback_issue}" - path: docs/source/user_guide/release_notes.md description: Release notes document updates: - action: "Add new release notes section" description: "Add complete release notes for the new version at the top of the file" - path: docs/source/community/versioning_policy.md description: Version compatibility and release policy updates: - action: "Update Release compatibility matrix" description: "Add row for new version with vLLM compatibility info" - action: "Update Release window section" description: "Update current release window dates" - action: "Update Branch states section" description: "Update branch status (stable/development)" - path: docs/source/community/contributors.md description: Contributors list updates: - action: "Add new contributors" description: "Add contributors from this release cycle" - path: .github/workflows/*.yml description: CI workflow files updates: - variable: VLLM_VERSION description: "Update vLLM version for CI testing" - path: docker/Dockerfile* description: Docker build files updates: - variable: VLLM_VERSION description: "Update base vLLM version" # Update order # Files should be updated in this order to ensure consistency update_order: 1: mkdocs.yml 2: README.md 3: README.zh.md 4: docs/source/faqs.md 5: docs/source/user_guide/release_notes.md 6: docs/source/community/versioning_policy.md 7: docs/source/community/contributors.md 8: .github/workflows/*.yml 9: docker/Dockerfile* # Validation # After updates, verify these conditions validation: - "grep -r 'X.X.X' docs/ README* | grep -v node_modules # Check for consistent version" - "grep 'vllm_ascend_version' mkdocs.yml # Verify version in mkdocs config" - "grep 'VLLM_VERSION' .github/workflows/*.yml # Check CI workflow versions" - "head -50 docs/source/user_guide/release_notes.md # Verify release notes added" - .agents/skills/vllm-ascend-release/scripts/collect_release_notes.py (17484 bytes)
#!/usr/bin/env python3 """ Collect release note fragments into a draft and audit for missing ones. Fragments live in release_notes/fragments/<PR number>.yaml. See release_notes/README.md for when a fragment is needed and the field rules. The default --fragments directory is resolved from the current directory first, then from the repository that contains this script, so it works from the repo root and from the skill directory. Usage: python collect_release_notes.py \ --version v0.31.0 \ --raw-commits output/v0.31.0/0-current-raw-commits.md \ --output output/v0.31.0/1-fragments-draft.md \ [--archive] """ import argparse import re import shutil import sys from dataclasses import dataclass, field from pathlib import Path import yaml DEFAULT_REPO = "vllm-project/vllm-ascend" # Maps each release notes section to the fragment categories that feed it, in the # order the sections appear in docs/source/user_guide/release_notes.md. SECTION_CATEGORIES: list[tuple[str, list[str]]] = [ ("Features", ["feature"]), ("Hardware and Operator Support", ["hardware"]), ("Performance", ["performance"]), ("Dependencies", ["dependency"]), ("Deprecation and Configuration Changes", ["breaking", "config"]), ("Documentation", ["documentation"]), ("Stability and Bug Fixes", ["bugfix"]), ("Other Changes", ["other"]), ] VALID_CATEGORIES = [category for _, categories in SECTION_CATEGORIES for category in categories] PR_LINK_IN_TEXT_RE = re.compile(r"\(#\d+\)|pull/\d+") PR_REF_RE = re.compile(r"\(#(\d+)\)|pull/(\d+)") FRAGMENT_NAME_RE = re.compile(r"^(\d+)(?:[-._].+)?$") @dataclass class Entry: pr: int category: str text: str highlight: bool = False model: str = "" hardware: list[str] = field(default_factory=list) @dataclass class MergedEntry: category: str text: str prs: list[int] = field(default_factory=list) highlight: bool = False model: str = "" hardware: list[str] = field(default_factory=list) def normalize_text(text: str) -> str: """Collapse whitespace so equivalent texts merge correctly.""" return " ".join(str(text).split()) def resolve_default_fragments() -> str: """Resolve release_notes/fragments from the current directory or the script's repository.""" default = Path("release_notes/fragments") if default.is_dir(): return str(default) try: repo_root = Path(__file__).resolve().parents[4] except IndexError: return str(default) candidate = repo_root / default return str(candidate if candidate.is_dir() else default) def parse_fragment(path: Path, errors: list[str]) -> list[Entry]: """Parse and validate a single fragment file.""" match = FRAGMENT_NAME_RE.match(path.stem) if not match: errors.append(f"{path}: file name must be <PR number>.yaml or <PR number>-<slug>.yaml") return [] try: content = path.read_text(encoding="utf-8") except (OSError, UnicodeDecodeError) as exc: errors.append(f"{path}: cannot read: {exc}") return [] try: data = yaml.safe_load(content) except yaml.YAMLError as exc: errors.append(f"{path}: invalid YAML: {exc}") return [] if not isinstance(data, dict): errors.append(f"{path}: top level must be a mapping with 'pr' and 'entries'") return [] pr = data.get("pr") if isinstance(pr, bool): pr = None if isinstance(pr, str) and pr.isdigit(): pr = int(pr) if not isinstance(pr, int) or pr <= 0: errors.append(f"{path}: 'pr' is required and must be the PR number") return [] if int(match.group(1)) != pr: errors.append(f"{path}: file name number {match.group(1)} does not match 'pr: {pr}'") raw_entries = data.get("entries") if not isinstance(raw_entries, list) or not raw_entries: errors.append(f"{path}: 'entries' is required and must be a non-empty list") return [] entries = [] for i, raw in enumerate(raw_entries): where = f"{path} entries[{i}]" if not isinstance(raw, dict): errors.append(f"{where}: must be a mapping") continue category = raw.get("category") if category not in VALID_CATEGORIES: errors.append(f"{where}: 'category' must be one of {', '.join(VALID_CATEGORIES)} (got {category!r})") continue raw_text = raw.get("text") if not isinstance(raw_text, str): errors.append(f"{where}: 'text' is required and must be a string") continue text = normalize_text(raw_text) if not text: errors.append(f"{where}: 'text' is required") continue if PR_LINK_IN_TEXT_RE.search(text): errors.append(f"{where}: do not append the PR link in 'text', the tooling adds it") highlight = raw.get("highlight", False) if not isinstance(highlight, bool): errors.append(f"{where}: 'highlight' must be true or false") continue model = raw.get("model", "") if model is None: model = "" if not isinstance(model, str): errors.append(f"{where}: 'model' must be a string") continue model = normalize_text(model) hardware = raw.get("hardware", []) if isinstance(hardware, str): hardware = [hardware] if not isinstance(hardware, list) or not all(isinstance(h, str) for h in hardware): errors.append(f"{where}: 'hardware' must be a string or a list of strings") continue hardware = [normalize_text(h) for h in hardware] if any(not h for h in hardware): errors.append(f"{where}: 'hardware' entries must not be empty") continue entries.append(Entry(pr=pr, category=category, text=text, highlight=highlight, model=model, hardware=hardware)) return entries def fragment_sort_key(path: Path) -> int: """Sort fragment files by their leading PR number.""" match = FRAGMENT_NAME_RE.match(path.stem) return int(match.group(1)) if match else 0 def load_fragments(fragments_dir: Path) -> tuple[list[Entry], list[Path], list[str]]: """Load all fragments in the directory, returning entries, files, and errors.""" if not fragments_dir.is_dir(): return [], [], [f"{fragments_dir}: fragments directory not found"] files = sorted( ( p for p in fragments_dir.iterdir() if p.is_file() and p.suffix.lower() in (".yaml", ".yml") and p.stem != "TEMPLATE" ), key=fragment_sort_key, ) entries: list[Entry] = [] errors: list[str] = [] for path in files: entries.extend(parse_fragment(path, errors)) return entries, files, errors def merge_entries(entries: list[Entry]) -> list[MergedEntry]: """Merge entries with identical text (for example cherry-picks) into one bullet.""" merged: dict[tuple[str, str], MergedEntry] = {} for entry in entries: key = (entry.category, entry.text.lower()) if key not in merged: merged[key] = MergedEntry(category=entry.category, text=entry.text) target = merged[key] if entry.pr not in target.prs: target.prs.append(entry.pr) target.highlight = target.highlight or entry.highlight if not target.model and entry.model: target.model = entry.model for hardware in entry.hardware: if hardware not in target.hardware: target.hardware.append(hardware) return list(merged.values()) def format_bullet(entry: MergedEntry, repo: str) -> str: links = " ".join(f"[#{pr}](https://github.com/{repo}/pull/{pr})" for pr in entry.prs) return f"- {entry.text} {links}" def escape_table_cell(value: str) -> str: """Escape '|' so metadata values do not break the markdown table.""" return value.replace("|", "\\|") def audit_raw_commits(raw_commits: Path, fragment_prs: set[int]) -> tuple[list[str], list[str], list[int]]: """Audit the raw commit list against the fragment PRs. Returns lines with a PR reference but no fragment, lines without a PR reference, and fragment PRs that never appear in the commit list (for example reverted PRs). """ missing: list[str] = [] no_pr: list[str] = [] seen_prs: set[int] = set() for raw_line in raw_commits.read_text(encoding="utf-8").splitlines(): line = raw_line.strip() if not line or line.startswith("#"): continue prs = {int(a or b) for a, b in PR_REF_RE.findall(line)} seen_prs |= prs if not prs: no_pr.append(line) elif prs - fragment_prs: missing.append(line) unmatched_prs = sorted(fragment_prs - seen_prs) return missing, no_pr, unmatched_prs def render_draft( version: str, merged: list[MergedEntry], repo: str, fragment_count: int, raw_commits: Path | None, audit_result: tuple[list[str], list[str], list[int]] | None = None, ) -> str: lines = [f"# Release Note Fragments Draft: {version}", ""] highlights = [entry for entry in merged if entry.highlight] lines.append(f"> {fragment_count} fragment(s), {len(merged)} entries, {len(highlights)} highlight candidate(s).") lines.append(">") lines.append( "> This is raw material: curate the wording and order, write the Highlights, then copy" " the result into `docs/source/user_guide/release_notes.md`." ) if audit_result is not None and audit_result[2]: lines.append(">") lines.append( f"> Warning: {len(audit_result[2])} fragment PR(s) are not in the commit list, likely leftovers from an" " earlier release. Review the Audit section before using this draft." ) lines.append("") lines.append("## Highlights (candidates)") lines.append("") if highlights: lines.extend(format_bullet(entry, repo) for entry in highlights) else: lines.append("_No entry is marked with `highlight: true`._") lines.append("") for section, categories in SECTION_CATEGORIES: lines.append(f"## {section}") lines.append("") section_entries = [entry for entry in merged if entry.category in categories] if section_entries: lines.extend(format_bullet(entry, repo) for entry in section_entries) else: lines.append("_(empty)_") lines.append("") matrix_entries = [entry for entry in merged if entry.model or entry.hardware] lines.append("## Support Matrix Hints") lines.append("") lines.append("| Model | Hardware | PRs |") lines.append("|-------|----------|-----|") if matrix_entries: for entry in matrix_entries: prs = ", ".join(f"#{pr}" for pr in entry.prs) model = escape_table_cell(entry.model) or "-" hardware = escape_table_cell(", ".join(entry.hardware)) or "-" lines.append(f"| {model} | {hardware} | {prs} |") else: lines.append("| - | - | - |") lines.append("") if raw_commits is not None and audit_result is not None: missing, no_pr, unmatched_prs = audit_result lines.append("## Audit: Code Changes Without a Fragment") lines.append("") lines.append(f"Raw commits source: `{raw_commits}`") lines.append("") fragment_prs = {pr for entry in merged for pr in entry.prs} lines.append(f"Covered by fragments: {len(fragment_prs)} PR(s), missing: {len(missing)} line(s).") lines.append("") lines.append("### Missing (review required)") lines.append("") if missing: lines.extend(f"- {line}" for line in missing) else: lines.append("_Every commit with a PR reference has a fragment._") lines.append("") lines.append("### Commits Without a PR Reference") lines.append("") if no_pr: lines.extend(f"- {line}" for line in no_pr) else: lines.append("_None._") lines.append("") lines.append("### Fragment PRs Not Found in the Commit List") lines.append("") if unmatched_prs: lines.append( "These fragments do not match any commit in this window. They are usually leftovers from" " an earlier release or PRs that were never merged, so archive or drop them before publishing:" ) lines.extend(f"- #{pr}" for pr in unmatched_prs) else: lines.append("_Every fragment PR appears in the commit list._") lines.append("") return "\n".join(lines) def main(): parser = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter) parser.add_argument("--version", required=True, help="Release version, e.g. v0.31.0") parser.add_argument( "--fragments", default=resolve_default_fragments(), help="Directory containing fragment YAML files (default: release_notes/fragments)", ) parser.add_argument( "--repo", default=DEFAULT_REPO, help=f"owner/repo used to build PR links (default: {DEFAULT_REPO})", ) parser.add_argument( "--raw-commits", help="Raw commits markdown file from fetch_commits.py, used for the missing-fragment audit", ) parser.add_argument("--output", required=True, help="Output file for the aggregated draft") parser.add_argument( "--archive", action="store_true", help="Move the fragment files to <archive-dir>/<version>/ after a successful run " "(use when preparing the release PR)", ) parser.add_argument( "--archive-dir", help="Root directory for archived fragments (default: <fragments parent>/archive)", ) args = parser.parse_args() if args.archive_dir and not args.archive: parser.error("--archive-dir requires --archive") if "/" in args.version or "\\" in args.version or args.version.startswith("."): parser.error("--version must be a plain version string like v0.31.0 (it is used as a directory name)") fragments_dir = Path(args.fragments) raw_commits = Path(args.raw_commits) if args.raw_commits else None if raw_commits is not None and not raw_commits.is_file(): print(f"Error: raw commits file not found: {raw_commits}", file=sys.stderr) return 1 entries, files, errors = load_fragments(fragments_dir) if errors: print("Fragment validation failed:", file=sys.stderr) for error in errors: print(f" - {error}", file=sys.stderr) return 1 archive_dir = None if args.archive: archive_root = Path(args.archive_dir) if args.archive_dir else fragments_dir.parent / "archive" archive_dir = archive_root - .agents/skills/vllm-ascend-release/scripts/fetch_commits.py (51855 bytes)
#!/usr/bin/env python3 """ Fetch all commits between two tags from a GitHub repository. Usage: python fetch_commits.py [--token YOUR_GITHUB_TOKEN] """ import argparse import os import re import dotenv import requests # Load .env.local first (higher priority), then .env as fallback dotenv.load_dotenv(".env.local") dotenv.load_dotenv() # .env as fallback def get_github_token(): """Get GitHub token from environment or argument.""" return os.environ.get("GITHUB_TOKEN") or os.environ.get("GH_TOKEN") def resolve_tag_to_sha(base_url: str, tag: str, headers: dict) -> str: """Resolve a tag name to its commit SHA.""" print(f"Resolving tag {tag}...") tag_resp = requests.get(f"{base_url}/git/refs/tags/{tag}", headers=headers) if tag_resp.status_code != 200: raise Exception(f"Failed to get tag {tag}: {tag_resp.text}") tag_data = tag_resp.json() sha = tag_data["object"]["sha"] # If it's an annotated tag, we need to get the commit it points to if tag_data["object"]["type"] == "tag": tag_obj_resp = requests.get(f"{base_url}/git/tags/{sha}", headers=headers) if tag_obj_resp.status_code == 200: sha = tag_obj_resp.json()["object"]["sha"] return sha def resolve_commit_sha(base_url: str, commit_ref: str, headers: dict) -> str: """Resolve a commit reference (SHA or short SHA) to full SHA.""" print(f"Resolving commit {commit_ref}...") commit_resp = requests.get(f"{base_url}/commits/{commit_ref}", headers=headers) if commit_resp.status_code != 200: raise Exception(f"Failed to get commit {commit_ref}: {commit_resp.text}") return commit_resp.json()["sha"] def get_default_branch_head(base_url: str, headers: dict) -> tuple[str, str]: """ Get the HEAD commit of the default branch. Returns: Tuple of (branch_name, head_sha) """ print("Getting default branch HEAD...") # Get repository info to find default branch repo_resp = requests.get(base_url, headers=headers) if repo_resp.status_code != 200: raise Exception(f"Failed to get repository info: {repo_resp.text}") default_branch = repo_resp.json()["default_branch"] print(f" Default branch: {default_branch}") # Get the HEAD commit of the default branch branch_resp = requests.get(f"{base_url}/branches/{default_branch}", headers=headers) if branch_resp.status_code != 200: raise Exception(f"Failed to get branch {default_branch}: {branch_resp.text}") head_sha = branch_resp.json()["commit"]["sha"] print(f" HEAD: {head_sha[:8]}") return (default_branch, head_sha) def get_all_tags(base_url: str, headers: dict) -> list[dict]: """Get all tags from the repository with their commit SHAs and dates.""" print("Fetching all tags...") all_tags = [] page = 1 per_page = 100 while True: resp = requests.get( f"{base_url}/tags", headers=headers, params={"per_page": per_page, "page": page}, ) if resp.status_code != 200: raise Exception(f"Failed to get tags: {resp.text}") tags = resp.json() if not tags: break all_tags.extend(tags) page += 1 if len(tags) < per_page: break print(f" Found {len(all_tags)} tags") return all_tags def get_commit_date(base_url: str, sha: str, headers: dict) -> str: """Get the commit date for a given SHA.""" commit_resp = requests.get(f"{base_url}/commits/{sha}", headers=headers) if commit_resp.status_code != 200: return None return commit_resp.json()["commit"]["committer"]["date"] def find_previous_tag( base_url: str, head_sha: str, headers: dict, tag_pattern: str | None = None ) -> tuple[str, str] | None: """ Find the most recent tag that is an ancestor of the given commit. Uses git history to find tags that are reachable from the commit. Args: base_url: GitHub API base URL head_sha: The commit SHA to search from headers: Request headers tag_pattern: Optional regex pattern to filter tags (e.g., r'^v\\d+\\.\\d+\\.\\d+$') Returns: Tuple of (tag_name, tag_sha) or None if no tag found """ print(f"Finding previous tag before commit {head_sha[:8]}...") # Get the date of the head commit head_date = get_commit_date(base_url, head_sha, headers) if not head_date: print(" Warning: Could not get head commit date") return None print(f" Head commit date: {head_date}") # Get all tags all_tags = get_all_tags(base_url, headers) # Filter tags by pattern if provided if tag_pattern: import re pattern = re.compile(tag_pattern) all_tags = [t for t in all_tags if pattern.match(t["name"])] print(f" After pattern filter: {len(all_tags)} tags") # For each tag, check if it's an ancestor of head_sha and get its date tag_candidates = [] for tag in all_tags: tag_name = tag["name"] tag_commit_sha = tag["commit"]["sha"] # Skip if this is the same commit as head if tag_commit_sha == head_sha: continue # Check if this tag's commit is an ancestor of head compare_resp = requests.get(f"{base_url}/compare/{tag_commit_sha}...{head_sha}", headers=headers) if compare_resp.status_code != 200: continue compare_data = compare_resp.json() # If tag is behind head (status = "behind" or "ahead"), it's an ancestor # We want tags where the comparison shows head is ahead if compare_data.get("status") in ["ahead", "diverged"]: # Get the tag's commit date tag_date = get_commit_date(base_url, tag_commit_sha, headers) if tag_date and tag_date < head_date: tag_candidates.append( { "name": tag_name, "sha": tag_commit_sha, "date": tag_date, "ahead_by": compare_data.get("ahead_by", 0), } ) print(f" Found candidate: {tag_name} ({compare_data.get('ahead_by', 0)} commits behind)") if not tag_candidates: print(" No previous tag found") return None # Sort by date (most recent first) or by ahead_by (smallest first) # Using ahead_by gives us the closest tag tag_candidates.sort(key=lambda x: x["ahead_by"]) best_tag = tag_candidates[0] print(f" Selected: {best_tag['name']} ({best_tag['ahead_by']} commits behind)") return (best_tag["name"], best_tag["sha"]) def fetch_commits_between_tags( owner: str, repo: str, base_tag: str, head_tag: str, token: str | None = None ) -> list[dict]: """ Fetch all commits between two tags by walking the commit graph. This method traverses from head_tag back to base_tag, collecting all commits. It properly handles the commit history and doesn't rely on date filtering. Args: owner: Repository owner (e.g., 'vllm-project') repo: Repository name (e.g., 'vllm') base_tag: Base tag (older, e.g., 'v0.11.2') head_tag: Head tag (newer, e.g., 'v0.12.0') token: Optional GitHub token for higher rate limits Returns: List of commit dictionaries """ headers = { "Accept": "application/vnd.github.v3+json", } if token: headers["Authorization"] = f"token {token}" base_url = f"https://api.github.com/repos/{owner}/{repo}" # Resolve tags to commit SHAs base_sha = resolve_tag_to_sha(base_url, base_tag, headers) head_sha = resolve_tag_to_sha(base_url, head_tag, headers) print(f"\nBase SHA ({base_tag}): {base_sha}") print(f"Head SHA ({head_tag}): {head_sha}") # First, use Compare API to get total commit count (for progress info) print(f"\nComparing {base_tag}...{head_tag}...") compare_resp = requests.get(f"{base_url}/compare/{base_sha}...{head_sha}", headers=headers) if compare_resp.status_code == 200: compare_data = compare_resp.json() total_commits = compare_data.get("total_commits", "unknown") print(f"Total commits to fetch: {total_commits}") # Walk the commit history from head to base # We use the commits API starting from head_sha and stop when we reach base_sha all_commits = [] seen_shas = set() seen_shas.add(base_sha) # Don't include the base commit itself # BFS traversal of commit graph to_visit = [head_sha] page_count = 0 print(f"\nFetching commits from {head_tag} back to {base_tag}...") while to_visit: current_sha = to_visit.pop(0) if current_sha in seen_shas: continue seen_shas.add(current_sha) # Fetch commit details commit_resp = requests.get(f"{base_url}/commits/{current_sha}", headers=headers) if commit_resp.status_code != 200: print(f" Warning: Failed to fetch commit {current_sha[:8]}") continue commit = commit_resp.json() all_commits.append(commit) # Add parent commits to visit queue for parent in commit.get("parents", []): parent_sha = parent["sha"] if parent_sha not in seen_shas: to_visit.append(parent_sha) # Progress logging if len(all_commits) % 50 == 0: page_count += 1 print(f" Fetched {len(all_commits)} commits...") print(f" Completed: {len(all_commits)} commits fetched") return all_commits def fetch_commits_by_date_range( owner: str, repo: str, since: str, until: str, token: str | None = None, branch: str | None = None, ) -> list[dict]: """ Fetch all commits within a date range. Args: owner: Repository owner (e.g., 'vllm-project') repo: Repository name (e.g., 'vllm') since: Start date (ISO 8601 format, e.g., '2025-01-01' or '2025-01-01T00:00:00Z') until: End date (ISO 8601 format, e.g., '2025-01-31' or '2025-01-31T23:59:59Z') token: Optional GitHub token for higher rate limits branch: Optional branch name (defaults to repository's default branch) Returns: List of commit dictionaries """ headers = { "Accept": "application/vnd.github.v3+json", } if token: headers["Authorization"] = f"token {token}" base_url = f"https://api.github.com/repos/{owner}/{repo}" per_page = 100 # Normalize date format - add time if not present if len(since) == 10: # YYYY-MM-DD format since = f"{since}T00:00:00Z" if len(until) == 10: # YYYY-MM-DD format until = f"{until}T23:59:59Z" print(f"\nFetching commits from {since} to {until}...") if branch: print(f" Branch: {branch}") all_commits = [] page = 1 while True: params = {"since": since, "until": until, "per_page": per_page, "page": page} if branch: params["sha"] = branch response = requests.get(f"{base_url}/commits", headers=headers, params=params) if response.status_code != 200: raise Exception(f"Failed to fetch commits: {response.text}") commits = response.json() if not commits: break all_commits.extend(commits) print(f" Page {page}: fetched {len(commits)} commits (total: {len(all_commits)})") if len(commits) < per_page: break page += 1 print(f" Completed: {len(all_commits)} commits fetched") return all_commits def get_merge_base(base_url: str, base_sha: str, head_sha: str, headers: dict) -> str | None: """ Get the merge base (common ancestor) of two commits. Args: base_url: GitHub API base URL for the repo base_sha: First commit SHA head_sha: Second commit SHA headers: Request headers Returns: Merge base commit SHA, or None if not found """ # GitHub Compare API returns merge_base_commit compare_resp = requests.get( f"{base_url}/compare/{base_sha}...{head_sha}", headers=headers, ) if compare_resp.status_code != 200: return None compare_data = compare_resp.json() merge_base = compare_data.get("merge_base_commit", {}).get("sha") return merge_base def fetch_commits_by_walking_history( base_url: str, base_sha: str, head_sha: str, base_tag: str, head_tag: str, headers: dict, stop_sha: str | None = None, ) -> list[dict]: """ Fetch commits by walking the commit history from head to a stop point. This method correctly handles release branches with cherry-picks. It walks the head's commit history until it reaches the stop commit. Args: base_url: GitHub API base URL for the repo base_sha: Base commit SHA (for display purposes) head_sha: Head commit SHA (newer) base_tag: Display name for base reference head_tag: Display name for head reference headers: Request headers stop_sha: SHA to stop at (if None, uses base_sha) Returns: List of commit dictionaries (excluding stop commit) """ per_page = 100 all_commits = [] page = 1 target_sha = stop_sha or base_sha print(f"\nWalking commit history from {head_tag} back to {base_tag}...") print(f" Stop SHA: {target_sha[:8]}") while True: response = requests.get( f"{base_url}/commits", headers=headers, params={"sha": head_sha, "per_page": per_page, "page": page}, ) if response.status_code != 200: print(f" Warning: API error on page {page}, stopping") break commits = response.json() if not commits: print(f" No more commits found on page {page}") break found_stop = False for commit in commits: if commit["sha"] == target_sha: # Reached stop commit, stop (don't include it) found_stop = True break all_commits.append(commit) print(f" Page {page}: fetched {len(commits)} commits (total: {len(all_commits)})") if found_stop: print(f" Reached stop commit ({target_sha[:8]})") break if len(commits) < per_page: print(" Warning: Reached end of history without finding stop commit") break page += 1 return all_commits def fetch_commits_between_tags_fast( owner: str, repo: str, base_tag: str, head_tag: str, token: str | None = None, head_is_commit: bool = False, base_is_commit: bool = False, ) -> list[dict]: """ Fetch all commits between two tags using GitHub Compare API with pagination. This properly fetches only the commits between th - .agents/skills/vllm-ascend-release/scripts/generate_announcement.py (5981 bytes)
#!/usr/bin/env python3 """ Generate release announcement for broadcasting. This script creates a formatted announcement message suitable for various channels (GitHub, blog, social media). Usage: python generate_announcement.py \ --version v0.15.0rc1 \ --release-notes release-notes.md \ --output announcement.md """ import argparse import re from pathlib import Path def extract_highlights(release_notes: str) -> list[str]: """Extract key highlights from release notes.""" highlights = [] # Look for ### Highlights section match = re.search(r"###\s+Highlights?\s*\n(.*?)(?=\n###|\Z)", release_notes, re.DOTALL) if match: section = match.group(1) # Extract bullet points for line in section.split("\n"): line = line.strip() if line.startswith("- ") or line.startswith("* "): highlights.append(line[2:]) return highlights[:5] # Limit to 5 highlights def generate_github_announcement(version: str, highlights: list[str]) -> str: """Generate GitHub release announcement.""" lines = [ f"# vLLM Ascend {version} Released!", "", f"We are excited to announce the release of vLLM Ascend {version}!", "", "## Highlights", "", ] for h in highlights: lines.append(f"- {h}") lines.extend( [ "", "## Getting Started", "", "Install via pip:", "```bash", f"pip install vllm-ascend=={version.lstrip('v')}", "```", "", "Or use Docker:", "```bash", f"docker pull quay.io/ascend/vllm-ascend:{version}", "```", "", "## Documentation", "", "- [Official Documentation](https://docs.vllm.ai/projects/ascend/en/latest)", "- [Release Notes](https://docs.vllm.ai/projects/ascend/en/latest/user_guide/release_notes.html)", "", "## Feedback", "", "Please report any issues or provide feedback on our [GitHub Issues](https://github.com/vllm-project/vllm-ascend/issues).", "", "Thank you to all contributors who made this release possible!", ] ) return "\n".join(lines) def generate_short_announcement(version: str, highlights: list[str]) -> str: """Generate short announcement for social media.""" lines = [ f"🚀 vLLM Ascend {version} is now available!", "", "Key highlights:", ] for h in highlights[:3]: lines.append(f"• {h}") lines.extend( [ "", f"📦 pip install vllm-ascend=={version.lstrip('v')}", "📖 https://docs.vllm.ai/projects/ascend", "", "#vLLM #Ascend #NPU #LLM #AI", ] ) return "\n".join(lines) def generate_chinese_announcement(version: str, highlights: list[str]) -> str: """Generate Chinese announcement.""" lines = [ f"# vLLM Ascend {version} 发布!", "", f"我们很高兴地宣布 vLLM Ascend {version} 正式发布!", "", "## 主要更新", "", ] for h in highlights: lines.append(f"- {h}") lines.extend( [ "", "## 快速开始", "", "通过 pip 安装:", "```bash", f"pip install vllm-ascend=={version.lstrip('v')}", "```", "", "或使用 Docker:", "```bash", f"docker pull quay.io/ascend/vllm-ascend:{version}", "```", "", "## 文档", "", "- [官方文档](https://docs.vllm.ai/projects/ascend/en/latest)", "- [发布说明](https://docs.vllm.ai/projects/ascend/en/latest/user_guide/release_notes.html)", "", "## 反馈", "", "如有问题或建议,请在 [GitHub Issues](https://github.com/vllm-project/vllm-ascend/issues) 反馈。", "", "感谢所有贡献者的付出!", ] ) return "\n".join(lines) def main(): parser = argparse.ArgumentParser(description="Generate release announcement") parser.add_argument("--version", required=True, help="Release version") parser.add_argument("--release-notes", required=True, help="Release notes file") parser.add_argument("--output", required=True, help="Output file path") args = parser.parse_args() release_notes_path = Path(args.release_notes) if release_notes_path.exists(): with open(release_notes_path) as f: release_notes = f.read() highlights = extract_highlights(release_notes) else: print(f"Warning: Release notes not found at {release_notes_path}") highlights = ["Various improvements and bug fixes"] if not highlights: highlights = ["Various improvements and bug fixes"] print(f"Generating announcement for {args.version}...") print(f"Found {len(highlights)} highlights") output_lines = [ "=" * 60, "GITHUB RELEASE ANNOUNCEMENT", "=" * 60, "", generate_github_announcement(args.version, highlights), "", "=" * 60, "SHORT ANNOUNCEMENT (Social Media)", "=" * 60, "", generate_short_announcement(args.version, highlights), "", "=" * 60, "CHINESE ANNOUNCEMENT", "=" * 60, "", generate_chinese_announcement(args.version, highlights), ] output = "\n".join(output_lines) output_path = Path(args.output) output_path.parent.mkdir(parents=True, exist_ok=True) with open(output_path, "w") as f: f.write(output) print(f"Announcement saved to {output_path}") return 0 if __name__ == "__main__": exit(main()) - .agents/skills/vllm-ascend-release/scripts/generate_checklist.py (3075 bytes)
#!/usr/bin/env python3 """ Generate release checklist issue body from template. Usage: python generate_checklist.py \ --version v0.15.0rc1 \ --branch main \ --date 2026.03.15 \ --manager wangxiyuan \ --feedback-issue 1234 \ --output release-checklist.md """ import argparse from pathlib import Path def load_template(template_path: Path) -> str: """Load the release checklist template.""" with open(template_path) as f: return f.read() def substitute_variables(template: str, variables: dict) -> str: """Substitute variables in the template.""" result = template for key, value in variables.items(): result = result.replace(f"${{{key}}}", value) return result def main(): parser = argparse.ArgumentParser(description="Generate release checklist from template") parser.add_argument("--version", required=True, help="Release version (e.g., v0.15.0rc1)") parser.add_argument("--branch", default="main", help="Release branch") parser.add_argument("--date", required=True, help="Target release date (e.g., 2026.03.15)") parser.add_argument("--manager", required=True, help="Release manager GitHub username") parser.add_argument("--feedback-issue", type=int, help="Feedback issue number") parser.add_argument("--output", required=True, help="Output file path") parser.add_argument( "--template", default=None, help="Template file path (default: templates/release-checklist-template.md)", ) args = parser.parse_args() # Determine template path script_dir = Path(__file__).parent if args.template: template_path = Path(args.template) else: template_path = script_dir.parent / "templates" / "release-checklist-template.md" if not template_path.exists(): print(f"Error: Template not found at {template_path}") return 1 # Load template template = load_template(template_path) # Build feedback issue URL feedback_issue_url = "" if args.feedback_issue: feedback_issue_url = f"https://github.com/vllm-project/vllm-ascend/issues/{args.feedback_issue}" # Prepare variables variables = { "VERSION": args.version, "BRANCH": args.branch, "DATE": args.date, "MANAGER": args.manager, "FEEDBACK_ISSUE_URL": feedback_issue_url, "RELEASE_NOTE_PR_URL": "", # To be filled later "BUG_LIST": "<!-- Run scan_release_bugs.py to populate -->", "PR_LIST": "<!-- Run identify_release_prs.py to populate -->", "FUNCTIONAL_TEST_RESULTS": "<!-- Run run_functional_tests.py to populate -->", } # Generate checklist checklist = substitute_variables(template, variables) # Write output output_path = Path(args.output) output_path.parent.mkdir(parents=True, exist_ok=True) with open(output_path, "w") as f: f.write(checklist) print(f"Generated release checklist at {output_path}") return 0 if __name__ == "__main__": exit(main()) - .agents/skills/vllm-ascend-release/scripts/scan_nightly_status.py (10048 bytes)
#!/usr/bin/env python3 """ Scan Nightly CI status for release readiness. This script: 1. Gets the latest Nightly-A3 and Nightly-A2 GitHub Action runs 2. Calls extract_and_analyze.py to analyze failures 3. Generates a markdown summary table for the release checklist Usage: python scan_nightly_status.py \ --repo vllm-project/vllm-ascend \ --output nightly-status.md """ from __future__ import annotations import argparse import json import subprocess import sys from dataclasses import dataclass, field from datetime import datetime from pathlib import Path REPO = "vllm-project/vllm-ascend" # Nightly workflow files NIGHTLY_WORKFLOWS = { "Nightly-A3": "nightly_test_vllm_main_a3.yaml", "Nightly-A2": "nightly_test_vllm_main.yaml", } # Path to extract_and_analyze.py (relative to this script) EXTRACT_SCRIPT = Path(__file__).parent.parent.parent / "main2main-error-analysis" / "scripts" / "extract_and_analyze.py" @dataclass class NightlyRun: """A nightly workflow run.""" workflow_name: str run_id: int run_url: str conclusion: str created_at: str total_jobs: int = 0 failed_jobs: int = 0 failed_tests: list[str] = field(default_factory=list) code_bugs: list[dict] = field(default_factory=list) env_flakes: list[dict] = field(default_factory=list) def gh_api_json(endpoint: str, **params) -> dict: """Call `gh api` and return parsed JSON.""" url = endpoint if params: qs = "&".join(f"{k}={v}" for k, v in params.items()) url = f"{endpoint}?{qs}" try: r = subprocess.run( ["gh", "api", url], capture_output=True, text=True, check=True, ) except FileNotFoundError: print( "ERROR: 'gh' CLI not found. Install it or run 'gh auth login'.", file=sys.stderr, ) sys.exit(1) except subprocess.CalledProcessError as e: print(f"ERROR: gh api {url} failed: {e.stderr.strip()}", file=sys.stderr) return {} return json.loads(r.stdout) def get_latest_workflow_run(repo: str, workflow_file: str) -> dict | None: """Get the latest run of a workflow.""" data = gh_api_json( f"/repos/{repo}/actions/workflows/{workflow_file}/runs", per_page="1", ) runs = data.get("workflow_runs", []) return runs[0] if runs else None def run_extract_and_analyze(run_id: int) -> dict | None: """Call extract_and_analyze.py script and get analysis result.""" if not EXTRACT_SCRIPT.exists(): print(f"ERROR: extract_and_analyze.py not found at {EXTRACT_SCRIPT}", file=sys.stderr) return None try: result = subprocess.run( [ sys.executable, str(EXTRACT_SCRIPT), "--run-id", str(run_id), ], capture_output=True, text=True, check=True, ) return json.loads(result.stdout) except subprocess.CalledProcessError as e: print(f"ERROR: extract_and_analyze.py failed: {e.stderr}", file=sys.stderr) return None except json.JSONDecodeError as e: print(f"ERROR: Failed to parse extract_and_analyze.py output: {e}", file=sys.stderr) return None def analyze_workflow_run(repo: str, workflow_name: str, run: dict) -> NightlyRun: """Analyze a workflow run using extract_and_analyze.py.""" run_id = run["id"] result = NightlyRun( workflow_name=workflow_name, run_id=run_id, run_url=run.get("html_url", ""), conclusion=run.get("conclusion", "unknown"), created_at=run.get("created_at", "")[:10], ) # If success, no need to analyze if run.get("conclusion") == "success": # Get job count anyway jobs_data = gh_api_json(f"/repos/{repo}/actions/runs/{run_id}/jobs", per_page="100") result.total_jobs = len(jobs_data.get("jobs", [])) return result # Call extract_and_analyze.py for detailed analysis print(f" Analyzing run {run_id} with extract_and_analyze.py...", file=sys.stderr) analysis = run_extract_and_analyze(run_id) if analysis: result.total_jobs = analysis.get("total_jobs", 0) result.failed_jobs = analysis.get("failed_jobs_count", 0) result.failed_tests = analysis.get("failed_tests", []) result.code_bugs = analysis.get("code_bugs", []) result.env_flakes = analysis.get("env_flakes", []) return result def generate_report(runs: list[NightlyRun], repo: str) -> str: """Generate a markdown report of nightly status.""" lines = [ "## Nightly CI Status", "", f"Repository: {repo}", f"Generated: {datetime.now().strftime('%Y-%m-%d %H:%M:%S')}", "", "### Summary", "", "| Workflow | Status | Jobs | Failed Tests | Code Bugs | Env Flakes | Run |", "|----------|--------|------|--------------|-----------|------------|-----|", ] for run in runs: status_icon = "✅" if run.conclusion == "success" else "❌" status = f"{status_icon} {run.conclusion}" if run.total_jobs > 0: jobs = f"{run.failed_jobs}/{run.total_jobs} failed" else: jobs = "-" tests = str(len(run.failed_tests)) bugs = str(len(run.code_bugs)) flakes = str(len(run.env_flakes)) link = f"[#{run.run_id}]({run.run_url})" lines.append(f"| {run.workflow_name} | {status} | {jobs} | {tests} | {bugs} | {flakes} | {link} |") lines.append("") # Detail section for each failed workflow for run in runs: if run.conclusion == "success": continue lines.append(f"### {run.workflow_name} Details") lines.append("") lines.append(f"**Run**: [{run.run_id}]({run.run_url})") lines.append(f"**Date**: {run.created_at}") lines.append("") if run.code_bugs: lines.append("#### Code Bugs (Need Fix)") lines.append("") lines.append("| Error Type | Message |") lines.append("|------------|---------|") for bug in run.code_bugs[:10]: msg = bug.get("error_message", "").replace("|", "\\|") if len(msg) > 80: msg = msg[:77] + "..." lines.append(f"| {bug.get('error_type', 'Unknown')} | {msg} |") if len(run.code_bugs) > 10: lines.append(f"| ... | {len(run.code_bugs) - 10} more |") lines.append("") if run.env_flakes: lines.append("#### Environment Flakes") lines.append("") lines.append("| Error Type | Message |") lines.append("|------------|---------|") for flake in run.env_flakes[:5]: msg = flake.get("error_message", "").replace("|", "\\|") if len(msg) > 80: msg = msg[:77] + "..." lines.append(f"| {flake.get('error_type', 'Unknown')} | {msg} |") if len(run.env_flakes) > 5: lines.append(f"| ... | {len(run.env_flakes) - 5} more |") lines.append("") if run.failed_tests: lines.append("#### Failed Tests") lines.append("") for test in run.failed_tests[:15]: lines.append(f"- `{test}`") if len(run.failed_tests) > 15: lines.append(f"- ... and {len(run.failed_tests) - 15} more") lines.append("") # Summary for checklist lines.append("### Summary for Release Checklist") lines.append("") lines.append("Copy the following to the 'Nightly Status' section:") lines.append("") lines.append("```markdown") all_passed = all(r.conclusion == "success" for r in runs) if all_passed: lines.append("- [x] All nightly tests passing") else: for run in runs: if run.conclusion == "success": lines.append(f"- [x] {run.workflow_name}: Passing") else: lines.append( f"- [ ] {run.workflow_name}: {len(run.code_bugs)} code bugs, {len(run.failed_tests)} failed tests" ) lines.append("```") return "\n".join(lines) def main(): parser = argparse.ArgumentParser(description="Scan Nightly CI status for release readiness") parser.add_argument("--repo", default=REPO, help="Repository") parser.add_argument("--output", required=True, help="Output file path") args = parser.parse_args() runs = [] for workflow_name, workflow_file in NIGHTLY_WORKFLOWS.items(): print(f"Fetching latest {workflow_name} run...", file=sys.stderr) run = get_latest_workflow_run(args.repo, workflow_file) if not run: print(f" No runs found for {workflow_name}", file=sys.stderr) continue print( f" Found run {run['id']}: {run.get('conclusion', 'unknown')}", file=sys.stderr, ) result = analyze_workflow_run(args.repo, workflow_name, run) runs.append(result) if not runs: print("No nightly runs found.", file=sys.stderr) return 1 # Generate report print("Generating report...", file=sys.stderr) report = generate_report(runs, args.repo) output_path = Path(args.output) output_path.parent.mkdir(parents=True, exist_ok=True) with open(output_path, "w") as f: f.write(report) print(f"Report saved to {output_path}", file=sys.stderr) # Print summary print("\nSummary:", file=sys.stderr) for run in runs: status = "✅" if run.conclusion == "success" else "❌" print( f" {status} {run.workflow_name}: {run.failed_jobs}/{run.total_jobs} jobs failed, " f"{len(run.code_bugs)} code bugs, {len(run.env_flakes)} env flakes", file=sys.stderr, ) return 0 if __name__ == "__main__": exit(main()) - .agents/skills/vllm-ascend-release/scripts/scan_release_bugs.py (12988 bytes)
#!/usr/bin/env python3 """ Scan GitHub issues since the last release to identify bugs and problems. This script: 1. Gets the release date of the previous version (including rc versions) 2. Fetches all issues created since that date 3. Lists issues with titles for quick browsing 4. Allows marking issues that need detailed review The output is designed for human review - browse titles quickly and flag issues that need more investigation. Usage: python scan_release_bugs.py \ --repo vllm-project/vllm-ascend \ --since-tag v0.15.0rc1 \ --output issue-scan.md """ import argparse import json import subprocess from dataclasses import dataclass, field from datetime import datetime from pathlib import Path @dataclass class Issue: """An issue from the repository.""" number: int title: str url: str state: str created_at: str author: str labels: list[str] = field(default_factory=list) comments_count: int = 0 reactions_count: int = 0 body_preview: str = "" needs_review: bool = False review_reason: str = "" def run_gh_command(args: list[str]) -> str: """Run a GitHub CLI command and return the output.""" result = subprocess.run(["gh"] + args, capture_output=True, text=True) if result.returncode != 0: raise Exception(f"gh command failed: {result.stderr}") return result.stdout def get_release_date(repo: str, tag: str) -> str: """Get the release date for a tag.""" try: output = run_gh_command( [ "release", "view", tag, "--repo", repo, "--json", "publishedAt", ] ) data = json.loads(output) return data.get("publishedAt", "")[:10] # Just the date part except Exception: # Fallback: try to get tag creation date try: output = run_gh_command( [ "api", f"repos/{repo}/git/refs/tags/{tag}", ] ) data = json.loads(output) sha = data.get("object", {}).get("sha", "") if sha: commit_output = run_gh_command( [ "api", f"repos/{repo}/commits/{sha}", ] ) commit_data = json.loads(commit_output) date = commit_data.get("commit", {}).get("committer", {}).get("date", "") return date[:10] if date else "" except Exception: pass return "" def fetch_issues_since(repo: str, since_date: str, state: str = "all") -> list[dict]: """Fetch issues created since a given date.""" output = run_gh_command( [ "issue", "list", "--repo", repo, "--state", state, "--search", f"created:>{since_date}", "--limit", "300", "--json", "number,title,url,state,createdAt,author,labels,comments,reactionGroups,body", ] ) return json.loads(output) def calculate_reactions(reaction_groups: list[dict]) -> int: """Calculate total reactions from reaction groups.""" total = 0 for group in reaction_groups: total += group.get("totalCount", 0) return total def should_flag_for_review(issue: dict) -> tuple[bool, str]: """Determine if an issue should be flagged for detailed review.""" title_lower = issue["title"].lower() labels = [label["name"].lower() for label in issue.get("labels", [])] reactions = calculate_reactions(issue.get("reactionGroups", [])) comments = len(issue.get("comments", [])) reasons = [] # High engagement indicates important issues if reactions >= 5: reasons.append(f"high reactions ({reactions})") if comments >= 5: reasons.append(f"many comments ({comments})") # Important labels important_labels = ["bug", "regression", "blocker", "priority:high", "critical"] for label in important_labels: if label in labels: reasons.append(f"has label '{label}'") # Keywords in title that suggest important issues important_keywords = [ "crash", "hang", "freeze", "oom", "memory", "leak", "error", "fail", "broken", "regression", "block", "urgent", "critical", ] for keyword in important_keywords: if keyword in title_lower: reasons.append(f"title contains '{keyword}'") break return len(reasons) > 0, ", ".join(reasons) def parse_issue(issue: dict) -> Issue: """Parse raw issue data into Issue object.""" should_review, reason = should_flag_for_review(issue) body = issue.get("body", "") or "" body_preview = body[:300].replace("\n", " ").strip() if len(body) > 300: body_preview += "..." return Issue( number=issue["number"], title=issue["title"], url=issue["url"], state=issue["state"], created_at=issue.get("createdAt", "")[:10], author=issue.get("author", {}).get("login", "unknown"), labels=[label["name"] for label in issue.get("labels", [])], comments_count=len(issue.get("comments", [])), reactions_count=calculate_reactions(issue.get("reactionGroups", [])), body_preview=body_preview, needs_review=should_review, review_reason=reason, ) def generate_report(issues: list[Issue], repo: str, since_tag: str, since_date: str) -> str: """Generate a markdown report for human review.""" lines = [ "## Issue Scan Report", "", f"Repository: {repo}", f"Since: {since_tag} ({since_date})", f"Generated: {datetime.now().strftime('%Y-%m-%d %H:%M:%S')}", f"Total issues: {len(issues)}", "", ] # Separate open and closed issues open_issues = [i for i in issues if i.state == "OPEN"] closed_issues = [i for i in issues if i.state == "CLOSED"] # Section 1: Issues flagged for review (high priority) flagged = [i for i in issues if i.needs_review] lines.append("### Flagged for Review") lines.append("") lines.append("These issues have been automatically flagged based on engagement or keywords.") lines.append("Please review the titles and check details if needed.") lines.append("") if flagged: for issue in flagged: state_icon = "🔴" if issue.state == "OPEN" else "🟢" lines.append(f"#### {state_icon} #{issue.number}: {issue.title}") lines.append("") lines.append(f"- **State**: {issue.state}") lines.append(f"- **Reason flagged**: {issue.review_reason}") lines.append(f"- **Labels**: {', '.join(issue.labels) if issue.labels else 'none'}") lines.append(f"- **Author**: @{issue.author}") lines.append(f"- **Created**: {issue.created_at}") lines.append(f"- **Engagement**: {issue.reactions_count} reactions, {issue.comments_count} comments") lines.append(f"- **Link**: {issue.url}") if issue.body_preview: lines.append(f"- **Preview**: {issue.body_preview}") lines.append("") else: lines.append("No issues flagged for review.") lines.append("") # Section 2: All open issues (quick browse) lines.append("### All Open Issues") lines.append("") lines.append("Quick browse - check titles and flag any that look important:") lines.append("") if open_issues: lines.append("| # | Title | Labels | Date | Engagement |") lines.append("|---|-------|--------|------|------------|") for issue in open_issues: labels_str = ", ".join(issue.labels[:3]) if issue.labels else "-" engagement = f"{issue.reactions_count}👍 {issue.comments_count}💬" # Escape pipe characters in title safe_title = issue.title.replace("|", "\\|") if len(safe_title) > 60: safe_title = safe_title[:57] + "..." lines.append( f"| [#{issue.number}]({issue.url}) | {safe_title} | {labels_str} | {issue.created_at} | {engagement} |" ) lines.append("") else: lines.append("No open issues found.") lines.append("") # Section 3: Recently closed issues (might be relevant) lines.append("### Recently Closed Issues") lines.append("") lines.append("These issues were closed but might still be relevant for release notes:") lines.append("") if closed_issues: # Sort by engagement and show top ones closed_issues.sort(key=lambda i: -(i.reactions_count + i.comments_count)) top_closed = closed_issues[:20] lines.append("| # | Title | Labels | Date | Engagement |") lines.append("|---|-------|--------|------|------------|") for issue in top_closed: labels_str = ", ".join(issue.labels[:3]) if issue.labels else "-" engagement = f"{issue.reactions_count}👍 {issue.comments_count}💬" safe_title = issue.title.replace("|", "\\|") if len(safe_title) > 60: safe_title = safe_title[:57] + "..." lines.append( f"| [#{issue.number}]({issue.url}) | {safe_title} | {labels_str} | {issue.created_at} | {engagement} |" ) if len(closed_issues) > 20: lines.append("") lines.append(f"*... and {len(closed_issues) - 20} more closed issues*") lines.append("") else: lines.append("No closed issues found.") lines.append("") # Summary for checklist lines.append("### Summary for Release Checklist") lines.append("") lines.append("After reviewing the issues above, add important ones to the checklist:") lines.append("") lines.append("```markdown") lines.append("### Bug need Solve") lines.append("") # Pre-fill with flagged open issues flagged_open = [i for i in flagged if i.state == "OPEN"] if flagged_open: for issue in flagged_open[:10]: lines.append(f"- [ ] #{issue.number} - {issue.title[:50]}") else: lines.append("- [ ] (Add issues after review)") lines.append("```") lines.append("") # Statistics lines.append("### Statistics") lines.append("") lines.append(f"- Total issues scanned: {len(issues)}") lines.append(f"- Open issues: {len(open_issues)}") lines.append(f"- Closed issues: {len(closed_issues)}") lines.append(f"- Flagged for review: {len(flagged)}") return "\n".join(lines) def main(): parser = argparse.ArgumentParser(description="Scan GitHub issues since the last release") parser.add_argument("--repo", default="vllm-project/vllm-ascend", help="Repository") parser.add_argument("--since-tag", required=True, help="Previous release tag (including rc)") parser.add_argument("--state", default="all", choices=["open", "closed", "all"], help="Issue state filter") parser.add_argument("--output", required=True, help="Output file path") args = parser.parse_args() # Get release date print(f"Getting release date for {args.since_tag}...") since_date = get_release_date(args.repo, args.since_tag) if not since_date: print(f"Warning: Could not get release date for {args.since_tag}") print("Please enter the date manually (YYYY-MM-DD):") since_date = input().strip() print(f"Scanning issues since {since_date}...") # Fetch issues print(f"Fetching issues from {args.repo}...") raw_issues = fetch_issues_since(args.repo, since_date, args.state) print(f"Found {len(raw_issues)} issues") # Parse issues issues = [parse_issue(issue) for issue in raw_issues] # Sort: flagged first, then by date descending issues.sort(key=lambda i: (not i.needs_review, i.created_at), reverse=True) # Generate report print("Generating report...") report = generate_report(issues, args.repo, args.since_tag, since_date) output_path = Path(args.output) output_path.parent.mkdir(parents=True, exist_ok=True) with open(output_path, "w") as f: f.write(report) print(f"Report saved to {output_path}") # Print summary flagged = [i for i in issues if i.needs_review] open_issues = [i for i in issues if i.state == "OPEN"] print("\nSummary:") print(f" Total issues: {len(issues)}") print(f" Open issues: {len(open_issues)}") print(f" Flagged for review: {len(flagged)}") print("\nPlease review the report and identify issues that need to be fixed before release.") return 0 if __name__ == "__main__": exit(main()) - .agents/skills/vllm-ascend-release/scripts/scan_test_coverage.py (17077 bytes)
#!/usr/bin/env python3 """ Scan for features and new models that need manual testing. This script identifies: 1. New features/models merged since the last release that may lack CI test coverage 2. Outstanding issues from the previous release's feedback issue These are typically features that: - Don't have corresponding test cases in CI - Were merged without tests due to environment constraints - New model support that needs validation Usage: python scan_test_coverage.py \ --repo vllm-project/vllm-ascend \ --since-tag v0.15.0rc1 \ --feedback-issue 1234 \ --output test-coverage-analysis.md """ import argparse import json import re import subprocess from dataclasses import dataclass, field from datetime import datetime from pathlib import Path @dataclass class FeatureItem: """A feature or model that may need testing.""" pr_number: int title: str url: str merged_at: str author: str labels: list[str] = field(default_factory=list) category: str = "" # "model", "feature", "optimization", etc. has_tests: bool = False test_files: list[str] = field(default_factory=list) reason: str = "" # Why it needs testing @dataclass class FeedbackItem: """An issue from the feedback thread.""" comment_id: int author: str body: str created_at: str linked_issue: int | None = None # Linked issue number if any status: str = "unknown" # "resolved", "unresolved", "unclear" resolution_note: str = "" # Keywords that indicate new model support MODEL_KEYWORDS = [ "support", "add", "enable", "model", "llm", "vlm", "moe", "mtp", "qwen", "deepseek", "llama", "glm", "kimi", "mixtral", "phi", "gemma", "internlm", "baichuan", "yi", "chatglm", ] # Keywords that indicate features needing testing FEATURE_KEYWORDS = [ "graph mode", "expert parallel", "tensor parallel", "pipeline parallel", "multimodal", "speculative", "chunked prefill", "prefix caching", "quantization", "awq", "gptq", "fp8", "w8a8", "performance", "optimization", ] # Labels that indicate new features FEATURE_LABELS = [ "enhancement", "feature", "new model", "new-model", "model support", ] def run_gh_command(args: list[str]) -> str: """Run a GitHub CLI command and return the output.""" result = subprocess.run(["gh"] + args, capture_output=True, text=True) if result.returncode != 0: raise Exception(f"gh command failed: {result.stderr}") return result.stdout def get_release_date(repo: str, tag: str) -> str: """Get the release date for a tag.""" try: output = run_gh_command( [ "release", "view", tag, "--repo", repo, "--json", "publishedAt", ] ) data = json.loads(output) return data.get("publishedAt", "") except Exception: # Fallback: try to get tag creation date try: output = run_gh_command( [ "api", f"repos/{repo}/git/refs/tags/{tag}", ] ) data = json.loads(output) # Get the commit date sha = data.get("object", {}).get("sha", "") if sha: commit_output = run_gh_command( [ "api", f"repos/{repo}/commits/{sha}", ] ) commit_data = json.loads(commit_output) return commit_data.get("commit", {}).get("committer", {}).get("date", "") except Exception: pass return "" def fetch_merged_prs(repo: str, since_date: str) -> list[dict]: """Fetch PRs merged since a given date.""" output = run_gh_command( [ "pr", "list", "--repo", repo, "--state", "merged", "--search", f"merged:>{since_date}", "--limit", "200", "--json", "number,title,url,mergedAt,author,labels,files", ] ) return json.loads(output) def check_pr_has_tests(pr: dict) -> tuple[bool, list[str]]: """Check if a PR includes test files.""" test_files = [] files = pr.get("files", []) for file in files: path = file.get("path", "") if "test" in path.lower() and path.endswith(".py"): test_files.append(path) return len(test_files) > 0, test_files def categorize_pr(pr: dict) -> str: """Categorize a PR based on its content.""" title_lower = pr["title"].lower() labels = [label["name"].lower() for label in pr.get("labels", [])] # Check for model support for keyword in ["model", "support", "add"]: if keyword in title_lower: for model in [ "qwen", "deepseek", "llama", "glm", "kimi", "mixtral", "phi", "gemma", "internlm", "baichuan", "yi", "chatglm", ]: if model in title_lower: return "model" # Check labels for label in labels: if "model" in label: return "model" if "feature" in label or "enhancement" in label: return "feature" # Check for specific features for feature in [ "graph", "parallel", "multimodal", "speculative", "chunked", "prefix", "quantization", "performance", ]: if feature in title_lower: return "feature" if "fix" in title_lower or "bug" in title_lower: return "bugfix" if "doc" in title_lower or "readme" in title_lower: return "docs" if "ci" in title_lower or "test" in title_lower: return "ci" return "other" def analyze_pr(pr: dict) -> FeatureItem | None: """Analyze a PR to determine if it needs testing.""" category = categorize_pr(pr) # Skip docs, CI changes, and bug fixes (they should be tested via CI) if category in ["docs", "ci", "bugfix", "other"]: return None has_tests, test_files = check_pr_has_tests(pr) # If it's a feature/model and has tests, it's likely covered by CI # But we still report it for visibility item = FeatureItem( pr_number=pr["number"], title=pr["title"], url=pr["url"], merged_at=pr.get("mergedAt", ""), author=pr.get("author", {}).get("login", "unknown"), labels=[label["name"] for label in pr.get("labels", [])], category=category, has_tests=has_tests, test_files=test_files, ) if not has_tests: item.reason = "No test files included in PR" else: item.reason = f"Has tests: {', '.join(test_files[:3])}" return item def fetch_feedback_comments(repo: str, issue_number: int) -> list[dict]: """Fetch comments from a feedback issue.""" output = run_gh_command( [ "issue", "view", str(issue_number), "--repo", repo, "--json", "comments", ] ) data = json.loads(output) return data.get("comments", []) def analyze_feedback_comment(comment: dict, repo: str) -> FeedbackItem | None: """Analyze a feedback comment to extract issues.""" body = comment.get("body", "") # Skip bot comments and simple reactions author = comment.get("author", {}).get("login", "") if "bot" in author.lower(): return None # Skip very short comments (likely reactions) if len(body.strip()) < 50: return None # Try to extract linked issue numbers issue_matches = re.findall(r"#(\d+)", body) linked_issue = int(issue_matches[0]) if issue_matches else None item = FeedbackItem( comment_id=comment.get("id", 0), author=author, body=body[:500] + "..." if len(body) > 500 else body, created_at=comment.get("createdAt", ""), linked_issue=linked_issue, ) # Check if there's a linked issue and if it's resolved if linked_issue: try: issue_output = run_gh_command( [ "issue", "view", str(linked_issue), "--repo", repo, "--json", "state,title", ] ) issue_data = json.loads(issue_output) if issue_data.get("state") == "CLOSED": item.status = "resolved" item.resolution_note = f"Issue #{linked_issue} is closed" else: item.status = "unresolved" item.resolution_note = f"Issue #{linked_issue} is still open" except Exception: item.status = "unknown" item.resolution_note = f"Could not fetch issue #{linked_issue}" else: # No linked issue, mark as needing review item.status = "needs_review" item.resolution_note = "No linked issue found, needs manual review" return item def generate_report( features: list[FeatureItem], feedback: list[FeedbackItem], repo: str, since_tag: str, ) -> str: """Generate a markdown report.""" lines = [ "## Test Coverage Analysis", "", f"Repository: {repo}", f"Since: {since_tag}", f"Generated: {datetime.now().strftime('%Y-%m-%d %H:%M:%S')}", "", ] # Section 1: Features/Models needing tests lines.append("### Features/Models Needing Manual Testing") lines.append("") lines.append("The following features/models were merged without test coverage in CI:") lines.append("") # Filter to only those without tests needs_testing = [f for f in features if not f.has_tests] if needs_testing: # Group by category models = [f for f in needs_testing if f.category == "model"] other_features = [f for f in needs_testing if f.category == "feature"] if models: lines.append("#### New Models") lines.append("") for f in models: lines.append(f"- [ ] **{f.title}**") lines.append(f" - PR: #{f.pr_number}") lines.append(f" - Author: @{f.author}") lines.append(f" - Reason: {f.reason}") lines.append("") if other_features: lines.append("#### New Features") lines.append("") for f in other_features: lines.append(f"- [ ] **{f.title}**") lines.append(f" - PR: #{f.pr_number}") lines.append(f" - Author: @{f.author}") lines.append(f" - Reason: {f.reason}") lines.append("") else: lines.append("All merged features/models appear to have test coverage.") lines.append("") # Section 2: Features with tests (for reference) has_tests = [f for f in features if f.has_tests] if has_tests: lines.append("### Features With Test Coverage (Reference)") lines.append("") lines.append("These features have tests and should be covered by CI:") lines.append("") for f in has_tests[:10]: # Limit to 10 lines.append(f"- {f.title} (#{f.pr_number})") if len(has_tests) > 10: lines.append(f"- ... and {len(has_tests) - 10} more") lines.append("") # Section 3: Feedback issue status lines.append("### Previous Release Feedback Status") lines.append("") if feedback: unresolved = [f for f in feedback if f.status == "unresolved"] needs_review = [f for f in feedback if f.status == "needs_review"] resolved = [f for f in feedback if f.status == "resolved"] if unresolved: lines.append("#### Unresolved Issues") lines.append("") for f in unresolved: lines.append(f"- [ ] Comment by @{f.author}") if f.linked_issue: lines.append(f" - Linked issue: #{f.linked_issue} (still open)") lines.append(f" - Preview: {f.body[:200]}...") lines.append("") if needs_review: lines.append("#### Needs Manual Review") lines.append("") for f in needs_review: lines.append(f"- [ ] Comment by @{f.author}") lines.append(f" - {f.resolution_note}") lines.append(f" - Preview: {f.body[:200]}...") lines.append("") if resolved: lines.append("#### Resolved") lines.append("") for f in resolved[:5]: lines.append(f"- [x] Comment by @{f.author} - {f.resolution_note}") if len(resolved) > 5: lines.append(f"- ... and {len(resolved) - 5} more resolved items") lines.append("") else: lines.append("No feedback comments found or no feedback issue specified.") lines.append("") # Summary for checklist lines.append("### Summary for Release Checklist") lines.append("") lines.append("Copy the following to the 'Functional Test' section:") lines.append("") lines.append("```markdown") lines.append("#### Manual Testing Required") lines.append("") for f in needs_testing[:10]: lines.append(f"- [ ] {f.title} (#{f.pr_number})") if len(needs_testing) > 10: lines.append(f"- [ ] ... and {len(needs_testing) - 10} more items") lines.append("") lines.append("#### Feedback Issues to Address") lines.append("") unresolved_count = len([f for f in feedback if f.status in ["unresolved", "needs_review"]]) if unresolved_count > 0: lines.append(f"- [ ] {unresolved_count} unresolved feedback items") else: lines.append("- [x] All feedback items addressed") lines.append("```") return "\n".join(lines) def main(): parser = argparse.ArgumentParser(description="Scan for features/models needing manual testing") parser.add_argument("--repo", default="vllm-project/vllm-ascend", help="Repository") parser.add_argument("--since-tag", required=True, help="Previous release tag") parser.add_argument("--feedback-issue", type=int, help="Previous release feedback issue number") parser.add_argument("--output", required=True, help="Output file path") args = parser.parse_args() # Get release date print(f"Getting release date for {args.since_tag}...") release_date = get_re - .agents/skills/vllm-ascend-release/scripts/update_checklist_section.py (4823 bytes)
#!/usr/bin/env python3 """ Update a specific section of the release checklist issue. This script uses the GitHub CLI to update the body of an issue, replacing or appending content to a specific section. Usage: python update_checklist_section.py \ --issue-number 6149 \ --section "Bug need Solve" \ --content-file bug-list.md """ import argparse import json import re import subprocess from pathlib import Path def run_gh_command(args: list[str]) -> str: """Run a GitHub CLI command and return the output.""" result = subprocess.run(["gh"] + args, capture_output=True, text=True) if result.returncode != 0: raise Exception(f"gh command failed: {result.stderr}") return result.stdout def get_issue_body(repo: str, issue_number: int) -> str: """Get the current body of an issue.""" output = run_gh_command( [ "issue", "view", str(issue_number), "--repo", repo, "--json", "body", ] ) data = json.loads(output) return data.get("body", "") def update_issue_body(repo: str, issue_number: int, new_body: str): """Update the body of an issue.""" run_gh_command( [ "issue", "edit", str(issue_number), "--repo", repo, "--body", new_body, ] ) def find_section(body: str, section_name: str) -> tuple[int, int]: """ Find the start and end positions of a section in the issue body. Returns: Tuple of (start_pos, end_pos) where end_pos is the start of the next section or end of the body. """ # Match section header (### Section Name) section_pattern = rf"^###\s+{re.escape(section_name)}\s*$" section_match = re.search(section_pattern, body, re.MULTILINE) if not section_match: return -1, -1 start_pos = section_match.end() # Find the next section (### followed by any text) next_section_pattern = r"^###\s+" next_match = re.search(next_section_pattern, body[start_pos:], re.MULTILINE) if next_match: end_pos = start_pos + next_match.start() else: end_pos = len(body) return start_pos, end_pos def update_section(body: str, section_name: str, new_content: str, append: bool = False) -> str: """ Update a section in the issue body. Args: body: The current issue body section_name: Name of the section to update new_content: New content for the section append: If True, append to existing content; if False, replace Returns: Updated body text """ start_pos, end_pos = find_section(body, section_name) if start_pos == -1: # Section not found, append at the end return body + f"\n\n### {section_name}\n\n{new_content}" if append: # Get existing content and append existing_content = body[start_pos:end_pos].strip() new_section_content = f"\n\n{existing_content}\n\n{new_content}\n\n" else: # Replace content new_section_content = f"\n\n{new_content}\n\n" return body[:start_pos] + new_section_content + body[end_pos:] def main(): parser = argparse.ArgumentParser(description="Update a section of the release checklist issue") parser.add_argument("--repo", default="vllm-project/vllm-ascend", help="Repository") parser.add_argument("--issue-number", type=int, required=True, help="Issue number") parser.add_argument("--section", required=True, help="Section name to update") parser.add_argument("--content-file", required=True, help="File containing new content") parser.add_argument("--append", action="store_true", help="Append to section instead of replace") parser.add_argument("--dry-run", action="store_true", help="Print changes without updating") args = parser.parse_args() content_path = Path(args.content_file) if not content_path.exists(): print(f"Error: Content file not found: {content_path}") return 1 with open(content_path) as f: new_content = f.read().strip() print(f"Fetching issue #{args.issue_number}...") body = get_issue_body(args.repo, args.issue_number) print(f"Updating section '{args.section}'...") updated_body = update_section(body, args.section, new_content, args.append) if args.dry_run: print("\n--- DRY RUN ---") print("Updated body would be:") print("-" * 40) print(updated_body) print("-" * 40) else: update_issue_body(args.repo, args.issue_number, updated_body) print(f"Successfully updated section '{args.section}' in issue #{args.issue_number}") return 0 if __name__ == "__main__": exit(main()) - .agents/skills/vllm-ascend-release/scripts/update_version_references.py (10701 bytes)
#!/usr/bin/env python3 """ Update version references across documentation files. This script loads configuration from references/version-files.yaml and updates version numbers and compatibility information in various files for a new release. Usage: python update_version_references.py \ --version v0.15.0rc1 \ --vllm-version v0.15.0 \ --feedback-issue https://github.com/vllm-project/vllm-ascend/issues/1234 """ import argparse import glob import re from datetime import datetime from pathlib import Path import yaml def load_config(config_path: Path) -> dict: """Load version files configuration from YAML.""" if not config_path.exists(): raise FileNotFoundError(f"Config file not found: {config_path}") with open(config_path) as f: return yaml.safe_load(f) def expand_glob_pattern(repo_root: Path, pattern: str) -> list[Path]: """Expand glob patterns to actual file paths.""" if "*" in pattern: # Use glob to expand the pattern full_pattern = str(repo_root / pattern) matches = glob.glob(full_pattern) return [Path(m) for m in matches] else: # Single file path return [repo_root / pattern] def build_regex_pattern(pattern_template: str) -> str: """ Convert a template pattern like 'pip install vllm-ascend==X.X.X' to a regex pattern like 'pip install vllm-ascend==[\\d.]+(?:rc\\d+)?' """ # Escape special regex characters except our placeholder escaped = re.escape(pattern_template) # Replace escaped X.X.X placeholder with version regex escaped = escaped.replace(r"X\.X\.X", r"[\d.]+(?:rc\d+)?") # Replace escaped XXXX (issue number) with digit regex escaped = escaped.replace("XXXX", r"\d+") return escaped def update_file_content( file_path: Path, updates: list[dict], variables: dict, dry_run: bool = False, ) -> tuple[bool, list[str]]: """ Update a single file with version references. Returns: Tuple of (modified: bool, messages: list of update messages) """ messages = [] if not file_path.exists(): messages.append(f" Warning: File not found: {file_path}") return False, messages with open(file_path) as f: content = f.read() modified = False for update in updates: # Check if this update has a regex pattern or a template pattern if "pattern" in update: pattern_template = update["pattern"] # Build regex from template pattern regex_pattern = build_regex_pattern(pattern_template) # Build replacement string if "replacement" in update: replacement = update["replacement"].format(**variables) else: # Use the pattern template with version substituted replacement = pattern_template.replace("X.X.X", variables.get("version_num", "")) replacement = replacement.replace("XXXX", variables.get("feedback_issue_num", "")) new_content = re.sub(regex_pattern, replacement, content) if new_content != content: content = new_content modified = True messages.append(f" Updated pattern: {pattern_template[:50]}...") elif "variable" in update: # Handle variable-based updates (e.g., VLLM_VERSION in workflow files) var_name = update["variable"] # Common patterns for variable definitions patterns = [ # YAML: VLLM_VERSION: "0.15.0" ( rf'({var_name}:\s*["\']?)[\d.]+(?:rc\d+)?(["\']?)', rf"\g<1>{variables.get('vllm_version_num', '')}\g<2>", ), # Shell: VLLM_VERSION=0.15.0 (rf"({var_name}=)[\d.]+(?:rc\d+)?", rf"\g<1>{variables.get('vllm_version_num', '')}"), # Docker ARG: ARG VLLM_VERSION=0.15.0 (rf"(ARG\s+{var_name}=)[\d.]+(?:rc\d+)?", rf"\g<1>{variables.get('vllm_version_num', '')}"), # ENV: ENV VLLM_VERSION=0.15.0 (rf"(ENV\s+{var_name}=)[\d.]+(?:rc\d+)?", rf"\g<1>{variables.get('vllm_version_num', '')}"), ] for pattern, replacement in patterns: new_content = re.sub(pattern, replacement, content) if new_content != content: content = new_content modified = True messages.append(f" Updated variable: {var_name}") break elif "action" in update: # This update requires manual action action = update["action"] description = update.get("description", "") messages.append(f" Manual action required: {action}") if description: messages.append(f" - {description}") if modified: if dry_run: messages.append(f" [DRY RUN] Would update {file_path}") else: with open(file_path, "w") as f: f.write(content) messages.append(f" Updated: {file_path}") return modified, messages def process_file_config( repo_root: Path, file_config: dict, variables: dict, dry_run: bool = False, ) -> tuple[list[str], list[str], list[str]]: """ Process a single file configuration entry. Returns: Tuple of (updated_files, manual_files, messages) """ updated_files = [] manual_files = [] messages = [] path_pattern = file_config["path"] updates = file_config.get("updates", []) # Expand glob patterns file_paths = expand_glob_pattern(repo_root, path_pattern) if not file_paths: messages.append(f"Processing {path_pattern}...") messages.append(f" Warning: No files matched pattern: {path_pattern}") return updated_files, manual_files, messages for file_path in file_paths: rel_path = file_path.relative_to(repo_root) if file_path.is_relative_to(repo_root) else file_path messages.append(f"Processing {rel_path}...") # Check if all updates are manual actions has_auto_updates = any("pattern" in u or "variable" in u for u in updates) has_manual_updates = any("action" in u for u in updates) if has_auto_updates: modified, update_messages = update_file_content(file_path, updates, variables, dry_run) messages.extend(update_messages) if modified: updated_files.append(str(rel_path)) if has_manual_updates: manual_files.append(str(rel_path)) for update in updates: if "action" in update: messages.append(f" Manual action: {update['action']}") if "description" in update: messages.append(f" - {update['description']}") return updated_files, manual_files, messages def main(): parser = argparse.ArgumentParser(description="Update version references across documentation") parser.add_argument("--version", required=True, help="New vLLM Ascend version (e.g., v0.15.0rc1)") parser.add_argument("--vllm-version", required=True, help="Compatible vLLM version (e.g., v0.15.0)") parser.add_argument("--feedback-issue", default="", help="Feedback issue URL") parser.add_argument("--repo-root", default=".", help="Repository root directory") parser.add_argument("--config", default=None, help="Config file path (default: references/version-files.yaml)") parser.add_argument("--dry-run", action="store_true", help="Print changes without updating") args = parser.parse_args() # Determine config path script_dir = Path(__file__).parent if args.config: config_path = Path(args.config) else: config_path = script_dir.parent / "references" / "version-files.yaml" # Load configuration try: config = load_config(config_path) except FileNotFoundError as e: print(f"Error: {e}") return 1 # Normalize versions version = args.version version_num = version.lstrip("v") vllm_version = args.vllm_version vllm_version_num = vllm_version.lstrip("v") # Extract feedback issue number if URL provided feedback_issue_num = "" if args.feedback_issue: match = re.search(r"/issues/(\d+)", args.feedback_issue) if match: feedback_issue_num = match.group(1) variables = { "version": version, "version_num": version_num, "vllm_version": vllm_version, "vllm_version_num": vllm_version_num, "feedback_issue": args.feedback_issue, "feedback_issue_num": feedback_issue_num, "date": datetime.now().strftime("%Y-%m-%d"), } repo_root = Path(args.repo_root).resolve() print(f"Updating version references to {version}...") print(f"Compatible vLLM version: {vllm_version}") print(f"Config file: {config_path}") print(f"Repository root: {repo_root}") print() all_updated_files = [] all_manual_files = [] # Get update order from config, or use file order update_order = config.get("update_order", {}) files_config = config.get("files", []) # Sort files by update order if specified if update_order: # Create a mapping from path to order path_to_order = {v: int(k) for k, v in update_order.items()} def get_order(file_config): path = file_config["path"] return path_to_order.get(path, 999) files_config = sorted(files_config, key=get_order) # Process each file configuration for file_config in files_config: updated, manual, messages = process_file_config(repo_root, file_config, variables, args.dry_run) all_updated_files.extend(updated) all_manual_files.extend(manual) for msg in messages: print(msg) print() # Print summary print("=" * 60) print("Summary") print("=" * 60) print(f"Files automatically updated: {len(all_updated_files)}") for f in all_updated_files: print(f" ✓ {f}") if all_manual_files: print() print(f"Files requiring manual updates: {len(set(all_manual_files))}") for f in sorted(set(all_manual_files)): print(f" ⚠ {f}") # Print validation hints from config validation = config.get("validation", []) if validation: print() print("Validation commands:") for cmd in validation: print(f" $ {cmd}") return 0 if __name__ == "__main__": exit(main()) - .agents/skills/vllm-ascend-release/SKILL.md (34617 bytes)
--- name: vllm-ascend-release description: "End-to-end release management skill for vLLM Ascend. Creates release checklist issues, triages critical bugs, audits test coverage, aggregates release note fragments, and guides through the complete release process." disable-model-invocation: true --- # vLLM Ascend Release Skill ## Overview This skill manages the complete end-to-end release process for vLLM Ascend, from creating the release checklist issue to final release announcement. It automates repetitive tasks while ensuring human oversight at critical decision points. ## When to Use This Skill Use this skill when: - Starting a new release cycle (RC or stable) - The release manager needs to track release progress - Preparing release artifacts (notes, documentation, tests) ## Prerequisites - GitHub CLI (`gh`) authenticated with write access to `vllm-project/vllm-ascend` - Access to Ascend NPU hardware for functional testing (or CI infrastructure) - Python environment with `uv` for running scripts ### Verify GitHub CLI Installation Before starting the release process, verify that `gh` CLI is installed and authenticated: ```bash # Check if gh is installed gh --version # If not installed, install gh CLI: # Ubuntu/Debian apt install gh -y # macOS brew install gh # OpenEuler yum install gh -y # Check authentication status gh auth status # If not authenticated, login with: gh auth login ``` Expected output for `gh auth status`: ``` github.com ✓ Logged in to github.com account <username> (keyring) - Active account: true - Git operations protocol: https - Token: gho_**** - Token scopes: 'gist', 'read:org', 'repo', 'workflow' ``` **Required scopes**: `repo` (for creating issues, PRs, releases) and `workflow` (for triggering CI workflows). ## Workflow Overview ``` ┌─────────────────────────────────────────────────────────────────────────────┐ │ vLLM Ascend Release Process │ ├─────────────────────────────────────────────────────────────────────────────┤ │ │ │ Phase 1: Initialization │ │ ├── Determine version & branch │ │ ├── Create feedback issue │ │ └── Create release checklist issue │ │ │ │ Phase 2: Bug Triage │ │ ├── Scan open bugs │ │ ├── Identify release-blocking bugs │ │ └── Update checklist with bug list │ │ │ │ Phase 3: PR Management │ │ ├── Identify must-merge PRs │ │ └── Update checklist with PR list │ │ │ │ Phase 4: Test Coverage Analysis │ │ ├── Scan PRs for features/models without tests │ │ ├── Check previous feedback issue status │ │ └── Update checklist with items needing manual testing │ │ │ │ Phase 5: Nightly Status │ │ ├── Get latest Nightly-A3 and Nightly-A2 runs │ │ ├── Analyze failures with extract_and_analyze.py │ │ └── Update checklist with nightly status table │ │ │ │ Phase 6: Release Notes │ │ ├── Aggregate release note fragments (release_notes/fragments) │ │ ├── Audit commits without fragments │ │ └── Create release notes PR │ │ │ │ Phase 7: Documentation & Artifacts │ │ └── Update version references (Docker/wheel built by CI automatically) │ │ │ │ Phase 8: Release Execution (requires human review) │ │ ├── Human review & approval │ │ ├── Merge release notes PR │ │ ├── Create GitHub release │ │ └── Verify automated pipelines (PyPI, Docker, ReadTheDocs) │ │ │ │ Phase 9: WeChat Article (微信公众号推文) │ │ ├── Collect release statistics (commits, contributors) │ │ ├── Generate WeChat article from template │ │ └── Review and publish to WeChat official account │ │ │ └─────────────────────────────────────────────────────────────────────────────┘ ``` ## Phase 1: Initialization ### 1.1 Gather Release Information Prompt the user for: - **Release Version**: e.g., `v0.15.0rc1`, `v0.15.0` - **Release Branch**: typically `main` - **Target Release Date**: e.g., `2026.03.15` - **Release Manager**: GitHub username ### 1.2 Determine Previous Version ```bash # Get the latest release tag gh release list --repo vllm-project/vllm-ascend --limit 5 # Or check existing tags git tag --sort=-creatordate | head -10 ``` ### 1.3 Create Feedback Issue Create a community feedback issue for the release: ```bash gh issue create --repo vllm-project/vllm-ascend \ --title "[Feedback]: v${VERSION} Release Feedback" \ --body "$(cat templates/feedback-issue-template.md)" \ --label "feedback" ``` ### 1.4 Create Release Checklist Issue Use the template in `templates/release-checklist-template.md`: ```bash # Generate the checklist from template python scripts/generate_checklist.py \ --version ${VERSION} \ --branch ${BRANCH} \ --date ${DATE} \ --manager ${MANAGER} \ --feedback-issue ${FEEDBACK_ISSUE_NUMBER} \ --output release-checklist.md # Create the issue gh issue create --repo vllm-project/vllm-ascend \ --title "[Release]: Release checklist for ${VERSION}" \ --body-file release-checklist.md \ --label "release" ``` ## Phase 2: Bug Triage ### 2.1 Scan Issues Since Last Release Run the issue scanning script to browse all issues since the last release: ```bash python scripts/scan_release_bugs.py \ --repo vllm-project/vllm-ascend \ --since-tag ${LAST_VERSION} \ --output issue-scan.md ``` The script: 1. Gets the release date of the previous version (including rc versions) 2. Fetches all issues created since that date 3. Generates a report with: - **Flagged issues**: Automatically flagged based on engagement or keywords - **All open issues**: Quick browse table with titles - **Recently closed issues**: May be relevant for release notes ### 2.2 Human Review Process The output is designed for quick human review: 1. **Check flagged issues first** - these have high engagement or concerning keywords 2. **Browse the open issues table** - scan titles, click to investigate if needed 3. **Review closed issues** - identify fixes that should be highlighted in release notes ### 2.3 Issue Flagging Criteria Issues are automatically flagged when they have: - High reactions (≥5) or many comments (≥5) - Labels: `bug`, `regression`, `blocker`, `priority:high`, `critical` - Keywords in title: crash, hang, freeze, oom, error, fail, etc. ### 2.4 Update Checklist After manual review, add important bugs to the release checklist: ```bash python scripts/update_checklist_section.py \ --issue-number ${CHECKLIST_ISSUE} \ --section "Bug need Solve" \ --content-file bug-list.md ``` ## Phase 3: PR Management ### 3.1 Identify Must-Merge PRs Scan for PRs that should be included in the release: ```bash # 1. [Priority] List open PRs/issues in the release milestone gh pr list --repo vllm-project/vllm-ascend \ --state open \ --search "milestone:${VERSION}" \ --json number,title,url,labels gh issue list --repo vllm-project/vllm-ascend \ --state open \ --search "milestone:${VERSION}" \ --json number,title,url,labels # 2. List open PRs with release-related labels gh pr list --repo vllm-project/vllm-ascend \ --state open \ --label "release-blocker" \ --json number,title,url # 3. List PRs merged since last release gh pr list --repo vllm-project/vllm-ascend \ --state merged \ --search "merged:>${LAST_RELEASE_DATE}" \ --json number,title,mergedAt ``` **Priority Order:** 1. PRs/Issues in the release milestone - these are explicitly targeted for this release 2. PRs with `release-blocker` label - critical items that must be merged 3. Recently merged PRs - for tracking what's already included ### 3.2 Update Checklist Update the checklist with PRs that need to be merged: ```bash python scripts/update_checklist_section.py \ --issue-number ${CHECKLIST_ISSUE} \ --section "PR need Merge" \ --content-file pr-list.md ``` ## Phase 4: Test Coverage Analysis ### 4.1 Identify Features/Models Needing Testing CI already covers most test cases. Manual testing is only needed for: - **New features** merged without test cases - **New models** added due to environment constraints (e.g., CI doesn't have the model) - **Issues** reported in the previous release's feedback Run the test coverage scanner: ```bash python scripts/scan_test_coverage.py \ --repo vllm-project/vllm-ascend \ --since-tag ${LAST_VERSION} \ --feedback-issue ${PREVIOUS_FEEDBACK_ISSUE} \ --output test-coverage-analysis.md ``` This script: 1. Scans PRs merged since the last release 2. Identifies features/models without corresponding test files 3. Checks the previous feedback issue for unresolved problems ### 4.2 Review the Analysis The output categorizes items: **Features/Models Needing Manual Testing:** - New model support (e.g., Kimi K2.5, GLM-5) - Features that couldn't be tested in CI **Previous Feedback Status:** - Unresolved issues from the feedback thread - Items that need manual verification ### 4.3 Manual Testing Checklist For items identified above, perform manual testing: ```markdown #### Manual Testing Required - [ ] Model: Kimi K2.5 - Basic inference works - [ ] Model: GLM-5 - Multimodal features work - [ ] Feature: Expert parallelism with 8 GPUs - [ ] Feedback: User reported slow startup (verify fixed) ``` ### 4.4 Update Checklist with Results ```bash python scripts/update_checklist_section.py \ --issue-number ${CHECKLIST_ISSUE} \ --section "Functional Test" \ --content-file test-results.md ``` ## Phase 5: Nightly Status ### 5.1 Analyze Nightly CI Runs Get the latest Nightly-A3 and Nightly-A2 CI runs and analyze failures: ```bash python scripts/scan_nightly_status.py \ --repo vllm-project/vllm-ascend \ --output nightly-status.md ``` This script: 1. Fetches the latest Nightly-A3 and Nightly-A2 workflow runs 2. Calls `extract_and_analyze.py` (from main2main-error-analysis skill) for failed runs 3. Extracts and categorizes errors: - **Code Bugs**: Real issues that need fixing - **Environment Flakes**: Transient issues (network, disk, etc.) ### 5.2 Review Output The output includes: | Workflow | Status | Failed Jobs | Code Bugs | Env Flakes | Run | |----------|--------|-------------|-----------|------------|-----| | Nightly-A3 | ✅ success | 0/15 | 0 | 0 | [#123](url) | | Nightly-A2 | ❌ failure | 3/12 | 2 | 1 | [#124](url) | For failed runs, it also shows: - Code bugs that need fixing before release - Failed test cases - Environment flakes (informational) ### 5.3 Update Checklist ```bash python scripts/update_checklist_section.py \ --issue-number ${CHECKLIST_ISSUE} \ --section "Nightly Status" \ --content-file nightly-status.md ``` ## Phase 6: Release Notes This phase assembles the release notes from the fragments collected in `release_notes/fragments/` (see `release_notes/README.md` for the fragment format and principles). Fragments are the primary source; the commit-level analysis is only a fallback audit for entries contributors forgot to declare. ### 6.1 Fetch Commits Fetch all commits between the previous and current version: ```bash # Create output directory mkdir -p output/${VERSION} # Fetch commits with contributor statistics uv run python scripts/fetch_commits.py \ --owner vllm-project \ --repo vllm-ascend \ --base-tag ${LAST_VERSION} \ --head-tag ${NEW_VERSION} \ --stats \ --output output/${VERSION}/0-current-raw-commits.md \ --stats-output output/${VERSION}/0-contributor-stats.md ``` The script outputs: - `0-current-raw-commits.md`: Raw commit list, used for the missing-fragment audit - `0-contributor-stats.md`: Contributor statistics including new contributors ### 6.2 Aggregate Fragments Validate, merge, and group all fragments into a draft: ```bash uv run python scripts/collect_release_notes.py \ --version ${NEW_VERSION} \ --raw-commits output/${VERSION}/0-current-raw-commits.md \ --output output/${VERSION}/1-fragments-draft.md ``` The script: 1. Validates the fragment schema and fails with precise errors on invalid files 2. Merges entries with identical text (for example cherry-picks) into a single bullet 3. Groups entries into release notes sections and lists `highlight: true` candidates 4. Emits the audit section: commits with a PR reference but no fragment, commits without a PR reference, and fragment PRs missing from the commit list It does not move anything unless `--archive` is passed. Output: `1-fragments-draft.md`. ### 6.3 Audit Missing Fragments Review the "Audit: Code Changes Without a Fragment" section of the draft. For every missing line, judge whether users can notice the change (the same principle as `release_notes/README.md`): - User-facing and worth a release note: check it with `gh pr view <number> --repo vllm-project/vllm-ascend`, add a fragment `release_notes/fragments/<number>.yaml`, and re-run 6.2 so the audit becomes clean and the entry is archived with the rest - Not user-facing (tests, CI, internal refactor, small docs): ignore it - Unsure and the PR is important: add a fragment, so the context is not lost Also confirm the "Fragment PRs Not Found in the Commit List" entries. Such a PR may be unmerged, reverte - .agents/skills/vllm-ascend-release/templates/feedback-issue-template.md (1610 bytes)
## vLLM Ascend ${VERSION} Release Feedback We are preparing to release **vLLM Ascend ${VERSION}**. This issue is for collecting community feedback before the official release. ### What's New <!-- Release highlights will be added here --> ### How to Test You can test the release candidate by: 1. **Using Docker (Recommended)**: ```bash docker pull quay.io/ascend/vllm-ascend:${VERSION} ``` 2. **Using pip**: ```bash pip install vllm-ascend==${VERSION} ``` 3. **Building from source**: ```bash git clone https://github.com/vllm-project/vllm-ascend.git cd vllm-ascend git checkout ${VERSION} pip install -e . ``` ### Feedback Guidelines Please share your feedback in the comments below: - **Bug Reports**: Describe the issue, steps to reproduce, and your environment - **Performance Issues**: Include benchmark results and comparison with previous versions - **Feature Requests**: Describe the use case and expected behavior - **Documentation Issues**: Point out unclear or missing documentation ### Environment Information When reporting issues, please include: ``` - vLLM Ascend version: - vLLM version: - CANN version: - torch_npu version: - Hardware (910B/310P/etc.): - OS: ``` ### Timeline - **RC Release Date**: ${DATE} - **Feedback Period**: 1 week - **Stable Release**: TBD based on feedback ### Related Links - [Release Checklist](${CHECKLIST_ISSUE_URL}) - [Documentation](https://docs.vllm.ai/projects/ascend/en/latest) - [Previous Releases](https://github.com/vllm-project/vllm-ascend/releases) --- Thank you for helping us improve vLLM Ascend! - .agents/skills/vllm-ascend-release/templates/release-checklist-template.md (2880 bytes)
### Release Checklist **Release Version**: ${VERSION} **Release Branch**: ${BRANCH} **Release Date**: ${DATE} **Release Manager**: @${MANAGER} ### Prepare Release Note - [ ] Create a new issue for release feedback ${FEEDBACK_ISSUE_URL} - [ ] Upgrade vllm version to the new version for CI and Dockerfile - [ ] Write the release note PR. ${RELEASE_NOTE_PR_URL} - [ ] Aggregate release note fragments and audit commits without fragments (release_notes/fragments) - [ ] Update the feedback issue link in docs/source/faqs.md - [ ] Add release note to docs/source/user_guide/release_notes.md - [ ] Archive the consumed fragments to release_notes/archive/<version>/ in the release PR - [ ] Update release version in README.md and README.zh.md (Getting Started and Branch section) - [ ] Update version info in docs/source/community/versioning_policy.md(Release compatibility matrix, Release window and Branch states section) - [ ] Update contributor info in docs/source/community/contributors.md - [ ] Update package version in mkdocs.yml (extra: block) ### Bug need Solve <!-- AI-GENERATED: This section will be populated by the release skill --> <!-- The skill scans open bugs and identifies release-blocking issues --> ${BUG_LIST} ### PR need Merge <!-- AI-GENERATED: This section will be populated by the release skill --> <!-- The skill identifies PRs that should be merged before release --> ${PR_LIST} ### Functional Test <!-- AI-GENERATED: This section will be populated by the release skill --> <!-- Lists features/models that need manual testing (not covered by CI) --> <!-- Also tracks unresolved items from previous release feedback --> #### Manual Testing Required ${MANUAL_TEST_ITEMS} #### Previous Feedback Status ${FEEDBACK_STATUS} ### Nightly Status <!-- AI-GENERATED: This section will be populated by the release skill --> <!-- Analyzes latest Nightly-A3 and Nightly-A2 CI runs --> <!-- Uses extract_and_analyze.py to extract and categorize failures --> | Workflow | Status | Failed Jobs | Code Bugs | Env Flakes | Run | |----------|--------|-------------|-----------|------------|-----| ${NIGHTLY_TABLE} #### Code Bugs Need Fix ${NIGHTLY_CODE_BUGS} ### Doc Test - [ ] Tutorial is updated. - [ ] User Guide is updated. - [ ] Developer Guide is updated. ### Prepare Artifacts - [ ] Docker image is ready. - [ ] Wheel package is ready. ### Release Step - [ ] Release note PR is merged. - [ ] Post the release on GitHub release page. - [ ] Generate official doc page on <https://app.readthedocs.org/dashboard/> - [ ] Wait for the wheel package to be available on <https://pypi.org/project/vllm-ascend> - [ ] Wait for the docker image to be available on <https://quay.io/ascend/vllm-ascend> - [ ] Upload 310p wheel to Github release page - [ ] Broadcast the release news (By message, blog , etc) - [ ] Close this issue - .claude/README.md (130 bytes)
If you want to use the skills in this repo with Claude code, please copy the skills directory `.agents/skills` to this directory.
- .gemini/config.yaml (479 bytes)
# https://developers.google.com/gemini-code-assist/docs/customize-gemini-behavior-github have_fun: false memory_config: disabled: false code_review: comment_severity_threshold: HIGH # Reduce quantity of comments max_review_comments: 10 # Avoid comment flooding on large PRs pull_request_opened: summary: false # PR descriptions are agent-generated and follow the template already ignore_patterns: - "tests/e2e/**" # Requires NPU hardware; CI validates - .gemini/styleguide.md (3040 bytes)
# Review Guidance for vllm-ascend vLLM Ascend is a hardware plugin integrating Ascend NPUs into vLLM. Formatting, PR title tags, and PR description templates are enforced by CI (`format.sh ci`, check gates) and by the coding agents developers already use — do NOT comment on them. Focus your review on issues that automated tooling cannot catch: ## NPU-specific correctness and performance - `tensor.item()`, `.cpu()`, `.tolist()`, or data-dependent control flow on device tensors in hot paths: each forces a host-device synchronization and can stall AsyncScheduler. Suggest device-side alternatives (`torch.argmax`, `torch.sum`, keeping values on device, batching). - Unnecessary CPU↔NPU transfers or avoidable copies in hot paths. - Long-running-process memory concerns: fragmentation, tensors that should be freed, per-step allocations that could be reused. ## Architecture rules (see AGENTS.md at repo root) - New environment variables need strong justification; user-facing config belongs in `--additional-config` via a typed sub-config in `vllm_ascend/ascend_config.py` (follow `EplbConfig`/`RlConfig` patterns). Flag scattered/hardcoded env var access instead of `vllm_ascend.envs`. - New patches under `vllm_ascend/patch/`: check the patch is minimal, targets the right layer (`platform/` vs `worker/`), and is a candidate for upstream contribution to vLLM. - Model runner changes (`vllm_ascend/worker/v2/model_runner.py`): check existing upstream extension points (inheritance, composition) were considered first, and flag device-side hot-path costs. - No new module-level mutable globals; constants (`ALL_UPPER_CASE`) are fine. - All imports at top of file, except circular imports, lazy worker-process loading, and `if TYPE_CHECKING:` blocks. ## Triton operators - JIT specialization / recompilation risks from changing arguments. - Warmup: required variants registered in the component's active registry window; runtime calls reuse the same wrapper as warmup. - Device-applicability across Ascend 910B / 910C / 310P. ## Docs consistency - Behavior changes (config keys, env vars, defaults) should be reflected in the relevant docs (`docs/source/user_guide/configuration/`, additional_config table). Flag docs that contradict the code in the same PR. ## Release notes - Every user-facing PR must add `release_notes/fragments/<PR number>.yaml` following `release_notes/TEMPLATE.yaml`. - Flag fragments that: describe internal refactors users cannot observe, state performance numbers without a scenario (model/workload/hardware) or an invented magnitude, or change config/env behavior without old→new plus the migration step. - Bug-fix fragments should describe the user-visible symptom, not the internal root cause. ## Testing - Changes to kernels, model runners, or config parsing should come with or update CPU-runnable unit tests under `tests/ut`. - Do not ask authors for NPU/E2E/performance runs when they lack hardware — that evidence comes from CI; just say so.
- AGENTS.md (16526 bytes)
# vLLM Ascend Development Guidelines This document provides instructions for contributors to the vLLM Ascend project. Please read and follow these guidelines to ensure code quality, maintainability, and consistency. Read linked documentation when it applies to the current task, using the relevant sections. Continue authorized local implementation and validation while preparing the evidence needed for PR review. --- ## Table of Contents - [Setup and Environment](#setup-and-environment) - [No New Environment Variables](#no-new-environment-variables) - [Configuration Review Requirement](#configuration-review-requirement) - [Testing](#testing) - [Code Style](#code-style) - [Python Conventions](#python-conventions) - [Naming Conventions](#naming-conventions) - [Triton Operators](#triton-operators) - [NPU-Specific Considerations](#npu-specific-considerations) - [Tensor item() Operations](#tensor-item-operations) - [Memory and Performance](#memory-and-performance) - [Model and Plugin Architecture](#model-and-plugin-architecture) - [vLLM Ascend Plugin Architecture](#vllm-ascend-plugin-architecture) - [Patching Requirement](#patching-requirement) - [Model Runner Changes](#model-runner-changes) - [Release Notes](#release-notes) - [Commit Messages and Pull Requests](#commit-messages-and-pull-requests) - [Commit Message Format](#commit-message-format) - [Quick Start for Contributors](#quick-start-for-contributors) - [References](#references) --- ## Setup and Environment ### No New Environment Variables Do not add new environment variables. User-facing configuration belongs in vLLM's additional config mechanism: a typed key under `--additional-config` (offline: `LLM(additional_config={...})`), following the existing sub-config patterns in [vllm_ascend/ascend_config.py](vllm_ascend/ascend_config.py) and documented in [Additional Configuration](docs/source/user_guide/configuration/additional_config.md). The project has already migrated most configuration from environment variables to additional config (see the [migration guide](docs/source/user_guide/configuration/additional_config.md#migration-guide)); do not reverse it. An environment variable is a last resort, only when additional config cannot cover the case — for example, a value needed before vLLM config exists (process bootstrap, subprocess launch). A strong reason is required; "it is simpler" is not one. When an environment variable is truly unavoidable, define it in the centralized `env_variables` dictionary in [vllm_ascend/envs.py](vllm_ascend/envs.py). **Requirements:** - Add documentation for each environment variable in the `env_variables` dict comment - Specify default values and valid ranges - Indicate whether the variable is sensitive (credentials, keys) **Never**: Hardcode environment variable names throughout the codebase. Reference them from the central module using `from vllm_ascend import envs`. ### Configuration Review Requirement Both configuration channels are strictly managed. The PR must justify the channel it chooses. For a **new additional config key** (the recommended channel), the PR must show: - Why existing configuration cannot cover the use case - Typed parsing and validation in `vllm_ascend/ascend_config.py`, following the existing sub-config pattern (e.g. `EplbConfig`, `RlConfig`), not raw dict access scattered across the code - A table entry in [Additional Configuration](docs/source/user_guide/configuration/additional_config.md) with type, default, and behavior - A name that generalizes beyond the model or workload that motivated it For a **new environment variable** (exceptional), the PR must show a strong reason on top: - Why additional config cannot cover the use case - The variable name follows the `VLLM_ASCEND_*` naming convention - The default value works for all supported hardware - Documentation is added to the `env_variables` dict and reflected in the user-facing [environment variables](docs/source/user_guide/configuration/env_vars.md) page --- ## Testing Run tests appropriate to the change and complete required checks. Broaden or repeat testing when new changes, failures, or unresolved risks justify it. When preparing or reviewing a PR, read the [PR testing requirements](docs/source/developer_guide/contribution/testing.md#pr-testing-requirements) for coverage, hardware evidence, and validation reporting. Use the other sections of the same guide for environment setup and test commands as needed. **What you can verify without NPU hardware:** - Unit tests run on CPU: `TORCH_DEVICE_BACKEND_AUTOLOAD=0 pytest -sv tests/ut` (commands and mocking conventions in the [Unit tests](docs/source/developer_guide/contribution/testing.md#unit-tests) section) - Format and lint checks: `bash format.sh ci` - Documentation build and link checks, when docs changed E2E, multi-card, and performance tests require real NPU hardware. Do not attempt them in a CPU-only environment; delegate that coverage to CI and say so in the PR. --- ## Code Style ### Python Conventions - **Imports**: All imports at the top of the file. Valid exceptions: - Circular imports (use inline imports) - Lazy loading for worker/isolation processes - Type-checking imports wrapped in `if TYPE_CHECKING:` - **Global Variables**: Avoid new global variables. Pass dependencies explicitly through function parameters. **Allowed:** - Constants named `ALL_UPPER_CASE` - Immutable configuration objects If mutable module-level state is necessary, explain its lifetime and concurrency implications in the PR. - **No Magic Numbers**: Use named constants with descriptive names. - **Descriptive Naming**: Use names that describe functionality, not implementation details. ### Naming Conventions - **Classes**: `PascalCase` (e.g., `NPUModelRunner`, `ACLGraphManager`) - **Functions/Methods**: `snake_case` (e.g., `forward_pass`, `compute_attention`) - **Constants**: `ALL_UPPER_CASE` - **Variables**: `snake_case` (e.g., `token_ids`, `sequence_lengths`) --- ## Triton Operators When adding, modifying, or reviewing Triton operators, follow the [Triton operator contribution guidelines](docs/source/developer_guide/contribution/triton_operators.md) for deliverables, functionality, accuracy, performance, and code quality. Contributors and reviewers must check: - The documented interface, production integration, and compatibility across applicable devices. - Operator and model accuracy and performance evidence, with test coverage appropriate to framework-wide or model-specific use. - Public wrapper usage, documented side effects, and JIT specialization and recompilation risks. - When warmup is used, required variants, registration in the selected component's active registry window, and reuse of the same wrapper for runtime calls. For warmup integration or changes, read [Triton JIT warmup on Ascend](docs/source/developer_guide/Design_Documents/triton_jit_warmup.md) for the startup flow, registration locations, and validation steps. --- ## NPU-Specific Considerations ### Tensor item() Operations **Warning**: `tensor.item()` operations cause synchronization overhead on NPU when the `tensor` is on device. If the `tensor` is a device tensor, calling `item()` triggers a synchronous data transfer from NPU to CPU. This can severely degrade performance in hot paths, causing `AsyncScheduler` to block here. **Review Requirements:** 1. Profile synchronization introduced in hot paths before merging 2. Consider alternative patterns: - Keep values on device when possible - Batch operations to reduce sync frequency - Use device-side operations (e.g., `torch.argmax`, `torch.sum`) 3. Document when `item()` is unavoidable (e.g., logging, conditional logic) ### Memory and Performance Additional NPU-specific best practices: - Avoid CPU-NPU memory transfers in hot paths - Prefer in-place operations where safe (e.g., `x.add_()`, `x.mul_()`) - Monitor memory fragmentation, especially for long-running processes - Test with realistic workloads on actual NPU hardware (e.g. Ascend 910B, 910C, 310P) --- ## Model and Plugin Architecture ### vLLM Ascend Plugin Architecture vLLM Ascend is a **hardware plugin** that integrates with upstream vLLM via the pluggable hardware interface. Prefer existing upstream implementations and extension points. Ascend-specific model adaptations are also registered in the [model registry](vllm_ascend/models/__init__.py); follow existing registration and inheritance patterns when working on these models. Choose the integration approach that fits the change: 1. **Patching** (in `vllm_ascend/patch/`): - `vllm_ascend/patch/platform/` - Platform-level patches (distributed, scheduling) - `vllm_ascend/patch/worker/` - Worker-level patches (model-specific behavior) - Patch is not the best solution for all cases. Use it when necessary. 2. **Inheritance**: - `NPUModelRunner(GPUModelRunner)` - Extend vLLM model runner with NPU-specific behavior - Add NPU-specific components via composition (e.g., `AclGraphManager`) - Custom Operators - NPU-specific custom operators (e.g., `AscendRMSNorm`) 3. **External upstream contributions** where appropriate When adding or adapting a model, follow the step-by-step walkthrough in the [vllm-ascend-model-adapter skill](.agents/skills/vllm-ascend-model-adapter/SKILL.md). ### Patching Requirement Explain new patches in the PR for architectural review. When adding or changing a patch, read the [patch strategy](docs/source/developer_guide/Design_Documents/patch.md) for lifecycle, registration, and removal-plan requirements. Your PR must explain: - Which upstream component the patch targets and why - How the patch stays minimal and focused - The performance implications - The long-term plan for upstream contribution ### Model Runner Changes Explain new `model_runner` behavior in the PR for architectural review. Your PR must explain: - Why existing extension points cannot cover the new behavior - The performance implications on NPU hardware - Compatibility with existing model implementations - Long-term maintainability and test coverage **NPU Model Runner Files:** - `vllm_ascend/worker/v2/model_runner.py` - vLLM v2 model runner --- ## Release Notes Every PR with a user-facing change must ship a release note fragment in the same PR. The fragment is written for users of the released wheel or Docker image, not for reviewers: a reader who never sees the diff should be able to tell what changed for them and whether they need to do anything. **Write a fragment for**: new models, hardware, or operators; new features; changes to user-facing APIs, parameters, environment variables, or configuration; breaking changes, deprecations, and default behavior changes; dependency and toolchain updates (CANN, torch_npu, triton-ascend, vLLM alignment); performance improvements; and bug fixes users could have hit (hangs, OOM, wrong outputs, broken metrics). **Do not write a fragment for**: internal refactors, code moves, logging and typing cleanups; tests, CI, build scripts, development tooling; documentation typo fixes and small edits (guides for brand-new scenarios are the exception, category `documentation`); and plumbing for future features that is still invisible to users. If the only honest sentence is "refactored X" or "prepared Y for Z", users cannot observe the change — no fragment. **How to write the text** (full rules and examples in [release_notes/README.md](release_notes/README.md#writing-the-text)): - Say what the user gets, not what the code does. Name user-facing switches exactly (env vars, config keys, CLI flags); skip file, class, and kernel-level narration. - Performance entries: state the scenario (model, workload, hardware) and the effect (measured metric and magnitude, or the concrete user-visible mechanism). Never invent numbers. - Breaking and config entries: old behavior → new behavior, the exact switch, the migration step, and what happens if the user does nothing. - Bug fixes: the user-visible symptom and when it occurs, not the internal root cause. - The unit of a release note is the net change relative to the previous release. A fix for a bug **already shipped in a released version** needs a `bugfix` fragment; a fix for a bug that never reached a released artifact needs no fragment of its own. A revert of a PR merged after the last release needs no fragment either. In both unreleased cases, remove the cancelled PR's not-yet-archived fragment in the fix/revert PR. See [release_notes/README.md](release_notes/README.md#changes-that-cancel-out-before-a-release). Add `release_notes/fragments/<PR number>.yaml` following `release_notes/TEMPLATE.yaml` — open the PR first to get its number, then push the fragment to the same PR. When in doubt about whether users can observe the change, add a fragment — the release manager drops extra entries, but a missing one is lost. Reviewers confirm the inclusion decision and the wording together with the code; there is no CI gate. For release management workflows (release checklist issues, critical bug triage, fragment aggregation), use the [vllm-ascend-release skill](.agents/skills/vllm-ascend-release/SKILL.md). --- ## Commit Messages and Pull Requests ### Commit Message Format Follow the [Conventional Commits](https://www.conventionalcommits.org/) format and **must include a sign-off**: ```bash git commit -s -m "<type>: <summary>" -m "<body - explaining what changed and why>" ``` **Valid Types**: `feat`, `fix`, `perf`, `refactor`, `test`, `docs`, `chore` ### Pull Request Title Format PR titles use bracketed change-type tags such as `[BugFix]`, `[Feature]`, or `[Doc]` (note: this differs from the `type:` prefix used in commit messages). The full accepted set is enforced by the CI check linked in the [PR submission checklist](docs/source/developer_guide/contribution/testing.md#pr-submission-checklist). ### Pull Request Template When creating a PR, please follow the template in `.github/PULL_REQUEST_TEMPLATE.md` and ensure the following sections are completed: > **Note**: The PR description will be automatically updated by GitHub Actions to include vLLM version info at the bottom. If you update the PR description via API or CLI, make sure to preserve the `- vLLM version:` and `- vLLM main:` lines. - **What this PR does / why we need it?** - Clearly describe the changes and their purpose - **Does this PR introduce _any_ user-facing change?** - Indicate if there are any user-visible changes - **How was this patch tested?** - Describe how you tested the changes. Examples: - Unit tests added/updated: list the test files - Manual testing: provide the test steps and commands - CI testing: indicate if only CI verification is needed **Only report test runs you actually performed.** Without NPU hardware, write "CI testing only" and let CI prod - aistorehub.json (439 bytes)
{ "name": "Vllm Ascend", "slug": "vllm-ascend", "version": "1.0.0", "type": "skill", "description": "Community maintained hardware plugin for vLLM on Huawei Ascend", "owner": "community", "repository": "catalog", "github_repo_url": "https://github.com/vllm-project/vllm-ascend", "tags": [ "trending", "vllm-project", "python", "github", "open-source" ], "imported_at": "2026-11-21T23:56:34.744Z" } - benchmarks/benchmark_kv_pool_metadata.py (13530 bytes)
# SPDX-License-Identifier: Apache-2.0 """Compare actual KV-pool CPU preparation methods with a mock store. Run in an installed vLLM/Ascend environment; no model, store service, or NPU operation is created. Both source trees must use the same dependency stack. This measures CPU metadata preparation, not a full scheduler tick or E2E serving. Example: python benchmarks/benchmark_kv_pool_metadata.py \ --baseline-root /tmp/kv-pool-baseline \ --candidate-root /tmp/kv-pool-candidate --output /tmp/kv-pool-perf.json """ from __future__ import annotations import argparse import hashlib import importlib.util import inspect import json import logging import platform import statistics import subprocess import sys import time from pathlib import Path from types import SimpleNamespace from typing import Any SCHEDULER_PATH = "vllm_ascend/distributed/kv_transfer/kv_pool/ascend_store/pool_scheduler.py" WORKER_PATH = "vllm_ascend/distributed/kv_transfer/kv_pool/ascend_store/layerwise/pool_worker.py" def load_source(root: Path, relative_path: str, label: str) -> Any: path = root / relative_path name = f"kv_pool_benchmark_{label}_{path.stem}" spec = importlib.util.spec_from_file_location(name, path) if spec is None or spec.loader is None: raise RuntimeError(f"Cannot load {path}") module = importlib.util.module_from_spec(spec) sys.modules[name] = module spec.loader.exec_module(module) return module class MockInfo: def __init__(self, address: int) -> None: self.address = address def size(self) -> int: return 128 def gva_list(self) -> list[int]: return [self.address] class MockStore: """Immediate deterministic metadata/lease replies; counts real method calls.""" def __init__(self, infos: dict[str, MockInfo]) -> None: self.infos = infos self.calls: dict[str, int] = {} def batch_get_key_info(self, keys: list[str], **kwargs: Any) -> list[MockInfo]: self.calls["metadata_calls"] = self.calls.get("metadata_calls", 0) + 1 self.calls["metadata_keys"] = self.calls.get("metadata_keys", 0) + len(keys) return [self.infos[key] for key in keys] def batch_add_lease(self, keys: list[str], ttl: int) -> list[int]: self.calls["lease_calls"] = self.calls.get("lease_calls", 0) + 1 self.calls["lease_keys"] = self.calls.get("lease_keys", 0) + len(keys) return [0] * len(keys) class SchedulerCase: def __init__(self, module: Any, count: int, all_cached: bool, *, unfinished_ids: list[str] | None = None) -> None: self.ids = [f"request-{index}" for index in range(count)] cached_ids = self.ids if all_cached else self.ids[::2] unfinished_ids = self.ids if unfinished_ids is None else unfinished_ids cached_id_set = set(cached_ids) self.expected_pending = [req_id for req_id in unfinished_ids if req_id not in cached_id_set] self.calls: list[str] = [] self.scheduler = module.KVPoolScheduler.__new__(module.KVPoolScheduler) for name, value in { "kv_role": "kv_both", "consumer_is_to_put": False, "can_put": False, "_request_trackers": {}, "_unfinished_requests": {req_id: (None, []) for req_id in unfinished_ids}, "_preempted_req_ids": set(), "_loading_req_ids": set(), }.items(): setattr(self.scheduler, name, value) self.scheduler._process_async_load_request = self.process_pending self.output = SimpleNamespace( finished_req_ids=set(), preempted_req_ids=set(), scheduled_new_reqs=[], scheduled_cached_reqs=SimpleNamespace(req_ids=cached_ids), ) def process_pending(self, req_id: str, *_: Any) -> None: self.calls.append(req_id) def reset(self) -> None: self.calls.clear() def run(self) -> Any: return self.scheduler.build_connector_meta(self.output) def verify(self, result: Any) -> dict[str, Any]: assert result.requests == [] assert self.calls == self.expected_pending return { "pending_callbacks": len(self.calls), "order_sha256": hashlib.sha256("\n".join(self.calls).encode()).hexdigest(), } class LeaseCase: def __init__(self, module: Any, count: int) -> None: import numpy as np from vllm_ascend.distributed.kv_transfer.kv_pool.ascend_store.metadata import ( LoadSpec, ReqMeta, block_hash_to_str, ) block_hashes = [index.to_bytes(8, "big") for index in range(count)] self.keys = [self.make_key(0, block_hash_to_str(block_hash)) for block_hash in block_hashes] self.addresses = np.arange(1_000_000, 1_000_000 + count * 128, 128, dtype=np.int64) self.store = MockStore( {key: MockInfo(int(address)) for key, address in zip(self.keys, self.addresses, strict=True)} ) self.worker = module.LayerwiseKVPoolWorker.__new__(module.LayerwiseKVPoolWorker) for name, value in { "use_layerwise_transfer": True, "use_eagle": False, "num_kv_cache_groups": 1, "cacheable_group_ids": [0], "grouped_block_size": [128], "hash_block_size": 128, "layerwise_offload": True, "_step_load_keys": set(), "m_store": self.store, "_compute_reachable_load_masks": lambda *_: None, "_layerwise_load_snapshot_start": lambda *_: None, "_resolve_layerwise_load_partial": lambda *_: (None, None), "_make_layerwise_full_key": self.make_key, }.items(): setattr(self.worker, name, value) ids = np.arange(count, dtype=np.int64) self.request = ReqMeta( req_id="benchmark-load", token_len_chunk=count * 128, target_token_len=count * 128, block_ids=ids.tolist(), block_ids_by_group=[ids.tolist()], block_ids_np=ids, block_ids_by_group_np=[ids], block_hashes=block_hashes, load_spec=LoadSpec(0, count * 128, True), ) @staticmethod def make_key(group: int, block_hash: str) -> str: return f"benchmark-model@group:{group}@head_or_tp_rank:0@{block_hash}" def reset(self) -> None: self.store.calls.clear() self.worker._step_load_keys.clear() def run(self) -> Any: return self.worker._prepare_load_gvas_for_request([self.request]) def verify(self, result: Any) -> dict[str, Any]: import numpy as np assert result is None assert self.request.load_keys == self.keys np.testing.assert_array_equal(self.request.load_block_gvas_by_group_np[0], self.addresses) assert self.worker._step_load_keys == set(self.keys) assert self.store.calls == { "metadata_calls": 2, "metadata_keys": 2 * len(self.keys), "lease_calls": 1, "lease_keys": len(self.keys), } return {**self.store.calls, "gva_sha256": hashlib.sha256(self.addresses.tobytes()).hexdigest()} def summarize(samples: list[float]) -> dict[str, Any]: ordered = sorted(samples) position = (len(ordered) - 1) * 0.95 low = int(position) high = min(low + 1, len(ordered) - 1) return { "samples": len(samples), "median_ms": statistics.median(samples), "p95_ms": ordered[low] + (ordered[high] - ordered[low]) * (position - low), "raw_ms": samples, } def measure(cases: dict[str, Any], warmup: int, rounds: int, iterations: int) -> dict[str, Any]: fingerprints = {} samples: dict[str, list[float]] = {label: [] for label in cases} for label, case in cases.items(): for _ in range(warmup): case.reset() fingerprints[label] = case.verify(case.run()) assert fingerprints["baseline"] == fingerprints["candidate"] for _ in range(rounds): for label in ("baseline", "candidate", "candidate", "baseline"): case = cases[label] for _ in range(iterations): case.reset() start = time.perf_counter_ns() result = case.run() elapsed_ms = (time.perf_counter_ns() - start) / 1_000_000 assert case.verify(result) == fingerprints[label] samples[label].append(elapsed_ms) return { **{label: summarize(values) for label, values in samples.items()}, "verified_operations": fingerprints["baseline"], } def git_revision(root: Path) -> str: result = subprocess.run(["git", "-C", str(root), "rev-parse", "HEAD"], capture_output=True, text=True, check=False) return result.stdout.strip() if result.returncode == 0 else "unavailable" def main() -> None: parser = argparse.ArgumentParser(description=__doc__) parser.add_argument("--baseline-root", type=Path, required=True) parser.add_argument("--candidate-root", type=Path, required=True) parser.add_argument("--output", type=Path, required=True) parser.add_argument("--warmup", type=int, default=3) parser.add_argument("--rounds", type=int, default=5) parser.add_argument("--iterations", type=int, default=3) args = parser.parse_args() if min(args.warmup, args.rounds, args.iterations) < 1: parser.error("warmup, rounds and iterations must be positive") roots = {"baseline": args.baseline_root.resolve(), "candidate": args.candidate_root.resolve()} sys.path.insert(0, str(roots["candidate"])) logging.getLogger("vllm").setLevel(logging.WARNING) modules = { label: (load_source(root, SCHEDULER_PATH, label), load_source(root, WORKER_PATH, label)) for label, root in roots.items() } report: dict[str, Any] = { "scope": "Native source modules and real NumPy, immediate mock store; CPU preparation only, no NPU/E2E claim", "python": platform.python_version(), "platform": platform.platform(), "order": "ABBA: baseline,candidate,candidate,baseline per round", "warmup_per_case_variant": args.warmup, "rounds": args.rounds, "iterations_per_ABBA_block": args.iterations, "sources": {}, "cases": [], } for label, (scheduler_module, worker_module) in modules.items(): report["sources"][label] = { "root": str(roots[label]), "git_head": git_revision(roots[label]), "files": { relative: hashlib.sha256((roots[label] / relative).read_bytes()).hexdigest() for relative in (SCHEDULER_PATH, WORKER_PATH) }, "loaded_scheduler": inspect.getsourcefile(scheduler_module.KVPoolScheduler), "loaded_worker": inspect.getsourcefile(worker_module.LayerwiseKVPoolWorker), } cases: dict[str, Any] for count in (128, 512, 1024, 4096): for all_cached in (True, False): cases = {label: SchedulerCase(pair[0], count, all_cached) for label, pair in modules.items()} report["cases"].append( { "phase": "build_connector_meta_filter", "active_requests": count, "all_cached": all_cached, **measure(cases, args.warmup, args.rounds, args.iterations), } ) # Decoding can schedule many cached requests with no or very few unfinished # connector requests. Include early/late matches as well as pending misses. for count in (128, 4096): for unfinished_count in (0, 1, 4): placements = ("empty",) if unfinished_count == 0 else ("head", "tail", "pending") for placement in placements: if placement == "head": unfinished_ids = [f"request-{index}" for index in range(unfinished_count)] elif placement == "tail": unfinished_ids = [f"request-{index}" for index in range(count - unfinished_count, count)] else: unfinished_ids = [f"pending-{index}" for index in range(unfinished_count)] cases = { label: SchedulerCase(pair[0], count, True, unfinished_ids=unfinished_ids) for label, pair in modules.items() } report["cases"].append( { "phase": "build_connector_meta_sparse", "scheduled_cached_requests": count, "unfinished_requests": unfinished_count, "unfinished_placement": placement, **measure(cases, args.warmup, args.rounds, args.iterations), } ) for count in (128, 512, 2048, 8192): cases = {label: LeaseCase(pair[1], count) for label, pair in modules.items()} report["cases"].append( { "phase": "prepare_layerwise_load_gvas", "blocks": count, **measure(cases, args.warmup, args.rounds, args.iterations), } ) args.output.parent.mkdir(parents=True, exist_ok=True) args.output.write_text(json.dumps(report, indent=2) + "\n", encoding="utf-8") print(json.dumps({"output": str(args.output), "cases": len(report["cases"])})) if __name__ == "__main__": main() - benchmarks/benchmark_ple_lookup.py (8872 bytes)
# SPDX-License-Identifier: Apache-2.0 # SPDX-FileCopyrightText: Copyright contributors to the vLLM project """Compare compact PLE host lookup with the same rows expanded to BF16. Run in a fresh process and TRITON_CACHE_DIR for cold-start compiler counts. This synthetic single-rank benchmark does not measure model serving. """ import argparse import json import statistics from collections import Counter from importlib.metadata import version import torch import torch_npu # noqa: F401 import vllm_ascend.ops # noqa: F401 # Register ops before importing DeviceOperator through Engram. from vllm_ascend.models.deepseek_v41.engram.npu import CHUNK_ROWS, HostUvaBuffer from vllm_ascend.models.qwen4_exp.ple_quant import AscendPLEQuantizedEmbeddingMethod from vllm_ascend.ops.triton.qwen4_exp.ple_lookup import _lookup, lookup from vllm_ascend.ops.triton.qwen4_exp.ple_lookup_quantized import _lookup_quantized, lookup_quantized def graph_latency(run, samples, repeats): graph = torch.npu.NPUGraph() try: with torch.npu.graph(graph): # Put repetitions inside the graph so Python replay submission # gaps are not divided into the reported operator latency. for _ in range(repeats): run() for _ in range(5): graph.replay() timings = [] for _ in range(samples): begin = torch.npu.Event(enable_timing=True) end = torch.npu.Event(enable_timing=True) begin.record() graph.replay() end.record() end.synchronize() timings.append(begin.elapsed_time(end) * 1000 / repeats) return statistics.median(timings), timings finally: torch.npu.synchronize() graph.reset() def main(args): torch.npu.set_device(args.device) torch.manual_seed(0) device = torch.device("npu", args.device) method = AscendPLEQuantizedEmbeddingMethod(mxfp8=args.format == "mxfp8") buffers: list[HostUvaBuffer] = [] owner = torch.nn.Module() compile_entries: Counter[str] = Counter() original_compilers = [(kernel, kernel._do_compile) for kernel in (_lookup, _lookup_quantized)] # Count the actual JIT dispatch entry, including persistent-cache lookups. # Only a fresh cache can make this a cold compilation count. for kernel, original_compile in original_compilers: def counted_compile(*positional, _compile=original_compile, _name=kernel.__name__, **keywords): compile_entries[_name] += 1 return _compile(*positional, **keywords) kernel._do_compile = counted_compile try: for columns, dtype in ( (args.width, method.weight_dtype), (args.width // 32, method.scale_dtype), (args.width, torch.bfloat16), ): buffers.append(HostUvaBuffer((args.rows, columns), dtype, device, f"table_{len(buffers)}", owner)) codes, scales, bf16 = buffers # Production tables contain tens of millions of rows. Bound temporary # CPU allocations while keeping all three registered tables resident. for begin in range(0, args.rows, 65536): end = min(begin + 65536, args.rows) count = end - begin values = torch.randint(-7, 8, (count, args.width)).to(method.weight_dtype) decoded = torch.linspace(0.25, 2, args.width // 32).expand(count, -1) if method.mxfp8: exponents = torch.arange(args.width // 32).remainder(4).add(125).to(torch.uint8) scales.tensor[begin:end].copy_(exponents.expand(count, -1)) decoded = torch.exp2(exponents.float() - 127).expand(count, -1) else: scales.tensor[begin:end].copy_(decoded) codes.tensor[begin:end].copy_(values) bf16.tensor[begin:end].copy_( (values.float().unflatten(-1, (-1, 32)) * decoded.unsqueeze(-1)).flatten(-2).bfloat16() ) def runners(ids, compact_output, baseline_output): def compact(): lookup_quantized( codes.ptrs, scales.ptrs, ids, compact_output, args.shard_start, args.shard_start + args.rows, CHUNK_ROWS, is_mxfp8=method.mxfp8, ) def baseline(): lookup(bf16.ptrs, ids, baseline_output, args.shard_start, args.shard_start + args.rows, CHUNK_ROWS) return compact, baseline ids = torch.full((1,), args.shard_start, dtype=torch.int64, device=device) output = torch.empty((1, args.width), dtype=torch.bfloat16, device=device) compact, baseline = runners(ids, output, torch.empty_like(output)) compact() baseline() torch.npu.synchronize() startup_entries = dict(compile_entries) print( json.dumps( dict( environment={name: version(name) for name in ("torch", "torch-npu", "triton-ascend")}, device=torch.npu.get_device_name(args.device), format=args.format, rows=args.rows, width=args.width, id_dtype="int64", start=args.shard_start, end=args.shard_start + args.rows, ownership_shards=args.ownership_shards, chunk=CHUNK_ROWS, block=1 << (args.width - 1).bit_length(), samples=args.samples, repeats=args.repeats, compact_storage_bytes=sum(b.tensor.numel() * b.tensor.element_size() for b in (codes, scales)), bf16_storage_bytes=bf16.tensor.numel() * bf16.tensor.element_size(), startup_jit_compile_entries=startup_entries, ) ), flush=True, ) for tokens in args.tokens: ids = torch.randint(args.rows * args.ownership_shards, (tokens,), dtype=torch.int64, device=device) ids += args.shard_start output = torch.empty((tokens, args.width), dtype=torch.bfloat16, device=device) baseline_output = torch.empty_like(output) compact, baseline = runners(ids, output, baseline_output) compact() baseline() torch.testing.assert_close(output, baseline_output, rtol=0, atol=0) owned_rows = int(((ids >= args.shard_start) & (ids < args.shard_start + args.rows)).sum().cpu()) compact_us, compact_samples = graph_latency(compact, args.samples, args.repeats) bf16_us, bf16_samples = graph_latency(baseline, args.samples, args.repeats) print( json.dumps( dict( tokens=tokens, owned_rows=owned_rows, compact_us=compact_us, bf16_us=bf16_us, compact_over_bf16=compact_us / bf16_us, compact_samples_us=compact_samples, bf16_samples_us=bf16_samples, ) ), flush=True, ) serving_entries = {key: count - startup_entries.get(key, 0) for key, count in compile_entries.items()} print(json.dumps(dict(synthetic_runtime_additional_jit_compile_entries=serving_entries)), flush=True) finally: for kernel, original_compile in original_compilers: kernel._do_compile = original_compile torch.npu.synchronize() for buffer in reversed(buffers): buffer.close() if __name__ == "__main__": parser = argparse.ArgumentParser(description=__doc__) parser.add_argument("--format", choices=("int8", "mxfp8"), required=True) parser.add_argument("--device", type=int, default=0) parser.add_argument("--rows", type=int, default=262144) parser.add_argument("--width", type=int, default=160) parser.add_argument("--ownership-shards", type=int, default=1, help="Uniform IDs over this many equal shards") parser.add_argument("--shard-start", type=int, default=0) parser.add_argument("--tokens", type=int, nargs="+", default=[1, 32, 256, 4096]) parser.add_argument("--samples", type=int, default=7) parser.add_argument("--repeats", type=int, default=50) args = parser.parse_args() if ( args.width <= 0 or args.width % 32 or args.shard_start < 0 or min(args.rows, args.samples, args.repeats, args.ownership_shards, *args.tokens) <= 0 ): parser.error("width must be a positive multiple of 32; sizes, samples and repeats must be positive") main(args) - benchmarks/benchmark_shared_engram.py (5828 bytes)
# SPDX-License-Identifier: Apache-2.0 """Real shared-pool API and unchanged native lookup, no mapping mocks.""" import argparse import json import statistics import time from functools import partial from pathlib import Path import torch import torch_npu # noqa: F401 import vllm_ascend.ops # noqa: F401 from vllm_ascend.models.deepseek_v41.engram.npu import ( gather_dequantize_engram_int8, gather_dequantize_host_uva, ) ROWS, WIDTH = 16384, 256 def source_tables(layer): codes = (torch.arange(ROWS * WIDTH).remainder(255) - 127).to(torch.int8).view(ROWS, WIDTH) scales = (torch.arange(ROWS * 8).remainder(8).float() + layer).view(ROWS, 8) / 16 return codes, scales def latency(fn): graph = torch.npu.NPUGraph() with torch.npu.graph(graph): output = fn() for _ in range(5): graph.replay() samples = [] for _ in range(8): start, end = torch.npu.Event(enable_timing=True), torch.npu.Event(enable_timing=True) start.record() for _ in range(50): graph.replay() end.record() end.synchronize() samples.append(start.elapsed_time(end) * 1000 / 50) return statistics.median(samples), output.cpu() def main(args): root = Path(args.root) from vllm_ascend.models.deepseek_v41.engram import shared_pool as pool root.mkdir(mode=0o700, parents=True, exist_ok=True) torch.npu.set_device(args.device) endpoint = args.endpoint if args.role == "owner": owner = pool.SharedPoolOwner(endpoint, "probe-model", device=f"npu:{args.device}", world_size=65) try: for layer in (1, 14): for key, values in zip(("codes", "scales"), source_tables(layer)): buffer = owner.add(f"{layer}.{key}", ROWS, values.shape[1], values.dtype) buffer.publish(0, values) owner.ready() print("POOL_READY", flush=True) deadline = time.monotonic() + 1800 while not (root / "stop").exists(): if time.monotonic() > deadline: raise TimeoutError("owner test deadline") time.sleep(0.2) finally: owner.close(timeout=30) print("OWNER_CLOSED_PASS", flush=True) return clients, views = [], {} try: for layer in (1, 14): client = pool.SharedPoolClient(endpoint, "probe-model", device=f"npu:{args.device}") clients.append(client) for key, width, dtype in (("codes", WIDTH, torch.int8), ("scales", 8, torch.float32)): views[layer, key] = client.view(f"{layer}.{key}", 0, ROWS, rows=ROWS, width=width, dtype=dtype) runtime = clients[0].runtime assert runtime is not None and runtime is clients[1].runtime, "layers must share the BM singleton" assert runtime.references == 2 for client in clients: for buffer in client.buffers.values(): assert buffer.handle.local_mem_size(buffer.runtime.bm.BmMemType.HOST) == 0 assert buffer.handle.local_mem_size(buffer.runtime.bm.BmMemType.DEVICE) == 0 result = dict( tag=args.tag, consumer_pool_hbm_bytes=0, consumer_pool_dram_bytes=0, hbm_reference_allocated_separately=True, cases=[], ) for tokens in (32, 128, 512): ids_cpu = torch.randint(0, ROWS, (tokens, 24), generator=torch.Generator().manual_seed(tokens)) ids = ids_cpu.npu() codes, scales = views[1, "codes"], views[1, "scales"] data, factors = source_tables(1) expected = ( data[ids_cpu.flatten()].float() * factors[ids_cpu.flatten()].repeat_interleave(32, 1) ).bfloat16() hbm_codes, hbm_scales = data.npu(), factors.npu() host_fn = partial(gather_dequantize_host_uva, codes, scales, ids, local_heads=24) hbm_fn = partial(gather_dequantize_engram_int8, hbm_codes, hbm_scales, ids, WIDTH, local_heads=24) torch.testing.assert_close(host_fn().cpu(), expected, rtol=0, atol=0) hbm_us, hbm_output = latency(hbm_fn) gva_us, gva_output = latency(host_fn) torch.testing.assert_close(gva_output, expected, rtol=0, atol=0) torch.testing.assert_close(hbm_output, expected, rtol=0, atol=0) result["cases"].append(dict(tokens=tokens, heads=24, hbm_us=hbm_us, gva_us=gva_us)) print(json.dumps(result["cases"][-1]), flush=True) del hbm_codes, hbm_scales, ids # One layer can detach while the other still reads the owner's table. clients[0].close() assert clients[1].runtime is runtime and runtime.references == 1 ids_cpu = torch.tensor([[0], [ROWS - 1], [17]], dtype=torch.int64) ids = ids_cpu.npu() actual = gather_dequantize_host_uva(views[14, "codes"], views[14, "scales"], ids) data, factors = source_tables(14) expected = (data[ids_cpu.flatten()].float() * factors[ids_cpu.flatten()].repeat_interleave(32, 1)).bfloat16() torch.testing.assert_close(actual.cpu(), expected, rtol=0, atol=0) clients[1].close() (root / f"{args.tag}.json").write_text(json.dumps(result, indent=2) + "\n") print("SHARED_POOL_DIRECT_LOOKUP_PASS", flush=True) finally: for client in clients: client.close() if __name__ == "__main__": parser = argparse.ArgumentParser() parser.add_argument("role", choices=("owner", "consumer")) parser.add_argument("--root", required=True) parser.add_argument("--device", type=int, default=0) parser.add_argument("--endpoint", required=True) parser.add_argument("--tag", default="consumer") main(parser.parse_args()) - benchmarks/prepare_indexer_indices.py (3943 bytes)
# SPDX-License-Identifier: Apache-2.0 # SPDX-FileCopyrightText: Copyright contributors to the vLLM project """Compare fused indexer postprocessing in an NPU graph. Run: python benchmarks/prepare_indexer_indices.py Times exclude compilation, graph capture and host tensor allocation. Repetition counts adapt to a warmup measurement so slow INT32-sort baselines stay bounded. """ import argparse import json import statistics from functools import partial import torch import torch_npu # noqa: F401 from vllm_ascend.ops.triton.prepare_indexer_indices import prepare_indexer_indices from vllm_ascend.ops.triton.triton_utils import init_device_properties_triton def reference_indices(selected, positions, compress_ratio): visible = ((positions + 1) // compress_ratio).unsqueeze(-1) valid = (selected >= 0) & (selected < visible) sentinel = torch.iinfo(torch.int32).max selected = torch.where(valid, selected, sentinel).sort(dim=-1).values return torch.where(selected == sentinel, -1, selected) def graph_latency_us(fn, value): for _ in range(3): fn(value) torch.npu.synchronize() start = torch.npu.Event(enable_timing=True) end = torch.npu.Event(enable_timing=True) start.record() fn(value) end.record() end.synchronize() estimate_ms = max(start.elapsed_time(end), 0.001) # Capture at most about 10 ms of work and measure about 50 ms per sample. batch = min(32, max(1, int(10 / estimate_ms))) repeats = min(20, max(1, int(50 / (batch * estimate_ms)))) graph = torch.npu.NPUGraph() with torch.npu.graph(graph, capture_error_mode="thread_local", auto_dispatch_capture=True): for _ in range(batch): output = fn(value) for _ in range(3): graph.replay() torch.npu.synchronize() samples = [] for _ in range(5): start = torch.npu.Event(enable_timing=True) end = torch.npu.Event(enable_timing=True) start.record() for _ in range(repeats): graph.replay() end.record() end.synchronize() samples.append(start.elapsed_time(end) * 1000 / (batch * repeats)) # Keep the captured outputs alive until timing completes. del output return statistics.median(samples) def benchmark(stage, value, reference, fused, **shape): expected, actual = reference(value), fused(value) if isinstance(expected, torch.Tensor): expected, actual = (expected,), (actual,) for output, ref in zip(actual, expected): torch.testing.assert_close(output, ref, rtol=0, atol=0) original_us = graph_latency_us(reference, value) fused_us = graph_latency_us(fused, value) print( json.dumps( { "stage": stage, **shape, "reference_us": original_us, "triton_us": fused_us, "speedup": original_us / fused_us, } ), flush=True, ) @torch.inference_mode() def main(): parser = argparse.ArgumentParser(description=__doc__) parser.add_argument("--tokens", type=int, nargs="+", default=[1, 32, 256, 4096]) parser.add_argument("--topk", type=int, nargs="+", default=[128, 2048]) args = parser.parse_args() torch.npu.set_device(0) init_device_properties_triton() torch.manual_seed(41) for tokens in args.tokens: for topk in args.topk: selected = torch.randint(-1, 4096, (tokens, topk), dtype=torch.int32, device="npu") positions = torch.full((tokens,), 4095, dtype=torch.int64, device="npu") benchmark( "indices", selected, partial(reference_indices, positions=positions, compress_ratio=2), partial(prepare_indexer_indices, positions=positions, compress_ratio=2), tokens=tokens, topk=topk, ) if __name__ == "__main__": main() - benchmarks/quantize_indexer_query.py (2922 bytes)
# SPDX-License-Identifier: Apache-2.0 # SPDX-FileCopyrightText: Copyright contributors to the vLLM project """Compare indexer query quantization latency inside an NPU graph. Run: python benchmarks/quantize_indexer_query.py Times exclude compilation, graph capture and host tensor allocation. """ import argparse import json import statistics import torch import torch_npu # noqa: F401 from vllm_ascend.ops.triton.quantize_indexer_query import quantize_indexer_query from vllm_ascend.ops.triton.triton_utils import init_device_properties_triton def reference(query): scale = (query.float().abs().amax(-1) / 127.0).half().clamp_min_(2.0**-24) quantized = (query.float() / scale.float().unsqueeze(-1)).round().clamp(-127, 127).to(torch.int8) return quantized, scale def graph_latency_us(fn, query, batch=32, repeats=20): for _ in range(3): fn(query) torch.npu.synchronize() graph = torch.npu.NPUGraph() with torch.npu.graph(graph, capture_error_mode="thread_local", auto_dispatch_capture=True): for _ in range(batch): output = fn(query) for _ in range(3): graph.replay() torch.npu.synchronize() samples = [] for _ in range(5): start = torch.npu.Event(enable_timing=True) end = torch.npu.Event(enable_timing=True) start.record() for _ in range(repeats): graph.replay() end.record() end.synchronize() samples.append(start.elapsed_time(end) * 1000 / (batch * repeats)) # Keep the captured outputs alive until timing completes. assert output[0].shape == query.shape return statistics.median(samples) @torch.inference_mode() def main(): parser = argparse.ArgumentParser(description=__doc__) parser.add_argument("--tokens", type=int, nargs="+", default=[1, 32, 256, 4096]) parser.add_argument("--heads", type=int, nargs="+", default=[32, 64]) args = parser.parse_args() torch.npu.set_device(0) init_device_properties_triton() torch.manual_seed(41) for tokens in args.tokens: for heads in args.heads: query = torch.randn(tokens, heads, 128, dtype=torch.bfloat16, device="npu") for actual, expected in zip(quantize_indexer_query(query), reference(query)): torch.testing.assert_close(actual, expected, rtol=0, atol=0) original_us = graph_latency_us(reference, query) fused_us = graph_latency_us(quantize_indexer_query, query) print( json.dumps( { "tokens": tokens, "heads": heads, "reference_us": original_us, "triton_us": fused_us, "speedup": original_us / fused_us, } ), flush=True, ) if __name__ == "__main__": main() - benchmarks/README.md (10811 bytes)
# vLLM Ascend Benchmarks ## Introduction This document outlines the benchmarking methodology for vllm-ascend, aimed at evaluating the performance under a variety of workloads. The primary goal is to help developers assess whether their pull requests improve or degrade vllm-ascend's performance. ## Overview **Benchmarking Coverage**: We measure latency, throughput, and fixed-QPS serving on the Atlas 800I A2 (see [quick_start](../docs/source/getting_started/quick_start.md) to learn more supported devices list), with different models(coming soon). - Latency tests - Input length: 32 tokens. - Output length: 128 tokens. - Batch size: fixed (8). - Models: Qwen2.5-7B-Instruct, Qwen3-8B. - Evaluation metrics: end-to-end latency (mean, median, p99). - Throughput tests - Input length: randomly sample 200 prompts from ShareGPT dataset (with fixed random seed). - Output length: the corresponding output length of these 200 prompts. - Batch size: dynamically determined by vllm to achieve maximum throughput. - Models: Qwen2.5-VL-7B-Instruct, Qwen2.5-7B-Instruct, Qwen3-8B. - Evaluation metrics: throughput. - Serving tests - Input length: randomly sample 200 prompts from ShareGPT dataset (with fixed random seed). - Output length: the corresponding output length of these 200 prompts. - Batch size: dynamically determined by vllm and the arrival pattern of the requests. - **Average QPS (query per second)**: 1, 4, 16 and inf. QPS = inf means all requests come at once. For other QPS values, the arrival time of each query is determined using a random Poisson process (with fixed random seed). - Models: Qwen2.5-VL-7B-Instruct, Qwen2.5-7B-Instruct, Qwen3-8B. - Evaluation metrics: throughput, TTFT (time to the first token, with mean, median and p99), ITL (inter-token latency, with mean, median and p99). **Benchmarking Duration**: about 800 seconds for single model. ## Quick Use ### Prerequisites Before running the benchmarks, ensure the following: - vllm and vllm-ascend are installed and properly set up in an NPU environment, as these scripts are specifically designed for NPU devices. - Install necessary dependencies for benchmarks: ```shell pip install -r benchmarks/requirements-bench.txt ``` - For performance benchmark, it is recommended to set the [load-format](https://github.com/vllm-project/vllm-ascend/blob/5897dc5bbe321ca90c26225d0d70bff24061d04b/benchmarks/tests/latency-tests.json#L7) as `dummy`, It will construct random weights based on the passed model without downloading the weights from internet, which can greatly reduce the benchmark time. - If you want to run a customized benchmark, feel free to add your own models and parameters in the [JSON](https://github.com/vllm-project/vllm-ascend/tree/main/benchmarks/tests), let's take `Qwen2.5-VL-7B-Instruct`as an example: ```json [ { "test_name": "serving_qwen2_5vl_7B_tp1", "qps_list": [ 1, 4, 16, "inf" ], "server_parameters": { "model": "Qwen/Qwen2.5-VL-7B-Instruct", "tensor_parallel_size": 1, "swap_space": 16, "disable_log_stats": "", "disable_log_requests": "", "trust_remote_code": "", "max_model_len": 16384 }, "client_parameters": { "model": "Qwen/Qwen2.5-VL-7B-Instruct", "backend": "openai-chat", "dataset_name": "hf", "hf_split": "train", "endpoint": "/v1/chat/completions", "dataset_path": "lmarena-ai/vision-arena-bench-v0.1", "num_prompts": 200 } } ] ``` this Json will be structured and parsed into server parameters and client parameters by the benchmark script. This configuration defines a test case named `serving_qwen2_5vl_7B_tp1`, designed to evaluate the performance of the `Qwen/Qwen2.5-VL-7B-Instruct` model under different request rates. The test includes both server and client parameters, for more parameters details, see vllm benchmark [cli](https://github.com/vllm-project/vllm/tree/main/vllm/benchmarks). - **Test Overview** - Test Name: serving_qwen2_5vl_7B_tp1 - Queries Per Second (QPS): The test is run at four different QPS levels: 1, 4, 16, and inf (infinite load, typically used for stress testing). - Server Parameters - Model: Qwen/Qwen2.5-VL-7B-Instruct - Tensor Parallelism: 1 (no model parallelism is used; the model runs on a single device or node) - Swap Space: 16 GB (used to handle memory overflow by swapping to disk) - disable_log_stats: disables logging of performance statistics. - disable_log_requests: disables logging of individual requests. - Trust Remote Code: enabled (allows execution of model-specific custom code) - Max Model Length: 16,384 tokens (maximum context length supported by the model) - Client Parameters - Model: Qwen/Qwen2.5-VL-7B-Instruct (same as the server) - Backend: openai-chat (suggests the client uses the OpenAI-compatible chat API format) - Dataset Source: Hugging Face (hf) - Dataset Split: train - Endpoint: /v1/chat/completions (the REST API endpoint to which chat requests are sent) - Dataset Path: lmarena-ai/vision-arena-bench-v0.1 (the benchmark dataset used for evaluation, hosted on Hugging Face) - Number of Prompts: 200 (the total number of prompts used during the test) ### Run benchmarks #### Use benchmark script The provided scripts automatically execute performance tests for serving, throughput, and latency. To start the benchmarking process, run command in the vllm-ascend root directory: ```shell bash benchmarks/scripts/run-performance-benchmarks.sh ``` Once the script completes, you can find the results in the benchmarks/results folder. The output files may resemble the following: ```shell . |-- serving_qwen2_5_7Bvl_tp1_qps_1.json |-- serving_qwen2_5_7Bvl_tp1_qps_16.json |-- serving_qwen2_5_7Bvl_tp1_qps_4.json |-- serving_qwen2_5_7Bvl_tp1_qps_inf.json |-- throughput_qwen2_5_7Bvl_tp1.json ``` These files contain detailed benchmarking results for further analysis. #### Use benchmark cli For more flexible and customized use, benchmark cli is also provided to run online/offline benchmarks Similarly, let's take `Qwen2.5-VL-7B-Instruct` benchmark as an example: ##### Online serving 1. Launch the server: ```shell vllm serve Qwen2.5-VL-7B-Instruct --max-model-len 16789 ``` 2. Running performance tests using cli ```shell vllm bench serve --model Qwen2.5-VL-7B-Instruct\ --endpoint-type "openai-chat" --dataset-name hf \ --hf-split train --endpoint "/v1/chat/completions" \ --dataset-path "lmarena-ai/vision-arena-bench-v0.1" \ --num-prompts 200 \ --request-rate 16 ``` ##### Offline - **Throughput** ```shell vllm bench throughput --output-json results/throughput_qwen2_5_7B_tp1.json \ --model Qwen/Qwen2.5-7B-Instruct --tensor-parallel-size 1 --load-format dummy \ --dataset-path /github/home/.cache/datasets/ShareGPT_V3_unfiltered_cleaned_split.json \ --num-prompts 200 --backend vllm ``` - **Latency** ```shell vllm bench latency --output-json results/latency_qwen2_5_7B_tp1.json \ --model Qwen/Qwen2.5-7B-Instruct --tensor-parallel-size 1 \ --load-format dummy --num-iters-warmup 5 --num-iters 15 ``` ## KV-pool CPU metadata microbenchmark [`benchmark_kv_pool_metadata.py`](benchmark_kv_pool_metadata.py) compares two AscendStore implementations of scheduler request filtering and layerwise leased GVA (global virtual address) preparation. It loads the actual source modules and uses real NumPy arrays with an immediate mock store. It does not start a model, contact a storage service, or execute NPU operations. Its timings cover CPU metadata preparation, not KV transfer, serving throughput, TTFT, or a complete scheduler step. ### Environment and usage Use a configured vLLM/vLLM-Ascend environment with NumPy and the dependencies needed to import the scheduler and worker modules. Both source revisions must be compatible with the same installed vLLM and dependency versions. A standalone Python installation is insufficient even though the measured work runs on CPU. Run from the candidate vLLM-Ascend checkout. The following Linux shell example uses the revision before the metadata lookup optimization as its baseline; replace `BASELINE_REF` to compare another compatible revision: ```shell BASELINE_REF=24a76e84394ae3427195e3715fc08e90769d398c git worktree add --detach ../vllm-ascend-kv-pool-baseline "$BASELINE_REF" export TORCH_DEVICE_BACKEND_AUTOLOAD=0 VLLM_TARGET_DEVICE=empty export OMP_NUM_THREADS=1 OPENBLAS_NUM_THREADS=1 MKL_NUM_THREADS=1 PYTHONPATH="$PWD${PYTHONPATH:+:$PYTHONPATH}" \ python3 benchmarks/benchmark_kv_pool_metadata.py \ --baseline-root ../vllm-ascend-kv-pool-baseline \ --candidate-root . \ --warmup 3 --rounds 5 --iterations 3 \ --output benchmarks/results/kv-pool-metadata.json ``` `--baseline-root`, `--candidate-root`, and `--output` are required. The positive integer options `--warmup`, `--rounds`, and `--iterations` default to 3, 5, and 3. Each round runs baseline, candidate, candidate, baseline (ABBA), giving 30 timed samples per variant and scenario with the defaults. Reset and result validation run outside the timing window; only the target method call is timed. The scheduler's asynchronous-load callback is replaced with an ID recorder to isolate filtering work. ### Scenarios and results The script runs 26 scenarios: - Eight filtering cases: 128/512/1,024/4,096 unfinished requests, with all or half already present in the scheduled cached-request list. This is a scheduling relationship, not a measured external KV-cache hit rate. - Fourteen sparse filtering cases: 128 or 4,096 scheduled cached IDs and 0, 1, or 4 unfinished IDs. Nonempty cases match the head or tail of the cached list, or remain pending because their IDs are absent from it. - Four layerwise GVA cases: 128/512/2,048/8,192 restored blocks. Read `cases[].baseline` and `cases[].candidate` in the output JSON for `median_ms`, interpolated `p95_ms`, `samples`, and every raw sample in `raw_ms`. Each case also records `verified_operations`: callback count/order fingerprints or GVA fingerprints and store operation counts. Assertions check equivalent results between variants and validate each timed call; there is no automatic performance pass/fail threshold. The JSON records Python/platform information, source roots, target-file SHA256 hashes, loaded module paths, and Git revisions when available. Use these fields to verify which implementations were compared. Compare results within the same environment and workload, including sparse cases that may have constant-factor tradeoffs; local metadata speedups do not establish end-to-end serving gains. - benchmarks/requirements-bench.txt (52 bytes)
pandas datasets modelscope<1.38 pyarrow<26 tabulate
- benchmarks/scripts/run-performance-benchmarks.sh (9254 bytes)
#!/bin/bash set -e check_npus() { # shellcheck disable=SC2155 declare -g npu_count=$(npu-smi info -l | grep "Total Count" | awk -F ':' '{print $2}' | tr -d ' ') if [[ -z "$npu_count" || "$npu_count" -eq 0 ]]; then echo "Need at least 1 NPU to run benchmarking." exit 1 else echo "found NPU conut: $npu_count" fi npu_type=$(npu-smi info | grep -E "^\| [0-9]+" | awk -F '|' '{print $2}' | awk '{$1=$1;print}' | awk '{print $2}') echo "NPU type is: $npu_type" } ensure_sharegpt_downloaded() { local FILE="/github/home/.cache/datasets/ShareGPT_V3_unfiltered_cleaned_split.json" local DIR DIR=$(dirname "$FILE") if [ ! -f "$FILE" ]; then echo "$FILE not found, downloading from hf-mirror ..." mkdir -p "$DIR" wget -O "$FILE" https://hf-mirror.com/datasets/anon8231489123/ShareGPT_Vicuna_unfiltered/resolve/main/ShareGPT_V3_unfiltered_cleaned_split.json if [ $? -ne 0 ]; then echo "Download failed!" >&2 return 1 fi echo "Download completed and saved to $FILE" else echo "$FILE already exists." fi } json2args() { # transforms the JSON string to command line args, and '_' is replaced to '-' # example: # input: { "model": "meta-llama/Llama-2-7b-chat-hf", "tensor_parallel_size": 1 } # output: --model meta-llama/Llama-2-7b-chat-hf --tensor-parallel-size 1 local json_string=$1 local args args=$( echo "$json_string" | jq -r ' to_entries | map("--" + (.key | gsub("_"; "-")) + " " + (.value | tostring)) | join(" ") ' ) echo "$args" } wait_for_server() { local waited=0 local timeout_sec=1200 while (( waited < timeout_sec )); do if curl -s -X GET localhost:8000/health > /dev/null; then return 0 fi echo "Waiting for vllm server to start..." sleep 1 ((waited++)) done echo "Timeout waiting for server" return 1 } get_cur_npu_id() { npu-smi info -l | awk -F ':' '/NPU ID/ {print $2+0; exit}' } kill_npu_processes() { ps -aux lsof -t -i:8000 | xargs -r kill -9 pgrep python3 | xargs -r kill -9 # vLLM now names the process with VLLM prefix after https://github.com/vllm-project/vllm/pull/21445 pgrep VLLM | xargs -r kill -9 sleep 4 rm -rf ~/.config/vllm } update_json_field() { local json_file="$1" local field_name="$2" local field_value="$3" jq --arg value "$field_value" \ --arg key "$field_name" \ '.[$key] = $value' "$json_file" > "${json_file}.tmp" && \ mv "${json_file}.tmp" "$json_file" } run_latency_tests() { # run latency tests using `benchmark_latency.py` # $1: a json file specifying latency test cases local latency_test_file latency_test_file=$1 # Iterate over latency tests jq -c '.[]' "$latency_test_file" | while read -r params; do # get the test name, and append the NPU type back to it. test_name=$(echo "$params" | jq -r '.test_name') if [[ ! "$test_name" =~ ^latency_ ]]; then echo "In latency-test.json, test_name must start with \"latency_\"." exit 1 fi # if TEST_SELECTOR is set, only run the test cases that match the selector if [[ -n "$TEST_SELECTOR" ]] && [[ ! "$test_name" =~ $TEST_SELECTOR ]]; then echo "Skip test case $test_name." continue fi # get arguments latency_params=$(echo "$params" | jq -r '.parameters') latency_args=$(json2args "$latency_params") latency_command="vllm bench latency \ --output-json $RESULTS_FOLDER/${test_name}.json \ $latency_args" echo "Running test case $test_name" echo "Latency command: $latency_command" # run the benchmark eval "$latency_command" # echo model_name to result file model_name=$(echo "$latency_params" | jq -r '.model') update_json_field "$RESULTS_FOLDER/${test_name}.json" "model_name" "$model_name" kill_npu_processes done } run_throughput_tests() { # run throughput tests using `benchmark_throughput.py` # $1: a json file specifying throughput test cases local throughput_test_file throughput_test_file=$1 # Iterate over throughput tests jq -c '.[]' "$throughput_test_file" | while read -r params; do # get the test name, and append the NPU type back to it. test_name=$(echo "$params" | jq -r '.test_name') if [[ ! "$test_name" =~ ^throughput_ ]]; then echo "In throughput-test.json, test_name must start with \"throughput_\"." exit 1 fi # if TEST_SELECTOR is set, only run the test cases that match the selector if [[ -n "$TEST_SELECTOR" ]] && [[ ! "$test_name" =~ $TEST_SELECTOR ]]; then echo "Skip test case $test_name." continue fi # get arguments throughput_params=$(echo "$params" | jq -r '.parameters') throughput_args=$(json2args "$throughput_params") throughput_command="vllm bench throughput \ --output-json $RESULTS_FOLDER/${test_name}.json \ $throughput_args" echo "Running test case $test_name" echo "Throughput command: $throughput_command" # run the benchmark eval "$throughput_command" # echo model_name to result file model_name=$(echo "$throughput_params" | jq -r '.model') update_json_field "$RESULTS_FOLDER/${test_name}.json" "model_name" "$model_name" kill_npu_processes done } run_serving_tests() { # run serving tests using `benchmark_serving.py` # $1: a json file specifying serving test cases local serving_test_file serving_test_file=$1 # Iterate over serving tests jq -c '.[]' "$serving_test_file" | while read -r params; do # get the test name, and append the NPU type back to it. test_name=$(echo "$params" | jq -r '.test_name') if [[ ! "$test_name" =~ ^serving_ ]]; then echo "In serving-test.json, test_name must start with \"serving_\"." exit 1 fi # if TEST_SELECTOR is set, only run the test cases that match the selector if [[ -n "$TEST_SELECTOR" ]] && [[ ! "$test_name" =~ $TEST_SELECTOR ]]; then echo "Skip test case $test_name." continue fi # get client and server arguments server_params=$(echo "$params" | jq -r '.server_parameters') client_params=$(echo "$params" | jq -r '.client_parameters') server_args=$(json2args "$server_params") client_args=$(json2args "$client_params") qps_list=$(echo "$params" | jq -r '.qps_list') qps_list=$(echo "$qps_list" | jq -r '.[] | @sh') echo "Running over qps list $qps_list" # check if server model and client model is aligned server_model=$(echo "$server_params" | jq -r '.model') client_model=$(echo "$client_params" | jq -r '.model') if [[ $server_model != "$client_model" ]]; then echo "Server model and client model must be the same. Skip testcase $test_name." continue fi server_command="python3 \ -m vllm.entrypoints.openai.api_server \ $server_args" # run the server echo "Running test case $test_name" echo "Server command: $server_command" bash -c "$server_command" & server_pid=$! # wait until the server is alive if wait_for_server; then echo "" echo "vllm server is up and running." else echo "" echo "vllm failed to start within the timeout period." fi # iterate over different QPS for qps in $qps_list; do # remove the surrounding single quote from qps if [[ "$qps" == *"inf"* ]]; then echo "qps was $qps" qps="inf" echo "now qps is $qps" fi new_test_name=$test_name"_qps_"$qps client_command="vllm bench serve \ --save-result \ --result-dir $RESULTS_FOLDER \ --result-filename ${new_test_name}.json \ --request-rate $qps \ $client_args" echo "Running test case $test_name with qps $qps" echo "Client command: $client_command" bash -c "$client_command" done # clean up kill -9 $server_pid kill_npu_processes done } cleanup() { rm -rf ./vllm_benchmarks } cleanup_on_error() { echo "An error occurred. Cleaning up results folder..." rm -rf $RESULTS_FOLDER } main() { START_TIME=$(date +%s) check_npus # dependencies (which wget && which curl) || (apt-get update && apt-get install -y wget curl) (which jq) || (apt-get update && apt-get -y install jq) (which lsof) || (apt-get update && apt-get install -y lsof) # get the current IP address, required by benchmark_serving.py # shellcheck disable=SC2155 export VLLM_HOST_IP=$(hostname -I | awk '{print $1}') # turn of the reporting of the status of each request, to clean up the terminal output export VLLM_LOG_LEVEL="WARNING" # set env export VLLM_USE_MODELSCOPE=True # prepare for benchmarking cd benchmarks || exit 1 trap cleanup EXIT QUICK_BENCHMARK_ROOT=./ declare -g RESULTS_FOLDER=results mkdir -p $RESULTS_FOLDER trap cleanup_on_error ERR ensure_sharegpt_downloaded # benchmarks run_serving_tests $QUICK_BENCHMARK_ROOT/tests/serving-tests.json run_latency_tests $QUICK_BENCHMARK_ROOT/tests/latency-tests.json run_throughput_tests $QUICK_BENCHMARK_ROOT/tests/throughput-tests.json END_TIME=$(date +%s) ELAPSED_TIME=$((END_TIME - START_TIME)) echo "Total execution time: $ELAPSED_TIME seconds" } main "$@" - CLAUDE.md (104 bytes)
IMPORTANT: Ensure you've thoroughly reviewed the [AGENTS.md](AGENTS.md) file before beginning any work.
- CMakeLists.txt (9523 bytes)
cmake_minimum_required(VERSION 3.16) project(vllm_ascend_C) # include(CheckCXXcompilerFlag) # check_cxx_compiler_flag("-std=c++17", COMPILER_SUPPORTS_CXX17) set(CMAKE_CXX_STANDARD 17) include(${CMAKE_CURRENT_LIST_DIR}/cmake/utils.cmake) # Suppress potential warnings about unused manually-specified variables set(ignoreMe "${VLLM_PYTHON_PATH}") set(PYTHON_SUPPORTED_VERSIONS "3.9" "3.10" "3.11" "3.12") find_package(pybind11 REQUIRED) append_cmake_prefix_path("torch" "torch.utils.cmake_prefix_path") set(VLLM_ASCEND_INSTALL_PATH "${CMAKE_INSTALL_PREFIX}") find_package(Torch REQUIRED) run_python(TORCH_VERSION "import torch; print(torch.__version__)" "Failed to locate torch path") # check torch version is 2.10.0 if(NOT ${TORCH_VERSION} VERSION_EQUAL "2.10.0") message(FATAL_ERROR "Expected PyTorch version 2.10.0, but found ${TORCH_VERSION}") endif() set(RUN_MODE "npu" CACHE STRING "cpu/sim/npu") set(SOC_VERSION ${SOC_VERSION}) message(STATUS "Detected SOC version: ${SOC_VERSION}") if (NOT CMAKE_BUILD_TYPE) set(CMAKE_BUILD_TYPE "Release" CACHE STRINGS "Build type Release/Debug (default Release)" FORCE) endif() if (CMAKE_INSTALL_PREFIX STREQUAL /usr/local) set(CMAKE_INSTALL_PREFIX "${CMAKE_CURRENT_LIST_DIR}/out" CACHE STRINGS "path to install()") endif() set(ASCEND_CANN_PACKAGE_PATH ${ASCEND_HOME_PATH}) if(EXISTS ${ASCEND_HOME_PATH}/tools/tikcpp/ascendc_kernel_cmake) set(ASCENDC_CMAKE_DIR ${ASCEND_HOME_PATH}/tools/tikcpp/ascendc_kernel_cmake) elseif(EXISTS ${ASCEND_HOME_PATH}/compiler/tikcpp/ascendc_kernel_cmake) set(ASCENDC_CMAKE_DIR ${ASCEND_HOME_PATH}/compiler/tikcpp/ascendc_kernel_cmake) elseif(EXISTS ${ASCEND_HOME_PATH}/ascendc_devkit/tikcpp/samples/cmake) set(ASCENDC_CMAKE_DIR ${ASCEND_HOME_PATH}/ascendc_devkit/tikcpp/samples/cmake) else() message(FATAL_ERROR "ascendc_kernel_cmake does not exist, please check whether the cann package is installed.") endif() include(${ASCENDC_CMAKE_DIR}/ascendc.cmake) file(GLOB KERNEL_FILES ${CMAKE_CURRENT_SOURCE_DIR}/csrc/kernels/*.cpp) set(VLLM_ASCEND_CUSTOM_OP ${KERNEL_FILES} ) if(SOC_VERSION MATCHES "ascend310p.*|ascend950") message(STATUS "Hardware ${SOC_VERSION} detected: skip vllm_ascend_kernels compile") else() ascendc_library(vllm_ascend_kernels SHARED ${VLLM_ASCEND_CUSTOM_OP} ) endif() message("TORCH_NPU_PATH is ${TORCH_NPU_PATH}") if(SOC_VERSION MATCHES "ascend310p.*") file(GLOB VLLM_ASCEND_SRC ${CMAKE_CURRENT_SOURCE_DIR}/csrc/*.cpp ${CMAKE_CURRENT_SOURCE_DIR}/csrc/aclnn_torch_adapter/*.cpp) else() file(GLOB VLLM_ASCEND_SRC ${CMAKE_CURRENT_SOURCE_DIR}/csrc/*.cpp ${CMAKE_CURRENT_SOURCE_DIR}/csrc/aclnn_torch_adapter/*.cpp) endif() # Sparse KV cache offloading is currently supported on Atlas A3 only. if(SOC_VERSION MATCHES "ascend910_93.*") find_package(OpenMP REQUIRED) endif() include_directories( ${pybind11_INCLUDE_DIRS} ${PYTHON_INCLUDE_PATH} ${TORCH_INCLUDE_DIRS} # Prefer the selected CANN toolkit headers over the ACL snapshot bundled in # some torch_npu wheels. Feature checks below use ASCEND_HOME_PATH as well. ${ASCEND_HOME_PATH}/include ${TORCH_NPU_PATH}/include ) set( INCLUDES ${TORCH_INCLUDE_DIRS} ${TORCH_NPU_INCLUDE_DIRS} ${ASCEND_HOME_PATH}/include ) pybind11_add_module(vllm_ascend_C ${VLLM_ASCEND_SRC}) if(SOC_VERSION MATCHES "ascend910_93.*") target_compile_definitions(vllm_ascend_C PRIVATE VLLM_ASCEND_ENABLE_SPARSE_KV_OFFLOAD) target_link_libraries(vllm_ascend_C PRIVATE OpenMP::OpenMP_CXX) endif() # Detect aclrtMemcpyBatchAsync availability (CANN 8.5+) # Can be overridden via VLLM_ASCEND_ENABLE_BATCH_MEMCPY env var (registered # in vllm_ascend/envs.py, forwarded by setup.py as a CMake variable): # VLLM_ASCEND_ENABLE_BATCH_MEMCPY=1 -> force enable # VLLM_ASCEND_ENABLE_BATCH_MEMCPY=0 -> force disable # unset -> auto-detect from CANN headers include(CheckCXXSourceCompiles) set(CMAKE_REQUIRED_INCLUDES ${ASCEND_HOME_PATH}/include) set(CMAKE_REQUIRED_LIBRARIES ascendcl) set(CMAKE_REQUIRED_LINK_OPTIONS "-L${ASCEND_HOME_PATH}/lib64") # Keep builds on older CANN versions working; endpoint selection must fail # explicitly if the installed CANN lacks the user-to-physical conversion. check_cxx_source_compiles(" #include <acl/acl_rt.h> int main() { int32_t physical_id = 0; return aclrtGetPhyDevIdByUserDevId(0, &physical_id); } " HAVE_ACLRT_GET_PHY_DEV_ID_BY_USER_DEV_ID) if(HAVE_ACLRT_GET_PHY_DEV_ID_BY_USER_DEV_ID) target_compile_definitions(vllm_ascend_C PRIVATE CANN_DEVICE_ID_MAPPING) endif() if(DEFINED VLLM_ASCEND_ENABLE_BATCH_MEMCPY) if("${VLLM_ASCEND_ENABLE_BATCH_MEMCPY}" STREQUAL "1") message(STATUS "aclrtMemcpyBatchAsync: force enabled via VLLM_ASCEND_ENABLE_BATCH_MEMCPY=1") target_compile_definitions(vllm_ascend_C PRIVATE CANN_MEMCPY_BATCH_ASYNC) else() message(STATUS "aclrtMemcpyBatchAsync: force disabled via VLLM_ASCEND_ENABLE_BATCH_MEMCPY=0") endif() else() # Test the full code pattern we actually use, including struct member access. # This ensures the macro is only defined when the API is fully compatible. check_cxx_source_compiles(" #include <acl/acl_rt.h> int main() { aclrtMemLocation loc = {}; loc.type = ACL_MEM_LOCATION_TYPE_HOST; loc.id = 0; aclrtMemcpyBatchAttr attr = {}; attr.srcLoc = loc; attr.dstLoc = loc; (void)aclrtMemcpyBatchAsync; return 0; } " HAVE_ACLRT_MEMCPY_BATCH_ASYNC) if(HAVE_ACLRT_MEMCPY_BATCH_ASYNC) message(STATUS "aclrtMemcpyBatchAsync: detected in CANN headers, enabling batch memcpy path") target_compile_definitions(vllm_ascend_C PRIVATE CANN_MEMCPY_BATCH_ASYNC) else() message(STATUS "aclrtMemcpyBatchAsync: not found in CANN headers, using fallback aclrtMemcpyAsync loop") endif() endif() # ECCPUConnector stores encoder-cache data in a shared mmap. Newer CANN releases # expose aclrtHostRegisterV2(..., ACL_HOST_REG_PINNED), which turns those mmap # pages into real page-locked host memory. Keep the custom op available on older # CANN releases as well, but make it fail with an actionable error at runtime # instead of silently claiming that pageable memory is pinned. check_cxx_source_compiles(" #include <acl/acl_rt.h> int main() { void* ptr = nullptr; (void)aclrtHostRegisterV2(ptr, 0, ACL_HOST_REG_PINNED); (void)aclrtHostUnregister; return 0; } " HAVE_ACLRT_HOST_REGISTER_V2_PINNED) if(HAVE_ACLRT_HOST_REGISTER_V2_PINNED) message(STATUS "aclrtHostRegisterV2: pinned host registration available") target_compile_definitions(vllm_ascend_C PRIVATE CANN_HOST_REGISTER_V2_PINNED) else() message(STATUS "aclrtHostRegisterV2: pinned host registration unavailable; ECCPUConnector will be rejected") endif() if(SOC_VERSION MATCHES "ascend310p.*") target_compile_definitions(vllm_ascend_C PRIVATE -DASCEND_PLATFORM_310P) endif() if(NOT (SOC_VERSION MATCHES "ascend310p.*|ascend950")) target_compile_definitions(vllm_ascend_C PRIVATE -DVLLM_ENABLE_ATB_AND_DIRECT_KERNELS) endif() target_link_directories( vllm_ascend_C PRIVATE ${TORCH_LIBRARY_DIRS} ${TORCH_NPU_PATH}/lib/ ${ASCEND_HOME_PATH}/lib64 ) set(VLLM_ASCEND_C_COMMON_LIBS ${TORCH_LIBRARIES} torch_npu ascendcl tiling_api register platform ascendalog dl opapi ) if(SOC_VERSION MATCHES "ascend310p.*|ascend950") target_link_libraries( vllm_ascend_C PUBLIC ${VLLM_ASCEND_C_COMMON_LIBS} ) else() target_link_libraries( vllm_ascend_C PUBLIC vllm_ascend_kernels ${VLLM_ASCEND_C_COMMON_LIBS} ) endif() target_link_options(vllm_ascend_C PRIVATE "-Wl,-rpath,$ORIGIN:$ORIGIN/lib:$ORIGIN/_cann_ops_custom/vendors/custom_transformer/op_api/lib") # A3 shared host Engram uses the optional CANN Fabric-handle API. if(SOC_VERSION MATCHES "ascend910_93.*") check_cxx_source_compiles(" #include <acl/acl.h> int main() { aclrtMemFabricHandle handle{}; (void)handle; (void)ACL_MEM_P2P_HUGE1G; (void)ACL_MEM_LOCATION_TYPE_HOST; (void)ACL_RT_VMM_EXPORT_FLAG_DISABLE_PID_VALIDATION; (void)aclrtMemExportToShareableHandleV2; (void)aclrtMemImportFromShareableHandleV2; return 0; } " HAVE_ENGRAM_HOST_VMM) if(HAVE_ENGRAM_HOST_VMM) find_package(Threads REQUIRED) add_library(engram_host_vmm SHARED csrc/engram_vmm/native.cpp) target_link_directories(engram_host_vmm PRIVATE ${ASCEND_HOME_PATH}/lib64) target_link_libraries(engram_host_vmm PRIVATE ascendcl Threads::Threads) add_dependencies(vllm_ascend_C engram_host_vmm) install(TARGETS engram_host_vmm DESTINATION ${VLLM_ASCEND_INSTALL_PATH}) endif() endif() if(EXISTS "${ASCEND_HOME_PATH}/include/hixl/hixl.h") find_library(ENGRAM_HIXL_LIBRARY cann_hixl PATHS ${ASCEND_HOME_PATH}/lib64) if(ENGRAM_HIXL_LIBRARY) add_library(engram_hixl SHARED csrc/engram_hixl/native.cpp) target_include_directories(engram_hixl PRIVATE ${ASCEND_HOME_PATH}/include) target_link_directories(engram_hixl PRIVATE ${ASCEND_HOME_PATH}/lib64) target_link_libraries(engram_hixl PRIVATE ${ENGRAM_HIXL_LIBRARY} ascendcl) add_dependencies(vllm_ascend_C engram_hixl) install(TARGETS engram_hixl DESTINATION ${VLLM_ASCEND_INSTALL_PATH}) endif() endif() if(SOC_VERSION MATCHES "ascend310p.*|ascend950") install(TARGETS vllm_ascend_C DESTINATION ${VLLM_ASCEND_INSTALL_PATH}) else() install(TARGETS vllm_ascend_C vllm_ascend_kernels DESTINATION ${VLLM_ASCEND_INSTALL_PATH}) endif() - CODE_OF_CONDUCT.md (5336 bytes)
# vLLM Code of Conduct ## Our Pledge We as members, contributors, and leaders pledge to make participation in our community a harassment-free experience for everyone, regardless of age, body size, visible or invisible disability, ethnicity, sex characteristics, gender identity and expression, level of experience, education, socioeconomic status, nationality, personal appearance, race, caste, color, religion, or sexual identity and orientation. We pledge to act and interact in ways that contribute to an open, welcoming, diverse, inclusive, and healthy community. ## Our Standards Examples of behavior that contributes to a positive environment for our community include: * Demonstrating empathy and kindness toward other people * Being respectful of differing opinions, viewpoints, and experiences * Giving and gracefully accepting constructive feedback * Accepting responsibility and apologizing to those affected by our mistakes, and learning from the experience * Focusing on what is best not just for us as individuals, but for the overall community Examples of unacceptable behavior include: * The use of sexualized language or imagery, and sexual attention or advances of any kind * Trolling, insulting or derogatory comments, and personal or political attacks * Public or private harassment * Publishing others' private information, such as a physical or email address, without their explicit permission * Other conduct which could reasonably be considered inappropriate in a professional setting ## Enforcement Responsibilities Community leaders are responsible for clarifying and enforcing our standards of acceptable behavior and will take appropriate and fair corrective action in response to any behavior that they deem inappropriate, threatening, offensive, or harmful. Community leaders have the right and responsibility to remove, edit, or reject comments, commits, code, wiki edits, issues, and other contributions that are not aligned to this Code of Conduct, and will communicate reasons for moderation decisions when appropriate. ## Scope This Code of Conduct applies within all community spaces, and also applies when an individual is officially representing the community in public spaces. Examples of representing our community include using an official email address, posting via an official social media account, or acting as an appointed representative at an online or offline/IRL event. ## Enforcement If you experience any abusive, harassing, or otherwise unacceptable behavior, please feel free to report it to us via [Ascend opensource assistant](docs/source/community/images/ascend_assistant.png) or <[email protected]>. All complaints will be reviewed and investigated promptly and fairly. All community leaders are obligated to respect the privacy and security of the reporter of any incident. ## Enforcement Guidelines Community leaders will follow these Community Impact Guidelines in determining the consequences for any action they deem in violation of this Code of Conduct: ### 1. Correction **Community Impact**: Use of inappropriate language or other behavior deemed unprofessional or unwelcome in the community. **Consequence**: A private, written warning from community leaders, providing clarity around the nature of the violation and an explanation of why the behavior was inappropriate. A public apology may be requested. ### 2. Warning **Community Impact**: A violation through a single incident or series of actions. **Consequence**: A warning with consequences for continued behavior. No interaction with the people involved, including unsolicited interaction with those enforcing the Code of Conduct, for a specified period of time. This includes avoiding interactions in community spaces as well as external channels like social media. Violating these terms may lead to a temporary or permanent ban. ### 3. Temporary Ban **Community Impact**: A serious violation of community standards, including sustained inappropriate behavior. **Consequence**: A temporary ban from any sort of interaction or public communication with the community for a specified period of time. No public or private interaction with the people involved, including unsolicited interaction with those enforcing the Code of Conduct, is allowed during this period. Violating these terms may lead to a permanent ban. ### 4. Permanent Ban **Community Impact**: Demonstrating a pattern of violation of community standards, including sustained inappropriate behavior, harassment of an individual, or aggression toward or disparagement of classes of individuals. **Consequence**: A permanent ban from any sort of public interaction within the community. ## Attribution This Code of Conduct is adapted from the [Contributor Covenant](https://www.contributor-covenant.org/), version 2.1, available at [v2.1](https://www.contributor-covenant.org/version/2/1/code_of_conduct.html). Community Impact Guidelines were inspired by [Mozilla's code of conduct enforcement ladder](https://github.com/mozilla/inclusion). For answers to common questions about this code of conduct, see the [Contributor Covenant FAQ](https://www.contributor-covenant.org/faq). Translations are available at [Contributor Covenant translations](https://www.contributor-covenant.org/translations).
- codecov.yml (886 bytes)
# # Copyright (c) 2025 Huawei Technologies Co., Ltd. All Rights Reserved. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # This file is a part of the vllm-ascend project. # coverage: status: # Patch coverage is mandatory and must be >= 80% patch: default: target: 80% # non-voting project: default: # non-voting informational: true - collect_env.py (14118 bytes)
# # Copyright 2023 The vLLM team. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # Adapted from https://github.com/vllm-project/vllm/blob/main/collect_env.py # import datetime import locale import os import subprocess import sys from collections import namedtuple import regex as re from vllm.envs import environment_variables try: import torch TORCH_AVAILABLE = True except (ImportError, NameError, AttributeError, OSError): TORCH_AVAILABLE = False # System Environment Information SystemEnv = namedtuple( "SystemEnv", [ "torch_version", "is_debug_build", "gcc_version", "clang_version", "cmake_version", "os", "libc_version", "python_version", "python_platform", "pip_version", # 'pip' or 'pip3' "pip_packages", "conda_packages", "cpu_info", "vllm_version", # vllm specific field "vllm_ascend_version", # vllm ascend specific field "env_vars", "npu_info", # ascend specific field "cann_info", # ascend specific field ], ) DEFAULT_CONDA_PATTERNS = { "torch", "numpy", "soumith", "mkl", "magma", "optree", "transformers", "zmq", "pynvml", "triton-ascend", } DEFAULT_PIP_PATTERNS = { "torch", "numpy", "mypy", "flake8", "optree", "onnx", "transformers", "zmq", "pynvml", "triton-ascend", } def run(command): """Return (return-code, stdout, stderr).""" shell = isinstance(command, str) p = subprocess.Popen(command, stdout=subprocess.PIPE, stderr=subprocess.PIPE, shell=shell) raw_output, raw_err = p.communicate() rc = p.returncode if get_platform() == "win32": enc = "oem" else: enc = locale.getpreferredencoding() output = raw_output.decode(enc) err = raw_err.decode(enc) return rc, output.strip(), err.strip() def run_and_read_all(run_lambda, command): """Run command using run_lambda; reads and returns entire output if rc is 0.""" rc, out, _ = run_lambda(command) if rc != 0: return None return out def run_and_parse_first_match(run_lambda, command, regex): """Run command using run_lambda, returns the first regex match if it exists.""" rc, out, _ = run_lambda(command) if rc != 0: return None match = re.search(regex, out) if match is None: return None return match.group(1) def run_and_return_first_line(run_lambda, command): """Run command using run_lambda and returns first line if output is not empty.""" rc, out, _ = run_lambda(command) if rc != 0: return None return out.split("\n")[0] def get_conda_packages(run_lambda, patterns=None): if patterns is None: patterns = DEFAULT_CONDA_PATTERNS conda = os.environ.get("CONDA_EXE", "conda") out = run_and_read_all(run_lambda, "{} list".format(conda)) if out is None: return out return "\n".join( line for line in out.splitlines() if not line.startswith("#") and any(name in line for name in patterns) ) def get_gcc_version(run_lambda): return run_and_parse_first_match(run_lambda, "gcc --version", r"gcc (.*)") def get_clang_version(run_lambda): return run_and_parse_first_match(run_lambda, "clang --version", r"clang version (.*)") def get_cmake_version(run_lambda): return run_and_parse_first_match(run_lambda, "cmake --version", r"cmake (.*)") def _parse_version(version, version_tuple): version_str = version_tuple[-1] if isinstance(version_str, str) and version_str.startswith("g"): if "." in version_str: git_sha = version_str.split(".")[0][1:] date = version_str.split(".")[-1][1:] return f"{version} (git sha: {git_sha}, date: {date})" else: git_sha = version_str[1:] # type: ignore return f"{version} (git sha: {git_sha})" return version def get_vllm_version(): from vllm import __version__, __version_tuple__ return _parse_version(__version__, __version_tuple__) def get_vllm_ascend_version(): from vllm_ascend._version import __version__, __version_tuple__ return _parse_version(__version__, __version_tuple__) def get_cpu_info(run_lambda): rc, out, err = 0, "", "" if get_platform() == "linux": rc, out, err = run_lambda("lscpu") elif get_platform() == "win32": rc, out, err = run_lambda( "wmic cpu get Name,Manufacturer,Family,Architecture,ProcessorType,DeviceID, \ CurrentClockSpeed,MaxClockSpeed,L2CacheSize,L2CacheSpeed,Revision /VALUE" ) elif get_platform() == "darwin": rc, out, err = run_lambda("sysctl -n machdep.cpu.brand_string") cpu_info = "None" if rc == 0: cpu_info = out else: cpu_info = err return cpu_info def get_platform(): if sys.platform.startswith("linux"): return "linux" elif sys.platform.startswith("win32"): return "win32" elif sys.platform.startswith("cygwin"): return "cygwin" elif sys.platform.startswith("darwin"): return "darwin" else: return sys.platform def get_mac_version(run_lambda): return run_and_parse_first_match(run_lambda, "sw_vers -productVersion", r"(.*)") def get_windows_version(run_lambda): system_root = os.environ.get("SYSTEMROOT", "C:\\Windows") wmic_cmd = os.path.join(system_root, "System32", "Wbem", "wmic") findstr_cmd = os.path.join(system_root, "System32", "findstr") return run_and_read_all(run_lambda, "{} os get Caption | {} /v Caption".format(wmic_cmd, findstr_cmd)) def get_lsb_version(run_lambda): return run_and_parse_first_match(run_lambda, "lsb_release -a", r"Description:\t(.*)") def check_release_file(run_lambda): return run_and_parse_first_match(run_lambda, "cat /etc/*-release", r'PRETTY_NAME="(.*)"') def get_os(run_lambda): from platform import machine platform = get_platform() if platform == "win32" or platform == "cygwin": return get_windows_version(run_lambda) if platform == "darwin": version = get_mac_version(run_lambda) if version is None: return None return "macOS {} ({})".format(version, machine()) if platform == "linux": # Ubuntu/Debian based desc = get_lsb_version(run_lambda) if desc is not None: return "{} ({})".format(desc, machine()) # Try reading /etc/*-release desc = check_release_file(run_lambda) if desc is not None: return "{} ({})".format(desc, machine()) return "{} ({})".format(platform, machine()) # Unknown platform return platform def get_python_platform(): import platform return platform.platform() def get_libc_version(): import platform if get_platform() != "linux": return "N/A" return "-".join(platform.libc_ver()) def get_pip_packages(run_lambda, patterns=None): """Return `pip list` output. Note: will also find conda-installed pytorch and numpy packages.""" if patterns is None: patterns = DEFAULT_PIP_PATTERNS # People generally have `pip` as `pip` or `pip3` # But here it is invoked as `python -mpip` def run_with_pip(pip): out = run_and_read_all(run_lambda, pip + ["list", "--format=freeze"]) return "\n".join(line for line in out.splitlines() if any(name in line for name in patterns)) pip_version = "pip3" if sys.version[0] == "3" else "pip" out = run_with_pip([sys.executable, "-mpip"]) return pip_version, out def get_npu_info(run_lambda): return run_and_read_all(run_lambda, "npu-smi info") def get_cann_info(run_lambda): out = run_and_read_all(run_lambda, "lscpu | grep Architecture:") cpu_arch = str(out).split()[-1] return run_and_read_all( run_lambda, "cat /usr/local/Ascend/ascend-toolkit/latest/{}-linux/ascend_toolkit_install.info".format(cpu_arch) ) def get_env_vars(): env_vars = "" secret_terms = ("secret", "token", "api", "access", "password") report_prefix = ("TORCH", "PYTORCH", "ASCEND_", "ATB_") for k, v in os.environ.items(): if any(term in k.lower() for term in secret_terms): continue if k in environment_variables: env_vars = env_vars + "{}={}".format(k, v) + "\n" if k.startswith(report_prefix): env_vars = env_vars + "{}={}".format(k, v) + "\n" return env_vars def get_env_info(): run_lambda = run pip_version, pip_list_output = get_pip_packages(run_lambda) if TORCH_AVAILABLE: version_str = torch.__version__ debug_mode_str = str(torch.version.debug) else: version_str = debug_mode_str = "N/A" sys_version = sys.version.replace("\n", " ") conda_packages = get_conda_packages(run_lambda) return SystemEnv( torch_version=version_str, is_debug_build=debug_mode_str, python_version="{} ({}-bit runtime)".format(sys_version, sys.maxsize.bit_length() + 1), python_platform=get_python_platform(), pip_version=pip_version, pip_packages=pip_list_output, conda_packages=conda_packages, os=get_os(run_lambda), libc_version=get_libc_version(), gcc_version=get_gcc_version(run_lambda), clang_version=get_clang_version(run_lambda), cmake_version=get_cmake_version(run_lambda), cpu_info=get_cpu_info(run_lambda), vllm_version=get_vllm_version(), vllm_ascend_version=get_vllm_ascend_version(), env_vars=get_env_vars(), npu_info=get_npu_info(run_lambda), cann_info=get_cann_info(run_lambda), ) env_info_fmt = """ PyTorch version: {torch_version} Is debug build: {is_debug_build} OS: {os} GCC version: {gcc_version} Clang version: {clang_version} CMake version: {cmake_version} Libc version: {libc_version} Python version: {python_version} Python platform: {python_platform} CPU: {cpu_info} Versions of relevant libraries: {pip_packages} {conda_packages} """.strip() # both the above code and the following code use `strip()` to # remove leading/trailing whitespaces, so we need to add a newline # in between to separate the two sections env_info_fmt += "\n" env_info_fmt += """ vLLM Version: {vllm_version} vLLM Ascend Version: {vllm_ascend_version} ENV Variables: {env_vars} NPU: {npu_info} CANN: {cann_info} """.strip() def pretty_str(envinfo): def replace_nones(dct, replacement="Could not collect"): for key in dct: if dct[key] is not None: continue dct[key] = replacement return dct def replace_bools(dct, true="Yes", false="No"): for key in dct: if dct[key] is True: dct[key] = true elif dct[key] is False: dct[key] = false return dct def prepend(text, tag="[prepend]"): lines = text.split("\n") updated_lines = [tag + line for line in lines] return "\n".join(updated_lines) def replace_if_empty(text, replacement="No relevant packages"): if text is not None and len(text) == 0: return replacement return text def maybe_start_on_next_line(string): # If `string` is multiline, prepend a \n to it. if string is not None and len(string.split("\n")) > 1: return "\n{}\n".format(string) return string mutable_dict = envinfo._asdict() # Replace True with Yes, False with No mutable_dict = replace_bools(mutable_dict) # Replace all None objects with 'Could not collect' mutable_dict = replace_nones(mutable_dict) # If either of these are '', replace with 'No relevant packages' mutable_dict["pip_packages"] = replace_if_empty(mutable_dict["pip_packages"]) mutable_dict["conda_packages"] = replace_if_empty(mutable_dict["conda_packages"]) # Tag conda and pip packages with a prefix # If they were previously None, they'll show up as ie '[conda] Could not collect' if mutable_dict["pip_packages"]: mutable_dict["pip_packages"] = prepend(mutable_dict["pip_packages"], "[{}] ".format(envinfo.pip_version)) if mutable_dict["conda_packages"]: mutable_dict["conda_packages"] = prepend(mutable_dict["conda_packages"], "[conda] ") mutable_dict["cpu_info"] = envinfo.cpu_info mutable_dict["npu_info"] = envinfo.npu_info mutable_dict["cann_info"] = envinfo.cann_info return env_info_fmt.format(**mutable_dict) def get_pretty_env_info(): return pretty_str(get_env_info()) def main(): print("Collecting environment information...") output = get_pretty_env_info() print(output) if TORCH_AVAILABLE and hasattr(torch, "utils") and hasattr(torch.utils, "_crash_handler"): minidump_dir = torch.utils._crash_handler.DEFAULT_MINIDUMP_DIR if sys.platform == "linux" and os.path.exists(minidump_dir): dumps = [os.path.join(minidump_dir, dump) for dump in os.listdir(minidump_dir)] latest = max(dumps, key=os.path.getctime) ctime = os.path.getctime(latest) creation_time = datetime.datetime.fromtimestamp(ctime).strftime("%Y-%m-%d %H:%M:%S") msg = ( "\n*** Detected a minidump at {} created on {}, ".format(latest, creation_time) + "if this is related to your bug please include it when you file a report ***" ) print(msg, file=sys.stderr) if __name__ == "__main__": main() - CONTRIBUTING.md (325 bytes)
# Contributing to vLLM Ascend You may find information about contributing to vLLM Ascend on [Developer Guide - Contributing](https://docs.vllm.ai/projects/ascend/en/latest/developer_guide/contribution/index.html), including step-by-step guide to help you setup development environment, contribute first PR and test locally.
- csrc/attention/attn_res_fwd/CMakeLists.txt (1197 bytes)
# ----------------------------------------------------------------------------------------------------------- # Copyright (c) 2026 Huawei Technologies Co., Ltd. # This program is free software, you can redistribute it and/or modify it under the terms and conditions of # CANN Open Software License Agreement Version 2.0 (the "License"). # Please refer to the License for details. You may not use this file except in compliance with the License. # THIS SOFTWARE IS PROVIDED ON AN "AS IS" BASIS, WITHOUT WARRANTIES OF ANY KIND, EITHER EXPRESS OR IMPLIED, # INCLUDING BUT NOT LIMITED TO NON-INFRINGEMENT, MERCHANTABILITY, OR FITNESS FOR A PARTICULAR PURPOSE. # See LICENSE in the root of the software repository for the full text of the License. # ----------------------------------------------------------------------------------------------------------- file(GLOB CURRENT_DIRS RELATIVE ${CMAKE_CURRENT_SOURCE_DIR} ${CMAKE_CURRENT_SOURCE_DIR}/*) if(NOT ENABLE_TEST AND NOT BENCHMARK) list(REMOVE_ITEM CURRENT_DIRS tests) endif() foreach(SUB_DIR ${CURRENT_DIRS}) if(EXISTS "${CMAKE_CURRENT_SOURCE_DIR}/${SUB_DIR}/CMakeLists.txt") add_subdirectory(${SUB_DIR}) endif() endforeach() - csrc/attention/attn_res_fwd/op_host/CMakeLists.txt (1851 bytes)
# ----------------------------------------------------------------------------------------------------------- # Copyright (c) 2026 Huawei Technologies Co., Ltd. # This program is free software, you can redistribute it and/or modify it under the terms and conditions of # CANN Open Software License Agreement Version 2.0 (the "License"). # Please refer to the License for details. You may not use this file except in compliance with the License. # THIS SOFTWARE IS PROVIDED ON AN "AS IS" BASIS, WITHOUT WARRANTIES OF ANY KIND, EITHER EXPRESS OR IMPLIED, # INCLUDING BUT NOT LIMITED TO NON-INFRINGEMENT, MERCHANTABILITY, OR FITNESS FOR A PARTICULAR PURPOSE. # See LICENSE in the root of the software repository for the full text of the License. # ----------------------------------------------------------------------------------------------------------- add_op_to_compiled_list() set(CONDITION_UNIT ${ASCEND_COMPUTE_UNIT}) if (BUILD_OPEN_PROJECT) target_sources(op_host_aclnnExc PRIVATE attn_res_fwd_def.cpp ) if(CONDITION_UNIT STREQUAL "ascend950") add_ops_compile_options( OP_NAME AttnResFwd OPTIONS -I${CMAKE_CURRENT_SOURCE_DIR}/../op_kernel -g --cce-auto-sync=off -Wno-deprecated-declarations -Werror -mllvm -cce-aicore-dcci-before-kernel-end=false ) else() add_ops_compile_options( OP_NAME AttnResFwd OPTIONS -I${CMAKE_CURRENT_SOURCE_DIR}/../op_kernel -g --cce-auto-sync=off -Wno-deprecated-declarations -Werror ) endif() endif() if(NOT BUILD_OPS_RTY_KERNEL) add_modules_sources(OPTYPE attn_res_fwd ACLNNTYPE aclnn_exclude) endif() - csrc/attention/attn_res_fwd/README.md (19642 bytes)
# AttnResFwd ## 当前统一 N 轴并行方案 本轮以正式 PR 的 `c730f7177` 为优化前对照。 BF16、H7168、B1/4/8/16、N1到8的目标混合用例全部使用新 kernel 沿候选行并行打分, 不再回退到旧 kernel 或单 worker token 打分。核组大小由候选行数、可用核数和 UB 预算计算; 输出仍由每个 token 的一个核处理完整 H,并保留一次就绪同步。 Token 核组路径中,输出 owner 的自身分数保留在 UB,只读取其他 worker 的分数;prefix 写回与打分重叠, 在就绪同步前保证写回完成。A2 另在融合加法时重叠读取 addend,省去一次中间 BF16 拷贝。 未改变计算顺序或两处 BF16 舍入边界。A2 小 N 相对正式 PR 仍有性能退化,继续保留新路径优化。 历史测量描述其对应冻结版本;最新四轮复测见本文末尾,不混用不同版本的性能数据。 ## 功能说明 `AttnResFwd` 是 Kimi K3 的注意力残差融合算子。单个 Torch 入口 `torch.ops._C_ascend.attn_res_fwd` 将可选的 BF16 加法、残差 bank 混合、 可选的输出 RMSNorm,以及 bank 写回放在一次调用中。源码注册覆盖 A2、A3 和 A5,小 batch 协作路径同时接入三类平台。已在 Ascend910B3 和 Ascend950PR_9579 上完成本仓库源码的原生构建、安装及完整算子精度验证, 包含大 H 分段用例。A5 使用独立 RegBase 协作实现;A3 仍需对应硬件验证。 令 `p` 为 `prefix_sum`(有 `addend` 时先做 BF16 加法),`v_i` 为 `block_residual` 中前 `num_valid_blocks` 行,并令最后一行 `v_N=p`。 启用 `mix` 时,每行先按 `norm_eps` 做 RMSNorm,再与 `norm_weight`、 `proj_weight` 相乘并沿隐藏维求和,所得分数经 softmax 后对原始 `v_i` 加权求和并舍入到 BF16。`mix=False` 时直接使用 `p`。传入 `output_norm_weight` 时,对混合结果再做一次 RMSNorm。 ## 接口与参数 ```python output, raw_prefix, materialized = torch.ops._C_ascend.attn_res_fwd( prefix_sum, addend, block_residual, proj_weight, norm_weight, norm_eps, num_valid_blocks, output_norm_weight=None, output_norm_eps=1e-5, block_write_idx=-1, return_materialized=False, mix=True, optimize_prefill=False, ) ``` | 参数 | 形状与约束 | 作用 | | --- | --- | --- | | `prefix_sum` | `[T,H]`,BF16,连续 | 当前层的残差前缀。 | | `addend` | `None` 或 `[T,H]`,BF16,连续 | 可选的前层输出;加法结果以 BF16 供后续计算。 | | `block_residual` | `[T,S,H]`,BF16;每行连续,token 间可有 stride | 残差 bank;仅前 `num_valid_blocks` 行参与混合。 | | `proj_weight`、`norm_weight` | 分别为 `[1,H]`、`[H]`,BF16,连续 | 混合分数的投影和 RMSNorm 权重。 | | `norm_eps`、`num_valid_blocks` | `norm_eps` 为 FP32 可表示的有限正数;`0 ≤ num_valid_blocks ≤ S` | 混合归一化参数和有效 bank 行数。 | | `output_norm_weight`、`output_norm_eps` | `None` 或 `[H]`,BF16,连续;传入权重时 epsilon 为 FP32 可表示的有限正数 | 可选的输出 RMSNorm。 | | `block_write_idx` | `-1` 或 `[num_valid_blocks,S)` 中的行号 | 将 `raw_prefix` 写入尚未参与本次混合的 bank 行;`-1` 表示不写回。 | | `return_materialized` | `bool` | 为真时单独保留归一化前的混合结果。 | | `mix` | `bool` | 为假时跳过 bank 混合,用于前层或只做加法的路径。 | | `optimize_prefill` | `bool` | 请求 prefill 策略;满足输出归一化、每核多 token 和 UB 容量条件时复用缓存权重。 | 所有 Tensor 必须位于同一设备。`output`、`raw_prefix` 和 `materialized` 均为 `[T,H]`、BF16。无 `addend` 时 `raw_prefix` 复用输入 `prefix_sum`;`return_materialized=False` 时 `materialized` 复用 `output`。写回只修改指定的 bank 行,不修改 `prefix_sum` 或 `addend`。`T=0` 可走空输出路径。 启用 bank 写回时,`block_residual` 不得与其他只读输入共享存储;Torch 入口会在 提交 kernel 前拒绝这类别名。关闭写回时仍允许只读输入共享 bank 存储。 底层 ACLNN 调用方同样需要保证写回 bank 与其他输入不重叠。 ## 分核与数据复用 令 token 数为 T、有效历史行数为 N、候选行数为 R=N+1、可用 AIV 核数为 C。 每次调用只启动一个 kernel。完整 H 能否驻留由 UB 容量决定,不以7168作为硬编码边界。 ### 全局行并行(40030) 1. 将 T*R 个完整 H 打分任务按核号步进分发,启动 `min(C,T*R)` 个核。 每行计算 RMSNorm 与投影点积,在独占的32字节 workspace 槽写一个 score。 2. 所有启动核执行一次 `SyncAll<true>()`,等待分数与融合 prefix 就绪。 3. 启动核数为 U 时,token t 的输出核为 `floor(t*U/T)`,每个 token 有唯一输出核。 该核对 R 个 score 做 softmax,读取原始候选行并对完整 H 加权累加。 4. 混合结果先舍入 BF16,再由该核完成可选的输出 RMSNorm,无第二次核间同步。 打分先处理 prefix,score 存储仍为历史行在前、prefix 在后。无任务循环覆盖时, 输出 owner 可复用驻留的 BF16 prefix;其余情况重新读取,避免使用已覆盖的中间量。 ### Token 核组(40040) 完整行满足 UB 预算且 `T>C/4` 时,Host 按候选行数和可用核数计算均匀 token 核组: `Gmax=min(R,C/T)`、`W=ceil(R/Gmax)`、`G=ceil(R/W)`,使用 `T*G` 个打分核。 它保留最大均匀核组的关键打分波数,并减少没有缩短关键路径的多余 worker。 低 token 占用保留全局行调度或既有的多 worker 核组;目标矩阵不回退到单核 token 打分。 核号 `core/groupSize` 确定 token,`core%groupSize` 确定组内位置。 leader 保留原始 BF16 prefix,并按 UB 余量缓存本核负责的历史候选行。 - 多 worker 组各自打分,leader 的自身分数保留在 UB,其他 worker 写独占 score 槽。 全体 AIV 执行一次就绪同步,leader 只读取远端分数,再完成 softmax、完整 H 加权和输出 RMSNorm。 - 本轮高 token 占用的完整行方案直接采用多 worker 核组,不再使用先前的收益准入门槛。 没有具体 B/N 白名单;不满足协作框架条件的其他场景仍保留原路径。 A5 的 UB 预算计入实际分配:直接从 `rowBf16_` 搬出,不分配输出队列; 核组均分配 GM 分数队列,余量用于缓存更多 BF16 候选行。 核组 leader 按 UB 余量缓存候选行,输出仍由一个 owner 完成完整 H。 当前 Host 不发射 `groupSize=1` 的40040,旧单 worker 对照分支及其专用累加 helper 已删除。 目标矩阵之外的老 kernel 仍采用单核 token 模式,相关历史优化见本文末尾。 完整行结果在 `V_MTE3` 事件后直接 `DataCopyPad` 写 GM,等待 `MTE3_V` 再复用。 A2/A3 另有 UB 地址 padding、循环外 Counter mask、BF16 多 repeat Copy 和归约尾块初始化; 这些调优不直接套用到 A5。A5 使用独立 RegBase 算术与归约实现。 ### 大 H 与其他路径 协作框架要求 T 不超过 C/2、H 至少512、R 在2到128之间,并由 Host 核对 UB 预算。 完整行放不下时保留 H 分段:先归并平方和求整行 invRms,再归并分段 score, 输出按 H 分段;启用输出 RMSNorm 时还要归并输出平方和。这需要额外同步, 不承诺一次 SyncAll。超过所有支持的分段预算时明确报错。 不满足协作框架条件的其他 shape、非混合及 prefill 路径保留对应实现。 当前 B=1/4/8/16、N=1到8、H=7168 的目标选路如下: | 平台 | B=1/4/8,N=1到8 | B=16,N=1到8 | | --- | --- | --- | | A2 | 新 kernel,N 行并行或多 worker token 核组 | G=2,共32个打分核 | | A5 | 新 kernel,N 行并行或多 worker token 核组 | N1/3为G=2,其余为G=3,共32/48个打分核 | Host 通过 `ShouldUseCooperativeKernel` 控制准入,仅接受非空且 `tokenGroupSize!=1` 的计划。 生产代码没有 B/N 白名单;上表是该通用规则在目标矩阵中的结果。 目标矩阵之外的老 kernel 仍根据 UB 容量选择 RESIDENT 或 RELOAD,不强制指定某个旧 tiling key。 此前 `T*(R-W) > 2*(P-T)` 的保守代理不再控制上述高占用完整行核组。 完整行不满足 UB 预算时仍走 H 分段。 ## 校验与测试 Torch、ACLNN 和 Tiling 检查 BF16 dtype、形状、stride、epsilon、整数溢出和 bank 写槽。 投影严格为 `[1,H]`,epsilon 须为 FP32 可表示的有限正数;写槽不得覆盖有效历史行。 空 token 同样先校验元数据与 dtype。Torch 拒绝写回 bank 与只读输入共享存储。 原生 ACLNN 调用方也须保证可写 bank 不与其他输入重叠。 - NPU:`tests/e2e/nightly/single_node/ops/singlecard_ops/test_attn_res_fwd.py` 和同目录 `test_attn_res_fwd_cooperative.py`。 - 上述 `41e163a30` 版本在 A2/A5 各通过153条 NPU 用例;含四条大 H、完整驻留尾块边界、 可选加法/归一化/materialized、带 stride 的 bank、写回和非法 FP16/FP32/INT32 拒绝。 ## 41e163a30 历史性能记录(本次选路调整前) 以下数据来自 `41e163a30`,不代表当前统一 N 轴并行版本的性能。 2026-10-30,在 Ascend910B3(40 AIV,卡6)及 Ascend950(56 AIV,卡3)验证最终源码。 最原始基线为 `ef983c314`,本轮前为 `bdcf63874`;三个版本均加 `-g`, 原始 A2 仅补一行平台注册,不改计算 kernel 与 tiling。 范围为 BF16、融合 add/输出 RMSNorm、B=1/4/8/16、N=1到8、H=7168。 四轮版本和 N 顺序交替,每组预热30次、采样100次,每 shape/版本400个采样。 每平台48份原始 `kernel_details.csv`、38,400个采样输出逐次 CPU 对照和 prefix 精确比较, 以及采集前/中/后设备进程占用记录均保留。实际 kernel 路径另外用 Host tiling 日志和 msprof BasicInfo 证明;BasicInfo 单次时长不混入性能均值。 | 平台 | B,N,H | 最原始 μs | 本轮前 bdcf μs | 最终 μs | 相对 bdcf 耗时降低 | | --- | --- | --- | --- | --- | --- | | A2 | 16,3,7168 | 14.2598 | 14.1454 | 10.8017 | 23.64% | | A2 | 1,8,7168 | 18.3325 | 8.9619 | 8.9163 | 0.51% | | A2 | 4,8,7168 | 18.9024 | 13.1860 | 13.0121 | 1.32% | | A2 | 8,8,7168 | 19.5897 | 15.2298 | 15.0765 | 1.01% | | A2 | 16,8,7168 | 21.4906 | 17.9058 | 17.8466 | 0.33% | | A5 | 16,1,7168 | 6.6922 | 6.2882 | 6.1736 | 1.82% | | A5 | 16,2,7168 | 7.7637 | 7.3115 | 7.1628 | 2.03% | | A5 | 16,4,7168 | 9.2523 | 9.3611 | 9.1622 | 2.12% | | A5 | 16,6,7168 | 11.3978 | 11.4417 | 11.1303 | 2.72% | | A5 | 1,8,7168 | 12.7652 | 7.5290 | 7.5200 | 0.12% | | A5 | 4,8,7168 | 13.1171 | 8.4787 | 8.4896 | -0.13% | | A5 | 8,8,7168 | 13.2714 | 9.4012 | 9.4248 | -0.25% | | A5 | 16,8,7168 | 13.5607 | 10.8586 | 10.7741 | 0.78% | 32组几何平均加速比为 A2 `1.011156x`、A5 `1.004305x`;相对 bdcf 分别18/32、21/32组 均值改善,最差分别慢1.67%和0.33%。本轮64个平台/shape组合的均值均低于同轮最原始基线。 A5 完整驻留的四组(B=16、N=1/2/4/6)四轮均改善,模式内几何平均加速2.224%。 A5 的整体增益低于初筛0.5%自动保留阈值;保留是为完成全场景优化框架覆盖,且受影响模式 持续受益,不声称全局显著加速。完整证据包含全部负收益和被弃用后重跑的占用窗口。 这些是 kernel 时延,不代表整网吞吐。全仓 CI、A3 真机和大 H 性能尚未覆盖。 完整矩阵保留全部负收益与逐轮波动,不以较好的单轮替代独立均值。 ## 首次单核 token 选路调整的历史验证(23c0b1631) 以下为23c0b1631首次回退规则的数据,不代表后续放宽准入后的全部场景。 当时 A2/A5 各153条NPU精度/参数用例通过,CPU五个测试方法通过。 以下为本次两轮对照,均带`-g`,H=7168、BF16、融合add/输出RMSNorm; 每shape/版本200个采样。正数为回退后耗时降低,负数表示暂时变慢。 | 平台 | B,N | 41e单核token μs | 老kernel μs | 耗时降低 | | --- | --- | --- | --- | --- | | A2 | 16,1 | 8.1260 | 10.6822 | -31.46% | | A2 | 16,2 | 9.4273 | 12.2654 | -30.11% | | A2 | 16,3 | 10.8906 | 13.7183 | -25.96% | | A5 | 16,1 | 6.1661 | 6.7894 | -10.11% | | A5 | 16,2 | 7.1636 | 7.8716 | -9.88% | | A5 | 16,4 | 9.1735 | 9.3727 | -2.17% | | A5 | 16,6 | 11.1321 | 11.5039 | -3.34% | 旧kernel算术尚未针对这些场景优化。本次按方案边界保留回退,性能数据作为后续老kernel优化基线, 不宣称回退本身有性能收益。每平台16份原始CSV、12,800个采样输出逐次CPU对照、 prefix精确比较和输入保护均通过;输入/参考、源码与安装包哈希和私有vendor映射已核对。 ## 资源驱动的保守准入验证 本轮移除固定省2行/6行的门槛,按关键路径工作与额外权重读取比较, 扩大旧kernel范围。本次157条NPU用例/平台通过;新增B12、B20及H1537、4096、8193, CPU每种架构53个候选kernel与5个旧路径场景通过,并扩展Host组合与边界断言。 A2设备6、A5设备1各做两轮配对对照:每shape/版本200采样、每平台16份CSV和12,800次输出检查。 本次H7168矩阵推导出A2 B16/N1到5、A5 B16/N1到6走旧kernel,其他保留跨核打分; 新增回退的实际key另经BasicInfo核对。其余B、H、核数和UB容量按相同规则计算。 | 新增回退场景 | 23c μs | 放宽后 μs | 耗时降低 | | --- | --- | --- | --- | | A2 B16/N4/H7168 | 14.6749 | 15.6604 | -6.72% | | A2 B16/N5/H7168 | 15.6810 | 17.1493 | -9.36% | | A5 B16/N3/H7168 | 7.9456 | 8.9305 | -12.40% | | A5 B16/N5/H7168 | 9.1089 | 10.5868 | -16.23% | 负数表示变慢;本次是按方案边界放宽旧kernel范围,旧kernel算术尚未优化。 以上数据作为后续旧kernel优化基线,不声称保守代理能预测所有shape的性能。 ## 历史旧路径优化与清理(2026-10-30) 保持上述资源驱动的选路,补齐旧路径的实际 UB 预算,融合 prefix 直接复用其专用 BF16 驻留区。A2/A3 批量执行 BF16 Copy,并在 Reload 中提前将 prefix/addend 放入已有双输入队列。 A5 Resident 的第一个候选在寄存器内从正零开始累加,保留独立 Mul/Add 和原 BF16 舍入边界。 A5 半行双路访问增加32字节地址对齐检查;不满足时走已有线性路径,修复 H1538 旧路径的访问异常。 `Process()` 分别调用 `ProcessRowParallel()` 和 `ProcessHSplit()`,区分完整 H 与 UB 受限的 H 分段。 删除未引用的多行归约、旧 meta 搬运、小维度归约和 A5 helper;保留测试使用的单 worker 对照路径, 以及标注为保留的 tiling 布局兼容字段。方法封装和删除未调用代码不改变同步与计算顺序。 以下为清理前冻结源码的独立四轮 B16/H7168 真机数据,BF16、addend 和输出 RMSNorm 开启。 原始版本为本 PR base `ef983c314`,计算和 tiling 不改;各版均加 `-g`, A2 原始版另加已批准的一行平台注册。每组每版预热30次/轮、采样100次/轮,共400次。 “本轮前”指上一轮 UB 优化快照,不是原始版。负数表示变慢。 | 平台 | N | 原始 μs | 最终 μs | 相对原始耗时降低 | 相对本轮前耗时降低 | | --- | --- | --- | --- | --- | --- | | A2 | 1 | 10.6718 | 9.3666 | 12.23% | -0.21% | | A2 | 2 | 13.1329 | 10.9669 | 16.49% | 0.29% | | A2 | 3 | 14.5668 | 13.4837 | 7.44% | 2.62% | | A2 | 4 | 16.0393 | 14.9483 | 6.80% | 2.19% | | A2 | 5 | 17.4734 | 16.3242 | 6.58% | 2.31% | | A2 | 6 | 18.9206 | 16.1635 | 14.57% | 1.25% | | A2 | 7 | 20.3483 | 17.8245 | 12.40% | 1.32% | | A2 | 8 | 21.8054 | 17.9280 | 17.78% | 1.40% | | A5 | 1 | 6.6774 | 6.5736 | 1.56% | 1.57% | | A5 | 2 | 7.7677 | 7.6489 | 1.53% | 1.37% | | A5 | 3 | 8.8377 | 8.7555 | 0.93% | 0.94% | | A5 | 4 | 9.2401 | 9.5265 | -3.10% | -2.44% | | A5 | 5 | 10.3161 | 10.5892 | -2.65% | -2.07% | | A5 | 6 | 11.3727 | 11.6910 | -2.80% | -2.17% | | A5 | 7 | 12.4672 | 10.5242 | 15.58% | -0.27% | | A5 | 8 | 13.5448 | 10.7377 | 20.72% | 0.42% | A5 N4到6相对仅对齐修复控制版基本持平;保留对齐修复是正确性决定,不声称有性能收益。 A2 N3到5最终各有一轮收益为负,不宣称每轮稳定获益。未修改的路径也有波动。 全矩阵覆盖 B1/4/8/16、N1到8;A2/A5 共112份接受的原始 CSV、89,600次采样输出检查。 两平台各172条 NPU 用例通过,包含四条大 H 精度、非对齐 H、正负零位模式和无效 dtype。 这些四轮性能数据发生在最终方法封装和未调用代码清理之前,不能作为清理后二进制的重新采集结果。 最终清理源码另在两平台重新完成原生编译、私有安装和精度回归:每平台172条协作/边界用例与 6条原路径用例通过。CPU六个测试方法通过,每种模拟架构执行53个协作候选与9个旧路径场景。 定向 Ruff、C++ 测试格式和空白检查通过;本机 `format.sh ci` 因缺少 `pre-commit` 未完成, 不声称全仓 CI 或 A3 真机验证通过。同步静态检查仍有已记录候选,不因精度通过而自动排除。 ## 当前统一 N 轴并行版本的独立复测(2026-10-30) 以下对照为本轮前 PR 提交 `c730f7177`,不是最初未修改的 PR base。 所有版本均带 `-g`,设备 ELF 的 `.debug_info` 已逐包核对;A2 卡6、A5 卡3, 每 shape/版本四轮共400次采样,保留全部负收益。B1/4/8/16、N1到8、H7168全部采用新 kernel 多核打分。 | 平台 | B | N | 本轮前 PR μs | 统一新路径 μs | 最终 μs | 相对本轮前耗时降低 | | --- | --- | --- | --- | --- | --- | --- | | A2 | 16 | 1 | 9.37876 | 10.78951 | 10.34071 | -10.26% | | A2 | 16 | 2 | 10.96546 | 12.16997 | 11.42181 | -4.16% | | A2 | 16 | 3 | 13.58345 | 12.33425 | 11.84478 | +12.80% | | A2 | 16 | 4 | 15.00211 | 13.95477 | 13.08395 | +12.79% | | A2 | 16 | 5 | 16.48741 | 14.24394 | 13.64717 | +17.23% | | A2 | 16 | 6 | 16.16281 | 15.77918 | 14.83648 | +8.21% | | A2 | 16 | 7 | 17.73577 | 16.09270 | 15.60024 | +12.04% | | A2 | 16 | 8 | 18.04828 | 17.73832 | 16.73555 | +7.27% | | A5 | 16 | 1 | 6.58669 | 6.47189 | 6.07565 | +7.76% | | A5 | 16 | 2 | 7.66126 | 7.30887 | 6.94755 | +9.32% | | A5 | 16 | 3 | 8.75248 | 7.96009 | 7.48109 | +14.53% | | A5 | 16 | 4 | 9.50099 | 8.85070 | 8.38101 | +11.79% | | A5 | 16 | 5 | 10.58661 | 9.04412 | 8.61022 | +18.67% | | A5 | 16 | 6 | 11.65197 | 10.27874 | 9.56163 | +17.94% | | A5 | 16 | 7 | 10.50956 | 10.48853 | 10.02614 | +4.60% | | A5 | 16 | 8 | 10.75578 | 10.72925 | 10.27616 | +4.46% | A2 B16/N1和N2仍分别慢10.26%和4.16%,按本轮要求保留新路径,不再继续专项调优。 A2 B16/N3到8降低7.27%到17.23%;A5 B16/N1到8降低4.46%到18.67%,均为四轮均值。 相对仅统一新路径,B16的细节优化使A2降低3.06%到6.24%,A5降低4.22%到6.98%; A2 N1有一轮细节优化收益为负,不能把均值改善描述为每轮稳定获益。 所选源码在两平台分别原生构建安装,172条协作/边界与6条原路径NPU用例通过,CPU六个方法通过。 最终三版本性能复测逐次校验每平台38,400个输出,所有元素满足混合容差;prefix精确比较和输入保护通过。 共96份最终原始CSV已下载本地,并核对源码、安装包、adapter及输入/参考哈希。 `format.sh ci`因本机未安装pre-commit未完成;定向Ruff、C++测试格式与git空白检查通过。 本次未覆盖A3真机、全仓CI或大H性能;保留既有大H精度和H分段后备实现。 - csrc/attention/chunk_kda_fwd/docs/api.md (4787 bytes)
# ChunkKdaFwd API ## Python 主入口 ```python from fla_npu.ops.ascendc import chunk_kda_fwd outputs = chunk_kda_fwd( q, k, v, g, beta, scale, chunk_size, layout="BSND", initial_state=None, output_final_state=False, cu_seqlens=None, chunk_indices=None, safe_gate=False, lower_bound=None, use_gate_in_kernel=False, A_log=None, dt_bias=None, disable_recompute=False, return_intermediate_states=False, state_v_first=False, ) ``` 返回: ```text (attn_out, final_state, gk, Aqk, Akk, w, u, qg, kg, v_new, h, initial_state) ``` 可选输出在 Python 层返回 `None`。`Aqk/Akk` 始终存在;其余保留策略见算子 README。 ## aclnn ```cpp aclnnStatus aclnnChunkKdaFwdGetWorkspaceSize( const aclTensor *q, const aclTensor *k, const aclTensor *v, const aclTensor *g, const aclTensor *beta, const aclTensor *aLogOptional, const aclTensor *dtBiasOptional, const aclTensor *initialStateOptional, const aclIntArray *cuSeqlensOptional, const aclIntArray *chunkIndicesOptional, const char *layout, double scale, int64_t chunkSize, bool safeGate, double lowerBound, bool useGateInKernel, bool stateVFirst, const aclTensor *attnOut, const aclTensor *finalStateOut, const aclTensor *gkOut, const aclTensor *aqkOut, const aclTensor *akkOut, const aclTensor *wOut, const aclTensor *uOut, const aclTensor *qgOut, const aclTensor *kgOut, const aclTensor *vNewOut, const aclTensor *hOut, uint64_t *workspaceSize, aclOpExecutor **executor); aclnnStatus aclnnChunkKdaFwd( void *workspace, uint64_t workspaceSize, aclOpExecutor *executor, aclrtStream stream); ``` aclnn L2 只描述张量与算法契约,不接收或解释 autograd 重计算策略: - `attnOut/aqkOut/akkOut` 是必选输出。 - `finalStateOut/gkOut/wOut/uOut/qgOut/kgOut/vNewOut/hOut` 均为相互独立的可选输出。 - `w/u/qg/kg/vNew/h` 的 L0 阶段固定写内部 compute 张量;对应可选输出非空时,L2 通过 `ViewCopy` 导出,为空时只保留前向内部生命周期。`gkOut` 非空时直接复用为 `gkCompute`, 避免目标场景额外复制整张 FP32 gate。 - `finalStateOut != nullptr` 同时表示本次需要计算并写出最终状态。 - `hCompute` 是 FwdH 到 Finalize 的内部必需 head-major 张量;`hOut` 是独立的公开可选输出。 `hOut == nullptr` 不会跳过内部 `hCompute`,只是不向调用方公开该中间状态;非空时由 L2 转为固定 sequence-major 后导出。 `output_final_state/disable_recompute/return_intermediate_states` 只存在于 Python 和 legacy torch 包装层,由上层按 FLA 的保留策略决定向 L2 传入哪些输出指针。 ## 输入与输出布局 `layout` 只解释 q/k/v/g/beta 输入。输出固定为: - `attnOut`: BSND 或 TND。 - `finalStateOut`: `[N,H_v,K,V]` 或 `stateVFirst=true` 时 `[N,H_v,V,K]`。 - `gkOut/AqkOut/AkkOut/wOut/uOut/qgOut/kgOut/vNewOut`: BNSD/NTD。 - `hOut`: dense 为 `[B,N_c,H_v,K,V]`,varlen 为 `[N_c,H_v,K,V]`; `stateVFirst=true` 时交换末两维。 完整 Shape 表见 [KDA 模型符号表](../../README.md#model-shape-symbols)。 ## Gate 语义 ```text useGateInKernel=false: gate = g useGateInKernel=true, safeGate=false: gate = -exp(A_log) * softplus(g + dt_bias) useGateInKernel=true, safeGate=true: gate = lowerBound * sigmoid(exp(A_log) * (g + dt_bias)) gk = chunk_local_cumsum(gate) / ln(2) ``` `safeGate` 的 true/false 都支持;`useGateInKernel=false` 时仍支持 `safeGate=true` 的后续稳定计算路径。 ## 示例 ```python import torch from fla_npu.ops.ascendc import chunk_kda_fwd B, T, H, K, V = 1, 128, 4, 128, 128 q = torch.randn(B, T, H, K, device="npu", dtype=torch.float16) k = torch.randn_like(q) v = torch.randn(B, T, H, V, device="npu", dtype=torch.float16) g = -torch.rand(B, T, H, K, device="npu", dtype=torch.float32) * 0.01 beta = torch.rand(B, T, H, device="npu", dtype=torch.float32) attn_out, final_state, *_ = chunk_kda_fwd( q, k, v, g, beta, K ** -0.5, 64, layout="BSND", output_final_state=True, safe_gate=True, ) assert attn_out.shape == (B, T, H, V) assert final_state.shape == (B, H, K, V) ``` ## 调用途径 | 路径 | 入口 | | --- | --- | | 稳定 Python | `fla_npu.ops.ascendc.chunk_kda_fwd` | | aclnn | `aclnnChunkKdaFwdGetWorkspaceSize/aclnnChunkKdaFwd` | | legacy | 显式加载后的 `torch.ops.npu.npu_chunk_kda_fwd` | | 受限直调样例 | `torch.ops.ascend_ops.chunk_kda_fwd_direct` | 直调样例仅覆盖 dense BNSD、K=128、V=128/256,并保留“调用方传入已累计 gk”的低层测试接口; 公开顶层语义以稳定 Python/aclnn 接口为准。 - csrc/attention/chunk_kda_fwd/docs/design.md (6320 bytes)
# ChunkKdaFwd 设计 ## 目标 1. 顶层接口对齐非 CP 的 FLA `chunk_kda_fwd`。 2. 不新增公开算子原型;A5 快路径复用既有 `ChunkKdaFwd` 原型和外层 kernel 入口。 3. A2/A3/A5 使用同一数学定义;A5 保留 regbase 双发射特化。 4. 输入 layout 与输出 layout 解耦。 5. FwdH 同时服务 KDA 与 GDN,并支持可选 scalar gate、key-wise gate 和 `state_v_first`。 ## L2 调度 ```text raw g -> ChunkKdaFwd[ gate cumsum -> Prepare/Post-WU -> FwdH -> Finalize ] -> attn_out ``` `aclnnChunkKdaFwd` 做公开 layout 的连续化和必要视图转换。A5 的 BF16、chunk=64、K=V=128 dense 对齐快路径保持单次物理 `ChunkKdaFwd` L0;A5 其他多 chunk 场景将同一个私有 L0 按 Gate/Prepare、Post-WU、FwdH、Finalize 四个阶段依次提交,使阶段间通过物理 launch 边界重置事件状态。 A2/A3 和单 chunk 场景仍使用单次物理 L0。阶段选择仅使用私有 `stage` 属性,不增加公开属性、 接口字段或独立算子原型。 ## 阶段职责 ### KdaGateCumsumVllm 将 raw/已激活 gate 转为 FP32 chunk-local log2 累计值: ```text gk = cumsum(gate) / ln(2) ``` 该算子同时保留独立 L2 接口供 GDN2 调用,输入和输出固定为 BNSD/NTD。 ### Prepare 只读取 `q/k/v/gk/beta` 及变长元数据,产生: ```text Aqk, Akk, qg, qg_scaled, w_seed, u_seed ``` 矩阵计算和三角求逆使用 FP32 累积;公开中间量在写回时转为 q dtype。 ### Post-WU 只读取 `k/gk/w_seed/Akk/u_seed`,产生: ```text w, u, kg, v_new_seed ``` `Akk` 的 head 循环按 `H_v` 执行,GQA 映射只在读取 q/k head 时换算,避免按 `H_k` 重复或漏算。 ### FwdH state propagation 读取 `kg/w/u/gk` 和可选 `initial_state`,计算 chunk 间递推: ```text v_new = u - w @ h_prev h_next = exp2(gk_last) * h_prev + kg^T @ v_new ``` arch35 路径复用与 `ChunkGatedDeltaRuleFwdHVllm` 相同的数学实现;其他场景在 `ChunkKdaFwd` 内嵌 共享 FwdH 实现。独立 GDN L0 原型继续保留给其他调用方,key-wise `gk` 固定使用 `exp2`。 ### Finalize 只读取 `qg_scaled/Aqk/v_new/h`,计算: ```text attn_out = qg_scaled @ h + Aqk @ v_new ``` kernel 内直接按 BSND/TND 写出 `attn_out`。供反向使用的中间量保持 BNSD/NTD。 ## 状态布局 内部递推统一使用 `[...,K,V]`。`state_v_first=true` 时,L2 在进入 FwdH 前转置 initial state。 内部 `hCompute` 始终保持 head-major 供 Finalize 消费;公开 `hOut` 在 L2 导出边界转为 sequence-major,并按 `state_v_first` 决定末两维顺序。`final_state` 按序列排列,与 FLA 顶层 输出一致。 ## 重计算策略 L2 不理解 autograd 重计算策略。`final_state/gk/w/u/qg/kg/v_new/h` 是相互独立的 `OPTIONAL_OUTPUT`;非空指针表示导出,空指针表示不公开该结果。单 launch 路径为隐藏输出传递 固定 ABI 占位,并由 tiling 在 kernel workspace 中承接实际中间结果。A5 四段 launch 路径将 阶段间依赖的 `gk/w/u/qg/kg/v_new/h/final_state` 和私有 `qg_scaled/u_seed` 物化为 executor 内部张量, 使后续 launch 不依赖前一 launch 的 kernel workspace。公开输出存在时直接作为内部目标使用。 Python/legacy 包装层对齐 fla-org `chunk_kda_fwd` 提交 `0f0f0c97af39343855b43bbbaddcedfda5cb9d77`: - `Aqk/Akk` 始终返回。 - `disable_recompute=false` 时不保留 `w/u/qg/kg/v_new`。 - `disable_recompute=true` 或 `return_intermediate_states=true` 时保留公开 `hOut`。 - `use_gate_in_kernel=false` 或 `disable_recompute=true` 时保留 `gk`。 - `final_state` 只在 `output_final_state=true` 时创建公开输出。 内部 `hCompute` 与公开 `hOut` 是两个生命周期:`hCompute` 是 FwdH 到 Finalize 的必需 head-major 阶段结果;`hOut` 为空时,单 launch 路径由 kernel workspace 承接,四段路径由 executor 内部张量承接。`hOut` 非空时,L2 提供 head-major 临时输出并在导出边界转为 sequence-major。 该规则对齐非 CP 的低层 12 返回值接口; 第 12 项 `initial_state` 由 Python 层原对象透传。 ## 模板化方案与 tiling key `ChunkKdaFwd` 只有一个外层 `op_kernel/chunk_kda_fwd.cpp` 入口和一个私有 L0 类型。A5 实现位于 `op_kernel/arch35/*.h`,host 侧 A5 模板选择位于 `op_host/arch35/*.h`。Prepare、Post-WU、 Finalize 的内部实现头与统一 kernel 入口同属 `chunk_kda_fwd/op_kernel/`,不存在对应的独立 L0 原型或 `.cpp` 入口。A5 四段路径只是用不同私有 `stage` 属性连续调用该入口。 - `tiling key=1`:非 chunk=64、K=V=128 场景的通用模板族。 - `tiling key=2`:chunk=64、K=V=128 模板族,包括 dense、tail 和 varlen。 两个 key 是同一 L0 的编译期场景变体,不是平台编号、独立算子或独立接口。A2/A3/A5 均生成两个 key;同一个 key 内再由编译架构选择根目录通用实现或 `arch35/` 实现。host 的 `SetTilingKey` 只检查 chunk、K、V,不检查 SoC。 在 arch35 上,key2 的 dense 对齐场景使用单 launch 和 arch35 FwdH;融合 score 写回在跳过 共享 PostWU 时会额外物化以块尾 gate 为参考的最终 `kg`,供 FwdH 和可选公开输出共同使用。 A5 多 chunk 的 tail/varlen 以及 key1 泛化场景使用四段 launch。其他架构在同一 key 下使用其 对应单 launch 实现。tiling key 和私有 `stage` 均不改变公开算子原型、输出契约或数学定义。 ## 性能设计 - Prepare 的右矩阵在 L1 驻留,避免 K/K^T 重复搬运和重复转置。 - AIC 使用 L1/L0 双缓冲组织 MTE2、MTE1、Cube、Fixpipe。 - AIV 使用输入 staging ping-pong,使下一 tile MTE2 与当前 tile VEC 重叠。 - A5 VEC 路径使用 regbase 双发射;数值主计算仍保持 FP32。 - inter-sub-chunk 合并使用独立 workspace 区域,避免阻塞主 tile 流水。 性能结论只使用 `msopprof`。目标回归 case 定义在 `tests/op_cases/chunk_kda_fwd.json`。 ## 验证矩阵 - 平台:A2/A3/A5。 - dtype:FP16/BF16。 - layout:BSND/BNSD/TND/NTD。 - gate:raw/已激活、safe true/false。 - Shape:K=128,V=128/256,chunk=64/128,dense/varlen/tail/GQA。 - 属性:final state、重计算策略、`state_v_first`。 - csrc/attention/chunk_kda_fwd/README.md (4744 bytes)
# ChunkKdaFwd ## 功能 `ChunkKdaFwd` 对齐不涉及 CP 切分的 FLA `chunk_kda_fwd` 顶层语义。公共接口接收 raw gate 或已激活的 自然对数 gate;Gate、Prepare、PostWu、FwdH 和 Finalize 均在一个物理 `ChunkKdaFwd` L0 内完成, L2 不再拼接或依次发射多个阶段 L0。 Shape 符号与布局约定见 [KDA 模型符号表](../README.md#model-shape-symbols)。 ## Gate 公式 令 `x = g + dt_bias`。逐 token、逐 K 维的自然对数衰减为: ```text use_gate_in_kernel = false: gate = g use_gate_in_kernel = true, safe_gate = false: gate = -exp(A_log) * softplus(x) use_gate_in_kernel = true, safe_gate = true: gate = lower_bound * sigmoid(exp(A_log) * x) ``` 随后在每个 chunk 内计算: ```text gk_i = cumsum(gate)_i / ln(2) ``` 因此后续 `exp2(gk)` 与自然指数 gate 严格绑定,不暴露额外 gate scale。 ## 输入 | 名称 | 必选性 | Shape/Dtype | 说明 | | --- | --- | --- | --- | | `q/k` | 必选 | 输入 layout 对应 Shape;FP16/BF16 | Query/Key | | `v` | 必选 | 输入 layout 对应 Shape;与 q 同 dtype | Value | | `g` | 必选 | 输入 layout 对应 K 维 Shape;FP32/BF16 | raw gate 或已激活自然对数 gate | | `beta` | 必选 | 去掉 g 的 K 维;FP32/BF16 | Delta 系数 | | `A_log` | 条件必选 | `[H_v]`,FP32 | `use_gate_in_kernel=true` 时必选 | | `dt_bias` | 可选 | `[H_v*K]`,FP32 | gate bias | | `initial_state` | 可选 | `[N,H_v,K,V]` 或 `[N,H_v,V,K]`,FP32 | 由 `state_v_first` 解释 | | `cu_seqlens` | 可选 | `[N+1]`,INT64 | 变长序列 | | `chunk_indices` | 可选 | `[2*N_c]`,INT64 | canonical chunk 顺序 | `layout` 只描述上述输入。BSND/TND 由 L2 使用 `l0op::Transpose` 转为内部 BNSD/NTD。 ## 输出 Python 返回顺序为: ```text (attn_out, final_state, gk, Aqk, Akk, w, u, qg, kg, v_new, h, initial_state) ``` - `attn_out` 固定为 BSND/TND。 - `final_state` 固定按序列排列,末两维服从 `state_v_first`。 - `Aqk/Akk` 始终返回,固定为 head-major。 - `gk/w/u/qg/kg/v_new` 是供反向使用的 head-major 中间量。 - 公开 `h` 固定为 sequence-major;内部 `hCompute` 保持 head-major 供 Finalize 使用。 - 第 12 个返回值是 Python 层对 `initial_state` 的原对象透传,不是 aclnn 输出。 输出保留策略对齐 fla-org [`chunk_kda_fwd`](https://github.com/fla-org/flash-linear-attention/blob/0f0f0c97af39343855b43bbbaddcedfda5cb9d77/fla/ops/kda/chunk_fwd.py) 提交 `0f0f0c97af39343855b43bbbaddcedfda5cb9d77`: | 条件 | 返回 | | --- | --- | | `output_final_state=true` | 返回 `final_state`,否则为 `None` | | `use_gate_in_kernel=false` 或 `disable_recompute=true` | 返回 `gk` | | 始终 | 返回 `Aqk/Akk` | | `disable_recompute=true` | 返回 `w/u/qg/kg/v_new` | | `disable_recompute=true` 或 `return_intermediate_states=true` | 返回 `h` | 这是 `fla_npu.ops.ascendc.chunk_kda_fwd` 的低层 12 返回值语义;不涉及 CP。aclnn L2 不接收 `output_final_state/disable_recompute/return_intermediate_states`,每个可选输出是否写出仅由对应 输出指针是否为空决定。`w/u/qg/kg/v_new/h` 的 L0 阶段固定写内部 compute 张量,L2 仅在 对应指针非空时通过 `ViewCopy` 导出;`gkOut` 非空时直接复用为 `gkCompute`,避免目标场景 额外复制整张 FP32 gate。内部 `hCompute` 是 FwdH 到 Finalize 的必需 head-major 阶段结果; 公开 `hOut` 非空时,L2 转为 sequence-major 后导出。`hOut` 为空时仍创建 `hCompute`,但不 作为第 11 个 Python 返回值公开。 ## 属性 | 名称 | 默认值 | 支持范围 | | --- | --- | --- | | `layout` | `BSND` | `BSND/BNSD/TND/NTD` | | `scale` | 必传 | 通常为 `K**-0.5` | | `chunk_size` | `64` | `64/128` | | `output_final_state` | `false` | bool | | `safe_gate` | `false` | bool | | `lower_bound` | `-5.0` | safe raw gate 时 `[-5,0)` | | `use_gate_in_kernel` | `false` | bool | | `disable_recompute` | `false` | bool | | `return_intermediate_states` | `false` | bool | | `state_v_first` | `false` | bool | ## 支持范围 - A2 (`ascend910b`)、A3 (`ascend910_93`)、Ascend 950PR&950DT 系列产品 (`ascend950`)。 - `K/V` 为 `[16,256]` 内 16 的倍数;交付重点覆盖 K=128、V=128/256。 - `chunk_size` 为 64/128。 - TND/NTD 均支持多 head。 - 变长调用最多 1024 条逻辑序列,rank-4 变长输入要求 B=1。 ## 验证 唯一用例规格是 `tests/op_cases/chunk_kda_fwd.json`。数值测试位于 `tests/operators/chunk_kda_fwd/accuracy/`,性能使用 `tests/operators/chunk_kda_fwd/performance/profile.py` 和 `msopprof`。 完整 API 见 [API 文档](docs/api.md),阶段和内存设计见 [设计文档](docs/design.md)。 - csrc/attention/CMakeLists.txt (1608 bytes)
# ----------------------------------------------------------------------------------------------------------- # Copyright (c) 2025 Huawei Technologies Co., Ltd. # This program is free software, you can redistribute it and/or modify it under the terms and conditions of # CANN Open Software License Agreement Version 2.0 (the "License"). # Please refer to the License for details. You may not use this file except in compliance with the License. # THIS SOFTWARE IS PROVIDED ON AN "AS IS" BASIS, WITHOUT WARRANTIES OF ANY KIND, EITHER EXPRESS OR IMPLIED, # INCLUDING BUT NOT LIMITED TO NON-INFRINGEMENT, MERCHANTABILITY, OR FITNESS FOR A PARTICULAR PURPOSE. # See LICENSE in the root of the software repository for the full text of the License. # ----------------------------------------------------------------------------------------------------------- set(OPTEST_NAME optest_${PKG_NAME}) file(GLOB CURRENT_DIRS RELATIVE ${CMAKE_CURRENT_SOURCE_DIR} ${CMAKE_CURRENT_SOURCE_DIR}/*) foreach(SUB_DIR ${CURRENT_DIRS}) if (DEFINED ASCEND_OP_NAME AND NOT "${ASCEND_OP_NAME}" STREQUAL "") if (NOT "${ASCEND_OP_NAME}" STREQUAL "all" AND NOT "${ASCEND_OP_NAME}" STREQUAL "ALL") if (NOT ${SUB_DIR} IN_LIST ASCEND_OP_NAME) continue() endif () endif () endif () if(EXISTS "${CMAKE_CURRENT_SOURCE_DIR}/${SUB_DIR}/CMakeLists.txt") add_subdirectory(${SUB_DIR}) else() if (EXISTS "${CMAKE_CURRENT_SOURCE_DIR}/${SUB_DIR}/op_host/CMakeLists.txt") add_subdirectory(${SUB_DIR}/op_host) endif() endif() endforeach() - csrc/attention/compressor/CMakeLists.txt (1197 bytes)
# ----------------------------------------------------------------------------------------------------------- # Copyright (c) 2026 Huawei Technologies Co., Ltd. # This program is free software, you can redistribute it and/or modify it under the terms and conditions of # CANN Open Software License Agreement Version 2.0 (the "License"). # Please refer to the License for details. You may not use this file except in compliance with the License. # THIS SOFTWARE IS PROVIDED ON AN "AS IS" BASIS, WITHOUT WARRANTIES OF ANY KIND, EITHER EXPRESS OR IMPLIED, # INCLUDING BUT NOT LIMITED TO NON-INFRINGEMENT, MERCHANTABILITY, OR FITNESS FOR A PARTICULAR PURPOSE. # See LICENSE in the root of the software repository for the full text of the License. # ----------------------------------------------------------------------------------------------------------- file(GLOB CURRENT_DIRS RELATIVE ${CMAKE_CURRENT_SOURCE_DIR} ${CMAKE_CURRENT_SOURCE_DIR}/*) if(NOT ENABLE_TEST AND NOT BENCHMARK) list(REMOVE_ITEM CURRENT_DIRS tests) endif() foreach(SUB_DIR ${CURRENT_DIRS}) if(EXISTS "${CMAKE_CURRENT_SOURCE_DIR}/${SUB_DIR}/CMakeLists.txt") add_subdirectory(${SUB_DIR}) endif() endforeach() - csrc/attention/compressor/README.md (29094 bytes)
# Compressor ## 产品支持情况 | 产品 | 是否支持 | | ------------------------------------------------------------ | :------: | |<term>Ascend 950PR&950DT 系列产品</term>| √ | |<term>Atlas A3 系列产品</term>| √ | |<term>Atlas A2 系列产品</term>| × | |<term>Atlas 200I/500 A2 推理产品</term>| × | |<term>Atlas 推理系列产品</term>| × | |<term>Atlas 训练系列产品</term>| × | ## 功能说明 - API功能:Compressor是推理场景下SMLA和QLI的前处理算子,用于将每4或128个token的KV cache压缩成一个,然后每个token与这些压缩的KV cache进行DSA计算。在长序列的情况下,Compressor可以有效地减少计算开销。 - 计算公式: 压缩阶段: 1. 计算矩阵乘法: - C4A: $\left[kv\_state^a, score\_state^a\right] = X @ \left[W^{aKV}, W^{aGate}\right], \left[kv\_state^b, score\_state^b\right] = X @ \left[W^{bKV}, W^{bGate}\right];$ - C128A: $\left[kv\_state, score\_state\right] = X @ \left[W^{KV}, W^{Gate}\right]$ 2. 计算分组加法: - C4A: $score\_state_i^\prime = \left[score\_state_{\left[4(i-1)+1:4i,:\right]}^a; score\_state_{\left[4i+1:4(i+1),:\right]}^b\right] + Ape,~i=1,2,\cdots, \frac{s}{4};$ - C128A: $score\_state_i^\prime = score\_state_{\left[128(i-1)+1:128i,:\right]} + Ape,~i=1,2,\cdots, \frac{s}{128};$ 3. 计算分组Softmax: - C4A: $S_i^\prime = softmax(score\_state_i^\prime),~i=1,2,\cdots, \frac{s}{4};$ - C128A: $S_i^\prime = softmax(score\_state_i^\prime),~i=1,2,\cdots, \frac{s}{128};$ 4. 计算Hadamard乘积: - C4A: $(S_H)_i = S_i^\prime \odot \left[kv\_state^a_{\left[4(i-1)+1:4i,:\right]} ; kv\_state^b_{\left[4i+1:4(i+1),:\right]}\right],~i=1,2,\cdots, \frac{s}{4};$ - C128A: $S_H = S_i^\prime \odot kv\_state;$ 5. 沿着压缩轴分组求和: - C4A: $C_{i}^{\text{Comp}} = \left[1\right]_{1\times8} @ (S_H)_i, ~i=1,2,\cdots, \frac{s}{4};$ - C128A: $C_{i}^{\text{Comp}} = \left[1\right]_{1\times128} @ (S_H)_i, ~i=1,2,\cdots, \frac{s}{128};$ 后处理阶段: 6. 计算RMSNorm: - $\text{RMS}(C^{\text{Comp}}) = \sqrt{\frac{1}{N} \sum_{i=j* N}^{(j+1)* N} {(C_{i}^{\text{Comp}})}^{\text{2}} + norm\_eps} ,N=head\_dim, ~j=1,2,\cdots, \frac{s}{cmp\_ratio}$ - $\text{RmsNorm}(C^{\text{Comp}}) = norm\_weight \cdot \frac{C_{i}^{\text{Comp}}}{\text{RMS}(C^{\text{Comp}})}$ 7. 计算Rope; - 主要计算过程为: 1. 将输入$X$与$W^{KV}$做Matmul运算得到$kv\_state$,将输入$X$与$W^{Gate}$做Matmul运算后再与$Ape$做Add运算得到$score\_state$,$kv\_state$与$score\_state$根据输入的start_pos及cu_seqlens完成更新。 2. 在coff为2的情况下对$kv\_state$和$score\_state$进行数据重排。 3. 对$score\_state$进行softmax运算将softmax结果与$kv\_state$做Mul计算,后进行ReduceSum运算。 4. 根据输入数据norm_weight、rope_sin、rope_cos,进行RMSNorm和Rope运算,得到$cmp\_kv$结果输出。 ## 参数说明 | 参数名 | 输入/输出/属性 | 描述 | 数据类型 | 数据格式 | |----------------------------|-----------|----------------------------------------------------------------------|----------------|------------| | x | 输入 | 公式中的$X$,表示原始不经压缩的数据。 | FLOAT16、BFLOAT16 | ND | | wkv | 输入 | 公式中的$W^{KV}$,表示kv压缩权重。 | FLOAT16、BFLOAT16 | ND | | wgate | 输入 | 公式中的$W^{Gate}$,表示gate压缩权重。 | FLOAT16、BFLOAT16 | ND | | state_cache | 输入 | 公式中的$\left[kv\_state, score\_state\right]$,表示kv\_state和score\_state的历史数据。 | FLOAT32 | ND | | ape | 输入 | 公式中的$Ape$,表示positional biases。 | FLOAT32 | ND | | norm\_weight | 输入 | 表示计算RmsNorm时的权重系数。 | FLOAT32 | ND | | rope\_sin | 输入 | 表示Rope计算时sin的权重系数。 | FLOAT32 | ND | | rope\_cos | 输入 | 表示Rope计算时cos的权重系数。 | FLOAT32 | ND | | rope\_head\_dim | 属性 | 表示rope_cos和rope_sin的hidden层最小单元大小,当前仅支持64。 | INT32 | - | | cmp\_ratio | 属性 | 用于稀疏计算,表示数据压缩率。 | INT32 | - | | state\_block\_table | 可选输入 | 表示state\_cache存储使用的block映射表。当其中元素的值为0时,表示当前位置无需进行更新state\_cache操作。 | INT32 | ND | | cu\_seqlens | 可选输入 | 表示不同Batch中的有效token数。 | INT32 | ND | | seqused | 可选输入 | 表示不同Batch中实际参与压缩的token数,如果指定为None时,表示和每个Batch上的Sequence Length长度相同。 | INT32 | ND | | start\_pos | 可选输入 | 表示计算起始位置。 | INT32 | ND | | coff | 可选属性 | 默认值1,支持1/2。当coff=1时,无需进行overlap数据重排。当coff=2时,需要进行overlap数据重排。 | INT32 | - | | norm\_eps | 可选属性 | 表示RmsNorm计算的权重系数。默认值1e-6。 | FLOAT32 | - | | rotary\_mode | 可选属性 | 表示Rope计算的模式。默认值1,支持1/2。rotary\_mode为1时,代表half模式。rotary\_mode为2时,代表interleave模式。 | INT32 | - | | cache\_mode | 可选属性 | 表示state_cache的存储模式,1表示连续buffer,2表示循环buffer。默认值1。**目前A3暂不支持输入2,且不支持0轴非连续**。 | INT32 | - | | cmp\_kv | 输出 | 表示压缩后的数据。 | FLOAT16、BFLOAT16 | ND | ## 约束说明 - x参数维度含义:B(Batch Size)表示输入样本批量大小、S(Sequence Length)表示输入样本序列长度、H(Head Size)表示hidden层的大小、D(Head Dim)表示hidden层的最小单元大小、T表示所有Batch输入样本序列长度的累加和。 - 输入shape限制: - wkv支持输入shape[coff* D,H] - wgate支持输入shape[coff* D,H] - state\_cache支持输入shape[block_num,block_size,2*coff* D],要求block_num>0,cache_mode=2时,需要满足block_size >= coff * cmp_ratio + S - 1。 - ape支持输入shape[cmp_ratio,coff* D] - norm\_weight支持输入shape[D,] - start\_pos支持输入shape[B,] - 若x的维度采用BS合轴,即x的输入shape为[T,H] - rope_sin、rope_cos要求输入shape为[min(T,T//cmp_ratio+B),rope_head_dim]。 - cu\_seqlens输入shape必须为[B+1,]。该参数中每个元素的值表示当前batch与之前所有batch的token数总和,即前缀和,因此后一个元素的值必须大于等于前一个元素的值,且第一位必须位0。 - seqused,支持输入shape[B,],要求每个Batch的有效token数要求小于等于对应Sequence Length长度,即seqused[n] <= cu\_seqlens[n+1] - cu\_seqlens[n],且不小于0。 - cache_mode=1时,state\_block\_table支持输入shape[B,ceil(Smax/block_size)]。Smax为每个Batch中最大的Sequence Length,即Smax=max(start\_pos)+max(cu\_seqlens[n+1] - cu\_seqlens[n])。cache_mode=2时,state\_block\_table支持输入shape[B]。 - cmp\_kv,输出shape为[min(T,T//cmp_ratio+B),D]:<batch0>compressed_tokens + <batch1>compressed_tokens + ... + <batchN>compressed_tokens + pad。 - 若x的维度不采用BS合轴,即x的输入shape为[B,S,H] - rope_sin、rope_cos要求输入shape为[B,ceil(S/cmp_ratio),rope_head_dim]。 - cu\_seqlens,参数必须为空。 - seqused,支持输入shape[B,],要求每个Batch的有效token数要求小于等于对应Sequence Length长度,即要求seqused[n] <= S,且不小于0。 - cache_mode=1时,state\_block\_table支持输入shape[B,ceil(Smax/block_size)]。Smax为每个Batch中最大的Sequence Length,即Smax=max(start\_pos)+S。cache_mode=2时,state\_block\_table支持输入shape[B]。 - cmp\_kv,输出shape为[B,ceil(S/cmp_ratio),D]:(<batch0>compressed_tokens+pad0) + (<batch1>compressed_tokens+pad1) + ... + (<batchN>compressed_tokens+padN)。 - 输入值域限制: - 该接口支持B、S泛化,且存在如下场景限制: - 部分长序列场景下,如果计算量过大可能会导致出现超过NPU内存的报错,注:这里计算量会受x输入shape的影响,值越大计算量越大。典型的长序列(即B、S的乘积或T较大)场景包括但不限于: <div style="overflow-x: auto;"> <table style="undefined;table-layout: fixed; width: 400px"><colgroup> <col style="width: 100px"> <col style="width: 100px"> </colgroup><thead> <tr> <th>B</th> <th>S</th> <th>H</th> </tr></thead> <tbody> <tr> <td>100</td> <td>65525</td> <td>4096</td> </tr> <tr> <td>25</td> <td>261120</td> <td>4096</td> </tr> <tr> <td>100</td> <td>131072</td> <td>4096</td> </tr> <tr> <td>100</td> <td>261120</td> <td>4096</td> </tr> </tbody> </table> </div> - 该接口支持B、S、T取0,即shape与B、S、T值相关的入参允许传入空tensor,其余入参不支持传入空tensor。该场景下state_cache不做更新,输出cmp_kv为空tensor。 - 输入属性限制: - 支持D为128/512。 - 支持H为1k~10k,512对齐。 - 支持block_size为1~1024。 - 支持cmp_ratio为2/4/8/16/32/64/128。支持如下三种典型组合场景: - C4A: D=512, coff=2, cmp_ratio=4; - C4Li: D=128, coff=2, cmp_ratio=4; - C128A: D=512, coff=1, cmp_ratio=128。 - 支持rotary_mode为2,Rope计算模式为interleave。 - 该接口支持aclgraph模式。 ## Atlas A3 系列产品 调用说明 - 单算子模式调用 ```python import torch import torch_npu import numpy as np import custom_ops import torch.nn as nn import math def get_seq_used_by_batch(batch_idx, S, seqused, cu_seqlens): if seqused is not None: return seqused[batch_idx] else: if cu_seqlens is not None: return cu_seqlens[batch_idx + 1] - cu_seqlens[batch_idx] else: return S data_type = torch.bfloat16 hidden_size = 4096 rope_head_dim = 64 norm_eps = 1e-6 coff = 1 # 1:no overlap 2:overlap cmp_ratio = 128 rotary_mode = 2 cache_mode = 1 head_dim = 512 cu_seqlens = [0, 1] # ------------- B = 1 S = 1 S_max = 0 block_size = 128 start_pos = [8191] * B # (B,) start_p=8191 seqused = None # (B,), None时cu_seqlens的数据全部参与计算,否则按传参实际值计算 # BS是否合轴 bs_combine_flag = True update_flag = 1 save_state_seqlens = None if seqused is not None: seqused = torch.tensor(seqused).to(torch.int32) if start_pos is not None: start_pos = torch.tensor(start_pos).to(torch.int32) else: start_pos = torch.full((B,), start_p, dtype=torch.int32) if bs_combine_flag: if cu_seqlens is None: T = B * S if T !=0: cu_seqlens = torch.arange(0, T + 1, S, dtype=torch.int32) else: cu_seqlens = torch.zeros((B+1), dtype=torch.int32) else: cu_seqlens = torch.tensor(cu_seqlens).to(torch.int32) for i in range(B): if start_pos[i] + cu_seqlens[i + 1] - cu_seqlens[i] > S_max: S_max = start_pos[i] + cu_seqlens[i + 1] - cu_seqlens[i] else: cu_seqlens = None S_max = max(start_pos) + S ### ======================== gen input data start ============================= # page state if cache_mode == 1: max_block_num_per_batch = (S_max + block_size - 1) // block_size block_num = B * max_block_num_per_batch next_block_id = 1 print(f"max_block_num_per_batch: {max_block_num_per_batch}") block_table = torch.zeros(size=(B, max_block_num_per_batch), dtype=torch.int32) for i in range(B): # 需要读取state的范围 cur_start = start_pos[i] // cmp_ratio * cmp_ratio - cmp_ratio cur_end = start_pos[i] // cmp_ratio * cmp_ratio + cmp_ratio if start_pos[i] % cmp_ratio == 0: cur_end = start_pos[i] cur_end = min(cur_end, start_pos[i] + S) cur_start_block_id = (cur_start // block_size) if cur_start >= 0 else 0 cur_end_block_id = (cur_end - 1) // block_size for j in range(cur_start_block_id, cur_end_block_id + 1): block_table[i][j] = next_block_id next_block_id = next_block_id + 1 # 需要写入state的范围 end_pos = get_seq_used_by_batch(i, S, seqused, cu_seqlens) if save_state_seqlens is not None: next_start = start_pos[i] + end_pos - save_state_seqlens[i] next_end = start_pos[i] + end_pos else: next_start = (start_pos[i] + end_pos) // cmp_ratio * cmp_ratio - cmp_ratio next_end = (start_pos[i] + end_pos) // cmp_ratio * cmp_ratio + cmp_ratio if (start_pos[i] + end_pos) % cmp_ratio == 0: next_end = start_pos[i] + end_pos next_end = min(next_end, start_pos[i] + end_pos) next_start_block_id = (next_start // block_size) if next_start >= 0 else 0 next_end_block_id = (next_end - 1) // block_size for j in range(next_start_block_id, next_end_block_id + 1): if block_table[i][j] == 0: block_table[i][j] = next_block_id next_block_id = next_block_id + 1 if B==0: kv_state = torch.tensor(np.random.uniform(-10, 10, (0, block_size, coff * head_dim))).to(torch.float32) score_state = torch.tensor(np.random.uniform(-10, 10, (0, block_size, coff * head_dim))).to(torch.float32) else: kv_state = torch.tensor(np.random.uniform(-10, 10, (torch.max(block_table) + 1, block_size, coff * head_dim))).to(torch.float32) score_state = torch.tensor(np.random.uniform(-10, 10, (torch.max(block_table) + 1, block_size, coff * head_dim))).to(torch.float32) else: block_table = torch.tensor(random.sample(list(range(B)), B), dtype=torch.int32) token_size = (2 * cmp_ratio + S - 1) if coff == 2 else (cmp_ratio + S - 1) if B==0: kv_state = torch.tensor(np.random.uniform(kv_state_datarange[0], kv_state_datarange[1], (0, token_size, coff * head_dim))).to(torch.float32) score_state = torch.tensor(np.random.uniform(score_state_datarange[0], score_state_datarange[1], (0, token_size, coff * head_dim))).to(torch.float32) else: kv_state = torch.tensor(np.random.uniform(kv_state_datarange[0], kv_state_datarange[1], (B, token_size, coff * head_dim))).to(torch.float32) score_state = torch.tensor(np.random.uniform(score_state_datarange[0], score_state_datarange[1], (B, token_size, coff * head_dim))).to(torch.float32) # other input if bs_combine_flag: x_shape = (cu_seqlens[-1], hidden_size) rope_sin_shape = (min(x_shape[0], x_shape[0] // cmp_ratio + B), rope_head_dim) rope_cos_shape = rope_sin_shape else: x_shape = (B, S, hidden_size) rope_sin_shape = (B, (S + cmp_ratio - 1) // cmp_ratio, rope_head_dim) rope_cos_shape = rope_sin_shape x = torch.tensor(np.random.uniform(-10.0, 10.0, x_shape)).to(data_type).npu() wkv = torch.tensor(np.random.uniform(-10, 10, (coff * head_dim, hidden_size))).to(data_type).npu() wgate = torch.tensor(np.random.uniform(-10, 10, (coff * head_dim, hidden_size))).to(data_type).npu() ape = torch.tensor(np.random.uniform(-10, 10, (cmp_ratio, coff * head_dim))).to(torch.float32).npu() norm_weight = torch.tensor(np.random.uniform(-10, 10, (head_dim))).to(torch.float32).npu() rope_sin = torch.tensor(np.random.uniform(-1, 1, rope_sin_shape)).to(torch.float32).npu() rope_cos = torch.tensor(np.random.uniform(-1, 1, rope_cos_shape)).to(torch.float32).npu() if cache_mode == 1: # 连续buffer state_cache = torch.zeros((kv_state.shape[0], kv_state.shape[1], 2*kv_state.shape[2])) state_cache = state_cache.npu() state_cache[:, :, :state_cache.shape[2]//2] = kv_state.clone() state_cache[:, :, state_cache.shape[2]//2:] = score_state.clone() else: layer_pad = random.randint(1, 50) layer_start_idx = random.randint(0, layer_pad-1) print(f"layer_pad: {layer_pad}") print(f"layer_start_idx: {layer_start_idx}") state_cache_pad = torch.zeros((kv_state.shape[0],kv_state.shape[1]*kv_state.shape[2]*2+layer_pad)) print(f"state_cache_pad: shape {state_cache_pad.shape}") state_cache_pad = state_cache_pad.to("npu:%s" % DEVICE_ID) state_cache = state_cache_pad[:, layer_start_idx : layer_start_idx + kv_state.shape[1]*kv_state.shape[2]*2].view(-1, kv_state.shape[1], kv_state.shape[2]*2) state_cache = state_cache.to("npu:%s" % DEVICE_ID) state_cache[:, :, :state_cache.shape[2]//2] = kv_state.clone() state_cache[:, :, state_cache. - csrc/CMakeLists.txt (21830 bytes)
# ----------------------------------------------------------------------------------------------------------- # Copyright (c) 2025 Huawei Technologies Co., Ltd. # This program is free software, you can redistribute it and/or modify it under the terms and conditions of # CANN Open Software License Agreement Version 2.0 (the "License"). # Please refer to the License for details. You may not use this file except in compliance with the License. # THIS SOFTWARE IS PROVIDED ON AN "AS IS" BASIS, WITHOUT WARRANTIES OF ANY KIND, EITHER EXPRESS OR IMPLIED, # INCLUDING BUT NOT LIMITED TO NON-INFRINGEMENT, MERCHANTABILITY, OR FITNESS FOR A PARTICULAR PURPOSE. # See LICENSE in the root of the software repository for the full text of the License. # ----------------------------------------------------------------------------------------------------------- cmake_minimum_required(VERSION 3.16) project(cann_ops-transformer) option(BUILD_OPEN_PROJECT "Build open ascend ops project." ON) option(BUILD_OPS_RTY_KERNEL "Build return yellow kernel." OFF) option(ENABLE_CCACHE "Enable ccache capability" ON) option(ENABLE_BUILT_IN "Enable built-in package" OFF) option(ENABLE_STATIC "Enable Static" OFF) option(ENABLE_EXPERIMENTAL "Enable experimental module" OFF) option(ENABLE_TEST "Enable test" OFF) option(ENABLE_UT_EXEC "Enable exec ut" OFF) option(ENABLE_ASAN "Enable asan" OFF) option(ENABLE_VALGRIND "Enable valgrind" OFF) option(OP_HOST_UT "Enable ophost ut" OFF) option(OP_API_UT "Enable opapi ut" OFF) option(OP_GRAPH_UT "Enable graph ut" OFF) option(OP_KERNEL_UT "Enable kernel ut" OFF) option(OP_KERNEL_AICPU_UT "Enable aicpu kernel ut" OFF) option(UT_TEST_ALL "Enable all ut" OFF) option(ENABLE_OOM "Enable kernel oom" OFF) set(ASCEND_COMPUTE_UNIT "ascend910b" CACHE STRING "soc that need to be compiled") set(ASCEND_OP_NAME "ALL" CACHE STRING "operators that need to be compiled") set(ARCH_DIRECTORY "" CACHE STRING "arch directory that need to be compiled") set(VENDOR_NAME "custom" CACHE STRING "vendor name") set(ASCEND_ALL_COMPUTE_UNIT "ascend310p;ascend910b;ascend910_93;ascend950;kirinx90" CACHE STRING "all soc list") set(SOC_VERSION_LIST ascend310p ascend910b ascend910_93 ascend950 kirinx90) set(ARCH_DIRECTORY_LIST arch22 arch32 arch32 arch35 arch32) if ("ascend950" IN_LIST ASCEND_COMPUTE_UNIT) message(STATUS "build with 3~8 packages........") set(BUILD_WITH_3_8_PACKAGE ON CACHE BOOL "build with 3~8 package and opsbase") endif() foreach(SOC_VERSION ${ASCEND_COMPUTE_UNIT}) list(FIND SOC_VERSION_LIST ${SOC_VERSION} INDEX) if(NOT INDEX EQUAL -1) list(GET ARCH_DIRECTORY_LIST ${INDEX} VAL) list(APPEND ARCH_DIRECTORY ${VAL}) else() message(STATUS "unsupported chip type") if ((NOT BUILD_OPS_RTY_KERNEL) AND (BUILD_OPEN_PROJECT)) include(cmake/build_empty_package.cmake) cpack_empty_package() return() endif() endif() endforeach() list(FIND ARCH_DIRECTORY "arch32" INDEX) if(NOT INDEX EQUAL -1) list(APPEND ARCH_DIRECTORY "arch22") endif() if(PROJECT_SOURCE_DIR STREQUAL CMAKE_SOURCE_DIR) message(STATUS "compile project with library") option(BUILD_WITH_INSTALLED_DEPENDENCY_CANN_PKG "Build ops-transformer with cann pkg" ON) else() message(STATUS "compile project with src") option(BUILD_WITH_INSTALLED_DEPENDENCY_CANN_PKG "Build ops-transformer with cann source" OFF) endif() if(UNIX) set(SYSTEM_PREFIX ${CMAKE_SYSTEM_PROCESSOR}-linux) endif() #外部传参 if(NOT ${CMAKE_BUILD_MODE} STREQUAL "FALSE") if(ENABLE_DEBUG) set(CMAKE_BUILD_MODE "${CMAKE_BUILD_MODE} -g") endif() set(COMPILE_OP_MODE ${CMAKE_BUILD_MODE}) else() if(ENABLE_TEST) set(COMPILE_OP_MODE "-O0 -g") endif() if(ENABLE_DEBUG) set(CMAKE_BUILD_MODE "-g") endif() endif() set(PKG_NAME transformer) set(OPS_TRANSFORMER_DIR ${CMAKE_CURRENT_SOURCE_DIR}) set(CMAKE_CXX_STANDARD 17 CACHE STRING "c++17 is needed for this project") set_directory_properties(PROPERTIES ADDITIONAL_MAKE_CLEAN_FILES "${CMAKE_BINARY_DIR}/_CPack_Packages" ) # Suppress warnings from catlass/tla third-party headers for CANN kernel compilation set(VLLM_ASCEND_CANN_COMPAT_HEADER "${OPS_TRANSFORMER_DIR}/common/include/cann_compat.h") list(APPEND OPS_COMPILE_OPTIONS -Wno-ignored-attributes) list(APPEND OPS_COMPILE_OPTIONS -includecann_compat.h) add_compile_options( $<$<COMPILE_LANGUAGE:CXX>:-include${VLLM_ASCEND_CANN_COMPAT_HEADER}> ) include(cmake/config.cmake) include(cmake/func.cmake) include(cmake/third_party/json.cmake) if (ENABLE_TEST) include(${PROJECT_SOURCE_DIR}/cmake/third_party/gtest.cmake) endif() include(${OPS_ADV_CMAKE_DIR}/ut.cmake) if (BUILD_OPEN_PROJECT) include(cmake/intf.cmake) add_definitions(-DBUILD_OPEN_PROJECT) if (BUILD_OPS_RTY_KERNEL) message(STATUS "Build return yellow kernel.") include(cmake/rty_obj_func.cmake) else() message(STATUS "Start building custom package.") include(ExternalProject) include(cmake/dependencies.cmake) include(cmake/variables.cmake) include(cmake/obj_func.cmake) include(cmake/third_party/abseil-cpp.cmake) include(cmake/third_party/ascend_protobuf.cmake) include(cmake/third_party/makeself-fetch.cmake) include(cmake/opbuild.cmake) include(cmake/custom_build.cmake) message(STATUS "End building custom package.") if (ENABLE_OPS_HOST) gen_aclnn_with_opdef() endif() if (ENABLE_STATIC) include(cmake/static.cmake) endif() if (ENABLE_AICPU OR AICPU_CUST_OBJ_TARGETS) include(cmake/symbol.cmake) gen_cust_aicpu_json_symbol() gen_cust_aicpu_kernel_symbol() endif() if (ENABLE_BUILT_IN) message(STATUS "Start building built-in package.") include(cmake/symbol.cmake) gen_norm_symbol() include(cmake/package.cmake) pack_built_in() else() include(cmake/package.cmake) pack_tiling_sink() endif() return() endif() else() include(cmake/dependencies.cmake) include(cmake/variables.cmake) include(cmake/opbuild.cmake) include(cmake/rty_obj_func.cmake) include(cmake/intf_pub_linux.cmake) endif() if (BUILD_OPS_RTY_KERNEL) set(CMAKE_MODULE_PATH ${CMAKE_MODULE_PATH} ${CMAKE_CURRENT_LIST_DIR}/cmake/modules ) set(CMAKE_PREFIX_PATH ${CMAKE_PREFIX_PATH} ${ASCEND_CANN_PACKAGE_PATH} ) set(_op_host_aclnn_link $<BUILD_INTERFACE:intf_pub> exe_graph register c_sec ) find_package(alog MODULE) if(NOT ${alog_FOUND}) add_definitions(-DALOG_NOT_FOUND) endif() add_library(op_host_aclnn SHARED EXCLUDE_FROM_ALL) target_link_libraries(op_host_aclnn PRIVATE ${_op_host_aclnn_link} ) target_compile_options(op_host_aclnn PRIVATE $<$<COMPILE_LANGUAGE:CXX>:-std=gnu++1z> ) add_library(op_host_aclnnInner SHARED EXCLUDE_FROM_ALL) target_link_libraries(op_host_aclnnInner PRIVATE ${_op_host_aclnn_link} ) target_compile_options(op_host_aclnnInner PRIVATE $<$<COMPILE_LANGUAGE:CXX>:-std=gnu++1z> ) add_library(op_host_aclnnExc SHARED EXCLUDE_FROM_ALL) target_link_libraries(op_host_aclnnExc PRIVATE ${_op_host_aclnn_link} ) target_compile_options(op_host_aclnnExc PRIVATE $<$<COMPILE_LANGUAGE:CXX>:-std=gnu++1z> ) # op proto add_library(opsproto SHARED) target_compile_options(opsproto PRIVATE $<$<COMPILE_LANGUAGE:CXX>:-std=c++11> -fvisibility=hidden ) target_compile_definitions(opsproto PRIVATE LOG_CPP PROCESS_LOG ) target_link_libraries(opsproto PRIVATE $<BUILD_INTERFACE:intf_pub> $<BUILD_INTERFACE:ops_transformer_utils_proto_headers> $<$<BOOL:${alog_FOUND}>:$<BUILD_INTERFACE:alog_headers>> -Wl,--whole-archive rt2_registry -Wl,--no-whole-archive -Wl,--no-as-needed exe_graph graph graph_base register ascendalog error_manager platform -Wl,--as-needed c_sec ) set_target_properties(opsproto PROPERTIES OUTPUT_NAME cust_opsproto_rt2.0 ) install(TARGETS opsproto LIBRARY DESTINATION packages/vendors/${VENDOR_NAME}_transformer/op_proto/lib/linux/${CMAKE_SYSTEM_PROCESSOR} ) add_ops_tiling_keys( OP_NAME "ALL" TILING_KEYS ${TILING_KEY} ) add_opc_config( OP_NAME "ALL" CONFIG ${OP_DEBUG_CONFIG} ) if(ADD_OPS_COMPILE_OPTION_V2) add_ops_compile_options( OP_NAME "ALL" OPTIONS ${OPS_COMPILE_OPTIONS} ) endif() endif () add_subdirectory(common) if (NOT BUILD_OPS_RTY_KERNEL) add_subdirectory(mc2) add_subdirectory(posembedding) endif() if(NOT DEFINED COMPILED_OPS) set(COMPILED_OPS ${COMPILED_OPS} CACHE STRING "Comp") set(COMPILED_OPS CACHE STRING "Compiled Ops" FORCE) set(COMPILED_OP_DIRS CACHE STRING "Compiled Ops Dirs" FORCE) endif() if(NOT DEFINED COMPILED_OP_DIRS) set(COMPILED_OP_DIRS CACHE STRING "Compiled Ops Dirs" FORCE) endif() set(OP_LIST) set(OP_DIR_LIST) op_add_subdirectory(OP_LIST OP_DIR_LIST) foreach (OP_DIR ${OP_DIR_LIST}) if (EXISTS "${OP_DIR}/op_host") add_subdirectory(${OP_DIR}/op_host) else() add_subdirectory(${OP_DIR}) endif() endforeach () add_subdirectory(moe) list(APPEND OP_LIST "moe_init_routing_v2") list(APPEND OP_LIST "moe_token_unpermute_with_ep_grad") list(APPEND OP_DIR_LIST ${CMAKE_CURRENT_SOURCE_DIR}/moe/moe_init_routing_v2) list(APPEND OP_DIR_LIST ${CMAKE_CURRENT_SOURCE_DIR}/moe/moe_token_unpermute_with_ep_grad) add_subdirectory(ffn) list(APPEND OP_LIST "ffn") list(APPEND OP_DIR_LIST ${CMAKE_CURRENT_SOURCE_DIR}/ffn/ffn) add_subdirectory(attention) list(APPEND OP_LIST ${COMPILED_OPS}) list(REMOVE_DUPLICATES OP_LIST) list(APPEND OP_DIR_LIST ${COMPILED_OP_DIRS}) list(REMOVE_DUPLICATES OP_DIR_LIST) add_subdirectory(gmm) list(REMOVE_DUPLICATES OP_LIST) list(APPEND OP_DIR_LIST ${COMPILED_OP_DIRS}) list(REMOVE_DUPLICATES OP_DIR_LIST) add_subdirectory(mc2) list(REMOVE_DUPLICATES OP_LIST) list(APPEND OP_DIR_LIST ${COMPILED_OP_DIRS}) list(REMOVE_DUPLICATES OP_DIR_LIST) set(OP_DEPEND_DIR_LIST) op_add_depend_directory( OP_LIST ${OP_LIST} OP_DIR_LIST OP_DEPEND_DIR_LIST ) foreach (OP_DEPEND_DIR ${OP_DEPEND_DIR_LIST}) if (EXISTS "${OP_DEPEND_DIR}/op_host") add_subdirectory(${OP_DEPEND_DIR}/op_host) else() add_subdirectory(${OP_DEPEND_DIR}) endif() endforeach () install(DIRECTORY ${OPS_ADV_ACT}/ DESTINATION ${IMPL_INSTALL_DIR}/ascendc/common/act ) install(DIRECTORY ${OPS_GROUPEDMATMUL_ACT}/ DESTINATION ${IMPL_INSTALL_DIR}/ascendc/common/groupedmatmul_act ) if (BUILD_OPS_RTY_KERNEL) get_target_property(base_aclnn_srcs op_host_aclnn SOURCES) get_target_property(base_aclnn_inner_srcs op_host_aclnnInner SOURCES) get_target_property(base_aclnn_exclude_srcs op_host_aclnnExc SOURCES) set(base_aclnn_binary_dir ${ASCEND_AUTOGEN_DIR}) set(generate_aclnn_srcs) set(generate_aclnn_inner_srcs) set(generate_aclnn_headers) set(generate_proto_dir ${base_aclnn_binary_dir}) set(generate_exclude_proto_srcs) set(generate_proto_srcs) set(generate_proto_headers) if (base_aclnn_srcs) foreach (_src ${base_aclnn_srcs}) string(REGEX MATCH "^${CMAKE_CURRENT_SOURCE_DIR}" is_match "${_src}") if (is_match) get_filename_component(name_without_ext ${_src} NAME_WE) string(REGEX REPLACE "_def$" "" _op_name ${name_without_ext}) list(APPEND generate_aclnn_srcs ${base_aclnn_binary_dir}/aclnn_${_op_name}.cpp) list(APPEND generate_aclnn_headers ${base_aclnn_binary_dir}/aclnn_${_op_name}.h) list(APPEND generate_proto_srcs ${generate_proto_dir}/${_op_name}_proto.cpp) list(APPEND generate_proto_headers ${generate_proto_dir}/${_op_name}_proto.h) endif () endforeach () else () add_custom_command(OUTPUT ${CMAKE_CURRENT_BINARY_DIR}/op_host_aclnn_stub.cpp COMMAND touch ${CMAKE_CURRENT_BINARY_DIR}/op_host_aclnn_stub.cpp ) target_sources(op_host_aclnn PRIVATE ${CMAKE_CURRENT_BINARY_DIR}/op_host_aclnn_stub.cpp ) endif () if (base_aclnn_inner_srcs) foreach (_src ${base_aclnn_inner_srcs}) string(REGEX MATCH "^${CMAKE_CURRENT_SOURCE_DIR}" is_match "${_src}") if (is_match) get_filename_component(name_without_ext ${_src} NAME_WE) string(REGEX REPLACE "_def$" "" _op_name ${name_without_ext}) list(APPEND generate_aclnn_inner_srcs ${base_aclnn_binary_dir}/inner/aclnnInner_${_op_name}.cpp) list(APPEND generate_proto_srcs ${generate_proto_dir}/inner/${_op_name}_proto.cpp) list(APPEND generate_proto_headers ${generate_proto_dir}/inner/${_op_name}_proto.h) endif () endforeach () else () add_custom_command(OUTPUT ${CMAKE_CURRENT_BINARY_DIR}/op_host_aclnn_inner_stub.cpp COMMAND touch ${CMAKE_CURRENT_BINARY_DIR}/op_host_aclnn_inner_stub.cpp ) target_sources(op_host_aclnnInner PRIVATE ${CMAKE_CURRENT_BINARY_DIR}/op_host_aclnn_inner_stub.cpp ) endif () if (base_aclnn_exclude_srcs) foreach (_src ${base_aclnn_exclude_srcs}) string(REGEX MATCH "^${CMAKE_CURRENT_SOURCE_DIR}" is_match "${_src}") if (is_match) get_filename_component(name_without_ext ${_src} NAME_WE) string(REGEX REPLACE "_def$" "" _op_name ${name_without_ext}) list(APPEND generate_exclude_proto_srcs ${generate_proto_dir}/exc/${_op_name}_proto.cpp) list(APPEND generate_proto_srcs ${generate_proto_dir}/exc/${_op_name}_proto.cpp) list(APPEND generate_proto_headers ${generate_proto_dir}/exc/${_op_name}_proto. - Dockerfile (4906 bytes)
# # Copyright (c) 2025 Huawei Technologies Co., Ltd. All Rights Reserved. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # This file is a part of the vllm-ascend project. # FROM swr.cn-north-12.myhuaweicloud.com/atlas_inference/cann:9.3.0_20261113161300-910b-ubuntu22.04-py3.12 WORKDIR /workspace # Install clang-15 (for triton-ascend) RUN apt-get update -y && \ apt-get install -y git vim wget curl protobuf-compiler libprotobuf-dev net-tools gcc g++ cmake numactl libnuma-dev libibverbs-dev libhiredis-dev clang-15 && \ update-alternatives --install /usr/bin/clang clang /usr/bin/clang-15 20 && \ update-alternatives --install /usr/bin/clang++ clang++ /usr/bin/clang++-15 20 && \ rm -rf /var/cache/apt/* && \ rm -rf /var/lib/apt/lists/* # Install modelscope (for fast download) and ray (for multinode) RUN python3 -m pip install 'modelscope<1.38' 'ray>=2.47.1,<=2.48.0' 'protobuf>3.20.0' && \ python3 -m pip cache purge # Install vLLM ARG VLLM_REPO=https://github.com/vllm-project/vllm.git ARG VLLM_TAG=v0.33.0 ARG VLLM_COMMIT="" ARG PYTORCH_INDEX_URL="https://download.pytorch.org/whl/cpu/" RUN if [ -n "$VLLM_COMMIT" ]; then \ git init /vllm-workspace/vllm && \ git -C /vllm-workspace/vllm fetch --depth 1 $VLLM_REPO "$VLLM_COMMIT" && \ git -C /vllm-workspace/vllm checkout FETCH_HEAD; \ else \ git clone --depth 1 -b $VLLM_TAG $VLLM_REPO /vllm-workspace/vllm; \ fi && \ VLLM_TARGET_DEVICE="empty" python3 -m pip install -e /vllm-workspace/vllm/[audio] --extra-index-url ${PYTORCH_INDEX_URL} && \ # In x86, triton will be installed by vllm. But in Ascend, triton doesn't work correctly. we need to uninstall it. python3 -m pip uninstall -y triton && \ # The same step also builds the optional Rust frontend (vllm-rs). cd /vllm-workspace/vllm && \ export PATH="/root/.cargo/bin:$PATH" && \ export PROTOC_INCLUDE=/usr/include && \ python3 -m pip install -r requirements/build/rust.txt && \ bash tools/build_rust.sh && \ python3 -m pip cache purge # Install vllm-ascend # Dependency versions used by the vllm-ascend build. ARG SOC_VERSION="ascend910b1" ENV DEBIAN_FRONTEND=noninteractive # Cap opc/c++/rust build parallelism to bound memory on shared buildkitd nodes: # the heavy vllm-ascend compile otherwise spawns unbounded jobs and can exhaust # a 128Gi node (OOM-killing kubelet). CI passes --build-arg MAX_JOBS=32 / # CARGO_BUILD_JOBS=32; leaving them unset keeps the default nproc parallelism. ARG MAX_JOBS ARG CARGO_BUILD_JOBS ENV SOC_VERSION=$SOC_VERSION \ TASK_QUEUE_ENABLE=1 \ OMP_NUM_THREADS=1 COPY . /vllm-workspace/vllm-ascend/ ARG ASCEND_INDEX_URL="https://mirrors.huaweicloud.com/ascend/repos/pypi \ https://mirrors.huaweicloud.com/ascend/repos/pypi/nightly" ARG COMPILE_CUSTOM_KERNELS=1 RUN export PIP_EXTRA_INDEX_URL="${ASCEND_INDEX_URL}" && \ export VLLM_BATCH_INVARIANT=1 && \ source /usr/local/Ascend/ascend-toolkit/set_env.sh && \ source /usr/local/Ascend/nnal/atb/set_env.sh && \ python3 -m pip install -e /vllm-workspace/vllm-ascend/ --extra-index-url ${PYTORCH_INDEX_URL} && \ python3 -m pip cache purge # ===== Install FLA and flash_attn_npu ===== ARG FLA_VERSION="26.10.0+feature" ARG FLASH_ATTN_NPU_VERSION="0.5.0" ARG MOONCAKE_VERSION="0.3.14" RUN source /usr/local/Ascend/ascend-toolkit/set_env.sh && \ source /usr/local/Ascend/nnal/atb/set_env.sh && \ source /usr/local/Ascend/cann/set_env.sh && \ export PIP_EXTRA_INDEX_URL="${ASCEND_INDEX_URL}" && \ # Install memfabric_hybrid kernels TORCH_DEVICE_BACKEND_AUTOLOAD=0 mfcli kernel install --soc-version A2 --install-for-all && \ # Install flash-linear-attention-npu from the Ascend mirror python3 -m pip install "flash-linear-attention-npu-a2===${FLA_VERSION}" --force-reinstall --no-deps && \ # Keep the CANN env and --no-build-isolation: the isolated build env lacks torch/torch_npu. FLASH_ATTN_BUILD_NPU=910 python3 -m pip install --no-build-isolation "flash_attn_npu==${FLASH_ATTN_NPU_VERSION}" && \ # Install mooncake-transfer-engine-npu from the Ascend nightly mirror python3 -m pip install "mooncake-transfer-engine-npu===${MOONCAKE_VERSION}" && \ python3 -m pip cache purge # Append `libascend_hal.so` path (devlib) to LD_LIBRARY_PATH RUN echo 'export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/usr/local/lib' >> ~/.bashrc CMD ["/bin/bash"] - format.sh (1498 bytes)
#!/usr/bin/env bash # # Copyright (c) 2025 Huawei Technologies Co., Ltd. All Rights Reserved. # Copyright 2023 The vLLM team. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # This file is a part of the vllm-ascend project. # Adapted from https://github.com/vllm-project/vllm/tree/main/tools # check_command() { if ! command -v "$1" &> /dev/null; then echo "❓❓$1 is not installed, please run:" echo "# Install lint deps" echo "pip install -r requirements-lint.txt" echo "# (optional) Enable git commit pre check" echo "pre-commit install" echo "" echo "See step by step contribution guide:" echo "https://docs.vllm.ai/projects/ascend/en/latest/developer_guide/contribution" exit 1 fi } check_command pre-commit # TODO: cleanup SC exclude export SHELLCHECK_OPTS="--exclude=SC2046,SC2006,SC2086" if [[ "$1" != 'ci' ]]; then pre-commit run --all-files else pre-commit run --all-files --hook-stage manual fi - INSTRUCTIONS.md (34617 bytes)
--- name: vllm-ascend-release description: "End-to-end release management skill for vLLM Ascend. Creates release checklist issues, triages critical bugs, audits test coverage, aggregates release note fragments, and guides through the complete release process." disable-model-invocation: true --- # vLLM Ascend Release Skill ## Overview This skill manages the complete end-to-end release process for vLLM Ascend, from creating the release checklist issue to final release announcement. It automates repetitive tasks while ensuring human oversight at critical decision points. ## When to Use This Skill Use this skill when: - Starting a new release cycle (RC or stable) - The release manager needs to track release progress - Preparing release artifacts (notes, documentation, tests) ## Prerequisites - GitHub CLI (`gh`) authenticated with write access to `vllm-project/vllm-ascend` - Access to Ascend NPU hardware for functional testing (or CI infrastructure) - Python environment with `uv` for running scripts ### Verify GitHub CLI Installation Before starting the release process, verify that `gh` CLI is installed and authenticated: ```bash # Check if gh is installed gh --version # If not installed, install gh CLI: # Ubuntu/Debian apt install gh -y # macOS brew install gh # OpenEuler yum install gh -y # Check authentication status gh auth status # If not authenticated, login with: gh auth login ``` Expected output for `gh auth status`: ``` github.com ✓ Logged in to github.com account <username> (keyring) - Active account: true - Git operations protocol: https - Token: gho_**** - Token scopes: 'gist', 'read:org', 'repo', 'workflow' ``` **Required scopes**: `repo` (for creating issues, PRs, releases) and `workflow` (for triggering CI workflows). ## Workflow Overview ``` ┌─────────────────────────────────────────────────────────────────────────────┐ │ vLLM Ascend Release Process │ ├─────────────────────────────────────────────────────────────────────────────┤ │ │ │ Phase 1: Initialization │ │ ├── Determine version & branch │ │ ├── Create feedback issue │ │ └── Create release checklist issue │ │ │ │ Phase 2: Bug Triage │ │ ├── Scan open bugs │ │ ├── Identify release-blocking bugs │ │ └── Update checklist with bug list │ │ │ │ Phase 3: PR Management │ │ ├── Identify must-merge PRs │ │ └── Update checklist with PR list │ │ │ │ Phase 4: Test Coverage Analysis │ │ ├── Scan PRs for features/models without tests │ │ ├── Check previous feedback issue status │ │ └── Update checklist with items needing manual testing │ │ │ │ Phase 5: Nightly Status │ │ ├── Get latest Nightly-A3 and Nightly-A2 runs │ │ ├── Analyze failures with extract_and_analyze.py │ │ └── Update checklist with nightly status table │ │ │ │ Phase 6: Release Notes │ │ ├── Aggregate release note fragments (release_notes/fragments) │ │ ├── Audit commits without fragments │ │ └── Create release notes PR │ │ │ │ Phase 7: Documentation & Artifacts │ │ └── Update version references (Docker/wheel built by CI automatically) │ │ │ │ Phase 8: Release Execution (requires human review) │ │ ├── Human review & approval │ │ ├── Merge release notes PR │ │ ├── Create GitHub release │ │ └── Verify automated pipelines (PyPI, Docker, ReadTheDocs) │ │ │ │ Phase 9: WeChat Article (微信公众号推文) │ │ ├── Collect release statistics (commits, contributors) │ │ ├── Generate WeChat article from template │ │ └── Review and publish to WeChat official account │ │ │ └─────────────────────────────────────────────────────────────────────────────┘ ``` ## Phase 1: Initialization ### 1.1 Gather Release Information Prompt the user for: - **Release Version**: e.g., `v0.15.0rc1`, `v0.15.0` - **Release Branch**: typically `main` - **Target Release Date**: e.g., `2026.03.15` - **Release Manager**: GitHub username ### 1.2 Determine Previous Version ```bash # Get the latest release tag gh release list --repo vllm-project/vllm-ascend --limit 5 # Or check existing tags git tag --sort=-creatordate | head -10 ``` ### 1.3 Create Feedback Issue Create a community feedback issue for the release: ```bash gh issue create --repo vllm-project/vllm-ascend \ --title "[Feedback]: v${VERSION} Release Feedback" \ --body "$(cat templates/feedback-issue-template.md)" \ --label "feedback" ``` ### 1.4 Create Release Checklist Issue Use the template in `templates/release-checklist-template.md`: ```bash # Generate the checklist from template python scripts/generate_checklist.py \ --version ${VERSION} \ --branch ${BRANCH} \ --date ${DATE} \ --manager ${MANAGER} \ --feedback-issue ${FEEDBACK_ISSUE_NUMBER} \ --output release-checklist.md # Create the issue gh issue create --repo vllm-project/vllm-ascend \ --title "[Release]: Release checklist for ${VERSION}" \ --body-file release-checklist.md \ --label "release" ``` ## Phase 2: Bug Triage ### 2.1 Scan Issues Since Last Release Run the issue scanning script to browse all issues since the last release: ```bash python scripts/scan_release_bugs.py \ --repo vllm-project/vllm-ascend \ --since-tag ${LAST_VERSION} \ --output issue-scan.md ``` The script: 1. Gets the release date of the previous version (including rc versions) 2. Fetches all issues created since that date 3. Generates a report with: - **Flagged issues**: Automatically flagged based on engagement or keywords - **All open issues**: Quick browse table with titles - **Recently closed issues**: May be relevant for release notes ### 2.2 Human Review Process The output is designed for quick human review: 1. **Check flagged issues first** - these have high engagement or concerning keywords 2. **Browse the open issues table** - scan titles, click to investigate if needed 3. **Review closed issues** - identify fixes that should be highlighted in release notes ### 2.3 Issue Flagging Criteria Issues are automatically flagged when they have: - High reactions (≥5) or many comments (≥5) - Labels: `bug`, `regression`, `blocker`, `priority:high`, `critical` - Keywords in title: crash, hang, freeze, oom, error, fail, etc. ### 2.4 Update Checklist After manual review, add important bugs to the release checklist: ```bash python scripts/update_checklist_section.py \ --issue-number ${CHECKLIST_ISSUE} \ --section "Bug need Solve" \ --content-file bug-list.md ``` ## Phase 3: PR Management ### 3.1 Identify Must-Merge PRs Scan for PRs that should be included in the release: ```bash # 1. [Priority] List open PRs/issues in the release milestone gh pr list --repo vllm-project/vllm-ascend \ --state open \ --search "milestone:${VERSION}" \ --json number,title,url,labels gh issue list --repo vllm-project/vllm-ascend \ --state open \ --search "milestone:${VERSION}" \ --json number,title,url,labels # 2. List open PRs with release-related labels gh pr list --repo vllm-project/vllm-ascend \ --state open \ --label "release-blocker" \ --json number,title,url # 3. List PRs merged since last release gh pr list --repo vllm-project/vllm-ascend \ --state merged \ --search "merged:>${LAST_RELEASE_DATE}" \ --json number,title,mergedAt ``` **Priority Order:** 1. PRs/Issues in the release milestone - these are explicitly targeted for this release 2. PRs with `release-blocker` label - critical items that must be merged 3. Recently merged PRs - for tracking what's already included ### 3.2 Update Checklist Update the checklist with PRs that need to be merged: ```bash python scripts/update_checklist_section.py \ --issue-number ${CHECKLIST_ISSUE} \ --section "PR need Merge" \ --content-file pr-list.md ``` ## Phase 4: Test Coverage Analysis ### 4.1 Identify Features/Models Needing Testing CI already covers most test cases. Manual testing is only needed for: - **New features** merged without test cases - **New models** added due to environment constraints (e.g., CI doesn't have the model) - **Issues** reported in the previous release's feedback Run the test coverage scanner: ```bash python scripts/scan_test_coverage.py \ --repo vllm-project/vllm-ascend \ --since-tag ${LAST_VERSION} \ --feedback-issue ${PREVIOUS_FEEDBACK_ISSUE} \ --output test-coverage-analysis.md ``` This script: 1. Scans PRs merged since the last release 2. Identifies features/models without corresponding test files 3. Checks the previous feedback issue for unresolved problems ### 4.2 Review the Analysis The output categorizes items: **Features/Models Needing Manual Testing:** - New model support (e.g., Kimi K2.5, GLM-5) - Features that couldn't be tested in CI **Previous Feedback Status:** - Unresolved issues from the feedback thread - Items that need manual verification ### 4.3 Manual Testing Checklist For items identified above, perform manual testing: ```markdown #### Manual Testing Required - [ ] Model: Kimi K2.5 - Basic inference works - [ ] Model: GLM-5 - Multimodal features work - [ ] Feature: Expert parallelism with 8 GPUs - [ ] Feedback: User reported slow startup (verify fixed) ``` ### 4.4 Update Checklist with Results ```bash python scripts/update_checklist_section.py \ --issue-number ${CHECKLIST_ISSUE} \ --section "Functional Test" \ --content-file test-results.md ``` ## Phase 5: Nightly Status ### 5.1 Analyze Nightly CI Runs Get the latest Nightly-A3 and Nightly-A2 CI runs and analyze failures: ```bash python scripts/scan_nightly_status.py \ --repo vllm-project/vllm-ascend \ --output nightly-status.md ``` This script: 1. Fetches the latest Nightly-A3 and Nightly-A2 workflow runs 2. Calls `extract_and_analyze.py` (from main2main-error-analysis skill) for failed runs 3. Extracts and categorizes errors: - **Code Bugs**: Real issues that need fixing - **Environment Flakes**: Transient issues (network, disk, etc.) ### 5.2 Review Output The output includes: | Workflow | Status | Failed Jobs | Code Bugs | Env Flakes | Run | |----------|--------|-------------|-----------|------------|-----| | Nightly-A3 | ✅ success | 0/15 | 0 | 0 | [#123](url) | | Nightly-A2 | ❌ failure | 3/12 | 2 | 1 | [#124](url) | For failed runs, it also shows: - Code bugs that need fixing before release - Failed test cases - Environment flakes (informational) ### 5.3 Update Checklist ```bash python scripts/update_checklist_section.py \ --issue-number ${CHECKLIST_ISSUE} \ --section "Nightly Status" \ --content-file nightly-status.md ``` ## Phase 6: Release Notes This phase assembles the release notes from the fragments collected in `release_notes/fragments/` (see `release_notes/README.md` for the fragment format and principles). Fragments are the primary source; the commit-level analysis is only a fallback audit for entries contributors forgot to declare. ### 6.1 Fetch Commits Fetch all commits between the previous and current version: ```bash # Create output directory mkdir -p output/${VERSION} # Fetch commits with contributor statistics uv run python scripts/fetch_commits.py \ --owner vllm-project \ --repo vllm-ascend \ --base-tag ${LAST_VERSION} \ --head-tag ${NEW_VERSION} \ --stats \ --output output/${VERSION}/0-current-raw-commits.md \ --stats-output output/${VERSION}/0-contributor-stats.md ``` The script outputs: - `0-current-raw-commits.md`: Raw commit list, used for the missing-fragment audit - `0-contributor-stats.md`: Contributor statistics including new contributors ### 6.2 Aggregate Fragments Validate, merge, and group all fragments into a draft: ```bash uv run python scripts/collect_release_notes.py \ --version ${NEW_VERSION} \ --raw-commits output/${VERSION}/0-current-raw-commits.md \ --output output/${VERSION}/1-fragments-draft.md ``` The script: 1. Validates the fragment schema and fails with precise errors on invalid files 2. Merges entries with identical text (for example cherry-picks) into a single bullet 3. Groups entries into release notes sections and lists `highlight: true` candidates 4. Emits the audit section: commits with a PR reference but no fragment, commits without a PR reference, and fragment PRs missing from the commit list It does not move anything unless `--archive` is passed. Output: `1-fragments-draft.md`. ### 6.3 Audit Missing Fragments Review the "Audit: Code Changes Without a Fragment" section of the draft. For every missing line, judge whether users can notice the change (the same principle as `release_notes/README.md`): - User-facing and worth a release note: check it with `gh pr view <number> --repo vllm-project/vllm-ascend`, add a fragment `release_notes/fragments/<number>.yaml`, and re-run 6.2 so the audit becomes clean and the entry is archived with the rest - Not user-facing (tests, CI, internal refactor, small docs): ignore it - Unsure and the PR is important: add a fragment, so the context is not lost Also confirm the "Fragment PRs Not Found in the Commit List" entries. Such a PR may be unmerged, reverte - LICENSE (11357 bytes)
Apache License Version 2.0, January 2004 http://www.apache.org/licenses/ TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION 1. Definitions. "License" shall mean the terms and conditions for use, reproduction, and distribution as defined by Sections 1 through 9 of this document. "Licensor" shall mean the copyright owner or entity authorized by the copyright owner that is granting the License. "Legal Entity" shall mean the union of the acting entity and all other entities that control, are controlled by, or are under common control with that entity. For the purposes of this definition, "control" means (i) the power, direct or indirect, to cause the direction or management of such entity, whether by contract or otherwise, or (ii) ownership of fifty percent (50%) or more of the outstanding shares, or (iii) beneficial ownership of such entity. "You" (or "Your") shall mean an individual or Legal Entity exercising permissions granted by this License. "Source" form shall mean the preferred form for making modifications, including but not limited to software source code, documentation source, and configuration files. "Object" form shall mean any form resulting from mechanical transformation or translation of a Source form, including but not limited to compiled object code, generated documentation, and conversions to other media types. "Work" shall mean the work of authorship, whether in Source or Object form, made available under the License, as indicated by a copyright notice that is included in or attached to the work (an example is provided in the Appendix below). "Derivative Works" shall mean any work, whether in Source or Object form, that is based on (or derived from) the Work and for which the editorial revisions, annotations, elaborations, or other modifications represent, as a whole, an original work of authorship. For the purposes of this License, Derivative Works shall not include works that remain separable from, or merely link (or bind by name) to the interfaces of, the Work and Derivative Works thereof. "Contribution" shall mean any work of authorship, including the original version of the Work and any modifications or additions to that Work or Derivative Works thereof, that is intentionally submitted to Licensor for inclusion in the Work by the copyright owner or by an individual or Legal Entity authorized to submit on behalf of the copyright owner. For the purposes of this definition, "submitted" means any form of electronic, verbal, or written communication sent to the Licensor or its representatives, including but not limited to communication on electronic mailing lists, source code control systems, and issue tracking systems that are managed by, or on behalf of, the Licensor for the purpose of discussing and improving the Work, but excluding communication that is conspicuously marked or otherwise designated in writing by the copyright owner as "Not a Contribution." "Contributor" shall mean Licensor and any individual or Legal Entity on behalf of whom a Contribution has been received by Licensor and subsequently incorporated within the Work. 2. Grant of Copyright License. Subject to the terms and conditions of this License, each Contributor hereby grants to You a perpetual, worldwide, non-exclusive, no-charge, royalty-free, irrevocable copyright license to reproduce, prepare Derivative Works of, publicly display, publicly perform, sublicense, and distribute the Work and such Derivative Works in Source or Object form. 3. Grant of Patent License. Subject to the terms and conditions of this License, each Contributor hereby grants to You a perpetual, worldwide, non-exclusive, no-charge, royalty-free, irrevocable (except as stated in this section) patent license to make, have made, use, offer to sell, sell, import, and otherwise transfer the Work, where such license applies only to those patent claims licensable by such Contributor that are necessarily infringed by their Contribution(s) alone or by combination of their Contribution(s) with the Work to which such Contribution(s) was submitted. If You institute patent litigation against any entity (including a cross-claim or counterclaim in a lawsuit) alleging that the Work or a Contribution incorporated within the Work constitutes direct or contributory patent infringement, then any patent licenses granted to You under this License for that Work shall terminate as of the date such litigation is filed. 4. Redistribution. You may reproduce and distribute copies of the Work or Derivative Works thereof in any medium, with or without modifications, and in Source or Object form, provided that You meet the following conditions: (a) You must give any other recipients of the Work or Derivative Works a copy of this License; and (b) You must cause any modified files to carry prominent notices stating that You changed the files; and (c) You must retain, in the Source form of any Derivative Works that You distribute, all copyright, patent, trademark, and attribution notices from the Source form of the Work, excluding those notices that do not pertain to any part of the Derivative Works; and (d) If the Work includes a "NOTICE" text file as part of its distribution, then any Derivative Works that You distribute must include a readable copy of the attribution notices contained within such NOTICE file, excluding those notices that do not pertain to any part of the Derivative Works, in at least one of the following places: within a NOTICE text file distributed as part of the Derivative Works; within the Source form or documentation, if provided along with the Derivative Works; or, within a display generated by the Derivative Works, if and wherever such third-party notices normally appear. The contents of the NOTICE file are for informational purposes only and do not modify the License. You may add Your own attribution notices within Derivative Works that You distribute, alongside or as an addendum to the NOTICE text from the Work, provided that such additional attribution notices cannot be construed as modifying the License. You may add Your own copyright statement to Your modifications and may provide additional or different license terms and conditions for use, reproduction, or distribution of Your modifications, or for any such Derivative Works as a whole, provided Your use, reproduction, and distribution of the Work otherwise complies with the conditions stated in this License. 5. Submission of Contributions. Unless You explicitly state otherwise, any Contribution intentionally submitted for inclusion in the Work by You to the Licensor shall be under the terms and conditions of this License, without any additional terms or conditions. Notwithstanding the above, nothing herein shall supersede or modify the terms of any separate license agreement you may have executed with Licensor regarding such Contributions. 6. Trademarks. This License does not grant permission to use the trade names, trademarks, service marks, or product names of the Licensor, except as required for reasonable and customary use in describing the origin of the Work and reproducing the content of the NOTICE file. 7. Disclaimer of Warranty. Unless required by applicable law or agreed to in writing, Licensor provides the Work (and each Contributor provides its Contributions) on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied, including, without limitation, any warranties or conditions of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A PARTICULAR PURPOSE. You are solely responsible for determining the appropriateness of using or redistributing the Work and assume any risks associated with Your exercise of permissions under this License. 8. Limitation of Liability. In no event and under no legal theory, whether in tort (including negligence), contract, or otherwise, unless required by applicable law (such as deliberate and grossly negligent acts) or agreed to in writing, shall any Contributor be liable to You for damages, including any direct, indirect, special, incidental, or consequential damages of any character arising as a result of this License or out of the use or inability to use the Work (including but not limited to damages for loss of goodwill, work stoppage, computer failure or malfunction, or any and all other commercial damages or losses), even if such Contributor has been advised of the possibility of such damages. 9. Accepting Warranty or Additional Liability. While redistributing the Work or Derivative Works thereof, You may choose to offer, and charge a fee for, acceptance of support, warranty, indemnity, or other liability obligations and/or rights consistent with this License. However, in accepting such obligations, You may act only on Your own behalf and on Your sole responsibility, not on behalf of any other Contributor, and only if You agree to indemnify, defend, and hold each Contributor harmless for any liability incurred by, or claims asserted against, such Contributor by reason of your accepting any such warranty or additional liability. END OF TERMS AND CONDITIONS APPENDIX: How to apply the Apache License to your work. To apply the Apache License to your work, attach the following boilerplate notice, with the fields enclosed by brackets "[]" replaced with your own identifying information. (Don't include the brackets!) The text should be enclosed in the appropriate comment syntax for the file format. We also recommend that a file or class name and description of purpose be included on the same "printed page" as the copyright notice for easier identification within third-party archives. Copyright [yyyy] [name of copyright owner] Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. - mkdocs.yml (16307 bytes)
# MkDocs configuration for vLLM Ascend documentation # Used for both English and Chinese builds. # Language is selected via the DOCS_LANG environment variable (default: en). # Chinese builds also set DOCS_DIR, DOCS_THEME_LANG, DOCS_SITE_NAME, # DOCS_SITE_URL, DOCS_PALETTE_*, DOCS_MACROS_DIR, DOCS_IS_RELEASE. # See tools/rtd_build.sh for the env vars used by the Chinese build. # # Deployed on Read the Docs with Material for MkDocs theme. site_name: !ENV [DOCS_SITE_NAME, 'vLLM Ascend'] site_url: !ENV [DOCS_SITE_URL, 'https://docs.vllm.ai/projects/ascend/en/latest/'] site_description: vLLM Ascend plugin - a community-maintained hardware plugin for running vLLM on the Ascend NPU. site_author: the vllm-ascend team repo_url: https://github.com/vllm-project/vllm-ascend repo_name: vllm-project/vllm-ascend edit_uri: edit/main/docs/source/ docs_dir: !ENV [DOCS_DIR, docs/source] use_directory_urls: false # ---- Theme: Material for MkDocs ---- theme: name: material language: !ENV [DOCS_THEME_LANG, en] logo: logos/vllm-ascend-logo-text-light.png favicon: logos/vllm-ascend-logo-text-light.png features: - navigation.tabs - navigation.tabs.sticky - navigation.sections - navigation.expand - navigation.indexes - navigation.top - navigation.tracking - toc.follow - search.suggest - search.highlight - search.share - content.code.copy - content.code.select - content.tabs.link - content.action.edit palette: - media: "(prefers-color-scheme: light)" scheme: default primary: !ENV [DOCS_PALETTE_LIGHT_PRIMARY, white] accent: indigo logo: logos/vllm-ascend-logo-text-light.png toggle: icon: material/brightness-7 name: !ENV [DOCS_PALETTE_LIGHT_TOGGLE_NAME, 'Switch to dark mode'] - media: "(prefers-color-scheme: dark)" scheme: slate primary: !ENV [DOCS_PALETTE_DARK_PRIMARY, black] accent: indigo logo: logos/vllm-ascend-logo-text-dark.png toggle: icon: material/brightness-4 name: !ENV [DOCS_PALETTE_DARK_TOGGLE_NAME, 'Switch to system preference'] icon: repo: fontawesome/brands/github custom_dir: docs/overrides # Exclusions are relative to the active docs_dir, including Chinese builds. # Include fragments remain readable by macros and the revision-date hook. exclude_docs: | /_build/ *.zh.md /_templates/ *.inc.md /release.template.md /zh/ __pycache__/ /locale/ # ---- Plugins ---- plugins: - search: separator: '[\s\-,:!=\[\]()"]+' # Both sites add ja for Lunr; English pages still index fine with ja enabled. lang: - en - ja - macros: module_name: docs/hooks/macros_main include_dir: !ENV [DOCS_MACROS_DIR, docs/source] on_undefined: keep # ---- Hooks ---- hooks: # Sets per-page and per-section titles for the active language. # See docs/hooks/nav_titles.py for the title mapping. - docs/hooks/nav_titles.py # Extend the TOC only on MiniMax-M3; other pages keep the site-wide depth. - docs/hooks/minimax_m3_toc.py # Include shared fragments and map Chinese pages to English Git history. - docs/hooks/revision_date.py # ---- Markdown extensions ---- markdown_extensions: # Python Markdown - abbr - admonition - attr_list - def_list - footnotes - md_in_html - pymdown_multimd_table: rowspan: true # Preserve blank-line boundaries between independent Markdown blocks. multibody: false - toc: permalink: true # Site-wide TOC display depth: affects every model guide. # Do not change this value to customize a single model's TOC. toc_depth: 3 # Python Markdown Extensions - pymdownx.arithmatex: generic: true - pymdownx.betterem: # Chinese text does not use spaces between words, so smart emphasis # treats valid constructs such as ``中文**加粗**文本`` as intraword # markup. The Chinese build enables smart handling only for # underscores, preserving identifiers such as ``foo_bar_baz``. smart_enable: !ENV [DOCS_BETTEREM_SMART_ENABLE, all] - pymdownx.caret - pymdownx.details - pymdownx.emoji: emoji_index: !!python/name:material.extensions.emoji.twemoji emoji_generator: !!python/name:material.extensions.emoji.to_svg - pymdownx.highlight: anchor_linenums: true line_spans: __span pygments_lang_class: true - pymdownx.inlinehilite - pymdownx.keys - pymdownx.mark - pymdownx.smartsymbols - pymdownx.superfences: custom_fences: - name: mermaid class: mermaid format: !!python/name:pymdownx.superfences.fence_code_format - pymdownx.tabbed: alternate_style: true - pymdownx.tasklist: custom_checkbox: true - pymdownx.tilde # ---- Extra CSS/JS ---- extra_css: - stylesheets/extra.css extra_javascript: - javascripts/mathjax.js - javascripts/tabbed-toc.js - javascripts/table-fullscreen.js - https://unpkg.com/mathjax@3/es5/tex-mml-chtml.js extra: # Enable dates on RTD; local builds and previews need no Git history. show_revision_date: !ENV [READTHEDOCS, false] # Toggle via DOCS_IS_RELEASE env var (set by .readthedocs.yaml for tag builds). is_release: !ENV [DOCS_IS_RELEASE, false] # Read the Docs build context, exposed to override templates (e.g. the # floating "report issue" button in docs/overrides/main.html). rtd_version: !ENV [READTHEDOCS_VERSION, latest] docs_lang: !ENV [DOCS_LANG, en] # ================== Released documentation stack =========================== # These values are used by Quickstart, Installation, release image/tag examples, # and the pip installation doctest. Update this group only after a vLLM Ascend # release and its complete compatible stack have been published and verified; # do not update it for main development. Git and container tags include the # leading "v" and are used across tutorials. vllm_version: v0.32.0 vllm_ascend_version: v0.32.0rc1 # Stable documentation target used by the developer-preview banner. stable_vllm_ascend_version: v0.30.0 # PyPI versions omit the leading "v" and are used by installation commands/tests. release_vllm_ascend_version: 0.32.0rc1 release_vllm_version: 0.32.0 # The supported Python range and the Python version inside release images differ. release_python_version: ">= 3.10, < 3.13" release_image_python_version: "3.12" release_hdk_version_a2_a3_300i_200i: "26.2.RC1" release_hdk_version_a5: "25.6.RC3.1" release_cann_version: "9.2.1" release_pytorch_version: "2.10.0" release_torch_npu_version: "2.10.0.post7" release_nnal_version: "9.2.1" release_triton_ascend_version: "3.6.0.post2" release_fla_npu_version: "26.9.1" # ================== Current main development stack ========================== # The compatibility policy and CPU-only build instructions use these values. # docs/hooks/macros_main.py reads framework versions from requirements.txt. # Update this group when the verified main build image changes. cann_image_tag: 9.2.1-910b-ubuntu22.04-py3.12 main_python_version: ">= 3.10, < 3.13" main_cann_version: "9.2.1" social: - icon: fontawesome/brands/github link: https://github.com/vllm-project/vllm-ascend generator: false # ---- Navigation ---- # File paths are listed without labels; titles come from each page's H1 # (or the bilingual mapping in docs/hooks/nav_titles.py). Section titles # are translated by the same hook based on the section's first child. nav: - index.md - Getting Started: - getting_started/overview.md - getting_started/quick_start.md - getting_started/installation.md - Model Tutorials: - tutorials/models/index.md - tutorials/models/Qwen3-Dense.md - tutorials/models/Qwen-VL-Dense.md - tutorials/models/Qwen3-30B-A3B.md - tutorials/models/Qwen3-235B-A22B.md - tutorials/models/Qwen3-VL-30B-A3B-Instruct.md - tutorials/models/Qwen3-VL-235B-A22B-Instruct.md - tutorials/models/Qwen3-Coder-30B-A3B.md - tutorials/models/Qwen3-Embedding.md - tutorials/models/Qwen3-VL-Embedding.md - tutorials/models/SigLIP2.md - tutorials/models/Step-5-Preview.md - tutorials/models/Qwen3-Reranker.md - tutorials/models/Qwen3-VL-Reranker.md - tutorials/models/Qwen3-Next.md - tutorials/models/Qwen3-Omni-30B-A3B-Thinking.md - tutorials/models/Qwen3.5-27B-Qwen3.6-27B.md - tutorials/models/Qwen3.5-Dense.md - tutorials/models/Qwen3.5-397B-A17B.md - tutorials/models/Qwen3.6-35B-A3B.md - tutorials/models/Qwen3.8-2.4T-A95B.md - tutorials/models/Qwen3.8-27B.md - tutorials/models/Qwen3.8-Flash-Next.md - tutorials/models/DeepSeek-V3.1.md - tutorials/models/DeepSeek-V3.2.md - tutorials/models/DeepSeek-V4-Flash.md - tutorials/models/DeepSeek-V4.1-Flash.md - tutorials/models/DeepSeek-V4-Pro.md - tutorials/models/DeepSeek-R1.md - tutorials/models/DeepSeekOCR2.md - tutorials/models/Gemma4.md - tutorials/models/GLM4.x.md - tutorials/models/GLM5.md - "GLM-5.2 & GLM-5.3": tutorials/models/GLM5.2.md - tutorials/models/GLM5.3-Flash.md - tutorials/models/Kimi-K2-Thinking.md - tutorials/models/Kimi-K2.5.md - tutorials/models/Kimi-K2.6.md - tutorials/models/Kimi-K3.md - tutorials/models/Kimi-K3.1-Pro.md - tutorials/models/PaddleOCR-VL.md - tutorials/models/MiniMax-M2.md - tutorials/models/MiniMax-M3.md - tutorials/models/MiniMax-M3.1-Flash.md - tutorials/models/Hunyuan-A13B-Instruct.md - tutorials/models/Hy3-preview.md - tutorials/models/Hy4-preview.md - tutorials/models/Hy4.md - tutorials/models/HY4-Flash.md - tutorials/models/Minitron-8B-Base.md - tutorials/models/LLaVA-OneVision-Qwen2-0.5B-OV.md - tutorials/models/gpt-oss-120b.md - tutorials/models/Mixtral-8x7B-Instruct-v0.1.md - tutorials/models/Qwen3-ASR-1.7B.md - tutorials/models/Cohere-Transcribe.md - tutorials/models/Qwen2.5-Math-RM-72B.md - tutorials/models/InternVL3.5.md - tutorials/models/Dots3-Note.md - Feature Tutorials: - tutorials/features/index.md - tutorials/features/pd_colocated_mooncake_multi_instance.md - tutorials/features/pd_disaggregation_mooncake_single_node.md - tutorials/features/pd_disaggregation_mooncake_multi_node.md - tutorials/features/ray.md - faqs.md - User Guide: - user_guide/index.md - Deployment Guide: - user_guide/deployment_guide/index.md - user_guide/deployment_guide/using_mindie_motor.md - user_guide/deployment_guide/using_volcano_kthena.md - Features and Models: - user_guide/support_matrix/index.md - user_guide/support_matrix/supported_models.md - user_guide/support_matrix/supported_features.md - user_guide/support_matrix/feature_matrix.md - Configuration: - user_guide/configuration/index.md - user_guide/configuration/device_selection.md - user_guide/configuration/env_vars.md - user_guide/configuration/additional_config.md - user_guide/configuration/runtime_config.md - user_guide/configuration/encoder_cache_manager.md - Feature Guide: - user_guide/feature_guide/index.md - user_guide/feature_guide/hisparse.md - user_guide/feature_guide/graph_mode.md - user_guide/feature_guide/cpu_binding.md - user_guide/feature_guide/Ai_QoS_introduction_en.md - user_guide/feature_guide/quantization.md - user_guide/feature_guide/sleep_mode.md - user_guide/feature_guide/runtime_guard.md - user_guide/feature_guide/kv_trace.md - user_guide/feature_guide/structured_output.md - user_guide/feature_guide/lora.md - user_guide/feature_guide/expert_parallelism_load_balancer.md - user_guide/feature_guide/netloader.md - user_guide/feature_guide/rfork.md - user_guide/feature_guide/epd_disaggregation.md - user_guide/feature_guide/kv_cache_pool_ascend_store.md - user_guide/feature_guide/mooncake_layerwise_adaptation_and_optimization.md - user_guide/feature_guide/mooncake_hybrid_attention.md - user_guide/feature_guide/layerwise_and_sparse_kv_cache_offloading.md - user_guide/feature_guide/kvpp.md - user_guide/feature_guide/kv_cache_cpu_offload.md - user_guide/feature_guide/encoder_cache_cpu_offload.md - user_guide/feature_guide/preempt_offload_connector.md - user_guide/feature_guide/large_scale_ep.md - user_guide/feature_guide/ucm_deployment.md - user_guide/feature_guide/Fine_grained_TP.md - user_guide/feature_guide/speculative_decoding.md - user_guide/feature_guide/context_parallel.md - user_guide/feature_guide/sequence_parallelism.md - user_guide/feature_guide/batch_invariance.md - user_guide/feature_guide/lmcache_ascend_deployment.md - user_guide/feature_guide/pipeline_parallel.md - user_guide/feature_guide/flash_attention.md - user_guide/feature_guide/short_request_first.md - user_guide/feature_guide/batch_job_aware_scheduler.md - user_guide/feature_guide/rl.md - user_guide/release_notes.md - Developer Guide: - developer_guide/index.md - Contribution: - developer_guide/contribution/index.md - developer_guide/contribution/testing.md - developer_guide/contribution/doc_writing.md - developer_guide/contribution/multi_node_test.md - developer_guide/contribution/nightly_ci_test.md - developer_guide/contribution/e2e_ci_test.md - developer_guide/contribution/dependency_version_management.md - developer_guide/contribution/triton_operators.md - Design Documents: - developer_guide/Design_Documents/index.md - developer_guide/Design_Documents/triton_jit_warmup.md - developer_guide/Design_Documents/persistent_csrc_build_cache.md - developer_guide/Design_Documents/patch.md - developer_guide/Design_Documents/cpu_binding.md - developer_guide/Design_Documents/ModelRunner_prepare_inputs.md - developer_guide/Design_Documents/disaggregated_prefill.md - developer_guide/Design_Documents/layerwise_and_sparse_kv_cache_offloading.md - developer_guide/Design_Documents/model_runner_v2_eplb.md - developer_guide/Design_Documents/ACL_Graph.md - developer_guide/Design_Documents/KV_Cache_Pool_Guide.md - developer_guide/Design_Documents/kvpp.md - developer_guide/Design_Documents/non_contiguous_kv_cache.md - developer_guide/Design_Documents/add_custom_aclnn_op.md - packages.txt (21 bytes)
git vim wget jq curl
- pyproject.toml (2224 bytes)
[build-system] # Should be mirrored in requirements.txt requires = [ "attrs", "cmake>=3.26", "decorator", "einops", "googleapis-common-protos", "numpy", "packaging", "pip", "pybind11", "pyyaml", "scipy", "pandas", "pandas-stubs", "psutil", "setuptools>=64", "setuptools-scm>=8", "transformers==5.19.0", "torch-npu==2.10.0.post7", "torch==2.10.0", "torchvision", "wheel", "msgpack", "quart", "numba", "xgrammar>=0.1.30", "fastapi>=0.133.0,<0.137.0", "fastokens==0.2.0", "compressed_tensors>=0.11.0", "arctic-inference==0.1.1", "memfabric-hybrid===1.3.0", "memcache-hybrid===1.3.0", "triton-ascend===3.6.0.post2" ] build-backend = "setuptools.build_meta" [tool.uv.extra-build-dependencies] pandas = ["setuptools<72"] [tool.pymarkdown] plugins.md004.style = "sublist" # ul-style plugins.md007.indent = 4 # ul-indent plugins.md007.start_indented = true # ul-indent plugins.md013.enabled = false # line-length plugins.md041.enabled = false # first-line-h1 plugins.md033.enabled = false # inline-html plugins.md046.enabled = false # code-block-style plugins.md024.allow_different_nesting = true # no-duplicate-headers plugins.md029.enabled = false # ol-prefix [tool.ruff] # TODO: according to PEP8, there should be 120 characters per line line-length = 120 # Folder to be modified exclude = [] [tool.ruff.lint] select = [ # pycodestyle "E", # Pyflakes "F", # pyupgrade "UP", # flake8-bugbear "B", # flake8-simplify "SIM", # isort "I", # flake8-logging-format "G", ] ignore = [ # star imports "F405", "F403", # lambda expression assignment "E731", # zip without `strict=` "B905", # Loop control variable not used within loop body "B007", # f-string format "UP032", "B904", "SIM108", "SIM102" ] logger-objects = ["vllm.logger.logger"] [tool.ruff.format] docstring-code-format = true [tool.pytest.ini_options] markers = [ "e2e_model: E2E test model identifier(s)", "e2e_coverage: E2E test coverage dimensions (arch, feature, parallel, deploy, hardware, quantization, graph_mode)", ] - README.md (10867 bytes)
<p align="center"> <picture> <source media="(prefers-color-scheme: dark)" srcset="https://raw.githubusercontent.com/vllm-project/vllm-ascend/main/docs/source/logos/vllm-ascend-logo-text-dark.png"> <img alt="vllm-ascend" src="https://raw.githubusercontent.com/vllm-project/vllm-ascend/main/docs/source/logos/vllm-ascend-logo-text-light.png" width=55%> </picture> </p> <h3 align="center"> vLLM Ascend Plugin </h3> <div align="center"> [](https://deepwiki.com/vllm-project/vllm-ascend) </div> <p align="center"> | <a href="https://www.hiascend.com/en/"><b>About Ascend</b></a> | <a href="https://docs.vllm.ai/projects/ascend/en/latest/"><b>Documentation</b></a> | <a href="https://docs.vllm.ai/projects/ascend/en/latest/user_guide/support_matrix/"><b>Support Matrix</b></a> | <a href="https://slack.vllm.ai"><b>#SIG-Ascend</b></a> | <a href="https://discuss.vllm.ai/c/hardware-support/vllm-ascend-support"><b>Users Forum</b></a> | <a href="https://tinyurl.com/vllm-ascend-meeting"><b>Weekly Meeting</b></a> | </p> <p align="center"> <a ><b>English</b></a> | <a href="README.zh.md"><b>中文</b></a> </p> --- *Latest News* 🔥 - [2026/10] We released the new official version [v0.30.0](https://github.com/vllm-project/vllm-ascend/releases/tag/v0.30.0)! Please follow the [official guide](https://docs.vllm.ai/projects/ascend/en/v0.30.0/) to start using vLLM Ascend Plugin on Ascend. - [2026/08] We released the new official version [v0.23.0](https://github.com/vllm-project/vllm-ascend/releases/tag/v0.23.0)! Please follow the [official guide](https://docs.vllm.ai/projects/ascend/en/v0.23.0/) to start using vLLM Ascend Plugin on Ascend. - [2026/05] We released the new official version [v0.18.0](https://github.com/vllm-project/vllm-ascend/releases/tag/v0.18.0)! Please follow the [official guide](https://docs.vllm.ai/projects/ascend/en/v0.18.0/) to start using vLLM Ascend Plugin on Ascend. - [2026/02] We released the new official version [v0.13.0](https://github.com/vllm-project/vllm-ascend/releases/tag/v0.13.0)! Please follow the [official guide](https://docs.vllm.ai/projects/ascend/en/v0.13.0/) to start using vLLM Ascend Plugin on Ascend. <details> <summary>More</summary> - [2025/12] We released the new official version [v0.11.0](https://github.com/vllm-project/vllm-ascend/releases/tag/v0.11.0)! Please follow the [official guide](https://docs.vllm.ai/projects/ascend/en/v0.11.0/) to start using vLLM Ascend Plugin on Ascend. - [2025/09] We released the new official version [v0.9.1](https://github.com/vllm-project/vllm-ascend/releases/tag/v0.9.1)! Please follow the [official guide](https://docs.vllm.ai/projects/ascend/en/v0.9.1/tutorials/large_scale_ep.html) to start deploying large-scale Expert Parallelism (EP) on Ascend. - [2025/08] We hosted the [vLLM Beijing Meetup](https://mp.weixin.qq.com/s/7n8OYNrCC_I9SJaybHA_-Q) with vLLM and Tencent! Please find the [meetup slides](https://drive.google.com/drive/folders/1Pid6NSFLU43DZRi0EaTcPgXsAzDvbBqF). - [2025/06] [User stories](https://docs.vllm.ai/projects/ascend/en/latest/community/user_stories/index.html) page is now live! It kicks off with LLaMA-Factory/verl/TRL/GPUStack to demonstrate how vLLM Ascend assists Ascend users in enhancing their experience across fine-tuning, evaluation, reinforcement learning (RL), and deployment scenarios. - [2025/06] [Contributors](https://docs.vllm.ai/projects/ascend/en/latest/community/contributors.html) page is now live! All contributions deserve to be recorded, thanks for all contributors. - [2025/05] We've released the first official version [v0.7.3](https://github.com/vllm-project/vllm-ascend/releases/tag/v0.7.3)! We collaborated with the vLLM community to publish a blog post sharing our practice: [Introducing vLLM Hardware Plugin, Best Practice from Ascend NPU](https://blog.vllm.ai/2025/05/12/hardware-plugin.html). - [2025/03] We hosted the [vLLM Beijing Meetup](https://mp.weixin.qq.com/s/VtxO9WXa5fC-mKqlxNUJUQ) with vLLM team! Please find the [meetup slides](https://drive.google.com/drive/folders/1Pid6NSFLU43DZRi0EaTcPgXsAzDvbBqF). - [2025/02] vLLM community officially created [vllm-project/vllm-ascend](https://github.com/vllm-project/vllm-ascend) repo for running vLLM seamlessly on the Ascend NPU. - [2024/12] We are working with the vLLM community to support [[RFC]: Hardware pluggable](https://github.com/vllm-project/vllm/issues/11162). </details> --- ## Overview vLLM Ascend (`vllm-ascend`) is a community maintained hardware plugin for running vLLM seamlessly on the Huawei Ascend NPU. It is the recommended approach for supporting the Ascend backend within the vLLM community. It adheres to the principles outlined in the [[RFC]: Hardware pluggable](https://github.com/vllm-project/vllm/issues/11162), providing a hardware-pluggable interface that decouples the integration of the Ascend NPU with vLLM. By using vLLM Ascend plugin, popular open-source models, including Transformer-like, Mixture-of-Experts (MoE), Embedding, Multi-modal LLMs can run seamlessly on the Ascend NPU. For detailed information on supported models and features, please refer to the [Support Matrix](https://docs.vllm.ai/projects/ascend/en/latest/user_guide/support_matrix/). ## DeepWiki [**DeepWiki**](https://deepwiki.com/vllm-project/vllm-ascend) is a dynamic knowledge base collaboratively maintained by the community and AI, designed to provide you with deeper technical insights beyond conventional documentation. If official documentation serves as a "quick start" guide to help you get going, the DeepWiki is your technical companion for "deep understanding". Here, you can explore: core architecture and design principles, key source code interpretations, technical context, and decision-making logic. Whether you are a performance tuner looking to deploy vllm-ascend in production or a contributor aiming to build upon it for secondary development, DeepWiki offers invaluable references for you. Welcome to explore now and join us in delving deep into the technical core of vLLM-Ascend. ## Prerequisites - Hardware: Atlas 800I A2 series, Atlas 800I A3 series, Atlas 300I Duo, 950PR&950DT Products - OS: Linux - Software: - Python == 3.12 - CANN == 9.2.1 (For Ascend HDK version, please refer to the [Release Notes](https://www.hiascend.com/document/detail/zh/CANNCommunityEdition/latest/softwareinst/releasenote/9.2.1/release-notes.md)) - PyTorch == 2.10.0, TorchNPU == 2.10.0.post7 - vLLM (the same version as vllm-ascend) ## Accessing Ascend NPU If you need to access Ascend NPU computing resources for development or testing, please visit the [HiDevLab - Online Development](https://hidevlab.huawei.com/online-develop-intro) page on the HiDevLab platform to apply for and use them. ## Getting Started Please use the following recommended versions to get started quickly: | Version | Release type | Doc | |------------|--------------|--------------------------------------| | v0.30.0 | Latest stable version | See [QuickStart](https://docs.vllm.ai/projects/ascend/en/v0.30.0/quick_start.html) and [Installation](https://docs.vllm.ai/projects/ascend/en/v0.30.0/installation.html) for more details | | v0.23.0 | Stable version | See [QuickStart](https://docs.vllm.ai/projects/ascend/en/v0.23.0/quick_start.html) and [Installation](https://docs.vllm.ai/projects/ascend/en/v0.23.0/installation.html) for more details | ## Branch vllm-ascend has a main branch and a dev branch. - **main**: main branch, corresponds to the vLLM main branch, and is continuously monitored for quality through Ascend CI. - **releases/vX.Y.Z**: development branch, created alongside new releases of vLLM. For example, `releases/v0.13.0` is the dev branch for vLLM `v0.13.0` version. Below are the maintained branches: | Branch | Status | Note | |------------------|--------------|--------------------------------------| | main | Maintained | CI commitment for vLLM main branch and vLLM v0.33.0 tag | | releases/v0.13.0 | Maintained | Only bug fixes are allowed, and no new release tags anymore. | | releases/v0.18.0 | Maintained | CI commitment for vLLM v0.18.0 version | | releases/v0.23.0 | Maintained | CI commitment for vLLM v0.23.0 version | | releases/v0.30.0 | Maintained | CI commitment for vLLM v0.30.0 version | | rfc/<feature-name> | Maintained | [Feature branches](https://docs.vllm.ai/projects/ascend/en/latest/community/versioning_policy.html#feature-branches) for collaboration | Please refer to [Versioning policy](https://docs.vllm.ai/projects/ascend/en/latest/community/versioning_policy.html) for more details. ## Contributing See [CONTRIBUTING](https://docs.vllm.ai/projects/ascend/en/latest/developer_guide/contribution/index.html) for more details, which is a step-by-step guide to help you set up the development environment, build and test. We welcome and value any contributions and collaborations: - Please let us know if you encounter a bug by [filing an issue](https://github.com/vllm-project/vllm-ascend/issues). - Please use [User forum](https://discuss.vllm.ai/c/hardware-support/vllm-ascend-support) for usage questions and help. ## Weekly Meeting - vLLM Ascend Weekly Meeting: <https://tinyurl.com/vllm-ascend-meeting> - Wednesday, 15:00 - 16:00 (UTC+8, [Convert to your timezone](https://dateful.com/convert/gmt8?t=15)) ## License Apache License 2.0, as found in the [LICENSE](./LICENSE) file. - README.zh.md (10373 bytes)
<p align="center"> <picture> <source media="(prefers-color-scheme: dark)" srcset="https://raw.githubusercontent.com/vllm-project/vllm-ascend/main/docs/source/logos/vllm-ascend-logo-text-dark.png"> <img alt="vllm-ascend" src="https://raw.githubusercontent.com/vllm-project/vllm-ascend/main/docs/source/logos/vllm-ascend-logo-text-light.png" width=55%> </picture> </p> <h3 align="center"> vLLM Ascend Plugin </h3> <div align="center"> [](https://deepwiki.com/vllm-project/vllm-ascend) </div> <p align="center"> | <a href="https://www.hiascend.com/"><b>关于昇腾</b></a> | <a href="https://docs.vllm.ai/projects/ascend/zh-cn/latest/"><b>官方文档</b></a> | <a href="https://docs.vllm.ai/projects/ascend/zh-cn/latest/user_guide/support_matrix/"><b>支持矩阵</b></a> | <a href="https://slack.vllm.ai"><b>#sig-ascend</b></a> | <a href="https://discuss.vllm.ai/c/hardware-support/vllm-ascend-support"><b>用户论坛</b></a> | <a href="https://tinyurl.com/vllm-ascend-meeting"><b>社区例会</b></a> | </p> <p align="center"> <a href="README.md"><b>English</b></a> | <a><b>中文</b></a> </p> --- *最新消息* 🔥 - [2026/10] 我们发布了新的正式版本 [v0.30.0](https://github.com/vllm-project/vllm-ascend/releases/tag/v0.30.0)!请按照[官方指南](https://docs.vllm.ai/projects/ascend/zh-cn/v0.30.0/)开始在 Ascend 上部署 vLLM Ascend Plugin。 - [2026/08] 我们发布了新的正式版本 [v0.23.0](https://github.com/vllm-project/vllm-ascend/releases/tag/v0.23.0)!请按照[官方指南](https://docs.vllm.ai/projects/ascend/zh-cn/v0.23.0/)开始在 Ascend 上部署 vLLM Ascend Plugin。 - [2026/05] 我们发布了新的正式版本 [v0.18.0](https://github.com/vllm-project/vllm-ascend/releases/tag/v0.18.0)!请按照[官方指南](https://docs.vllm.ai/projects/ascend/zh-cn/v0.18.0/)开始在 Ascend 上部署vLLM Ascend Plugin。 - [2026/02] 我们发布了新的正式版本 [v0.13.0](https://github.com/vllm-project/vllm-ascend/releases/tag/v0.13.0)!请按照[官方指南](https://docs.vllm.ai/projects/ascend/zh-cn/v0.13.0/)开始在 Ascend 上部署vLLM Ascend Plugin。 <details> <summary>更多内容</summary> - [2025/12] 我们发布了新的正式版本 [v0.11.0](https://github.com/vllm-project/vllm-ascend/releases/tag/v0.11.0)!请按照[官方指南](https://docs.vllm.ai/projects/ascend/zh-cn/v0.11.0/)开始在Ascend上部署vLLM Ascend Plugin。 - [2025/09] 我们发布了新的正式版本 [v0.9.1](https://github.com/vllm-project/vllm-ascend/releases/tag/v0.9.1)!请按照[官方指南](https://docs.vllm.ai/projects/ascend/zh-cn/v0.9.1/user_guide/feature_guide/Large_Scale_Expert_Parallelism.html)开始在Ascend上部署大型专家并行 (EP)。 - [2025/08] 我们与vLLM和腾讯合作举办了[vLLM北京Meetup](https://mp.weixin.qq.com/s/7n8OYNrCC_I9SJaybHA_-Q)!请查阅[活动幻灯片](https://drive.google.com/drive/folders/1Pid6NSFLU43DZRi0EaTcPgXsAzDvbBqF)。 - [2025/06] [用户案例](https://docs.vllm.ai/projects/ascend/zh-cn/latest/community/user_stories/index.html)现已上线!展示了LLaMA-Factory/verl/TRL/GPUStack等用户案例,展示了vLLM Ascend如何帮助昇腾用户在模型微调、评估、强化学习 (RL) 以及部署等场景中提升体验。 - [2025/06] [贡献者](https://docs.vllm.ai/projects/ascend/zh-cn/latest/community/contributors.html)页面现已上线!所有的贡献都值得被记录,感谢所有的贡献者。 - [2025/05] 我们发布了首个正式版本 [v0.7.3](https://github.com/vllm-project/vllm-ascend/releases/tag/v0.7.3)!我们与 vLLM 社区合作发布了一篇博客文章,分享了我们的实践:[Introducing vLLM Hardware Plugin, Best Practice from Ascend NPU](https://blog.vllm.ai/2025/05/12/hardware-plugin.html)。 - [2025/03] 我们和vLLM团队举办了[vLLM Beijing Meetup](https://mp.weixin.qq.com/s/CGDuMoB301Uytnrkc2oyjg)!请查阅[活动幻灯片](https://drive.google.com/drive/folders/1Pid6NSFLU43DZRi0EaTcPgXsAzDvbBqF)。 - [2025/02] vLLM社区正式创建了[vllm-project/vllm-ascend](https://github.com/vllm-project/vllm-ascend)仓库,让vLLM可以无缝运行在Ascend NPU。 - [2024/12] 我们正在与 vLLM 社区合作,以支持 [[RFC]: Hardware pluggable](https://github.com/vllm-project/vllm/issues/11162)。 </details> --- ## 总览 vLLM 昇腾插件 (`vllm-ascend`) 是一个由社区维护的让vLLM在 Huawei Ascend NPU 上无缝运行的后端插件。 此插件是 vLLM 社区中支持昇腾后端的推荐方式。它遵循[[RFC]: Hardware pluggable](https://github.com/vllm-project/vllm/issues/11162)所述原则:通过解耦的方式提供了vLLM对Ascend NPU的支持。 使用 vLLM 昇腾插件,可以让类Transformer、混合专家(MoE)、嵌入、多模态等流行的大语言模型在 Ascend NPU 上无缝运行。 支持的模型详细信息,请参考[模型支持列表](https://docs.vllm.ai/projects/ascend/zh-cn/latest/user_guide/support_matrix/supported_models.html)。 ## DeepWiki [**DeepWiki**](https://deepwiki.com/vllm-project/vllm-ascend) 是一个由社区和 AI 协同维护的动态知识库,旨在为您提供超越常规文档的、更深层次的技术解析。 如果说官方文档是帮助您“快速上手”的操作指南,那么DeepWiki就是陪伴您“深入理解”的技术伙伴。在这里,您可以探索:核心架构与设计原理、关键源码解读、技术背景与决策逻辑。 无论您是希望将 vllm-ascend 用于生产部署的性能调优者,还是想基于它进行二次开发的贡献者,DeepWiki 都能为您提供极具价值的参考。欢迎立即探索,与我们一起深入 vllm-ascend 的技术内核。 ## 准备 - 硬件:Atlas 800I A2 系列、Atlas 800I A3 系列、Atlas 300I Duo、Ascend 950PR&950DT 系列产品 - 操作系统:Linux - 软件: - Python == 3.12 - CANN == 9.2.1 (Ascend HDK 版本详见 [版本说明](https://www.hiascend.com/document/detail/zh/CANNCommunityEdition/latest/softwareinst/releasenote/9.2.1/release-notes.md)) - PyTorch == 2.10.0, TorchNPU == 2.10.0.post7 - vLLM (与vllm-ascend版本一致) ## 访问昇腾NPU 如果您需要访问昇腾NPU算力资源进行开发或测试,请进入HiDevLab平台的[HiDevLab-在线开发](https://hidevlab.huawei.com/online-develop-intro) 页面申请并使用算力。 ## 开始使用 推荐您使用以下版本快速开始使用: | Version | Release type | Doc | |------------|--------------|--------------------------------------| | v0.30.0 | 最新正式/稳定版本 | 请查看[快速开始](https://docs.vllm.ai/projects/ascend/zh-cn/v0.30.0/quick_start.html)和[安装指南](https://docs.vllm.ai/projects/ascend/zh-cn/v0.30.0/installation.html)了解更多 | | v0.23.0 | 正式/稳定版本 | 请查看[快速开始](https://docs.vllm.ai/projects/ascend/zh-cn/v0.23.0/quick_start.html)和[安装指南](https://docs.vllm.ai/projects/ascend/zh-cn/v0.23.0/installation.html)了解更多 | ## 分支策略 vllm-ascend有主干分支和开发分支。 - **main**: 主干分支,与vLLM的主干分支对应,并通过昇腾CI持续进行质量看护。 - **releases/vX.Y.Z**: 开发分支,随vLLM部分新版本发布而创建,比如`releases/v0.13.0`是vllm-ascend针对vLLM `v0.13.0` 版本的开发分支。 下面是维护中的分支: | 分支 | 状态 | 备注 | |------------------|--------------|----------------------| | main | Maintained | 基于vLLM main分支和vLLM最新版本(v0.33.0)CI看护 | | releases/v0.13.0 | Maintained | 只允许Bug修复,不会再发布新版本 | | releases/v0.18.0 | Maintained | 基于vLLM v0.18.0版本CI看护 | | releases/v0.23.0 | Maintained | 基于vLLM v0.23.0版本CI看护 | | releases/v0.30.0 | Maintained | 基于vLLM v0.30.0版本CI看护 | | rfc/<feature-name> | Maintained | 为协作创建的[特性分支](https://docs.vllm.ai/projects/ascend/zh-cn/latest/community/versioning_policy.html#_7) | 请参阅[版本策略](https://docs.vllm.ai/projects/ascend/zh-cn/latest/community/versioning_policy.html)了解更多详细信息。 ## 贡献 请参考[CONTRIBUTING](https://docs.vllm.ai/projects/ascend/zh-cn/latest/developer_guide/contribution/index.html)文档了解更多关于开发环境搭建、功能测试以及 PR 提交规范的信息。 我们欢迎并重视任何形式的贡献与合作: - 请通过[Issue](https://github.com/vllm-project/vllm-ascend/issues)来告知我们您遇到的任何Bug。 - 请通过[用户论坛](https://discuss.vllm.ai/c/hardware-support/vllm-ascend-support)来交流使用问题和寻求帮助。 ## 社区例会 - vLLM Ascend 每周社区例会:<https://tinyurl.com/vllm-ascend-meeting> - 每周三下午,15:00 - 16:00 (UTC+8, 查看[您的时区](https://dateful.com/convert/gmt8?t=15)) ## 许可证 Apache 许可证 2.0,如 [LICENSE](./LICENSE) 文件中所示。 - requirements-dev.txt (378 bytes)
-r requirements-lint.txt -r requirements.txt modelscope<1.38 openai pytest >= 6.0,<9.0.0 pytest-asyncio pytest-mock lm-eval==0.4.12 types-jsonschema zmq types-psutil pytest-cov sentence_transformers pyarrow<26 ray>=2.47.1,<=2.48.0 protobuf>3.20.0 jiwer librosa soundfile mindstudio-probe>=8.3.0 xlite==0.2.0rc5 uc-manager ninja coverage slipcover>=1.0.18 pytest-xdist diff-cover
- requirements-lint.txt (118 bytes)
# formatting pre-commit==4.0.1 # type checking mypy==1.11.1 types-PyYAML types-regex types-requests types-setuptools
- requirements-nightly.txt (264 bytes)
-r requirements-lint.txt openai pytest >= 6.0,<9.0.0 pytest-asyncio pytest-mock lm-eval==0.4.12 types-jsonschema zmq types-psutil pytest-cov sentence_transformers pyarrow<26 jiwer librosa soundfile mindstudio-probe>=8.3.0 xlite==0.2.0rc5 uc-manager ninja coverage
- requirements.txt (643 bytes)
# Should be mirrored in pyproject.toml cmake>=3.26 decorator einops numpy packaging pip pybind11 pyyaml regex scipy pandas psutil setuptools>=64 setuptools-scm>=8 torch==2.10.0 torch-npu==2.10.0.post7 torchvision==0.25.0 torchaudio==2.10.0 triton-ascend===3.6.0.post2 wheel xgrammar>=0.1.30 pandas-stubs compressed_tensors>=0.11.0 # Required for disaggregated prefill msgpack quart # Required for N-gram speculative decoding numba # Required for kvpool supports memcache backend memfabric-hybrid===1.3.0 memcache-hybrid===1.3.0 arctic-inference==0.1.1 transformers==5.19.0 fastapi>=0.133.0,<0.137.0 fastokens==0.2.0 concurrent-log-handler
- setup.py (19288 bytes)
# # Copyright (c) 2025 Huawei Technologies Co., Ltd. All Rights Reserved. # Copyright 2023 The vLLM team. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # This file is a part of the vllm-ascend project. # Adapted from https://github.com/vllm-project/vllm/blob/main/setup.py # import importlib.util import logging import os import subprocess import sys from sysconfig import get_paths from setuptools import Command, Extension, find_packages, setup from setuptools.command.build_ext import build_ext from setuptools.command.build_py import build_py from setuptools.command.develop import develop from setuptools.command.install import install from setuptools_scm import get_version def load_module_from_path(module_name, path): spec = importlib.util.spec_from_file_location(module_name, path) module = importlib.util.module_from_spec(spec) sys.modules[module_name] = module spec.loader.exec_module(module) return module ROOT_DIR = os.path.dirname(__file__) logger = logging.getLogger(__name__) def check_or_set_default_env(cmake_args, env_name, env_variable, default_path=""): if env_variable is None: logging.warning( "No %s found in your environment, please try to set %s if you customize the installation path of this " "library, otherwise default path will be adapted during build this project", env_name, env_name, ) logging.warning("Set default %s: %s", env_name, default_path) env_variable = default_path else: logging.info("Found existing %s: %s", env_name, env_variable) # cann package seems will check this environments in cmake, need write this env variable back. if env_name == "ASCEND_HOME_PATH": os.environ["ASCEND_HOME_PATH"] = env_variable cmake_args += [f"-D{env_name}={env_variable}"] return cmake_args def get_value_from_lines(lines: list[str], key: str) -> str: for line in lines: line = " ".join(line.split()) if key in line: return line.split(":")[-1].strip() return "" def get_chip_type() -> str: try: # Get NPU ID npu_info_lines = subprocess.check_output(["npu-smi", "info", "-l"]).decode().strip().split("\n") npu_id = int(get_value_from_lines(npu_info_lines, "NPU ID")) # Stage 1: query board info without -c flag board_info_lines = ( subprocess.check_output(["npu-smi", "info", "-t", "board", "-i", str(npu_id)]).decode().strip().split("\n") ) # Check if Chip Name exists (Ascend950 includes it directly) chip_name = get_value_from_lines(board_info_lines, "Chip Name") # Stage 2: query with -c flag only if Chip Name not found (A2/A3/310P) if not chip_name: chip_info_lines = ( subprocess.check_output(["npu-smi", "info", "-t", "board", "-i", str(npu_id), "-c", "0"]) .decode() .strip() .split("\n") ) else: # Ascend950 already has complete info chip_info_lines = board_info_lines # Extract required fields chip_name = get_value_from_lines(chip_info_lines, "Chip Name") chip_type = get_value_from_lines(chip_info_lines, "Chip Type") npu_name = get_value_from_lines(chip_info_lines, "NPU Name") if "310" in chip_name: # 310P case assert chip_type return (chip_type + chip_name).lower() elif "910" in chip_name: if chip_type: # A2 case assert not npu_name return (chip_type + chip_name).lower() else: # A3 case assert npu_name return (chip_name + "_" + npu_name).lower() elif "950" in chip_name: assert npu_name return (chip_name + "_" + npu_name).lower() elif "a2g3" in chip_name.lower(): # A2 case: CH version of the HDK return "ascend910b1" else: raise ValueError(f"Unable to recognize chip name: {chip_name}, please manually set env SOC_VERSION") except subprocess.CalledProcessError as e: raise RuntimeError(f"Get chip info failed: {e}") except FileNotFoundError: logging.warning( "npu-smi command not found, if this is an npu envir, please check if npu driver is installed correctly." ) return "" envs = load_module_from_path("envs", os.path.join(ROOT_DIR, "vllm_ascend", "envs.py")) hardware = load_module_from_path( "vllm_ascend_hardware", os.path.join(ROOT_DIR, "vllm_ascend", "device", "hardware.py"), ) if not envs.SOC_VERSION: soc_version = get_chip_type() if not soc_version: raise RuntimeError( "Could not determine chip type automatically via 'npu-smi'. " "This can happen in a CPU-only environment. " "Please set the 'SOC_VERSION' environment variable to specify the target chip, for example:\n" ' - Atlas A2: export SOC_VERSION="ascend910b1"\n' ' - Atlas A3: export SOC_VERSION="ascend910_9391"\n' ' - Atlas 300I: export SOC_VERSION="ascend310p1"\n' ' - Atlas A5: export SOC_VERSION="<value starting with ascend950>"\n' "You can also refer to the SOC_VERSION defaults in Dockerfile*." ) envs.SOC_VERSION = soc_version def gen_build_info(): soc_version = envs.SOC_VERSION device_type = hardware.device_type_from_soc_version(soc_version).name package_dir = os.path.join(ROOT_DIR, "vllm_ascend", "_build_info.py") with open(package_dir, "w+") as f: f.write("# Auto-generated file\n") f.write(f"__device_type__ = '{device_type}'\n") logging.info("Generated _build_info.py with SOC version: %s", soc_version) class CMakeExtension(Extension): def __init__(self, name: str, cmake_lists_dir: str = ".", **kwargs) -> None: super().__init__(name, sources=[], py_limited_api=False, **kwargs) self.cmake_lists_dir = os.path.abspath(cmake_lists_dir) class custom_develop(develop): def run(self): gen_build_info() super().run() class custom_build_info(build_py): def run(self): gen_build_info() super().run() class build_and_install_aclnn(Command): description = "Build and install AclNN by running build_aclnn.sh" user_options = [] def initialize_options(self): pass def finalize_options(self): pass def run(self): try: print("Running bash build_aclnn.sh ...") subprocess.check_call(["bash", "csrc/build_aclnn.sh", ROOT_DIR, envs.SOC_VERSION]) print("build_aclnn.sh executed successfully!") except subprocess.CalledProcessError as e: print(f"Error running build_aclnn.sh: {e}") raise SystemExit(e.returncode) class cmake_build_ext(build_ext): # A dict of extension directories that have been configured. did_config: dict[str, bool] = {} # # Determine number of compilation jobs # def compute_num_jobs(self): # `num_jobs` is either the value of the MAX_JOBS environment variable # (if defined) or the number of CPUs available. num_jobs = envs.MAX_JOBS if num_jobs is not None: num_jobs = int(num_jobs) logger.info("Using MAX_JOBS=%d as the number of jobs.", num_jobs) else: try: # os.sched_getaffinity() isn't universally available, so fall # back to os.cpu_count() if we get an error here. num_jobs = len(os.sched_getaffinity(0)) except AttributeError: num_jobs = os.cpu_count() num_jobs = max(1, num_jobs) return num_jobs # # Perform cmake configuration for a single extension. # def configure(self, ext: CMakeExtension) -> None: build_temp = self.build_temp os.makedirs(build_temp, exist_ok=True) python_executable = sys.executable cmake_args = ["cmake"] # Default use release mode to compile the csrc code # Turbo now support compiled with Release, Debug and RelWithDebugInfo if envs.CMAKE_BUILD_TYPE is None or envs.CMAKE_BUILD_TYPE not in [ "Debug", "Release", "RelWithDebugInfo", ]: envs.CMAKE_BUILD_TYPE = "Release" cmake_args += [f"-DCMAKE_BUILD_TYPE={envs.CMAKE_BUILD_TYPE}"] # Default dump the compile commands for lsp cmake_args += ["-DCMAKE_EXPORT_COMPILE_COMMANDS=1"] if envs.CXX_COMPILER is not None: cmake_args += [f"-DCMAKE_CXX_COMPILER={envs.CXX_COMPILER}"] if envs.C_COMPILER is not None: cmake_args += [f"-DCMAKE_C_COMPILER={envs.C_COMPILER}"] if envs.VERBOSE: cmake_args += ["-DCMAKE_VERBOSE_MAKEFILE=ON"] # find ASCEND_HOME_PATH check_or_set_default_env( cmake_args, "ASCEND_HOME_PATH", envs.ASCEND_HOME_PATH, "/usr/local/Ascend/ascend-toolkit/latest", ) # find PYTHON_EXECUTABLE check_or_set_default_env(cmake_args, "PYTHON_EXECUTABLE", sys.executable) # find PYTHON_INCLUDE_PATH check_or_set_default_env(cmake_args, "PYTHON_INCLUDE_PATH", get_paths()["include"]) # ccache and ninja can not be applied at ascendc kernels now try: # if pybind11 is installed via pip pybind11_cmake_path = ( subprocess.check_output([python_executable, "-m", "pybind11", "--cmakedir"]).decode().strip() ) except subprocess.CalledProcessError as e: # else specify pybind11 path installed from source code on CI container raise RuntimeError(f"CMake configuration failed: {e}") install_path = os.path.join(ROOT_DIR, self.build_lib) if isinstance(self.distribution.get_command_obj("develop"), develop): install_path = os.path.join(ROOT_DIR, "vllm_ascend") # add CMAKE_INSTALL_PATH cmake_args += [f"-DCMAKE_INSTALL_PREFIX={install_path}"] cmake_args += [f"-DCMAKE_PREFIX_PATH={pybind11_cmake_path}"] soc_version_map = { "910b": "ascend910b1", "910c": "ascend910_9392", "310p": "ascend310p1", } CANN_SOC_VERSION = soc_version_map.get(envs.SOC_VERSION, envs.SOC_VERSION) cmake_args += [f"-DSOC_VERSION={CANN_SOC_VERSION}"] # Override the base directory for FetchContent downloads to $ROOT/.deps # This allows sharing dependencies between profiles, # and plays more nicely with sccache. # To override this, set the FETCHCONTENT_BASE_DIR environment variable. fc_base_dir = os.path.join(ROOT_DIR, ".deps") fc_base_dir = os.environ.get("FETCHCONTENT_BASE_DIR", fc_base_dir) cmake_args += ["-DFETCHCONTENT_BASE_DIR={}".format(fc_base_dir)] torch_npu_command = "python3 -m pip show torch-npu | grep '^Location:' | awk '{print $2}'" try: torch_npu_path = subprocess.check_output(torch_npu_command, shell=True).decode().strip() torch_npu_path += "/torch_npu" except subprocess.CalledProcessError as e: raise RuntimeError(f"Retrieve torch version version failed: {e}") # add TORCH_NPU_PATH cmake_args += [f"-DTORCH_NPU_PATH={torch_npu_path}"] # Pass VLLM_ASCEND_ENABLE_BATCH_MEMCPY to CMake if explicitly set. # When unset (None), CMake will auto-detect from CANN headers. if envs.VLLM_ASCEND_ENABLE_BATCH_MEMCPY is not None: cmake_args += [f"-DVLLM_ASCEND_ENABLE_BATCH_MEMCPY={envs.VLLM_ASCEND_ENABLE_BATCH_MEMCPY}"] cmake_args += [ext.cmake_lists_dir] logging.info("cmake config command: %s", cmake_args) try: subprocess.check_call(cmake_args, cwd=self.build_temp) except subprocess.CalledProcessError as e: raise RuntimeError(f"CMake configuration failed: {e}") def build_extensions(self) -> None: if not envs.COMPILE_CUSTOM_KERNELS: return # Ensure that CMake is present and working try: subprocess.check_output(["cmake", "--version"]) except OSError as e: raise RuntimeError(f"Cannot find CMake executable: {e}") # Create build directory if it does not exist. if not os.path.exists(self.build_temp): os.makedirs(self.build_temp) targets = [] os.makedirs(os.path.join(self.build_lib, "vllm_ascend"), exist_ok=True) def target_name(s: str) -> str: return s.removeprefix("vllm_ascend.") # Build all the extensions for ext in self.extensions: self.configure(ext) targets.append(target_name(ext.name)) num_jobs = self.compute_num_jobs() build_args = [ "--build", ".", f"-j={num_jobs}", *[f"--target={name}" for name in targets], ] try: subprocess.check_call(["cmake", *build_args], cwd=self.build_temp) except OSError as e: raise RuntimeError(f"Build library failed: {e}") # Install the libraries install_args = [ "cmake", "--install", ".", ] try: subprocess.check_call(install_args, cwd=self.build_temp) except OSError as e: raise RuntimeError(f"Install library failed: {e}") # copy back to build folder for editable build if isinstance(self.distribution.get_command_obj("develop"), develop): import shutil for root, _, files in os.walk(self.build_temp): for file in files: if file.endswith(".so"): src_path = os.path.join(root, file) dst_path = os.path.join(self.build_lib, "vllm_ascend", file) shutil.copy(src_path, dst_path) print(f"Copy: {src_path} -> {dst_path}") # copy back _cann_ops_custom directory src_cann_ops_custom = os.path.join(ROOT_DIR, "vllm_ascend", "_cann_ops_custom") dst_cann_ops_custom = os.path.join(self.build_lib, "vllm_ascend", "_cann_ops_custom") if os.path.exists(src_cann_ - SKILL.md (8748 bytes)
--- name: vllm-ascend-model-adapter description: "Adapt and debug existing or new models for vLLM on Ascend NPU. Implement in /vllm-workspace/vllm and /vllm-workspace/vllm-ascend, validate via direct vllm serve from /workspace, and deliver one signed commit in the current repo." disable-model-invocation: true --- # vLLM Ascend Model Adapter ## Overview Adapt Hugging Face or local models to run on `vllm-ascend` with minimal changes, deterministic validation, and single-commit delivery. This skill is for both already-supported models and new architectures not yet registered in vLLM. ## Read order 1. Start with `references/workflow-checklist.md`. 2. Read `references/multimodal-ep-aclgraph-lessons.md` (feature-first checklist). 3. If startup/inference fails, read `references/troubleshooting.md`. 4. If checkpoint is fp8-on-NPU, read `references/fp8-on-npu-lessons.md`. 5. Before handoff, read `references/deliverables.md`. ## Hard constraints - Never upgrade `transformers`. - Primary implementation roots are fixed by Dockerfile: - `/vllm-workspace/vllm` - `/vllm-workspace/vllm-ascend` - Start `vllm serve` from `/workspace` with direct command by default. - Default API port is `8000` unless user explicitly asks otherwise. - Feature-first default: try best to validate ACLGraph / EP / flashcomm1 / MTP / multimodal out-of-box. - `--enable-expert-parallel` and flashcomm1 checks are MoE-only; for non-MoE models mark as not-applicable with evidence. - If any feature cannot be enabled, keep evidence and explain reason in final report. - Do not rely on `PYTHONPATH=<modified-src>:$PYTHONPATH` unless debugging fallback is strictly needed. - Keep code changes minimal and focused on the target model. - Final deliverable commit must be one single signed commit in the current working repo (`git commit -sm ...`). - Keep final docs in Chinese and compact. - **Dummy-first is encouraged for speed, but dummy is NOT fully equivalent to real weights.** - **Never sign off adaptation using dummy-only evidence; real-weight gate is mandatory.** ## Execution playbook ### 1) Collect context - Confirm model path (default `/models/<model-name>`; if environment differs, confirm with user explicitly). - Confirm implementation roots (`/vllm-workspace/vllm`, `/vllm-workspace/vllm-ascend`). - Confirm delivery root (the current git repo where the final commit is expected). - Confirm runtime import path points to `/vllm-workspace/*` install. - Use default expected feature set: ACLGraph + EP + flashcomm1 + MTP + multimodal (if model has VL capability). - User requirements extend this baseline, not replace it. ### 2) Analyze model first - Inspect `config.json`, processor files, modeling files, tokenizer files. - Identify architecture class, attention variant, quantization type, and multimodal requirements. - Check state-dict key prefixes (and safetensors index) to infer mapping needs. - Decide whether support already exists in `vllm/model_executor/models/registry.py`. ### 3) Choose adaptation strategy (new-model capable) - Reuse existing vLLM architecture if compatible. - If architecture is missing or incompatible, implement native support: - add model adapter under `vllm/model_executor/models/`; - add processor under `vllm/transformers_utils/processors/` when needed; - register architecture in `vllm/model_executor/models/registry.py`; - implement explicit weight loading/remap rules (including fp8 scale pairing, KV/QK norm sharding, rope variants). - If remote code needs newer transformers symbols, do not upgrade dependency. - If unavoidable, copy required modeling files from sibling transformers source and keep scope explicit. - If failure is backend-specific (kernel/op/platform), patch minimal required code in `/vllm-workspace/vllm-ascend`. ### 4) Implement minimal code changes (in implementation roots) - Touch only files required for this model adaptation. - Keep weight mapping explicit and auditable. - Avoid unrelated refactors. ### 5) Two-stage validation on Ascend (direct run) #### Stage A: dummy fast gate (recommended first) - Run from `/workspace` with `--load-format dummy`. - Goal: fast validate architecture path / operator path / API path. - Do not treat `Application startup complete` as pass by itself; request smoke is mandatory. - Require at least: - startup readiness (`/v1/models` 200), - one text request 200, - if VL model, one text+image request 200, - ACLGraph evidence where expected. #### Stage B: real-weight mandatory gate (must pass before sign-off) - Remove `--load-format dummy` and validate with real checkpoint. - Goal: validate real-only risks: - weight key mapping, - fp8/fp4 dequantization path, - KV/QK norm sharding with real tensor shapes, - load-time/runtime stability. - Require HTTP 200 and non-empty output before declaring success. - Do not pass Stage B on startup-only evidence. ### 6) Validate inference and features - Send `GET /v1/models` first. - Send at least one OpenAI-compatible text request. - For multimodal models, require at least one text+image request. - Validate architecture registration and loader path with logs (no unresolved architecture, no fatal missing-key errors). - Try feature-first validation: EP + ACLGraph path first; eager path as fallback/isolation. - If startup succeeds but first request crashes (false-ready), treat as runtime failure and continue root-cause isolation. - For `torch._dynamo` + `interpolate` + `NPU contiguous` failures on VL paths, try `TORCHDYNAMO_DISABLE=1` as diagnostic/stability fallback. - For multimodal processor API mismatch (for example `skip_tensor_conversion` signature mismatch), use text-only isolation (`--limit-mm-per-prompt` set image/video/audio to 0) to separate processor issues from core weight loading issues. - Capacity baseline by default (single machine): `max-model-len=128k` + `max-num-seqs=16`. - Then expand concurrency (e.g., 32/64) if requested or feasible. ### 7) Backport, generate artifacts, and commit in delivery repo - If implementation happened in `/vllm-workspace/*`, backport minimal final diff to current working repo. - Generate test config YAML at `tests/e2e/models/configs/<ModelName>.yaml` following the schema of existing configs (must include `model_name`, `hardware`, `tasks` with accuracy metrics, and `num_fewshot`). Use accuracy results from evaluation to populate metric values. - Generate tutorial markdown at `docs/source/tutorials/models/<ModelName>.md` following the standard template (Introduction, Supported Features, Environment Preparation with docker tabs, Deployment with serve script, Functional Verification with curl example, Accuracy Evaluation, Performance). Fill in model-specific details: HF path, hardware requirements, TP size, max-model-len, served-model-name, sample curl, and accuracy table. - Update `docs/source/tutorials/models/index.md` to include the new tutorial. - Confirm test config YAML and tutorial doc are included in the staged files. - Commit code changes once (single signed commit). ### 8) Prepare handoff artifacts - Write comprehensive Chinese analysis report. - Write compact Chinese runbook for server startup and validation commands. - Include feature status matrix (supported / unsupported / checkpoint-missing / not-applicable). - Include dummy-vs-real validation matrix and explicit non-equivalence notes. - Include changed-file list, key logs, and final commit hash. - Post the SKILL.md content (or a link to it) as a comment on the originating GitHub issue to document the AI-assisted workflow. ## Quality gate before final answer - Service starts successfully from `/workspace` with direct command. - OpenAI-compatible inference request succeeds (not startup-only). - Key feature set is attempted and reported: ACLGraph / EP / flashcomm1 / MTP / multimodal. - Capacity baseline (`128k + bs16`) result is reported, or explicit reason why not feasible. - **Dummy stage evidence is present (if used), and real-weight stage evidence is present (mandatory).** - Test config YAML exists at `tests/e2e/models/configs/<ModelName>.yaml` and follows the established schema (`model_name`, `hardware`, `tasks`, `num_fewshot`). - Tutorial doc exists at `docs/source/tutorials/models/<ModelName>.md` and follows the standard template (Introduction, Supported Features, Environment Preparation, Deployment, Functional Verification, Accuracy Evaluation, Performance). - Tutorial index at `docs/source/tutorials/models/index.md` includes the new model entry. - Exactly one signed commit contains all code changes in current working repo. - Final response includes commit hash, file paths, key commands, known limits, and failure reasons where applicable. - typos.toml (3769 bytes)
[files] # these files may be written in non english words extend-exclude = [".pre-commit-config.yaml", "docs/source/community/contributors.md", "docs/source/locale/",] ignore-hidden = true ignore-files = true ignore-dot = true ignore-vcs = true ignore-global = true ignore-parent = true [default] binary = false check-filename = false check-file = true unicode = true ignore-hex = true identifier-leading-digits = false locale = "en" extend-ignore-identifiers-re = [".*Unc.*", ".*_thw", ".*(UE8M0|ue8m0).*", ".*(UE4M3|ue4m3).*", ".*eles.*", ".*fo.*", ".*ba.*", ".*ot.*", ".*[Tt]h[rR].*", # tools/ai_qos.py: align with driver qos_mata_config / original_sdma_mata (not "meta") ".*(?i)mata.*"] extend-ignore-words-re = ["CANN", "cann","ND","alog","nd","BA","datas","ful","udo","artifcat",] extend-ignore-re = [] [default.extend-identifiers] nd_to_nz_2d = "nd_to_nz_2d" bbc5b7ede = "bbc5b7ede" womens_doubles = "womens_doubles" v_2nd = "v_2nd" splitted_input = "splitted_input" NOOPs = "NOOPs" typ = "typ" nin_shortcut = "nin_shortcut" UperNetDecoder = "UperNetDecoder" subtile = "subtile" SFOuput = "SFOuput" # huggingface transformers repo uses these words depthwise_seperable_out_channel = "depthwise_seperable_out_channel" DepthWiseSeperableConv1d = "DepthWiseSeperableConv1d" depthwise_seperable_CNN = "depthwise_seperable_CNN" [default.extend-words] mata = "mata" iy = "iy" tendencias = "tendencias" PARD = "PARD" # intel cpu features tme = "tme" dout = "dout" Pn = "Pn" arange = "arange" ptd = "ptd" # AscendC operator build tree of the fla_npu OPP (op_impl/ai_core/tbe) tbe = "tbe" [type.py] extend-glob = [] extend-ignore-identifiers-re = [] extend-ignore-words-re = [] extend-ignore-re = [] [type.py.extend-identifiers] arange = "arange" NDArray = "NDArray" EOFError = "EOFError" [type.py.extend-words] [type.cpp] extend-glob = [] extend-ignore-identifiers-re = [] extend-ignore-words-re = [] extend-ignore-re = [] [type.cpp.extend-identifiers] countr_one = "countr_one" [type.cpp.extend-words] [type.rust] extend-glob = [] extend-ignore-identifiers-re = [] extend-ignore-words-re = [] extend-ignore-re = [] [type.rust.extend-identifiers] flate2 = "flate2" [type.rust.extend-words] ser = "ser" [type.lock] extend-glob = [] check-file = false extend-ignore-identifiers-re = [] extend-ignore-words-re = [] extend-ignore-re = [] [type.lock.extend-identifiers] [type.lock.extend-words] [type.jl] extend-glob = [] extend-ignore-identifiers-re = [] extend-ignore-words-re = [] extend-ignore-re = [] [type.jl.extend-identifiers] [type.jl.extend-words] modul = "modul" egals = "egals" usig = "usig" egal = "egal" [type.go] extend-glob = [] extend-ignore-identifiers-re = [] extend-ignore-words-re = [] extend-ignore-re = [] [type.go.extend-identifiers] flate = "flate" [type.go.extend-words] [type.css] extend-glob = [] extend-ignore-identifiers-re = [] extend-ignore-words-re = [] extend-ignore-re = [] [type.css.extend-identifiers] nd = "nd" [type.css.extend-words] [type.man] extend-glob = [] extend-ignore-identifiers-re = [] extend-ignore-words-re = [] extend-ignore-re = [] [type.man.extend-identifiers] Nd = "Nd" [type.man.extend-words] [type.cert] extend-glob = [] check-file = false extend-ignore-identifiers-re = [] extend-ignore-words-re = [] extend-ignore-re = [] [type.cert.extend-identifiers] [type.cert.extend-words] [type.sh] extend-glob = [] extend-ignore-identifiers-re = [] extend-ignore-words-re = [] extend-ignore-re = [] [type.sh.extend-identifiers] stap = "stap" ot = "ot" [type.sh.extend-words] [type.vimscript] extend-glob = [] extend-ignore-identifiers-re = [] extend-ignore-words-re = [] extend-ignore-re = [] [type.vimscript.extend-identifiers] windo = "windo" [type.vimscript.extend-words]