AIStoreHub Package Vault: Transformers
Package Slug: transformers
Archetype: tool
Upstream Repository: https://github.com/huggingface/transformers
Workspace Files & Specifications (67 files)
- AGENTS.md (13 bytes)
.ai/AGENTS.md
- aistorehub.json (508 bytes)
{ "name": "Transformers", "slug": "transformers", "version": "1.0.0", "type": "tool", "description": "State-of-the-art Machine Learning for PyTorch, TensorFlow, and JAX with thousands of pretrained models.", "owner": "community", "repository": "catalog", "github_repo_url": "https://github.com/huggingface/transformers", "tags": [ "transformers", "nlp", "deep-learning", "huggingface", "pytorch", "machine-learning" ], "imported_at": "2026-11-21T23:56:27.435Z" } - awesome-transformers.md (39641 bytes)
# Awesome projects built with Transformers This page lists awesome projects built on top of Transformers. Transformers is more than a toolkit to use pretrained models: it's a community of projects built around it and the Hugging Face Hub. We want Transformers to enable developers, researchers, students, professors, engineers, and anyone else to build their dream projects. In this list, we showcase incredibly impactful and novel projects that have pushed the field forward. We celebrate 100 of these projects as we reach the milestone of 100k stars as a community; but we're very open to pull requests adding other projects to the list. If you believe a project should be here and it's not, then please, open a PR to add it. ## [◉ Universal Intelligence](https://github.com/blueraai/universal-intelligence) [Universal Intelligence](https://github.com/blueraai/universal-intelligence) aims to standardize models, tools, and agents —transforming them into simple, composable, portable, interoperable, framework-agnostic, hardware-agnostic interfaces (through auto-negotiation and resource sharing); for fast and accessible development of AI applications. Keywords: Protocol, Open-source, LLMs, Large Language Models, Agents, Low-code ## [gpt4all](https://github.com/nomic-ai/gpt4all) [gpt4all](https://github.com/nomic-ai/gpt4all) is an ecosystem of open-source chatbots trained on massive collections of clean assistant data including code, stories and dialogue. It offers open-source, large language models such as LLaMA and GPT-J trained in an assistant-style. Keywords: Open-source, LLaMa, GPT-J, instruction, assistant ## [recommenders](https://github.com/recommenders-team/recommenders) This repository contains examples and best practices for building recommendation systems, provided as Jupyter notebooks. It goes over several aspects required to build efficient recommendation systems: data preparation, modeling, evaluation, model selection & optimization, as well as operationalization Keywords: Recommender systems, AzureML ## [IOPaint](https://github.com/Sanster/IOPaint) Image inpainting tool powered by Stable Diffusion. Remove any unwanted object, defect, people from your pictures or erase and replace anything on your pictures. Keywords: inpainting, SD, Stable Diffusion ## [flair](https://github.com/flairNLP/flair) FLAIR is a powerful PyTorch NLP framework, covering several important tasks: NER, sentiment-analysis, part-of-speech tagging, text and document embeddings, among other things. Keywords: NLP, text embedding, document embedding, biomedical, NER, PoS, sentiment-analysis ## [mindsdb](https://github.com/mindsdb/mindsdb) MindsDB is a low-code ML platform, which automates and integrates several ML frameworks into the data stack as "AI Tables" to streamline the integration of AI into applications, making it accessible to developers of all skill levels. Keywords: Database, low-code, AI table ## [langchain](https://github.com/langchain-ai/langchain) [langchain](https://github.com/langchain-ai/langchain) is aimed at assisting in the development of apps merging both LLMs and other sources of knowledge. The library allows chaining calls to applications, creating a sequence across many tools. Keywords: LLMs, Large Language Models, Agents, Chains ## [LlamaIndex](https://github.com/run-llama/llama_index) [LlamaIndex](https://github.com/run-llama/llama_index) is a project that provides a central interface to connect your LLM's with external data. It provides various kinds of indices and retrieval mechanisms to perform different LLM tasks and obtain knowledge-augmented results. Keywords: LLMs, Large Language Models, Data Retrieval, Indices, Knowledge Augmentation ## [ParlAI](https://github.com/facebookresearch/ParlAI) [ParlAI](https://github.com/facebookresearch/ParlAI) is a python framework for sharing, training and testing dialogue models, from open-domain chitchat, to task-oriented dialogue, to visual question answering. It provides more than 100 datasets under the same API, a large zoo of pretrained models, a set of agents, and has several integrations. Keywords: Dialogue, Chatbots, VQA, Datasets, Agents ## [sentence-transformers](https://github.com/UKPLab/sentence-transformers) This framework provides an easy method to compute dense vector representations for sentences, paragraphs, and images. The models are based on transformer networks like BERT / RoBERTa / XLM-RoBERTa etc. and achieve state-of-the-art performance in various task. Text is embedding in vector space such that similar text is close and can efficiently be found using cosine similarity. Keywords: Dense vector representations, Text embeddings, Sentence embeddings ## [ludwig](https://github.com/ludwig-ai/ludwig) Ludwig is a declarative machine learning framework that makes it easy to define machine learning pipelines using a simple and flexible data-driven configuration system. Ludwig is targeted at a wide variety of AI tasks. It provides a data-driven configuration system, training, prediction, and evaluation scripts, as well as a programmatic API. Keywords: Declarative, Data-driven, ML Framework ## [InvokeAI](https://github.com/invoke-ai/InvokeAI) [InvokeAI](https://github.com/invoke-ai/InvokeAI) is an engine for Stable Diffusion models, aimed at professionals, artists, and enthusiasts. It leverages the latest AI-driven technologies through CLI as well as a WebUI. Keywords: Stable-Diffusion, WebUI, CLI ## [PaddleNLP](https://github.com/PaddlePaddle/PaddleNLP) [PaddleNLP](https://github.com/PaddlePaddle/PaddleNLP) is an easy-to-use and powerful NLP library particularly targeted at the Chinese languages. It has support for multiple pre-trained model zoos, and supports a wide-range of NLP tasks from research to industrial applications. Keywords: NLP, Chinese, Research, Industry ## [stanza](https://github.com/stanfordnlp/stanza) The Stanford NLP Group's official Python NLP library. It contains support for running various accurate natural language processing tools on 60+ languages and for accessing the Java Stanford CoreNLP software from Python. Keywords: NLP, Multilingual, CoreNLP ## [DeepPavlov](https://github.com/deeppavlov/DeepPavlov) [DeepPavlov](https://github.com/deeppavlov/DeepPavlov) is an open-source conversational AI library. It is designed for the development of production ready chat-bots and complex conversational systems, as well as research in the area of NLP and, particularly, of dialog systems. Keywords: Conversational, Chatbot, Dialog ## [alpaca-lora](https://github.com/tloen/alpaca-lora) Alpaca-lora contains code for reproducing the Stanford Alpaca results using low-rank adaptation (LoRA). The repository provides training (fine-tuning) as well as generation scripts. Keywords: LoRA, Parameter-efficient fine-tuning ## [imagen-pytorch](https://github.com/lucidrains/imagen-pytorch) An open-source Implementation of Imagen, Google's closed-source Text-to-Image Neural Network that beats DALL-E2. As of release, it is the new SOTA for text-to-image synthesis. Keywords: Imagen, Text-to-image ## [adapters](https://github.com/adapter-hub/adapters) [adapters](https://github.com/adapter-hub/adapters) is an extension of HuggingFace's Transformers library, integrating adapters into state-of-the-art language models by incorporating AdapterHub, a central repository for pre-trained adapter modules. It is a drop-in replacement for transformers, which is regularly updated to stay up-to-date with the developments of transformers. Keywords: Adapters, LoRA, Parameter-efficient fine-tuning, Hub ## [NeMo](https://github.com/NVIDIA/NeMo) NVIDIA [NeMo](https://github.com/NVIDIA/NeMo) is a conversational AI toolkit built for researchers working on automatic speech recognition (ASR), text-to-speech synthesis (TTS), large language models (LLMs), and natural language processing (NLP). The primary objective of [NeMo](https://github.com/NVIDIA/NeMo) is to help researchers from industry and academia to reuse prior work (code and pretrained models) and make it easier to create new https://developer.nvidia.com/conversational-ai#started. Keywords: Conversational, ASR, TTS, LLMs, NLP ## [Runhouse](https://github.com/run-house/runhouse) [Runhouse](https://github.com/run-house/runhouse) allows to send code and data to any of your compute or data infra, all in Python, and continue to interact with them normally from your existing code and environment. Runhouse developers mention: > Think of it as an expansion pack to your Python interpreter that lets it take detours to remote machines or manipulate remote data. Keywords: MLOps, Infrastructure, Data storage, Modeling ## [MONAI](https://github.com/Project-MONAI/MONAI) [MONAI](https://github.com/Project-MONAI/MONAI) is a PyTorch-based, open-source framework for deep learning in healthcare imaging, part of PyTorch Ecosystem. Its ambitions are: - developing a community of academic, industrial and clinical researchers collaborating on a common foundation; - creating state-of-the-art, end-to-end training workflows for healthcare imaging; - providing researchers with the optimized and standardized way to create and evaluate deep learning models. Keywords: Healthcare imaging, Training, Evaluation ## [simpletransformers](https://github.com/ThilinaRajapakse/simpletransformers) Simple Transformers lets you quickly train and evaluate Transformer models. Only 3 lines of code are needed to initialize, train, and evaluate a model. It supports a wide variety of NLP tasks. Keywords: Framework, simplicity, NLP ## [JARVIS](https://github.com/microsoft/JARVIS) [JARVIS](https://github.com/microsoft/JARVIS) is a system attempting to merge LLMs such as GPT-4 with the rest of the open-source ML community: leveraging up to 60 downstream models in order to perform tasks identified by the LLM. Keywords: LLM, Agents, HF Hub ## [transformers.js](https://github.com/huggingface/transformers.js/) [transformers.js](https://github.com/huggingface/transformers.js/) is a JavaScript library targeted at running models from transformers directly within the browser. Keywords: Transformers, JavaScript, browser ## [bumblebee](https://github.com/elixir-nx/bumblebee) Bumblebee provides pre-trained Neural Network models on top of Axon, a neural networks library for the Elixir language. It includes integration with 🤗 Models, allowing anyone to download and perform Machine Learning tasks with few lines of code. Keywords: Elixir, Axon ## [argilla](https://github.com/argilla-io/argilla) Argilla is an open-source platform providing advanced NLP labeling, monitoring, and workspaces. It is compatible with many open source ecosystems such as Hugging Face, Stanza, FLAIR, and others. Keywords: NLP, Labeling, Monitoring, Workspaces ## [haystack](https://github.com/deepset-ai/haystack) Haystack is an open source NLP framework to interact with your data using Transformer models and LLMs. It offers production-ready tools to quickly build complex decision making, question answering, semantic search, text generation applications, and more. Keywords: NLP, Framework, LLM ## [spaCy](https://github.com/explosion/spaCy) [spaCy](https://github.com/explosion/spaCy) is a library for advanced Natural Language Processing in Python and Cython. It's built on the very latest research, and was designed from day one to be used in real products. It offers support for transformers models through its third party package, spacy-transformers. Keywords: NLP, Framework ## [speechbrain](https://github.com/speechbrain/speechbrain) SpeechBrain is an open-source and all-in-one conversational AI toolkit based on PyTorch. The goal is to create a single, flexible, and user-friendly toolkit that can be used to easily develop state-of-the-art speech technologies, including systems for speech recognition, speaker recognition, speech enhancement, speech separation, language identification, multi-microphone signal processing, and many others. Keywords: Conversational, Speech ## [skorch](https://github.com/skorch-dev/skorch) Skorch is a scikit-learn compatible neural network library that wraps PyTorch. It has support for models within transformers, and tokenizers from tokenizers. Keywords: Scikit-Learn, PyTorch ## [bertviz](https://github.com/jessevig/bertviz) BertViz is an interactive tool for visualizing attention in Transformer language models such as BERT, GPT2, or T5. It can be run inside a Jupyter or Colab notebook through a simple Python API that supports most Huggingface models. Keywords: Visualization, Transformers ## [mesh-transformer-jax](https://github.com/kingoflolz/mesh-transformer-jax) [mesh-transformer-jax](https://github.com/kingoflolz/mesh-transformer-jax) is a haiku library using the xmap/pjit operators in JAX for model parallelism of transformers. This library is designed for scalability up to approximately 40B parameters on TPUv3s. It was the library used to train the GPT-J model. Keywords: Haiku, Model parallelism, LLM, TPU ## [deepchem](https://github.com/deepchem/deepchem) DeepChem aims to provide a high quality open-source toolchain that democratizes the use of deep-learning in drug discovery, materials science, quantum chemistry, and biology. Keywords: Drug discovery, Materials Science, Quantum Chemistry, Biology ## [OpenNRE](https://github.com/thunlp/OpenNRE) An Open-Source Package for Neural Relation Extraction (NRE). It is targeted at a wide range of users, from newcomers to relation extraction, to developers, researchers, or students. Keywords: Neural Relation Extraction, Framework ## [pycorrector](https://github.com/shibing624/pycorrector) PyCorrector is a Chinese Text Error Correction Tool. It uses a language model to detect errors, pinyin feature and shape feature to correct Chinese text errors. it can be used for Chinese Pinyin and stroke input method. Keywords: Chinese, Error correction tool, Language model, Pinyin ## [nlpaug](https://github.com/makcedward/nlpaug) This python library helps you with augmenting nlp for machine learning projects. It is a lightweight library featuring synthetic data generation for improving model performance, support for audio and text, and compatibility with several ecosystems (scikit-learn, pytorch, tensorflow). Keywords: Data augmentation, Synthetic data generation, Audio, NLP ## [dream-textures](https://github.com/carson-katri/dream-textures) [dream-textures](https://github.com/carson-katri/dream-textures) is a library targeted at bringing stable-diffusion support within Blender. It supports several use-cases, such as image generation, texture projection, inpainting/outpainting, ControlNet, and upscaling. Keywords: Stable-Diffusion, Blender ## [seldon-core](https://github.com/SeldonIO/seldon-core) Seldon core converts your ML models (Tensorflow, Pytorch, H2o, etc.) or language wrappers (Python, Java, etc.) into production REST/GRPC microservices. Seldon handle
- CLAUDE.md (13 bytes)
.ai/AGENTS.md
- CODE_OF_CONDUCT.md (5489 bytes)
# Contributor Covenant Code of Conduct ## Our Pledge We as members, contributors, and leaders pledge to make participation in our community a harassment-free experience for everyone, regardless of age, body size, visible or invisible disability, ethnicity, sex characteristics, gender identity and expression, level of experience, education, socio-economic status, nationality, personal appearance, race, caste, color, religion, or sexual identity and orientation. We pledge to act and interact in ways that contribute to an open, welcoming, diverse, inclusive, and healthy community. ## Our Standards Examples of behavior that contributes to a positive environment for our community include: * Demonstrating empathy and kindness toward other people * Being respectful of differing opinions, viewpoints, and experiences * Giving and gracefully accepting constructive feedback * Accepting responsibility and apologizing to those affected by our mistakes, and learning from the experience * Focusing on what is best not just for us as individuals, but for the overall community Examples of unacceptable behavior include: * The use of sexualized language or imagery, and sexual attention or advances of any kind * Trolling, insulting or derogatory comments, and personal or political attacks * Public or private harassment * Publishing others' private information, such as a physical or email address, without their explicit permission * Other conduct which could reasonably be considered inappropriate in a professional setting ## Enforcement Responsibilities Community leaders are responsible for clarifying and enforcing our standards of acceptable behavior and will take appropriate and fair corrective action in response to any behavior that they deem inappropriate, threatening, offensive, or harmful. Community leaders have the right and responsibility to remove, edit, or reject comments, commits, code, wiki edits, issues, and other contributions that are not aligned to this Code of Conduct, and will communicate reasons for moderation decisions when appropriate. ## Scope This Code of Conduct applies within all community spaces, and also applies when an individual is officially representing the community in public spaces. Examples of representing our community include using an official e-mail address, posting via an official social media account, or acting as an appointed representative at an online or offline event. ## Enforcement Instances of abusive, harassing, or otherwise unacceptable behavior may be reported to the community leaders responsible for enforcement at [email protected]. All complaints will be reviewed and investigated promptly and fairly. All community leaders are obligated to respect the privacy and security of the reporter of any incident. ## Enforcement Guidelines Community leaders will follow these Community Impact Guidelines in determining the consequences for any action they deem in violation of this Code of Conduct: ### 1. Correction **Community Impact**: Use of inappropriate language or other behavior deemed unprofessional or unwelcome in the community. **Consequence**: A private, written warning from community leaders, providing clarity around the nature of the violation and an explanation of why the behavior was inappropriate. A public apology may be requested. ### 2. Warning **Community Impact**: A violation through a single incident or series of actions. **Consequence**: A warning with consequences for continued behavior. No interaction with the people involved, including unsolicited interaction with those enforcing the Code of Conduct, for a specified period of time. This includes avoiding interactions in community spaces as well as external channels like social media. Violating these terms may lead to a temporary or permanent ban. ### 3. Temporary Ban **Community Impact**: A serious violation of community standards, including sustained inappropriate behavior. **Consequence**: A temporary ban from any sort of interaction or public communication with the community for a specified period of time. No public or private interaction with the people involved, including unsolicited interaction with those enforcing the Code of Conduct, is allowed during this period. Violating these terms may lead to a permanent ban. ### 4. Permanent Ban **Community Impact**: Demonstrating a pattern of violation of community standards, including sustained inappropriate behavior, harassment of an individual, or aggression toward or disparagement of classes of individuals. **Consequence**: A permanent ban from any sort of public interaction within the community. ## Attribution This Code of Conduct is adapted from the [Contributor Covenant][homepage], version 2.1, available at [https://www.contributor-covenant.org/version/2/1/code_of_conduct.html][v2.1]. Community Impact Guidelines were inspired by [Mozilla's code of conduct enforcement ladder][Mozilla CoC]. For answers to common questions about this code of conduct, see the FAQ at [https://www.contributor-covenant.org/faq][FAQ]. Translations are available at [https://www.contributor-covenant.org/translations][translations]. [homepage]: https://www.contributor-covenant.org [v2.1]: https://www.contributor-covenant.org/version/2/1/code_of_conduct.html [Mozilla CoC]: https://github.com/mozilla/diversity [FAQ]: https://www.contributor-covenant.org/faq [translations]: https://www.contributor-covenant.org/translations
- conftest.py (23538 bytes)
# Copyright 2020 The HuggingFace Team. All rights reserved. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # tests directory-specific settings - this file is run automatically # by pytest before any tests are run import doctest import errno import functools import math import os import re import sys import tempfile import warnings from os.path import abspath, dirname, join from unittest import mock import _pytest import pytest from transformers.testing_utils import ( HfDoctestModule, HfDocTestParser, get_cpu_ram_total_gib, install_ci_source_deps_tracker, is_torch_available, patch_psutil_cpu_memory, patch_testing_methods_to_collect_info, patch_torch_compile_force_graph, record_ci_source_deps_by_file, record_ci_source_deps_chains, scan_preloaded_test_module_deps, set_ci_collecting_file, set_ci_deps_phase, write_ci_module_graph, ) from transformers.utils import enable_tf32 from transformers.utils.network_logging import register_network_debug_plugin _ci_fallback_cache_dir = None # Repos that already fell back to `_ci_fallback_cache_dir` in this process. Later lookups for # them must go to the tmp cache first: their files are not in the read-only shared cache, so # e.g. a `local_files_only=True` reload would otherwise fail to find them. _ci_fallback_repo_ids = set() # Directory holding one append-only file per process recording the repo/file id for every # call retried through the read-only cache fallback. A directory (rather than an in-memory # list) is used so the count survives `pytest-xdist`: tests run in worker subprocesses, but # `pytest_terminal_summary` runs on the controller, which aggregates all the files. The path # is shared with workers via an env var (they inherit it at spawn, like DISABLE_SAFETENSORS_CONVERSION). _ci_fallback_events_dir = None def _record_fallback_event(repo_id): """Append `repo_id` to this process's fallback-event file (xdist-safe, best-effort).""" if not _ci_fallback_events_dir: return worker = os.environ.get("PYTEST_XDIST_WORKER", "main") try: with open(os.path.join(_ci_fallback_events_dir, f"{worker}-{os.getpid()}.log"), "a", encoding="utf-8") as fh: fh.write(f"{repo_id}\n") except OSError: pass def _collect_fallback_events(): """Aggregate fallback events recorded by every process (controller + xdist workers).""" events = [] if _ci_fallback_events_dir and os.path.isdir(_ci_fallback_events_dir): for name in sorted(os.listdir(_ci_fallback_events_dir)): try: with open(os.path.join(_ci_fallback_events_dir, name), encoding="utf-8") as fh: events.extend(line.strip() for line in fh if line.strip()) except OSError: pass return events # Rust's `std::io::Error` renders OS-level failures as ``"<strerror> (os error <N>)"`` # where ``<N>`` is the raw platform errno. The trailing ``(os error N)`` fragment is # emitted by Rust in English regardless of locale, so the numeric code -- not the # human-readable ``<strerror>`` text -- is the stable signal to key off. _OS_ERROR_CODE_RE = re.compile(r"os error (\d+)", re.IGNORECASE) def _is_readonly_fs_error(e): """Return True if `e` signals a read-only filesystem (EROFS). The plain download path raises `OSError` with `errno == EROFS`. The Xet path (`hf_xet` Rust library) instead raises a bare `RuntimeError` with no `.errno` set, carrying a Rust-rendered message that ends with ``(os error <N>)``. We extract that numeric code and compare it to `errno.EROFS` rather than matching the locale-dependent strerror text, which is far more robust. The cause/context chain is followed so a wrapped error is still recognised. """ # Follow the cause/context chain so a wrapped error is still recognised. while e is not None: # Standard path: an OSError carrying the EROFS errno. if isinstance(e, OSError) and e.errno == errno.EROFS: return True # hf_xet path: a bare RuntimeError with the raw OS errno rendered as "(os error N)". if isinstance(e, RuntimeError) and any(int(c) == errno.EROFS for c in _OS_ERROR_CODE_RE.findall(str(e))): return True e = e.__cause__ or e.__context__ return False def _with_tmpdir_cache_fallback(fn): """Decorator that retries `fn` with a writable tmp cache dir if it raises EROFS. In CI, the shared HF cache is read-only. Most models are pre-populated there and work fine. Only downloads of new or updated files fail with EROFS. On such failure, a session-scoped tmp dir is created once (via `tempfile.mkdtemp`, which is atomic and process-safe) and the call is retried with `cache_dir` set to the tmp dir. The retry also disables Xet (``HF_HUB_DISABLE_XET``) so the download takes the plain HTTP path, which writes into the redirected `cache_dir`. This is required because the Xet path (`hf_xet` Rust library) uses a process-wide session singleton that binds to the original read-only `HF_XET_CACHE` the first time it runs; redirecting `HF_XET_CACHE` on retry has no effect on that already-created session, so the write keeps failing. `HF_XET_CACHE` is still redirected too, as a belt-and-suspenders measure. """ def call_with_fallback_cache(args, kwargs): import huggingface_hub.constants as hf_constants with ( mock.patch.object(hf_constants, "HF_HUB_DISABLE_XET", True), mock.patch.dict(os.environ, {"HF_HUB_DISABLE_XET": "1"}), mock.patch.object(hf_constants, "HF_XET_CACHE", _ci_fallback_cache_dir), mock.patch.dict(os.environ, {"HF_XET_CACHE": _ci_fallback_cache_dir}), ): return fn(*args, **{**kwargs, "cache_dir": _ci_fallback_cache_dir}) @functools.wraps(fn) def wrapper(*args, **kwargs): # `cached_files` names it `path_or_repo_id`; `hf_hub_download` names it `repo_id`. repo_id = str(kwargs.get("path_or_repo_id") or kwargs.get("repo_id") or (args[0] if args else "?")) if repo_id in _ci_fallback_repo_ids and kwargs.get("cache_dir") is None: try: return call_with_fallback_cache(args, kwargs) except (OSError, RuntimeError): # The repo may be split across both caches (only some files were new); fall # through to the shared cache, which retries via the tmp dir again on EROFS. pass try: return fn(*args, **kwargs) except (OSError, RuntimeError) as e: if not _is_readonly_fs_error(e): raise global _ci_fallback_cache_dir if _ci_fallback_cache_dir is None: _ci_fallback_cache_dir = tempfile.mkdtemp(prefix="ci_fallback_tmpdir_cache_dir_") _ci_fallback_repo_ids.add(repo_id) _record_fallback_event(repo_id) # Emit a marker as well; captured per-test by pytest, but the guaranteed-visible # count is reported in `pytest_terminal_summary` at the end of the run. print( f"[CI_CACHE_FALLBACK] read-only cache hit for {repo_id!r} ({type(e).__name__}); " "retrying via writable tmp cache_dir with Xet disabled", file=sys.stderr, flush=True, ) return call_with_fallback_cache(args, kwargs) return wrapper def _with_fallback_cache_revision(resolve_revision): """Decorator for `HfApi.resolve_revision` that resolves repos which already fell back against `_ci_fallback_cache_dir`. `from_pretrained` resolves the revision once before downloading anything. Offline (e.g. `local_files_only=True`), it reads `refs/<revision>` from the cache, and the read-only shared cache may point to an older commit than the one the fallback downloaded into the tmp cache, so the tmp cache lookups would then miss. """ @functools.wraps(resolve_revision) def wrapper(self, repo_id, *args, **kwargs): if repo_id in _ci_fallback_repo_ids and kwargs.get("cache_dir") is None: kwargs = {**kwargs, "cache_dir": _ci_fallback_cache_dir} return resolve_revision(self, repo_id, *args, **kwargs) wrapper._ci_fallback_wrapped = True return wrapper # In K8S instance-sharing CI, each runner sees the full machine's CPU RAM (~750 GB) even though it only # owns a fraction. This causes `device_map="auto"` to overfill GPU+CPU with nothing offloaded to disk, # leading to GPU OOM at runtime. Cap psutil.virtual_memory to the actual per-runner limit so the entire # test session sees a realistic budget. `get_cpu_ram_total_gib` prefers the cgroup limit (accurate inside # a pod) and the machine's physical RAM, falling back to the CI budget only when neither is available. _cpu_ram_total_gib = get_cpu_ram_total_gib() if not math.isinf(_cpu_ram_total_gib): patch_psutil_cpu_memory(int(_cpu_ram_total_gib * 1024**3)) NOT_DEVICE_TESTS = { "test_tokenization", "test_tokenization_mistral_common", "test_processing", "test_beam_constraints", "test_configuration_utils", "test_data_collator", "test_trainer_callback", "test_trainer_utils", "test_feature_extraction", "test_image_processing", "test_image_processor", "test_image_transforms", "test_optimization", "test_retrieval", "test_config", "test_from_pretrained_no_checkpoint", "test_keep_in_fp32_modules", "test_gradient_checkpointing_backward_compatibility", "test_gradient_checkpointing_enable_disable", "test_torch_save_load", "test_forward_signature", "test_model_get_set_embeddings", "test_model_main_input_name", "test_correct_missing_keys", "test_can_use_safetensors", "test_load_save_without_tied_weights", "test_tied_weights_keys", "test_model_weights_reload_no_missing_tied_weights", "test_can_load_ignoring_mismatched_shapes", "test_model_is_small", "ModelTest::test_pipeline_", # None of the pipeline tests from PipelineTesterMixin (of which XxxModelTest inherits from) are running on device "ModelTester::test_pipeline_", "/repo_utils/", "/utils/", "/conftest_tests/", } # allow having multiple repository checkouts and not needing to remember to rerun # `pip install -e '.[dev]'` when switching between checkouts and running tests. git_repo_path = abspath(join(dirname(__file__), "src")) sys.path.insert(1, git_repo_path) # silence FutureWarning warnings in tests since often we can't act on them until # they become normal warnings - i.e. the tests still need to test the current functionality warnings.simplefilter(action="ignore", category=FutureWarning) def pytest_configure(config): # Shared directory for the read-only cache fallback events. The controller creates it and # exports the path; xdist workers (spawned later) inherit the env var and write into the # same directory, so `pytest_terminal_summary` on the controller can aggregate all of them. global _ci_fallback_events_dir _ci_fallback_events_dir = os.environ.get("CI_FALLBACK_EVENTS_DIR") if _ci_fallback_events_dir is None: _ci_fallback_events_dir = tempfile.mkdtemp(prefix="ci_fallback_events_") os.environ["CI_FALLBACK_EVENTS_DIR"] = _ci_fallback_events_dir import transformers.utils.hub as _hub _hub.cached_files = _with_tmpdir_cache_fallback(_hub.cached_files) # `transformers.utils.hub.resolve_revision` is imported by name in many modules, but all of # them go through `HfApi.resolve_revision`, so patch the method once. from huggingface_hub import HfApi if not getattr(HfApi.resolve_revision, "_ci_fallback_wrapped", False): HfApi.resolve_revision = _with_fallback_cache_revision(HfApi.resolve_revision) # `cached_files` is transformers' entry point, but libraries such as PEFT call # `huggingface_hub.hf_hub_download` directly (e.g. PeftConfig.from_pretrained fetching # adapter_config.json), bypassing it. Wrap that too and re-bind it wherever it has already # been imported via `from huggingface_hub import hf_hub_download`; modules imported later # (peft.config is loaded lazily during the test) pick up the wrapped version automatically. import huggingface_hub _original_hf_hub_download = huggingface_hub.hf_hub_download if not getattr(_original_hf_hub_download, "_ci_fallback_wrapped", False): _wrapped_hf_hub_download = _with_tmpdir_cache_fallback(_original_hf_hub_download) _wrapped_hf_hub_download._ci_fallback_wrapped = True huggingface_hub.hf_hub_download = _wrapped_hf_hub_download # Re-bind already-imported *third-party* consumers (e.g. peft). Skip transformers and # huggingface_hub themselves: transformers funnels downloads through the already-wrapped # `cached_files`, and huggingface_hub's own `snapshot_download` calls `hf_hub_download` # internally -- wrapping either inner call would redirect only the download to the tmp dir # while the caller still resolves the file against the original (read-only) cache_dir, # producing a spurious "missing file". Consumers imported later (peft.config is loaded # lazily during the test) pick up the wrapped function via the package attribute above. for _mod in list(sys.modules.values()): _name = getattr(_mod, "__name__", "") or "" if _name.split(".", 1)[0] in ("transformers", "huggingface_hub"): continue if getattr(_mod, "hf_hub_download", None) is _original_hf_hub_download: _mod.hf_hub_download = _wrapped_hf_hub_download # Special case: third-party libraries such as `kernels` call HfApi methods # (`api.hf_hub_download`, `api.snapshot_download`). Both methods use *local* # imports inside the method body, e.g.: # `from .file_download import hf_hub_download` (HfApi.hf_hub_download) # `from ._snapshot_download import snapshot_download` (HfApi.snapshot_download) # Local imports read the source module's __dict__ at *call time*, so patching # those module-level names is the right interception point. # Module-level references already imported by _snapshot_download.py itself # (its own `hf_hub_download`) are frozen before conftest runs; we wrap # `s - CONTRIBUTING.md (14535 bytes)
<!--- Copyright 2020 The HuggingFace Team. All rights reserved. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. --> # Contribute to 🤗 Transformers ## Contribution rules There are two ways to open a PR: fix an issue that's open for contributions, or open a standalone PR that carries its own evidence. New models can also be added with a standalone PR, see [Adding a new model](#adding-a-new-model). Hugging Face staff, maintainers and some long-standing contributors are exempt from these rules. **Fixing an issue.** Link the issue with a closing keyword like `Fixes #123` in the PR description. You can work on: - an issue you opened, once a maintainer has labeled it `triaged` - an issue labeled [`contributions-welcome`](https://github.com/huggingface/transformers/labels/contributions-welcome) - an issue labeled [`New model`](https://github.com/huggingface/transformers/labels/New%20model), see [Adding a new model](#adding-a-new-model) Other people's issues aren't open for PRs unless a maintainer has said so. PRs that link an issue you can't work on will be closed. You don't need to ask before working on an issue that qualifies, and we don't assign issues to people. Comments like "can I work on this?" or "please assign this to me" just add noise, so skip them and open your PR. The first PR for an issue gets priority. Before working on a `contributions-welcome` or `New model` issue, check whether someone has already opened a PR for it. If so, don't open another one. If you spot mistakes, leave suggestions on the existing PR instead. Only open a competing PR if the existing one is fundamentally flawed, or stale: its author hasn't responded to review comments or pushed changes for two weeks. A PR that's waiting on a maintainer review isn't stale. Either way, explain why in the issue before opening yours. **Standalone PRs.** If nobody has reported the problem yet, you can open a PR without an issue. Standalone PRs are for short changes with real impact, like a bug fix, a performance improvement, or a fix for a broken example in the docs. The PR description needs the same evidence as an issue: - for a bug, a short script that fails on `main` and passes with your fix, using a published checkpoint, a default config, or a documented workflow - for a performance improvement, before and after numbers on a realistic workload, like a published checkpoint or default config, and the hardware you ran on. A faster microbenchmark of a function that isn't on a hot path doesn't count. If an issue already exists for the problem, your PR goes through that issue whether you link it or not. Standalone PRs that fix an existing issue, or that come without evidence, will be closed. These requirements don't apply to new models. **Changes that need an issue first.** Mechanical cleanups that don't affect correctness or performance, and bulk fixes to typos or wording in the docs, need an issue before a PR. Propose the change, how you'd make it, and how many files it touches. A single typo fix is too small for a PR on either path. **Reviewing other people's PRs.** Reviews from the community are welcome when they tell us something new, but maintainers read every comment on a PR, and we already run our own review tools. Before you comment on a PR you didn't open, make sure you have something specific to say: - If you found a problem or have a concrete change to propose, leave an inline comment on the relevant lines, ideally as a GitHub suggestion the author can apply directly. A reproducer helps if you're reporting a bug. - If you reported the issue the PR fixes, a quick note saying whether it fixes your case is really useful. - Please don't post general summaries or "I tested this and it works" reports. Approvals from non-maintainers don't affect whether a PR gets merged, and a long comment that restates the PR costs us time without changing the outcome. How you write your code is up to you, but you're responsible for everything you submit, and we may block accounts that repeatedly open low-quality issues or PRs. Please also respect our [code of conduct](https://github.com/huggingface/transformers/blob/main/CODE_OF_CONDUCT.md). <details> <summary> What we look for in issues and PRs </summary> - Clear diagnosis of bugs. Quick workarounds often cause code bloat or incompatibilities with other models. Spending time to track down the exact cause of a problem, and in particular locating the first commit where it appeared (for example with [git bisect](https://git-scm.com/docs/git-bisect)) is valuable. - Minimize the diff. Check your PR to eliminate any unnecessary changes. Ensure that you did not commit any testing scripts or unrelated files. Add comments only if they're really necessary. If your PR can be a 1-line fix, make it a 1-line fix. This makes the PR much easier to review and improves the chances that it will be accepted. - Take the time to reproduce the problem. Very often when a user reports an issue, the issue is actually caused by environment issues on their machine, or they misdiagnose the problem and suggest an invalid solution. Trusting the report too much leads to bad solutions, sometimes for problems that do not exist! Writing a simple reproducer script and running it to make sure you see the problem is valuable. - Compare against other models. The Transformers repo is very large, and many models are doing similar things. When fixing a bug, it's valuable to see if the bug exists in other models. If your PR says "fixed by using the same approach as (other model)", with a link to the relevant code, that is very helpful for maintainers, because it tells us that the fix is likely to be correct and compatible with the rest of the codebase. - Avoid "busywork" PRs. We don't have time for small style changes or typo fixes in comments, and we don't accept fixes for "theoretical bugs" found by code analysis that rarely if ever cause problems in practice, unless they can be exploited by an attacker. - Verify tests locally and in the CI. Before opening a PR, run `make fix-repo` and use `utils/tests_fetcher.py` to see a list of tests that cover the files you have changed in your PR branch. Run those tests locally, and make sure they pass before you open a PR. After you open your PR, please verify that the CI is green and fix any issues before pinging anyone for review! This reduces notification spam a lot, which keeps maintainers sane. </details> ## Set up 1. Fork [Transformers](https://github.com/huggingface/transformers) with the **Fork** button on GitHub to get a copy under your account, then clone it locally. ```bash git clone [email protected]:<your Github handle>/transformers.git cd transformers ``` 2. Add the Transformers repository as a remote named `upstream`. `origin` points at your fork and `upstream` points at the source of truth. That lets you sync your local `main` before pushing to `origin` and opening a PR. ```bash git remote add upstream https://github.com/huggingface/transformers.git git remote -v # origin [email protected]:<your Github handle>/transformers.git (fetch) # origin [email protected]:<your Github handle>/transformers.git (push) # upstream https://github.com/huggingface/transformers.git (fetch) # upstream https://github.com/huggingface/transformers.git (push) ``` 3. Keep your fork up to date and start a branch to work on. You can also sync your fork from the GitHub UI with the **Sync fork** button. ```bash git checkout main && git pull upstream main && git push origin main git switch -c a-descriptive-name-for-my-branch ``` 4. Install the library in editable mode. Use `[dev]` for most contributions because it includes the development tools needed for style and quality checks. Use `[torch,testing]` for model contributions, or `[quality]` for docs-only changes and small fixes when the full development dependency set is not needed. ```bash pip install -e ".[dev]" # for model contributions pip install -e ".[torch,testing]" # for docs-only changes and small fixes pip install -e ".[quality]" ``` ### Windows On Windows, unless you're using [WSL](https://learn.microsoft.com/en-us/windows/wsl/), configure git to convert `CRLF` line endings to `LF`. ```bash git config core.autocrlf input ``` One way to run the `make` command on Windows is with MSYS2. 1. [Download MSYS2](https://www.msys2.org/) (we assume it's installed in `C:\msys64`). 2. Open the command line `C:\msys64\msys2.exe` (it should be available from the **Start** menu). 3. Run in the shell: `pacman -Syu` and install `make` with `pacman -S make`. 4. Add `C:\msys64\usr\bin` to your PATH environment variable. You can now use `make` from any terminal (PowerShell, cmd.exe, etc.). ## Opening issues Use the issue [templates](https://github.com/huggingface/transformers/tree/main/.github/ISSUE_TEMPLATE), and make sure the issue wasn't already reported (use the search bar on GitHub under Issues). A bug report should be about a bug in Transformers, not in your own code, and it should affect real use and real checkpoints; a "theoretical bug" found by reading the code that requires a very nonstandard config to trigger won't be triaged unless it's a security issue. Include a short, self-contained script that reproduces the bug on the latest release or `main`, using a published checkpoint or default config. Bug reports without a runnable reproducer won't be triaged. If you're unsure whether the bug is in your code or the library, ask in the [forum](https://discuss.huggingface.co/) or on [Discord](https://discord.com/invite/hugging-face-879548962464493619) first. That helps keep GitHub focused on actionable library issues. ## Adding a new model Adding a model to Transformers makes it available for anyone to load and fine-tune it or run inference. Before you start the implementation, isolate the differences between your architecture and recent state-of-the-art models that are already in Transformers. Explain those differences in the issue or PR so reviewers can quickly understand what challenges the model brings, what parts can reuse existing patterns, and what should be standardized across models. Follow the guides below. 1. [Add a model with modular Transformers](https://github.com/huggingface/transformers/blob/main/docs/source/en/modular_transformers.md) — implement the model using the modular file and generate standalone modeling files. 2. [Auto-generate docstrings](https://github.com/huggingface/transformers/blob/main/docs/source/en/auto_docstring.md) — use the `@auto_docstring` decorator to generate consistent docstrings without boilerplate. 3. [Testing](https://github.com/huggingface/transformers/blob/main/docs/source/en/testing.md) — write and run model tests to verify correctness and keep the contribution maintainable. For causal language models, inherit from the causal LM tester where possible. Prioritize integration tests because they verify the full processor, tokenizer, and model path against real checkpoints. 4. [Model structure rules](https://github.com/huggingface/transformers/blob/main/docs/source/en/modeling_rules.md) — check your files pass the static model structure rules enforced by `make typing`. Some model additions need extra integration work. - [Add vision processing components](https://github.com/huggingface/transformers/blob/main/docs/source/en/add_vision_processing_components.md) if your model requires image or video inputs. - [Dynamic weight loading](https://github.com/huggingface/transformers/blob/main/docs/source/en/weightconverter.md) if published checkpoint parameter names do not match the Transformers implementation. ### Model addition timeline There are four timelines for model additions depending on the model contributor and community demand for an architecture. - Day-0 integration: make the model available in Transformers on release day with a new version release. We provide help to support the most important core features (quantization, FlashAttention, KV-cache, etc.), review early drafts, and can also take care of the model implementation depending on timelines. Email [email protected] a few weeks ahead, especially for novel architectures. We'll iterate with you on a private fork until your checkpoint and release are ready. - Same-week integration: high-demand models usually land within a week, even when the author doesn't reach out. Open an issue with the [new model template](https://github.com/huggingface/transformers/issues/new?assignees=&labels=New+model&projects=&template=new-model-addition.yml) to request one. Issues with more activity move up the queue faster. - Post-release integration: models without strong demand, or that we don't have bandwidth to take on, land after the upstream release. Outside contributors can open a PR for a new model directly, with or without an issue. Before you start, check the open ["New model"](https://github.com/huggingface/transformers/issues?q=is%3Aopen+is%3Aissue+label%3A%22New+model%22) issues and the open PRs to make sure nobody is already adding it. We'll review and guide you through it. - Hub-first release: ship your model directly on the Hub via Transformers' [remote-code](https://github.com/huggingface/transformers/blob/main/docs/source/en/models.md#custom-models) support, with no upstream PR required. Popular Hub-first models often get integrated into Transformers later, which unlocks first-class docs, maintenance, and optimization. Hub-first is the lowest-friction way to add a model. However, it can be more fragile if a model requires weight conversion or other backward compatibility issues. ## Docs For API reference generated from source files, use the [`@auto_docstring`](https://github.com/huggingface/transformers/blob/main/docs/source/en/auto_docstring.md) decorator when it applies. Refer to the docs [README](https://github.com/huggingface/transformers/blob/main/docs/README.md) for more details about how to edit the docs and the syntax we use. - docs/source/en/serve-cli/tiny_agents.md (2450 bytes)
<!--Copyright 2026 The HuggingFace Team. All rights reserved. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the ⚠️ Note that this file is in Markdown but contains specific syntax for our doc-builder (similar to MDX) that may not be rendered properly in your Markdown viewer. --> # tiny-agents [tiny-agents](https://huggingface.co/blog/python-tiny-agents) offers a minimal pattern for building tool-using agents. A small Python loop connects an MCP client to tools from MCP servers. Hugging Face Spaces work as MCP servers. Find more compatible Spaces on the [Hub](https://huggingface.co/spaces?filter=mcp-server). Create a config file that points to your local model and the tool server. This example uses an image generation [Space](https://evalstate-flux1-schnell.hf.space/). ```json { "model": "Menlo/Jan-nano", "endpointUrl": "http://localhost:8000", "servers": [ { "type": "sse", "url": "https://evalstate-flux1-schnell.hf.space/gradio_api/mcp/sse" } ] } ``` Run the agent with your config. ```bash tiny-agents run path/to/your/config.json ``` Ensure `transformers serve` is running in the background. The agent will use your local model to run the image generation tool. ```bash Agent loaded with 1 tools: • flux1_schnell_infer » Generate an image of a cat on the moon <Tool req_0_tool_call>flux1_schnell_infer {"prompt": "a cat on the moon", "seed": 42, "randomize_seed": true, "width": 1024, "height": 1024, "num_inference_steps": 4} Tool req_0_tool_call [Binary Content: Image image/webp, 57732 bytes] The task is complete and the content accessible to the User Image URL: https://evalstate-flux1-schnell.hf.space/gradio_api/file=/tmp/gradio/3dbddc0e53b5a865ed56a4e3dbdd30f3f61cf3b8aabf1b456f43e5241bd968b8/image.webp 380576952 I have generated an image of a cat on the moon using the Flux 1 Schnell Image Generator. The image is 1024x1024 pixels and was created with 4 inference steps. Let me know if you would like to make any changes or need further assistance! ``` - docs/source/ko/tiny_agents.md (2162 bytes)
### `tiny-agents` CLI 및 MCP 도구[[tiny-agents-cli-and-mcp-tools]] MCP 도구의 사용을 보여주기 위해 [`tiny-agents`](https://huggingface.co/blog/python-tiny-agents) CLI와 `transformers serve` 서버를 연동하는 방법을 살펴보겠습니다. > [!TIP] > 이 예시처럼 많은 Hugging Face Spaces를 MCP 서버로 활용할 수 있습니다. 호환 가능한 모든 Spaces는 [여기](https://huggingface.co/spaces?filter=mcp-server)에서 찾을 수 있습니다. MCP 도구를 사용하려면 먼저 모델에 사용 가능한 도구를 알려야 합니다. 예를 들어, [이미지 생성 MCP 서버](https://evalstate-flux1-schnell.hf.space/)를 참조하는 `tiny-agents` 설정 파일을 살펴보겠습니다. ```json { "model": "Menlo/Jan-nano", "endpointUrl": "http://localhost:8000", "servers": [ { "type": "sse", "url": "https://evalstate-flux1-schnell.hf.space/gradio_api/mcp/sse" } ] } ``` 그런 다음 아래 명령어로 `tiny-agents` 채팅 인터페이스를 실행할 수 있습니다. ```bash tiny-agents run path/to/your/config.json ``` 백그라운드에서 `transformers serve`가 실행 중이라면, 이제 로컬 모델에서 MCP 도구를 사용할 수 있습니다. 다음은 `tiny-agents`와의 채팅 세션 예시입니다. ```bash Agent loaded with 1 tools: • flux1_schnell_infer » Generate an image of a cat on the moon <Tool req_0_tool_call>flux1_schnell_infer {"prompt": "a cat on the moon", "seed": 42, "randomize_seed": true, "width": 1024, "height": 1024, "num_inference_steps": 4} Tool req_0_tool_call [Binary Content: Image image/webp, 57732 bytes] The task is complete and the content accessible to the User Image URL: https://evalstate-flux1-schnell.hf.space/gradio_api/file=/tmp/gradio/3dbddc0e53b5a865ed56a4e3dbdd30f3f61cf3b8aabf1b456f43e5241bd968b8/image.webp 380576952 Flux 1 Schnell 이미지 생성기를 사용하여 달 위의 고양이 이미지를 생성했습니다. 이미지는 1024x1024 픽셀이며 4번의 추론 단계를 거쳐 생성되었습니다. 변경 사항이 필요하거나 추가 도움이 필요하시면 알려주세요! ``` - doctest_list.txt (0 bytes)
- INSTRUCTIONS.md (526 bytes)
You are the autonomous runtime coordinator for Transformers. System Archetype: tool. Description: State-of-the-art Machine Learning for PyTorch, TensorFlow, and JAX with thousands of pretrained models. Core Directives: 1. Execute operations deterministically following verified repository manifests. 2. Strictly enforce safe input validation and local-first execution boundaries. 3. Provide structured outputs, formatted logs, and actionable resolution steps. 4. Preserve data integrity and respect local workspace isolation.
- LICENSE (11418 bytes)
Copyright 2018- The Hugging Face team. All rights reserved. Apache License Version 2.0, January 2004 http://www.apache.org/licenses/ TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION 1. Definitions. "License" shall mean the terms and conditions for use, reproduction, and distribution as defined by Sections 1 through 9 of this document. "Licensor" shall mean the copyright owner or entity authorized by the copyright owner that is granting the License. "Legal Entity" shall mean the union of the acting entity and all other entities that control, are controlled by, or are under common control with that entity. For the purposes of this definition, "control" means (i) the power, direct or indirect, to cause the direction or management of such entity, whether by contract or otherwise, or (ii) ownership of fifty percent (50%) or more of the outstanding shares, or (iii) beneficial ownership of such entity. "You" (or "Your") shall mean an individual or Legal Entity exercising permissions granted by this License. "Source" form shall mean the preferred form for making modifications, including but not limited to software source code, documentation source, and configuration files. "Object" form shall mean any form resulting from mechanical transformation or translation of a Source form, including but not limited to compiled object code, generated documentation, and conversions to other media types. "Work" shall mean the work of authorship, whether in Source or Object form, made available under the License, as indicated by a copyright notice that is included in or attached to the work (an example is provided in the Appendix below). "Derivative Works" shall mean any work, whether in Source or Object form, that is based on (or derived from) the Work and for which the editorial revisions, annotations, elaborations, or other modifications represent, as a whole, an original work of authorship. For the purposes of this License, Derivative Works shall not include works that remain separable from, or merely link (or bind by name) to the interfaces of, the Work and Derivative Works thereof. "Contribution" shall mean any work of authorship, including the original version of the Work and any modifications or additions to that Work or Derivative Works thereof, that is intentionally submitted to Licensor for inclusion in the Work by the copyright owner or by an individual or Legal Entity authorized to submit on behalf of the copyright owner. For the purposes of this definition, "submitted" means any form of electronic, verbal, or written communication sent to the Licensor or its representatives, including but not limited to communication on electronic mailing lists, source code control systems, and issue tracking systems that are managed by, or on behalf of, the Licensor for the purpose of discussing and improving the Work, but excluding communication that is conspicuously marked or otherwise designated in writing by the copyright owner as "Not a Contribution." "Contributor" shall mean Licensor and any individual or Legal Entity on behalf of whom a Contribution has been received by Licensor and subsequently incorporated within the Work. 2. Grant of Copyright License. Subject to the terms and conditions of this License, each Contributor hereby grants to You a perpetual, worldwide, non-exclusive, no-charge, royalty-free, irrevocable copyright license to reproduce, prepare Derivative Works of, publicly display, publicly perform, sublicense, and distribute the Work and such Derivative Works in Source or Object form. 3. Grant of Patent License. Subject to the terms and conditions of this License, each Contributor hereby grants to You a perpetual, worldwide, non-exclusive, no-charge, royalty-free, irrevocable (except as stated in this section) patent license to make, have made, use, offer to sell, sell, import, and otherwise transfer the Work, where such license applies only to those patent claims licensable by such Contributor that are necessarily infringed by their Contribution(s) alone or by combination of their Contribution(s) with the Work to which such Contribution(s) was submitted. If You institute patent litigation against any entity (including a cross-claim or counterclaim in a lawsuit) alleging that the Work or a Contribution incorporated within the Work constitutes direct or contributory patent infringement, then any patent licenses granted to You under this License for that Work shall terminate as of the date such litigation is filed. 4. Redistribution. You may reproduce and distribute copies of the Work or Derivative Works thereof in any medium, with or without modifications, and in Source or Object form, provided that You meet the following conditions: (a) You must give any other recipients of the Work or Derivative Works a copy of this License; and (b) You must cause any modified files to carry prominent notices stating that You changed the files; and (c) You must retain, in the Source form of any Derivative Works that You distribute, all copyright, patent, trademark, and attribution notices from the Source form of the Work, excluding those notices that do not pertain to any part of the Derivative Works; and (d) If the Work includes a "NOTICE" text file as part of its distribution, then any Derivative Works that You distribute must include a readable copy of the attribution notices contained within such NOTICE file, excluding those notices that do not pertain to any part of the Derivative Works, in at least one of the following places: within a NOTICE text file distributed as part of the Derivative Works; within the Source form or documentation, if provided along with the Derivative Works; or, within a display generated by the Derivative Works, if and wherever such third-party notices normally appear. The contents of the NOTICE file are for informational purposes only and do not modify the License. You may add Your own attribution notices within Derivative Works that You distribute, alongside or as an addendum to the NOTICE text from the Work, provided that such additional attribution notices cannot be construed as modifying the License. You may add Your own copyright statement to Your modifications and may provide additional or different license terms and conditions for use, reproduction, or distribution of Your modifications, or for any such Derivative Works as a whole, provided Your use, reproduction, and distribution of the Work otherwise complies with the conditions stated in this License. 5. Submission of Contributions. Unless You explicitly state otherwise, any Contribution intentionally submitted for inclusion in the Work by You to the Licensor shall be under the terms and conditions of this License, without any additional terms or conditions. Notwithstanding the above, nothing herein shall supersede or modify the terms of any separate license agreement you may have executed with Licensor regarding such Contributions. 6. Trademarks. This License does not grant permission to use the trade names, trademarks, service marks, or product names of the Licensor, except as required for reasonable and customary use in describing the origin of the Work and reproducing the content of the NOTICE file. 7. Disclaimer of Warranty. Unless required by applicable law or agreed to in writing, Licensor provides the Work (and each Contributor provides its Contributions) on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied, including, without limitation, any warranties or conditions of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A PARTICULAR PURPOSE. You are solely responsible for determining the appropriateness of using or redistributing the Work and assume any risks associated with Your exercise of permissions under this License. 8. Limitation of Liability. In no event and under no legal theory, whether in tort (including negligence), contract, or otherwise, unless required by applicable law (such as deliberate and grossly negligent acts) or agreed to in writing, shall any Contributor be liable to You for damages, including any direct, indirect, special, incidental, or consequential damages of any character arising as a result of this License or out of the use or inability to use the Work (including but not limited to damages for loss of goodwill, work stoppage, computer failure or malfunction, or any and all other commercial damages or losses), even if such Contributor has been advised of the possibility of such damages. 9. Accepting Warranty or Additional Liability. While redistributing the Work or Derivative Works thereof, You may choose to offer, and charge a fee for, acceptance of support, warranty, indemnity, or other liability obligations and/or rights consistent with this License. However, in accepting such obligations, You may act only on Your own behalf and on Your sole responsibility, not on behalf of any other Contributor, and only if You agree to indemnify, defend, and hold each Contributor harmless for any liability incurred by, or claims asserted against, such Contributor by reason of your accepting any such warranty or additional liability. END OF TERMS AND CONDITIONS APPENDIX: How to apply the Apache License to your work. To apply the Apache License to your work, attach the following boilerplate notice, with the fields enclosed by brackets "[]" replaced with your own identifying information. (Don't include the brackets!) The text should be enclosed in the appropriate comment syntax for the file format. We also recommend that a file or class name and description of purpose be included on the same "printed page" as the copyright notice for easier identification within third-party archives. Copyright [yyyy] [name of copyright owner] Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. - Makefile (3175 bytes)
# make sure to test the local checkout in scripts and not the pre-installed one (don't use quotes!) export PYTHONPATH = src .PHONY: style typing check-code-quality check-repository-consistency check-repo fix-repo test run-mic codex claude clean-ai # Checker lists. The two CI jobs (CircleCI runs `make check-code-quality` and # `make check-repository-consistency` in parallel) own the canonical sets below. # Local convenience targets `check-repo` and `fix-repo` are *derived* from them, # so they can never drift out of sync (e.g. silently dropping `auto_mappings` # from CI, as happened in #45018 → fixed in #45774). STYLE_CHECKERS := ruff_check, ruff_format, init_isort, sort_auto_mappings, noisy_comments TYPING_CHECKERS := types, modeling_structure CODE_QUALITY_CHECKERS := $(TYPING_CHECKERS), $(STYLE_CHECKERS) REPO_CONSISTENCY_CHECKERS := \ auto_mappings, \ imports, \ import_complexity, \ copies, \ modular_conversion, \ inits, \ doc_toc, \ reviewers, \ modeling_rules_doc, \ docstrings, \ dummies, \ repo, \ pipeline_typing, \ config_docstrings, \ config_attributes, \ doctest_list, \ update_metadata, \ add_dates, \ deps_table ALL_CHECKERS := $(CODE_QUALITY_CHECKERS), $(REPO_CONSISTENCY_CHECKERS) # Runs all linting/formatting scripts, most notably ruff style: @python utils/checkers.py $(STYLE_CHECKERS) --fix # Runs ty type checker and model structure rules typing: @python utils/checkers.py $(TYPING_CHECKERS) # Runs typing, ruff linting/formatting, import-order checks and auto-mappings check-code-quality: @python utils/checkers.py $(CODE_QUALITY_CHECKERS) # Runs a full repository consistency check. check-repository-consistency: @python utils/checkers.py $(REPO_CONSISTENCY_CHECKERS) # Runs typing and formatting checks + repository consistency check (ignores errors) check-repo: @python utils/checkers.py $(ALL_CHECKERS) --keep-going # Run all repo checks for which there is an automatic fix, most notably modular conversions fix-repo: @python utils/checkers.py $(ALL_CHECKERS) --fix --keep-going # Run tests for the library, requires pytest-random-order test: python -m pytest -p random_order -n auto --dist=loadfile -s -v --random-order-bucket=module ./tests/ # Run pinned MIC fixtures against this checkout (MODEL=bert, ALL=1, BASE=origin/main, FLEET=prototypes|control, JOBS=4). # Opt-in checker (network + model runtimes), deliberately not in any CI checker list above. run-mic: export MODEL := $(MODEL) run-mic: export ALL := $(ALL) run-mic: export BASE := $(BASE) run-mic: export FLEET := $(FLEET) run-mic: export JOBS := $(JOBS) run-mic: @python utils/checkers.py mic codex: mkdir -p .agents rm -rf .agents/skills ln -snf ../.ai/skills .agents/skills claude: mkdir -p .claude rm -rf .claude/skills ln -snf ../.ai/skills .claude/skills clean-ai: rm -rf .agents/skills .claude/skills # Release stuff pre-release: python utils/release.py pre-patch: python utils/release.py --patch post-release: python utils/release.py --post_release post-patch: python utils/release.py --post_release --patch build-release: rm -rf dist rm -rf build python setup.py bdist_wheel python setup.py sdist
- MIGRATION_GUIDE_V5.md (45883 bytes)
<!--- Copyright 2025 The HuggingFace Team. All rights reserved. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. --> # Version 5 Migration guide ## Library-wide changes with widespread impact ### Removal of TensorFlow and Jax We're removing the TensorFlow and Jax parts of the library. This will help us focus fully on `torch` going forward and will greatly reduce the maintenance cost of models. We are working with tools from the Jax ecosystem still (such as MaxText) in order to see how we can remain compatible with their tool while keeping `torch` as the only backend for now. Linked PR: https://github.com/huggingface/transformers/pull/40760 ### Dynamic weight loading We introduce a new weight loading API in `transformers`, which significantly improves on the previous API. This weight loading API is designed to apply operations to the checkpoints loaded by transformers. Instead of loading the checkpoint exactly as it is serialized within the model, these operations can reshape, merge, and split the layers according to how they're defined in this new API. These operations are often a necessity when working with quantization or parallelism algorithms. This new API is centered around the new `WeightConverter` class: ```python class WeightConverter(WeightTransform): operations: list[ConversionOps] source_keys: Union[str, list[str]] target_keys: Union[str, list[str]] ``` The weight converter is designed to apply a list of operations on the source keys, resulting in target keys. A common operation done on the attention layers is to fuse the query, key, values layers. Doing so with this API would amount to defining the following conversion: ```python conversion = WeightConverter( ["self_attn.q_proj", "self_attn.k_proj", "self_attn.v_proj"], # The input layers "self_attn.qkv_proj", # The single layer as output operations=[Concatenate(dim=0)], ) ``` In this situation, we apply the `Concatenate` operation, which accepts a list of layers as input and returns a single layer. This allows us to define a mapping from architecture to a list of weight conversions. Applying those weight conversions can apply arbitrary transformations to the layers themselves. This significantly simplified the `from_pretrained` method and helped us remove a lot of technical debt that we accumulated over the past few years. This results in several improvements: - Much cleaner definition of transformations applied to the checkpoint - Reversible transformations, so loading and saving a checkpoint should result in the same checkpoint - Faster model loading thanks to scheduling of tensor materialization - Enables complex mix of transformations that wouldn't otherwise be possible (such as quantization + MoEs, or TP + MoEs) While this is being implemented, expect varying levels of support across different release candidates. Linked PR: https://github.com/huggingface/transformers/pull/41580 ## Tokenization Just as we moved towards a single backend library for model definition, we want our tokenizers, and the `Tokenizer` object to be a lot more intuitive. With v5, tokenizer definition is much simpler; one can now initialize an empty `LlamaTokenizer` and train it directly on your corpus. Defining a new tokenizer object should be as simple as this: ```python from transformers import TokenizersBackend, generate_merges from tokenizers import pre_tokenizers, Tokenizer from tokenizers.model import BPE class Llama5Tokenizer(TokenizersBackend): def __init__(self, unk_token="<unk>",bos_token="<s>", eos_token="</s>", vocab=None, merges=None ): if vocab is None: self._vocab = { str(unk_token): 0, str(bos_token): 1, str(eos_token): 2, } else: self._vocab = vocab self._merges = merges or [] self._tokenizer = Tokenizer( BPE(vocab=self._vocab, merges=self._merges, fuse_unk=True) ) self._tokenizer.pre_tokenizer = pre_tokenizers.Metaspace( replacement="▁", prepend_scheme=_get_prepend_scheme(self.add_prefix_space, self), split=False ) super().__init__( tokenizer_object=self._tokenizer, unk_token=unk_token, bos_token=bos_token, eos_token=eos_token, ) ``` Once the tokenizer is defined as above, you can load it with the following: `Llama5Tokenizer()`. Doing this returns you an empty, trainable tokenizer that follows the definition of the authors of `Llama5` (it does not exist yet :wink:). The above is the main motivation towards refactoring tokenization: we want tokenizers to behave similarly to models: trained or empty, and with exactly what is defined in their class definition. ### Backend Architecture Changes: moving away from the slow/fast tokenizer separation Up to now, transformers maintained two parallel implementations for many tokenizers: - "Slow" tokenizers (`tokenization_<model>.py`) - Python-based implementations, often using [SentencePiece](https://github.com/google/sentencepiece) as the backend. - "Fast" tokenizers (`tokenization_<model>_fast.py`) - Rust-based implementations using the 🤗 [tokenizers](https://github.com/huggingface/tokenizers) library. In v5, we consolidate to a single tokenizer file per model: `tokenization_<model>.py`. This file will use the most appropriate backend available: 1. **TokenizersBackend** (preferred): Rust-based tokenizers from the 🤗 [tokenizers](https://github.com/huggingface/tokenizers) library. In general it provides optimal performance, but it also offers a lot more features that are commonly adopted across the ecosystem: - handling additional tokens - a full python API for setting and updating - automatic parallelization, - automatic offsets - customization - training 2. **SentencePieceBackend**: for tokenizers requiring the `sentencepiece` library. It inherits from `PythonBackend`. 3. **PythonBackend**: a Python implementations of the features provided by `tokenizers`. Basically allows adding tokens. 4. **MistralCommonBackend**: relies on `MistralCommon`'s tokenization library. (Previously known as the `MistralCommonTokenizer`) The `AutoTokenizer` automatically selects the appropriate backend based on available files and dependencies. This is transparent, you continue to use `AutoTokenizer.from_pretrained()` as before. This allows transformers to be future-proof and modular to easily support future backends. ### Defining a tokenizers outside of the existing backends We enable users and tokenizer builders to define their own tokenizers from top to bottom. Tokenizers are usually defined using a backend such as `tokenizers`, `sentencepiece` or `mistral-common`, but we offer the possibility to design the tokenizer at a higher-level, without relying on those backends. To do so, you can import the `PythonBackend` (which was previously known as `PreTrainedTokenizer`). This class encapsulates all the logic related to added tokens, encoding, and decoding. If you want something even higher up the stack, then `PreTrainedTokenizerBase` is what `PythonBackend` inherits from. It contains the very basic tokenizer API features: - `encode` - `decode` - `vocab_size` - `get_vocab` - `convert_tokens_to_ids` - `convert_ids_to_tokens` - `from_pretrained` - `save_pretrained` - among a few others **Note for implementing new tokenizers:** When creating a tokenizer class that loads from SentencePiece files, you can override the `convert_from_spm` class method in your converter to customize vocabulary structure, normalizers, regexes and anything that you would want to be passed to the tokenizers your are converting. This is useful if the model requires specific token ordering or special split regex patterns. See existing converter classes in `convert_slow_tokenizer.py` for examples. ### API Changes #### 1. Direct tokenizer initialization with vocab and merges Starting with v5, we now enable initializing blank, untrained `tokenizers`-backed tokenizers: ```py from transformers import LlamaTokenizer tokenizer = LlamaTokenizer() ``` This tokenizer will therefore follow the definition of the `LlamaTokenizer` as defined in its class definition. It can then be trained on a corpus as can be seen in [the `tokenizers` documentation](https://huggingface.co/docs/tokenizers/training_from_memory). These tokenizers can also be initialized from vocab and merges (if necessary), like the previous "slow" tokenizers: ```py from transformers import LlamaTokenizer vocab = {"<unk>": 0, "<s>": 1, "</s>": 2, "hello": 3, "world": 4} merges = [("h", "e"), ("l", "l"), ("o", " ")] tokenizer = LlamaTokenizer(vocab=vocab, merges=merges) ``` This tokenizer will behave as a Llama-like tokenizer, with an updated vocabulary. This allows comparing different tokenizer classes with the same vocab; therefore enabling the comparison of different pre-tokenizers, normalizers, etc. **Simplified file loading:** Support is added for passing`vocab` and `merges` as file paths directly to tokenizer initialization. The tokenizer will automatically detect the format (SentencePiece `.model`, Tekken `tekken.json`, or plain vocab/merges files) for loading. For BPE tokenizers, if a vocab is provided but no merges, merges will be automatically generated (excluding special tokens). Note: Loading from file paths with `vocab="<path_to_a_file>"`'s primary goal is to allow you to do some quick testing, but for `BPE` models for example we don't check whether you properly passed the merges or not. #### 2. Simplified decoding API The `batch_decode` and `decode` methods have been unified to reflect behavior of the `encode` method. Both single and batch decoding now use the same `decode` method. See an example of the new behavior below: ```python from transformers import AutoTokenizer tokenizer = AutoTokenizer.from_pretrained("t5-small") inputs = ["hey how are you?", "fine"] tokenizer.decode(tokenizer.encode(inputs)) ``` Gives: ```diff - 'hey how are you?</s> fine</s>' + ['hey how are you?</s>', 'fine</s>'] ``` We expect `encode` and `decode` to behave, as two sides of the same coin: `encode`, `process`, `decode`, should work. > [!NOTE] > A common use-case would be: `encode`, `model.generate`, `decode`. However, using `generate` would return `list[list[int]]`, which would then be incompatible with `decode`. #### 3. Unified encoding API The `encode_plus` method is deprecated in favor of the single `__call__` method. #### 4. `apply_chat_template` returns `BatchEncoding` Previously, `apply_chat_template` returned `input_ids` for backward compatibility. Starting with v5, it now consistently returns a `BatchEncoding` dict like other tokenizer methods. ```python # v5 messages = [ {"role": "user", "content": "Hello!"}, {"role": "assistant", "content": "Hi there!"} ] # Now returns BatchEncoding with input_ids, attention_mask, etc. outputs = tokenizer.apply_chat_template(messages, return_tensors="pt") print(outputs.keys()) # dict_keys(['input_ids', 'attention_mask']) ``` #### 5. Removed legacy configuration file saving: We simplify the serialization of tokenization attributes: - `special_tokens_map.json` - special tokens are now stored in `tokenizer_config.json`. - `added_tokens.json` - added tokens are now stored in `tokenizer.json`. - `added_tokens_decoder` is only stored when there is no `tokenizer.json`. - `add_bos_token` and `add_eos_token` - these are no longer saved in `tokenizer_config.json`. When a `tokenizer.json` file exists, these settings are defined in the tokenizer class or `tokenizer.json` itself. **Backend synchronization removed:** The automatic synchronization logic that updated backend tokenizer settings (like `add_prefix_space`, `do_lower_case`, `strip_accents`, `tokenize_chinese_chars`) after initialization has been removed. Tokenizer behavior is now fully determined by the `tokenizer.json` file or class definition at initialization time. When loading older tokenizers, these files are still read for backward compatibility, but new saves use the consolidated format. We're gradually moving towards consolidating attributes to fewer files so that other libraries and implementations may depend on them more reliably. #### 6. Model-Specific Changes Several models that had identical tokenizers now import from their base implementation: - **LayoutLM** → uses BertTokenizer - **LED** → uses BartTokenizer - **Longformer** → uses RobertaTokenizer - **LXMert** → uses BertTokenizer - **MT5** → uses T5Tokenizer - **MVP** → uses BartTokenizer These modules will eventually be removed altogether. **Removed T5-specific workarounds** The internal `_eventually_correct_t5_max_length` method has been removed. T5 tokenizers now handle max length consistently with other models. ### Testing Changes A few testing changes specific to tokenizers have been applied: - Model-specific tokenization test files now focus on integration tests. - Common tokenization API tests (e.g., `add_tokens`, `encode`, `decode`) are now centralized and automatically applied across all tokenizers. This reduces test duplication and ensures consistent behavior For legacy implementations, the original BERT Python tokenizer code (including `WhitespaceTokenizer`, `BasicTokenizer`, etc.) is preserved in `bert_legacy.py` for reference purposes. #### 7. Deprecated / Modified Features **Special Tokens Structure:** - `SpecialTokensMixin`: Merged into `PreTrainedTokenizerBase` to simplify the tokenizer architecture. - `special_tokens_map`: Now only stores named special token attributes (e.g., `bos_token`, `eos_token`). Use `extra_special_tokens` for additional special tokens (formerly `additional_special_tokens`). `all_special_tokens` includes both named and extra tokens. ```python # v4 tokenizer.special_tokens_map # Included 'additional_special_tokens' # v5 tokenizer.special_tokens_map # Only named tokens tokenizer.extra_special_tokens # Additional tokens ``` - `special_tokens_map_extended` and `all_special_tokens_extended`: Removed. Access `AddedToken` objects directly from `_special_tokens_map` or `_extra_special_tokens` if needed. - `additional_special_tokens`: Automatically converted to `extra_special_tokens` during initialization. - `additional_special_tokens_ids`: Removed. Use `extra_special_tokens_ids` instead. - `extra_special_tokens`: Only accepts list/tuple format and is intended for use during tokenizer initialization. For model-s - pyproject.toml (6901 bytes)
[tool.coverage.run] source = ["transformers"] omit = [ "*/convert_*", "*/__main__.py" ] [tool.coverage.report] exclude_lines = [ "pragma: no cover", "raise", "except", "register_parameter" ] [tool.ruff] target-version = "py311" line-length = 119 [tool.ruff.lint] # `PLW1514` is still a preview rule, so `preview` has to be on for it. Pair it with # `explicit-preview-rules` so only the preview rules named in `select` below turn on, # instead of every preview rule of the selected families. preview = true explicit-preview-rules = true select = [ "E", # pycodestyle errors "F", # Pyflakes "I", # isort "W", # pycodestyle warnings "UP", # pyupgrade "FURB", # refurb "SIM", # flake8-simplify "S110", # bandit's try-except-pass rule "C4", # flake8-comprehensions "C901", # McCabe complexity "RUF013", # Checks for the use of implicit Optional in type annotations when the default parameter value is None. "PERF102", # Checks for uses of dict.items() that discard either the key or the value when iterating over the dictionary. "PLC1802", # Checks for len calls on sequences in a boolean test context. "PLC0208", # Checks for iteration over a set literal where each element in the set is itself a literal value. "PIE794", # Checks for duplicate field definitions in classes. "PLW1514", # Checks that `open()` in text mode is called with an explicit encoding. ] ignore = [ "E501", # Checks for lines that exceed the specified maximum character length. "E741", # Checks for the use of the characters 'l', 'O', or 'I' as variable names. "SIM1", # All SIM1XX rules "SIM905", # Checks for static str.split calls that can be replaced with list literals. "UP015", # Checks for redundant open mode arguments. "UP031", # Checks for printf-style string formatting, and offers to replace it with str.format calls. ] extend-safe-fixes = [ "UP006", # Checks for the use of generics that can be replaced with standard library variants based on PEP 585. ] # Codes owned by other checkers, so ruff keeps `# noqa: <code>` for them instead of warning it is invalid. external = [ "NC", # utils/check_noisy_comments.py ] # Ignore import violations in all `__init__.py` files. [tool.ruff.lint.per-file-ignores] "__init__.py" = ["E402", "F401", "F403", "F811"] "examples/**/*.py" = ["C901"] "src/transformers/file_utils.py" = ["F401"] "src/transformers/utils/dummy_*.py" = ["F401"] [tool.ruff.lint.isort] lines-after-imports = 2 known-first-party = ["transformers"] [tool.ruff.format] # Like Black, use double quotes for strings. quote-style = "double" # Like Black, indent with spaces, rather than tabs. indent-style = "space" # Like Black, respect magic trailing commas. skip-magic-trailing-comma = false # Like Black, automatically detect the appropriate line ending. line-ending = "auto" [tool.ruff.lint.mccabe] max-complexity = 75 [tool.pytest.ini_options] addopts = "--doctest-glob='**/*.md'" doctest_optionflags="NUMBER NORMALIZE_WHITESPACE ELLIPSIS" markers = [ "all_flash_attn_test: marks tests related to all mainline flash attentions at once (deselect with '-m \"not all_flash_attn_test\"')", "flash_attn_4_test: marks tests related to flash attention 4 (deselect with '-m \"not flash_attn_4_test\"')", "flash_attn_3_test: marks tests related to flash attention 3 (deselect with '-m \"not flash_attn_3_test\"')", "flash_attn_test: marks tests related to flash attention (deselect with '-m \"not flash_attn_test\"')", "bitsandbytes: select (or deselect with `not`) bitsandbytes integration tests", "generate: marks tests that use the GenerationTesterMixin", "is_training_test: marks tests that use the TrainingTesterMixin (deselect with '-m \"not is_training_test\"')", "is_tensor_parallel_test: marks tests that use the TensorParallelTesterMixin (deselect with '-m \"not is_tensor_parallel_test\"')", "is_expert_parallel_test: marks tests that use the ExpertParallelTesterMixin (deselect with '-m \"not is_expert_parallel_test\"')", "is_fsdp_test: marks tests that use the FSDPTesterMixin (deselect with '-m \"not is_fsdp_test\"')", ] log_cli = 1 log_cli_level = "WARNING" asyncio_default_fixture_loop_scope = "function" filterwarnings = [ # The above pytest-asyncio rule emits unnessecary warnings when it's not installed, so skip it here "ignore:Unknown config option.*asyncio_default_fixture_loop_scope", # Latest sentencepiece (v0.2.1) triggers those warnings when installed due to its build with afflicted version of swig - skip them "ignore:builtin type SwigPyPacked has no __module__ attribute:DeprecationWarning", "ignore:builtin type SwigPyObject has no __module__ attribute:DeprecationWarning", "ignore:builtin type swigvarlink has no __module__ attribute:DeprecationWarning", ] env = [ # Avoid any ReadTimeout when making requests to the Hub (default is 10) # Note: 'D:' means default value from laptop or CI won't be overwritten "D:HF_HUB_DOWNLOAD_TIMEOUT=60", ] [tool.ty] # ty type checker configuration # Using default settings for comprehensive type checking [tool.ty.rules] # Disable specific rules that produce false positives or are too strict for this codebase invalid-method-override = "ignore" # Parameter name differences are acceptable (e.g., x vs input, new_embeddings vs value) not-subscriptable = "ignore" # False positives on tensor slicing (e.g., self.position_ids[:, :seq_length]) no-matching-overload = "ignore" # False positives on torch.zeros and similar functions accepting Size/tuple unsupported-operator = "ignore" # False positives on tuple concatenation with += when properly initialized unresolved-import = "ignore" # Optional dependencies (mlx, torch_npu, habana_frameworks, etc.) checked at runtime call-non-callable = "ignore" # Mixin pattern issues where classes are used as both types and callables unresolved-reference = "ignore" # Forward references with noqa: F821 that ty doesn't respect invalid-argument-type = "ignore" # Complex type narrowing and union type issues not-iterable = "ignore" # Complex async/Future type patterns invalid-return-type = "ignore" # Return type mismatches that would require refactoring deprecated = "ignore" # Deprecation warnings from dependencies invalid-assignment = "ignore" # Low-level assignments that are runtime-safe unused-ignore-comment = "ignore" # Ignore comments that became unnecessary after adding broader per-file-ignores unused-type-ignore-comment = "ignore" # A `# type: ignore` counts as "used" only when its suppressed error fires, which needs optional deps (torch, openai, ...) installed. CI's lean `quality` env lacks them, so directives used locally look unused in CI. - README.md (17010 bytes)
<!--- Copyright 2020 The HuggingFace Team. All rights reserved. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. --> <p align="center"> <picture> <source media="(prefers-color-scheme: dark)" srcset="https://huggingface.co/datasets/huggingface/documentation-images/raw/main/transformers-logo-dark.svg"> <source media="(prefers-color-scheme: light)" srcset="https://huggingface.co/datasets/huggingface/documentation-images/raw/main/transformers-logo-light.svg"> <img alt="Hugging Face Transformers Library" src="https://huggingface.co/datasets/huggingface/documentation-images/raw/main/transformers-logo-light.svg" width="352" height="59" style="max-width: 100%;"> </picture> <br/> <br/> </p> <p align="center"> <a href="https://huggingface.com/models"><img alt="Checkpoints on Hub" src="https://img.shields.io/endpoint?url=https://huggingface.co/api/shields/models&color=brightgreen"></a> <a href="https://circleci.com/gh/huggingface/transformers"><img alt="Build" src="https://img.shields.io/circleci/build/github/huggingface/transformers/main"></a> <a href="https://github.com/huggingface/transformers/blob/main/LICENSE"><img alt="GitHub" src="https://img.shields.io/github/license/huggingface/transformers.svg?color=blue"></a> <a href="https://huggingface.co/docs/transformers/index"><img alt="Documentation" src="https://img.shields.io/website/http/huggingface.co/docs/transformers/index.svg?down_color=red&down_message=offline&up_message=online"></a> <a href="https://github.com/huggingface/transformers/releases"><img alt="GitHub release" src="https://img.shields.io/github/release/huggingface/transformers.svg"></a> <a href="https://github.com/huggingface/transformers/blob/main/CODE_OF_CONDUCT.md"><img alt="Contributor Covenant" src="https://img.shields.io/badge/Contributor%20Covenant-v2.0%20adopted-ff69b4.svg"></a> <a href="https://zenodo.org/badge/latestdoi/155220641"><img src="https://zenodo.org/badge/155220641.svg" alt="DOI"></a> </p> <h4 align="center"> <p> <b>English</b> | <a href="i18n/README_zh-hans.md">简体中文</a> | <a href="i18n/README_zh-hant.md">繁體中文</a> | <a href="i18n/README_ko.md">한국어</a> | <a href="i18n/README_es.md">Español</a> | <a href="i18n/README_ja.md">日本語</a> | <a href="i18n/README_hd.md">हिन्दी</a> | <a href="i18n/README_mr.md">मराठी</a> | <a href="i18n/README_ru.md">Русский</a> | <a href="i18n/README_pt-br.md">Português</a> | <a href="i18n/README_te.md">తెలుగు</a> | <a href="i18n/README_fr.md">Français</a> | <a href="i18n/README_de.md">Deutsch</a> | <a href="i18n/README_it.md">Italiano</a> | <a href="i18n/README_vi.md">Tiếng Việt</a> | <a href="i18n/README_ar.md">العربية</a> | <a href="i18n/README_ur.md">اردو</a> | <a href="i18n/README_bn.md">বাংলা</a> | <a href="i18n/README_fa.md">فارسی</a> | <a href="i18n/README_ro.md">Română</a> | <a href="i18n/README_tr.md">Türkçe</a> </p> </h4> <h3 align="center"> <p>State-of-the-art pretrained models for inference and training</p> </h3> <h3 align="center"> <img src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/transformers_as_a_model_definition.png"/> </h3> Transformers acts as the model-definition framework for state-of-the-art machine learning with text, computer vision, audio, video, and multimodal models, for both inference and training. It centralizes the model definition so that this definition is agreed upon across the ecosystem. `transformers` is the pivot across frameworks: if a model definition is supported, it will be compatible with the majority of training frameworks (Axolotl, Unsloth, DeepSpeed, FSDP, PyTorch-Lightning, ...), inference engines (vLLM, SGLang, TGI, ...), and adjacent modeling libraries (llama.cpp, mlx, ...) which leverage the model definition from `transformers`. We pledge to help support new state-of-the-art models and democratize their usage by having their model definition be simple, customizable, and efficient. There are over 1M+ Transformers [model checkpoints](https://huggingface.co/models?library=transformers&sort=trending) on the [Hugging Face Hub](https://huggingface.co/models) you can use. Explore the [Hub](https://huggingface.co/) today to find a model and use Transformers to help you get started right away. ## Installation Transformers works with Python 3.11+, and [PyTorch](https://pytorch.org/get-started/locally/) 2.6+. Create and activate a virtual environment with [venv](https://docs.python.org/3/library/venv.html) or [uv](https://docs.astral.sh/uv/), a fast Rust-based Python package and project manager. ```py # venv python -m venv .my-env source .my-env/bin/activate # uv uv venv .my-env source .my-env/bin/activate ``` Install Transformers in your virtual environment. ```py # pip pip install "transformers[torch]" # uv uv pip install "transformers[torch]" ``` Install Transformers from source if you want the latest changes in the library or are interested in contributing. However, the *latest* version may not be stable. Feel free to open an [issue](https://github.com/huggingface/transformers/issues) if you encounter an error. ```shell git clone https://github.com/huggingface/transformers.git cd transformers # pip pip install '.[torch]' # uv uv pip install '.[torch]' ``` ## Quickstart Get started with Transformers right away with the [Pipeline](https://huggingface.co/docs/transformers/pipeline_tutorial) API. The `Pipeline` is a high-level inference class that supports text, audio, vision, and multimodal tasks. It handles preprocessing the input and returns the appropriate output. Instantiate a pipeline and specify model to use for text generation. The model is downloaded and cached so you can easily reuse it again. Finally, pass some text to prompt the model. ```py from transformers import pipeline pipeline = pipeline(task="text-generation", model="Qwen/Qwen2.5-1.5B") pipeline("the secret to baking a really good cake is ") [{'generated_text': 'the secret to baking a really good cake is 1) to use the right ingredients and 2) to follow the recipe exactly. the recipe for the cake is as follows: 1 cup of sugar, 1 cup of flour, 1 cup of milk, 1 cup of butter, 1 cup of eggs, 1 cup of chocolate chips. if you want to make 2 cakes, how much sugar do you need? To make 2 cakes, you will need 2 cups of sugar.'}] ``` To chat with a model, the usage pattern is the same. The only difference is you need to construct a chat history (the input to `Pipeline`) between you and the system. > [!TIP] > You can also chat with a model directly from the command line, as long as [`transformers serve` is running](https://huggingface.co/docs/transformers/main/en/serving). > ```shell > transformers chat Qwen/Qwen2.5-0.5B-Instruct > ``` ```py import torch from transformers import pipeline chat = [ {"role": "system", "content": "You are a sassy, wise-cracking robot as imagined by Hollywood circa 1986."}, {"role": "user", "content": "Hey, can you tell me any fun things to do in New York?"} ] pipeline = pipeline(task="text-generation", model="meta-llama/Meta-Llama-3-8B-Instruct", dtype=torch.bfloat16, device_map="auto") response = pipeline(chat, max_new_tokens=512) print(response[0]["generated_text"][-1]["content"]) ``` Expand the examples below to see how `Pipeline` works for different modalities and tasks. <details> <summary>Automatic speech recognition</summary> ```py from transformers import pipeline pipeline = pipeline(task="automatic-speech-recognition", model="openai/whisper-large-v3") pipeline("https://huggingface.co/datasets/Narsil/asr_dummy/resolve/main/mlk.flac") {'text': ' I have a dream that one day this nation will rise up and live out the true meaning of its creed.'} ``` </details> <details> <summary>Image classification</summary> <h3 align="center"> <a><img src="https://huggingface.co/datasets/Narsil/image_dummy/raw/main/parrots.png"></a> </h3> ```py from transformers import pipeline pipeline = pipeline(task="image-classification", model="facebook/dinov2-small-imagenet1k-1-layer") pipeline("https://huggingface.co/datasets/Narsil/image_dummy/raw/main/parrots.png") [{'label': 'macaw', 'score': 0.997848391532898}, {'label': 'sulphur-crested cockatoo, Kakatoe galerita, Cacatua galerita', 'score': 0.0016551691805943847}, {'label': 'lorikeet', 'score': 0.00018523589824326336}, {'label': 'African grey, African gray, Psittacus erithacus', 'score': 7.85409429227002e-05}, {'label': 'quail', 'score': 5.502637941390276e-05}] ``` </details> <details> <summary>Visual question answering</summary> <h3 align="center"> <a><img src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/tasks/idefics-few-shot.jpg"></a> </h3> ```py from transformers import pipeline pipeline = pipeline(task="visual-question-answering", model="Salesforce/blip-vqa-base") pipeline( image="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/tasks/idefics-few-shot.jpg", question="What is in the image?", ) [{'answer': 'statue of liberty'}] ``` </details> ## Why should I use Transformers? 1. Easy-to-use state-of-the-art models: - High performance on natural language understanding & generation, computer vision, audio, video, and multimodal tasks. - Low barrier to entry for researchers, engineers, and developers. - Few user-facing abstractions with just three classes to learn. - A unified API for using all our pretrained models. 1. Lower compute costs, smaller carbon footprint: - Share trained models instead of training from scratch. - Reduce compute time and production costs. - Hundreds of model architectures with 1M+ pretrained checkpoints across all modalities. 1. Choose the right framework for every part of a model's lifetime: - Train state-of-the-art models in 3 lines of code. - Move a single model between PyTorch/JAX/TF2.0 frameworks at will. - Pick the right framework for training, evaluation, and production. 1. Easily customize a model or an example to your needs: - We provide examples for each architecture to reproduce the results published by its original authors. - Model internals are exposed as consistently as possible. - Model files can be used independently of the library for quick experiments. <a target="_blank" href="https://huggingface.co/enterprise"> <img alt="Hugging Face Enterprise Hub" src="https://github.com/user-attachments/assets/247fb16d-d251-4583-96c4-d3d76dda4925"> </a><br> ## When shouldn't I use Transformers? - This library is not a modular toolbox of building blocks for neural nets. The code in the model files is not refactored with additional abstractions on purpose, so that researchers can quickly iterate on each of the models without diving into additional abstractions/files. - The training API is optimized to work with PyTorch models provided by Transformers. For generic machine learning loops, you should use another library like [Accelerate](https://huggingface.co/docs/accelerate). ## 100 projects using Transformers Transformers is more than a toolkit to use pretrained models, it's a community of projects built around it and the Hugging Face Hub. We want Transformers to enable developers, researchers, students, professors, engineers, and anyone else to build their dream projects. In order to celebrate Transformers 100,000 stars, we wanted to put the spotlight on the community with the [awesome-transformers](./awesome-transformers.md) page which lists 100 incredible projects built with Transformers. If you own or use a project that you believe should be part of the list, please open a PR to add it! ## Example models You can test most of our models directly on their [Hub model pages](https://huggingface.co/models). Expand each modality below to see a few example models for various use cases. <details> <summary>Audio</summary> - Audio classification with [CLAP](https://huggingface.co/laion/clap-htsat-fused) - Automatic speech recognition with [Parakeet](https://huggingface.co/nvidia/parakeet-ctc-1.1b#transcribing-using-transformers-%F0%9F%A4%97), [Whisper](https://huggingface.co/openai/whisper-large-v3-turbo), [GLM-ASR](https://huggingface.co/zai-org/GLM-ASR-Nano-2512) and [Moonshine-Streaming](https://huggingface.co/UsefulSensors/moonshine-streaming-medium) - Keyword spotting with [Wav2Vec2](https://huggingface.co/superb/wav2vec2-base-superb-ks) - Speech to speech generation with [Moshi](https://huggingface.co/kyutai/moshiko-pytorch-bf16) - Text to audio with [MusicGen](https://huggingface.co/facebook/musicgen-large) - Text to speech with [CSM](https://huggingface.co/sesame/csm-1b) </details> <details> <summary>Computer vision</summary> - Automatic mask generation with [SAM](https://huggingface.co/facebook/sam-vit-base) - Depth estimation with [DepthPro](https://huggingface.co/apple/DepthPro-hf) - Image classification with [DINO v2](https://huggingface.co/facebook/dinov2-base) - Keypoint detection with [SuperPoint](https://huggingface.co/magic-leap-community/superpoint) - Keypoint matching with [SuperGlue](https://huggingface.co/magic-leap-community/superglue_outdoor) - Object detection with [RT-DETRv2](https://huggingface.co/PekingU/rtdetr_v2_r50vd) - Pose Estimation with [VitPose](https://huggingface.co/usyd-community/vitpose-base-simple) - Universal segmentation with [OneFormer](https://huggingface.co/shi-labs/oneformer_ade20k_swin_large) - Video classification with [VideoMAE](https://huggingface.co/MCG-NJU/videomae-large) </details> <details> <summary>Multimodal</summary> - Audio or text to text with [Voxtral](https://huggingface.co/mistralai/Voxtral-Mini-3B-2507), [Audio Flamingo](https://huggingface.co/nvidia/audio-flamingo-3-hf) - Document question answering with [LayoutLMv3](https://huggingface.co/microsoft/layoutlmv3-base) - Image or text to text with [Qwen-VL](https://huggingface.co/Qwen/Qwen2.5-VL-3B-Instruct) - Image captioning [BLIP-2](https://huggingface.co/Salesforce/blip2-opt-2.7b) - OCR-based document understanding with [GOT-OCR2](https://huggingface.co/stepfun-ai/GOT-OCR-2.0-hf) - Table question answering with [TAPAS](https://huggingface.co/google/tapas-base) - Unified multimodal understanding and generation with [Emu3](https://huggingface.co/BAAI/Emu3-Gen) - Vision to text with - SECURITY.md (9246 bytes)
# Security Policy ## Hugging Face Hub, remote artefacts, and remote code Transformers is open-source software that is tightly coupled to the Hugging Face Hub. While you have the ability to use it offline with pre-downloaded model weights, it provides a very simple way to download, use, and manage models locally. When downloading artefacts that have been uploaded by others on any platform, you expose yourself to risks. Please read below for the security recommendations in order to keep your runtime and local environment safe. ### Remote artefacts Models uploaded on the Hugging Face Hub come in different formats. We heavily recommend uploading and downloading models in the [`safetensors`](https://github.com/huggingface/safetensors) format (which is the default prioritized by the transformers library), as developed specifically to prevent arbitrary code execution on your system. To avoid loading models from unsafe formats (e.g. [pickle](https://docs.python.org/3/library/pickle.html), you should use the `use_safetensors` parameter. If doing so, in the event that no .safetensors file is present, transformers will error when loading the model. ### Remote code #### Modeling Transformers supports many model architectures, but is also the bridge between your Python runtime and models that are stored in model repositories on the Hugging Face Hub. These models require the `trust_remote_code=True` parameter to be set when using them; please **always** verify the content of the modeling files when using this argument. We recommend setting a revision in order to ensure you protect yourself from updates on the repository. ## Supported Versions Security fixes ship in the latest release. Please reproduce on the current released version before reporting. ## Reporting a Vulnerability Report privately — **do not open a public issue or PR for a suspected vulnerability.** - **Preferred:** GitHub private vulnerability reporting — the **"Report a vulnerability"** button under this repo's **Security** tab. This routes to the maintainers, keeps the report private until a fix is ready, and lets us issue a CVE through GitHub if warranted. - **Email:** [email protected] We acknowledge valid, in-scope reports and will keep you updated on remediation. Please give us a reasonable window to fix before any public disclosure. ## Recognition We do not offer a monetary bounty. For a valid, in-scope report we credit you on the published GitHub Security Advisory and name you as the reporter in the associated CVE. Let us know how you'd like to be credited (name or handle). ## What your report must include We receive a high volume of reports. To be triaged, a report **must** follow the structure below. Copy this block into your submission and fill in every field. Reports missing the version, the proof of concept, or the impact are returned as incomplete and are not investigated until provided. ```markdown ### Summary One sentence: what the vulnerability is and where. ### Affected version / commit Exact released version or commit SHA you reproduced on (e.g. v5.22.2 / a1b2c3d). Not "latest" or "main". ### Affected component The public API, module, or entry point involved (e.g. `AutoModel.from_pretrained`, `apply_chat_template`). ### Vulnerability class Type and CWE if known (e.g. deserialization / CWE-502, path traversal / CWE-22). ### Attack vector & preconditions - How is the vulnerable code reached? (which API call / input / config) - Who is the attacker and what do they control? - What must be true for the attack to work? (auth, a user action, a non-default setting, a malicious file being loaded, etc.) ### Proof of concept A minimal, self-contained script or step sequence that runs on a clean install of the version above. Include: - the exact commands / code to run, - any input files needed (attach them, or give a script that generates them), - the **expected** behavior vs. the **actual** behavior you observed. A snippet showing that a function *exists* or *could* be misused is not a PoC. ### Impact What an attacker gains in a realistic deployment. "Could theoretically…" without a working chain is not an impact. ### Scope Which trust boundary (see below) does this cross? If your finding touches anything in the "Out of scope" list, name which item and explain why it is nonetheless a violation of a guarantee we make. ### Suggested severity (optional) We assign the final severity. Include a CVSS v3.1 vector only if you have one. ### Suggested fix (optional) ``` > The bar is a **reproducible PoC against a supported version, with a concrete impact that crosses a trust boundary we actually defend** (see scope below). Reports that are theoretical, auto-generated by a scanner or LLM, or that restate documented behavior will be closed without detailed review. ## Threat model & trust boundaries Understanding these saves everyone time — most of what we close as not-a-vulnerability falls inside one of them. **Loading artifacts you did not create is a trust decision.** Models, datasets, tokenizers, and configuration files can carry code or instructions that run when loaded. That a maliciously crafted artifact can execute code, read local files, or otherwise act when you load it is the **documented risk of loading untrusted content** — not a vulnerability in this library. Protect yourself by: - preferring the `safetensors` format over pickle-based formats (see [Remote artefacts](#remote-artefacts)), - pinning a specific, reviewed revision, - enabling remote-code execution (`trust_remote_code=True`) only for repositories whose modeling files you have inspected. We *will* treat as a vulnerability anything that breaks one of these protections — e.g. code executing despite `safetensors`-only loading, or a pinned revision being bypassed. **Chat templates are rendered in a sandbox.** `apply_chat_template` renders templates in a sandboxed Jinja environment so that a template from an untrusted repository cannot run code on your machine. An escape from that sandbox (running code, reading files, or reaching Python objects outside the template's inputs) is in scope. A template that renders a misleading prompt, loops, or uses excessive memory is not. **Conversion, CLI, and developer utilities are operator tools.** Scripts you run yourself against inputs you chose (checkpoint conversion scripts, training/utility CLIs, dev helpers) are not a library API attack surface. Behaviors such as a script deserializing a file you pointed it at are within the operator's trust. ## In scope We treat as vulnerabilities issues in the **published package code** — the library's own API surface — that an attacker can trigger without the victim having opted into a documented risk. For example: - code execution, memory corruption, or file access reachable through a normal API call on input that is **not** an untrusted model/artifact the user chose to load; - a control we advertise being bypassed (e.g. code running despite `safetensors`-only loading, an escape from the chat-template sandbox, or a pinned revision being ignored); - exposure or mishandling of credentials, tokens, or another user's data by the library; - CI/CD or supply-chain issues in this repository. ## Out of scope The following are **not** treated as vulnerabilities in `transformers`. If your finding touches one of these, the report must explain why it is nonetheless a violation of a guarantee we make — otherwise it will be closed. - Issues that require loading an untrusted artifact and amount to the documented load-time risk above (code execution / file access on load of a malicious model, dataset, config, or pickle). An escape from the chat-template sandbox is the exception, see above. - Anything that requires `trust_remote_code=True`: running the repository's code is what that flag does. - Findings in the repository's top-level `examples/`, `scripts/` and `utils/` folders, documentation, tests, or other non-packaged reference material. - Local denial-of-service from feeding pathological input to a function on your own machine (high memory, slow parse, panic), absent a multi-tenant or remote-service impact. - Model behavior: jailbreaks, alignment failures, prompt injection, or harmful generations. Model weights are authored by their uploaders; report these to the model owner. - Vulnerabilities in third-party dependencies we do not vendor — report upstream (we'll bump once fixed). - Theoretical issues without a working proof of concept, and reports auto-generated from scanners or LLMs without a verified, reproducible chain. - Best-practice or hardening suggestions with no demonstrated impact — missing email-authentication or transport records (MTA-STS, TLS-RPT, DMARC/SPF tuning), missing HTTP security headers, TLS configuration preferences, and similar scanner or config-checker output presented without a working exploit chain. ## Safe harbor Good-faith research that respects these guidelines, avoids privacy violations and service disruption, and gives us a reasonable disclosure window will not be pursued by us. Do not access data that isn't yours and do not run tests against Hugging Face production infrastructure.
- setup.py (12743 bytes)
# Copyright 2021 The HuggingFace Team. All rights reserved. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. """ Simple check list from AllenNLP repo: https://github.com/allenai/allennlp/blob/main/setup.py To create the package for pypi. 1. Create the release branch named: v<RELEASE>-release, for example v4.19-release. For a patch release checkout the current release branch. If releasing on a special branch, copy the updated README.md on the main branch for the commit you will make for the post-release and run `make fix-repo` on the main branch as well. 2. Run `make pre-release` (or `make pre-patch` for a patch release) and commit these changes with the message: "Release: <VERSION>" and push. 3. Go back to the main branch and run `make post-release` then `make fix-repo`. Commit these changes with the message "v<NEXT_VERSION>.dev.0" and push to main. # If you were just cutting the branch in preparation for a release, you can stop here for now. 4. Wait for the tests on the release branch to be completed and be green (otherwise revert and fix bugs) 5. On the release branch, add a tag in git to mark the release: "git tag v<VERSION> -m 'Adds tag v<VERSION> for pypi' " Push the tag to git: git push --tags origin v<RELEASE>-release 6. Have a core maintainer review and approve the deployment to pypi. """ import re import shutil import sys from pathlib import Path from setuptools import Command, find_packages, setup # Supported Python version range (min, max) SUPPORTED_PYTHON_VERSIONS = (11, 14) # 3.11 to 3.14 PYTHON_MINOR_VERSION = sys.version_info.minor # Remove stale transformers.egg-info directory to avoid https://github.com/pypa/pip/issues/5466 stale_egg_info = Path(__file__).parent / "transformers.egg-info" if stale_egg_info.exists(): print( f"Warning: {stale_egg_info} exists.\n\n" "If you recently updated transformers to 3.0 or later, this is expected,\n" "but it may prevent transformers from installing in editable mode.\n\n" "This directory is automatically generated by Python's packaging tools.\n" "I will remove it now.\n\n" "See https://github.com/pypa/pip/issues/5466 for details.\n" ) shutil.rmtree(stale_egg_info) # IMPORTANT: # 1. all dependencies should be listed here with their version requirements if any # 2. once modified, run: `make fix-repo` to update src/transformers/dependency_versions_table.py _deps = [ "Pillow>=10.0.1,<=15.0", "accelerate>=1.1.0", "av", "beautifulsoup4", "blobfile", "codecarbon>=2.8.1", "datasets>=2.15.0", # We need either this pin or pyarrow<21.0.0 "deepspeed>=0.9.3", "diffusers==0.35.2", # Needed for VibeVoice TTS integration tests "faiss-cpu", "fastapi", "filelock", "fugashi>=1.0", "GitPython<3.1.19", "hf-doc-builder>=0.6.0", "huggingface-hub>=1.31.0,<3.0", "ipadic>=1.0.0,<2.0", "jinja2>=3.1.6", "kernels>=0.18.2,<0.19", "librosa", "mistral-common[image]>=1.11.7", "nltk<=3.8.1", "num2words", "numpy>=1.17", "openai>=1.98.0", "opencv-python", "optimum-benchmark>=0.3.0", "optuna", "packaging>=20.0", "parameterized>=0.9", # older version of parameterized cause pytest collection to fail on .expand "peft>=0.22.0", "phonemizer", "protobuf", "psutil", "pyyaml>=5.1", "pydantic>=2", "pytest>=7.2.0,<9.0.0", "pytest-asyncio>=1.2.0", "pytest-random-order", "pytest-rerunfailures<16.0", "pytest-timeout", "pytest-env", "pytest-xdist", "pytest-order", "python>=3.11.0", "regex>=2025.10.22", "rhoknp>=1.1.0,<1.3.1", "rjieba", "ruff==0.14.10", # When bumping `transformers-mlinter`, sync repo-local rule overrides from # `utils/rules.toml` back into the released package. "transformers-mlinter==0.1.6", "ty==0.0.20", "sacremoses", "safetensors>=0.8.0", "sagemaker>=2.31.0", "schedulefree>=1.2.6", "scikit-learn", "scipy", "sentencepiece>=0.1.91,!=0.1.92", "soundfile>=0.13.0", # first version with wheels on every supported platform and Python "soxr>=0.5.0", # first version with wheels on every supported platform and Python "starlette", "sudachipy>=0.6.6", "sudachidict_core>=20220729", "tensorboard", "timeout-decorator", "tiktoken", "timm>=1.0.23", "tokenizers>=0.23.1,<0.24.0", "torch>=2.6", "torchaudio", "torchvision", "tqdm>=4.60", "transformers-mic[runtime]>=0.1.3", "typer", "unidic>=1.0.2", "unidic_lite>=1.0.7", "urllib3<2.0.0", "uvicorn", "pytest-rich", "libcst", "rich", "ray[tune]>=2.7.0", ] # This is a lookup table with items like: {"tokenizers": "tokenizers==0.9.4", "packaging": "packaging"}, i.e. # some of the values are versioned whereas others aren't. deps = {b: a for a, b in (re.findall(r"^(([^!=<>~ ]+)(?:[!=<>~ ].*)?$)", x)[0] for x in _deps)} def deps_list(*pkgs): return [deps[pkg] for pkg in pkgs] extras = {} extras["torch"] = deps_list("torch", "accelerate") extras["vision"] = deps_list("torchvision", "Pillow") extras["audio"] = deps_list("torchaudio", "soundfile", "soxr") extras["video"] = deps_list("av") extras["timm"] = deps_list("timm") extras["quality"] = deps_list( "datasets", "ruff", "GitPython", "urllib3", "libcst", "rich", "ty", "transformers-mlinter" ) extras["docs"] = deps_list("hf-doc-builder") extras["kernels"] = deps_list("kernels") extras["sentencepiece"] = deps_list("sentencepiece", "protobuf") extras["tiktoken"] = deps_list("tiktoken", "blobfile") extras["mistral-common"] = deps_list("mistral-common[image]") extras["chat_template"] = deps_list("jinja2") extras["sklearn"] = deps_list("scikit-learn") extras["accelerate"] = deps_list("accelerate") extras["retrieval"] = deps_list("faiss-cpu", "datasets") extras["sagemaker"] = deps_list("sagemaker") extras["deepspeed"] = deps_list("deepspeed", "accelerate") extras["optuna"] = deps_list("optuna") extras["integrations"] = deps_list("kernels", "optuna", "codecarbon") if PYTHON_MINOR_VERSION < 14: extras["ray"] = deps_list("ray[tune]") extras["integrations"] += extras["ray"] extras["codecarbon"] = deps_list("codecarbon") extras["serving"] = deps_list("openai", "pydantic", "uvicorn", "fastapi", "starlette", "rich") + extras["torch"] extras["num2words"] = deps_list("num2words") extras["benchmark"] = deps_list("optimum-benchmark") extras["ja"] = deps_list("fugashi", "ipadic", "unidic_lite", "unidic", "rhoknp") if PYTHON_MINOR_VERSION < 14: extras["ja"] += deps_list("sudachipy", "sudachidict_core") extras["testing"] = ( deps_list( "pytest", "pytest-asyncio", "pytest-random-order", "pytest-rich", "pytest-xdist", "pytest-order", "pytest-rerunfailures", "pytest-timeout", "pytest-env", "timeout-decorator", "transformers-mic[runtime]", "parameterized", "psutil", "diffusers", # Needed for VibeVoice TTS integration tests "librosa", # reference implementation in audio tests "phonemizer", # Wav2Vec2Phoneme / VITS tokenizer tests (also needs the system `espeak-ng`) "nltk", "sacremoses", "rjieba", "beautifulsoup4", "tensorboard", "filelock", # filesystem locks, e.g., to prevent parallel downloads ) + extras["docs"] + extras["quality"] + extras["retrieval"] + extras["sentencepiece"] + extras["serving"] ) extras["testing"] += extras["mistral-common"] extras["deepspeed-testing"] = extras["deepspeed"] + extras["testing"] + extras["optuna"] + extras["sentencepiece"] extras["all"] = ( extras["torch"] + extras["vision"] + extras["audio"] + extras["video"] + extras["kernels"] + extras["timm"] + extras["sentencepiece"] + extras["tiktoken"] + extras["chat_template"] + extras["num2words"] ) extras["all"] += extras["mistral-common"] extras["dev"] = extras["all"] + extras["testing"] + extras["ja"] + extras["sklearn"] # Those define the hard dependencies of `transformers` install_requires = [ deps["huggingface-hub"], deps["numpy"], deps["packaging"], # utilities from PyPA to e.g., compare versions deps["pyyaml"], # used for the model cards metadata deps["regex"], # for OpenAI GPT deps["tokenizers"], deps["typer"], # CLI utilities. In practice, already a dependency of huggingface_hub but we use it as well deps["safetensors"], deps["tqdm"], # progress bars in model download and training scripts ] class DepsTableUpdateCommand(Command): """ A custom distutils command that updates the dependency table. usage: python setup.py deps_table_update """ description = "build runtime dependency table" user_options = [ # format: (long option, short option, description). ("dep-table-update", None, "updates src/transformers/dependency_versions_table.py"), ] def initialize_options(self): pass def finalize_options(self): pass def run(self): if SUPPORTED_PYTHON_VERSIONS[0] > PYTHON_MINOR_VERSION: print( f"Table updated only when running 3.{SUPPORTED_PYTHON_VERSIONS[0]}.x, detected version is {sys.version}." ) return entries = "\n".join([f' "{k}": "{v}",' for k, v in deps.items()]) content = [ "# THIS FILE HAS BEEN AUTOGENERATED. To update:", "# 1. modify the `_deps` dict in setup.py", "# 2. run `make fix-repo``", "deps = {", entries, "}", "", ] target = "src/transformers/dependency_versions_table.py" with open(target, "w", encoding="utf-8", newline="\n") as f: f.write("\n".join(content)) if __name__ == "__main__": # Generate python_requires from supported version range min_version, max_version = SUPPORTED_PYTHON_VERSIONS python_requires = f">=3.{min_version}.0" # Generate Python version classifiers dynamically python_classifiers = ["Programming Language :: Python :: 3"] for minor in range(min_version, max_version + 1): python_classifiers.append(f"Programming Language :: Python :: 3.{minor}") setup( name="transformers", version="5.24.0.dev0", # expected format is one of x.y.z.dev0, or x.y.z.rc1 or x.y.z (no to dashes, yes to dots) author="The Hugging Face team (past and future) with the help of all our contributors (https://github.com/huggingface/transformers/graphs/contributors)", author_email="[email protected]", description="Transformers: the model-definition framework for state-of-the-art machine learning models in text, vision, audio, and multimodal models, for both inference and training.", long_description=open("README.md", "r", encoding="utf-8").read(), long_description_content_type="text/markdown", keywords="machine-learning nlp python pytorch transformer llm vlm deep-learning inference training model-hub pretrained-models llama gemma qwen", license="Apache 2.0 License", url="https://github.com/huggingface/transformers", package_dir={"": "src"}, packages=find_packages("src"), include_package_data=True, package_data={"": ["**/*.cu", "**/*.cpp", "**/*.cuh", "**/*.h", "**/*.pyx", "py.typed"]}, zip_safe=False, extras_require=extras, entry_points={"console_scripts": ["transformers=transformers.cli.transformers:main"]}, python_requires=python_requires, install_requires=list(install_requires), classifiers=[ "Development Status :: 5 - Production/Stable", "Intended Audience :: Developers", "Intended Audience :: Education", "Intended Audience :: Science/Research", "Operating System :: OS Independent", ] + python_classifiers + [ "Topic :: Scientific/Engineering :: Artificial Intelligence", ], cmdclass={"deps_table_update": DepsTableUpdateCommand}, ) - src/transformers/__init__.py (39148 bytes)
# Copyright 2020 The HuggingFace Team. All rights reserved. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # When adding a new object to this init, remember to add it twice: once inside the `_import_structure` dictionary and # once inside the `if TYPE_CHECKING` branch. The `TYPE_CHECKING` should have import statements as usual, but they are # only there for type checking. The `_import_structure` is a dictionary submodule to list of object names, and is used # to defer the actual importing for when the objects are requested. This way `import transformers` provides the names # in the namespace without actually importing anything (and especially none of the backends). __version__ = "5.24.0.dev0" import re import sys from pathlib import Path from typing import TYPE_CHECKING # Check the dependencies satisfy the minimal versions required. from . import dependency_versions_check from .utils import ( OptionalDependencyNotAvailable, _LazyModule, is_essentia_available, is_g2p_en_available, is_librosa_available, is_mistral_common_available, is_mlx_available, is_numba_available, is_pretty_midi_available, ) # Note: the following symbols are deliberately exported with `as` # so that mypy, pylint or other static linters can recognize them, # given that they are not exported using `__all__` in this file. from .utils import is_bitsandbytes_available as is_bitsandbytes_available from .utils import is_scipy_available as is_scipy_available from .utils import is_sentencepiece_available as is_sentencepiece_available from .utils import is_speech_available as is_speech_available from .utils import is_timm_available as is_timm_available from .utils import is_tokenizers_available as is_tokenizers_available from .utils import is_torch_available as is_torch_available from .utils import is_torchaudio_available as is_torchaudio_available from .utils import is_torchvision_available as is_torchvision_available from .utils import is_vision_available as is_vision_available from .utils import logging as logging from .utils.import_utils import _LegacyModuleAliasFinder, define_import_structure logger = logging.get_logger(__name__) # pylint: disable=invalid-name # Base objects, independent of any specific backend _import_structure = { "audio_utils": [], "cli": [], "configuration_utils": ["PreTrainedConfig", "PretrainedConfig"], "convert_slow_tokenizers_checkpoints_to_fast": [], "data": [], "data.data_collator": [ "DataCollator", "DataCollatorForLanguageModeling", "DataCollatorForMultipleChoice", "DataCollatorForPermutationLanguageModeling", "DataCollatorForSeq2Seq", "DataCollatorForTokenClassification", "DataCollatorWithFlattening", "DataCollatorWithPadding", "DefaultDataCollator", "default_data_collator", ], "debug_utils": [], "decision": ["DecisionConfig"], "dependency_versions_check": [], "dependency_versions_table": [], "distributed": ["DistributedConfig"], "dynamic_module_utils": [], "exporters": [], "feature_extraction_sequence_utils": ["SequenceFeatureExtractor"], "feature_extraction_utils": ["BatchFeature", "FeatureExtractionMixin"], "file_utils": [], "generation": [ "AsyncTextIteratorStreamer", "CompileConfig", "ContinuousBatchingConfig", "GenerationConfig", "TextDiffusionStreamer", "TextIteratorStreamer", "TextStreamer", "WatermarkingConfig", ], "hf_argparser": ["HfArgumentParser"], "hyperparameter_search": [], "image_processing_utils_fast": [], "image_transforms": [], "integrations": [ "is_clearml_available", "is_comet_available", "is_dvclive_available", "is_neptune_available", "is_optuna_available", "is_ray_available", "is_ray_tune_available", "is_swanlab_available", "is_tensorboard_available", "is_trackio_available", "is_wandb_available", ], "loss": [], "pipelines": [ "AnyToAnyPipeline", "AudioClassificationPipeline", "AutomaticSpeechRecognitionPipeline", "CsvPipelineDataFormat", "DecisionPipeline", "DepthEstimationPipeline", "DocumentQuestionAnsweringPipeline", "FeatureExtractionPipeline", "FillMaskPipeline", "ImageClassificationPipeline", "ImageFeatureExtractionPipeline", "ImageSegmentationPipeline", "ImageTextToTextPipeline", "JsonPipelineDataFormat", "KeypointMatchingPipeline", "MaskGenerationPipeline", "NerPipeline", "ObjectDetectionPipeline", "PipedPipelineDataFormat", "Pipeline", "PipelineDataFormat", "TableQuestionAnsweringPipeline", "TextClassificationPipeline", "TextGenerationPipeline", "TextToAudioPipeline", "TokenClassificationPipeline", "VideoClassificationPipeline", "ZeroShotAudioClassificationPipeline", "ZeroShotClassificationPipeline", "ZeroShotImageClassificationPipeline", "ZeroShotObjectDetectionPipeline", "pipeline", ], "processing_utils": [ "AudioKwargs", "ImagesKwargs", "ProcessingKwargs", "ProcessorMixin", "TextKwargs", "VideosKwargs", ], "quantizers": [], "testing_utils": [], "tokenization_python": ["PreTrainedTokenizer", "PythonBackend"], "tokenization_utils": [], "tokenization_utils_base": [ "AddedToken", "BatchEncoding", "CharSpan", "PreTrainedTokenizerBase", "TokenSpan", ], "tokenization_utils_fast": [], "tokenization_utils_sentencepiece": ["SentencePieceBackend"], "trainer_callback": [ "DefaultFlowCallback", "EarlyStoppingCallback", "PrinterCallback", "ProgressCallback", "TrainerCallback", "TrainerControl", "TrainerState", ], "trainer_utils": [ "EvalPrediction", "IntervalStrategy", "SchedulerType", "enable_full_determinism", "set_seed", ], "training_args": ["TrainingArguments"], "training_args_seq2seq": ["Seq2SeqTrainingArguments"], "utils": [ "CONFIG_NAME", "MODEL_CARD_NAME", "SPIECE_UNDERLINE", "WEIGHTS_NAME", "TensorType", "add_end_docstrings", "add_start_docstrings", "is_apex_available", "is_av_available", "is_bitsandbytes_available", "is_datasets_available", "is_faiss_available", "is_matplotlib_available", "is_mlx_available", "is_phonemizer_available", "is_psutil_available", "is_py3nvml_available", "is_pyctcdecode_available", "is_sacremoses_available", "is_scipy_available", "is_sentencepiece_available", "is_sklearn_available", "is_speech_available", "is_timm_available", "is_tokenizers_available", "is_torch_available", "is_torch_hpu_available", "is_torch_mlu_available", "is_torch_musa_available", "is_torch_neuroncore_available", "is_torch_npu_available", "is_torch_supa_available", "is_torchvision_available", "is_torch_xla_available", "is_torch_xpu_available", "is_vision_available", "logging", ], "utils.import_utils": ["requires_backends"], "utils.kernel_config": ["KernelConfig"], "utils.quantization_config": [ "AqlmConfig", "AutoRoundConfig", "AwqConfig", "BitNetQuantConfig", "BitsAndBytesConfig", "CompressedTensorsConfig", "EetqConfig", "FbgemmFp8Config", "FineGrainedFP8Config", "FourOverSixConfig", "FPQuantConfig", "GemmaQuantizationConfig", "GgufConfig", "GPTQConfig", "HiggsConfig", "HqqConfig", "MetalConfig", "Mxfp4Config", "NVFP4Config", "QuantoConfig", "QuarkConfig", "SinqConfig", "SpQRConfig", "TorchAoConfig", "VptqConfig", ], "video_utils": [], } # tokenizers-backed objects try: if not is_tokenizers_available(): raise OptionalDependencyNotAvailable() except OptionalDependencyNotAvailable: from .utils import dummy_tokenizers_objects _import_structure["utils.dummy_tokenizers_objects"] = [ name for name in dir(dummy_tokenizers_objects) if not name.startswith("_") ] else: # Fast tokenizers structure _import_structure["tokenization_utils_tokenizers"] = [ "PreTrainedTokenizerFast", "TokenizersBackend", ] try: if not (is_sentencepiece_available() and is_tokenizers_available()): raise OptionalDependencyNotAvailable() except OptionalDependencyNotAvailable: from .utils import dummy_sentencepiece_and_tokenizers_objects _import_structure["utils.dummy_sentencepiece_and_tokenizers_objects"] = [ name for name in dir(dummy_sentencepiece_and_tokenizers_objects) if not name.startswith("_") ] else: _import_structure["convert_slow_tokenizer"] = [ "SLOW_TO_FAST_CONVERTERS", "convert_slow_tokenizer", ] try: if not (is_mistral_common_available()): raise OptionalDependencyNotAvailable() except OptionalDependencyNotAvailable: from .utils import dummy_mistral_common_objects _import_structure["utils.dummy_mistral_common_objects"] = [ name for name in dir(dummy_mistral_common_objects) if not name.startswith("_") ] else: _import_structure["tokenization_mistral_common"] = ["MistralCommonBackend"] # Vision-specific objects try: if not is_vision_available(): raise OptionalDependencyNotAvailable() except OptionalDependencyNotAvailable: from .utils import dummy_vision_objects _import_structure["utils.dummy_vision_objects"] = [ name for name in dir(dummy_vision_objects) if not name.startswith("_") ] else: _import_structure["image_processing_backends"] = ["PilBackend"] _import_structure["image_processing_base"] = ["ImageProcessingMixin"] _import_structure["image_processing_utils"] = ["BaseImageProcessor"] _import_structure["image_utils"] = ["ImageFeatureExtractionMixin"] try: if not is_torchvision_available(): raise OptionalDependencyNotAvailable() except OptionalDependencyNotAvailable: from .utils import dummy_torchvision_objects _import_structure["utils.dummy_torchvision_objects"] = [ name for name in dir(dummy_torchvision_objects) if not name.startswith("_") ] else: _import_structure.setdefault("image_processing_backends", []) _import_structure["image_processing_backends"] += ["TorchvisionBackend"] _import_structure["video_processing_utils"] = ["BaseVideoProcessor", "VideoProcessorMixin"] # PyTorch-backed objects try: if not is_torch_available(): raise OptionalDependencyNotAvailable() except OptionalDependencyNotAvailable: from .utils import dummy_pt_objects _import_structure["utils.dummy_pt_objects"] = [name for name in dir(dummy_pt_objects) if not name.startswith("_")] else: _import_structure["activations"] = [] _import_structure["backbone_utils"] = ["BackboneConfigMixin", "BackboneMixin"] _import_structure["cache_utils"] = [ "Cache", "CacheLayerMixin", "DynamicCache", "DynamicIndexedLayer", "DynamicLayer", "EncoderDecoderCache", "Fp8QuantizedLayer", "HQQQuantizedLayer", "QuantizedCache", "QuantoQuantizedLayer", "StaticCache", "StaticIndexedLayer", "StaticLayer", "StaticSlidingWindowLayer", ] _import_structure["core_model_loading"] = [ "Chunk", "Concatenate", "ConversionOps", "GroupWeightRename", "MergeModulelist", "PermuteForRope", "SplitModulelist", "VisionFuseAndPermuteForRope", "VisionUnfuseAndPermuteForRope", "WeightConverter", ] _import_structure["generation"].extend( [ "AlternatingCodebooksLogitsProcessor", "BayesianDetectorConfig", "BayesianDetectorModel", "ClassifierFreeGuidanceLogitsProcessor", "ContinuousBatchingManager", "ContinuousMixin", "EncoderNoRepeatNGramLogitsProcessor", "EncoderRepetitionPenaltyLogitsProcessor", "EosTokenCriteria", "EpsilonLogitsWarper", "EtaLogitsWarper", "ExponentialDecayLengthPenalty", "ForcedBOSTokenLogitsProcessor", "ForcedEOSTokenLogitsProcessor", "GenerationMixin", "InfNanRemoveLogitsProcessor", "LogitNormalization", "LogitsProcessor", "LogitsProcessorList", "MaxLengthCriteria", "MaxTimeCriteria", "MinLengthLogitsProcessor", "MinNewTokensLengthLogitsProcessor", "MinPLogitsWarper", "NoBadWordsLogitsProcessor", "NoRepeatNGramLogitsProcessor", "PrefixConstrainedLogitsProcessor", "RepetitionPenaltyLogitsProcessor", "SequenceBiasLogitsProcessor", "StoppingCriteria", "StoppingCriteriaList", "StopStringCriteria", "SuppressTokensAtBeginLogitsProcessor", "SuppressTokensLogitsProcessor", "SynthIDTextWatermarkDetector", "SynthIDTextWatermarkingConfig", "SynthIDTextWatermarkLogitsProcessor", "TemperatureLogitsWarper", "TopHLogitsWarper", "TopKLogitsWarper", "TopPLogitsWarper", "TypicalLogitsWarper", "UnbatchedClassifierFreeGuidanceLogitsProcessor", "WatermarkDetector", "WatermarkLogitsProcessor", "WhisperTimeStampLogitsProcessor", ] ) # PyTorch domain libraries integration _import_structure["integrations.executorch"] = [ "TorchExportableModuleWithStaticCache", "convert_and_export_with_cache", ] _import_structure["integrations.hub_kernels"] = ["kernelize"] _import_structure["masking_utils"] = ["AttentionMaskInterface"] _import_s - src/transformers/_typing.py (6981 bytes)
# Copyright 2026 The HuggingFace Inc. team. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. """Typing helpers shared across the Transformers library.""" from __future__ import annotations import logging from collections.abc import Mapping, MutableMapping from os import PathLike from typing import TYPE_CHECKING, Any, Protocol, TypeAlias if TYPE_CHECKING: import torch # A few helpful type aliases Level: TypeAlias = int ExcInfo: TypeAlias = ( None | bool | BaseException | tuple[type[BaseException], BaseException, object] # traceback is `types.TracebackType`, but keep generic here ) DeviceMeshLike: TypeAlias = Any # PyTorch stubs do not model torch.distributed.device_mesh consistently yet. class TransformersLogger(Protocol): # ---- Core Logger identity / configuration ---- name: str level: int parent: logging.Logger | None propagate: bool disabled: bool handlers: list[logging.Handler] # Exists on Logger; default is True. (Not heavily used, but is part of API.) raiseExceptions: bool # ---- Standard methods ---- def setLevel(self, level: Level) -> None: ... def isEnabledFor(self, level: Level) -> bool: ... def getEffectiveLevel(self) -> int: ... def getChild(self, suffix: str) -> logging.Logger: ... def addHandler(self, hdlr: logging.Handler) -> None: ... def removeHandler(self, hdlr: logging.Handler) -> None: ... def hasHandlers(self) -> bool: ... # ---- Logging calls ---- def debug(self, msg: object, *args: object, **kwargs: object) -> None: ... def info(self, msg: object, *args: object, **kwargs: object) -> None: ... def warning(self, msg: object, *args: object, **kwargs: object) -> None: ... def warn(self, msg: object, *args: object, **kwargs: object) -> None: ... def error(self, msg: object, *args: object, **kwargs: object) -> None: ... def exception(self, msg: object, *args: object, exc_info: ExcInfo = True, **kwargs: object) -> None: ... def critical(self, msg: object, *args: object, **kwargs: object) -> None: ... def fatal(self, msg: object, *args: object, **kwargs: object) -> None: ... # The lowest-level primitive def log(self, level: Level, msg: object, *args: object, **kwargs: object) -> None: ... # ---- Record-level / formatting ---- def makeRecord( self, name: str, level: Level, fn: str, lno: int, msg: object, args: tuple[object, ...] | Mapping[str, object], exc_info: ExcInfo, func: str | None = None, extra: Mapping[str, object] | None = None, sinfo: str | None = None, ) -> logging.LogRecord: ... def handle(self, record: logging.LogRecord) -> None: ... def findCaller( self, stack_info: bool = False, stacklevel: int = 1, ) -> tuple[str, int, str, str | None]: ... def callHandlers(self, record: logging.LogRecord) -> None: ... def getMessage(self) -> str: ... # NOTE: actually on LogRecord; included rarely; safe to omit if you want def _log( self, level: Level, msg: object, args: tuple[object, ...] | Mapping[str, object], exc_info: ExcInfo = None, extra: Mapping[str, object] | None = None, stack_info: bool = False, stacklevel: int = 1, ) -> None: ... # ---- Filters ---- def addFilter(self, filt: logging.Filter) -> None: ... def removeFilter(self, filt: logging.Filter) -> None: ... @property def filters(self) -> list[logging.Filter]: ... def filter(self, record: logging.LogRecord) -> bool: ... # ---- Convenience helpers ---- def setFormatter(self, fmt: logging.Formatter) -> None: ... # mostly on handlers; present on adapters sometimes def debugStack(self, msg: object, *args: object, **kwargs: object) -> None: ... # not std; safe no-op if absent # ---- stdlib dictConfig-friendly / extra storage ---- # Logger has `manager` and can have arbitrary attributes; Protocol can't express arbitrary attrs, # but we can at least include `__dict__` to make "extra attributes" less painful. __dict__: MutableMapping[str, Any] # ---- Transformers logger specific methods ---- def warning_advice(self, msg: object, *args: object, **kwargs: object) -> None: ... def warning_once(self, msg: object, *args: object, **kwargs: object) -> None: ... def info_once(self, msg: object, *args: object, **kwargs: object) -> None: ... class GenerativePreTrainedModel(Protocol): """Protocol for the model interface that GenerationMixin expects. GenerationMixin is designed to be mixed into PreTrainedModel subclasses. This Protocol documents the attributes and methods the mixin relies on from its host class. It is *not* used at runtime — its purpose is to help the ``ty`` type checker resolve ``self.<attr>`` accesses inside the mixin. """ config: Any # PretrainedConfig — kept as Any to avoid circular imports device: torch.device dtype: torch.dtype main_input_name: str base_model_prefix: str _is_stateful: bool hf_quantizer: Any encoder: Any hf_device_map: dict[str, Any] _previous_max_cache_length: int generation_config: Any # GenerationConfig def __getattr__(self, name: str) -> Any: ... def forward(self, *args: Any, **kwargs: Any) -> Any: ... def __call__(self, *args: Any, **kwargs: Any) -> Any: ... def can_generate(self) -> bool: ... def get_encoder(self) -> Any: ... def get_output_embeddings(self) -> Any: ... def get_input_embeddings(self) -> Any: ... def set_output_embeddings(self, value: Any) -> None: ... def set_input_embeddings(self, value: Any) -> None: ... def get_compiled_call(self, compile_config: Any) -> Any: ... def set_experts_implementation(self, *args: Any, **kwargs: Any) -> Any: ... def _supports_logits_to_keep(self) -> bool: ... class StringValuedEnumLike(Protocol): value: str class PeftConfigLike(Protocol): peft_type: StringValuedEnumLike is_prompt_learning: bool base_model_name_or_path: str | PathLike[str] | None inference_mode: bool def save_pretrained(self, save_directory: str | PathLike[str], **kwargs: Any) -> None: ... class WhisperGenerationConfigLike(Protocol): """Protocol for Whisper-specific generation config fields accessed in generation internals.""" no_timestamps_token_id: int - src/transformers/activations.py (13477 bytes)
# Copyright 2020 The HuggingFace Team. All rights reserved. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. import functools import math from collections import OrderedDict import torch from torch import Tensor, nn from .integrations.hub_kernels import use_kernel_forward_from_hub from .utils import logging from .utils.import_utils import is_torchdynamo_compiling logger = logging.get_logger(__name__) @use_kernel_forward_from_hub("GeluTanh") class GELUTanh(nn.Module): """ A fast C implementation of the tanh approximation of the GeLU activation function. See https://huggingface.co/papers/1606.08415. This implementation is equivalent to NewGELU and FastGELU but much faster. However, it is not an exact numerical match due to rounding errors. """ def __init__(self, use_gelu_tanh_python: bool = False): super().__init__() if use_gelu_tanh_python: self.act = self._gelu_tanh_python else: self.act = functools.partial(nn.functional.gelu, approximate="tanh") def _gelu_tanh_python(self, input: Tensor) -> Tensor: return input * 0.5 * (1.0 + torch.tanh(math.sqrt(2.0 / math.pi) * (input + 0.044715 * torch.pow(input, 3.0)))) def forward(self, input: Tensor) -> Tensor: return self.act(input) # Added for compatibility with autoawq which is archived now and imports PytorchGELUTanh from activations.py PytorchGELUTanh = GELUTanh @use_kernel_forward_from_hub("NewGELU") class NewGELUActivation(nn.Module): """ Implementation of the GELU activation function currently in Google BERT repo (identical to OpenAI GPT). Also see the Gaussian Error Linear Units paper: https://huggingface.co/papers/1606.08415 """ def forward(self, input: Tensor) -> Tensor: return 0.5 * input * (1.0 + torch.tanh(math.sqrt(2.0 / math.pi) * (input + 0.044715 * torch.pow(input, 3.0)))) @use_kernel_forward_from_hub("GeLU") class GELUActivation(nn.Module): """ Original Implementation of the GELU activation function in Google BERT repo when initially created. For information: OpenAI GPT's GELU is slightly different (and gives slightly different results): 0.5 * x * (1 + torch.tanh(math.sqrt(2 / math.pi) * (x + 0.044715 * torch.pow(x, 3)))) This is now written in C in nn.functional Also see the Gaussian Error Linear Units paper: https://huggingface.co/papers/1606.08415 """ def __init__(self, use_gelu_python: bool = False): super().__init__() if use_gelu_python: self.act = self._gelu_python else: self.act = nn.functional.gelu def _gelu_python(self, input: Tensor) -> Tensor: return input * 0.5 * (1.0 + torch.erf(input / math.sqrt(2.0))) def forward(self, input: Tensor) -> Tensor: return self.act(input) @use_kernel_forward_from_hub("SiLU") class SiLUActivation(nn.Module): """ See Gaussian Error Linear Units (Hendrycks et al., https://arxiv.org/abs/1606.08415) where the SiLU (Sigmoid Linear Unit) was originally introduced and coined, and see Sigmoid-Weighted Linear Units for Neural Network Function Approximation in Reinforcement Learning (Elfwing et al., https://arxiv.org/abs/1702.03118) and Swish: a Self-Gated Activation Function (Ramachandran et al., https://arxiv.org/abs/1710.05941v1) where the SiLU was experimented with later. """ def forward(self, input: Tensor) -> Tensor: return nn.functional.silu(input) @use_kernel_forward_from_hub("FastGELU") class FastGELUActivation(nn.Module): """ Applies GELU approximation that is slower than QuickGELU but more accurate. See: https://github.com/hendrycks/GELUs """ def forward(self, input: Tensor) -> Tensor: return 0.5 * input * (1.0 + torch.tanh(input * 0.7978845608 * (1.0 + 0.044715 * input * input))) @use_kernel_forward_from_hub("QuickGELU") class QuickGELUActivation(nn.Module): """ Applies GELU approximation that is fast but somewhat inaccurate. See: https://github.com/hendrycks/GELUs """ def forward(self, input: Tensor) -> Tensor: return input * torch.sigmoid(1.702 * input) class ClippedGELUActivation(nn.Module): """ Clip the range of possible GeLU outputs between [min, max]. This is especially useful for quantization purpose, as it allows mapping negatives values in the GeLU spectrum. For more information on this trick, please refer to https://huggingface.co/papers/2004.09602. Gaussian Error Linear Unit. Original Implementation of the gelu activation function in Google Bert repo when initially created. For information: OpenAI GPT's gelu is slightly different (and gives slightly different results): 0.5 * x * (1 + torch.tanh(math.sqrt(2 / math.pi) * (x + 0.044715 * torch.pow(x, 3)))). See https://huggingface.co/papers/1606.08415 """ def __init__(self, min: float, max: float): if min > max: raise ValueError(f"min should be < max (got min: {min}, max: {max})") super().__init__() self.min = min self.max = max def forward(self, x: Tensor) -> Tensor: return torch.clip(gelu(x), self.min, self.max) class AccurateGELUActivation(nn.Module): """ Applies GELU approximation that is faster than default and more accurate than QuickGELU. See: https://github.com/hendrycks/GELUs Implemented along with MEGA (Moving Average Equipped Gated Attention) """ def __init__(self): super().__init__() self.precomputed_constant = math.sqrt(2 / math.pi) def forward(self, input: Tensor) -> Tensor: return 0.5 * input * (1 + torch.tanh(self.precomputed_constant * (input + 0.044715 * torch.pow(input, 3)))) class MishActivation(nn.Module): """ See Mish: A Self-Regularized Non-Monotonic Activation Function (Misra., https://huggingface.co/papers/1908.08681). Also visit the official repository for the paper: https://github.com/digantamisra98/Mish """ def __init__(self): super().__init__() self.act = nn.functional.mish def _mish_python(self, input: Tensor) -> Tensor: return input * torch.tanh(nn.functional.softplus(input)) def forward(self, input: Tensor) -> Tensor: return self.act(input) class LinearActivation(nn.Module): """ Applies the linear activation function, i.e. forwarding input directly to output. """ def forward(self, input: Tensor) -> Tensor: return input class LaplaceActivation(nn.Module): """ Applies elementwise activation based on Laplace function, introduced in MEGA as an attention activation. See https://huggingface.co/papers/2209.10655 Inspired by squared relu, but with bounded range and gradient for better stability """ def forward(self, input, mu=0.707107, sigma=0.282095): input = (input - mu).div(sigma * math.sqrt(2.0)) return 0.5 * (1.0 + torch.erf(input)) class ReLUSquaredActivation(nn.Module): """ Applies the relu^2 activation introduced in https://huggingface.co/papers/2109.08668 """ def forward(self, input): relu_applied = nn.functional.relu(input) squared = torch.square(relu_applied) return squared class SqrtSoftplusActivation(nn.Module): """sqrt(softplus(x)) — the router scoring function used by DeepSeek V4.""" def forward(self, input): return nn.functional.softplus(input).sqrt() class ClassInstantier(OrderedDict): def __getitem__(self, key): content = super().__getitem__(key) cls, kwargs = content if isinstance(content, tuple) else (content, {}) return cls(**kwargs) class XIELUActivation(nn.Module): """ Applies the xIELU activation function introduced in https://arxiv.org/abs/2411.13010 If the user has installed the nickjbrowning/XIELU wheel, we import xIELU CUDA Otherwise, we emit a single warning and use xIELU Python """ def __init__( self, alpha_p_init=0.8, alpha_n_init=0.8, beta=0.5, eps=-1e-6, dtype=torch.bfloat16, with_vector_loads=False, ): super().__init__() self.alpha_p = nn.Parameter(torch.log(torch.expm1(torch.tensor(alpha_p_init, dtype=dtype))).unsqueeze(0)) self.alpha_n = nn.Parameter( torch.log(torch.expm1(torch.tensor(alpha_n_init - beta, dtype=dtype))).unsqueeze(0) ) self.beta = nn.Buffer(torch.tensor(beta, dtype=dtype)) self.eps = nn.Buffer(torch.tensor(eps, dtype=dtype)) self.with_vector_loads = with_vector_loads # Temporary until xIELU CUDA fully implemented self._beta_scalar = float(beta) self._eps_scalar = float(eps) self._xielu_cuda_obj = None try: import xielu.ops # noqa: F401 self._xielu_cuda_obj = torch.classes.xielu.XIELU() msg = "Using experimental xIELU CUDA." try: from torch.compiler import allow_in_graph self._xielu_cuda_fn = allow_in_graph(self._xielu_cuda) msg += " Enabled torch._dynamo for xIELU CUDA." except Exception as err: msg += f" Could not enable torch._dynamo for xIELU ({err}) - this may result in slower performance." self._xielu_cuda_fn = self._xielu_cuda logger.warning_once(msg) except Exception as err: logger.warning_once( f"CUDA-fused xIELU not available ({err}) – falling back to a Python version.\n" "For CUDA xIELU (experimental), `pip install git+https://github.com/nickjbrowning/XIELU`" ) def _xielu_python(self, x: Tensor) -> Tensor: alpha_p = nn.functional.softplus(self.alpha_p) alpha_n = self.beta + nn.functional.softplus(self.alpha_n) return torch.where( x > 0, alpha_p * x * x + self.beta * x, (torch.expm1(torch.min(x, self.eps)) - x) * alpha_n + self.beta * x, ) def _xielu_cuda(self, x: Tensor) -> Tensor: """Firewall function to prevent torch.compile from seeing .item() calls""" original_shape = x.shape # CUDA kernel expects 3D tensors, reshape if needed while x.dim() < 3: x = x.unsqueeze(0) if x.dim() > 3: x = x.view(-1, 1, x.size(-1)) if original_shape != x.shape: logger.warning_once( "Warning: xIELU input tensor expects 3 dimensions but got (shape: %s). Reshaping to (shape: %s).", original_shape, x.shape, ) result = self._xielu_cuda_obj.forward( x, self.alpha_p.to(x.dtype), self.alpha_n.to(x.dtype), # Temporary until xIELU CUDA fully implemented -> self.{beta,eps}.item() self._beta_scalar, self._eps_scalar, self.with_vector_loads, ) return result.view(original_shape) def forward(self, input: Tensor) -> Tensor: if self._xielu_cuda_obj is not None and input.is_cuda: if not is_torchdynamo_compiling(): return self._xielu_cuda_fn(input) else: logger.warning_once("torch._dynamo is compiling, using Python version of xIELU.") return self._xielu_python(input) ACT2CLS = { "gelu": GELUActivation, "gelu_10": (ClippedGELUActivation, {"min": -10, "max": 10}), "gelu_fast": FastGELUActivation, "gelu_new": NewGELUActivation, "gelu_python": (GELUActivation, {"use_gelu_python": True}), "gelu_pytorch_tanh": GELUTanh, "gelu_python_tanh": (GELUTanh, {"use_gelu_tanh_python": True}), "gelu_accurate": AccurateGELUActivation, "hardswish": nn.Hardswish, "laplace": LaplaceActivation, "leaky_relu": nn.LeakyReLU, "linear": LinearActivation, "mish": MishActivation, "quick_gelu": QuickGELUActivation, "relu": nn.ReLU, "relu2": ReLUSquaredActivation, "relu6": nn.ReLU6, "sigmoid": nn.Sigmoid, "silu": SiLUActivation, "sqrtsoftplus": SqrtSoftplusActivation, "swish": nn.SiLU, "tanh": nn.Tanh, "prelu": nn.PReLU, "xielu": XIELUActivation, } ACT2FN = ClassInstantier(ACT2CLS) def get_activation(activation_string): if activation_string in ACT2FN: return ACT2FN[activation_string] else: raise KeyError(f"function {activation_string} not found in ACT2FN mapping {list(ACT2FN.keys())}") # For backwards compatibility with: from activations import gelu_python gelu_python = get_activation("gelu_python") gelu_new = get_activation("gelu_new") gelu = get_activation("gelu") gelu_fast = get_activation("gelu_fast") gelu_pytorch_tanh = get_activation("gelu_pytorch_tanh") quick_gelu = get_activation("quick_gelu") silu = get_activation("silu") mish = get_activation("mish") linear_act = get_activation("linear") - src/transformers/audio_utils.py (68942 bytes)
# Copyright 2023 The HuggingFace Inc. team and the librosa & torchaudio authors. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. """ Audio processing functions to extract features from audio waveforms. This code is pure numpy to support all frameworks and remove unnecessary dependencies. """ import base64 import io import math import os import warnings from collections.abc import Sequence from io import BytesIO from typing import TYPE_CHECKING, Any, Union from urllib.parse import urlparse import numpy as np from huggingface_hub.utils import httpx from .utils import ( is_numpy_array, is_soundfile_available, is_soxr_available, is_torch_tensor, is_torchaudio_available, is_torchcodec_greater_or_equal, is_torchcodec_usable, requires_backends, ) from .utils.generic import retry if TYPE_CHECKING: import torch if is_soundfile_available(): import soundfile as sf if is_soxr_available(): import soxr if is_torchaudio_available(): import torchaudio AudioInput = Union[np.ndarray, "torch.Tensor", Sequence[np.ndarray], Sequence["torch.Tensor"]] @retry(exceptions=(httpx.HTTPError,)) def _fetch_audio_bytes(url: str, timeout: float | None = 10.0) -> bytes: """Fetch audio bytes from a URL with automatic retry and exponential backoff.""" response = httpx.get(url, follow_redirects=True, timeout=timeout) response.raise_for_status() return response.content _NEEDS_TORCHCODEC = ( "Loading audio from this source needs torchcodec>=0.3.0 with FFmpeg, which is not installed or can't be loaded. " "See https://github.com/meta-pytorch/torchcodec#installing-torchcodec." ) TORCHCODEC_ONLY_FILETYPES = frozenset( { "3gp", "aac", "ac3", "amr", "avi", "flv", "m4a", "m4v", "mkv", "mov", "mp4", "mpg", "ogv", "sox", "ts", "webm", "wma", "wmv", "wv", } ) def _format_from_source(audio: str) -> "str | None": """Best-effort format token from the source *string* — the file extension (paths and URLs) or the media subtype (`data:` URIs) — without resolving or decoding it. Returns None when the string carries no hint, e.g. a raw base64 payload.""" if audio.startswith("data:"): media_type = audio[len("data:") :].split(",", 1)[0].split(";", 1)[0] return media_type.rpartition("/")[2].removeprefix("x-") or None path = urlparse(audio).path if audio.startswith(("http://", "https://")) else audio return os.path.splitext(path)[1].lstrip(".").lower() or None def get_audio_filetype(data: bytes) -> str: """Identify a file's container/codec from its magic bytes. A few extensions are byte-identical in their headers and collapse to a canonical type: ``wavex`` -> ``wav`` and ``m4v``/``hevc.mp4`` -> ``mp4`` (all carry the ``isom`` ftyp brand). Raises ValueError if the bytes match no supported filetype. """ head = data[:64] # Containers that host several filetypes -> sniff a bit deeper. if head[4:8] == b"ftyp": # ISO-BMFF: m4v & hevc share the 'isom' brand -> mp4 brand = head[8:12] return ( "3gp" if brand[:3] == b"3gp" else "m4a" if brand[:3] == b"M4A" else "mov" if brand[:2] == b"qt" else "mp4" ) if head[:4] == b"RIFF" and head[8:12] in (b"WAVE", b"AVI "): return "wav" if head[8:12] == b"WAVE" else "avi" if head[:4] == b"riff" and head[4:8] == bytes.fromhex("2e91cf11"): # Wave64 return "w64" if head[:4] == bytes.fromhex("1a45dfa3"): # EBML: Matroska vs WebM return "webm" if b"webm" in head else "mkv" if head[:4] == b"OggS": # OGG: Opus / Theora (ogv) / Vorbis (ogg) page = data[:128] return "opus" if b"OpusHead" in page else "ogv" if b"theora" in page else "ogg" if head[:16] == bytes.fromhex("3026b2758e66cf11a6d900aa0062ce6c"): # ASF return "wmv" if bytes.fromhex("c0ef19bc4d5bcf11a8fd00805f5c442b") in data else "wma" if head[:1] == b"\xff" and len(head) > 1 and head[1] & 0xE0 == 0xE0: # MPEG/AAC sync if head[1] & 0xF6 == 0xF0: # ADTS layer bits 00 -> AAC return "aac" layer = head[1] >> 1 & 0x3 # MPEG audio layer field (II -> mp2, III -> mp3) if layer in (0b10, 0b01): return "mp2" if layer == 0b10 else "mp3" if head[:1] == b"\x47" and len(data) > 188 and data[188] == 0x47: return "ts" if head[:4] == b"FORM" and head[8:12] in (b"AIFF", b"AIFC"): return "aiff" # Single fixed-signature formats, keyed by their leading bytes. signatures = { b"fLaC": "flac", b"RF64": "rf64", b"caff": "caf", b".snd": "au", b"#!AMR": "amr", b"wvpk": "wv", b".SoX": "sox", b"XoS.": "sox", b"Creative Voice File": "voc", b"\x64\xa3\x01\x00": "sf", b"\x00\x01\xa3\x64": "sf", b"\x0b\x77": "ac3", b"\x00\x00\x01\xba": "mpg", b"FLV": "flv", b"ID3": "mp3", } for sig, filetype in signatures.items(): if head.startswith(sig): return filetype raise ValueError("not supported filetype") def _resolve_audio_source(audio: str, timeout: float | None = None) -> "str | bytes": """Resolve an audio source string to a local file path or raw bytes for a decoder. Accepts `http(s)://` URLs (fetched with retry), local file paths (returned unchanged), and base64 strings (optionally wrapped as a `data:...` URI). """ if audio.startswith(("http://", "https://")): return _fetch_audio_bytes(audio, timeout=timeout) if os.path.isfile(audio): return audio # Not a URL or a local path — assume base64, optionally wrapped as a `data:<media-type>;base64,` URI if audio.startswith("data:"): audio = audio.split(",", 1)[1] try: return base64.b64decode(audio) except Exception as e: raise ValueError( "Incorrect audio source. Must be a valid URL starting with `http://` or `https://`, " f"a valid path to an audio file, or a base64 encoded string. Got {audio}. Failed with {e}" ) def _load_audio_soundfile(source: "str | bytes", sampling_rate: int) -> np.ndarray: """Decode with soundfile and resample with soxr: the same steps as `librosa.load`, so the output is identical (downmix by averaging, `soxr_hq` resampling, padding/trimming to `ceil(num_samples * sampling_rate / src_rate)`).""" requires_backends(load_audio, ["audio"]) with sf.SoundFile(BytesIO(source) if isinstance(source, bytes) else source) as f: src_sampling_rate = f.samplerate waveform = f.read(dtype="float32", always_2d=True).T waveform = waveform.mean(axis=0) if waveform.shape[0] > 1 else waveform[0] if src_sampling_rate != sampling_rate: num_samples = math.ceil(waveform.shape[-1] * sampling_rate / src_sampling_rate) waveform = soxr.resample(waveform, src_sampling_rate, sampling_rate, quality="soxr_hq") waveform = np.pad(waveform, (0, max(0, num_samples - len(waveform))))[:num_samples] return np.ascontiguousarray(waveform, dtype=np.float32) def load_audio(audio: str | np.ndarray, sampling_rate=16000, timeout=None, backend: str = "auto") -> np.ndarray: """ Loads `audio` to an np.ndarray object. Args: audio (`str` or `np.ndarray`): The audio to be loaded to the numpy array format. If a `str`, it can be an `http(s)://` URL, a local file path, or a base64-encoded string (optionally wrapped as a `data:<media-type>;base64,` URI). sampling_rate (`int`, *optional*, defaults to 16000): The sampling rate to be used when loading the audio. It should be same as the sampling rate the model you will be using further was trained with. timeout (`float`, *optional*): The timeout value in seconds for the URL request. backend (`str`, *optional*, defaults to `"auto"`): Decoding backend: `"auto"` uses torchcodec when available (>=0.3.0) and falls back to soundfile; `"torchcodec"`, `"soundfile"` or `"torchaudio"` force that backend (and error if it is missing). `"soundfile"` decodes with `soundfile` and resamples with `soxr` (identical to `librosa.load`). `"torchaudio"` decodes with `torchaudio.load` and resamples with `torchaudio.functional.resample` (matches serving stacks such as sglang bit-for-bit). `"librosa"` is a deprecated alias of `"soundfile"`. Returns: `np.ndarray`: A numpy array representing the audio. """ if isinstance(audio, np.ndarray): return audio if not isinstance(audio, str): raise TypeError( "Incorrect format used for `audio`. Should be a numpy array or a `str`: an `http(s)://` URL, " f"a local file path, or a base64-encoded string (optionally wrapped as a `data:...` URI) but got {type(audio)}." ) # torchcodec handles audio/video; soundfile only plain audio. `backend` lets callers pin one. if backend == "librosa": warnings.warn( '`backend="librosa"` is deprecated and will be removed in Transformers v6. It now decodes with soundfile ' 'and resamples with soxr, with output identical to `librosa.load`. Use `backend="soundfile"` instead.', FutureWarning, ) backend = "soundfile" if backend == "auto": resolved_backend = ( "torchcodec" if is_torchcodec_greater_or_equal("0.3.0") and is_torchcodec_usable(decode="audio") else "soundfile" ) elif backend in ("torchcodec", "soundfile", "torchaudio"): resolved_backend = backend else: raise ValueError(f"Unknown backend {backend!r}; expected 'auto', 'torchcodec', 'soundfile', or 'torchaudio'.") # soundfile-based backends (soundfile / torchaudio) cannot decode the video-ish formats below. use_torchcodec = resolved_backend == "torchcodec" # 1. Identify the format from the source string (extension / `data:` media type), without fetching. filetype = _format_from_source(audio) # 2. Without torchcodec, fail fast and clearly on a format soundfile cannot decode. if not use_torchcodec and filetype in TORCHCODEC_ONLY_FILETYPES: raise RuntimeError( f"The audio source is a '{filetype}' file, which soundfile cannot decode. {_NEEDS_TORCHCODEC}" ) # 3. Resolve to local path or bytes; sniff format for raw base64 payloads before passing to soundfile. source = _resolve_audio_source(audio, timeout=timeout) if not use_torchcodec and filetype is None and isinstance(source, bytes): try: filetype = get_audio_filetype(source) except ValueError: filetype = None if filetype in TORCHCODEC_ONLY_FILETYPES: raise RuntimeError( f"The audio source is a '{filetype}' file, which soundfile cannot decode. {_NEEDS_TORCHCODEC}" ) # 4. Decode with the selected backend (`requires_backends` raises a clear error if it is missing). if use_torchcodec: requires_backends(load_audio, ["torchcodec"]) from torchcodec.decoders import AudioDecoder # `num_channels=1` matches what most models expect and the soundfile backend. return AudioDecoder(source, sample_rate=sampling_rate, num_channels=1).get_all_samples().data[0].numpy() if resolved_backend == "torchaudio": requires_backends(load_audio, ["torchaudio"]) waveform, src_sampling_rate = torchaudio.load(BytesIO(source) if isinstance(source, bytes) else source) waveform = waveform.mean(dim=0) # to mono if src_sampling_rate != sampling_rate: waveform = torchaudio.functional.resample(waveform, orig_freq=src_sampling_rate, new_freq=sampling_rate) return waveform.numpy().astype(np.float32) return _load_audio_soundfile(source, sampling_rate) def load_audio_torchcodec(audio: str | np.ndarray, sampling_rate=16000, timeout=None) -> np.ndarray: """Deprecated. Use [`load_audio`] instead (equivalent to `backend="torchcodec"`).""" warnings.warn( "`load_audio_torchcodec` is deprecated and will be removed in a future version. " 'Use `load_audio(..., backend="torchcodec")` instead.', FutureWarning, ) return load_audio(audio, sampling_rate=sampling_rate, timeout=timeout, backend="torchcodec") def load_audio_librosa(audio: str | np.ndarray, sampling_rate=16000, timeout=None) -> np.ndarray: """Deprecated. Use [`load_audio`] instead (equivalent to `backend="soundfile"`).""" warnings.warn( "`load_audio_librosa` is deprecated and will be removed in a future version. " 'Use `load_audio(..., backend="soundfile")` instead.', FutureWarning, ) return load_audio(audio, sampling_rate=sampling_rate, timeout=timeout, backend="soundfile") def load_audio_as( audio: str, return_format: str, timeout: int | None = None, force_mono: bool = False, sampling_rate: int | None = None, ) -> str | dict[str, Any] | io.BytesIO | None: """ Load audio from either a local file path or URL and return in specified format. Args: audio (`str`): Either a local file path or a URL to an audio file return_format (`str`): Format to return the audio in: - "base64": Base64 encoded string - "dict": Dictionary with data and format - "buffer": BytesIO object timeout (`int`, *optional*): Timeout for URL requests in seconds force_mono (`bool`): Whether to convert stereo audio to mono sampling_rate (`int`, *optional*): If provided, the audio will be resampled to the specified sampling rate. Returns: `Union[str, Dict[str, Any], io.BytesIO, None]`: - `str`: Base64 encoded audio data (if return_format="base64") - `dict`: Dictionary with 'data' (base64 encoded audio data) and 'format' keys (if return_format="dict") - `io.BytesIO`: BytesIO object containing audio data (if return_format="buffer") """ requires_backends(load_audio_as, ["soundfile"]) if sampling_rate is not None: requires_backends(load_audio_as, ["soxr"]) # only needed to resample if return_format not in [" - src/transformers/backbone_utils.py (17556 bytes)
# Copyright 2026 The HuggingFace Inc. team. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. """Collection of utils to be used by backbones and their components.""" import enum import functools import inspect from .utils import hf_api, logging from .utils.output_capturing import ( maybe_install_capturing_hooks, resolve_hidden_states_indices, selected_hidden_states, ) logger = logging.get_logger(__name__) class BackboneType(enum.Enum): TIMM = "timm" TRANSFORMERS = "transformers" class BackboneConfigMixin: """ A Mixin to support handling the `out_features` and `out_indices` attributes for the backbone configurations. """ def set_output_features_output_indices( self, out_features: list | None, out_indices: list | None, ): """ Sets output indices and features to new values and aligns them with the given `stage_names`. If one of the inputs is not given, find the corresponding `out_features` or `out_indices` for the given `stage_names`. Args: out_features (`list[str]`, *optional*): The names of the features for the backbone to output. Defaults to `config._out_features` if not provided. out_indices (`list[int]` or `tuple[int]`, *optional*): The indices of the features for the backbone to output. Defaults to `config._out_indices` if not provided. """ self._out_features = out_features self._out_indices = list(out_indices) if isinstance(out_indices, tuple) else out_indices # First verify that the out_features and out_indices are valid self.verify_out_features_out_indices() # Align output features with indices out_features, out_indices = self._out_features, self._out_indices if out_indices is None and out_features is None: out_indices = [len(self.stage_names) - 1] out_features = [self.stage_names[-1]] elif out_indices is None and out_features is not None: out_indices = [self.stage_names.index(layer) for layer in out_features] elif out_features is None and out_indices is not None: out_features = [self.stage_names[idx] for idx in out_indices] # Update values and verify that the aligned out_features and out_indices are valid self._out_features, self._out_indices = out_features, out_indices self.verify_out_features_out_indices() def verify_out_features_out_indices(self): """ Verify that out_indices and out_features are valid for the given stage_names. """ if self.stage_names is None: raise ValueError("Stage_names must be set for transformers backbones") if self._out_features is not None: if not isinstance(self._out_features, (list,)): raise ValueError(f"out_features must be a list got {type(self._out_features)}") if any(feat not in self.stage_names for feat in self._out_features): raise ValueError( f"out_features must be a subset of stage_names: {self.stage_names} got {self._out_features}" ) if len(self._out_features) != len(set(self._out_features)): raise ValueError(f"out_features must not contain any duplicates, got {self._out_features}") if self._out_features != ( sorted_feats := [feat for feat in self.stage_names if feat in self._out_features] ): raise ValueError( f"out_features must be in the same order as stage_names, expected {sorted_feats} got {self._out_features}" ) if self._out_indices is not None: if not isinstance(self._out_indices, list): raise ValueError(f"out_indices must be a list, got {type(self._out_indices)}") # Convert negative indices to their positive equivalent: [-1,] -> [len(stage_names) - 1,] positive_indices = tuple(idx % len(self.stage_names) if idx < 0 else idx for idx in self._out_indices) if any(idx for idx in positive_indices if idx not in range(len(self.stage_names))): raise ValueError( f"out_indices must be valid indices for stage_names {self.stage_names}, got {self._out_indices}" ) if len(positive_indices) != len(set(positive_indices)): msg = f"out_indices must not contain any duplicates, got {self._out_indices}" msg += f"(equivalent to {positive_indices}))" if positive_indices != self._out_indices else "" raise ValueError(msg) if positive_indices != tuple(sorted(positive_indices)): sorted_negative = [ idx for _, idx in sorted(zip(positive_indices, self._out_indices), key=lambda x: x[0]) ] raise ValueError( f"out_indices must be in the same order as stage_names, expected {sorted_negative} got {self._out_indices}" ) if self._out_features is not None and self._out_indices is not None: if len(self._out_features) != len(self._out_indices): raise ValueError("out_features and out_indices should have the same length if both are set") if self._out_features != [self.stage_names[idx] for idx in self._out_indices]: raise ValueError("out_features and out_indices should correspond to the same stages if both are set") @property def out_features(self): return self._out_features @out_features.setter def out_features(self, out_features: list[str]): """ Set the out_features attribute. This will also update the out_indices attribute to match the new out_features. """ self.set_output_features_output_indices(out_features=out_features, out_indices=None) @property def out_indices(self): return self._out_indices @out_indices.setter def out_indices(self, out_indices: tuple[int, ...] | list[int]): """ Set the out_indices attribute. This will also update the out_features attribute to match the new out_indices. """ out_indices = list(out_indices) if out_indices is not None else out_indices self.set_output_features_output_indices(out_features=None, out_indices=out_indices) def to_dict(self): """ Serializes this instance to a Python dictionary. Override the default `to_dict()` from `PreTrainedConfig` to include the `out_features` and `out_indices` attributes. """ output = super().to_dict() output["out_features"] = output.pop("_out_features", None) output["out_indices"] = output.pop("_out_indices", None) return output def filter_output_hidden_states(forward_function): """ Wrapper for backbone forwards. Backbones always compute `hidden_states` to build their feature maps, so this forces `output_hidden_states=True` on the wrapped forward and then removes `hidden_states` from the returned object unless the caller explicitly requested them. NOTE: We assume a `can_return_tuple` decorator to be applied before so that we always expect a dict like object to remove the hidden states. """ @functools.wraps(forward_function) def wrapper(self, *args, **kwargs): output_hidden_states = kwargs.get("output_hidden_states", getattr(self.config, "output_hidden_states", False)) kwargs["output_hidden_states"] = True output = forward_function(self, *args, **kwargs) if not output_hidden_states: filtered_output_data = {k: v for k, v in output.items() if k != "hidden_states"} output = type(output)(**filtered_output_data) elif (selected := selected_hidden_states(output_hidden_states)) is not None: # Only return the selected hidden states, as `capture_outputs` would. Unlike there, all of them already exist selected = resolve_hidden_states_indices(self, selected, num_hidden_states=len(output["hidden_states"])) output["hidden_states"] = [h if i in selected else None for i, h in enumerate(output["hidden_states"])] return output return wrapper class BackboneMixin: backbone_type: BackboneType | None = None # Attribute to indicate if the backbone has attention and can return attention outputs. # Should be set to `False` for conv-based models to be able to run `forward_with_filtered_kwargs` has_attentions: bool = True def __init__(self, *args, **kwargs) -> None: """ Method to initialize the backbone. This method is called by the constructor of the base class after the pretrained model weights have been loaded. """ super().__init__(*args, **kwargs) timm_backbone = kwargs.pop("timm_backbone", None) if timm_backbone is not None: self.backbone_type = BackboneType.TIMM else: self.backbone_type = BackboneType.TRANSFORMERS if self.backbone_type == BackboneType.TIMM: self._init_timm_backbone(backbone=timm_backbone) elif self.backbone_type == BackboneType.TRANSFORMERS: self._init_transformers_backbone() else: raise ValueError(f"backbone_type {self.backbone_type} not supported.") def post_init(self): """ Override `post_init` to always install capturing hooks, as backbone will ALWAYS capture outputs. We need to do it in `post_init`, as modules need to be already instantiated. It avoids some mixups with `torch.compile`, as the first hook installation will need/create a graph break, which can clash with external user call such as `model = torch.compile(model...)`. """ # NOTE: Since this class is ALWAYS used as a Mixin with another PreTrainedModel class, this `super` call # will call the PreTrained's `post_init` super().post_init() maybe_install_capturing_hooks(self) def _init_timm_backbone(self, backbone) -> None: """ Initialize the backbone model from timm. The backbone must already be loaded to backbone """ out_features_from_config = getattr(self.config, "out_features", None) stage_names_from_config = getattr(self.config, "stage_names", None) # These will disagree with the defaults for the transformers models e.g. for resnet50 # the transformer model has out_features = ['stem', 'stage1', 'stage2', 'stage3', 'stage4'] # the timm model has out_features = ['act', 'layer1', 'layer2', 'layer3', 'layer4'] self.stage_names = [stage["module"] for stage in backbone.feature_info.info] self.num_features = [stage["num_chs"] for stage in backbone.feature_info.info] out_indices = list(backbone.feature_info.out_indices) out_features = backbone.feature_info.module_name() if out_features_from_config is not None and out_features_from_config != out_features: raise ValueError( f"Config has `out_features` set to {out_features_from_config} which doesn't match `out_features` " "from backbone's feature_info. Please check if your checkpoint has correct out features/indices saved." ) if stage_names_from_config is not None and stage_names_from_config != self.stage_names: raise ValueError( f"Config has `stage_names` set to {stage_names_from_config} which doesn't match `stage_names` " "from backbone's feature_info. Please check if your checkpoint has correct `stage_names` saved." ) # We set, align and verify out indices, out features and stage names self.config.stage_names = self.stage_names self.config.set_output_features_output_indices(out_features, out_indices) def _init_transformers_backbone(self) -> None: self.stage_names = self.config.stage_names self.config.verify_out_features_out_indices() # Number of channels for each stage. This is set in the transformer backbone model init self.num_features = None @property def out_features(self): return self.config._out_features @out_features.setter def out_features(self, out_features: list[str]): """ Set the out_features attribute. This will also update the out_indices attribute to match the new out_features. """ self.config.out_features = out_features @property def out_indices(self): return self.config._out_indices @out_indices.setter def out_indices(self, out_indices: tuple[int] | list[int]): """ Set the out_indices attribute. This will also update the out_features attribute to match the new out_indices. """ self.config.out_indices = out_indices @property def out_feature_channels(self): # the current backbones will output the number of channels for each stage # even if that stage is not in the out_features list. return {stage: self.num_features[i] for i, stage in enumerate(self.stage_names)} @property def channels(self): return [self.out_feature_channels[name] for name in self.out_features] def forward_with_filtered_kwargs(self, *args, **kwargs): if not self.has_attentions: kwargs.pop("output_attentions", None) if self.backbone_type == BackboneType.TIMM: signature = dict(inspect.signature(self.forward).parameters) kwargs = {k: v for k, v in kwargs.items() if k in signature} return self(*args, **kwargs) def forward( self, pixel_values, output_hidden_states: bool | None = None, output_attentions: bool | None = None, return_dict: bool | None = None, ): raise NotImplementedError("This method should be implemented by the derived class.") def consolidate_backbone_kwargs_to_config( backbone_config, default_backbone: str | None = None, default_config_type: str | None = None, default_config_kwargs: dict | None = None, timm_default_kwargs: dict | None = None, **kwargs, ): # Lazy import to avoid circular import issues. Can be imported properly # after deleting ref to `BackboneMixin` in `utils/backbone_utils.py` from .con - src/transformers/cache_utils.py (120474 bytes)
from abc import ABC, abstractmethod from collections import UserDict from collections.abc import Iterable import torch from .configuration_utils import PreTrainedConfig from .utils import ( is_hqq_available, is_optimum_quanto_available, is_quanto_greater, is_torch_greater_or_equal, is_torchdynamo_compiling, ) if is_hqq_available(): from hqq.core.quantize import Quantizer as HQQQuantizer _is_torch_greater_or_equal_than_2_7 = is_torch_greater_or_equal("2.7", accept_dev=True) class CacheLayerMixin(ABC): """Base, abstract class for a single layer's cache.""" is_compileable = False is_croppable = False supports_early_init = True # Subclasses can set `_layer_type` to auto-register themselves in the mappings, if the class definition lives in a modeling # file instead of this file. This allows to update the mapping only when the modeling file is imported, which simplifies imports _layer_type: str | None = None def __init_subclass__(cls, **kwargs): super().__init_subclass__(**kwargs) if cls._layer_type is not None: if issubclass(cls, StaticLayer): STATIC_LAYER_TYPE_MAPPING[cls._layer_type] = cls else: DYNAMIC_LAYER_TYPE_MAPPING[cls._layer_type] = cls def __init__(self, **kwargs): self.keys: torch.Tensor | None = None self.values: torch.Tensor | None = None self.is_initialized = False def __repr__(self): return f"{self.__class__.__name__}" @abstractmethod def lazy_initialization(self, key_states: torch.Tensor, value_states: torch.Tensor) -> None: ... @abstractmethod def update( self, key_states: torch.Tensor, value_states: torch.Tensor, *args, **kwargs ) -> tuple[torch.Tensor, torch.Tensor]: ... @abstractmethod def get_mask_sizes(self, query_length: int) -> tuple[int, int]: ... @abstractmethod def get_seq_length(self) -> int: ... @abstractmethod def get_max_length(self) -> int: """ Returns the maximum sequence length the layer can hold. A value of `-1` means no maximum, or an undefined maximum, for example a dynamic attention layer that grows indefinitely or a linear attention layer that has no sequence length dimension. """ ... def offload(self): """Offload this layer's data to CPU device.""" if self.is_initialized: self.keys = self.keys.to("cpu", non_blocking=True) self.values = self.values.to("cpu", non_blocking=True) def prefetch(self): """In case of layer offloading, this allows to move the data back to the layer's device ahead of time.""" if self.is_initialized and self.keys.device != self.device: self.keys = self.keys.to(self.device, non_blocking=True) self.values = self.values.to(self.device, non_blocking=True) def reset(self) -> None: """Resets the cache values while preserving the objects""" if self.is_initialized: self.keys.zero_() self.values.zero_() # This attribute is set on several Layers if hasattr(self, "cumulative_length"): # It can either be an int for dynamic layers, or a tensor for static layers if isinstance(self.cumulative_length, int): self.cumulative_length = 0 else: self.cumulative_length.zero_() def reorder_cache(self, beam_idx: torch.LongTensor) -> None: """Reorders this layer's cache for beam search.""" if self.get_seq_length() > 0: self.keys = self.keys.index_select(0, beam_idx.to(self.keys.device)) self.values = self.values.index_select(0, beam_idx.to(self.values.device)) class DynamicLayer(CacheLayerMixin): """ A cache layer that grows dynamically as more tokens are generated. This is the default for generative models. It stores the key and value states as tensors of shape `[batch_size, num_heads, seq_len, head_dim]`. """ is_sliding = False is_croppable = True def lazy_initialization(self, key_states: torch.Tensor, value_states: torch.Tensor) -> None: self.device = key_states.device self.keys = torch.tensor([], dtype=key_states.dtype, device=self.device) self.values = torch.tensor([], dtype=value_states.dtype, device=self.device) self.is_initialized = True def update( self, key_states: torch.Tensor, value_states: torch.Tensor, *args, **kwargs ) -> tuple[torch.Tensor, torch.Tensor]: """ Update the key and value caches in-place, and return the necessary keys and value states. Args: key_states (`torch.Tensor`): The new key states to cache. value_states (`torch.Tensor`): The new value states to cache. Returns: tuple[`torch.Tensor`, `torch.Tensor`]: The key and value states. """ # Lazy initialization if not self.is_initialized: self.lazy_initialization(key_states, value_states) self.keys = torch.cat([self.keys, key_states], dim=-2) self.values = torch.cat([self.values, value_states], dim=-2) return self.keys, self.values def get_mask_sizes(self, query_length: int) -> tuple[int, int]: """Return the length and offset of the cache, used to generate the mask""" kv_offset = 0 kv_length = self.get_seq_length() + query_length return kv_length, kv_offset def get_seq_length(self) -> int: """Returns the sequence length of the cached states.""" if not self.is_initialized or self.keys.numel() == 0: return 0 return self.keys.shape[-2] def get_max_length(self) -> int: """Returns the maximum sequence length of the cache object. DynamicLayer does not have a maximum length.""" return -1 def reset(self) -> None: """Resets the cache values while preserving the objects.""" # Dropped rather than zeroed, as `update` grows them by concatenation. Clearing `is_initialized` first skips # the zeroing in `super`, which is still called to reset the `cumulative_length` of the inheriting layers. self.keys = self.values = None self.is_initialized = False super().reset() def crop(self, tokens_to_remove: int) -> None: """ Remove `tokens_to_remove` tokens from the current cache layer. """ # Legacy path: `tokens_to_remove` represents the final absolute size that the cache should have if tokens_to_remove > 0: raise ValueError( f"Calling `crop` with a positive value (`{tokens_to_remove}`) is not supported. " "Please use a negative integer to remove that number of tokens from the cache instead." ) # Nothing to do in these cases: 0 tokens to crop or the layer is not initialized (can happen with KV sharing) if tokens_to_remove == 0 or not self.is_initialized: return # Crop the cache self.keys = self.keys[..., : -abs(tokens_to_remove), :] self.values = self.values[..., : -abs(tokens_to_remove), :] def batch_repeat_interleave(self, repeats: int) -> None: """Repeat the cache `repeats` times in the batch dimension.""" if self.get_seq_length() > 0: self.keys = self.keys.repeat_interleave(repeats, dim=0) self.values = self.values.repeat_interleave(repeats, dim=0) def batch_select_indices(self, indices: torch.Tensor) -> None: """Only keep the `indices` in the batch dimension of the cache.""" if self.get_seq_length() > 0: self.keys = self.keys[indices, ...] self.values = self.values[indices, ...] class DynamicSlidingWindowLayer(DynamicLayer): """ A cache layer that grows dynamically as more tokens are generated, up until the sliding window size. It stores the key and value states as tensors of shape `[batch_size, num_heads, min(seq_len, sliding_window), head_dim]`. """ is_sliding = True def __init__(self, sliding_window: int, **kwargs): super().__init__() self.sliding_window = sliding_window self.cumulative_length = 0 self._sliding_window_tensor = torch.tensor(self.sliding_window, dtype=torch.long) self.record_past = False def activate_past_recording(self): """ Calling this function will activate past state recording, meaning that a call to `update` will wait for a call to `crop` before restricting the size of the `k/v_states` to `sliding_window`, to be able to retrieve previous full states. """ self.record_past = True def lazy_initialization(self, key_states: torch.Tensor, value_states: torch.Tensor) -> None: super().lazy_initialization(key_states, value_states) self._sliding_window_tensor = self._sliding_window_tensor.to(self.device) def update( self, key_states: torch.Tensor, value_states: torch.Tensor, *args, **kwargs ) -> tuple[torch.Tensor, torch.Tensor]: """ Update the key and value caches in-place, and return the necessary keys and value states. Args: key_states (`torch.Tensor`): The new key states to cache. value_states (`torch.Tensor`): The new value states to cache. Returns: tuple[`torch.Tensor`, `torch.Tensor`]: The key and value states. """ # Lazy initialization if not self.is_initialized: self.lazy_initialization(key_states, value_states) self.cumulative_length += key_states.shape[-2] # Compute the full states full_key_states = torch.cat([self.keys, key_states], dim=-2) full_value_states = torch.cat([self.values, value_states], dim=-2) # Only cache the last `self.sliding_window - 1` tokens (or all of them if lower than that) if not self.record_past: self.keys = full_key_states[:, :, -self.sliding_window + 1 :, :] self.values = full_value_states[:, :, -self.sliding_window + 1 :, :] # Return the full states return full_key_states, full_value_states # If we record the past, we keep them all for now, and they'll be restricted to the window size in `crop` else: self.keys = full_key_states self.values = full_value_states # In theory, when we record the past we always have a call to `crop` after every `forward`, so returning the full states # similar to the non-past case should be enough. However, in case several `forward` are run in a row without calling `crop` # in-between (as is the case in some assisted decoding method, where the assistant itself calls `generate` with a past-aware # Cache), we need to slice to only return the necesary states that are advertized to the mask by `get_mask_sizes` num_visible = self.sliding_window - 1 + key_states.shape[-2] return full_key_states[:, :, -num_visible:, :], full_value_states[:, :, -num_visible:, :] def get_mask_sizes(self, query_length: int) -> tuple[int, int]: """Return the length and offset of the cache, used to generate the attention mask""" is_full = self.cumulative_length >= self.sliding_window kv_offset = max(self.cumulative_length - self.sliding_window + 1, 0) if is_full: kv_length = self.sliding_window - 1 + query_length else: kv_length = self.cumulative_length + query_length return kv_length, kv_offset def get_seq_length(self) -> int: """Returns the sequence length of the cached states.""" return self.cumulative_length def get_max_length(self) -> int: """Return the maximum cache shape of the cache""" return self.sliding_window def crop(self, tokens_to_remove: int) -> None: """ Remove `tokens_to_remove` tokens from the current cache layer. This will also restrict the size of the cached states back to their minimal working size, i.e. `sliding_window - 1` if they reached the sliding window length. This means that `crop(0)` will not necessarily always be a no-op, as it may still remove useless states (i.e. states that are not needed for the next `forward`). """ # If we are beyond the sliding window, we need to be more careful if self.get_seq_length() >= self.sliding_window: if not self.record_past: raise RuntimeError( "`crop` was called, but the current layer does not track past states, and the sliding window size was already " "reached. Call `activate_past_recording` before `crop` to be able to rollback the cache." ) if tokens_to_remove > 0: raise RuntimeError( "Once the sliding window size has been reached, `DynamicSlidingWindowLayer` can only be cropped by passing a " "negative int, to specify how many tokens to remove" ) # In this case, simply restrict the size back to sliding window without cropping if tokens_to_remove == 0: self.keys = self.keys[:, :, -self.sliding_window + 1 :, :] self.values = self.values[:, :, -self.sliding_window + 1 :, :] # In this case, we crop and restrict the size back to the sliding window if still larger else: tokens_to_remove = abs(tokens_to_remove) self.keys = self.keys[:, :, -self.sliding_window + 1 - tokens_to_remove : -tokens_to_remove, :] self.values = self.values[:, :, -self.sliding_window + 1 - tokens_to_remove : -tokens_to_remove, :] self.cumulative_length = self.cumulative_length - tokens_to_remove # If we did not reach the sliding window, we can do the same as for a full attention layer else: super().crop(tokens_to_remove) self.cumulative_length = self.keys.shape[-2] class DynamicIndexedLayer(DynamicLayer): """ A cache layer that extends `DynamicLayer` with an extra indexer key cache for Dynamic Sparse Attention (DSA) models (e.g. GLM MoE DSA, DeepSeek V32). The main K/V cache stores tensors of shape `[batch_size, num_heads, seq_len, head_dim]` (inherited). The indexer key cache stores a tensor of shape `[batch_size, seq_len, index_head_dim]` (3D, single-head). """ def __init__(self, **kwargs): super().__init__() self.indexer_keys: torch.Tensor | None = None # Optional indexer positions, shaped [position_dims, batch_size, seq_len]. self.indexer_position_ids: torch.Tensor | None = None self.is_indexer_initialized: bool = False def lazy_initialization - src/transformers/cli/__init__.py (606 bytes)
# Copyright 2025 The HuggingFace Team. All rights reserved. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License.
- src/transformers/cli/add_new_model_like.py (32626 bytes)
# Copyright 2021 The HuggingFace Team. All rights reserved. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. import difflib import os import re import subprocess import textwrap from collections.abc import Callable from datetime import date from pathlib import Path from typing import Annotated, Any, cast import typer from ..utils import is_libcst_available # We protect this import to avoid requiring it for all `transformers` CLI commands - however it is actually # strictly required for this one (we need it both for modular and for the following Visitor) if is_libcst_available(): import libcst as cst from libcst import CSTVisitor from libcst import matchers as m class ClassFinder(CSTVisitor): """ A visitor to find all classes in a python module. """ def __init__(self): self.classes: list = [] self.public_classes: list = [] self.is_in_class = False def visit_ClassDef(self, node: cst.ClassDef) -> None: """Record class names. We assume classes always only appear at top-level (i.e. no class definition in function or similar)""" self.classes.append(node.name.value) self.is_in_class = True def leave_ClassDef(self, node: cst.ClassDef): self.is_in_class = False def visit_SimpleStatementLine(self, node: cst.SimpleStatementLine): """Record all public classes inside the `__all__` assignment.""" simple_top_level_assign_structure = m.SimpleStatementLine( body=[m.Assign(targets=[m.AssignTarget(target=m.Name())])] ) if not self.is_in_class and m.matches(node, simple_top_level_assign_structure): stmt = cast(cst.Assign, node.body[0]) assigned_variable = cast(cst.Name, stmt.targets[0].target).value if assigned_variable == "__all__": elements = cast(cst.Tuple, stmt.value).elements self.public_classes = [cast(cst.SimpleString, element.value).value for element in elements] CURRENT_YEAR = date.today().year REPO_PATH = Path(__file__).parents[3] COPYRIGHT = f""" # coding=utf-8 # Copyright {CURRENT_YEAR} the HuggingFace Team. All rights reserved. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. """.lstrip() # Don't modify the following dict unless you changed the format of `models/auto/auto_mappings.py` _AUTO_MAPPING_NAMES = { "image_processing_auto.py": "IMAGE_PROCESSOR_MAPPING_NAMES", "video_processing_auto.py": "VIDEO_PROCESSOR_MAPPING_NAMES", "processing_auto.py": "PROCESSOR_MAPPING_NAMES", "feature_extraction_auto.py": "FEATURE_EXTRACTOR_MAPPING_NAMES", } ### Entrypoint def add_new_model_like( repo_path: Annotated[ str | None, typer.Argument(help="When not using an editable install, the path to the Transformers repo.") ] = None, ): """ Add a new model to the library, based on an existing one. """ ( old_model_infos, new_lowercase_name, new_model_paper_name, filenames_to_add, ) = get_user_input() _add_new_model_like_internal( repo_path=Path(repo_path) if repo_path is not None else REPO_PATH, old_model_infos=old_model_infos, new_lowercase_name=new_lowercase_name, new_model_paper_name=new_model_paper_name, filenames_to_add=filenames_to_add, ) ### Core logic class ModelInfos: """ Retrieve the basic information about an existing model classes. """ def __init__(self, lowercase_name: str): from ..models.auto.configuration_auto import CONFIG_MAPPING_NAMES from ..models.auto.feature_extraction_auto import FEATURE_EXTRACTOR_MAPPING_NAMES from ..models.auto.image_processing_auto import IMAGE_PROCESSOR_MAPPING_NAMES from ..models.auto.processing_auto import PROCESSOR_MAPPING_NAMES from ..models.auto.tokenization_auto import TOKENIZER_MAPPING_NAMES from ..models.auto.video_processing_auto import VIDEO_PROCESSOR_MAPPING_NAMES # Just to make sure it's indeed lowercase self.lowercase_name = lowercase_name.lower().replace(" ", "_").replace("-", "_") if self.lowercase_name not in CONFIG_MAPPING_NAMES: self.lowercase_name.replace("_", "-") if self.lowercase_name not in CONFIG_MAPPING_NAMES: raise ValueError(f"{lowercase_name} is not a valid model name") self.config_class = CONFIG_MAPPING_NAMES[self.lowercase_name] self.camelcase_name = self.config_class.replace("Config", "") # Get tokenizer class if self.lowercase_name in TOKENIZER_MAPPING_NAMES: self.tokenizer_class = None self.fast_tokenizer_class = TOKENIZER_MAPPING_NAMES[self.lowercase_name] self.fast_tokenizer_class = ( None if self.fast_tokenizer_class == "PreTrainedTokenizerFast" else self.fast_tokenizer_class ) else: self.tokenizer_class, self.fast_tokenizer_class = None, None self.image_processor_classes = IMAGE_PROCESSOR_MAPPING_NAMES.get(self.lowercase_name, None) self.video_processor_class = VIDEO_PROCESSOR_MAPPING_NAMES.get(self.lowercase_name, None) self.feature_extractor_class = FEATURE_EXTRACTOR_MAPPING_NAMES.get(self.lowercase_name, None) self.processor_class = PROCESSOR_MAPPING_NAMES.get(self.lowercase_name, None) def add_content_to_file(file_name: str | os.PathLike, new_content: str, add_after: str): """ A utility to add some content inside a given file. Args: file_name (`str` or `os.PathLike`): The name of the file in which we want to insert some content. new_content (`str`): The content to add. add_after (`str`): The new content is added just after the first instance matching it. """ with open(file_name, "r", encoding="utf-8") as f: old_content = f.read() before, after = old_content.split(add_after, 1) new_content = before + add_after + new_content + after with open(file_name, "w", encoding="utf-8") as f: f.write(new_content) def add_model_to_auto_mappings( repo_path: Path, old_model_infos: ModelInfos, new_lowercase_name: str, new_model_paper_name: str, filenames_to_add: list[tuple[str, bool]], ): """ Add a model to all the relevant mappings in the auto module. Args: old_model_infos (`ModelInfos`): The structure containing the class information of the old model. new_lowercase_name (`str`): The new lowercase model name. new_model_paper_name (`str`): The fully cased name (as in the official paper name) of the new model. filenames_to_add (`list[tuple[str, bool]]`): A list of tuples of all potential filenames to add for a new model, along a boolean flag describing if we should add this file or not. For example, [(`modeling_xxx.px`, True), (`configuration_xxx.py`, True), (`tokenization_xxx.py`, False),...] """ new_cased_name = "".join(x.title() for x in new_lowercase_name.replace("-", "_").split("_")) old_lowercase_name = old_model_infos.lowercase_name old_cased_name = old_model_infos.camelcase_name filenames_to_add = [ (filename.replace(old_lowercase_name, "auto"), to_add) for filename, to_add in filenames_to_add[1:] ] # fast tokenizer has the same auto mappings as normal ones corrected_filenames_to_add = [] for file, to_add in filenames_to_add: if "tokenization_auto_fast.py" in file: previous_file, previous_to_add = corrected_filenames_to_add[-1] corrected_filenames_to_add[-1] = (previous_file, previous_to_add or to_add) else: corrected_filenames_to_add.append((file, to_add)) # Add the config and image/video processor mappings directly as the handling is a bit different add_content_to_file( repo_path / "src" / "transformers" / "models" / "auto" / "auto_mappings.py", new_content=f'("{new_lowercase_name}", "{new_cased_name}Config"),\n ', add_after="CONFIG_MAPPING_NAMES = OrderedDict(\n [\n ", ) autofile = (repo_path / "src" / "transformers" / "models" / "auto" / "auto_mappings.py").read_text() for filename, to_add in corrected_filenames_to_add: if to_add: if filename in _AUTO_MAPPING_NAMES: # These are saved in `auto_mapping.py` and require a slightly diff regex match mapping_name = _AUTO_MAPPING_NAMES[filename] filename = "auto_mappings.py" # use the unified mapping filename to write content! block_match = re.search( rf"[^\w_]{mapping_name}\s*=\s*OrderedDict\(\s*\[(.*?)\]\s*\)", autofile, re.DOTALL ) block = block_match.group(1) # type: ignore matching_lines = re.findall( rf'( {{8,12}}\(\s*"{old_lowercase_name}",.*?\),\n)(?: {{4,12}}\(|\])', block, re.DOTALL ) else: # These auto mappings are filled-in manually (tokenization and modeling files) filename = filename.replace("_fast.py", ".py") file = (repo_path / "src" / "transformers" / "models" / "auto" / filename).read_text() # The regex has to be a bit complex like this as the tokenizer mapping has new lines everywhere matching_lines = re.findall( rf'( {{8,12}}\(\s*"{old_lowercase_name}",.*?\),\n)(?: {{4,12}}\(|\])', file, re.DOTALL ) for match in matching_lines: add_content_to_file( repo_path / "src" / "transformers" / "models" / "auto" / filename, new_content=match.replace(old_lowercase_name, new_lowercase_name).replace( old_cased_name, new_cased_name ), add_after=match, ) def create_doc_file(new_paper_name: str, public_classes: list[str]): """ Create a new doc file to fill for the new model. Args: new_paper_name (`str`): The fully cased name (as in the official paper name) of the new model. public_classes (`list[str]`): A list of all the public classes that the model will have in the library. """ added_note = ( "\n\n⚠️ Note that this file is in Markdown but contain specific syntax for our doc-builder (similar to MDX) that " "may not be rendered properly in your Markdown viewer.\n\n-->\n\n" ) copyright_for_markdown = re.sub(r"# ?", "", COPYRIGHT).replace("coding=utf-8\n", "<!--") + added_note doc_template = textwrap.dedent( f""" # {new_paper_name} ## Overview The {new_paper_name} model was proposed in [<INSERT PAPER NAME HERE>](<INSERT PAPER LINK HERE>) by <INSERT AUTHORS HERE>. <INSERT SHORT SUMMARY HERE> The abstract from the paper is the following: <INSERT PAPER ABSTRACT HERE> Tips: <INSERT TIPS ABOUT MODEL HERE> This model was contributed by [INSERT YOUR HF USERNAME HERE](https://huggingface.co/<INSERT YOUR HF USERNAME HERE>). The original code can be found [here](<INSERT LINK TO GITHUB REPO HERE>). ## Usage examples <INSERT SOME NICE EXAMPLES HERE> """ ) # Add public classes doc doc_for_classes = [] for class_ in public_classes: doc = f"## {class_}\n\n[[autodoc]] {class_}" if "Model" in class_: doc += "\n - forward" doc_for_classes.append(doc) class_doc = "\n\n".join(doc_for_classes) return copyright_for_markdown + doc_template + class_doc def insert_model_in_doc_toc( repo_path: Path, old_lowercase_name: str, new_lowercase_name: str, new_model_paper_name: str ): """ Insert the new model in the doc `_toctree.yaml`, in the same section as the old model. Args: old_lowercase_name (`str`): The old lowercase model name. new_lowercase_name (`str`): The new lowercase model name. new_model_paper_name (`str`): The fully cased name (as in the official paper name) of the new model. """ toc_file = repo_path / "docs" / "source" / "en" / "_toctree.yml" with open(toc_file, "r", encoding="utf-8") as f: content = f.read() toc_match = re.search(rf"- local: model_doc/{old_lowercase_name}\n {{8}}title: .*?\n", content) if toc_match is None: raise ValueError(f"Could not find TOC entry for {old_lowercase_name}") old_model_toc = toc_match.group(0) new_toc = f" - local: model_doc/{new_lowercase_name}\n title: {new_model_paper_name}\n" add_content_to_file( repo_path / "docs" / "source" / "en" / "_toctree.yml", new_content=new_toc, add_after=old_model_toc ) def create_init_file(old_lowercase_name: str, new_lowercase_name: str, filenames_to_add: list[tuple[str, bool]]): """ Create the `__init__.py` file to add in the new model folder. Args: old_lowercase_name (`str`): The old lowercase model name. new_lowercase_name (`str`): The new lowercase model name. filenames_to_add (`list[tuple[str, bool]]`): A list of tuples of all potential filenames to add for a new model, along a boolean flag describing if we should add this file or not. For example, [(`modeling_xxx.px`, True), (`configuration_xxx.py`, True), (`tokenization_xxx.py`, False),...] """ filenames_to_add = [ (filename.replace(old_lowercase_name, new_lowercase_name).replace(".py", ""), to_add) for filename, to_add in filenames_to_add ] imports = "\n ".join(f"from .{file} import *" for file, to_add in filenames - src/transformers/cli/chat.py (29700 bytes)
# Copyright 2025 The HuggingFace Team. All rights reserved. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. import asyncio import json import os import platform import re import string import time from collections.abc import AsyncIterator, Awaitable from typing import Annotated, Any from urllib.parse import urljoin, urlparse, urlunparse import typer import yaml from huggingface_hub import AsyncInferenceClient, ChatCompletionStreamOutput from huggingface_hub.utils import httpx from transformers import GenerationConfig from transformers.utils import is_rich_available try: import readline # noqa importing this enables GNU readline capabilities except ImportError: # some platforms may not support readline: https://docs.python.org/3/library/readline.html pass if platform.system() != "Windows": import pwd if is_rich_available(): from rich import filesize from rich.console import Console from rich.live import Live from rich.markdown import Markdown from rich.progress import BarColumn, Progress, ProgressColumn, TextColumn, TimeElapsedColumn from rich.text import Text DEFAULT_HTTP_ENDPOINT = {"hostname": "localhost", "port": 8000} ALLOWED_KEY_CHARS = set(string.ascii_letters + string.whitespace) ALLOWED_VALUE_CHARS = set( string.ascii_letters + string.digits + string.whitespace + r".!\"#$%&'()*+,\-/:<=>?@[]^_`{|}~" ) DEFAULT_EXAMPLES = { "llama": {"text": "There is a Llama in my lawn, how can I get rid of it?"}, "code": { "text": ( "Write a Python function that integrates any Python function f(x) numerically over an arbitrary " "interval [x_start, x_end]." ), }, "helicopter": {"text": "How many helicopters can a human eat in one sitting?"}, "numbers": {"text": "Count to 10 but skip every number ending with an 'e'"}, "birds": {"text": "Why aren't birds real?"}, "socks": {"text": "Why is it important to eat socks after meditating?"}, "numbers2": {"text": "Which number is larger, 9.9 or 9.11?"}, } # Printed at the start of a chat session HELP_STRING_MINIMAL = """ **TRANSFORMERS CHAT INTERFACE** Chat interface to try out a model. Besides chatting with the model, here are some basic commands: - **!help**: shows all available commands (set generation settings, save chat, etc.) - **!status**: shows the current status of the model and generation settings - **!clear**: clears the current conversation and starts a new one - **!exit**: closes the interface """ # Printed when the user types `help` in the chat session HELP_STRING = f""" **TRANSFORMERS CHAT INTERFACE HELP** Full command list: - **!help**: shows this help message - **!clear**: clears the current conversation and starts a new one - **!status**: shows the current status of the model and generation settings - **!example {{NAME}}**: loads example named `{{NAME}}` from the config and uses it as the user input. Available example names: `{"`, `".join(DEFAULT_EXAMPLES.keys())}` - **!set {{ARG_1}}={{VALUE_1}} {{ARG_2}}={{VALUE_2}}** ...: changes the system prompt or generation settings (multiple settings are separated by a space). Accepts the same flags and format as the `generate_flags` CLI argument. If you're a new user, check this basic flag guide: https://huggingface.co/docs/transformers/llm_tutorial#common-options - **!save {{SAVE_NAME}} (optional)**: saves the current chat and settings to file by default to `./chat_history/{{MODEL_ID}}/chat_{{DATETIME}}.yaml` or `{{SAVE_NAME}}` if provided - **!exit**: closes the interface """ def get_service_root_url(base_url: str) -> str: """Return the service root of `base_url`, where the serve management endpoints live.""" parsed = urlparse(base_url) path = parsed.path.rstrip("/").removesuffix("/v1") return urlunparse((parsed.scheme, parsed.netloc, path, "", "", "")) class RichInterface: def __init__(self, model_id: str, user_id: str, base_url: str): self._console = Console() self.model_id = model_id self.user_id = user_id self.base_url = base_url async def stream_output( self, stream: Awaitable[AsyncIterator[ChatCompletionStreamOutput]] ) -> tuple[str, str | Any | None]: self._console.print(f"[bold blue]<{self.model_id}>:") with Live(console=self._console, refresh_per_second=4) as live: text = "" completion_tokens = 0 start_time = time.time() finish_reason: str | None = None async for token in await stream: outputs = token.choices[0].delta.content finish_reason = getattr(token.choices[0], "finish_reason", finish_reason) usage = getattr(token, "usage", None) if usage is not None: completion_tokens = getattr(usage, "completion_tokens", completion_tokens) if not outputs: continue # Escapes single words encased in <>, e.g. <think> -> \<think\>, for proper rendering in Markdown. # It only escapes single words that may have `_`, optionally following a `/` (e.g. </think>) outputs = re.sub(r"<(/*)(\w*)>", r"\<\1\2\>", outputs) text += outputs # Render the accumulated text as Markdown # NOTE: this is a workaround for the rendering "unstandard markdown" # in rich. The chatbots output treat "\n" as a new line for # better compatibility with real-world text. However, rendering # in markdown would break the format. It is because standard markdown # treat a single "\n" in normal text as a space. # Our workaround is adding two spaces at the end of each line. # This is not a perfect solution, as it would # introduce trailing spaces (only) in code block, but it works well # especially for console output, because in general the console does not # care about trailing spaces. lines = [] for line in text.splitlines(): lines.append(line) if line.startswith("```"): # Code block marker - do not add trailing spaces, as it would # break the syntax highlighting lines.append("\n") else: lines.append(" \n") markdown = Markdown("".join(lines).strip(), code_theme="github-dark") # Update the Live console output live.update(markdown, refresh=True) elapsed = time.time() - start_time if elapsed > 0 and completion_tokens > 0: tok_per_sec = completion_tokens / elapsed self._console.print() self._console.print(f"[dim]{completion_tokens} tokens in {elapsed:.1f}s ({tok_per_sec:.1f} tok/s)[/dim]") self._console.print() return text, finish_reason def input(self) -> str: """Gets user input from the console.""" input = self._console.input(f"[bold red]<{self.user_id}>:\n") self._console.print() return input def clear(self): """Clears the console.""" self._console.clear() def print_user_message(self, text: str): """Prints a user message to the console.""" self._console.print(f"[bold red]<{self.user_id}>:[/ bold red]\n{text}") self._console.print() def print_color(self, text: str, color: str): """Prints text in a given color to the console.""" self._console.print(f"[bold {color}]{text}") self._console.print() def confirm(self, message: str, default: bool = False) -> bool: """Displays a yes/no prompt to the user, returning True for confirmation.""" default_hint = "Y/n" if default else "y/N" response = self._console.input(f"[bold yellow]{message} ({default_hint}): ") self._console.print() response = response.strip().lower() if not response: return default return response in {"y", "yes"} def print_help(self, minimal: bool = False): """Prints the help message to the console.""" self._console.print(Markdown(HELP_STRING_MINIMAL if minimal else HELP_STRING)) self._console.print() def print_model_load(self, model: str): class StatsColumn(ProgressColumn): def render(self, task): if not task.total: return Text("") if task.fields.get("unit") == "bytes": done = filesize.decimal(int(task.completed)) tot = filesize.decimal(int(task.total)) speed = f" {filesize.decimal(int(task.speed))}/s" if task.speed else "" if task.time_remaining is not None: eta = f" {int(task.time_remaining // 60)}:{int(task.time_remaining % 60):02d}" else: eta = "" return Text(f"{done}/{tot}{speed}{eta}", style="progress.download") return Text(f"{int(task.completed)}/{int(task.total)}") stage_labels = { "processor": "Loading processor", "config": "Loading config", "download": "Downloading files", "weights": "Loading into memory", } # Include the model name prefix in descriptions only when the terminal is wide enough. # The bar, stats, and elapsed columns need ~70 chars; the model prefix needs len(model)+5. show_model_prefix = self._console.width >= len(model) + 5 + 70 def _label(stage_key): stage_text = stage_labels.get(stage_key, stage_key) if show_model_prefix: return f"{model} → {stage_text}" return stage_text progress = Progress( TextColumn("[bold]{task.description}"), BarColumn(bar_width=40), StatsColumn(), TimeElapsedColumn(), console=self._console, ) task_id = progress.add_task(_label("processor"), total=None) cached = False url = urljoin(get_service_root_url(self.base_url) + "/", "load_model") with ( httpx.stream("POST", url, json={"model": model}, timeout=None) as response, Live(progress, console=self._console, transient=True), ): response.raise_for_status() for line in response.iter_lines(): if not line or not line.startswith("data: "): continue event = json.loads(line[6:]) status = event.get("status") if status == "ready": cached = event.get("cached", False) break if status == "error": raise RuntimeError(event.get("message", "Unknown error")) if status == "loading": stage = event.get("stage") prog = event.get("progress") label = _label(stage) if prog: unit = "bytes" if stage == "download" else "items" progress.update( task_id, description=label, completed=prog["current"], total=prog.get("total"), unit=unit ) else: progress.update(task_id, description=label, completed=0, total=None) if cached: self._console.print(Markdown(f"_*{model} was already loaded.*_")) else: self._console.print(Markdown(f"_*{model} is warm.*_")) self._console.print() def print_status(self, config: GenerationConfig): """Prints the status of the model and generation settings to the console.""" self._console.print(f"[bold blue]Model: {self.model_id}\n") self._console.print(f"[bold blue]{config}") self._console.print() class Chat: """Chat with a model from the command line.""" # Defining a class to help with internal state but in practice it's just a method to call # TODO: refactor into a proper module with helpers + 1 main method def __init__( self, model_id: Annotated[str, typer.Argument(help="ID of the model to use (e.g. 'HuggingFaceTB/SmolLM3-3B').")], base_url: Annotated[ str | None, typer.Argument(help="Base url to connect to (e.g. http://localhost:8000/v1).") ] = f"http://{DEFAULT_HTTP_ENDPOINT['hostname']}:{DEFAULT_HTTP_ENDPOINT['port']}", generate_flags: Annotated[ list[str] | None, typer.Argument( help=( "Flags to pass to `generate`, using a space as a separator between flags. Accepts booleans, numbers, " "and lists of integers, more advanced parameterization should be set through --generation-config. " "Example: `transformers chat <model_id> <base_url> max_new_tokens=100 do_sample=False eos_token_id=[1,2]`. " "If you're a new user, check this basic flag guide: " "https://huggingface.co/docs/transformers/llm_tutorial#common-options" ) ), ] = None, # General settings user: Annotated[ str | None, typer.Option(help="Username to display in chat interface. Defaults to the current user's name."), ] = None, system_prompt: Annotated[str | None, typer.Option(help="System prompt.")] = None, save_folder: Annotated[str, typer.Option(help="Folder to save chat history.")] = "./chat_history/", examples_path: Annotated[str | None, typer.Option(help="Path to a yaml file with examples.")] = None, # Generation settings generation_config: Annotated[ str | None, typer.Option( help="Path to a local generation config file or to a HuggingFace repo containing a `generation_config.json` file. Other generation settings passed as CLI arguments will be applied on top of this generation config." ), ] = None, ) -> None: """Chat with a model from the command line.""" self.base_url = base_url parsed = urlparse(self.base_url) if parsed.h - src/transformers/cli/download.py (1692 bytes)
# Copyright 2020 The HuggingFace Team. All rights reserved. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. from typing import Annotated import typer def download( model_id: Annotated[str, typer.Argument(help="The model ID to download")], cache_dir: Annotated[str | None, typer.Option(help="Directory where to save files.")] = None, force_download: Annotated[ bool, typer.Option(help="If set, the files will be downloaded even if they are already cached locally.") ] = False, trust_remote_code: Annotated[ bool, typer.Option( help="Whether or not to allow for custom models defined on the Hub in their own modeling files. Use only if you've reviewed the code as it will execute on your local machine" ), ] = False, ): """Download a model and its tokenizer from the Hub.""" from ..models.auto import AutoModel, AutoTokenizer AutoModel.from_pretrained( model_id, cache_dir=cache_dir, force_download=force_download, trust_remote_code=trust_remote_code ) AutoTokenizer.from_pretrained( model_id, cache_dir=cache_dir, force_download=force_download, trust_remote_code=trust_remote_code ) - src/transformers/cli/serve.py (10432 bytes)
# Copyright 2025 The HuggingFace Team. All rights reserved. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. """ CLI entry point for `transformers serve`. """ import asyncio import enum import json import threading from typing import Annotated import typer from transformers.utils import logging from transformers.utils.import_utils import is_serve_available from .serving.utils import set_torch_seed logger = logging.get_logger(__name__) class ReasoningMode(str, enum.Enum): ON = "on" OFF = "off" AUTO = "auto" class Serve: def __init__( self, force_model: Annotated[ str | None, typer.Argument( help=( "Model to preload and use for all requests. GGUF weights are named `<repo>:<file>.gguf`, " "and their config and tokenizer come from the file itself." ) ), ] = None, # Model options continuous_batching: Annotated[ bool, typer.Option(help="Enable continuous batching with paged attention. Configure with --cb-* flags."), ] = False, attn_implementation: Annotated[ str | None, typer.Option(help="Attention implementation (e.g. flash_attention_2).") ] = None, compile: Annotated[bool, typer.Option(help="Enable torch.compile for faster inference.")] = False, quantization: Annotated[ str | None, typer.Option(help="Quantization method: 'bnb-4bit' or 'bnb-8bit'.") ] = None, reasoning: Annotated[ ReasoningMode, typer.Option( help=( "Reasoning mode. 'auto' uses the chat template default. Only applies to models that " "support reasoning via their chat template (e.g. Qwen3, Gemma 4) — for other models " "this flag has no effect." ) ), ] = ReasoningMode.AUTO.value, # type: ignore[invalid-parameter-default] chat_template_kwargs: Annotated[ str | None, typer.Option( help=( "Default JSON kwargs forwarded to apply_chat_template " "(e.g. '{\"enable_thinking\": true}'); per-request chat_template_kwargs override these." ) ), ] = None, decision_config: Annotated[ str | None, typer.Option(help="Path to a JSON file with decision labels, temperatures, and a chat template."), ] = None, device: Annotated[str, typer.Option(help="Device for inference (e.g. 'auto', 'cuda:0', 'cpu').")] = "auto", dtype: Annotated[str | None, typer.Option(help="Override model dtype. 'auto' derives from weights.")] = "auto", trust_remote_code: Annotated[bool, typer.Option(help="Trust remote code when loading.")] = False, model_timeout: Annotated[ int, typer.Option(help="Seconds before idle model is unloaded. Ignored when force_model is set.") ] = 300, # Continuous batching tuning cb_block_size: Annotated[ int | None, typer.Option(help="KV cache block size in tokens for continuous batching.") ] = None, cb_num_blocks: Annotated[ int | None, typer.Option(help="Number of KV cache blocks for continuous batching.") ] = None, cb_max_batch_tokens: Annotated[ int | None, typer.Option(help="Maximum tokens per batch for continuous batching.") ] = None, cb_max_memory_percent: Annotated[ float | None, typer.Option(help="Max GPU memory fraction for KV cache (0.0-1.0).") ] = None, cb_use_cuda_graph: Annotated[ bool | None, typer.Option(help="Enable CUDA graphs for continuous batching.") ] = None, # Server options host: Annotated[str, typer.Option(help="Server listen address.")] = "localhost", port: Annotated[int, typer.Option(help="Server listen port.")] = 8000, enable_cors: Annotated[bool, typer.Option(help="Enable permissive CORS.")] = False, log_level: Annotated[str, typer.Option(help="Logging level (e.g. 'info', 'warning').")] = "warning", default_seed: Annotated[int | None, typer.Option(help="Default torch seed.")] = None, non_blocking: Annotated[ bool, typer.Option(hidden=True, help="Run server in a background thread. Used by tests.") ] = False, ) -> None: if not is_serve_available(): raise ImportError("Missing dependencies for serving. Install with `pip install transformers[serving]`") import uvicorn from .serving.chat_completion import ChatCompletionHandler from .serving.completion import CompletionHandler from .serving.model_manager import ModelManager from .serving.response import ResponseHandler from .serving.server import build_server from .serving.systemone import SystemOneHandler from .serving.transcription import TranscriptionHandler from .serving.utils import GenerationState # Seed if default_seed is not None: set_torch_seed(default_seed) # Logging transformers_logger = logging.get_logger("transformers") transformers_logger.setLevel(logging.log_levels[log_level.lower()]) self._model_manager = ModelManager( device=device, dtype=dtype, trust_remote_code=trust_remote_code, attn_implementation=attn_implementation, continuous_batching=continuous_batching, quantization=quantization, model_timeout=model_timeout, force_model=force_model, ) from transformers import ContinuousBatchingConfig cb_kwargs = { k: v for k, v in { "block_size": cb_block_size, "num_blocks": cb_num_blocks, "max_batch_tokens": cb_max_batch_tokens, "max_memory_percent": cb_max_memory_percent, "use_cuda_graph": cb_use_cuda_graph, }.items() if v is not None } cb_config = ContinuousBatchingConfig(**cb_kwargs) if cb_kwargs else None self._generation_state = GenerationState( continuous_batching=continuous_batching, compile=compile, cb_config=cb_config, ) if chat_template_kwargs: chat_template_kwargs = json.loads(chat_template_kwargs) if not isinstance(chat_template_kwargs, dict): raise typer.BadParameter("--chat-template-kwargs must be a JSON object") else: chat_template_kwargs = {} if reasoning == ReasoningMode.ON: chat_template_kwargs["enable_thinking"] = True elif reasoning == ReasoningMode.OFF: chat_template_kwargs["enable_thinking"] = False self._chat_handler = ChatCompletionHandler( model_manager=self._model_manager, generation_state=self._generation_state, chat_template_kwargs=chat_template_kwargs, ) self._completion_handler = CompletionHandler( model_manager=self._model_manager, generation_state=self._generation_state, ) self._response_handler = ResponseHandler( model_manager=self._model_manager, generation_state=self._generation_state, chat_template_kwargs=chat_template_kwargs, ) self._transcription_handler = TranscriptionHandler(self._model_manager, self._generation_state) self._systemone_handler = SystemOneHandler( model_manager=self._model_manager, generation_state=self._generation_state, chat_template_kwargs=chat_template_kwargs, decision_config=decision_config, ) app = build_server( self._model_manager, self._chat_handler, completion_handler=self._completion_handler, response_handler=self._response_handler, transcription_handler=self._transcription_handler, systemone_handler=self._systemone_handler, generation_state=self._generation_state, enable_cors=enable_cors, ) config = uvicorn.Config(app, host=host, port=port, log_level="info") self.server = uvicorn.Server(config) if non_blocking: self.start_server() else: self.server.run() def start_server(self): def _run(): loop = asyncio.new_event_loop() asyncio.set_event_loop(loop) loop.run_until_complete(self.server.serve()) self._thread = threading.Thread(target=_run, name="uvicorn-thread", daemon=False) self._thread.start() def reset_loaded_models(self): """Clear all loaded models from memory.""" self._model_manager.shutdown() def kill_server(self): self._generation_state.shutdown() self._model_manager.shutdown() if not self._thread or not self._thread.is_alive(): return self.server.should_exit = True self._thread.join(timeout=2) Serve.__doc__ = """ Run a FastAPI server to serve models on-demand with an OpenAI compatible API. Models will be loaded and unloaded automatically based on usage and a timeout. \b Endpoints: POST /v1/chat/completions — Chat completions (streaming + non-streaming). POST /v1/completions — Legacy text completions from a prompt. POST /v1/systemone — System One typed decisions (choice, score, yes/no) with probabilities. GET /v1/models — Lists available models. GET /health — Health check. Requires FastAPI and Uvicorn: pip install transformers[serving] """ - src/transformers/cli/serving/__init__.py (708 bytes)
# Copyright 2025 The HuggingFace Team. All rights reserved. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. from .model_manager import ModelManager from .server import build_server from .utils import Modality
- src/transformers/cli/serving/chat_completion.py (17344 bytes)
# Copyright 2026 The HuggingFace Team. All rights reserved. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. """ Handler for the /v1/chat/completions endpoint. Supports streaming (SSE via DirectStreamer) and non-streaming (JSON) responses. """ import asyncio import time from collections.abc import AsyncGenerator from typing import TYPE_CHECKING from ...utils import logging from ...utils.import_utils import is_serve_available if is_serve_available(): from fastapi.responses import JSONResponse, StreamingResponse from openai.types.chat import ( ChatCompletion, ChatCompletionMessage, ChatCompletionMessageToolCall, ) from openai.types.chat.chat_completion import Choice from openai.types.chat.chat_completion_chunk import ( ChatCompletionChunk, ChoiceDelta, ChoiceDeltaToolCall, ) from openai.types.chat.chat_completion_chunk import Choice as ChoiceChunk from openai.types.chat.completion_create_params import CompletionCreateParamsStreaming from openai.types.completion_usage import CompletionUsage from .utils import ( BaseGenerateManager, BaseHandler, Modality, ReasoningText, ToolCall, _StreamError, build_response_parser, parse_assistant_message, ) if TYPE_CHECKING: from transformers import GenerationConfig, PreTrainedModel, PreTrainedTokenizerFast, ProcessorMixin class TransformersCompletionCreateParamsStreaming(CompletionCreateParamsStreaming, total=False): generation_config: str seed: int chat_template_kwargs: dict # Fields accepted by the OpenAI schema but not yet supported. # Receiving these raises an error to avoid silent misbehaviour. # NOTE: "stop" is NOT in this set — we map it to stop_strings. UNUSED_CHAT_COMPLETION_FIELDS = { "audio", "function_call", "functions", "logprobs", "metadata", "modalities", "n", "parallel_tool_calls", "prediction", "presence_penalty", "reasoning_effort", "response_format", "service_tier", "store", "stream_options", "tool_choice", "top_logprobs", "user", "web_search_options", } logger = logging.get_logger(__name__) class ChatCompletionHandler(BaseHandler): """Handler for the `/v1/chat/completions` endpoint. Supports both streaming (SSE) and non-streaming (JSON) responses. """ _valid_params_class = TransformersCompletionCreateParamsStreaming _unused_fields = UNUSED_CHAT_COMPLETION_FIELDS async def handle_request(self, body: dict, request_id: str) -> StreamingResponse | JSONResponse: """Validate the request, load the model, and dispatch to streaming or non-streaming. Args: body (`dict`): The raw JSON request body (OpenAI chat completion format). request_id (`str`): Unique request identifier (from header or auto-generated). Returns: `StreamingResponse | JSONResponse`: SSE stream or JSON depending on ``body["stream"]``. """ self._validate_request(body) model_id, model, processor = self._resolve_model(body) modality = self.model_manager.get_model_modality(model, processor=processor) use_cb = self.generation_state.use_continuous_batching(model, modality) logger.warning(f"[Request received] Model: {model_id}, CB: {use_cb}") gen_manager = self.generation_state.get_manager(model_id, use_cb=use_cb) processor_inputs = self.get_processor_inputs_from_messages(body["messages"], modality) has_video = any( c.get("type") == "video" for msg in processor_inputs for c in (msg.get("content") if isinstance(msg.get("content"), list) else []) ) # Default to 32 frames for video (Gemma 4 default); some processors load all frames otherwise. # Merge order (later wins): custom default -> server default → request-level kwargs. chat_template_kwargs: dict = {} if has_video: chat_template_kwargs["num_frames"] = 32 chat_template_kwargs.update(self.chat_template_kwargs) chat_template_kwargs.update(body.get("chat_template_kwargs", {})) inputs = processor.apply_chat_template( processor_inputs, add_generation_prompt=True, tools=body.get("tools"), return_tensors=None if use_cb else "pt", return_dict=True, tokenize=True, load_audio_from_video=modality == Modality.MULTIMODAL and has_video, **chat_template_kwargs, ) if not use_cb: inputs = inputs.to(model.device) # type: ignore[union-attr] gen_config = self._build_generation_config(body, model.generation_config, use_cb=use_cb) # TODO: remove when CB supports per-request generation config if use_cb: gen_manager.init_cb(model, gen_config) streaming = body.get("stream") if streaming: return self._streaming( request_id, model, processor, model_id, inputs, gen_config, gen_manager=gen_manager, tools=body.get("tools"), ) else: return await self._non_streaming( request_id, model, processor, model_id, inputs, gen_config, gen_manager=gen_manager, tools=body.get("tools"), ) # ----- streaming ----- def _streaming( self, request_id: str, model: "PreTrainedModel", processor: "ProcessorMixin | PreTrainedTokenizerFast", model_id: str, inputs: dict, gen_config: "GenerationConfig", gen_manager: BaseGenerateManager, tools=None, ) -> StreamingResponse: """Stream tokens as SSE via DirectStreamer.""" response_parser = build_response_parser(processor, model, inputs["input_ids"], tools=tools) queue, streamer = gen_manager.generate_streaming( model, processor, inputs, gen_config, request_id=request_id, response_parser=response_parser, ) input_ids = inputs["input_ids"] # CB returns plain lists, regular path returns tensors input_len = len(input_ids) if isinstance(input_ids, list) else input_ids.shape[-1] async def sse_gen() -> AsyncGenerator[str, None]: try: yield self._build_chunk_sse(request_id, role="assistant", model=model_id) tool_call_index = 0 has_tool_calls = False done = False while not done: item = await queue.get() batch = [item] try: while True: batch.append(queue.get_nowait()) except asyncio.QueueEmpty: pass sse_parts: list[str] = [] for item in batch: if item is None: done = True break if isinstance(item, _StreamError): sse_parts.append(f'data: {{"error": "{item.msg}"}}\n\n') yield "".join(sse_parts) return if isinstance(item, ToolCall): has_tool_calls = True sse_parts.append( self._build_chunk_sse( request_id, model=model_id, tool_calls=[ ChoiceDeltaToolCall( index=tool_call_index, type="function", id=f"{request_id}_tool_call_{tool_call_index}", function={"name": item.name, "arguments": item.arguments}, ) ], ) ) tool_call_index += 1 elif isinstance(item, ReasoningText): sse_parts.append(self._build_chunk_sse(request_id, model=model_id, reasoning_content=item)) else: sse_parts.append(self._build_chunk_sse(request_id, model=model_id, content=item)) if sse_parts: yield "".join(sse_parts) hit_max = gen_config.max_new_tokens is not None and streamer.total_tokens >= gen_config.max_new_tokens if has_tool_calls: finish_reason = "tool_calls" elif hit_max: finish_reason = "length" else: finish_reason = "stop" usage = CompletionUsage( prompt_tokens=input_len, completion_tokens=streamer.total_tokens, total_tokens=input_len + streamer.total_tokens, ) yield self._build_chunk_sse( request_id, finish_reason=finish_reason, model=model_id, usage=usage, ) except (GeneratorExit, asyncio.CancelledError): # Client disconnected — abort generation to free GPU. # Re-raise is mandatory: Python raises RuntimeError if GeneratorExit is swallowed. streamer.cancel() raise return StreamingResponse(sse_gen(), media_type="text/event-stream") # ----- non-streaming ----- async def _non_streaming( self, request_id: str, model: "PreTrainedModel", processor: "ProcessorMixin | PreTrainedTokenizerFast", model_id: str, inputs: dict, gen_config: "GenerationConfig", gen_manager: BaseGenerateManager, tools=None, ) -> JSONResponse: """Run generation and return a JSONResponse.""" content, input_len, generated_ids = await gen_manager.generate_non_streaming( model, processor, inputs, gen_config, request_id=request_id ) hit_max = gen_config.max_new_tokens is not None and len(generated_ids) >= gen_config.max_new_tokens completion_tokens = len(generated_ids) usage = CompletionUsage( prompt_tokens=input_len, completion_tokens=completion_tokens, total_tokens=input_len + completion_tokens, ) content, reasoning_content, parsed_tool_calls = parse_assistant_message( processor, model, generated_ids, input_ids=inputs["input_ids"], cleaned_content=content, tools=tools ) tool_calls = None if parsed_tool_calls: tool_calls = [ ChatCompletionMessageToolCall( id=f"{request_id}_tool_call_{i}", type="function", function={"name": tc.name, "arguments": tc.arguments}, ) for i, tc in enumerate(parsed_tool_calls) ] if tool_calls is not None: finish_reason = "tool_calls" elif hit_max: finish_reason = "length" else: finish_reason = "stop" return JSONResponse( self._build_completion( request_id, content, model_id, finish_reason=finish_reason, usage=usage, tool_calls=tool_calls, reasoning_content=reasoning_content, ), media_type="application/json", ) # ----- helpers ----- def _build_generation_config(self, body: dict, model_generation_config: "GenerationConfig", use_cb: bool = False): """Apply Chat Completions params (``max_completion_tokens``, ``max_tokens``, ``frequency_penalty``, ``logit_bias``, ``stop``) on top of the base generation config.""" generation_config = super()._build_generation_config(body, model_generation_config, use_cb=use_cb) if body.get("max_completion_tokens") is not None: generation_config.max_new_tokens = int(body["max_completion_tokens"]) elif body.get("max_tokens") is not None: generation_config.max_new_tokens = int(body["max_tokens"]) if body.get("frequency_penalty") is not None: generation_config.repetition_penalty = 1.0 + float(body["frequency_penalty"]) if body.get("logit_bias") is not None: generation_config.sequence_bias = {(int(k),): v for k, v in body["logit_bias"].items()} if body.get("stop") is not None: generation_config.stop_strings = body["stop"] return generation_config # ----- response builders ----- def _build_completion( self, request_id: str, content: str, model_id: str, finish_reason: str, usage: CompletionUsage | None = None, tool_calls: list[dict] | None = None, reasoning_content: str | None = None, ) -> dict: """Build a non-streaming ChatCompletion response dict. Args: request_id (`str`): Unique request identifier. content (`str`): The generated text. model_id (`str`): Model ID to include in the response. finish_reason (`str`): Why generation stopped (``"stop"``, ``"length"``, ``"tool_calls"``). usage (`CompletionUsage`, *optional*): Token usage statistics. tool_calls (`list[dict]`, *optional*): Parsed tool calls, if any. reasoning_content (`str`, *optional*): Chain-of-thought content extracted from the response. Returns: `dict`: Serialized ``ChatCompletion`` ready for JSON response. """ # reasoning_content is added as an extra field (base types set extra="allow") # we use model_validate rather than __init__ to avoid ty raising errors for the extra field message = ChatCompleti - src/transformers/cli/serving/completion.py (10223 bytes)
# Copyright 2026 The HuggingFace Team. All rights reserved. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. """ Handler for the /v1/completions endpoint (OpenAI legacy Completions API). Accepts a freeform text prompt (no chat template) and returns generated text in choices[].text. Supports streaming and non-streaming modes, and suffix for fill-in-the-middle text insertion. """ import asyncio import time from collections.abc import AsyncGenerator from typing import TYPE_CHECKING from ...utils import logging from ...utils.import_utils import is_serve_available if is_serve_available(): from fastapi import HTTPException from fastapi.responses import JSONResponse, StreamingResponse from openai.types import Completion, CompletionChoice, CompletionUsage from openai.types.completion_create_params import CompletionCreateParamsBase from .utils import BaseGenerateManager, BaseHandler, _StreamError if TYPE_CHECKING: from transformers import GenerationConfig, PreTrainedModel, PreTrainedTokenizerFast, ProcessorMixin class TransformersTextCompletionCreateParams(CompletionCreateParamsBase, total=False): generation_config: str seed: int stream: bool # Fields accepted by the OpenAI schema but not yet supported. UNUSED_LEGACY_COMPLETION_FIELDS = { "best_of", "echo", "logprobs", "n", "presence_penalty", "stream_options", "user", } logger = logging.get_logger(__name__) class CompletionHandler(BaseHandler): """Handler for the `/v1/completions` endpoint. Takes a raw text ``prompt`` (no chat template) and generates text returned in ``choices[].text``. Supports streaming (SSE) and non-streaming (JSON) responses, and ``suffix`` for fill-in-the-middle insertion. """ _valid_params_class = TransformersTextCompletionCreateParams _unused_fields = UNUSED_LEGACY_COMPLETION_FIELDS async def handle_request(self, body: dict, request_id: str) -> "StreamingResponse | JSONResponse": """Validate the request, load the model, and dispatch to streaming or non-streaming. Args: body (`dict`): The raw JSON request body (OpenAI legacy completions format). request_id (`str`): Unique request identifier (from header or auto-generated). Returns: `StreamingResponse | JSONResponse`: SSE stream or JSON depending on ``body["stream"]``. """ self._validate_request(body) prompt = body.get("prompt", "") if not isinstance(prompt, str): raise HTTPException(status_code=400, detail="prompt must be a string.") model_id, model, processor = self._resolve_model(body) modality = self.model_manager.get_model_modality(model, processor=processor) use_cb = self.generation_state.use_continuous_batching(model, modality) logger.warning(f"[Request received] Model: {model_id}, CB: {use_cb}") gen_manager = self.generation_state.get_manager(model_id, use_cb=use_cb) tokenizer = getattr(processor, "tokenizer", processor) inputs = tokenizer(prompt, return_tensors=None if use_cb else "pt") if not use_cb: inputs = inputs.to(model.device) gen_config = self._build_generation_config(body, model.generation_config, use_cb=use_cb) if use_cb: gen_manager.init_cb(model, gen_config) suffix = body.get("suffix") streaming = body.get("stream") if streaming: return self._streaming(request_id, model, processor, model_id, inputs, gen_config, gen_manager, suffix) else: return await self._non_streaming( request_id, model, processor, model_id, inputs, gen_config, gen_manager, suffix ) # ----- streaming ----- def _streaming( self, request_id: str, model: "PreTrainedModel", processor: "ProcessorMixin | PreTrainedTokenizerFast", model_id: str, inputs: dict, gen_config: "GenerationConfig", gen_manager: BaseGenerateManager, suffix: str | None = None, ) -> "StreamingResponse": """Stream tokens as SSE.""" queue, streamer = gen_manager.generate_streaming(model, processor, inputs, gen_config, request_id=request_id) input_ids = inputs["input_ids"] input_len = len(input_ids) if isinstance(input_ids, list) else input_ids.shape[-1] async def sse_gen() -> AsyncGenerator[str, None]: try: done = False while not done: text = await queue.get() batch = [text] try: while True: batch.append(queue.get_nowait()) except asyncio.QueueEmpty: pass sse_parts: list[str] = [] for text in batch: if text is None: done = True break if isinstance(text, _StreamError): sse_parts.append(f'data: {{"error": "{text.msg}"}}\n\n') yield "".join(sse_parts) return sse_parts.append(self._build_chunk_sse(request_id, model_id, text=text)) if sse_parts: yield "".join(sse_parts) hit_max = gen_config.max_new_tokens is not None and streamer.total_tokens >= gen_config.max_new_tokens finish_reason = "length" if hit_max else "stop" if suffix is not None: yield self._build_chunk_sse(request_id, model_id, text=suffix) usage = CompletionUsage( prompt_tokens=input_len, completion_tokens=streamer.total_tokens, total_tokens=input_len + streamer.total_tokens, ) yield self._build_chunk_sse(request_id, model_id, finish_reason=finish_reason, usage=usage) except (GeneratorExit, asyncio.CancelledError): streamer.cancel() raise return StreamingResponse(sse_gen(), media_type="text/event-stream") # ----- non-streaming ----- async def _non_streaming( self, request_id: str, model: "PreTrainedModel", processor: "ProcessorMixin | PreTrainedTokenizerFast", model_id: str, inputs: dict, gen_config: "GenerationConfig", gen_manager: BaseGenerateManager, suffix: str | None = None, ) -> "JSONResponse": """Run generation and return a JSONResponse.""" text, input_len, generated_ids = await gen_manager.generate_non_streaming( model, processor, inputs, gen_config, request_id=request_id ) if suffix is not None: text = text + suffix completion_tokens = len(generated_ids) hit_max = gen_config.max_new_tokens is not None and completion_tokens >= gen_config.max_new_tokens finish_reason = "length" if hit_max else "stop" usage = CompletionUsage( prompt_tokens=input_len, completion_tokens=completion_tokens, total_tokens=input_len + completion_tokens, ) result = Completion( id=request_id, created=int(time.time()), model=model_id, choices=[ CompletionChoice( text=text, index=0, logprobs=None, finish_reason=finish_reason, ) ], object="text_completion", usage=usage, ) return JSONResponse(result.model_dump(exclude_none=True), media_type="application/json") # ----- helpers ----- def _build_chunk_sse( self, request_id: str, model_id: str, text: str = "", finish_reason: str | None = None, usage: "CompletionUsage | None" = None, ) -> str: """Build a streaming ``Completion`` chunk and format it as an SSE ``data:`` line. Uses ``model_construct`` to bypass pydantic validation so that ``finish_reason`` can be ``None`` for mid-stream chunks (the OpenAI SDK's ``CompletionChoice`` only accepts literal values). """ chunk = Completion.model_construct( id=request_id, object="text_completion", created=int(time.time()), model=model_id, choices=[ CompletionChoice.model_construct( text=text, index=0, logprobs=None, finish_reason=finish_reason, ) ], usage=usage, ) return self.chunk_to_sse(chunk) # ----- generation config ----- def _build_generation_config(self, body: dict, model_generation_config: "GenerationConfig", use_cb: bool = False): """Apply legacy completion params (``max_tokens``, ``frequency_penalty``, ``stop``) on top of base config.""" generation_config = super()._build_generation_config(body, model_generation_config, use_cb=use_cb) if body.get("max_tokens") is not None: generation_config.max_new_tokens = int(body["max_tokens"]) if body.get("frequency_penalty") is not None: generation_config.repetition_penalty = 1.0 + float(body["frequency_penalty"]) if body.get("stop") is not None: generation_config.stop_strings = body["stop"] return generation_config - src/transformers/cli/serving/model_manager.py (23054 bytes)
# Copyright 2026 The HuggingFace Team. All rights reserved. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. """ Model loading, caching, and lifecycle management. """ import asyncio import gc import json import os import threading from collections.abc import Callable from typing import TYPE_CHECKING from huggingface_hub import CacheNotFound import transformers from transformers import BitsAndBytesConfig, PreTrainedTokenizerBase from ...utils import logging from .utils import ( Modality, get_gen_models, make_progress_tqdm_class, model_owner, reset_torch_cache, split_model_id, split_revision, ) if TYPE_CHECKING: from transformers import PreTrainedModel, PreTrainedTokenizerFast, ProcessorMixin logger = logging.get_logger(__name__) # Metal flash attention, the default for continuous batching on MPS where `kernels` can fetch it. METAL_FLASH_ATTN = "kernels-community/metal-flash-sdpa" class TimedModel: """Wraps a model + processor and auto-unloads them after a period of inactivity. Args: model: The loaded model. timeout_seconds: Seconds of inactivity before auto-unload. Use -1 to disable. processor: The associated processor or tokenizer. on_unload: Optional callback invoked after the model is unloaded from memory. """ def __init__( self, model: "PreTrainedModel", timeout_seconds: int, processor: "ProcessorMixin | PreTrainedTokenizerFast | None" = None, on_unload: "Callable | None" = None, ): self.model = model self._name_or_path = str(model.name_or_path) self.processor = processor self.timeout_seconds = timeout_seconds self._on_unload = on_unload self._timer = threading.Timer(self.timeout_seconds, self._timeout_reached) self._timer.start() def reset_timer(self) -> None: """Reset the inactivity timer (called on each request).""" self._timer.cancel() self._timer = threading.Timer(self.timeout_seconds, self._timeout_reached) self._timer.start() def delete_model(self) -> None: """Delete the model and processor, free GPU memory.""" if hasattr(self, "model") and self.model is not None: del self.model del self.processor self.model = None self.processor = None gc.collect() reset_torch_cache() self._timer.cancel() if self._on_unload is not None: self._on_unload() def _timeout_reached(self) -> None: if self.timeout_seconds > 0: self.delete_model() logger.info(f"{self._name_or_path} was removed from memory after {self.timeout_seconds}s of inactivity") class ModelManager: """Loads, caches, and manages the lifecycle of models. Handlers receive a reference to this and call `load_model_and_processor()` to get a model ready for inference. Args: device: Device to place models on (e.g. "auto", "cuda", "cpu"). dtype: Torch dtype override. "auto" derives from model weights. trust_remote_code: Whether to trust remote code when loading models. attn_implementation: Attention implementation override (e.g. "flash_attention_2"). continuous_batching: Whether continuous batching is enabled, to default to a Metal flash kernel on MPS. quantization: Quantization method ("bnb-4bit" or "bnb-8bit"). model_timeout: Seconds before an idle model is unloaded. -1 disables. force_model: If set, preload this model at init time. """ def __init__( self, device: str = "auto", dtype: str | None = "auto", trust_remote_code: bool = False, attn_implementation: str | None = None, continuous_batching: bool = False, quantization: str | None = None, model_timeout: int = 300, force_model: str | None = None, ): self.loaded_models: dict[str, TimedModel] = {} # Thread-safety for concurrent load_model_and_processor calls self._model_locks: dict[str, threading.Lock] = {} self._model_locks_guard = threading.Lock() # Tracks in-flight loads for fan-out to multiple SSE subscribers (used by load_model_streaming) self._loading_subscribers: dict[str, list[asyncio.Queue[str | None]]] = {} self._loading_tasks: dict[str, asyncio.Task] = {} # Convert numeric device strings (e.g. "0") to int so device_map works correctly self.device = int(device) if device.isdigit() else device self.dtype = self._resolve_dtype(dtype) self.trust_remote_code = trust_remote_code self.attn_implementation = self._resolve_attn_implementation( attn_implementation, self.device, continuous_batching ) self.quantization = quantization self.model_timeout = model_timeout self.force_model = force_model self._validate_args() # Preloaded models should never be auto-unloaded if force_model is not None: self.model_timeout = -1 # Preload the forced model after all state is initialized if force_model is not None: repo, gguf_file = split_model_id(force_model) self.load_model_and_processor(self.process_model_name(repo), gguf_file=gguf_file) @staticmethod def _resolve_dtype(dtype: str | None): import torch if dtype in ("auto", None): return dtype resolved = getattr(torch, dtype, None) if not isinstance(resolved, torch.dtype): raise ValueError( f"Unsupported dtype: '{dtype}'. Must be 'auto' or a valid torch dtype (e.g. 'float16', 'bfloat16')." ) return resolved @staticmethod def _resolve_attn_implementation( attn_implementation: str | None, device: str | int, continuous_batching: bool ) -> str | None: """Default to the Metal flash kernel for continuous batching on `mps`, when `kernels` is installed.""" if attn_implementation is not None or not continuous_batching: return attn_implementation import torch from ...utils.import_utils import is_kernels_available is_mps_device = ( isinstance(device, str) and device.startswith("mps") or (device == "auto" and torch.backends.mps.is_available() and not torch.cuda.is_available()) ) if is_mps_device and is_kernels_available(): logger.warning_once( f"Continuous batching on MPS with `kernels` installed: defaulting attention to `{METAL_FLASH_ATTN}`. " "Pass `--attn-implementation sdpa` to opt out." ) return METAL_FLASH_ATTN return attn_implementation def _validate_args(self): if self.quantization is not None and self.quantization not in ("bnb-4bit", "bnb-8bit"): raise ValueError( f"Unsupported quantization method: '{self.quantization}'. Must be 'bnb-4bit' or 'bnb-8bit'." ) from ...integrations.hub_kernels import is_kernel VALID_ATTN_IMPLEMENTATIONS = {"eager", "sdpa", "flash_attention_2", "flash_attention_3", "flex_attention"} if ( self.attn_implementation is not None and not is_kernel(self.attn_implementation) and self.attn_implementation not in VALID_ATTN_IMPLEMENTATIONS ): raise ValueError( f"Unsupported attention implementation: '{self.attn_implementation}'. " f"Must be one of {VALID_ATTN_IMPLEMENTATIONS} or a hub kernel (e.g. 'kernels-community/flash-attn2')." ) @staticmethod def process_model_name(model_id: str) -> str: """Canonical name of a model, the one it is cached under. - A Hub repo is named `'model_id@revision'`: `"org/model"` -> `"org/model@main"`, `"org/model@dev"` is kept. - An existing local path is kept as it is: it has no revision, and an `@` in it is part of the path. """ if os.path.exists(model_id): # A local path has no revision, and an `@` in it is part of the path. return model_id if "@" in model_id: # A Hub repo that already names its revision, e.g. `"org/model@dev"`. return model_id # A Hub repo with no revision: default to `main`. return f"{model_id}@main" def get_quantization_config(self) -> BitsAndBytesConfig | None: """Return a BitsAndBytesConfig based on the `quantization` setting, or None.""" if self.quantization == "bnb-4bit": return BitsAndBytesConfig( load_in_4bit=True, bnb_4bit_quant_type="nf4", bnb_4bit_use_double_quant=True, ) elif self.quantization == "bnb-8bit": return BitsAndBytesConfig(load_in_8bit=True) return None def _load_processor( self, model_id_and_revision: str, gguf_file: str | None = None ) -> "ProcessorMixin | PreTrainedTokenizerFast": """Load a processor for the given model. Args: model_id_and_revision: Model ID in ``'model_id@revision'`` format. """ from transformers import AutoProcessor, AutoTokenizer model_id, revision = split_revision(model_id_and_revision) if gguf_file is not None: # A GGUF repo ships no tokenizer files: the vocabulary is read out of the file's metadata. return AutoTokenizer.from_pretrained( model_id, gguf_file=gguf_file, revision=revision, trust_remote_code=self.trust_remote_code ) return AutoProcessor.from_pretrained(model_id, revision=revision, trust_remote_code=self.trust_remote_code) def _load_model( self, model_id_and_revision: str, tqdm_class: type | None = None, progress_callback: Callable | None = None, gguf_file: str | None = None, ) -> "PreTrainedModel": """Load a model. Args: model_id_and_revision (`str`): Model ID in ``'model_id@revision'`` format. tqdm_class (*optional*): tqdm subclass for progress bars during ``from_pretrained``. progress_callback (`Callable`, *optional*): Called with progress dicts during loading. Returns: `PreTrainedModel`: The loaded model. """ from transformers import AutoConfig model_id, revision = split_revision(model_id_and_revision) model_kwargs = { "revision": revision, "attn_implementation": self.attn_implementation, "dtype": self.dtype, "device_map": self.device, "trust_remote_code": self.trust_remote_code, "tqdm_class": tqdm_class, } quantization_config = self.get_quantization_config() if quantization_config is not None: model_kwargs["quantization_config"] = quantization_config if gguf_file is not None: model_kwargs["gguf_file"] = gguf_file if progress_callback is not None: progress_callback({"status": "loading", "model": model_id_and_revision, "stage": "config"}) config = AutoConfig.from_pretrained(model_id, **model_kwargs) from transformers import AutoModelForDecision from transformers.models.auto.modeling_auto import MODEL_FOR_MULTIMODAL_LM_MAPPING_NAMES # A decision model, or a checkpoint shipping a decision head next to its weights: loaded like the pipeline does try: return AutoModelForDecision.from_pretrained(model_id, **model_kwargs) except ValueError: # not a decision model, nor a checkpoint with a decision head: as the pipeline does pass if config.model_type in MODEL_FOR_MULTIMODAL_LM_MAPPING_NAMES: from transformers import AutoModelForMultimodalLM return AutoModelForMultimodalLM.from_pretrained(model_id, **model_kwargs) architecture = getattr(transformers, config.architectures[0]) return architecture.from_pretrained(model_id, **model_kwargs) def load_model_and_processor( self, model_id_and_revision: str, progress_callback: Callable | None = None, tqdm_class: type | None = None, gguf_file: str | None = None, ) -> "tuple[PreTrainedModel, ProcessorMixin | PreTrainedTokenizerFast]": """Load a model (or return it from cache), resetting its inactivity timer. Args: model_id_and_revision: Model ID in ``'model_id@revision'`` format. progress_callback: If provided, called with dicts like ``{"status": "loading", "model": ..., "stage": ...}`` during loading. tqdm_class: Optional tqdm subclass for progress bars during ``from_pretrained``. """ # Cached under the weights, not the repository: two quantizations of one repo are two models. key = model_id_and_revision if gguf_file is None else f"{model_id_and_revision}:{gguf_file}" # Per-model lock prevents duplicate loads when concurrent requests arrive with self._model_locks_guard: lock = self._model_locks.setdefault(key, threading.Lock()) with lock: if key not in self.loaded_models: logger.warning(f"Loading {key}") if progress_callback is not None: progress_callback({"status": "loading", "model": key, "stage": "processor"}) processor = self._load_processor(model_id_and_revision, gguf_file=gguf_file) model = self._load_model( model_id_and_revision, tqdm_class=tqdm_class, progress_callback=progress_callback, gguf_file=gguf_file, ) self.loaded_models[key] = TimedModel( model, timeout_seconds=self.model_timeout, processor=processor, on_unload=lambda cached=key: self.loaded_models.pop(cached, None), ) if progress_callback is not None: progress_callback({"status": "ready", "model": key, "cached": False}) else: self.loaded_models[key].reset_timer() - src/transformers/cli/serving/response.py (29530 bytes)
# Copyright 2026 The HuggingFace Team. All rights reserved. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. """ Handler for the /v1/responses endpoint (OpenAI Responses API). Supports streaming (SSE) and non-streaming (JSON) responses. """ import asyncio import time from collections.abc import AsyncGenerator from typing import TYPE_CHECKING from ...utils import logging from ...utils.import_utils import is_serve_available if is_serve_available(): from fastapi import HTTPException from fastapi.responses import JSONResponse, StreamingResponse from openai.types.responses import ( Response, ResponseCompletedEvent, ResponseContentPartAddedEvent, ResponseContentPartDoneEvent, ResponseCreatedEvent, ResponseError, ResponseErrorEvent, ResponseFailedEvent, ResponseFunctionCallArgumentsDoneEvent, ResponseFunctionToolCall, ResponseInProgressEvent, ResponseOutputItemAddedEvent, ResponseOutputItemDoneEvent, ResponseOutputMessage, ResponseOutputText, ResponseReasoningItem, ResponseReasoningTextDeltaEvent, ResponseReasoningTextDoneEvent, ResponseTextDeltaEvent, ResponseTextDoneEvent, ) from openai.types.responses.response_create_params import ResponseCreateParamsStreaming from openai.types.responses.response_usage import InputTokensDetails, OutputTokensDetails, ResponseUsage from .utils import ( BaseGenerateManager, BaseHandler, Modality, ReasoningText, ToolCall, _StreamError, build_response_parser, parse_assistant_message, ) if TYPE_CHECKING: from transformers import GenerationConfig, PreTrainedModel, PreTrainedTokenizerFast, ProcessorMixin logger = logging.get_logger(__name__) class TransformersResponseCreateParamsStreaming(ResponseCreateParamsStreaming, total=False): generation_config: str seed: int UNUSED_RESPONSE_FIELDS = { "background", "include", "max_tool_calls", "previous_response_id", "prompt", "service_tier", "store", "text", "tool_choice", "top_logprobs", "truncation", "user", } class _ResponseStreamBuilder: """Builds SSE events for one streaming Responses API generation.""" def __init__(self, *, request_id: str, response_defaults: dict): self._response_defaults = response_defaults self.resp_id = f"resp_{request_id}" self.msg_id = f"msg_{request_id}" self.reasoning_id = f"rs_{request_id}" self.seq = 0 self.output_index = 0 self.content = "" self.full_reasoning = "" self.reasoning_open = False self.message_open = False self.reasoning_item: ResponseReasoningItem | None = None self.message_item: ResponseOutputMessage | None = None self.tool_calls: list[ResponseFunctionToolCall] = [] def _emit(self, event) -> str: sse = BaseHandler.chunk_to_sse(event) self.seq += 1 return sse def _response(self, status: str, **extra) -> "Response": return Response(**self._response_defaults, status=status, **extra) def start_response(self) -> list[str]: return [ self._emit( ResponseCreatedEvent( type="response.created", sequence_number=self.seq, response=self._response("queued", output=[]), ) ), self._emit( ResponseInProgressEvent( type="response.in_progress", sequence_number=self.seq, response=self._response("in_progress", output=[]), ) ), ] def start_reasoning(self) -> list[str]: self.reasoning_open = True return [ self._emit( ResponseOutputItemAddedEvent( type="response.output_item.added", sequence_number=self.seq, output_index=self.output_index, item=ResponseReasoningItem( id=self.reasoning_id, type="reasoning", summary=[], content=[], status="in_progress" ), ) ) ] def reasoning_delta(self, text: str) -> list[str]: self.full_reasoning += text return [ self._emit( ResponseReasoningTextDeltaEvent( type="response.reasoning_text.delta", item_id=self.reasoning_id, sequence_number=self.seq, output_index=self.output_index, content_index=0, delta=text, ) ) ] def finish_reasoning(self) -> list[str]: self.reasoning_item = ResponseReasoningItem( id=self.reasoning_id, type="reasoning", summary=[], content=[{"type": "reasoning_text", "text": self.full_reasoning}], status="completed", ) parts = [ self._emit( ResponseReasoningTextDoneEvent( type="response.reasoning_text.done", item_id=self.reasoning_id, sequence_number=self.seq, output_index=self.output_index, content_index=0, text=self.full_reasoning, ) ), self._emit( ResponseOutputItemDoneEvent( type="response.output_item.done", sequence_number=self.seq, output_index=self.output_index, item=self.reasoning_item, ) ), ] self.reasoning_open = False self.output_index += 1 return parts def start_message(self) -> list[str]: self.message_open = True return [ self._emit( ResponseOutputItemAddedEvent( type="response.output_item.added", sequence_number=self.seq, output_index=self.output_index, item=ResponseOutputMessage( id=self.msg_id, type="message", status="in_progress", role="assistant", content=[] ), ) ), self._emit( ResponseContentPartAddedEvent( type="response.content_part.added", item_id=self.msg_id, sequence_number=self.seq, output_index=self.output_index, content_index=0, part=ResponseOutputText(type="output_text", text="", annotations=[]), ) ), ] def text_delta(self, text: str) -> list[str]: self.content += text return [ self._emit( ResponseTextDeltaEvent( type="response.output_text.delta", item_id=self.msg_id, sequence_number=self.seq, output_index=self.output_index, content_index=0, delta=text, logprobs=[], ) ) ] def finish_message(self) -> list[str]: output_text_part = ResponseOutputText(type="output_text", text=self.content, annotations=[]) self.message_item = ResponseOutputMessage( id=self.msg_id, type="message", status="completed", role="assistant", content=[output_text_part], annotations=[], # type: ignore[call-arg] ) parts = [ self._emit( ResponseTextDoneEvent( type="response.output_text.done", item_id=self.msg_id, sequence_number=self.seq, output_index=self.output_index, content_index=0, text=self.content, logprobs=[], ) ), self._emit( ResponseContentPartDoneEvent( type="response.content_part.done", item_id=self.msg_id, sequence_number=self.seq, output_index=self.output_index, content_index=0, part=output_text_part, ) ), self._emit( ResponseOutputItemDoneEvent( type="response.output_item.done", sequence_number=self.seq, output_index=self.output_index, item=self.message_item, ) ), ] self.message_open = False return parts def tool_call(self, tc_id: str, name: str, arguments: str) -> list[str]: self.output_index += 1 item = ResponseFunctionToolCall( id=tc_id, call_id=tc_id, type="function_call", name=name, arguments=arguments, status="completed", ) self.tool_calls.append(item) return [ self._emit( ResponseOutputItemAddedEvent( type="response.output_item.added", sequence_number=self.seq, output_index=self.output_index, item=item, ) ), self._emit( ResponseFunctionCallArgumentsDoneEvent( type="response.function_call_arguments.done", sequence_number=self.seq, item_id=tc_id, output_index=self.output_index, arguments=arguments, name=name, ) ), self._emit( ResponseOutputItemDoneEvent( type="response.output_item.done", sequence_number=self.seq, output_index=self.output_index, item=item, ) ), ] def error(self, msg: str) -> list[str]: return [ self._emit(ResponseErrorEvent(type="error", sequence_number=self.seq, message=msg)), self._emit( ResponseFailedEvent( type="response.failed", sequence_number=self.seq, response=self._response( "failed", output=[], error=ResponseError(code="server_error", message=msg) ), ) ), ] def completed(self, usage) -> list[str]: all_output: list = [] if self.reasoning_item is not None: all_output.append(self.reasoning_item) all_output.append(self.message_item) all_output.extend(self.tool_calls) return [ self._emit( ResponseCompletedEvent( type="response.completed", sequence_number=self.seq, response=self._response("completed", output=all_output, usage=usage), ) ) ] class ResponseHandler(BaseHandler): """Handler for the ``/v1/responses`` endpoint.""" _valid_params_class = TransformersResponseCreateParamsStreaming _unused_fields = UNUSED_RESPONSE_FIELDS async def handle_request(self, body: dict, request_id: str) -> StreamingResponse | JSONResponse: """Validate, load model, dispatch to streaming or non-streaming. Args: body (`dict`): The raw JSON request body (OpenAI Responses API format). request_id (`str`): Unique request identifier (from header or auto-generated). Returns: `StreamingResponse | JSONResponse`: SSE stream or JSON depending on ``body["stream"]``. """ self._validate_request(body) model_id, model, processor = self._resolve_model(body) modality = self.model_manager.get_model_modality(model, processor=processor) use_cb = self.generation_state.use_continuous_batching(model, modality) logger.warning(f"[Request received] Model: {model_id}, CB: {use_cb}") gen_manager = self.generation_state.get_manager(model_id, use_cb=use_cb) # Two-step input conversion (chat completions skips step 1 since messages are already standard): # 1. Normalize Responses API input (string/list/dict + instructions) → standard messages list # 2. Transform message content for the HF processor (VLM image handling, text joining, etc.) messages = self._normalize_input(body) processor_inputs = self.get_processor_inputs_from_messages(messages, modality) has_video = any( c.get("type") == "video" for msg in processor_inputs for c in (msg.get("content") if isinstance(msg.get("content"), list) else []) ) # Default to 32 frames for video (Gemma 4 default); some processors load all frames otherwise. # Merge order (later wins): custom default -> server default → request-level kwargs. chat_template_kwargs: dict = {} if has_video: chat_template_kwargs["num_frames"] = 32 chat_template_kwargs.update(self.chat_template_kwargs) chat_template_kwargs.update(body.get("chat_template_kwargs") or {}) # updates the flat tool structure to the one expected by the `apply_chat_template` method. tools = self._normalize_tools(body.get("tools")) inputs = processor.apply_chat_template( processor_inputs, add_generation_prompt=True, tools=tools, return_tensors=None if use_cb else "pt", return_dict=True, tokenize=True, load_audio_from_video=modality == Modality.MULTIMODAL and has_video, **chat_template_kwargs, ) if not use_cb: inputs = inputs.to(model.device) # type: ignore[union-attr] gen_config = self._build_generation_config(body, model.generation_config, use_cb=use_cb) # TODO: remove when CB supports per-request generation config if use_cb: gen_manager.init_cb(model, gen_config) stream - src/transformers/cli/serving/server.py (5668 bytes)
# Copyright 2026 The HuggingFace Team. All rights reserved. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. """ FastAPI app factory. """ import uuid from contextlib import asynccontextmanager from typing import TYPE_CHECKING from ...utils import logging from ...utils.import_utils import is_serve_available if is_serve_available(): from fastapi import FastAPI, Request from fastapi.middleware.cors import CORSMiddleware from fastapi.responses import JSONResponse, StreamingResponse if TYPE_CHECKING: from .chat_completion import ChatCompletionHandler from .completion import CompletionHandler from .response import ResponseHandler from .systemone import SystemOneHandler from .transcription import TranscriptionHandler from .model_manager import ModelManager from .utils import X_REQUEST_ID, CBWorkerDeadError, GenerationState, split_model_id logger = logging.get_logger(__name__) def build_server( model_manager: ModelManager, chat_handler: "ChatCompletionHandler", completion_handler: "CompletionHandler", response_handler: "ResponseHandler", transcription_handler: "TranscriptionHandler", systemone_handler: "SystemOneHandler", generation_state: GenerationState, enable_cors: bool = False, ) -> "FastAPI": """Build and return a configured FastAPI application. Args: model_manager: Handles model loading, caching, and cleanup. chat_handler: Handles `/v1/chat/completions` requests. response_handler: Handles `/v1/responses` requests. systemone_handler: Handles `/v1/systemone` requests. generation_state: Owns the per-model generation managers (regular and CB). Passed in here so `/health` can check whether the CB worker has died and respond with 503 instead of a misleading 200. enable_cors: If `True`, adds permissive CORS middleware (allow all origins). Returns: A FastAPI app ready to be passed to uvicorn. """ @asynccontextmanager async def lifespan(app: FastAPI): yield model_manager.shutdown() app = FastAPI(lifespan=lifespan) @app.exception_handler(CBWorkerDeadError) async def _cb_dead_handler(_request: Request, exc: CBWorkerDeadError): # Map CBWorkerDeadError to 503; otherwise it'd fall through to Starlette's default 500. return JSONResponse({"error": str(exc)}, status_code=503) if enable_cors: app.add_middleware( CORSMiddleware, allow_origins=["*"], allow_credentials=True, allow_methods=["*"], allow_headers=["*"], ) logger.warning_once("CORS allow origin is set to `*`. Not recommended for production.") # ---- Middleware ---- @app.middleware("http") async def request_id_middleware(request: Request, call_next): """Get or set the request ID in the header.""" request_id = request.headers.get(X_REQUEST_ID) or str(uuid.uuid4()) request.state.request_id = request_id response = await call_next(request) response.headers[X_REQUEST_ID] = request_id return response # ---- Routes ---- @app.post("/v1/chat/completions") async def chat_completions(request: Request, body: dict): return await chat_handler.handle_request(body, request.state.request_id) @app.post("/v1/completions") async def completions(request: Request, body: dict): return await completion_handler.handle_request(body, request.state.request_id) @app.post("/v1/responses") async def responses(request: Request, body: dict): return await response_handler.handle_request(body, request.state.request_id) @app.post("/v1/systemone") async def systemone(request: Request, body: dict): return await systemone_handler.handle_request(body, request.state.request_id) @app.post("/v1/audio/transcriptions") async def audio_transcriptions(request: Request): return await transcription_handler.handle_request(request) @app.post("/load_model") async def load_model(body: dict): from fastapi import HTTPException model = body.get("model") if model is None: raise HTTPException(status_code=422, detail="Missing `model` field in the request body.") model, gguf_file = split_model_id(model) model_id_and_revision = model_manager.process_model_name(model) return StreamingResponse( model_manager.load_model_streaming(model_id_and_revision, gguf_file=gguf_file), media_type="text/event-stream", ) @app.post("/reset") def reset(): model_manager.shutdown() return JSONResponse({"status": "ok"}) @app.get("/v1/models") @app.options("/v1/models") def list_models(): return JSONResponse({"object": "list", "data": model_manager.list_models()}) @app.get("/health") def health(): if not generation_state.is_cb_alive(): return JSONResponse({"status": "unhealthy", "reason": "cb_worker_dead"}, status_code=503) return JSONResponse({"status": "ok"}) return app - src/transformers/cli/serving/systemone.py (6754 bytes)
# Copyright 2026 The HuggingFace Team. All rights reserved. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. """ Handler for the /v1/systemone endpoint (System One typed decisions, in Jev's request and answer shapes). Answers typed questions (`choice`, `score`, `noul`) about a `state`, and the images of its `assets`, with a probability for every allowed answer, by running the served model through [`DecisionPipeline`]. No text is generated. """ from typing import TYPE_CHECKING, Any from ...decision import DecisionConfig, parse_questions from ...utils import logging from ...utils.import_utils import is_serve_available if is_serve_available(): from fastapi import HTTPException from fastapi.responses import JSONResponse from pydantic import BaseModel, ConfigDict, Field, ValidationError from .utils import BaseHandler, GenerateManager if TYPE_CHECKING: from transformers import PreTrainedModel, PreTrainedTokenizerFast, ProcessorMixin from transformers.pipelines import DecisionPipeline from .model_manager import ModelManager from .utils import GenerationState logger = logging.get_logger(__name__) class SystemOneRequest(BaseModel): """A System One request (https://api.typesafe.ai/openapi.json), answered by [`DecisionPipeline`].""" model_config = ConfigDict(extra="forbid") model: str state: str | dict[str, Any] | list[Any] # Validated by `parse_questions`, like the pipeline's questions: dict[str, dict[str, Any]] = Field(min_length=1, max_length=64) # Images keyed by the placeholder they replace in the state, instructions or criteria; checked by the template assets: dict[str, dict[str, Any]] = Field(default_factory=dict, max_length=4) class SystemOneHandler(BaseHandler): """Handler for the `/v1/systemone` endpoint. Requests are answered by a [`DecisionPipeline`] around the served model, so the prompts, images, heads and answers are the pipeline's. Returns one JSON response; streaming is not supported. """ def __init__( self, model_manager: "ModelManager", generation_state: "GenerationState", chat_template_kwargs: dict | None = None, decision_config: DecisionConfig | str | None = None, ): super().__init__(model_manager, generation_state, chat_template_kwargs) # Without one, each model's pipeline reads the decision config of its repo self.decision_config = ( DecisionConfig.from_file(decision_config) if isinstance(decision_config, str) else decision_config ) self.pipelines: dict[str, DecisionPipeline] = {} def _validate_request(self, body: dict) -> SystemOneRequest: """Validate the entire request, questions included, before resolving or loading a model.""" if "model" not in body and self.model_manager.force_model is not None: body = {**body, "model": self.model_manager.force_model} try: request = SystemOneRequest.model_validate(body, strict=True) except ValidationError as error: detail = [{**item, "loc": ("body", *item["loc"])} for item in error.errors(include_url=False)] raise HTTPException(status_code=422, detail=detail) from error try: parse_questions(request.questions, self.decision_config or DecisionConfig()) except ValueError as error: detail = [{"loc": ("body", "questions"), "msg": str(error), "type": "value_error"}] raise HTTPException(status_code=422, detail=detail) from error return request def _pipeline( self, model_id: str, model: "PreTrainedModel", processor: "ProcessorMixin | PreTrainedTokenizerFast" ) -> "DecisionPipeline": """The decision pipeline of the model, built again when the model was reloaded.""" from transformers import ProcessorMixin from transformers.pipelines import DecisionPipeline pipe = self.pipelines.get(model_id) if pipe is None or pipe.model is not model: pipe = self.pipelines[model_id] = DecisionPipeline( model=model, tokenizer=getattr(processor, "tokenizer", processor), processor=processor if isinstance(processor, ProcessorMixin) else None, decision_config=self.decision_config, ) return pipe @staticmethod def _answer(pipe: "DecisionPipeline", request: dict[str, Any]) -> tuple[dict[str, Any], int]: """The pipeline's steps, in the inference thread: the answers and the number of input tokens.""" model_inputs = pipe.preprocess(request) answers = pipe.postprocess(pipe.forward(model_inputs)) return answers, sum(inputs["input_ids"].shape[-1] for inputs, _ in model_inputs["prompts"]) async def handle_request(self, body: dict, request_id: str) -> "JSONResponse": """Validate the request, load the model, and answer every question. Args: body (`dict`): The raw JSON request body (System One format). request_id (`str`): Unique request identifier (from header or auto-generated). Returns: `JSONResponse`: `model`, `answers` keyed by question name, and `usage`. """ request = self._validate_request(body) model_id, model, processor = self._resolve_model(body) logger.warning(f"[Request received] Model: {model_id}, questions: {len(request.questions)}") gen_manager: GenerateManager = self.generation_state.get_manager(model_id, use_cb=False) # type: ignore[assignment] pipe = self._pipeline(model_id, model, processor) try: answers, input_tokens = await gen_manager.async_submit( self._answer, pipe, request.model_dump(include={"state", "questions", "assets"}) ) except ValueError as error: # e.g. labels that aren't single tokens, or images for a text model raise HTTPException(status_code=400, detail=str(error)) from error return JSONResponse( { "model": body["model"], "answers": answers, "usage": {"input_tokens": input_tokens, "output_tokens": 0}, } ) - src/transformers/cli/serving/transcription.py (7814 bytes)
# Copyright 2026 The HuggingFace Team. All rights reserved. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. """ Handler for the /v1/audio/transcriptions endpoint. """ import base64 from typing import TYPE_CHECKING from ...audio_utils import load_audio from ...utils import logging from ...utils.import_utils import is_serve_available if is_serve_available(): from fastapi import HTTPException, Request from fastapi.responses import JSONResponse, StreamingResponse from openai.types.audio.transcription_create_params import TranscriptionCreateParamsBase from .model_manager import ModelManager from .utils import DirectStreamer, GenerateManager, GenerationState, _StreamError if TYPE_CHECKING: from transformers import PreTrainedModel, ProcessorMixin logger = logging.get_logger(__name__) class TransformersTranscriptionCreateParams(TranscriptionCreateParamsBase, total=False): stream: bool UNUSED_TRANSCRIPTION_FIELDS = { "chunking_strategy", "include", "language", "prompt", "response_format", "temperature", "timestamp_granularities", } class TranscriptionHandler: """Handler for ``POST /v1/audio/transcriptions``. Accepts a multipart/form-data request with an audio file and model name, runs speech-to-text, and returns an OpenAI-compatible Transcription response. Standalone (does not extend :class:`BaseHandler`) because audio requests use multipart form data, not JSON bodies, and don't need generation config or validation. Shares the :class:`GenerationState` for thread safety. """ def __init__(self, model_manager: ModelManager, generation_state: GenerationState): """ Args: model_manager (`ModelManager`): Handles model loading, caching, and lifecycle. generation_state (`GenerationState`): Shared generation state for thread safety. """ self.model_manager = model_manager self.generation_state = generation_state def _validate_request(self, form_keys: set[str]) -> None: """Validate transcription request fields.""" unexpected = form_keys - getattr(TransformersTranscriptionCreateParams, "__mutable_keys__", set()) if unexpected: raise HTTPException(status_code=422, detail=f"Unexpected fields in the request: {unexpected}") unused = form_keys & UNUSED_TRANSCRIPTION_FIELDS if unused: logger.warning_once(f"Ignoring unsupported fields in the request: {unused}") async def handle_request(self, request: Request) -> JSONResponse | StreamingResponse: """Parse multipart form, run transcription, return result. Args: request (`Request`): FastAPI request containing multipart form data with ``file`` (audio bytes), ``model`` (model ID), and optional ``stream`` flag. Returns: `JSONResponse | StreamingResponse`: Transcription result or SSE stream. """ from transformers.utils.import_utils import is_multipart_available if not is_multipart_available(): raise ImportError( "Missing python-multipart dependency for file uploads. Install with `pip install python-multipart`" ) async with request.form() as form: self._validate_request(set(form.keys())) file_field = form["file"] if isinstance(file_field, str): raise HTTPException(status_code=422, detail="Expected file upload, got string") file_bytes = await file_field.read() model = form["model"] if not isinstance(model, str): raise HTTPException(status_code=422, detail="Expected model name as string") stream = str(form.get("stream", "false")).lower() == "true" model_id_and_revision = self.model_manager.process_model_name(model) audio_model, audio_processor = self.model_manager.load_model_and_processor(model_id_and_revision) base_manager = self.generation_state.get_manager(model_id_and_revision) if not isinstance(base_manager, GenerateManager): raise HTTPException(status_code=400, detail="Audio transcription requires sequential generation (not CB)") gen_manager = base_manager audio_inputs = self._prepare_audio_inputs(file_bytes, audio_processor, audio_model) if stream: return self._streaming(gen_manager, audio_model, audio_processor, audio_inputs) return await self._non_streaming(gen_manager, audio_model, audio_processor, audio_inputs) @staticmethod def _prepare_audio_inputs( file_bytes: bytes, audio_processor: "ProcessorMixin", audio_model: "PreTrainedModel" ) -> dict: """Load audio bytes and convert to model inputs.""" sampling_rate = audio_processor.feature_extractor.sampling_rate audio_array = load_audio(base64.b64encode(file_bytes).decode("ascii"), sampling_rate=sampling_rate) audio_inputs = audio_processor(audio_array, sampling_rate=sampling_rate, return_tensors="pt").to( audio_model.device ) audio_inputs["input_features"] = audio_inputs["input_features"].to(audio_model.dtype) return audio_inputs async def _non_streaming( self, gen_manager: GenerateManager, audio_model: "PreTrainedModel", audio_processor: "ProcessorMixin", audio_inputs: dict, ) -> JSONResponse: # Audio models have different inputs (input_features) and decode (batch_decode) # than text models, so we use async_submit() directly instead of # generate_non_streaming() from openai.types.audio import Transcription generated_ids = await gen_manager.async_submit(audio_model.generate, **audio_inputs) text = audio_processor.batch_decode(generated_ids, skip_special_tokens=True)[0] return JSONResponse(Transcription(text=text).model_dump(exclude_none=True)) def _streaming( self, gen_manager: GenerateManager, audio_model: "PreTrainedModel", audio_processor: "ProcessorMixin", audio_inputs: dict, ) -> StreamingResponse: # Same as _non_streaming — uses submit() directly because audio inputs # differ from text. import asyncio tokenizer = audio_processor.tokenizer if hasattr(audio_processor, "tokenizer") else audio_processor loop = asyncio.get_running_loop() queue: asyncio.Queue = asyncio.Queue() streamer = DirectStreamer(tokenizer._tokenizer, loop, queue) gen_kwargs = {**audio_inputs, "streamer": streamer} def _run(): try: audio_model.generate(**gen_kwargs) except Exception as e: loop.call_soon_threadsafe(queue.put_nowait, _StreamError(str(e))) gen_manager.submit(_run) async def sse_gen(): while True: text = await queue.get() if text is None: break if isinstance(text, _StreamError): yield f'data: {{"error": "{text.msg}"}}\n\n' return yield f"data: {text}\n\n" return StreamingResponse(sse_gen(), media_type="text/event-stream") - src/transformers/cli/serving/utils.py (56555 bytes)
# Copyright 2026 The HuggingFace Team. All rights reserved. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. """ Shared types, constants, and utilities for the serving layer. """ import asyncio import copy import enum import json import os import threading from abc import ABC, abstractmethod from collections.abc import Callable from concurrent.futures import Future from dataclasses import dataclass from functools import lru_cache from queue import Queue from typing import TYPE_CHECKING from huggingface_hub import scan_cache_dir from tqdm import tqdm from transformers.utils import logging from transformers.utils.chat_parsing import ResponseParser from transformers.utils.chat_template_utils import _REASONING_FIELDS if TYPE_CHECKING: import pydantic import tokenizers import torch from transformers import ( ContinuousBatchingConfig, GenerationConfig, PreTrainedModel, PreTrainedTokenizerFast, ProcessorMixin, ) from transformers.generation.continuous_batching.continuous_api import ContinuousBatchingManager from transformers.generation.continuous_batching.requests import GenerationOutput from transformers.generation.continuous_batching.scheduler import Scheduler from .model_manager import ModelManager logger = logging.get_logger(__name__) X_REQUEST_ID = "x-request-id" def split_model_id(model_id: str) -> tuple[str, str | None]: """Split the GGUF file off a model name: `"org/model:weights.gguf"` -> `("org/model", "weights.gguf")`. Only a `.gguf` file after the last colon counts. Any other colon is part of the name, as in a local """ name, colon, gguf_file = model_id.rpartition(":") names_gguf_file = bool(colon) and gguf_file.endswith(".gguf") if not names_gguf_file: return model_id, None return name, gguf_file def split_revision(model_id_and_revision: str) -> tuple[str, str | None]: """Split the revision off a model name: `"org/model@dev"` -> `("org/model", "dev")`. An existing local path has no revision, and an `@` in it is part of the path: `"/tmp/ckpt@1000"` -> `("/tmp/ckpt@1000", None)`. """ if os.path.exists(model_id_and_revision): return model_id_and_revision, None model_id, _, revision = model_id_and_revision.partition("@") return model_id, revision or None def model_owner(model_id: str) -> str: """Owner of a model for `/v1/models`: `"local"` for a local path, the organization of a Hub repo.""" if os.path.exists(model_id): return "local" return model_id.split("/")[0] if "/" in model_id else "" @lru_cache def get_gen_models(cache_dir: str | None = None) -> list[dict]: """List generative models (LLMs and VLMs) available in the HuggingFace cache. Args: cache_dir (`str`, *optional*): Path to the HuggingFace cache directory. Defaults to the standard cache location. Returns: `list[dict]`: OpenAI-compatible model list entries with ``id``, ``object``, etc. """ from transformers.models.auto.modeling_auto import ( MODEL_FOR_CAUSAL_LM_MAPPING_NAMES, MODEL_FOR_IMAGE_TEXT_TO_TEXT_MAPPING_NAMES, MODEL_FOR_MULTIMODAL_LM_MAPPING_NAMES, ) generative_architectures = { *MODEL_FOR_CAUSAL_LM_MAPPING_NAMES.values(), *MODEL_FOR_IMAGE_TEXT_TO_TEXT_MAPPING_NAMES.values(), *MODEL_FOR_MULTIMODAL_LM_MAPPING_NAMES.values(), } generative_models = [] for repo in tqdm(scan_cache_dir(cache_dir).repos): if repo.repo_type != "model": continue owned_by = repo.repo_id.split("/")[0] if "/" in repo.repo_id else "" for ref, revision_info in repo.refs.items(): # One quantization per file, each a separate model, so list files rather than the repo. gguf_files = sorted(f.file_name for f in revision_info.files if f.file_name.endswith(".gguf")) model_ids = [f"{repo.repo_id}:{gguf_file}" for gguf_file in gguf_files] # The repo itself is listed when its config names a generative architecture. config_path = next((f.file_path for f in revision_info.files if f.file_name == "config.json"), None) config = json.loads(config_path.read_text()) if config_path is not None else None architectures = (config.get("architectures") or []) if isinstance(config, dict) else [] if any(architecture in generative_architectures for architecture in architectures): model_ids.append(repo.repo_id if ref == "main" else f"{repo.repo_id}@{ref}") for model_id in model_ids: generative_models.append( {"owned_by": owned_by, "id": model_id, "object": "model", "created": int(repo.last_modified)} ) return generative_models class Modality(enum.Enum): LLM = "LLM" VLM = "VLM" MULTIMODAL = "MULTIMODAL" # supports text, image, video, and audio STT = "STT" TTS = "TTS" class _StreamError: """Sentinel to signal an error from the generate thread.""" def __init__(self, msg: str): self.msg = msg class _GenerationCancelled(Exception): """Raised inside ``DirectStreamer.put()`` to abort ``model.generate()``.""" class ReasoningText(str): """Tagged str subclass: text chunk belonging to a thinking/reasoning block. Streamers wrap reasoning text with this so handlers can route it to ``reasoning_content`` deltas instead of ``content``. """ class CBWorkerDeadError(RuntimeError): """Raised when a request is submitted to a CB worker that has died. Surfaced as 503 by the FastAPI exception handler. Carries the original error message that killed the worker so the client knows why the server is in this state. """ # Fallback response_template per model_type for tokenizers that don't ship one. _RESPONSE_TEMPLATE_FALLBACKS = { ( "qwen2", "qwen2_moe", "qwen2_vl", "qwen2_5_vl", "qwen3", "qwen3_moe", "qwen3_next", "qwen3_vl", "qwen3_vl_moe", ): { "defaults": {"role": "assistant"}, "start_anchor": "<|im_start|>assistant\n", "fields": { "reasoning_content": { "open": "<think>", "close": "</think>", "content": "text", }, "tool_calls": { # Leading \s* swallows the separator newline between consecutive tool calls (and the # blank line after content) so it can't open an empty implicit `content` region, # whose close would overwrite the real content. "open_pattern": r"\s*<tool_call>", "close": "</tool_call>", "repeats": True, "content": "json", "transform": {"type": "function", "function": "{content}"}, }, "content": { # Leading \s* keeps the whitespace between two end markers (e.g. Qwen 3.5's # "<|im_end|>\n<|endoftext|>" tail) out of the implicit content region — an # empty region close would overwrite the real content with "". "close_pattern": r"\s*(?:<\|im_end\|>|<\|endoftext\|>|<\|eot_id\|>)", "content": "text", }, }, }, # Qwen 3.5 family wraps tool calls in <tool_call>...</tool_call> around an inner # <function=NAME><parameter=KEY>VALUE</parameter></function> markup that holds the call data. # The chat template prefills the assistant turn with either "<think>\n" (thinking on) or # "<think>\n\n</think>\n\n" (default), which prefix-aware parsing picks up via start_anchor. ("qwen3_5", "qwen3_5_text", "qwen3_5_moe", "qwen3_5_moe_text"): { "defaults": {"role": "assistant"}, "start_anchor": "<|im_start|>assistant\n", "fields": { "reasoning_content": { "open": "<think>", "close": "</think>", "content": "text", }, "tool_calls": { # Leading \s* swallows the separator newline between consecutive tool calls (and the # blank line after content) so it can't open an empty implicit `content` region, # whose close would overwrite the real content. "open_pattern": r"\s*<tool_call>\s*<function=(?P<name>[^>\n]+)>", "close_pattern": r"</function>\s*</tool_call>", "repeats": True, "content": "xml-inline", "content_args": { "tag_pattern": r"<parameter=(?P<key>[^>\n]+)>\s*(?P<value>.*?)\s*</parameter>", }, "transform": {"type": "function", "function": {"name": "{name}", "arguments": "{content}"}}, }, "content": { # See the qwen2 entry: Qwen 3.5's eos is <|endoftext|>, so generations end with # "<|im_end|>\n<|endoftext|>"; the \s* keeps that "\n" from clobbering content. "close_pattern": r"\s*(?:<\|im_end\|>|<\|endoftext\|>)", "content": "text", }, }, }, ("gemma4",): { "defaults": {"role": "assistant"}, "start_anchor": ["<|turn>model\n", "<tool_response|>"], "fields": { "reasoning_content": { "open": "<|channel>thought\n", "close": "<channel|>", "content": "text", }, "tool_calls": { "open_pattern": r"<\|tool_call>call:(?P<name>\w+)", "close": "<tool_call|>", "repeats": True, "content": "json", "content_args": { "unquoted_keys": True, "string_delims": [['<|"|>', '<|"|>']], }, "transform": {"type": "function", "function": {"name": "{name}", "arguments": "{content}"}}, }, "content": { "close": ["<turn|>", "<|tool_response>", "<eos>"], "content": "text", }, }, }, } @dataclass class ToolCall: """A parsed tool call surfaced through the stream queue when the parser closes a `tool_calls` region. ``arguments`` is always a JSON string.""" name: str arguments: str def get_response_template(processor, model: "PreTrainedModel") -> dict | None: """Return the response template (new declarative format) for this model. Resolution order: 1. ``tokenizer.response_template`` if set on the tokenizer. 2. Fallback in :data:`_RESPONSE_TEMPLATE_FALLBACKS` matched by ``model.config.model_type``. 3. ``None`` (no parsing performed; raw text is streamed through). """ tokenizer = getattr(processor, "tokenizer", processor) tmpl = getattr(tokenizer, "response_template", None) if tmpl is not None: return tmpl model_type = model.config.model_type return next((v for types, v in _RESPONSE_TEMPLATE_FALLBACKS.items() if model_type in types), None) def build_response_parser( processor, model: "PreTrainedModel", input_ids, tools=None, ) -> "ResponseParser | None": """Build a streaming :class:`ResponseParser` for the current generation. Returns ``None`` when no template is available for the model (then text streams through unparsed). `input_ids` is decoded and passed to the parser as `prefix=` so templates that declare a `start_anchor` get prefill-aware parsing (e.g. for a future prefilled `<think>\\n` opener). Templates without `start_anchor` fail closed and ignore the prefix — safe for chat templates where the anchor isn't at the actual generation point (e.g. Gemma 4 tool-result followups). """ template = get_response_template(processor, model) if template is None: return None tokenizer = getattr(processor, "tokenizer", processor) return ResponseParser(template, prefix=_decode_prefix(tokenizer, input_ids), tools=tools) def _normalize_tool_call(value: dict) -> ToolCall: """Convert one parsed ``tool_calls`` entry into a :class:`ToolCall`. Every template in this codebase shapes a tool call as ``{"type": "function", "function": {"name": ..., "arguments": ...}}`` — the parser already extracted ``name`` and ``arguments`` for us. The only work left is re-serializing ``arguments`` to a JSON string, since the field uses ``"content": "json"`` so the parser hands us a parsed dict while the OpenAI API expects a string. """ fn = value["function"] args = fn["arguments"] if not isinstance(args, str): args = json.dumps(args) return ToolCall(name=fn["name"], arguments=args) def _decode_prefix(tokenizer, input_ids) -> str: """Decode ``input_ids`` to the chat prompt string the parser expects as ``prefix=``. Serve always handles one request at a time, but the upstream shape varies: ``apply_chat_template(return_tensors="pt")`` hands us a 2D ``[1, N]`` tensor while continuous batching hands us a flat ``list[int]``. Both forms hold a single sequence; this helper unwraps it before decoding.""" if hasattr(input_ids, "tolist"): input_ids = input_ids.tolist() if input_ids and isinstance(input_ids[0], list): input_ids = input_ids[0] return tokenizer.decode(input_ids) def parse_assistant_message( processor, model: "PreTrainedModel", generated_ids, input_ids, cleaned_content: str = "", tools=None, ) -> tuple[str, str | None, list[ToolCall] | None]: """Parse a full assistant message into ``(content, reasoning, tool_calls)``. Uses :meth:`tokenizer.parse_response` with the resolved response template (see :func:`get_response_template`). When no template is available, returns ``(cleaned_content, None, None)`` — i.e., surface the raw model decode. ``cleaned_content`` is the ``skip_special_tokens=True`` decode of ``generated_ids``. The parser itself runs on the raw decode (keeping special tokens so it can match delimiters like Gemma 4's ``<|tool_call>``); ``cleaned_content`` is only used on the no-template path.""" template = get_response_template(processor, model) if template is None: return cleaned_content, None, None tokenizer = getattr(processor, "tokenizer", processor) parsed = tokenizer.parse - src/transformers/cli/system.py (5932 bytes)
# Copyright 2025 The HuggingFace Team. All rights reserved. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. """Contains commands to print information about the environment and version. Usage: transformers env transformers version """ import contextlib import io import os import platform from typing import Annotated import huggingface_hub import typer from packaging.version import parse from .. import __version__ from ..integrations.deepspeed import is_deepspeed_available from ..utils import ( get_torch_version, is_accelerate_available, is_torch_available, is_torch_hpu_available, is_torch_npu_available, is_torch_supa_available, is_torch_xpu_available, ) from ..utils.import_utils import ACCELERATE_MIN_VERSION, _is_package_available def env( accelerate_config_file: Annotated[ str | None, typer.Argument(help="The accelerate config file to use for the default values in the launching script."), ] = None, ) -> None: """Print information about the environment.""" import safetensors # TODO: remove hasattr guard once safetensors >= 0.8.0 is released (adds __version__) safetensors_version = safetensors.__version__ if hasattr(safetensors, "__version__") else "unknown" accelerate_available, accelerate_version = _is_package_available("accelerate", return_version=True) if not accelerate_available: accelerate_version = "not installed" elif parse(accelerate_version) < parse(ACCELERATE_MIN_VERSION): accelerate_version += " (not in expected version range)" accelerate_config = accelerate_config_str = "not found" if is_accelerate_available(): from accelerate.commands.config import default_config_file, load_config_from_file # Get the default from the config file. if accelerate_config_file is not None or os.path.isfile(default_config_file): accelerate_config = load_config_from_file(accelerate_config_file).to_dict() accelerate_config_str = ( "\n".join([f"\t- {prop}: {val}" for prop, val in accelerate_config.items()]) if isinstance(accelerate_config, dict) else f"\t{accelerate_config}" ) pt_version = get_torch_version() pt_cuda_available = "NA" pt_accelerator = "NA" pt_version_warn = "" if pt_version == "N/A": pt_version = "not installed" elif not is_torch_available(): pt_version_warn = " (not in expected version range)" if is_torch_available(): import torch pt_version = torch.__version__ pt_cuda_available = torch.cuda.is_available() pt_xpu_available = is_torch_xpu_available() pt_npu_available = is_torch_npu_available() pt_hpu_available = is_torch_hpu_available() pt_supa_available = is_torch_supa_available() if pt_supa_available: pt_accelerator = "SUPA" elif pt_cuda_available: pt_accelerator = "CUDA" elif pt_xpu_available: pt_accelerator = "XPU" elif pt_npu_available: pt_accelerator = "NPU" elif pt_hpu_available: pt_accelerator = "HPU" deepspeed_version = "not installed" if is_deepspeed_available(): # Redirect command line output to silence deepspeed import output. with contextlib.redirect_stdout(io.StringIO()): import deepspeed deepspeed_version = deepspeed.__version__ info = { "`transformers` version": __version__, "Platform": platform.platform(), "Python version": platform.python_version(), "Huggingface_hub version": huggingface_hub.__version__, "Safetensors version": f"{safetensors_version}", "Accelerate version": f"{accelerate_version}", "Accelerate config": f"{accelerate_config_str}", "DeepSpeed version": f"{deepspeed_version}", "PyTorch version (accelerator?)": f"{pt_version} ({pt_accelerator}){pt_version_warn}", "Using distributed or parallel set-up in script?": "<fill in>", } if is_torch_available(): if pt_supa_available: info["Using SUPA in script?"] = "<fill in>" info["SUPA type"] = torch.supa.get_device_name() if hasattr(torch.version, "supa"): info["SUPA version"] = torch.version.supa elif pt_cuda_available: info["Using GPU in script?"] = "<fill in>" info["GPU type"] = torch.cuda.get_device_name() elif pt_xpu_available: info["Using XPU in script?"] = "<fill in>" info["XPU type"] = torch.xpu.get_device_name() elif pt_hpu_available and hasattr(torch, "hpu"): info["Using HPU in script?"] = "<fill in>" info["HPU type"] = torch.hpu.get_device_name() elif pt_npu_available and hasattr(torch, "npu"): info["Using NPU in script?"] = "<fill in>" info["NPU type"] = torch.npu.get_device_name() if hasattr(torch.version, "cann"): info["CANN version"] = torch.version.cann print("\nCopy-and-paste the text below in your GitHub issue and FILL OUT the two last points.\n") print(_format_dict(info)) return info def version() -> None: """Print CLI version.""" print(__version__) def _format_dict(d: dict) -> str: return "\n".join([f"- {prop}: {val}" for prop, val in d.items()]) + "\n" - src/transformers/cli/transformers.py (1248 bytes)
# Copyright 2025 The HuggingFace Team. All rights reserved. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. """Transformers CLI.""" from huggingface_hub import check_cli_update, typer_factory from transformers.cli.add_new_model_like import add_new_model_like from transformers.cli.chat import Chat from transformers.cli.download import download from transformers.cli.serve import Serve from transformers.cli.system import env, version app = typer_factory(help="Transformers CLI") app.command()(add_new_model_like) app.command(name="chat")(Chat) app.command()(download) app.command()(env) app.command(name="serve")(Serve) app.command()(version) def main(): check_cli_update("transformers") app() if __name__ == "__main__": main() - src/transformers/configuration_utils.py (82335 bytes)
# Copyright 2018 The Google AI Language Team Authors and The HuggingFace Inc. team. # Copyright (c) 2018, NVIDIA CORPORATION. All rights reserved. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. """Configuration base class and utilities.""" from __future__ import annotations import copy import json import math import os import warnings from collections.abc import Sequence from dataclasses import MISSING, dataclass, fields from functools import wraps from typing import TYPE_CHECKING, Any, ClassVar, Literal, TypeVar from huggingface_hub.dataclasses import strict from packaging import version from typing_extensions import dataclass_transform from . import __version__ from .backbone_utils import consolidate_backbone_kwargs_to_config from .dynamic_module_utils import custom_object_save from .generation.configuration_utils import GenerationConfig from .integrations.heterogeneity import HeterogeneousConfigMixin from .modeling_rope_utils import RotaryEmbeddingConfigMixin from .utils import ( CONFIG_NAME, PushToHubMixin, cached_file, copy_func, hf_api, is_torch_available, logging, resolve_revision, ) from .utils.generic import is_timm_config_dict if TYPE_CHECKING: import torch logger = logging.get_logger(__name__) # type hinting: specifying the type of config class that inherits from PreTrainedConfig SpecificPreTrainedConfigType = TypeVar("SpecificPreTrainedConfigType", bound="PreTrainedConfig") _FLOAT_TAG_KEY = "__float__" _FLOAT_TAG_VALUES = {"Infinity": float("inf"), "-Infinity": float("-inf"), "NaN": float("nan")} ALLOWED_ATTN_LAYER_TYPES = ( "full_attention", "sliding_attention", "chunked_attention", "window_attention", # non-overlapping windows usually in ViT "indexed_attention", # For indexer-based attentions "compressed_sparse_attention", # CSA, used in deepseek_v4 "heavily_compressed_attention", # HCA, used in deepseek_v4 "minimax_m3_sparse", # lightning-index sparse attention, used in minimax_m3_vl "conv", "moe", # for nemotron_h, which uses either attention, mamba or moe "hybrid", # layers that combine attention + mamba/linear-attention-shaped states (zamba2, falcon_h1, zaya1) "hybrid_sliding", # layers that combine sliding attention + linear-attention-shaped states (zaya1) "reference_sliding_attention", # UnlimitedOcr: full attention over prefill + ring-buffered sliding window over generated tokens # Recurrent layers (mamba / mamba2 / GDN / minimax-lightning) "linear_attention", ) ALLOWED_MLP_LAYER_TYPES = ( "sparse", "dense", ) # Keep a complete list of layer types as well for BC ALLOWED_LAYER_TYPES = ALLOWED_ATTN_LAYER_TYPES + ALLOWED_MLP_LAYER_TYPES # Mapping from old names to new names _LEGACY_LAYER_TYPE_REMAP = { "mamba": "linear_attention", "attention": "full_attention", "deepseek_sparse_attention": "indexed_attention", # for models with DSA indexer (GLM MoE DSA, DeepSeek V32, ...) "qwen_sparse_attention": "indexed_attention", # QSA with block-compressed indexer keys (Qwen4-Exp) } def remap_legacy_layer_types( layer_types: list[str] | None = None, config: PreTrainedConfig | None = None ) -> list[str] | None: """ Remap legacy layer types to newer convention names. Any name that does not fit one of the `_LEGACY_LAYER_TYPE_REMAP` patterns is returned unchanged. This function can either take a list of `layer_types`, in which case a remapped list is returned, or a `config`, in which case the config's `layer_types` and `mtp_layer_types` will be modified in-place, and nothing will be returned. Args: layer_types (`list[str]`, optional): Layer type names that may include legacy values. config (`PreTrainedConfig`, optional): Config on which `layer_types` and `mtp_layer_types` will be remapped in-plce if they exist. Returns: `list[str]` if `layer_types` is passed, or `None` if `config` is passed. """ if (layer_types is None) ^ (config is not None): raise ValueError("This function must take exactly one of `layer_types` or `config`") if layer_types is not None: return [_LEGACY_LAYER_TYPE_REMAP.get(t, t) for t in layer_types] else: if getattr(config, "layer_types", None) is not None: # This check should not be needed, but sometimes `layer_types` is a read-only @property (already following # correct conventions), so this avoids error when trying to `setattr` it if (remapped := remap_legacy_layer_types(config.layer_types)) != config.layer_types: config.layer_types = remapped if getattr(config, "mtp_layer_types", None) is not None: # This check should not be needed, but sometimes `mtp_layer_types` is a read-only @property (already following # correct conventions), so this avoids error when trying to `setattr` it if (remapped := remap_legacy_layer_types(config.mtp_layer_types)) != config.mtp_layer_types: config.mtp_layer_types = remapped # copied from huggingface_hub.dataclasses.strict when `accept_kwargs=True` def wrap_init_to_accept_kwargs(cls: dataclass): # Get the original dataclass-generated __init__ original_init = cls.__init__ @wraps(original_init) def __init__(self, *args, **kwargs: Any) -> None: # Extract only the fields that are part of the dataclass dataclass_fields = {f.name for f in fields(cls)} standard_kwargs = {k: v for k, v in kwargs.items() if k in dataclass_fields} # We need to call bare `__init__` without `__post_init__` but the `original_init` of # any dataclas contains a call to post-init at the end (without kwargs) if len(args) > 0: raise ValueError( f"{cls.__name__} accepts only keyword arguments, but found `{len(args)}` positional args." ) for f in fields(cls): # type: ignore if f.name in standard_kwargs: setattr(self, f.name, standard_kwargs[f.name]) elif f.default is not MISSING: setattr(self, f.name, f.default) elif f.default_factory is not MISSING: setattr(self, f.name, f.default_factory()) else: raise TypeError(f"Missing required field - '{f.name}'") # Pass any additional kwargs to `__post_init__` and let the object # decide whether to set the attr or use for different purposes (e.g. BC checks) additional_kwargs = {} for name, value in kwargs.items(): if name not in dataclass_fields: additional_kwargs[name] = value self.__post_init__(**additional_kwargs) cls.__init__ = __init__ return cls class SubConfigSpec: def __init__( self, config_class, model_type: str | None = None, init_kwargs: dict | None = None, optional: bool = False, ): """ Specification for initializing sub-configs within a composite configuration class. Args: config_class (`PretrainedConfig` or `type`): The config class used to instantiate the subconfig (e.g. `Qwen2VLVisionConfig`) or `AutoConfig` if subconfig should be resolved dynamically from input kwargs.` model_type (`str`, *optional*): Must be provided only if `config_class=AutoConfig` - indicates a default model type associated with the subconfig. init_kwargs (`dict`, *optional*): Additional keyword arguments to pass when instantiating the subconfig. optional (`bool`, *optional*, defaults to `False`): Whether this subconfig is optional in the composite model hierarchy. If not optional, an error is raised when it's explicitly set to `None` in input kwargs. """ self.init_kwargs = init_kwargs if init_kwargs is not None else {} self.config_class = config_class self.optional = optional # we can have `AutoConfig` with a default model-type (eg. LLaVA), or a subconfig # with `cls.model_type` attr set to non-empty value (e.g. Qwen2VLVisionConfig) if model_type is None and not hasattr(config_class, "model_type"): raise ValueError( "You have to provide either a valid `model_type` or an specific `config_class` " f"to init subconfigs correctly, but got model_type={model_type} and config_class={config_class}" ) self.model_type = model_type if model_type is not None else config_class.model_type def get_config_class(self, model_type: str | None = None): # Avoid circular imports - we only need the static mapping here to map from `string` to config class from transformers.models.auto.configuration_auto import CONFIG_MAPPING if model_type not in CONFIG_MAPPING: raise ValueError( f"Unrecognized model type for subconfig: {model_type}. Should contain one of {', '.join(CONFIG_MAPPING.keys())}" ) return CONFIG_MAPPING[model_type] def create_subconfig(self, key, subconfig=None, **kwargs): """ Construct a subconfig class either from provided input or from spec defaults. In case the provided input is a `dict` without `model_type`, we fallback to spec's default model-type. Provided input can only be one of - [`PreTrainedConfig`, `dict`, `None`] """ # early exit if sub-config is already a config instance if isinstance(subconfig, PreTrainedConfig): return subconfig, kwargs # Vision model backbones have their own utility for BC/Timm if key == "backbone_config": backbone_config, kwargs = consolidate_backbone_kwargs_to_config( backbone_config=subconfig, default_config_type=self.model_type, default_config_kwargs=self.init_kwargs, **kwargs, ) return backbone_config, kwargs # For BC with released models where `config.json` might contain non-existant model types, we don't try to map # config by `model_type` if the spec points to a particular class (e.g. `MyModeltextConfig`). Model-type # resolution takes action only when the spec holds `config_class=AutoConfig` if not issubclass(self.config_class, PreTrainedConfig): model_type = subconfig.get("model_type", self.model_type) if subconfig is not None else self.model_type if not model_type: raise ValueError(f"Cannot resolve `{key}`: no `model_type` found inputs or in `sub_configs_defaults`.") subconfig_cls = self.get_config_class(model_type) else: subconfig_cls = self.config_class # Copy the dict to not mutate it in-place if isinstance(subconfig, dict): pass elif subconfig is None: subconfig = self.init_kwargs logger.info(f"`{key}` is None, initializing {subconfig_cls.__name__} with default values.") else: raise TypeError(f"`{key}` must be a `dict`, `PreTrainedConfig`, or `None`, got `{type(subconfig)}`") return subconfig_cls(**subconfig), kwargs @property def default_config(self): return self.get_config_class(self.model_type)(**self.init_kwargs) def default_config_fields(self): """ Collect default subconfig kwargs resolved from subconfig's dataclass defaults and `self.init_kwargs` """ default_subconfig_class = ( self.get_config_class(self.model_type) if not issubclass(self.config_class, PreTrainedConfig) else self.config_class ) subconfig_default_fields = default_subconfig_class.default_config_fields() subconfig_default_fields.update(self.init_kwargs) return subconfig_default_fields # a small helper for BC - substitutes a class attribute class classproperty(property): def __get__(self, owner_self, owner_cls): return self.fget(owner_cls) @dataclass_transform(kw_only_default=True) @strict(accept_kwargs=True) @dataclass(repr=False) class PreTrainedConfig(PushToHubMixin, RotaryEmbeddingConfigMixin, HeterogeneousConfigMixin): # no-format r""" Base class for all configuration classes. Handles a few parameters common to all models' configurations as well as methods for loading/downloading/saving configurations. <Tip> A configuration file can be loaded and saved to disk. Loading the configuration file and using this file to initialize a model does **not** load the model weights. It only affects the model's configuration. </Tip> Class attributes (overridden by derived classes): - **model_type** (`str`) -- An identifier for the model type, serialized into the JSON file, and used to recreate the correct object in [`~transformers.AutoConfig`]. - **attribute_map** (`dict[str, str]`) -- A dict that maps model specific attribute names to the standardized naming of attributes. - **base_model_tp_plan** (`dict[str, Any]`) -- A dict that maps sub-modules FQNs of a base model to a tensor parallel plan applied to the sub-module when `model.tensor_parallel` is called. - **base_model_fsdp_plan** (`dict[Any, str]`) -- A dict that maps sub-modules of a base model to an FSDP2 sharding strategy (e.g. `"free_full_weight"` / `"keep_full_weight"`). Keys can be wildcard module paths (e.g. `"layers.*"`) or tuples of paths (grouped into a single `fully_shard` call). - **base_model_pp_plan** (`dict[str, tuple[list[str]]]`) -- A dict that maps child-modules of a base model to a pipeline parallel plan that enables users to place the child-module on the appropriate device. Common attributes (present in all subclasses): - **vocab_size** (`int`) -- The number of tokens in the vocabulary, which is also the first dimension of the embeddings matrix (this attribute may be missing for models that don't have a text modality like ViT). - **hidden_size** (`int`) -- The hidden size of the model. - **num_attention_heads** (`int`) -- The number of attention heads used in the multi-head attention layers of the model. - **num_hidden_layers** (`int`) -- The - src/transformers/conversion_mapping.py (156565 bytes)
# Copyright (C) 2025 the HuggingFace Inc. team. All rights reserved. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. from __future__ import annotations from collections import defaultdict from copy import deepcopy from typing import TYPE_CHECKING from .core_model_loading import ( Chunk, Concatenate, ErnieFuseAndSplitTextVisionExperts, GroupWeightRename, Inkling2IndexOffsetWeightRename, Inkling2LoopDequantizeExperts, Inkling2LoopMergeMessageQueries, Inkling2UnpadVocab, Interleave, LayerPairCollapse, MergeModulelist, PermuteForRope, PrefixChange, Split, Transpose, WeightConverter, WeightRenaming, WeightTransform, ) if TYPE_CHECKING: from .modeling_utils import PreTrainedModel from .quantizers import HfQuantizer _MODEL_TO_CONVERSION_PATTERN = { # Mixtral-style MoE "minimax": "mixtral", "minimax_m2": "mixtral", # Qwen2-style MoE "Dots3NoteTextModel": "qwen2_moe", "afmoe": "qwen2_moe", "deepseek_v2": "qwen2_moe", "deepseek_v3": "qwen2_moe", "deepseek_v32": "qwen2_moe", "dots1": "qwen2_moe", "ernie4_5_moe": "qwen2_moe", "glm4_moe": "qwen2_moe", "glm4_moe_lite": "qwen2_moe", "glm_moe_dsa": "qwen2_moe", "glm4v_moe": "qwen2_moe", "longcat_flash": "qwen2_moe", "solar_open": "qwen2_moe", "mellum": "qwen2_moe", "qwen3_moe": "qwen2_moe", "beam": "qwen2_moe", "qwen3_omni_moe": "qwen2_moe", "qwen3_omni_moe_thinker": "qwen2_moe", "qwen3_omni_moe_text": "qwen2_moe", "qwen3_next": "qwen2_moe", "hunyuan_v1_moe": "qwen2_moe", "flex_olmo": "qwen2_moe", "olmoe": "qwen2_moe", "sarvam_mla": "qwen2_moe", "exaone_moe": "qwen2_moe", "cohere2_moe": "qwen2_moe", "rt_detr_v2": "rt_detr", "pp_doclayout_v2": "rt_detr", "pp_doclayout_v3": "rt_detr", "sam3_tracker_video": "sam3_tracker", "AltCLIPVisionModel": "CLIPVisionModel", "ChineseCLIPVisionModel": "CLIPVisionModel", "CLIPSegVisionModel": "CLIPVisionModel", "MetaClip2VisionModel": "CLIPVisionModel", "mlcd_vision": "CLIPVisionModel", "mlcd": "CLIPVisionModel", "SiglipVisionModel": "CLIPVisionModel", "Siglip2VisionModel": "CLIPVisionModel", "xCLIPVisionModel": "CLIPVisionModel", "CLIPSegTextModel": "CLIPTextModel", "MetaClip2TextModel": "CLIPTextModel", "SiglipTextModel": "CLIPTextModel", "Siglip2TextModel": "CLIPTextModel", "xCLIPTextModel": "CLIPTextModel", "aria": "llava", "paligemma": "llava", "aya_vision": "llava", "got_ocr2": "llava", "gemma3": "llava", "internvl": "llava", "vipllava": "llava", "mistral3": "llava", "pp_chart2table": "llava", "voxtral": "qwen2_audio", "voxtral_realtime": "qwen2_audio", "audioflamingo3": "qwen2_audio", "glmasr": "qwen2_audio", "musicflamingo": "qwen2_audio", "granite_speech_plus": "granite_speech", "granitemoeshared": "granitemoe", "granitemoehybrid": "granitemoe", "gemma3n_text": "qwen3_5_text", "glm5_next_text": "glm5_next", "hy_v4_vl_text": "hy_v4", "qwen3_5_moe_text": "qwen3_5_text", "llava_next_video": "llava_next", "llava_onevision": "llava_next", # class-based mappings "AriaModel": "LlavaModel", "PaliGemmaModel": "LlavaModel", "AyaVisionModel": "LlavaModel", "GotOcr2Model": "LlavaModel", "Gemma3Model": "LlavaModel", "InternVLModel": "LlavaModel", "VipLlavaModel": "LlavaModel", "Mistral3Model": "LlavaModel", "PPChart2TableModel": "LlavaModel", "LlavaNextModel": "LlavaModel", "LlavaNextVideoModel": "LlavaModel", "LlavaOnevisionModel": "LlavaModel", "FuyuModel": "LlavaModel", "MllamaModel": "LlavaModel", "VoxtralModel": "Qwen2AudioModel", "VoxtralRealtimeModel": "Qwen2AudioModel", "AudioFlamingo3Model": "Qwen2AudioModel", "GlmAsrModel": "Qwen2AudioModel", "MusicFlamingoModel": "Qwen2AudioModel", "GraniteSpeechPlusModel": "GraniteSpeechModel", "MaskFormerDetrDecoder": "DetrModel", "Qwen2_5_VLModel": "Qwen2VLModel", "Qwen2_5_VLForConditionalGeneration": "Qwen2VLForConditionalGeneration", "qwen2_5_omni_thinker": "Qwen2_5OmniThinkerForConditionalGeneration", "Qwen3OmniMoeThinkerModel": "Qwen2_5OmniThinkerModel", "Qwen3OmniMoeThinkerForConditionalGeneration": "Qwen2_5OmniThinkerForConditionalGeneration", "Tipsv2VisionBackbone": "Tipsv2VisionModel", "UnlimitedOcrTextModel": "qwen2_moe", # ViT-style vision models (old HuggingFace checkpoint format → new modular format) "ASTModel": "ViTModel", "BeitModel": "ViTModel", "DeiTModel": "ViTModel", "IJepaModel": "ViTModel", "ViTMAEModel": "ViTModel", "ViTMSNModel": "ViTModel", "VivitModel": "ViTModel", "Dinov2Backbone": "Dinov2Model", "Dinov2WithRegistersBackbone": "Dinov2WithRegistersModel", } def _build_checkpoint_conversion_mapping(): mapping = { # Dory checkpoints store each decoder layer as an attention block followed by an mlp block "dory": [ LayerPairCollapse(), WeightRenaming(r"\.self_attn\.g_proj\.", ".self_attn.gate_proj."), WeightRenaming(r"^backbone\.", "model."), WeightRenaming(r"\.embeddings\.", ".embed_tokens."), ], "hy_v4": [ # General HC prefix which is dropped WeightRenaming(r"\.hc_pre\.hc_", ".hc_"), # Attn gating + sinks WeightRenaming(r"\.learnable_sink_param", ".sinks"), WeightRenaming(r"\.linear_gate", ".gate_proj"), # Follow DSv4 HC standards WeightRenaming(r"\.hc_attn_layer\.hc_fn", ".attn_hc.fn"), WeightRenaming(r"\.hc_attn_layer\.hc_base", ".attn_hc.base"), WeightRenaming(r"\.hc_attn_layer\.hc_scale", ".attn_hc.scale"), WeightRenaming(r"\.hc_mlp_layer\.hc_fn", ".ffn_hc.fn"), WeightRenaming(r"\.hc_mlp_layer\.hc_base", ".ffn_hc.base"), WeightRenaming(r"\.hc_mlp_layer\.hc_scale", ".ffn_hc.scale"), WeightRenaming(r"\.hc_head_fn", ".hc_fn"), WeightRenaming(r"\.hc_head_base", ".hc_base"), WeightRenaming(r"\.hc_head_scale", ".hc_scale"), ], # HY4-VL: text renames come from the scoped `hy_v4_vl_text` mapping after module placement. # Vision renames follow `hunyuan_vl`, with only `before_rms` and no image begin/end/sep embeddings. "hy_v4_vl": [ # -- vision tower: Megatron-style MLP and pre/post norm names -- WeightRenaming(source_patterns="mlp.dense_h_to_4h", target_patterns="mlp.fc1"), WeightRenaming(source_patterns="mlp.dense_4h_to_h", target_patterns="mlp.fc2"), WeightRenaming( source_patterns=r"^vit\.layers\.(\d+)\.input_layernorm\.", target_patterns=r"vit.layers.\1.layer_norm1.", ), WeightRenaming( source_patterns=r"^vit\.layers\.(\d+)\.post_attention_layernorm\.", target_patterns=r"vit.layers.\1.layer_norm2.", ), # -- projector: normalize the RMS name and give the convolutions explicit module names -- WeightRenaming(source_patterns=r"(perceive|patch_merger)\.norm\.", target_patterns=r"\1.before_rms."), WeightRenaming(source_patterns=r"perceive\.proj\.0\.", target_patterns="perceive.proj_conv."), WeightRenaming(source_patterns=r"perceive\.proj\.2\.", target_patterns="perceive.proj_out."), WeightRenaming(source_patterns=r"perceive\.", target_patterns="patch_merger."), # -- module placement: text under `language_model`, tower under `vision_tower` -- WeightRenaming( source_patterns=r"^model(?!\.(language_model|vit|vision_tower))", target_patterns="model.language_model", ), WeightRenaming(source_patterns=r"^vit", target_patterns="model.vision_tower"), ], # Cosmos3 Edge's composite checkpoint stores its dense reasoner text tower as conventional attention + MLP # blocks. The visual/projector tensors already use their native module names and intentionally need no mapping. "cosmos3_edge": [ WeightRenaming(r"^embed_tokens\.", "model.language_model.embed_tokens."), WeightRenaming(r"^norm\.", "model.language_model.norm."), WeightRenaming(r"^layers\.", "model.language_model.layers."), WeightRenaming(r"\.self_attn\.to_q\.", ".self_attn.q_proj."), WeightRenaming(r"\.self_attn\.to_k\.", ".self_attn.k_proj."), WeightRenaming(r"\.self_attn\.to_v\.", ".self_attn.v_proj."), WeightRenaming(r"\.self_attn\.to_out\.", ".self_attn.o_proj."), WeightRenaming(r"\.mlp\.up_proj\.", ".mlp.fc1."), WeightRenaming(r"\.mlp\.down_proj\.", ".mlp.fc2."), ], "inkling_mm_model": [ WeightRenaming(source_patterns=r"model\.llm\.layers", target_patterns=r"model.language_model.layers"), WeightRenaming( source_patterns=r"model\.llm\.embed_norm\.weight", target_patterns=r"model.language_model.embed_tokens.embed_norm.weight", ), WeightRenaming( source_patterns=r"model\.llm\.embed\.weight", target_patterns=r"model.language_model.embed_tokens.weight", ), WeightRenaming( source_patterns=r"model\.llm\.norm\.weight", target_patterns=r"model.language_model.norm.weight" ), WeightRenaming(source_patterns=r"model\.llm\.unembed\.weight", target_patterns=r"lm_head.weight"), WeightRenaming(source_patterns=r"model\.audio\.", target_patterns=r"model.audio_tower."), WeightRenaming(source_patterns=r"model\.visual", target_patterns=r"model.vision_tower"), # Vision encoder internals (run after the tower namespace renames) WeightRenaming( source_patterns=r"vision_tower.layers.linear_(\d+)", target_patterns=r"vision_tower.encoder_layers.\1.projection", ), WeightRenaming( source_patterns=r"vision_tower.layers.norm_(\d+)", target_patterns=r"vision_tower.encoder_layers.\1.layer_norm", ), # Audio tower internals. These run AFTER the generic `model.audio.` -> `model.audio_tower.` # rename above and substring-match its output. The audio dMel embedding moved from # `audio.encoder` to an `InklingAudioModelEmbeddings` submodule, and `final_norm` was # renamed to `norm`. WeightRenaming( source_patterns=r"audio_tower.encoder.weight", target_patterns=r"audio_tower.embed_audio_tokens.embed_audio_tokens.weight", ), WeightRenaming( source_patterns=r"audio_tower.final_norm.weight", target_patterns=r"audio_tower.norm.weight" ), # MoE and MLP # no Transpose ops here: the TP loader shards the raw tensor but validates the shard # shape on the target param, so dim-permuting conversions break sharded loads cc @cyril WeightConverter( source_patterns="shared_w13_weight", target_patterns=["gate_proj", "up_proj"], operations=[Interleave(dim=1), Chunk(dim=1)], ), WeightRenaming(source_patterns=r"shared_w2_weight", target_patterns=r"down_proj"), WeightConverter( source_patterns="mlp.experts.w13_weight", target_patterns=["mlp.experts.gate_up_proj"], operations=[Interleave(dim=1)], ), WeightRenaming(source_patterns=r"mlp.experts.w2_weight", target_patterns=r"mlp.experts.down_proj"), WeightConverter( source_patterns="mlp.w13_dn.weight", target_patterns=["mlp.gate_proj.weight", "mlp.up_proj.weight"], operations=[Interleave(dim=0), Chunk(dim=0)], ), WeightRenaming(source_patterns=r"mlp.w2_md.weight", target_patterns=r"mlp.down_proj.weight"), WeightRenaming(source_patterns=r"mlp.gate.bias", target_patterns=r"mlp.gate.e_score_correction_bias"), # Attn WeightRenaming(source_patterns=r"attn\.wq_du", target_patterns=r"self_attn.q_proj"), WeightRenaming(source_patterns=r"attn\.wk_dv", target_patterns=r"self_attn.k_proj"), WeightRenaming(source_patterns=r"attn\.wv_dv", target_patterns=r"self_attn.v_proj"), WeightRenaming(source_patterns=r"attn\.wr_du", target_patterns=r"self_attn.r_proj"), WeightRenaming(source_patterns=r"attn\.wo_ud", target_patterns=r"self_attn.o_proj"), WeightRenaming(source_patterns=r"\.attn\.q_norm", target_patterns=r".self_attn.q_norm"), WeightRenaming(source_patterns=r"\.attn\.k_norm", target_patterns=r".self_attn.k_norm"), WeightRenaming(source_patterns=r"\.attn\.k_sconv", target_patterns=r".self_attn.k_sconv.conv1d"), WeightRenaming(source_patterns=r"\.attn\.v_sconv", target_patterns=r".self_attn.v_sconv.conv1d"), WeightRenaming(source_patterns=r"\.attn\.rel_logits_proj", target_patterns=r".self_attn.rel_logits_proj"), WeightRenaming(source_patterns=r"attn_sconv\.weight", target_patterns=r"attn_sconv.conv1d.weight"), WeightRenaming(source_patterns=r"mlp_sconv\.weight", target_patterns=r"mlp_sconv.conv1d.weight"), WeightRenaming(source_patterns=r"mlp_norm", target_patterns=r"post_attention_layernorm"), WeightRenaming(source_patterns=r"attn_norm", target_patterns=r"input_layernorm"), ], "inkling2": [ WeightRenaming(source_patterns=r"model\.llm\.unembed\.", target_patterns=r"lm_head."), WeightConverter( source_patterns=r"lm_head\.weight", target_patterns=r"lm_head.weight", operations=[Inkling2UnpadVocab()], ), WeightRenaming(source_patterns=r"model\.llm\.", target_patterns=r"model.language_model."), WeightRenaming(source_patterns=r"model\.visual\.", target_patterns=r"model.vision_tower."), WeightRenaming( source_patterns=r"vision_tower\.backbone\.embeddings\.", - src/transformers/convert_slow_tokenizer.py (78570 bytes)
# Copyright 2018 The HuggingFace Inc. team. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. """ Utilities to convert slow tokenizers in their fast tokenizers counterparts. All the conversions are grouped here to gather SentencePiece dependencies outside of the fast tokenizers files and allow to make our dependency on SentencePiece optional. """ import os import re import warnings from packaging import version from tokenizers import AddedToken, Regex, Tokenizer, decoders, normalizers, pre_tokenizers, processors from tokenizers.models import BPE, Unigram, WordPiece from .integrations.mistral.constants import is_tekken_vocab_filename from .tokenization_utils_base import _get_prepend_scheme, generate_merges from .utils import is_protobuf_available, is_sentencepiece_available, logging, requires_backends from .utils.import_utils import PROTOBUF_IMPORT_ERROR logger = logging.get_logger(__name__) MBART_LANGUAGES = [ "ar_AR", "cs_CZ", "de_DE", "en_XX", "es_XX", "et_EE", "fi_FI", "fr_XX", "gu_IN", "hi_IN", "it_IT", "ja_XX", "kk_KZ", "ko_KR", "lt_LT", "lv_LV", "my_MM", "ne_NP", "nl_XX", "ro_RO", "ru_RU", "si_LK", "tr_TR", "vi_VN", "zh_CN", ] MBART50_LANGUAGES = MBART_LANGUAGES + [ "af_ZA", "az_AZ", "bn_IN", "fa_IR", "he_IL", "hr_HR", "id_ID", "ka_GE", "km_KH", "mk_MK", "ml_IN", "mn_MN", "mr_IN", "pl_PL", "ps_AF", "pt_XX", "sv_SE", "sw_KE", "ta_IN", "te_IN", "th_TH", "tl_XX", "uk_UA", "ur_PK", "xh_ZA", "gl_ES", "sl_SI", ] def import_protobuf(error_message=""): if is_sentencepiece_available(): from sentencepiece import sentencepiece_model_pb2 return sentencepiece_model_pb2 if is_protobuf_available(): import google.protobuf if version.parse(google.protobuf.__version__) < version.parse("4.0.0"): from transformers.utils import sentencepiece_model_pb2 else: from transformers.utils import sentencepiece_model_pb2_new as sentencepiece_model_pb2 return sentencepiece_model_pb2 else: raise ImportError(PROTOBUF_IMPORT_ERROR.format(error_message)) class SentencePieceExtractor: """ Extractor implementation for SentencePiece trained models. https://github.com/google/sentencepiece """ def __init__(self, model: str): requires_backends(self, "sentencepiece") requires_backends(self, "protobuf") # from .utils import sentencepiece_model_pb2 as model_pb2 model_pb2 = import_protobuf() m = model_pb2.ModelProto() with open(model, "rb") as f: m.ParseFromString(f.read()) self.proto = m def extract(self, model_type, **kwargs) -> tuple[dict[str, int], list[tuple]]: """ By default will return vocab and merges with respect to their order, by sending `vocab_scores` we're going to order the merges with respect to the piece scores instead. """ self.proto.trainer_spec.unk_id if model_type is None: from tokenizers.models import BPE, Unigram model_type = Unigram if self.proto.trainer_spec.model_type == 1 else BPE vocab = [(piece.piece, piece.score) for piece in self.proto.pieces] if model_type.__name__ != "BPE": kwargs["unk_id"] = self.proto.trainer_spec.unk_id kwargs["vocab"] = vocab else: vocab = {word: i for i, (word, score) in enumerate(vocab)} kwargs["vocab"] = vocab merges = generate_merges(vocab) kwargs["merges"] = merges # control tokens are special # user defined symbols are not # both user and control tokens are AddedTokens # Add user defined symbols (type == 4) from sentencepiece (https://github.com/google/sentencepiece/blob/6225e08edb2577757163b3f5dbba4c0b670ef445/src/sentencepiece_model.proto#L299C29-L299C33) spm_added_tokens = [(id, p.piece, p.type == 3) for id, p in enumerate(self.proto.pieces) if p.type in [3, 4]] kwargs["additional_special_tokens"] = [ AddedToken(token, normalized=False, special=special) for id, token, special in sorted(spm_added_tokens, key=lambda x: x[0]) ] kwargs["_spm_precompiled_charsmap"] = getattr(self.proto.normalizer_spec, "precompiled_charsmap", None) or None return kwargs class GemmaSentencePieceExtractor(SentencePieceExtractor): def extract(self, vocab_scores=None) -> tuple[dict[str, int], list[tuple]]: """ By default will return vocab and merges with respect to their order, by sending `vocab_scores` we're going to order the merges with respect to the piece scores instead. """ sp = self.sp vocab = {sp.id_to_piece(index): index for index in range(sp.GetPieceSize())} # If "\t" is missing in the vocab, we have to do this to support merges # "<0x09>" is the bytefallback for `\t` if "\t" not in vocab: vocab["\t"] = vocab.get("<0x09>") merges = generate_merges(vocab, vocab_scores) return vocab, merges def check_number_comma(piece: str) -> bool: return len(piece) < 2 or piece[-1] != "," or not piece[-2].isdigit() class Converter: def __init__(self, original_tokenizer): self.original_tokenizer = original_tokenizer def converted(self) -> Tokenizer: raise NotImplementedError() class BertConverter(Converter): def converted(self) -> Tokenizer: vocab = self.original_tokenizer.vocab tokenizer = Tokenizer(WordPiece(vocab, unk_token=str(self.original_tokenizer.unk_token))) tokenize_chinese_chars = False strip_accents = False do_lower_case = False if hasattr(self.original_tokenizer, "basic_tokenizer"): tokenize_chinese_chars = self.original_tokenizer.basic_tokenizer.tokenize_chinese_chars strip_accents = self.original_tokenizer.basic_tokenizer.strip_accents do_lower_case = self.original_tokenizer.basic_tokenizer.do_lower_case tokenizer.normalizer = normalizers.BertNormalizer( clean_text=True, handle_chinese_chars=tokenize_chinese_chars, strip_accents=strip_accents, lowercase=do_lower_case, ) tokenizer.pre_tokenizer = pre_tokenizers.BertPreTokenizer() cls = str(self.original_tokenizer.cls_token) sep = str(self.original_tokenizer.sep_token) cls_token_id = self.original_tokenizer.cls_token_id sep_token_id = self.original_tokenizer.sep_token_id tokenizer.post_processor = processors.TemplateProcessing( single=f"{cls}:0 $A:0 {sep}:0", pair=f"{cls}:0 $A:0 {sep}:0 $B:1 {sep}:1", special_tokens=[ (cls, cls_token_id), (sep, sep_token_id), ], ) tokenizer.decoder = decoders.WordPiece(prefix="##") return tokenizer class SplinterConverter(Converter): def converted(self) -> Tokenizer: vocab = self.original_tokenizer.vocab tokenizer = Tokenizer(WordPiece(vocab, unk_token=str(self.original_tokenizer.unk_token))) tokenize_chinese_chars = False strip_accents = False do_lower_case = False if hasattr(self.original_tokenizer, "basic_tokenizer"): tokenize_chinese_chars = self.original_tokenizer.basic_tokenizer.tokenize_chinese_chars strip_accents = self.original_tokenizer.basic_tokenizer.strip_accents do_lower_case = self.original_tokenizer.basic_tokenizer.do_lower_case tokenizer.normalizer = normalizers.BertNormalizer( clean_text=True, handle_chinese_chars=tokenize_chinese_chars, strip_accents=strip_accents, lowercase=do_lower_case, ) tokenizer.pre_tokenizer = pre_tokenizers.BertPreTokenizer() cls = str(self.original_tokenizer.cls_token) sep = str(self.original_tokenizer.sep_token) question = str(self.original_tokenizer.question_token) dot = "." cls_token_id = self.original_tokenizer.cls_token_id sep_token_id = self.original_tokenizer.sep_token_id question_token_id = self.original_tokenizer.question_token_id dot_token_id = self.original_tokenizer.convert_tokens_to_ids(".") if self.original_tokenizer.padding_side == "right": pair = f"{cls}:0 $A:0 {question} {dot} {sep}:0 $B:1 {sep}:1" else: pair = f"{cls}:0 $A:0 {sep}:0 $B:1 {question} {dot} {sep}:1" tokenizer.post_processor = processors.TemplateProcessing( single=f"{cls}:0 $A:0 {sep}:0", pair=pair, special_tokens=[ (cls, cls_token_id), (sep, sep_token_id), (question, question_token_id), (dot, dot_token_id), ], ) tokenizer.decoder = decoders.WordPiece(prefix="##") return tokenizer class FunnelConverter(Converter): def converted(self) -> Tokenizer: vocab = self.original_tokenizer.vocab tokenizer = Tokenizer(WordPiece(vocab, unk_token=str(self.original_tokenizer.unk_token))) tokenize_chinese_chars = False strip_accents = False do_lower_case = False if hasattr(self.original_tokenizer, "basic_tokenizer"): tokenize_chinese_chars = self.original_tokenizer.basic_tokenizer.tokenize_chinese_chars strip_accents = self.original_tokenizer.basic_tokenizer.strip_accents do_lower_case = self.original_tokenizer.basic_tokenizer.do_lower_case tokenizer.normalizer = normalizers.BertNormalizer( clean_text=True, handle_chinese_chars=tokenize_chinese_chars, strip_accents=strip_accents, lowercase=do_lower_case, ) tokenizer.pre_tokenizer = pre_tokenizers.BertPreTokenizer() cls = str(self.original_tokenizer.cls_token) sep = str(self.original_tokenizer.sep_token) cls_token_id = self.original_tokenizer.cls_token_id sep_token_id = self.original_tokenizer.sep_token_id tokenizer.post_processor = processors.TemplateProcessing( single=f"{cls}:2 $A:0 {sep}:0", # token_type_id is 2 for Funnel transformer pair=f"{cls}:2 $A:0 {sep}:0 $B:1 {sep}:1", special_tokens=[ (cls, cls_token_id), (sep, sep_token_id), ], ) tokenizer.decoder = decoders.WordPiece(prefix="##") return tokenizer class MPNetConverter(Converter): def converted(self) -> Tokenizer: vocab = self.original_tokenizer.vocab tokenizer = Tokenizer(WordPiece(vocab, unk_token=str(self.original_tokenizer.unk_token))) tokenize_chinese_chars = False strip_accents = False do_lower_case = False if hasattr(self.original_tokenizer, "basic_tokenizer"): tokenize_chinese_chars = self.original_tokenizer.basic_tokenizer.tokenize_chinese_chars strip_accents = self.original_tokenizer.basic_tokenizer.strip_accents do_lower_case = self.original_tokenizer.basic_tokenizer.do_lower_case tokenizer.normalizer = normalizers.BertNormalizer( clean_text=True, handle_chinese_chars=tokenize_chinese_chars, strip_accents=strip_accents, lowercase=do_lower_case, ) tokenizer.pre_tokenizer = pre_tokenizers.BertPreTokenizer() cls = str(self.original_tokenizer.cls_token) sep = str(self.original_tokenizer.sep_token) cls_token_id = self.original_tokenizer.cls_token_id sep_token_id = self.original_tokenizer.sep_token_id tokenizer.post_processor = processors.TemplateProcessing( single=f"{cls}:0 $A:0 {sep}:0", pair=f"{cls}:0 $A:0 {sep}:0 {sep}:0 $B:1 {sep}:1", # MPNet uses two [SEP] tokens special_tokens=[ (cls, cls_token_id), (sep, sep_token_id), ], ) tokenizer.decoder = decoders.WordPiece(prefix="##") return tokenizer class OpenAIGPTConverter(Converter): def converted(self) -> Tokenizer: vocab = self.original_tokenizer.encoder merges = list(self.original_tokenizer.bpe_ranks.keys()) unk_token = self.original_tokenizer.unk_token tokenizer = Tokenizer( BPE( vocab=vocab, merges=merges, dropout=None, unk_token=str(unk_token), end_of_word_suffix="</w>", fuse_unk=False, ) ) if tokenizer.token_to_id(str(unk_token)) is not None: tokenizer.add_special_tokens([str(unk_token)]) tokenizer.normalizer = normalizers.BertNormalizer(lowercase=True) tokenizer.pre_tokenizer = pre_tokenizers.BertPreTokenizer() tokenizer.decoder = decoders.BPEDecoder(suffix="</w>") return tokenizer class GPT2Converter(Converter): def converted(self, vocab: dict[str, int] | None = None, merges: list[tuple[str, str]] | None = None) -> Tokenizer: if not vocab: vocab = self.original_tokenizer.encoder if not merges: merges = list(self.original_tokenizer.bpe_ranks) tokenizer = Tokenizer( BPE( vocab=vocab, merges=merges, dropout=None, continuing_subword_prefix="", end_of_word_suffix="", fuse_unk=False, ) ) add_prefix_space = getattr(self.original_tokenizer, "add_prefix_space", False) tokenizer.pre_tokenizer = pre_tokenizers.ByteLevel(add_prefix_space=add_prefix_space) tokenizer.decoder = decoders.ByteLevel() if getattr(self.original_tokenizer, "add_bos_token", False): bos = self.original_tokenizer.bos_token bos_token_id = self.original_tokenizer.bos_token_id tokenizer.post_processor = processors.TemplateProcessing( single=f"{bos}:0 $A:0", pair=f"{bos}:0 $A:0 $B:1", special_tokens=[ (bos, bos_token_id), ], - src/transformers/convert_slow_tokenizers_checkpoints_to_fast.py (5790 bytes)
# Copyright 2018 The HuggingFace Inc. team. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. """Convert slow tokenizers checkpoints in fast (serialization format of the `tokenizers` library)""" import argparse import os from pathlib import Path import transformers from .convert_slow_tokenizer import SLOW_TO_FAST_CONVERTERS from .utils import logging logging.set_verbosity_info() logger = logging.get_logger(__name__) TOKENIZER_CLASSES = {} for name in SLOW_TO_FAST_CONVERTERS: # Special cases for tokenizers that don't have their own Fast tokenizer if name == "Phi3Tokenizer": tokenizer_class_name = "LlamaTokenizerFast" elif name == "ElectraTokenizer": tokenizer_class_name = "BertTokenizerFast" else: tokenizer_class_name = name + "Fast" try: TOKENIZER_CLASSES[name] = getattr(transformers, tokenizer_class_name) except AttributeError: # Skip tokenizers that don't have a Fast version pass def convert_slow_checkpoint_to_fast(tokenizer_name, checkpoint_name, dump_path, force_download): if tokenizer_name is not None and tokenizer_name not in TOKENIZER_CLASSES: raise ValueError(f"Unrecognized tokenizer name, should be one of {list(TOKENIZER_CLASSES.keys())}.") if tokenizer_name is None: tokenizer_names = TOKENIZER_CLASSES else: tokenizer_names = {tokenizer_name: getattr(transformers, tokenizer_name + "Fast")} logger.info(f"Loading tokenizer classes: {tokenizer_names}") for tokenizer_name in tokenizer_names: tokenizer_class = TOKENIZER_CLASSES[tokenizer_name] add_prefix = True if checkpoint_name is None: checkpoint_names = list(tokenizer_class.max_model_input_sizes.keys()) else: checkpoint_names = [checkpoint_name] logger.info(f"For tokenizer {tokenizer_class.__class__.__name__} loading checkpoints: {checkpoint_names}") for checkpoint in checkpoint_names: logger.info(f"Loading {tokenizer_class.__class__.__name__} {checkpoint}") # Load tokenizer tokenizer = tokenizer_class.from_pretrained(checkpoint, force_download=force_download) # Save fast tokenizer logger.info(f"Save fast tokenizer to {dump_path} with prefix {checkpoint} add_prefix {add_prefix}") # For organization names we create sub-directories if "/" in checkpoint: checkpoint_directory, checkpoint_prefix_name = checkpoint.split("/") dump_path_full = os.path.join(dump_path, checkpoint_directory) # Security check try: Path(dump_path_full).resolve().relative_to(Path(dump_path).resolve()) except ValueError: raise ValueError( f"Invalid checkpoint path: '{checkpoint}' attempts to escape `dump_path`: {dump_path}" ) elif add_prefix: checkpoint_prefix_name = checkpoint dump_path_full = dump_path else: checkpoint_prefix_name = None dump_path_full = dump_path logger.info(f"=> {dump_path_full} with prefix {checkpoint_prefix_name}, add_prefix {add_prefix}") if checkpoint in list(tokenizer.pretrained_vocab_files_map.values())[0]: file_path = list(tokenizer.pretrained_vocab_files_map.values())[0][checkpoint] next_char = file_path.split(checkpoint)[-1][0] if next_char == "/": dump_path_full = os.path.join(dump_path_full, checkpoint_prefix_name) checkpoint_prefix_name = None logger.info(f"=> {dump_path_full} with prefix {checkpoint_prefix_name}, add_prefix {add_prefix}") file_names = tokenizer.save_pretrained( dump_path_full, legacy_format=False, filename_prefix=checkpoint_prefix_name ) logger.info(f"=> File names {file_names}") for file_name in file_names: if not file_name.endswith("tokenizer.json"): os.remove(file_name) logger.info(f"=> removing {file_name}") if __name__ == "__main__": parser = argparse.ArgumentParser() # Required parameters parser.add_argument( "--dump_path", default=None, type=str, required=True, help="Path to output generated fast tokenizer files." ) parser.add_argument( "--tokenizer_name", default=None, type=str, help=( f"Optional tokenizer type selected in the list of {list(TOKENIZER_CLASSES.keys())}. If not given, will " "download and convert all the checkpoints from AWS." ), ) parser.add_argument( "--checkpoint_name", default=None, type=str, help="Optional checkpoint name. If not given, will download and convert the canonical checkpoints from AWS.", ) parser.add_argument( "--force_download", action="store_true", help="Re-download checkpoints.", ) args = parser.parse_args() convert_slow_checkpoint_to_fast(args.tokenizer_name, args.checkpoint_name, args.dump_path, args.force_download) - src/transformers/core_model_loading.py (96699 bytes)
# Copyright 2025 The HuggingFace Inc. team. All rights reserved. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. """Core helpers for loading model checkpoints.""" from __future__ import annotations import math import os import re import traceback from abc import ABC, abstractmethod from collections import defaultdict from collections.abc import Callable from concurrent.futures import Future, ThreadPoolExecutor from contextlib import contextmanager from copy import deepcopy from itertools import chain from typing import TYPE_CHECKING, Any import torch from .distributed.sharding_utils import DtensorShardOperation, _dtensor_from_local_like from .distributed.utils import is_dtensor from .integrations.accelerate import get_device, offload_weight from .utils import is_env_variable_true from .utils.loading_report import LoadStateDictInfo from .utils.logging import get_logger, tqdm if TYPE_CHECKING: from .modeling_utils import LoadStateDictConfig, PreTrainedModel from .quantizers import HfQuantizer logger = get_logger(__name__) def build_glob_alternation( globs: list[WeightRenaming | WeightConverter | str], ) -> tuple[re.Pattern, dict[str, str], dict[str, str]]: """ Build a single alternation regex with one named group per glob. """ src_group_to_glob: dict[str, str] = {} tgt_group_to_glob: dict[str, str] = {} branches: list[str] = [] i = 0 for glob in globs: if isinstance(glob, (WeightRenaming, WeightConverter)): for src in glob.source_patterns: group_name = f"g{i}" src_group_to_glob[group_name] = src i += 1 body = src.replace("*", r".*") branches.append(f"(?P<{group_name}>{body})") tgt_group_to_glob[group_name] = glob.target_patterns[0] # we index with the first target else: group_name = f"g{i}" src_group_to_glob[group_name] = glob i += 1 body = glob body = body.replace("*", r".*") branches.append(f"(?P<{group_name}>{body})") tgt_group_to_glob[group_name] = glob alternation = re.compile("|".join(branches)) return alternation, src_group_to_glob, tgt_group_to_glob class ConversionOps(ABC): """Base class for weight conversion operations.""" # Set True on an op that maps several sources to several targets (e.g. Ernie VL experts). # Ops outside core can opt in without core importing them. supports_many_to_many = False def __repr__(self): if hasattr(self, "dim"): return f"{self.__class__.__name__}(dim={self.dim})" else: return f"{self.__class__.__name__}" @abstractmethod def convert( self, input_dict: dict[str, Any], source_patterns: list[str], target_patterns: list[str], **kwargs ) -> dict[str, list[torch.Tensor]]: raise NotImplementedError @property def reverse_op(self) -> ConversionOps: raise NotImplementedError class _IdentityOp(ConversionOps): """Pass-through reverse op for dequantize operations. Dequantized weights are already in their target dtype and should be saved as-is without any conversion. """ def convert(self, input_dict: dict[str, Any], **kwargs) -> dict[str, Any]: return input_dict class Chunk(ConversionOps): """Split a tensor along `dim` into equally sized chunks. Additionally, `num_shards_attribute` is a config field to read to know how many tensors to chunk into. Useful when concatenating an arbitrary number of tensors.""" def __init__(self, dim: int = 0, num_shards_attribute: str | None = None): self.dim = dim self.num_shards_attribute = num_shards_attribute @torch.no_grad def convert( self, input_dict: dict[str, torch.Tensor], source_patterns: list[str], target_patterns: list[str], **kwargs ) -> dict[str, torch.Tensor]: if len(input_dict) > 1: raise ValueError("Undefined Operation encountered") tensors = next(iter(input_dict.values())) tensor = tensors[0] if isinstance(tensors, list) else tensors targets = self.get_target_patterns(target_patterns, **kwargs) num_shards = len(targets) chunks = tuple(chunk.contiguous() for chunk in torch.chunk(tensor, num_shards, dim=self.dim)) return dict(zip(targets, chunks)) def get_target_patterns(self, target_patterns: list[str], **kwargs) -> list[str]: if self.num_shards_attribute is None: return target_patterns # In this case we need to use the config to know how many chunks to create else: if len(target_patterns) > 1: raise ValueError("Undefined Operation encountered") subconfig = kwargs["config"].get_text_config() num_shards = getattr(subconfig, self.num_shards_attribute) target_patterns = [target_patterns[0].replace("*", str(i)) for i in range(num_shards)] return target_patterns @property def reverse_op(self) -> ConversionOps: return Concatenate(self.dim, self.num_shards_attribute) class Split(ConversionOps): """Split a tensor along `dim` into one piece per target, sized like that target's model parameter. WARNING this op is not TP-safe, a sharded fused tensor needs each target's local slice, not the full parameter sizes.""" def __init__(self, dim: int = 0): self.dim = dim @torch.no_grad def convert( self, input_dict: dict[str, torch.Tensor], source_patterns: list[str], target_patterns: list[str], full_layer_name: str, model: PreTrainedModel, **kwargs, ) -> dict[str, torch.Tensor]: if len(input_dict) > 1: raise ValueError("Undefined Operation encountered") tensors = next(iter(input_dict.values())) tensor = tensors[0] if isinstance(tensors, list) else tensors sizes = [ model.get_parameter(full_layer_name.replace(target_patterns[0], target)).shape[self.dim] for target in target_patterns ] chunks = torch.split(tensor, sizes, dim=self.dim) return {target: chunk.contiguous() for target, chunk in zip(target_patterns, chunks)} @property def reverse_op(self) -> ConversionOps: return Concatenate(self.dim) class Concatenate(ConversionOps): """Concatenate tensors along `dim`. Additionally, if concatenating an aribitrary number of tensors, `num_shards_attribute` is a config field to read to know how many tensors to recreate when using the opposite Ops.""" def __init__(self, dim: int = 0, num_shards_attribute: str | None = None): self.dim = dim self.num_shards_attribute = num_shards_attribute @torch.no_grad def convert( self, input_dict: dict[str, list[torch.Tensor]], source_patterns: list[str], target_patterns: list[str], **kwargs, ) -> dict[str, torch.Tensor]: target_pattern = self.get_target_pattern(target_patterns) all_tensors = [] # Very important to keep the relative order of the source patterns here, so we iterate over them not the # input directly as it's unordered! Skip patterns that prior ops in the chain (e.g. `Fp8Dequantize`) # have already consumed and dropped from `input_dict`. for source_pattern in source_patterns: if source_pattern not in input_dict: continue # Immediately free the input_dict, so that we do not keep many copies simultaneously - otherwise we have to # wait for this function to return to be able to clean-up, which will not get garbage collected as fast as if # everything is freed right now tensors = input_dict.pop(source_pattern) if isinstance(tensors, list): all_tensors.extend(tensors) else: all_tensors.append(tensors) return {target_pattern: torch.cat(all_tensors, dim=self.dim)} def get_target_pattern(self, target_patterns: list[str]) -> str: # Here we always return the target pattern if len(target_patterns) > 1: raise ValueError("Undefined Operation encountered!") return target_patterns[0] @property def reverse_op(self) -> ConversionOps: return Chunk(self.dim, self.num_shards_attribute) class Interleave(ConversionOps): """Deinterleaves a tensor along `dim` by splitting in two and transposing. Reshapes param back to its original size.""" def __init__(self, dim: int = 0, inverse: bool = False): self.dim = dim self.inverse = inverse def convert(self, input_dict, source_patterns, target_patterns, **kwargs): tensor = next(iter(input_dict.values())) tensor = tensor[0] if isinstance(tensor, list) else tensor # Split into two in given dim and transpose to interleave along it shape = list(tensor.shape) if self.inverse: shape[self.dim : self.dim + 1] = [2, shape[self.dim] // 2] else: shape[self.dim : self.dim + 1] = [shape[self.dim] // 2, 2] tensor = tensor.reshape(shape).transpose(self.dim, self.dim + 1).reshape(tensor.shape).contiguous() return {target_patterns[0]: tensor} @property def reverse_op(self) -> ConversionOps: # can use the same dim and it will inverse it back return Interleave(self.dim, inverse=not self.inverse) class MergeModulelist(ConversionOps): """ Merge a list of tensors into a single tensor along the first dimension. We explicitly define this because for EP or TP you want to make sure you know what you are doing! """ def __init__(self, dim: int = 0): self.dim = dim @torch.no_grad def convert( self, input_dict: dict[str, list[torch.Tensor]], source_patterns: list[str], target_patterns: list[str], **kwargs, ) -> dict[str, torch.Tensor]: input_size = len(input_dict) merged: dict[str, torch.Tensor] = {} for source_pattern in list(input_dict.keys()): # Immediately free the input dict, so that we do not keep many copies simultaneously - otherwise we have to # wait for this function to return to be able to clean-up, and if the size of the input_dict is larger than 1 # (such as the MoEs' gate_proj/up_proj merging), we are wasting quite some memory tensors = input_dict.pop(source_pattern) target_pattern = self.get_target_pattern(input_size, source_pattern, target_patterns) # CompressedTensorsDequantize pre-allocates a stacked tensor to avoid holding N individual # decompressed tensors simultaneously. Pass it through to skip the redundant copy # that torch.stack would otherwise make. if isinstance(tensors, torch.Tensor): merged[target_pattern] = tensors else: merged[target_pattern] = torch.stack(tensors, dim=self.dim) return merged def get_target_pattern(self, input_size: int, source_pattern: str, target_patterns: list[str]) -> str: # Here it's a single operation, so we use the target if input_size == 1: if len(target_patterns) == 1: return target_patterns[0] else: raise ValueError("Undefined Operation encountered!") # Here it's the first operation in a chain, so we use the source as they were replaced before in the chain else: return source_pattern @property def reverse_op(self) -> ConversionOps: return SplitModulelist(self.dim) class SplitModulelist(ConversionOps): """Inverse of `MergeModulelist` using explicit split sizes per group.""" def __init__(self, dim: int = 0): self.dim = dim @torch.no_grad def convert( self, input_dict: dict[str, torch.Tensor], source_patterns: list[str], target_patterns: list[str], **kwargs ) -> dict[str, torch.Tensor]: all_tensors = {} for source_pattern, tensors in input_dict.items(): tensor = tensors[0] if isinstance(tensors, list) else tensors # We split in the number of tensors present in the given dim sizes = tensor.size(self.dim) targets = self.get_target_patterns(input_dict, source_pattern, target_patterns, sizes) chunks = torch.chunk(tensor, sizes, dim=self.dim) # Squeeze only the split dim, so a tensor keeps its own size-1 dims all_tensors.update({target: chunk.squeeze(self.dim) for target, chunk in zip(targets, chunks)}) return all_tensors def get_target_patterns( self, input_dict: dict, source_pattern: str, target_patterns: list[str], sizes: int ) -> list[str]: # Here it's a single operation, so we use the target if len(input_dict) == 1: if len(target_patterns) == 1: return [target_patterns[0].replace("*", f"{i}") for i in range(sizes)] else: raise ValueError("Undefined Operation encountered!") # Here it's the last operation in a chain, so we use the source as they were replaced before in the chain else: return [source_pattern.replace("*", f"{i}") for i in range(sizes)] @property def reverse_op(self) -> ConversionOps: return MergeModulelist(self.dim) class Transpose(ConversionOps): """ Transposes the given tensor along dim0 and dim1. """ def __init__(self, dim0: int = 0, dim1: int = 1, check_dims: bool = False): self.dim0 = dim0 self.dim1 = dim1 self.check_dims = check_dims @torch.no_grad def convert( self, input_dict: dict[str, torch.Tensor], source_patterns: list[str], target_patterns: list[str], **kwargs ) -> dict[str, torch.Tensor]: target_pattern = self.get_target_pattern(input_dict, source_patterns, target_patterns) tensors = next(iter(input_dict.values())) tensor = tensors[0] if isinstance(tensors, list) else tensors # In this case, always transpose if not self.check_dims: return {target_pattern: torch.transpose(tensor, dim0=self.dim0, dim1=self.dim1).contiguous()} # In this case, check the shapes bef - src/transformers/data/__init__.py (939 bytes)
# Copyright 2020 The HuggingFace Team. All rights reserved. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. from .data_collator import ( DataCollatorForLanguageModeling, DataCollatorForMultipleChoice, DataCollatorForPermutationLanguageModeling, DataCollatorForSeq2Seq, DataCollatorForTokenClassification, DataCollatorWithFlattening, DataCollatorWithPadding, DefaultDataCollator, default_data_collator, ) - src/transformers/data/data_collator.py (66381 bytes)
# Copyright 2020 The HuggingFace Team. All rights reserved. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. import multiprocessing as mp import os import warnings from collections.abc import Callable, Mapping from dataclasses import dataclass from random import randint from typing import Any import numpy as np from ..tokenization_utils_base import PreTrainedTokenizerBase from ..utils import PaddingStrategy InputDataClass = Any """ A DataCollator is a function that takes a list of samples from a Dataset and collate them into a batch, as a dictionary of PyTorch tensors or NumPy arrays. """ DataCollator = Callable[[list[InputDataClass]], dict[str, Any]] class DataCollatorMixin: def __call__(self, features, return_tensors: str | None = None): if return_tensors is None: return_tensors = self.return_tensors if return_tensors == "pt": return self.torch_call(features) elif return_tensors == "np": return self.numpy_call(features) else: raise ValueError(f"Framework '{return_tensors}' not recognized!") def pad_without_fast_tokenizer_warning(tokenizer, *pad_args, **pad_kwargs): """ Pads without triggering the warning about how using the pad function is sub-optimal when using a fast tokenizer. """ # To avoid errors when using Feature extractors if not hasattr(tokenizer, "deprecation_warnings"): return tokenizer.pad(*pad_args, **pad_kwargs) # Save the state of the warning, then disable it warning_state = tokenizer.deprecation_warnings.get("Asking-to-pad-a-fast-tokenizer", False) tokenizer.deprecation_warnings["Asking-to-pad-a-fast-tokenizer"] = True try: padded = tokenizer.pad(*pad_args, **pad_kwargs) finally: # Restore the state of the warning. tokenizer.deprecation_warnings["Asking-to-pad-a-fast-tokenizer"] = warning_state return padded def default_data_collator(features: list[InputDataClass], return_tensors="pt") -> dict[str, Any]: """ Very simple data collator that simply collates batches of dict-like objects and performs special handling for potential keys named: - `label`: handles a single value (int or float) per object - `label_ids`: handles a list of values per object Does not do any additional preprocessing: property names of the input object will be used as corresponding inputs to the model. See glue and ner for example of how it's useful. """ # In this function we'll make the assumption that all `features` in the batch # have the same attributes. # So we will look at the first element as a proxy for what attributes exist # on the whole batch. if return_tensors == "pt": return torch_default_data_collator(features) elif return_tensors == "np": return numpy_default_data_collator(features) @dataclass class DefaultDataCollator(DataCollatorMixin): """ Very simple data collator that simply collates batches of dict-like objects and performs special handling for potential keys named: - `label`: handles a single value (int or float) per object - `label_ids`: handles a list of values per object Does not do any additional preprocessing: property names of the input object will be used as corresponding inputs to the model. See glue and ner for example of how it's useful. This is an object (like other data collators) rather than a pure function like default_data_collator. This can be helpful if you need to set a return_tensors value at initialization. Args: return_tensors (`str`, *optional*, defaults to `"pt"`): The type of Tensor to return. Allowable values are "np", or "pt". """ return_tensors: str = "pt" def __call__(self, features: list[dict[str, Any]], return_tensors=None) -> dict[str, Any]: if return_tensors is None: return_tensors = self.return_tensors return default_data_collator(features, return_tensors) def torch_default_data_collator(features: list[InputDataClass]) -> dict[str, Any]: import torch if not isinstance(features[0], Mapping): features = [vars(f) for f in features] first = features[0] batch = {} # Special handling for labels. # Ensure that tensor is created with the correct type # (it should be automatically the case, but let's make sure of it.) if "label" in first and first["label"] is not None: label = first["label"].item() if isinstance(first["label"], torch.Tensor) else first["label"] dtype = torch.long if isinstance(label, (int, np.integer)) else torch.float batch["labels"] = torch.tensor([f["label"] for f in features], dtype=dtype) elif "label_ids" in first and first["label_ids"] is not None: if isinstance(first["label_ids"], torch.Tensor): batch["labels"] = torch.stack([f["label_ids"] for f in features]) else: dtype = torch.long if isinstance(first["label_ids"][0], (int, np.integer)) else torch.float batch["labels"] = torch.tensor([f["label_ids"] for f in features], dtype=dtype) # Handling of all other possible keys. # Again, we will use the first element to figure out which key/values are not None for this model. for k, v in first.items(): if k not in ("label", "label_ids") and v is not None and not isinstance(v, str): if isinstance(v, torch.Tensor): batch[k] = torch.stack([f[k] for f in features]) elif isinstance(v, np.ndarray): batch[k] = torch.from_numpy(np.stack([f[k] for f in features])) else: batch[k] = torch.tensor([f[k] for f in features]) return batch def numpy_default_data_collator(features: list[InputDataClass]) -> dict[str, Any]: if not isinstance(features[0], Mapping): features = [vars(f) for f in features] first = features[0] batch = {} # Special handling for labels. # Ensure that tensor is created with the correct type # (it should be automatically the case, but let's make sure of it.) if "label" in first and first["label"] is not None: label = first["label"].item() if isinstance(first["label"], np.ndarray) else first["label"] dtype = np.int64 if isinstance(label, (int, np.integer)) else np.float32 batch["labels"] = np.array([f["label"] for f in features], dtype=dtype) elif "label_ids" in first and first["label_ids"] is not None: if isinstance(first["label_ids"], np.ndarray): batch["labels"] = np.stack([f["label_ids"] for f in features]) else: dtype = np.int64 if isinstance(first["label_ids"][0], (int, np.integer)) else np.float32 batch["labels"] = np.array([f["label_ids"] for f in features], dtype=dtype) # Handling of all other possible keys. # Again, we will use the first element to figure out which key/values are not None for this model. for k, v in first.items(): if k not in ("label", "label_ids") and v is not None and not isinstance(v, str): if isinstance(v, np.ndarray): batch[k] = np.stack([f[k] for f in features]) else: batch[k] = np.array([f[k] for f in features]) return batch @dataclass class DataCollatorWithPadding: """ Data collator that will dynamically pad the inputs received. Args: tokenizer ([`PreTrainedTokenizer`] or [`PreTrainedTokenizerFast`]): The tokenizer used for encoding the data. padding (`bool`, `str` or [`~utils.PaddingStrategy`], *optional*, defaults to `True`): Select a strategy to pad the returned sequences (according to the model's padding side and padding index) among: - `True` or `'longest'` (default): Pad to the longest sequence in the batch (or no padding if only a single sequence is provided). - `'max_length'`: Pad to a maximum length specified with the argument `max_length` or to the maximum acceptable input length for the model if that argument is not provided. - `False` or `'do_not_pad'`: No padding (i.e., can output a batch with sequences of different lengths). max_length (`int`, *optional*): Maximum length of the returned list and optionally padding length (see above). pad_to_multiple_of (`int`, *optional*): If set will pad the sequence to a multiple of the provided value. This is especially useful to enable the use of Tensor Cores on NVIDIA hardware with compute capability >= 7.0 (Volta). return_tensors (`str`, *optional*, defaults to `"pt"`): The type of Tensor to return. Allowable values are "np", or "pt". """ tokenizer: PreTrainedTokenizerBase padding: bool | str | PaddingStrategy = True max_length: int | None = None pad_to_multiple_of: int | None = None return_tensors: str = "pt" def __call__(self, features: list[dict[str, Any]]) -> dict[str, Any]: batch = pad_without_fast_tokenizer_warning( self.tokenizer, features, padding=self.padding, max_length=self.max_length, pad_to_multiple_of=self.pad_to_multiple_of, return_tensors=self.return_tensors, ) if "label" in batch: batch["labels"] = batch["label"] del batch["label"] if "label_ids" in batch: batch["labels"] = batch["label_ids"] del batch["label_ids"] return batch @dataclass class DataCollatorForTokenClassification(DataCollatorMixin): """ Data collator that will dynamically pad the inputs received, as well as the labels. Args: tokenizer ([`PreTrainedTokenizer`] or [`PreTrainedTokenizerFast`]): The tokenizer used for encoding the data. padding (`bool`, `str` or [`~utils.PaddingStrategy`], *optional*, defaults to `True`): Select a strategy to pad the returned sequences (according to the model's padding side and padding index) among: - `True` or `'longest'` (default): Pad to the longest sequence in the batch (or no padding if only a single sequence is provided). - `'max_length'`: Pad to a maximum length specified with the argument `max_length` or to the maximum acceptable input length for the model if that argument is not provided. - `False` or `'do_not_pad'`: No padding (i.e., can output a batch with sequences of different lengths). max_length (`int`, *optional*): Maximum length of the returned list and optionally padding length (see above). pad_to_multiple_of (`int`, *optional*): If set will pad the sequence to a multiple of the provided value. This is especially useful to enable the use of Tensor Cores on NVIDIA hardware with compute capability >= 7.0 (Volta). label_pad_token_id (`int`, *optional*, defaults to -100): The id to use when padding the labels (-100 will be automatically ignore by PyTorch loss functions). return_tensors (`str`, *optional*, defaults to `"pt"`): The type of Tensor to return. Allowable values are "np", or "pt". """ tokenizer: PreTrainedTokenizerBase padding: bool | str | PaddingStrategy = True max_length: int | None = None pad_to_multiple_of: int | None = None label_pad_token_id: int = -100 return_tensors: str = "pt" def torch_call(self, features): import torch label_name = "label" if "label" in features[0] else "labels" labels = [feature[label_name] for feature in features] if label_name in features[0] else None no_labels_features = [{k: v for k, v in feature.items() if k != label_name} for feature in features] batch = pad_without_fast_tokenizer_warning( self.tokenizer, no_labels_features, padding=self.padding, max_length=self.max_length, pad_to_multiple_of=self.pad_to_multiple_of, return_tensors="pt", ) if labels is None: return batch sequence_length = batch["input_ids"].shape[1] padding_side = self.tokenizer.padding_side def to_list(tensor_or_iterable): if isinstance(tensor_or_iterable, torch.Tensor): return tensor_or_iterable.tolist() return list(tensor_or_iterable) if padding_side == "right": batch[label_name] = [ to_list(label) + [self.label_pad_token_id] * (sequence_length - len(label)) for label in labels ] else: batch[label_name] = [ [self.label_pad_token_id] * (sequence_length - len(label)) + to_list(label) for label in labels ] batch[label_name] = torch.tensor(batch[label_name], dtype=torch.int64) return batch def numpy_call(self, features): label_name = "label" if "label" in features[0] else "labels" labels = [feature[label_name] for feature in features] if label_name in features[0] else None no_labels_features = [{k: v for k, v in feature.items() if k != label_name} for feature in features] batch = pad_without_fast_tokenizer_warning( self.tokenizer, no_labels_features, padding=self.padding, max_length=self.max_length, pad_to_multiple_of=self.pad_to_multiple_of, return_tensors="np", ) if labels is None: return batch sequence_length = np.array(batch["input_ids"]).shape[1] padding_side = self.tokenizer.padding_side if padding_side == "right": batch[label_name] = [ list(label) + [self.label_pad_token_id] * (sequence_length - len(label)) for label in labels ] else: batch[label_name] = [ [self.label_pad_token_id] * (sequence_length - len(label)) + list(label) for label in labels ] batch[label_name] = np.array(batch[label_name], dtype=np.int64) return batch def _torch_collate_batch(examples, tokenizer, pad_to_multiple_of: int | None = None): """Collate `examples` into a batch, using the information in `tokenizer` for paddin - src/transformers/debug_utils.py (12964 bytes)
# Copyright 2020 The HuggingFace Team. All rights reserved. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. import collections from .utils import ExplicitEnum, is_torch_available, logging if is_torch_available(): import torch logger = logging.get_logger(__name__) class DebugUnderflowOverflow: """ This debug class helps detect and understand where the model starts getting very large or very small, and more importantly `nan` or `inf` weight and activation elements. There are 2 working modes: 1. Underflow/overflow detection (default) 2. Specific batch absolute min/max tracing without detection Mode 1: Underflow/overflow detection To activate the underflow/overflow detection, initialize the object with the model : ```python debug_overflow = DebugUnderflowOverflow(model) ``` then run the training as normal and if `nan` or `inf` gets detected in at least one of the weight, input or output elements this module will throw an exception and will print `max_frames_to_save` frames that lead to this event, each frame reporting 1. the fully qualified module name plus the class name whose `forward` was run 2. the absolute min and max value of all elements for each module weights, and the inputs and output For example, here is the header and the last few frames in detection report for `google/mt5-small` run in fp16 mixed precision : ``` Detected inf/nan during batch_number=0 Last 21 forward frames: abs min abs max metadata [...] encoder.block.2.layer.1.DenseReluDense.wi_0 Linear 2.17e-07 4.50e+00 weight 1.79e-06 4.65e+00 input[0] 2.68e-06 3.70e+01 output encoder.block.2.layer.1.DenseReluDense.wi_1 Linear 8.08e-07 2.66e+01 weight 1.79e-06 4.65e+00 input[0] 1.27e-04 2.37e+02 output encoder.block.2.layer.1.DenseReluDense.wo Linear 1.01e-06 6.44e+00 weight 0.00e+00 9.74e+03 input[0] 3.18e-04 6.27e+04 output encoder.block.2.layer.1.DenseReluDense T5DenseGatedGeluDense 1.79e-06 4.65e+00 input[0] 3.18e-04 6.27e+04 output encoder.block.2.layer.1.dropout Dropout 3.18e-04 6.27e+04 input[0] 0.00e+00 inf output ``` You can see here, that `T5DenseGatedGeluDense.forward` resulted in output activations, whose absolute max value was around 62.7K, which is very close to fp16's top limit of 64K. In the next frame we have `Dropout` which renormalizes the weights, after it zeroed some of the elements, which pushes the absolute max value to more than 64K, and we get an overflow. As you can see it's the previous frames that we need to look into when the numbers start going into very large for fp16 numbers. The tracking is done in a forward hook, which gets invoked immediately after `forward` has completed. By default the last 21 frames are printed. You can change the default to adjust for your needs. For example : ```python debug_overflow = DebugUnderflowOverflow(model, max_frames_to_save=100) ``` To validate that you have set up this debugging feature correctly, and you intend to use it in a training that may take hours to complete, first run it with normal tracing enabled for one of a few batches as explained in the next section. Mode 2. Specific batch absolute min/max tracing without detection The second work mode is per-batch tracing with the underflow/overflow detection feature turned off. Let's say you want to watch the absolute min and max values for all the ingredients of each `forward` call of a given batch, and only do that for batches 1 and 3. Then you instantiate this class as : ```python debug_overflow = DebugUnderflowOverflow(model, trace_batch_nums=[1, 3]) ``` And now full batches 1 and 3 will be traced using the same format as explained above. Batches are 0-indexed. This is helpful if you know that the program starts misbehaving after a certain batch number, so you can fast-forward right to that area. Early stopping: You can also specify the batch number after which to stop the training, with : ```python debug_overflow = DebugUnderflowOverflow(model, trace_batch_nums=[1, 3], abort_after_batch_num=3) ``` This feature is mainly useful in the tracing mode, but you can use it for any mode. **Performance**: As this module measures absolute `min`/``max` of each weight of the model on every forward it'll slow the training down. Therefore remember to turn it off once the debugging needs have been met. Args: model (`nn.Module`): The model to debug. max_frames_to_save (`int`, *optional*, defaults to 21): How many frames back to record trace_batch_nums(`list[int]`, *optional*, defaults to `[]`): Which batch numbers to trace (turns detection off) abort_after_batch_num (`int``, *optional*): Whether to abort after a certain batch number has finished """ def __init__(self, model, max_frames_to_save=21, trace_batch_nums=None, abort_after_batch_num=None): if trace_batch_nums is None: trace_batch_nums = [] self.model = model self.trace_batch_nums = trace_batch_nums self.abort_after_batch_num = abort_after_batch_num # keep a LIFO buffer of frames to dump as soon as inf/nan is encountered to give context to the problem emergence self.frames = collections.deque([], max_frames_to_save) self.frame = [] self.batch_number = 0 self.total_calls = 0 self.detected_overflow = False self.prefix = " " self.analyse_model() self.register_forward_hook() def save_frame(self, frame=None): if frame is not None: self.expand_frame(frame) self.frames.append("\n".join(self.frame)) self.frame = [] # start a new frame def expand_frame(self, line): self.frame.append(line) def trace_frames(self): print("\n".join(self.frames)) self.frames = [] def reset_saved_frames(self): self.frames = [] def dump_saved_frames(self): print(f"\nDetected inf/nan during batch_number={self.batch_number}") print(f"Last {len(self.frames)} forward frames:") print(f"{'abs min':8} {'abs max':8} metadata") print("\n".join(self.frames)) print("\n\n") self.frames = [] def analyse_model(self): # extract the fully qualified module names, to be able to report at run time. e.g.: # encoder.block.2.layer.0.SelfAttention.o # # for shared weights only the first shared module name will be registered self.module_names = {m: name for name, m in self.model.named_modules()} # self.longest_module_name = max(len(v) for v in self.module_names.values()) def analyse_variable(self, var, ctx): if torch.is_tensor(var): self.expand_frame(get_abs_min_max(var, ctx)) if detect_overflow(var, ctx): self.detected_overflow = True elif var is None: self.expand_frame(f"{'None':>17} {ctx}") else: self.expand_frame(f"{'not a tensor':>17} {ctx}") def batch_start_frame(self): self.expand_frame(f"\n\n{self.prefix} *** Starting batch number={self.batch_number} ***") self.expand_frame(f"{'abs min':8} {'abs max':8} metadata") def batch_end_frame(self): self.expand_frame(f"{self.prefix} *** Finished batch number={self.batch_number - 1} ***\n\n") def create_frame(self, module, input, output): self.expand_frame(f"{self.prefix} {self.module_names[module]} {module.__class__.__name__}") # params for name, p in module.named_parameters(recurse=False): self.analyse_variable(p, name) # inputs if isinstance(input, tuple): for i, x in enumerate(input): self.analyse_variable(x, f"input[{i}]") else: self.analyse_variable(input, "input") # outputs if isinstance(output, tuple): for i, x in enumerate(output): # possibly a tuple of tuples if isinstance(x, tuple): for j, y in enumerate(x): self.analyse_variable(y, f"output[{i}][{j}]") else: self.analyse_variable(x, f"output[{i}]") else: self.analyse_variable(output, "output") self.save_frame() def register_forward_hook(self): self.model.apply(self._register_forward_hook) def _register_forward_hook(self, module): module.register_forward_hook(self.forward_hook) def forward_hook(self, module, input, output): # - input is a tuple of packed inputs (could be non-Tensors) # - output could be a Tensor or a tuple of Tensors and non-Tensors last_frame_of_batch = False trace_mode = self.batch_number in self.trace_batch_nums if trace_mode: self.reset_saved_frames() if self.total_calls == 0: self.batch_start_frame() self.total_calls += 1 # count batch numbers - the very first forward hook of the batch will be called when the # batch completes - i.e. it gets called very last - we know this batch has finished if module == self.model: self.batch_number += 1 last_frame_of_batch = True self.create_frame(module, input, output) # if last_frame_of_batch: # self.batch_end_frame() if trace_mode: self.trace_frames() if last_frame_of_batch: self.batch_start_frame() if self.detected_overflow and not trace_mode: self.dump_saved_frames() # now we can abort, as it's pointless to continue running raise ValueError( "DebugUnderflowOverflow: inf/nan detected, aborting as there is no point running further. " "Please scroll up above this traceback to see the activation values prior to this event." ) # abort after certain batch if requested to do so if self.abort_after_batch_num is not None and self.batch_number > self.abort_after_batch_num: raise ValueError( f"DebugUnderflowOverflow: aborting after {self.batch_number} batches due to" f" `abort_after_batch_num={self.abort_after_batch_num}` arg" ) def get_abs_min_max(var, ctx): abs_var = var.abs() return f"{abs_var.min():8.2e} {abs_var.max():8.2e} {ctx}" def detect_overflow(var, ctx): """ Report whether the tensor contains any `nan` or `inf` entries. This is useful for detecting overflows/underflows and best to call right after the function that did some math that modified the tensor in question. This function contains a few other helper features that you can enable and tweak directly if you want to track various other things. Args: var: the tensor variable to check ctx: the message to print as a context Return: `True` if `inf` or `nan` was detected, `False` otherwise """ detected = False if torch.isnan(var).any().item(): detected = True print(f"{ctx} has nans") if torch.isinf(var).any().item(): detected = True print(f"{ctx} has infs") # if needed to monitor large elements can enable the following if 0: # and detected: n100 = var[torch.ge(var.abs(), 100)] if n100.numel() > 0: print(f"{ctx}: n100={n100.numel()}") n1000 = var[torch.ge(var.abs(), 1000)] if n1000.numel() > 0: print(f"{ctx}: n1000={n1000.numel()}") n10000 = var[torch.ge(var.abs(), 10000)] if n10000.numel() > 0: print(f"{ctx}: n10000={n10000.numel()}") if 0: print(f"min={var.min():9.2e} max={var.max():9.2e}") if 0: print(f"min={var.min():9.2e} max={var.max():9.2e} var={var.var():9.2e} mean={var.mean():9.2e} ({ctx})") return detected class DebugOption(ExplicitEnum): UNDERFLOW_OVERFLOW = "underflow_overflow" TPU_METRICS_DEBUG = "tpu_metrics_debug" - src/transformers/decision/__init__.py (989 bytes)
# Copyright 2026 The HuggingFace Team. All rights reserved. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. from typing import TYPE_CHECKING from ..utils import _LazyModule from ..utils.import_utils import define_import_structure if TYPE_CHECKING: from .configuration_utils import * from .template_utils import * else: import sys _file = globals()["__file__"] sys.modules[__name__] = _LazyModule(__name__, _file, define_import_structure(_file), module_spec=__spec__) - src/transformers/decision/configuration_utils.py (5511 bytes)
# Copyright 2026 The HuggingFace Team. All rights reserved. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. """Settings of decision models, saved as `decision.json` and `decision_template.jinja`.""" import json from dataclasses import asdict, dataclass, field, fields from pathlib import Path from huggingface_hub.dataclasses import strict from ..utils import cached_file DECISION_CONFIG_NAME, DECISION_TEMPLATE_NAME = "decision.json", "decision_template.jinja" @strict @dataclass class DecisionConfig: """ Settings of a decision model, saved as `decision.json` (and its chat template as `decision_template.jinja`). Args: labels (`dict[str, list[str]]`, *optional*): Single-token labels of the options per question type, for heads that read them from the logits. Defaults to `Yes`/`No` for `noul` and `A`, `B`, `C`, ... otherwise. temperature (`dict[str, float]`, *optional*): Temperature per question type, applied to the option scores before the softmax. Defaults to 1. chat_template (`str`, *optional*): Jinja template rendering `state` with `question` (one prompt per question) or `questions` (one prompt for all of them). Defaults to a generic prompt in the tokenizer's chat template. noul_options (`list[str]`, *optional*, defaults to `["true", "false"]`): Order in which the template receives the two options of a `noul` question. sanitize_control_tokens (`bool`, *optional*, defaults to `True`): Whether control tokens in the request are encoded as text. Templates that escape them themselves disable it, to keep the input ids of their original code. adapter (`str`, *optional*): Subfolder of the LoRA that the pipeline loads on the model, for repos that don't keep it at their root. """ labels: dict[str, list[str]] = field(default_factory=dict) temperature: dict[str, float | int] = field(default_factory=dict) chat_template: str | None = None noul_options: list[str] = field(default_factory=lambda: ["true", "false"]) sanitize_control_tokens: bool = True adapter: str | None = None def validate_noul_options(self): if sorted(self.noul_options) != ["false", "true"]: raise ValueError(f"noul_options must order 'true' and 'false', got {self.noul_options!r}.") def get_logits_processor(self, question_type: str): """The [`LogitsProcessorList`] applied to the option scores of a question of type `question_type`.""" from ..generation import LogitsProcessorList, TemperatureLogitsWarper return LogitsProcessorList([TemperatureLogitsWarper(float(self.temperature.get(question_type, 1.0)))]) @property def logits_processors(self) -> dict: """The [`LogitsProcessorList`] of each question type, built once.""" if getattr(self, "_logits_processors", None) is None: from .template_utils import Question self._logits_processors = {kind: self.get_logits_processor(kind) for kind in Question.questions_types} return self._logits_processors def prepare_inputs(self, request, processor) -> list[tuple[dict, list]]: """The tokenized `(inputs, questions)` of each prompt of `request`, see [`~decision.prepare_inputs`].""" from .template_utils import prepare_inputs return prepare_inputs(self, request, processor) @classmethod def from_file(cls, path: str | Path | None = None) -> "DecisionConfig": config = {} if path is None else json.loads(Path(path).read_text(encoding="utf-8")) # Settings this version doesn't know (e.g. older exports' `sort_choices`) are ignored return cls(**{key: value for key, value in config.items() if key in {f.name for f in fields(cls)}}) @classmethod def from_pretrained( cls, pretrained_model_name_or_path: str | Path, config_name: str = DECISION_CONFIG_NAME, **kwargs ) -> "DecisionConfig": # The default settings for a repo without a decision config, or a model that has no repo if not pretrained_model_name_or_path: return cls() kwargs["_raise_exceptions_for_missing_entries"] = False config = cls.from_file(cached_file(pretrained_model_name_or_path, config_name, **kwargs)) if (template := cached_file(pretrained_model_name_or_path, DECISION_TEMPLATE_NAME, **kwargs)) is not None: config.chat_template = Path(template).read_text(encoding="utf-8") return config def save_pretrained(self, save_directory: str | Path): path, config = Path(save_directory), asdict(self) path.mkdir(parents=True, exist_ok=True) if (template := config.pop("chat_template")) is not None: (path / DECISION_TEMPLATE_NAME).write_text(template, encoding="utf-8") (path / DECISION_CONFIG_NAME).write_text(json.dumps(config, indent=2) + "\n", encoding="utf-8") __all__ = ["DecisionConfig"] - src/transformers/decision/modeling_utils.py (6447 bytes)
# Copyright 2026 The HuggingFace Team. All rights reserved. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. """ `decide`, the `generate` of decision models: like the `GenerationMixin` of the models that generate. """ import json from pathlib import Path import torch from ..configuration_utils import PreTrainedConfig from ..utils import cached_file from .configuration_utils import DecisionConfig from .template_utils import Question def process_scores(decision_config, input_ids, marks, questions, scores) -> list[torch.Tensor]: """The logits processors of the decision config on the option scores of each question of a prompt.""" tail = input_ids[0, int(marks[-1]) :] # the tokens after the last mark, where the templates write the state return [decision_config.logits_processors[q.type](tail, s.float()) for q, s in zip(questions, scores)] def _hub_kwargs(kwargs: dict) -> dict: return {key: kwargs[key] for key in ("cache_dir", "local_files_only", "revision", "token") if key in kwargs} class DecisionMixin: """ The `decide` of the `XxxForDecision` models, with their `decision_config`: the decision config of their repo, or the one of the checkpoint they read the head of, like `generation_config` for `generate`. """ @classmethod def from_pretrained(cls, pretrained_model_name_or_path, *args, config=None, decision_config=None, **kwargs): """ Also loads the checkpoints that ship a decision head next to their weights, in their own format: the config of the checkpoint is the `text_config` of the decision model, and the config files of its head (`config_class.extra_config_files`) give the rest of its config, in `__post_init__`. """ if not isinstance(config, cls.config_class): # none, or the config of the model the head ships next to head_config, decision_config = cls.checkpoint_config(pretrained_model_name_or_path, **kwargs) if head_config is None and config is not None: raise ValueError(f"{pretrained_model_name_or_path} has no {cls.__name__} head next to its weights.") config = head_config or config model = super().from_pretrained(pretrained_model_name_or_path, *args, config=config, **kwargs) if decision_config is None: # the decision config of the repo, read now like `generation_config` decision_config = DecisionConfig.from_pretrained(pretrained_model_name_or_path, **_hub_kwargs(kwargs)) model.decision_config = decision_config return model @classmethod def checkpoint_config( cls, pretrained_model_name_or_path, **kwargs ) -> tuple[PreTrainedConfig, DecisionConfig] | tuple[None, None]: """ The config and decision config of the decision model of a checkpoint that ships its head next to its weights, read from its files, or `None`s for any other repo (e.g. one saved by `save_pretrained`, whose `config.json` is the one of the decision model). """ hub_kwargs = _hub_kwargs(kwargs) head_configs = {} for filename in cls.config_class.extra_config_files: path = cached_file( pretrained_model_name_or_path, filename, _raise_exceptions_for_missing_entries=False, **hub_kwargs ) if path is not None: head_configs[Path(filename).stem] = json.loads(Path(path).read_text(encoding="utf-8")) if not head_configs: return None, None text_config, _ = PreTrainedConfig.get_config_dict(pretrained_model_name_or_path, **hub_kwargs) decision_config = cls.config_class.decision_config(head_configs, text_config) if decision_config is None or text_config.get("model_type") == cls.config_class.model_type: return None, None # not a checkpoint of this decision model return cls.config_class(text_config=text_config, **head_configs), decision_config def post_init(self): """The default decision config, replaced by the one of the repo in `from_pretrained`, like `generation_config`.""" super().post_init() self.decision_config = DecisionConfig() def save_pretrained(self, save_directory, *args, **kwargs): """Also saves the decision config, like `generation_config`: the saved repo reloads as the same decision model.""" super().save_pretrained(save_directory, *args, **kwargs) if kwargs.get("is_main_process", True): self.decision_config.save_pretrained(save_directory) @torch.no_grad() def decide( self, input_ids: torch.LongTensor, marks: torch.LongTensor, decision_questions: list[Question], decision_config: DecisionConfig | None = None, **kwargs, ) -> list[torch.Tensor]: """ Scores the options of each question of a prompt from `decision_config.prepare_inputs`: one forward pass, then the logits processors of the decision config. Returns the scores of each question, which `Question.answer` turns into answers. """ decision_config = decision_config or self.decision_config inputs = {"input_ids": input_ids, **kwargs} num_options = torch.tensor([[len(question.options) for question in decision_questions]]) question_types = [Question.questions_types.index(question.type) for question in decision_questions] scores = self( marks=marks[None].to(self.device), num_options=num_options.to(self.device), question_type_ids=torch.tensor([question_types], device=self.device), **{key: value.to(self.device) if hasattr(value, "to") else value for key, value in inputs.items()}, ) return process_scores( decision_config, inputs["input_ids"], marks, decision_questions, scores.split(num_options[0].tolist()) ) __all__ = ["DecisionMixin"] - src/transformers/decision/template_utils.py (18531 bytes)
# Copyright 2026 The HuggingFace Team. All rights reserved. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. """ TLDR: decision models answer a question in a single forward pass, by reading the answer at a known position of the prompt. A request is `{"state": ..., "questions": {name: question}}`, rendered with a chat template into a prompt per question (or one for all of them, when the template uses `questions`). A model with a `decision_template.jinja` places two sentinel tokens: - `<|mark|>`: where the head reads, which depends on the head: jev ... [decision]:<|mark|> reads 1 token: the one before each mark (one per question) kev ... opt A<|mark|> opt B<|mark|> opt C<|mark|> state<|mark|> reads 1 token before each mark, one mark after each option von ... [MASK]<|mark|> [MASK]<|mark|> ... <|mark|> reads 1 token per option; the last mark ends the block clef INSTRUCTION: <|mark|>What is the tone?<|mark|> OPTION 1: <|mark|>{"option_id":"calm"}<|mark|> OPTION 2: <|mark|>{"option_id":"pushy"}<|mark|> reads every token between a pair of marks, pooled (hidden states + output embeddings of the span): the spans sit mid-sequence, not at the end - `<|split|>`: is added to enforce splitting the encoding and is only there for "safety", ensuring user text never parses any control / special tokens. Both are dropped from the input ids: [`~DecisionMixin.decide`] keeps the token ids and the number of tokens before each `<|mark|>`. We otherwise can turn any model into a decision model with a `DECISION_PROMPT`, where the logits of the option labels (`A`, `B`, ... or `Yes`/`No`) are read after the generation prompt, e.g. for `Qwen/Qwen2.5-0.5B-Instruct`: ``` <|im_start|>system You are Qwen, created by Alibaba Cloud. You are a helpful assistant.<|im_end|> <|im_start|>user {"message": "WIN a FREE iPhone, click here!"} What is the tone? A. calm B. pushy: Pressures the reader Answer with the letter of the option that fits best (A to B) only.<|im_end|> <|im_start|>assistant ``` The rest of this file handles the different input validation, escaping user text from the mark/split tokens, and replacing images with the corresponding assets. We support arbitrary kind of inputs, users should be able to expand it to his needs. """ import functools import json import re import string from dataclasses import asdict, dataclass from typing import Any, Literal from huggingface_hub.dataclasses import strict from huggingface_hub.errors import StrictDataclassError from ..image_utils import load_image from ..tokenization_utils_base import AddedToken from ..utils.chat_template_utils import _compile_jinja_template, _get_template_variables from .configuration_utils import DecisionConfig MARK, SPLIT = "<|mark|>", "<|split|>" Content = str | dict | list @strict @dataclass class NoulQuestion: """{"type": "noul", "instructions": "Is this message spam?", "criteria": {"true": "Unsolicited ads", "false": "A real conversation"}}""" type: Literal["noul"] = "noul" instructions: Content | None = None criteria: dict[Literal["true", "false"], Content | None] | None = None @strict @dataclass class ChoiceQuestion: """{"type": "choice", "instructions": "What is the tone?", "criteria": {"calm": "A polite message", "angry": null}}""" criteria: dict[str, Content | None] type: Literal["choice"] = "choice" instructions: Content | None = None @strict @dataclass class ScoreQuestion: """{"type": "score", "instructions": "How urgent is it?", "criteria": ["Can wait", "This week", "Today"]}""" criteria: list[Content] type: Literal["score"] = "score" instructions: Content | None = None # Models that don't ship a decision_template.jinja default to this one, the prompt of the `v1/systemone` endpoint of # `transformers serve`: any chat model answers with the next-token logits of the option labels DECISION_PROMPT = """ {%- macro text(x) %}{{ "" if x is none else x if x is string else x|tojson }}{% endmacro -%} {%- set ns = namespace(lines=[text(question.instructions)]) -%} {%- for option in question.options -%} {%- set label, answer = question.labels[loop.index0], text(question.descriptions[loop.index0]) -%} {%- if question.type == "noul" -%} {%- set answer = answer or (option if question.labels != ["Yes", "No"] else "") -%} {%- set ns.lines = ns.lines + ([label ~ ": " ~ answer] if answer else []) -%} {%- else -%} {%- set answer = (option ~ ": " ~ answer if answer else option) if question.type == "choice" else answer -%} {%- set ns.lines = ns.lines + [label ~ ". " ~ answer] -%} {%- endif -%} {%- endfor -%} {{- text(state) ~ "\\n\\n" ~ ns.lines|select|join("\\n") ~ "\\n" -}} {%- if question.type == "noul" -%} {{- "Answer " ~ question.labels[0] ~ " or " ~ question.labels[1] ~ " only." -}} {%- else -%} {%- set kind = "letter" if question.labels == "ABCDEFGHIJKLMNOPQRSTUVWXYZ"[: question.labels|length]|list else "label" -%} {{- "Answer with the " ~ kind ~ " of the " ~ ("option" if question.type == "choice" else "level") -}} {{- " that fits best (" ~ question.labels[0] ~ " to " ~ question.labels[-1] ~ ") only." -}} {%- endif -%} """ @dataclass class Question: """One System One question, with its options (choice keys, score levels or true/false) in template order.""" name: str type: Literal["noul", "choice", "score"] instructions: Any options: list[str] labels: list[str] descriptions: list[Any] questions_schemes = {"noul": NoulQuestion, "choice": ChoiceQuestion, "score": ScoreQuestion} questions_types = tuple(questions_schemes) @classmethod def from_request(cls, name: str, question: dict[str, Any], config: DecisionConfig) -> "Question": kind = question.get("type") if isinstance(question, dict) else None if kind not in cls.questions_schemes: got = kind if isinstance(question, dict) else question raise ValueError( f"Question {name!r}: `type` must be one of {', '.join(cls.questions_types)}, got {got!r}." ) schema = cls.questions_schemes[kind] try: question = asdict(schema(**question)) except (TypeError, StrictDataclassError) as error: raise ValueError(f"Question {name!r}: {error} (a {kind} question looks like {schema.__doc__})") from error criteria = question["criteria"] if kind != "noul" and not criteria: raise ValueError(f"Question {name!r}: a {kind} question needs `criteria` with its options.") if kind == "noul": criteria = {option: (criteria or {}).get(option) for option in config.noul_options} elif kind == "score": criteria = {str(index): level for index, level in enumerate(criteria)} labels = config.labels.get(kind) or (["Yes", "No"] if kind == "noul" else list(string.ascii_uppercase)) if len(labels) < len(criteria): raise ValueError( f"Question {name!r}: this model reads at most {len(labels)} options for a {kind} question, got " f"{len(criteria)} (`labels.{kind}` in its decision config)." ) options, descriptions = list(criteria), list(criteria.values()) return cls(name, kind, question.get("instructions"), options, labels[: len(options)], descriptions) def answer(self, probabilities: list[float]) -> dict[str, Any]: """The System One answer, from the probabilities of the options.""" by_option, n_options = dict(zip(self.options, probabilities)), len(probabilities) if self.type == "noul": return {"type": "noul", "noul": by_option["true"]} best = max(by_option, key=by_option.get) if self.type == "choice": confidence = 1.0 if n_options == 1 else max(0.0, (by_option[best] - 1 / n_options) / (1 - 1 / n_options)) return {"type": "choice", "choice": best, "confidence": confidence, "probabilities": by_option} distance = sum(probability * abs(level - int(best)) for level, probability in enumerate(probabilities)) uniform_distance = sum(abs(level - (n_options - 1) / 2) for level in range(n_options)) / n_options confidence = 1.0 if n_options == 1 else max(0.0, 1 - distance / uniform_distance) score = sum(level * probability for level, probability in enumerate(probabilities)) return { "type": "score", "score": score, "legend": dict(zip(self.options, self.descriptions)), "confidence": confidence, "probabilities": by_option, } def label_token_ids(tokenizer, labels: tuple[str, ...]) -> list[int]: """ The ids of the `labels` of a question, which must be distinct single known tokens. Only for models without a decision head (e.g. `ArthurZ/d3-nano`): the score of each option is the logit of its label token, at the last token before the mark. Heads (clef, jev, ...) read hidden states instead. """ ids = tokenizer(list(labels), add_special_tokens=False)["input_ids"] label_ids = [token_ids[0] for token_ids in ids if len(token_ids) == 1 and token_ids[0] != tokenizer.unk_token_id] if len(set(label_ids)) != len(ids): raise ValueError(f"The labels {list(labels)} are not distinct single tokens for this model.") return label_ids def parse_questions(questions: dict[str, Any], config: DecisionConfig) -> list[Question]: """ Validates the `questions` of a System One request and puts them in template order, raising one `ValueError` that lists every invalid question, each with an example of a valid one. """ if not isinstance(questions, dict) or not questions: raise ValueError(f"`questions` maps names to questions, with at least one question; got {questions!r}.") parsed, errors = [], [] for name, question in questions.items(): try: parsed.append(Question.from_request(name, question, config)) except ValueError as error: errors.append(str(error)) if errors: raise ValueError("\n".join(errors)) return parsed def _split_at_placeholders(text: str, assets: dict[str, Any], load) -> list[dict[str, Any]]: """ This goes from the System One way of passing an image (a placeholder in the text, the image in `assets`) to the chat template way (an image block in the content), which the processor turns into image tokens, e.g. for Qwen2.5-VL: "Is <img-1> a cat?", {"img-1": ...} -> [{"type": "text", "text": "Is "}, {"type": "image", "image": ...}, {"type": "text", "text": " a cat?"}] -> "Is <|vision_start|><|image_pad|>...<|image_pad|><|vision_end|> a cat?" (one `<|image_pad|>` per patch) """ if not assets: return [{"type": "text", "text": text}] pattern = re.compile("|".join(re.escape(name) for name in sorted(assets, key=len, reverse=True))) content, last = [], 0 for match in pattern.finditer(text): content += [{"type": "text", "text": text[last : match.start()]}, {"type": "image", "image": load(match[0])}] last = match.end() content.append({"type": "text", "text": text[last:]}) return [block for block in content if block.get("text") != ""] def _insert_images(template_inputs: dict[str, Any], assets: dict[str, Any], load) -> tuple[dict[str, Any], list]: """ Puts the images of a request into the inputs of a decision template (`state`, `question`), where an image is a placeholder anywhere in a string, e.g. deep in the state: template_inputs = {"state": {"photo": "img-1"}, "question": {"instructions": "Is it like img-2?"}} assets = {"img-1": <red>, "img-2": <blue>} Returns: - the inputs with each such string as content blocks, so the template prints the model's image tokens there: {"state": {"photo": [{"type": "image", "image": <red>}]}, "question": {"instructions": [{"type": "text", "text": "Is it like "}, {"type": "image", "image": <blue>}, ...]}} - the images, `[<red>, <blue>]`, for the processor: it only collects images from `messages`, not from template inputs. They are in dict order, the order templates print them in (state, instructions, options). """ images = [] def insert(value): # visits the values in dict order if isinstance(value, dict): return {key: insert(item) for key, item in value.items()} if isinstance(value, list): return [insert(item) for item in value] if isinstance(value, str): parts = _split_at_placeholders(value, assets, load) found = [part["image"] for part in parts if part["type"] == "image"] images.extend(found) return parts if found else value return value return insert(template_inputs), images def read_marks(inputs: dict[str, Any], sentinel_ids: list[int]) -> tuple[dict[str, Any], Any]: """ Drops the sentinels from the inputs of a prompt (a batch of one): the marks are the number of tokens before each `<|mark|>`, or all of them for a prompt without marks (read after the generation prompt). """ mark, split = sentinel_ids input_ids = inputs["input_ids"][0] keep = (input_ids != mark) & (input_ids != split) marks = keep.cumsum(0)[input_ids == mark] inputs = { key: value[:, keep] if getattr(value, "shape", ())[-1:] == input_ids.shape else value for key, value in inputs.items() if key != "attention_mask" } return inputs, marks if len(marks) else keep.sum(0, keepdim=True) def prepare_inputs( config: DecisionConfig, request: dict[str, Any] | str, processor ) -> list[tuple[dict[str, Any], list[Question]]]: """ Tokenizes a decision `request` (`{"state": ..., "questions": ..., "assets": ...}`, or its JSON) into one prompt per question, or a single prompt when the template reads `questions`. Returns `(inputs, questions)` per prompt: `inputs` has no sentinels and gives `marks`, the positions [`~DecisionMixin.decide`] reads. `assets` map placeholders in the strings of the request to images: a System One asset (`{"data_base64": ...}`), or anything [`~image_utils.load_image`] takes. """ tokenizer = getattr(processo - src/transformers/dependency_versions_check.py (2099 bytes)
# Copyright 2020 The HuggingFace Team. All rights reserved. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. from .dependency_versions_table import deps from .utils.versions import require_version, require_version_core # define which module versions we always want to check at run time # (usually the ones defined in `install_requires` in setup.py) # # order specific notes: # - tqdm must be checked before tokenizers pkgs_to_check_at_runtime = [ "python", "tqdm", "regex", "packaging", "filelock", "numpy", "tokenizers", "huggingface-hub", "safetensors", "accelerate", "pyyaml", ] for pkg in pkgs_to_check_at_runtime: if pkg in deps: if pkg == "tokenizers": # must be loaded here, or else tqdm check may fail from .utils import is_tokenizers_available if not is_tokenizers_available(): continue # not required, check version only if installed elif pkg == "accelerate": # must be loaded here, or else tqdm check may fail from .utils import is_accelerate_available # Maybe switch to is_torch_available in the future here so that Accelerate is hard dep of # Transformers with PyTorch if not is_accelerate_available(): continue # not required, check version only if installed require_version_core(deps[pkg]) else: raise ValueError(f"can't find {pkg} in {deps.keys()}, check dependency_versions_table.py") def dep_version_check(pkg, hint=None): require_version(deps[pkg], hint) - src/transformers/dependency_versions_table.py (3037 bytes)
# THIS FILE HAS BEEN AUTOGENERATED. To update: # 1. modify the `_deps` dict in setup.py # 2. run `make fix-repo`` deps = { "Pillow": "Pillow>=10.0.1,<=15.0", "accelerate": "accelerate>=1.1.0", "av": "av", "beautifulsoup4": "beautifulsoup4", "blobfile": "blobfile", "codecarbon": "codecarbon>=2.8.1", "datasets": "datasets>=2.15.0", "deepspeed": "deepspeed>=0.9.3", "diffusers": "diffusers==0.35.2", "faiss-cpu": "faiss-cpu", "fastapi": "fastapi", "filelock": "filelock", "fugashi": "fugashi>=1.0", "GitPython": "GitPython<3.1.19", "hf-doc-builder": "hf-doc-builder>=0.6.0", "huggingface-hub": "huggingface-hub>=1.31.0,<3.0", "ipadic": "ipadic>=1.0.0,<2.0", "jinja2": "jinja2>=3.1.6", "kernels": "kernels>=0.18.2,<0.19", "librosa": "librosa", "mistral-common[image]": "mistral-common[image]>=1.11.7", "nltk": "nltk<=3.8.1", "num2words": "num2words", "numpy": "numpy>=1.17", "openai": "openai>=1.98.0", "opencv-python": "opencv-python", "optimum-benchmark": "optimum-benchmark>=0.3.0", "optuna": "optuna", "packaging": "packaging>=20.0", "parameterized": "parameterized>=0.9", "peft": "peft>=0.22.0", "phonemizer": "phonemizer", "protobuf": "protobuf", "psutil": "psutil", "pyyaml": "pyyaml>=5.1", "pydantic": "pydantic>=2", "pytest": "pytest>=7.2.0,<9.0.0", "pytest-asyncio": "pytest-asyncio>=1.2.0", "pytest-random-order": "pytest-random-order", "pytest-rerunfailures": "pytest-rerunfailures<16.0", "pytest-timeout": "pytest-timeout", "pytest-env": "pytest-env", "pytest-xdist": "pytest-xdist", "pytest-order": "pytest-order", "python": "python>=3.11.0", "regex": "regex>=2025.10.22", "rhoknp": "rhoknp>=1.1.0,<1.3.1", "rjieba": "rjieba", "ruff": "ruff==0.14.10", "transformers-mlinter": "transformers-mlinter==0.1.6", "ty": "ty==0.0.20", "sacremoses": "sacremoses", "safetensors": "safetensors>=0.8.0", "sagemaker": "sagemaker>=2.31.0", "schedulefree": "schedulefree>=1.2.6", "scikit-learn": "scikit-learn", "scipy": "scipy", "sentencepiece": "sentencepiece>=0.1.91,!=0.1.92", "soundfile": "soundfile>=0.13.0", "soxr": "soxr>=0.5.0", "starlette": "starlette", "sudachipy": "sudachipy>=0.6.6", "sudachidict_core": "sudachidict_core>=20220729", "tensorboard": "tensorboard", "timeout-decorator": "timeout-decorator", "tiktoken": "tiktoken", "timm": "timm>=1.0.23", "tokenizers": "tokenizers>=0.23.1,<0.24.0", "torch": "torch>=2.6", "torchaudio": "torchaudio", "torchvision": "torchvision", "tqdm": "tqdm>=4.60", "transformers-mic[runtime]": "transformers-mic[runtime]>=0.1.3", "typer": "typer", "unidic": "unidic>=1.0.2", "unidic_lite": "unidic_lite>=1.0.7", "urllib3": "urllib3<2.0.0", "uvicorn": "uvicorn", "pytest-rich": "pytest-rich", "libcst": "libcst", "rich": "rich", "ray[tune]": "ray[tune]>=2.7.0", } - src/transformers/distributed/__init__.py (1992 bytes)
# Copyright 2025 The HuggingFace Team. All rights reserved. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. from typing import TYPE_CHECKING from ..utils import _LazyModule _import_structure = { "checkpoint": [ "consolidate_distributed_checkpoint", "_load_model_checkpoint_distributed", "load_optimizer_distributed", "_save_model_checkpoint_distributed", "save_optimizer_distributed", ], "configuration_utils": ["DistributedConfig"], "fsdp": ["is_fsdp_enabled", "is_fsdp_managed_module", "verify_fsdp_plan"], "mixin": ["DistributedMixin"], "utils": [ "gather_full_state_dict", "initialize_distributed_mesh", "initialize_fully_sharded_data_parallelism", ], } if TYPE_CHECKING: from .checkpoint import ( _load_model_checkpoint_distributed, _save_model_checkpoint_distributed, consolidate_distributed_checkpoint, load_optimizer_distributed, save_optimizer_distributed, ) from .configuration_utils import ( DistributedConfig, ) from .fsdp import is_fsdp_enabled, is_fsdp_managed_module, verify_fsdp_plan from .mixin import DistributedMixin from .utils import ( gather_full_state_dict, initialize_distributed_mesh, initialize_fully_sharded_data_parallelism, ) else: import sys sys.modules[__name__] = _LazyModule(__name__, globals()["__file__"], _import_structure, module_spec=__spec__) - src/transformers/distributed/checkpoint.py (12502 bytes)
# Copyright 2026 The HuggingFace Team. All rights reserved. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. from __future__ import annotations import os import re from typing import TYPE_CHECKING from ..utils import ( is_torch_available, is_torch_greater_or_equal, logging, ) from .utils import ( _check_distributed_checkpointing_available, _distributed_barrier, is_dtensor, ) if TYPE_CHECKING: from ..quantizers import HfQuantizer logger = logging.get_logger(__name__) if is_torch_available(): import torch from torch.utils._pytree import tree_map if _check_distributed_checkpointing_available(): from torch.distributed.checkpoint.hf_storage import HuggingFaceStorageReader, HuggingFaceStorageWriter from torch.distributed.tensor import Shard from torch.distributed.tensor.placement_types import _StridedShard def _redistribute_state_dict(state_dict, reference_state_dict: dict | None = None): """ The DTensor hooks used by DCP to save/load a state dict are broken for `_StridedShard` placements. Example: Context: - Global tensor: [10, 11, 12, 13, 14, 15, 16, 17] - Placement: _StridedShard(dim=0, split_factor=2) - Mesh size: 2 - Rank 0 local: [10, 11, 14, 15] Torch DCP machinery uses DTensor hooks (`__create_write_items__`, `__create_chunk_list__`, and `__get_tensor_shard__`) to save/load a state dict. The issue is that they expose the local storage as a single contiguous region, which would be saved as a single chunk: ``` __create_write_items__: [WriteItem(name="weight", global_shape=[8], offset=[0], size=[4])] __create_chunk_list__: [ChunkStorageMetadata(offsets=[0], sizes=[4])] __get_tensor_shard__(MetadataIndex("weight", offset=[0])): [10, 11, 14, 15] ``` While the data is correct, the chunk metadata is misleading because it implies that the local storage corresponds to a single contiguous region of the global tensor, which is not the case. The hooks should expose the local storage as two disjoint regions, which should be saved as two chunks: ``` __create_write_items__: [WriteItem(name="weight", global_shape=[8], offset=[0], size=[2]), WriteItem(name="weight", global_shape=[8], offset=[4], size=[2])] __create_chunk_list__: [ChunkStorageMetadata(offsets=[0], sizes=[2]), ChunkStorageMetadata(offsets=[4], sizes=[2])] __get_tensor_shard__(MetadataIndex("weight", offset=[0])): [10, 11] __get_tensor_shard__(MetadataIndex("weight", offset=[4])): [14, 15] ``` A solution would be to fix this machinary in PyTorch or by extending the DCP API to support it. Until then, one workaround is to redistribute the DTensors with `_StridedShard` placements to equivalent `Shard` placements before saving the state dict. If `reference_state_dict` is given, the DTensors are instead redistributed back to the placements of the matching DTensors in `reference_state_dict`, which undoes a previous call (e.g. after loading). """ if not _check_distributed_checkpointing_available(): raise OSError("Distributed checkpointing requires `torch>=2.8` with `torch.distributed` available.") def redistribute(value, reference=None): if not is_dtensor(value): return value if reference is not None: placements = reference.placements else: placements = tuple(Shard(p.dim) if isinstance(p, _StridedShard) else p for p in value.placements) if placements == value.placements: return value return value.redistribute(placements=placements) # tree_map allows to map a function on arbitrarily nested structures. if reference_state_dict is None: return tree_map(redistribute, state_dict) return tree_map(redistribute, state_dict, reference_state_dict) def _save_model_checkpoint_distributed(model, checkpoint_dir: str | os.PathLike) -> None: """Save rank-local model shards as safetensors with DCP.""" if not _check_distributed_checkpointing_available(): raise OSError("Distributed checkpointing requires `torch>=2.8` with `torch.distributed` available.") # Import here because otherwise it emits a warning every time it's imported on some hardware - this keeps the warning from # being emitted if the function is not used import torch.distributed.checkpoint as dcp from torch.distributed.checkpoint.state_dict import get_model_state_dict # DCP describes each DTensor as one rectangular chunk, which cannot represent strided shards. # We redistribute any strided shards to contiguous shards so DCP can write them out. # Sub-optimal compared to a future DCP that can write strided shards directly, but works for now. state_dict = _redistribute_state_dict(get_model_state_dict(model)) writer = HuggingFaceStorageWriter( path=checkpoint_dir, save_distributed=True, enable_consolidation=False, ) dcp.save(state_dict, storage_writer=writer) # All ranks wait until the checkpoint is ready for loading or consolidation. _distributed_barrier() def consolidate_distributed_checkpoint( checkpoint_dir: str | os.PathLike, output_dir: str | os.PathLike | None = None ) -> None: """Consolidate rank-local safetensors files. Call from a single process or rank.""" if not _check_distributed_checkpointing_available() or not is_torch_greater_or_equal("2.9"): raise OSError( "Consolidating a distributed checkpoint requires `torch>=2.9` with `torch.distributed` available." ) from torch.distributed.checkpoint._consolidate_hf_safetensors import consolidate_safetensors_files if output_dir is None: output_dir = checkpoint_dir metadata = HuggingFaceStorageReader(str(checkpoint_dir)).read_metadata() os.makedirs(output_dir, exist_ok=True) consolidate_safetensors_files( input_dir=str(checkpoint_dir), output_dir=str(output_dir), fqn_to_index_mapping=dict.fromkeys(metadata.state_dict_metadata, 1), ) def is_sharded_checkpoint(checkpoint_dir: str | os.PathLike) -> bool: """Return True if the checkpoint directory contains sharded safetensors files.""" pattern = r"^shard-[0-9]{5}-model-[0-9]{5}-of-[0-9]{5}\.safetensors$" if not os.path.isdir(checkpoint_dir): return False return any(re.match(pattern, name) for name in os.listdir(checkpoint_dir)) def _load_model_checkpoint_distributed(model, checkpoint_dir: str | os.PathLike, strict: bool = True) -> None: """Load rank-local DCP files from checkpoint_dir into an initialized model, preserving its mesh and placements.""" if not _check_distributed_checkpointing_available(): raise OSError("Distributed checkpointing requires `torch>=2.8` with `torch.distributed` available.") # Import here because otherwise it emits a warning every time it's imported on some hardware - this keeps the warning from # being emitted if the function is not used import torch.distributed.checkpoint as dcp from torch.distributed.checkpoint.default_planner import DefaultLoadPlanner from torch.distributed.checkpoint.state_dict import get_model_state_dict, set_model_state_dict reader = HuggingFaceStorageReader(str(checkpoint_dir)) original_state = get_model_state_dict(model) if any(value.is_meta for value in original_state.values() if isinstance(value, torch.Tensor)): raise ValueError("Materialize the model's tensors before loading a distributed checkpoint.") state = _redistribute_state_dict(original_state) # `allow_partial_load=False` (i.e. strict) raises if a key in `state` (the model's own params) has no # matching entry in the checkpoint, checkpoint keys absent from `state` are always silently ignored. dcp.load(state, storage_reader=reader, planner=DefaultLoadPlanner(allow_partial_load=not strict)) state = _redistribute_state_dict(state, reference_state_dict=original_state) set_model_state_dict(model, state) def save_optimizer_distributed(model, optimizer, checkpoint_dir: str) -> None: """Save optimizer state via DCP. All ranks must call.""" if not _check_distributed_checkpointing_available(): raise OSError("Distributed checkpointing requires `torch>=2.8` with `torch.distributed` available.") # Import here because otherwise it emits a warning every time it's imported on some hardware - this keeps the warning from # being emitted if the function is not used import torch.distributed.checkpoint as dcp from torch.distributed.checkpoint.state_dict import StateDictOptions, get_optimizer_state_dict # Flatten optimizer state and group options into fully qualified name entries. # A mesh change can alter which parameters are DTensors and therefore how optimizer groups are constructed, fully # qualified name keys make the checkpoint independent of grouping. options = StateDictOptions(flatten_optimizer_state_dict=True) optimizer_state_dict = _redistribute_state_dict(get_optimizer_state_dict(model, optimizer, options=options)) dcp.save({"optimizer": optimizer_state_dict}, checkpoint_id=checkpoint_dir) def load_optimizer_distributed(model, optimizer, checkpoint_dir: str) -> None: """Load optimizer state from a DCP directory.""" if not _check_distributed_checkpointing_available(): raise OSError("Distributed checkpointing requires `torch>=2.8` with `torch.distributed` available.") # Import here because otherwise it emits a warning every time it's imported on some hardware - this keeps the warning from # being emitted if the function is not used import torch.distributed.checkpoint as dcp from torch.distributed.checkpoint.state_dict import ( StateDictOptions, get_optimizer_state_dict, set_optimizer_state_dict, ) # Use the same fully qualified name keys format as for saving so PyTorch can map the checkpoint into the destination # optimizer's current groups, even when a mesh change has altered their structure. options = StateDictOptions(flatten_optimizer_state_dict=True) optimizer_state_dict = get_optimizer_state_dict(model, optimizer, options=options) checkpoint_state_dict = _redistribute_state_dict(optimizer_state_dict) dcp.load({"optimizer": checkpoint_state_dict}, checkpoint_id=checkpoint_dir) optimizer_state_dict = _redistribute_state_dict(checkpoint_state_dict, reference_state_dict=optimizer_state_dict) set_optimizer_state_dict(model, optimizer, optimizer_state_dict) def get_distributed_checkpoint_dir( pretrained_model_name_or_path: str | os.PathLike | None, subfolder: str, gguf_file: str | None, state_dict: dict | None, ) -> str | os.PathLike | None: if pretrained_model_name_or_path is None or gguf_file is not None or state_dict is not None: return None candidate_dir = os.path.join(pretrained_model_name_or_path, subfolder) if is_sharded_checkpoint(candidate_dir): return candidate_dir return None def _get_resolved_distributed_checkpoint_files( distributed_checkpoint_dir: str | os.PathLike, hf_quantizer: HfQuantizer | None = None, ) -> list[str]: if not _check_distributed_checkpointing_available(): raise OSError("Loading a distributed checkpoint requires torch>=2.8.") if hf_quantizer is not None: raise ValueError("Quantization is not supported when loading a distributed checkpoint.") return sorted( os.path.join(distributed_checkpoint_dir, name) for name in os.listdir(distributed_checkpoint_dir) if name.endswith(".safetensors") ) - src/transformers/distributed/configuration_utils.py (11759 bytes)
# Copyright 2025 The HuggingFace Team. All rights reserved. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. import json import os import warnings from dataclasses import asdict, dataclass from typing import Literal from ..utils import is_torch_greater_or_equal from .utils import _get_torch_distributed_rank @dataclass class DistributedConfig: """ Configuration for native distributed inference and training with tensor, pipeline, or FSDP2 parallelism. Args: tp_size (`int`, *optional*): Number of devices for tensor parallelism. If `None` and `tp_plan` is set, defaults to `WORLD_SIZE // (other_parallel_size)`. If `None` and no `tp_plan` is set, defaults to 1. tp_plan (`dict[str, str]` or `"auto"`, *optional*): Tensor parallel sharding plan. Pass `"auto"`, or leave as `None` when `tp_size` is set, to use the model's predefined `base_model_tp_plan`. Pass a dictionary to override individual rules of that plan; unspecified rules are kept. An empty dictionary applies no tensor parallel rule. enable_sequence_parallel (`bool`, *optional*, defaults to `False`): Reserved for sequence parallelism. Not wired up yet. enable_expert_parallel (`bool`, *optional*, defaults to `False`): Deprecated, removed in v5.21. Without `ep_size`, same as `ep_size=tp_size, tp_plan={}`: the expert-parallel plan alone across `tp_size` ranks. An explicit `ep_size` takes precedence. This flag does not change `tp_size` or `fsdp_size`. fsdp_size (`int`, *optional*): Number of devices for FSDP (data parallelism). If `None` and `tp_size` is set, defaults to 1. dp_replicate_size (`int`, *optional*): Number of FSDP replicas (HSDP). Parameters are sharded across `fsdp_size` devices and replicated `dp_replicate_size` times: gradients are reduce-scattered within a replica, then all-reduced across replicas. Defaults to 1. We suggest it for multi-node training: plain FSDP all-gathers parameters and reduce-scatters gradients across all devices at every step, so this traffic goes over the slower inter-node links. Set `dp_replicate_size` to the number of nodes and `fsdp_size * tp_size` to the number of devices per node: the sharding traffic then stays on the fast intra-node links, and only the gradient all-reduce crosses nodes. The cost is more memory per device, since parameters are sharded within a node. fsdp_plan (`dict[str, str]`, *optional*): FSDP2 sharding plan. Leave as `None` to use the model's predefined `base_model_fsdp_plan` and class-level `_fsdp_plan`. Pass a dictionary to override individual rules; unspecified rules are kept. Applied only when `dp_replicate_size * fsdp_size > 1`. fsdp_cpu_offload (`bool`, *optional*, defaults to `False`): Whether to enable CPU offloading for FSDP2. fsdp_mixed_precision (`bool`, *optional*, defaults to `False`): Whether to enable mixed precision for FSDP2. pp_size (`int`, *optional*): Number of devices for pipeline parallelism. If `None` and another parallel mode is set, defaults to 1. ep_size (`int`, *optional*): Number of devices owning distinct expert shards. Defaults to 1. Set it explicitly to enable EP. Must be a multiple of `tp_size` and divide `fsdp_size * tp_size`. `ep_size > tp_size` needs an `"ep_dispatch_experts"` rule with an `ep_implementation` that exchanges tokens, such as `"all_to_all"`. ep_plan (`dict[str, str]`, *optional*): Expert parallel sharding plan. Leave as `None` to use the model's predefined `base_model_ep_plan`. Pass a dictionary to override individual rules of that plan; unspecified rules are kept. Applied only when `ep_size > 1`, and its rules take precedence over `tp_plan` rules for the same modules. An `"ep_dispatch_experts"` rule lets `ep_implementation` pick how tokens reach the experts. ep_implementation (`str`, *optional*, defaults to `"auto"`): How an `"ep_dispatch_experts"` rule brings the experts their tokens. It names an `EPImplementation` registered in `EPImplementationInterface`: - `"all_to_all"` exchanges the routed tokens with all-to-all collectives. - `"all_reduce"` runs each rank's experts on the batch the EP group shares, and sums the outputs. It requires `ep_size == tp_size`. - `"deepgemm_megamoe"` hands that batch to DeepGEMM Mega MoE, whose kernel exchanges the tokens itself. Loading with `experts_implementation="deepgemm_megamoe"` selects it. - `"auto"` runs `"all_reduce"` at inference when the EP group shares one batch, and `"all_to_all"` otherwise. """ tp_size: int | None = None tp_plan: dict[str, str] | Literal["auto"] | None = None enable_sequence_parallel: bool = False enable_expert_parallel: bool = False fsdp_size: int | None = None dp_replicate_size: int | None = None fsdp_plan: dict[str, str] | None = None fsdp_cpu_offload: bool = False fsdp_mixed_precision: bool = False pp_size: int | None = None ep_size: int | None = None ep_plan: dict[str, str] | None = None ep_implementation: str = "auto" @property def efsdp_size(self) -> int: """ The number of ranks that own the same experts and FSDP-shard them between each other. Dense and expert parameters are laid out over the same world size: pp x fsdp x tp == pp x efsdp x ep => efsdp = fsdp x tp // ep Experts are not tensor-parallel, so EP and expert-FSDP together cover all the ranks that dense parameters split between FSDP and TP. Example with 16 ranks, ep=8: ep groups : {0..7} {8..15} the 8 ranks of a group together hold all experts (num_experts / 8 each), tokens are routed within it efsdp groups : {0,8} {1,9} ... {7,15} each pair holds the same experts, FSDP-sharded on dim 0 (the expert dim) and all-gathered for compute - ep == tp : efsdp == fsdp - ep == fsdp * tp: efsdp == 1, every expert lives whole on a single rank Sharding over `efsdp` is applied by the EP token-dispatch path (`ep_dispatch_experts`). """ return self.fsdp_size * self.tp_size // self.ep_size @property def dp_size(self) -> int: """The number of ranks that each see a distinct batch: `dp_replicate_size * fsdp_size`.""" return self.dp_replicate_size * self.fsdp_size def __post_init__(self): self._resolve_parallelism() self._validate_mesh_config() def _resolve_parallelism(self): """Resolve parallel sizes and legacy EP settings.""" for value in (self.tp_size, self.fsdp_size, self.dp_replicate_size, self.pp_size, self.ep_size): if value is not None and value < 1: raise ValueError(f"Parallelism sizes must be >= 1, got {value}.") if self.fsdp_size is None: self.fsdp_size = 1 if self.dp_replicate_size is None: self.dp_replicate_size = 1 if self.pp_size is None: self.pp_size = 1 if self.tp_size is None and self.tp_plan is not None: world_size = int(os.environ.get("WORLD_SIZE", 1)) other_parallel_size = self.dp_replicate_size * self.fsdp_size * self.pp_size if world_size % other_parallel_size != 0: raise ValueError( f"WORLD_SIZE ({world_size}) must be divisible by dp_replicate_size * fsdp_size * pp_size " f"({other_parallel_size}) to derive tp_size." ) self.tp_size = world_size // other_parallel_size elif self.tp_size is None: self.tp_size = 1 if self.enable_expert_parallel and self.ep_size is None: # TODO: remove once deprecation cycle is over self.ep_size = self.tp_size if self.tp_plan is None: self.tp_plan = {} if _get_torch_distributed_rank() == 0: warnings.warn( f"`enable_expert_parallel` without `ep_size` is deprecated and will be removed in v5.21. Use " f"ep_size={self.ep_size}, tp_plan={{}} instead, or drop tp_plan to also shard the dense modules.", FutureWarning, stacklevel=4, ) if self.ep_size is None: self.ep_size = 1 # Retain the legacy attribute for callers; internal EP decisions use ep_size. self.enable_expert_parallel = self.ep_size > 1 def _validate_mesh_config(self): """Validate mesh sizes before the model's expert plan is available.""" if self.ep_plan is not None and not isinstance(self.ep_plan, dict): raise ValueError("`ep_plan` must be a dictionary or None.") if self.fsdp_plan is not None and not isinstance(self.fsdp_plan, dict): raise ValueError("`fsdp_plan` must be a dictionary or None.") if self.ep_size > 1: if self.ep_size % self.tp_size: raise ValueError("`ep_size` must be a multiple of `tp_size`.") if (self.fsdp_size * self.tp_size) % self.ep_size: raise ValueError("`ep_size` must divide `fsdp_size * tp_size`.") if self.dp_size > 1 and self.pp_size > 1: raise ValueError( "Combining FSDP with pipeline parallelism is not supported yet. " "Use DistributedConfig(tp_size=N, fsdp_size=M), or combine TP and PP." ) def _validate_resolved_ep_plan(self, ep_plan: dict[str, str]): """Validate the layout against the resolved EP plan, once the model's defaults and overrides are merged.""" if self.ep_size <= 1 or not ep_plan: return if "ep_dispatch_experts" in ep_plan.values(): if self.pp_size > 1: raise ValueError("Combining token dispatch with pipeline parallelism is not supported/tested yet.") if not is_torch_greater_or_equal("2.7"): raise OSError("Expert-parallel token dispatch requires `torch>=2.7`.") @classmethod def from_dict(cls, config_dict: dict, **kwargs) -> "DistributedConfig": merged = {**config_dict, **kwargs} valid_keys = {f.name for f in cls.__dataclass_fields__.values()} return cls(**{k: v for k, v in merged.items() if k in valid_keys}) def to_dict(self) -> dict: return asdict(self) def to_json_string(self) -> str: return json.dumps(self.to_dict(), indent=2) + "\n" def to_json_file(self, json_file_path: str | os.PathLike): with open(json_file_path, "w", encoding="utf-8") as f: f.write(self.to_json_string()) def __repr__(self): return f"{self.__class__.__name__} {self.to_json_string()}" - src/transformers/distributed/expert_parallel.py (22674 bytes)
# Copyright 2026 The HuggingFace Team. All rights reserved. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. """Expert parallelism (EP) for MoE models: the EP styles and the plan application.""" from __future__ import annotations import contextlib from typing import TYPE_CHECKING, NamedTuple from ..utils import logging from ..utils.generic import GeneralInterface from ..utils.import_utils import is_torch_available, is_torch_distributed_available from .parallel_utils import ALL_PARALLEL_STYLES, _use_local_dtensor_params if TYPE_CHECKING: from torch import nn from torch.distributed.device_mesh import DeviceMesh logger = logging.get_logger(__name__) if is_torch_available(): import torch if is_torch_distributed_available(): from torch.distributed import all_gather_into_tensor, all_to_all_single, get_process_group_ranks from torch.distributed._functional_collectives import all_to_all_single_autograd from torch.distributed.tensor import DTensor, Replicate, Shard, distribute_tensor from .parallel_utils import _AllReduceBackward, _AllReduceForward, _WaitTensor class ExpertParallelLayer: """Base class of the expert-parallel styles, with the same hooks as `TensorParallelLayer`.""" def should_use_local_tensors(self, module): """Whether this module's forward requires local inputs and parameters.""" return False def validate_param(self, module, param, mesh, parameter_name=None): """Validate a parameter before applying this EP style.""" pass def shard_param(self, module, param, mesh): """Wrap ONE parameter as a DTensor placeholder. Default: no-op.""" pass def transform_inputs_pre_forward(self, module, args, kwargs, mesh): return args, kwargs def context_around_forward(self, module, mesh): if self.should_use_local_tensors(module): return _use_local_dtensor_params(module) return contextlib.nullcontext() def transform_output_post_forward(self, module, output, mesh): return output def install_forward(self, module, mesh): """Install pre / around / post transforms by replacing module.forward.""" original_forward = module.forward def ep_forward(*args, **kwargs): args, kwargs = self.transform_inputs_pre_forward(module, args, kwargs, mesh) with self.context_around_forward(module, mesh): output = original_forward(*args, **kwargs) return self.transform_output_post_forward(module, output, mesh) module.forward = ep_forward return module class MoEParamShard(ExpertParallelLayer): """Param-only EP style for MoE expert weights (``grouped_gemm``). Shards dim 0 and updates module.num_experts to the per-rank local count, so the experts forward skips the expert ids the EP implementations mark as off-rank. """ def __init__(self, placement, *, shards_expert_dim: bool = False): self.placement = placement self.shards_expert_dim = shards_expert_dim def shard_param(self, module, param, mesh): meta = module._parameters.get(param) if meta is None: return if self.shards_expert_dim and hasattr(module, "num_experts"): global_num_experts = meta.shape[0] expert_parallel_size = mesh.size() if global_num_experts % expert_parallel_size != 0: raise ValueError( f"Cannot evenly shard {global_num_experts} experts across " f"{expert_parallel_size} expert-parallel ranks." ) module.num_experts = global_num_experts // expert_parallel_size # The experts forward masks sentinel rows only when its experts are actually split. module._is_expert_parallel = True module._parameters[param] = torch.nn.Parameter( distribute_tensor(meta, mesh, [self.placement], src_data_rank=None), requires_grad=meta.requires_grad, ) class EPImplementation: """ How the experts of an `ep_dispatch_experts` module get their tokens. One instance is built per experts module when the plan is applied, so an implementation can keep state across forwards (communication buffers, a layout it reuses). Calling it with `(hidden_states, top_k_index, top_k_weights)`, the batch this rank holds (global expert ids), returns the weighted expert output for each of its tokens. It runs this rank's experts with `experts_forward`, on whatever rows, local expert ids and weights it brought them. Gradients must reach every rank that holds a token, routing weights included, so a collective that autograd does not differentiate has to be wrapped in a `torch.autograd.Function`. """ # Set by implementations that only run when every rank of the EP group holds the same batch, e.g. `AllReduceEP`. needs_shared_batch = False def __init__(self, experts: nn.Module, ep_mesh: DeviceMesh, tp_mesh: DeviceMesh): self.experts_forward = experts.forward # the experts' own forward, on local expert ids self.num_local_experts = experts.num_experts self.ep_mesh = ep_mesh self.tp_mesh = tp_mesh # Every rank of the EP group holds the same batch when the EP group is the TP group. tp_ranks, ep_ranks = (get_process_group_ranks(mesh.get_group()) for mesh in (tp_mesh, ep_mesh)) self.shares_batch = tp_ranks == ep_ranks if self.needs_shared_batch and not self.shares_batch: raise ValueError( f"{type(self).__name__} needs every rank of the expert group to hold the same batch (`ep_size == tp_size`)." ) def __call__(self, hidden_states, top_k_index, top_k_weights): raise NotImplementedError class _AllToAllState(NamedTuple): top_k_weights: torch.Tensor order: torch.Tensor send_sizes: list[int] recv_sizes: list[int] num_tokens: int class AllToAllEP(EPImplementation): """ The ranks of a TP group keep disjoint slices of the batch they share, send each (token, expert) pair to the expert's rank with an all-to-all and get the results back with a second one, then gather the slices back over TP. The split sizes are read back to the host every layer, so it does not capture in a CUDA graph. Example: 8 experts [E0, E7] with DistributedConfig(tp_size=2, fsdp_size=4, ep_size=4): - fsdp_size=4: 4 batches [B0, B3] - tp_size=2: the 2 ranks of a TP group share one batch - efsdp_size = fsdp_size * tp_size / ep_size = 4 * 2 / 4 = 2 copies of each expert GPU 0 1 2 3 4 5 6 7 | | | | | | | | dense view --------------------------------------------------------- batch [====B0====] [====B1====] [====B2====] [====B3====] tp_size [___________ 0 ___________] [___________ 1 ___________] fsdp_size 0 1 2 3 expert view --------------------------------------------------------- experts E0E1 E2E3 E4E5 E6E7 E0E1 E2E3 E4E5 E6E7 ep_size 0 1 2 3 0 1 2 3 efsdp_size [___________ 0 ___________] [___________ 1 ___________] The TP slicing keeps a token from being computed twice. Say token 1 of B0 picks E4, which lives on rank 2. B0 sits on ranks 0 and 1, so if both sent it, E4 would compute it twice: token 1 falls in rank 0's slice, so rank 0 sends it and rank 1 does not. """ def __call__(self, hidden_states, top_k_index, top_k_weights): tokens, expert_ids, weights, state = self._dispatch_tokens(hidden_states, top_k_index, top_k_weights) if tokens.shape[0] == 0: # Keep the empty rows connected to backward, so the reverse exchange and FSDP's reduction still run. expert_output = self.experts_forward( tokens.sum(dim=0, keepdim=True), expert_ids.new_zeros((1, 1)), weights.new_ones((1, 1)) )[:0] else: expert_output = self.experts_forward(tokens, expert_ids, weights) return self._combine_tokens(expert_output, state) def _dispatch_tokens(self, hidden_states, top_k_index, top_k_weights): """Each (token, expert) pair to the expert's rank: the rows this rank's experts run on, their local expert ids and weights, and what `_combine_tokens` needs to bring the results back.""" num_tokens = hidden_states.size(0) if self.tp_mesh.size() > 1: # Replicate -> Shard(0) is a local chunk (no communication); its backward all-gathers the row gradients. hidden_states, top_k_index, top_k_weights = ( DTensor.from_local(t, self.tp_mesh, [Replicate()], run_check=False) .redistribute(self.tp_mesh, [Shard(0)]) .to_local() for t in (hidden_states, top_k_index, top_k_weights) ) ep_group, ep_size = self.ep_mesh.get_group(), self.ep_mesh.size() num_top_k = top_k_index.size(-1) # Sorting the selected pairs by expert groups them by owner rank, since each rank owns a contiguous range of # experts, and the per-expert counts tell every receiver which expert each token it gets is for. The split # sizes are the one host sync of the layer. expert_ids = top_k_index.reshape(-1) order = torch.argsort(expert_ids) send_tokens = hidden_states[order // num_top_k] send_counts = torch.zeros(self.num_local_experts * ep_size, dtype=torch.long, device=hidden_states.device) send_counts = send_counts.scatter_add_(0, expert_ids, torch.ones_like(expert_ids)) send_counts = send_counts.view(ep_size, self.num_local_experts) recv_counts = torch.empty_like(send_counts) all_to_all_single(recv_counts, send_counts, group=ep_group) send_sizes, recv_sizes = torch.stack([send_counts.sum(dim=1), recv_counts.sum(dim=1)]).tolist() recv_tokens = all_to_all_single_autograd(send_tokens, recv_sizes, send_sizes, ep_group) # hand the experts the received rows, not the all-to-all's pending result (whose pointer is null until # waited on): https://github.com/huggingface/transformers/pull/49421 recv_tokens = _WaitTensor.apply(recv_tokens) recv_expert_ids = torch.arange(self.num_local_experts, device=hidden_states.device).repeat(ep_size) recv_expert_ids = recv_expert_ids.repeat_interleave(recv_counts.reshape(-1), output_size=sum(recv_sizes)) recv_expert_ids = recv_expert_ids.unsqueeze(-1) # Experts run top-1 with unit weights; the routing weights are applied in `_combine_tokens`, on the sender. recv_weights = torch.ones_like(recv_expert_ids, dtype=recv_tokens.dtype) state = _AllToAllState(top_k_weights, order, send_sizes, recv_sizes, num_tokens) return recv_tokens, recv_expert_ids, recv_weights, state def _combine_tokens(self, expert_output, state): """The expert outputs back to the ranks that sent the tokens, weighted and summed per token.""" hidden_dim = expert_output.size(-1) recv_out = all_to_all_single_autograd( expert_output, state.send_sizes, state.recv_sizes, self.ep_mesh.get_group() ) # Restore the original (token, top-k slot) order, then apply routing weights. token_outputs = torch.empty_like(recv_out) token_outputs[state.order] = recv_out token_outputs = token_outputs.view(*state.top_k_weights.shape, hidden_dim) output = (token_outputs * state.top_k_weights.unsqueeze(-1)).sum(dim=1) if self.tp_mesh.size() == 1: return output # Shard(0) -> Replicate is one all-gather of the row slices, uneven or empty ones too. output = DTensor.from_local( output.contiguous(), self.tp_mesh, [Shard(0)], shape=(state.num_tokens, hidden_dim), stride=(hidden_dim, 1), ) return output.full_tensor() class AllReduceEP(EPImplementation): """ When the EP group shares one batch (`ep_size == tp_size`): every rank runs its own experts on the whole batch, masking the routes to other ranks' experts (local id `num_local_experts`, weight 0), and an all-reduce sums the partial outputs. No tokens move and no split size is read back, so it captures in a CUDA graph; the masked routes still go through the experts' gather and elementwise passes, so its cost grows with the batch. Example: 4 experts on 2 ranks, top-2. Token t picks E1 (weight w1) and E2 (weight w2). GPU 0 1 | | batch [====== B ======] [====== B ======] the same tokens on both ranks experts E0 E1 E2 E3 top_k_index[t] [1, 2] [1, 2] global expert ids local index[t] [1, 2] [2, 0] 2 = not on this rank, masked top_k_weights[t] [w1, 0] [0, w2] expert output[t] w1 * E1(t) w2 * E2(t) after all-reduce w1 * E1(t) + w2 * E2(t) on both ranks """ needs_shared_batch = True def __call__(self, hidden_states, top_k_index, top_k_weights): ep_group = self.ep_mesh.get_group() # Each rank backpropagates through its own experts only, so the gradients of the shared inputs it computes are # partial: sum them over the group (the forward all-reduce has an identity backward). hidden_states = _AllReduceBackward.apply(hidden_states, ep_group) top_k_weights = _AllReduceBackward.apply(top_k_weights, ep_group) off_rank = (top_k_index // self.num_local_experts) != self.ep_mesh.get_local_rank() local_index = torch.where(off_rank, self.num_local_experts, top_k_index % self.num_local_experts) expert_output = self.experts_forward(hidden_states, local_index, top_k_weights.masked_fill(off_rank, 0.0)) return _AllReduceForward.apply(expert_output, ep_group) def _uses_megamoe(experts: nn.Module) -> bool: """Mega MoE experts exchange the tokens inside their kernel, so only `MegaMoeEP` can run them.""" return getattr(getattr(experts, "config", None), - src/transformers/distributed/fsdp.py (11983 bytes)
# Copyright 2026 The HuggingFace Team. All rights reserved. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. from __future__ import annotations import inspect import os from typing import TYPE_CHECKING, Any from ..utils import is_torch_available, is_torch_distributed_available, logging, strtobool from ..utils.quantization_config import QuantizationMethod from .parallel_utils import replace_layer_number_by_wildcard from .utils import _is_torch_distributed_initialized, _warning_once_on_rank_zero if TYPE_CHECKING: import torch.nn as nn from .configuration_utils import DistributedConfig from .utils import TransformersDeviceMesh if is_torch_available(): import torch if is_torch_distributed_available(): from torch.distributed._composable.fsdp import fully_shard from torch.distributed.fsdp import CPUOffloadPolicy, MixedPrecisionPolicy logger = logging.get_logger(__name__) # FSDP plan styles: whether a module frees its full weight after forward (and re-gathers it for backward) or keeps it. ALL_FSDP_STYLES = ("free_full_weight", "keep_full_weight") def is_fsdp_enabled() -> bool: """Check if FSDP is active via Accelerate (env var based) — covers FSDP1 only.""" return ( _is_torch_distributed_initialized() and strtobool(os.environ.get("ACCELERATE_USE_FSDP", "False")) == 1 and strtobool(os.environ.get("FSDP_CPU_RAM_EFFICIENT_LOADING", "False")) == 1 ) def is_fsdp_managed_module(module: nn.Module) -> bool: """Check if a module is managed by FSDP (1 or 2).""" if not is_torch_distributed_available(): return False # FSDP2: attribute set by apply_fsdp2() if getattr(module, "_is_fsdp_managed_module", False): return True # FSDP1: wrapped by FullyShardedDataParallel from torch.distributed.fsdp import FullyShardedDataParallel return isinstance(module, FullyShardedDataParallel) def _get_fsdp_policy_kwargs(distributed_config: DistributedConfig | None) -> dict[str, Any]: """Build ``fully_shard`` policy kwargs from ``DistributedConfig`` runtime flags.""" if distributed_config is None: return {} fsdp_policy_kwargs = {} if distributed_config.fsdp_cpu_offload: fsdp_policy_kwargs["offload_policy"] = CPUOffloadPolicy() if distributed_config.fsdp_mixed_precision: fsdp_policy_kwargs["mp_policy"] = MixedPrecisionPolicy( param_dtype=torch.bfloat16, reduce_dtype=torch.float32, output_dtype=None, ) return fsdp_policy_kwargs def _get_input_output_embeddings(model: nn.Module) -> tuple[nn.Module | None, nn.Module | None]: input_embed = None output_head = None if hasattr(model, "get_input_embeddings"): input_embed = model.get_input_embeddings() if hasattr(model, "get_output_embeddings"): output_head = model.get_output_embeddings() return input_embed, output_head def is_norm_and_head_pair(no_reshard_targets: list[tuple[str, nn.Module]], model: nn.Module) -> bool: if len(no_reshard_targets) != 2: return False input_embed, output_head = _get_input_output_embeddings(model) head_modules = {module for module in (input_embed, output_head) if module is not None} names, modules = [], [] for name, module in no_reshard_targets: names.append(name) modules.append(module) has_final_norm = any(name == "norm" or name.endswith(".norm") for name in names) has_output_head = any(module in head_modules for module in modules) return has_final_norm and has_output_head def _resolve_tied_embed_lm_head_plan( fsdp_plan: dict[str, str], model: nn.Module, ) -> dict[str, str]: """ Rewrite the plan so tied embed/lm_head weights are wrapped once. Example: {"model.embed_tokens": "free_full_weight", "model.layers.*": "free_full_weight", "model.norm": "keep_full_weight", "lm_head": "keep_full_weight"} -> {"model.layers.*": "free_full_weight", "model.norm": "keep_full_weight", "model.embed_tokens": "keep_full_weight"} """ tied_keys = getattr(model, "all_tied_weights_keys", None) or {} if not tied_keys: return fsdp_plan input_embed, output_head = _get_input_output_embeddings(model) name_by_module = {module: name for name, module in model.named_modules()} embed_module = name_by_module.get(input_embed) head_module = name_by_module.get(output_head) if embed_module is None or head_module is None: return fsdp_plan adapted_plan = fsdp_plan.copy() adapted_plan.pop(embed_module, None) if fsdp_plan.get(head_module) == "keep_full_weight": adapted_plan.pop(head_module, None) adapted_plan[embed_module] = "keep_full_weight" return adapted_plan def verify_fsdp_plan(module_names: list[str], fsdp_plan: dict[str, str] | None) -> None: """ Verify the FSDP plan of the model, log a warning if plan keys were not applied or strategies are invalid. """ _warning_once_on_rank_zero( "`verify_fsdp_plan` is deprecated and will be removed in v5.23: the model validates its plans itself when " "they are set and in `_resolve_parallel_plans`." ) if not fsdp_plan: return name_lookup = dict.fromkeys(module_names) unused_rules: dict[str, str] = {} invalid_strategies: dict[str, str] = {} for key, strategy in fsdp_plan.items(): if strategy not in {"free_full_weight", "keep_full_weight"}: invalid_strategies[key] = strategy elif key not in name_lookup and not any(replace_layer_number_by_wildcard(name) == key for name in name_lookup): unused_rules[key] = strategy if invalid_strategies: logger.warning(f"The following FSDP entries have unknown strategies: {invalid_strategies}") if unused_rules: logger.warning(f"The following FSDP rules were not applied to any module: {unused_rules}") def apply_fully_sharded_data_parallelism( model: nn.Module, mesh_manager: TransformersDeviceMesh, fsdp_plan: dict[str, str], ep_plan: dict[str, str], ) -> nn.Module: """ Apply FSDP2 (fully_shard) to a model: dispatched experts on `efsdp` and the rest of the modules on `fsdp`. Torch availability, distributed initialization and the version requirement are asserted upstream by `initialize_distributed_mesh`. """ distributed_config = getattr(model.config, "distributed_config", None) if distributed_config is not None and distributed_config.dp_replicate_size > 1: fsdp_mesh = mesh_manager.get_mesh(("dp_replicate", "fsdp")) else: fsdp_mesh = mesh_manager.get_mesh("fsdp") fsdp_policy_kwargs = _get_fsdp_policy_kwargs(distributed_config) if distributed_config is not None and distributed_config.ep_size > 1: if distributed_config.dp_replicate_size > 1: expert_mesh = mesh_manager.get_mesh(("dp_replicate", "efsdp")) else: expert_mesh = mesh_manager.get_mesh("efsdp") for module_name, module in model.named_modules(): if ep_plan.get(module_name) == "ep_dispatch_experts": fully_shard(module, mesh=expert_mesh, reshard_after_forward=True, **fsdp_policy_kwargs) # an EP group holds ep_size / tp_size distinct batches # the efsdp reduce the sums on efsdp_size copies of that expert. # The expert gradient therefore covers ep_size / tp_size * efsdp_size = fsdp_size batches # not efsdp_size batches (the relationship is efsdp_size = fsdp_size * tp_size / ep_size) # With HSDP, the all-reduce across replicas adds dp_replicate_size times more: dp_size batches. module.set_gradient_divide_factor(float(distributed_config.dp_size)) # FSDP2 applies a divide factor other than the group size through PREMUL_SUM, which non-NCCL backends # reject and NCCL zeroes in bfloat16 up to torch 2.13 (pytorch/pytorch#190747): sum, divide locally. if distributed_config.efsdp_size > 1: # No reduce-scatter if efsdp_size == 1: FSDP2 already divides, so a forced sum would divide twice. module.set_force_sum_reduction_for_comms(True) reshard_targets, no_reshard_targets = [], [] for name, style in fsdp_plan.items(): if style == "keep_full_weight": no_reshard_targets.append((name, model.get_submodule(name))) else: reshard_targets.append((name, model.get_submodule(name))) for name, module in reshard_targets: fully_shard(module, mesh=fsdp_mesh, reshard_after_forward=True, **fsdp_policy_kwargs) logger.debug(f"Applied fully_shard to {name} (reshard=True)") # Optimization: when the keep buffer is exactly the (final_norm, lm_head/embed) # tail pair, bundle them into one fully_shard so that we dont need to do all-gather during backward pass. if is_norm_and_head_pair(no_reshard_targets, model): names, modules = zip(*no_reshard_targets) fully_shard(list(modules), mesh=fsdp_mesh, reshard_after_forward=False, **fsdp_policy_kwargs) logger.debug(f"Grouped tail {names} (reshard=False)") else: for name, module in no_reshard_targets: fully_shard(module, mesh=fsdp_mesh, reshard_after_forward=False, **fsdp_policy_kwargs) logger.debug(f"Applied fully_shard to {name} (reshard=False)") # Apply FSDP2 to the root module fully_shard(model, mesh=fsdp_mesh, **fsdp_policy_kwargs) logger.info(f"FSDP2 applied to model via _fsdp_plan: {len(fsdp_plan)} entries") # Used by generation code to detect FSDP and enable synced_gpus. model._is_fsdp_managed_module = True # NOTE(3outeille): No need to tie the word embeddings here, it will be done _finalize_model_loading in modeling_utils.py return model # ========================= PEFT compatibility ========================= # TODO(3outeille): make sure new FSDP works with PEFT def get_fsdp_ckpt_kwargs(): """ Returns checkpoint kwargs for FSDP model saving. Checks if the `adapter_only` parameter is supported by `save_fsdp_model` from accelerate and returns the appropriate kwargs. """ from accelerate.utils import save_fsdp_model if "adapter_only" in list(inspect.signature(save_fsdp_model).parameters): return {"adapter_only": True} else: return {} def update_fsdp_plugin_peft(model, accelerator): """ Updates the FSDP plugin for PEFT LoRA/QLoRA compatibility. When using FSDP with PEFT LoRA, the auto wrap policy needs to be updated to additionally wrap LoRA trainable layers separately. When using FSDP with QLoRA, the mixed precision policy needs to be updated to use the quantization storage data type. """ from peft import PeftConfig from peft.utils.other import fsdp_auto_wrap_policy if isinstance(model.active_peft_config, PeftConfig): accelerator.state.fsdp_plugin.auto_wrap_policy = fsdp_auto_wrap_policy(model) if ( getattr(model, "quantization_method", None) == QuantizationMethod.BITS_AND_BYTES and model.hf_quantizer.quantization_config.bnb_4bit_quant_storage.is_floating_point ): accelerator.state.fsdp_plugin.set_mixed_precision( model.hf_quantizer.quantization_config.bnb_4bit_quant_storage, override=True ) - src/transformers/distributed/mixin.py (22449 bytes)
# Copyright 2026 The HuggingFace Team. All rights reserved. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. from __future__ import annotations import os import re import shutil from itertools import chain from typing import TYPE_CHECKING import torch from ..core_model_loading import build_glob_alternation from ..dynamic_module_utils import custom_object_save from ..utils import is_torch_greater_or_equal, logging from ..utils.hub import create_and_tag_model_card from ..utils.loading_report import LoadStateDictInfo from .checkpoint import ( _load_model_checkpoint_distributed, _save_model_checkpoint_distributed, consolidate_distributed_checkpoint, ) from .configuration_utils import DistributedConfig from .expert_parallel import apply_expert_parallelism from .fsdp import ALL_FSDP_STYLES, _resolve_tied_embed_lm_head_plan, apply_fully_sharded_data_parallelism from .parallel_utils import ALL_PARALLEL_STYLES, _get_parameter_plan, replace_layer_number_by_wildcard from .pipeline_parallel import apply_pipeline_parallelism from .tensor_parallel import apply_tensor_parallelism, gather_state_dict_for_save from .utils import ( TransformersDeviceMesh, _check_distributed_checkpointing_available, _distributed_barrier, _get_torch_distributed_rank, _is_torch_distributed_initialized, gather_full_state_dict, initialize_distributed_mesh, ) logger = logging.get_logger(__name__) if TYPE_CHECKING: import torch.nn as nn class DistributedMixin: """Distributed orchestration and save/load hooks for [`PreTrainedModel`].""" _device_mesh = None _is_distributed_loading_by_transformers = False _mesh_manager: TransformersDeviceMesh | None = None _tp_plan: dict[str, str] | None = None _ep_plan: dict[str, str] | None = None _tp_size = None _fsdp_size = None _pp_plan: dict[str, tuple[str, str]] | None = None _fsdp_plan: dict[str, str] | None = None @property def is_distributed_loading_by_transformers(self) -> bool: """ Whether `from_pretrained(distributed_config=...)` sharded the model at load time (each rank reads only its shard through `DtensorShardOperation`), rather than a wrapper such as Accelerate sharding it afterwards. """ return self._is_distributed_loading_by_transformers def init_parallel_plans(self) -> None: """Copy class-level plans onto the instance and merge config/children contributions.""" model_cls = type(self) self._tp_plan = dict(getattr(model_cls, "_tp_plan", None) or {}) self._ep_plan = dict(getattr(model_cls, "_ep_plan", None) or {}) self._pp_plan = dict(getattr(model_cls, "_pp_plan", None) or {}) self._fsdp_plan = dict(getattr(model_cls, "_fsdp_plan", None) or {}) if self.base_model is self: self._pp_plan.update(self.config.base_model_pp_plan or {}) self._tp_plan.update(self.config.base_model_tp_plan or {}) self._ep_plan.update(self.config.base_model_ep_plan or {}) self._fsdp_plan.update(self.config.base_model_fsdp_plan or {}) # Shard the input embeddings on the vocabulary unless the plan already names them. if self._tp_plan and isinstance(getattr(self, "embed_tokens", None), torch.nn.Embedding): self._tp_plan = {"embed_tokens": "embedding_rowwise", **self._tp_plan} for name, module in self.named_children(): if plan := getattr(module, "_ep_plan", None): self._ep_plan.update({f"{name}.{k}": v for k, v in plan.copy().items()}) if plan := getattr(module, "_tp_plan", None): self._tp_plan.update({f"{name}.{k}": v for k, v in plan.copy().items()}) if plan := getattr(module, "_pp_plan", None): self._pp_plan.update({f"{name}.{k}": v for k, v in plan.copy().items()}) if plan := getattr(module, "_fsdp_plan", None): self._fsdp_plan.update({f"{name}.{k}": v for k, v in plan.copy().items()}) @property def tp_plan(self) -> dict[str, str]: """The full tensor parallel plan for the model's modules.""" return self._tp_plan @property def ep_plan(self) -> dict[str, str]: """The full expert parallel plan for the model's modules, kept separate from `tp_plan`.""" return self._ep_plan @property def fsdp_plan(self) -> dict[str, str]: return self._fsdp_plan @property def pp_plan(self) -> dict[str, tuple[str, str]]: return self._pp_plan @tp_plan.setter def tp_plan(self, plan: dict[str, str] | None): if plan is None: self._tp_plan = {} return if not isinstance(plan, dict): raise ValueError("Can only set a dictionary as `tp_plan`") self._validate_parallel_plan(plan, ALL_PARALLEL_STYLES, "tp_plan") self._tp_plan = plan @ep_plan.setter def ep_plan(self, plan: dict[str, str] | None): if plan is None: self._ep_plan = {} return if not isinstance(plan, dict): raise ValueError("Can only set a dictionary as `ep_plan`") self._validate_parallel_plan(plan, ALL_PARALLEL_STYLES, "ep_plan") self._ep_plan = plan @fsdp_plan.setter def fsdp_plan(self, plan: dict[str, str] | None): if plan is None: self._fsdp_plan = {} return if not isinstance(plan, dict): raise ValueError("Can only set a dictionary as `fsdp_plan`") self._validate_parallel_plan(plan, ALL_FSDP_STYLES, "fsdp_plan") self._fsdp_plan = plan @pp_plan.setter def pp_plan(self, plan: dict[str, tuple[str, str]] | None): if plan is None: self._pp_plan = {} return if not isinstance(plan, dict): raise ValueError("Can only set a dictionary as `pp_plan`") self._pp_plan = plan @classmethod def prepare_distribute_model( cls, distributed_config: DistributedConfig | dict | None, device_map=None, ) -> tuple[DistributedConfig | None, object, TransformersDeviceMesh | None]: if distributed_config is None: return None, device_map, None if isinstance(distributed_config, dict): distributed_config = DistributedConfig.from_dict(distributed_config) if distributed_config.tp_size == 1 and distributed_config.dp_size == 1 and distributed_config.pp_size == 1: return distributed_config, device_map, None if distributed_config.tp_size > 1 and device_map is not None: raise ValueError("Tensor parallelism and `device_map` are mutually exclusive.") if distributed_config.dp_size > 1 and not is_torch_greater_or_equal("2.8"): raise OSError("FSDP2 requires `torch>=2.8` (distributed checkpoint save/load).") device_map, mesh_manager = initialize_distributed_mesh(distributed_config) return distributed_config, device_map, mesh_manager def _validate_parallel_plan( self, plan: dict[str, str], supported_styles, plan_name: str, names: set[str] | None = None ) -> None: """Raise on a style that `supported_styles` does not know, warn on a rule that names nothing in the model.""" unsupported_styles = {style for style in plan.values() if style not in supported_styles} if unsupported_styles: raise ValueError( f"Unsupported `{plan_name}` styles: {unsupported_styles}. Supported styles are {list(supported_styles)}" ) if names is None: names = { replace_layer_number_by_wildcard(n) for n, _ in chain(self.named_modules(), self.named_parameters()) } if unused := plan.keys() - names: if _get_torch_distributed_rank() == 0: logger.warning( f"`{plan_name}` rules {sorted(unused)} match nothing in {type(self).__name__} and will not be applied." ) def _expand_parallel_plan(self, plan: dict[str, str]) -> dict[str, str]: """Replace the `*` of the plan's rules with the layer numbers of the model, one entry per module or parameter.""" expanded_plan = {} for module_name, module in self.named_modules(): module_style = _get_parameter_plan(parameter_name=module_name, plan=plan, is_weight=False) if module_style in ALL_FSDP_STYLES: # FSDP: list the module to wrap. expanded_plan[module_name] = module_style else: # TP/EP: list the module's own parameters (to shard) before the module (to hook). for param_name, _ in module.named_parameters(recurse=False): name = f"{module_name}.{param_name}" if module_name else param_name param_style = _get_parameter_plan(parameter_name=name, plan=plan, is_weight=True) if param_style is not None: expanded_plan[name] = param_style if module_style is not None: expanded_plan[module_name] = module_style return expanded_plan def _resolve_parallel_plans( self, distributed_config: DistributedConfig ) -> tuple[dict[str, str], dict[str, str], dict[str, str]]: """Build the final TP, EP and FSDP plans: apply the config overrides, then replace `*` with the layer numbers. Each returned plan maps one concrete module or parameter name to a style, e.g. `model.layers.3.mlp.up_proj` instead of `model.layers.*.mlp.up_proj`. A module that has an EP rule keeps no TP rule. """ module_names = {replace_layer_number_by_wildcard(n) for n, _ in self.named_modules()} param_names = {replace_layer_number_by_wildcard(n) for n, _ in self.named_parameters()} names = module_names | param_names for plan_name in ("tp_plan", "ep_plan", "fsdp_plan"): override = getattr(distributed_config, plan_name) if isinstance(override, dict): plan = getattr(self, plan_name) if unknown := override.keys() - names - plan.keys(): raise ValueError(f"`{plan_name}` keys {sorted(unknown)} match nothing in {type(self).__name__}.") setattr(self, f"_{plan_name}", {} if plan_name == "tp_plan" and not override else plan | override) tp_plan = dict(self.tp_plan) if distributed_config.tp_size > 1 else {} ep_plan = dict(self.ep_plan) if distributed_config.ep_size > 1 else {} # HSDP shards each replica with the FSDP plan. fsdp_plan = dict(self.fsdp_plan) if distributed_config.dp_size > 1 else {} for kind, plan in (("tp", tp_plan), ("ep", ep_plan), ("fsdp", fsdp_plan)): size = distributed_config.dp_size if kind == "fsdp" else getattr(distributed_config, f"{kind}_size") if size > 1 and not plan: raise ValueError( f"`{kind}_size={size}` was requested, but `{type(self).__name__}` has no `{kind}_plan`. Pass " f"`{kind}_plan` in `DistributedConfig`, add `base_model_{kind}_plan` to the model's config, " f"or set `{kind}_size=1`." ) if "ep_router" in ep_plan.values() or "moe_tp_experts" in ep_plan.values(): raise ValueError( "The `ep_router` style was removed, and `moe_tp_experts` is a tensor-parallel style that belongs in " "`tp_plan`. Drop the router rule and shard the experts by expert with `ep_dispatch_experts` and " "`grouped_gemm`, e.g. `ep_plan={'layers.*.mlp.experts': 'ep_dispatch_experts', " "'layers.*.mlp.experts.gate_up_proj': 'grouped_gemm', 'layers.*.mlp.experts.down_proj': " "'grouped_gemm'}` (most MoE models default to this plan, so `ep_plan` can usually be left out). " '`DistributedConfig(ep_implementation="all_reduce")` gives the masking and all-reduce of the old ' "`ep_router` + `moe_tp_experts` plan." ) # EP rules take precedence: drop TP rules on EP modules and their children. is_expert = re.compile(rf"(?:{'|'.join(map(re.escape, ep_plan))})(?:\..+)?").fullmatch tp_plan = {name: style for name, style in tp_plan.items() if not is_expert(name)} self._validate_parallel_plan(tp_plan, ALL_PARALLEL_STYLES, "tp_plan", names) self._validate_parallel_plan(ep_plan, ALL_PARALLEL_STYLES, "ep_plan", names) self._validate_parallel_plan(fsdp_plan, ALL_FSDP_STYLES, "fsdp_plan", names) distributed_config._validate_resolved_ep_plan(ep_plan) # Tied embed/lm_head weights are wrapped once: fold the head's FSDP rule into the embedding's. fsdp_plan = _resolve_tied_embed_lm_head_plan(fsdp_plan, self) # FSDP2 needs one group per dtype: give the modules kept in float32 their own group. if fsdp_plan and (dtype_plan := self._get_dtype_plan(self.dtype)): kept_in_fp32, _, _ = build_glob_alternation(list(dtype_plan)) fp32_modules = {name.rpartition(".")[0] for name in param_names if kept_in_fp32.search(name)} fsdp_plan = dict.fromkeys(fp32_modules, "free_full_weight") | fsdp_plan tp_plan = self._expand_parallel_plan(tp_plan) ep_plan = self._expand_parallel_plan(ep_plan) fsdp_plan = self._expand_parallel_plan(fsdp_plan) # FSDP2 needs a submodule sharded before the module containing it, e.g. `mlp.gate` before its layer. fsdp_plan = dict(sorted(fsdp_plan.items(), key=lambda item: item[0].count("."), reverse=True)) return tp_plan, ep_plan, fsdp_plan @classmethod def maybe_distribute_model( cls, model: nn.Module, distributed_config: DistributedConfig | None, mesh_manager: TransformersDeviceMesh | None, ): """Apply pipeline, tensor and expert parallelism, then FSDP2, after model init and before weight loading.""" if mesh_manager is None: return model model.config.distributed_config = distributed_config model._mesh_manager = mesh_manager model._device_mesh = mesh_manager.get_mesh(("pp", "fsdp", "tp")) model._is_distributed_loading_by_transformers = True model._tp_size = distributed_config.tp_size model._fs - src/transformers/distributed/parallel_utils.py (5985 bytes)
# Copyright 2026 The HuggingFace Team. All rights reserved. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. import contextlib import re from ..utils.generic import GeneralInterface from ..utils.import_utils import is_torch_available, is_torch_distributed_available if is_torch_available(): import torch if is_torch_distributed_available(): import torch.distributed as dist from torch.distributed._functional_collectives import wait_tensor from torch.distributed.tensor import DTensor class ParallelInterface(GeneralInterface): """Registry of the named parallel styles: `tensor_parallel.py` registers the TP styles, `expert_parallel.py` the EP ones.""" _global_mapping = {} ALL_PARALLEL_STYLES: ParallelInterface = ParallelInterface() def replace_layer_number_by_wildcard(name: str) -> str: """ Replace the numbers in the `name` by wildcards, only if they are in-between dots (`.`) or if they are between a dot (`.`) and the end of the string. This matches how modules are named/numbered when using a nn.ModuleList or nn.Sequential, but will NOT match numbers in a parameter name itself, e.g. if the param is named `"w1"` or `"w2"`. """ return re.sub(r"\.\d+(\.|$)", lambda m: ".*" + m.group(1), name) def _get_parameter_plan(parameter_name: str, plan: dict[str, str], is_weight=True) -> str | None: """ Get the parallel style for a parameter or module from a TP or EP plan. The plan is a dictionary that maps parameter or module names to parallel styles. The parameter name can be a generic name with wildcards (e.g. "*.weight") or a specific name (e.g. "layer_1.weight"). The `is_weight` is important because for weights, we want to support `.weights` and `.bias` cases seamlessly! but not parent classes for `post_init` calls """ generic_param_name = replace_layer_number_by_wildcard(parameter_name) if generic_param_name in plan: return plan[generic_param_name] elif is_weight and "." in generic_param_name and (module_name := generic_param_name.rsplit(".", 1)[0]) in plan: return plan[module_name] return None @contextlib.contextmanager def _use_local_dtensor_params(module): # Kernels as DeepGEMM require local tensors rather than DTensors. # We temporarily convert the DTensors to local tensors for the duration of forward() and swap them back after. originals = {name: param for name, param in module.named_parameters(recurse=False) if isinstance(param, DTensor)} local_params = {name: param.to_local() for name, param in originals.items()} module._parameters.update(local_params) try: yield finally: for name, original in originals.items(): current = module._parameters[name] if current is local_params[name]: module._parameters[name] = original # Some kernels such as Megamoe performs changing on FP4 weights and scale factors during its forward pass. # That implies creating a new Parameter so we should not restore the original DTensor. elif current is not None and not isinstance(current, DTensor): replacement = DTensor.from_local( current, original.device_mesh, original.placements, run_check=False, ) module._parameters[name] = torch.nn.Parameter( replacement, requires_grad=current.requires_grad, ) if is_torch_distributed_available(): class _AllReduceForward(torch.autograd.Function): """Allreduce-sum forward, identity backward.""" @staticmethod def forward(ctx, x, process_group): if dist.get_world_size(process_group) > 1: dist.all_reduce(x, group=process_group) return x @staticmethod def backward(ctx, grad): return grad, None class _AllReduceBackward(torch.autograd.Function): """Identity forward, allreduce-sum backward. Used for MoE routing weights: the forward value is replicated (same on all ranks), but the backward gradient is partial (each rank has 1/tp_size from its expert shard). We need to sum the partial gradients without dividing by world_size, which is what DTensor's Replicate backward does incorrectly. `apply` returns `x` as is when no gradient can flow: at inference the per-call overhead of an `autograd.Function` compounds across layers, e.g. in eager decode. """ @classmethod def apply(cls, x, process_group): if torch.is_grad_enabled() and x.requires_grad: return super().apply(x, process_group) return x @staticmethod def forward(ctx, x, process_group): ctx.process_group = process_group return x @staticmethod def backward(ctx, grad): grad = grad.contiguous() dist.all_reduce(grad, group=ctx.process_group) return grad, None class _WaitTensor(torch.autograd.Function): """Wait on a functional collective's result, identity backward (`wait_tensor` has no autograd kernel before torch 2.11).""" @staticmethod def forward(ctx, x): return wait_tensor(x) @staticmethod def backward(ctx, grad): return grad - src/transformers/distributed/pipeline_parallel.py (12241 bytes)
# Copyright 2026 The HuggingFace Team. All rights reserved. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. from __future__ import annotations import inspect from functools import wraps from typing import TYPE_CHECKING from ..modeling_outputs import CausalLMOutputWithPast from ..utils import is_torch_available, is_torch_distributed_available if TYPE_CHECKING: import torch.nn as nn if is_torch_available(): import torch import torch.nn as nn if is_torch_distributed_available(): import torch.distributed as dist def _bind_forward_kwargs(forward_signature: inspect.Signature, args: tuple, kwargs: dict) -> dict: bound = forward_signature.bind_partial(*args, **kwargs) bound.apply_defaults() return dict(bound.arguments) class PipelineIdentityLayer(nn.Identity): """An identity layer replacing modules not owned by the current pipeline stage.""" def __init__(self, *args, **kwargs): super().__init__() def forward(self, *args, **kwargs): """Return the first arg from args or the first value from kwargs as nothing is used.""" return args[0] if args else next(iter(kwargs.values())) class PipelineStage: """Pipeline-parallel stage metadata derived from a 1-D PP device mesh.""" def __init__(self, pp_mesh: torch.distributed.device_mesh.DeviceMesh): self.pp_rank = pp_mesh.get_local_rank() self.pp_size = pp_mesh.size() self.pp_group = pp_mesh.get_group() self.pp_is_first_stage = self.pp_rank == 0 self.pp_is_last_stage = self.pp_rank == self.pp_size - 1 self.pp_prev_rank = self.pp_rank - 1 if self.pp_rank > 0 else None self.pp_next_rank = self.pp_rank + 1 if self.pp_rank < self.pp_size - 1 else None self.comm_on_cpu = dist.get_backend(self.pp_group) == "gloo" def communicate( self, operation: str, *, device: torch.device, dtype: torch.dtype, tensor: torch.Tensor | None = None, shape: tuple[int, ...] | None = None, ) -> torch.Tensor | None: """Point-to-point pipeline communication between adjacent stages.""" comm_device = torch.device("cpu") if self.comm_on_cpu else device src = dest = None if operation == "recv_forward": if self.pp_is_first_stage: return None # Receive hidden states from the previous stage. src = self.pp_prev_rank # Shape is provided by the caller (derived from input_ids / inputs_embeds). tensor = torch.empty(shape, dtype=dtype, device=comm_device) elif operation == "send_forward": if self.pp_is_last_stage: return None # Send hidden states to the next stage. dest = self.pp_next_rank tensor = tensor.to(device=comm_device, dtype=dtype).contiguous() else: raise ValueError(f"Unsupported pipeline communication operation: {operation}") # Shared P2P: one isend/irecv with the adjacent rank. is_send = operation.startswith("send") peer_group_rank = dest if is_send else src peer_rank = dist.get_global_rank(self.pp_group, peer_group_rank) op = dist.P2POp(dist.isend if is_send else dist.irecv, tensor, peer_rank, group=self.pp_group) # Wait for the communication to complete. for req in dist.batch_isend_irecv([op]): req.wait() if comm_device.type == "cuda": torch.cuda.synchronize() return None if is_send else tensor.to(device=device, dtype=dtype) def layer_range_for_rank(self, rank: int, num_layers: int) -> tuple[int, int]: """ Return [start, end) owned by rank. The remainder goes to the first ranks, one layer each. Example with pp_size=3 and num_layers=32: rank 0 -> [0, 11) # layers 0-10 rank 1 -> [11, 22) # layers 11-21 rank 2 -> [22, 32) # layers 22-31 """ # TODO(3outeille): Balance stages by layer type or parameter bytes instead of splitting solely by layer count. layers_per_rank, remainder = divmod(num_layers, self.pp_size) start_layer = rank * layers_per_rank + min(rank, remainder) end_layer = start_layer + layers_per_rank + (rank < remainder) return start_layer, end_layer def find_rank_for_key(self, key: str, num_layers: int, base_model_prefix: str) -> int | None: """Return the PP rank that owns a checkpoint parameter key, or ``None`` if unknown.""" base_prefix = f"{base_model_prefix}." if key.startswith(f"{base_prefix}embed_tokens."): return 0 if key.startswith(f"{base_prefix}norm.") or key.startswith("lm_head."): return self.pp_size - 1 layers_prefix = f"{base_prefix}layers." if not key.startswith(layers_prefix): return None layer_idx = int(key.split(".")[2]) for rank in range(self.pp_size): start_layer, end_layer = self.layer_range_for_rank(rank, num_layers) if start_layer <= layer_idx < end_layer: return rank return None def broadcast_from_last( self, tensor: torch.Tensor | None, *, dtype: torch.dtype, device: torch.device, ) -> torch.Tensor: """Broadcast logits from the last PP rank to every rank. Only the last stage computes logits, but every rank must return them so `generate()` works without PP-specific changes.""" if self.pp_size <= 1: return tensor last_rank = dist.get_global_rank(self.pp_group, self.pp_size - 1) comm_device = torch.device("cpu") if self.comm_on_cpu else device # Logits are always (batch, seq_len, vocab_size). logits_ndim = 3 # last rank sends the logits vector shape; other ranks receive it to size their buffer. if self.pp_is_last_stage: assert tensor.ndim == logits_ndim, ( f"Expected logits with {logits_ndim} dims (batch, seq_len, vocab_size), got {tensor.ndim}" ) logits = tensor.to(device=comm_device, dtype=dtype).contiguous() shape_msg = torch.tensor(list(logits.shape), dtype=torch.long, device=comm_device) else: shape_msg = torch.empty(logits_ndim, dtype=torch.long, device=comm_device) dist.broadcast(shape_msg, src=last_rank, group=self.pp_group) # last rank sends the actual logits value; other ranks receive into the pre-allocated buffer. if not self.pp_is_last_stage: logits = torch.empty(tuple(shape_msg.tolist()), dtype=dtype, device=comm_device) dist.broadcast(logits, src=last_rank, group=self.pp_group) return logits.to(device=device, dtype=dtype) def apply_pipeline_parallelism(model: nn.Module, pp_mesh: torch.distributed.device_mesh.DeviceMesh) -> nn.Module: """Naive even split of `base_model.layers` across PP ranks.""" # TODO(3outeille): involves pp_plan to do the split ? stage = PipelineStage(pp_mesh) model._pp_stage = stage base_model = getattr(model, model.base_model_prefix) layers = base_model.layers num_layers = len(layers) start_layer, end_layer = stage.layer_range_for_rank(stage.pp_rank, num_layers) tied = getattr(model.config, "tie_word_embeddings", False) # When tied, keep embed_tokens on the last stage too so _finalize_model_loading in modeling_utils.py can tie lm_head locally. keep_embed_tokens = stage.pp_is_first_stage or (tied and stage.pp_is_last_stage) if not keep_embed_tokens: base_model.embed_tokens = PipelineIdentityLayer() for layer_idx in range(num_layers): if layer_idx < start_layer or layer_idx >= end_layer: layers[layer_idx] = PipelineIdentityLayer() if not stage.pp_is_last_stage: base_model.norm = PipelineIdentityLayer() model.lm_head = PipelineIdentityLayer() # let _finalize_model_loading know that we want to tie the lm_head only in the last rank if tied and not stage.pp_is_last_stage: model.all_tied_weights_keys = {} # TODO(3outeille): dispatch to different pipeline parallelism schedules (gpipe, 1f1b, etc.) if not getattr(model, "_pp_forward_wrapped", False): original_forward = model.forward forward_signature = inspect.signature(original_forward) @wraps(original_forward) def pp_naive_forward(*args, **kwargs): return pipeline_parallel_naive_forward(model, original_forward, forward_signature, *args, **kwargs) # @wraps(original_forward) # def pp_gpipe_forward(*args, **kwargs): # return pipeline_parallel_gpipe_forward(model, original_forward, *args, **kwargs) model.forward = pp_naive_forward model._pp_forward_wrapped = True return model def _hidden_states_shape(fwd_kwargs: dict, hidden_size: int) -> tuple[int, ...]: if (inputs_embeds := fwd_kwargs.get("inputs_embeds")) is not None: return tuple(inputs_embeds.shape) if (input_ids := fwd_kwargs.get("input_ids")) is not None: return (*input_ids.shape[:2], hidden_size) raise ValueError("Cannot determine hidden states shape for pipeline recv_forward") def _feed_hidden_states_as_input_embeds(fwd_kwargs: dict, hidden_states: torch.Tensor) -> dict: """Stage 0 uses input_ids; later stages use received hidden states as inputs_embeds.""" fwd_kwargs.pop("input_ids", None) fwd_kwargs["inputs_embeds"] = hidden_states return fwd_kwargs def pipeline_parallel_naive_forward( model: nn.Module, original_forward, forward_signature: inspect.Signature, *args, **kwargs ): stage = model._pp_stage device = next(model.parameters()).device dtype = next(model.parameters()).dtype # Bind the forward function to the arguments so that we can easily access them regardless of the number of arguments or the way we called the forward function. caller_kwargs = _bind_forward_kwargs(forward_signature, args, kwargs) fwd_kwargs = caller_kwargs # Non-first stages: recv activations from prev stage and use them as inputs_embeds. if not stage.pp_is_first_stage: shape = _hidden_states_shape(caller_kwargs, model.config.hidden_size) hidden_states = stage.communicate("recv_forward", device=device, dtype=dtype, shape=shape) fwd_kwargs = _feed_hidden_states_as_input_embeds(caller_kwargs, hidden_states) if stage.pp_is_last_stage: # Last stage: compute the logits. outputs = original_forward(**fwd_kwargs) logits, cache = outputs.logits, outputs.cache else: # Non-last stages: Compute and send the activations to the next stage. # We need the base models because hidden_states is getting slices in the causal_lm models + we want last_hidden_state base_model = getattr(model, model.base_model_prefix) base_kwargs = {k: v for k, v in fwd_kwargs.items() if k not in {"labels", "logits_to_keep"}} base_outputs = base_model(**base_kwargs) stage.communicate("send_forward", device=device, dtype=dtype, tensor=base_outputs.last_hidden_state) logits, cache = None, base_outputs.cache # Only the last stage computed the logits, so broadcast them to every rank (they will be waiting) # TODO(3outeille): very naive implementation to make it work seamless with generate(). # In long term, we want only the last stage to hold the logits not every stages that will require changes in generate() directly logits = stage.broadcast_from_last(logits, dtype=dtype, device=device) return CausalLMOutputWithPast(logits=logits, cache=cache) - src/transformers/distributed/sharding_utils.py (18868 bytes)
# Copyright 2026 The HuggingFace Team. All rights reserved. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. from __future__ import annotations import math from typing import TYPE_CHECKING from ..utils import is_torch_available, is_torch_distributed_available if TYPE_CHECKING: import torch from torch.distributed.tensor import DTensor if is_torch_available(): import torch if is_torch_distributed_available(): import torch.distributed as dist from torch.distributed.tensor import DTensor from torch.distributed.tensor._utils import compute_local_shape_and_global_offset from torch.distributed.tensor.placement_types import Shard def _local_shard_size_and_offset(size_on_dim: int, num_chunks: int, rank: int) -> tuple[int, int]: """Local shard size and offset for this rank, torch renamed this API across 2.6 to 2.10+.""" # torch >= 2.10: public, non-underscored name. if hasattr(Shard, "local_shard_size_and_offset"): return Shard.local_shard_size_and_offset(size_on_dim, num_chunks, rank) # torch 2.8 / 2.9: same signature, underscored name. if hasattr(Shard, "_local_shard_size_and_offset"): return Shard._local_shard_size_and_offset(size_on_dim, num_chunks, rank) # torch 2.6 / 2.7: only `_local_shard_size_on_dim`, which returns (size, offset) when asked to. return Shard._local_shard_size_on_dim(size_on_dim, num_chunks, rank, return_offset=True) class DtensorShardOperation: """Shard-on-read: slice a full disk tensor down to this rank's local DTensor shard, for any combination of placements on a 1-D or n-D mesh. It's on read because instructions are made so the cpu only fetches on disk the parts we want. Placements primer ----------------- Each mesh dim carries one placement describing how it slices the tensor: | Placement | Local data on each rank of the mesh dim | |--------------------------|-------------------------------------------------| | Replicate | full tensor (no slicing) | | Shard(d) | contiguous chunk of dim d (rows r*c .. (r+1)*c) | | _StridedShard(d, sf=N) | one chunk from each of N groups along dim d, | | | concatenated together (interleaved layout) | Different scenarios of different placements ------------------------------------------ Placement tuples are ordered outermost-first; for a 2-D (fsdp, tp) mesh the tuple is (fsdp_placement, tp_placement). | Scenario | Placements | |---------------------------------------------------|---------------------------------------------| | TP-only, non-fused (e.g. q_proj/k_proj/v_proj) | [Shard(d)] | | TP-only, fused gate/up | [_StridedShard(d, sf=2)] | | TP + FSDP, same tensor dim (contiguous TP case) | [Shard(d), Shard(d)] | | TP + FSDP, same tensor dim (fused/interleaved TP) | [_StridedShard(d, sf=tp_size), Shard(d)] | | TP + FSDP, different dims | [Shard(d1), Shard(d2)] | Loading (this class) -------------------- During from_pretrained, each rank looks up every tensor key, but does not load the weight bytes yet (safetensors get_slice). When a weight is needed, shard_tensor indexes that slice to read only this rank's local shard through Dtensor logic which provides placement logics Depending on how the checkpoint was saved, the weight loader either gives us one big tensor or many small ones: 1. One stacked tensor: the checkpoint has one key with all experts together, shaped [num_experts, in, out]. The weight loader passes it straight through; we slice out this rank's piece. 2. One tensor per expert: the checkpoint has a separate key for each expert (expert 0, expert 1, …), each shaped [in, out]. The weight loader feeds them in one at a time. If this rank doesn't own a given expert, we skip it. Later, the converter (e.g. `MergeModulelist`) stacks the owned experts we kept into the rank's local shard """ def __init__(self, param: DTensor): self.device_mesh = param.device_mesh self.placements = tuple(param.placements) self.param_ndim = param.ndim local_shape, offsets = compute_local_shape_and_global_offset(param.shape, self.device_mesh, self.placements) # Axis-0 range owned by this rank (used to filter per-expert pieces) # [_axis0_offset, _axis0_offset + _axis0_local_size) self._axis0_offset = offsets[0] self._axis0_local_size = local_shape[0] self._axis0_size = param.shape[0] def shard_tensor( self, source: torch.Tensor, tensor_idx: int | None = None, device=None, dtype=None ) -> torch.Tensor | None: """Return this rank's local shard of a checkpoint tensor. Two layouts (example param shape [N, in, out]): - a stacked [N, in, out] tensor: slice every sharded dim (including axis 0). - one [in, out] slice of axis 0 (e.g. one expert), the `tensor_idx`-th of the list a converter collects: return None if this rank does not own that slice, else slice inner dims only. Surviving slices are stacked by the converter into this rank's local [n_local, in, out] shard. A list can fill several params of the same axis-0 size back to back (e.g. text then vision experts as in ernie), so `tensor_idx` wraps around it. """ source_shape = list(source.shape) if isinstance(source, torch.Tensor) else source.get_shape() if tensor_idx is not None and len(source_shape) != self.param_ndim - 1: tensor_idx = None # not a slice of axis 0, the list holds whole tensors dim_placements = [ (mesh_dim, placement) for mesh_dim, placement in enumerate(self.placements) if hasattr(placement, "dim") ] # Dense path if tensor_idx is None: if not dim_placements: return source[...].to(device=device, dtype=dtype) # Determine for each tensor dimension, which type of sharding operations to apply (_StridedShard or Shard) and which rank to apply it to. # i.e: dim 0 -> [ Strided(rank0, size=2, sf=2), Shard(rank1, size=2) ] # i.e: dim 1 -> [ Shard(rank0, size=2)] planned_ops_by_dim = [[] for _ in source_shape] for mesh_dim, placement in dim_placements: sub_mesh = self._get_sub_mesh(mesh_dim) rank, world_size = sub_mesh.get_local_rank(), sub_mesh.size() dim_idx = self._normalize_param_dim(placement.dim) planned_ops_by_dim[dim_idx].append((placement, rank, world_size)) # prepare the slices to fetch on disk for each tensor dimension. intervals_by_dim = [[(0, size)] for size in source_shape] for dim_idx, planned_ops in enumerate(planned_ops_by_dim): intervals = intervals_by_dim[dim_idx] for placement, rank, world_size in planned_ops: if placement.is_shard(): intervals = self._compute_contiguous_slice(intervals, rank, world_size) else: intervals = self._compute_strided_slice(intervals, rank, world_size, placement.split_factor) intervals_by_dim[dim_idx] = intervals has_strided_shard = any(not placement.is_shard() for _, placement in dim_placements) # finally fetch from the disk only the slices # finally fetch from the disk only the slices if has_strided_shard: # Multi-interval dim: read each piece separately, then concatenate. return self._slice_and_cat(source, intervals_by_dim, device, dtype) else: slice_parts = [] for intervals in intervals_by_dim: start, end = intervals[0] if len(intervals) > 0 else (0, 0) slice_parts.append(slice(start, end)) return source[tuple(slice_parts)].to(device=device, dtype=dtype) # MoE path # tensor_idx identifies the axis-0 piece in param space (not in source.shape). normalized_dim_placements = [ (mesh_dim, placement, self._normalize_param_dim(placement.dim)) for mesh_dim, placement in dim_placements ] # if this rank owns expert `tensor_idx` along axis 0, we need to slice the inner dimensions, else we drop it has_axis0_shard = any(param_dim == 0 for _, _, param_dim in normalized_dim_placements) tensor_idx %= self._axis0_size owns_tensor_idx = self._axis0_offset <= tensor_idx < self._axis0_offset + self._axis0_local_size if has_axis0_shard and not owns_tensor_idx: return None # `param_dim` indexes the full parameter layout [N, in, out] (expert axis first). # In per-expert loading, leading axis is absent ([in, out]). # Therefore we need to shift the dimensions by 1 to the left so that sharding operations can be applied to the inner dimensions. # param [N, in, out] ↔ source [in, out] # Shard(param dim 1) → slice source dim 0 # Shard(param dim 2) → slice source dim 1 planned_ops_by_source_dim = [[] for _ in source_shape] for mesh_dim, _, param_dim in normalized_dim_placements: if param_dim > 0: source_dim = param_dim - 1 sub_mesh = self._get_sub_mesh(mesh_dim) rank, world_size = sub_mesh.get_local_rank(), sub_mesh.size() planned_ops_by_source_dim[source_dim].append((rank, world_size)) intervals_by_source_dim = [[(0, size)] for size in source_shape] for source_dim, planned_ops in enumerate(planned_ops_by_source_dim): intervals = intervals_by_source_dim[source_dim] for rank, world_size in planned_ops: intervals = self._compute_contiguous_slice(intervals, rank, world_size) intervals_by_source_dim[source_dim] = intervals slice_parts = [] for intervals in intervals_by_source_dim: start, end = intervals[0] if intervals else (0, 0) slice_parts.append(slice(start, end)) return source[tuple(slice_parts)].to(device=device, dtype=dtype) def _compute_strided_slice( self, intervals: list[tuple[int, int]], rank: int, world_size: int, split_factor: int ) -> list[tuple[int, int]]: local_intervals = [] for interval_start, interval_end in intervals: # For each interval, we break it into split_factor consecutive groups. group_width = math.ceil((interval_end - interval_start) / split_factor) for group_idx in range(split_factor): # Compute this group's boundaries in source coordinates. group_start = interval_start + group_idx * group_width group_end = min(group_start + group_width, interval_end) group_len = group_end - group_start if group_len > 0: # Inside each group, we do a normal contiguous shard across world_size ranks. local_shard_size, local_shard_offset = _local_shard_size_and_offset(group_len, world_size, rank) if local_shard_size > 0: # Convert group-local offset back to global source coordinates. shard_start = group_start + local_shard_offset local_intervals.append((shard_start, shard_start + local_shard_size)) return local_intervals def _compute_contiguous_slice( self, intervals: list[tuple[int, int]], rank: int, world_size: int ) -> list[tuple[int, int]]: # We apply contiguous sharding to a list of intervals. Two cases: # 1. The intervals is a single interval -> we return the local interval for this rank. # 2. The intervals are disjoint (i.e: 2D TP + FSDP, same tensor dim (fused/interleaved TP = [StridedShard(d, sf=tp_size), Shard(d)]) # -> find overlapping local intervals across the disjoint intervals. # Compute rank's local length and start offset when the total length is partitioned across world_size ranks. flat_total_len = sum(end - start for start, end in intervals) local_flat_len, local_flat_start = _local_shard_size_and_offset(flat_total_len, world_size, rank) local_flat_end = local_flat_start + local_flat_len if local_flat_len == 0: return [] # Single-interval case. if len(intervals) == 1: source_start, _ = intervals[0] return [(source_start + local_flat_start, source_start + local_flat_end)] # Disjoint intervals case. # 1) Build flat mapping from source intervals. # Example: intervals=[(0,3), (10,15)] -> flat segments: (0,3,0) and (3,8,10) # meaning flat [0,3) maps to source [0,3), and flat [3,8) maps to source [10,15). flat_segments = [] # (interval_flat_start, interval_flat_end, source_start) idx = 0 for source_start, source_end in intervals: interval_len = source_end - source_start if interval_len > 0: flat_segments.append((idx, idx + interval_len, source_start)) idx += interval_len # 2) Intersect this rank's flat span with each flat segment, then map overlap # back to source coordinates. # Example: local flat span [2,6) with segments above gives: # overlap with [0,3) => [2,3) -> source (2,3) # overlap with [3,8) => [3,6) -> source (10,13) # result: local_intervals = [(2,3), (10,13)] local_intervals = [] for interval_flat_start, interval_flat_end, source_start in flat_segments: overlap_flat_start = max(interval_flat_start, local_flat_start) overlap_flat_end = min(interval_flat_end, local_flat_end) if overlap_flat_start < overlap_flat_end: source_overlap_start = source_start + (overlap_flat_start - interval_flat_start) source_overlap_ - src/transformers/distributed/tensor_parallel.py (32203 bytes)
# Copyright 2026 The HuggingFace Team. All rights reserved. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. from __future__ import annotations import contextlib import re import warnings from typing import TYPE_CHECKING from ..utils import logging from ..utils.import_utils import is_torch_available, is_torch_distributed_available from .parallel_utils import ( ALL_PARALLEL_STYLES, ParallelInterface, # noqa: F401 _use_local_dtensor_params, replace_layer_number_by_wildcard, ) from .utils import _get_torch_distributed_rank, _warning_once_on_rank_zero if TYPE_CHECKING: from torch import nn from torch.distributed.device_mesh import DeviceMesh from ..modeling_utils import PreTrainedModel from .configuration_utils import DistributedConfig logger = logging.get_logger(__name__) if is_torch_available(): import torch if is_torch_distributed_available(): import torch.distributed as dist from torch.distributed.tensor import DTensor, Partial, Replicate, Shard, distribute_tensor from torch.distributed.tensor.placement_types import _StridedShard from .parallel_utils import _AllReduceBackward, _AllReduceForward def verify_tp_plan(expected_keys: list[str], tp_plan: dict[str, str] | None): """ Verify the TP plan of the model, log a warning if the layers that were not sharded and the rules that were not applied. """ if tp_plan is None: return generic_keys = {replace_layer_number_by_wildcard(key) for key in expected_keys} unsharded_layers = set(generic_keys) unused_rules = tp_plan.copy() for key in generic_keys: param_name = key.rsplit(".", 1)[0] if "." in key else key generic_param_name = re.sub(r"\d+", "*", param_name) if generic_param_name in tp_plan: unused_rules.pop(generic_param_name, None) unsharded_layers.discard(key) elif "." in generic_param_name and (parent_param_name := generic_param_name.rsplit(".", 1)[0]) in tp_plan: unused_rules.pop(parent_param_name, None) unsharded_layers.discard(key) if len(unused_rules) > 0: logger.warning(f"The following TP rules were not applied on any of the layers: {unused_rules}") if len(unsharded_layers) > 0: logger.warning(f"The following layers were not sharded: {', '.join(unsharded_layers)}") def get_kv_heads_per_rank(tp_plan: dict[str, str], num_heads: int | list[int], tp_size: int) -> int | list[int] | None: """Returns the KV heads each rank holds under `tp_plan`, or `None` if they cannot be evenly sharded.""" # Projections whose output holds the attention keys, fused or not key_projections = ("k_proj", "qkv_proj", "query_key_value") if not any(key.rsplit(".", 1)[-1] in key_projections and style == "colwise" for key, style in tp_plan.items()): # Unsharded heads: no plan, a gathered output, or experts only return num_heads layer_heads = [num_heads] if isinstance(num_heads, int) else num_heads if any(heads % tp_size for heads in layer_heads): return None # A scalar must stay scalar: `early_initialization` broadcasts it, but wants one entry per layer in a list return num_heads // tp_size if isinstance(num_heads, int) else [h // tp_size for h in layer_heads] class TensorParallelLayer: def should_use_local_tensors(self, module): """Whether this module's forward requires local inputs and parameters.""" return False def validate_param(self, module, param, mesh, parameter_name=None): """Validate a parameter before applying this TP style.""" pass def shard_param(self, module, param, mesh): """Wrap ONE parameter as a DTensor placeholder. Default: no-op.""" pass def transform_inputs_pre_forward(self, module, args, kwargs, mesh): return args, kwargs def context_around_forward(self, module, mesh): if self.should_use_local_tensors(module): return _use_local_dtensor_params(module) return contextlib.nullcontext() def transform_output_post_forward(self, module, output, mesh): return output @staticmethod def _from_local_last_dim_shard(local, mesh, size): """Create a `DTensor` for a tensor sharded across its last dimension. The local shard can have a different size from the other ranks when `size` is not divisible by the TP size.""" shape = (*local.shape[:-1], size) return DTensor.from_local( local, mesh, [Shard(-1)], run_check=False, shape=shape, stride=torch.empty(shape, device="meta").stride() ) def install_forward(self, module, mesh): """Install pre / around / post transforms by replacing module.forward.""" original_forward = module.forward def tp_forward(*args, **kwargs): args, kwargs = self.transform_inputs_pre_forward(module, args, kwargs, mesh) with self.context_around_forward(module, mesh): output = original_forward(*args, **kwargs) return self.transform_output_post_forward(module, output, mesh) module.forward = tp_forward return module class ColwiseParallel(TensorParallelLayer): """Column-wise for linear layers: weight & bias → Shard(0); input replicated, output Shard(-1). The output width (`out_features`) need not divide by the TP size: the last rank then holds a shorter slice.""" def __init__(self, *, input_layouts=None, output_layouts=None, use_local_output: bool = True): self.input_layouts = input_layouts or Replicate() self.output_layouts = output_layouts if output_layouts is not None else Shard(-1) self.use_local_output = use_local_output def should_use_local_tensors(self, module): use_local_quantized_path = getattr(module, "_hf_quantized_needs_local_tp", False) uses_local_inference_kernel = isinstance(module, torch.nn.Linear) and not torch.is_grad_enabled() return use_local_quantized_path or uses_local_inference_kernel def install_forward(self, module, mesh): if not hasattr(module, "out_features"): raise ValueError(f"{type(module).__name__} has no `out_features`: colwise shards linear layers.") return super().install_forward(module, mesh) def shard_param(self, module, param, mesh): meta = module._parameters.get(param) if meta is None: return placement = Shard(meta.ndim - 2) module._parameters[param] = torch.nn.Parameter( distribute_tensor(meta, mesh, [placement], src_data_rank=None), requires_grad=meta.requires_grad, ) def transform_inputs_pre_forward(self, module, args, kwargs, mesh): x = args[0] is_local_input = not isinstance(x, DTensor) # regular Linear inference fast path: the input is already replicated, the # local weight owns a shard of the output features, and no backward is needed. if is_local_input and isinstance(module, torch.nn.Linear) and not torch.is_grad_enabled(): return args, kwargs # Plain-local quantized fast path. The input is already replicated, so avoid # wrapping it as a DTensor. If its gradient is needed, synchronize the partial # input-gradient contributions explicitly. if is_local_input and getattr(module, "_hf_quantized_needs_local_tp", False): process_group = mesh.get_group() if mesh.ndim == 1 else mesh.get_group("tp") x = _AllReduceBackward.apply(x, process_group) return (x,) + args[1:], kwargs # DTensor path: handle regular training and layout redistribution. if is_local_input: x = DTensor.from_local(x, mesh, [self.input_layouts], run_check=False) if x.placements != (Replicate(),): x = x.redistribute(placements=[Replicate()]) if self.should_use_local_tensors(module): x = x.to_local(grad_placements=[Partial()]) return (x,) + args[1:], kwargs def transform_output_post_forward(self, module, output, mesh): # The local forward produced this rank's shard of the output features (last dim). output_is_local_shard = ( not isinstance(output, DTensor) and isinstance(self.output_layouts, Shard) and self.output_layouts.dim in (-1, output.dim() - 1) ) if self.should_use_local_tensors(module) and self.use_local_output and output_is_local_shard: return output if not isinstance(output, DTensor): # `out_features`, not the weight's shape: a packed or quantized weight doesn't show the full width output = self._from_local_last_dim_shard(output, mesh, module.out_features) if output.placements != (self.output_layouts,): output = output.redistribute(placements=[self.output_layouts]) return output.to_local() if self.use_local_output else output class RowwiseParallel(TensorParallelLayer): """Row-wise for linear layers: weight → Shard(1), bias → Replicate; input sharded on the last dim. The module forward produces a Partial output which the boundary redistribute reduces to output_layouts (Replicate → allreduce, Shard(1) → reduce-scatter). The input width (`in_features`) need not divide by the TP size. """ def __init__(self, *, input_layouts=None, output_layouts=None, use_local_output: bool = True): self.input_layouts = input_layouts or Shard(-1) self.output_layouts = output_layouts or Replicate() self.use_local_output = use_local_output def should_use_local_tensors(self, module): use_local_quantized_path = getattr(module, "_hf_quantized_needs_local_tp", False) uses_local_inference_kernel = isinstance(module, torch.nn.Linear) and not torch.is_grad_enabled() return use_local_quantized_path or uses_local_inference_kernel def shard_param(self, module, param, mesh): meta = module._parameters.get(param) if meta is None: return # bias is replicated (added after the row-reduce); weight shards on input dim (-1) placement = Replicate() if param == "bias" else Shard(-1) module._parameters[param] = torch.nn.Parameter( distribute_tensor(meta, mesh, [placement], src_data_rank=None), requires_grad=meta.requires_grad, ) def install_forward(self, module, mesh): if not hasattr(module, "in_features"): raise ValueError(f"{type(module).__name__} has no `in_features`: rowwise shards linear layers.") return super().install_forward(module, mesh) def transform_inputs_pre_forward(self, module, args, kwargs, mesh): desired = Shard(-1) x = args[0] # A local kernel can use a plain input when the previous layer already split it. # avoid a redundant Tensor -> DTensor -> Tensor round trip. input_has_desired_layout = self.input_layouts == desired if self.should_use_local_tensors(module) and input_has_desired_layout and not isinstance(x, DTensor): return args, kwargs if not isinstance(x, DTensor): if isinstance(self.input_layouts, Shard): # the input is this rank's slice of the module's declared `in_features` x = self._from_local_last_dim_shard(x, mesh, module.in_features) else: x = DTensor.from_local(x, mesh, [self.input_layouts], run_check=False) if x.placements != (desired,): x = x.redistribute(placements=[desired]) if self.should_use_local_tensors(module): x = x.to_local() return (x,) + args[1:], kwargs @contextlib.contextmanager def context_around_forward(self, module, mesh): if not self.should_use_local_tensors(module): yield else: # A rowwise local forward must produce only its partial matmul. If we don't hide # the bias, we will be adding the bias x world_size times which is not correct. # We should add it once after the all_reduce (redistribute). bias = module._parameters.get("bias") if bias is not None: module._parameters["bias"] = None try: with _use_local_dtensor_params(module): yield finally: if bias is not None: module._parameters["bias"] = bias def transform_output_post_forward(self, module, output, mesh): use_local_inference_path = ( isinstance(module, torch.nn.Linear) and not isinstance(output, DTensor) and isinstance(self.output_layouts, Replicate) and not output.requires_grad ) if use_local_inference_path: process_group = mesh.get_group() if mesh.ndim == 1 else mesh.get_group("tp") dist.all_reduce(output, group=process_group) if (bias := module._parameters.get("bias")) is not None: output = output + (bias.to_local() if isinstance(bias, DTensor) else bias) else: # Dtensor tracks whether the result is partial, sharded, or replicated. if not isinstance(output, DTensor): output = DTensor.from_local(output, mesh, [Partial()], run_check=False) if output.placements != (self.output_layouts,): output = output.redistribute(placements=[self.output_layouts]) if self.should_use_local_tensors(module) and (bias := module._parameters.get("bias")) is not None: output = output + bias if self.use_local_output: output = output.to_local() return output class EmbeddingParallel(TensorParallelLayer): """Embedding with the token ids replicated on every rank. `shard_dim=0` is vocab-parallel: each rank holds a slice of the vocabulary, and the ranks' partial lookups are summed. `shard_dim=1` shards the hidden dim: each rank looks up a slice of every vector, and the slices are gathered. The sharded dim need not divide by the TP size.""" def __init__(self, *, shard_dim: int = 0, use_local_output: bool = True): self.shard_dim = shard_dim self.use_local_output = use_local_output def should_use_local_tensors(self, module): return getattr(module, "_hf_quantized_needs_local_ - src/transformers/distributed/utils.py (14826 bytes)
# Copyright 2026 The HuggingFace Team. All rights reserved. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. from __future__ import annotations import math import os import warnings from datetime import timedelta from typing import TYPE_CHECKING, TypeGuard from ..utils import ( is_torch_available, is_torch_distributed_available, is_torch_greater_or_equal, logging, ) logger = logging.get_logger(__name__) if TYPE_CHECKING: from torch.distributed.device_mesh import DeviceMesh from torch.distributed.tensor import DTensor from .configuration_utils import DistributedConfig if is_torch_available(): import torch def _check_distributed_checkpointing_available() -> bool: # `torch.distributed.checkpoint.hf_storage` (HuggingFaceStorageReader/Writer) first ships in torch 2.8. return is_torch_distributed_available() and is_torch_greater_or_equal("2.8") def _is_torch_distributed_initialized() -> bool: if not is_torch_distributed_available(): return False return torch.distributed.is_initialized() def is_dtensor(obj: object) -> TypeGuard[DTensor]: if not is_torch_distributed_available(): return False from torch.distributed.tensor import DTensor return isinstance(obj, DTensor) def _get_torch_distributed_rank() -> int: if not _is_torch_distributed_initialized(): return 0 return torch.distributed.get_rank() def _warning_once_on_rank_zero(message: str) -> None: """Log a warning once per process, on rank 0 only.""" if _get_torch_distributed_rank() == 0: logger.warning_once(message) def _get_torch_distributed_world_size() -> int: if not _is_torch_distributed_initialized(): return 1 return torch.distributed.get_world_size() def is_local_dist_rank_0() -> bool: return _is_torch_distributed_initialized() and int(os.environ.get("LOCAL_RANK", "-1")) == 0 DISTRIBUTED_BACKEND_MAP = { "cuda": "nccl", "cpu": "gloo", "xpu": "xccl", "hpu": "hccl", "neuron": "neuron", "tpu": "tpu_dist", } def get_distributed_backend(device_type: str) -> str | None: """Return the `torch.distributed` backend for `device_type`, or `None` if it has no dedicated one.""" dist_backend = DISTRIBUTED_BACKEND_MAP.get(device_type) if dist_backend is None: raise ValueError(f"No distributed backend found for device type '{device_type}'") return dist_backend def _bind_local_device(device_type: str, local_rank: int) -> torch.device: """Bind this process to its accelerator and return the device (no binding on CPU).""" if device_type == "cpu": return torch.device(device_type) # torch_tpu gives each process a single chip, always at index 0 device_index = torch.accelerator.current_device_index() if device_type == "tpu" else local_rank torch.accelerator.set_device_index(device_index) return torch.device(device_type, device_index) def _ensure_torch_distributed(device_type: str | None = None): """Initialize torch.distributed if not already initialized. If `device_type` is not given, it is detected from the current accelerator. """ if not torch.distributed.is_initialized(): if device_type is None: device_type = torch._C._get_accelerator().type if device_type == "mps": logger.warning_once( "PyTorch's built-in DeviceMesh/DTensor stack does not support an MPS mesh. Falling back to CPU." ) device_type = "cpu" try: rank = int(os.environ["RANK"]) local_rank = int(os.environ["LOCAL_RANK"]) world_size = int(os.environ["WORLD_SIZE"]) backend = get_distributed_backend(device_type) # Bind the accelerator before init so the process group is created with a # device_id, otherwise collectives like barrier() warn (and may spin up an # extra NCCL comm) about the missing device binding. device = _bind_local_device(device_type, local_rank) # Only pass device_id to backends that support communicator splitting, for now we limit # it to NCCL: a bound device makes DeviceMesh create its subgroups with `split_group`, # which fails on backends without split support (e.g. Neuron). device_id = device if backend == "nccl" else None torch.distributed.init_process_group( backend=backend, rank=rank, world_size=world_size, device_id=device_id, # Sharded loading takes tens of minutes with high rank skew; the default 10-minute watchdog is too short timeout=timedelta(hours=2), ) except Exception as e: raise OSError( "We tried to initialize torch.distributed for you, but it failed. Make " "sure you init torch distributed in your script to use distributed training." ) from e def _distributed_barrier(): """Barrier bound to the current accelerator device. Passing `device_ids` is required when the process group was initialized without a `device_id`; with it, the call is a no-op compared to plain `barrier()`. Safe to call when torch.distributed has not been initialized — returns immediately. """ if not _is_torch_distributed_initialized(): return device_type = torch._C._get_accelerator().type if device_type != "cpu": torch.distributed.barrier(device_ids=[torch.accelerator.current_device_index()]) else: torch.distributed.barrier() class TransformersDeviceMesh: """ Holds the device meshes used by a model. dense layers and experts are sharded differently, so they need different views of the same ranks. dense : (pp, dp_replicate, fsdp, tp) attention, dense MLPs, embeddings, lm_heads expert : (pp, dp_replicate, efsdp, ep) experts Both views cover the same world, so pp * dp_replicate * fsdp * tp == pp * dp_replicate * efsdp * ep. efsdp is not something you pick, it is whatever is left once ep is fixed. The relationship is as follow: efsdp = fsdp * tp / ep. It is the FSDP axis for expert weights same role `fsdp` plays for the dense params. There is no etp (expert tensor parallel) axis: with EP on, experts are never tensor-sharded. (for now) For experts, EP plays the role TP plays for dense layers (weights stay sharded during compute), and efsdp plays the role of fsdp (weights all-gathered before compute). If one were ever added, the relationship would become pp * efsdp * ep * etp == pp * fsdp * tp and efsdp would shrink by etp (efsdp = fsdp * tp / (ep * etp)) When ep_size == tp_size, efsdp == fsdp (given the relationship efsdp = fsdp * tp / ep), thus same axis, same size and same groups of ranks. In that case experts could reuse the dense mesh's fsdp axis. When ep_size != tp_size, efsdp != fsdp, so we can't reuse the same mesh as they don't have the same groups of ranks This explains why experts need their own efsdp axis. Regarding ep value, one can decide to default the value to node width (8 on most machines) so that all-to-all never leaves the node. That has several implications on efsdp value given your setup: - On a single node, ep == fsdp * tp thus efsdp = 1, the axis does nothing. - On several nodes, we still keep ep at node width, since all-to-all across nodes is expensive. However, each node then holds a full copy of the expert group and efsdp is the number of copies, which is where FSDP happens for the experts i.e: 2 nodes x 8 GPUs -> efsdp = 16 / 8 = 2, one EP group per node, two copies, sharded over efsdp. """ def __init__(self, dense_mesh: DeviceMesh, expert_mesh: DeviceMesh): self._dense_mesh = dense_mesh self._expert_mesh = expert_mesh def get_mesh(self, dims: str | tuple[str, ...]) -> DeviceMesh: """Select expert axes for `ep`/`efsdp`, otherwise dense axes; DeviceMesh handles slicing.""" dims = (dims,) if isinstance(dims, str) else dims mesh = self._expert_mesh if "ep" in dims or "efsdp" in dims else self._dense_mesh return mesh[dims] # Retained for the legacy transformers.integrations.tensor_parallel API. def initialize_tensor_parallelism( tp_plan: str | dict[str, str] | None, tp_size: int | None = None, device_mesh=None, device_map=None ): r""" Sets up the device mesh and initialized the backend for tensor parallelism. This function is called when the model is loaded and the TP plan is set to 'auto'. """ warnings.warn( "`initialize_tensor_parallelism` is deprecated and will be removed in a future release. " "Use `initialize_distributed_mesh` with a `DistributedConfig` instead.", FutureWarning, stacklevel=2, ) if tp_size is not None and tp_plan is None: raise ValueError("tp_plan has to be set when tp_size is passed.") if tp_plan is not None and device_map is not None: raise ValueError("`tp_plan` and `device_map` are mutually exclusive. Choose either one for parallelization.") if device_mesh is None: # Detect the accelerator on the machine. If no accelerator is available, it returns CPU. device_type = torch._C._get_accelerator().type if device_type == "mps": logger.warning_once( "PyTorch's built-in DeviceMesh/DTensor stack does not support an MPS mesh. Falling back to CPU." ) device_type = "cpu" if device_type != "cpu": tp_device = _bind_local_device(device_type, int(os.environ.get("LOCAL_RANK", 0))) device_map = tp_device else: tp_device = torch.device(device_type) device_map = device_type or {} device_mesh = torch.distributed.init_device_mesh(tp_device.type, (tp_size,)) else: if device_mesh.ndim > 1: if "tp" not in device_mesh.mesh_dim_names: raise ValueError( "When using `tp_plan` and n-d `device_mesh`, it must contain a 'tp' dimension. " "Please provide a valid `device_mesh`." ) device_mesh = device_mesh["tp"] device_map = _bind_local_device(device_mesh.device_type, int(os.environ.get("LOCAL_RANK", 0))) return device_map, device_mesh def initialize_fully_sharded_data_parallelism(distributed_config: DistributedConfig): warnings.warn( "`initialize_fully_sharded_data_parallelism` is deprecated and will be removed in a future release. " "Use `initialize_distributed_mesh` with a `DistributedConfig` instead.", FutureWarning, stacklevel=2, ) # `fully_shard` itself only needs torch>=2.6, but distributed checkpoint save/load # (DCP + HuggingFaceStorageWriter) needs 2.8, so that is the effective requirement. if distributed_config.fsdp_size > 1 and not is_torch_greater_or_equal("2.8"): raise OSError("FSDP2 requires `torch>=2.8` (distributed checkpoint save/load).") device_type = torch._C._get_accelerator().type device_map = _bind_local_device(device_type, int(os.environ.get("LOCAL_RANK", 0))) fsdp_size = distributed_config.fsdp_size dims, names = [], [] if fsdp_size > 1: dims.append(fsdp_size) names.append("fsdp") # Build the N-dimensional device mesh mesh = torch.distributed.init_device_mesh(device_type, tuple(dims), mesh_dim_names=tuple(names)) # If N > 1, create a flattened sub-mesh so all-reduces across the world mesh ae done in one collective if len(dims) > 1: mesh._flatten("_".join(names)) return device_map, mesh def initialize_distributed_mesh( distributed_config: DistributedConfig, ) -> tuple[torch.device | None, TransformersDeviceMesh | None]: """Create a device mesh containing every configured parallel dimension.""" mesh_shape = ( distributed_config.pp_size, distributed_config.dp_replicate_size, distributed_config.fsdp_size, distributed_config.tp_size, ) if mesh_shape == (1, 1, 1, 1): return None, None device_type = torch._C._get_accelerator().type if distributed_config.tp_size > 1 and device_type == "mps": raise RuntimeError("Tensor parallelism is not supported on MPS devices.") _ensure_torch_distributed(device_type) world_size = torch.distributed.get_world_size() expected_world_size = math.prod(mesh_shape) if expected_world_size != world_size: raise RuntimeError( f"The parallel mesh requires {expected_world_size} processes, but world_size is {world_size}." ) device_map = _bind_local_device(device_type, int(os.environ.get("LOCAL_RANK", 0))) dense_mesh = torch.distributed.init_device_mesh( device_type, mesh_shape, mesh_dim_names=("pp", "dp_replicate", "fsdp", "tp"), ) expert_mesh = torch.distributed.init_device_mesh( device_type, ( distributed_config.pp_size, distributed_config.dp_replicate_size, distributed_config.efsdp_size, distributed_config.ep_size, ), mesh_dim_names=("pp", "dp_replicate", "efsdp", "ep"), ) return device_map, TransformersDeviceMesh(dense_mesh, expert_mesh) def gather_full_state_dict(model) -> dict[str, torch.Tensor]: """Gather FSDP-sharded params to full plain CPU tensors. Only rank 0 accumulates the result; other ranks return ``{}``. """ if not _check_distributed_checkpointing_available(): raise OSError("Distributed checkpointing requires `torch>=2.8` with `torch.distributed` available.") # Import here because otherwise it emits a warning every time it's imported on some hardware - this keeps the warning from # being emitted if the function is not used from torch.distributed.checkpoint.state_dict import StateDictOptions, get_model_state_dict options = StateDictOptions(full_state_dict=True, cpu_offload=True) full_state_dict = get_model_state_dict(model, options=options) if _get_torch_distributed_rank() == 0: return full_state_dict return {} - src/transformers/dynamic_module_utils.py (36834 bytes)
# Copyright 2021 The HuggingFace Inc. team. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. """Utilities to dynamically load objects from the Hub.""" import ast import filecmp import hashlib import importlib import importlib.metadata import importlib.util import keyword import os import re import shutil import signal import sys import threading from pathlib import Path from types import ModuleType from typing import Any from huggingface_hub import ResolvedRevision, is_offline_mode, try_to_load_from_cache from packaging import version from .utils import ( HF_MODULES_CACHE, TRANSFORMERS_DYNAMIC_MODULE_NAME, cached_file, extract_commit_hash, logging, ) from .utils.import_utils import VersionComparison, split_package_version logger = logging.get_logger(__name__) # pylint: disable=invalid-name def _sanitize_module_name(name: str) -> str: r""" Tries to sanitize a module name so that it can be used as a Python module. The following transformations are applied: 1. Replace `.` in module names with `_dot_`. 2. Replace `-` in module names with `_hyphen_`. 3. If the module name starts with a digit, prepend it with `_`. 4. Warn if the sanitized name is a Python reserved keyword or not a valid identifier. If the input name is already a valid identifier, it is returned unchanged. """ # We not replacing `\W` characters with `_` to avoid collisions. Because `_` is a very common # separator used in module names, replacing `\W` with `_` would create too many collisions. # Once a module is imported, it is cached in `sys.modules` and the second import would return # the first module, which might not be the expected behavior if name collisions happen. new_name = name.replace(".", "_dot_").replace("-", "_hyphen_") if new_name and new_name[0].isdigit(): new_name = f"_{new_name}" if keyword.iskeyword(new_name): logger.warning( f"The module name {new_name} (originally {name}) is a reserved keyword in Python. " "Please rename the original module to avoid import issues." ) elif not new_name.isidentifier(): logger.warning( f"The module name {new_name} (originally {name}) is not a valid Python identifier. " "Please rename the original module to avoid import issues." ) return new_name _HF_REMOTE_CODE_LOCK = threading.Lock() def init_hf_modules(): """ Creates the cache directory for modules with an init, and adds it to the Python path. """ # This function has already been executed if HF_MODULES_CACHE already is in the Python path. if HF_MODULES_CACHE in sys.path: return sys.path.append(HF_MODULES_CACHE) os.makedirs(HF_MODULES_CACHE, exist_ok=True) init_path = Path(HF_MODULES_CACHE) / "__init__.py" if not init_path.exists(): init_path.touch() importlib.invalidate_caches() def create_dynamic_module(name: str | os.PathLike) -> None: """ Creates a dynamic module in the cache directory for modules. Args: name (`str` or `os.PathLike`): The name of the dynamic module to create. """ init_hf_modules() dynamic_module_path = (Path(HF_MODULES_CACHE) / name).resolve() # If the parent module does not exist yet, recursively create it. if not dynamic_module_path.parent.exists(): create_dynamic_module(dynamic_module_path.parent) os.makedirs(dynamic_module_path, exist_ok=True) init_path = dynamic_module_path / "__init__.py" if not init_path.exists(): init_path.touch() # It is extremely important to invalidate the cache when we change stuff in those modules, or users end up # with errors about module that do not exist. Same for all other `invalidate_caches` in this file. importlib.invalidate_caches() def get_relative_imports(module_file: str | os.PathLike) -> list[str]: """ Get the list of modules that are relatively imported in a module file. Args: module_file (`str` or `os.PathLike`): The module file to inspect. Returns: `list[str]`: The list of relative imports in the module. """ with open(module_file, encoding="utf-8") as f: content = f.read() # Imports of the form `import .xxx` relative_imports = re.findall(r"^\s*import\s+\.(\S+)\s*$", content, flags=re.MULTILINE) # Imports of the form `from .xxx import yyy` relative_imports += re.findall(r"^\s*from\s+\.(\S+)\s+import", content, flags=re.MULTILINE) # Unique-ify return list(set(relative_imports)) def get_relative_import_files(module_file: str | os.PathLike) -> list[str]: """ Get the list of all files that are needed for a given module. Note that this function recurses through the relative imports (if a imports b and b imports c, it will return module files for b and c). Args: module_file (`str` or `os.PathLike`): The module file to inspect. Returns: `list[str]`: The list of all relative imports a given module needs (recursively), which will give us the list of module files a given module needs. """ no_change = False files_to_check = [module_file] all_relative_imports = [] # Let's recurse through all relative imports while not no_change: new_imports = [] for f in files_to_check: new_imports.extend(get_relative_imports(f)) module_path = Path(module_file).parent new_import_files = [f"{str(module_path / m)}.py" for m in new_imports] files_to_check = [f for f in new_import_files if f not in all_relative_imports] no_change = len(files_to_check) == 0 all_relative_imports.extend(files_to_check) return all_relative_imports def get_imports(filename: str | os.PathLike) -> list[str]: """ Extracts all the libraries (not relative imports this time) that are imported in a file. Args: filename (`str` or `os.PathLike`): The module file to inspect. Returns: `list[str]`: The list of all packages required to use the input module. """ with open(filename, encoding="utf-8") as f: content = f.read() imported_modules = set() import transformers.utils def recursive_look_for_imports(node): if isinstance(node, ast.Try): return # Don't recurse into Try blocks and ignore imports in them elif isinstance(node, ast.If): test = node.test for condition_node in ast.walk(test): if isinstance(condition_node, ast.Call): check_function = getattr(condition_node.func, "id", "") if ( check_function.endswith("available") and check_function.startswith("is_flash_attn") or hasattr(transformers.utils.import_utils, check_function) ): # Don't recurse into "if flash_attn_available()" or any "if library_available" blocks # that appears in `transformers.utils.import_utils` and ignore imports in them return elif isinstance(node, ast.Import): # Handle 'import x' statements for alias in node.names: top_module = alias.name.split(".")[0] if top_module: imported_modules.add(top_module) elif isinstance(node, ast.ImportFrom): # Handle 'from x import y' statements, ignoring relative imports if node.level == 0 and node.module: top_module = node.module.split(".")[0] if top_module: imported_modules.add(top_module) # Recursively visit all children for child in ast.iter_child_nodes(node): recursive_look_for_imports(child) tree = ast.parse(content) recursive_look_for_imports(tree) return sorted(imported_modules) def check_imports(filename: str | os.PathLike) -> list[str]: """ Check if the current Python environment contains all the libraries that are imported in a file. Will raise if a library is missing. Args: filename (`str` or `os.PathLike`): The module file to check. Returns: `list[str]`: The list of relative imports in the file. """ imports = get_imports(filename) missing_packages = [] for imp in imports: try: importlib.import_module(imp) except ImportError as exception: logger.warning(f"Encountered exception while importing {imp}: {exception}") # Some packages can fail with an ImportError because of a dependency issue. # This check avoids hiding such errors. # See https://github.com/huggingface/transformers/issues/33604 if "No module named" in str(exception): missing_packages.append(imp) else: raise if len(missing_packages) > 0: raise ImportError( "This modeling file requires the following packages that were not found in your environment: " f"{', '.join(missing_packages)}. Run `pip install {' '.join(missing_packages)}`" ) return get_relative_imports(filename) def get_class_in_module( class_name: str, module_path: str | os.PathLike, *, force_reload: bool = False, ) -> type: """ Import a module on the cache directory for modules and extract a class from it. Args: class_name (`str`): The name of the class to import. module_path (`str` or `os.PathLike`): The path to the module to import. force_reload (`bool`, *optional*, defaults to `False`): Whether to reload the dynamic module from file if it already exists in `sys.modules`. Otherwise, the module is only reloaded if the file has changed. Returns: `typing.Type`: The class looked for. """ name = os.path.normpath(module_path) name = name.removesuffix(".py") name = name.replace(os.path.sep, ".") module_file: Path = Path(HF_MODULES_CACHE) / module_path with _HF_REMOTE_CODE_LOCK: if force_reload: sys.modules.pop(name, None) importlib.invalidate_caches() cached_module: ModuleType | None = sys.modules.get(name) module_spec = importlib.util.spec_from_file_location(name, location=module_file) # Hash the module file and all its relative imports to check if we need to reload it module_files: list[Path] = [module_file] + sorted(map(Path, get_relative_import_files(module_file))) module_hash: str = hashlib.sha256(b"".join(bytes(f) + f.read_bytes() for f in module_files)).hexdigest() module: ModuleType if cached_module is None: module = importlib.util.module_from_spec(module_spec) # insert it into sys.modules before any loading begins sys.modules[name] = module else: module = cached_module # reload in both cases, unless the module is already imported and the hash hits if getattr(module, "__transformers_module_hash__", "") != module_hash: module_spec.loader.exec_module(module) module.__transformers_module_hash__ = module_hash return getattr(module, class_name) def _compute_local_source_files_hash( pretrained_model_name_or_path: str | os.PathLike, resolved_module_file: str | os.PathLike, ) -> str: """ Computes a stable hash from the bytes of the local source file and its relative-import source files. """ model_path = Path(pretrained_model_name_or_path).resolve() resolved_module_file = Path(resolved_module_file) def _resolve_relative_source_path(source_file_path: Path) -> str: canonical_path = source_file_path.parent.resolve() / source_file_path.name try: return canonical_path.relative_to(model_path).as_posix() except ValueError: return canonical_path.as_posix() files_to_hash = [ (_resolve_relative_source_path(resolved_module_file), resolved_module_file), ] for source_file in get_relative_import_files(resolved_module_file): source_file_path = Path(source_file) files_to_hash.append((_resolve_relative_source_path(source_file_path), source_file_path)) source_files_hash = hashlib.sha256() for relative_path, file_path in sorted(files_to_hash, key=lambda entry: entry[0]): source_files_hash.update(relative_path.encode("utf-8")) source_files_hash.update(file_path.read_bytes()) return source_files_hash.hexdigest()[:16] def get_cached_module_file( pretrained_model_name_or_path: str | os.PathLike, module_file: str, cache_dir: str | os.PathLike | None = None, force_download: bool = False, proxies: dict[str, str] | None = None, token: bool | str | None = None, revision: str | None = None, local_files_only: bool = False, repo_type: str | None = None, **deprecated_kwargs, ) -> str: """ Prepares Downloads a module from a local folder or a distant repo and returns its path inside the cached Transformers module. Args: pretrained_model_name_or_path (`str` or `os.PathLike`): This can be either: - a string, the *model id* of a pretrained model configuration hosted inside a model repo on huggingface.co. - a path to a *directory* containing a configuration file saved using the [`~PreTrainedTokenizer.save_pretrained`] method, e.g., `./my_model_directory/`. module_file (`str`): The name of the module file containing the class to look for. cache_dir (`str` or `os.PathLike`, *optional*): Path to a directory in which a downloaded pretrained model configuration should be cached if the standard cache should not be used. force_download (`bool`, *optional*, defaults to `False`): Whether or not to force to (re-)download the configuration files and override the cached versions if they exist. proxies (`dict[str, str]`, *optional*): A dictionary of proxy servers to use by protocol or endpoint, e.g., `{'http': 'foo.bar:3128', 'http://hostname': 'foo.bar:4012'}.` The proxies are used on each request. token (`str` or *bool*, *optional*) - src/transformers/exporters/__init__.py (2729 bytes)
# Copyright 2026 The HuggingFace Inc. team. All rights reserved. # Modifications Copyright (C) 2025, Advanced Micro Devices, Inc. All rights reserved. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. from typing import TYPE_CHECKING from ..utils import _LazyModule _import_structure = { "auto": ["AutoExportConfig", "AutoHfExporter", "register_backend"], "base": ["ExportedComponent", "ExportedModel", "HfExporter"], "configs": [ "DynamoConfig", "ExecutorchConfig", "ExportConfigMixin", "ExportFormat", "OnnxConfig", "OpenVINOConfig", ], "exporter_dynamo": ["DynamoExporter"], "exporter_executorch": ["ExecutorchExporter"], "exporter_onnx": ["OnnxExporter"], "exporter_openvino": ["OpenVINOExporter"], "metadata": ["ExportMetadata"], "quantizers": [ "ExportQuantizer", "NNCFOnnxQuantizer", "NNCFOpenVINOQuantizer", "NNCFQuantizationConfig", "NNCFTorchFXQuantizer", "NNCFWeightCompressionConfig", "OnnxRuntimeDynamicConfig", "OnnxRuntimeQuantizer", "OnnxRuntimeStaticConfig", "PT2EQuantizer", ], "runners": ["ModelRunner"], } if TYPE_CHECKING: from .auto import AutoExportConfig, AutoHfExporter, register_backend from .base import ExportedComponent, ExportedModel, HfExporter from .configs import DynamoConfig, ExecutorchConfig, ExportConfigMixin, ExportFormat, OnnxConfig, OpenVINOConfig from .exporter_dynamo import DynamoExporter from .exporter_executorch import ExecutorchExporter from .exporter_onnx import OnnxExporter from .exporter_openvino import OpenVINOExporter from .metadata import ExportMetadata from .quantizers import ( ExportQuantizer, NNCFOnnxQuantizer, NNCFOpenVINOQuantizer, NNCFQuantizationConfig, NNCFTorchFXQuantizer, NNCFWeightCompressionConfig, OnnxRuntimeDynamicConfig, OnnxRuntimeQuantizer, OnnxRuntimeStaticConfig, PT2EQuantizer, ) from .runners import ModelRunner else: import sys sys.modules[__name__] = _LazyModule(__name__, globals()["__file__"], _import_structure, module_spec=__spec__)