# Project export: MetalTranslate: CTranslate2 Unleashed on Apple Silicon This document was generated by HackStack to give an AI agent context about a hackathon project. Sections are labeled with their provenance; content marked as truncated was cut to keep this document small. ## Project metadata - Hackathon: OpenAI Build Week - Tagline: Built with Codex, MetalTranslate adds native Metal MPS acceleration to CTranslate2 for fast, private Apple Silicon inference, up to 1.64× faster than CPU with FP16, BF16, and INT8. - Devpost: https://devpost.com/software/metaltranslate-ctranslate2-unleashed-on-apple-silicon - GitHub: https://github.com/TBO22/CTranslate2 - Video: https://www.youtube.com/embed/0UOFot-fDhQ?enablejsapi=1&hl=en_US&rel=0&start=&version=3&wmode=transparent - Team: 1 GitHub contributor(s) — Talha Bin Omar (11 commits) ## Devpost submission (written by the team) ### Inspiration I started MetalTranslate because I regularly use CTranslate2 for local translation and speech projects, especially for my locally trained Roman Pashto translation and Pashto speech-to-text models. CTranslate2 is a fast and easy-to-use C++ inference library for Transformer models. It supports models such as Marian and Whisper, and it is also used by popular projects such as Faster Whisper. Many transcription, translation, and local AI tools are built on top of CTranslate2 because it is lightweight, fast, and much easier to deploy than a full deep learning framework. The problem was that CTranslate2 supported optimized CPU inference and NVIDIA CUDA, but it had no native Apple Metal or MPS backend. This is also an issue currently open in its repository. This was a major limitation for anyone testing local models on a Mac. Apple Silicon devices have powerful and efficient integrated GPUs, but models running through CTranslate2 were forced to stay on the CPU. Other frameworks such as PyTorch could use MPS, but they introduce Python overhead, and CTranslate2 applications could not simply use: That meant projects based on Faster Whisper, Marian, and other CTranslate2-supported models could not take advantage of the GPU inside Apple Silicon devices. I originally started working on this backend several months ago, but it remained unfinished because adding MPS support required much more than writing one GPU kernel. It meant understanding a large C++ inference engine, Objective-C++ integration, Metal synchronization, tensor layouts, quantization, decoding, memory management, and many different operators. Codex helped me finally turn that unfinished experiment into a working native backend. ### What it does MetalTranslate adds native Apple Metal and Metal Performance Shaders support to CTranslate2. It works through the normal CTranslate2 API, so users do not need to move their models into PyTorch or rewrite their application around a different framework. Behind that small API change is a complete backend written in C++, Objective-C++, Metal Performance Shaders, and custom Metal kernels. The backend supports FP32, FP16, BF16, and INT8 inference. It also includes the main operations required for real Transformer inference, including matrix multiplication, matrix-vector multiplication, LayerNorm, RMSNorm, softmax, reductions, rotary embeddings, ALiBi, TopK, TopP, sampling, Conv1D, gather, split, concat, tile, transpose, quantization, dequantization, and decoder cache operations. It also supports asynchronous Metal execution, persistent command buffers, compute and blit encoder reuse, reduced synchronization, batched operations, transposed layouts, broadcast strides, odd dimensions, and unaligned tensor shapes. This is not a Python wrapper around another framework. MPS is integrated directly into the CTranslate2 runtime as a real device backend. ### How we built it The first step was understanding how CTranslate2 actually executes a model. A Transformer layer does not directly call a GPU kernel. The operation moves through multiple parts of the library, including model layers, operators, tensor storage, device primitives, memory allocators, quantization code, search logic, and backend-specific implementations. Codex helped me trace these paths and compare the existing CPU and CUDA backends. One of the most important parts was understanding autoregressive decoding. During single-token generation, many projection operations have a shape similar to: $$ A_{1 \times K} B_{K \times N} = C_{1 \times N} $$ Because (M = 1), this behaves more like a matrix-vector multiplication than a large matrix multiplication. Large GPU matrix multiplications are relatively easy to accelerate, but autoregressive decoding is different. It contains many small operations where command submission, memory movement, synchronization, temporary allocations, and cache updates are very important. Because of that, I implemented a specialized FP16 GEMV path for batch-size-one decoding, along with tiled GEMM for larger operations. The first version submitted a separate Metal command buffer for almost every operation. It worked, but it introduced too much overhead. A single generated token can require dozens of operations, so constantly creating and committing command buffers made the GPU wait for the CPU. I replaced that with a persistent asynchronous Metal execution stream. The backend keeps an active command buffer, reuses compute and blit encoders, and only synchronizes when the CPU actually needs the result. I also added configurable submission limits for compute and copy operations. One interesting result was that accumulating too many operations made performance worse because the GPU started executing later. On the tested Apple M1 workload, a threshold of around 16 compute operations produced the best result. I also added reduced-precision support. For INT8, values are quantized using symmetric scaling: $$ s = \frac{\max_j |x_j|}{127} $$ $$ q_i = \operatorname{round}\left(\frac{x_i}{s}\right) $$ The GPU performs signed INT8 multiplication with INT32 accumulation, followed by dequantization: $$y_{mn}=s_{A,m}s_{B,n}\sum_{k=1}^{K}q^A_{mk}q^B_{kn}$$ For BF16, tensors remain stored in BF16, while important accumulation operations are performed in FP32 before being converted back. The backend was developed by repeatedly inspecting an execution path, implementing one change, compiling it, running tests, comparing CPU and MPS output, and then benchmarking the result. ### Challenges we ran into The hardest problem was not making the backend fast. It was making it fast while keeping the output correct. At one point, the MPS backend appeared to be more than twice as fast as CPU inference. The problem was that the translations were completely wrong. The model produced repeated unrelated words, different output lengths, and sometimes continued generating until the maximum token limit. The backend was fast, but some tensors were being corrupted during asynchronous execution. I treated every performance result as invalid until the output matched the CPU backend. The issue made me investigate command ordering, tensor broadcasting, shared-memory visibility, buffer offsets, buffer lifetimes, gather operations, encoder transitions, aliasing, and synchronization boundaries. I added tests for transposed matrices, odd dimensions, unaligned tensor sizes, batched strides, broadcast strides, interior buffer offsets, TopK ties, quantization, grouped Conv1D, sampling, and deterministic translation output. Another challenge was that lower precision did not always mean better performance. INT8 worked correctly and reduced model storage, but it did not outperform FP16 during batch-size-one Marian decoding on my M1 MacBook Air. Because of that, MetalTranslate selects FP16 for automatic compute mode while still allowing users to choose INT8 manually. This project made it very clear that a fast backend is useless if the output is wrong. ### Accomplishments we're proud of The biggest accomplishment is that MPS now works through the normal CTranslate2 API. Users do not need to rewrite their model, move their application into PyTorch, or change the overall structure of their inference pipeline. On an Apple M1 MacBook Air with a 7-core GPU, using a Release build and a real FP16 Marian Roman Pashto translation model, MetalTranslate achieved the following result: The throughput improvement was: $$ \frac{187.21}{114.32} \approx 1.64 $$ The latency reduction was: $$ \frac{306.15 - 186.96}{306.15} \times 100 \approx 38.9% $$ That means MetalTranslate achieved up to 1.64 times higher throughput and around 39 percent lower latency than optimized CPU inference on this workload. The final output was correct and deterministic. The backend passed all 175 MPS tests, including FP32, FP16, BF16, INT8, quantization, dequantization, quantized Conv1D, grouped Conv1D, transposed matrices, odd dimensions, batched operations, buffer offset handling, TopK, sampling, and CPU versus MPS translation comparisons. A separate CPU-only build also passed 190 tests. I am also proud that the project was driven by a real low-resource language use case. The model used during development was a Roman Pashto translation model. Pashto has far fewer local translation and speech tools than major languages. Making these models faster and easier to run on everyday Apple Silicon hardware has practical value beyond a synthetic benchmark. ### What we learned I learned that Apple unified memory does not remove the need for synchronization. The CPU and GPU may share physical memory, but the order in which operations happen still has to be controlled carefully. I learned that autoregressive inference is often limited by latency, memory movement, and command submission rather than theoretical GPU compute performance. I also learned that one incorrect broadcast or stale buffer can pass through many Transformer layers before finally destroying the output logits. Another important lesson was that INT8 is not automatically faster on every architecture or workload. Most importantly, I learned not to trust a performance result until the output is correct, deterministic, and tested against a known-good backend. Codex also changed how I approached the project. Instead of using it only to generate code, I used it to inspect the codebase, compare implementations, form hypotheses, debug incorrect output, write tests, analyze benchmarks, resolve merge conflicts, and rebase several months of work onto the latest CTranslate2 version. ### What's next MetalTranslate is functional, but there is still a lot of room for improvement. The next steps include better packed-weight layouts, more GEMV tuning for newer Apple chips, fused attention kernels, fused KV-cache operations, FlashAttention-style Metal kernels, larger-vocabulary GPU TopP, precompiled Metal libraries, and native Apple Silicon Python wheels. I also want to test the backend on M2, M3, and M4 devices. Another major goal is benchmarking Whisper and Faster Whisper workloads. Since Faster Whisper is built on CTranslate2, native MPS support could eventually allow many transcription tools to use the Apple GPU without switching frameworks. The long-term goal is simple: Apple Silicon should become a first-class inference target for CTranslate2. Built with Codex, C++, Objective-C++, Metal, Metal Performance Shaders, Metal Shading Language, Apple Silicon, CTranslate2, Python, Pybind11, CMake, GoogleTest, Marian, FP16, BF16, INT8 quantization, machine translation, natural language processing, GPU computing, and high-performance computing. Try it out Source code View the MetalTranslate source code MetalTranslate documentation Read the Apple Silicon MPS backend documentation Original CTranslate2 project View the original CTranslate2 project ## README (from the GitHub repository) [![CI](https://github.com/OpenNMT/CTranslate2/workflows/CI/badge.svg)](https://github.com/OpenNMT/CTranslate2/actions?query=workflow%3ACI) [![PyPI version](https://badge.fury.io/py/ctranslate2.svg)](https://badge.fury.io/py/ctranslate2) [![Documentation](https://img.shields.io/badge/docs-latest-blue.svg)](https://opennmt.net/CTranslate2/) [![Gitter](https://badges.gitter.im/OpenNMT/CTranslate2.svg)](https://gitter.im/OpenNMT/CTranslate2?utm_source=badge&utm_medium=badge&utm_campaign=pr-badge) [![Forum](https://img.shields.io/discourse/status?server=https%3A%2F%2Fforum.opennmt.net%2F)](https://forum.opennmt.net/) # CTranslate2 CTranslate2 is a C++ and Python library for efficient inference with Transformer models. The project implements a custom runtime that applies many performance optimization techniques such as weights quantization, layers fusion, batch reordering, etc., to [accelerate and reduce the memory usage](#benchmarks) of Transformer models on CPU and GPU. > [!NOTE] > This fork is the home of **MetalTranslate**, a native Apple Metal/MPS backend > for CTranslate2 built for the Codex hackathon. It enables correct, private, > on-device Transformer inference on Apple Silicon and reached **1.64x the > performance of the optimized CPU backend** on the correctness-validated > Marian translation workload described below. Jump to the > [Apple Silicon setup guide](#apple-silicon-mps-backend-experimental) to try it. The following model types are currently supported: * Encoder-decoder models: Transformer base/big, M2M-100, NLLB, BART, mBART, Pegasus, T5, Whisper, T5Gemma, T5Gemma2, MADLAD-400 * Decoder-only models: GPT-2, GPT-J, GPT-NeoX, OPT, BLOOM, MPT, Llama, Mistral, Gemma, CodeGen, GPTBigCode, Falcon, Qwen2 * Encoder-only models: BERT, DistilBERT, XLM-RoBERTa Compatible models should be first converted into an optimized model format. The library includes converters for multiple frameworks: * [OpenNMT-py](https://opennmt.net/CTranslate2/guides/opennmt_py.html) * [OpenNMT-tf](https://opennmt.net/CTranslate2/guides/opennmt_tf.html) * [Fairseq](https://opennmt.net/CTranslate2/guides/fairseq.html) * [Marian](https://opennmt.net/CTranslate2/guides/marian.html) * [OPUS-MT](https://opennmt.net/CTranslate2/guides/opus_mt.html) * [Transformers](https://opennmt.net/CTranslate2/guides/transformers.html) The project is production-oriented and comes with [backward compatibility guarantees](https://opennmt.net/CTranslate2/versioning.html), but it also includes experimental features related to model compression and inference acceleration. ## Key features * **Fast and efficient execution on CPU and GPU**
The execution [is significantly faster and requires less resources](#benchmarks) than general-purpose deep learning frameworks on supported models and tasks thanks to many advanced optimizations: layer fusion, padding removal, batch reordering, in-place operations, caching mechanism, etc. * **Quantization and reduced precision**
The model serialization and computation support weights with [reduced precision](https://opennmt.net/CTranslate2/quantization.html): 16-bit floating points (FP16), 16-bit brain floating points (BF16), 16-bit integers (INT16), 8-bit integers (INT8) and AWQ quantization (INT4). * **Multiple CPU architectures support**
The project supports x86-64 and AArch64/ARM64 processors and integrates multiple backends that are optimized for these platforms: [Intel MKL](https://software.intel.com/content/www/us/en/develop/tools/oneapi/components/onemkl.html), [oneDNN](https://github.com/oneapi-src/oneDNN), [OpenBLAS](https://www.openblas.net/), [Ruy](https://github.com/google/ruy), and [Apple Accelerate](https://developer.apple.com/documentation/accelerate). * **Automatic CPU detection and code dispatch**
One binary can include multiple backends (e.g. Intel MKL and oneDNN) and instruction set architectures (e.g. AVX, AVX2) that are automatically selected at runtime based on the CPU information. * **Parallel and asynchronous execution**
Multiple batches can be processed in parallel and asynchronously using multiple GPUs or CPU cores. * **Dynamic memory usage**
The memory usage changes dynamically depending on the request size while still meeting performance requirements thanks to caching allocators on both CPU and GPU. * **Lightweight on disk**
Quantization can make the models 4 times smaller on disk with minimal accuracy loss. * **Simple integration**
The project has few dependencies and exposes simple APIs in [Python](https://opennmt.net/CTranslate2/python/overview.html) and C++ to cover most integration needs. * **Configurable and interactive decoding**
[Advanced decoding features](https://opennmt.net/CTranslate2/decoding.html) allow autocompleting a partial sequence and returning alternatives at a specific location in the sequence. * **Support tensor parallelism for distributed inference**
Very large model can be split into multiple GPUs. Following this [documentation](docs/parallel.md#model-and-tensor-parallelism) to set up the required environment. Some of these features are difficult to achieve with standard deep learning frameworks and are the motivation for this project. ## Installation and usage CTranslate2 can be installed with pip: ```bash pip install ctranslate2 ``` The Python module is used to convert models and can translate or generate text with few lines of code: ```python translator = ctranslate2.Translator(translation_model_path) translator.translate_batch(tokens) generator = ctranslate2.Generator(generation_model_path) generator.generate_batch(start_tokens) ``` See the [documentation](https://opennmt.net/CTranslate2) for more information and examples. If you have an AMD ROCm GPU, we provide specific Python wheels on the [releases page](https://github.com/OpenNMT/CTranslate2/releases/). ### Apple Silicon MPS backend (experimental) MetalTranslate is implemented directly inside the CTranslate2 runtime. It is not a wrapper around PyTorch: model layers dispatch through CTranslate2's C++ operator system into Objective-C++ and Metal kernels. The backend includes a persistent asynchronous command stream, decode-specific FP16 GEMV, tiled and batched GEMM, GPU search and sampling, reductions, layout operations, and quantized execution. The MPS build is currently source-only and requires: * An Apple Silicon Mac running macOS 11 or newer * Xcode Command Line Tools (`xcode-select --install`) * CMake and a C++17 compiler * Python 3.9 or newer for the optional Python API #### 1. Clone and create a Python environment ```bash git clone --recursive https://github.com/TBO22/CTranslate2.git cd CTranslate2 python3 -m venv .venv source .venv/bin/activate python -m pip install --upgrade pip ``` > [!IMPORTANT] > CTranslate2 uses Git submodules for build dependencies such as `cxxopts`, > `googletest`, and `spdlog`. GitHub's **Download ZIP** archive does not include > their contents. Use the recursive clone command above for a source build. If the repository was already cloned without `--recursive`, initialize the missing dependencies before running CMake: ```bash git submodule update --init --recursive ``` If CMake reports that `third_party/googletest` has no `CMakeLists.txt` or that `cxxopts` is missing, the submodules were not initialized; the command above fixes both errors. #### 2. Build and install the C++ library ```bash cmake -S . -B build-mps \ -DCMAKE_BUILD_TYPE=Release \ -DBUILD_TESTS=ON \ -DWITH_MPS=ON \ -DWITH_ACCELERATE=ON \ -DWITH_MKL=OFF \ -DOPENMP_RUNTIME=NONE cmake --build build-mps -j 4 cmake --install build-mps --prefix "$PWD/install-mps" ``` `WITH_MPS` cannot be combined with CUDA or HIP in the same build. #### 3. Install the Python extension The `CTRANSLATE2_ROOT` value makes the extension compile and link against the MPS-enabled library that was just installed. `ARCHFLAGS` prevents an invalid x86_64 slice from being added to the native Apple Silicon [README truncated for size] ## Detected evidence (automated analysis) Indexed codebase: 378 recognized source files, 3352 KB. - C (language) — detected in the code - C++ (language) — detected in the code - CSS (language) — detected in the code - Hugging Face (technology) — detected in the code - Python (language) — detected in the code - PyTorch (technology) — detected in the code ## Codebase structure (from repository index) ### Files (120 of 516) ``` .dockerignore .github/workflows/ci.yml .gitignore .gitmodules CHANGELOG.md cli/CMakeLists.txt cli/translator.cc cmake/ctranslate2Config.cmake cmake/FindNCCL.cmake CMakeLists.txt CONTRIBUTING.md docker/build_all.sh docker/Dockerfile docker/Dockerfile_rocm docs/_static/custom.css docs/conf.py docs/conversion.md docs/decoding.md docs/encoding.md docs/environment_variables.md docs/faq.md docs/generate.py docs/generation.md docs/guides/fairseq.md docs/guides/marian.md docs/guides/opennmt_py.md docs/guides/opennmt_tf.md docs/guides/opus_mt.md docs/guides/transformers.md docs/hardware_support.md docs/index.rst docs/installation.md docs/memory.md docs/parallel.md docs/performance.md docs/quantization.md docs/quickstart.md docs/README.md docs/requirements.txt docs/speech_recognition.md docs/translation.md docs/versioning.md examples/llama2/chat.py examples/llama2/README.md examples/llama2/requirements.txt examples/llama3/chat.py examples/llama3/README.md examples/llama3/requirements.txt examples/wngt2020/CMakeLists.txt examples/wngt2020/Dockerfile.cpu examples/wngt2020/Dockerfile.gpu examples/wngt2020/main.cc examples/wngt2020/README.md examples/wngt2020/run.sh include/ctranslate2/allocator.h include/ctranslate2/batch_reader.h include/ctranslate2/bfloat16.h include/ctranslate2/buffered_translation_wrapper.h include/ctranslate2/decoding_utils.h include/ctranslate2/decoding.h include/ctranslate2/devices.h include/ctranslate2/encoder.h include/ctranslate2/encoding.h include/ctranslate2/filesystem.h include/ctranslate2/generation.h include/ctranslate2/generator.h include/ctranslate2/layers/attention_layer.h include/ctranslate2/layers/attention.h include/ctranslate2/layers/common.h include/ctranslate2/layers/decoder.h include/ctranslate2/layers/encoder.h include/ctranslate2/layers/flash_attention.h include/ctranslate2/layers/layers.h include/ctranslate2/layers/transformer.h include/ctranslate2/layers/wav2vec2.h include/ctranslate2/layers/wav2vec2bert.h include/ctranslate2/layers/whisper.h include/ctranslate2/logging.h include/ctranslate2/models/language_model.h include/ctranslate2/models/model_factory.h include/ctranslate2/models/model_reader.h include/ctranslate2/models/model.h include/ctranslate2/models/sequence_to_sequence.h include/ctranslate2/models/transformer.h include/ctranslate2/models/wav2vec2.h include/ctranslate2/models/wav2vec2bert.h include/ctranslate2/models/whisper.h include/ctranslate2/ops/activation.h include/ctranslate2/ops/add.h include/ctranslate2/ops/alibi_add.h include/ctranslate2/ops/awq/dequantize_awq.h include/ctranslate2/ops/awq/gemm.h include/ctranslate2/ops/awq/gemv.h include/ctranslate2/ops/bias_add.h include/ctranslate2/ops/concat.h include/ctranslate2/ops/conv1d.h include/ctranslate2/ops/cos.h include/ctranslate2/ops/dequantize.h include/ctranslate2/ops/flash_attention.h include/ctranslate2/ops/flash-attention/alibi.h include/ctranslate2/ops/flash-attention/block_info.h include/ctranslate2/ops/flash-attention/flash_fwd_kernel.h include/ctranslate2/ops/flash-attention/flash_fwd_launch_template.h include/ctranslate2/ops/flash-attention/flash.h include/ctranslate2/ops/flash-attention/kernel_traits.h include/ctranslate2/ops/flash-attention/mask.h include/ctranslate2/ops/flash-attention/rotary.h include/ctranslate2/ops/flash-attention/softmax.h include/ctranslate2/ops/flash-attention/static_switch.h include/ctranslate2/ops/flash-attention/utils.h include/ctranslate2/ops/gather.h include/ctranslate2/ops/gelu.h include/ctranslate2/ops/gemm.h include/ctranslate2/ops/gumbel_max.h include/ctranslate2/ops/identity.h include/ctranslate2/ops/layer_norm.h include/ctranslate2/ops/log.h include/ctranslate2/ops/matmul.h include/ctranslate2/ops/mean.h include/ctranslate2/ops/median_filter.h [396 more files omitted for size] ``` ### Dependencies - docs/requirements.txt: myst-parser@==0.18.*, sphinx@==5.3.*, sphinx-rtd-theme@==1.0.* - examples/llama2/requirements.txt: accelerate, ctranslate2@>=3.17.1,<4, sentencepiece, transformers[torch]@==4.31.* - examples/llama3/requirements.txt: accelerate, ctranslate2@>=4.2.1, transformers[torch]@==4.40.* - python/tests/requirements.txt: protobuf, pytest, sentencepiece, torch@==2.12, transformers@==5.9.0.*, wurlitzer@==3.1.* - tools/benchmark_tensor_parallel/requirements.txt: ctranslate2@>=4.1.0, GPUtil, sentencepiece - tools/benchmark/requirements.txt: docker@==4.3.1, gputil@==1.4.0, sacrebleu@==1.4.14 ### Recent commits (newest first) - Fix CUDA half precision dispatch linking - Document recursive clone for source builds - Fix MPS async lifetimes and Whisper decoding - Optimize MPS decode GEMV and fused epilogues - Correct attribution of MPS research and Codex work - Make Codex development story more personal - Document MetalTranslate setup and Codex development - Fix Python package formatting - Add MPS BF16 and INT8 inference support - Fix MPS block broadcast correctness - Add Apple MPS backend on current upstream - Bump version to 4.8.1 (#2071) - Version 4.8.1 (#2070) - Harden legacy converter checkpoint loading (#2036) - Fix model load heap overflow (#2068) - Fix process-killing integer division by zero when Whisper align() gets a window with no frames (#2065) - CI/CD compilation fixes for Windows (#2069) - Optimize attention softmax buffer reuse (#2066) - Support for gemma4 12b dense model (#2060) - Version 4.80 (#2061) ## Key source files (fetched from GitHub, selected and truncated for size) ### SECURITY.md ```markdown # Security Policy ## Supported Versions Security fixes are provided for the latest release series only. ## Reporting a Vulnerability Do not open public issues for security vulnerabilities. Please report security vulnerabilities privately to Jordi Mas . ## What to Include in Your Report To help us triage effectively, please include: - **Severity assessment**: Critical / High / Medium / Low, with a short justification. - **Impact**: what an attacker can achieve (RCE, information disclosure, DoS, etc.). - **Reproduction steps** with enough detail for maintainers to understand and validate the issue. ## Response SLA - We aim to acknowledge new reports within **2 weeks** of submission. - We aim to provide a fix or mitigation within **90 days** of acknowledgement. ``` ### CONTRIBUTING.md ```markdown # Contributing This document provides some information to help you contribute to the CTranslate2. ## Reporting issues We use GitHub issues for bugs in the code that are **reproducible**. A good bug report should contain every information needed to reproduce it. Before opening a new issue, make sure to: * **use the GitHub issue search** for existing and fixed bugs; * **check if the issue has been fixed** in a more recent version; * **isolate the problem** to give as much context as possible. If you have questions on how to use the project or have trouble getting started with it, consider using [our forum](https://forum.opennmt.net/) instead and tagging your topic with the *ctranslate2* tag. ## Requesting features Do you think a feature is missing or would be a great addition to the project? Please open a GitHub issue to describe it. ## Developing code * If you want to contribute with code but are unsure what to do, * search for *TODO* comments in the code: these are small dev tasks that should be addressed at some point. * look for GitHub issues marked with the *help wanted* label: these are developments that we find particularly suited for community contributions. * If you are planning to make a large change to the existing code, consider asking first on [the forum](https://forum.opennmt.net/) to confirm that it is welcome. ## Contribution rules CTranslate2 is a low-level, performance-critical codebase. A single misplaced pointer or inefficient memory allocation (which LLMs often get wrong) can take hours to debug. To maintain code integrity and manage maintainer workload, we apply the following policy: * Use of AI tools for brainstorming or minor assistance is acceptable, but contributors must explicitly disclose how AI was used and remain fully responsible for correctness, performance, and design. Submissions that appear generated without deep understanding will be declined. Verifying AI output for correctness and performance is more time-consuming than writing code manually. * Mandatory Deep Understanding: Contributors must fully understand their code and be prepared to justify the purpose of part of the code base. * Please contribute within your area of expertise. If you are not familiar with the core codebase, consider contributing to documentation, examples, or Hugging Face integrations. ### Building the sources See [Install from sources](https://opennmt.net/CTranslate2/installation.html#install-from-sources). ### Running the tests #### C++ To enable the C++ tests, you should configure the project with `cmake -DBUILD_TESTS=ON`. The binary `tests/ctranslate2_test` runs all tests using [Google Test](https://github.com/google/googletest). It expects the path to the test data as argument: ```bash ./tests/ctranslate2_test ../tests/data ``` #### Python The Python tests can be run with `pytest`: ```bash cd python pip install -r tests/requirements.txt pytest tests/ ``` The code should also be checked with `black` (auto [truncated — 6310 more characters] ``` ### docs/requirements.txt ``` myst-parser==0.18.* sphinx-rtd-theme==1.0.* sphinx==5.3.* ``` ### python/pyproject.toml ``` [build-system] requires = ["setuptools", "wheel", "pybind11==2.11.1"] build-backend = "setuptools.build_meta" ``` ### docker/Dockerfile ``` FROM nvidia/cuda:12.8.1-cudnn-devel-ubuntu22.04 AS builder RUN apt-get update && \ apt-get install -y --no-install-recommends \ python3-dev \ python3-pip \ wget \ && \ apt-get clean && \ rm -rf /var/lib/apt/lists/* WORKDIR /root ENV ONEAPI_VERSION=2025.3 RUN wget -q https://apt.repos.intel.com/intel-gpg-keys/GPG-PUB-KEY-INTEL-SW-PRODUCTS.PUB && \ apt-key add *.PUB && \ rm *.PUB && \ echo "deb https://apt.repos.intel.com/oneapi all main" > /etc/apt/sources.list.d/oneAPI.list && \ apt-get update && \ apt-get install -y --no-install-recommends \ intel-oneapi-mkl-devel-$ONEAPI_VERSION \ && \ apt-get clean && \ rm -rf /var/lib/apt/lists/* RUN python3 -m pip --no-cache-dir install cmake==3.22.* ENV ONEDNN_VERSION=3.1.1 RUN wget -q https://github.com/oneapi-src/oneDNN/archive/refs/tags/v${ONEDNN_VERSION}.tar.gz && \ tar xf *.tar.gz && \ rm *.tar.gz && \ cd oneDNN-* && \ cmake -DCMAKE_BUILD_TYPE=Release -DONEDNN_LIBRARY_TYPE=STATIC -DONEDNN_BUILD_EXAMPLES=OFF -DONEDNN_BUILD_TESTS=OFF -DONEDNN_ENABLE_WORKLOAD=INFERENCE -DONEDNN_ENABLE_PRIMITIVE="CONVOLUTION;REORDER" -DONEDNN_BUILD_GRAPH=OFF . && \ make -j$(nproc) install && \ cd .. && \ rm -r oneDNN-* ENV OPENMPI_VERSION=4.1.6 RUN wget -q https://download.open-mpi.org/release/open-mpi/v4.1/openmpi-${OPENMPI_VERSION}.tar.bz2 && \ tar xf *.tar.bz2 && \ rm *.tar.bz2 && \ cd openmpi-* && \ ./configure && \ make -j$(nproc) install && \ cd .. && \ rm -r openmpi-* COPY third_party third_party COPY cli cli COPY include include COPY src src COPY cmake cmake COPY python python COPY CMakeLists.txt . ARG CXX_FLAGS ENV CXX_FLAGS=${CXX_FLAGS:-"-msse4.1"} ARG CUDA_NVCC_FLAGS ENV CUDA_NVCC_FLAGS=${CUDA_NVCC_FLAGS:-"-Xfatbin=-compress-all"} ARG CUDA_ARCH_LIST ENV CUDA_ARCH_LIST=${CUDA_ARCH_LIST:-"Common"} ENV CTRANSLATE2_ROOT=/opt/ctranslate2 ENV LD_LIBRARY_PATH=/usr/local/lib/:${LD_LIBRARY_PATH} RUN mkdir build_tmp && \ cd build_tmp && \ cmake -DCMAKE_INSTALL_PREFIX=${CTRANSLATE2_ROOT} \ -DWITH_CUDA=ON -DWITH_CUDNN=ON -DWITH_MKL=ON -DWITH_DNNL=ON -DOPENMP_RUNTIME=COMP \ -DCMAKE_BUILD_TYPE=Release -DCMAKE_CXX_FLAGS="${CXX_FLAGS}" \ -DCUDA_NVCC_FLAGS="${CUDA_NVCC_FLAGS}" -DCUDA_ARCH_LIST="${CUDA_ARCH_LIST}" -DWITH_TENSOR_PARALLEL=ON .. && \ VERBOSE=1 make -j$(nproc) install ENV LANG=en_US.UTF-8 COPY README.md . RUN cd python && \ python3 -m pip --no-cache-dir install -r install_requirements.txt && \ python3 setup.py bdist_wheel --dist-dir $CTRANSLATE2_ROOT FROM nvidia/cuda:12.8.1-base-ubuntu22.04 # We remove the cuda-compat package because it conflicts with the CUDA Enhanced Compatibility. # See e.g. https://github.com/NVIDIA/nvidia-docker/issues/1515 RUN apt-get update && \ apt-get install -y --no-install-recommends \ libcublas-12-8=12.8.4.1-1 \ libcudnn9-cuda-12=9.10.2.21-1 \ libnccl2=2.26.2-1+cuda12.8 \ libopenmpi3=4.1.2-2ubuntu1 \ openmpi-bin \ libgomp1 \ python3-pip \ && \ apt-get purge -y cuda-compat-12-8 && \ apt-get clean && \ rm -rf /var/lib/apt/lists/* ENV CTRANSLATE2_ROOT=/opt/ctranslate2 ENV LD_LIBRARY_PATH=$LD_LIBRARY_PATH:$CTRANSLATE2_ROOT/lib COPY --from=builder $CTRANSLATE2_ROOT $CTRANSLATE2_ROOT RUN python3 -m pip --no-cache-dir install $CTRANSLATE2_ROOT/*.whl && \ rm $CTRANSLATE2_ROOT/*.whl ENTRYPOINT ["/opt/ctranslate2/bin/ct2-translator"] ``` ### tools/benchmark_tensor_parallel/requirements.txt ``` ctranslate2>=4.1.0 sentencepiece GPUtil ``` ### tools/benchmark/requirements.txt ``` docker==4.3.1 gputil==1.4.0 sacrebleu==1.4.14 ``` ### examples/llama3/requirements.txt ``` ctranslate2>=4.2.1 transformers[torch]==4.40.* accelerate ``` ### examples/llama2/requirements.txt ``` ctranslate2>=3.17.1,<4 sentencepiece transformers[torch]==4.31.* accelerate ``` ### python/tests/requirements.txt ``` transformers==5.9.0.*;platform_system=='Linux' sentencepiece;platform_system=='Linux' protobuf;platform_system=='Linux' pytest wurlitzer==3.1.*;sys_platform=='linux' torch==2.12 ``` [368 more indexed source files omitted to keep this export small. The full file list is in the Codebase structure section above.]