Project Info

MetalTranslate: CTranslate2 Unleashed on Apple Silicon

Inspiration

I started MetalTranslate because I regularly use CTranslate2 for local translation and speech projects, especially for my locally trained Roman Pashto translation and Pashto speech-to-text models. CTranslate2 is a fast and easy-to-use C++ inference library for Transformer models. It supports models such as Marian and Whisper, and it is also used by popular projects such as Faster Whisper. Many transcription, translation, and local AI tools are built on top of CTranslate2 because it is lightweight, fast, and much easier to deploy than a full deep learning framework. The problem was that CTranslate2 supported optimized CPU inference and NVIDIA CUDA, but it had no native Apple Metal or MPS backend. This is also an issue currently open in its repository. This was a major limitation for anyone testing local models on a Mac. Apple Silicon devices have powerful and efficient integrated GPUs, but models running through CTranslate2 were forced to stay on the CPU. Other frameworks such as PyTorch could use MPS, but they introduce Python overhead, and CTranslate2 applications could not simply use: That meant projects based on Faster Whisper, Marian, and other CTranslate2-supported models could not take advantage of the GPU inside Apple Silicon devices. I originally started working on this backend several months ago, but it remained unfinished because adding MPS support required much more than writing one GPU kernel. It meant understanding a large C++ inference engine, Objective-C++ integration, Metal synchronization, tensor layouts, quantization, decoding, memory management, and many different operators. Codex helped me finally turn that unfinished experiment into a working native backend.

What it does

MetalTranslate adds native Apple Metal and Metal Performance Shaders support to CTranslate2. It works through the normal CTranslate2 API, so users do not need to move their models into PyTorch or rewrite their application around a different framework. Behind that small API change is a complete backend written in C++, Objective-C++, Metal Performance Shaders, and custom Metal kernels. The backend supports FP32, FP16, BF16, and INT8 inference. It also includes the main operations required for real Transformer inference, including matrix multiplication, matrix-vector multiplication, LayerNorm, RMSNorm, softmax, reductions, rotary embeddings, ALiBi, TopK, TopP, sampling, Conv1D, gather, split, concat, tile, transpose, quantization, dequantization, and decoder cache operations. It also supports asynchronous Metal execution, persistent command buffers, compute and blit encoder reuse, reduced synchronization, batched operations, transposed layouts, broadcast strides, odd dimensions, and unaligned tensor shapes. This is not a Python wrapper around another framework. MPS is integrated directly into the CTranslate2 runtime as a real device backend.

How we built it

The first step was understanding how CTranslate2 actually executes a model. A Transformer layer does not directly call a GPU kernel. The operation moves through multiple parts of the library, including model layers, operators, tensor storage, device primitives, memory allocators, quantization code, search logic, and backend-specific implementations. Codex helped me trace these paths and compare the existing CPU and CUDA backends. One of the most important parts was understanding autoregressive decoding. During single-token generation, many projection operations have a shape similar to: $$ A_{1 \times K} B_{K \times N} = C_{1 \times N} $$ Because (M = 1), this behaves more like a matrix-vector multiplication than a large matrix multiplication. Large GPU matrix multiplications are relatively easy to accelerate, but autoregressive decoding is different. It contains many small operations where command submission, memory movement, synchronization, temporary allocations, and cache updates are very important. Because of that, I implemented a specialized FP16 GEMV path for batch-size-one decoding, along with tiled GEMM for larger operations. The first version submitted a separate Metal command buffer for almost every operation. It worked, but it introduced too much overhead. A single generated token can require dozens of operations, so constantly creating and committing command buffers made the GPU wait for the CPU. I replaced that with a persistent asynchronous Metal execution stream. The backend keeps an active command buffer, reuses compute and blit encoders, and only synchronizes when the CPU actually needs the result. I also added configurable submission limits for compute and copy operations. One interesting result was that accumulating too many operations made performance worse because the GPU started executing later. On the tested Apple M1 workload, a threshold of around 16 compute operations produced the best result. I also added reduced-precision support. For INT8, values are quantized using symmetric scaling: $$ s = \frac{\max_j |x_j|}{127} $$ $$ q_i = \operatorname{round}\left(\frac{x_i}{s}\right) $$ The GPU performs signed INT8 multiplication with INT32 accumulation, followed by dequantization: $$y_{mn}=s_{A,m}s_{B,n}\sum_{k=1}^{K}q^A_{mk}q^B_{kn}$$ For BF16, tensors remain stored in BF16, while important accumulation operations are performed in FP32 before being converted back. The backend was developed by repeatedly inspecting an execution path, implementing one change, compiling it, running tests, comparing CPU and MPS output, and then benchmarking the result.

Challenges we ran into

The hardest problem was not making the backend fast. It was making it fast while keeping the output correct. At one point, the MPS backend appeared to be more than twice as fast as CPU inference. The problem was that the translations were completely wrong. The model produced repeated unrelated words, different output lengths, and sometimes continued generating until the maximum token limit. The backend was fast, but some tensors were being corrupted during asynchronous execution. I treated every performance result as invalid until the output matched the CPU backend. The issue made me investigate command ordering, tensor broadcasting, shared-memory visibility, buffer offsets, buffer lifetimes, gather operations, encoder transitions, aliasing, and synchronization boundaries. I added tests for transposed matrices, odd dimensions, unaligned tensor sizes, batched strides, broadcast strides, interior buffer offsets, TopK ties, quantization, grouped Conv1D, sampling, and deterministic translation output. Another challenge was that lower precision did not always mean better performance. INT8 worked correctly and reduced model storage, but it did not outperform FP16 during batch-size-one Marian decoding on my M1 MacBook Air. Because of that, MetalTranslate selects FP16 for automatic compute mode while still allowing users to choose INT8 manually. This project made it very clear that a fast backend is useless if the output is wrong.

Accomplishments we're proud of

The biggest accomplishment is that MPS now works through the normal CTranslate2 API. Users do not need to rewrite their model, move their application into PyTorch, or change the overall structure of their inference pipeline. On an Apple M1 MacBook Air with a 7-core GPU, using a Release build and a real FP16 Marian Roman Pashto translation model, MetalTranslate achieved the following result: The throughput improvement was: $$ \frac{187.21}{114.32} \approx 1.64 $$ The latency reduction was: $$ \frac{306.15 - 186.96}{306.15} \times 100 \approx 38.9% $$ That means MetalTranslate achieved up to 1.64 times higher throughput and around 39 percent lower latency than optimized CPU inference on this workload. The final output was correct and deterministic. The backend passed all 175 MPS tests, including FP32, FP16, BF16, INT8, quantization, dequantization, quantized Conv1D, grouped Conv1D, transposed matrices, odd dimensions, batched operations, buffer offset handling, TopK, sampling, and CPU versus MPS translation comparisons. A separate CPU-only build also passed 190 tests. I am also proud that the project was driven by a real low-resource language use case. The model used during development was a Roman Pashto translation model. Pashto has far fewer local translation and speech tools than major languages. Making these models faster and easier to run on everyday Apple Silicon hardware has practical value beyond a synthetic benchmark.

What we learned

I learned that Apple unified memory does not remove the need for synchronization. The CPU and GPU may share physical memory, but the order in which operations happen still has to be controlled carefully. I learned that autoregressive inference is often limited by latency, memory movement, and command submission rather than theoretical GPU compute performance. I also learned that one incorrect broadcast or stale buffer can pass through many Transformer layers before finally destroying the output logits. Another important lesson was that INT8 is not automatically faster on every architecture or workload. Most importantly, I learned not to trust a performance result until the output is correct, deterministic, and tested against a known-good backend. Codex also changed how I approached the project. Instead of using it only to generate code, I used it to inspect the codebase, compare implementations, form hypotheses, debug incorrect output, write tests, analyze benchmarks, resolve merge conflicts, and rebase several months of work onto the latest CTranslate2 version.

What's next

MetalTranslate is functional, but there is still a lot of room for improvement. The next steps include better packed-weight layouts, more GEMV tuning for newer Apple chips, fused attention kernels, fused KV-cache operations, FlashAttention-style Metal kernels, larger-vocabulary GPU TopP, precompiled Metal libraries, and native Apple Silicon Python wheels. I also want to test the backend on M2, M3, and M4 devices. Another major goal is benchmarking Whisper and Faster Whisper workloads. Since Faster Whisper is built on CTranslate2, native MPS support could eventually allow many transcription tools to use the Apple GPU without switching frameworks. The long-term goal is simple: Apple Silicon should become a first-class inference target for CTranslate2. Built with Codex, C++, Objective-C++, Metal, Metal Performance Shaders, Metal Shading Language, Apple Silicon, CTranslate2, Python, Pybind11, CMake, GoogleTest, Marian, FP16, BF16, INT8 quantization, machine translation, natural language processing, GPU computing, and high-performance computing. Try it out Source code View the MetalTranslate source code MetalTranslate documentation Read the Apple Silicon MPS backend documentation Original CTranslate2 project View the original CTranslate2 project

CTranslate2

CTranslate2 is a C++ and Python library for efficient inference with Transformer models.

The project implements a custom runtime that applies many performance optimization techniques such as weights quantization, layers fusion, batch reordering, etc., to accelerate and reduce the memory usage of Transformer models on CPU and GPU.

[!NOTE] This fork is the home of MetalTranslate, a native Apple Metal/MPS backend for CTranslate2 built for the Codex hackathon. It enables correct, private, on-device Transformer inference on Apple Silicon and reached 1.64x the performance of the optimized CPU backend on the correctness-validated Marian translation workload described below. Jump to the Apple Silicon setup guide to try it.

The following model types are currently supported:

Encoder-decoder models: Transformer base/big, M2M-100, NLLB, BART, mBART, Pegasus, T5, Whisper, T5Gemma, T5Gemma2, MADLAD-400
Decoder-only models: GPT-2, GPT-J, GPT-NeoX, OPT, BLOOM, MPT, Llama, Mistral, Gemma, CodeGen, GPTBigCode, Falcon, Qwen2
Encoder-only models: BERT, DistilBERT, XLM-RoBERTa

Compatible models should be first converted into an optimized model format. The library includes converters for multiple frameworks:

The project is production-oriented and comes with backward compatibility guarantees, but it also includes experimental features related to model compression and inference acceleration.

Key features

Fast and efficient execution on CPU and GPUThe execution is significantly faster and requires less resources than general-purpose deep learning frameworks on supported models and tasks thanks to many advanced optimizations: layer fusion, padding removal, batch reordering, in-place operations, caching mechanism, etc.
Quantization and reduced precisionThe model serialization and computation support weights with reduced precision: 16-bit floating points (FP16), 16-bit brain floating points (BF16), 16-bit integers (INT16), 8-bit integers (INT8) and AWQ quantization (INT4).
Multiple CPU architectures supportThe project supports x86-64 and AArch64/ARM64 processors and integrates multiple backends that are optimized for these platforms: Intel MKL, oneDNN, OpenBLAS, Ruy, and Apple Accelerate.
Automatic CPU detection and code dispatchOne binary can include multiple backends (e.g. Intel MKL and oneDNN) and instruction set architectures (e.g. AVX, AVX2) that are automatically selected at runtime based on the CPU information.
Parallel and asynchronous executionMultiple batches can be processed in parallel and asynchronously using multiple GPUs or CPU cores.
Dynamic memory usageThe memory usage changes dynamically depending on the request size while still meeting performance requirements thanks to caching allocators on both CPU and GPU.
Lightweight on diskQuantization can make the models 4 times smaller on disk with minimal accuracy loss.
Simple integrationThe project has few dependencies and exposes simple APIs in Python and C++ to cover most integration needs.
Configurable and interactive decodingAdvanced decoding features allow autocompleting a partial sequence and returning alternatives at a specific location in the sequence.
Support tensor parallelism for distributed inferenceVery large model can be split into multiple GPUs. Following this documentation to set up the required environment.

Some of these features are difficult to achieve with standard deep learning frameworks and are the motivation for this project.

Installation and usage

CTranslate2 can be installed with pip:

pip install ctranslate2

The Python module is used to convert models and can translate or generate text with few lines of code:

translator = ctranslate2.Translator(translation_model_path)
translator.translate_batch(tokens)

generator = ctranslate2.Generator(generation_model_path)
generator.generate_batch(start_tokens)

See the documentation for more information and examples.

If you have an AMD ROCm GPU, we provide specific Python wheels on the releases page.

Apple Silicon MPS backend (experimental)

MetalTranslate is implemented directly inside the CTranslate2 runtime. It is not a wrapper around PyTorch: model layers dispatch through CTranslate2's C++ operator system into Objective-C++ and Metal kernels. The backend includes a persistent asynchronous command stream, decode-specific FP16 GEMV, tiled and batched GEMM, GPU search and sampling, reductions, layout operations, and quantized execution.

The MPS build is currently source-only and requires:

An Apple Silicon Mac running macOS 11 or newer
Xcode Command Line Tools (xcode-select --install)
CMake and a C++17 compiler
Python 3.9 or newer for the optional Python API

1. Clone and create a Python environment

git clone --recursive https://github.com/TBO22/CTranslate2.git
cd CTranslate2

python3 -m venv .venv
source .venv/bin/activate
python -m pip install --upgrade pip

[!IMPORTANT] CTranslate2 uses Git submodules for build dependencies such as cxxopts, googletest, and spdlog. GitHub's Download ZIP archive does not include their contents. Use the recursive clone command above for a source build.

If the repository was already cloned without --recursive, initialize the missing dependencies before running CMake:

git submodule update --init --recursive

If CMake reports that third_party/googletest has no CMakeLists.txt or that cxxopts is missing, the submodules were not initialized; the command above fixes both errors.

2. Build and install the C++ library

cmake -S . -B build-mps \
  -DCMAKE_BUILD_TYPE=Release \
  -DBUILD_TESTS=ON \
  -DWITH_MPS=ON \
  -DWITH_ACCELERATE=ON \
  -DWITH_MKL=OFF \
  -DOPENMP_RUNTIME=NONE
cmake --build build-mps -j 4
cmake --install build-mps --prefix "$PWD/install-mps"

WITH_MPS cannot be combined with CUDA or HIP in the same build.

3. Install the Python extension

The CTRANSLATE2_ROOT value makes the extension compile and link against the MPS-enabled library that was just installed. ARCHFLAGS prevents an invalid x86_64 slice from being added to the native Apple Silicon extension.

cd python
python -m pip install -r install_requirements.txt

export CTRANSLATE2_ROOT="$(cd ../install-mps && pwd)"
export ARCHFLAGS="-arch arm64"
python -m pip install -e .
cd ..

Verify the installation:

python -c 'import ctranslate2; print(ctranslate2.get_mps_device_count()); print(ctranslate2.get_supported_compute_types("mps"))'

On a supported Mac, the first value should be at least 1. The compute-type list should include float32, float16, bfloat16, and the INT8 hybrid modes.

4. Run inference

import ctranslate2

translator = ctranslate2.Translator(
    "path/to/converted/model",
    device="mps",
    compute_type="float16",
)

results = translator.translate_batch([["▁Hello", "▁world", "!"]])
print(results[0].hypotheses[0])

FP16 is the recommended compute type and is selected by compute_type="auto" on MPS. INT8 reduces model weight size, but is not necessarily faster on Apple GPUs, particularly during batch-size-1 decoding.

5. Run the MPS correctness tests

Metal API validation is useful during development because it catches invalid resource use and encoder mistakes:

MTL_DEBUG_LAYER=1 CT2_MPS_MAX_OPS=16 \
  ./build-mps/tests/ctranslate2_test tests/data \
  --gtest_filter='MPS/*:MPSBackendTest.*:TranslatorTest.MPS*'

The final validation run passed 175 of 175 MPS tests, including CPU-versus-MPS translation, FP32/FP16/BF16 operators, INT8 quantization, batched GEMM, TopK, sampling, and quantized grouped Conv1D. A separate CPU-only build passed 190 tests with 2 expected skips.

6. Benchmark and profile

Build-time benchmarks cover decode GEMM, prefill GEMM, argmax, and copy-heavy operations:

./build-mps/tests/benchmark_mps all

Set CT2_MPS_PROFILE=1 to print command-buffer, synchronization, dispatch, copy, GEMM-path, TopK, allocation, and buffer-lookup counters. See environment variables for all tuning and diagnostic options.

Correctness-validated result

This representative measurement used a Release build, an Apple M1 MacBook Air with a 7-core GPU, a real FP16 Marian Roman Pashto translation model, batch size 1, and greedy decoding:

Backend	Inference time	Throughput
Optimized CTranslate2 CPU	306.15 ms	114.32 tokens/s
MetalTranslate FP16	186.96 ms	187.21 tokens/s

The result is a 1.64x speedup and approximately 39% lower latency. Absolute performance varies by model, sequence length, search configuration, and Apple chip. Earlier results produced a larger number while generating incorrect tokens; those measurements were rejected rather than reported as a speedup.

Supported precision and operations

The backend supports FP32, FP16, BF16, and the int8_float32, int8_float16, and int8_bfloat16 hybrid compute types. BF16 values are stored in BF16 while GEMM and reduction accumulation use FP32. The INT8 path uses signed INT8 matrices, INT32 accumulation, per-row activation scales, and a fused dequantization/output kernel.

The backend keeps common decoding operations such as small TopK/argmax, TopP masking, multinomial/Gumbel sampling, ALiBi, median filtering, and quantized or dilated Conv1D on the GPU.

Current MPS limitations include FlashAttention, AWQ INT4, INT16 GEMM, distributed collectives, and packed/shifted-u8 INT8 GEMM. GPU TopP currently supports up to 1024 classes, and the optimized small TopK path supports k = 1, 2, 4, 8. Metal kernels are compiled on first use, so performance measurements should include warmup runs. See hardware support, quantization, and installation for additional details.

My research, and where Codex and GPT-5.6 helped

I started this port about six months before the hackathon, and most of the research behind it was work I had already done myself. I spent months reading Apple's MLX Metal backend, especially its matmul, matvec, command submission, tiling, and SIMD-group code. I compared that with the approaches used by GGML/llama.cpp and PyTorch MPS, then compared all of them with CTranslate2's CUDA backend and its own operator and tensor conventions.

Those implementations solve different problems. MLX was designed around Apple hardware from the beginning. GGML is heavily shaped by quantized LLM decoding and its own weight layouts. PyTorch MPS has to provide broad framework coverage. CUDA has mature libraries, streams, and a more explicit device-memory model. I could not just copy one of them into CTranslate2. I had to understand which ideas made sense for CTranslate2's StorageView, primitives, model loading, weight layouts, and autoregressive search.

I also spent a lot of time understanding unified memory. Apple Silicon lets the CPU and GPU use the same physical memory, but that does not make execution automatically synchronized. The CPU can still read data before the GPU has finished writing it. That distinction shaped the allocator, buffer registry, persistent command stream, and every place where the host genuinely needs a result back from Metal.

By the time I brought Codex and GPT-5.6 into the project, I was not starting from a blank prompt. I already had the backend direction, early kernels, and a real Roman Pashto model exposing the problems. What I needed was help turning that research into a complete integration without spending another six months moving through a large C++ codebase one file at a time.

I used Codex directly inside the repository as a pair programmer. I would give it a specific failure, a trace, or a benchmark result. It helped follow the call path across C++, CUDA, Objective-C++, and Metal, implement the next piece, compile it, add tests, and run the model again. This was especially useful for finishing missing operations such as Gather, extending the test matrix to odd shapes and batch strides, rebuilding the Python extension, resolving the 31-commit upstream gap, and cleaning up CI and documentation.

The optimization decisions still came from research and measurement on my machine. I tested command-buffer limits of 16, 32, 64, and 128. I compared the custom GEMV with the general matrix path. I found that INT8 worked but was slower than FP16 on my M1, so I kept FP16 as the automatic default. When an early run appeared more than twice as fast but produced repeated, corrupted words, I rejected that number and kept debugging until the translation was stable.

The fairest description is that I researched the architecture, chose the direction, and validated the result on my own hardware and models. Codex and GPT-5.6 helped me implement, debug, test, and finish that work much faster. This project came from my Metal research; Codex helped me finally turn it into a working CTranslate2 backend.

Web Server

ctranslate2-web-server is a web server built on top of CTranslate2 that exposes an OpenAI-compatible REST API, making it easy to integrate CTranslate2 models into applications that already support the OpenAI API.

Benchmarks

We translate the En->De test set newstest2014 with multiple models:

OpenNMT-tf WMT14: a base Transformer trained with OpenNMT-tf on the WMT14 dataset (4.5M lines)
OpenNMT-py WMT14: a base Transformer trained with OpenNMT-py on the WMT14 dataset (4.5M lines)
OPUS-MT: a base Transformer trained with Marian on all OPUS data available on 2020-02-26 (81.9M lines)

The benchmark reports the number of target tokens generated per second (higher is better). The results are aggregated over multiple runs. See the benchmark scripts for more details and reproduce these numbers.

Please note that the results presented below are only valid for the configuration used during this benchmark: absolute and relative performance may change with different settings.

CPU

	Tokens per second	Max. memory	BLEU
OpenNMT-tf WMT14 model
OpenNMT-tf 2.31.0 (with TensorFlow 2.11.0)	209.2	2653MB	26.93
OpenNMT-py WMT14 model
OpenNMT-py 3.0.4 (with PyTorch 1.13.1)	275.8	2012MB	26.77
- int8	323.3	1359MB	26.72
CTranslate2 3.6.0	658.8	849MB	26.77
- int16	733.0	672MB	26.82
- int8	860.2	529MB	26.78
- int8 + vmap	1126.2	598MB	26.64
OPUS-MT model
Transformers 4.26.1 (with PyTorch 1.13.1)	147.3	2332MB	27.90
Marian 1.11.0	344.5	7605MB	27.93
- int16	330.2	5901MB	27.65
- int8	355.8	4763MB	27.27
CTranslate2 3.6.0	525.0	721MB	27.92
- int16	596.1	660MB	27.53
- int8	696.1	516MB	27.65

Executed with 4 threads on a c5.2xlarge Amazon EC2 instance equipped with an Intel(R) Xeon(R) Platinum 8275CL CPU.

GPU

	Tokens per second	Max. GPU memory	Max. CPU memory	BLEU
OpenNMT-tf WMT14 model
OpenNMT-tf 2.31.0 (with TensorFlow 2.11.0)	1483.5	3031MB	3122MB	26.94
OpenNMT-py WMT14 model
OpenNMT-py 3.0.4 (with PyTorch 1.13.1)	1795.2	2973MB	3099MB	26.77
FasterTransformer 5.3	6979.0	2402MB	1131MB	26.77
- float16	8592.5	1360MB	1135MB	26.80
CTranslate2 3.6.0	6634.7	1261MB	953MB	26.77
- int8	8567.2	1005MB	807MB	26.85
- float16	10990.7	941MB	807MB	26.77
- int8 + float16	8725.4	813MB	800MB	26.83
OPUS-MT model
Transformers 4.26.1 (with PyTorch 1.13.1)	1022.9	4097MB	2109MB	27.90
Marian 1.11.0	3241.0	3381MB	2156MB	27.92
- float16	3962.4	3239MB	1976MB	27.94
CTranslate2 3.6.0	5876.4	1197MB	754MB	27.92
- int8	7521.9	1005MB	792MB	27.79
- float16	9296.7	909MB	814MB	27.90
- int8 + float16	8362.7	813MB	766MB	27.90

Executed with CUDA 11 on a g5.xlarge Amazon EC2 instance equipped with a NVIDIA A10G GPU (driver version: 510.47.03).

Contributing

CTranslate2 is a community-driven project. We welcome contributions of all kinds:

New Model Support: Help us implement more Transformer architectures.
Performance: Propose optimizations for CPU or GPU kernels.
Bug Reports: Open an issue if you find something not working as expected.
Documentation: Improve our guides or add new examples.

Check out our Contributing Guide to learn how to set up your development environment.

Additional resources

Analysis

Compare with all teams

View

Metric

Talha Bin Omar
11

Figures cover GitHub contributors during the hackathon window. A co-authored commit counts in full for each author, so per-member totals add up to more than the whole-team figures.

Technology

Found in codeClaimed only

CIn code
C++In code
CSSIn code
Hugging FaceIn code
PythonIn code
PyTorchIn code

6 of 6 appear in the indexed code.

AI coding agents

No AI coding agent signals were found in this repository.

Detected from committed agent config files and commit authorship. Absence of a signal is not proof an agent was unused.

Codebase size

Source size

3.3 MB

Source files

378

Counts recognized source files only; vendored directories, binaries and lockfiles are excluded, so this is smaller than the repository on disk.