# Project export: Ghost in the Machine This document was generated by HackStack to give an AI agent context about a hackathon project. Sections are labeled with their provenance; content marked as truncated was cut to keep this document small. ## Project metadata - Hackathon: TreeHacks 2026 - Tagline: I put myself in an rPi Zero. On an LLM. Trained from scratch. - Devpost: https://devpost.com/software/ghost-in-the-machine-4je80s - GitHub: https://github.com/matthewover137/pi-ris-public - Video: https://www.youtube.com/embed/KBqjMb9z13o?enablejsapi=1&hl=en_US&rel=0&start=&version=3&wmode=transparent - Team: 1 GitHub contributor(s) — Matthew Li (2 commits) ## Devpost submission (written by the team) ### Inspiration It's Valentines weekend and I'm not able to spend it with my girlfriend. Therefore, I decided to make her a funny little gift: a clone of myself (based on our 100k+ messages). I thought some kind of LLM would be a good place to start, and because she doesn't have much coding experience or access to hardware, I decided to make this LLM run on the cheapest computer around: the Raspberry-Pi Zero (512MB RAM, $15). I don't think you can get much smaller than this. On a broader scale, this project explores how small we can make language models so they can still learn domain specific knowledge, and what the bare minimum hardware is for serving these language models. I believe these to be important questions because they provide answers to how we can distribute LLMs at fraction of their cost today. This is a key step in democratizing AI technology. ### What it does I put myself in a Raspberry Pi! You plug in a Raspberry Pi containing an LLM to your computer and you can converse with it, not like a chatbot, but like an actual human: It will talk to you if you leave it alone for too long. It won't automatically respond to every message you send. This is achieved by: Training a 15M parameter LLM from-scratch on my text messages to mimic my conversations with my girlfriend. Creating the protocol to inference it (in C) on a bare-metal (no operating system) Raspberry-Pi Zero. Making a minimalistic messaging webapp (mostly just for demonstration purposes). ### How we built it 1. Training the Model Data Preparation: Preparing 1.5 years of Facebook Messenger messages with my girlfriend. Synthetic Data: Generating synthetic conversation data using Gemini-2.0-Flash to make more training examples (~20k messages total). Pipeline: Using training pipelines from the llama2.c repository (Ack) to train a custom tokenizer and the model. Optimization: Quantizing the model to fp8 for inference speed. 2. Bare-Metal Inference Bootloader: Making bootloader code to quickly load the model into Pi RAM. Inference: Adapting inference code from llama2.c to be able to do forward passes on the Pi. Communication: Creating a UART communication protocol to actually simulate messaging someone with the model outputs. ### Challenges we ran into I can bucket the main challenge of this project into roughly 3 places: Model Goodness: Getting the model to produce coherent English and proper turn-based dialogue. Speed of Inference: Getting the model to run fast enough on the Pi. Connecting everything together: Handling hardware/software integration. Here are some things I found interesting for each: Skill Expression in Data: I tested extensively with training 15M and 42M parameter models with many different sweeps of hyperparameters. I ended up feeling like there was not much skill expression here. HOWEVER, I found a lot of skill expression in how the dataset was made. I cleaned 1.5 years of messenger data (80k messages). Initially, just sampling from true messages, I found the model was very incoherent until I refined the cleaning process. Inference Constraints: This was probably the biggest hurdle. I needed to throw out the idea of models larger than 15M parameters because it didn't make sense to inference them on the Pi. I ended up partially solving this by quantizing my models (into fp8) and training a custom tokenizer (vocab size 2048). One forward pass takes ~2.2s still D: Race Conditions: Many race conditions appeared regarding the user messages and how the Pi is supposed to respond, especially if user messages are sent while the Pi is in the middle of generating tokens. I solved this by developing a handshaking protocol that standardizes communication between the Pi and my computer. ### Accomplishments we're proud of Learning about LLMs and pulling everything together. ### What we learned This was my first time training an LLM. It was amazing to learn how developed the model training ecosystem really is, between having to write very little actual training code because PyTorch wraps everything so well, and having unbelievable insight into my training process with just a few lines of code using WandB. These tools really amazed me. Also, it was very cool to witness firsthand how my data turned randomly initialized weights into something with structure. On one hand, I understand the "black box" sentiment more now, because I literally can't comprehend that the model learned my data so well even though I intentionally chose the simplest training pipeline possible. On the other hand, it was cool to learn that I still have some control. For instance, it was really cool to see my validation loss curve invert itself when I added some dropout. I also learned a lot regarding how the Transformer actually works. One thing I learned that turns out to be of extreme significance to my project is how the KV cache is computed and saved. When turning an LLM into a chat application, I figured out that you can save a lot on inference time by checkpointing your KV cache (and not recomputing it needlessly). Ultimately there is probably a lot more hiding here in terms of unlocking faster inference. There are actually so many cool things I learned this weekend. Please come ask me about this if you want to hear more. ### What's next Model Training I was unable to download my Facebook Messenger data from the past year in time to train my model with this data. I feel like this is a significant chunk of conversational data I have that's real, so I'm eager to retrain once this finally downloads. I also wonder how training on a more general conversation dataset first and then fine-tuning with only certain layers unfrozen would change things. Also, I'm generally curious what the pitfalls were of my synthetic data generation and how much I can scale this. Speedier Inference There's definitely so much to unlock here. I did not rewrite much of Karpathy's original inference code to conform to my use case, and I am aware of the repo llama.cpp which has a lot of speedup tricks which I want to take a look at. In terms of Pi stuff, the Pi Zero does have a GPU (maybe not in the traditional sense but still), and maybe it will be faster at doing the matmuls. I feel like I still need a better understanding of how the transformer actually works, so it will be good to step through this after TreeHacks. Acknowledgements llama2.c repo - training codes, would not be possible without it. Eric Chen - guidance during model training process. Iris Nguyen - Idea + data :) ## README (from the GitHub repository) # Ghost in the Machine [![Demo Video](https://img.youtube.com/vi/KBqjMb9z13o/maxresdefault.jpg)](https://youtu.be/KBqjMb9z13o) I put myself in a Raspberry Pi. This is a 15M parameter LLM trained from scratch on my text messages with my girlfriend, running bare-metal (no operating system) on a Raspberry Pi Zero (512MB RAM, $15). You plug the Pi into your computer and chat with it through a messaging webapp. It'll message you if you leave it alone for too long, and it won't automatically respond to every message you send. This is the public version of this repo (without my messages). If you want to see the full commit history or private data, come ask me. ## Why It's Valentine's weekend and I can't spend it with my girlfriend, so I made a clone of myself based on our 100k+ messages. I thought an LLM would be a good starting point, and because she doesn't have much coding experience or access to hardware, I wanted this to run on the cheapest computer around. On a broader scale: how small can we make language models that still learn domain-specific knowledge, and what's the bare minimum hardware to serve them? These are important questions for distributing LLMs at a fraction of their current cost. ## How It Works Three pieces: 1. **Training**: Fine-tune a 15M param Llama 2-style model on my text messages using PyTorch. 2. **Inference**: Run the quantized (int8) model in C on a bare-metal Raspberry Pi Zero. 3. **Webapp**: A messaging UI connected to the Pi over serial (UART) via a Python bridge. ``` Chat History → Tokenize → Fine-tune → Export → Bake into Pi kernel ↓ Raspberry Pi (bare metal) ↓ (serial @ 115200 baud) Host PC (server.py) ↓ (WebSocket) Browser (index.html) ``` ## Setup ### Prerequisites - `arm-none-eabi-gcc` (for cross-compiling to Pi) - A Raspberry Pi Zero (or similar) with a serial connection to your host ### 1. Training the Model Everything lives in `llama2.c/`. **Install dependencies:** ```bash cd llama2.c pip install -r requirements.txt ``` **Download a base model** (pre-trained on TinyStories): ```bash # 15M params (fits in Pi RAM after quantization) wget https://huggingface.co/karpathy/tinyllamas/resolve/main/stories15M.pt # 42M params (if you're not targeting the Pi) wget https://huggingface.co/karpathy/tinyllamas/resolve/main/stories42M.pt -P finetunes ``` **Prepare your data:** Put your cleaned chat history in `private/chat_history_train.txt` and `private/chat_history_val.txt`. Format is lines prefixed with `I:` (you) and `M:` (the other person): ``` I: hey what's up M: not much just got home I: nice want to get food M: yes please I'm starving ``` Then tokenize and preprocess: ```bash python chat_history.py pretokenize python preprocess_chat.py --input_dir ../private --output_dir ../private ``` This creates binary token files and turn-offset indices so the training loop can sample from complete conversation turns (not random positions in the middle of a sentence). **Fine-tune:** On CUDA: ```bash python finetune_msg.py \ --device=cuda \ --compile=True \ --batch_size=64 \ --warmup_iters=500 \ --learning_rate=1e-5 \ --dropout=0.1 \ --wandb_log=True \ --wandb_project=pi-ris \ --wandb_run_name=run_nn ``` On Apple Silicon (MPS): ```bash python finetune_msg.py \ --device=mps \ --compile=False \ --batch_size=32 \ --wandb_log=True \ --wandb_project=pi-ris \ --wandb_run_name=run_nn ``` The model fine-tunes from a TinyStories checkpoint (`stories15M.pt` by default). To use 42M params instead, pass `--ckpt_path=stories42M.pt`. Output checkpoint lands in `finetunes/ckpt.pt`. **Test your model:** ```bash python sample.py \ --checkpoint=finetunes/ckpt.pt \ --device=cuda \ --compile=True \ --num_samples=5 \ --max_new_tokens=100 \ --start="I: Hello\nM:" ``` ### 2. Deploying to the Pi The bare-metal inference code lives in `pi-baremetal/inference-server/`. **Export your model to binary:** ```bash python export.py finetunes/ckpt.pt ``` This creates a quantized `.bin` file that the C inference engine can read. **Build the kernel image:** ```bash cd pi-baremetal/inference-server make MODEL=path/to/your/quantized_model.bin ``` This does a few things: - Compiles `main.c` (protocol handler) and `llama2.c` (inference engine) with libpi - Uses `objcopy` to bake the model weights and tokenizer directly into the kernel binary - Outputs `kernel.img` (~17-18 MB) **Flash to SD card:** Copy `kernel.img` to your Pi's SD card (alongside `bootcode.bin` and `start.elf` from the Raspberry Pi firmware). The bootloader loads the kernel into RAM at `0x8000` and starts inference. One forward pass takes ~2.2s on the Pi Zero. Not fast, but it works. ### 3. Running the Webapp The webapp lives in `webapp/`. **Install dependencies:** ```bash cd webapp pip install -r requirements.txt ``` **Start the bridge:** ```bash python server.py /dev/ttyUSB0 115200 ``` (Replace `/dev/ttyUSB0` with your actual serial port — on Mac it's usually something like `/dev/tty.usbserial-*`.) This starts: - An HTTP server on `http://localhost:8080` (serves the chat UI) - A WebSocket server on `ws://localhost:8765` (real-time comms) - A serial bridge to the Pi **Open the chat:** Go to `http://localhost:8080` in your browser. The UI mimics iMessage. It'll show a loading state until the Pi handshake completes, then you can start chatting. **Debug tool:** If something's weird with the serial connection: ```bash python debug_serial.py # listen mode (hex + ASCII dump) python debug_serial.py --send # handshake + send a test prompt python debug_serial.py --raw # raw mode (no protocol) ``` ## Architecture Details ### Model - Llama 2-style transformer (from [llama2.c](https://github.com/karpathy/llama2.c)) - 288-dim, 6 layers, 6 heads - Vocab size: 32,000 (Llama 2 SentencePiece tokenizer) - Sequence length: 256 tokens - Quantized to int8 for inference (4x smaller weights) - Pre-trained on TinyStories, fine-tuned on ~20k chat messages ### Bare-Metal Inference - Pure C, no OS, no stdlib (custom libpi) - Hardware floating point (VFP) enabled - KV cache kept in float32 for quality, weights in int8 for size - Model baked directly into kernel binary via `objcopy` - Custom UART protocol with handshaking, cancel support, and retry logic ### Communication Protocol The Pi and host communicate over serial with a custom protocol: 1. **Handshake**: Pi sends `READY` every 500ms until host responds with `ACK` 2. **Prompt**: Host sends `STX + prompt + ETX`, Pi responds with `OK` 3. **Generation**: Pi sends `TYPING` → `MSG:` → `IDLE` as it generates 4. **Cancel**: Host sends `CAN` byte, Pi acknowledges and stops The protocol handles race conditions (messages sent mid-generation) and retries (if the model doesn't produce a valid `M:` response). ## Things I Learned - There's way more skill expression in how you prepare data than in hyperparameter tuning. My model went from incoherent garbage to passable English mostly by cleaning the training data better. - The KV cache is everything. You can checkpoint it between turns instead of recomputing from scratch, which saves a ton of inference time. - Watching randomly initialized weights learn my actual speech patterns was wild. I still can't fully explain how the simplest possible training pipeline produced something that sounds like me. - Dropout actually works. My val loss curve literally inverted itself when I added some. ## What's Next - **More data**: I couldn't download my last year of Messenger data in time. That's a significant chunk. - **Faster inference**: Haven't really optimized the C code beyond quantization. The Pi Zero has a GPU of sorts [README truncated for size] ## Detected evidence (automated analysis) Indexed codebase: 150 recognized source files, 558 KB. - C (language) — detected in the code - FastAPI (technology) — detected in the code - HTML (language) — detected in the code - Hugging Face (technology) — detected in the code - Python (language) — detected in the code - PyTorch (technology) — detected in the code - Streamlit (technology) — detected in the code ## Codebase structure (from repository index) ### Files (120 of 295) ``` .DS_Store .gitignore llama2.c/.DS_Store llama2.c/.github/workflows/build.yml llama2.c/chat_history.py llama2.c/configurator.py llama2.c/doc/stories260K.md llama2.c/doc/train_llama_tokenizer.md llama2.c/export.py llama2.c/finetune_msg.py llama2.c/gemini_generate.py llama2.c/LICENSE llama2.c/Makefile llama2.c/model.py llama2.c/preprocess_chat.py llama2.c/README.md llama2.c/requirements.txt llama2.c/run llama2.c/run.c llama2.c/run.ipynb llama2.c/runq llama2.c/runq.c llama2.c/sample.py llama2.c/test_all.py llama2.c/test.c llama2.c/tinystories.py llama2.c/tokenizer.model llama2.c/tokenizer.py llama2.c/train.py llama2.c/wandb_data/.DS_Store llama2.c/wandb_data/debug-internal.log llama2.c/wandb_data/debug.log llama2.c/wandb_data/latest-run llama2.c/wandb_data/run-20260214_053939-ps1xwflu/files/config.yaml llama2.c/wandb_data/run-20260214_053939-ps1xwflu/files/output.log llama2.c/wandb_data/run-20260214_053939-ps1xwflu/files/wandb-metadata.json llama2.c/wandb_data/run-20260214_053939-ps1xwflu/files/wandb-summary.json llama2.c/wandb_data/run-20260214_053939-ps1xwflu/logs/debug-core.log llama2.c/wandb_data/run-20260214_053939-ps1xwflu/logs/debug-internal.log llama2.c/wandb_data/run-20260214_053939-ps1xwflu/logs/debug.log llama2.c/wandb_data/run-20260214_053939-ps1xwflu/run-ps1xwflu.wandb llama2.c/wandb_data/run-20260214_055907-3vq9dqfq/files/config.yaml llama2.c/wandb_data/run-20260214_055907-3vq9dqfq/files/output.log llama2.c/wandb_data/run-20260214_055907-3vq9dqfq/files/wandb-metadata.json llama2.c/wandb_data/run-20260214_055907-3vq9dqfq/files/wandb-summary.json llama2.c/wandb_data/run-20260214_055907-3vq9dqfq/logs/debug-core.log llama2.c/wandb_data/run-20260214_055907-3vq9dqfq/logs/debug-internal.log llama2.c/wandb_data/run-20260214_055907-3vq9dqfq/logs/debug.log llama2.c/wandb_data/run-20260214_055907-3vq9dqfq/run-3vq9dqfq.wandb llama2.c/wandb/.DS_Store llama2.c/wandb/debug-cli.ubuntu.log llama2.c/wandb/debug-internal.log llama2.c/wandb/debug.log llama2.c/wandb/latest-run llama2.c/wandb/offline-run-20260215_014033-d0ihevn1/logs/debug-core.log llama2.c/wandb/offline-run-20260215_014033-d0ihevn1/logs/debug-internal.log llama2.c/wandb/offline-run-20260215_014033-d0ihevn1/logs/debug.log llama2.c/wandb/offline-run-20260215_014033-d0ihevn1/run-d0ihevn1.wandb llama2.c/wandb/offline-run-20260215_014228-vyq1pcgk/logs/debug-core.log llama2.c/wandb/offline-run-20260215_014228-vyq1pcgk/logs/debug-internal.log llama2.c/wandb/offline-run-20260215_014228-vyq1pcgk/logs/debug.log llama2.c/wandb/offline-run-20260215_014228-vyq1pcgk/run-vyq1pcgk.wandb llama2.c/wandb/run-20260214_053939-ps1xwflu/files/requirements.txt llama2.c/wandb/run-20260214_055907-3vq9dqfq/files/requirements.txt llama2.c/wandb/run-20260214_151151-t713v9yq/files/config.yaml llama2.c/wandb/run-20260214_151151-t713v9yq/files/output.log llama2.c/wandb/run-20260214_151151-t713v9yq/files/wandb-metadata.json llama2.c/wandb/run-20260214_151151-t713v9yq/files/wandb-summary.json llama2.c/wandb/run-20260214_151151-t713v9yq/logs/debug-core.log llama2.c/wandb/run-20260214_151151-t713v9yq/logs/debug-internal.log llama2.c/wandb/run-20260214_151151-t713v9yq/logs/debug.log llama2.c/wandb/run-20260214_151151-t713v9yq/run-t713v9yq.wandb llama2.c/wandb/run-20260214_153214-1m9km32j/files/config.yaml llama2.c/wandb/run-20260214_153214-1m9km32j/files/output.log llama2.c/wandb/run-20260214_153214-1m9km32j/files/wandb-metadata.json llama2.c/wandb/run-20260214_153214-1m9km32j/files/wandb-summary.json llama2.c/wandb/run-20260214_153214-1m9km32j/logs/debug-core.log llama2.c/wandb/run-20260214_153214-1m9km32j/logs/debug-internal.log llama2.c/wandb/run-20260214_153214-1m9km32j/logs/debug.log llama2.c/wandb/run-20260214_153214-1m9km32j/run-1m9km32j.wandb llama2.c/wandb/run-20260214_153643-xyku14yq/files/config.yaml llama2.c/wandb/run-20260214_153643-xyku14yq/files/output.log llama2.c/wandb/run-20260214_153643-xyku14yq/files/wandb-metadata.json llama2.c/wandb/run-20260214_153643-xyku14yq/files/wandb-summary.json llama2.c/wandb/run-20260214_153643-xyku14yq/logs/debug-core.log llama2.c/wandb/run-20260214_153643-xyku14yq/logs/debug-internal.log llama2.c/wandb/run-20260214_153643-xyku14yq/logs/debug.log llama2.c/wandb/run-20260214_153643-xyku14yq/run-xyku14yq.wandb llama2.c/wandb/run-20260214_154135-6z1ep46l/files/config.yaml llama2.c/wandb/run-20260214_154135-6z1ep46l/files/output.log llama2.c/wandb/run-20260214_154135-6z1ep46l/files/wandb-metadata.json llama2.c/wandb/run-20260214_154135-6z1ep46l/files/wandb-summary.json llama2.c/wandb/run-20260214_154135-6z1ep46l/logs/debug-core.log llama2.c/wandb/run-20260214_154135-6z1ep46l/logs/debug-internal.log llama2.c/wandb/run-20260214_154135-6z1ep46l/logs/debug.log llama2.c/wandb/run-20260214_154135-6z1ep46l/run-6z1ep46l.wandb llama2.c/wandb/run-20260214_161002-ul66jpqy/files/config.yaml llama2.c/wandb/run-20260214_161002-ul66jpqy/files/output.log llama2.c/wandb/run-20260214_161002-ul66jpqy/files/wandb-metadata.json llama2.c/wandb/run-20260214_161002-ul66jpqy/files/wandb-summary.json llama2.c/wandb/run-20260214_161002-ul66jpqy/logs/debug-core.log llama2.c/wandb/run-20260214_161002-ul66jpqy/logs/debug-internal.log llama2.c/wandb/run-20260214_161002-ul66jpqy/logs/debug.log llama2.c/wandb/run-20260214_161002-ul66jpqy/run-ul66jpqy.wandb llama2.c/wandb/run-20260214_162215-eqngabo3/files/config.yaml llama2.c/wandb/run-20260214_162215-eqngabo3/files/output.log llama2.c/wandb/run-20260214_162215-eqngabo3/files/wandb-metadata.json llama2.c/wandb/run-20260214_162215-eqngabo3/files/wandb-summary.json llama2.c/wandb/run-20260214_162215-eqngabo3/logs/debug-core.log llama2.c/wandb/run-20260214_162215-eqngabo3/logs/debug-internal.log llama2.c/wandb/run-20260214_162215-eqngabo3/logs/debug.log llama2.c/wandb/run-20260214_162215-eqngabo3/run-eqngabo3.wandb llama2.c/wandb/run-20260215_013357-h1k929vh/files/config.yaml llama2.c/wandb/run-20260215_013357-h1k929vh/files/output.log llama2.c/wandb/run-20260215_013357-h1k929vh/files/wandb-metadata.json llama2.c/wandb/run-20260215_013357-h1k929vh/files/wandb-summary.json llama2.c/wandb/run-20260215_013357-h1k929vh/logs/debug-core.log llama2.c/wandb/run-20260215_013357-h1k929vh/logs/debug-internal.log llama2.c/wandb/run-20260215_013357-h1k929vh/logs/debug.log llama2.c/wandb/run-20260215_013357-h1k929vh/run-h1k929vh.wandb [175 more files omitted for size] ``` ### Dependencies - llama2.c/requirements.txt: numpy@==1.23.5, pytest@==7.4.0, Requests@==2.31.0, sentencepiece@==0.1.99, torch@==2.0.1, tqdm@==4.64.1, wandb@==0.15.5 - llama2.c/wandb/run-20260214_053939-ps1xwflu/files/requirements.txt: altair@==5.5.0, annotated-types@==0.7.0, antlr4-python3-runtime@==4.9.3, anyio@==3.7.1, appnope@==0.1.4, arviz@==0.23.1, asttokens@==3.0.0, attrs@==25.4.0, autogluon.common@==1.4.0, autogluon.core@==1.4.0, autogluon.features@==1.4.0, autogluon.tabular@==1.4.0, backoff@==2.2.1, beautifulsoup4@==4.14.3, blinker@==1.9.0, boto3@==1.42.0, botocore@==1.41.6, cachetools@==6.2.1, certifi@==2025.8.3, cffi@==2.0.0, charset-normalizer@==3.4.3, click@==8.3.0, cloudpickle@==3.1.2, comm@==0.2.3, cons@==0.4.7, contourpy@==1.3.3, cryptography@==44.0.2, cycler@==0.12.1, DateTime@==5.5, debugpy@==1.8.17, decorator@==5.2.1, distro@==1.9.0, einops@==0.8.1, einx@==0.3.0, etuples@==0.3.10, eval_type_backport@==0.3.0, executing@==2.2.1, fastapi@==0.104.1, filelock@==3.20.0, fonttools@==4.60.0, frozendict@==2.4.7, fsspec@==2025.9.0, gitdb@==4.0.12, GitPython@==3.1.45, h11@==0.16.0, h5netcdf@==1.8.0, h5py@==3.15.1, hf-xet@==1.2.0, httptools@==0.7.1, huggingface-hub@==0.36.0, idna@==3.10, ipykernel@==6.30.1, ipython@==9.5.0, ipython_pygments_lexers@==1.1.1, ipywidgets@==8.1.8, jedi@==0.19.2, Jinja2@==3.1.6, jmespath@==1.0.1, joblib@==1.5.2, jsonschema@==4.25.1, jsonschema-specifications@==2025.9.1, jupyter_client@==8.6.3, jupyter_core@==5.8.1, jupyterlab_widgets@==3.0.16, kalshi-python@==2.1.4, kditransform@==1.2.0, kiwisolver@==1.4.9, lazy_imports@==1.0.1, lightgbm@==4.6.0, llvmlite@==0.45.1, logical-unification@==0.4.7, loguru@==0.7.3, lxml@==6.0.2, markdown-it-py@==4.0.0, MarkupSafe@==3.0.3, matplotlib@==3.10.6, matplotlib-inline@==0.1.7, mdurl@==0.1.2, miniKanren@==1.0.5, mpmath@==1.3.0, multipledispatch@==1.0.0, narwhals@==2.8.0, nest-asyncio@==1.6.0, networkx@==3.5, numba@==0.62.1, numpy@==2.3.3, omegaconf@==2.3.0, packaging@==25.0, pandas@==2.3.3, parso@==0.8.5, patsy@==1.0.2, pexpect@==4.9.0, pillow@==11.3.0, pip@==25.2, platformdirs@==4.4.0, posthog@==6.9.3, prompt_toolkit@==3.0.52, protobuf@==6.33.0, psutil@==7.0.0, ptyprocess@==0.7.0, pure_eval@==0.2.3, pyarrow@==20.0.0, pybaseball@==2.2.7, pycparser@==2.23, pydantic@==2.11.9, pydantic_core@==2.33.2, pydantic-settings@==2.12.0, pydeck@==0.9.1, PyGithub@==2.8.1, Pygments@==2.19.2, PyJWT@==2.10.1, pymc@==5.27.0, PyNaCl@==1.6.1, pyobjc-core@==12.1, pyobjc-framework-Cocoa@==12.1, pyobjc-framework-Metal@==12.1, pyparsing@==3.2.5, pyreadr@==0.5.4, pyserial@==3.5, pytensor@==2.36.3, python-dateutil@==2.9.0.post0, python-dotenv@==1.0.1, pytz@==2025.2, PyYAML@==6.0.3, pyzmq@==27.1.0, referencing@==0.37.0, regex@==2025.11.3, requests@==2.32.5, rich@==14.2.0, rpds-py@==0.27.1, s3transfer@==0.16.0, safetensors@==0.7.0, scikit-learn@==1.7.2, scipy@==1.16.2, seaborn@==0.13.2, sentencepiece@==0.2.1, sentry-sdk@==2.52.0, setuptools@==80.9.0, six@==1.17.0, smmap@==5.0.2, sniffio@==1.3.1, soupsieve@==2.8, stack-data@==0.6.3, starlette@==0.27.0, statsmodels@==0.14.6, streamlit@==1.50.0, sympy@==1.14.0, tabpfn@==6.0.6, tabpfn-common-utils@==0.2.10, tenacity@==9.1.2, threadpoolctl@==3.6.0, tokenizers@==0.22.1, toml@==0.10.2, toolz@==1.1.0, torch@==2.7.1, tornado@==6.5.2, tqdm@==4.67.1, traitlets@==5.14.3, transformers@==4.57.3, typing_extensions@==4.15.0, typing-inspection@==0.4.1, tzdata@==2025.2, urllib3@==2.3.0, uv@==0.9.13, uvicorn@==0.24.0, uvloop@==0.22.1, wandb@==0.25.0, watchfiles@==1.1.1, wcwidth@==0.2.14, websockets@==12.0, widgetsnbextension@==4.0.15, xarray@==2025.12.0, xarray-einstats@==0.9.1, xgboost@==3.1.2, xmodem@==0.4.7, zope.interface@==8.0.1 - llama2.c/wandb/run-20260214_055907-3vq9dqfq/files/requirements.txt: altair@==5.5.0, annotated-types@==0.7.0, antlr4-python3-runtime@==4.9.3, anyio@==3.7.1, appnope@==0.1.4, arviz@==0.23.1, asttokens@==3.0.0, attrs@==25.4.0, autogluon.common@==1.4.0, autogluon.core@==1.4.0, autogluon.features@==1.4.0, autogluon.tabular@==1.4.0, backoff@==2.2.1, beautifulsoup4@==4.14.3, blinker@==1.9.0, boto3@==1.42.0, botocore@==1.41.6, cachetools@==6.2.1, certifi@==2025.8.3, cffi@==2.0.0, charset-normalizer@==3.4.3, click@==8.3.0, cloudpickle@==3.1.2, comm@==0.2.3, cons@==0.4.7, contourpy@==1.3.3, cryptography@==44.0.2, cycler@==0.12.1, DateTime@==5.5, debugpy@==1.8.17, decorator@==5.2.1, distro@==1.9.0, einops@==0.8.1, einx@==0.3.0, etuples@==0.3.10, eval_type_backport@==0.3.0, executing@==2.2.1, fastapi@==0.104.1, filelock@==3.20.0, fonttools@==4.60.0, frozendict@==2.4.7, fsspec@==2025.9.0, gitdb@==4.0.12, GitPython@==3.1.45, h11@==0.16.0, h5netcdf@==1.8.0, h5py@==3.15.1, hf-xet@==1.2.0, httptools@==0.7.1, huggingface-hub@==0.36.0, idna@==3.10, ipykernel@==6.30.1, ipython@==9.5.0, ipython_pygments_lexers@==1.1.1, ipywidgets@==8.1.8, jedi@==0.19.2, Jinja2@==3.1.6, jmespath@==1.0.1, joblib@==1.5.2, jsonschema@==4.25.1, jsonschema-specifications@==2025.9.1, jupyter_client@==8.6.3, jupyter_core@==5.8.1, jupyterlab_widgets@==3.0.16, kalshi-python@==2.1.4, kditransform@==1.2.0, kiwisolver@==1.4.9, lazy_imports@==1.0.1, lightgbm@==4.6.0, llvmlite@==0.45.1, logical-unification@==0.4.7, loguru@==0.7.3, lxml@==6.0.2, markdown-it-py@==4.0.0, MarkupSafe@==3.0.3, matplotlib@==3.10.6, matplotlib-inline@==0.1.7, mdurl@==0.1.2, miniKanren@==1.0.5, mpmath@==1.3.0, multipledispatch@==1.0.0, narwhals@==2.8.0, nest-asyncio@==1.6.0, networkx@==3.5, numba@==0.62.1, numpy@==2.3.3, omegaconf@==2.3.0, packaging@==25.0, pandas@==2.3.3, parso@==0.8.5, patsy@==1.0.2, pexpect@==4.9.0, pillow@==11.3.0, pip@==25.2, platformdirs@==4.4.0, posthog@==6.9.3, prompt_toolkit@==3.0.52, protobuf@==6.33.0, psutil@==7.0.0, ptyprocess@==0.7.0, pure_eval@==0.2.3, pyarrow@==20.0.0, pybaseball@==2.2.7, pycparser@==2.23, pydantic@==2.11.9, pydantic_core@==2.33.2, pydantic-settings@==2.12.0, pydeck@==0.9.1, PyGithub@==2.8.1, Pygments@==2.19.2, PyJWT@==2.10.1, pymc@==5.27.0, PyNaCl@==1.6.1, pyobjc-core@==12.1, pyobjc-framework-Cocoa@==12.1, pyobjc-framework-Metal@==12.1, pyparsing@==3.2.5, pyreadr@==0.5.4, pyserial@==3.5, pytensor@==2.36.3, python-dateutil@==2.9.0.post0, python-dotenv@==1.0.1, pytz@==2025.2, PyYAML@==6.0.3, pyzmq@==27.1.0, referencing@==0.37.0, regex@==2025.11.3, requests@==2.32.5, rich@==14.2.0, rpds-py@==0.27.1, s3transfer@==0.16.0, safetensors@==0.7.0, scikit-learn@==1.7.2, scipy@==1.16.2, seaborn@==0.13.2, sentencepiece@==0.2.1, sentry-sdk@==2.52.0, setuptools@==80.9.0, six@==1.17.0, smmap@==5.0.2, sniffio@==1.3.1, soupsieve@==2.8, stack-data@==0.6.3, starlette@==0.27.0, statsmodels@==0.14.6, streamlit@==1.50.0, sympy@==1.14.0, tabpfn@==6.0.6, tabpfn-common-utils@==0.2.10, tenacity@==9.1.2, threadpoolctl@==3.6.0, tokenizers@==0.22.1, toml@==0.10.2, toolz@==1.1.0, torch@==2.7.1, tornado@==6.5.2, tqdm@==4.67.1, traitlets@==5.14.3, transformers@==4.57.3, typing_extensions@==4.15.0, typing-inspection@==0.4.1, tzdata@==2025.2, urllib3@==2.3.0, uv@==0.9.13, uvicorn@==0.24.0, uvloop@==0.22.1, wandb@==0.25.0, watchfiles@==1.1.1, wcwidth@==0.2.14, websockets@==12.0, widgetsnbextension@==4.0.15, xarray@==2025.12.0, xarray-einstats@==0.9.1, xgboost@==3.1.2, xmodem@==0.4.7, zope.interface@==8.0.1 - webapp/requirements.txt: pyserial@>=3.5, websockets@>=12.0 ### Recent commits (newest first) - readme - redme - init public ## Key source files (fetched from GitHub, selected and truncated for size) ### llama2.c/doc/stories260K.md ```markdown # stories260K [Stories260K huggginface link](https://huggingface.co/karpathy/tinyllamas) The 260K model is a tiny model used for testing, and was trained as follows: ``` python train.py \ --out_dir="outmini" \ --batch_size=128 \ --max_seq_len=512 \ --gradient_accumulation_steps=1 \ --vocab_source="custom" \ --vocab_size=512 \ --dim=64 \ --n_layers=5 \ --n_heads=8 \ --n_kv_heads=4 \ --multiple_of=4 \ --learning_rate=1e-3 \ --dropout=0.05 \ --weight_decay=0.01 \ --max_iters=100000 \ --beta2=0.99 \ --warmup_iters=1000 \ --eval_interval=2000 \ --eval_iters=100 \ --compile=True ``` You'll notice that `n_kv_heads` is 4 while `n_heads` is 8, so two heads at a time share their key,value projections, i.e. this model is 2X multiquery. You'll also notice that we're using a custom tokenizer with 512 tokens. The model trained for ~10 minutes (?) on my A100 and achieves validation loss of 1.2968. Sampling this model at temperature 0.0 (i.e. deterministic greedy argmax sampling) gives: ``` $ ./run stories260K/stories260K.bin -z stories260K/tok512.bin -t 0.0 Once upon a time, there was a little girl named Lily. She loved to play outside in the park. One day, she saw a big, red ball. She wanted to play with it, but it was too high. Lily's mom said, "Lily, let's go to the park." Lily was sad and didn't know what to do. She said, "I want to play with your ball, but I can't find it." Lily was sad and didn't know what to do. She said, "I'm sorry, Lily. I didn't know what to do." Lily didn't want to help her mom, so she said, "I'm sorry, mom. I didn't know what to do." Her mom said, "Don't worry, Lily. We can help you. ``` You can reproduce the same in Python by running `sample.py`: ``` $ python sample.py --checkpoint=stories260K/stories260K.pt --tokenizer=stories260K/tok512.model --temperature=0.0 --max_new_tokens=257 ``` I hardcoded max tokens to be 257 manually because the `sample.py` script doesn't currently terminate on the special BOS token like the run.c script does. Sampling at 1.0 with topp of 0.9 gives a bit more reasonable samples: ``` $ ./run stories260K/stories260K.bin -z stories260K/tok512.bin -t 1.0 -p 0.9 -s 133742 Once upon a time, there was a little boy named Timmy. Timmy loved to play with his toys and eat sandwiches. One day, Timmy's mom told him it was time to rest for a while. Timmy's friend Billy came over and took him a down. Timmy's mom saw that Timmy was sad, but Timmy said, "I didn't understand what is it! We need to find some leafs." Timmy thought about it and took a deep breath on a spoon. He hoped it was important to be kind and continued to find its image next time. After they finished getting, Timmy's dad came up to his house and promised to help Timmy. ``` Hey you can't expect too much from a 260K parameter model. I'm even mildly shocked we get this far :D ``` ### llama2.c/doc/train_llama_tokenizer.md ```markdown # training llama tokenizer How does Meta train their sentencepiece tokenizer? You can print the config as follows: ```python import sentencepiece.sentencepiece_model_pb2 mp = sentencepiece.sentencepiece_model_pb2.ModelProto() mp.ParseFromString(open("tokenizer.model", "rb").read()) print(mp.trainer_spec) print(mp.normalizer_spec) ``` this gives: ``` trainer_spec { input: "/large_experiments/theorem/datasets/MERGED/all.test1.merged" model_prefix: "spm_model_32k_200M_charcov099995_allowWSO__v2" model_type: BPE vocab_size: 32000 self_test_sample_size: 0 input_format: "text" character_coverage: 0.9999499917030334 input_sentence_size: 200000000 seed_sentencepiece_size: 1000000 shrinking_factor: 0.75 num_threads: 80 num_sub_iterations: 2 max_sentence_length: 4192 shuffle_input_sentence: true max_sentencepiece_length: 16 split_by_unicode_script: true split_by_whitespace: true split_by_number: true treat_whitespace_as_suffix: false split_digits: true allow_whitespace_only_pieces: true vocabulary_output_piece_score: true hard_vocab_limit: true use_all_vocab: false byte_fallback: true required_chars: "" unk_id: 0 bos_id: 1 eos_id: 2 pad_id: -1 unk_surface: " \342\201\207 " unk_piece: "" bos_piece: "" eos_piece: "" pad_piece: "" train_extremely_large_corpus: false enable_differential_privacy: false differential_privacy_noise_level: 0.0 differential_privacy_clipping_threshold: 0 } normalizer_spec { name: "identity" precompiled_charsmap: "" add_dummy_prefix: true remove_extra_whitespaces: false normalization_rule_tsv: "" } ``` We can use the sentencepiece spm_train to train the same models, but optionally smaller. Here are their [options docs](https://github.com/google/sentencepiece/blob/master/doc/options.md) we can refer to. It's not much but it helps. We'll depart on one setting, I recommend changing `character_coverage` -> 1.0. We also want to make sure to note the following important settings that come up in the paper and are not necessarily the default sentencepiece settings: ``` --split-digits = true --allow_whitespace_only_pieces = true --byte_fallback = true --normalization_rule_name = identity ``` With this in mind we can train a sentencepiece vocab in what I believe is probably the same to how Meta trained theirs as: ``` spm_train --input="$input" \ --model_prefix="$model_prefix" \ --model_type=bpe \ --vocab_size="$vocab_size" \ --self_test_sample_size=0 \ --input_format="text" \ --character_coverage=1.0 \ --num_threads="$(nproc)" \ --split_digits=true \ --allow_whitespace_only_pieces=true \ --byte_fallback=true \ --unk_surface=" \342\201\207 " \ --normalization_rule_name=identity \ ``` Where $input is the input file, $model_prefix is the output path prefix, vocab_size is the desired vocab, and we're by default taking [truncated — 532 more characters] ``` ### webapp/requirements.txt ``` websockets>=12.0 pyserial>=3.5 ``` ### llama2.c/requirements.txt ``` numpy==1.23.5 pytest==7.4.0 Requests==2.31.0 sentencepiece==0.1.99 torch==2.0.1 tqdm==4.64.1 wandb==0.15.5 ``` ### llama2.c/wandb/run-20260214_053939-ps1xwflu/files/requirements.txt ``` xgboost==3.1.2 uvicorn==0.24.0 threadpoolctl==3.6.0 lightgbm==4.6.0 pyreadr==0.5.4 pydantic_core==2.33.2 pyobjc-framework-Cocoa==12.1 pexpect==4.9.0 PyNaCl==1.6.1 GitPython==3.1.45 cloudpickle==3.1.2 h5netcdf==1.8.0 pip==25.2 jupyter_core==5.8.1 requests==2.32.5 toolz==1.1.0 idna==3.10 fonttools==4.60.0 rich==14.2.0 starlette==0.27.0 traitlets==5.14.3 tokenizers==0.22.1 charset-normalizer==3.4.3 nest-asyncio==1.6.0 pymc==5.27.0 ptyprocess==0.7.0 pydantic==2.11.9 boto3==1.42.0 pytz==2025.2 toml==0.10.2 streamlit==1.50.0 typing-inspection==0.4.1 scikit-learn==1.7.2 pytensor==2.36.3 pydeck==0.9.1 jsonschema-specifications==2025.9.1 executing==2.2.1 platformdirs==4.4.0 lazy_imports==1.0.1 statsmodels==0.14.6 autogluon.core==1.4.0 cons==0.4.7 blinker==1.9.0 cryptography==44.0.2 tenacity==9.1.2 transformers==4.57.3 botocore==1.41.6 contourpy==1.3.3 zope.interface==8.0.1 seaborn==0.13.2 numpy==2.3.3 jedi==0.19.2 einx==0.3.0 click==8.3.0 regex==2025.11.3 PyGithub==2.8.1 cffi==2.0.0 jupyter_client==8.6.3 asttokens==3.0.0 kiwisolver==1.4.9 tqdm==4.67.1 urllib3==2.3.0 torch==2.7.1 sentry-sdk==2.52.0 pyparsing==3.2.5 tzdata==2025.2 ipython_pygments_lexers==1.1.1 appnope==0.1.4 pyobjc-framework-Metal==12.1 certifi==2025.8.3 prompt_toolkit==3.0.52 parso==0.8.5 decorator==5.2.1 rpds-py==0.27.1 ipykernel==6.30.1 xarray==2025.12.0 uv==0.9.13 stack-data==0.6.3 tornado==6.5.2 sympy==1.14.0 PyJWT==2.10.1 jsonschema==4.25.1 autogluon.features==1.4.0 pydantic-settings==2.12.0 markdown-it-py==4.0.0 posthog==6.9.3 pybaseball==2.2.7 python-dotenv==1.0.1 jmespath==1.0.1 watchfiles==1.1.1 loguru==0.7.3 ipython==9.5.0 h11==0.16.0 gitdb==4.0.12 sniffio==1.3.1 psutil==7.0.0 setuptools==80.9.0 httptools==0.7.1 s3transfer==0.16.0 sentencepiece==0.2.1 mdurl==0.1.2 patsy==1.0.2 networkx==3.5 python-dateutil==2.9.0.post0 mpmath==1.3.0 PyYAML==6.0.3 soupsieve==2.8 kditransform==1.2.0 logical-unification==0.4.7 narwhals==2.8.0 xmodem==0.4.7 antlr4-python3-runtime==4.9.3 ipywidgets==8.1.8 protobuf==6.33.0 anyio==3.7.1 typing_extensions==4.15.0 kalshi-python==2.1.4 beautifulsoup4==4.14.3 packaging==25.0 fsspec==2025.9.0 frozendict==2.4.7 cycler==0.12.1 hf-xet==1.2.0 Jinja2==3.1.6 widgetsnbextension==4.0.15 referencing==0.37.0 pandas==2.3.3 jupyterlab_widgets==3.0.16 matplotlib==3.10.6 MarkupSafe==3.0.3 websockets==12.0 wandb==0.25.0 multipledispatch==1.0.0 pure_eval==0.2.3 DateTime==5.5 eval_type_backport==0.3.0 distro==1.9.0 xarray-einstats==0.9.1 arviz==0.23.1 Pygments==2.19.2 tabpfn==6.0.6 attrs==25.4.0 h5py==3.15.1 etuples==0.3.10 einops==0.8.1 filelock==3.20.0 matplotlib-inline==0.1.7 smmap==5.0.2 scipy==1.16.2 pyobjc-core==12.1 pillow==11.3.0 backoff==2.2.1 debugpy==1.8.17 pyserial==3.5 tabpfn-common-utils==0.2.10 safetensors==0.7.0 autogluon.tabular==1.4.0 uvloop==0.22.1 fastapi==0.104.1 altair==5.5.0 lxml==6.0.2 numba==0.62.1 joblib==1.5.2 annotated-types==0.7.0 wcwidth==0.2.14 llvmlite==0.45.1 omegaconf==2.3.0 six==1.17.0 huggingface-hub==0.36.0 pycparser==2.23 pyzmq==27.1.0 pyarrow==20.0.0 autogluon.common==1.4.0 cachetools==6.2.1 miniKanren==1.0.5 comm==0.2.3 ``` ### llama2.c/wandb/run-20260214_055907-3vq9dqfq/files/requirements.txt ``` xgboost==3.1.2 uvicorn==0.24.0 threadpoolctl==3.6.0 lightgbm==4.6.0 pyreadr==0.5.4 pydantic_core==2.33.2 pyobjc-framework-Cocoa==12.1 pexpect==4.9.0 PyNaCl==1.6.1 GitPython==3.1.45 cloudpickle==3.1.2 h5netcdf==1.8.0 pip==25.2 jupyter_core==5.8.1 requests==2.32.5 toolz==1.1.0 idna==3.10 fonttools==4.60.0 rich==14.2.0 starlette==0.27.0 traitlets==5.14.3 tokenizers==0.22.1 charset-normalizer==3.4.3 nest-asyncio==1.6.0 pymc==5.27.0 ptyprocess==0.7.0 pydantic==2.11.9 boto3==1.42.0 pytz==2025.2 toml==0.10.2 streamlit==1.50.0 typing-inspection==0.4.1 scikit-learn==1.7.2 pytensor==2.36.3 pydeck==0.9.1 jsonschema-specifications==2025.9.1 executing==2.2.1 platformdirs==4.4.0 lazy_imports==1.0.1 statsmodels==0.14.6 autogluon.core==1.4.0 cons==0.4.7 blinker==1.9.0 cryptography==44.0.2 tenacity==9.1.2 transformers==4.57.3 botocore==1.41.6 contourpy==1.3.3 zope.interface==8.0.1 seaborn==0.13.2 numpy==2.3.3 jedi==0.19.2 einx==0.3.0 click==8.3.0 regex==2025.11.3 PyGithub==2.8.1 cffi==2.0.0 jupyter_client==8.6.3 asttokens==3.0.0 kiwisolver==1.4.9 tqdm==4.67.1 urllib3==2.3.0 torch==2.7.1 sentry-sdk==2.52.0 pyparsing==3.2.5 tzdata==2025.2 ipython_pygments_lexers==1.1.1 appnope==0.1.4 pyobjc-framework-Metal==12.1 certifi==2025.8.3 prompt_toolkit==3.0.52 parso==0.8.5 decorator==5.2.1 rpds-py==0.27.1 ipykernel==6.30.1 xarray==2025.12.0 uv==0.9.13 stack-data==0.6.3 tornado==6.5.2 sympy==1.14.0 PyJWT==2.10.1 jsonschema==4.25.1 autogluon.features==1.4.0 pydantic-settings==2.12.0 markdown-it-py==4.0.0 posthog==6.9.3 pybaseball==2.2.7 python-dotenv==1.0.1 jmespath==1.0.1 watchfiles==1.1.1 loguru==0.7.3 ipython==9.5.0 h11==0.16.0 gitdb==4.0.12 sniffio==1.3.1 psutil==7.0.0 setuptools==80.9.0 httptools==0.7.1 s3transfer==0.16.0 sentencepiece==0.2.1 mdurl==0.1.2 patsy==1.0.2 networkx==3.5 python-dateutil==2.9.0.post0 mpmath==1.3.0 PyYAML==6.0.3 soupsieve==2.8 kditransform==1.2.0 logical-unification==0.4.7 narwhals==2.8.0 xmodem==0.4.7 antlr4-python3-runtime==4.9.3 ipywidgets==8.1.8 protobuf==6.33.0 anyio==3.7.1 typing_extensions==4.15.0 kalshi-python==2.1.4 beautifulsoup4==4.14.3 packaging==25.0 fsspec==2025.9.0 frozendict==2.4.7 cycler==0.12.1 hf-xet==1.2.0 Jinja2==3.1.6 widgetsnbextension==4.0.15 referencing==0.37.0 pandas==2.3.3 jupyterlab_widgets==3.0.16 matplotlib==3.10.6 MarkupSafe==3.0.3 websockets==12.0 wandb==0.25.0 multipledispatch==1.0.0 pure_eval==0.2.3 DateTime==5.5 eval_type_backport==0.3.0 distro==1.9.0 xarray-einstats==0.9.1 arviz==0.23.1 Pygments==2.19.2 tabpfn==6.0.6 attrs==25.4.0 h5py==3.15.1 etuples==0.3.10 einops==0.8.1 filelock==3.20.0 matplotlib-inline==0.1.7 smmap==5.0.2 scipy==1.16.2 pyobjc-core==12.1 pillow==11.3.0 backoff==2.2.1 debugpy==1.8.17 pyserial==3.5 tabpfn-common-utils==0.2.10 safetensors==0.7.0 autogluon.tabular==1.4.0 uvloop==0.22.1 fastapi==0.104.1 altair==5.5.0 lxml==6.0.2 numba==0.62.1 joblib==1.5.2 annotated-types==0.7.0 wcwidth==0.2.14 llvmlite==0.45.1 omegaconf==2.3.0 six==1.17.0 huggingface-hub==0.36.0 pycparser==2.23 pyzmq==27.1.0 pyarrow==20.0.0 autogluon.common==1.4.0 cachetools==6.2.1 miniKanren==1.0.5 comm==0.2.3 ``` ### webapp/server.py ```python #!/usr/bin/env python3 """ Bridge between the iMessage web UI and the Pi inference server. Protocol: 1. Pi sends READY every 500ms. Host sees READY → sends ACK. Pi stops. 2. Host sends STX + prompt + ETX. Pi reads it, sends OK. 3. Pi generates, sends TYPING/MSG/IDLE/NOREPLY/RETRY. 4. Host sends CAN to cancel. Pi sends CANCEL_ACK. Usage: python server.py [serial_port] [baud_rate] """ import asyncio import json import glob import sys import os import threading from http.server import HTTPServer, SimpleHTTPRequestHandler from functools import partial import serial import websockets # Protocol constants (must match llama2.h) STX = 0x02 ETX = 0x03 SOH = 0x04 ACK = 0x06 CAN = 0x18 HTTP_PORT = 8080 WS_PORT = 8765 BAUD_RATE = 115200 def find_serial_port(): patterns = [ "/dev/cu.usbserial-*", "/dev/cu.SLAB_USB*", "/dev/ttyUSB*", "/dev/ttyACM*", ] for pattern in patterns: ports = glob.glob(pattern) if ports: return sorted(ports)[0] return None class Bridge: def __init__(self, port, baud): self.port = port self.baud = baud self.ser = None self.ws = None self.running = True self.pi_ready = False self.pi_busy = False # True when Pi is generating/retrying # Events for synchronization self.ok_event = asyncio.Event() self.cancel_ack_event = asyncio.Event() def connect_serial(self): self.ser = serial.Serial(self.port, self.baud, timeout=0.1) self.ser.reset_input_buffer() print(f" Serial: {self.port} @ {self.baud}") # --- Debug logging --- CTRL_NAMES = {0x02: "STX", 0x03: "ETX", 0x04: "SOH", 0x06: "ACK", 0x18: "CAN"} def _fmt_bytes(self, data): """Format bytes as readable string: control chars as names, rest as ASCII.""" parts = [] for b in data: if b in self.CTRL_NAMES: parts.append(f"[{self.CTRL_NAMES[b]}]") elif b == 0x0a: parts.append("\\n") elif b == 0x0d: parts.append("\\r") elif 32 <= b < 127: parts.append(chr(b)) else: parts.append(f"[{b:02x}]") return "".join(parts) def _log_rx(self, data): if data: print(f" RX: {self._fmt_bytes(data)}") def _log_tx(self, data): if data: print(f" TX: {self._fmt_bytes(data)}") # --- Protocol line parser (shared by handshake + main loop) --- def _parse_bytes(self, data, line_buf, in_protocol): """Parse raw serial bytes into protocol lines. Returns (lines, line_buf, in_protocol).""" self._log_rx(data) lines = [] for byte in data: if byte == SOH: in_protocol = True line_buf = bytearray() elif byte == ord("\n") and in_protocol: lines.append(line_buf.decode("utf-8", errors="replace")) in_protocol = False elif in_protocol: line_buf.append(byte) return lines, line_buf, in_protocol # --- Phase 1: Handshake --- async def handshake(self): """Wait for READY from Pi, send ACK.""" loop = asyncio.get_event_loop() print(" Waiting for Pi READY...") line_buf = bytearray() in_proto = False while True: try: data = await loop.run_in_executor(None, self.ser.read, 256) except Exception as e: print(f" Serial error: {e}") await asyncio.sleep(1) continue if not data: continue lines, line_buf, in_proto = self._parse_bytes(data, line_buf, in_proto) for line in lines: if line == "READY": self._log_tx(bytes([ACK])) self.ser.write(bytes([ACK])) self.pi_ready = True print(" Got READY → sent ACK. Handshake complete!") return # --- Serial reader (main loop after handshake) --- async def serial_reader(self): loop = asyncio.get_event_loop() line_buf = bytearray() in_proto = False while self.running: try: data = await loop.run_in_executor(None, self.ser.read, 256) except serial.SerialException as e: print(f" Serial error: {e}") await asyncio.sleep(1) continue except Exception as e: print(f" Read error: {e}") await asyncio.sleep(0.5) continue if not data: continue lines, line_buf, in_proto = self._parse_bytes(data, line_buf, in_proto) for line in lines: await self.handle_protocol_line(line) async def handle_protocol_line(self, line): print(f" Pi → {line}") # Internal handshake signals if line == "OK": self.pi_busy = True self.ok_event.set() return if line == "CANCEL_ACK": self.pi_busy = False self.cancel_ack_event.set() return if line == "READY": # Pi rebooted — re-ACK self._log_tx(bytes([ACK])) self.ser.write(bytes([ACK])) print(" Pi rebooted — re-sent ACK") return if line.startswith("DBG:"): # Debug output from generation — log only, don't forward print(f" Pi DBG: {line[4:]}") return # Forward to browser if not self.ws: return try: if line == "TYPING": await self.ws.send(json.dumps({"type": "typing"})) elif line.startswith("MSG:"): await self.ws.send(json.dumps({"type": "message", "text": line[4:]})) [truncated — 4176 more characters] ``` ### llama2.c/chat_history.py ```python #ADDED CODE - MLi import argparse import os import numpy as np from tokenizer import Tokenizer train_file = "./../private/chat_history_train.txt" val_file = "./../private/chat_history_val.txt" def pretokenize(): enc = Tokenizer() with open(train_file, "r") as f: train_tokens = enc.encode(f.read(), bos=False, eos=False) with open(val_file, "r") as f: val_tokens = enc.encode(f.read(), bos=False, eos=False) train_tokens = np.array(train_tokens, dtype=np.uint16) val_tokens = np.array(val_tokens, dtype=np.uint16) train_bin = train_file.replace(".txt", ".bin") val_bin = val_file.replace(".txt", ".bin") with open(train_bin, "wb") as f: f.write(train_tokens.tobytes()) with open(val_bin, "wb") as f: f.write(val_tokens.tobytes()) print(f"Saved tokenized train: {train_bin} and val: {val_bin}") if __name__ == "__main__": """ These stages are designed to be run in order. To tokenize data with the Llama 2 tokenizer: python chat_history.py pretokenize """ parser = argparse.ArgumentParser() parser.add_argument("stage", type=str, choices=["pretokenize"]) args = parser.parse_args() # depending on the stage call the appropriate function if args.stage == "pretokenize": pretokenize() else: raise ValueError(f"Unknown stage {args.stage}") ``` ### llama2.c/win.h ```c #ifndef _WIN_H_ #define _WIN_H_ #define WIN32_LEAN_AND_MEAN // Exclude rarely-used stuff from Windows headers #include #include #include #define ssize_t int64_t #define ftell _ftelli64 // Below code is originally from mman-win32 // /* * sys/mman.h * mman-win32 */ #ifndef _WIN32_WINNT // Allow use of features specific to Windows XP or later. #define _WIN32_WINNT 0x0501 // Change this to the appropriate value to target other versions of Windows. #endif /* All the headers include this file. */ #ifndef _MSC_VER #include <_mingw.h> #endif #include #ifdef __cplusplus extern "C" { #endif #define PROT_NONE 0 #define PROT_READ 1 #define PROT_WRITE 2 #define PROT_EXEC 4 #define MAP_FILE 0 #define MAP_SHARED 1 #define MAP_PRIVATE 2 #define MAP_TYPE 0xf #define MAP_FIXED 0x10 #define MAP_ANONYMOUS 0x20 #define MAP_ANON MAP_ANONYMOUS #define MAP_FAILED ((void *)-1) /* Flags for msync. */ #define MS_ASYNC 1 #define MS_SYNC 2 #define MS_INVALIDATE 4 /* Flags for portable clock_gettime call. */ #define CLOCK_REALTIME 0 void* mmap(void *addr, size_t len, int prot, int flags, int fildes, ssize_t off); int munmap(void *addr, size_t len); int mprotect(void *addr, size_t len, int prot); int msync(void *addr, size_t len, int flags); int mlock(const void *addr, size_t len); int munlock(const void *addr, size_t len); int clock_gettime(int clk_id, struct timespec *tp); #ifdef __cplusplus }; #endif #endif /* _WIN_H_ */ ``` ### llama2.c/configurator.py ```python """ Poor Man's Configurator. Probably a terrible idea. Example usage: $ python train.py config/override_file.py --batch_size=32 this will first run config/override_file.py, then override batch_size to 32 The code in this file will be run as follows from e.g. train.py: >>> exec(open('configurator.py').read()) So it's not a Python module, it's just shuttling this code away from train.py The code in this script then overrides the globals() I know people are not going to love this, I just really dislike configuration complexity and having to prepend config. to every single variable. If someone comes up with a better simple Python solution I am all ears. """ import sys from ast import literal_eval for arg in sys.argv[1:]: if '=' not in arg: # assume it's the name of a config file assert not arg.startswith('--') config_file = arg print(f"Overriding config with {config_file}:") with open(config_file) as f: print(f.read()) exec(open(config_file).read()) else: # assume it's a --key=value argument assert arg.startswith('--') key, val = arg.split('=') key = key[2:] if key in globals(): try: # attempt to eval it it (e.g. if bool, number, or etc) attempt = literal_eval(val) except (SyntaxError, ValueError): # if that goes wrong, just use the string attempt = val # ensure the types match ok assert type(attempt) == type(globals()[key]) # cross fingers print(f"Overriding: {key} = {attempt}") globals()[key] = attempt else: raise ValueError(f"Unknown config key: {key}") ``` [136 more indexed source files omitted to keep this export small. The full file list is in the Codebase structure section above.]