# Project export: Nexus

This document was generated by HackStack to give an AI agent context about a hackathon project. Sections are labeled with their provenance; content marked as truncated was cut to keep this document small.

## Project metadata

- Hackathon: TreeHacks 2026
- Tagline: Rent idle GPUs for fast, ultra-cheap AI inference with speculative decoding on a distributed marketplace.
- Devpost: https://devpost.com/software/nexus-nqpb8m
- GitHub: https://github.com/Teddygat0r/treehacks
- Video: https://www.youtube.com/embed/xA119G6RxEY?enablejsapi=1&hl=en_US&rel=0&start=&version=3&wmode=transparent
- Team: 2 GitHub contributor(s) — Joshua Zhang (3 commits), Ryan Co (2 commits)

## Devpost submission (written by the team)

### Inspiration

The environmental impact of a single AI conversation is equivalent to running a microwave for one minute and throwing away a bottle of water. Yet today’s AI inference stack still assumes you must run everything inside large, expensive GPU clusters inside data centers. That leaves a massive inefficiency: millions of consumer GPUs (laptops/desktops) sit idle most of the day as cheap and efficient sources of compute. Nexus is motivated by a simple observation: most of the “wasted” work in LLM inference happens during sequential token-by-token generation. Newly developed techniques like speculative decoding can shift much of that work onto smaller models. If we can offload the right compute to otherwise idle GPUs, we can reduce cost and environmental impact without sacrificing output quality.

### What it does

Nexus is a global marketplace for AI inference built on speculative decoding. Users can rent out their GPUs to run small, fast draft models that propose a sequence of tokens which are efficiently verified by a large model in parallel. This architecture enables: Lower latency: fewer expensive, sequential target-model decoding steps Higher throughput: the verifier does fewer passes per response, and draft generation is massively parallelizable Lower cost per generated token: expensive GPU time is reserved for verification rather than brute-force generation Better hardware utilization: idle consumer GPUs do useful work (drafting) instead of sitting unused From a user perspective, Nexus exposes a simple inference API. Under the hood, it routes requests across the marketplace, orchestrates draft generation, batches verification, and returns final outputs with high accuracy.

### How we built it

Our inference platform was built on the speculative decoding algorithm, where a small draft model predicts multiple tokens ahead and a large target model verifies them in parallel, reducing the number of expensive decoding steps. To run the small draft model, we used vLLM, an engine for open-source AI inference with optimizations like KV-cache management, continuous batching, and high-throughput serving. While our architecture is model-agnostic and can support any draft-verifier pair that shares a tokenizer, we selected Qwen 2.5 0.5B as the draft model and Qwen 2.5 70B as the verifier. This pairing provided the strongest empirical performance gains in our speculative decoding framework. We hosted our Qwen 2.5 70B verifier model on Modal to enable reliable, high-throughput GPU inference without managing infrastructure. Modal’s autoscaling and GPU orchestration allowed us to rapidly iterate on system design and support concurrent draft-model streams. This reduced operational overhead and let us focus on optimizing the speculative decoding pipeline rather than deployment engineering. Finally, our frontend was designed and hosted using Vercel v0. We chose Vercel for its simplicity and fast deployment workflow, which allowed us to quickly ship and update the application with minimal configuration overhead. Its managed hosting environment provided reliable performance and scalability without requiring us to maintain custom frontend infrastructure.

### Challenges we ran into

Tokenization alignment between draft and target models required strict consistency, small mismatches caused verification failures. Acceptance rate tuning was delicate. Too small a draft model reduced acceptance length and too much draft compute reduced cost savings. Network overhead between distributed draft nodes and the verifier had to be carefully batched to prevent latency from offsetting compute gains. Serving a 70B model under concurrency introduced GPU memory pressure, requiring careful KV-cache and batching strategies.

### Accomplishments we're proud of

Nexus inference is 20% cheaper than comparable modern cloud inference options by combining speculative decoding with distributed idle GPUs. Here’s how we did it: Built an end-to-end speculative decoding pipeline with a real draft/verifier pair. Demonstrated stable concurrent draft streams feeding a centralized verifier. Reduced verifier passes per response by accepting longer draft prefixes in single verification steps. Achieved measurable cost reduction without changing the final model output

### What we learned

We learned that speculative decoding is less about “clever prompting” and more about systems-level efficiency: memory bandwidth, batching, and acceptance rate distributions matter as much as raw FLOPs. We implemented and validated three core optimizations: Multi-candidate drafting: Instead of generating one draft continuation, we generate multiple candidates per step and batch them for verification. This increases the probability that the verifier can accept a longer prefix in a single pass, improving throughput. Optimistic verification: We structured the pipeline so the system proceeds under the assumption that most draft tokens will be accepted, only falling back when divergence is detected. This reduces unnecessary synchronization overhead and improves wall-clock latency. KV caching: Efficient cache reuse and management significantly improved throughput, especially under concurrency, and made the verifier layer more stable.

### What's next

We believe the concept of Nexus can significantly alleviate the current shortage of computing we have, while giving access to the general population to contribute and reap the rewards of the continued usage of LLMs. While we have made significant strides in optimizing Nexus, there are still many optimizations that we want to implement to cut down from 20% to 50% cost savings: Decentralize the verifier layer to support multiple providers, route verification to the cheapest/closest GPU that meets latency constraints, add fallback strategies to preserve reliability Dynamic draft selection and adaptive speculation to automatically choose draft size and length based on the prompt and acceptance rates Real market with spot pricing for draft GPUs, cost tiers, and smart routing based on node performance These optimizations will make the case for Nexus even stronger and solidify it as a robust platform for distributed model inference.

## README (from the GitHub repository)

yay


## Detected evidence (automated analysis)

Indexed codebase: 60 recognized source files, 369 KB.
- Anthropic (technology) — detected in the code
- CSS (language) — detected in the code
- FastAPI (technology) — detected in the code
- Hugging Face (technology) — detected in the code
- Next.js (technology) — detected in the code
- OpenAI (technology) — detected in the code
- Python (language) — detected in the code
- PyTorch (technology) — detected in the code
- React (technology) — detected in the code
- Tailwind CSS (technology) — detected in the code
- TypeScript (language) — detected in the code
- Vercel (technology) — claimed on Devpost, not found in the code
- AI coding agent: Claude Code — evidence: config files committed to the repository

## Codebase structure (from repository index)

### Files (74 of 74)

```
.gitignore
app/globals.css
app/layout.tsx
app/network/page.tsx
app/page.tsx
app/provider/page.tsx
app/visualize/page.tsx
CLAUDE.md
components/activity-chart.tsx
components/chat-input.tsx
components/chat-panel.tsx
components/connected-hardware-card.tsx
components/dashboard-body.tsx
components/dashboard-header.tsx
components/draft-nodes-table.tsx
components/live-metrics.tsx
components/network-explorer.tsx
components/network-stat-card.tsx
components/network-visualizer.tsx
components/provider-dashboard.tsx
components/provider-header.tsx
components/recent-payouts-table.tsx
components/speculative-visualizer.tsx
components/target-nodes-table.tsx
components/total-earnings-card.tsx
components/ui/badge.tsx
components/ui/card.tsx
components/ui/progress.tsx
components/ui/scroll-area.tsx
data/speculative-log.json
lib/api.ts
lib/types.ts
lib/utils.ts
next-env.d.ts
next.config.mjs
package.json
postcss.config.mjs
proto/.gitignore
proto/common_pb2_grpc.py
proto/common_pb2.py
proto/common.proto
proto/frontend_service.proto
proto/README.md
proto/router_service.proto
proto/speculative_decoding_pb2_grpc.py
proto/speculative_decoding_pb2.py
proto/speculative_decoding.proto
README.md
requirements-mac.txt
requirements.txt
router/README.md
router/server.py
scripts/README.md
tsconfig.json
workers/.gitignore
workers/benchmark_gsm8k.py
workers/BENCHMARK_USAGE.md
workers/draft_node/client.py
workers/draft_node/README.md
workers/frontend_bridge/server.py
workers/paper.txt
workers/QUICK_REFERENCE.md
workers/quick_test.py
workers/README.md
workers/start_bridge.sh
workers/start_draft_client.sh
workers/start_router.sh
workers/start_target_server.sh
workers/target_node/README.md
workers/target_node/server_modal.py
workers/target_node/server.py
workers/target_node/verification_strategies.py
workers/test_strategies.py
workers/TESTING_GUIDE.md
```

### Dependencies

- package.json: @radix-ui/react-progress@^1.1, @radix-ui/react-scroll-area@^1.2, @radix-ui/react-slot@^1.1, @tailwindcss/postcss@^4, @types/node@^22, @types/react@^19, @types/react-dom@^19, class-variance-authority@^0.7, clsx@^2, framer-motion@^12, lucide-react@^0.469, next@^16, react@^19, react-dom@^19, recharts@^2.15, tailwind-merge@^3, tailwindcss@^4, typescript@^5
- requirements.txt: aiohappyeyeballs@==2.6.1, aiohttp@==3.13.3, aiosignal@==1.4.0, annotated-doc@==0.0.4, annotated-types@==0.7.0, anthropic@==0.79.0, anyio@==4.12.1, apache-tvm-ffi@==0.1.8.post2, astor@==0.8.1, attrs@==25.4.0, blake3@==1.0.8, cachetools@==7.0.1, cbor2@==5.8.0, certifi@==2026.1.4, cffi@==2.0.0, charset-normalizer@==3.4.4, click@==8.3.1, cloudpickle@==3.1.2, compressed-tensors@==0.13.0, cryptography@==46.0.5, cuda-bindings@==13.1.1, cuda-pathfinder@==1.3.4, cuda-python@==13.1.1, cupy-cuda12x@==13.6.0, depyf@==0.20.0, dill@==0.4.1, diskcache@==5.6.3, distro@==1.9.0, dnspython@==2.8.0, docstring_parser@==0.17.0, einops@==0.8.2, email-validator@==2.3.0, fastapi@==0.129.0, fastapi-cli@==0.0.21, fastapi-cloud-cli@==0.12.0, fastar@==0.8.0, fastrlock@==0.8.3, filelock@==3.23.0, flashinfer-python@==0.6.1, frozenlist@==1.8.0, fsspec@==2026.2.0, gguf@==0.17.1, grpcio@==1.78.0, grpcio-reflection@==1.78.0, grpcio-tools@==1.78.0, h11@==0.16.0, hf-xet@==1.2.0, httpcore@==1.0.9, httptools@==0.7.1, httpx@==0.28.1, httpx-sse@==0.4.3, huggingface_hub@==0.36.2, idna@==3.11, ijson@==3.4.0.post0, interegular@==0.3.3, Jinja2@==3.1.6, jiter@==0.13.0, jmespath@==1.1.0, jsonschema@==4.26.0, jsonschema-specifications@==2025.9.1, lark@==1.2.2, llguidance@==1.3.0, llvmlite@==0.44.0, lm-format-enforcer@==0.11.3, loguru@==0.7.3, markdown-it-py@==4.0.0, MarkupSafe@==3.0.3, mcp@==1.26.0, mdurl@==0.1.2, mistral_common@==1.9.1, model-hosting-container-standards@==0.1.13, mpmath@==1.3.0, msgpack@==1.1.2, msgspec@==0.20.0, multidict@==6.7.1, networkx@==3.6.1, ninja@==1.13.0, numba@==0.61.2, numpy@==2.2.6, nvidia-cublas@==13.0.0.19, nvidia-cublas-cu11@==11.11.3.6, nvidia-cublas-cu12@==12.8.4.1, nvidia-cuda-cupti@==13.0.48, nvidia-cuda-cupti-cu11@==11.8.87, nvidia-cuda-cupti-cu12@==12.8.90, nvidia-cuda-nvrtc@==13.0.48, nvidia-cuda-nvrtc-cu11@==11.8.89, nvidia-cuda-nvrtc-cu12@==12.8.93, nvidia-cuda-runtime@==13.0.48, nvidia-cuda-runtime-cu11@==11.8.89, nvidia-cuda-runtime-cu12@==12.8.90, nvidia-cudnn-cu11@==9.1.0.70, nvidia-cudnn-cu12@==9.10.2.21, nvidia-cudnn-cu13@==9.13.0.50, nvidia-cudnn-frontend@==1.18.0, nvidia-cufft@==12.0.0.15, nvidia-cufft-cu11@==10.9.0.58, nvidia-cufft-cu12@==11.3.3.83, nvidia-cufile@==1.15.0.42, nvidia-cufile-cu12@==1.13.1.3, nvidia-curand@==10.4.0.35, nvidia-curand-cu11@==10.3.0.86, nvidia-curand-cu12@==10.3.9.90, nvidia-cusolver@==12.0.3.29, nvidia-cusolver-cu11@==11.4.1.48, nvidia-cusolver-cu12@==11.7.3.90, nvidia-cusparse@==12.6.2.49, nvidia-cusparse-cu11@==11.7.5.86, nvidia-cusparse-cu12@==12.5.8.93, nvidia-cusparselt-cu12@==0.7.1, nvidia-cusparselt-cu13@==0.8.0, nvidia-cutlass-dsl@==4.4.0, nvidia-cutlass-dsl-libs-base@==4.4.0, nvidia-ml-py@==13.590.48, nvidia-nccl-cu11@==2.21.5, nvidia-nccl-cu12@==2.27.5, nvidia-nccl-cu13@==2.27.7, nvidia-nvjitlink@==13.0.39, nvidia-nvjitlink-cu12@==12.8.93, nvidia-nvshmem-cu12@==3.3.20, nvidia-nvshmem-cu13@==3.3.24, nvidia-nvtx@==13.0.39, nvidia-nvtx-cu11@==11.8.86, nvidia-nvtx-cu12@==12.8.90, openai@==2.21.0, openai-harmony@==0.0.8, opencv-python-headless@==4.13.0.92, outlines_core@==0.2.11, packaging@==26.0, partial-json-parser@==0.2.1.1.post7, pillow@==12.1.1, prometheus_client@==0.24.1, prometheus-fastapi-instrumentator@==7.1.0, propcache@==0.4.1, protobuf@==6.33.5, psutil@==7.2.2, py-cpuinfo@==9.0.0, pybase64@==1.4.3, pycountry@==24.6.1, pycparser@==3.0, pydantic@==2.12.5, pydantic_core@==2.41.5, pydantic-extra-types@==2.11.0, pydantic-settings@==2.12.0, Pygments@==2.19.2, PyJWT@==2.11.0, python-dotenv@==1.2.1, python-json-logger@==4.0.0, python-multipart@==0.0.22, PyYAML@==6.0.3, pyzmq@==27.1.0, ray@==2.53.0, referencing@==0.37.0, regex@==2026.1.15, requests@==2.32.5, rich@==14.3.2, rich-toolkit@==0.19.4, rignore@==0.7.6, rpds-py@==0.30.0, safetensors@==0.7.0, sentencepiece@==0.2.1, sentry-sdk@==2.52.0, setproctitle@==1.3.7, setuptools@==80.10.2, shellingham@==1.5.4, six@==1.17.0, sniffio@==1.3.1, sse-starlette@==3.2.0, starlette@==0.52.1, supervisor@==4.3.0, sympy@==1.14.0, tabulate@==0.9.0, tiktoken@==0.12.0, tokenizers@==0.22.2, torch@==2.9.1, torchaudio@==2.9.1, torchvision@==0.24.1, tqdm@==4.67.3, transformers@==4.57.6, triton@==3.5.1, typer@==0.23.1, typing_extensions@==4.15.0, typing-inspection@==0.4.2, urllib3@==2.6.3, uvicorn@==0.40.0, uvloop@==0.22.1, vllm@==0.15.1, watchfiles@==1.1.1, websockets@==16.0, xgrammar@==0.1.29, yarl@==1.22.0

### Recent commits (newest first)

- Merge remote-tracking branch 'origin/ryan-ui' into router-real
- lowk works now
- claude save 2
- Merge ryan-ui: resolve model name and client config conflicts
- claude save 2
- finish frontend
- update visualization
- claude save
- multiple candiates
- chore: stop tracking __pycache__ files
- visualization
- add modal
- decent responses
- Merge pull request #2 from RyanPCo/v0/copierceryan-7677-ca08f7f1
- Merge pull request #3 from RyanPCo/network-explorer-page
- working verification
- feat: add packet events and visualizer for network steps
- feat: make NetworkVisualizer fully responsive
- refactor: strip right-side content from NetworkVisualizer
- refactor: centralize token animation state in DashboardBody

## Key source files (fetched from GitHub, selected and truncated for size)

### CLAUDE.md

```markdown
# Speculative Decoding System

## Architecture
Distributed speculative decoding: a **draft node** (small model, local GPU) generates candidate tokens, a **target node** (large model, Modal cloud GPU) verifies them.

### Key Components
- `workers/draft_node/client.py` — `DraftNodeClient`: generates draft tokens with vLLM, sends to target for verification. Supports multi-candidate speculative decoding (N candidates per round).
- `workers/target_node/server_modal.py` — `VerificationService`: Modal-deployed target model. Methods: `verify_draft` (single candidate), `verify_multi_candidate` (N candidates, single forward pass).
- `workers/target_node/server.py` — gRPC-based local target server (alternative to Modal).
- `workers/target_node/verification_strategies.py` — Pluggable verification strategies: `deterministic`, `probabilistic`, `threshold`, `greedy`. Supports `verify()` (single) and `verify_batch()` (vectorized N-candidate).
- `workers/benchmark_gsm8k.py` — Benchmark harness for GSM8K and HumanEval datasets.
- `proto/` — Protobuf definitions (`common.proto`, `speculative_decoding.proto`).

### Multi-Candidate Speculative Decoding
- Draft node generates N candidates per round using `SamplingParams(n=N)`.
- Target runs ONE forward pass for all candidates (they share the prefix).
- Best candidate selected by: longest accepted prefix, tie-break by target logprob sum.
- Adaptive N: reduces candidate count if acceptance rate stays below 20% over 5 rounds.
- CLI: `--num-candidates N`, `--candidate-temp`, `--candidate-top-p`.

## Models
- Draft: `Qwen/Qwen2.5-0.5B-Instruct` (default in benchmark) or `Qwen/Qwen3-1.7B-Instruct`
- Target: `Qwen/Qwen2.5-3B-Instruct` (Modal A100) or `Qwen/Qwen3-32B-Instruct`

## Commands
```bash
# Deploy target service to Modal
modal deploy workers/target_node/server_modal.py

# Run benchmark (single candidate)
python workers/benchmark_gsm8k.py --num-samples 5 --max-tokens 64

# Run benchmark (multi-candidate)
python workers/benchmark_gsm8k.py --num-samples 5 --max-tokens 64 --num-candidates 2

# Run local target server (gRPC, no Modal)
python workers/target_node/server.py --model Qwen/Qwen2.5-0.5B-Instruct --strategy deterministic

# Run draft client directly
python workers/draft_node/client.py
```

## Tech Stack
- Python 3.12, vLLM (inference), Modal (cloud GPU), gRPC + protobuf (local comms)
- Frontend: Next.js (in `app/`, `components/`)
- macOS ARM (Apple Silicon) for local dev; Modal A100 for target

## Conventions
- Activate venv before running: `source .venv/bin/activate`
- Proto stubs are pre-generated in `proto/` (no build step needed)
- Benchmark results saved to `workers/logs/` as JSON

```

### workers/QUICK_REFERENCE.md

```markdown
# Quick Reference Guide

## 🚀 Running the System

### 1. Start Target Server
```bash
./start_target_server.sh                          # Default: deterministic
./start_target_server.sh --strategy probabilistic # SLED paper algorithm
./start_target_server.sh --verbose                # See token comparisons
```

### 2. Test/Benchmark
```bash
# Quick test (3 prompts, ~30s)
python quick_test.py

# GSM8K benchmark (10 questions, ~2-3 min)
python benchmark_gsm8k.py

# Full benchmark (50 questions, ~10-15 min)
python benchmark_gsm8k.py --num-samples 50 --use-hf
```

## 📊 Available Tools

| Tool | Purpose | Runtime | Output |
|------|---------|---------|--------|
| `quick_test.py` | Quick acceptance check | ~30s | Console stats |
| `benchmark_gsm8k.py` | Math benchmark | ~2-15min | Console + JSON |
| `test_strategies.py` | Compare strategies | Manual | Console |

## 🎯 Verification Strategies

| Strategy | Description | Best For | Acceptance |
|----------|-------------|----------|------------|
| `deterministic` | Exact match only | Baseline testing | 85-95% |
| `probabilistic` | SLED paper (α=min(1, p_t/p_d)) | Research/paper replication | 75-90% |
| `threshold` | Accept if p_target > threshold | Tuning tradeoffs | 80-95% |
| `greedy` | Always use target | Debugging | Varies |

## 📈 Performance Targets

### OPT-350m → OPT-1.3b (Recommended)
- **Acceptance:** 85-95%
- **Speed:** 60-70 tokens/sec
- **Speedup:** 1.4-1.5x vs target alone

### OPT-125m → OPT-1.3b (Not Recommended)
- **Acceptance:** 40-50% ❌
- **Speed:** 35-45 tokens/sec
- **Speedup:** Minimal

## 🔧 Common Commands

### Basic Testing
```bash
# Test current setup
python quick_test.py

# Benchmark 10 questions
python benchmark_gsm8k.py
```

### Change Draft Model
```bash
# Edit draft_node/client.py, line 22:
draft_model="facebook/opt-350m"  # Change this
```

### Change Verification Strategy
```bash
# Restart server with different strategy
./start_target_server.sh --strategy probabilistic
python quick_test.py
```

### Debug Low Acceptance
```bash
# See token-by-token comparisons
./start_target_server.sh --strategy deterministic --verbose
python quick_test.py
```

## 📖 Documentation

- **README.md** - System overview and setup
- **TESTING_GUIDE.md** - Strategy testing details
- **BENCHMARK_USAGE.md** - GSM8K benchmark guide
- **QUICK_REFERENCE.md** - This file

## 🐛 Troubleshooting

| Problem | Solution |
|---------|----------|
| Low acceptance (<50%) | Use opt-350m instead of opt-125m |
| Out of memory | Reduce gpu_memory_utilization in server.py |
| Server won't start | Check port 50051 not in use |
| Connection refused | Start server first: `./start_target_server.sh` |

## 💡 Tips

1. **Always use greedy (temp=0)** for best acceptance
2. **Start with quick_test.py** before long benchmarks
3. **Use opt-350m** as draft model (good balance)
4. **Save benchmark results** for comparison
5. **Check verbose logs** if acceptance is low

```

### package.json

```
{
  "name": "nexus-dashboard",
  "version": "0.1.0",
  "private": true,
  "scripts": {
    "dev": "next dev --turbopack",
    "build": "next build",
    "start": "next start",
    "lint": "next lint"
  },
  "dependencies": {
    "next": "^16",
    "react": "^19",
    "react-dom": "^19",
    "framer-motion": "^12",
    "lucide-react": "^0.469",
    "class-variance-authority": "^0.7",
    "clsx": "^2",
    "tailwind-merge": "^3",
    "recharts": "^2.15",
    "@radix-ui/react-progress": "^1.1",
    "@radix-ui/react-scroll-area": "^1.2",
    "@radix-ui/react-slot": "^1.1"
  },
  "devDependencies": {
    "@types/node": "^22",
    "@types/react": "^19",
    "@types/react-dom": "^19",
    "typescript": "^5",
    "tailwindcss": "^4",
    "@tailwindcss/postcss": "^4"
  }
}

```

### requirements.txt

```
aiohappyeyeballs==2.6.1
aiohttp==3.13.3
aiosignal==1.4.0
annotated-doc==0.0.4
annotated-types==0.7.0
anthropic==0.79.0
anyio==4.12.1
apache-tvm-ffi==0.1.8.post2
astor==0.8.1
attrs==25.4.0
blake3==1.0.8
cachetools==7.0.1
cbor2==5.8.0
certifi==2026.1.4
cffi==2.0.0
charset-normalizer==3.4.4
click==8.3.1
cloudpickle==3.1.2
compressed-tensors==0.13.0
cryptography==46.0.5
cuda-bindings==13.1.1
cuda-pathfinder==1.3.4
cuda-python==13.1.1
cupy-cuda12x==13.6.0
depyf==0.20.0
dill==0.4.1
diskcache==5.6.3
distro==1.9.0
dnspython==2.8.0
docstring_parser==0.17.0
einops==0.8.2
email-validator==2.3.0
fastapi==0.129.0
fastapi-cli==0.0.21
fastapi-cloud-cli==0.12.0
fastar==0.8.0
fastrlock==0.8.3
filelock==3.23.0
flashinfer-python==0.6.1
frozenlist==1.8.0
fsspec==2026.2.0
gguf==0.17.1
grpcio==1.78.0
grpcio-reflection==1.78.0
grpcio-tools==1.78.0
h11==0.16.0
hf-xet==1.2.0
httpcore==1.0.9
httptools==0.7.1
httpx==0.28.1
httpx-sse==0.4.3
huggingface_hub==0.36.2
idna==3.11
ijson==3.4.0.post0
interegular==0.3.3
Jinja2==3.1.6
jiter==0.13.0
jmespath==1.1.0
jsonschema==4.26.0
jsonschema-specifications==2025.9.1
lark==1.2.2
llguidance==1.3.0
llvmlite==0.44.0
lm-format-enforcer==0.11.3
loguru==0.7.3
markdown-it-py==4.0.0
MarkupSafe==3.0.3
mcp==1.26.0
mdurl==0.1.2
mistral_common==1.9.1
model-hosting-container-standards==0.1.13
mpmath==1.3.0
msgpack==1.1.2
msgspec==0.20.0
multidict==6.7.1
networkx==3.6.1
ninja==1.13.0
numba==0.61.2
numpy==2.2.6
nvidia-cublas==13.0.0.19
nvidia-cublas-cu11==11.11.3.6
nvidia-cublas-cu12==12.8.4.1
nvidia-cuda-cupti==13.0.48
nvidia-cuda-cupti-cu11==11.8.87
nvidia-cuda-cupti-cu12==12.8.90
nvidia-cuda-nvrtc==13.0.48
nvidia-cuda-nvrtc-cu11==11.8.89
nvidia-cuda-nvrtc-cu12==12.8.93
nvidia-cuda-runtime==13.0.48
nvidia-cuda-runtime-cu11==11.8.89
nvidia-cuda-runtime-cu12==12.8.90
nvidia-cudnn-cu11==9.1.0.70
nvidia-cudnn-cu12==9.10.2.21
nvidia-cudnn-cu13==9.13.0.50
nvidia-cudnn-frontend==1.18.0
nvidia-cufft==12.0.0.15
nvidia-cufft-cu11==10.9.0.58
nvidia-cufft-cu12==11.3.3.83
nvidia-cufile==1.15.0.42
nvidia-cufile-cu12==1.13.1.3
nvidia-curand==10.4.0.35
nvidia-curand-cu11==10.3.0.86
nvidia-curand-cu12==10.3.9.90
nvidia-cusolver==12.0.3.29
nvidia-cusolver-cu11==11.4.1.48
nvidia-cusolver-cu12==11.7.3.90
nvidia-cusparse==12.6.2.49
nvidia-cusparse-cu11==11.7.5.86
nvidia-cusparse-cu12==12.5.8.93
nvidia-cusparselt-cu12==0.7.1
nvidia-cusparselt-cu13==0.8.0
nvidia-cutlass-dsl==4.4.0
nvidia-cutlass-dsl-libs-base==4.4.0
nvidia-ml-py==13.590.48
nvidia-nccl-cu11==2.21.5
nvidia-nccl-cu12==2.27.5
nvidia-nccl-cu13==2.27.7
nvidia-nvjitlink==13.0.39
nvidia-nvjitlink-cu12==12.8.93
nvidia-nvshmem-cu12==3.3.20
nvidia-nvshmem-cu13==3.3.24
nvidia-nvtx==13.0.39
nvidia-nvtx-cu11==11.8.86
nvidia-nvtx-cu12==12.8.90
openai==2.21.0
openai-harmony==0.0.8
opencv-python-headless==4.13.0.92
outlines_core==0.2.11
packaging==26.0
partial-json-parser==0.2.1.1.post7
pillow==12.1.1
prometheus-fastapi-instrumentator==7.1.0
prometheus_client==0.24.1
propcache==0.4.1
protobuf==6.33.5
psutil==7.2.2
py-cpuinfo==9.0.0
pybase64==1.4.3
pycountry==24.6.1
pycparser==3.0
pydantic==2.12.5
pydantic-extra-types==2.11.0
pydantic-settings==2.12.0
pydantic_core==2.41.5
Pygments==2.19.2
PyJWT==2.11.0
python-dotenv==1.2.1
python-json-logger==4.0.0
python-multipart==0.0.22
PyYAML==6.0.3
pyzmq==27.1.0
ray==2.53.0
referencing==0.37.0
regex==2026.1.15
requests==2.32.5
rich==14.3.2
rich-toolkit==0.19.4
rignore==0.7.6
rpds-py==0.30.0
safetensors==0.7.0
sentencepiece==0.2.1
sentry-sdk==2.52.0
setproctitle==1.3.7
setuptools==80.10.2
shellingham==1.5.4
six==1.17.0
sniffio==1.3.1
sse-starlette==3.2.0
starlette==0.52.1
supervisor==4.3.0
sympy==1.14.0
tabulate==0.9.0
tiktoken==0.12.0
tokenizers==0.22.2
torch==2.9.1
torchaudio==2.9.1
torchvision==0.24.1
tqdm==4.67.3
transformers==4.57.6
triton==3.5.1
typer==0.23.1
typing-inspection==0.4.2
typing_extensions==4.15.0
urllib3==2.6.3
uvicorn==0.40.0
uvloop==0.22.1
vllm==0.15.1
watchfiles==1.1.1
websockets==16.0
xgrammar==0.1.29
yarl==1.22.0

```

### app/page.tsx

```typescript
import { DashboardHeader } from "@/components/dashboard-header"
import { DashboardBody } from "@/components/dashboard-body"

export default function Home() {
  return (
    <main className="flex h-screen flex-col overflow-hidden bg-background">
      <DashboardHeader />
      <DashboardBody />
    </main>
  )
}

```

### app/layout.tsx

```typescript
import type { Metadata, Viewport } from "next"
import { Inter, JetBrains_Mono, Space_Grotesk } from "next/font/google"
import "./globals.css"

const spaceGrotesk = Space_Grotesk({
  subsets: ["latin"],
  variable: "--font-space-grotesk",
  weight: ["500", "600", "700"],
})

const inter = Inter({
  subsets: ["latin"],
  variable: "--font-inter",
})

const jetbrainsMono = JetBrains_Mono({
  subsets: ["latin"],
  variable: "--font-jetbrains",
})

export const metadata: Metadata = {
  title: "Nexus - Distributed Speculative Decoding",
  description:
    "Nexus is a distributed LLM inference engine using speculative decoding with edge draft GPUs and cloud target GPUs.",
}

export const viewport: Viewport = {
  themeColor: "#0a0a0f",
}

export default function RootLayout({
  children,
}: {
  children: React.ReactNode
}) {
  return (
    <html lang="en" className="dark">
      <body className={`${spaceGrotesk.variable} ${inter.variable} ${jetbrainsMono.variable} font-sans antialiased`}>
        {children}
      </body>
    </html>
  )
}

```

### router/server.py

```python
"""HTTP router service connecting frontend bridge requests to draft nodes."""

from __future__ import annotations

import argparse
from dataclasses import dataclass, field
import threading
import time
import uuid

from fastapi import FastAPI
from pydantic import BaseModel, Field
import uvicorn


DEFAULT_HEARTBEAT_INTERVAL_MS = 5000
DEFAULT_HEARTBEAT_TIMEOUT_S = 30


class ModelInfo(BaseModel):
    model_id: str = ""
    model_name: str = ""
    version: str = ""


class ResourceStats(BaseModel):
    gpu_utilization: float = 0.0
    memory_used_bytes: int = 0
    memory_total_bytes: int = 0
    active_requests: int = 0
    tokens_per_second: float = 0.0


class WorkerRegistration(BaseModel):
    worker_id: str = ""
    address: str
    worker_type: str = "target"
    model_info: ModelInfo = Field(default_factory=ModelInfo)
    gpu_model: str = ""
    gpu_memory_bytes: int = 0
    gpu_count: int = 0
    max_concurrent_requests: int = 1
    max_batch_size: int = 1


class WorkerRegistrationResponse(BaseModel):
    accepted: bool
    message: str
    assigned_worker_id: str = ""


class WorkerHeartbeatRequest(BaseModel):
    worker_id: str
    stats: ResourceStats = Field(default_factory=ResourceStats)


class WorkerHeartbeatResponse(BaseModel):
    acknowledged: bool
    next_heartbeat_interval_ms: int


class DraftNodeRegistration(BaseModel):
    draft_node_id: str = ""
    address: str
    model_info: ModelInfo = Field(default_factory=ModelInfo)
    gpu_model: str = ""
    gpu_memory_bytes: int = 0
    max_draft_tokens: int = 5


class DraftNodeRegistrationResponse(BaseModel):
    accepted: bool
    message: str
    assigned_node_id: str = ""


class DraftNodeHeartbeatRequest(BaseModel):
    draft_node_id: str
    stats: ResourceStats = Field(default_factory=ResourceStats)
    available_capacity: int = 1


class DraftNodeHeartbeatResponse(BaseModel):
    acknowledged: bool
    next_heartbeat_interval_ms: int


class RouteRequestMessage(BaseModel):
    request_id: str
    prompt: str
    model_id: str = ""
    priority: int = 0


class RouteRequestResponse(BaseModel):
    request_id: str
    assigned_draft_node_id: str = ""
    assigned_draft_node_address: str = ""
    status: str
    message: str
    estimated_queue_time_ms: int = 0


class WorkerAssignmentRequest(BaseModel):
    request_id: str
    draft_node_id: str = ""
    model_id: str = ""


class WorkerAssignmentResponse(BaseModel):
    request_id: str
    worker_id: str = ""
    worker_address: str = ""
    model_info: ModelInfo = Field(default_factory=ModelInfo)
    status: str
    message: str


@dataclass
class DraftNodeRecord:
    draft_node_id: str
    address: str
    model_info: ModelInfo
    gpu_model: str
    gpu_memory_bytes: int
    max_draft_tokens: int
    available_capacity: int = 1
    stats: ResourceStats = field(default_factory=ResourceStats)
    last_heartbeat_s: float = field(default_factory=time.time)


@dataclass
class WorkerRecord:
    worker_id: str
    address: str
    worker_type: str
    model_info: ModelInfo
    gpu_model: str
    gpu_memory_bytes: int
    gpu_count: int
    max_concurrent_requests: int
    max_batch_size: int
    stats: ResourceStats = field(default_factory=ResourceStats)
    last_heartbeat_s: float = field(default_factory=time.time)


class RouterState:
    """Thread-safe in-memory registry and routing state."""

    def __init__(self, heartbeat_timeout_s: int = DEFAULT_HEARTBEAT_TIMEOUT_S):
        self.heartbeat_timeout_s = heartbeat_timeout_s
        self._draft_nodes: dict[str, DraftNodeRecord] = {}
        self._workers: dict[str, WorkerRecord] = {}
        self._draft_rr_cursor: dict[str, int] = {}
        self._worker_rr_cursor: dict[str, int] = {}
        self._lock = threading.Lock()

    def _is_active(self, last_heartbeat_s: float) -> bool:
        return (time.time() - last_heartbeat_s) <= self.heartbeat_timeout_s

    def _purge_stale_locked(self) -> None:
        stale_drafts = [
            node_id
            for node_id, record in self._draft_nodes.items()
            if not self._is_active(record.last_heartbeat_s)
        ]
        for node_id in stale_drafts:
            del self._draft_nodes[node_id]

        stale_workers = [
            worker_id
            for worker_id, record in self._workers.items()
            if not self._is_active(record.last_heartbeat_s)
        ]
        for worker_id in stale_workers:
            del self._workers[worker_id]

    def register_draft_node(self, request: DraftNodeRegistration) -> str:
        assigned_id = request.draft_node_id.strip() or f"draft-{uuid.uuid4().hex[:8]}"
        with self._lock:
            self._draft_nodes[assigned_id] = DraftNodeRecord(
                draft_node_id=assigned_id,
                address=request.address,
                model_info=request.model_info,
                gpu_model=request.gpu_model,
                gpu_memory_bytes=request.gpu_memory_bytes,
                max_draft_tokens=request.max_draft_tokens,
            )
        return assigned_id

    def draft_heartbeat(self, request: DraftNodeHeartbeatRequest) -> bool:
        with self._lock:
            record = self._draft_nodes.get(request.draft_node_id)
            if record is None:
                return False
            record.stats = request.stats
            record.available_capacity = request.available_capacity
            record.last_heartbeat_s = time.time()
            return True

    def register_worker(self, request: WorkerRegistration) -> str:
        assigned_id = request.worker_id.strip() or f"worker-{uuid.uuid4().hex[:8]}"
        with self._lock:
            self._workers[assigned_id] = WorkerRecord(
                worker_id=assigned_id,
                address=request.address,
                worker_type=request.worker_type,
                model_info=request.model_info,
                gpu_model=request.gpu_model,
                gpu_memory_bytes=request.gpu_memory_bytes,
                gpu_count=request.gp
[truncated — 6523 more characters]
```

### app/visualize/page.tsx

```typescript
import { DashboardHeader } from "@/components/dashboard-header"
import { SpeculativeVisualizer } from "@/components/speculative-visualizer"

export default function VisualizePage() {
  return (
    <main className="flex h-screen flex-col overflow-hidden bg-background">
      <DashboardHeader />
      <SpeculativeVisualizer />
    </main>
  )
}

```

### app/network/page.tsx

```typescript
import { DashboardHeader } from "@/components/dashboard-header"
import { NetworkExplorer } from "@/components/network-explorer"

export const metadata = {
  title: "Network - Nexus",
  description: "Live view of available Draft and Target nodes on the Nexus distributed inference network.",
}

export default function NetworkPage() {
  return (
    <main className="flex h-screen flex-col overflow-hidden bg-background">
      <DashboardHeader />
      <NetworkExplorer />
    </main>
  )
}

```

### app/provider/page.tsx

```typescript
import { DashboardHeader } from "@/components/dashboard-header"
import { ProviderDashboard } from "@/components/provider-dashboard"

export const metadata = {
  title: "Earnings Dashboard - Nexus",
  description:
    "Track your Draft Node earnings, acceptance rate, and recent payouts on the Nexus network.",
}

export default function ProviderPage() {
  return (
    <main className="flex h-screen flex-col overflow-hidden bg-background">
      <DashboardHeader />
      <ProviderDashboard />
    </main>
  )
}

```

[43 more indexed source files omitted to keep this export small. The full file list is in the Codebase structure section above.]