# Project export: Cache Me If You Can

This document was generated by HackStack to give an AI agent context about a hackathon project. Sections are labeled with their provenance; content marked as truncated was cut to keep this document small.

## Project metadata

- Hackathon: UC Berkeley AI Hackathon 2025
- Tagline: A faster, cheaper GPT by breaking down user prompts into components, checking if any were previously answered, and selectively reusing those results, saving time, tokens, and energy.
- Devpost: https://devpost.com/software/cache-22
- GitHub: https://github.com/MingkuanY/cache-22/
- Team: 1 GitHub contributor(s) — Mingkuan Yan (7 commits)

## Devpost submission (written by the team)

### Inspiration

Large language models are compute-hungry, expensive, and slow at scale. I wanted to build something that could dramatically reduce the cost and latency of using ChatGPT, while making it more sustainable and accessible — especially in real-time, edge, or low-resource environments.

### What it does

Cache-22 is "ChatGPT, but faster and cheaper." It breaks down user prompts into semantically meaningful components, checks if any were previously answered, and selectively reuses those results — saving time, tokens, and energy. Responses are synthesized into a coherent final answer.

### How we built it

Prompt Decomposition: Prompts are split into atomic questions using GPT-3.5-turbo. Similarity-Based Caching: Components are embedded with SentenceTransformer and compared via FAISS. Selective Generation: GPT-4 is only called for novel components; cached responses are reused otherwise. Lightweight Synthesis: GPT-3.5-turbo stitches partial answers into one fluent response. Full-stack app built with Next.js, FastAPI, and OpenAI's API.

### Challenges we ran into

Ensuring decomposed prompts preserve numeric and semantic fidelity (especially for math problems). Managing token accounting and performance metrics. Making responses feel natural when stitched from multiple sources. Hooking up the frontend without introducing latency.

### Accomplishments we're proud of

Meaningful compute/token savings (up to 66% reuse on follow-up prompts). Live working demo with metrics, real-time streaming, and a ChatGPT-style frontend. Generalizable framework for caching at the subprompt level.

### What we learned

GPT models are surprisingly good at prompt decomposition and synthesis. Subprompt-level caching offers a promising middle ground between full memory and raw generation. Even small tweaks (like preserving numeric values) can make or break similarity-based caching.

### What's next

for Cache-22 Faster, smoother frontend to feel identical to real chatbots. Smarter decomposition with better number handling. Persistent vector store and cache across users. Fine-tuned synthesis for domain-specific use cases (e.g., customer support, coding).

## README (from the GitHub repository)

# Cache-22: ChatGPT, but faster and cheaper

Built for Berkeley AI Hacks with the goal of dramatically reducing transformer compute by caching GPT responses at a granular level. Instead of generating an entire response from scratch each time, Cache-22 breaks down user prompts into components, checks if any were previously answered, and selectively reuses those results — saving time, tokens, and energy.

## Why?

Large language models are compute-hungry, expensive, and slow at scale. By reusing knowledge, Cache-22 opens the door to more sustainable and accessible AI — especially in edge settings, real-time apps, or low-resource environments.

## How It Works

Prompt Decomposition
- The user prompt is broken down into semantically meaningful components using GPT-3.5-turbo (e.g., from "What is ChatGPT and how do I use it" to "What is ChatGPT?" and "How do I use ChatGPT?").

Similarity-Based Caching
- Each component is embedded using SentenceTransformer and compared to a vector cache using FAISS.
- If a similar question has been asked before (based on cosine or L2 similarity), its answer is reused.

Selective Generation
- If no similar component is found, the system queries the full GPT model (e.g., GPT-4) to generate a new response.
- Otherwise, cached results are reused.

Lightweight Synthesis
- Once all component responses are collected, they are passed to a small synthesis model (GPT-3.5-turbo) to stitch together a coherent, final reply.
- This drastically reduces the need for full-model inference end-to-end.


## Detected evidence (automated analysis)

Indexed codebase: 16 recognized source files, 17 KB.
- CSS (language) — detected in the code
- Next.js (technology) — detected in the code
- Python (language) — detected in the code
- React (technology) — detected in the code
- Tailwind CSS (technology) — detected in the code
- TypeScript (language) — detected in the code
- FastAPI (technology) — claimed on Devpost, not found in the code

## Codebase structure (from repository index)

### Files (22 of 22)

```
.gitignore
cache/cache_manager.py
client/.gitignore
client/eslint.config.mjs
client/next.config.ts
client/package.json
client/postcss.config.mjs
client/README.md
client/src/app/api/prompt/route.ts
client/src/app/globals.css
client/src/app/layout.tsx
client/src/app/page.tsx
client/src/utils/run_prompt.py
client/tsconfig.json
embedding/embedder.py
embedding/real_embedder.py
llm/openai_client.py
main.py
README.md
server/__init__.py
server/api.py
server/simulate.py
```

### Dependencies

- client/package.json: @eslint/eslintrc@^3, @tailwindcss/postcss@^4, @types/node@^20, @types/react@^19, @types/react-dom@^19, eslint@^9, eslint-config-next@15.3.4, next@15.3.4, react@^19.0.0, react-dom@^19.0.0, tailwindcss@^4, typescript@^5

### Recent commits (newest first)

- Add basic chatbot UI with FastAPI backend
- Show metrics for compute costs saved
- Add real response generated by GPT-4-turbo for novel prompts synthesized by GPT-3.5-turbo
- Add prompt decomposition through GPT-3.5-turbo
- Create basic working cache with sentence formation and cosine similarity
- Create README.md
- first commit

## Key source files (fetched from GitHub, selected and truncated for size)

### client/package.json

```
{
  "name": "client",
  "version": "0.1.0",
  "private": true,
  "scripts": {
    "dev": "next dev --turbopack",
    "build": "next build",
    "start": "next start",
    "lint": "next lint"
  },
  "dependencies": {
    "react": "^19.0.0",
    "react-dom": "^19.0.0",
    "next": "15.3.4"
  },
  "devDependencies": {
    "typescript": "^5",
    "@types/node": "^20",
    "@types/react": "^19",
    "@types/react-dom": "^19",
    "@tailwindcss/postcss": "^4",
    "tailwindcss": "^4",
    "eslint": "^9",
    "eslint-config-next": "15.3.4",
    "@eslint/eslintrc": "^3"
  }
}

```

### main.py

```python
# Only used for CLI testing, obsolete for full stack app

from cache.cache_manager import CacheManager
from embedding.real_embedder import embed
import time
import numpy as np
import tiktoken
from llm.openai_client import decompose_prompt, gpt4_generate_response, gpt3_5_synthesize

cache = CacheManager()
encoder = tiktoken.encoding_for_model("gpt-4-turbo")  # For local token counting

def count_tokens(text):
    return len(encoder.encode(text))

def show_metrics_terminal(
    components, hits, misses, time_taken, tokens_used, tokens_saved, api_calls
):
    print("\n=== GPT Cache Performance ===")
    print(f"Total components processed   : {components}")
    print(f"Cache hits                   : {hits}")
    print(f"Cache misses                 : {misses}")
    print(f"API Calls (compute used)     : {api_calls}")
    print(f"Tokens used (compute spent)  : {tokens_used}")
    print(f"Tokens saved (cache reuse)   : {tokens_saved}")
    print(f"Time taken (s)               : {time_taken:.2f}")
    
    hit_rate = (hits / components * 100) if components > 0 else 0
    print(f"Cache hit rate               : {hit_rate:.1f}%")

    estimated_total_tokens = tokens_used + tokens_saved
    token_savings_pct = (tokens_saved / estimated_total_tokens * 100) if estimated_total_tokens else 0
    print(f"→ Token savings              : {token_savings_pct:.1f}% of total token usage")

    estimated_total_api_calls = components
    api_call_savings = estimated_total_api_calls - api_calls
    api_savings_pct = (api_call_savings / estimated_total_api_calls * 100) if estimated_total_api_calls else 0
    print(f"→ API call savings           : {api_savings_pct:.1f}% fewer calls than full generation")

    print("\n💡 Efficiency Summary:")
    print(f"By reusing cached results, we reduced compute by {int(token_savings_pct)}% and avoided {api_call_savings} full model calls.")
    print(f"That means faster answers, lower cost, and less energy use. 🚀")
    print()


def simulate_prompt_flow(prompt):
    # Decompose into components
    components = decompose_prompt(prompt)
    print("\n--- Decomposed Components ---")
    for i, comp in enumerate(components, 1):
        print(f"{i}. {comp}")
    print()
    
    hits, misses = 0, 0
    tokens_used_api = 0
    tokens_saved_by_cache = 0
    api_calls = 0
    start_time = time.perf_counter()
    
    responses = []

    print("\n--- Processing Components ---")
    for comp in components:
        vec = embed(comp)
        print(f"Vector shape: {np.array(vec).shape}")

        cached = cache.check_cache(vec)

        if cached:
            hits += 1
            estimated_saved = len(encoder.encode(cached))
            tokens_saved_by_cache += estimated_saved
            print(f"[HIT]   \"{comp}\" → Using cached response (estimated tokens saved: {estimated_saved})")
            responses.append(cached)
        else:
            misses += 1
            api_calls += 1
            response, tokens_used = gpt4_generate_response(comp)
            tokens_used_api += tokens_used
            print(f"[MISS]  \"{comp}\" → Generated new response (tokens used: {tokens_used})")
            cache.add_to_cache(vec, response)
            responses.append(response)

    time_taken = time.perf_counter() - start_time

    final_answer = gpt3_5_synthesize(responses)

    return {
        "final_answer": final_answer,
        "metrics": {
            "components": len(components),
            "hits": hits,
            "misses": misses,
            "api_calls": api_calls,
            "tokens_used": tokens_used_api,
            "tokens_saved": tokens_saved_by_cache,
            "time_taken": time_taken,
        }
    }



if __name__ == "__main__":
    print("Welcome to Cache-22!")
    print("Enter 'exit' to quit.\n")

    while True:
        user_prompt = input("Enter your prompt: ")
        if user_prompt.lower() in {"exit", "quit"}:
            break
        result = simulate_prompt_flow(user_prompt)
        print("\n=== Synthesized Final Response ===")
        print(result["final_answer"])
        show_metrics_terminal(**result["metrics"])


```

### client/src/app/layout.tsx

```typescript
import type { Metadata } from "next";
import { Geist, Geist_Mono } from "next/font/google";
import "./globals.css";

const geistSans = Geist({
  variable: "--font-geist-sans",
  subsets: ["latin"],
});

const geistMono = Geist_Mono({
  variable: "--font-geist-mono",
  subsets: ["latin"],
});

export const metadata: Metadata = {
  title: "Cache-22",
  description: "Faster, cheaper ChatGPT",
};

export default function RootLayout({
  children,
}: Readonly<{
  children: React.ReactNode;
}>) {
  return (
    <html lang="en">
      <body
        className={`${geistSans.variable} ${geistMono.variable} antialiased`}
      >
        {children}
      </body>
    </html>
  );
}

```

### client/src/app/page.tsx

```typescript
// File: app/page.tsx (or pages/index.tsx if not using app dir)
'use client'

import { useState } from 'react'

export default function Home() {
  const [prompt, setPrompt] = useState('')
  const [messages, setMessages] = useState<{ role: string; content: string }[]>([])
  const [loading, setLoading] = useState(false)

  const handleSubmit = async (e: React.FormEvent) => {
    e.preventDefault()
    if (!prompt.trim()) return

    const userMessage = { role: 'user', content: prompt }
    setMessages((prev) => [...prev, userMessage])
    setPrompt('')
    setLoading(true)

    try {
      const res = await fetch('http://localhost:8000/api/query', {
        method: 'POST',
        headers: { 'Content-Type': 'application/json' },
        body: JSON.stringify({ prompt }),
      })

      if (!res.ok) throw new Error('Server error')

      const data = await res.json()

      const botMessage = { role: 'assistant', content: data.final_response }
      setMessages((prev) => [...prev, botMessage])
      const metrics = data.metrics
      console.log("metrics: ", metrics)

    } catch (err) {
      const errorMessage = {
        role: 'assistant',
        content: '⚠️ Sorry, something went wrong while contacting the server.',
      }
      setMessages((prev) => [...prev, errorMessage])
      console.error(err)
    }

    setLoading(false)
  }


  return (
    <main className="max-w-2xl mx-auto p-6">
      <h1 className="text-2xl font-semibold mb-4">Cache-22 Chat</h1>

      <div className="border rounded p-4 h-[400px] overflow-y-auto mb-4 bg-gray-50">
        {messages.map((m, i) => (
          <div key={i} className={`mb-2 ${m.role === 'user' ? 'text-right' : 'text-left'}`}>
            <div className={`inline-block px-4 py-2 rounded-lg ${m.role === 'user' ? 'bg-blue-200' : 'bg-gray-300'}`}>
              {m.content}
            </div>
          </div>
        ))}
        {loading && <div className="text-gray-500">Thinking...</div>}
      </div>

      <form onSubmit={handleSubmit} className="flex gap-2">
        <input
          className="flex-1 border rounded px-4 py-2"
          value={prompt}
          onChange={(e) => setPrompt(e.target.value)}
          placeholder="Ask me anything..."
        />
        <button className="bg-blue-500 text-white px-4 py-2 rounded" type="submit">
          Send
        </button>
      </form>
    </main>
  )
}

```

### embedding/embedder.py

```python
def embed(text):
    # Fake embedding: hash to simulate different vectors
    return hash(text) % 10000

```

### client/next.config.ts

```typescript
import type { NextConfig } from "next";

const nextConfig: NextConfig = {
  /* config options here */
};

export default nextConfig;

```

### embedding/real_embedder.py

```python
from sentence_transformers import SentenceTransformer
import numpy as np

# Load the model once
model = SentenceTransformer("all-MiniLM-L6-v2")

def embed(text: str) -> np.ndarray:
    # Return a 1D numpy array of shape (384,)
    vec = model.encode([text], convert_to_numpy=True)  # This returns shape (1, 384)
    return vec[0].astype(np.float32)  # Return the first row

if __name__ == "__main__":
    import sys
    text = sys.argv[1] if len(sys.argv) > 1 else "What is gravity"
    vec = embed(text)
    print(f"Vector shape: {vec.shape}")
    print(f"Vector preview: {vec[:5]}")


```

### server/api.py

```python
from fastapi import FastAPI, Request
from fastapi.middleware.cors import CORSMiddleware
from server.simulate import simulate_prompt_flow  # or however you import it

app = FastAPI()

# CORS settings for dev
app.add_middleware(
    CORSMiddleware,
    allow_origins=["http://localhost:3000"],  # You can limit to ["http://localhost:3000"] if you want
    allow_credentials=True,
    allow_methods=["*"],
    allow_headers=["*"],
)

@app.post("/api/query")
async def query(request: Request):
    data = await request.json()
    prompt = data.get("prompt")
    if not prompt:
        return {"error": "No prompt provided"}

    result = simulate_prompt_flow(prompt)
    return result

```

### cache/cache_manager.py

```python
import faiss
import numpy as np

class CacheManager:
  def __init__(self):
    self.index = faiss.IndexFlatIP(384)  # IP = inner product ≈ cosine similarity
    self.stored_vectors = []
    self.responses = []

  def add_to_cache(self, vec, response):
    vec = vec / np.linalg.norm(vec)
    vec_np = np.array([vec]).astype("float32")
    self.index.add(vec_np)
    self.stored_vectors.append(vec)
    self.responses.append(response)

  def check_cache(self, vec, threshold=0.8):  # bump up threshold since IP goes from -1 to 1
    if len(self.stored_vectors) == 0:
        return None

    vec = vec / np.linalg.norm(vec)
    vec_np = np.array([vec]).astype("float32")
    D, I = self.index.search(vec_np, 1)  # top-1

    similarity = D[0][0]
    print(f"Similarity: {similarity:.4f}")
    if similarity >= threshold:
        return self.responses[I[0][0]]

    return None


```

### llm/openai_client.py

```python
import os
from openai import OpenAI
from dotenv import load_dotenv

load_dotenv()

client = OpenAI(api_key=os.getenv("OPENAI_API_KEY"))

def decompose_prompt(prompt: str, model="gpt-3.5-turbo"):
    response = client.chat.completions.create(
        model=model,
        messages=[
            {
                "role": "system",
                "content": "You're a tool that rewrites a long user prompt into 2–4 reusable subquestions. Do not number or label the subquestions. Just return each one on a new line as plain text."
            },
            {
                "role": "user",
                "content": f"Break this prompt down into atomic subquestions. Do not number them:\n{prompt}"
            }
        ],
        temperature=0.3,
    )
    text = response.choices[0].message.content
    return [line.strip("- ").strip() for line in text.split("\n") if line.strip()]

def gpt4_generate_response(prompt: str, model="gpt-4-turbo"):
    response = client.chat.completions.create(
        model=model,
        messages=[
            {"role": "user", "content": prompt}
        ],
        temperature=0.7,
    )
    text = response.choices[0].message.content.strip()
    tokens_used = response.usage.total_tokens
    return text, tokens_used

def gpt3_5_synthesize(responses: list[str], model="gpt-3.5-turbo"):
    prompt = (
        "Synthesize the following responses into one coherent, concise answer:\n\n"
        + "\n\n".join(responses)
    )
    response = client.chat.completions.create(
        model=model,
        messages=[{"role": "user", "content": prompt}],
        temperature=0.3,
    )
    text = response.choices[0].message.content.strip()
    tokens_used = response.usage.total_tokens
    return text, tokens_used

```

[3 more indexed source files omitted to keep this export small. The full file list is in the Codebase structure section above.]