Project Info
This project did not submit a demo video on Devpost.
Inspiration
Large language models are compute-hungry, expensive, and slow at scale. I wanted to build something that could dramatically reduce the cost and latency of using ChatGPT, while making it more sustainable and accessible — especially in real-time, edge, or low-resource environments.
What it does
Cache-22 is "ChatGPT, but faster and cheaper." It breaks down user prompts into semantically meaningful components, checks if any were previously answered, and selectively reuses those results — saving time, tokens, and energy. Responses are synthesized into a coherent final answer.
How we built it
Prompt Decomposition: Prompts are split into atomic questions using GPT-3.5-turbo. Similarity-Based Caching: Components are embedded with SentenceTransformer and compared via FAISS. Selective Generation: GPT-4 is only called for novel components; cached responses are reused otherwise. Lightweight Synthesis: GPT-3.5-turbo stitches partial answers into one fluent response. Full-stack app built with Next.js, FastAPI, and OpenAI's API.
Challenges we ran into
Ensuring decomposed prompts preserve numeric and semantic fidelity (especially for math problems). Managing token accounting and performance metrics. Making responses feel natural when stitched from multiple sources. Hooking up the frontend without introducing latency.
Accomplishments we're proud of
Meaningful compute/token savings (up to 66% reuse on follow-up prompts). Live working demo with metrics, real-time streaming, and a ChatGPT-style frontend. Generalizable framework for caching at the subprompt level.
What we learned
GPT models are surprisingly good at prompt decomposition and synthesis. Subprompt-level caching offers a promising middle ground between full memory and raw generation. Even small tweaks (like preserving numeric values) can make or break similarity-based caching.
What's next
for Cache-22 Faster, smoother frontend to feel identical to real chatbots. Smarter decomposition with better number handling. Persistent vector store and cache across users. Fine-tuned synthesis for domain-specific use cases (e.g., customer support, coding).
Cache-22: ChatGPT, but faster and cheaper
Built for Berkeley AI Hacks with the goal of dramatically reducing transformer compute by caching GPT responses at a granular level. Instead of generating an entire response from scratch each time, Cache-22 breaks down user prompts into components, checks if any were previously answered, and selectively reuses those results — saving time, tokens, and energy.
Why?
Large language models are compute-hungry, expensive, and slow at scale. By reusing knowledge, Cache-22 opens the door to more sustainable and accessible AI — especially in edge settings, real-time apps, or low-resource environments.
How It Works
Prompt Decomposition
- The user prompt is broken down into semantically meaningful components using GPT-3.5-turbo (e.g., from "What is ChatGPT and how do I use it" to "What is ChatGPT?" and "How do I use ChatGPT?").
Similarity-Based Caching
- Each component is embedded using SentenceTransformer and compared to a vector cache using FAISS.
- If a similar question has been asked before (based on cosine or L2 similarity), its answer is reused.
Selective Generation
- If no similar component is found, the system queries the full GPT model (e.g., GPT-4) to generate a new response.
- Otherwise, cached results are reused.
Lightweight Synthesis
- Once all component responses are collected, they are passed to a small synthesis model (GPT-3.5-turbo) to stitch together a coherent, final reply.
- This drastically reduces the need for full-model inference end-to-end.
Analysis
View
Metric
- 7
Figures cover GitHub contributors during the hackathon window. A co-authored commit counts in full for each author, so per-member totals add up to more than the whole-team figures.
Technology
- CSSIn code
- Next.jsIn code
- PythonIn code
- ReactIn code
- Tailwind CSSIn code
- TypeScriptIn code
- FastAPIClaimed
6 of 7 appear in the indexed code. 1 claimed on Devpost could not be matched to code, which may simply mean the tool leaves no trace in the repository.
AI coding agents
No AI coding agent signals were found in this repository.
Detected from committed agent config files and commit authorship. Absence of a signal is not proof an agent was unused.
Codebase size
Source size
17 KB
Source files
16
Counts recognized source files only; vendored directories, binaries and lockfiles are excluded, so this is smaller than the repository on disk.
Repository
MingkuanY/cache-22
28 files · 231 KB · @ 88aa08d
Structure
Interface
3 files · 11%Screens, components and styles rendered to the user.
API & routing
4 files · 14%Request entry points: routes, handlers and controllers.
Application logic
6 files · 21%Domain rules, services and shared utilities.
Supporting
Layers are inferred from where files sit in the tree, not from reading the code. A project that names its directories unconventionally will read oddly here — open the file browser to check anything the diagram implies.
Languages
- Python61%
- TypeScript19%
- Markdown18%
- CSS3%
Share of indexed source by file size. Binary and vendored files are excluded.
Dependencies
client/package.json
npm · 12- next
- react
- react-dom
- +9 more
Declared in the repository’s manifests at the indexed commit. A declared package is not proof it is used, and runtime dependencies are listed first.
This project’s features have not been analysed yet.
Export this project's context (description, README, evidence, key source files) to chat with an AI agent elsewhere.