# Project export: akira

This document was generated by HackStack to give an AI agent context about a hackathon project. Sections are labeled with their provenance; content marked as truncated was cut to keep this document small.

## Project metadata

- Hackathon: UC Berkeley AI Hackathon 2025
- Tagline: Friendly Companion
- Devpost: https://devpost.com/software/akira-br92vo
- GitHub: https://github.com/maxakiralee/akira
- Team: 1 GitHub contributor(s) — Max (26 commits)

## Devpost submission (written by the team)

### Inspiration

More than half of Americans reported feeling lonely this year. Moreso, everyone needs someone to talk to, yet finding a listening ear when you need it most isn’t always easy. It’s difficult to find someone who truly listens, remembers, and cares, no matter the time of day or topic.

### What it does

Akira is your true companion, a fully conversational ai powered by a multi-agent memory architecture you can personalize both visually and functionally. Key features include: Memory-Powered: Learns and remembers what matters. Memory-Powered: Learns and remembers what matters. Multi-Modal: fully conversational, analyzes photos, and fetches live web data on demand. Multi-Modal: fully conversational, analyzes photos, and fetches live web data on demand. Emotion-Aware: Adapts itself to how you feel. Emotion-Aware: Adapts itself to how you feel. Friendly UI: Engaging avatar you can style and interact with. Friendly UI: Engaging avatar you can style and interact with.

### How we built it

Backend Memory Agent Deployed a two-agent architecture on Letta Cloud: Low-Latency Agent uses a lightweight model to deliver instant responses. Context Agent leverages a more powerful model to retrieve and feed relevant context to the low-latency agent. Memory Agent Deployed a two-agent architecture on Letta Cloud: Low-Latency Agent uses a lightweight model to deliver instant responses. Context Agent leverages a more powerful model to retrieve and feed relevant context to the low-latency agent. Multi-Modal Pipeline Voice: Vapi Cloud’s STT → custom Letta LLM → TTS workflow for seamless spoken conversations. Images: Next.js API routes send uploads to Claude-sonnet-4 for image-to-text processing, then inject captions into the Vapi conversation stream. Web Search: Letta’s tool use fetches live web data on demand and augments the dialogue. Multi-Modal Pipeline Voice: Vapi Cloud’s STT → custom Letta LLM → TTS workflow for seamless spoken conversations. Images: Next.js API routes send uploads to Claude-sonnet-4 for image-to-text processing, then inject captions into the Vapi conversation stream. Web Search: Letta’s tool use fetches live web data on demand and augments the dialogue. Emotion-Aware Dialogue Tuned Letta agents and Vapi configurations to produce natural, empathetic flow—mirroring real human engagement. Emotion-Aware Dialogue Tuned Letta agents and Vapi configurations to produce natural, empathetic flow—mirroring real human engagement. Auth & Data Secure user authentication and database management ensure controlled access, encrypted storage, and efficient handling of conversation and profile data. Auth & Data Secure user authentication and database management ensure controlled access, encrypted storage, and efficient handling of conversation and profile data. Frontend Framework & Styling Built with Next.js, TypeScript, and Tailwind CSS for a fast, responsive, and modern UI. Framework & Styling Built with Next.js, TypeScript, and Tailwind CSS for a fast, responsive, and modern UI. 3D Avatar & Customization Three.js renders an interactive avatar you can style—change cosmetics, expressions, and animations—and engage with through gestures and reactions in real time. 3D Avatar & Customization Three.js renders an interactive avatar you can style—change cosmetics, expressions, and animations—and engage with through gestures and reactions in real time.

### Challenges we ran into

Emotion & Speech Tuning Fine-tuning the Letta memory agents and Vapi’s pipeline to produce natural, empathetic speech took extensive iteration. Emotion & Speech Tuning Fine-tuning the Letta memory agents and Vapi’s pipeline to produce natural, empathetic speech took extensive iteration. Integration Complexity Orchestrating Vapi Cloud and Letta Cloud for seamless multimodal functionality (voice, text, image, web search) required careful design and execution across multiple services and frameworks. Integration Complexity Orchestrating Vapi Cloud and Letta Cloud for seamless multimodal functionality (voice, text, image, web search) required careful design and execution across multiple services and frameworks.

### Accomplishments we're proud of

Deployed a fully functional multimodal, memory-enabled AI companion in under 24 hours. Seamlessly combined Vapi Cloud’s STT→Letta LLM→TTS pipeline with Letta Cloud’s two-agent memory architecture for real-time, context-rich conversations. Built a friendly, interactive UI and a fully customizable 3D avatar using Three.js, Next.js, TypeScript, and Tailwind CSS—bringing Akira to life.

### What we learned

The power of a two-agent memory system for smooth, engaging conversations by balancing low-latency responses with deep contextual recall. Orchestrating Vapi Cloud’s STT→LLM→TTS pipeline alongside Letta Cloud’s memory agents revealed best practices for multi-cloud service integration and error handling. Iteratively tuning emotion and speech parameters underscored how small prosody adjustments can dramatically improve perceived empathy and naturalness. Building multi-modal support (voice, text, image, web search) showed us the importance of designing flexible data flows and fallbacks for each modality. Empowering users with avatar customization highlighted how personalization drives deeper connection and sustained engagement.

### What's next

Multi-Language Support Expand beyond English to enable truly global companionship in the user’s native language or dialect. Multi-Language Support Expand beyond English to enable truly global companionship in the user’s native language or dialect. Voice-Call Integration Allow Akira to “call in” and check on you via phone or integrate directly with VoIP for hands-free conversations. Voice-Call Integration Allow Akira to “call in” and check on you via phone or integrate directly with VoIP for hands-free conversations. Long-Term Memory Enhancements Introduce “memory pruning” and “highlight reels” so Akira can surface your most important moments and learn over months or years. Long-Term Memory Enhancements Introduce “memory pruning” and “highlight reels” so Akira can surface your most important moments and learn over months or years. Third-Party Integrations Plug into calendars, music services (Spotify, Apple Music), fitness trackers, and smart-home devices to make Akira an even more useful companion. Third-Party Integrations Plug into calendars, music services (Spotify, Apple Music), fitness trackers, and smart-home devices to make Akira an even more useful companion. More Avatar Configuration Offer finer-grained customization options—hair styles, outfits, expressions, and dynamic animations—so users can craft a unique companion. More Avatar Configuration Offer finer-grained customization options—hair styles, outfits, expressions, and dynamic animations—so users can craft a unique companion. Immersive Three.js World Build a virtual environment where your avatar can explore, interact, and host mini-experiences, turning chats into immersive encounters. Immersive Three.js World Build a virtual environment where your avatar can explore, interact, and host mini-experiences, turning chats into immersive encounters.

## README (from the GitHub repository)

This is a [Next.js](https://nextjs.org/) project bootstrapped with [`create-next-app`](https://github.com/vercel/next.js/tree/canary/packages/create-next-app).

## Getting Started

First, run the development server:

```bash
npm run dev
# or
yarn dev
# or
pnpm dev
# or
bun dev
```

Open [http://localhost:3000](http://localhost:3000) with your browser to see the result.

You can start editing the page by modifying `pages/index.tsx`. The page auto-updates as you edit the file.

[API routes](https://nextjs.org/docs/api-routes/introduction) can be accessed on [http://localhost:3000/api/hello](http://localhost:3000/api/hello). This endpoint can be edited in `pages/api/hello.ts`.

The `pages/api` directory is mapped to `/api/*`. Files in this directory are treated as [API routes](https://nextjs.org/docs/api-routes/introduction) instead of React pages.

This project uses [`next/font`](https://nextjs.org/docs/basic-features/font-optimization) to automatically optimize and load Inter, a custom Google Font.

## Learn More

To learn more about Next.js, take a look at the following resources:

- [Next.js Documentation](https://nextjs.org/docs) - learn about Next.js features and API.
- [Learn Next.js](https://nextjs.org/learn) - an interactive Next.js tutorial.

You can check out [the Next.js GitHub repository](https://github.com/vercel/next.js/) - your feedback and contributions are welcome!

## Deploy on Vercel

The easiest way to deploy your Next.js app is to use the [Vercel Platform](https://vercel.com/new?utm_medium=default-template&filter=next.js&utm_source=create-next-app&utm_campaign=create-next-app-readme) from the creators of Next.js.

Check out our [Next.js deployment documentation](https://nextjs.org/docs/deployment) for more details.


## Detected evidence (automated analysis)

Indexed codebase: 52 recognized source files, 91 KB.
- Anthropic (technology) — detected in the code
- CSS (language) — detected in the code
- JavaScript (language) — detected in the code
- Next.js (technology) — detected in the code
- React (technology) — detected in the code
- Supabase (technology) — detected in the code
- Tailwind CSS (technology) — detected in the code
- TypeScript (language) — detected in the code

## Codebase structure (from repository index)

### Files (73 of 73)

```
.eslintrc.json
.gitignore
assistants/assistant.ts
components.json
components/app/.CosmeticsPanel.tsx.swp
components/app/assistant.tsx
components/app/assistantButton.tsx
components/app/CosmeticsButton.tsx
components/app/CosmeticsPanel.tsx
components/app/createAssistant.tsx
components/app/Experience.tsx
components/app/ImageUploadButton.tsx
components/app/ImageUploader.tsx
components/app/ImageUploadPanel.tsx
components/app/Robot.tsx
components/app/RobotContext.tsx
components/auth/AuthForm.tsx
components/auth/AuthProvider.tsx
components/ui/button.tsx
components/ui/fluidBackground.tsx
config/env.config.ts
example.env
hooks/useCreateAssistant.ts
hooks/useVapi.ts
lib/fluidThree/modules/Advection.js
lib/fluidThree/modules/Common.js
lib/fluidThree/modules/Divergence.js
lib/fluidThree/modules/ExternalForce.js
lib/fluidThree/modules/glsl/sim/advection.frag
lib/fluidThree/modules/glsl/sim/color.frag
lib/fluidThree/modules/glsl/sim/divergence.frag
lib/fluidThree/modules/glsl/sim/externalForce.frag
lib/fluidThree/modules/glsl/sim/face.vert
lib/fluidThree/modules/glsl/sim/line.vert
lib/fluidThree/modules/glsl/sim/mouse.vert
lib/fluidThree/modules/glsl/sim/poisson.frag
lib/fluidThree/modules/glsl/sim/pressure.frag
lib/fluidThree/modules/glsl/sim/viscous.frag
lib/fluidThree/modules/Mouse.js
lib/fluidThree/modules/Output.js
lib/fluidThree/modules/Poisson.js
lib/fluidThree/modules/Pressure.js
lib/fluidThree/modules/ShaderPass.js
lib/fluidThree/modules/Simulation.js
lib/fluidThree/modules/Viscous.js
lib/fluidThree/modules/WebGL.js
lib/fluidThree/utils/EventBus.js
lib/fluidThree/utils/math.js
lib/fluidThree/utils/utils.js
lib/supabase/client.ts
lib/supabase/middleware.ts
lib/supabase/server.ts
lib/types/conversation.type.ts
lib/utils.ts
lib/vapi.sdk.ts
LICENSE
middleware.ts
next.config.mjs
package.json
pages/_app.tsx
pages/_document.tsx
pages/api/analyzeImage.ts
pages/api/createAssistant.ts
pages/api/getUserAgent.ts
pages/auth/confirm.tsx
pages/index.tsx
postcss.config.js
public/models/hat.glb
public/models/robot.glb
README.md
styles/globals.css
tailwind.config.ts
tsconfig.json
```

### Dependencies

- package.json: @anthropic-ai/sdk@^0.54.0, @letta-ai/letta-client@^0.1.132, @radix-ui/react-slot@^1.0.2, @react-three/drei@^9.122.0, @react-three/fiber@^8.18.0, @supabase/auth-helpers-nextjs@^0.10.0, @supabase/supabase-js@^2.50.0, @types/node@^20, @types/react@^18, @types/react-dom@^18, @types/three@^0.177.0, @vapi-ai/web@^1.0.266, autoprefixer@^10.0.1, class-variance-authority@^0.7.0, clsx@^2.1.0, eslint@^8, eslint-config-next@14.1.0, fuse.js@^7.0.0, lucide-react@^0.330.0, next@14.1.0, postcss@^8, raw-loader@^4.0.2, react@^18, react-dom@^18, tailwind-merge@^2.2.1, tailwindcss@^3.3.0, tailwindcss-animate@^1.0.7, three@^0.177.0, typescript@^5

### Recent commits (newest first)

- image vapi.send issue
- bug fix
- voice experiments
- experiment with voices
- automate fluid-three
- added hat to cosmetics panel
- image upload panel
- cosmetics panel + body color
- image upload feature
- fluid-three background integrated
- fluid-three background
- neumorphism redesign
- cleaned up ui
- added supabase auth+db, 1 voice agent architecture per user
- auth ui setup
- blinking eyes :)
- talking mouth :)
- gltf to component
- setup threejs and import robot model
- added assistant transcript

## Key source files (fetched from GitHub, selected and truncated for size)

### package.json

```
{
  "name": "satisfi-broadway-poc-next",
  "version": "0.1.0",
  "private": true,
  "scripts": {
    "dev": "next dev",
    "build": "next build",
    "start": "next start",
    "lint": "next lint"
  },
  "dependencies": {
    "@anthropic-ai/sdk": "^0.54.0",
    "@letta-ai/letta-client": "^0.1.132",
    "@radix-ui/react-slot": "^1.0.2",
    "@react-three/drei": "^9.122.0",
    "@react-three/fiber": "^8.18.0",
    "@supabase/auth-helpers-nextjs": "^0.10.0",
    "@supabase/supabase-js": "^2.50.0",
    "@vapi-ai/web": "^1.0.266",
    "class-variance-authority": "^0.7.0",
    "clsx": "^2.1.0",
    "fuse.js": "^7.0.0",
    "lucide-react": "^0.330.0",
    "next": "14.1.0",
    "react": "^18",
    "react-dom": "^18",
    "tailwind-merge": "^2.2.1",
    "tailwindcss-animate": "^1.0.7",
    "three": "^0.177.0"
  },
  "devDependencies": {
    "@types/node": "^20",
    "@types/react": "^18",
    "@types/react-dom": "^18",
    "@types/three": "^0.177.0",
    "autoprefixer": "^10.0.1",
    "eslint": "^8",
    "eslint-config-next": "14.1.0",
    "postcss": "^8",
    "raw-loader": "^4.0.2",
    "tailwindcss": "^3.3.0",
    "typescript": "^5"
  }
}

```

### pages/index.tsx

```typescript
import Image from "next/image";
import { Inter } from "next/font/google";
import { Assistant } from "@/components/app/assistant";

const inter = Inter({ subsets: ["latin"] });

export default function Home() {
  return (
    <main
      className={`h-screen w-screen overflow-hidden ${inter.className}`}
    >
      <Assistant />
    </main>
  );
}

```

### lib/supabase/server.ts

```typescript
import { createServerComponentClient } from '@supabase/auth-helpers-nextjs'
import { cookies } from 'next/headers'

export const createClient = () => createServerComponentClient({ cookies }) 
```

### postcss.config.js

```javascript
module.exports = {
  plugins: {
    tailwindcss: {},
    autoprefixer: {},
  },
};

```

### tailwind.config.ts

```typescript
import type { Config } from "tailwindcss";

const config: Config = {
  content: [
    "./pages/**/*.{js,ts,jsx,tsx,mdx}",
    "./components/**/*.{js,ts,jsx,tsx,mdx}",
    "./app/**/*.{js,ts,jsx,tsx,mdx}",
  ],
  theme: {
    extend: {
      backgroundImage: {
        "gradient-radial": "radial-gradient(var(--tw-gradient-stops))",
        "gradient-conic":
          "conic-gradient(from 180deg at 50% 50%, var(--tw-gradient-stops))",
      },
    },
  },
  plugins: [],
};
export default config;

```

### middleware.ts

```typescript
import { createMiddlewareClient } from '@supabase/auth-helpers-nextjs'
import { NextResponse } from 'next/server'
import type { NextRequest } from 'next/server'

export async function middleware(req: NextRequest) {
  const res = NextResponse.next()
  const supabase = createMiddlewareClient({ req, res })

  // Refresh session if expired - required for Server Components
  await supabase.auth.getSession()

  return res
}

export const config = {
  matcher: [
    /*
     * Match all request paths except for the ones starting with:
     * - _next/static (static files)
     * - _next/image (image optimization files)
     * - favicon.ico (favicon file)
     * - public (public files)
     */
    '/((?!_next/static|_next/image|favicon.ico|public).*)',
  ],
} 
```

### lib/vapi.sdk.ts

```typescript
import Vapi from "@vapi-ai/web";
import { envConfig } from "@/config/env.config";

export const vapi = new Vapi(envConfig.vapi.token);

```

### lib/utils.ts

```typescript
import { type ClassValue, clsx } from "clsx"
import { twMerge } from "tailwind-merge"

export function cn(...inputs: ClassValue[]) {
  return twMerge(clsx(inputs))
}

```

### config/env.config.ts

```typescript
export const envConfig = {
  vapi: {
    apiUrl: process.env.NEXT_PUBLIC_VAPI_API_URL ?? "https://api.vapi.ai",
    token: process.env.NEXT_PUBLIC_VAPI_WEB_TOKEN ?? "vapi-web-token",
  },
};

```

### styles/globals.css

```css
@tailwind base;
@tailwind components;
@tailwind utilities;

.btn-active {
  @apply bg-red-500 hover:bg-red-700;
}

.btn-loading {
  @apply bg-orange-500 hover:bg-orange-700;
}
.btn-idle {
  @apply bg-green-500 hover:bg-green-700;
}

```

[42 more indexed source files omitted to keep this export small. The full file list is in the Codebase structure section above.]