# Project export: Clarity Coach

This document was generated by HackStack to give an AI agent context about a hackathon project. Sections are labeled with their provenance; content marked as truncated was cut to keep this document small.

## Project metadata

- Hackathon: TreeHacks 2026
- Tagline: Build speaking confidence through interactive conversations and smart coaching!
- Devpost: https://devpost.com/software/clarity-coach-1oxkvl
- GitHub: https://github.com/SamhitaK10/clarity-coach
- Video: https://www.youtube.com/embed/pnJdJuRz2vA?enablejsapi=1&hl=en_US&rel=0&start=&version=3&wmode=transparent
- Team: 1 GitHub contributor(s) — Samhita (9 commits)

## Devpost submission (written by the team)

### Inspiration

Roughly 1 in 5 U.S. residents grew up speaking a language other than English at home. ESL speakers often face invisible friction: interviewing for jobs, speaking up in class, or even navigating casual social interactions can feel intimidating when language confidence lags behind ability – especially in modern-day linguistically homogenous societies. All four of our team members grew up in bilingual households. We know firsthand the hesitation before speaking, the anxiety after mispronouncing a word, and the pervading feeling of being evaluated rather than understood. We realized that while language apps teach vocabulary and grammar, very few platforms simulate real-world speaking scenarios with personalized feedback. So we built Clarity Coach: an AI-powered speaking platform that helps users build confidence in their second-language communication skills. Through real-time video recording, personalized speech and body language feedback, visual and audio analytics, and gamified progress tracking, Clarity Coach helps individuals acclimate to unfamiliar linguistic environments in a safe, welcoming, yet productive learning platform.

### What it does

Clarity Coach lets users record a 45 second video of themselves and analyzes their posture, body language, eye contact, and speech. The website then provides users with a specialized analysis of their presentation, rating on posture, eye contact, clarity, and pacing. The user can ask specific questions relating to their performance and improvement to a human-sounding conversational AI. Finally, Clarity Coach tracks prior runs of recordings and saves them and respective stats to show user progress over time.

### How we built it

We built Clarity Coach as a full-stack AI application with the following stack: • Backend: FastAPI + Uvicorn for asynchronous API endpoints; Pydantic for structured validation; python-multipart and aiofiles for video uploads; MoviePy for audio extraction; OpenCV + MediaPipe for video/pose processing; OpenAI + Anthropic APIs for transcript evaluation and feedback generation; Modal for scalable processing • Frontend: HTML/CSS/JS for general web development; MediaRecorder API for in-browser recording

### Challenges we ran into

We ran into lots of fickle issues along the way, requiring us to relentlessly and strategically debug. Some of the main challenges we faced were: • Slow computing times of video and audio analyzing models → Solved by offloading inference to GPU-backed Modal containers to accelerate model processing • Creating a natural conversational coaching experience → Solved by designing structured LLM outputs and prompt constraints to produce short voice-ready responses and a follow-up prompt, enabling iterative turn-taking • Realtime voice sessions connecting but not returning audio → Solved by properly attaching WebRTC audio tracks and using ephemeral session tokens for authenticated streaming • Synchronizing multiple AI services in one pipeline → Solved by sequencing transcription, LLM analysis, and text-to-speech generation to ensure reliable end-to-end responses

### What we learned

As a team of beginner hackers, we learned SO MUCH within this period of 36 hours. Since the initial check-in, we’ve been living in and breathing the innovative air of Huang basement. Whether it was learning how to reason our idea into fruition or how to debug a seemingly trivial issue, we’ve become more inventive, more creative, and more confident in our abilities to solve problems in our communities. In terms of concrete technical abilities, we learned: • how to capture live audio using the MediaRecorder API and navigator.mediaDevices.getUserMedia() • how to process chunks of recorded media and send them to a backend server • how to structure a full-stack app with separated frontend and backend logic • how to design feedback systems that turn raw model outputs into substantial insights • how to stream microphone input as Blob chunks, encode them, and transmit via multipart/form-data to an Express backend • how to orchestrate a real-time speech pipeline combining Whisper transcription, Claude LLM analysis, and ElevenLabs text-to-speech synthesis • and lots more!

### What's next

Next, we want to bring Clarity Coach beyond a standalone web app and make it something users can access wherever they practice speaking. We’re exploring browser extensions and lightweight integrations that provide feedback directly inside tools people already use! We’re also excited about building interactive AI avatars that simulate different interviewers or audiences (from a fast-paced recruiter to a skeptical panelist) so users can practice adapting their delivery in more realistic, dynamic environments.

## README (from the GitHub repository)

## 🎤 Interview Coaching API

Express-based backend powering the conversational audio coaching system for **Clarity Coach**, an AI-driven communication training platform.

This service captures spoken responses, performs speech transcription, evaluates communication clarity using large language models, and generates spoken coaching feedback to enable iterative, conversational practice.

Devpost: https://devpost.com/software/clarity-coach-1oxkvl

---

## 🎯 System Overview

The Interview Coaching API enables real-time interview practice by transforming spoken responses into structured communication insights and spoken coaching feedback.

The system evaluates clarity, delivery, phrasing, pacing, and filler word usage, then produces actionable coaching guidance and follow-up prompts to simulate realistic interview dialogue.

This service is designed for low-latency feedback loops and conversational continuity.

---

## 🧠 Design Goals

• enable realistic interview simulation  
• provide actionable communication feedback  
• support iterative practice through conversational loops  
• maintain low-latency audio processing  
• separate frontend capture from backend analysis  
• support scalable AI-driven coaching workflows  

---

## ⚙️ End-to-End Processing Pipeline

Microphone Capture  
→ Browser MediaRecorder stream  
→ Multipart audio upload  
→ OpenAI Whisper API transcription  
→ Claude API communication analysis  
→ structured coaching generation  
→ ElevenLabs text-to-speech synthesis  
→ audio playback to client  
→ AI follow-up prompt generation  
→ continued conversational loop  

---

## ✨ Core Capabilities

### Speech Processing
• browser-based audio capture  
• multipart audio ingestion  
• high-accuracy speech-to-text transcription  

### Communication Analysis
• clarity evaluation  
• grammar and phrasing refinement  
• filler word detection and reduction strategies  
• delivery and pacing feedback  
• confidence and tone guidance  

### Conversational Coaching
• AI-generated follow-up questions  
• iterative practice loop  
• conversational response scaffolding  
• interview simulation flow  

### Voice Feedback
• low-latency TTS synthesis  
• natural spoken coaching delivery  
• base64 audio streaming to client  

---

## 🧠 Conversational Coaching Loop

1. user records interview response  
2. audio is captured via MediaRecorder  
3. Whisper API transcribes speech  
4. Claude API analyzes communication quality  
5. coaching feedback is generated  
6. ElevenLabs synthesizes spoken coaching  
7. AI produces follow-up prompt  
8. user responds and continues loop  

This loop creates a dynamic interview simulation environment rather than static feedback.

---

## 🏗 Architecture Role

This service powers the **audio coaching pipeline** within Clarity Coach.

It integrates with the video analysis backend to provide full-spectrum communication feedback.



## Detected evidence (automated analysis)

Indexed codebase: 6 recognized source files, 10 KB.
- Anthropic (technology) — detected in the code
- Express (technology) — detected in the code
- HTML (language) — detected in the code
- JavaScript (language) — detected in the code
- OpenAI (technology) — detected in the code
- CSS (language) — claimed on Devpost, not found in the code
- Node.js (technology) — claimed on Devpost, not found in the code
- Python (language) — claimed on Devpost, not found in the code

## Codebase structure (from repository index)

### Files (9 of 9)

```
.env.example
.gitignore
package.json
README.md
routes/analyze.js
routes/transcribe.js
routes/voice-feedback.js
server.js
test.html
```

### Dependencies

- package.json: @anthropic-ai/sdk@^0.32.1, cors@^2.8.5, dotenv@^16.4.5, elevenlabs@^1.59.0, elevenlabs-node@^2.0.3, express@^4.21.0, multer@^2.0.0, node-fetch@^3.3.2, openai@^4.73.0

### Recent commits (newest first)

- Update README.md
- Revise README for improved clarity and detail
- Revise README for improved clarity and structure
- Revise README for improved clarity and structure
- Enhance README with features and setup details
- Delete .cursor/rules directory
- Refactor coaching system and feedback response
- Update README to remove ELEVENLABS_VOICE_ID
- clean backend commit
- Backend complete: transcription, coaching, and voice feedback

## Key source files (fetched from GitHub, selected and truncated for size)

### package.json

```
{
  "name": "interview-coaching-api",
  "version": "1.0.0",
  "description": "Backend API for interview coaching app",
  "main": "server.js",
  "scripts": {
    "start": "node server.js",
    "dev": "node --watch server.js"
  },
  "dependencies": {
    "@anthropic-ai/sdk": "^0.32.1",
    "cors": "^2.8.5",
    "dotenv": "^16.4.5",
    "elevenlabs": "^1.59.0",
    "elevenlabs-node": "^2.0.3",
    "express": "^4.21.0",
    "multer": "^2.0.0",
    "node-fetch": "^3.3.2",
    "openai": "^4.73.0"
  }
}

```

### server.js

```javascript
require('dotenv').config();
console.log("Anthropic key loaded:", !!process.env.ANTHROPIC_API_KEY);
const express = require('express');
const cors = require('cors');

const transcribeRouter = require('./routes/transcribe');
const analyzeRouter = require('./routes/analyze');
const voiceFeedbackRouter = require('./routes/voice-feedback');

const app = express();
const PORT = process.env.PORT || 3000;

app.use(cors());
app.use(express.json());

app.use('/api/transcribe', transcribeRouter);
app.use('/api/analyze', analyzeRouter);
app.use('/api/voice-feedback', voiceFeedbackRouter);

app.get('/health', (req, res) => {
  res.json({ status: 'ok', message: 'Interview coaching API is running' });
});

app.use((err, req, res, next) => {
  console.error(err.stack);
  res.status(err.status || 500).json({
    error: err.message || 'Internal server error',
  });
});

app.listen(PORT, () => {
  console.log(`Server running on http://localhost:${PORT}`);
});

```

### test.html

```html
<!DOCTYPE html>
<html>
<body>
  <h2>Upload Audio Test</h2>
  <form action="http://localhost:3000/api/transcribe" method="post" enctype="multipart/form-data">
    <input type="file" name="audio" />
    <button type="submit">Upload</button>
  </form>
</body>
</html>

```

### routes/voice-feedback.js

```javascript
const express = require("express");
const fetch = require("node-fetch");

const router = express.Router();

router.post("/", async (req, res) => {
  try {
    const { text } = req.body;

    const response = await fetch(
      "https://api.elevenlabs.io/v1/text-to-speech/EXAVITQu4vr4xnSDxMaL",
      {
        method: "POST",
        headers: {
          "xi-api-key": process.env.ELEVENLABS_API_KEY,
          "Content-Type": "application/json",
        },
        body: JSON.stringify({
          text,
          model_id: "eleven_multilingual_v2",
        }),
      }
    );

    const audioBuffer = await response.arrayBuffer();
    const base64Audio = Buffer.from(audioBuffer).toString("base64");

    res.json({ audio: base64Audio });
  } catch (err) {
    console.error(err);
    res.status(500).json({ error: "Voice generation failed" });
  }
});

module.exports = router;

```

### routes/transcribe.js

```javascript
const express = require('express');
const multer = require('multer');
const OpenAI = require('openai');
const path = require('path');
const fs = require('fs');
const fetch = require('node-fetch');

const router = express.Router();
const openai = new OpenAI({ apiKey: process.env.OPENAI_API_KEY });

/* ---------- upload setup ---------- */

const uploadDir = path.join(__dirname, '..', 'uploads');
if (!fs.existsSync(uploadDir)) {
  fs.mkdirSync(uploadDir, { recursive: true });
}

const storage = multer.diskStorage({
  destination: (req, file, cb) => cb(null, uploadDir),
  filename: (req, file, cb) => {
    const unique = Date.now() + '-' + Math.round(Math.random() * 1e9);
    const ext = path.extname(file.originalname) || '.webm';
    cb(null, unique + ext);
  },
});

const upload = multer({ storage });

/* ---------- main route ---------- */

router.post('/', upload.single('audio'), async (req, res) => {
  if (!req.file) {
    return res.status(400).json({ error: 'No audio file uploaded' });
  }

  const filePath = req.file.path;

  try {
    // 1️⃣ Transcribe audio
    const buffer = await fs.promises.readFile(filePath);
    const file = await OpenAI.toFile(buffer, path.basename(filePath));

    const transcription = await openai.audio.transcriptions.create({
      file,
      model: 'whisper-1'
    });

    const transcript = transcription.text || transcription;

    // 2️⃣ Get coaching feedback
    const analyzeRes = await fetch("http://localhost:3000/api/analyze", {
      method: "POST",
      headers: { "Content-Type": "application/json" },
      body: JSON.stringify({ text: transcript })
    });

    const feedback = await analyzeRes.json();

    // 3️⃣ Generate spoken coaching tip
    const voiceRes = await fetch("http://localhost:3000/api/voice-feedback", {
      method: "POST",
      headers: { "Content-Type": "application/json" },
      body: JSON.stringify({
        text: feedback.exampleSentence
      })
    });

    const voice = await voiceRes.json();

    // 4️⃣ Return everything
    res.json({
      transcript,
      feedback,
      audio: voice.audio
    });

  } catch (err) {
    console.error(err);
    res.status(500).json({ error: "Processing failed" });
  } finally {
    fs.unlink(filePath, () => {});
  }
});

module.exports = router;

```

### routes/analyze.js

```javascript
const express = require('express');
const Anthropic = require('@anthropic-ai/sdk');

const router = express.Router();
const anthropic = new Anthicropic({ apiKey: process.env.ANTHROPIC_API_KEY });

const COACHING_SYSTEM = `
You are a supportive interview coach helping non-native English speakers improve clarity and confidence.

Given an interview answer, respond ONLY with valid JSON using these keys:

- clarity: Brief feedback on clarity and how to improve it.
- grammar: Specific grammar corrections if needed.
- phrasing: Suggest more natural or idiomatic phrasing.
- fillerWords: Identify filler words and advise reducing them.
- exampleSentence: ONE improved example sentence.
- followUp: Ask ONE short supportive follow-up question to help the user improve their answer.
- reply: A short, natural spoken coaching response (max 2 sentences) that sounds like a real coach speaking directly to the user.

Guidelines:
Keep tone supportive and human.
Keep responses concise.
End the reply with the follow-up question.
Respond ONLY with JSON.
`;

router.post('/', express.json(), async (req, res, next) => {
  if (!process.env.ANTHROPIC_API_KEY) {
    return res.status(503).json({ error: 'Analysis not configured: ANTHROPIC_API_KEY missing' });
  }

  const { transcript, question } = req.body;
  const text = transcript || req.body.text;

  if (!text || typeof text !== 'string') {
    return res.status(400).json({ error: 'Request body must include "transcript" or "text".' });
  }

  const userContent = question
    ? `Question: ${question}\n\nAnswer:\n${text}`
    : `Answer:\n${text}`;

  try {
    const message = await anthropic.messages.create({
      model: 'claude-sonnet-4-20250514',
      max_tokens: 800,
      system: COACHING_SYSTEM,
      messages: [{ role: 'user', content: userContent }],
    });

    const block = message.content.find((b) => b.type === 'text');
    const raw = block?.text ?? '';

    let feedback = null;
    try {
      const jsonMatch = raw.match(/\{[\s\S]*\}/);
      feedback = jsonMatch ? JSON.parse(jsonMatch[0]) : null;
    } catch {
      feedback = null;
    }

    if (!feedback) {
      return res.json({ error: "Coaching generation failed." });
    }

    // conversational voice reply (used by ElevenLabs)
    const coachReply =
      feedback.reply ||
      `${feedback.exampleSentence} ${feedback.followUp}`;

    res.json({
      clarity: feedback.clarity ?? '',
      grammar: feedback.grammar ?? '',
      phrasing: feedback.phrasing ?? '',
      fillerWords: feedback.fillerWords ?? '',
      exampleSentence: feedback.exampleSentence ?? '',
      followUp: feedback.followUp ?? 'Try answering again more concisely.',
      reply: feedback.reply ?? '',
      coachReply
    });

  } catch (err) {
    next(err);
  }
});

module.exports = router;

```