# Project export: Noq

This document was generated by HackStack to give an AI agent context about a hackathon project. Sections are labeled with their provenance; content marked as truncated was cut to keep this document small.

## Project metadata

- Hackathon: Cal Hacks 11.0
- Tagline: AI-Powered Real-Time Note Taker
- Devpost: https://devpost.com/software/noq-92iysh
- GitHub: https://github.com/pnavab/calhacks11
- Video: https://www.youtube.com/embed/NWODx14Z6pw?enablejsapi=1&hl=en_US&rel=0&start=&version=3&wmode=transparent
- Result: winner (Groq: The Best on Groq)
- Team: 4 GitHub contributor(s) — joshua-demo (31 commits), pnavab (12 commits), Jonathan Nguyen (4 commits), gursheyss (3 commits)

## Devpost submission (written by the team)

### Inspiration

Our team has consistently tried to improve our personal school experiences slowly but surely by optimizing every small process we can. This tool takes it to the next level and completely streamlines the learning process up to the point where you can start studying!

### What it does

Noq listens into your lecture, compiling and categorizing all of the lecture’s most important points for you. They are automatically expanded and revised, separating all the different ideas to be ready for vectorization. Once entered into the vectorstore, the RAG search engine allows for context-driven searching through your notes to easily find what you need. You can also generate diagrams for specific lines within a note, easily visualizing what has been summarized for the user.

### How we built it

Using Groq as the main infrastructure behind our application, we leveraged its highly fast speeds to run multiple inference calls in our backend AI agent network almost instantly. ChromaDB also served as the vectorstore to store the notes and run our semantic search engine for easy querying. Our actual website was also built in Next.js and Tailwind.

### Challenges we ran into

It was hard creating a complex real-time app without streaming with minimal delay and having AI agents acting in the background

### Accomplishments we're proud of

Being able to integrate external tools and technologies such as embedding models with Groq, as well as maintaining multiple servers interacting with each other to create a seamless and extremely fast interaction

### What we learned

Frontend is hard

### What's next

Completely integrate it with personal calendars and email, using tool calling for extracting upcoming events and deadlines and automatically creating reminders for those. This is a product we could easily see becoming a staple in our daily lives, and will definitely continue improving upon it!

## README (from the GitHub repository)

This is a [Next.js](https://nextjs.org) project bootstrapped with [`create-next-app`](https://nextjs.org/docs/app/api-reference/cli/create-next-app).

## Getting Started

First, run the development server:

```bash
npm run dev
# or
yarn dev
# or
pnpm dev
# or
bun dev
```

Open [http://localhost:3000](http://localhost:3000) with your browser to see the result.

You can start editing the page by modifying `app/page.tsx`. The page auto-updates as you edit the file.

This project uses [`next/font`](https://nextjs.org/docs/app/building-your-application/optimizing/fonts) to automatically optimize and load [Geist](https://vercel.com/font), a new font family for Vercel.

## Learn More

To learn more about Next.js, take a look at the following resources:

- [Next.js Documentation](https://nextjs.org/docs) - learn about Next.js features and API.
- [Learn Next.js](https://nextjs.org/learn) - an interactive Next.js tutorial.

You can check out [the Next.js GitHub repository](https://github.com/vercel/next.js) - your feedback and contributions are welcome!

## Deploy on Vercel

The easiest way to deploy your Next.js app is to use the [Vercel Platform](https://vercel.com/new?utm_medium=default-template&filter=next.js&utm_source=create-next-app&utm_campaign=create-next-app-readme) from the creators of Next.js.

Check out our [Next.js deployment documentation](https://nextjs.org/docs/app/building-your-application/deploying) for more details.


## Detected evidence (automated analysis)

Indexed codebase: 35 recognized source files, 318 KB.
- CSS (language) — detected in the code
- JavaScript (language) — detected in the code
- Next.js (technology) — detected in the code
- OpenAI (technology) — detected in the code
- Python (language) — detected in the code
- React (technology) — detected in the code
- Tailwind CSS (technology) — detected in the code
- TypeScript (language) — detected in the code
- Vercel AI SDK (technology) — detected in the code

## Codebase structure (from repository index)

### Files (42 of 42)

```
.DS_Store
.gitignore
app/actions/groq.js
app/api/diagram/route.ts
app/api/groq/route.js
app/api/search/route.js
app/api/summarize/route.js
app/api/transcribe/route.js
app/diagram/page.tsx
app/globals.css
app/groq/page.jsx
app/layout.tsx
app/page.tsx
app/search/page.js
app/testing/page.jsx
chroma/chroma.sqlite3
components.json
components/app-sidebar.tsx
components/diagram-generator.tsx
components/diagram-list.tsx
components/diagram-modal.tsx
components/real-time-transcription-note-app.tsx
components/search-engine.tsx
components/ui/button.tsx
components/ui/input.tsx
components/ui/separator.tsx
components/ui/sheet.tsx
components/ui/sidebar.tsx
components/ui/skeleton.tsx
components/ui/textarea.tsx
components/ui/tooltip.tsx
hooks/use-mobile.tsx
hooks/useDebounce.tsx
lib/utils.ts
next-env.d.ts
next.config.mjs
package.json
postcss.config.mjs
README.md
server.py
tailwind.config.ts
tsconfig.json
```

### Dependencies

- package.json: @ai-sdk/groq@^0.0.1, @radix-ui/react-dialog@^1.1.2, @radix-ui/react-separator@^1.1.0, @radix-ui/react-slot@^1.1.0, @radix-ui/react-tooltip@^1.1.3, @types/node@^20, @types/react@^18, @types/react-dom@^18, ai@^3.4.16, chromadb@^1.9.2, chromadb-default-embed@^2.13.2, class-variance-authority@^0.7.0, clsx@^2.1.1, eslint@^8, eslint-config-next@14.2.15, groq-sdk@^0.3.2, lucide-react@^0.453.0, mermaid@^11.3.0, next@14.2.15, openai@^4.68.1, postcss@^8, react@^18, react-dom@^18, react-markdown@^9.0.1, tailwind-merge@^2.5.4, tailwindcss@^3.4.1, tailwindcss-animate@^1.0.7, typescript@^5, zod@^3.23.8

### Recent commits (newest first)

- better proompting
- updated seo
- better diagram gen model
- better proompting for diagram gen
- instant debounce when pasting
- centered modal
- renders based on section in markdown
- diagram generation works way better
- better diagram outputs
- added diagram gen on main page
- expanded live transcription text area
- editable markdown
- fixed ui bug that exnotes not expanded
- added search side bar
- search functionality
- backend rag searching works
- better prompting for transcribe endpoint
- work
- Merge branch 'main' of https://github.com/pnavab/calhacks11
- rendered markdown output

## Key source files (fetched from GitHub, selected and truncated for size)

### package.json

```
{
  "name": "calhacks11",
  "version": "0.1.0",
  "private": true,
  "scripts": {
    "dev": "next dev --turbo",
    "build": "next build",
    "start": "next start",
    "lint": "next lint"
  },
  "dependencies": {
    "@ai-sdk/groq": "^0.0.1",
    "@radix-ui/react-dialog": "^1.1.2",
    "@radix-ui/react-separator": "^1.1.0",
    "@radix-ui/react-slot": "^1.1.0",
    "@radix-ui/react-tooltip": "^1.1.3",
    "ai": "^3.4.16",
    "chromadb": "^1.9.2",
    "chromadb-default-embed": "^2.13.2",
    "class-variance-authority": "^0.7.0",
    "clsx": "^2.1.1",
    "groq-sdk": "^0.3.2",
    "lucide-react": "^0.453.0",
    "mermaid": "^11.3.0",
    "next": "14.2.15",
    "openai": "^4.68.1",
    "react": "^18",
    "react-dom": "^18",
    "react-markdown": "^9.0.1",
    "tailwind-merge": "^2.5.4",
    "tailwindcss-animate": "^1.0.7",
    "zod": "^3.23.8"
  },
  "devDependencies": {
    "@types/node": "^20",
    "@types/react": "^18",
    "@types/react-dom": "^18",
    "eslint": "^8",
    "eslint-config-next": "14.2.15",
    "postcss": "^8",
    "tailwindcss": "^3.4.1",
    "typescript": "^5"
  }
}

```

### server.py

```python
from fastapi import FastAPI, Request, HTTPException
from fastapi.middleware.cors import CORSMiddleware
import uvicorn
import chromadb
from uuid import uuid4
from sentence_transformers import SentenceTransformer
embedding_model = SentenceTransformer('all-mpnet-base-v2')
print("embedding model loaded", embedding_model)

app = FastAPI()
origins = ["*"]
app.add_middleware(
    CORSMiddleware,
    allow_origins=origins,
    allow_credentials=True,
    allow_methods=["*"],
    allow_headers=["*"],
)

chroma_client = chromadb.PersistentClient(
    path="./chroma",  # Directory for storing the database
)
# chroma_client.delete_collection("collection")
collection = chroma_client.get_or_create_collection("collection")

def upload_documents(dict_list):
    ids = [str(uuid4()) for _ in dict_list]
    documents = [d['content'] for d in dict_list]
    metadatas = [{'title': d['title']} for d in dict_list]
    embeddings = embedding_model.encode(documents)

    collection.add(
        documents=documents,
        ids=ids,
        metadatas=metadatas,
        embeddings=embeddings.tolist(),
    )

@app.post("/upload")
async def process_notes(request: Request):
    try:
        # Parse JSON body directly from the request
        notes = await request.json()
        print("received notes:", notes)
        # Ensure notes is a list
        if not isinstance(notes, list):
            raise HTTPException(status_code=400, detail="Invalid format: Expected a list of notes.")

        upload_documents(notes)

        return {"status": "success", "message": "Notes processed and stored successfully."}
    
    except Exception as e:
        raise HTTPException(status_code=500, detail=str(e))
    
@app.get("/notes")
async def get_notes():
    # Get all documents from the collection
    documents = collection.get()
    print("retrieved notes:", documents)
    return documents

@app.post("/search")
async def search_notes(request: Request):
    try:
        # Parse JSON body directly from the request
        query = await request.json()
        print("received query:", query)

        query_embedding = embedding_model.encode([query])
        results = collection.query(
            query_embeddings=query_embedding.tolist(),
            n_results=5
        )
        return results
    except Exception as e:
        print(str(e))
        raise HTTPException(status_code=500, detail=str(e))

if __name__ == "__main__":
    uvicorn.run("server:app", port=8000, reload=True)
```

### app/page.tsx

```typescript
import Component from "@/components/real-time-transcription-note-app";

export default function Home() {
  return (
    <Component />
  );
}

```

### app/layout.tsx

```typescript
import type { Metadata } from "next";
import localFont from "next/font/local";
import "./globals.css";
import { SidebarTrigger, SidebarProvider } from "@/components/ui/sidebar";
import { AppSidebar } from "@/components/app-sidebar";

export const metadata: Metadata = {
  title: "Noq",
  description: "Notes with Groq",
};

export default function RootLayout({
  children,
}: Readonly<{
  children: React.ReactNode;
}>) {
  return (
    <html>
      <body>
        <SidebarProvider>
          <AppSidebar />
          <main className="w-full">
            <SidebarTrigger />
            {children}
          </main>
        </SidebarProvider>
      </body>
    </html>
  );
}

```

### app/search/page.js

```javascript
'use client';
import SearchEngine from "@/components/search-engine";

export default function SearchPage() {
  return <SearchEngine />;
}
```

### app/diagram/page.tsx

```typescript
import DiagramGenerator from "@/components/diagram-generator";

export default function DiagramTestPage() {
  return <DiagramGenerator />;
}

```

### app/groq/page.jsx

```javascript
// make a page that uses the groq.js server action
'use client';
import { groq, createGroq } from '@ai-sdk/groq';
import { generateText } from 'ai';
import { streamText } from '@/app/actions/groq';
import { useEffect } from 'react';

// const groq = createGroq();

export default function Page() {

  useEffect(() => {
    useEffectFunction();
  }, []);

  const useEffectFunction = async () => {
    const { text } = await streamText();
    // console.log(text);
  };

  return (
    <div>
      <h1>Groq Page</h1>
      {/* <p>{text}</p> */}
    </div>
  );
} 
```

### app/testing/page.jsx

```javascript
'use client'

import { useState, useRef } from 'react';

const WebMRecorder = () => {
  const [recording, setRecording] = useState(false);
  const [recordedBlob, setRecordedBlob] = useState(null);
  const mediaRecorder = useRef(null);

  const startRecording = async () => {
    try {
      const stream = await navigator.mediaDevices.getUserMedia({ audio: true, video: false });
      mediaRecorder.current = new MediaRecorder(stream);
      const chunks = [];

      mediaRecorder.current.ondataavailable = (event) => {
        if (event.data.size > 0) {
          chunks.push(event.data);
        }
      };

      mediaRecorder.current.onstop = () => {
        const blob = new Blob(chunks, { type: 'audio/webm' });
        setRecordedBlob(blob);
      };

      mediaRecorder.current.start();
      setRecording(true);

      // Stop recording after 1 second
      setTimeout(() => {
        mediaRecorder.current.stop();
        setRecording(false);
        stream.getTracks().forEach(track => track.stop());
      }, 1000);
    } catch (error) {
      console.error('Error accessing microphone:', error);
    }
  };

  const saveRecording = async () => {
    if (recordedBlob) {
      try {
        // Check if the File System Access API is supported
        if ('showSaveFilePicker' in window) {
          const fileHandle = await window.showSaveFilePicker({
            suggestedName: 'recording.webm',
            types: [{
              description: 'WebM Files',
              accept: {'audio/webm': ['.webm']},
            }],
          });
          
          const writable = await fileHandle.createWritable();
          await writable.write(recordedBlob);
          await writable.close();
          
          console.log('File saved successfully');
        } else {
          throw new Error('File System Access API not supported');
        }
      } catch (error) {
        console.error('Error saving file:', error);
        // Fallback to download method if File System Access API is not supported
        const url = URL.createObjectURL(recordedBlob);
        const a = document.createElement('a');
        document.body.appendChild(a);
        a.style = 'display: none';
        a.href = url;
        a.download = 'recording.webm';
        a.click();
        URL.revokeObjectURL(url);
      }
    }
  };

  return (
    <div className="p-4">
      <button
        onClick={startRecording}
        disabled={recording}
        className="bg-blue-500 hover:bg-blue-700 text-white font-bold py-2 px-4 rounded mr-2"
      >
        {recording ? 'Recording...' : 'Start Recording'}
      </button>
      <button
        onClick={saveRecording}
        disabled={!recordedBlob}
        className="bg-green-500 hover:bg-green-700 text-white font-bold py-2 px-4 rounded"
      >
        Save Recording
      </button>
    </div>
  );
};

export default WebMRecorder;
```

### app/api/groq/route.js

```javascript
'use server';

import { groq } from '@ai-sdk/groq';
import { createGroq } from '@ai-sdk/groq';
import { generateText } from 'ai';
import { z } from 'zod';

// Create a new instance of groq
const groqInstance = createGroq();

// Define a Zod schema for the response structure
const responseSchema = z.object({
  response: z.string(),
});

// Define a Zod schema for the request body
const requestBodySchema = z.object({
  prompt: z.string(),
});

export async function POST(request) {
  try {
    // Parse and validate the request body
    const body = await request.json();
    const { prompt } = requestBodySchema.parse(body);

    // Generate the text using the provided model and user prompt
    const { text } = await generateText({
      model: groqInstance('gemma2-9b-it'),
      prompt,
    });

    // Validate the generated text against the schema
    const validatedData = responseSchema.parse({ response: text });

    // Respond with the validated data in JSON format
    return new Response(JSON.stringify(validatedData), {
      headers: { 'Content-Type': 'application/json' },
    });
  } catch (error) {
    console.error('Error generating response:', error);
    return new Response(JSON.stringify({ error: 'Failed to generate response' }), {
      status: 500,
      headers: { 'Content-Type': 'application/json' },
    });
  }
}
```

### app/api/search/route.js

```javascript
'use server';

import { createGroq } from '@ai-sdk/groq';
import { generateText } from 'ai';
import { z } from 'zod';
import { NextResponse } from 'next/server';

// Create a new instance of groq
const groqInstance = createGroq();

const searchRequestBodySchema = z.object({
  query: z.string(),
  
});

const searchResponseSchema = z.object({
  response: z.string(),
});

export async function POST(request) {
  try {
    // Parse and validate request body
    const body = await request.json();
    let { query } = searchRequestBodySchema.parse(body);

    let context = await fetch("http://localhost:8000/search", {
      method: "POST",
      headers: {
        "Content-Type": "application/json",
      },
      body: JSON.stringify({ query }),
    });
    context = await context.json();
    context = context.documents[0]
    console.log(context)
    let response = await summarizeRAG(context, query);
    console.log(response)

    // Validate and respond with new summary and context
    const validatedData = searchResponseSchema.parse({ response: response });
    // console.log({validatedData})
    return NextResponse.json(validatedData, { status: 200 });
  } catch (error) {
    console.error('Error generating response:', error);
    return NextResponse.json({ error: 'Failed to generate response' }, { status: 500 });
  }
}

async function summarizeRAG(context, query) {
  const prompt = `You are an assistant for answering questions given a large set of possible context. Use the following retrieved context to help form an answer to the user's query, using ONLY information from the provided context to generate an answer. If you do not know the answer, just say that you don't know.
  
  Here is the provided context: "${context}".
  
  The user's query is: "${query}".
  
  Remember, only answer the user's query using information from the provided context and nothing else, do not mention the context. Only return a comprehensible response.`;
  const { text } = await generateText({
    model: groqInstance('llama-3.1-70b-versatile'),
    prompt,
  });
  console.log(text);
  return text;
}

```

[24 more indexed source files omitted to keep this export small. The full file list is in the Codebase structure section above.]