# Project export: Notefly - AI Meeting Notes on the Fly This document was generated by HackStack to give an AI agent context about a hackathon project. Sections are labeled with their provenance; content marked as truncated was cut to keep this document small. ## Project metadata - Hackathon: Cal Hacks 11.0 - Tagline: Notefly is your real time online meeting note taker. See meeting transcript and AI summary while on the call. Always stay on point with short notes from your personal typist for meetings. - Devpost: https://devpost.com/software/notefly-ai-meeting-notes-on-the-fly - GitHub: https://github.com/Almasx/calhack - Video: https://www.youtube.com/embed/8wdY5N1KWZE?enablejsapi=1&hl=en_US&rel=0&start=&version=3&wmode=transparent - Result: winner (Agora: Best Innovation of TEN) - Team: 3 GitHub contributor(s) — Almas Sapar (7 commits), Arlan Marat (5 commits), shovelfishy (2 commits) ## Devpost submission (written by the team) ### Inspiration We are students at Minerva University which uses innovative curriculum and learning methods. Online forum classes is how we learn, they require active participation. While on the classes, we might doze off or forget about something discussed, which diminishes our grades. We always wanted a live, automatic AI note taker so we won't need to take notes all the time. ### What it does Our product, Notefly, turns the talks in the meetings into live transcript text. Then it turns the transcripts into short note blocks, a 40 words speech becomes 10 words. The short note blocks turn into bullet points and bullet points turn into main idea summary notes. All these summaries can be seen while in the meeting so everyone knows what we are discussing and important details, like a customers mail address. ### How we built it Started by using agora's SDK to create a basic web conferencing software using next.js for the frontend. Then used agoras transcription features and figured out how we retrieve, store and process that data using fetch.ai agents. Used gemini for processing and mongo.db for storing data ### Challenges we ran into Our tech stack didn't exactly matched with the project. Storing transcripts was a BIG bugger, we thought agora had place to store data but they didn't. we tried to send to singlestore but that didn't worked properly as well. Making fetch.ai agents work and make them retrieve and send data, and communicate with each other was also a problem. Making the app live update was also a challenge. ### Accomplishments we're proud of IT FUC'IN WORRKKS!!! ### What we learned How to create a video calling app, something that we have 0 knowledge about, using tools that we have 0 knowledge about. Also deciding on the idea took much longer than it should, we will focus on not the perfect but the most applicable and team friendly idea next time. ### What's next After an all-nighter, the app is far from done, there could be more improvements in the summary logic. We can turn this into a more scalable product that allows people to host their own meetings in the cloud, its on a local host that lives in agoras serves now, we sadly can't send try us links rn. ## README (from the GitHub repository) This is a [Next.js](https://nextjs.org) project bootstrapped with [`create-next-app`](https://nextjs.org/docs/app/api-reference/cli/create-next-app). ## Getting Started First, run the development server: ```bash npm run dev # or yarn dev # or pnpm dev # or bun dev ``` Open [http://localhost:3000](http://localhost:3000) with your browser to see the result. You can start editing the page by modifying `app/page.tsx`. The page auto-updates as you edit the file. This project uses [`next/font`](https://nextjs.org/docs/app/building-your-application/optimizing/fonts) to automatically optimize and load [Geist](https://vercel.com/font), a new font family for Vercel. ## Learn More To learn more about Next.js, take a look at the following resources: - [Next.js Documentation](https://nextjs.org/docs) - learn about Next.js features and API. - [Learn Next.js](https://nextjs.org/learn) - an interactive Next.js tutorial. You can check out [the Next.js GitHub repository](https://github.com/vercel/next.js) - your feedback and contributions are welcome! ## Deploy on Vercel The easiest way to deploy your Next.js app is to use the [Vercel Platform](https://vercel.com/new?utm_medium=default-template&filter=next.js&utm_source=create-next-app&utm_campaign=create-next-app-readme) from the creators of Next.js. Check out our [Next.js deployment documentation](https://nextjs.org/docs/app/building-your-application/deploying) for more details. ## Detected evidence (automated analysis) Indexed codebase: 37 recognized source files, 104 KB. - CSS (language) — detected in the code - JavaScript (language) — detected in the code - MongoDB (technology) — detected in the code - Next.js (technology) — detected in the code - React (technology) — detected in the code - Tailwind CSS (technology) — detected in the code - TypeScript (language) — detected in the code - Google Gemini (technology) — claimed on Devpost, not found in the code ## Codebase structure (from repository index) ### Files (45 of 45) ``` .eslintrc.json .gitignore bun.lockb next.config.mjs package.json postcss.config.mjs README.md src/app/api/subtitles/route.ts src/app/api/summary/[id]/route.ts src/app/api/summary/route.ts src/app/api/transcribe/route.ts src/app/channel/[channelName]/page.tsx src/app/globals.css src/app/hook/stream.ts src/app/hook/utils.ts src/app/layout.tsx src/app/page.tsx src/components/ai-panel/hook.ts src/components/ai-panel/index.tsx src/components/ai-panel/notes.tsx src/components/ai-panel/summary.tsx src/components/call-panel/default-video.tsx src/components/call-panel/index.tsx src/components/call-panel/video-grid.tsx src/components/control-panel/action-button.tsx src/components/control-panel/index.tsx src/components/transcript-panel.tsx src/layouts/sidebar.tsx src/lib/agora/const.ts src/lib/agora/utils.ts src/lib/db/queries.ts src/lib/events.ts src/lib/mongodb.ts src/lib/protobuf/genJsCode.sh src/lib/protobuf/SttMessage_es6.js src/lib/protobuf/SttMessage.js src/lib/protobuf/SttMessage.proto src/lib/protobuf/SttMessage001.js src/lib/protobuf/SttMessage002.js src/lib/protobuf/SttMessage003.js src/store/subtitle-store.ts src/store/transcription-store.ts src/store/ui-store.ts tailwind.config.ts tsconfig.json ``` ### Dependencies - package.json: @types/node@^20, @types/react@^18, @types/react-dom@^18, agora-rtc-react@^2.3.0, class-variance-authority@^0.7.0, clsx@^2.1.1, eslint@^8, eslint-config-next@14.2.15, framer-motion@^11.11.9, lucide-react@^0.453.0, mongodb@^6.9.0, next@14.2.15, postcss@^8, protobufjs@^7.4.0, react@^18, react-dom@^18, tailwindcss@^3.4.1, typescript@^5, zustand@^5.0.0 ### Recent commits (newest first) - feat: full call - feat: full call - feat: nice transcribe - display - design - arla11n - arla11n - arlan - arlan - rrere - feat: nice transcribe - almas gay - feat: 1 - init - Initial commit from Create Next App ## Key source files (fetched from GitHub, selected and truncated for size) ### package.json ``` { "name": "calhacks", "version": "0.1.0", "private": true, "scripts": { "dev": "next dev", "build": "next build", "start": "next start", "lint": "next lint" }, "dependencies": { "agora-rtc-react": "^2.3.0", "class-variance-authority": "^0.7.0", "clsx": "^2.1.1", "framer-motion": "^11.11.9", "lucide-react": "^0.453.0", "mongodb": "^6.9.0", "next": "14.2.15", "protobufjs": "^7.4.0", "react": "^18", "react-dom": "^18", "zustand": "^5.0.0" }, "devDependencies": { "@types/node": "^20", "@types/react": "^18", "@types/react-dom": "^18", "eslint": "^8", "eslint-config-next": "14.2.15", "postcss": "^8", "tailwindcss": "^3.4.1", "typescript": "^5" } } ``` ### src/app/layout.tsx ```typescript import type { Metadata } from "next"; import localFont from "next/font/local"; import "./globals.css"; const geistMono = localFont({ src: "./fonts/GeistMonoVF.woff", variable: "--font-geist-mono", weight: "100 900", }); export const metadata: Metadata = { title: "Create Next App", description: "Generated by create next app", }; export default function RootLayout({ children, }: Readonly<{ children: React.ReactNode; }>) { return ( {children} ); } ``` ### src/app/page.tsx ```typescript "use client"; import { ChevronRight } from "lucide-react"; import { useRouter } from "next/navigation"; import { Summary } from "~/components/ai-panel/summary"; import { Headline } from "~/lib/db/queries"; const PRODUCT_DEMO_HEADLINES: Headline[] = [ { _id: "1", text: "The next big thing in Real Time AI is here", timestamp: new Date(), }, { _id: "2", text: "AI is changing the game", timestamp: new Date(), }, { _id: "3", text: "AI is the future", timestamp: new Date(), }, ]; export default function Home() { const router = useRouter(); return (
{ e.preventDefault(); const target = e.target as typeof e.target & { channel: { value: string }; }; router.push(`/channel/${target.channel.value}`); }} className="flex flex-col mt-6 w-full" >
); } ``` ### src/components/ai-panel/index.tsx ```typescript import { Notes } from "./notes"; import { Summary } from "./summary"; import { Headline, Summary as ISummary } from "~/lib/db/queries"; export const AIPanel = ({ summaries, headlines, }: { summaries: ISummary[]; headlines: Headline[]; }) => { return ( <> ); }; ``` ### src/components/call-panel/index.tsx ```typescript "use client"; import { useEffect } from "react"; import AgoraRTC, { AgoraRTCProvider, useRTCClient } from "agora-rtc-react"; import { ControlPanel } from "../control-panel"; import useStreamSubscription from "~/app/hook/stream"; import { useTranscriptionStore } from "~/store/transcription-store"; import { VideoGrid } from "./video-grid"; export const Call = ({ appId, channelName, }: { appId: string; channelName: string; }) => { const client = useRTCClient( AgoraRTC.createClient({ codec: "vp8", mode: "rtc" }) ); useStreamSubscription(client); const { startTranscription, stopTranscription, isTranscribing } = useTranscriptionStore(); useEffect(() => { startTranscription(channelName); return () => { if (isTranscribing) { stopTranscription(); } }; }, [channelName, startTranscription, stopTranscription, isTranscribing]); return (
); }; ``` ### src/components/control-panel/index.tsx ```typescript "use client"; import React, { useCallback, useMemo } from "react"; import { MicIcon, MicOffIcon, VideoIcon, VideoOffIcon, Phone, } from "lucide-react"; import { useUIStore } from "../../store/ui-store"; import { ActionButton } from "./action-button"; import { useRouter } from "next/navigation"; interface ControlAction { label: string; icon: React.ReactNode; onClick: () => void; variant?: "default" | "active" | "danger"; } const AIIcon = () => ( ); const TranscriptIcon = () => ( ); export const ControlPanel: React.FC = () => { const { isAIOpen, isTranscriptOpen, toggleAI, toggleTranscript, isMuted, isVideoOff, toggleMute, toggleVideo, } = useUIStore(); const router = useRouter(); const handleLeaveCall = useCallback(() => { router.push("/"); }, [router]); const audioAction = useMemo( (): ControlAction => ({ label: isMuted ? "Unmute" : "Mute", icon: isMuted ? ( ) : ( ), onClick: toggleMute, variant: isMuted ? "active" : "default", }), [isMuted, toggleMute] ); const videoAction = useMemo( (): ControlAction => ({ label: isVideoOff ? "Turn on camera" : "Turn off camera", icon: isVideoOff ? ( ) : ( ), onClick: toggleVideo, variant: isVideoOff ? "active" : "default", }), [isVideoOff, toggleVideo] ); const leaveCallAction = useMemo( (): ControlAction => ({ label: "Leave call", icon: , onClick: handleLeaveCall, variant: "danger", }), [handleLeaveCall] ); const aiAction = useMemo( (): ControlAction => ({ label: "AI", icon: , onClick: toggleAI, variant: isAIOpen ? "active" : "default", }), [isAIOpen, toggleAI] ); const transcriptAction = useMemo( (): ControlAction => ({ label: "Transcript", icon: , onClick: toggleTranscript, variant: isTranscriptOpen ? "active" : "default", }), [isTranscriptOpen, toggleTranscript] ); const controlActions = useMemo( () => [ audioAction, videoAction, leaveCallAction, aiAction, transcriptAction, ], [audioAction, videoAction, leaveCallAction, aiAction, transcriptAction] ); return (
{controlActions.map((action) => ( ))}
); }; ``` ### src/app/api/summary/route.ts ```typescript import { NextResponse } from "next/server"; import { getSummaries } from "~/lib/db/queries"; export async function GET() { try { const summaries = await getSummaries(); return NextResponse.json(summaries); } catch (error) { console.error("Error fetching summaries:", error); return NextResponse.json( { error: "Failed to fetch summaries" }, { status: 500 } ); } } ``` ### src/app/channel/[channelName]/page.tsx ```typescript import { Call } from "~/components/call-panel"; import { Sidebar } from "~/layouts/sidebar"; import { AGORA_APP_ID } from "~/lib/agora/const"; import { getSubtitles, getSummaries, getHeadlines, clearHeadlines, } from "~/lib/db/queries"; interface PageProps { params: { channelName: string }; } export default async function ChannelPage({ params }: PageProps) { const { channelName } = params; try { const [subtitles, summaries, headlines] = await Promise.all([ getSubtitles(), getSummaries(), getHeadlines(), ]); return (
s)} headlines={headlines} />
); } catch (error) { console.error("Error fetching data:", error); return
Error loading channel data. Please try again later.
; } } ``` ### src/app/api/subtitles/route.ts ```typescript // app/api/subtitles/route.ts import { NextRequest, NextResponse } from "next/server"; import clientPromise from "~/lib/mongodb"; export async function POST(request: NextRequest) { try { const client = await clientPromise; const db = client.db(process.env.MONGODB_DB || "myproject"); const subtitle = await request.json(); const result = await db.collection("subtitles").insertOne(subtitle); return NextResponse.json( { message: "Subtitle saved", data: result }, { status: 201 } ); } catch (error: any) { console.error("Error saving subtitle:", error); return NextResponse.json({ error: error.message }, { status: 500 }); } } export async function GET() { try { const client = await clientPromise; const db = client.db(process.env.MONGODB_DB || "myproject"); const subtitles = await db.collection("subtitles").find().toArray(); return NextResponse.json(subtitles, { status: 200 }); } catch (error: any) { console.error("Error fetching subtitles:", error); return NextResponse.json({ error: error.message }, { status: 500 }); } } ``` ### src/app/api/transcribe/route.ts ```typescript // ~/app/api/transcription/route.ts import { NextRequest, NextResponse } from "next/server"; import { apiSTTStartTranscription, apiSTTStopTranscription, apiSTTAcquireToken, } from "~/lib/agora/utils"; // POST: Start Transcription export async function POST(request: NextRequest) { try { const { channel } = await request.json(); const { taskId, builderToken } = await handleStartTranscription(channel); return NextResponse.json({ taskId, builderToken }); } catch (error) { return NextResponse.json( { error: (error as Error).message }, { status: 500 } ); } } // DELETE: Stop Transcription export async function DELETE(request: NextRequest) { try { const { taskId, builderToken, channel, uid } = await request.json(); await apiSTTStopTranscription({ taskId, token: builderToken, uid, channel, }); return NextResponse.json({ message: "Transcription stopped successfully" }); } catch (error) { return NextResponse.json( { error: (error as Error).message }, { status: 500 } ); } } // Helper: Start Transcription Logic async function handleStartTranscription(channel: string) { const uid = "almas"; // Example UID, ideally this should come from the client const tokenData = await apiSTTAcquireToken({ channel, uid }); const { taskId } = await apiSTTStartTranscription({ uid, channel, token: tokenData.tokenName, }); return { taskId, builderToken: tokenData.tokenName }; } ``` [26 more indexed source files omitted to keep this export small. The full file list is in the Codebase structure section above.]