# Project export: Athena This document was generated by HackStack to give an AI agent context about a hackathon project. Sections are labeled with their provenance; content marked as truncated was cut to keep this document small. ## Project metadata - Hackathon: TreeHacks 2024 - Tagline: Athena serves as your AI-powered Content Management Assistant. - Devpost: https://devpost.com/software/athena-2so98j - GitHub: https://github.com/Nyumat/TreeHacks10 - Team: 3 GitHub contributor(s) — Sankalp Patil (16 commits), Thomas (15 commits), TanzHasan (13 commits) ## Devpost submission (written by the team) ### Inspiration The inspiration for our project, Athena, comes from our experience as students with busy lives. Often, it isn't easy to keep track of the vast amounts of media we encounter (Lectures, webinars, TedTalks, etc). With Athena, people can have one AI-powered store for all their content, allowing them to save time slogging through hours of material in search of information. Our aim is to enhance productivity and empower our users to explore, engage, and learn in a way that truly values their time. What it Does In Athena, we empower our users to manage and query all forms of content. You have the flexibility to choose how you organize and interact with your material. Whether you prefer grouping content by course and using focused queries or rewatching lectures with a custom-trained chatbot at your fingertips, our application Athena has got you covered. We allow users to either perform multimodal vectorized searches across all their documents, enhancing information accessibility, or explore a single document with more depth and nuance using a custom-trained LLm model. With Athena, the power of information is in your hands, and the choice is yours to make. ### How we built it We built our application using a multitude of services/frameworks/tools: React.js for the core client frontend TypeScript for robust typing and abstraction support Tailwind for a utility-first CSS framework ShadCN for animations and UI components Clerk for a seamless and drop-in OAuth provider React-icons for drop-in pixel-perfect icons NextJS for server-side rendering and enhanced SEO Convex for vector search over our database App-router for client-side navigation Convex for real-time server and end-to-end type safety Challenges We Ran Into Navigating new services and needing to read a lot of documentation -- since this was the first time any of us had used vector search with Convex, it took a lot of research and heads-down coding to get Athena working. Being awake to work as a team -- since this hackathon is both in-person and through the weekend, we had many sleepless nights to ensure we can successfully produce Athena. ### Accomplishments we're proud of Finishing our project and getting it working! We were honestly surprised at our progress this weekend and are super proud of our end product Athena. Learning a ton of new technologies we would have never come across without Tree Hacks. Being able to code for at times 12-16 hours straight and still be having fun! ### What we learned Tools are tools for a reason! Embrace them, learn from them, and utilize them to make your applications better. Sometimes, more sleep is better -- as humans, sleep can sometimes be the basis for our mental ability! How to work together on a team project with many commits and iterate fast on our moving parts. ### What's next Create more options for users to group their content in different ways. Establish the ability for users to share content with others, increasing knowledge bases. Allow for more types of content upload apart from Videos and PDFs ## README (from the GitHub repository) # TreeHacks 2024 🌲 Our project submission for TreeHacks 2024, the 10th anniversary of TreeHacks, the annual hackathon at Stanford University. ## Tech Stack - Convex - Bun - Together.ai - Next.js - React - Langchain - shadcn/ui - Clerk - TypeScript - OpenAI GPT-3.5 Turbo - Tailwind CSS - FastAPI - Python - Vercel - YouTube Transcript Library - Zod ## Inspiration Have you ever had to watch a video for a class, but you didn't have time to watch the whole thing? Or maybe you just wanted to skip to the important parts? We've all been there. That's why we created Athena and Minerva, two AI-powered tools that help you get the most out of your educational videos. ## What it does Our project consists of two components. - Athena, a full-stack Next.js application powered by Convex, Bun, and shadcn. - Minerva, a FastAPI backend powered by Langchain, Together.ai, and OpenAI GPT-3.5 Turbo. Athena allows you to ask questions to your documents. You can ask questions like "What is the main idea of this video?" or "What are the key points?" and Athena will give you a summary of the video. Athena uses natural language processing to understand your questions and provide you with the most relevant information. Minerva allows you to search for specific topics within your documents. You can search for keywords like "calculus" or "quantum mechanics" and Minerva will show you the most relevant parts of the video. ## Detected evidence (automated analysis) Indexed codebase: 59 recognized source files, 133 KB. - CSS (language) — detected in the code - FastAPI (technology) — detected in the code - JavaScript (language) — detected in the code - LangChain (technology) — detected in the code - Next.js (technology) — detected in the code - OpenAI (technology) — detected in the code - Python (language) — detected in the code - React (technology) — detected in the code - Tailwind CSS (technology) — detected in the code - TypeScript (language) — detected in the code ## Codebase structure (from repository index) ### Files (73 of 73) ``` athena/.eslintrc.json athena/.gitignore athena/bun.lockb athena/components.json athena/convex/_generated/api.d.ts athena/convex/_generated/api.js athena/convex/_generated/dataModel.d.ts athena/convex/_generated/server.d.ts athena/convex/_generated/server.js athena/convex/auth.config.js athena/convex/clerk.ts athena/convex/http.ts athena/convex/langchain/db.ts athena/convex/messaging.ts athena/convex/README.md athena/convex/schema.ts athena/convex/test.json athena/convex/tsconfig.json athena/convex/users.ts athena/convex/vectors.ts athena/next.config.mjs athena/package.json athena/postcss.config.js athena/README.md athena/sampleData.jsonl athena/src/(components)/chat-component.tsx athena/src/(components)/ChatBot.tsx athena/src/(components)/ChatComponent.tsx athena/src/(components)/ConvexClientProvider.tsx athena/src/(components)/FileInput.tsx athena/src/(components)/navbar.tsx athena/src/(components)/QuizPage/index.tsx athena/src/(components)/QuizPage/questions.json athena/src/(components)/Selector.tsx athena/src/(components)/sidebar.tsx athena/src/(components)/theme-provider.tsx athena/src/(components)/theme-toggle.tsx athena/src/(components)/ui/badge.tsx athena/src/(components)/ui/button.tsx athena/src/(components)/ui/card.tsx athena/src/(components)/ui/checkbox.tsx athena/src/(components)/ui/command.tsx athena/src/(components)/ui/dialog.tsx athena/src/(components)/ui/dropdown-menu.tsx athena/src/(components)/ui/form.tsx athena/src/(components)/ui/input.tsx athena/src/(components)/ui/label.tsx athena/src/(components)/ui/multi-select.tsx athena/src/(components)/ui/resizable.tsx athena/src/(components)/ui/tabs.tsx athena/src/(components)/ui/toast.tsx athena/src/(components)/ui/toaster.tsx athena/src/(components)/ui/toggle-group.tsx athena/src/(components)/ui/toggle.tsx athena/src/(components)/ui/use-toast.ts athena/src/(components)/UploadForm.tsx athena/src/app/chat/page.tsx athena/src/app/home/page.tsx athena/src/app/layout.tsx athena/src/app/page.tsx athena/src/app/quiz/page.tsx athena/src/app/upload/[...slug]/page.tsx athena/src/lib/utils.ts athena/src/middleware.ts athena/src/styles/globals.css athena/tailwind.config.ts athena/tsconfig.json minerva/.gitignore minerva/init.py minerva/main.py minerva/requirements.txt README.md schema.txt ``` ### Dependencies - athena/package.json: @clerk/nextjs@^4.29.7, @emotion/react@^11.11.3, @emotion/styled@^11.11.0, @hookform/resolvers@^3.3.4, @langchain/community@^0.0.30, @langchain/openai@^0.0.14, @mui/material@^5.15.10, @radix-ui/react-checkbox@^1.0.4, @radix-ui/react-dialog@^1.0.5, @radix-ui/react-dropdown-menu@^2.0.6, @radix-ui/react-icons@^1.3.0, @radix-ui/react-label@^2.0.2, @radix-ui/react-slot@^1.0.2, @radix-ui/react-tabs@^1.0.4, @radix-ui/react-toast@^1.1.5, @radix-ui/react-toggle@^1.0.3, @radix-ui/react-toggle-group@^1.0.4, @types/node@^20, @types/react@^18, @types/react-dom@^18, @xixixao/uploadstuff@^0.0.5, autoprefixer@^10.0.1, axios@^1.6.7, cheerio@^1.0.0-rc.12, class-variance-authority@^0.7.0, clsx@^2.1.0, cmdk@^0.2.1, convex@^1.9.0, eslint@^8, eslint-config-next@14.1.0, framer-motion@^11.0.5, langchain@^0.1.20, lucide-react@^0.331.0, next@14.1.0, next-themes@^0.2.1, node-fetch@^3.3.2, postcss@^8, react@^18, react-dom@^18, react-dropzone@^14.2.3, react-hook-form@^7.50.1, react-player@^2.14.1, react-resizable-panels@^2.0.9, react-router-dom@^6.22.1, react-spinners@^0.13.8, svix@^1.18.0, tailwind-merge@^2.2.1, tailwindcss@^3.3.0, tailwindcss-animate@^1.0.7, typescript@^5, validata@^5.0.6, youtube-transcript@^1.0.6, zod@^3.22.4 - minerva/requirements.txt: fastapi, ffmpeg, lanchain, moviepy, openai, openai-whisper, pdf2image, pdfminer, pytesseract, python-multipart, unstructured, uvicorn ### Recent commits (newest first) - 🌲 Final Commit 🌲 - Landing Page + full stack chat - add chat - Merge pull request #2 from Nyumat/sandy - Merge branch 'master' into sandy - chat - Swapped to together - change - chat - together api plugin - YouTube Transcript - Upload Youtube Transcription Vectors to Convex - add embedding route - upload url - Merge branch 'master' of https://github.com/Nyumat/TreeHacks10 - add quiz - Content Selector, Upload Slug - multi selector input - shadcn: ui/ux components - progress ## Key source files (fetched from GitHub, selected and truncated for size) ### minerva/requirements.txt ``` openai openai-whisper pdfminer pdf2image unstructured lanchain ffmpeg moviepy pytesseract fastapi uvicorn python-multipart ``` ### athena/package.json ``` { "name": "athena", "version": "0.1.0", "private": true, "scripts": { "dev": "next dev", "build": "next build", "start": "next start", "lint": "next lint" }, "dependencies": { "@clerk/nextjs": "^4.29.7", "@emotion/react": "^11.11.3", "@emotion/styled": "^11.11.0", "@hookform/resolvers": "^3.3.4", "@langchain/community": "^0.0.30", "@langchain/openai": "^0.0.14", "@mui/material": "^5.15.10", "@radix-ui/react-checkbox": "^1.0.4", "@radix-ui/react-dialog": "^1.0.5", "@radix-ui/react-dropdown-menu": "^2.0.6", "@radix-ui/react-icons": "^1.3.0", "@radix-ui/react-label": "^2.0.2", "@radix-ui/react-slot": "^1.0.2", "@radix-ui/react-tabs": "^1.0.4", "@radix-ui/react-toast": "^1.1.5", "@radix-ui/react-toggle": "^1.0.3", "@radix-ui/react-toggle-group": "^1.0.4", "@xixixao/uploadstuff": "^0.0.5", "axios": "^1.6.7", "cheerio": "^1.0.0-rc.12", "class-variance-authority": "^0.7.0", "clsx": "^2.1.0", "cmdk": "^0.2.1", "convex": "^1.9.0", "framer-motion": "^11.0.5", "langchain": "^0.1.20", "lucide-react": "^0.331.0", "next": "14.1.0", "next-themes": "^0.2.1", "node-fetch": "^3.3.2", "react": "^18", "react-dom": "^18", "react-dropzone": "^14.2.3", "react-hook-form": "^7.50.1", "react-player": "^2.14.1", "react-resizable-panels": "^2.0.9", "react-router-dom": "^6.22.1", "react-spinners": "^0.13.8", "svix": "^1.18.0", "tailwind-merge": "^2.2.1", "tailwindcss-animate": "^1.0.7", "validata": "^5.0.6", "youtube-transcript": "^1.0.6", "zod": "^3.22.4" }, "devDependencies": { "@types/node": "^20", "@types/react": "^18", "@types/react-dom": "^18", "autoprefixer": "^10.0.1", "eslint": "^8", "eslint-config-next": "14.1.0", "postcss": "^8", "tailwindcss": "^3.3.0", "typescript": "^5" } } ``` ### minerva/main.py ```python from dotenv import load_dotenv from fastapi import FastAPI, File, HTTPException from langchain_community.document_loaders import PyPDFLoader from fastapi.responses import JSONResponse import pytesseract from moviepy.editor import VideoFileClip from PIL import Image import openai import time import io import os import whisper import httpx import fitz from llama_index.embeddings.together import TogetherEmbedding import together import json from pydantic import BaseModel load_dotenv() client = openai.OpenAI(api_key=os.environ["OPENAI_API_KEY"]) embed_model = TogetherEmbedding( model_name="WhereIsAI/UAE-Large-V1", api_key=os.environ["TOGETHER_API_KEY"], ) together.api_key = os.environ["TOGETHER_API_KEY"] app = FastAPI() @app.get("/") def read_root(): return {"message": "Hello, FastAPI!"} @app.get("/items/{item_id}") def read_item(item_id: int, query_param: str = None): return {"item_id": item_id, "query_param": query_param} class ImageURL(BaseModel): url: str @app.post("/upload-image/") async def create_upload_file(image_url: ImageURL): async with httpx.AsyncClient() as client: try: response = await client.get(image_url.url) response.raise_for_status() except httpx.RequestError as exc: raise HTTPException(status_code=400, detail=f"Error fetching image from {image_url.url}: {str(exc)}") except httpx.HTTPStatusError as exc: raise HTTPException(status_code=exc.response.status_code, detail=f"Error fetching image from {image_url.url}: HTTP Status Code {exc.response.status_code}") image = Image.open(io.BytesIO(response.content)) text = pytesseract.image_to_string(image) response = embed_model.get_text_embedding(text) return JSONResponse(content={"text": text, "embedding": response}) class VideoURL(BaseModel): url: str @app.post("/upload-video/") async def create_upload_video(video_url: VideoURL): async with httpx.AsyncClient() as client: try: response = await client.get(video_url.url) response.raise_for_status() except httpx.RequestError as exc: raise HTTPException(status_code=400, detail=f"Error fetching video from {video_url.url}: {str(exc)}") except httpx.HTTPStatusError as exc: raise HTTPException(status_code=exc.response.status_code, detail=f"Error fetching video from {video_url.url}: HTTP Status Code {exc.response.status_code}") video_path = f"temp_video.mp4" with open(video_path, "wb") as f: f.write(response.content) try: clip = VideoFileClip(video_path) audio_path = video_path + ".mp3" clip.audio.write_audiofile(audio_path) model = whisper.load_model("base") result = model.transcribe(audio_path) finally: clip.close() os.remove(video_path) os.remove(audio_path) return JSONResponse(content=generate_embeddings_for_segments(result)) def generate_embeddings_for_segments(data): embeddings = [] for segment in data["segments"]: attempt = 0 while True: try: segment["embedding"] = embed_model.get_text_embedding(segment["text"]) embeddings.append(segment) break except Exception as e: attempt += 1 # wait_time = min(2**attempt + random.random(), 60) wait_time = 1 print( f"Rate limit hit, waiting {wait_time:.2f} seconds before retrying..." ) if attempt == 10: break time.sleep(wait_time) keys_to_remove = ['temperature', 'avg_logprob', 'compression_ratio', 'no_speech_prob', 'tokens'] cleaned_data_array = [ {key: value for key, value in element.items() if key not in keys_to_remove} for element in embeddings ] return cleaned_data_array class PDFURL(BaseModel): url: str @app.post("/upload-pdf/") async def upload_pdf(pdf_url: PDFURL): async with httpx.AsyncClient() as client: try: response = await client.get(pdf_url.url) response.raise_for_status() except httpx.RequestError as exc: raise HTTPException(status_code=400, detail=f"Error fetching PDF from {pdf_url.url}: {str(exc)}") except httpx.HTTPStatusError as exc: raise HTTPException(status_code=exc.response.status_code, detail=f"Error fetching PDF from {pdf_url.url}: HTTP Status Code {exc.response.status_code}") pdf_path = "temp_document.pdf" with open(pdf_path, "wb") as f: f.write(response.content) text = "" response = [] with fitz.open(pdf_path) as doc: text = [page.get_text() for page in doc] for tex in text: attempt = 0 while True: try: segment = embed_model.get_text_embedding(tex[:2000]) response.append(segment) break except Exception as e: attempt += 1 # wait_time = min(2**attempt + random.random(), 60) wait_time = 1 print( f"Rate limit hit, waiting {wait_time:.2f} seconds before retrying..." ) if attempt == 10: break time.sleep(wait_time) os.remove(pdf_path) return JSONResponse(content={"text": ' '.join(text), "embedding": response}) class QuizRequest(BaseModel): text: str @app.post("/generate-quiz/") def generate_quiz(request: QuizRequest): prompt = f"{request.text}\n\n create a json schema about the above with the following: {{\nquestion: ['Right Answer', 'Wrong', 'Wrong', 'Wrong']\n}}" [truncated — 359 more characters] ``` ### athena/src/app/layout.tsx ```typescript import ConvexClientProvider from "@/(components)/ConvexClientProvider"; import { Navbar } from "@/(components)/navbar"; import { ThemeProvider } from "@/(components)/theme-provider"; import { Toaster } from "@/(components)/ui/toaster"; import { cn } from "@/lib/utils"; import "@/styles/globals.css"; import type { Metadata } from "next"; import { Inter as FontSans } from "next/font/google"; export const fontSans = FontSans({ subsets: ["latin"], variable: "--font-sans", }); export const metadata: Metadata = { title: "Athena", description: "The ai-powered cms of the future", }; export default function RootLayout({ children, }: { children: React.ReactNode; }) { return ( {children} ); } ``` ### athena/src/app/page.tsx ```typescript "use client"; import { Button } from "@/(components)/ui/button"; import { Input } from "@/(components)/ui/input"; import { useRouter } from "next/navigation"; import { useState } from "react"; // Thanks, Next.js, for making me do this lol. export const dynamic = "force-dynamic"; export default function Home() { // Maybe we should turn this into the root of our dashboard? I'm too sleepy to think about it. const [youtubeVideo, setYoutubeVideo] = useState(""); const [errors, setErrors] = useState([]); const [loading, setLoading] = useState(false); const router = useRouter(); const handleClick = async (e: any) => { e.preventDefault(); if (youtubeVideo === "") { setErrors(["Please enter a valid YouTube URL"]); return; } if (youtubeVideo !== "") { setErrors([]); setLoading(true); router.push(`/home?video=${youtubeVideo}`); } }; return (

Next Generation {" "} Multi-Modal
Tooling Powered by{" "} AI

Athena is all you need to query your data, structured, unstructured, and multi-modal. We have it all.

Give us a YouTube Link, We'll Do The Rest.

setYoutubeVideo(e.target.value)} />
); } ``` ### athena/convex/_generated/server.js ```javascript /* eslint-disable */ /** * Generated utilities for implementing server-side Convex query and mutation functions. * * THIS CODE IS AUTOMATICALLY GENERATED. * * Generated by convex@1.9.0. * To regenerate, run `npx convex dev`. * @module */ import { actionGeneric, httpActionGeneric, queryGeneric, mutationGeneric, internalActionGeneric, internalMutationGeneric, internalQueryGeneric, } from "convex/server"; /** * Define a query in this Convex app's public API. * * This function will be allowed to read your Convex database and will be accessible from the client. * * @param func - The query function. It receives a {@link QueryCtx} as its first argument. * @returns The wrapped query. Include this as an `export` to name it and make it accessible. */ export const query = queryGeneric; /** * Define a query that is only accessible from other Convex functions (but not from the client). * * This function will be allowed to read from your Convex database. It will not be accessible from the client. * * @param func - The query function. It receives a {@link QueryCtx} as its first argument. * @returns The wrapped query. Include this as an `export` to name it and make it accessible. */ export const internalQuery = internalQueryGeneric; /** * Define a mutation in this Convex app's public API. * * This function will be allowed to modify your Convex database and will be accessible from the client. * * @param func - The mutation function. It receives a {@link MutationCtx} as its first argument. * @returns The wrapped mutation. Include this as an `export` to name it and make it accessible. */ export const mutation = mutationGeneric; /** * Define a mutation that is only accessible from other Convex functions (but not from the client). * * This function will be allowed to modify your Convex database. It will not be accessible from the client. * * @param func - The mutation function. It receives a {@link MutationCtx} as its first argument. * @returns The wrapped mutation. Include this as an `export` to name it and make it accessible. */ export const internalMutation = internalMutationGeneric; /** * Define an action in this Convex app's public API. * * An action is a function which can execute any JavaScript code, including non-deterministic * code and code with side-effects, like calling third-party services. * They can be run in Convex's JavaScript environment or in Node.js using the "use node" directive. * They can interact with the database indirectly by calling queries and mutations using the {@link ActionCtx}. * * @param func - The action. It receives an {@link ActionCtx} as its first argument. * @returns The wrapped action. Include this as an `export` to name it and make it accessible. */ export const action = actionGeneric; /** * Define an action that is only accessible from other Convex functions (but not from the client). * * @param func - The function. It receives an {@link ActionCtx} as its first argument. * @returns The wrapped function. Include this as an `export` to name it and make it accessible. */ export const internalAction = internalActionGeneric; /** * Define a Convex HTTP action. * * @param func - The function. It receives an {@link ActionCtx} as its first argument, and a `Request` object * as its second. * @returns The wrapped endpoint function. Route a URL path to this function in `convex/http.js`. */ export const httpAction = httpActionGeneric; ``` ### athena/src/app/chat/page.tsx ```typescript import ChatBot from '@/(components)/ChatBot' export default function Chat() { return ; } ``` ### athena/src/app/quiz/page.tsx ```typescript import QuizPage from '../../(components)/QuizPage/index'; // Assuming your index.js file contains the QuizPage component import questionsData from '../../(components)/QuizPage/questions.json'; export default function Quiz() { return ; } ``` ### athena/src/app/home/page.tsx ```typescript "use client"; import { ChatComponent } from "@/(components)/chat-component"; import { useUser } from "@clerk/nextjs"; import { useAction } from "convex/react"; import { useEffect } from "react"; import { api } from "../../../convex/_generated/api"; export default function Chat() { const searchParams = new URLSearchParams(window.location.search); const video = searchParams.get("video"); const fetchAndEmbedSingle = useAction(api.vectors.fetchAndEmbedSingle); const auth = useUser().user?.id; useEffect(() => { if (video) { fetchAndEmbedSingle({ url: video, userId: auth ?? "" }); } }, [video, auth]); return ( video && ( <> ) ); } ``` ### athena/src/(components)/QuizPage/index.tsx ```typescript "use client"; import React, { useState, useEffect } from 'react'; import Link from 'next/link'; import { CardHeader, CardContent, Card } from "@/(components)/ui/card" import { Button } from "@/(components)/ui/button" interface Question { question: string; choices: { choice: string; isCorrect: boolean }[]; } interface QuizPageProps { questionsData: { [key: string]: string[] }; } const QuizPage: React.FC = ({ questionsData }) => { const [currentQuestionIndex, setCurrentQuestionIndex] = useState(0); const [selectedAnswer, setSelectedAnswer] = useState(''); const [results, setResults] = useState([]); const [shuffledQuestions, setShuffledQuestions] = useState([]); const [quizCompleted, setQuizCompleted] = useState(false); useEffect(() => { // Shuffle the choices for each question while marking the correct answer const shuffled = Object.entries(questionsData).map(([question, choices]) => { const correctAnswer = choices[0]; const shuffledChoices = shuffleArray(choices); const choicesWithIsCorrect = shuffledChoices.map(choice => ({ choice, isCorrect: choice === correctAnswer })); return { question, choices: choicesWithIsCorrect }; }); setShuffledQuestions(shuffled); }, [questionsData]); const handleAnswerSelect = (answer: string) => { setSelectedAnswer(answer); }; const handleNextQuestion = () => { const currentQuestion = shuffledQuestions[currentQuestionIndex]; const isCorrect = currentQuestion.choices.find(choice => choice.choice === selectedAnswer)?.isCorrect || false; setResults([...results, isCorrect]); setSelectedAnswer(''); if (currentQuestionIndex + 1 < shuffledQuestions.length) { setCurrentQuestionIndex(currentQuestionIndex + 1); } else { setQuizCompleted(true); } }; if (shuffledQuestions.length === 0) { return (

No quiz could be made with that content

Back to Home
© 2024 Athena. All rights reserved.
); } if (quizCompleted) { return ; } function BookOpenIcon(props: any) { return ( ) } const currentQuestion = shuffledQuestions[currentQuestionIndex]; return (

Question {currentQuestionIndex + 1}

{currentQuestionIndex + 1} of {shuffledQuestions.length}

{currentQuestion.question}

    {currentQuestion.choices.map((choice, index) => (