# Project export: scholara.ai This document was generated by HackStack to give an AI agent context about a hackathon project. Sections are labeled with their provenance; content marked as truncated was cut to keep this document small. ## Project metadata - Hackathon: Cal Hacks 11.0 - Tagline: Unlock Research: AI-Powered Clarity for Complex Papers - Devpost: https://devpost.com/software/scholora-ai - GitHub: https://github.com/aryamankukal/CalHacks - Team: 2 GitHub contributor(s) — Alex Luo (6 commits), akh1799 (1 commits) ## Devpost submission (written by the team) ### Inspiration It all started when our friends asked us a simple question: "How does ChatGPT work?" While we tried to explain it, we realized how difficult it can be to convey something complex to someone with limited experience in a field. And as college students, we've all conducted research before, and see many of our peers going into research. But from our own experience, we've seen how papers use complex jargon and technical concepts that make it difficult to understand, especially when most students lack the required experience. Therefore, we propose scholora.ai—a tool that can explain these complex topics as if we were talking to an elementary schooler. Using visual diagrams generated by the Manim Library and a Voice Agent for real-time conversations, students can understand research papers in a fun, engaging way. ### What it does First up, we have the Prerequisite Learner. Ever opened a research paper and realized you need to understand five other concepts first? Our Prerequisite Learner has got your back. It provides curated YouTube links that cover the basics you need before diving into the methods of the paper. Think of it as your guided roadmap to make sure you're ready before the real journey begins. Next, there's the Knowledge Graph. Research papers don’t exist in isolation—they build upon each other in a web of ideas. Our Knowledge Graph shows you which papers influenced the current one, giving you more context and a deeper understanding of how the field evolved. It also suggests papers that reference the one you're reading, opening doors for further exploration. Understanding complex ideas is much easier when you can visualize them. That’s why we’ve included Manim animations that break down the key concepts. Instead of struggling through dense equations, watch as the theory comes alive in engaging, step-by-step visualizations. To keep you on track and ensure you're really grasping the content, we've added a Quiz feature. After each section of the paper, you'll be prompted with questions to test your understanding before moving on. It’s like having a friendly checkpoint that makes sure the journey isn’t just about moving forward, but also truly comprehending. And finally, our Voicebot is here to make learning conversational. Got a question about the paper? Just ask the Voicebot, and it'll give you an easy-to-understand summary. No more struggling with jargon or feeling stuck—the Voicebot turns those confusing sections into a quick, friendly explanation. ### How we built it For the frontend, we utilized Next.js and React to craft a responsive, dynamic user interface. We leaned on Tailwind for streamlined, modern styling, and used a blend of CSS, HTML, and JavaScript to piece together the UI layout in a way that is not only functional but visually engaging. The goal was to ensure users had a seamless, enjoyable experience navigating the platform. On the backend, things got even more exciting. We integrated Google Gemini's API to handle paper summarization and provide curated YouTube links for prerequisite learning. This helped make sure users could understand the key concepts before diving in too deep. We leveraged Chroma's API for a powerful semantic search. By analyzing papers in similar categories on arXiv, it identifies both the foundational papers that influenced the current one, as well as subsequent work that was influenced by it. This provided the backbone for our Knowledge Graph feature, allowing users to explore the interconnected world of academic research. To bring visualizations to life, we used the Hyperbolic API paired with Llama 3.1 to generate Manim code that breaks down key concepts in the paper. This approach allows us to transform theoretical ideas into animations that are easier to grasp and far more engaging. For the interactive voicebot, we used Vapi.ai’s API alongside Groq's Llama 3.1 405b-reasoning as the backend model, with Cartesia for voice synthesis. This allowed users to get summaries about a section, ask questions, and receive easy-to-understand answers—all through a conversational interface. To enhance user control, we also implemented Flask to connect a simple "start" and "stop" button for activating and deactivating the Vapi.ai voicebot. ### Challenges we ran into We ran into challenges with generating the Manim code, as we had to make the Manim code generate diagrams for a large variety of concepts. We eventually solved this through using the Hyperbolic API to generate Manim Code based on a scene description. ### Accomplishments we're proud of We're proud of integrating the Vapi API, as users can communicate directly with the voice agent. We're also happy that we could create a visual knowledge graph that allows users to see the history and related works of the paper. ### What we learned We learned a lot about integrating LLMs into full-stack web applications, and styling our web app using Tailwind.js. ### What's next for scholora.ai We hope to add a slider that varies responses based on the user's skill level. ## README (from the GitHub repository) No README available. ## Detected evidence (automated analysis) Indexed codebase: 13 recognized source files, 35 KB. - CSS (language) — detected in the code - Flask (technology) — detected in the code - JavaScript (language) — detected in the code - Next.js (technology) — detected in the code - OpenAI (technology) — detected in the code - Python (language) — detected in the code - React (technology) — detected in the code - Tailwind CSS (technology) — detected in the code - Google Gemini (technology) — claimed on Devpost, not found in the code - HTML (language) — claimed on Devpost, not found in the code ## Codebase structure (from repository index) ### Files (23 of 23) ``` .DS_Store .gitignore AI Backend/chroma.py AI Backend/vapi.py backend/prereq_search.py frontend/.eslintrc.json frontend/.gitignore frontend/components.json frontend/jsconfig.json frontend/next.config.mjs frontend/package.json frontend/postcss.config.mjs frontend/README.md frontend/src/app/components/comb.js frontend/src/app/components/Main.js frontend/src/app/components/ui/select.jsx frontend/src/app/components/ui/slider.jsx frontend/src/app/globals.css frontend/src/app/layout.js frontend/src/app/page.js frontend/src/lib/utils.js frontend/tailwind.config.js requirements.txt ``` ### Dependencies - frontend/package.json: @radix-ui/react-icons@^1.3.0, @radix-ui/react-select@^2.1.2, @radix-ui/react-slider@^1.2.1, axios@^1.7.7, class-variance-authority@^0.7.0, clsx@^2.1.1, eslint@^8, eslint-config-next@14.2.15, framer-motion@^11.11.9, lucide-react@^0.453.0, next@^14.2.15, postcss@^8, react@^18, react-dom@^18, tailwind-merge@^2.5.4, tailwindcss@^3.4.1, tailwindcss-animate@^1.0.7 - requirements.txt: black@==21.6b0, elevenlabs@==0.2.24, flake8@==3.9.2, Flask@==2.0.1, Flask-CORS@==3.0.10, numpy@==1.21.0, openai@==0.27.0, pyaudio@==0.2.11, pydub@==0.25.1, pytest@==6.2.4, python-dotenv@==0.19.0, scipy@==1.7.0, sounddevice@==0.4.1, SpeechRecognition@==3.8.1 ### Recent commits (newest first) - idek - Connect voice stuff with frontend + add knowledge graph - idek - fixed ui and questions - Video stuff - Merge branch 'main' of https://github.com/aryamankukal/CalHacks - Pdf stuff - tried video stuff, not working: - app name???? - added search paper part of upload page - colors - Merge branch 'main' of https://github.com/aryamankukal/CalHacks - changed scroll bar color - API stuff - API stuff - API stuff - API stuff - Merged - Add AI backend files - translate skill to words ## Key source files (fetched from GitHub, selected and truncated for size) ### requirements.txt ``` # Backend dependencies Flask==2.0.1 Flask-CORS==3.0.10 python-dotenv==0.19.0 openai==0.27.0 elevenlabs==0.2.24 SpeechRecognition==3.8.1 pyaudio==0.2.11 pydub==0.25.1 numpy==1.21.0 scipy==1.7.0 sounddevice==0.4.1 # Frontend dependencies (for reference, not installed via pip) # These would typically be in package.json for a React app # react==17.0.2 # react-dom==17.0.2 # axios==0.21.1 # framer-motion==4.1.17 # Development tools pytest==6.2.4 black==21.6b0 flake8==3.9.2 ``` ### frontend/package.json ``` { "name": "frontend", "version": "0.1.0", "private": true, "scripts": { "dev": "next dev", "build": "next build", "start": "next start", "lint": "next lint" }, "dependencies": { "@radix-ui/react-icons": "^1.3.0", "@radix-ui/react-select": "^2.1.2", "@radix-ui/react-slider": "^1.2.1", "axios": "^1.7.7", "class-variance-authority": "^0.7.0", "clsx": "^2.1.1", "framer-motion": "^11.11.9", "lucide-react": "^0.453.0", "next": "^14.2.15", "react": "^18", "react-dom": "^18", "tailwind-merge": "^2.5.4", "tailwindcss-animate": "^1.0.7" }, "devDependencies": { "eslint": "^8", "eslint-config-next": "14.2.15", "postcss": "^8", "tailwindcss": "^3.4.1" } } ``` ### frontend/src/app/page.js ```javascript // import FileUpload from './components/FileUpload'; // import WordWeb from './components/RelatedPaperMap'; // import Main from './components/Main'; import App from './components/comb'; export default function Home() { return (
{/*
*/}
); } ``` ### frontend/src/app/layout.js ```javascript import localFont from "next/font/local"; import "./globals.css"; const geistSans = localFont({ src: "./fonts/GeistVF.woff", variable: "--font-geist-sans", weight: "100 900", }); const geistMono = localFont({ src: "./fonts/GeistMonoVF.woff", variable: "--font-geist-mono", weight: "100 900", }); export const metadata = { title: "Create Next App", description: "Generated by create next app", }; export default function RootLayout({ children }) { return ( {children} ); } ``` ### frontend/src/app/components/Main.js ```javascript 'use client'; import React, { useState } from 'react'; function App() { const [progress, setProgress] = useState(30); // Example progress state return (
{/* Header */}

I am a college student

Teach me about vectors

{/* Main Content */}
{/* Vector Visualization */}
{/* Placeholder for an SVG or Image */} Vector Visualization
{/* Answer Input Box */}

If Vector1 had a magnitude of 10 and the second vector had a magnitude of 5, what will the resultant vector's magnitude be on the assumption that Vector1 and Vector2 are at a 0-degree angle?

{/* Footer with Chunked Progress Bar */}
); } export default App; ``` ### AI Backend/vapi.py ```python from flask import Flask, request, jsonify from vapi_python import Vapi app = Flask(__name__) vapi = Vapi(api_key='3661f60f-c3ea-4dfd-acf5-f4d302258e7a') assistant = { 'firstMessage': 'Would you like to start?', 'context': 'You are an AI assistant that explains research papers concisely and in a easy to understand way. Your task is to provide clear and concise explanations at a high level of academic papers the user is interested in as if you were explaining the paper to an elementary school student with no prior knowledge of the topics.', 'model': { 'provider': 'groq', 'model': 'llama-3.1-405b-reasoning', 'knowledgeBase': { "provider": "canonical", "fileIds": ["8c4d6d2d-6cbd-4ca7-a54b-56791cffba7f"] }, }, 'voice': { 'provider': 'cartesia', 'voiceId': '638efaaa-4d0c-442e-b701-3fae16aad012' }, 'interruptionsEnabled': False, 'recordingEnabled': True, 'endCallMessage': 'Thank you' } @app.route('/api/start-voicebot', methods=['POST']) def start_voicebot(): vapi.start(assistant=assistant) return jsonify({"message": "Voicebot started"}), 200 @app.route('/api/stop-voicebot', methods=['POST']) def stop_voicebot(): vapi.stop() return jsonify({"message": "Voicebot stopped"}), 200 if __name__ == '__main__': app.run(debug=True, port=5001) ``` ### frontend/tailwind.config.js ```javascript /** @type {import('tailwindcss').Config} */ module.exports = { darkMode: ["class"], content: [ "./src/pages/**/*.{js,ts,jsx,tsx,mdx}", "./src/components/**/*.{js,ts,jsx,tsx,mdx}", "./src/app/**/*.{js,ts,jsx,tsx,mdx}", ], theme: { extend: { colors: { background: 'hsl(var(--background))', foreground: 'hsl(var(--foreground))', card: { DEFAULT: 'hsl(var(--card))', foreground: 'hsl(var(--card-foreground))' }, popover: { DEFAULT: 'hsl(var(--popover))', foreground: 'hsl(var(--popover-foreground))' }, primary: { DEFAULT: 'hsl(var(--primary))', foreground: 'hsl(var(--primary-foreground))' }, secondary: { DEFAULT: 'hsl(var(--secondary))', foreground: 'hsl(var(--secondary-foreground))' }, muted: { DEFAULT: 'hsl(var(--muted))', foreground: 'hsl(var(--muted-foreground))' }, accent: { DEFAULT: 'hsl(var(--accent))', foreground: 'hsl(var(--accent-foreground))' }, destructive: { DEFAULT: 'hsl(var(--destructive))', foreground: 'hsl(var(--destructive-foreground))' }, border: 'hsl(var(--border))', input: 'hsl(var(--input))', ring: 'hsl(var(--ring))', chart: { '1': 'hsl(var(--chart-1))', '2': 'hsl(var(--chart-2))', '3': 'hsl(var(--chart-3))', '4': 'hsl(var(--chart-4))', '5': 'hsl(var(--chart-5))' } }, borderRadius: { lg: 'var(--radius)', md: 'calc(var(--radius) - 2px)', sm: 'calc(var(--radius) - 4px)' } } }, plugins: [require("tailwindcss-animate")], }; ``` ### backend/prereq_search.py ```python # -*- coding: utf-8 -*- """prereq_yt.ipynb Automatically generated by Colab. Original file is located at https://colab.research.google.com/drive/1BbgEnUzf1jgf8aU-hRHDdr6LIaT12kXP """ import urllib.request import re import ssl import spacy import PyPDF2 from sklearn.feature_extraction.text import TfidfVectorizer CUSTOM_STOPWORDS = set([ "methods", "introduction", "related", "work", "conclusion", "references", "abstract", "author", "study", "figure", "table", "journal", "doi", "section" ]) def clean_keyword(keyword): keyword = re.sub(r'\s+', ' ', keyword) keyword = re.sub(r'[^\w\s]', '', keyword) return keyword.strip() def searchVideoForKeyword(searchKeyword): allvideos = [] searchKeyword = clean_keyword(searchKeyword) if len(searchKeyword.split(" ")) > 1: searchKeyword = searchKeyword.replace(" ", "+") url = f"https://www.youtube.com/results?search_query={searchKeyword}" gcontext = ssl.SSLContext() try: html = urllib.request.urlopen(url, context=gcontext) video_ids = re.findall(r"watch\?v=(\S{11})", html.read().decode()) if video_ids: allvideos.append(f"https://www.youtube.com/embed/{video_ids[0]}") except Exception as e: print(f"Error fetching video for keyword '{searchKeyword}': {e}") return allvideos def extract_section(text, section_name, next_sections=None): section_start = re.search(rf'\b{section_name}\b', text, re.IGNORECASE) if not section_start: return "" if next_sections: next_section = re.search(rf'\b({"|".join(next_sections)})\b', text[section_start.end():], re.IGNORECASE) if next_section: return text[section_start.end():section_start.end() + next_section.start()] return text[section_start.end():] def extract_keywords_from_pdf(pdf_path, num_keywords=5): with open(pdf_path, 'rb') as pdf_file: reader = PyPDF2.PdfReader(pdf_file) text = "" for page in reader.pages: text += page.extract_text() methods_text = extract_section(text, 'Methods', ['Results', 'Discussion', 'Conclusion']) intro_text = extract_section(text, 'Introduction', ['Methods', 'Discussion', 'Conclusion']) conclusion_text = extract_section(text, 'Conclusion', ['References']) combined_text = methods_text + intro_text + conclusion_text if not combined_text: return [] # Process text with spacy nlp = spacy.load("en_core_web_sm") doc = nlp(combined_text) # Extract noun chunks raw_chunks = [chunk.text.lower() for chunk in doc.noun_chunks if len(chunk.text.split()) > 1] cleaned_chunks = [clean_keyword(chunk) for chunk in raw_chunks if chunk not in CUSTOM_STOPWORDS and len(chunk) > 2] vectorizer = TfidfVectorizer(max_features=num_keywords, stop_words='english') X = vectorizer.fit_transform(cleaned_chunks) feature_names = vectorizer.get_feature_names_out() return list(feature_names) def search_videos_from_pdf(pdf_path): keywords = extract_keywords_from_pdf(pdf_path, num_keywords=10) print(f"Extracted Keywords: {keywords}") # Search for videos based on the extracted keywords videos = {} for keyword in keywords: video_links = searchVideoForKeyword(keyword) videos[keyword] = video_links return videos pdf_path = "/content/1706.03762v7.pdf" video_results = search_videos_from_pdf(pdf_path) for keyword, video_links in video_results.items(): print(f"Keyword: {keyword}") for video in video_links: print(f"Video Link: {video}") ``` ### AI Backend/chroma.py ```python import chromadb from chromadb.utils import embedding_functions import requests import xml.etree.ElementTree as ET from urllib.parse import urlencode, quote_plus from datetime import datetime # Initialize Chroma client chroma_client = chromadb.Client() # Use OpenAI's embedding function (ensure you have an API key for this) # Replace 'YOUR_OPENAI_API_KEY' with your actual API key embedding_function = embedding_functions.OpenAIEmbeddingFunction( api_key='OPENAI_API_KEY', model_name="text-embedding-ada-002" ) # Create or get collections for before and after papers before_collection = chroma_client.get_or_create_collection(name="before_papers", embedding_function=embedding_function) after_collection = chroma_client.get_or_create_collection(name="after_papers", embedding_function=embedding_function) def search_arxiv_paper(title): base_url = 'http://export.arxiv.org/api/query?' query = 'search_query=ti:"{}"&max_results=1'.format(title) url = base_url + query response = requests.get(url) if response.status_code != 200: print(f"Error fetching paper: {response.status_code} - {response.text}") return None # Parse the XML response root = ET.fromstring(response.content) entries = root.findall('{http://www.w3.org/2005/Atom}entry') if entries: return entries[0] else: print("Paper not found.") return None def get_paper_metadata(entry): paper = {} paper['id'] = entry.find('{http://www.w3.org/2005/Atom}id').text paper['title'] = entry.find('{http://www.w3.org/2005/Atom}title').text.strip() paper['summary'] = entry.find('{http://www.w3.org/2005/Atom}summary').text.strip() paper['authors'] = [author.find('{http://www.w3.org/2005/Atom}name').text for author in entry.findall('{http://www.w3.org/2005/Atom}author')] paper['categories'] = [category.attrib['term'] for category in entry.findall('{http://www.w3.org/2005/Atom}category')] paper['published'] = entry.find('{http://www.w3.org/2005/Atom}published').text return paper def search_similar_papers(paper, when='before', max_results=50): base_url = 'http://export.arxiv.org/api/query?' # Use the primary categories and keywords from the title categories = ' OR '.join(['cat:' + cat for cat in paper['categories']]) title_keywords = ' '.join(quote_plus(word) for word in paper['title'].split() if len(word) > 3) # Date filter published_date = paper['published'][:10] # Extract date in YYYY-MM-DD if when == 'before': date_from = '1990-01-01' date_to = published_date elif when == 'after': date_from = published_date date_to = datetime.now().strftime('%Y-%m-%d') else: date_from = '1990-01-01' date_to = datetime.now().strftime('%Y-%m-%d') date_filter = f'submittedDate:[{date_from} TO {date_to}]' search_query = f'({categories}) AND ({title_keywords}) AND {date_filter}' query = urlencode({'search_query': search_query, 'start': 0, 'max_results': max_results}) url = base_url + query response = requests.get(url) if response.status_code != 200: print(f"Error fetching similar papers ({when}): {response.status_code} - {response.text}") return [] # Parse the XML response root = ET.fromstring(response.content) entries = root.findall('{http://www.w3.org/2005/Atom}entry') # Exclude the main paper if it's in the results similar_entries = [entry for entry in entries if entry.find('{http://www.w3.org/2005/Atom}id').text != paper['id']] return similar_entries def main(): # Get user input title = input("Enter the title of the research paper: ").strip() # Search for the paper entry = search_arxiv_paper(title) if not entry: return main_paper = get_paper_metadata(entry) print(f"\nMain paper: {main_paper['title']}") print(f"Published on: {main_paper['published'][:10]}") # Search for similar papers published before the main paper before_entries = search_similar_papers(main_paper, when='before', max_results=50) print(f"\nFound {len(before_entries)} similar papers published before the main paper.") before_papers = [get_paper_metadata(entry) for entry in before_entries] # Search for similar papers published after the main paper after_entries = search_similar_papers(main_paper, when='after', max_results=50) print(f"\nFound {len(after_entries)} similar papers published after the main paper.") after_papers = [get_paper_metadata(entry) for entry in after_entries] # Add all before papers to their collection before_docs = [paper['title'] + ': ' + paper['summary'] for paper in before_papers] before_ids = [paper['id'] for paper in before_papers] before_metadatas = [{ 'id': paper['id'], 'title': paper.get('title', ''), 'authors': ', '.join(paper.get('authors', [])), 'published': paper.get('published', '')[:10] } for paper in before_papers] before_collection.add(documents=before_docs, metadatas=before_metadatas, ids=before_ids) # Add all after papers to their collection after_docs = [paper['title'] + ': ' + paper['summary'] for paper in after_papers] after_ids = [paper['id'] for paper in after_papers] after_metadatas = [{ 'id': paper['id'], 'title': paper.get('title', ''), 'authors': ', '.join(paper.get('authors', [])), 'published': paper.get('published', '')[:10] } for paper in after_papers] after_collection.add(documents=after_docs, metadatas=after_metadatas, ids=after_ids) print("\nPapers added to Chroma collections for future queries.") # Query the 'before' papers print("\nSearching for top 3 similar papers from the 'before' set:") query_text = main_paper['title'] + ' ' + main_paper['summary'] results_before = before_collection.query( query_texts=[query_text], n_results=3 ) print("\nTop 3 similar papers [truncated — 661 more characters] ``` ### frontend/src/lib/utils.js ```javascript import { clsx } from "clsx"; import { twMerge } from "tailwind-merge" export function cn(...inputs) { return twMerge(clsx(inputs)); } ``` [4 more indexed source files omitted to keep this export small. The full file list is in the Codebase structure section above.]