# Project export: STUDYAI This document was generated by HackStack to give an AI agent context about a hackathon project. Sections are labeled with their provenance; content marked as truncated was cut to keep this document small. ## Project metadata - Hackathon: Cal Hacks 10.0 - Tagline: Systematic Teaching Using Dynamic Yielding and Autonomous Intelligence (StudyAI) is an advanced, voice-activated study assistant designed to aid students in learning and comprehending various subjects - Devpost: https://devpost.com/software/studyai-ru8lf4 - GitHub: https://github.com/IdkwhatImD0ing/STUDYAI - Team: 2 GitHub contributor(s) — Bill Zhang (18 commits), Varun Swaminathan (5 commits) ## Devpost submission (written by the team) ### Overview *S*ystematic *T*eaching *U*sing *D*ynamic *Y*ielding and *A*utonomous *I*ntelligence Overview StudyAI is a groundbreaking, voice-activated study assistant engineered to redefine the self-study experience. Utilizing cutting-edge machine learning, voice recognition, and natural language understanding technologies, this platform offers an array of features designed to facilitate efficient and effective study sessions. ### Inspiration In an era flooded with information, the conventional methods of self-study are fraught with challenges like information overload, disorganized note management, and a lack of personalized learning experiences. Study AI tackles these issues head-on, offering a revolutionary approach to academic learning and comprehension. Goals Our mission is to revolutionize self-study by providing an intelligent, voice-activated assistant that delivers a seamless and intuitive user experience. Study AI aims to become the go-to platform for students, offering features like text and video summarization, explanatory image generation, educational video recommendations, and personalized note management. Built With Frontend: Reflex.dev with custom React components as plugins Backend: FastAPI (via Reflex.dev) Voice Recognition and Text-to-Speech: 11Labs, Whisper Autonomous Agents: OpenAI function calling agents Text Summarization: Using mistralai/Mistral-7B-Instruct-v0.1 from TogetherAI Image Generation: Using stabilityai/stable-diffusion-2-1 from TogetherAI Challenges Implementing real-time voice recognition and natural language understanding with minimal latency (less than 5 seconds response time) Ensuring seamless integration of multiple technologies, including machine learning models and external APIs. Achieving scalability while maintaining high performance and reliability. Ensuring data privacy and security. Accomplishments Successfully developed a voice-activated command parsing module. Implemented autonomous decision-making capabilities using OpenAI function calling. Engineered data fetching and summarization modules with high accuracy and efficiency. Integrated YouTube API for enriched educational content. What We Learned How to integrate machine learning and voice recognition technologies to create a seamless user experience. The importance of a modular architecture for scalability and future expansions. How to manage and manipulate large datasets for quick and accurate retrieval. The nuances of user experience design, particularly for an educational tool. What's Next Implementing a personalized learning path based on user behavior and preferences. Expanding the database to include more academic resources and journals. Launching a mobile application to make StudyAI accessible on the go. Exploring partnerships with educational institutions for broader reach and impact. Features Voice Activation and Command Parsing: Understands user needs through voice commands. Autonomous Decision Making: Adapts to the user's needs, activating the most useful tools or functionalities. Text Summarization: Offers concise, yet comprehensive, academic information. Generate Visualizations: Provides visual aids for complex topics. YouTube Video Suggestions: Recommends relevant educational videos from YouTube. ## README (from the GitHub repository) # S.T.U.D.Y.A.I Systematic Teaching Using Dynamic Yielding and Autonomous Intelligence --- ## Overview Studyai is a groundbreaking, voice-activated study assistant engineered to redefine the self-study experience. Utilizing cutting-edge machine learning, voice recognition, and natural language understanding technologies, this platform offers an array of features designed to facilitate efficient and effective study sessions. --- ## Inspiration In an era flooded with information, the conventional methods of self-study are fraught with challenges like information overload, disorganized note management, and a lack of personalized learning experiences. Autostudy Buddy was conceived to tackle these issues head-on, offering a revolutionary approach to academic learning and comprehension. --- ## Goals Our mission is to revolutionize self-study by providing an intelligent, voice-activated assistant that delivers a seamless and intuitive user experience. Autostudy Buddy aims to become the go-to platform for students, offering features like text and video summarization, explanatory image generation, educational video recommendations, and personalized note management. --- ## Built With - **Frontend**: Reflex.dev with React components as plugins - **Backend**: FastAPI - **Voice Recognition and Text-to-Speech**: 11Labs, LLM - **Autonomous Agents**: OpenAI function calling agents - **Text Summarization**: TogetherAI - **Image Generation**: TogetherAI - **Database**: MindsDB - **YouTube Integration**: MindsDB with YouTube API --- ## Challenges - Implementing real-time voice recognition and natural language understanding. - Ensuring seamless integration of multiple technologies, including machine learning models and external APIs. - Achieving scalability while maintaining high performance and reliability. - Ensuring data privacy and security. --- ## Accomplishments - Successfully developed a voice-activated command parsing module. - Implemented autonomous decision-making capabilities using OpenAI. - Engineered data fetching and summarization modules with high accuracy and efficiency. - Integrated YouTube API for enriched educational content. - Designed an effective note management system using MindsDB. --- ## What We Learned - How to integrate machine learning and voice recognition technologies to create a seamless user experience. - The importance of a modular architecture for scalability and future expansions. - How to manage and manipulate large datasets for quick and accurate retrieval. - The nuances of user experience design, particularly for an educational tool. --- ## What's Next - Implementing a personalized learning path based on user behavior and preferences. - Expanding the database to include more academic resources and journals. - Launching a mobile application to make Autostudy Buddy accessible on the go. - Exploring partnerships with educational institutions for broader reach and impact. --- ## Functions - **Voice Activation and Command Parsing**: Understands user needs through voice commands. - **Autonomous Decision Making**: Adapts to the user's needs, activating the most useful tools or functionalities. - **Text Summarization**: Offers concise, yet comprehensive, academic information. - **Image Generation**: Provides visual aids for complex topics. - **YouTube Video Suggestions**: Recommends relevant educational videos. - **Note Management**: Organizes user-uploaded study notes for easy retrieval. ## Detected evidence (automated analysis) Indexed codebase: 41 recognized source files, 135 KB. - CSS (language) — detected in the code - JavaScript (language) — detected in the code - Next.js (technology) — detected in the code - Python (language) — detected in the code - React (technology) — detected in the code - Tailwind CSS (technology) — detected in the code - OpenAI (technology) — claimed on Devpost, not found in the code ## Codebase structure (from repository index) ### Files (56 of 56) ``` .env .env.example .gitignore .python-version .web/.gitignore .web/components/reflex/chakra_color_mode_provider.js .web/components/reflex/radix_themes_color_mode_provider.js .web/env.json .web/jsconfig.json .web/next.config.js .web/package.json .web/pages/_app.js .web/pages/_document.js .web/pages/404.js .web/pages/dashboard.js .web/pages/index.js .web/pages/recorder.js .web/pages/settings.js .web/postcss.config.js .web/public/AutoRecorder.js .web/public/voice-processor.js .web/reflex.json .web/styles/code/prism.js .web/styles/styles.css .web/styles/tailwind.css .web/tailwind.config.js .web/utils/client_side_routing.js .web/utils/components.js .web/utils/context.js .web/utils/helpers/dataeditor.js .web/utils/state.js .web/utils/theme.js assets/AutoRecorder.js assets/voice-processor.js automatic_chat.py Experiments/hp0.txt Experiments/speechToTest.ipynb Experiments/textToSpeech.ipynb Experiments/youtube_transcript.ipynb py README.md requirements.txt rxconfig.py STUDYAI/__init__.py STUDYAI/components/__init__.py STUDYAI/components/sidebar.py STUDYAI/pages/__init__.py STUDYAI/pages/dashboard.py STUDYAI/pages/index.py STUDYAI/pages/recorder.py STUDYAI/pages/settings.py STUDYAI/state.py STUDYAI/STUDYAI.py STUDYAI/styles.py STUDYAI/templates/__init__.py STUDYAI/templates/template.py ``` ### Dependencies - .web/package.json: @chakra-ui/icons@^2.0.19, @chakra-ui/react@^2.6.1, @chakra-ui/system@^2.5.7, @emotion/react@^11.11.0, @emotion/styled@^11.11.0, autoprefixer@10.4.14, axios@1.4.0, focus-visible@5.2.0, framer-motion@^10.16.4, json5@2.2.3, next@13.5.4, next-sitemap@4.1.8, next-themes@0.2.0, postcss@8.4.24, react@18.2.0, react-dom@18.2.0, react-markdown@^8.0.7, react-syntax-highlighter@^15.5.0, recordrtc@5.6.2, rehype-katex@^6.0.3, rehype-raw@^6.1.1, remark-gfm@^3.0.1, remark-math@^5.1.1, socket.io-client@4.6.1, tailwindcss@^3.3.2, universal-cookie@4.0.4 - requirements.txt: reflex@==0.3.1 ### Recent commits (newest first) - MAnual addition - Merge pull request #5 from IdkwhatImD0ing/readme - FINAL FINAL REQUIREMTNS - Requirements - Merge pull request #4 from IdkwhatImD0ing/reflexState - Fixing concurrency - It works sometimes - No longer using pydantic in streaming audio - AutoRecorder works - Merge changes - Update requirements.txt - Working - Merge pull request #1 from IdkwhatImD0ing/youtube-transcription - Merge pull request #2 from IdkwhatImD0ing/automatic - Transcribe youtube video - Update automatic_chat - Bug fix 2 - Bug fix - Using faster whisper - Audomatic sst ## Key source files (fetched from GitHub, selected and truncated for size) ### requirements.txt ``` ��aif360==0.5.0 faster-whisper==0.9.0 flask-cors==3.0.10 ipykernel==6.20.1 mccabe==0.6.1 openai==0.28.1 opencv-python==4.7.0.68 pip-chill==1.0.3 pyaudio==0.2.13 pycodestyle==2.5.0 pyflakes==2.1.1 python-dotenv==1.0.0 reflex==0.3.1 sklearn==0.0.post1 tensorboard==2.11.2 virtualenv==20.17.1 wave==0.0.2 yapf==0.32.0 reflex==0.3.1 ``` ### .web/package.json ``` { "name": "reflex", "scripts": { "dev": "next dev", "export": "next build && next export -o _static", "export-sitemap": "next build && next-sitemap && next export -o _static", "prod": "next start" }, "dependencies": { "@chakra-ui/icons": "^2.0.19", "@chakra-ui/react": "^2.6.1", "@chakra-ui/system": "^2.5.7", "@emotion/react": "^11.11.0", "@emotion/styled": "^11.11.0", "axios": "1.4.0", "focus-visible": "5.2.0", "framer-motion": "^10.16.4", "json5": "2.2.3", "next": "13.5.4", "next-sitemap": "4.1.8", "next-themes": "0.2.0", "react": "18.2.0", "react-dom": "18.2.0", "react-markdown": "^8.0.7", "react-syntax-highlighter": "^15.5.0", "recordrtc": "5.6.2", "rehype-katex": "^6.0.3", "rehype-raw": "^6.1.1", "remark-gfm": "^3.0.1", "remark-math": "^5.1.1", "socket.io-client": "4.6.1", "tailwindcss": "^3.3.2", "universal-cookie": "4.0.4" }, "devDependencies": { "autoprefixer": "10.4.14", "postcss": "8.4.24" } } ``` ### STUDYAI/pages/index.py ```python """The home page of the app.""" from STUDYAI import styles from STUDYAI.templates import template import reflex as rx @template(route="/", title="Home", image="/github.svg") def index() -> rx.Component: """The home page. Returns: The UI for the home page. """ with open("README.md", encoding="utf-8") as readme: content = readme.read() return rx.markdown(content, component_map=styles.markdown_style) ``` ### .web/pages/index.js ```javascript import { Fragment, useContext, useEffect, useRef, useState } from "react" import { useRouter } from "next/router" import { Event, getAllLocalStorageItems, getRefValue, getRefValues, isTrue, preventDefault, refs, spreadArraysOrObjects, uploadFiles, useEventLoop } from "/utils/state" import { ColorModeContext, EventLoopContext, initialEvents, StateContext } from "/utils/context.js" import "focus-visible/dist/focus-visible" import "katex/dist/katex.min.css" import { Box, Center, Code, Heading, HStack, Image, Link, ListItem, Menu, MenuButton, MenuDivider, MenuItem, MenuList, Modal, ModalBody, ModalContent, ModalHeader, ModalOverlay, OrderedList, Spacer, Text, UnorderedList, VStack } from "@chakra-ui/react" import { getEventURL } from "/utils/state.js" import NextLink from "next/link" import ReactMarkdown from "react-markdown" import remarkMath from "remark-math" import remarkGfm from "remark-gfm" import rehypeKatex from "rehype-katex" import rehypeRaw from "rehype-raw" import { Prism } from "react-syntax-highlighter" import { light } from "/styles/code/prism" import { HamburgerIcon } from "@chakra-ui/icons" import NextHead from "next/head" export default function Component() { const state = useContext(StateContext) const router = useRouter() const [ colorMode, toggleColorMode ] = useContext(ColorModeContext) const focusRef = useRef(); // Main event loop. const [addEvents, connectError] = useContext(EventLoopContext) // Set focus to the specified element. useEffect(() => { if (focusRef.current) { focusRef.current.focus(); } }) // Route after the initial page hydration. useEffect(() => { const change_complete = () => addEvents(initialEvents()) router.events.on('routeChangeComplete', change_complete) return () => { router.events.off('routeChangeComplete', change_complete) } }, [router]) return ( {isTrue(connectError !== null) ? ( {`Connection Error`} {`Cannot connect to server: `} {(connectError !== null) ? connectError.message : ''} {`. Check if server is reachable at `} {getEventURL().href} ) : ( )}
{`Home`} {`Dashboard`} {`Recorder`} {`Settings`}