# Project export: Polarity This document was generated by HackStack to give an AI agent context about a hackathon project. Sections are labeled with their provenance; content marked as truncated was cut to keep this document small. ## Project metadata - Hackathon: TreeHacks 2024 - Tagline: Polarity elevates your math and science lectures from audio to perfectly formatted LaTeX notes, transforming every spoken word into an equation and concept with textbook precision. - Devpost: https://devpost.com/software/polarity-zp0w5m - GitHub: https://github.com/rish1p/polarity - Team: 0 GitHub contributor(s) — ## Devpost submission (written by the team) ### Inspiration Our project, Polarity, was born in the midst of an educational crisis, where overwhelmed educators are often unable to provide written notes for their lectures. This not only places a significant burden on students to accurately document these lectures but also takes away valuable time that could be spent on understanding and internalizing the material. We saw a unique opportunity to use technology to fill this gap by creating a tool that not only aids students in generating their own precise notes effortlessly but also empowers teachers. ### What it does With Polarity, educators can record their teachings, and the system will automatically transcribe and convert these recordings into detailed, well-formatted LaTeX notes, with every mathematical equation and concept captured accurately from voice to paper. This approach not only ensures students have access to high-quality notes but also streamlines the note-distribution process, making it more efficient for teachers to provide essential study materials to their students. ### How we built it We first created a dataset of LaTeX files manually based on audio clips from a variety different math and physics lectures, in order to maintain a more representative data sample. After that, we fine-tuned our model utilizing data pairs of audio clip transcripts (from speech to text) to LaTeX code. We turned to establish our formal data pipeline at this point, where we utilized MonsterAPI speech->text and then GPT4 text->LaTeX. Since the data was trained, and the pipeline was fully working at this point, the only task left to tackle was building a website using the innovative Reflex.dev framework. We were able to do this and connect it to our API chain (MonsterAPI/GPT4), finalizing our work. ### Challenges we ran into One of the biggest challenges we faced was dealing with model limitations earlier in the hackathon. We initially tried to train a PredictionGuard suggested model, DeepSeek, on our task. However, we faced limitations with the token response being very low for these models. After switching to GPT4, nearly none of these issues were relevant anymore. The next major issue we had was the efficiency of the model - it became too slow to the point of not being able to function properly, causing runtime errors with our environment. We had to make several changes to make the model faster, including changing our prompts and our data pipeline. ### Accomplishments we're proud of Polarity’s capability to intelligently organize lecture content into a structured format stands out remarkably. Not only does it transcribe words but it also discerns the flow of the lecture to create meaningful headers and sections autonomously. Beyond that, the accuracy with which Polarity renders mathematical equations, symbols, and concepts is unparalleled. Through rigorous training, our system has mastered the complex language of mathematics, allowing it to replicate the exact equations and symbols discussed in lectures. This capability ensures that every notation, from simple algebraic expressions to intricate calculus, is flawlessly represented in LaTeX (textbook) format. Polarity is also equipped with advanced speech processing algorithms that filter out filler words commonly found in spoken language. This ensures that the final notes are concise and free of clutter, containing only the critical information that students need to focus on. This feature is especially beneficial in maintaining the high quality and readability of the notes, ensuring they serve as an effective study tool. ### What's next We plan to roll out Polarity as a product for schools and universities across America, hopefully easing a burden off of overworked educators by providing a method to generate class notes for students with ease. While many AI models already exist for general information, there is not a currently feasible model that can specifically help implement work in LaTeX. We are working to create an API that allows educators to specifically tailor a AI model to their class structure and format. ## README (from the GitHub repository) No README available. ## Detected evidence (automated analysis) Indexed codebase: 3 recognized source files, 5 KB. - Python (language) — detected in the code ## Codebase structure (from repository index) ### Files (5 of 5) ``` .gitignore Polarity/__init__.py Polarity/Polarity.py requirements.txt rxconfig.py ``` ### Dependencies - requirements.txt: reflex@==0.4.0 ### Recent commits (newest first) - Update Polarity.py - Push ## Key source files (fetched from GitHub, selected and truncated for size) ### requirements.txt ``` reflex==0.4.0 ``` ### rxconfig.py ```python import reflex as rx config = rx.Config( app_name="Polarity", ) ``` ### Polarity/Polarity.py ```python import reflex as rx import requests import mimetypes import os import time import logging from openai import OpenAI import json LaTeX = "" def wait_for_run_completion(client, thread_id, run_id, sleep_interval=5): while True: try: run = client.beta.threads.runs.retrieve(thread_id=thread_id, run_id=run_id) if run.completed_at: elapsed_time = run.completed_at - run.created_at formatted_elapsed_time = time.strftime( "%H:%M:%S", time.gmtime(elapsed_time) ) # print(f"Run completed in {formatted_elapsed_time}") logging.info(f"Run completed in {formatted_elapsed_time}") # Get messages here once Run is completed! messages = client.beta.threads.messages.list(thread_id=thread_id) last_message = messages.data[0] response = last_message.content[0].text.value print(f"{response}") LaTeX = f"{response}" break except Exception as e: logging.error(f"An error occurred while retrieving the run: {e}") break logging.info("Waiting for run to complete...") time.sleep(sleep_interval) def text_to_latex(text): message = text # Define your OpenAI API key directly api_key = "___" # Create an OpenAI client with the API key client = OpenAI(api_key=api_key) thread_id = "thread_nxgLyathgYSnRQHJHKZWczQG" assistant_id = "asst_mE1PhOgATRi7GgfAvqTSoZH6" message = client.beta.threads.messages.create( thread_id=thread_id, role="user", content=message ) run = client.beta.threads.runs.create( thread_id=thread_id, assistant_id="asst_mE1PhOgATRi7GgfAvqTSoZH6" ) # === Run === wait_for_run_completion(client=client, thread_id=thread_id, run_id=run.id) def speech_to_text(filepath): url = "https://api.monsterapi.ai/v1/generate/whisper" API_Key = "___" payload = {"diarize": "true", "language": "en"} file_name = os.path.basename(filepath) files = { "file": (file_name, open(filepath, "rb"), mimetypes.guess_type(filepath)[0]) } headers = {"accept": "application/json", "authorization": f"Bearer {API_Key}"} response = requests.post(url, data=payload, files=files, headers=headers) # FETCHING!!! process_id = (response.json())["process_id"] url = f"https://api.monsterapi.ai/v1/status/{process_id}" headers = { "accept": "application/json", "authorization": "Bearer ___" } status = "IN_PROGRESS" while status != "COMPLETED": response = requests.get(url, headers=headers) status = response.json()["status"] speakers = (response.json()["result"]["text"])['Sequence'] text = "" for speaker in speakers: text += speaker['transcription'] print(text) text_to_latex(text) class State(rx.State): """The app state.""" img: list[str] = [] text_result: str = "" async def handle_upload(self, files: list[rx.UploadFile]): for file in files: upload_data = await file.read() outfile = f".web/public/{file.filename}" # Save the file. with open(outfile, "wb") as file_object: file_object.write(upload_data) # Update the img var. self.img.append(file.filename) #!!! result = speech_to_text(outfile) self.text_result = result def index(): """The main view.""" return rx.vstack( rx.text_area(LaTeX), rx.upload( rx.vstack( rx.button("Select File", color="rgb(107,99,246)", bg="white", border="1px solid rgb(107,99,246)"), rx.text("Drag and drop files here or click to select files"), ), multiple=False, accept={ "audio/mpeg": [".mp3"], "video/mp4": [".mp4"] }, max_files=1, disabled=False, on_keyboard=True, border="1px dotted rgb(107,99,246)", padding="5em", ), rx.button( "Upload", on_click=lambda: State.handle_upload(rx.upload_files()), ), rx.text(State.text_result), # Display the speech-to-text result rx.chakra.responsive_grid( rx.foreach( State.img, lambda img: rx.vstack( rx.image(src=img), rx.text(img), ), ), columns=[2], spacing="5px", ), padding="5em", ) app = rx.App() app.add_page(index) app.compile() ```