HackStack

Project Info

texTalk

Devpost

Inspiration

The genesis of this project was an extra credit opportunity in a discrete mathematics course, where the challenge was to transcribe class notes into LaTeX format. Having experienced firsthand the tediousness and time-consuming nature of manually converting these notes, I was inspired to develop a solution that could streamline this process. This project, thus, aims to bridge the gap between spoken mathematical concepts and their formal documentation in LaTeX, making the transcription process not only faster but also more accessible to students and educators alike.

What it does

TexTalk revolutionizes mathematical documentation by seamlessly converting spoken words into LaTeX code, complete with detailed explanations and solutions. This code is then transformed into a user-friendly, accessible pdf document. The dual-output system not only streamlines document creation but also deepens comprehension by clarifying each step of the equations. Furthermore, it supports accessibility with a dictation feature that audibly reads back the document, ensuring vision-impaired users can fully engage with the content.

How we built it

TexTalk was crafted using a modular Python approach, integrating specialized libraries for each functionality. We implemented voice capture with PvRecorder, then utilized speech recognition coupled with speech synthesis and custom instructions for transcription, converting speech to LaTeX. The resulting LaTeX code is rendered visually and, where applicable, solved for immediate results. AI-powered using a custom fine-tuned version of gpt4 and Toyomi Hayashi's speech synthesis through our LLM bridge module, the system generates step-by-step explanations, which are then formatted for clarity by our steps converter. The process, from recording to detailed LaTeX documents, is both efficient and user-friendly, streamlining the creation of complex mathematical documentation. Additionally, this modular approach allowed us to efficiently integrate the dictation feature because we had access to the original text formatted input which we then text-to-speech back to the user. Steps Converter | Voice Transcription | Dictation

Challenges we ran into

One of the foremost challenges was achieving high accuracy in speech recognition for mathematical terminology, which often includes highly specialized symbols and expressions. Fine-tuning and debugging the LaTeX conversion engine required a deep dive into both linguistic processing and mathematical structuring, ensuring the translation from spoken word to LaTeX code was both accurate and logically formatted. Additionally, crafting the explanation module to produce clear, step-by-step solutions demanded a strong understanding of mathematical problem-solving.

Accomplishments we're proud of

Successfully creating a tool that not only transcribes but also breaks down and explains mathematical equations was a great feeling. TexTalk stands as a testament to the potential of integrating technology with education, offering a novel approach to mathematical documentation. Witnessing TexTalk accurately convert complex spoken equations into LaTeX documents, complete with elucidative breakdowns, has been incredibly rewarding.

What we learned

This project deepened our understanding of speech recognition technologies, NLP, and LaTeX formatting, highlighting the interdisciplinary nature of developing educational tools. We gained insights into the complexities of mathematical notation and the challenges of translating it from speech to structured documents. The development process also honed our skills in fine-tuning and prompt engineering, particularly in creating logic that interprets and structures mathematical content.

What's next

for TexTalk Looking forward, we aim to enhance TexTalk's accuracy and expand its vocabulary to encompass a broader range of mathematical fields. Integrating machine learning to refine the contextual understanding of equations and exploring real-time transcription are key objectives. Additionally, we plan to develop an interactive interface that allows users to edit and refine generated LaTeX documents and explanations directly, fostering a more integrated and user-friendly experience.

Analysis

Compare with all teams

View

Metric

Figures cover GitHub contributors during the hackathon window. A co-authored commit counts in full for each author, so per-member totals add up to more than the whole-team figures.

Technology

Found in codeClaimed only
  • CSSIn code
  • JavaScriptIn code
  • Next.jsIn code
  • PythonIn code
  • ReactIn code
  • OpenAIClaimed

5 of 6 appear in the indexed code. 1 claimed on Devpost could not be matched to code, which may simply mean the tool leaves no trace in the repository.

AI coding agents

No AI coding agent signals were found in this repository.

Detected from committed agent config files and commit authorship. Absence of a signal is not proof an agent was unused.

Codebase size

Source size

15 KB

Source files

13

Counts recognized source files only; vendored directories, binaries and lockfiles are excluded, so this is smaller than the repository on disk.

0 stars