# Project export: texTalk
This document was generated by HackStack to give an AI agent context about a hackathon project. Sections are labeled with their provenance; content marked as truncated was cut to keep this document small.
## Project metadata
- Hackathon: TreeHacks 2024
- Tagline: Transform Your Voice into Detailed LaTeX Documents: Elevate Mathematical Communication with Instant Voice-to-Equation Conversion and Comprehensive Step-by-Step Solutions
- Devpost: https://devpost.com/software/textalk-edkh0w
- GitHub: https://github.com/Alans44/TexTalk-
- Video: https://www.youtube.com/embed/jt5gBS2f_lQ?enablejsapi=1&hl=en_US&rel=0&start=&version=3&wmode=transparent
- Team: 1 GitHub contributor(s) — Alans44 (4 commits)
## Devpost submission (written by the team)
### Inspiration
The genesis of this project was an extra credit opportunity in a discrete mathematics course, where the challenge was to transcribe class notes into LaTeX format. Having experienced firsthand the tediousness and time-consuming nature of manually converting these notes, I was inspired to develop a solution that could streamline this process. This project, thus, aims to bridge the gap between spoken mathematical concepts and their formal documentation in LaTeX, making the transcription process not only faster but also more accessible to students and educators alike.
### What it does
TexTalk revolutionizes mathematical documentation by seamlessly converting spoken words into LaTeX code, complete with detailed explanations and solutions. This code is then transformed into a user-friendly, accessible pdf document. The dual-output system not only streamlines document creation but also deepens comprehension by clarifying each step of the equations. Furthermore, it supports accessibility with a dictation feature that audibly reads back the document, ensuring vision-impaired users can fully engage with the content.
### How we built it
TexTalk was crafted using a modular Python approach, integrating specialized libraries for each functionality. We implemented voice capture with PvRecorder, then utilized speech recognition coupled with speech synthesis and custom instructions for transcription, converting speech to LaTeX. The resulting LaTeX code is rendered visually and, where applicable, solved for immediate results. AI-powered using a custom fine-tuned version of gpt4 and Toyomi Hayashi's speech synthesis through our LLM bridge module, the system generates step-by-step explanations, which are then formatted for clarity by our steps converter. The process, from recording to detailed LaTeX documents, is both efficient and user-friendly, streamlining the creation of complex mathematical documentation. Additionally, this modular approach allowed us to efficiently integrate the dictation feature because we had access to the original text formatted input which we then text-to-speech back to the user. Steps Converter | Voice Transcription | Dictation
### Challenges we ran into
One of the foremost challenges was achieving high accuracy in speech recognition for mathematical terminology, which often includes highly specialized symbols and expressions. Fine-tuning and debugging the LaTeX conversion engine required a deep dive into both linguistic processing and mathematical structuring, ensuring the translation from spoken word to LaTeX code was both accurate and logically formatted. Additionally, crafting the explanation module to produce clear, step-by-step solutions demanded a strong understanding of mathematical problem-solving.
### Accomplishments we're proud of
Successfully creating a tool that not only transcribes but also breaks down and explains mathematical equations was a great feeling. TexTalk stands as a testament to the potential of integrating technology with education, offering a novel approach to mathematical documentation. Witnessing TexTalk accurately convert complex spoken equations into LaTeX documents, complete with elucidative breakdowns, has been incredibly rewarding.
### What we learned
This project deepened our understanding of speech recognition technologies, NLP, and LaTeX formatting, highlighting the interdisciplinary nature of developing educational tools. We gained insights into the complexities of mathematical notation and the challenges of translating it from speech to structured documents. The development process also honed our skills in fine-tuning and prompt engineering, particularly in creating logic that interprets and structures mathematical content.
### What's next
for TexTalk Looking forward, we aim to enhance TexTalk's accuracy and expand its vocabulary to encompass a broader range of mathematical fields. Integrating machine learning to refine the contextual understanding of equations and exploring real-time transcription are key objectives. Additionally, we plan to develop an interactive interface that allows users to edit and refine generated LaTeX documents and explanations directly, fostering a more integrated and user-friendly experience.
## README (from the GitHub repository)
# TexTalk
https://devpost.com/software/textalk-edkh0w?ref_content=my-projects-tab&ref_feature=my_projects
## Detected evidence (automated analysis)
Indexed codebase: 13 recognized source files, 15 KB.
- CSS (language) — detected in the code
- JavaScript (language) — detected in the code
- Next.js (technology) — detected in the code
- Python (language) — detected in the code
- React (technology) — detected in the code
- OpenAI (technology) — claimed on Devpost, not found in the code
## Codebase structure (from repository index)
### Files (21 of 21)
```
.gitattributes
.gitignore
jsconfig.json
LICENSE
llm_bridge.py
main.py
package.json
pages/_app.js
pages/index.js
README.md
recorder.py
renderer.py
resources/Equation1.flac
resources/output.flac
resources/speech.flac
solver.py
speaker.py
steps_converter.py
styles/globals.css
styles/index.module.css
transcript.py
```
### Dependencies
- package.json: next@^latest_version, react@^latest_version, react-dom@^latest_version
### Recent commits (newest first)
- Update README.md
- example output
- llm bridge + starter
- converting from local
- Initial commit
## Key source files (fetched from GitHub, selected and truncated for size)
### package.json
```
{
"name": "textalk-",
"version": "1.0.0",
"description": "",
"main": "index.js",
"scripts": {
"dev": "next dev",
"build": "next build",
"start": "next start",
"test": "echo \"Error: no test specified\" && exit 1"
},
"dependencies": {
"next": "^latest_version",
"react": "^latest_version",
"react-dom": "^latest_version"
},
"author": "",
"license": "ISC"
}
```
### main.py
```python
import recorder
import transcript
import renderer
import solver
import steps_converter
import llm_bridge
if __name__ == "__main__":
record = recorder.start_record()
formula = transcript.audio_to_latex(record)
renderer.render(formula)
result = solver.evaluate(formula)
prompt_solve = llm_bridge.build_prompt_solve(formula, result)
print(prompt_solve)
res = llm_bridge.get_response("gpt-4", prompt_solve)
print(res)
steps = steps_converter.convert(res)
steps_converter.render_steps(steps)
```
### pages/index.js
```javascript
// Assuming Label and Button are components that accept className props
import React from 'react';
import { Label } from "@/components/ui/label";
import Button from "@/components/ui/button";
import styles from '../styles/index.module.css';
export default function Component() {
return (
);
}
function MicIcon(props) {
// Ensure the styles.micIcon is defined in your CSS module to apply styles to MicIcon
return (
);
}
```
### solver.py
```python
from latex2sympy2 import latex2sympy, latex2latex
from sympy import *
import llm_bridge
def evaluate(tmp):
latex = tmp.replace('\x0c','\\f')
latex = llm_bridge.get_response("gpt-4", llm_bridge.build_prompt_correct(latex))
print(latex)
return(latex2sympy(latex).doit())
```
### speaker.py
```python
import requests
from dotenv import load_dotenv, find_dotenv
import os
import soundfile as sf
from IPython.display import Audio
import numpy as np
import torch
load_dotenv(find_dotenv())
API_URL = "https://api-inference.huggingface.co/models/espnet/kan-bayashi_ljspeech_vits"
HF_API_KEY = os.getenv("HF_API_KEY")
headers = {"Authorization": f"Bearer {HF_API_KEY}"}
def query(payload):
response = requests.post(API_URL, headers=headers, json=payload)
return response
def tts(text, index):
audio = query({
"inputs": text,
})
with open(f"resources/more/.speech{index}.wav", "wb") as f:
f.write(audio.content)
return audio.content
```
### transcript.py
```python
import requests
from dotenv import load_dotenv, find_dotenv
import os
import llm_bridge
load_dotenv(find_dotenv())
API_URL = "https://api-inference.huggingface.co/models/openai/whisper-large-v3"
HF_API_KEY = os.getenv("HF_API_KEY")
headers = {"Authorization": f"Bearer {HF_API_KEY}"}
def query(filename):
with open(filename, "rb") as f:
data = f.read()
response = requests.post(API_URL, headers=headers, data=data)
return response.json()
def audio_to_latex(filename):
print("Sending to api...")
output = query(filename)
print(output['text'])
prompt = llm_bridge.build_prompt_latex(output["text"])
res = llm_bridge.get_response("gpt-4", prompt)
print(res)
return res
```
### renderer.py
```python
import matplotlib.pyplot as plt
from PIL import Image
from pdf2image import convert_from_path
import numpy as np
from pylatex import Document, Section, Subsection, Tabular, Math, TikZ, Axis, \
Plot, Figure, Matrix, Alignat
from pylatex.utils import italic
import os
def format_str(input_str):
result = ""
for char in input_str:
if char == '\\':
result += '\\\\'
else:
result += char
return result
def render(formula, path="resources/result.png"):
fig = plt.figure()
plt.axis("off")
#formula = format_str(formula)
plt.text(0.5, 0.5, f"${formula}$", size=50, ha="center", va="center")
pdf_path = "resources/.result.pdf"
png_path = path
plt.savefig(pdf_path, format="pdf", bbox_inches="tight", pad_inches=0.4)
plt.close(fig)
images = convert_from_path(pdf_path)
images[0].save(png_path, "PNG")
return png_path
if __name__ == "__main__":
latex_formula = r"\int_{0}^{2} 3x \, dx"
render(latex_formula)
```
### recorder.py
```python
from pvrecorder import PvRecorder
import soundfile as sf
import time
import transcript
import numpy as np
import speech_recognition as sr
def audio_transcription_thread(audio_file):
transcript.audio_to_latex(audio_file)
def start_record():
recognizer = sr.Recognizer()
for index, device in enumerate(PvRecorder.get_available_devices()):
print(f"[{index}] {device}")
#change to microphone index you want to use
recorder = PvRecorder(device_index=0, frame_length=512)
audio = []
t_start = time.time()
try:
recorder.start()
while recorder.is_recording:
frame = recorder.read()
audio.extend(frame)
# t_elapsed = time.time() - t_start
#print(f"Recording duration: {t_elapsed:.2f} seconds", end='\r')
# add real time api calls
#save temp audio file
# audio_np = np.array(audio, dtype='int16')
# with sf.SoundFile("resources/.temp.flac", 'w', samplerate=16000, channels=1, subtype="PCM_16") as f:
# f.write(audio_np)
# Multithreading for audio transcription
# thread = threading.Thread(target=audio_transcription_thread, args=("ressources/.temp.flac",))
# thread.start()
except KeyboardInterrupt:
recorder.stop()
audio_np = np.array(audio, dtype='int16')
with sf.SoundFile("resources/output.flac", 'w', samplerate=16000, channels=1, subtype="PCM_16") as f:
f.write(audio_np)
finally:
recorder.delete()
# transcript.audio_to_latex("resources/output.flac")
return "resources/output.flac"
if __name__ == "__main__":
start_record()
```
### steps_converter.py
```python
import renderer
import speaker
_steps = r"""Got it, let's break down the solution:
1. **Antiderivative**:
- EXP: \int 3x \, dx = \frac{3}{2}x^2 + C
- EXPL: Integrate \(3x\) with respect to \(x\) to find the antiderivative. The constant of integration \(C\) accounts for any constant shift.
2. **Evaluate at Limits**:
- EXP: \left[\frac{3}{2}x^2\right]_{0}^{2}
- EXPL: Substitute \(x = 2\) into the antiderivative and subtract the result when \(x = 0\) to find the definite integral over the interval \([0, 2]\).
3. **Substitute Upper Limit**:
- EXP: \left[\frac{3}{2}(2)^2\right] - \left[\frac{3}{2}(0)^2\right]
- EXPL: Calculate the antiderivative at the upper limit.
4. **Calculate**:
- EXP: \left[\frac{3}{2}(4)\right] - \left[0\right]
- EXPL: Simplify the expression by evaluating \((2)^2 = 4\) and \((0)^2 = 0\).
5. **Simplify Further**:
- EXP: \frac{3}{2}(4)
- EXPL: Multiply \(\frac{3}{2}\) by \(4\).
6. **Final Result**:
- EXP: 6
- EXPL: Evaluate the expression to get the final result.
This leads us to the conclusion that the definite integral \(\int_{0}^{2} 3x \, dx\) evaluates to \(6\)."""
def convert(response: str):
# response = renderer.format_str(response)
steps = {"EXP": [], "EXPL": []}
lines = response.split("\n")
for line in lines:
if "EXP:" in line:
steps["EXP"].append(line[line.index("EXP:") + len("EXP:"):].strip())
elif "EXPL:" in line:
steps["EXPL"].append(line[line.index("EXPL:") + len("EXPL:"):].strip())
return steps
def render_steps(steps_dict: dict):
i = 0
for exp in steps_dict["EXP"]:
renderer.render(exp, f"resources/more/.step_{i}.png")
speaker.tts(steps_dict["EXPL"][i], i)
print(exp)
i+=1
if __name__ == "__main__":
steps_dict = convert(_steps)
i = 0
for exp in steps_dict["EXP"]:
renderer.render(exp, f"resources/more/.step_{i}.png")
speaker.tts(steps_dict["EXPL"][i], i)
print(exp)
i+=1
```
### llm_bridge.py
```python
import openai
import os
from dotenv import load_dotenv, find_dotenv
load_dotenv(find_dotenv())
context = ""
# Use the API key from the environment variable
openai.api_key = str(os.getenv("OPENAI_API_KEY"))
def build_prompt_latex(equation):
prompt_template = f"Context:\n\
You are a math assistant. \
Your only goal is to transform a written math equation \
to a latex code expression. \
Only output the code for the math expression. \
Do not include any other latex code appart from the math expression. \n\
Input equation: {equation}\n\
Output expression:"
return prompt_template
def build_prompt_solve(equation, result):
prompt_template = f"Context:\n\
You are a math teacher. \
Your goal is to explain how to solve a given equation. \
You are given two arguments. \
EQUATION is the equation to solve, \
RESULT is the answer to this equation. \
Give all the steps that lead to this result. \
Divide each step in two parts. \n\
'EXP:' is the latex formula of the step expression and \
'EXPL:' is where you explain what you are doing. \n\
Do not write anything outside of 'EXP' and 'EXPL'\n\
EQUATION: {equation} \n\
RESULT: {result}"
return prompt_template
def build_prompt_correct(equation):
eq1 = r"\int_{0}^{2} 3x \, dx"
_eq1 = r"\int_{0}^{2} 3x"
eq2 = r"\int_{0}^{4} \frac{49 dx}{(49 - x^2)^{3/2}}"
_eq2 = r"\int_{0}^{4} \frac{49}{(49 - x^2)^{3/2}}"
prompt_template = f"Context:\n\
Your goal is to modify a latex formula to make it easier to \
understand for a programming language. \
For exemple, with integrals, you should \
remove the dx but also the coma ',' and \
any backslash that do not make sense. \
ONLY OUTPUT THE MODIFIED FORMULA. \
Do not right anything that is not \
the output formula. \
Do not write 'INPUT' or 'OUTPUT' or \
the non-modified formula. Only write the output formula\
Exemple: \
INPUT: {eq1}\
OUTPUT: {_eq1}\
INPUT: {eq2} \
OUTPUT: {_eq2} \
INPUT: {equation}"
return prompt_template
def get_response(model, message):
global context
messages=[{"role": "system",
"content": context},
{"role": "user",
"content": message}]
response = openai.chat.completions.create(
model=model,
max_tokens=1000,
temperature=0.2,
messages = messages)
# Access the generated text
generated_text = response.choices[0].message.content.strip()
context += "User: " + message + "\n" + "System : " + generated_text + "\n"
return generated_text
if __name__ == "__main__":
input_model = int(input("Chose your model: \n '1': gpt-3.5-turbo \n '2': gpt-4 \n Choice (1/2): "))
model = "gpt-4" if input_model == 2 else "gpt-3.5-turbo"
while True:
input_message = input(">>> ")
if(input_message == "exit"):
break
get_response(model, input_message)
print("")
```
[3 more indexed source files omitted to keep this export small. The full file list is in the Codebase structure section above.]
