HackStack

Project Info

SpeechScaffold

Devpost

Inspiration

This project was born out of the desire to combine speech to text AI with local LLMs to create a product that improves the developer experience by a large factor.

What it does

SpeechScaffold is a tool which developers run alongside the IDE. The tool allows a developer to rapidly build and iterate on the style and functionality of their React components simply by describing the changes they want out loud. It integrates directly with the IDE, so the developer can instantly edit and use code they generate. How I built it I used React for the frontend, since this is a very popular web framework and also most likely to draw good results from Codellama and/or Mistral. The AI agent, written in Python, is a heavily modified version of an AI assistant link which, with the modifications, is able to fetch the current state of the code, iterate it based on the developer's voice commands, and output a new and improved version of the code each time.

Challenges we ran into

I originally used Svelte for the framework, since it is more modern than React and has much simpler code in many cases. However, the LLMs do not output Svelte at the same quality, likely due to it being a newer framework. Thus, I had to retool the entire pipeline to React.

Accomplishments we're proud of

I am proud of the fact that the tool is able to achieve fairly consistent and accurate results, which means it is several factors faster than writing the code for the same components by hand.

What we learned

I learned a lot about implementing OpenAI's whisper model, inferencing with Ollama and streaming its output, parsing and cleaning LLM output data to fit rigorous standards for code, and prompt engineering for code iteration.

What's next

The next improvements include improving the list of special commands by adding memory with undo, multiple generations with choice of best, and improving the speed of generation via external APIs rather than self hosting the LLM.

Analysis

Compare with all teams

View

Metric

Figures cover GitHub contributors during the hackathon window. A co-authored commit counts in full for each author, so per-member totals add up to more than the whole-team figures.

Technology

Found in codeClaimed only
  • CSSIn code
  • HTMLIn code
  • JavaScriptIn code
  • Next.jsIn code
  • OpenAIIn code
  • PythonIn code
  • PyTorchIn code
  • ReactIn code
  • SvelteIn code
  • Tailwind CSSIn code
  • TypeScriptIn code
  • Mistral AIClaimed
  • OllamaClaimed

11 of 13 appear in the indexed code. 2 claimed on Devpost could not be matched to code, which may simply mean the tool leaves no trace in the repository.

AI coding agents

No AI coding agent signals were found in this repository.

Detected from committed agent config files and commit authorship. Absence of a signal is not proof an agent was unused.

Codebase size

Source size

33 KB

Source files

25

Counts recognized source files only; vendored directories, binaries and lockfiles are excluded, so this is smaller than the repository on disk.

0 stars