# Project export: SpeechScaffold

This document was generated by HackStack to give an AI agent context about a hackathon project. Sections are labeled with their provenance; content marked as truncated was cut to keep this document small.

## Project metadata

- Hackathon: TreeHacks 2024
- Tagline: Build React components at the speed of sound
- Devpost: https://devpost.com/software/speechscaffold
- GitHub: https://github.com/SambhavG/voice-to-website
- Video: https://www.youtube.com/embed/TbYreuNG9tc?enablejsapi=1&hl=en_US&rel=0&start=&version=3&wmode=transparent
- Team: 1 GitHub contributor(s) — SambhavG (10 commits)

## Devpost submission (written by the team)

### Inspiration

This project was born out of the desire to combine speech to text AI with local LLMs to create a product that improves the developer experience by a large factor.

### What it does

SpeechScaffold is a tool which developers run alongside the IDE. The tool allows a developer to rapidly build and iterate on the style and functionality of their React components simply by describing the changes they want out loud. It integrates directly with the IDE, so the developer can instantly edit and use code they generate. How I built it I used React for the frontend, since this is a very popular web framework and also most likely to draw good results from Codellama and/or Mistral. The AI agent, written in Python, is a heavily modified version of an AI assistant link which, with the modifications, is able to fetch the current state of the code, iterate it based on the developer's voice commands, and output a new and improved version of the code each time.

### Challenges we ran into

I originally used Svelte for the framework, since it is more modern than React and has much simpler code in many cases. However, the LLMs do not output Svelte at the same quality, likely due to it being a newer framework. Thus, I had to retool the entire pipeline to React.

### Accomplishments we're proud of

I am proud of the fact that the tool is able to achieve fairly consistent and accurate results, which means it is several factors faster than writing the code for the same components by hand.

### What we learned

I learned a lot about implementing OpenAI's whisper model, inferencing with Ollama and streaming its output, parsing and cleaning LLM output data to fit rigorous standards for code, and prompt engineering for code iteration.

### What's next

The next improvements include improving the list of special commands by adding memory with undo, multiple generations with choice of best, and improving the speed of generation via external APIs rather than self hosting the LLM.

## README (from the GitHub repository)

# SpeechScaffold
SpeechScaffold is a tool which developers run alongside the IDE. The tool allows a developer to rapidly build and iterate on the style and functionality of their React components simply by describing the changes they want out loud. It integrates directly with the IDE, so the developer can instantly edit and use code they generate.

## Installation
You'll need to run Ollama (for local LLM; I used codellama) and you'll need to download a whisper model to /assistant/whisper (I used medium.en.pt). Then, simply run the Ollama server (should be automatic), the assistant (with python3 assistant.py in the folder /assistant), and the client (with npm run dev in the folder /autowebsite).

## Detected evidence (automated analysis)

Indexed codebase: 25 recognized source files, 33 KB.
- CSS (language) — detected in the code
- HTML (language) — detected in the code
- JavaScript (language) — detected in the code
- Next.js (technology) — detected in the code
- OpenAI (technology) — detected in the code
- Python (language) — detected in the code
- PyTorch (technology) — detected in the code
- React (technology) — detected in the code
- Svelte (technology) — detected in the code
- Tailwind CSS (technology) — detected in the code
- TypeScript (language) — detected in the code
- Mistral AI (technology) — claimed on Devpost, not found in the code
- Ollama (technology) — claimed on Devpost, not found in the code

## Codebase structure (from repository index)

### Files (42 of 42)

```
.gitignore
.vscode/extensions.json
assistant/.gitignore
assistant/assistant.py
assistant/assistant.yaml
assistant/LICENSE
assistant/prompt.txt
assistant/README.md
assistant/requirements.txt
assistant/syntaxCorrectionPrompt.txt
autowebsite/.eslintrc.json
autowebsite/.gitignore
autowebsite/jsconfig.json
autowebsite/next.config.mjs
autowebsite/package.json
autowebsite/postcss.config.js
autowebsite/public/Worktable.txt
autowebsite/README.md
autowebsite/src/app/globals.css
autowebsite/src/app/layout.js
autowebsite/src/app/lib/CodeDisplay.jsx
autowebsite/src/app/lib/commands.txt
autowebsite/src/app/lib/user-prompt.js
autowebsite/src/app/lib/Worktable.jsx
autowebsite/src/app/page.css
autowebsite/src/app/page.jsx
autowebsite/tailwind.config.js
index.html
jsconfig.json
old_src/app.css
old_src/App.svelte
old_src/lib/CodeDisplay.svelte
old_src/lib/forceRefresh.svelte
old_src/lib/user-prompt.js
old_src/lib/Worktable.svelte
old_src/lib/Worktable.txt
old_src/main.js
old_src/vite-env.d.ts
package.json
README.md
svelte.config.js
vite.config.js
```

### Dependencies

- assistant/requirements.txt: blobfile@==2.1.1, openai@==1.7.0, openai-whisper@@ git+https://github.com/openai/whisper.git@ba3f3cd54b0e5b8ce1ab3de13e32122d0d5f98ab, PyAudio@==0.2.14, pygame@==2.5.2, pyObjC@==9.0.1, pyttsx3@==2.90, PyYAML@==6.0.1, soundfile@==0.12.1, torch@==2.1.2, torchaudio@==2.1.2, torchvision@==0.16.2, Wave@==0.0.2
- autowebsite/package.json: autoprefixer@^10.0.1, eslint@^8, eslint-config-next@14.1.0, next@14.1.0, postcss@^8, react@^18, react-dom@^18, react-syntax-highlighter@^15.5.0, tailwind-styled-components@^2.2.0, tailwindbase@^1.0.0, tailwindcss@^3.3.0
- package.json: @sveltejs/vite-plugin-svelte@^3.0.2, highlight.js@^11.9.0, prism-svelte@^0.5.0, prismjs@^1.29.0, svelte@^4.2.10, svelte-highlight@^7.6.0, vite@^5.1.0

### Recent commits (newest first)

- Update readme
- Final code push
- Adapt to react
- Add react
- Syntax correction agent
- Update CodeDisplay and add force refreshing
- Add code display
- Update assistant.py
- Add ollama-voice-mac and remove extraneous files
- first commit

## Key source files (fetched from GitHub, selected and truncated for size)

### package.json

```
{
  "name": "autowebsite",
  "private": true,
  "version": "0.0.0",
  "type": "module",
  "scripts": {
    "dev": "vite",
    "build": "vite build",
    "preview": "vite preview"
  },
  "devDependencies": {
    "@sveltejs/vite-plugin-svelte": "^3.0.2",
    "svelte": "^4.2.10",
    "svelte-highlight": "^7.6.0",
    "vite": "^5.1.0"
  },
  "dependencies": {
    "highlight.js": "^11.9.0",
    "prism-svelte": "^0.5.0",
    "prismjs": "^1.29.0"
  }
}

```

### assistant/requirements.txt

```
torch==2.1.2
torchvision==0.16.2
torchaudio==2.1.2
pyttsx3==2.90
blobfile==2.1.1
openai==1.7.0
Wave==0.0.2
PyAudio==0.2.14
PyYAML==6.0.1
pygame==2.5.2
soundfile==0.12.1
pyObjC==9.0.1
openai-whisper @ git+https://github.com/openai/whisper.git@ba3f3cd54b0e5b8ce1ab3de13e32122d0d5f98ab
```

### autowebsite/package.json

```
{
  "name": "autowebsite",
  "version": "0.1.0",
  "private": true,
  "scripts": {
    "dev": "next dev",
    "build": "next build",
    "start": "next start",
    "lint": "next lint"
  },
  "dependencies": {
    "next": "14.1.0",
    "react": "^18",
    "react-dom": "^18",
    "react-syntax-highlighter": "^15.5.0",
    "tailwind-styled-components": "^2.2.0",
    "tailwindbase": "^1.0.0"
  },
  "devDependencies": {
    "autoprefixer": "^10.0.1",
    "eslint": "^8",
    "eslint-config-next": "14.1.0",
    "postcss": "^8",
    "tailwindcss": "^3.3.0"
  }
}

```

### old_src/main.js

```javascript
import './app.css'
import App from './App.svelte'

const app = new App({
  target: document.getElementById('app'),
})

export default app

```

### autowebsite/src/app/layout.js

```javascript
import { Inter } from "next/font/google";
import "./globals.css";

const inter = Inter({ subsets: ["latin"] });

export const metadata = {
  title: "SpeechScaffold",
  description: "Build websites with your voice",
};

export default function RootLayout({ children }) {
  return (
    <html lang="en">
      <body className={inter.className}>{children}</body>
    </html>
  );
}

```

### autowebsite/src/app/page.jsx

```javascript
'use client';

import React, { useState, useEffect } from "react";
import Worktable from "./lib/Worktable";
import userPrompt from "./lib/user-prompt";
import CodeDisplay from "./lib/CodeDisplay";
import "./page.css";

const App = () => {
  const [code, setCode] = useState("console.log('Hello world')");
  const [error, setError] = useState(false);

  useEffect(() => {
    fetch("Worktable.txt")
      .then((res) => res.text())
      .then((text) => setCode(text));
  }, []);

  return (
    <main>
      <div className="header">
        <div className="user-prompt">Prompt: {userPrompt}</div>
      </div>
      
      <div className="contentPanels">
        <div className="panel">
          <div className="panelHeader">Code</div>
          <div className="code">
            <CodeDisplay code={code} />
          </div>
        </div>
        <div className="panel">
          <div className="panelHeader">Output</div>
          <div className="component">
            <ErrorBoundary>
              {!error && <Worktable onError={() => setError(true)} />}
              {error && <div class="error-container">
                <div class="error-text">Error in generation</div>
              </div>}
            </ErrorBoundary>
          </div>
        </div>
      </div>
      <div className="fixed bottom-0 right-0 p-2 bg-gray-800 text-white text-sm rounded-tl-md text-2xl">
        <div>Keyphrases:</div>
          <ul>
            <li>Delete all</li>
            <li>Fix error</li>
          </ul>
      </div>
    </main>
  );
};

class ErrorBoundary extends React.Component {
  constructor(props) {
    super(props);
    this.state = { hasError: false };
  }

  static getDerivedStateFromError(error) {
    return { hasError: true };
  }

  componentDidCatch(error, errorInfo) {
    console.error("Error caught:", error, errorInfo);
  }

  render() {
    if (this.state.hasError) {
      return <div class="w-full h-full flex justify-center items-center">
      <div class="text-4xl">Error in generation ☹️</div>
    </div>;
    }
    return this.props.children;
  }
}

export default App;

```

### vite.config.js

```javascript
import { defineConfig } from 'vite'
import { svelte } from '@sveltejs/vite-plugin-svelte'

// https://vitejs.dev/config/
export default defineConfig({
  plugins: [svelte()],
})

```

### svelte.config.js

```javascript
import { vitePreprocess } from '@sveltejs/vite-plugin-svelte'

export default {
  // Consult https://svelte.dev/docs#compile-time-svelte-preprocess
  // for more information about preprocessors
  preprocess: vitePreprocess(),
}

```

### index.html

```html
<!doctype html>
<html lang="en">
  <head>
    <meta charset="UTF-8" />
    <link rel="icon" type="image/svg+xml" href="/vite.svg" />
    <meta name="viewport" content="width=device-width, initial-scale=1.0" />
    <title>Vite + Svelte</title>
  </head>
  <body>
    <div id="app"></div>
    <script type="module" src="/src/main.js"></script>
  </body>
</html>

```

### old_src/vite-env.d.ts

```typescript
/// <reference types="svelte" />
/// <reference types="vite/client" />

```

[15 more indexed source files omitted to keep this export small. The full file list is in the Codebase structure section above.]