# Project export: Verbalist.ai This document was generated by HackStack to give an AI agent context about a hackathon project. Sections are labeled with their provenance; content marked as truncated was cut to keep this document small. ## Project metadata - Hackathon: Cal Hacks 11.0 - Tagline: Verbalist.ai can automatically apply text edits verbally described by natural language, thereby allowing programmers to use their voice to speed up their development speeds. - Devpost: https://devpost.com/software/verbalism-ai - GitHub: https://github.com/sayhilel/Verbalist - Demo: https://prezi.com/view/2WRBQNSb4uMzyJzgNxsG/ - Video: https://www.youtube.com/embed/5OVrEaMtPq8?enablejsapi=1&hl=en_US&rel=0&start=&version=3&wmode=transparent - Team: 3 GitHub contributor(s) — Sahil Sinha (27 commits), Akilan Babu (8 commits), Lucien (4 commits) ## Devpost submission (written by the team) ### Inspiration Cursorless is a voice interface for manipulating text (e.g., code). We saw its potential as a bold new interface for text editing. However, it is very unintuitive and learning Cursorless amounts to learning a new language, with unnatural and complicated syntax. This was the inspiration behind Verbalist. We want to harness the power of voice (and AI) to greatly improve productivity while editing text, especially code. Most other AI products access user data. We also want to ensure data security of our product. ### What it does Verbalist is a VSCode extension that enables the use of voice to edit their code. After a user downloads and configures the extension, users can record small voice snippets describing the high-level actions they want to take on text. Then, our AI models decide the specific actions to execute in order to do the high-levels actions--all without processing the content of the file. Challenges and what we learned We learned some limitations of using large-language models on difficult, real-world tasks. For example, the LLM model we used often struggled to identify a correct, intuitive sequence of actions to perform the user's specified action. We spent a long time refining prompts; we learned that our final results were very sensitive to the quality of our prompts. We also spent a while setting up the interaction between our main extension TypeScript file and our Python file, which handled the recording and AI processing. Through this process, we learned how to set up inter-process communication and extensively using the standard libraries (e.g., input/output streams) of both Python and TypeScript. Accomplishments we're proud of Our extension allows users to use natural language to manipulate collections of lines and perform simple find-and-replace operations. T We also built on top of the VSCode text editing API to allow for higher-level operations without providing any file contents to AI. ### What's next The concepts behind this prototype can easily be extended to a fully-functional extension that adds a functionality not present in any other software today. We can implement more high-level, detailed actions for the AI to perform; for example, the ability to rename a variable, surround an expression in parentheses, or perform actions across multiple files. The voice interface can become a natural extension of the keyboard, one that allows programmers to spend less time thinking and more time doing. ## README (from the GitHub repository) # Verbalist Verbalist is a productivity & accessibility VScode extention developed as a team project for Calhacks 2024. Verbalist is designed to bring the productivity of Vim motions to VScode using speech. Users can request the extention to delete, insert or replace any text in real time using large language model and VScode API. ## Detected evidence (automated analysis) Indexed codebase: 10 recognized source files, 18 KB. - FastAPI (technology) — detected in the code - JavaScript (language) — detected in the code - OpenAI (technology) — detected in the code - Python (language) — detected in the code - TypeScript (language) — detected in the code ## Codebase structure (from repository index) ### Files (24 of 24) ``` .env .gitignore .vscode-test.mjs .vscode/extensions.json .vscode/launch.json .vscode/settings.json .vscode/tasks.json .vscodeignore backend/audioRecorder.py backend/groqCommands.py backend/IPC.py backend/LICENSE backend/requirements.txt backend/test.py eslint.config.mjs package.json README.md requirements.txt src/editor.ts src/extension.ts src/test/extension.test.ts tsconfig.json vsc-extension-quickstart.md webpack.config.js ``` ### Dependencies - backend/requirements.txt: groq@==0.11.0, numpy@==2.1.2, python-dotenv@==1.0.1, sounddevice@==0.5.1 - package.json: @types/mocha@^10.0.8, @types/node@20.x, @types/vscode@^1.94.0, @typescript-eslint/eslint-plugin@^8.7.0, @typescript-eslint/parser@^8.7.0, @vscode/test-cli@^0.0.10, @vscode/test-electron@^2.4.1, eslint@^9.11.1, ts-loader@^9.5.1, typescript@^5.6.2, webpack@^5.95.0, webpack-cli@^5.1.4 - requirements.txt: annotated-types@==0.7.0, anyio@==4.4.0, asttokens@==2.4.1, blinker@==1.4, boto3@==1.34.137, botocore@==1.34.137, Brotli@==1.1.0, BTrees@==5.2, certifi@==2024.6.2, cffi@==1.16.0, charset-normalizer@==3.3.2, click@==8.1.7, comm@==0.2.2, command-not-found@==0.3, contourpy@==1.2.1, cryptography@==3.4.8, cycler@==0.12.1, dbus-python@==1.2.18, debugpy@==1.8.2, decorator@==5.1.1, distro@==1.7.0, distro-info@==1.1+ubuntu0.2, dnspython@==2.6.1, email_validator@==2.2.0, exceptiongroup@==1.2.1, executing@==2.0.1, fastapi@==0.111.0, fastapi-cli@==0.0.4, fonttools@==4.53.1, h11@==0.14.0, httpcore@==1.0.5, httplib2@==0.20.2, httptools@==0.6.1, httpx@==0.27.0, idna@==3.7, importlib-metadata@==4.6.4, iniconfig@==2.0.0, ipykernel@==6.29.5, ipython@==8.26.0, jedi@==0.19.1, jeepney@==0.7.1, Jinja2@==3.1.4, jmespath@==1.0.1, jupyter_client@==8.6.2, jupyter_core@==5.7.2, keyring@==23.5.0, kiwisolver@==1.4.5, launchpadlib@==1.10.16, lazr.restfulclient@==0.14.4, lazr.uri@==1.0.6, markdown-it-py@==3.0.0, MarkupSafe@==2.1.5, matplotlib@==3.9.1, matplotlib-inline@==0.1.7, mdurl@==0.1.2, more-itertools@==8.10.0, mutagen@==1.47.0, mypy@==1.8.0, mypy-extensions@==1.0.0, nest-asyncio@==1.6.0, netifaces@==0.11.0, numpy@==2.0.0, oauthlib@==3.2.0, openai@==1.35.10, orjson@==3.10.6, packaging@==24.1, pandas@==2.2.2, parso@==0.8.4, persistent@==5.1, pexpect@==4.9.0, pillow@==10.4.0, platformdirs@==4.2.2, pluggy@==1.5.0, prompt_toolkit@==3.0.47, psutil@==6.0.0, ptyprocess@==0.7.0, pure-eval@==0.2.2, pycparser@==2.21, pycryptodomex@==3.20.0, pydantic@==2.8.0, pydantic_core@==2.20.0, Pygments@==2.18.0, PyGObject@==3.42.1, PyJWT@==2.3.0, pyparsing@==2.4.7, pyplate-hte@==0.4.7, pytest@==8.3.2, python-apt@==2.4.0+ubuntu4, python-dateutil@==2.9.0.post0, python-dotenv@==1.0.1, python-multipart@==0.0.9, pytube@==15.0.0, pytz@==2024.1, PyYAML@==5.4.1, pyzmq@==26.0.3, replicate@==0.27.0, requests@==2.32.3, rich@==13.7.1, s3transfer@==0.10.2, SecretStorage@==3.3.1, shellingham@==1.5.4, six@==1.16.0, sniffio@==1.3.1, sounddevice@==0.5.1, stack-data@==0.6.3, starlette@==0.37.2, systemd-python@==234, tabulate@==0.9.0, tomli@==2.0.1, tornado@==6.4.1, tqdm@==4.66.4, traitlets@==5.14.3, typer@==0.12.3, typing_extensions@==4.12.2, tzdata@==2024.1, ubuntu-pro-client@==8001, ufw@==0.36.1, ujson@==5.10.0, unattended-upgrades@==0.1, urllib3@==2.2.2, uvicorn@==0.30.1, uvloop@==0.19.0, wadllib@==1.3.6, watchfiles@==0.22.0, wcwidth@==0.2.13, websockets@==12.0, youtube-dl@==2021.12.17, yt-dlp@==2024.7.9, zipp@==1.0.0, zope.interface@==6.1 ### Recent commits (newest first) - Update README.md - Merge branch 'prod' - polish notifications - added toggleRecording command - updated package json for toggleRecording - Woops API Key - v1.2 - Revert "v1.1" - v1.1 - Init v.10 - Parsing changes - Updated cursorposition prompt v4 - Updated cursorposition prompt v3 - Updated cursorposition prompt v2 - Updated cursorposition prompt - Updated edit DOT Builder - attempt to run command using new api - Updated gronk - V0.1 - V0.1 ## Key source files (fetched from GitHub, selected and truncated for size) ### vsc-extension-quickstart.md ```markdown # Welcome to your VS Code Extension ## What's in the folder * This folder contains all of the files necessary for your extension. * `package.json` - this is the manifest file in which you declare your extension and command. * The sample plugin registers a command and defines its title and command name. With this information VS Code can show the command in the command palette. It doesn’t yet need to load the plugin. * `src/extension.ts` - this is the main file where you will provide the implementation of your command. * The file exports one function, `activate`, which is called the very first time your extension is activated (in this case by executing the command). Inside the `activate` function we call `registerCommand`. * We pass the function containing the implementation of the command as the second parameter to `registerCommand`. ## Setup * install the recommended extensions (amodio.tsl-problem-matcher, ms-vscode.extension-test-runner, and dbaeumer.vscode-eslint) ## Get up and running straight away * Press `F5` to open a new window with your extension loaded. * Run your command from the command palette by pressing (`Ctrl+Shift+P` or `Cmd+Shift+P` on Mac) and typing `Hello World`. * Set breakpoints in your code inside `src/extension.ts` to debug your extension. * Find output from your extension in the debug console. ## Make changes * You can relaunch the extension from the debug toolbar after changing code in `src/extension.ts`. * You can also reload (`Ctrl+R` or `Cmd+R` on Mac) the VS Code window with your extension to load your changes. ## Explore the API * You can open the full set of our API when you open the file `node_modules/@types/vscode/index.d.ts`. ## Run tests * Install the [Extension Test Runner](https://marketplace.visualstudio.com/items?itemName=ms-vscode.extension-test-runner) * Run the "watch" task via the **Tasks: Run Task** command. Make sure this is running, or tests might not be discovered. * Open the Testing view from the activity bar and click the Run Test" button, or use the hotkey `Ctrl/Cmd + ; A` * See the output of the test result in the Test Results view. * Make changes to `src/test/extension.test.ts` or create new test files inside the `test` folder. * The provided test runner will only consider files matching the name pattern `**.test.ts`. * You can create folders inside the `test` folder to structure your tests any way you want. ## Go further * Reduce the extension size and improve the startup time by [bundling your extension](https://code.visualstudio.com/api/working-with-extensions/bundling-extension). * [Publish your extension](https://code.visualstudio.com/api/working-with-extensions/publishing-extension) on the VS Code extension marketplace. * Automate builds by setting up [Continuous Integration](https://code.visualstudio.com/api/working-with-extensions/continuous-integration). ``` ### package.json ``` { "name": "verbalist", "displayName": "verbalist", "description": "Voice activated vim, but better", "version": "0.0.1", "engines": { "vscode": "^1.94.0" }, "categories": [ "Other" ], "activationEvents": [], "main": "./dist/extension.js", "contributes": { "commands": [ { "command": "verbalist.captureAudio", "title": "Start Audio Recording" }, { "command": "verbalist.stopAudio", "title": "Audio Stop" }, { "command": "verbalist.runCommand", "title": "Run Command" }, { "command": "verbalist.RUN_COMMAND", "title": "RUN COMMAND" }, { "command": "verbalist.toggleRecording", "title": "Toggle the recording", "key": "shift+z", "mac": "shift+z", "linux": "shift+z", "win": "shift+z" } ] }, "scripts": { "vscode:prepublish": "npm run package", "compile": "webpack", "watch": "webpack --watch", "package": "webpack --mode production --devtool hidden-source-map", "compile-tests": "tsc -p . --outDir out", "watch-tests": "tsc -p . -w --outDir out", "pretest": "npm run compile-tests && npm run compile && npm run lint", "lint": "eslint src", "test": "vscode-test" }, "devDependencies": { "@types/vscode": "^1.94.0", "@types/mocha": "^10.0.8", "@types/node": "20.x", "@typescript-eslint/eslint-plugin": "^8.7.0", "@typescript-eslint/parser": "^8.7.0", "eslint": "^9.11.1", "typescript": "^5.6.2", "ts-loader": "^9.5.1", "webpack": "^5.95.0", "webpack-cli": "^5.1.4", "@vscode/test-cli": "^0.0.10", "@vscode/test-electron": "^2.4.1" } } ``` ### requirements.txt ``` annotated-types==0.7.0 anyio==4.4.0 asttokens==2.4.1 blinker==1.4 boto3==1.34.137 botocore==1.34.137 Brotli==1.1.0 BTrees==5.2 certifi==2024.6.2 cffi==1.16.0 charset-normalizer==3.3.2 click==8.1.7 comm==0.2.2 command-not-found==0.3 contourpy==1.2.1 cryptography==3.4.8 cycler==0.12.1 dbus-python==1.2.18 debugpy==1.8.2 decorator==5.1.1 distro==1.7.0 distro-info==1.1+ubuntu0.2 dnspython==2.6.1 email_validator==2.2.0 exceptiongroup==1.2.1 executing==2.0.1 fastapi==0.111.0 fastapi-cli==0.0.4 fonttools==4.53.1 h11==0.14.0 httpcore==1.0.5 httplib2==0.20.2 httptools==0.6.1 httpx==0.27.0 idna==3.7 importlib-metadata==4.6.4 iniconfig==2.0.0 ipykernel==6.29.5 ipython==8.26.0 jedi==0.19.1 jeepney==0.7.1 Jinja2==3.1.4 jmespath==1.0.1 jupyter_client==8.6.2 jupyter_core==5.7.2 keyring==23.5.0 kiwisolver==1.4.5 launchpadlib==1.10.16 lazr.restfulclient==0.14.4 lazr.uri==1.0.6 markdown-it-py==3.0.0 MarkupSafe==2.1.5 matplotlib==3.9.1 matplotlib-inline==0.1.7 mdurl==0.1.2 more-itertools==8.10.0 mutagen==1.47.0 mypy==1.8.0 mypy-extensions==1.0.0 nest-asyncio==1.6.0 netifaces==0.11.0 numpy==2.0.0 oauthlib==3.2.0 openai==1.35.10 orjson==3.10.6 packaging==24.1 pandas==2.2.2 parso==0.8.4 persistent==5.1 pexpect==4.9.0 pillow==10.4.0 platformdirs==4.2.2 pluggy==1.5.0 prompt_toolkit==3.0.47 psutil==6.0.0 ptyprocess==0.7.0 pure-eval==0.2.2 pycparser==2.21 pycryptodomex==3.20.0 pydantic==2.8.0 pydantic_core==2.20.0 Pygments==2.18.0 PyGObject==3.42.1 PyJWT==2.3.0 pyparsing==2.4.7 pyplate-hte==0.4.7 pytest==8.3.2 python-apt==2.4.0+ubuntu4 python-dateutil==2.9.0.post0 python-dotenv==1.0.1 python-multipart==0.0.9 pytube==15.0.0 pytz==2024.1 PyYAML==5.4.1 pyzmq==26.0.3 replicate==0.27.0 requests==2.32.3 rich==13.7.1 s3transfer==0.10.2 SecretStorage==3.3.1 shellingham==1.5.4 six==1.16.0 sniffio==1.3.1 sounddevice==0.5.1 stack-data==0.6.3 starlette==0.37.2 systemd-python==234 tabulate==0.9.0 tomli==2.0.1 tornado==6.4.1 tqdm==4.66.4 traitlets==5.14.3 typer==0.12.3 typing_extensions==4.12.2 tzdata==2024.1 ubuntu-pro-client==8001 ufw==0.36.1 ujson==5.10.0 unattended-upgrades==0.1 urllib3==2.2.2 uvicorn==0.30.1 uvloop==0.19.0 wadllib==1.3.6 watchfiles==0.22.0 wcwidth==0.2.13 websockets==12.0 youtube-dl==2021.12.17 yt-dlp==2024.7.9 zipp==1.0.0 zope.interface==6.1 ``` ### backend/requirements.txt ``` groq==0.11.0 numpy==2.1.2 python-dotenv==1.0.1 sounddevice==0.5.1 ``` ### webpack.config.js ```javascript //@ts-check 'use strict'; const path = require('path'); //@ts-check /** @typedef {import('webpack').Configuration} WebpackConfig **/ /** @type WebpackConfig */ const extensionConfig = { target: 'node', // VS Code extensions run in a Node.js-context 📖 -> https://webpack.js.org/configuration/node/ mode: 'none', // this leaves the source code as close as possible to the original (when packaging we set this to 'production') entry: './src/extension.ts', // the entry point of this extension, 📖 -> https://webpack.js.org/configuration/entry-context/ output: { // the bundle is stored in the 'dist' folder (check package.json), 📖 -> https://webpack.js.org/configuration/output/ path: path.resolve(__dirname, 'dist'), filename: 'extension.js', libraryTarget: 'commonjs2' }, externals: { vscode: 'commonjs vscode' // the vscode-module is created on-the-fly and must be excluded. Add other modules that cannot be webpack'ed, 📖 -> https://webpack.js.org/configuration/externals/ // modules added here also need to be added in the .vscodeignore file }, resolve: { // support reading TypeScript and JavaScript files, 📖 -> https://github.com/TypeStrong/ts-loader extensions: ['.ts', '.js'] }, module: { rules: [ { test: /\.ts$/, exclude: /node_modules/, use: [ { loader: 'ts-loader' } ] } ] }, devtool: 'nosources-source-map', infrastructureLogging: { level: "log", // enables logging required for problem matchers }, }; module.exports = [ extensionConfig ]; ``` ### backend/test.py ```python import audioRecorder import groqCommands import os audio_file = os.path.dirname(os.path.abspath(__file__)) + "/Audio/tmp.wav" def recordTest(): recorder = audioRecorder.AudioRecorder(output_filename=audio_file) recorder.start_recording() import time time.sleep(5) recorder.stop_recording() groqCommands.get_command(audio_file) recordTest() ``` ### backend/IPC.py ```python import sys import audioRecorder import groqCommands import os ofile = os.getcwd() + "/audio.wav" recorder = audioRecorder.AudioRecorder(output_filename=ofile) def listen_for_commands(): try: while True: # Read a line from stdin and strip any whitespace command = sys.stdin.readline().strip() #eprint(f"Received command: {command}") # Respond to the "start" command if command == "start": if not recorder.is_recording: #print("Received 'start' command.") recorder.start_recording() # Delete file continue else: #print("Already recording.") pass # Respond to the "stop" command if command == "stop": if recorder.is_recording: #print("Received 'stop' command.") recorder.stop_recording() print("COMMAND_START") print(groqCommands.get_command(ofile)) print("COMMAND_END") break except KeyboardInterrupt: print("\nProgram interrupted. Exiting...") sys.exit(0) listen_for_commands() ``` ### backend/audioRecorder.py ```python import sounddevice as sd import numpy as np import wave class AudioRecorder: def __init__(self, samplerate=16000, channels=1, output_filename="output.wav"): self.samplerate = samplerate self.channels = channels self.output_filename = output_filename self.frames = [] self.is_recording = False self.stream = None def _callback(self, indata, frames, time, status): if status: #print(status, flush=True) pass if self.is_recording: self.frames.append(indata.copy()) # save the audio record def start_recording(self): if not self.is_recording: #print("Recording started...") self.frames = [] # 清空之前的录音数据 self.is_recording = True self.stream = sd.InputStream(samplerate=self.samplerate, channels=self.channels, callback=self._callback) self.stream.start() # start recording def stop_recording(self): if self.is_recording: #print("Stopping recording...") self.is_recording = False if self.stream: self.stream.stop() # 停止录音 self.stream.close() self._save_audio() def _save_audio(self): # 将录制的音频保存为 WAV 文件 audio_data = np.concatenate(self.frames, axis=0) audio_data = (audio_data * 32767).astype(np.int16) # change float32 to 16-bit PCM format with wave.open(self.output_filename, 'wb') as wf: wf.setnchannels(self.channels) wf.setsampwidth(2) # 2 bytes per sample (16 bits) wf.setframerate(self.samplerate) wf.writeframes(audio_data.tobytes()) #print(f"Recording saved to {self.output_filename}") ``` ### src/editor.ts ```typescript import * as vscode from "vscode"; export function searchAndEditWithRegex( regex: RegExp, // Accepts a regular expression editCallback: (editBuilder: vscode.TextEditorEdit, range: vscode.Range) => void // Callback for handling edit operations ) { // Get the active text editor const editor = vscode.window.activeTextEditor; if (editor) { const document = editor.document; const text = document.getText(); // Get the entire text of the document let match; const rangesToEdit: vscode.Range[] = []; // Iterate over all matches of the regular expression while ((match = regex.exec(text)) !== null) { // Get the start and end positions of the match const startPos = document.positionAt(match.index); const endPos = document.positionAt(match.index + match[0].length); // Create the range to be edited (deleted or replaced) rangesToEdit.push(new vscode.Range(startPos, endPos)); } // Perform edit operations using the provided callback editor.edit(editBuilder => { // Process the ranges in reverse order to avoid affecting the subsequent ranges for (let i = rangesToEdit.length - 1; i >= 0; i--) { editCallback(editBuilder, rangesToEdit[i]); // Invoke the callback for each range } }).then(success => { if (success) { console.log(`All matches for regex ${regex} have been processed.`); } else { console.error('Failed to process the matches.'); } }); } else { console.log('No active editor found.'); } } //function to run vs code commands export default function runEditorAction(command: string) { // Get the active text editor const editor = vscode.window.activeTextEditor; if (!editor) { vscode.window.showInformationMessage("No active editor found."); return; } const editCommandString = "editor.edit((editBuilder) => {\n" + command + "});"; try { eval(editCommandString); console.log("passed eval"); } catch (e) { console.error(e); } } ``` ### src/extension.ts ```typescript import { spawn } from "child_process"; import * as path from "path"; import * as vscode from "vscode"; import runEditorAction from "./editor"; let recordingProcess: any = null; export function activate(context: vscode.ExtensionContext) { const extensionPath = context.extensionPath; const pythonPath = path.join(extensionPath, "venv", "bin", "python3"); const filePath = path.join(extensionPath, "backend", "IPC.py"); //---------------------------------------------- // Buffer to accumulate incoming data from stdin let buffer = ""; // Function to process command output from stdin function processCommandInput() { const commandStart = buffer.indexOf("COMMAND_START"); const commandEnd = buffer.indexOf("COMMAND_END"); if (commandStart !== -1 && commandEnd !== -1) { // Extract command between the delimiters const commandString = buffer .slice(commandStart + "COMMAND_START".length, commandEnd) .trim() .replace("\n", ""); // Call runEditorAction with the extracted command if (commandString) { runEditorAction(commandString); buffer = ""; } // Remove the processed command from the buffer buffer = buffer.slice(commandEnd + "COMMAND_END".length); } } //----------------------------------------------- context.subscriptions.push( vscode.commands.registerCommand("verbalist.captureAudio", () => { const cmd = pythonPath; const args = ["-u", filePath]; const options = { cwd: path.join(extensionPath, "backend"), }; recordingProcess = spawn(cmd, args, options); recordingProcess.stdout.on("data", (data: any) => { console.log(`${data}`); buffer += data; processCommandInput(); }); recordingProcess.stderr.on("data", (data: any) => { console.log(`STDERR:${data}`); }); // vscode.window.showInformationMessage("spawned recording process"); recordingProcess.stdin.write("start" + "\n"); vscode.window.showInformationMessage("Recording started"); return recordingProcess; }) ); context.subscriptions.push( vscode.commands.registerCommand("verbalist.stopAudio", () => { if (recordingProcess) { recordingProcess.stdin.write("stop\n"); vscode.window.showInformationMessage("Stopped recording"); recordingProcess.on("exit", () => { recordingProcess = null; console.log("EXITED PROC"); vscode.window.showInformationMessage("Action executed"); }); } else { vscode.window.showInformationMessage("no recording process to kill"); } }) ); context.subscriptions.push( vscode.commands.registerCommand("verbalist.runCommand", () => {}) ); context.subscriptions.push( vscode.commands.registerCommand("verbalist.toggleRecording", () => { if (recordingProcess) { vscode.commands.executeCommand("verbalist.stopAudio"); } else { vscode.commands.executeCommand("verbalist.captureAudio"); } }) ); } export function deactivate() {} ``` [2 more indexed source files omitted to keep this export small. The full file list is in the Codebase structure section above.]