# Project export: Zero This document was generated by HackStack to give an AI agent context about a hackathon project. Sections are labeled with their provenance; content marked as truncated was cut to keep this document small. ## Project metadata - Hackathon: Cal Hacks 10.0 - Tagline: Redefining Human-Computer Interaction - Devpost: https://devpost.com/software/zero-913v4q - GitHub: https://github.com/utkarshdubey/browser-extension - Team: 1 GitHub contributor(s) — Utkarsh Dubey (3 commits) ## Devpost submission (written by the team) ### Inspiration Inspired by Sci-fi, we imagine humans to be one with technology. With zero restrictions, zero accessibility barriers. We see this as the foundation for the next generation of human computer interfaces coming in the intelligence era. Building accessibility first, we try to echo what BCIs are meant for. All the utilities, at some point, in the future would be abstracted away, we will only have screens for entertainment. ### What it does We built a copilot which becomes one with you, interact with the internet seamlessly with voice commands in any language, having zero barriers to anything. Order a coffee, Write an email, absolutely anything that you can think of. ### How we built it We think of it as using LLMs as an assistant, for each task that the user prompts, we parse the Document Object Model for each page we land on, query the LLM to figure which action to take and then we take the action. ### Challenges we ran into One of the biggest challenges we ran into was narrowing the vision down, building it niche first and then scaling from there. Technically another challenge, optimizing the DOM to be smaller as to not hit the token limits on the LLMs, Building some usecases. ### Accomplishments we're proud of This project was inspired by another model architecture called Large Action Models, a Model that operates on Actions. Us managing to create the same use with large language models, Using Large Language models for the same use as Large Action models, without the LAM bottlenecks was one achievement. ### What we learned We learned about accessibility and how computers are still not acessible as much as we'd think. ### What's next We want to ultimately build a really powerful large action model that serves as an operating system for the next generation of Human Machine Interfaces, this project was a proof of concept for what can be possible. In the future, we imagine this to be used by companies like wispr as the OS for their BCI and we also imagine this to be the accessibility first ubiquitous way to interact with computers of the future. ## README (from the GitHub repository) No README available. ## Detected evidence (automated analysis) Indexed codebase: 64 recognized source files, 473 KB. - CSS (language) — detected in the code - HTML (language) — detected in the code - JavaScript (language) — detected in the code - OpenAI (technology) — detected in the code - React (technology) — detected in the code - TypeScript (language) — detected in the code ## Codebase structure (from repository index) ### Files (73 of 73) ``` .babelrc .eslintrc .gitignore .prettierrc .tool-versions .vscode/settings.json jest.config.ts package.json src/common/App.tsx src/common/AutosizeTextarea.tsx src/common/CopyButton.tsx src/common/ModelDropdown.tsx src/common/OptionsDropdown.tsx src/common/RunTaskButton.tsx src/common/SetAPIKey.tsx src/common/TaskHistory.tsx src/common/TaskStatus.tsx src/common/TaskUI.tsx src/common/TokenCount.tsx src/constants.ts src/globals.d.ts src/helpers/availableActions.ts src/helpers/chromeDebugger.ts src/helpers/countTokens.ts src/helpers/determineNextAction.ts src/helpers/disableExtensions.ts src/helpers/domActions.ts src/helpers/pageRPC.ts src/helpers/parseResponse.test.ts src/helpers/parseResponse.ts src/helpers/runtimeFunctionStrings.ts src/helpers/shrinkHTML/tagsSelfClose.ts src/helpers/shrinkHTML/templatize.test.ts src/helpers/shrinkHTML/templatize.ts src/helpers/simplifyDom.ts src/helpers/utils.ts src/manifest.json src/pages/Background/index.js src/pages/Content/content.styles.css src/pages/Content/copyToClipboard.ts src/pages/Content/getAnnotatedDOM.ts src/pages/Content/index.ts src/pages/Content/ripple.ts src/pages/Devtools/index.html src/pages/Devtools/index.js src/pages/Newtab/index.css src/pages/Newtab/index.html src/pages/Newtab/index.jsx src/pages/Newtab/Newtab.css src/pages/Newtab/Newtab.jsx src/pages/Newtab/Newtab.scss src/pages/Options/index.css src/pages/Options/index.html src/pages/Options/index.jsx src/pages/Options/Options.css src/pages/Options/Options.tsx src/pages/Panel/index.css src/pages/Panel/index.html src/pages/Panel/index.jsx src/pages/Panel/Panel.tsx src/pages/Popup/index.css src/pages/Popup/index.html src/pages/Popup/index.jsx src/pages/Popup/Popup.tsx src/state/currentTask.ts src/state/settings.ts src/state/store.ts src/state/ui.ts tsconfig.json utils/build.js utils/env.js utils/webserver.js webpack.config.js ``` ### Dependencies - package.json: @azure/openai@1.0.0-beta.7, @babel/core@^7.20.12, @babel/plugin-proposal-class-properties@^7.18.6, @babel/preset-env@^7.20.2, @babel/preset-react@^7.18.6, @babel/preset-typescript@^7.23.2, @chakra-ui/icons@^2.0.16, @chakra-ui/react@^2.4.8, @emotion/babel-preset-css-prop@^11.10.0, @emotion/react@^11.10.5, @emotion/styled@^11.10.5, @jest/globals@^29.5.0, @mozilla/readability@^0.4.2, @types/chrome@^0.0.202, @types/prettier@^2.7.2, @types/react@^18.0.26, @types/react-dom@^18.0.10, @types/react-syntax-highlighter@^15.5.6, @typescript-eslint/eslint-plugin@^5.57.0, @typescript-eslint/parser@^5.57.0, babel-eslint@^10.1.0, babel-jest@^29.5.0, babel-loader@^8.3.0, babel-preset-react-app@^10.0.1, clean-webpack-plugin@^4.0.0, copy-webpack-plugin@^7.0.0, css-loader@^6.7.3, dotenv-webpack@^8.0.1, eslint@^8.31.0, eslint-config-prettier@^8.8.0, eslint-config-react-app@^7.0.1, eslint-plugin-flowtype@^8.0.3, eslint-plugin-import@^2.27.4, eslint-plugin-jsx-a11y@^6.7.1, eslint-plugin-prettier@^4.2.1, eslint-plugin-react@^7.32.0, eslint-plugin-react-hooks@^4.6.0, eslint-plugin-unused-imports@^2.0.0, file-loader@^6.2.0, framer-motion@^8.4.3, fs-extra@^10.1.0, html-loader@^3.1.0, html-webpack-plugin@^5.5.0, immer@^9.0.21, jest@^29.5.0, jest-environment-jsdom@^29.5.0, lodash@^4.17.21, mic-recorder-to-mp3@^2.2.2, node-sass@^8.0.0, openai@^3.2.1, prettier@^2.8.4, react@^18.2.0, react-dom@^18.2.0, react-hot-loader@^4.13.1, react-icons@^4.8.0, react-syntax-highlighter@^15.5.0, react-textarea-autosize@^8.4.1, react-use@^17.4.0, sass-loader@^13.2.0, source-map-loader@^3.0.1, stream-browserify@^3.0.0, style-loader@^3.3.1, terser-webpack-plugin@^5.3.6, ts-loader@^9.4.2, typescript@^4.9.4, webpack@^5.75.0, webpack-cli@^4.10.0, webpack-dev-server@^4.11.1, zustand@^4.3.6 ### Recent commits (newest first) - added: voice module - removed bloat - init ## Key source files (fetched from GitHub, selected and truncated for size) ### package.json ``` { "name": "zero", "version": "1.1.0", "description": "zer0 agent", "license": "MIT", "scripts": { "build": "node utils/build.js", "start": "node utils/webserver.js", "prettier": "prettier --write '**/*.{js,jsx,ts,tsx,json,css,scss,md}'", "test": "jest", "lint": "eslint src/ --ext .js,.jsx,.ts,.tsx" }, "dependencies": { "@azure/openai": "1.0.0-beta.7", "@chakra-ui/icons": "^2.0.16", "@chakra-ui/react": "^2.4.8", "@emotion/react": "^11.10.5", "@emotion/styled": "^11.10.5", "@mozilla/readability": "^0.4.2", "@typescript-eslint/eslint-plugin": "^5.57.0", "@typescript-eslint/parser": "^5.57.0", "babel-jest": "^29.5.0", "dotenv-webpack": "^8.0.1", "eslint-plugin-unused-imports": "^2.0.0", "framer-motion": "^8.4.3", "immer": "^9.0.21", "lodash": "^4.17.21", "mic-recorder-to-mp3": "^2.2.2", "openai": "^3.2.1", "prettier": "^2.8.4", "react": "^18.2.0", "react-dom": "^18.2.0", "react-hot-loader": "^4.13.1", "react-icons": "^4.8.0", "react-syntax-highlighter": "^15.5.0", "react-textarea-autosize": "^8.4.1", "react-use": "^17.4.0", "zustand": "^4.3.6" }, "devDependencies": { "@babel/core": "^7.20.12", "@babel/plugin-proposal-class-properties": "^7.18.6", "@babel/preset-env": "^7.20.2", "@babel/preset-react": "^7.18.6", "@babel/preset-typescript": "^7.23.2", "@emotion/babel-preset-css-prop": "^11.10.0", "@jest/globals": "^29.5.0", "@types/chrome": "^0.0.202", "@types/prettier": "^2.7.2", "@types/react": "^18.0.26", "@types/react-dom": "^18.0.10", "@types/react-syntax-highlighter": "^15.5.6", "babel-eslint": "^10.1.0", "babel-loader": "^8.3.0", "babel-preset-react-app": "^10.0.1", "clean-webpack-plugin": "^4.0.0", "copy-webpack-plugin": "^7.0.0", "css-loader": "^6.7.3", "eslint": "^8.31.0", "eslint-config-prettier": "^8.8.0", "eslint-config-react-app": "^7.0.1", "eslint-plugin-flowtype": "^8.0.3", "eslint-plugin-import": "^2.27.4", "eslint-plugin-jsx-a11y": "^6.7.1", "eslint-plugin-prettier": "^4.2.1", "eslint-plugin-react": "^7.32.0", "eslint-plugin-react-hooks": "^4.6.0", "file-loader": "^6.2.0", "fs-extra": "^10.1.0", "html-loader": "^3.1.0", "html-webpack-plugin": "^5.5.0", "jest": "^29.5.0", "jest-environment-jsdom": "^29.5.0", "node-sass": "^8.0.0", "sass-loader": "^13.2.0", "source-map-loader": "^3.0.1", "stream-browserify": "^3.0.0", "style-loader": "^3.3.1", "terser-webpack-plugin": "^5.3.6", "ts-loader": "^9.4.2", "typescript": "^4.9.4", "webpack": "^5.75.0", "webpack-cli": "^4.10.0", "webpack-dev-server": "^4.11.1" } } ``` ### src/common/App.tsx ```typescript import { Box, ChakraProvider, Heading, HStack } from '@chakra-ui/react'; import React from 'react'; import { useAppState } from '../state/store'; import ModelDropdown from './ModelDropdown'; import SetAPIKey from './SetAPIKey'; import TaskUI from './TaskUI'; import OptionsDropdown from './OptionsDropdown'; import logo from '../assets/img/icon-128.png'; const App = () => { const openAIKey = useAppState((state) => state.settings.openAIKey); return ( logo zer0 {openAIKey ? : } ); }; export default App; ``` ### src/pages/Devtools/index.js ```javascript chrome.devtools.panels.create('zer0', 'icon-34.png', 'panel.html'); ``` ### src/pages/Background/index.js ```javascript // console.log('This is the background page.'); // console.log('Put the background scripts here.'); ``` ### src/pages/Content/index.ts ```typescript // console.log('Content script loaded..'); import { watchForRPCRequests } from '../../helpers/pageRPC'; watchForRPCRequests(); ``` ### src/pages/Newtab/index.jsx ```javascript import React from 'react'; import { render } from 'react-dom'; import Newtab from './Newtab'; import './index.css'; render(, window.document.querySelector('#app-container')); if (module.hot) module.hot.accept(); ``` ### src/pages/Options/index.jsx ```javascript import React from 'react'; import { render } from 'react-dom'; import Options from './Options'; import './index.css'; render( , window.document.querySelector('#app-container') ); if (module.hot) module.hot.accept(); ``` ### src/pages/Panel/index.jsx ```javascript import React from 'react'; import { createRoot } from 'react-dom/client'; import Panel from './Panel'; import './index.css'; const container = window.document.querySelector('#app-container'); const root = createRoot(container); root.render(); if (module.hot) module.hot.accept(); ``` ### src/pages/Popup/index.jsx ```javascript import React from 'react'; import { createRoot } from 'react-dom/client'; import Popup from './Popup'; import './index.css'; const container = window.document.querySelector('#app-container'); const root = createRoot(container); root.render(); if (module.hot) module.hot.accept(); ``` ### webpack.config.js ```javascript var webpack = require('webpack'), path = require('path'), fileSystem = require('fs-extra'), env = require('./utils/env'), CopyWebpackPlugin = require('copy-webpack-plugin'), HtmlWebpackPlugin = require('html-webpack-plugin'), TerserPlugin = require('terser-webpack-plugin'), Dotenv = require('dotenv-webpack'); var { CleanWebpackPlugin } = require('clean-webpack-plugin'); const ASSET_PATH = process.env.ASSET_PATH || '/'; var alias = {}; // load the secrets var secretsPath = path.join(__dirname, 'secrets.' + env.NODE_ENV + '.js'); var fileExtensions = [ 'jpg', 'jpeg', 'png', 'gif', 'eot', 'otf', 'svg', 'ttf', 'woff', 'woff2', ]; if (fileSystem.existsSync(secretsPath)) { alias['secrets'] = secretsPath; } var options = { mode: process.env.NODE_ENV || 'development', entry: { newtab: path.join(__dirname, 'src', 'pages', 'Newtab', 'index.jsx'), options: path.join(__dirname, 'src', 'pages', 'Options', 'index.jsx'), popup: path.join(__dirname, 'src', 'pages', 'Popup', 'index.jsx'), background: path.join(__dirname, 'src', 'pages', 'Background', 'index.js'), contentScript: path.join(__dirname, 'src', 'pages', 'Content', 'index.ts'), devtools: path.join(__dirname, 'src', 'pages', 'Devtools', 'index.js'), panel: path.join(__dirname, 'src', 'pages', 'Panel', 'index.jsx'), }, chromeExtensionBoilerplate: { notHotReload: ['background', 'contentScript', 'devtools'], }, output: { filename: '[name].bundle.js', path: path.resolve(__dirname, 'build'), clean: true, publicPath: ASSET_PATH, }, module: { rules: [ { // look for .css or .scss files test: /\.(css|scss)$/, // in the `src` directory use: [ { loader: 'style-loader', }, { loader: 'css-loader', }, { loader: 'sass-loader', options: { sourceMap: true, }, }, ], }, { test: new RegExp('.(' + fileExtensions.join('|') + ')$'), type: 'asset/resource', exclude: /node_modules/, // loader: 'file-loader', // options: { // name: '[name].[ext]', // }, }, { test: /\.html$/, loader: 'html-loader', exclude: /node_modules/, }, // { test: /\.(ts|tsx)$/, loader: 'babel-loader', exclude: /node_modules/ }, { test: /\.(js|jsx|ts|tsx)$/, use: [ { loader: 'source-map-loader', }, { loader: 'babel-loader', }, ], exclude: /node_modules/, }, ], }, resolve: { alias: alias, fallback: { stream: false, }, extensions: fileExtensions .map((extension) => '.' + extension) .concat(['.js', '.jsx', '.ts', '.tsx', '.css']), }, plugins: [ new CleanWebpackPlugin({ verbose: false }), new webpack.ProgressPlugin(), // expose and write the allowed env vars on the compiled bundle new webpack.EnvironmentPlugin(['NODE_ENV']), new CopyWebpackPlugin({ patterns: [ { from: 'src/manifest.json', to: path.join(__dirname, 'build'), force: true, transform: function (content, path) { // generates the manifest file using the package.json informations return Buffer.from( JSON.stringify({ description: process.env.npm_package_description, version: process.env.npm_package_version, ...JSON.parse(content.toString()), }) ); }, }, ], }), new CopyWebpackPlugin({ patterns: [ { from: 'src/pages/Content/content.styles.css', to: path.join(__dirname, 'build'), force: true, }, ], }), new CopyWebpackPlugin({ patterns: [ { from: 'src/assets/img/icon-128.png', to: path.join(__dirname, 'build'), force: true, }, ], }), new CopyWebpackPlugin({ patterns: [ { from: 'src/assets/img/icon-34.png', to: path.join(__dirname, 'build'), force: true, }, ], }), new HtmlWebpackPlugin({ template: path.join(__dirname, 'src', 'pages', 'Newtab', 'index.html'), filename: 'newtab.html', chunks: ['newtab'], cache: false, }), new HtmlWebpackPlugin({ template: path.join(__dirname, 'src', 'pages', 'Options', 'index.html'), filename: 'options.html', chunks: ['options'], cache: false, }), new HtmlWebpackPlugin({ template: path.join(__dirname, 'src', 'pages', 'Popup', 'index.html'), filename: 'popup.html', chunks: ['popup'], cache: false, }), new HtmlWebpackPlugin({ template: path.join(__dirname, 'src', 'pages', 'Devtools', 'index.html'), filename: 'devtools.html', chunks: ['devtools'], cache: false, }), new HtmlWebpackPlugin({ template: path.join(__dirname, 'src', 'pages', 'Panel', 'index.html'), filename: 'panel.html', chunks: ['panel'], cache: false, }), new Dotenv({ path: path.resolve(__dirname, '.env') }), ], infrastructureLogging: { level: 'info', }, experiments: { asyncWebAssembly: true, }, }; if (env.NODE_ENV === 'development') { options.devtool = 'cheap-module-source-map'; } else { options.optimization = { minimize: true, minimizer: [ new TerserPlugin({ extractComments: false, }), ], }; } module.exports = options; ``` [52 more indexed source files omitted to keep this export small. The full file list is in the Codebase structure section above.]