# Project export: LLMaOS This document was generated by HackStack to give an AI agent context about a hackathon project. Sections are labeled with their provenance; content marked as truncated was cut to keep this document small. ## Project metadata - Hackathon: TreeHacks 2025 - Tagline: A voice-controlled operating system - it is general-purpose, low-latency, transparent, user-friendly and possesses search and screen-analysis capabilities. - Devpost: https://devpost.com/software/llmaos - GitHub: https://github.com/jianzhi-1/LLMaOS - Video: https://www.youtube.com/embed/ECTOJ7sqNEE?enablejsapi=1&hl=en_US&rel=0&start=&version=3&wmode=transparent - Team: 1 GitHub contributor(s) — jianzhi-1 (16 commits) ## Devpost submission (written by the team) ### Inspiration The average typing speed is 40 words per minute, which is considerably slower than the average speaking speed (150 words per minute). However, the keyboard is still the main way in which most people interact with their devices, and it has stayed that way for decades. There are existing voice applications like Siri and Alexa, but those products did not process other streams of information such as the display. I propose a voice-controlled operating system, with the entire stack (the assembler, the processor) enhanced by AI components. This should remind people of J.A.R.V.I.S. and the robots in WALL-E. ### What it does As an operating system, LLMaOS simplifies the way you interact with the computer. Below are some examples (non-exhaustive). (1) "Play me the song Espresso" LLMaOS launches Chrome, enters youtube.com in the URL bar, enters Espresso in the YouTube search bar, clicks the first non-ad entry, clicks Skip Ads, enters fullscreen (2) "What is the score between Manchester City and Real Madrid?" LLMaOS launches Chrome, enters google.com in the URL bar, enters "Man City vs Real Madrid" in the search bar, analyses the screen, and tells you the score. (3) "When is the next Codeforces contest?" LLMaOS launches Chrome, enters codeforces.com in the URL bar, analyses the screen, and tells you the time of the next contest. LLMaOS is voice controlled, transparent (you keep a log of its "assembly-level" instructions and can see what it is doing), and possesses screen-processing capabilities. ### How we built it There are three modular components: User interface: This (application) layer is for voice control and is powered by OpenAI's RealTime API. In the RealTime session, user transcripts were decoded and sent to the Assembler for code generation. User interface: This (application) layer is for voice control and is powered by OpenAI's RealTime API. In the RealTime session, user transcripts were decoded and sent to the Assembler for code generation. Assembler: The purpose of this (operating system-compiler) layer is to generate "assembly-like" instructions for the processor. Such instructions include any non-dangerous UNIX commands, device commands (LEFT_CLICK x y, KEYBOARD string) and the screen-processing command ANALYSIS. Assembler: The purpose of this (operating system-compiler) layer is to generate "assembly-like" instructions for the processor. Such instructions include any non-dangerous UNIX commands, device commands (LEFT_CLICK x y, KEYBOARD string) and the screen-processing command ANALYSIS. Processor: The processor executes the instructions generated by the assembler. For example, the special instruction ANALYSIS takes a screenshot and uses 3 AI models (NVIDIA's NeVA, Mistral AI's Pixtral, OpenAI's GPT) in parallel (Python's asyncio) to extract information from it. The collated information is fed back to both the assembler and RealTime. This layer of LLMaOS deviates from the traditional computer architecture in the sense that instructions are generated on the fly. For example, ANALYSIS on an image must be done first before determining the x and y arguments for the next LEFT_CLICK instruction. Much like a motherboard, the processor can offload tasks to large models' API endpoints, analogous to specialised hardware accelerators. Processor: The processor executes the instructions generated by the assembler. For example, the special instruction ANALYSIS takes a screenshot and uses 3 AI models (NVIDIA's NeVA, Mistral AI's Pixtral, OpenAI's GPT) in parallel (Python's asyncio) to extract information from it. The collated information is fed back to both the assembler and RealTime. This layer of LLMaOS deviates from the traditional computer architecture in the sense that instructions are generated on the fly. For example, ANALYSIS on an image must be done first before determining the x and y arguments for the next LEFT_CLICK instruction. Much like a motherboard, the processor can offload tasks to large models' API endpoints, analogous to specialised hardware accelerators. ### Challenges we ran into The main challenge lies in image analysis. For example, it is difficult to detect a cross symbol on a high-pixel-density screen. It is also difficult to understand the texts on the screen. Due to these, models' confidence in bounding boxes sharply decreased, which made it hard for the assembler, particularly the LEFT_CLICK x y instructions. To alleviate this problem, I threw three giant models at it, but there are still failure cases. In the end, I had to hardcode some portions for demonstration purposes. (I would also have liked to use NVIDIA's ocdrnet model, but there is a tradeoff between the image sizes it supports and the clarity of texts.) The latency and user-friendliness of the operating system are also crucial - the calls to the three API endpoints are parallelised (using Python's async). However, there is still significant latency due to image-processing the screen. The context switching between LLMaOS (especially maintaining the RealTime session) and other applications like Chrome is not very significant but still noticeable. ### Accomplishments we're proud of Made a demo-able (though slightly hackish) prototype. Come by to try it out! ### What we learned (1) The power of abstraction and (2) connecting many layers of the computer stack together. ### What's next I'll focus on improving the AI subcomponents first, by using/training better models that are specifically for this task. Then, add more features and optimisations to the LLMaOS stack. ## README (from the GitHub repository) # LLMaOS A voice-controlled operating system that is general-purposed, low-latency, transparent, user-friendly and possesses search and screen-analysis capabilities. ### Set Up ```shell conda activate berkos export OPENAI_API_KEY= export MISTRAL_API_KEY= export NVIDIA_API_KEY= ``` ### Architecture 1. User interface: This (application) layer is for voice control and is powered by OpenAI's RealTime API. In the RealTime session, user transcripts were decoded and sent to the Assembler for code generation. 2. Assembler: The purpose of this (operating system-compiler) layer is to generate "assembly-like" instructions for the processor. Such instructions include any non-dangerous UNIX commands, device commands (`LEFT_CLICK x y`, `KEYBOARD string`) and the screen-processing command `ANALYSIS`. 3. Processor: The processor executes the instructions generated by the assembler. For example, the special instruction `ANALYSIS` takes a screenshot and uses 3 AI models (NVIDIA's NeVA, Mistral AI's Pixtral, OpenAI's GPT) in parallel (Python's asyncio) to extract information from it. The collated information is fed back to both the assembler and RealTime. This layer of LLMaOS deviates from the traditional computer architecture in the sense that instructions are generated on the fly. For example, `ANALYSIS` on an image must be done first before determining the `x` and `y` arguments for the next `LEFT_CLICK` instruction. Much like a motherboard, the processor can offload tasks to large models' API endpoints, analogous to specialised hardware accelerators. ### Supported Use Cases (1) "Play me the song Espresso" - LLMaOS launches Chrome, enters youtube.com in the URL bar, enters Espresso in the YouTube search bar, clicks the first non-ad entry, clicks Skip Ads, enters fullscreen (2) "What is the score between Manchester City and Real Madrid?" - LLMaOS launches Chrome, enters google.com in the URL bar, enters "Man City vs Real Madrid" in the search bar, analyses the screen, and tells you the score. (3) "When is the next Codeforces contest?" - LLMaOS launches Chrome, enters codeforces.com in the URL bar, analyses the screen, and tells you the time of the next contest. LLMaOS is voice controlled, transparent (you keep a log of its "assembly-level" instructions and can see what it is doing), and possesses screen-processing capabilities. ## Detected evidence (automated analysis) Indexed codebase: 416 recognized source files, 1806 KB. - Python (language) — detected in the code ## Codebase structure (from repository index) ### Files (120 of 420) ``` .gitignore app/app.py app/audio_util.py pyproject.toml README.md src/llmaos.py src/openai/__init__.py src/openai/__main__.py src/openai/_base_client.py src/openai/_client.py src/openai/_compat.py src/openai/_constants.py src/openai/_exceptions.py src/openai/_extras/__init__.py src/openai/_extras/_common.py src/openai/_extras/numpy_proxy.py src/openai/_extras/pandas_proxy.py src/openai/_files.py src/openai/_legacy_response.py src/openai/_models.py src/openai/_module_client.py src/openai/_qs.py src/openai/_resource.py src/openai/_response.py src/openai/_streaming.py src/openai/_types.py src/openai/_utils/__init__.py src/openai/_utils/_logs.py src/openai/_utils/_proxy.py src/openai/_utils/_reflection.py src/openai/_utils/_streams.py src/openai/_utils/_sync.py src/openai/_utils/_transform.py src/openai/_utils/_typing.py src/openai/_utils/_utils.py src/openai/_version.py src/openai/cli/__init__.py src/openai/cli/_api/__init__.py src/openai/cli/_api/_main.py src/openai/cli/_api/audio.py src/openai/cli/_api/chat/__init__.py src/openai/cli/_api/chat/completions.py src/openai/cli/_api/completions.py src/openai/cli/_api/files.py src/openai/cli/_api/image.py src/openai/cli/_api/models.py src/openai/cli/_cli.py src/openai/cli/_errors.py src/openai/cli/_models.py src/openai/cli/_progress.py src/openai/cli/_tools/__init__.py src/openai/cli/_tools/_main.py src/openai/cli/_tools/fine_tunes.py src/openai/cli/_tools/migrate.py src/openai/cli/_utils.py src/openai/lib/__init__.py src/openai/lib/_old_api.py src/openai/lib/_parsing/__init__.py src/openai/lib/_parsing/_completions.py src/openai/lib/_pydantic.py src/openai/lib/_tools.py src/openai/lib/_validators.py src/openai/lib/.keep src/openai/lib/azure.py src/openai/lib/streaming/__init__.py src/openai/lib/streaming/_assistants.py src/openai/lib/streaming/_deltas.py src/openai/lib/streaming/chat/__init__.py src/openai/lib/streaming/chat/_completions.py src/openai/lib/streaming/chat/_events.py src/openai/lib/streaming/chat/_types.py src/openai/pagination.py src/openai/py.typed src/openai/resources/__init__.py src/openai/resources/audio/__init__.py src/openai/resources/audio/audio.py src/openai/resources/audio/speech.py src/openai/resources/audio/transcriptions.py src/openai/resources/audio/translations.py src/openai/resources/batches.py src/openai/resources/beta/__init__.py src/openai/resources/beta/assistants.py src/openai/resources/beta/beta.py src/openai/resources/beta/chat/__init__.py src/openai/resources/beta/chat/chat.py src/openai/resources/beta/chat/completions.py src/openai/resources/beta/realtime/__init__.py src/openai/resources/beta/realtime/realtime.py src/openai/resources/beta/realtime/sessions.py src/openai/resources/beta/threads/__init__.py src/openai/resources/beta/threads/messages.py src/openai/resources/beta/threads/runs/__init__.py src/openai/resources/beta/threads/runs/runs.py src/openai/resources/beta/threads/runs/steps.py src/openai/resources/beta/threads/threads.py src/openai/resources/beta/vector_stores/__init__.py src/openai/resources/beta/vector_stores/file_batches.py src/openai/resources/beta/vector_stores/files.py src/openai/resources/beta/vector_stores/vector_stores.py src/openai/resources/chat/__init__.py src/openai/resources/chat/chat.py src/openai/resources/chat/completions/__init__.py src/openai/resources/chat/completions/completions.py src/openai/resources/chat/completions/messages.py src/openai/resources/completions.py src/openai/resources/embeddings.py src/openai/resources/files.py src/openai/resources/fine_tuning/__init__.py src/openai/resources/fine_tuning/fine_tuning.py src/openai/resources/fine_tuning/jobs/__init__.py src/openai/resources/fine_tuning/jobs/checkpoints.py src/openai/resources/fine_tuning/jobs/jobs.py src/openai/resources/images.py src/openai/resources/models.py src/openai/resources/moderations.py src/openai/resources/uploads/__init__.py src/openai/resources/uploads/parts.py src/openai/resources/uploads/uploads.py src/openai/types/__init__.py src/openai/types/audio_model.py [300 more files omitted for size] ``` ### Dependencies - pyproject.toml: anyio@>=3.5.0, <5, distro@>=1.7.0, <2, httpx@>=0.23.0, <1, jiter@>=0.4.0, <1, numpy@>= 1, pandas@>= 1.2.3, pandas-stubs@>= 1.1.0.11, pydantic@>=1.9.0, <3, sniffio, tqdm@> 4, typing-extensions@>=4.11, <5, websockets@>= 13, < 15 ### Recent commits (newest first) - updated README - clean up - integrated pixtral, best performing model so far - reduced instruction set and made instructions more robust - clean up - renamed files - added nvidia-neva api endpoint - implemented optimisation that removes tts-1 - performance enhancement - no need additional tts-1 model, removed time.sleep - added processor worker and fixed analysis bug - added analysis stage - added codegen and execution - added initial version of os - added initial files - took pyproject.toml without change - first commit ## Key source files (fetched from GitHub, selected and truncated for size) ### pyproject.toml ``` [project] name = "openai" version = "1.63.0" description = "The official Python library for the openai API" dynamic = ["readme"] license = "Apache-2.0" authors = [ { name = "OpenAI", email = "support@openai.com" }, ] dependencies = [ "httpx>=0.23.0, <1", "pydantic>=1.9.0, <3", "typing-extensions>=4.11, <5", "anyio>=3.5.0, <5", "distro>=1.7.0, <2", "sniffio", "tqdm > 4", "jiter>=0.4.0, <1", ] requires-python = ">= 3.8" classifiers = [ "Typing :: Typed", "Intended Audience :: Developers", "Programming Language :: Python :: 3.8", "Programming Language :: Python :: 3.9", "Programming Language :: Python :: 3.10", "Programming Language :: Python :: 3.11", "Programming Language :: Python :: 3.12", "Operating System :: OS Independent", "Operating System :: POSIX", "Operating System :: MacOS", "Operating System :: POSIX :: Linux", "Operating System :: Microsoft :: Windows", "Topic :: Software Development :: Libraries :: Python Modules", "License :: OSI Approved :: Apache Software License" ] [project.urls] Homepage = "https://github.com/openai/openai-python" Repository = "https://github.com/openai/openai-python" [project.scripts] openai = "openai.cli:main" [project.optional-dependencies] realtime = ["websockets >= 13, < 15"] datalib = ["numpy >= 1", "pandas >= 1.2.3", "pandas-stubs >= 1.1.0.11"] [tool.rye] managed = true # version pins are in requirements-dev.lock dev-dependencies = [ "pyright>=1.1.359", "mypy", "respx", "pytest", "pytest-asyncio", "ruff", "time-machine", "nox", "dirty-equals>=0.6.0", "importlib-metadata>=6.7.0", "rich>=13.7.1", "inline-snapshot >=0.7.0", "azure-identity >=1.14.1", "types-tqdm > 4", "types-pyaudio > 0", "trio >=0.22.2", "nest_asyncio==1.6.0", ] [tool.rye.scripts] format = { chain = [ "format:ruff", "format:docs", "fix:ruff", # run formatting again to fix any inconsistencies when imports are stripped "format:ruff", ]} "format:docs" = "python scripts/utils/ruffen-docs.py README.md api.md" "format:ruff" = "ruff format" "lint" = { chain = [ "check:ruff", "typecheck", "check:importable", ]} "check:ruff" = "ruff check ." "fix:ruff" = "ruff check --fix ." "check:importable" = "python -c 'import openai'" typecheck = { chain = [ "typecheck:pyright", "typecheck:mypy" ]} "typecheck:pyright" = "pyright" "typecheck:verify-types" = "pyright --verifytypes openai --ignoreexternal" "typecheck:mypy" = "mypy ." [build-system] requires = ["hatchling", "hatch-fancy-pypi-readme"] build-backend = "hatchling.build" [tool.hatch.build] include = [ "src/*" ] [tool.hatch.build.targets.wheel] packages = ["src/openai"] [tool.hatch.build.targets.sdist] # Basically everything except hidden files/directories (such as .github, .devcontainers, .python-version, etc) include = [ "app/*", "src/*", ] [tool.hatch.metadata.hooks.fancy-pypi-readme] content-type = "text/markdown" [[tool.hatch.metadata.hooks.fancy-pypi-readme.fragments]] path = "README.md" [[tool.hatch.metadata.hooks.fancy-pypi-readme.substitutions]] # replace relative links with absolute links pattern = '\[(.+?)\]\(((?!https?://)\S+?)\)' replacement = '[\1](https://github.com/openai/openai-python/tree/main/\g<2>)' [tool.pytest.ini_options] testpaths = ["tests"] addopts = "--tb=short" xfail_strict = true asyncio_mode = "auto" asyncio_default_fixture_loop_scope = "session" filterwarnings = [ "error" ] [tool.pyright] # this enables practically every flag given by pyright. # there are a couple of flags that are still disabled by # default in strict mode as they are experimental and niche. typeCheckingMode = "strict" pythonVersion = "3.8" exclude = [ "_dev", ".venv", ".nox", # uses inline `uv` script dependencies # which means it can't be type checked "examples/realtime/audio_util.py", "examples/realtime/push_to_talk_app.py" ] reportImplicitOverride = true reportImportCycles = false reportPrivateUsage = false [tool.ruff] line-length = 120 output-format = "grouped" target-version = "py37" [tool.ruff.format] docstring-code-format = true [tool.ruff.lint] select = [ # isort "I", # bugbear rules "B", # remove unused imports "F401", # bare except statements "E722", # unused arguments "ARG", # print statements "T201", "T203", # misuse of typing.TYPE_CHECKING "TC004", # import rules "TID251", ] ignore = [ # mutable defaults "B006", ] unfixable = [ # disable auto fix for print statements "T201", "T203", ] [tool.ruff.lint.flake8-tidy-imports.banned-api] "functools.lru_cache".msg = "This function does not retain type information for the wrapped function's arguments; The `lru_cache` function from `_utils` should be used instead" [tool.ruff.lint.isort] length-sort = true length-sort-straight = true combine-as-imports = true extra-standard-library = ["typing_extensions"] known-first-party = ["openai", "tests"] [tool.ruff.lint.per-file-ignores] "scripts/**.py" = ["T201", "T203"] "app/**.py" = ["T201", "T203"] ``` ### app/app.py ```python #!/usr/bin/env uv run #################################################################### # Sample TUI app with a push to talk interface to the Realtime API # # If you have `uv` installed and the `OPENAI_API_KEY` # # environment variable set, you can run this example with just # # # # `./app/push_to_talk_app.py` # #################################################################### # # /// script # requires-python = ">=3.9" # dependencies = [ # "textual", # "numpy", # "pyaudio", # "pydub", # "sounddevice", # "openai[realtime]", # "pyautogui", # "requests", # "mistralai" # ] # # [tool.uv.sources] # openai = { path = "../", editable = true } # /// from __future__ import annotations import base64 import asyncio from typing import Any, cast, Tuple, Union, List from typing_extensions import override import logging import queue from pydub import AudioSegment from pydub.playback import play from textual import events from audio_util import CHANNELS, SAMPLE_RATE, AudioPlayerAsync from textual.app import App, ComposeResult from textual.widgets import Button, Static, RichLog from textual.reactive import reactive from textual.containers import Container from openai import AsyncOpenAI from openai.types.beta.realtime.session import Session from openai.resources.beta.realtime.realtime import AsyncRealtimeConnection from llmaos import LLMAOS logger = logging.getLogger(__name__) logging.basicConfig(filename='berk.log', encoding='utf-8', level=logging.WARN) class SessionDisplay(Static): """A widget that shows the current session ID.""" session_id = reactive("") @override def render(self) -> str: return f"Session ID: {self.session_id}" if self.session_id else "Connecting..." class AudioStatusIndicator(Static): """A widget that shows the current audio recording status.""" is_recording = reactive(False) @override def render(self) -> str: status = ( "🔴 Recording... (Press K to stop)" if self.is_recording else "⚪ Press K to start recording (Q to quit)" ) return status class RealtimeApp(App[None]): CSS = """ Screen { background: #1a1b26; /* Dark blue-grey background */ } Container { border: double rgb(91, 164, 91); } Horizontal { width: 100%; } #input-container { height: 5; /* Explicit height for input container */ margin: 1 1; padding: 1 2; } Input { width: 80%; height: 3; /* Explicit height for input */ } Button { width: 20%; height: 3; /* Explicit height for button */ } #bottom-pane { width: 100%; height: 82%; /* Reduced to make room for session display */ border: round rgb(205, 133, 63); content-align: center middle; } #status-indicator { height: 3; content-align: center middle; background: #2a2b36; border: solid rgb(91, 164, 91); margin: 1 1; } #session-display { height: 3; content-align: center middle; background: #2a2b36; border: solid rgb(91, 164, 91); margin: 1 1; } Static { color: white; } """ client: AsyncOpenAI should_send_audio: asyncio.Event audio_player: AudioPlayerAsync last_audio_item_id: str | None connection: AsyncRealtimeConnection | None session: Session | None connected: asyncio.Event def __init__(self) -> None: super().__init__() self.connection = None self.session = None self.client = AsyncOpenAI() self.audio_player = AudioPlayerAsync() self.last_audio_item_id = None self.should_send_audio = asyncio.Event() self.connected = asyncio.Event() self.REALTIME_MODEL = "gpt-4o-realtime-preview" self.processor_queue = queue.Queue() self.llmaos = LLMAOS(self.client, self, neva_flag=True, pixtral_flag=True) @override def compose(self) -> ComposeResult: """Create child widgets for the app.""" with Container(): yield SessionDisplay(id="session-display") yield AudioStatusIndicator(id="status-indicator") yield RichLog(id="bottom-pane", wrap=True, highlight=True, markup=True) async def on_mount(self) -> None: self.run_worker(self.handle_realtime_connection()) self.run_worker(self.send_mic_audio()) self.run_worker(self.processor()) async def handle_realtime_connection(self) -> None: async with self.client.beta.realtime.connect( model=self.REALTIME_MODEL, ) as conn: self.connection = conn self.connected.set() # note: this is the default and can be omitted # if you want to manually handle VAD yourself, then set `'turn_detection': None` await conn.session.update(session={ "turn_detection": {"type": "server_vad", "threshold": 0.8, "silence_duration_ms": 1000}, "instructions": """ You are an operating system named Berk, helping a user named Jay. Be very less verbose. Do not list stuff. If Jay asks you to perform operations relating to software or web search (such as sports), answer positively that you are initiating the commands. Do not answer "I can't", just say you will initiate commands to do so. """, "input_audio_transcription": { "model":"whisper-1", "language":"en" }, "voice":"sage" }) [truncated — 5391 more characters] ``` ### app/audio_util.py ```python from __future__ import annotations import io import base64 import asyncio import threading from typing import Callable, Awaitable import numpy as np import pyaudio import sounddevice as sd from pydub import AudioSegment from openai.resources.beta.realtime.realtime import AsyncRealtimeConnection CHUNK_LENGTH_S = 0.05 # 100ms SAMPLE_RATE = 24000 FORMAT = pyaudio.paInt16 CHANNELS = 1 # pyright: reportUnknownMemberType=false, reportUnknownVariableType=false, reportUnknownArgumentType=false def audio_to_pcm16_base64(audio_bytes: bytes) -> bytes: # load the audio file from the byte stream audio = AudioSegment.from_file(io.BytesIO(audio_bytes)) print(f"Loaded audio: {audio.frame_rate=} {audio.channels=} {audio.sample_width=} {audio.frame_width=}") # resample to 24kHz mono pcm16 pcm_audio = audio.set_frame_rate(SAMPLE_RATE).set_channels(CHANNELS).set_sample_width(2).raw_data return pcm_audio class AudioPlayerAsync: def __init__(self): self.queue = [] self.lock = threading.Lock() self.stream = sd.OutputStream( callback=self.callback, samplerate=SAMPLE_RATE, channels=CHANNELS, dtype=np.int16, blocksize=int(CHUNK_LENGTH_S * SAMPLE_RATE), ) self.playing = False self._frame_count = 0 def callback(self, outdata, frames, time, status): # noqa with self.lock: data = np.empty(0, dtype=np.int16) # get next item from queue if there is still space in the buffer while len(data) < frames and len(self.queue) > 0: item = self.queue.pop(0) frames_needed = frames - len(data) data = np.concatenate((data, item[:frames_needed])) if len(item) > frames_needed: self.queue.insert(0, item[frames_needed:]) self._frame_count += len(data) # fill the rest of the frames with zeros if there is no more data if len(data) < frames: data = np.concatenate((data, np.zeros(frames - len(data), dtype=np.int16))) outdata[:] = data.reshape(-1, 1) def reset_frame_count(self): self._frame_count = 0 def get_frame_count(self): return self._frame_count def add_data(self, data: bytes): with self.lock: # bytes is pcm16 single channel audio data, convert to numpy array np_data = np.frombuffer(data, dtype=np.int16) self.queue.append(np_data) if not self.playing: self.start() def start(self): self.playing = True self.stream.start() def stop(self): self.playing = False self.stream.stop() with self.lock: self.queue = [] def terminate(self): self.stream.close() async def send_audio_worker_sounddevice( connection: AsyncRealtimeConnection, should_send: Callable[[], bool] | None = None, start_send: Callable[[], Awaitable[None]] | None = None, ): sent_audio = False device_info = sd.query_devices() print(device_info) read_size = int(SAMPLE_RATE * 0.02) stream = sd.InputStream( channels=CHANNELS, samplerate=SAMPLE_RATE, dtype="int16", ) stream.start() try: while True: if stream.read_available < read_size: await asyncio.sleep(0) continue data, _ = stream.read(read_size) if should_send() if should_send else True: if not sent_audio and start_send: await start_send() await connection.send( {"type": "input_audio_buffer.append", "audio": base64.b64encode(data).decode("utf-8")} ) sent_audio = True elif sent_audio: print("Done, triggering inference") await connection.send({"type": "input_audio_buffer.commit"}) await connection.send({"type": "response.create", "response": {}}) sent_audio = False await asyncio.sleep(0) except KeyboardInterrupt: pass finally: stream.stop() stream.close() ``` ### src/llmaos.py ```python import pyautogui import time import subprocess import base64 import asyncio import os from typing import Union import requests from random import choice from string import ascii_uppercase import logging from mistralai import Mistral class LLMAOS(): def __init__(self, client, network, neva_flag=True, pixtral_flag=True): self.KEYWORDS = [""] self.GPT_MODEL = "gpt-4o-mini" self.PIXTRAL_MODEL = "pixtral-12b-2409" self.a0 = None # accumulator, stores return value self.t0 = None # task instruction self.screenshot_name_history = [] self.screenshot_name_length = 20 self.client = client self.network = network self.neva_flag = neva_flag self.pixtral_flag = pixtral_flag pass async def execute(self, instructions: list[str], transcript: str) -> None: for _inst in instructions: inst = _inst.strip("'").strip('"').strip() time.sleep(1.5) if "KEYBOARD" in inst: if inst[9:] in self.KEYWORDS: # key if inst[9:] == "": pyautogui.press('enter') else: self.keyboard_write(inst[9:].strip("'")) elif "LEFT_CLICK" in inst: ls = inst.split(" ") x, y = int(ls[1]), int(ls[2]) self.left_click(x, y) elif "WAIT" in inst: ls = inst.split(" ") t = float(ls[1]) time.sleep(t) elif "OPEN_TERMINAL" in inst: pyautogui.keyDown('command') pyautogui.press('space') pyautogui.keyUp('command') time.sleep(1.0) self.keyboard_write("terminal") time.sleep(1.0) pyautogui.press('enter') time.sleep(1.0) elif "ANALYSIS" in inst: self.screenshot() await self.analysis_powerful(transcript) elif "NO-OP" in inst: pass else: self.shell(inst) def shell(self, command: Union[str, list[str]]) -> None: if isinstance(command, str): command = command.strip("'").strip('"').strip() command = command.split(" ") result = subprocess.run(command, capture_output=True, text=True, check=True) print(result.stdout) def generate_hash(self, n: int) -> str: return "".join(choice(ascii_uppercase) for _ in range(n)) def screenshot(self, name: str="") -> None: if len(name) == 0: name = self.generate_hash(self.screenshot_name_length) screenshot = pyautogui.screenshot() screenshot.save(f"{name}.png") self.screenshot_name_history.append(name) logging.getLogger(__name__).warn(f"Screenshot name = {name}") async def assemble_and_run(self, transcript): instructions = await self.codegen(transcript) await self.execute(instructions, transcript) async def codegen(self, transcript) -> list[str]: """ Acts as an intermediate language. """ response = await self.client.chat.completions.create( model=self.GPT_MODEL, messages=[ { "role": "developer", "content": """ You are a compiler. Given the user's task, your response should be a sequence of the following instructions. If the user is not giving a task, then only output NO-OP. Do not insert numbering or ordered list. Just separate each command by enter.: 1. An executable UNIX command that is not dangerous. For example, rm -rf is dangerous. 2. LEFT_CLICK x y (x from 0=left to 1792=right, y from 0=up to 1120=down are integers): left clicks on (x, y) on the 1792 x 1120 screen. 3. OPEN_TERMINAL: opens up a terminal 4. ANALYSIS: takes a screenshot and send it to API endpoints for analysis 5. NO-OP: does nothing, used when Jay says something non-task related 6. KEYBOARD x (x is some string in single quotes or a special keyword like ): inputs x on the keyboard 7. WAIT x (x is integer): waits x seconds, for a website to load for example """ }, { "role": "developer", "content": """ For example, if the user asks for the score of football team X versus football team Y, output the following (do not change the integers): "open https://www.google.com/" "LEFT_CLICK 605 561" "KEYBOARD 'X vs Y'" "KEYBOARD " "ANALYSIS" """ }, { "role": "developer", "content": """ For example, if the user asks to open YouTube and search X, output the following (do not change the integers): "open https://www.youtube.com/" "LEFT_CLICK 598 142" "KEYBOARD 'X'" "KEYBOARD " "LEFT_CLICK 638 645" "WAIT 6" "LEFT_CLICK 1245 797" "LEFT_CLICK 1293 814" """ }, { "role": "developer", "content": """ For example, if the user says press enter, output the following: "KEYBOARD " """ }, { "role": "developer", "content": """ For example, if the user says type x: [truncated — 7975 more characters] ``` ### src/openai/__main__.py ```python from .cli import main main() ``` ### src/openai/version.py ```python from ._version import __version__ VERSION: str = __version__ ``` ### src/openai/_version.py ```python # File generated from our OpenAPI spec by Stainless. See CONTRIBUTING.md for details. __title__ = "openai" __version__ = "1.63.0" # x-release-please-version ``` ### src/openai/_constants.py ```python # File generated from our OpenAPI spec by Stainless. See CONTRIBUTING.md for details. import httpx RAW_RESPONSE_HEADER = "X-Stainless-Raw-Response" OVERRIDE_CAST_TO_HEADER = "____stainless_override_cast_to" # default timeout is 10 minutes DEFAULT_TIMEOUT = httpx.Timeout(timeout=600, connect=5.0) DEFAULT_MAX_RETRIES = 2 DEFAULT_CONNECTION_LIMITS = httpx.Limits(max_connections=1000, max_keepalive_connections=100) INITIAL_RETRY_DELAY = 0.5 MAX_RETRY_DELAY = 8.0 ``` ### src/openai/_resource.py ```python # File generated from our OpenAPI spec by Stainless. See CONTRIBUTING.md for details. from __future__ import annotations import time from typing import TYPE_CHECKING import anyio if TYPE_CHECKING: from ._client import OpenAI, AsyncOpenAI class SyncAPIResource: _client: OpenAI def __init__(self, client: OpenAI) -> None: self._client = client self._get = client.get self._post = client.post self._patch = client.patch self._put = client.put self._delete = client.delete self._get_api_list = client.get_api_list def _sleep(self, seconds: float) -> None: time.sleep(seconds) class AsyncAPIResource: _client: AsyncOpenAI def __init__(self, client: AsyncOpenAI) -> None: self._client = client self._get = client.get self._post = client.post self._patch = client.patch self._put = client.put self._delete = client.delete self._get_api_list = client.get_api_list async def _sleep(self, seconds: float) -> None: await anyio.sleep(seconds) ``` ### src/openai/_module_client.py ```python # File generated from our OpenAPI spec by Stainless. See CONTRIBUTING.md for details. from typing_extensions import override from . import resources, _load_client from ._utils import LazyProxy class ChatProxy(LazyProxy[resources.Chat]): @override def __load__(self) -> resources.Chat: return _load_client().chat class BetaProxy(LazyProxy[resources.Beta]): @override def __load__(self) -> resources.Beta: return _load_client().beta class FilesProxy(LazyProxy[resources.Files]): @override def __load__(self) -> resources.Files: return _load_client().files class AudioProxy(LazyProxy[resources.Audio]): @override def __load__(self) -> resources.Audio: return _load_client().audio class ImagesProxy(LazyProxy[resources.Images]): @override def __load__(self) -> resources.Images: return _load_client().images class ModelsProxy(LazyProxy[resources.Models]): @override def __load__(self) -> resources.Models: return _load_client().models class BatchesProxy(LazyProxy[resources.Batches]): @override def __load__(self) -> resources.Batches: return _load_client().batches class EmbeddingsProxy(LazyProxy[resources.Embeddings]): @override def __load__(self) -> resources.Embeddings: return _load_client().embeddings class CompletionsProxy(LazyProxy[resources.Completions]): @override def __load__(self) -> resources.Completions: return _load_client().completions class ModerationsProxy(LazyProxy[resources.Moderations]): @override def __load__(self) -> resources.Moderations: return _load_client().moderations class FineTuningProxy(LazyProxy[resources.FineTuning]): @override def __load__(self) -> resources.FineTuning: return _load_client().fine_tuning chat: resources.Chat = ChatProxy().__as_proxied__() beta: resources.Beta = BetaProxy().__as_proxied__() files: resources.Files = FilesProxy().__as_proxied__() audio: resources.Audio = AudioProxy().__as_proxied__() images: resources.Images = ImagesProxy().__as_proxied__() models: resources.Models = ModelsProxy().__as_proxied__() batches: resources.Batches = BatchesProxy().__as_proxied__() embeddings: resources.Embeddings = EmbeddingsProxy().__as_proxied__() completions: resources.Completions = CompletionsProxy().__as_proxied__() moderations: resources.Moderations = ModerationsProxy().__as_proxied__() fine_tuning: resources.FineTuning = FineTuningProxy().__as_proxied__() ``` [401 more indexed source files omitted to keep this export small. The full file list is in the Codebase structure section above.]