# Project export: Horus This document was generated by HackStack to give an AI agent context about a hackathon project. Sections are labeled with their provenance; content marked as truncated was cut to keep this document small. ## Project metadata - Hackathon: TreeHacks 2024 - Tagline: We're developing a live video monitoring system, able to take any video stream, identify incidents deemed “flaggable” by users, and in real-time contextually communicate these incidents with users. - Devpost: https://devpost.com/software/horus-c5wapt - GitHub: https://github.com/LiuLawrence45/treehacks - Video: https://www.youtube.com/embed/4wh-zVzDJko?enablejsapi=1&hl=en_US&rel=0&start=&version=3&wmode=transparent - Team: 1 GitHub contributor(s) — LiuLawrence45 (41 commits) ## Devpost submission (written by the team) ### Inspiration As a team, we're all close with our grandparents. As they get older, we want to be able to support them in any way that we can, even when we can't be there. Our ideation focus started out on elderly care homes--we wanted to devise a system to help keep an eye out for elderly patients, especially those who could be prone to falling or wandering off due to a condition like Alzheimer's. But we realized that interactive and accessible security isn't just limited to elderly care homes. From consumer usage with parents, to enterprise-level usage at companies that need real-time security monitoring, interactive security is largely lacking. ### What it does Our pipeline takes in live stream of videos, and leveraging zero-shot action recognition, we're able to provide real time feedback on chunks of videos that are of interest. Users are able to input with natural language, what they would consider "flaggable" behavior. We match chunks with flaggable behavior, and then, if an incident is flagged, real-time deployment of Twilio API calls the user who deployed the software. The user is then notified of the incident, and able to talk to Twilio about 1. Context of the incident 2. Possible next steps to take 3. Extractive video ### How we built it Computer Vision. Azure integration, to provide holistic view on people actions during frame, with natural language. Multi-threading allows us to achieve this in real-time without much latency. Chunking video into actions from the stream to be saved into our cache. Cache detection runs through GPT, and submits a POST request to our flask server when an action is deemed to be flaggable. Twilio retrieves. Twilio Separate tunneled local server on ngrok so Twilio can access. Then, server receives live requests that uses TwixML to give appropriate responses to users. Utilizing whisper, users are able to have real-time conversation with Twilio, which has access to custom defined toolkits. ### Challenges we ran into A lot of networking challenges. Some API keys disappeared here and there (we reached our limit), and spent like 2 hours debugging for no reason. ### Accomplishments we're proud of As beginners at our first hackathon, we really got to get hands on with building a project, and become a super cohesive team! This was a lot of our first times fully building out a deployable application, and it was really great just learning how to take our skills in class to apply to the real world. It's super inspiring to know that what we make can change the world. ### What we learned So so much. From multi-threading to lightsaber-duels, the knowledge was everywhere. ### What's next Huge emphasis on stopping implicit bias, especially in CV algorithms. We planning to focus on the ethical implications of software like Horus, stressing that implicit biases don't make their way into the input, detection, or communication pipelines. ## README (from the GitHub repository) # Horus ## Inspiration As a team, we're all close with our grandparents. As they get older, we want to be able to support them in any way that we can, even when we can't be there. Our ideation focus started out on elderly care homes--we wanted to devise a system to help keep an eye out for elderly patients, especially those who could be prone to falling or wandering off due to a condition like Alzheimer's. But we realized that interactive and accessible security isn't just limited to elderly care homes. From consumer usage with parents, to enterprise-level usage at companies that need real-time security monitoring, interactive security is largely lacking. ## What it does Our pipeline takes in live stream of videos, and leveraging zero-shot action recognition, we're able to provide real time feedback on chunks of videos that are of interest. Users are able to input with natural language, what they would consider "flaggable" behavior. We match chunks with flaggable behavior, and then, if an incident is flagged, real-time deployment of Twilio API calls the user who deployed the software. The user is then notified of the incident, and able to talk to Twilio about 1. Context of the incident 2. Possible next steps to take 3. Extractive video all sent to their phone in real time w/ real phone-call interaction. ## How we built it 1. Computer Vision. Azure + YoloV8 integration, to provide holistic view on people actions during frame, with natural language. Multi-threading allows us to achieve this in real-time without much latency. Chunking video based on sections of "similar" actions, to be saved in our DB. etc etc Retrieval when necessary, and then all sent to Twilio. 2. Twilio Tunneled local server on ngrok so Twilio can access. Then, server receives live requests that uses TwixML to give appropriate responses to users, after running through our fine-tuned GPT. Utilizing whisper, users are able to have real-time conversation with Twilio, who is able to use custom functions and utilities. etc etc ## Running 1. In directory treehacks: ngrok http 3000 --domain=treehacks.ngrok.app (or any ngrok domain). 2. In directory treehacks/call-gpt: npm run dev 3. In treehacks/processing/cv: python flaskPrompt.py In treehacks, add .env with AZURE_API_KEY="" VISION_ENDPOINT="" In /call-gpt, add .env with //Your ngrok or server URL (E.g. 123.ngrok.io or myserver.fly.dev) SERVER="treehacks.ngrok.app" //Service API Keys OPENAI_API_KEY="" DEEPGRAM_API_KEY="" XI_API_KEY="a38ad5190e33965faec7f22098a8f531" // Available models at a signed GET request to /v1/models XI_MODEL_ID="" // Uses "Rachel" voice by default See https://api.elevenlabs.io/v1/voices or visit https://elevenlabs.io/voice-library for a list of all available voices VOICE_ID="21m00Tcm4TlvDq8ikWAM" // Configure your Twilio credentials if you want to make test calls using '$ npm test'. TWILIO_ACCOUNT_SID="" TWILIO_AUTH_TOKEN="" FROM_NUMBER='' TO_NUMBER='' ## Detected evidence (automated analysis) Indexed codebase: 38 recognized source files, 97 KB. - Express (technology) — detected in the code - Flask (technology) — detected in the code - JavaScript (language) — detected in the code - OpenAI (technology) — detected in the code - Python (language) — detected in the code - PyTorch (technology) — detected in the code - TensorFlow (technology) — detected in the code - React (technology) — claimed on Devpost, not found in the code ## Codebase structure (from repository index) ### Files (75 of 75) ``` .gitignore call-gpt/.gitignore call-gpt/app-initial.json call-gpt/app.js call-gpt/functions/checkInventory.js call-gpt/functions/checkPrice.js call-gpt/functions/function-manifest.js call-gpt/functions/makeEmergencyCall.js call-gpt/functions/placeOrder.js call-gpt/package.json call-gpt/README.md call-gpt/scripts/inbound-call.js call-gpt/scripts/outbound-call.js call-gpt/scripts/requirements.txt call-gpt/services/gpt-service-prompts.json call-gpt/services/gpt-service.js call-gpt/services/stream-service.js call-gpt/services/transcription-service.js call-gpt/services/tts-service.js call-gpt/test/checkInventory.test.js call-gpt/test/checkPrice.test.js call-gpt/test/placeOrder.test.js package.json processing/__init__.py processing/.DS_Store processing/cv/__init__.py processing/cv/action_archive/asyncInitial.py processing/cv/action_archive/asyncQueue.py processing/cv/action_archive/asyncUpdated.py processing/cv/action_archive/demo-save.py processing/cv/action_archive/detectAction.py processing/cv/action_archive/detectActionAsync.py processing/cv/action_archive/old_prompt processing/cv/asyncSave.py processing/cv/cache/1080.txt processing/cv/cache/120.txt processing/cv/cache/1200.txt processing/cv/cache/1320.txt processing/cv/cache/1440.txt processing/cv/cache/240.txt processing/cv/cache/360.txt processing/cv/cache/480.txt processing/cv/cache/600.txt processing/cv/cache/720.txt processing/cv/cache/840.txt processing/cv/cache/960.txt processing/cv/cache/Error.txt processing/cv/cache/Summary_1080.txt processing/cv/cache/Summary_120.txt processing/cv/cache/Summary_1200.txt processing/cv/cache/Summary_1320.txt processing/cv/cache/Summary_1440.txt processing/cv/cache/Summary_240.txt processing/cv/cache/Summary_360.txt processing/cv/cache/Summary_480.txt processing/cv/cache/Summary_600.txt processing/cv/cache/Summary_720.txt processing/cv/cache/Summary_840.txt processing/cv/cache/Summary_960.txt processing/cv/createCache.py processing/cv/flaskPrompt.py processing/cv/medium-YOLO.pt processing/cv/shared_resources.py processing/cv/smallest-YOLO.pt processing/cv/yolov8.py processing/twilio-outdated/answer_phone.py processing/twilio-outdated/gpt_response.py processing/twilio-outdated/speechtotext.py processing/twilio-outdated/userinteraction.py processing/twilio/answer_phone.py processing/twilio/gpt_response.py processing/twilio/speechtotext.py processing/twilio/userinteraction.py README.md requirements.txt ``` ### Dependencies - call-gpt/package.json: @deepgram/sdk@^2.4.0, @flydotio/dockerfile@^0.4.11, colors@^1.4.0, dotenv@^16.3.1, express@^4.18.2, express-ws@^5.0.2, jest@^29.7.0, node-fetch@^2.7.0, nodemon@^3.0.2, openai@^4.20.1, twilio@^4.19.3, uuid@^9.0.1, wavefile@^11.0.0 - call-gpt/scripts/requirements.txt: absl-py@==2.1.0, adal@==1.2.7, aiohttp@==3.9.3, aiohttp-retry@==2.8.3, aiosignal@==1.3.1, annotated-types@==0.6.0, anyio@==4.2.0, astunparse@==1.6.3, async-timeout@==4.0.3, attrs@==23.2.0, azure@==4.0.0, azure-ai-formrecognizer@==3.3.2, azure-ai-vision-imageanalysis@==1.0.0b2, azure-applicationinsights@==0.1.1, azure-batch@==4.1.3, azure-cognitiveservices-vision-computervision@==0.9.0, azure-cognitiveservices-vision-contentmoderator@==1.0.0, azure-cognitiveservices-vision-customvision@==3.1.0, azure-common@==1.1.28, azure-core@==1.30.0, azure-cosmosdb-nspkg@==2.0.2, azure-cosmosdb-table@==1.0.6, azure-datalake-store@==0.0.53, azure-eventgrid@==1.3.0, azure-graphrbac@==0.40.0, azure-keyvault@==1.1.0, azure-loganalytics@==0.1.1, azure-mgmt@==4.0.0, azure-mgmt-advisor@==1.0.1, azure-mgmt-applicationinsights@==0.1.1, azure-mgmt-authorization@==0.50.0, azure-mgmt-batch@==5.0.1, azure-mgmt-batchai@==2.0.0, azure-mgmt-billing@==0.2.0, azure-mgmt-cdn@==3.1.0, azure-mgmt-cognitiveservices@==3.0.0, azure-mgmt-commerce@==1.0.1, azure-mgmt-compute@==4.6.2, azure-mgmt-consumption@==2.0.0, azure-mgmt-containerinstance@==1.5.0, azure-mgmt-containerregistry@==2.8.0, azure-mgmt-containerservice@==4.4.0, azure-mgmt-cosmosdb@==0.4.1, azure-mgmt-datafactory@==0.6.0, azure-mgmt-datalake-analytics@==0.6.0, azure-mgmt-datalake-nspkg@==3.0.1, azure-mgmt-datalake-store@==0.5.0, azure-mgmt-datamigration@==1.0.0, azure-mgmt-devspaces@==0.1.0, azure-mgmt-devtestlabs@==2.2.0, azure-mgmt-dns@==2.1.0, azure-mgmt-eventgrid@==1.0.0, azure-mgmt-eventhub@==2.6.0, azure-mgmt-hanaonazure@==0.1.1, azure-mgmt-iotcentral@==0.1.0, azure-mgmt-iothub@==0.5.0, azure-mgmt-iothubprovisioningservices@==0.2.0, azure-mgmt-keyvault@==1.1.0, azure-mgmt-loganalytics@==0.2.0, azure-mgmt-logic@==3.0.0, azure-mgmt-machinelearningcompute@==0.4.1, azure-mgmt-managementgroups@==0.1.0, azure-mgmt-managementpartner@==0.1.1, azure-mgmt-maps@==0.1.0, azure-mgmt-marketplaceordering@==0.1.0, azure-mgmt-media@==1.0.1, azure-mgmt-monitor@==0.5.2, azure-mgmt-msi@==0.2.0, azure-mgmt-network@==2.7.0, azure-mgmt-notificationhubs@==2.1.0, azure-mgmt-nspkg@==3.0.2, azure-mgmt-policyinsights@==0.1.0, azure-mgmt-powerbiembedded@==2.0.0, azure-mgmt-rdbms@==1.9.0, azure-mgmt-recoveryservices@==0.3.0, azure-mgmt-recoveryservicesbackup@==0.3.0, azure-mgmt-redis@==5.0.0, azure-mgmt-relay@==0.1.0, azure-mgmt-reservations@==0.2.1, azure-mgmt-resource@==2.2.0, azure-mgmt-scheduler@==2.0.0, azure-mgmt-search@==2.1.0, azure-mgmt-servicebus@==0.5.3, azure-mgmt-servicefabric@==0.2.0, azure-mgmt-signalr@==0.1.1, azure-mgmt-sql@==0.9.1, azure-mgmt-storage@==2.0.0, azure-mgmt-subscription@==0.2.0, azure-mgmt-trafficmanager@==0.50.0, azure-mgmt-web@==0.35.0, azure-nspkg@==3.0.2, azure-servicebus@==0.21.1, azure-servicefabric@==6.3.0.0, azure-servicemanagement-legacy@==0.20.7, azure-storage-blob@==1.5.0, azure-storage-common@==1.4.2, azure-storage-file@==1.4.0, azure-storage-queue@==1.4.0, blinker@==1.7.0, cachetools@==5.3.2, certifi@==2024.2.2, cffi@==1.16.0, charset-normalizer@==3.3.2, click@==8.1.7, contourpy@==1.2.0, cryptography@==42.0.3, cv@==1.0.0, cycler@==0.12.1, distro@==1.9.0, exceptiongroup@==1.2.0, filelock@==3.13.1, Flask@==3.0.2, Flask-Sockets@==0.2.1, flatbuffers@==23.5.26, fonttools@==4.49.0, frozenlist@==1.4.1, fsspec@==2024.2.0, gast@==0.5.4, gevent@==24.2.1, gevent-websocket@==0.10.1, google-auth@==2.28.0, google-auth-oauthlib@==1.2.0, google-pasta@==0.2.0, greenlet@==3.0.3, grpcio@==1.60.1, h11@==0.14.0, h5py@==3.10.0, httpcore@==1.0.3, httpx@==0.26.0, idna@==3.6, importlib-metadata@==7.0.1, importlib-resources@==6.1.1, isodate@==0.6.1, itsdangerous@==2.1.2, Jinja2@==3.1.3, keras@==2.15.0, kiwisolver@==1.4.5, libclang@==16.0.6, Markdown@==3.5.2, MarkupSafe@==2.1.5, matplotlib@==3.8.3, ml-dtypes@==0.2.0, mpmath@==1.3.0, msal@==1.26.0, msrest@==0.7.1, msrestazure@==0.6.4, multidict@==6.0.5, networkx@==3.2.1, ngrok@==1.0.0, numpy@==1.26.4, oauthlib@==3.2.2, openai@==1.12.0, opencv-contrib-python@==4.9.0.80, opencv-python@==4.9.0.80, opt-einsum@==3.3.0, packaging@==23.2, pandas@==2.2.0, pillow@==10.2.0, protobuf@==3.20.3, psutil@==5.9.8, py-cpuinfo@==9.0.0, pyasn1@==0.5.1, pyasn1-modules@==0.3.0, pycparser@==2.21, pydantic@==2.6.1, pydantic_core@==2.16.2, PyJWT@==2.8.0, pyparsing@==3.1.1, python-dateutil@==2.8.2, python-dotenv@==1.0.1, pytz@==2024.1, PyYAML@==6.0.1, requests@==2.31.0, requests-oauthlib@==1.3.1, rsa@==4.9, scipy@==1.12.0, seaborn@==0.13.2, six@==1.16.0, sniffio@==1.3.0, sympy@==1.12, tensorboard@==2.15.2, tensorboard-data-server@==0.7.2, tensorflow@==2.15.0, tensorflow-estimator@==2.15.0, tensorflow-io-gcs-filesystem@==0.36.0, termcolor@==2.4.0, thop@==0.1.1.post2209072238, torch@==2.2.0, torchvision@==0.17.0, tqdm@==4.66.2, twilio@==8.13.0, typing_extensions@==4.9.0, tzdata@==2024.1, ultralytics@==8.1.15, urllib3@==2.2.0, Werkzeug@==3.0.1, wrapt@==1.14.1, yarl@==1.9.4, zipp@==3.17.0, zope.event@==5.0, zope.interface@==6.2 - package.json: twilio-cli@^5.18.0 - requirements.txt: absl-py@==2.1.0, adal@==1.2.7, aiohttp@==3.9.3, aiohttp-retry@==2.8.3, aiosignal@==1.3.1, annotated-types@==0.6.0, anyio@==4.2.0, astunparse@==1.6.3, async-timeout@==4.0.3, attrs@==23.2.0, azure@==4.0.0, azure-ai-formrecognizer@==3.3.2, azure-ai-vision-imageanalysis@==1.0.0b2, azure-applicationinsights@==0.1.1, azure-batch@==4.1.3, azure-cognitiveservices-vision-computervision@==0.9.0, azure-cognitiveservices-vision-contentmoderator@==1.0.0, azure-cognitiveservices-vision-customvision@==3.1.0, azure-common@==1.1.28, azure-core@==1.30.0, azure-cosmosdb-nspkg@==2.0.2, azure-cosmosdb-table@==1.0.6, azure-datalake-store@==0.0.53, azure-eventgrid@==1.3.0, azure-graphrbac@==0.40.0, azure-keyvault@==1.1.0, azure-loganalytics@==0.1.1, azure-mgmt@==4.0.0, azure-mgmt-advisor@==1.0.1, azure-mgmt-applicationinsights@==0.1.1, azure-mgmt-authorization@==0.50.0, azure-mgmt-batch@==5.0.1, azure-mgmt-batchai@==2.0.0, azure-mgmt-billing@==0.2.0, azure-mgmt-cdn@==3.1.0, azure-mgmt-cognitiveservices@==3.0.0, azure-mgmt-commerce@==1.0.1, azure-mgmt-compute@==4.6.2, azure-mgmt-consumption@==2.0.0, azure-mgmt-containerinstance@==1.5.0, azure-mgmt-containerregistry@==2.8.0, azure-mgmt-containerservice@==4.4.0, azure-mgmt-cosmosdb@==0.4.1, azure-mgmt-datafactory@==0.6.0, azure-mgmt-datalake-analytics@==0.6.0, azure-mgmt-datalake-nspkg@==3.0.1, azure-mgmt-datalake-store@==0.5.0, azure-mgmt-datamigration@==1.0.0, azure-mgmt-devspaces@==0.1.0, azure-mgmt-devtestlabs@==2.2.0, azure-mgmt-dns@==2.1.0, azure-mgmt-eventgrid@==1.0.0, azure-mgmt-eventhub@==2.6.0, azure-mgmt-hanaonazure@==0.1.1, azure-mgmt-iotcentral@==0.1.0, azure-mgmt-iothub@==0.5.0, azure-mgmt-iothubprovisioningservices@==0.2.0, azure-mgmt-keyvault@==1.1.0, azure-mgmt-loganalytics@==0.2.0, azure-mgmt-logic@==3.0.0, azure-mgmt-machinelearningcompute@==0.4.1, azure-mgmt-managementgroups@==0.1.0, azure-mgmt-managementpartner@==0.1.1, azure-mgmt-maps@==0.1.0, azure-mgmt-marketplaceordering@==0.1.0, azure-mgmt-media@==1.0.1, azure-mgmt-monitor@==0.5.2, azure-mgmt-msi@==0.2.0, azure-mgmt-network@==2.7.0, azure-mgmt-notificationhubs@==2.1.0, azure-mgmt-nspkg@==3.0.2, azure-mgmt-policyinsights@==0.1.0, azure-mgmt-powerbiembedded@==2.0.0, azure-mgmt-rdbms@==1.9.0, azure-mgmt-recoveryservices@==0.3.0, azure-mgmt-recoveryservicesbackup@==0.3.0, azure-mgmt-redis@==5.0.0, azure-mgmt-relay@==0.1.0, azure-mgmt-reservations@==0.2.1, azure-mgmt-resource@==2.2.0, azure-mgmt-scheduler@==2.0.0, azure-mgmt-search@==2.1.0, azure-mgmt-servicebus@==0.5.3, azure-mgmt-servicefabric@==0.2.0, azure-mgmt-signalr@==0.1.1, azure-mgmt-sql@==0.9.1, azure-mgmt-storage@==2.0.0, azure-mgmt-subscription@==0.2.0, azure-mgmt-trafficmanager@==0.50.0, azure-mgmt-web@==0.35.0, azure-nspkg@==3.0.2, azure-servicebus@==0.21.1, azure-servicefabric@==6.3.0.0, azure-servicemanagement-legacy@==0.20.7, azure-storage-blob@==1.5.0, azure-storage-common@==1.4.2, azure-storage-file@==1.4.0, azure-storage-queue@==1.4.0, blinker@==1.7.0, cachetools@==5.3.2, certifi@==2024.2.2, cffi@==1.16.0, charset-normalizer@==3.3.2, click@==8.1.7, contourpy@==1.2.0, cryptography@==42.0.3, cv@==1.0.0, cycler@==0.12.1, distro@==1.9.0, exceptiongroup@==1.2.0, filelock@==3.13.1, Flask@==3.0.2, Flask-Sockets@==0.2.1, flatbuffers@==23.5.26, fonttools@==4.49.0, frozenlist@==1.4.1, fsspec@==2024.2.0, gast@==0.5.4, gevent@==24.2.1, gevent-websocket@==0.10.1, google-auth@==2.28.0, google-auth-oauthlib@==1.2.0, google-pasta@==0.2.0, greenlet@==3.0.3, grpcio@==1.60.1, h11@==0.14.0, h5py@==3.10.0, httpcore@==1.0.3, httpx@==0.26.0, idna@==3.6, importlib-metadata@==7.0.1, importlib-resources@==6.1.1, isodate@==0.6.1, itsdangerous@==2.1.2, Jinja2@==3.1.3, keras@==2.15.0, kiwisolver@==1.4.5, libclang@==16.0.6, Markdown@==3.5.2, MarkupSafe@==2.1.5, matplotlib@==3.8.3, ml-dtypes@==0.2.0, mpmath@==1.3.0, msal@==1.26.0, msrest@==0.7.1, msrestazure@==0.6.4, multidict@==6.0.5, networkx@==3.2.1, ngrok@==1.0.0, numpy@==1.26.4, oauthlib@==3.2.2, openai@==1.12.0, opencv-contrib-python@==4.9.0.80, opencv-python@==4.9.0.80, opt-einsum@==3.3.0, packaging@==23.2, pandas@==2.2.0, pillow@==10.2.0, protobuf@==3.20.3, psutil@==5.9.8, py-cpuinfo@==9.0.0, pyasn1@==0.5.1, pyasn1-modules@==0.3.0, pycparser@==2.21, pydantic@==2.6.1, pydantic_core@==2.16.2, PyJWT@==2.8.0, pyparsing@==3.1.1, python-dateutil@==2.8.2, python-dotenv@==1.0.1, pytz@==2024.1, PyYAML@==6.0.1, requests@==2.31.0, requests-oauthlib@==1.3.1, rsa@==4.9, scipy@==1.12.0, seaborn@==0.13.2, six@==1.16.0, sniffio@==1.3.0, sympy@==1.12, tensorboard@==2.15.2, tensorboard-data-server@==0.7.2, tensorflow@==2.15.0, tensorflow-estimator@==2.15.0, tensorflow-io-gcs-filesystem@==0.36.0, termcolor@==2.4.0, thop@==0.1.1.post2209072238, torch@==2.2.0, torchvision@==0.17.0, tqdm@==4.66.2, twilio@==8.13.0, typing_extensions@==4.9.0, tzdata@==2024.1, ultralytics@==8.1.15, urllib3@==2.2.0, Werkzeug@==3.0.1, wrapt@==1.14.1, yarl@==1.9.4, zipp@==3.17.0, zope.event@==5.0, zope.interface@==6.2 ### Recent commits (newest first) - Update README.md - Update README.md - Update README.md - Update README.md - Merge branch 'main' of https://github.com/LiuLawrence45/treehacks - prettify beginning - Update README.md - app demo ready (draft 1) - thread freezing after prompt call - prompting for payload - updated; but need to stop repeat subprocesses - finished model prompting with relevance - new prompt - model prompting - asyncSave and process in new thread. all successful. next steps linking with twilio - cache with futures thread to finish processing - Merge branch 'main' of https://github.com/LiuLawrence45/treehacks - cache gpt summary - Merge branch 'main' of https://github.com/LiuLawrence45/treehacks - call-gpt changes ## Key source files (fetched from GitHub, selected and truncated for size) ### package.json ``` { "dependencies": { "twilio-cli": "^5.18.0" } } ``` ### requirements.txt ``` absl-py==2.1.0 adal==1.2.7 aiohttp==3.9.3 aiohttp-retry==2.8.3 aiosignal==1.3.1 annotated-types==0.6.0 anyio==4.2.0 astunparse==1.6.3 async-timeout==4.0.3 attrs==23.2.0 azure==4.0.0 azure-ai-formrecognizer==3.3.2 azure-ai-vision-imageanalysis==1.0.0b2 azure-applicationinsights==0.1.1 azure-batch==4.1.3 azure-cognitiveservices-vision-computervision==0.9.0 azure-cognitiveservices-vision-contentmoderator==1.0.0 azure-cognitiveservices-vision-customvision==3.1.0 azure-common==1.1.28 azure-core==1.30.0 azure-cosmosdb-nspkg==2.0.2 azure-cosmosdb-table==1.0.6 azure-datalake-store==0.0.53 azure-eventgrid==1.3.0 azure-graphrbac==0.40.0 azure-keyvault==1.1.0 azure-loganalytics==0.1.1 azure-mgmt==4.0.0 azure-mgmt-advisor==1.0.1 azure-mgmt-applicationinsights==0.1.1 azure-mgmt-authorization==0.50.0 azure-mgmt-batch==5.0.1 azure-mgmt-batchai==2.0.0 azure-mgmt-billing==0.2.0 azure-mgmt-cdn==3.1.0 azure-mgmt-cognitiveservices==3.0.0 azure-mgmt-commerce==1.0.1 azure-mgmt-compute==4.6.2 azure-mgmt-consumption==2.0.0 azure-mgmt-containerinstance==1.5.0 azure-mgmt-containerregistry==2.8.0 azure-mgmt-containerservice==4.4.0 azure-mgmt-cosmosdb==0.4.1 azure-mgmt-datafactory==0.6.0 azure-mgmt-datalake-analytics==0.6.0 azure-mgmt-datalake-nspkg==3.0.1 azure-mgmt-datalake-store==0.5.0 azure-mgmt-datamigration==1.0.0 azure-mgmt-devspaces==0.1.0 azure-mgmt-devtestlabs==2.2.0 azure-mgmt-dns==2.1.0 azure-mgmt-eventgrid==1.0.0 azure-mgmt-eventhub==2.6.0 azure-mgmt-hanaonazure==0.1.1 azure-mgmt-iotcentral==0.1.0 azure-mgmt-iothub==0.5.0 azure-mgmt-iothubprovisioningservices==0.2.0 azure-mgmt-keyvault==1.1.0 azure-mgmt-loganalytics==0.2.0 azure-mgmt-logic==3.0.0 azure-mgmt-machinelearningcompute==0.4.1 azure-mgmt-managementgroups==0.1.0 azure-mgmt-managementpartner==0.1.1 azure-mgmt-maps==0.1.0 azure-mgmt-marketplaceordering==0.1.0 azure-mgmt-media==1.0.1 azure-mgmt-monitor==0.5.2 azure-mgmt-msi==0.2.0 azure-mgmt-network==2.7.0 azure-mgmt-notificationhubs==2.1.0 azure-mgmt-nspkg==3.0.2 azure-mgmt-policyinsights==0.1.0 azure-mgmt-powerbiembedded==2.0.0 azure-mgmt-rdbms==1.9.0 azure-mgmt-recoveryservices==0.3.0 azure-mgmt-recoveryservicesbackup==0.3.0 azure-mgmt-redis==5.0.0 azure-mgmt-relay==0.1.0 azure-mgmt-reservations==0.2.1 azure-mgmt-resource==2.2.0 azure-mgmt-scheduler==2.0.0 azure-mgmt-search==2.1.0 azure-mgmt-servicebus==0.5.3 azure-mgmt-servicefabric==0.2.0 azure-mgmt-signalr==0.1.1 azure-mgmt-sql==0.9.1 azure-mgmt-storage==2.0.0 azure-mgmt-subscription==0.2.0 azure-mgmt-trafficmanager==0.50.0 azure-mgmt-web==0.35.0 azure-nspkg==3.0.2 azure-servicebus==0.21.1 azure-servicefabric==6.3.0.0 azure-servicemanagement-legacy==0.20.7 azure-storage-blob==1.5.0 azure-storage-common==1.4.2 azure-storage-file==1.4.0 azure-storage-queue==1.4.0 blinker==1.7.0 cachetools==5.3.2 certifi==2024.2.2 cffi==1.16.0 charset-normalizer==3.3.2 click==8.1.7 contourpy==1.2.0 cryptography==42.0.3 cv==1.0.0 cycler==0.12.1 distro==1.9.0 exceptiongroup==1.2.0 filelock==3.13.1 Flask==3.0.2 Flask-Sockets==0.2.1 flatbuffers==23.5.26 fonttools==4.49.0 frozenlist==1.4.1 fsspec==2024.2.0 gast==0.5.4 gevent==24.2.1 gevent-websocket==0.10.1 google-auth==2.28.0 google-auth-oauthlib==1.2.0 google-pasta==0.2.0 greenlet==3.0.3 grpcio==1.60.1 h11==0.14.0 h5py==3.10.0 httpcore==1.0.3 httpx==0.26.0 idna==3.6 importlib-metadata==7.0.1 importlib-resources==6.1.1 isodate==0.6.1 itsdangerous==2.1.2 Jinja2==3.1.3 keras==2.15.0 kiwisolver==1.4.5 libclang==16.0.6 Markdown==3.5.2 MarkupSafe==2.1.5 matplotlib==3.8.3 ml-dtypes==0.2.0 mpmath==1.3.0 msal==1.26.0 msrest==0.7.1 msrestazure==0.6.4 multidict==6.0.5 networkx==3.2.1 ngrok==1.0.0 numpy==1.26.4 oauthlib==3.2.2 openai==1.12.0 opencv-contrib-python==4.9.0.80 opencv-python==4.9.0.80 opt-einsum==3.3.0 packaging==23.2 pandas==2.2.0 pillow==10.2.0 protobuf==3.20.3 psutil==5.9.8 py-cpuinfo==9.0.0 pyasn1==0.5.1 pyasn1-modules==0.3.0 pycparser==2.21 pydantic==2.6.1 pydantic_core==2.16.2 PyJWT==2.8.0 pyparsing==3.1.1 python-dateutil==2.8.2 python-dotenv==1.0.1 pytz==2024.1 PyYAML==6.0.1 requests==2.31.0 requests-oauthlib==1.3.1 rsa==4.9 scipy==1.12.0 seaborn==0.13.2 six==1.16.0 sniffio==1.3.0 sympy==1.12 tensorboard==2.15.2 tensorboard-data-server==0.7.2 tensorflow==2.15.0 tensorflow-estimator==2.15.0 tensorflow-io-gcs-filesystem==0.36.0 termcolor==2.4.0 thop==0.1.1.post2209072238 torch==2.2.0 torchvision==0.17.0 tqdm==4.66.2 twilio==8.13.0 typing_extensions==4.9.0 tzdata==2024.1 ultralytics==8.1.15 urllib3==2.2.0 Werkzeug==3.0.1 wrapt==1.14.1 yarl==1.9.4 zipp==3.17.0 zope.event==5.0 zope.interface==6.2 ``` ### call-gpt/package.json ``` { "name": "genai-phone", "version": "1.1.0", "description": "", "main": "index.js", "scripts": { "inbound": "node ./scripts/inbound-call.js", "outbound": "node ./scripts/outbound-call.js", "test": "jest", "dev": "nodemon app.js", "start": "node app.js" }, "keywords": [], "author": "Charlie Weems", "license": "MIT", "dependencies": { "@deepgram/sdk": "^2.4.0", "colors": "^1.4.0", "dotenv": "^16.3.1", "express": "^4.18.2", "express-ws": "^5.0.2", "node-fetch": "^2.7.0", "openai": "^4.20.1", "twilio": "^4.19.3", "uuid": "^9.0.1", "wavefile": "^11.0.0" }, "devDependencies": { "@flydotio/dockerfile": "^0.4.11", "jest": "^29.7.0", "nodemon": "^3.0.2" } } ``` ### call-gpt/scripts/requirements.txt ``` absl-py==2.1.0 adal==1.2.7 aiohttp==3.9.3 aiohttp-retry==2.8.3 aiosignal==1.3.1 annotated-types==0.6.0 anyio==4.2.0 astunparse==1.6.3 async-timeout==4.0.3 attrs==23.2.0 azure==4.0.0 azure-ai-formrecognizer==3.3.2 azure-ai-vision-imageanalysis==1.0.0b2 azure-applicationinsights==0.1.1 azure-batch==4.1.3 azure-cognitiveservices-vision-computervision==0.9.0 azure-cognitiveservices-vision-contentmoderator==1.0.0 azure-cognitiveservices-vision-customvision==3.1.0 azure-common==1.1.28 azure-core==1.30.0 azure-cosmosdb-nspkg==2.0.2 azure-cosmosdb-table==1.0.6 azure-datalake-store==0.0.53 azure-eventgrid==1.3.0 azure-graphrbac==0.40.0 azure-keyvault==1.1.0 azure-loganalytics==0.1.1 azure-mgmt==4.0.0 azure-mgmt-advisor==1.0.1 azure-mgmt-applicationinsights==0.1.1 azure-mgmt-authorization==0.50.0 azure-mgmt-batch==5.0.1 azure-mgmt-batchai==2.0.0 azure-mgmt-billing==0.2.0 azure-mgmt-cdn==3.1.0 azure-mgmt-cognitiveservices==3.0.0 azure-mgmt-commerce==1.0.1 azure-mgmt-compute==4.6.2 azure-mgmt-consumption==2.0.0 azure-mgmt-containerinstance==1.5.0 azure-mgmt-containerregistry==2.8.0 azure-mgmt-containerservice==4.4.0 azure-mgmt-cosmosdb==0.4.1 azure-mgmt-datafactory==0.6.0 azure-mgmt-datalake-analytics==0.6.0 azure-mgmt-datalake-nspkg==3.0.1 azure-mgmt-datalake-store==0.5.0 azure-mgmt-datamigration==1.0.0 azure-mgmt-devspaces==0.1.0 azure-mgmt-devtestlabs==2.2.0 azure-mgmt-dns==2.1.0 azure-mgmt-eventgrid==1.0.0 azure-mgmt-eventhub==2.6.0 azure-mgmt-hanaonazure==0.1.1 azure-mgmt-iotcentral==0.1.0 azure-mgmt-iothub==0.5.0 azure-mgmt-iothubprovisioningservices==0.2.0 azure-mgmt-keyvault==1.1.0 azure-mgmt-loganalytics==0.2.0 azure-mgmt-logic==3.0.0 azure-mgmt-machinelearningcompute==0.4.1 azure-mgmt-managementgroups==0.1.0 azure-mgmt-managementpartner==0.1.1 azure-mgmt-maps==0.1.0 azure-mgmt-marketplaceordering==0.1.0 azure-mgmt-media==1.0.1 azure-mgmt-monitor==0.5.2 azure-mgmt-msi==0.2.0 azure-mgmt-network==2.7.0 azure-mgmt-notificationhubs==2.1.0 azure-mgmt-nspkg==3.0.2 azure-mgmt-policyinsights==0.1.0 azure-mgmt-powerbiembedded==2.0.0 azure-mgmt-rdbms==1.9.0 azure-mgmt-recoveryservices==0.3.0 azure-mgmt-recoveryservicesbackup==0.3.0 azure-mgmt-redis==5.0.0 azure-mgmt-relay==0.1.0 azure-mgmt-reservations==0.2.1 azure-mgmt-resource==2.2.0 azure-mgmt-scheduler==2.0.0 azure-mgmt-search==2.1.0 azure-mgmt-servicebus==0.5.3 azure-mgmt-servicefabric==0.2.0 azure-mgmt-signalr==0.1.1 azure-mgmt-sql==0.9.1 azure-mgmt-storage==2.0.0 azure-mgmt-subscription==0.2.0 azure-mgmt-trafficmanager==0.50.0 azure-mgmt-web==0.35.0 azure-nspkg==3.0.2 azure-servicebus==0.21.1 azure-servicefabric==6.3.0.0 azure-servicemanagement-legacy==0.20.7 azure-storage-blob==1.5.0 azure-storage-common==1.4.2 azure-storage-file==1.4.0 azure-storage-queue==1.4.0 blinker==1.7.0 cachetools==5.3.2 certifi==2024.2.2 cffi==1.16.0 charset-normalizer==3.3.2 click==8.1.7 contourpy==1.2.0 cryptography==42.0.3 cv==1.0.0 cycler==0.12.1 distro==1.9.0 exceptiongroup==1.2.0 filelock==3.13.1 Flask==3.0.2 Flask-Sockets==0.2.1 flatbuffers==23.5.26 fonttools==4.49.0 frozenlist==1.4.1 fsspec==2024.2.0 gast==0.5.4 gevent==24.2.1 gevent-websocket==0.10.1 google-auth==2.28.0 google-auth-oauthlib==1.2.0 google-pasta==0.2.0 greenlet==3.0.3 grpcio==1.60.1 h11==0.14.0 h5py==3.10.0 httpcore==1.0.3 httpx==0.26.0 idna==3.6 importlib-metadata==7.0.1 importlib-resources==6.1.1 isodate==0.6.1 itsdangerous==2.1.2 Jinja2==3.1.3 keras==2.15.0 kiwisolver==1.4.5 libclang==16.0.6 Markdown==3.5.2 MarkupSafe==2.1.5 matplotlib==3.8.3 ml-dtypes==0.2.0 mpmath==1.3.0 msal==1.26.0 msrest==0.7.1 msrestazure==0.6.4 multidict==6.0.5 networkx==3.2.1 ngrok==1.0.0 numpy==1.26.4 oauthlib==3.2.2 openai==1.12.0 opencv-contrib-python==4.9.0.80 opencv-python==4.9.0.80 opt-einsum==3.3.0 packaging==23.2 pandas==2.2.0 pillow==10.2.0 protobuf==3.20.3 psutil==5.9.8 py-cpuinfo==9.0.0 pyasn1==0.5.1 pyasn1-modules==0.3.0 pycparser==2.21 pydantic==2.6.1 pydantic_core==2.16.2 PyJWT==2.8.0 pyparsing==3.1.1 python-dateutil==2.8.2 python-dotenv==1.0.1 pytz==2024.1 PyYAML==6.0.1 requests==2.31.0 requests-oauthlib==1.3.1 rsa==4.9 scipy==1.12.0 seaborn==0.13.2 six==1.16.0 sniffio==1.3.0 sympy==1.12 tensorboard==2.15.2 tensorboard-data-server==0.7.2 tensorflow==2.15.0 tensorflow-estimator==2.15.0 tensorflow-io-gcs-filesystem==0.36.0 termcolor==2.4.0 thop==0.1.1.post2209072238 torch==2.2.0 torchvision==0.17.0 tqdm==4.66.2 twilio==8.13.0 typing_extensions==4.9.0 tzdata==2024.1 ultralytics==8.1.15 urllib3==2.2.0 Werkzeug==3.0.1 wrapt==1.14.1 yarl==1.9.4 zipp==3.17.0 zope.event==5.0 zope.interface==6.2 ``` ### call-gpt/app.js ```javascript require("dotenv").config(); const express = require("express"); const ExpressWs = require("express-ws"); const colors = require('colors'); const { GptService } = require("./services/gpt-service"); const { StreamService } = require("./services/stream-service"); const { TranscriptionService } = require("./services/transcription-service"); const { TextToSpeechService } = require("./services/tts-service"); const app = express(); ExpressWs(app); const PORT = process.env.PORT || 3000; // app.post("/incoming", (req, res) => { // res.status(200); // res.type("text/xml"); // res.end(` // // // // // // `); // }); app.post("/incoming", (req, res) => { res.status(200); res.type("text/xml"); res.end(` `); }); app.ws("/connection", (ws, req) => { ws.on("error", console.error); console.log((console.error).red) // Filled in from start message let streamSid; const gptService = new GptService(); const streamService = new StreamService(ws); const transcriptionService = new TranscriptionService(); const ttsService = new TextToSpeechService({}); let marks = [] let interactionCount = 0 // Incoming from MediaStream ws.on("message", async function message(data) { const msg = JSON.parse(data); if (msg.event === "start") { streamSid = msg.start.streamSid; streamService.setStreamSid(streamSid); console.log(`Twilio -> Starting Media Stream for ${streamSid}`.underline.red); try { const response = await fetch('http://127.0.0.1:5000/data'); // Replace with your actual URL if (!response.ok) { throw new Error('Network response was not ok'); } const data = await response.json(); const initialMessage = data.assistantPrompt; // Replace fieldName with the actual field you need // Use the fetched initialMessage in your ttsService.generate call ttsService.generate({ partialResponseIndex: null, partialResponse: initialMessage }, 1); } catch (error) { console.error('Failed to fetch initial message:', error); // Handle error, possibly send an error message back via WebSocket } } else if (msg.event === "media") { transcriptionService.send(msg.media.payload); } else if (msg.event === "mark") { const label = msg.mark.name; console.log(`Twilio -> Audio completed mark (${msg.sequenceNumber}): ${label}`.red) marks = marks.filter(m => m !== msg.mark.name) } else if (msg.event === "stop") { console.log(`Twilio -> Media stream ${streamSid} ended.`.underline.red) } }); transcriptionService.on("utterance", async (text) => { // This is a bit of a hack to filter out empty utterances if(marks.length > 0 && text?.length > 5) { console.log("Twilio -> Interruption, Clearing stream".red) ws.send( JSON.stringify({ streamSid, event: "clear", }) ); } }); transcriptionService.on("transcription", async (text) => { if (!text) { return; } console.log(`Interaction ${interactionCount} – STT -> GPT: ${text}`.yellow); gptService.completion(text, interactionCount); interactionCount += 1; }); gptService.on('gptreply', async (gptReply, icount) => { console.log(`Interaction ${icount}: GPT -> TTS: ${gptReply.partialResponse}`.green ) ttsService.generate(gptReply, icount); }); ttsService.on("speech", (responseIndex, audio, label, icount) => { console.log(`Interaction ${icount}: TTS -> TWILIO: ${label}`.blue); streamService.buffer(responseIndex, audio); }); streamService.on('audiosent', (markLabel) => { marks.push(markLabel); }) }); app.listen(PORT); console.log(`Server running on port ${PORT}`); ``` ### processing/cv/shared_resources.py ```python from threading import Lock # Initialize a lock and a flag execution_lock = Lock() has_executed = False ``` ### processing/cv/yolov8.py ```python from ultralytics import YOLO model = YOLO('processing/cv/smallest-YOLO.pt') results = model(source = 0, show = True, conf = 0.3, save = True) ``` ### call-gpt/test/placeOrder.test.js ```javascript const placeOrder = require('../functions/placeOrder'); test("Expect placeOrder to return an object with a price and order number", () => { const order = JSON.parse(placeOrder({model: 'airpods pro', quantity: 10})) expect(order).toHaveProperty('orderNumber') expect(order).toHaveProperty('price') }) ``` ### call-gpt/functions/checkPrice.js ```javascript function checkPrice(functionArgs) { let model = functionArgs.model; console.log("GPT -> called checkPrice function"); if (model?.toLowerCase().includes("pro")) { return JSON.stringify({ price: 249 }); } else if (model?.toLowerCase().includes("max")) { return JSON.stringify({ price: 549 }); } else { return JSON.stringify({ price: 149 }); } } module.exports = checkPrice; ``` ### call-gpt/functions/checkInventory.js ```javascript function checkInventory(functionArgs) { const model = functionArgs.model; console.log("GPT -> called checkInventory function"); if (model?.toLowerCase().includes("pro")) { return JSON.stringify({ stock: 10 }); } else if (model?.toLowerCase().includes("max")) { return JSON.stringify({ stock: 0 }); } else { return JSON.stringify({ stock: 100 }); } } module.exports = checkInventory; ``` [28 more indexed source files omitted to keep this export small. The full file list is in the Codebase structure section above.]