# Project export: ephemeral
This document was generated by HackStack to give an AI agent context about a hackathon project. Sections are labeled with their provenance; content marked as truncated was cut to keep this document small.
## Project metadata
- Hackathon: TreeHacks 2024
- Tagline: always helping but you never notice
- Devpost: https://devpost.com/software/invisible-me
- GitHub: https://github.com/pgasawa/ephemeral
- Video: https://www.youtube.com/embed/thlFX-JA9cE?enablejsapi=1&hl=en_US&rel=0&start=&version=3&wmode=transparent
- Result: winner (Together.ai: Best Use of Together API ($2.5k Credits + 1x AirPods Max))
- Team: 4 GitHub contributor(s) — pgasawa (69 commits), Jason Ding (67 commits), Ayushi Batwara (52 commits), arvindrajaraman (27 commits)
## Devpost submission (written by the team)
### Inspiration
Have you ever wished you had…another you? This thought has crossed all of our heads countless times as we find ourselves swamped in too many tasks, unable to keep up in meetings as information flies over our head, or wishing we had the feedback of a third perspective. Our goal was to build an autonomous agent that could be that person for you — an AI that learns from your interactions and proactively takes actions, provides answers, offers advice, and more, to give back your time to you.
### What it does
Ephemeral is an autonomous AI agent that interacts with the world primarily through the modality of voice. It can sit in on meetings, calls, anywhere you have your computer out. It’s power is the ability to take what it hears and proactively carry out repetitive actions for you such as be a real-time AI assistant in meetings, draft emails directly in your Google inbox, schedule calendar events and invite attendees, search knowledge corpuses or the web for answers to questions, image generation, and more. Multiple users (in multiple languages!) can use the technology simultaneously through the server/client architecture that efficiently handles multiprocessing.
### How we built it
Languages: Python ∙ JavaScript ∙ HTML ∙ CSS Frameworks and Tools: React.js ∙ PyTorch ∙ Flask ∙ LangChain ∙ OpenAI ∙ TogetherAI ∙ Many More 1. Audio to Text We utilized OpenAI’s Whisper model and the python speech_recognition library to convert audio in real-time to text that can be used by downstream functions. 2. Client → Server via Socket Connection We use socket connections between the client and server to pass over the textual query to the server for it to determine a particular action and action parameters. The socket connections enable us to support multiprocessing as multiple clients can connect to the server simultaneously while performing concurrent logic (such as real-time, personalized agentic actions during a meeting). 3. Neural Network Action Classifier We trained a neural network from scratch to handle the multi-class classification problem that is going from text to action (or none at all). Because the agent is constantly listening, we need a way to efficiently and accurately determine if each transcribed chunk necessitates a particular action (if so, which?) or none at all (most commonly). We generated data for this task utilizing data augmentation sources such as ChatGPT (web). 4. LLM Logic: Query → Function Parameters We use in-context learning via few-shot prompting and RAG to query the LLM for various agentic tasks. We built a RAG pipeline over the conversation history and past related, relevant meetings for context. The agentic tasks take in function parameters, which are generated by the LLM. 5. Server → Client Parameters via Socket Connection We pass back the function parameters as a JSON object from the server socket to the client. 6. Client Side Handler: API Call A client side handler receives a JSON object that includes which action (if any) was chosen by the Action Planner in step 3, then passes control to the appropriate handler function which handles authorizations and makes API calls to various services such as Google’s Gmail Client, Calendar API, text-to-speech, and more. 7. Client Action Notifications → File (monitored by Flask REST API) After the completion of each action, the client server writes the results of the action down to a file which is then read by the React Web App to display ephemeral updates on a UI, in addition to suggestions/answers/discussion questions/advice on a polling basis. 8. React Web App and Ephemeral UI To communicate updates to the user (specifically notifications and suggestions from Ephemeral), we poll the Flask API for any updates and serve it to the user via a React web app. Our app is called Ephemeral because we show information minimally yet expressively to the user, in order to promote focus in meetings.
### Challenges we ran into
We spent a significant amount of our time optimizing for lower latency, which is important for a real-time consumer-facing application. In order to do this, we created sockets to enable 2-way communication between the client(s) and the server. Then, in order to support concurrent and parallel execution, we added support for multithreading on the server-side. Choosing action spaces that can be precisely articulated enough in text such that a language model can carry out actions was a troublesome task. We went through a lot of experimentation on different tasks to figure out which would have the highest value to humans and also the highest correctness guarantee.
### Accomplishments we're proud of
Successful integration of numerous OSS and closed source models into a working product, including Llama-70B-Chat, Mistral-7B, Stable Diffusion 2.1, OpenAI TTS, OpenAI Whisper, and more. Integration of real actions that we can see ourselves directly using was very cool to see go from a hypothetical to a reality. The potential for impact of this general workflow in various domains is not lost on us, as while the general productivity purpose stands, there are many more specific gains to be seen in fields such as digital education, telemedicine, and more!
### What we learned
The possibility of powerful autonomous agents to supplement human workflows signals the shift of a new paradigm where more and more our imprecise language can be taken by these programs and turned into real actions on behalf of us.
### What's next
for Ephemeral An agent is only constrained by the size of the action space you give it. We think that Ephemeral has the potential to grow boundlessly as more powerful actions are integrated into its planning capabilities and it returns more of a user’s time to them.
## README (from the GitHub repository)
# To Run:
## Setup
After the first clone, make a virtual environment using ```virtualenv treehacks24-venv```
Before pushing:
- Run ```pip freeze >> requirements.txt```
After pulling:
- Run ```pip install -r requirements.txt```
### Example .env configuration
```
TOGETHER_API = "secret"
USER_EMAIL = "pgasawa@berkeley.edu"
HOST = "1.1.1.1"
SERVER_LISTEN_PORT = "12342"
SERVER_SEND_PORT = "12343"
NAME = "Parth"
OPENAI_API_KEY = "secret"
```
You can find your host IP address by running ```ifconfig | grep "inet "```. You will need to generate a Together AI API Key and an OpenAI API Key.
### Google Cloud Setup
Create a Google Cloud project with email and calendar scopes. Add the emails of accounts you want to use, and then download credentials.json and put it into the ```client``` folder.
## Run
1. Run ```flask --app main run``` from the ```flask``` folder
2. Run ```npm start``` from the ```frontend``` folder
3. Run ```python server.py``` from the ```server``` folder
3. Run ```python client.py``` from the ```client``` folder
3. Run ```python client_ping.py``` from the ```client``` folder
# Information (from [Devpost](https://devpost.com/software/invisible-me))
## Demo Video
[](https://www.youtube.com/watch?v=thlFX-JA9cE)
## Inspiration
Have you ever wished you had…another you?
This thought has crossed all of our heads countless times as we find ourselves swamped in too many tasks, unable to keep up in meetings as information flies over our head, or wishing we had the feedback of a third perspective.
Our goal was to build an **autonomous agent** that could be that person for you — an AI that learns from your interactions and proactively takes **actions**, provides **answers**, offers advice, and more, to give back your time to you.
## What it does
Ephemeral is an **autonomous AI agent** that interacts with the world primarily through the modality of **voice**. It can sit in on meetings, calls, anywhere you have your computer out.
It’s power is the ability to take what it hears and proactively carry out repetitive actions for you such as be a real-time AI assistant in meetings, draft emails directly in your Google inbox, schedule calendar events and invite attendees, search knowledge corpuses or the web for answers to questions, image generation, and more.
Multiple users (in multiple languages!) can use the technology simultaneously through the server/client architecture that efficiently handles multiprocessing.
## How we built it
**Languages**: Python ∙ JavaScript ∙ HTML ∙ CSS
**Frameworks and Tools**: React.js ∙ PyTorch ∙ Flask ∙ LangChain ∙ OpenAI ∙ TogetherAI ∙ Many More
### 1. Audio to Text
We utilized OpenAI’s Whisper model and the python speech_recognition library to convert audio in real-time to text that can be used by downstream functions.
### 2. Client → Server via Socket Connection
We use socket connections between the client and server to pass over the textual query to the server for it to determine a particular action and action parameters. The socket connections enable us to support multiprocessing as multiple clients can connect to the server simultaneously while performing concurrent logic (such as real-time, personalized agentic actions during a meeting).
### 3. Neural Network Action Classifier
We trained a neural network from scratch to handle the multi-class classification problem that is going from text to action (or none at all). Because the agent is constantly listening, we need a way to efficiently and accurately determine if each transcribed chunk necessitates a particular action (if so, which?) or none at all (most commonly).
We generated data for this task utilizing data augmentation sources such as ChatGPT (web).
### 4. LLM Logic: Query → Function Parameters
We use in-context learning via few-shot prompting and RAG to query the LLM for various agentic tasks. We built a RAG pipeline over the conversation history and past related, relevant meetings for context. The agentic tasks take in function parameters, which are generated by the LLM.
### 5. Server → Client Parameters via Socket Connection
We pass back the function parameters as a JSON object from the server socket to the client.
### 6. Client Side Handler: API Call
A client side handler receives a JSON object that includes which action (if any) was chosen by the Action Planner in step 3, then passes control to the appropriate handler function which handles authorizations and makes API calls to various services such as Google’s Gmail Client, Calendar API, text-to-speech, and more.
### 7. Client Action Notifications → File (monitored by Flask REST API)
After the completion of each action, the client server writes the results of the action down to a file which is then read by the React Web App to display ephemeral updates on a UI, in addition to suggestions/answers/discussion questions/advice on a polling basis.
### 8. React Web App and Ephemeral UI
To communicate updates to the user (specifically notifications and suggestions from Ephemeral), we poll the Flask API for any updates and serve it to the user via a React web app. Our app is called Ephemeral because we show information minimally yet expressively to the user, in order to promote focus in meetings.
## Challenges we ran into
We spent a significant amount of our time optimizing for lower latency, which is important for a real-time consumer-facing application. In order to do this, we created sockets to enable 2-way communication between the client(s) and the server. Then, in order to support concurrent and parallel execution, we added support for multithreading on the server-side.
Choosing action spaces that can be precisely articulated enough in text such that a language model can carry out actions was a troublesome task. We went through a lot of experimentation on different tasks to figure out which would have the highest value to humans and also the highest correctness guarantee.
## Accomplishments that we're proud of
Successful integration of numerous OSS and closed source models into a working product, including Llama-70B-Chat, Mistral-7B, Stable Diffusion 2.1, OpenAI TTS, OpenAI Whisper, and more.
Integration of real actions that we can see ourselves directly using was very cool to see go from a hypothetical to a reality. The potential for impact of this general workflow in various domains is not lost on us, as while the general productivity purpose stands, there are many more specific gains to be seen in fields such as digital education, telemedicine, and more!
## What we learned
The possibility of powerful autonomous agents to supplement human workflows signals the shift of a new paradigm where more and more our imprecise language can be taken by these programs and turned into real actions on behalf of us.
## What's next for Ephemeral
An agent is only constrained by the size of the action space you give it. We think that Ephemeral has the potential to grow boundlessly as more powerful actions are integrated into its planning capabilities and it returns more of a user’s time to them.
## Detected evidence (automated analysis)
Indexed codebase: 39 recognized source files, 95 KB.
- CSS (language) — detected in the code
- HTML (language) — detected in the code
- Hugging Face (technology) — detected in the code
- JavaScript (language) — detected in the code
- LangChain (technology) — detected in the code
- OpenAI (technology) — detected in the code
- Python (language) — detected in the code
- PyTorch (technology) — detected in the code
- React (technology) — detected in the code
- Flask (technology) — claimed on Devpost, not found in the code
## Codebase structure (from repository index)
### Files (59 of 59)
```
.gitignore
.ipynb_checkpoints/action_classifier-checkpoint.ipynb
action_classifier.ipynb
archive/llm.py
archive/socket_client.py
archive/speech_to_text.py
archive/test.py
client/agents/clarify.py
client/agents/mailing.py
client/agents/scheduling.py
client/client_agent_handler.py
client/client_ping.py
client/client.py
client/speak.py
data/action_classification/clarify.txt
data/action_classification/email.txt
data/action_classification/link.txt
data/action_classification/schedule.txt
data/action_classification/todos.txt
data/action_classification/unknown.txt
flask/main.py
flask/package.json
frontend/.gitignore
frontend/package.json
frontend/public/index.html
frontend/public/manifest.json
frontend/public/robots.txt
frontend/README.md
frontend/src/App.css
frontend/src/App.js
frontend/src/App.test.js
frontend/src/favicon_io/site.webmanifest
frontend/src/index.css
frontend/src/index.js
frontend/src/notifications/CalendarInviteCard.js
frontend/src/notifications/ClarifyCard.js
frontend/src/notifications/EmailDraftCard.js
frontend/src/notifications/Notification.js
frontend/src/notifications/OpenLinkCard.js
frontend/src/notifications/SuggestionCard.js
frontend/src/reportWebVitals.js
frontend/src/setupTests.js
README.md
requirements.txt
server/action_classifier.pt
server/actions/assistant.py
server/actions/clarify_search.py
server/actions/contacts.txt
server/actions/create_email.py
server/actions/get_link.py
server/actions/imageGen.py
server/actions/links.txt
server/actions/rag.py
server/actions/schedule_event.py
server/actions/suggestions.py
server/classifier.py
server/llm.py
server/requirements.txt
server/server.py
```
### Dependencies
- flask/package.json: axios@^1.6.7
- frontend/package.json: @testing-library/jest-dom@^5.17.0, @testing-library/react@^13.4.0, @testing-library/user-event@^13.5.0, axios@^1.6.7, bootstrap@^5.3.2, react@^18.2.0, react-bootstrap@^2.10.1, react-bootstrap-icons@^1.11.3, react-dom@^18.2.0, react-scripts@5.0.1, styled-components@^6.1.8, web-vitals@^2.1.4
- requirements.txt: aiohttp@==3.9.3, aiosignal@==1.3.1, annotated-types@==0.6.0, anyio@==4.2.0, async-timeout@==4.0.3, attrs@==23.2.0, backoff@==2.2.1, beautifulsoup4@==4.12.3, cachetools@==5.3.2, certifi@==2024.2.2, cffi@==1.16.0, chardet@==5.2.0, charset-normalizer@==3.3.2, click@==8.1.7, dataclasses-json@==0.6.4, dataclasses-json-speakeasy@==0.5.11, distro@==1.9.0, emoji@==2.10.1, exceptiongroup@==1.2.0, faiss-cpu@==1.7.4, filelock@==3.13.1, filetype@==1.2.0, frozenlist@==1.4.1, fsspec@==2024.2.0, google-api-core@==2.17.1, google-api-python-client@==2.118.0, google-auth@==2.28.0, google-auth-httplib2@==0.2.0, google-auth-oauthlib@==1.2.0, googleapis-common-protos@==1.62.0, h11@==0.14.0, httpcore@==1.0.3, httplib2@==0.22.0, httpx@==0.26.0, huggingface-hub@==0.20.3, idna@==3.6, Jinja2@==3.1.3, joblib@==1.3.2, jsonpatch@==1.33, jsonpath-python@==1.0.6, jsonpointer@==2.4, langchain@==0.1.7, langchain-community@==0.0.20, langchain-core@==0.1.23, langchain-together@==0.0.2.post1, langdetect@==1.0.9, langsmith@==0.0.87, llvmlite@==0.42.0, lxml@==5.1.0, MarkupSafe@==2.1.5, marshmallow@==3.20.2, more-itertools@==10.2.0, mpmath@==1.3.0, multidict@==6.0.5, mypy-extensions@==1.0.0, networkx@==3.2.1, nltk@==3.8.1, numba@==0.59.0, numpy@==1.26.4, oauthlib@==3.2.2, openai@==1.12.0, openai-whisper@@ git+https://github.com/openai/whisper.git@ba3f3cd54b0e5b8ce1ab3de13e32122d0d5f98ab, packaging@==23.2, pillow@==10.2.0, protobuf@==4.25.3, pyasn1@==0.5.1, pyasn1-modules@==0.3.0, pyaudio, PyAudio@==0.2.14, pycparser@==2.21, pydantic@==2.6.1, pydantic_core@==2.16.2, pyparsing@==3.1.1, python-dateutil@==2.8.2, python-dotenv@==1.0.1, python-iso639@==2024.2.7, python-magic@==0.4.27, PyYAML@==6.0.1, rapidfuzz@==3.6.1, regex@==2023.12.25, requests@==2.31.0, requests-oauthlib@==1.3.1, rsa@==4.9, safetensors@==0.4.2, scikit-learn@==1.4.1.post1, scipy@==1.12.0, sentence-transformers@==2.3.1, sentencepiece@==0.1.99, six@==1.16.0, sniffio@==1.3.0, soundfile@==0.12.1, soupsieve@==2.5, SpeechRecognition@==3.10.1, SQLAlchemy@==2.0.27, sseclient-py@==1.8.0, sympy@==1.12, tabulate@==0.9.0, tenacity@==8.2.3, threadpoolctl@==3.3.0, tiktoken@==0.6.0, together@==0.2.11, tokenizers@==0.15.2, torch@==2.2.0, tqdm@==4.66.2, transformers@==4.37.2, typer@==0.9.0, typing_extensions@==4.9.0, typing-inspect@==0.9.0, unstructured@==0.12.4, unstructured-client@==0.18.0, uritemplate@==4.1.1, urllib3@==2.2.0, whisper@==1.1.10, wrapt@==1.16.0, yarl@==1.9.4
- server/requirements.txt: aiohttp@==3.9.3, aiosignal@==1.3.1, annotated-types@==0.6.0, anyio@==4.2.0, async-timeout@==4.0.3, attrs@==23.2.0, backoff@==2.2.1, beautifulsoup4@==4.12.3, cachetools@==5.3.2, certifi@==2024.2.2, chardet@==5.2.0, charset-normalizer@==3.3.2, click@==8.1.7, dataclasses-json@==0.6.4, dataclasses-json-speakeasy@==0.5.11, emoji@==2.10.1, exceptiongroup@==1.2.0, faiss-cpu@==1.7.4, filelock@==3.13.1, filetype@==1.2.0, frozenlist@==1.4.1, fsspec@==2024.2.0, google-api-core@==2.17.1, google-api-python-client@==2.118.0, google-auth@==2.28.0, google-auth-httplib2@==0.2.0, google-auth-oauthlib@==1.2.0, googleapis-common-protos@==1.62.0, httplib2@==0.22.0, huggingface-hub@==0.20.3, idna@==3.6, Jinja2@==3.1.3, joblib@==1.3.2, jsonpatch@==1.33, jsonpath-python@==1.0.6, jsonpointer@==2.4, langchain@==0.1.7, langchain-community@==0.0.20, langchain-core@==0.1.23, langdetect@==1.0.9, langsmith@==0.0.87, lxml@==5.1.0, MarkupSafe@==2.1.5, marshmallow@==3.20.2, mpmath@==1.3.0, multidict@==6.0.5, mypy-extensions@==1.0.0, networkx@==3.2.1, nltk@==3.8.1, numpy@==1.26.4, oauthlib@==3.2.2, packaging@==23.2, pillow@==10.2.0, protobuf@==4.25.3, pyasn1@==0.5.1, pyasn1-modules@==0.3.0, PyAudio@==0.2.14, pydantic@==2.6.1, pydantic_core@==2.16.2, pyparsing@==3.1.1, python-dateutil@==2.8.2, python-dotenv@==1.0.1, python-iso639@==2024.2.7, python-magic@==0.4.27, PyYAML@==6.0.1, rapidfuzz@==3.6.1, regex@==2023.12.25, requests@==2.31.0, requests-oauthlib@==1.3.1, rsa@==4.9, safetensors@==0.4.2, scikit-learn@==1.4.1.post1, scipy@==1.12.0, sentence-transformers@==2.3.1, sentencepiece@==0.1.99, six@==1.16.0, sniffio@==1.3.0, soupsieve@==2.5, SpeechRecognition@==3.10.1, SQLAlchemy@==2.0.27, sseclient-py@==1.8.0, sympy@==1.12, tabulate@==0.9.0, tenacity@==8.2.3, threadpoolctl@==3.3.0, together@==0.2.11, tokenizers@==0.15.2, torch@==2.2.0, tqdm@==4.66.2, transformers@==4.37.2, typer@==0.9.0, typing_extensions@==4.9.0, typing-inspect@==0.9.0, unstructured@==0.12.4, unstructured-client@==0.18.0, uritemplate@==4.1.1, urllib3@==2.2.0, wrapt@==1.16.0, yarl@==1.9.4
### Recent commits (newest first)
- Update README.md
- Update README.md
- Update README.md
- Add example env configuration
- Typo
- Update README
- Merge branch 'main' of https://github.com/pgasawa/treehacks24-dev
- logoogogoog
- Fix
- Merge branch 'main' of https://github.com/pgasawa/treehacks24-dev
- making height height
- Merge branch 'main' of https://github.com/pgasawa/treehacks24-dev
- Updated model
- name
- finishing title
- merge
- minor ui changes
- Merge branch 'main' of https://github.com/pgasawa/treehacks24-dev
- Small update
- Merge branch 'main' of https://github.com/pgasawa/treehacks24-dev
## Key source files (fetched from GitHub, selected and truncated for size)
### requirements.txt
```
SpeechRecognition
pyaudio
aiohttp==3.9.3
aiosignal==1.3.1
annotated-types==0.6.0
async-timeout==4.0.3
attrs==23.2.0
certifi==2024.2.2
charset-normalizer==3.3.2
click==8.1.7
frozenlist==1.4.1
idna==3.6
multidict==6.0.5
pydantic==2.6.1
pydantic_core==2.16.2
requests==2.31.0
sseclient-py==1.8.0
tabulate==0.9.0
together==0.2.11
tqdm==4.66.2
typer==0.9.0
typing_extensions==4.9.0
urllib3==2.2.0
yarl==1.9.4
aiohttp==3.9.3
aiosignal==1.3.1
annotated-types==0.6.0
async-timeout==4.0.3
attrs==23.2.0
certifi==2024.2.2
charset-normalizer==3.3.2
click==8.1.7
frozenlist==1.4.1
idna==3.6
multidict==6.0.5
pydantic==2.6.1
pydantic_core==2.16.2
python-dotenv==1.0.1
requests==2.31.0
sseclient-py==1.8.0
tabulate==0.9.0
together==0.2.11
tqdm==4.66.2
typer==0.9.0
typing_extensions==4.9.0
urllib3==2.2.0
yarl==1.9.4
aiohttp==3.9.3
aiosignal==1.3.1
annotated-types==0.6.0
async-timeout==4.0.3
attrs==23.2.0
cachetools==5.3.2
certifi==2024.2.2
charset-normalizer==3.3.2
click==8.1.7
frozenlist==1.4.1
google-api-core==2.17.1
google-api-python-client==2.118.0
google-auth==2.28.0
google-auth-httplib2==0.2.0
google-auth-oauthlib==1.2.0
googleapis-common-protos==1.62.0
httplib2==0.22.0
idna==3.6
multidict==6.0.5
oauthlib==3.2.2
protobuf==4.25.3
pyasn1==0.5.1
pyasn1-modules==0.3.0
PyAudio==0.2.14
pydantic==2.6.1
pydantic_core==2.16.2
pyparsing==3.1.1
python-dotenv==1.0.1
requests==2.31.0
requests-oauthlib==1.3.1
rsa==4.9
SpeechRecognition==3.10.1
sseclient-py==1.8.0
tabulate==0.9.0
together==0.2.11
tqdm==4.66.2
typer==0.9.0
typing_extensions==4.9.0
uritemplate==4.1.1
urllib3==2.2.0
yarl==1.9.4
aiohttp==3.9.3
aiosignal==1.3.1
annotated-types==0.6.0
anyio==4.2.0
async-timeout==4.0.3
attrs==23.2.0
backoff==2.2.1
beautifulsoup4==4.12.3
certifi==2024.2.2
cffi==1.16.0
chardet==5.2.0
charset-normalizer==3.3.2
click==8.1.7
dataclasses-json==0.6.4
dataclasses-json-speakeasy==0.5.11
distro==1.9.0
emoji==2.10.1
exceptiongroup==1.2.0
filelock==3.13.1
filetype==1.2.0
frozenlist==1.4.1
fsspec==2024.2.0
h11==0.14.0
httpcore==1.0.3
httpx==0.26.0
idna==3.6
Jinja2==3.1.3
joblib==1.3.2
jsonpatch==1.33
jsonpath-python==1.0.6
jsonpointer==2.4
langchain==0.1.7
langchain-community==0.0.20
langchain-core==0.1.23
langchain-together==0.0.2.post1
langdetect==1.0.9
langsmith==0.0.87
llvmlite==0.42.0
lxml==5.1.0
MarkupSafe==2.1.5
marshmallow==3.20.2
more-itertools==10.2.0
mpmath==1.3.0
multidict==6.0.5
mypy-extensions==1.0.0
networkx==3.2.1
nltk==3.8.1
numba==0.59.0
numpy==1.26.4
openai==1.12.0
openai-whisper @ git+https://github.com/openai/whisper.git@ba3f3cd54b0e5b8ce1ab3de13e32122d0d5f98ab
packaging==23.2
PyAudio==0.2.14
pycparser==2.21
pydantic==2.6.1
pydantic_core==2.16.2
python-dateutil==2.8.2
python-dotenv==1.0.1
python-iso639==2024.2.7
python-magic==0.4.27
PyYAML==6.0.1
rapidfuzz==3.6.1
regex==2023.12.25
requests==2.31.0
six==1.16.0
sniffio==1.3.0
soundfile==0.12.1
soupsieve==2.5
SpeechRecognition==3.10.1
SQLAlchemy==2.0.27
sseclient-py==1.8.0
sympy==1.12
tabulate==0.9.0
tenacity==8.2.3
tiktoken==0.6.0
together==0.2.11
torch==2.2.0
tqdm==4.66.2
typer==0.9.0
typing-inspect==0.9.0
typing_extensions==4.9.0
unstructured==0.12.4
unstructured-client==0.18.0
urllib3==2.2.0
whisper==1.1.10
wrapt==1.16.0
yarl==1.9.4
aiohttp==3.9.3
aiosignal==1.3.1
annotated-types==0.6.0
anyio==4.2.0
async-timeout==4.0.3
attrs==23.2.0
backoff==2.2.1
beautifulsoup4==4.12.3
cachetools==5.3.2
certifi==2024.2.2
cffi==1.16.0
chardet==5.2.0
charset-normalizer==3.3.2
click==8.1.7
dataclasses-json==0.6.4
dataclasses-json-speakeasy==0.5.11
emoji==2.10.1
exceptiongroup==1.2.0
faiss-cpu==1.7.4
filelock==3.13.1
filetype==1.2.0
frozenlist==1.4.1
fsspec==2024.2.0
google-api-core==2.17.1
google-api-python-client==2.118.0
google-auth==2.28.0
google-auth-httplib2==0.2.0
google-auth-oauthlib==1.2.0
googleapis-common-protos==1.62.0
httplib2==0.22.0
huggingface-hub==0.20.3
idna==3.6
Jinja2==3.1.3
joblib==1.3.2
jsonpatch==1.33
jsonpath-python==1.0.6
jsonpointer==2.4
langchain==0.1.7
langchain-community==0.0.20
langchain-core==0.1.23
langdetect==1.0.9
langsmith==0.0.87
llvmlite==0.42.0
lxml==5.1.0
MarkupSafe==2.1.5
marshmallow==3.20.2
more-itertools==10.2.0
mpmath==1.3.0
multidict==6.0.5
mypy-extensions==1.0.0
networkx==3.2.1
nltk==3.8.1
numba==0.59.0
numpy==1.26.4
oauthlib==3.2.2
openai-whisper @ git+https://github.com/openai/whisper.git@ba3f3cd54b0e5b8ce1ab3de13e32122d0d5f98ab
packaging==23.2
pillow==10.2.0
protobuf==4.25.3
pyasn1==0.5.1
pyasn1-modules==0.3.0
PyAudio==0.2.14
pycparser==2.21
pydantic==2.6.1
pydantic_core==2.16.2
pyparsing==3.1.1
python-dateutil==2.8.2
python-dotenv==1.0.1
python-iso639==2024.2.7
python-magic==0.4.27
PyYAML==6.0.1
rapidfuzz==3.6.1
regex==2023.12.25
requests==2.31.0
requests-oauthlib==1.3.1
rsa==4.9
safetensors==0.4.2
scikit-learn==1.4.1.post1
scipy==1.12.0
sentence-transformers==2.3.1
sentencepiece==0.1.99
six==1.16.0
sniffio==1.3.0
soundfile==0.12.1
soupsieve==2.5
SpeechRecognition==3.10.1
SQLAlchemy==2.0.27
sseclient-py==1.8.0
sympy==1.12
tabulate==0.9.0
tenacity==8.2.3
threadpoolctl==3.3.0
tiktoken==0.6.0
together==0.2.11
tokenizers==0.15.2
torch==2.2.0
tqdm==4.66.2
transformers==4.37.2
typer==0.9.0
typing-inspect==0.9.0
typing_extensions==4.9.0
unstructured==0.12.4
unstructured-client==0.18.0
uritemplate==4.1.1
urllib3==2.2.0
whisper==1.1.10
wrapt==1.16.0
yarl==1.9.4
```
### flask/package.json
```
{
"dependencies": {
"axios": "^1.6.7"
}
}
```
### frontend/package.json
```
{
"name": "frontend",
"version": "0.1.0",
"private": true,
"dependencies": {
"@testing-library/jest-dom": "^5.17.0",
"@testing-library/react": "^13.4.0",
"@testing-library/user-event": "^13.5.0",
"axios": "^1.6.7",
"bootstrap": "^5.3.2",
"react": "^18.2.0",
"react-bootstrap": "^2.10.1",
"react-bootstrap-icons": "^1.11.3",
"react-dom": "^18.2.0",
"react-scripts": "5.0.1",
"styled-components": "^6.1.8",
"web-vitals": "^2.1.4"
},
"scripts": {
"start": "react-scripts start",
"build": "react-scripts build",
"test": "react-scripts test",
"eject": "react-scripts eject"
},
"eslintConfig": {
"extends": [
"react-app",
"react-app/jest"
]
},
"browserslist": {
"production": [
">0.2%",
"not dead",
"not op_mini all"
],
"development": [
"last 1 chrome version",
"last 1 firefox version",
"last 1 safari version"
]
}
}
```
### server/requirements.txt
```
aiohttp==3.9.3
aiosignal==1.3.1
annotated-types==0.6.0
anyio==4.2.0
async-timeout==4.0.3
attrs==23.2.0
backoff==2.2.1
beautifulsoup4==4.12.3
cachetools==5.3.2
certifi==2024.2.2
chardet==5.2.0
charset-normalizer==3.3.2
click==8.1.7
dataclasses-json==0.6.4
dataclasses-json-speakeasy==0.5.11
emoji==2.10.1
exceptiongroup==1.2.0
faiss-cpu==1.7.4
filelock==3.13.1
filetype==1.2.0
frozenlist==1.4.1
fsspec==2024.2.0
google-api-core==2.17.1
google-api-python-client==2.118.0
google-auth==2.28.0
google-auth-httplib2==0.2.0
google-auth-oauthlib==1.2.0
googleapis-common-protos==1.62.0
httplib2==0.22.0
huggingface-hub==0.20.3
idna==3.6
Jinja2==3.1.3
joblib==1.3.2
jsonpatch==1.33
jsonpath-python==1.0.6
jsonpointer==2.4
langchain==0.1.7
langchain-community==0.0.20
langchain-core==0.1.23
langdetect==1.0.9
langsmith==0.0.87
lxml==5.1.0
MarkupSafe==2.1.5
marshmallow==3.20.2
mpmath==1.3.0
multidict==6.0.5
mypy-extensions==1.0.0
networkx==3.2.1
nltk==3.8.1
numpy==1.26.4
oauthlib==3.2.2
packaging==23.2
pillow==10.2.0
protobuf==4.25.3
pyasn1==0.5.1
pyasn1-modules==0.3.0
PyAudio==0.2.14
pydantic==2.6.1
pydantic_core==2.16.2
pyparsing==3.1.1
python-dateutil==2.8.2
python-dotenv==1.0.1
python-iso639==2024.2.7
python-magic==0.4.27
PyYAML==6.0.1
rapidfuzz==3.6.1
regex==2023.12.25
requests==2.31.0
requests-oauthlib==1.3.1
rsa==4.9
safetensors==0.4.2
scikit-learn==1.4.1.post1
scipy==1.12.0
sentence-transformers==2.3.1
sentencepiece==0.1.99
six==1.16.0
sniffio==1.3.0
soupsieve==2.5
SpeechRecognition==3.10.1
SQLAlchemy==2.0.27
sseclient-py==1.8.0
sympy==1.12
tabulate==0.9.0
tenacity==8.2.3
threadpoolctl==3.3.0
together==0.2.11
tokenizers==0.15.2
torch==2.2.0
tqdm==4.66.2
transformers==4.37.2
typer==0.9.0
typing-inspect==0.9.0
typing_extensions==4.9.0
unstructured==0.12.4
unstructured-client==0.18.0
uritemplate==4.1.1
urllib3==2.2.0
wrapt==1.16.0
yarl==1.9.4
```
### flask/main.py
```python
from flask import Flask
from flask_cors import CORS, cross_origin
import os
import json
app = Flask(__name__)
CORS(app)
app.config['CORS_HEADERS'] = 'Content-Type'
@app.route("/")
@cross_origin()
def read_from_data():
content = []
data_folder = './data'
for filename in os.listdir(data_folder):
with open(os.path.join(data_folder, filename), 'r') as file:
data = file.read()
content.append(json.loads(data))
# for filename in os.listdir(data_folder):
# os.remove(os.path.join(data_folder, filename))
return content
```
### server/server.py
```python
import socket
import threading
import sys
import os
from dotenv import load_dotenv
from classifier import classify
from actions.suggestions import get_suggestions
load_dotenv()
action_socket = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
HOST = os.environ["HOST"]
PORT = int(os.environ["SERVER_LISTEN_PORT"])
send_socket = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
SEND_HOST = os.environ["HOST"]
SEND_PORT = int(os.environ["SERVER_SEND_PORT"])
FILE = open("actions/central-log.txt", "w")
FILE.seek(0)
FILE.truncate()
FILE.close()
def handle_client(client_socket):
# This function will handle the communication with each client
while True:
file = open("actions/central-log.txt", "a")
# Receive data from the client
data = client_socket.recv(1024)
if not data:
break # Break the loop if no data is received
print(f"Received data from {client_socket.getpeername()}: {data.decode('utf-8')}")
data = data.decode('utf-8')
print(f"Write? {data.find(':')} + 2 < {len(data)}")
if (data.find(":") + 2 < len(data)):
file.write(data + "\n")
json_result = classify(data)
# file.write(data)
if json_result:
client_socket.sendall(json_result.encode('utf-8'))
else:
client_socket.sendall("No results".encode('utf-8'))
file.close()
# Close the connection when the client disconnects
print(f"Connection with {client_socket.getpeername()} closed.")
file.close()
client_socket.close()
def start_server():
# Bind the socket to a specific address and port
server_address = (HOST, PORT)
action_socket.bind(server_address)
send_server_address = (SEND_HOST, SEND_PORT)
send_socket.bind(send_server_address)
# Listen for incoming connections (maximum 5 clients in the queue)
action_socket.listen(5)
send_socket.listen()
print("Listening server is listening for incoming connections...")
try:
while True:
# Accept a connection from a client
client_socket, client_address = action_socket.accept()
print(f"Accepted connection from {client_address}")
# Create a new thread to handle the communication with the client
client_handler = threading.Thread(target=handle_client, args=(client_socket,))
client_handler.start()
send_client_socket, send_client_address = send_socket.accept()
print(f"Accepted send connection from {send_client_address}")
send_client_handler = threading.Thread(target=send_insights, args=(send_client_socket,))
send_client_handler.start()
except KeyboardInterrupt:
print("KeyboardInterrupt: Shutting down the server.")
shutdown_server()
def send_insights(send_client_socket):
while True:
data = send_client_socket.recv(1024)
if not data:
break
results = get_suggestions()
print("-========START========-")
print("Results: " + results)
print("-=========END=========-")
send_client_socket.sendall(results.encode('utf-8'))
def shutdown_server():
print("Shutting down the listening server.")
FILE.close()
sys.exit(0)
start_server()
```
### frontend/src/index.js
```javascript
import React from 'react';
import ReactDOM from 'react-dom/client';
import './index.css';
import App from './App';
import reportWebVitals from './reportWebVitals';
import 'bootstrap/dist/css/bootstrap.min.css';
const root = ReactDOM.createRoot(document.getElementById('root'));
root.render(
);
// If you want to start measuring performance in your app, pass a function
// to log results (for example: reportWebVitals(console.log))
// or send to an analytics endpoint. Learn more: https://bit.ly/CRA-vitals
reportWebVitals();
```
### frontend/src/App.js
```javascript
import './App.css';
import axios from 'axios';
import Container from 'react-bootstrap/Container';
import Navbar from 'react-bootstrap/Navbar';
import Row from 'react-bootstrap/Row';
import Col from 'react-bootstrap/Col';
import Tab from 'react-bootstrap/Tab';
import Tabs from 'react-bootstrap/Tabs';
import CalendarInviteCard from './notifications/CalendarInviteCard';
import EmailDraftCard from './notifications/EmailDraftCard';
import OpenLinkCard from './notifications/OpenLinkCard';
import ClarifyCard from './notifications/ClarifyCard';
import SuggestionCard from './notifications/SuggestionCard';
import React, { useState, useEffect } from 'react';
import backgroundImage from './background.gif'
function App() {
const now = new Date();
const [notifications, setNotifications] = useState([]);
const [suggestions, setSuggestions] = useState([]);
const fetchData = async () => {
try {
const response = await axios.get('http://127.0.0.1:5000');
const sortedData = response.data.sort((a, b) => b.creation_time - a.creation_time);
const sortedNotifications = sortedData.filter(item => item.action !== "suggestion");
const sortedSuggestions = sortedData.filter(item => item.action === "suggestion").slice(-1);
setNotifications(sortedNotifications);
setSuggestions(sortedSuggestions);
} catch (error) {
console.error('Error fetching data:', error);
}
};
useEffect(() => {
const intervalId = setInterval(() => {
fetchData();
}, 1000);
return () => clearInterval(intervalId);
}, []);
return (
Ephemeral
{notifications.map((item, index) => {
switch (item.action) {
case 'email':
return (
);
case 'schedule':
return (
);
case 'link':
return (
);
case 'clarify':
return (
)
default:
return null;
}
})}
{suggestions.map((item, index) => (
))}
);
}
export default App;
```
### archive/test.py
```python
import subprocess
def send_notification(message, title="Notification Title"):
script = f'display notification "{message}" with title "{title}"'
subprocess.run(["osascript", "-e", script])
# Example usage
send_notification("Your message here", "Custom Title")
```
### archive/socket_client.py
```python
import socket
HOST = '10.32.82.125' # Replace with server's IP address
PORT = 12345 # Same port as server
def send_message(message: str):
with socket.socket(socket.AF_INET, socket.SOCK_STREAM) as s:
s.connect((HOST, PORT))
s.sendall(message.encode("utf-8"))
```
[31 more indexed source files omitted to keep this export small. The full file list is in the Codebase structure section above.]