# Project export: BAIB This document was generated by HackStack to give an AI agent context about a hackathon project. Sections are labeled with their provenance; content marked as truncated was cut to keep this document small. ## Project metadata - Hackathon: TreeHacks 2024 - Tagline: Powerful semantic search for your life. Pronounced "babe". - Devpost: https://devpost.com/software/baib - GitHub: https://github.com/arjunpat/treehacks24 - Video: https://www.youtube.com/embed/KXF0eHWXbp0?enablejsapi=1&hl=en_US&rel=0&start=&version=3&wmode=transparent - Team: 4 GitHub contributor(s) — Jonathan Liu (38 commits), Tony Xin (32 commits), Clement Chan (24 commits), Arjun Patrawala (22 commits) ## Devpost submission (written by the team) ### Inspiration Powerful semantic search for your life does not currently exist. Google and ChatGPT have brought the world’s information to our fingertips, yet our personal search engines — Spotlight on Mac, and search on iOS or Android — are insufficient. Google Assistant and Siri tried to solve these problems by allowing us to search and perform tasks with just our voice, yet their use remains limited to a narrow range of tasks. Recent advancement in large language models has enabled a significant transformation in what's possible with our devices. ### What it does That's why we made Best AI Buddy, or BAIB for short. BAIB (pronounced "babe") is designed to seamlessly answer natural language queries about your life. BAIB builds an index of your personal data — text messages, emails, photos, among others — and runs a search pipeline on top of that data to answer questions. For example, you can ask BAIB to give you gift recommendations for a close friend. BAIB looks pertinent interactions you've had with that friend and generates gift ideas based on their hobbies, interests, and personality. To support its recommendations, BAIB cites parts of past text conversations you had with that friend. Or you can ask BAIB to tell you about what happened the last time you went skiing with friends. BAIB intelligently combines information from the ski group chat, AirBnB booking information from email, and your Google Photos to provide you a beautiful synopsis of your recent adventure. BAIB understands “hidden deadlines” — that form you need to fill out by Friday or that internship decision deadline due next week — and keeps track of them for you, sending you notifications as these “hidden deadlines” approach. Privacy is an essential concern. BAIB currently only runs on M1+ Macs. We are working on running a full-fledged LLM on the Apple Neural Engine to ensure that all information and processing is kept on-device. We believe that this is the only future of BAIB that is both safe and maximally helpful. ### How we built it Eventually, we plan to build a full-fledged desktop application, but for now we have built a prototype using the SvelteKit framework and the Skeleton.dev UI library. We use Bun as our TypeScript runtime & toolkit. Python backend. Our backend is built in Python with FastAPI, using a few hacks (check out our GitHub) to connect to your Mac’s contacts and iMessage database. We use the Google API to connect to Gmail + photos. LLM-guided search. A language model makes the decisions about what information should be retrieved — what keywords to search through different databases — and when to generate a response or continue accumulating more information. A beautiful, concise answer to a user query is often a result of many LLM prompts and aggregation events. Retrieval augmented generation. We experimented with vector databases and context-window based RAG, finding the latter to be more effective. Notifications. We have a series of “notepads” on which the LLM can jot down information, such as deadlines. We then later use a language model to generate notifications to ensure you don’t miss any crucial events. ### Challenges we ran into Speed. LLM-guided search is inherently slow, bottlenecked by inference performance. We had a lot of difficulty filtering data before giving it to the LLM for summarization and reasoning in a way that maximizes flexibility while minimizing cost. Prompt engineering. LLMs don’t do what you tell them, especially the smaller ones. Learning to deal with it in a natural way and work around the LLMs idiosyncrasies was important for achieving good results in the end. Vector search. Had issues with InterSystems and getting the vector database to work. ### Accomplishments we're proud of BAIB is significantly more powerful than we thought. As we played around with BAIB and asked fun questions like “what are the weirdest texts that Tony has sent me?”, its in-depth analysis on Tony’s weird texts were incredibly accurate: “Tony mentions that maybe his taste buds have become too American… This reflection on cultural and dietary shifts is interesting and a bit unusual in the context of a casual conversation.” This has increased our conviction in the long-term potential of this idea. We truly believe that this product must and will exist with or without us. Our team organization was good (for a hackathon). We split our team into the backend team and the frontend team. We’re proud that we made something useful and beautiful. ### What we learned Prompt engineering is very important. As we progressed through the project, we were able to speed up the response significantly and increase the quality by just changing the way we framed our question. ChatGPT 4.0 is more expensive than we thought. Further conviction that personal assistants will have a huge stake in the future. Energy drinks were not as effective as expected. ### What's next Building this powerful prototype gave us a glimpse of what BAIB could really become. We believe that BAIB can be integrated into all aspects of life. For example, integrating with other communication methods like Discord, Slack, and Facebook will allow the personal assistant to gain a level of organization and analysis that would not be previously possible. Imagine getting competing offers at different companies and being able to ask BAIB, who can combine the knowledge of the internet with the context of your family and friends to help give you enough information to make a decision. We want to continue the development of BAIB after this hackathon and build it as an app on your phone to truly become the Best AI Buddy. ## README (from the GitHub repository) ## Inspiration **Powerful semantic search for your life does not currently exist.** Google and ChatGPT have brought the world’s information to our fingertips, yet our personal search engines — Spotlight on Mac, and search on iOS or Android — are insufficient. Google Assistant and Siri tried to solve these problems by allowing us to search and perform tasks with just our voice, yet their use remains limited to a narrow range of tasks. **Recent advancement in large language models has enabled a significant transformation in what's possible with our devices.** ## What it does [See it in action on YouTube](http://www.youtube.com/watch?v=KXF0eHWXbp0) That's why we made Best AI Buddy, or BAIB for short. **BAIB (pronounced "babe") is designed to seamlessly answer natural language queries about your life.** BAIB builds an index of your personal data — text messages, emails, photos, among others — and runs a search pipeline on top of that data to answer questions. For example, you can ask BAIB to give you gift recommendations for a close friend. BAIB looks pertinent interactions you've had with that friend and generates gift ideas based on their hobbies, interests, and personality. To support its recommendations, **BAIB cites parts of past text conversations you had with that friend.** Or you can ask BAIB to tell you about what happened the last time you went skiing with friends. BAIB intelligently combines information from the ski group chat, AirBnB booking information from email, and your Google Photos to provide you a beautiful synopsis of your recent adventure. **BAIB understands “hidden deadlines”** — that form you need to fill out by Friday or that internship decision deadline due next week — and keeps track of them for you, sending you notifications as these “hidden deadlines” approach. **Privacy is an essential concern.** BAIB currently only runs on M1+ Macs. We are working on running a full-fledged LLM on the Apple Neural Engine to ensure that all information and processing is kept on-device. We believe that this is the only future of BAIB that is both safe and maximally helpful. ## How we built it Eventually, we plan to build a full-fledged desktop application, but for now we have built a prototype using the SvelteKit framework and the Skeleton.dev UI library. We use **Bun as our TypeScript runtime & toolkit.** **Python backend.** Our backend is built in Python with FastAPI, using a few hacks (check out our GitHub) to connect to your Mac’s contacts and iMessage database. We use the Google API to connect to Gmail + photos. **LLM-guided search.** A language model makes the decisions about what information should be retrieved — what keywords to search through different databases — and when to generate a response or continue accumulating more information. A beautiful, concise answer to a user query is often a result of many LLM prompts and aggregation events. **Retrieval augmented generation.** We experimented with vector databases and context-window based RAG, finding the latter to be more effective. **Notifications.** We have a series of “notepads” on which the LLM can jot down information, such as deadlines. We then later use a language model to generate notifications to ensure you don’t miss any crucial events. ## Challenges we ran into **Speed.** LLM-guided search is inherently slow, bottlenecked by inference performance. We had a lot of difficulty filtering data before giving it to the LLM for summarization and reasoning in a way that maximizes flexibility while minimizing cost. **Prompt engineering.** LLMs don’t do what you tell them, especially the smaller ones. Learning to deal with it in a natural way and work around the LLMs idiosyncrasies was important for achieving good results in the end. **Vector search.** Had issues with InterSystems and getting the vector database to work. ## Accomplishments that we're proud of **BAIB is significantly more powerful than we thought.** As we played around with BAIB and asked fun questions like “what are the weirdest texts that Tony has sent me?”, its in-depth analysis on Tony’s weird texts were incredibly accurate: “Tony mentions that maybe his taste buds have become too American… This reflection on cultural and dietary shifts is interesting and a bit unusual in the context of a casual conversation.” This has increased our conviction in the long-term potential of this idea. We truly believe that this product must and will exist with or without us. **Our team organization was good (for a hackathon).** We split our team into the backend team and the frontend team. We’re proud that we made something useful and beautiful. ## What we learned Prompt engineering is very important. As we progressed through the project, we were able to speed up the response significantly and increase the quality by just changing the way we framed our question. ChatGPT 4.0 is more expensive than we thought. Further conviction that personal assistants will have a huge stake in the future. Energy drinks were not as effective as expected. ## What's next for BAIB Building this powerful prototype gave us a glimpse of what BAIB could really become. We believe that BAIB can be integrated into all aspects of life. For example, integrating with other communication methods like Discord, Slack, and Facebook will allow the personal assistant to gain a level of organization and analysis that would not be previously possible. Imagine getting competing offers at different companies and being able to ask BAIB, who can combine the knowledge of the internet with the context of your family and friends to help give you enough information to make a decision. We want to continue the development of BAIB after this hackathon and build it as an app on your phone to truly become the Best AI Buddy. ## Detected evidence (automated analysis) Indexed codebase: 30 recognized source files, 79 KB. - FastAPI (technology) — detected in the code - HTML (language) — detected in the code - Hugging Face (technology) — detected in the code - JavaScript (language) — detected in the code - LangChain (technology) — detected in the code - LlamaIndex (technology) — detected in the code - OpenAI (technology) — detected in the code - Python (language) — detected in the code - PyTorch (technology) — detected in the code - Svelte (technology) — detected in the code - Tailwind CSS (technology) — detected in the code - TypeScript (language) — detected in the code ## Codebase structure (from repository index) ### Files (44 of 44) ``` .gitignore api.py baib/.gitignore baib/.npmrc baib/.prettierignore baib/.prettierrc baib/.vscode/settings.json baib/bun.lockb baib/package.json baib/postcss.config.cjs baib/README.md baib/src/app.d.ts baib/src/app.html baib/src/app.postcss baib/src/lib/components/AnimatedCheck.svelte baib/src/lib/components/AnswerSection.svelte baib/src/lib/components/ImageSection.svelte baib/src/lib/components/PromptInput.svelte baib/src/lib/components/Source.svelte baib/src/lib/components/Todo.svelte baib/src/lib/components/Todos.svelte baib/src/lib/index.ts baib/src/lib/stores/index.ts baib/src/lib/types/index.ts baib/src/routes/+layout.svelte baib/src/routes/+page.svelte baib/src/routes/response/+page.svelte baib/svelte.config.js baib/tailwind.config.ts baib/tsconfig.json baib/vite.config.ts chat.py chattest.py contacts.py main.py messages.py photos_gmail.py README.md requirements.txt retrieve.ipynb retrieve.py testing.ipynb textsearch.py wstest.py ``` ### Dependencies - baib/package.json: @floating-ui/dom@1.6.3, @iconify-json/mdi@^1.1.64, @skeletonlabs/skeleton@2.8.0, @skeletonlabs/tw-plugin@0.3.1, @sveltejs/adapter-auto@^3.0.0, @sveltejs/kit@^2.0.0, @sveltejs/vite-plugin-svelte@^3.0.0, @tailwindcss/forms@0.5.7, @tailwindcss/typography@0.5.10, @types/node@20.11.19, @types/showdown@^2.0.6, autoprefixer@10.4.17, highlight.js@11.9.0, postcss@8.4.35, prettier@^3.1.1, prettier-plugin-svelte@^3.1.2, showdown@^2.1.0, svelte@^4.2.7, svelte-check@^3.6.0, tailwindcss@3.4.1, tslib@^2.4.1, typescript@^5.0.0, unplugin-icons@^0.18.5, vite@^5.0.3, vite-plugin-tailwind-purgecss@0.2.0 - requirements.txt: aiohttp@==3.9.3, aiosignal@==1.3.1, annotated-types@==0.6.0, anyio@==4.0.0, appnope@==0.1.3, argon2-cffi@==23.1.0, argon2-cffi-bindings@==21.2.0, arrow@==1.3.0, asttokens@==2.4.1, async-lru@==2.0.4, async-timeout@==4.0.3, attrs@==23.1.0, Babel@==2.13.1, beautifulsoup4@==4.12.2, bleach@==6.1.0, cachetools@==5.3.2, certifi@==2023.7.22, cffi@==1.16.0, charset-normalizer@==3.3.2, click@==8.1.7, coloredlogs@==15.0.1, comm@==0.2.0, dataclasses-json@==0.6.4, debugpy@==1.8.0, decorator@==5.1.1, defusedxml@==0.7.1, Deprecated@==1.2.14, dirtyjson@==1.0.8, distro@==1.9.0, docker@==7.0.0, exceptiongroup@==1.1.3, executing@==2.0.1, fastapi@==0.109.2, fastapi-utils, fastembed@==0.2.1, fastjsonschema@==2.19.0, filelock@==3.13.1, flatbuffers@==23.5.26, fqdn@==1.5.1, frozenlist@==1.4.1, fsspec@==2024.2.0, google-api-core@==2.17.1, google-api-python-client@==2.118.0, google-auth@==2.28.0, google-auth-httplib2@==0.2.0, google-auth-oauthlib@==1.2.0, googleapis-common-protos@==1.62.0, greenlet@==3.0.3, h11@==0.14.0, httpcore@==1.0.3, httplib2@==0.22.0, httpx@==0.26.0, huggingface-hub@==0.20.3, humanfriendly@==10.0, idna@==3.4, install@==1.3.5, ipykernel@==6.26.0, ipython@==8.17.2, isoduration@==20.11.0, jedi@==0.19.1, Jinja2@==3.1.2, joblib@==1.3.2, json5@==0.9.14, jsonpatch@==1.33, jsonpointer@==2.4, jsonschema@==4.19.2, jsonschema-specifications@==2023.11.1, jupyter_client@==8.6.0, jupyter_core@==5.5.0, jupyter_server@==2.10.0, jupyter_server_terminals@==0.4.4, jupyter-events@==0.9.0, jupyter-lsp@==2.2.0, jupyterlab@==4.0.8, jupyterlab_server@==2.25.1, jupyterlab-pygments@==0.2.2, langchain@==0.0.348, langchain-core@==0.0.13, langchain-iris@==0.1.1, langsmith@==0.0.92, llama-index@==0.9.48, llama-iris@==0.3.0, loguru@==0.7.2, MarkupSafe@==2.1.3, marshmallow@==3.20.2, matplotlib-inline@==0.1.6, mistune@==3.0.2, mpmath@==1.3.0, multidict@==6.0.5, mypy-extensions@==1.0.0, nbclient@==0.9.0, nbconvert@==7.11.0, nbformat@==5.9.2, nest-asyncio@==1.5.8, networkx@==3.2.1, nltk@==3.8.1, notebook_shim@==0.2.3, numpy@==1.26.2, oauthlib@==3.2.2, onnx@==1.15.0, onnxruntime@==1.17.0, openai@==1.12.0, overrides@==7.4.0, packaging@==23.2, pandas@==2.2.0, pandocfilters@==1.5.0, parso@==0.8.3, pexpect@==4.8.0, phonenumbers@==8.13.30, pillow@==10.2.0, platformdirs@==4.0.0, prometheus-client@==0.18.0, prompt-toolkit@==3.0.41, protobuf@==4.25.3, psutil@==5.9.6, ptyprocess@==0.7.0, pure-eval@==0.2.2, pyasn1@==0.5.1, pyasn1-modules@==0.3.0, pycparser@==2.21, pydantic@==2.6.1, pydantic_core@==2.16.2, Pygments@==2.16.1, pyobjc-core@==10.0, pyobjc-framework-Cocoa@==10.0, pyobjc-framework-Contacts@==10.0, pyparsing@==3.1.1, python-dateutil@==2.8.2, python-dotenv@==1.0.1, python-json-logger@==2.0.7, pytypedstream@==0.1.0, pytz@==2023.3.post1, PyYAML@==6.0.1, pyzmq@==25.1.1, referencing@==0.31.0, regex@==2023.12.25, requests@==2.31.0, requests-oauthlib@==1.3.1, rfc3339-validator@==0.1.4, rfc3986-validator@==0.1.1, rpds-py@==0.12.0, rsa@==4.9, safetensors@==0.4.2, scikit-learn@==1.4.1.post1, scipy@==1.12.0, Send2Trash@==1.8.2, sentence-transformers@==2.3.1, sentencepiece@==0.1.99, six@==1.16.0, sniffio@==1.3.0, soupsieve@==2.5, SQLAlchemy@==2.0.27, sqlalchemy-iris@==0.13.1, stack-data@==0.6.3, starlette@==0.36.3, sympy@==1.12, tenacity@==8.2.3, terminado@==0.18.0, testcontainers-core@==0.0.1rc1, testcontainers-iris@==1.1.0, threadpoolctl@==3.3.0, tiktoken@==0.6.0, tinycss2@==1.2.1, tokenizers@==0.15.2, tomli@==2.0.1, torch@==2.2.0, tornado@==6.3.3, tqdm@==4.66.2, traitlets@==5.13.0, transformers@==4.37.2, types-python-dateutil@==2.8.19.14, typing_extensions@==4.8.0, typing-inspect@==0.9.0, tzdata@==2023.3, uri-template@==1.3.0, uritemplate@==4.1.1, urllib3@==2.1.0, uvicorn@==0.23.2, wcwidth@==0.2.10, webcolors@==1.13, webencodings@==0.5.1, websocket-client@==1.6.4, websockets, wrapt@==1.16.0, yarl@==1.9.4 ### Recent commits (newest first) - Create README.md - changes - fix - Merge branch 'main' of https://github.com/arjunpat/treehacks24 - email + photos - Remove unused node modules folder - Clean up package files - more updates - Merge branch 'main' of https://github.com/arjunpat/treehacks24 - better styling - Remove json - Allow query with Enter key - fixed - Wrap for super long chat words - some changes - Merge branch 'main' of https://github.com/arjunpat/treehacks24 - fixed bug - NAMED BAIB - Fix code again again - Fix code again ## Key source files (fetched from GitHub, selected and truncated for size) ### requirements.txt ``` aiohttp==3.9.3 aiosignal==1.3.1 annotated-types==0.6.0 anyio==4.0.0 appnope==0.1.3 argon2-cffi==23.1.0 argon2-cffi-bindings==21.2.0 arrow==1.3.0 asttokens==2.4.1 async-lru==2.0.4 async-timeout==4.0.3 attrs==23.1.0 Babel==2.13.1 beautifulsoup4==4.12.2 bleach==6.1.0 cachetools==5.3.2 certifi==2023.7.22 cffi==1.16.0 charset-normalizer==3.3.2 click==8.1.7 coloredlogs==15.0.1 comm==0.2.0 dataclasses-json==0.6.4 debugpy==1.8.0 decorator==5.1.1 defusedxml==0.7.1 Deprecated==1.2.14 dirtyjson==1.0.8 distro==1.9.0 docker==7.0.0 exceptiongroup==1.1.3 executing==2.0.1 fastapi==0.109.2 fastembed==0.2.1 fastjsonschema==2.19.0 filelock==3.13.1 flatbuffers==23.5.26 fqdn==1.5.1 frozenlist==1.4.1 fsspec==2024.2.0 google-api-core==2.17.1 google-api-python-client==2.118.0 google-auth==2.28.0 google-auth-httplib2==0.2.0 google-auth-oauthlib==1.2.0 googleapis-common-protos==1.62.0 greenlet==3.0.3 h11==0.14.0 httpcore==1.0.3 httplib2==0.22.0 httpx==0.26.0 huggingface-hub==0.20.3 humanfriendly==10.0 idna==3.4 install==1.3.5 ipykernel==6.26.0 ipython==8.17.2 isoduration==20.11.0 jedi==0.19.1 Jinja2==3.1.2 joblib==1.3.2 json5==0.9.14 jsonpatch==1.33 jsonpointer==2.4 jsonschema==4.19.2 jsonschema-specifications==2023.11.1 jupyter-events==0.9.0 jupyter-lsp==2.2.0 jupyter_client==8.6.0 jupyter_core==5.5.0 jupyter_server==2.10.0 jupyter_server_terminals==0.4.4 jupyterlab==4.0.8 jupyterlab-pygments==0.2.2 jupyterlab_server==2.25.1 langchain==0.0.348 langchain-core==0.0.13 langchain-iris==0.1.1 langsmith==0.0.92 llama-index==0.9.48 llama-iris==0.3.0 loguru==0.7.2 MarkupSafe==2.1.3 marshmallow==3.20.2 matplotlib-inline==0.1.6 mistune==3.0.2 mpmath==1.3.0 multidict==6.0.5 mypy-extensions==1.0.0 nbclient==0.9.0 nbconvert==7.11.0 nbformat==5.9.2 nest-asyncio==1.5.8 networkx==3.2.1 nltk==3.8.1 notebook_shim==0.2.3 numpy==1.26.2 oauthlib==3.2.2 onnx==1.15.0 onnxruntime==1.17.0 openai==1.12.0 overrides==7.4.0 packaging==23.2 pandas==2.2.0 pandocfilters==1.5.0 parso==0.8.3 pexpect==4.8.0 phonenumbers==8.13.30 pillow==10.2.0 platformdirs==4.0.0 prometheus-client==0.18.0 prompt-toolkit==3.0.41 protobuf==4.25.3 psutil==5.9.6 ptyprocess==0.7.0 pure-eval==0.2.2 pyasn1==0.5.1 pyasn1-modules==0.3.0 pycparser==2.21 pydantic==2.6.1 pydantic_core==2.16.2 Pygments==2.16.1 pyobjc-core==10.0 pyobjc-framework-Cocoa==10.0 pyobjc-framework-Contacts==10.0 pyparsing==3.1.1 python-dateutil==2.8.2 python-dotenv==1.0.1 python-json-logger==2.0.7 pytypedstream==0.1.0 pytz==2023.3.post1 PyYAML==6.0.1 pyzmq==25.1.1 referencing==0.31.0 regex==2023.12.25 requests==2.31.0 requests-oauthlib==1.3.1 rfc3339-validator==0.1.4 rfc3986-validator==0.1.1 rpds-py==0.12.0 rsa==4.9 safetensors==0.4.2 scikit-learn==1.4.1.post1 scipy==1.12.0 Send2Trash==1.8.2 sentence-transformers==2.3.1 sentencepiece==0.1.99 six==1.16.0 sniffio==1.3.0 soupsieve==2.5 SQLAlchemy==2.0.27 sqlalchemy-iris==0.13.1 stack-data==0.6.3 starlette==0.36.3 sympy==1.12 tenacity==8.2.3 terminado==0.18.0 testcontainers-core==0.0.1rc1 testcontainers-iris==1.1.0 threadpoolctl==3.3.0 tiktoken==0.6.0 tinycss2==1.2.1 tokenizers==0.15.2 tomli==2.0.1 torch==2.2.0 tornado==6.3.3 tqdm==4.66.2 traitlets==5.13.0 transformers==4.37.2 types-python-dateutil==2.8.19.14 typing-inspect==0.9.0 typing_extensions==4.8.0 tzdata==2023.3 uri-template==1.3.0 uritemplate==4.1.1 urllib3==2.1.0 uvicorn==0.23.2 wcwidth==0.2.10 webcolors==1.13 webencodings==0.5.1 websocket-client==1.6.4 wrapt==1.16.0 yarl==1.9.4 websockets fastapi-utils ``` ### baib/package.json ``` { "name": "baib", "version": "0.0.1", "private": true, "scripts": { "dev": "vite dev", "build": "vite build", "preview": "vite preview", "check": "svelte-kit sync && svelte-check --tsconfig ./tsconfig.json", "check:watch": "svelte-kit sync && svelte-check --tsconfig ./tsconfig.json --watch", "lint": "prettier --check .", "format": "prettier --write ." }, "devDependencies": { "@iconify-json/mdi": "^1.1.64", "@skeletonlabs/skeleton": "2.8.0", "@skeletonlabs/tw-plugin": "0.3.1", "@sveltejs/adapter-auto": "^3.0.0", "@sveltejs/kit": "^2.0.0", "@sveltejs/vite-plugin-svelte": "^3.0.0", "@tailwindcss/forms": "0.5.7", "@tailwindcss/typography": "0.5.10", "@types/node": "20.11.19", "autoprefixer": "10.4.17", "postcss": "8.4.35", "prettier": "^3.1.1", "prettier-plugin-svelte": "^3.1.2", "svelte": "^4.2.7", "svelte-check": "^3.6.0", "tailwindcss": "3.4.1", "tslib": "^2.4.1", "typescript": "^5.0.0", "unplugin-icons": "^0.18.5", "vite": "^5.0.3", "vite-plugin-tailwind-purgecss": "0.2.0" }, "type": "module", "dependencies": { "@floating-ui/dom": "1.6.3", "@types/showdown": "^2.0.6", "highlight.js": "11.9.0", "showdown": "^2.1.0" } } ``` ### main.py ```python import asyncio import json import os import pickle import re from dataclasses import dataclass from datetime import datetime import chat import textsearch from contacts import Contact, get_contacts from messages import Message, read_imessages db_path = os.path.expanduser("~/Downloads/chat.db") contacts = get_contacts() # messages = read_imessages(db_path) # with open("message_cache.pkl", "wb") as f: # pickle.dump(messages, f) with open("message_cache.pkl", "rb") as f: messages = pickle.load(f) dir_path = os.path.dirname(os.path.realpath(__file__)) persona_notepad_file = os.path.join(dir_path, "persona_notepad.json") with open(persona_notepad_file, "r") as f: persona_notepad = json.loads(f.read()) # What is Tony's door code? # What did I do in Hawaii last summer? # Can you suggest gifts for Gaurav Chak for his birthday? def retrieve_treehacks_email(): return """ Subject: Welcome to TreeHacks! Parking Information Inside 🚗 Dear Hackers, We are thrilled to have you join us for this year's TreeHacks event! As you embark on this exciting journey of innovation and creativity, we want to ensure your experience is as smooth and hassle-free as possible, starting with parking. For your convenience, we have arranged for two primary parking locations: Roble Field Garage - Located conveniently near the main event area, providing easy access to all the hackathon activities. Via Ortega Garage - A short walk from the event venue, this garage offers ample space and is an excellent option for those arriving from different parts of the campus or city. Parking Hours and Fees: We understand the importance of accessible parking, which is why we're pleased to inform you that parking at both Roble Field Garage and Via Ortega Garage is free after 4 PM on weekdays and completely free all day on weekends. This should accommodate your needs throughout the hackathon, allowing you to focus on what truly matters - bringing your innovative ideas to life! Need Assistance? Should you encounter any issues with parking or have any questions, please do not hesitate to contact us. We are here to assist you in any way we can to make your TreeHacks experience memorable and enjoyable. We are eagerly looking forward to seeing the incredible projects you will create and the unique solutions you will develop. Let's make this hackathon a remarkable event filled with creativity, collaboration, and innovation. Safe travels, and see you soon! Warm regards, The TreeHacks Organizing Team """ def retrieve_persona_notepad(phone_number: str): if phone_number not in messages: return "Expected a valid phone number of a contact." if phone_number not in persona_notepad: return f"Persona notepad for {phone_number} currently empty. Please add to it when you acquire new information" return "\n".join(persona_notepad[phone_number]) def save_to_persona_notepad(phone_number: str, info: str): if phone_number not in persona_notepad: persona_notepad[phone_number] = [] persona_notepad[phone_number].append(info) with open(persona_notepad_file, "w") as f: f.write(json.dumps(persona_notepad)) return "Succesfully saved to persona notepad." def format_datetime(dt): """ Formats a datetime object into a string with the format 'dd MMM yyyy at hh:mm AM/PM'. Args: dt (datetime): A datetime object to be formatted. Returns: str: The formatted datetime string. """ # return dt.strftime("%d %b %Y at %I:%M %p") return dt.strftime("%d %b %Y") def query_contacts_by_name(name: str): potential_contact_list = [] # TODO: can be improved, ask to list multiple spellings name = name.strip().lower().split(" ") for con in contacts: keywords = [con.first_name.lower(), con.last_name.lower()] keywords = [e for e in keywords if e is not None and e != ""] for i in range(len(name)): for j in range(len(keywords)): if name[i] in keywords[j] or keywords[j] in name[i]: potential_contact_list.append(con) return "\n".join([str(c) for c in potential_contact_list]) def remove_non_numbers(s): # Use a list comprehension to filter out non-digit characters filtered_chars = [char for char in s if char.isdigit()] # Join the list back into a string return "".join(filtered_chars) def get_contact_from_phone_number(phone_number: str) -> list[Contact]: potential_contact_list = [] for con in contacts: pns = [remove_non_numbers(c) for c in con.phone_numbers if isinstance(c, str)] # if con.first_name == "Stephan": # print(remove_non_numbers(phone_number), pns) if any([remove_non_numbers(phone_number) == c for c in pns]): potential_contact_list.append(con) return potential_contact_list """ def query_all_text_messages_from_contact(phone_number: str): if phone_number not in messages: return "Could not find any messages with this phone number" contact_list = get_contact_from_phone_number(phone_number) assert len(contact_list) > 0 contact = contact_list[0] contact_name = f"{contact.first_name} {contact.last_name}" msg_list = "" for msg in messages[phone_number].messages: date_str = format_datetime(msg.date) person = contact_name if msg.sender == phone_number else msg.sender msg_list += f"{person} - {date_str}: {msg.text}\n" return msg_list """ def query_text_messages_from_contact(phone_number: str, query: str): if phone_number not in messages: return "Could not find any messages with this phone number" contact_list = get_contact_from_phone_number(phone_number) assert len(contact_list) > 0 contact = contact_list[0] contact_name = f"{contact.first_name} {contact.last_name}" indices = set() CONTEXT_LEN = 4 if "code" in query: code_convo = """(1) Arjun - Nov 21, 2019, 12:49 pm: yo [truncated — 9714 more characters] ``` ### baib/src/lib/index.ts ```typescript // place files you want to import through the `$lib` alias in this folder. ``` ### baib/src/lib/stores/index.ts ```typescript import { writable } from 'svelte/store'; export const query = writable(''); export const action_items: any = writable([]) ``` ### baib/src/lib/types/index.ts ```typescript export type SourceType = 'message' | 'email' | 'photo'; export type AnswerContent = { type: AnswerType; content: TextContent | MessageContent; }; export type AnswerType = 'text' | 'message'; export type TextContent = { text: string; }; export type MessageContent = { [id: string]: Message; }; export type Message = { speaker: 'other' | 'self'; text: string; }; export type Progress = { done: boolean; text: string; }; ``` ### wstest.py ```python import asyncio import websockets import json async def receive_progress(): async with websockets.connect('ws://localhost:8000/generate') as websocket: await websocket.send(json.dumps({"question": "When is Clement's birthday?"})) while True: message = await websocket.recv() body = json.loads(message) print(body) if body["status"] != "progress": break asyncio.run(receive_progress()) ``` ### retrieve.py ```python import getpass import os from dotenv import load_dotenv from llama_index import Document, ServiceContext, SimpleDirectoryReader, StorageContext from llama_index.indices.vector_store import VectorStoreIndex from llama_iris import IRISVectorStore load_dotenv(override=True) if not os.environ.get("OPENAI_API_KEY"): os.environ["OPENAI_API_KEY"] = getpass.getpass("OpenAI API Key:") import os from contacts import get_contacts from messages import read_imessages username = "SUPERUSER" password = "SYS2" # Replace password with password you set hostname = "localhost" port = "1972" namespace = "USER" CONNECTION_STRING = f"iris://{username}:{password}@{hostname}:{port}/{namespace}" db_path = os.path.expanduser("~/Downloads/chat.db") # contacts = get_contacts() # chats = read_imessages(db_path) documents = [] chats = { "a": [ "hello there", "do you know when we're going", "sooooo excited for the beach", ], "b": [ "hey just letting you know my party is tomorrow", "ayyy that's sick i'll see you at The Tower, gotta be a lot of fun", "yup yup see you at 9", ], } for k, v in chats.items(): # text = "\n".join([msg.text for msg in v.messages]) text = "\n".join(v) documents.append(Document(text=text)) print(documents) vector_store = IRISVectorStore.from_params( connection_string=CONNECTION_STRING, table_name="texts", embed_dim=1536, # openai embedding dimension ) storage_context = StorageContext.from_defaults(vector_store=vector_store) # service_context = ServiceContext.from_defaults( # embed_model=embed_model, llm=None # ) index = VectorStoreIndex.from_documents( documents, storage_context=storage_context, show_progress=True, # service_context=service_context, ) # index = VectorStoreIndex.from_vector_store(vector_store=vector_store) query_engine = index.as_query_engine() response = query_engine.query("Where is the event occuring?") print(response) ``` ### textsearch.py ```python import getpass import os from dotenv import load_dotenv from llama_index import Document, ServiceContext, SimpleDirectoryReader, StorageContext from llama_index.indices.vector_store import VectorStoreIndex from llama_iris import IRISVectorStore load_dotenv(override=True) if not os.environ.get("OPENAI_API_KEY"): os.environ["OPENAI_API_KEY"] = getpass.getpass("OpenAI API Key:") username = "SUPERUSER" password = "SYS2" # Replace password with password you set hostname = "localhost" # hostname = "0.tcp.ngrok.io" port = "1972" # port = "18659" namespace = "USER" CONNECTION_STRING = f"iris://{username}:{password}@{hostname}:{port}/{namespace}" # semantics searcher class TextSearcher: def __init__(self): self.documents = [] # load a chat from a person into the vector database def load(self, chat, name): # ok whole thing gooo # self.documents.append(Document(text=chat, metadata={"name": name} or {})) for message in chat: doc = Document(text=message, metadata={"name": name} or {}) self.documents.append(doc) # print("loaded messages") def search(self, query): vector_store = IRISVectorStore.from_params( connection_string=CONNECTION_STRING, table_name="texts", embed_dim=1536, # openai embedding dimension ) # print("vector store made") storage_context = StorageContext.from_defaults(vector_store=vector_store) # service_context = ServiceContext.from_defaults( # embed_model=embed_model, llm=None # ) # print("content made") index = VectorStoreIndex.from_documents( self.documents, storage_context=storage_context, show_progress=True, # service_context=service_context, ) # print("index made") query_engine = index.as_query_engine() # print("querying...") response = query_engine.query(query) print(str(response)) return str(response) ``` ### contacts.py ```python import Contacts import objc import phonenumbers class Contact: def __init__( self, first_name: str, last_name: str, phone_numbers: list[str] ) -> None: self.first_name = first_name self.last_name = last_name self.phone_numbers = phone_numbers def __repr__(self) -> str: return f"Contact(first_name={self.first_name}, last_name={self.last_name}, phone_numbers={self.phone_numbers})" def __str__(self): return repr(self) def get_CNContacts_list() -> list[Contacts.CNContact]: """Returns a list of contacts""" # Create a contact store store = Contacts.CNContactStore.alloc().init() # Define keys that you want to fetch (e.g., name) keys_to_fetch = [ Contacts.CNContactGivenNameKey, Contacts.CNContactFamilyNameKey, Contacts.CNContactPhoneNumbersKey, ] # Fetch all contacts fetch_request = Contacts.CNContactFetchRequest.alloc().initWithKeysToFetch_( keys_to_fetch ) error = objc.nil # Create an empty list to hold contacts contact_list = [] # Process each contact def contact_handler(contact, stop): contact_list.append(contact) store.enumerateContactsWithFetchRequest_error_usingBlock_( fetch_request, error, contact_handler ) return contact_list def sanitize_phone_num(phone_num: str) -> str: """Sanitizes a phone number""" try: phone_num = bytes(phone_num, "utf-8").decode("utf-8", "ignore") phone_num = phone_num.strip().replace(" ", "") return phonenumbers.format_number( phonenumbers.parse(phone_num, "US"), phonenumbers.PhoneNumberFormat.E164, ) except Exception as e: print(e, phone_num) def get_contacts() -> list[Contact]: """Returns a list of contacts""" contact_list = get_CNContacts_list() return list( map( lambda e: Contact( e.givenName(), e.familyName(), list( map( lambda ea: sanitize_phone_num(str(ea.value().stringValue())), e.phoneNumbers(), ) ), ), contact_list, ) ) ``` [20 more indexed source files omitted to keep this export small. The full file list is in the Codebase structure section above.]