# Project export: The Guild This document was generated by HackStack to give an AI agent context about a hackathon project. Sections are labeled with their provenance; content marked as truncated was cut to keep this document small. ## Project metadata - Hackathon: TreeHacks 2024 - Tagline: Personalized and parallelized communication with your business clients for WhatsApp merchants around the world. - Devpost: https://devpost.com/software/agentic-ai-marketing-platform - GitHub: https://github.com/mikezhang25/TreeHacks2024 - Team: 3 GitHub contributor(s) — SJayYang (21 commits), michaelzhang (14 commits), Gautham Raghupathi (7 commits) ## Devpost submission (written by the team) ### Inspiration We were inspired by our friend's brother, a small business owner who spoke about the challenges of communicating with their many clients through Whatsapp. We decided to work on this idea to help small business owners all around the world who communicate with their customers directly. ### What it does We built a platform that helps small business owners interact with their customers using AI agents. We are the first to build an AI agent first WhatsApp/messaging application, that's personalized based on the customers' previous interests and wants. ### How we built it We built our tools using MonsterAPI for their LLM inference interface (using the Zephyr LLM model), Reflex.dev for their frontend in Python, and Fetch AI for creating an environments for our AI agents. We used RAG, for customer specific knowledge search. ### Challenges we ran into We ran into many issues with API calls and connecting our frontend to our backends. We used a lot of new and exciting technologies implemented in new libraries, in which there wasn't too much documentation for us to work off of. This likely slowed down our development speed. ### Accomplishments we're proud of We built a working AI agent framework for communication with real phone numbers! ### What we learned We learned a lot about the different tools used within the AI agent framework and interactions fro creating a full stack application. ### What's next for Agentic AI Marketing Platform The future of AI agents looks pretty bright! We used the newest tools for LLM inference and creation of agents, and we got a MVP working within about 24 hrs of focused work. We're really excited by the quick development speed enabled by these new frameworks! ## README (from the GitHub repository) # TreeHacks2024 ## Frontend ``` cd reflex-chat/webui reflex init reflect run ``` # AI Engine NOTE: initiate backend before running this, otherwise client list is `None` ``` python3 manager.py ``` ## API Backend ``` node whatsapp/app.js ``` ## Detected evidence (automated analysis) Indexed codebase: 30 recognized source files, 79 KB. - Express (technology) — detected in the code - FastAPI (technology) — detected in the code - HTML (language) — detected in the code - Hugging Face (technology) — detected in the code - JavaScript (language) — detected in the code - LlamaIndex (technology) — detected in the code - OpenAI (technology) — detected in the code - Python (language) — detected in the code - PyTorch (technology) — detected in the code - Redis (technology) — detected in the code ## Codebase structure (from repository index) ### Files (69 of 69) ``` .gitignore agent1q0k7nuj7vz_data.json agent1q2g8ask7we_data.json agent1q2pnfjzyp3_data.json agent1qfcjz3kmhl_data.json agent1qfhh5zrd3x_data.json agent1qg7cv3g4em_data.json agent1qgxfwsp4sn_data.json agent1qthlh5xv9c_data.json agent1qv3d84sh7v_data.json agent1qvthy3a6xp_data.json agent1qvzfn63tel_data.json agent1qwm5uf0qxe_data.json agent1qwts34jt3y_data.json agent1qww3ju3h6k_data.json api_test.py clients.csv data/business/business.txt data/clients/14084315248/chat.txt data/clients/14084809325/chat.txt data/clients/14129801240/chat.txt data/clients/15023880975/chat.txt data/clients/16476767815/chat.txt data/clients/16502502196/chat.txt data/clients/16502835469/chat.txt data/clients/16692220216/chat.txt data/clients/19089173100/chat.txt data/clients/19178539101/chat.txt data/clients/5213319410786/chat.txt data/clients/918390785130/chat.txt data/clients/919811266476/chat.txt data/clients/error.txt data/clients/error/chat.txt fetchai_examples/agent1qww3ju3h6k_data.json fetchai_examples/agents_basic.py fetchai_examples/agents_communication.py fetchai_examples/agents_restaurant.py fetchai_examples/book.py main.py manager.py protocol.py rag_src/__init__.py rag_src/get_rag_data.sh rag_src/install_rag_packages.sh rag_src/zephyr_rag.py README.md reflex-chat/.gitignore reflex-chat/LICENSE reflex-chat/README.md reflex-chat/webui/.gitignore reflex-chat/webui/requirements.txt reflex-chat/webui/rxconfig.py reflex-chat/webui/webui/__init__.py reflex-chat/webui/webui/components/__init__.py reflex-chat/webui/webui/components/chat.py reflex-chat/webui/webui/components/explainbar.py reflex-chat/webui/webui/components/loading_icon.py reflex-chat/webui/webui/components/modal.py reflex-chat/webui/webui/components/navbar.py reflex-chat/webui/webui/components/sidebar.py reflex-chat/webui/webui/state.py reflex-chat/webui/webui/styles.py reflex-chat/webui/webui/webui.py requirements.txt whatsapp/.wwebjs_cache/2.2407.2.html whatsapp/.wwebjs_cache/2.2407.3.html whatsapp/.wwebjs_cache/2.2411.2.html whatsapp/app.js whatsapp/package.json ``` ### Dependencies - reflex-chat/webui/requirements.txt: openai@==0.28, reflex@>=0.2.0 - requirements.txt: aiohttp@==3.9.3, aiosignal@==1.3.1, alembic@==1.13.1, annotated-types@==0.6.0, anyio@==4.2.0, async-timeout@==4.0.3, attrs@==23.2.0, beautifulsoup4@==4.12.3, bidict@==0.23.0, bs4@==0.0.2, certifi@==2024.2.2, charset-normalizer@==3.3.2, click@==8.1.7, cloudpickle@==2.2.1, dataclasses-json@==0.6.4, Deprecated@==1.2.14, dirtyjson@==1.0.8, distro@==1.9.0, docopt@==0.6.2, exceptiongroup@==1.2.0, fastapi@==0.96.1, filelock@==3.13.1, frozenlist@==1.4.1, fsspec@==2024.2.0, greenlet@==3.0.3, gunicorn@==20.1.0, h11@==0.14.0, httpcore@==1.0.3, httpx@==0.25.2, huggingface-hub@==0.20.3, idna@==3.6, Jinja2@==3.1.3, joblib@==1.3.2, llama-index@==0.10.5, llama-index-agent-openai@==0.1.1, llama-index-core@==0.10.5, llama-index-embeddings-huggingface@==0.1.1, llama-index-embeddings-openai@==0.1.1, llama-index-legacy@==0.9.48, llama-index-llms-monsterapi@==0.1.1, llama-index-llms-openai@==0.1.2, llama-index-multi-modal-llms-openai@==0.1.1, llama-index-program-openai@==0.1.2, llama-index-question-gen-openai@==0.1.1, llama-index-readers-file@==0.1.3, Mako@==1.3.2, markdown-it-py@==3.0.0, MarkupSafe@==2.1.5, marshmallow@==3.20.2, mdurl@==0.1.2, monsterapi@==1.0.6, mpmath@==1.3.0, multidict@==6.0.5, mypy-extensions@==1.0.0, nest-asyncio@==1.6.0, networkx@==3.2.1, nltk@==3.8.1, numpy@==1.26.4, openai@==0.28.0, packaging@==23.2, pandas@==2.2.0, pillow@==10.2.0, pipdeptree@==2.14.0, pipreqs@==0.4.13, platformdirs@==3.11.0, psutil@==5.9.8, pydantic@==1.10.14, pydantic_core@==2.16.2, Pygments@==2.17.2, PyMuPDF@==1.23.22, PyMuPDFb@==1.23.22, pypdf@==4.0.1, python-dateutil@==2.8.2, python-engineio@==4.9.0, python-multipart@==0.0.5, python-socketio@==5.11.1, pytz@==2024.1, PyYAML@==6.0.1, redis@==4.6.0, reflex@==0.4.0, reflex-hosting-cli@==0.1.8, regex@==2023.12.25, requests@==2.31.0, requests-toolbelt@==1.0.0, rich@==13.7.0, safetensors@==0.4.2, scikit-learn@==1.4.1.post1, scipy@==1.12.0, sentence-transformers@==2.3.1, sentencepiece@==0.1.99, simple-websocket@==1.0.0, six@==1.16.0, sniffio@==1.3.0, soupsieve@==2.5, SQLAlchemy@==2.0.27, sqlmodel@==0.0.14, starlette@==0.27.0, starlette-admin@==0.9.0, sympy@==1.12, tabulate@==0.9.0, tenacity@==8.2.3, threadpoolctl@==3.3.0, tiktoken@==0.6.0, tokenizers@==0.15.2, torch@==2.2.0, tqdm@==4.66.2, transformers@==4.37.2, typer@==0.9.0, typing_extensions@==4.9.0, typing-inspect@==0.9.0, tzdata@==2024.1, urllib3@==2.2.0, uvicorn@==0.27.1, watchdog@==2.3.1, watchfiles@==0.19.0, websockets@==12.0, wrapt@==1.16.0, wsproto@==1.2.0, yarg@==0.1.9, yarl@==1.9.4 - whatsapp/package.json: express@^4.18.2, qrcode-terminal@^0.12.0, whatsapp-web.js@^1.23.0 ### Recent commits (newest first) - Documented how to get project running - added logging - no more acoustic - make prompt more liberal in filtering. - Remove word template from prompts. - working explainbar at bottom - push explainbar - Merge remote-tracking branch 'refs/remotes/origin/main' - wrong code for explainbar - mod prompt. - Update prompts - make prompts more readable in code. - fresh push refactoring name - refactored to remove name - add in onboarding message code for frontend - added onboarding ability - putting stuff together - connect frontend to backend - Add POST send-message. Update fetchMessages to get chat by id. - got endpoints to work ## Key source files (fetched from GitHub, selected and truncated for size) ### requirements.txt ``` aiohttp==3.9.3 aiosignal==1.3.1 alembic==1.13.1 annotated-types==0.6.0 anyio==4.2.0 async-timeout==4.0.3 attrs==23.2.0 beautifulsoup4==4.12.3 bidict==0.23.0 bs4==0.0.2 certifi==2024.2.2 charset-normalizer==3.3.2 click==8.1.7 cloudpickle==2.2.1 dataclasses-json==0.6.4 Deprecated==1.2.14 dirtyjson==1.0.8 distro==1.9.0 docopt==0.6.2 exceptiongroup==1.2.0 fastapi==0.96.1 filelock==3.13.1 frozenlist==1.4.1 fsspec==2024.2.0 greenlet==3.0.3 gunicorn==20.1.0 h11==0.14.0 httpcore==1.0.3 httpx==0.25.2 huggingface-hub==0.20.3 idna==3.6 Jinja2==3.1.3 joblib==1.3.2 llama-index==0.10.5 llama-index-agent-openai==0.1.1 llama-index-core==0.10.5 llama-index-embeddings-huggingface==0.1.1 llama-index-embeddings-openai==0.1.1 llama-index-legacy==0.9.48 llama-index-llms-monsterapi==0.1.1 llama-index-llms-openai==0.1.2 llama-index-multi-modal-llms-openai==0.1.1 llama-index-program-openai==0.1.2 llama-index-question-gen-openai==0.1.1 llama-index-readers-file==0.1.3 Mako==1.3.2 markdown-it-py==3.0.0 MarkupSafe==2.1.5 marshmallow==3.20.2 mdurl==0.1.2 monsterapi==1.0.6 mpmath==1.3.0 multidict==6.0.5 mypy-extensions==1.0.0 nest-asyncio==1.6.0 networkx==3.2.1 nltk==3.8.1 numpy==1.26.4 openai==0.28.0 packaging==23.2 pandas==2.2.0 pillow==10.2.0 pipdeptree==2.14.0 pipreqs==0.4.13 platformdirs==3.11.0 psutil==5.9.8 pydantic==1.10.14 pydantic_core==2.16.2 Pygments==2.17.2 PyMuPDF==1.23.22 PyMuPDFb==1.23.22 pypdf==4.0.1 python-dateutil==2.8.2 python-engineio==4.9.0 python-multipart==0.0.5 python-socketio==5.11.1 pytz==2024.1 PyYAML==6.0.1 redis==4.6.0 reflex==0.4.0 reflex-hosting-cli==0.1.8 regex==2023.12.25 requests==2.31.0 requests-toolbelt==1.0.0 rich==13.7.0 safetensors==0.4.2 scikit-learn==1.4.1.post1 scipy==1.12.0 sentence-transformers==2.3.1 sentencepiece==0.1.99 simple-websocket==1.0.0 six==1.16.0 sniffio==1.3.0 soupsieve==2.5 SQLAlchemy==2.0.27 sqlmodel==0.0.14 starlette==0.27.0 starlette-admin==0.9.0 sympy==1.12 tabulate==0.9.0 tenacity==8.2.3 threadpoolctl==3.3.0 tiktoken==0.6.0 tokenizers==0.15.2 torch==2.2.0 tqdm==4.66.2 transformers==4.37.2 typer==0.9.0 typing-inspect==0.9.0 typing_extensions==4.9.0 tzdata==2024.1 urllib3==2.2.0 uvicorn==0.27.1 watchdog==2.3.1 watchfiles==0.19.0 websockets==12.0 wrapt==1.16.0 wsproto==1.2.0 yarg==0.1.9 yarl==1.9.4 ``` ### whatsapp/package.json ``` { "name": "whatsapp-js-bot", "version": "1.0.0", "description": "", "main": "index.js", "scripts": { "test": "echo \"Error: no test specified\" && exit 1" }, "keywords": [], "author": "", "license": "ISC", "dependencies": { "express": "^4.18.2", "qrcode-terminal": "^0.12.0", "whatsapp-web.js": "^1.23.0" } } ``` ### reflex-chat/webui/requirements.txt ``` reflex>=0.2.0 openai==0.28 ``` ### main.py ```python if __name__ == "__main__": # Your main function code here ``` ### whatsapp/app.js ```javascript const qrcode = require('qrcode-terminal'); const { Client, LocalAuth } = require('whatsapp-web.js'); const express = require('express'); const client = new Client(); client.on('qr', (qr) => { qrcode.generate(qr, { small: true }); }); client.on('ready', async () => { console.log('Client is ready!'); }); client.initialize(); //--------------------------------------- const app = express(); app.use(express.json()) // for parsing application/json app.use(express.urlencoded({ extended: true })) // for parsing application/x-www-form-urlencoded app.get('/phone-numbers', async (req, res) => { try { const chats = await client.getChats(); const phoneNumbers = chats.map(chat => chat.id.user); res.json(phoneNumbers); } catch (error) { res.status(500).json({ error: 'Failed to fetch chats' }); } }); app.get('/fetch-messages/:phoneNumber', async (req, res) => { const phoneNumber = req.params.phoneNumber; try { const chat = await client.getChatById(`${phoneNumber}@c.us`); if (chat) { const messages = await chat.fetchMessages(); const messageBodies = messages.map(message => message.body); res.json(messageBodies); } else { res.status(404).json({ error: 'Chat not found' }); } } catch (error) { res.status(500).json({ error: 'Failed to fetch messages' }); } }); app.post('/send-message', async (req, res) => { const phoneNumber = req.body.phoneNumber; const message = req.body.message; try { const chat = await client.getChatById(`${phoneNumber}@c.us`); if (chat) { await chat.sendMessage(message); res.json({ message: 'Message sent successfully' }); } else { res.status(404).json({ error: 'Chat not found' }); } } catch (error) { res.status(500).json({ error: 'Failed to send message' }); } }); // For debugging purposes app.get('/chats', async (req, res) => { try { const chats = await client.getChats(); res.json(chats); } catch (error) { res.status(500).json({ error: 'Failed to fetch chats' }); } }); const PORT = 3001; app.listen(PORT, () => { console.log(`Server is running on http://localhost:${PORT}`); }); ``` ### api_test.py ```python from uagents import Agent, Context, Model from uagents.setup import fund_agent_if_low class Message(Model): message: str RECIPIENT_ADDRESS="agent1qfjcg2h5c2d2qkzksc8wntkpcyflntz0w8lsh2q6nwqpe6a2dn5ps88aqq3" sigmar = Agent( name="sigmar", port=8001, seed="sigmar secret phrase", endpoint=["http://0.0.0.0:8001/submit"], ) fund_agent_if_low(sigmar.wallet.address()) @sigmar.on_interval(period=10.0) async def send_message(ctx: Context): await ctx.send(RECIPIENT_ADDRESS, Message(message=input("prompt:"))) @sigmar.on_message(model=Message) async def message_handler(ctx: Context, sender: str, msg: Message): ctx.logger.info(f"Received message from {sender}: {msg.message}") if __name__ == "__main__": sigmar.run() ``` ### protocol.py ```python from uagents import Protocol, Model, Context class MessageRequest(Model): message: str class MessageResponse(Model): response_text: str message_proto = Protocol() @message_proto.on_message(model=MessageRequest, replies=MessageResponse) async def handle_message_request(ctx: Context, sender: str, msg: MessageRequest): # Process the message request here # For example, simply echo back the received message with some modification response_text = f"Received your message: {msg.message}" # Log the received message and the response ctx.logger.info(f"Received message from {sender}: {msg.message}. Responding with: {response_text}") # Send back a response to the sender await ctx.send(sender, MessageResponse(response_text=response_text)) ``` ### manager.py ```python from uagents import Agent, Bureau, Context, Model from uagents.setup import fund_agent_if_low from rag_src.zephyr_rag import ZephyrRAG API_ENDPT = "http://localhost:3001" company = "Sand Hill Pharmaceuticals" import requests import asyncio async def send_message(phone_number, message): try: payload = {'phoneNumber': phone_number, 'message': message} response = requests.post(f"{API_ENDPT}/send-message", json=payload) if response.status_code == 200: print(f"Message sent successfully to {phone_number}") else: print(f"Failed to send message to {phone_number}: {response.json().get('error')}") except Exception as e: print(f"Exception occurred while sending message to {phone_number}: {str(e)}") def fetch_phone_numbers(): try: response = requests.get(f"{API_ENDPT}/phone-numbers") excluded_numbers = ["5213319410786","14129801240","918390785130","14084315248","15023880975","15103568431","15127610849","447506916750", "16502835469"] phone_numbers = response.json() phone_numbers = list(filter(lambda number: number not in excluded_numbers, phone_numbers)) return phone_numbers or [] except Exception as e: print(f"Failed to fetch phone numbers: {str(e)}") def fetch_messages(phone_number): try: response = requests.get(f"{API_ENDPT}/fetch-messages/{phone_number}") messages = response.json() return messages or [] except Exception as e: print(f"Failed to fetch messages for {phone_number}: {str(e)}") class Manager: def __init__(self, bureau, data_path, clients) -> None: print(f"Initializing Manager") self.agent = Agent(name="Manager", seed=f"manager recovery phrase") fund_agent_if_low(self.agent.wallet.address()) self.bureau = bureau self.rag = ZephyrRAG( model="zephyr-7b-beta", temperature=0.75, context_window=1024, embed_model="local:BAAI/bge-small-en-v1.5", init_data_path=data_path ) self.rag.start_rag() bureau.add(self.agent) self.clients = clients @self.agent.on_event("startup") async def start_handler(ctx: Context): for client in self.clients: await ctx.send(client.agent.address, Message(message=f"Establishing manager contact")) @self.agent.on_message(model=OnBoard) async def onboard_client(ctx: Context, sender: str, msg: OnBoard): ctx.logger.info(f"Registering new client with number {msg.phone}") client_data_path = f"{base_path}/data/clients/{msg.phone}" if not os.path.exists(client_data_path): os.makedirs(client_data_path) with open(os.path.join(client_data_path, "chat.txt"), "w") as chat_file: chat_file.write("") newClient = Client(msg.phone, msg.phone, self.bureau, client_data_path) self.clients.append(newClient) send_message(msg.number, newClient.rag.query(f""" You are the owner of a business. There is a new client. Here's some context: !START OF CONTEXT! {msg.context} !END OF CONTEXT! You must craft a two-sentence welcome text, that embodies at least one key idea from the context. Remember, your output must STRICTLY BE TWO SENTENCES! Begin your output with "Welcome". """)) @self.agent.on_message(model=Message) async def message_handler(ctx: Context, sender: str, message: Message): print(f"Received message: {message.message}") # generate template user_prompt = message.message user_template = self.rag.query(f""" Generate a general response to a customer based on the following instructions from {company}. Instructions: \"\"\"{user_prompt}\"\"\" Your response is a text message, it should be worded in that manner. """).response #user_template = "boilerplate template" print(f"Generated template: {user_template}") # send combined prompt and prompt for client in self.clients: await ctx.send(client.agent.address, Directive(template=user_template, prompt=user_prompt)) def refresh_clients(self): for client in self.clients: fetch_messages(client.phone) def add_prompt(self, prompt): self.prompt_buffer.append(prompt) self.send_directive() class Client: def __init__(self, name, phone, bureau, chat_path) -> None: print(f"Initializing {name}") self.agent = Agent(name=name, seed=f"{name} recovery phrase") self.agent.storage.set("phone", phone) fund_agent_if_low(self.agent.wallet.address()) bureau.add(self.agent) self.rag = ZephyrRAG( model="zephyr-7b-beta", temperature=0.75, context_window=1024, embed_model="local:BAAI/bge-small-en-v1.5", init_data_path=chat_path ) self.rag.start_rag() self.refresh_chat() @self.agent.on_message(model=Message) async def message_handler(ctx: Context, sender: str, msg: Message): ctx.logger.info(f"{ctx.name} received message: {msg.message}") @self.agent.on_message(model=Directive) async def directive_handler(ctx: Context, sender: str, msg: Directive): # TODO: call API to refresh chat history in data/[name] fits_prompt = self.rag.query(f""" You are the representative for a customer. As a representative, your job is to filter out requests to contact your customer from business owners. In order to filter the requests, you use the chat history of the customer to de [truncated — 6070 more characters] ``` ### rag_src/get_rag_data.sh ```shell rm -r ./data mkdir -p data && cd data && curl 'https://bookspz.b-cdn.net/UMLE%20STEP%201%20CLOUD/First%20Aid%20for%20the%20USMLE%20Step%201-2019.pdf' -o "health.pdf" ``` ### rag_src/install_rag_packages.sh ```shell # For the RAG script python3 -m pip install llama-index python3 -m pip install llama-index-llms-monsterapi python3 -m pip install monsterapi python3 -m pip install sentence_transformers python3 -m pip install pypdf #this modeule is required for pdf parsing python3 -m pip install llama-index-embeddings-huggingface ``` [19 more indexed source files omitted to keep this export small. The full file list is in the Codebase structure section above.]