# Project export: Rondo - A Search Engine for the Elderly Made Easy This document was generated by HackStack to give an AI agent context about a hackathon project. Sections are labeled with their provenance; content marked as truncated was cut to keep this document small. ## Project metadata - Hackathon: TreeHacks 2024 - Tagline: Today's seniors struggle with tech and are averse to the web. Rondo is an innovative AI-powered search engine tool that makes navigating the internet accessible for those who lack digital literacy. - Devpost: https://devpost.com/software/rondo-a-search-engine-for-the-elderly-made-easy - GitHub: https://github.com/ericz23/TreeHacksProject - Team: 2 GitHub contributor(s) — maxdittgen (24 commits), ericz23 (1 commits) ## Devpost submission (written by the team) ### Inspiration As of today, over a third of the US population is over the age of 50 and this number is rapidly rising. One of the biggest issues facing today's elderly is an inability to understand and effectively utilize modern technology such as the Internet. As they did not grow up with these tools, many seniors find technology to be intimidating and become averse to its usage entirely. As a result, the elderly population suffers from an inhibited access to information and services, limiting their autonomy. This can be particularly impactful when seniors are unable to access important health-related information. For example, a senior may be experiencing chest pain and want to learn more about their condition. Simply searching "chest pain" on Google returns over three billion search results, many of which are long articles about general chest pain and the possible causes, symptoms, and treatment options. While this may be helpful and digestible for someone who has grown up with the Internet, for the elderly, such a wide scope of information may be daunting and difficult to parse through. There are so many different possible causes for chest pain and each has its own set of symptoms and conditions. It would be much more effective for them to include in their search query more specifics about their situation (ie, 'sharp upper chest pain lasting for three weeks'). We hope to help them leverage the Internet in the ways that they would like, but don't know how to. ### What it does Rondo is an AI-powered search-engine tool that makes the Internet truly accessible for the elderly population. Rondo takes in a query input from the user like any other regular old search engine. However, based off the specificity of the query, Rondo will prompt the user to answer a specific, well defined multiple choice question related to the original query in order to generate a more specific, better defined search query with more tailored results. Along with the question prompting, Rondo also automatically summarizes the top ten search results from the current query in order to make the content of each link more digestible to the user, who can read the brief summaries and decide which link to click on. These two functionalities run in parallel with question prompting on the left half of the screen and the article summaries on the right hand side. This way, the user can see what the current search query returns and can click on an article they find to be appropriate at any time. If the search results are still too broad or not what they are looking for, they can continue along with the prompted question to generate a better search query. Once the query reaches a certain level of specificity, Rondo will stop asking follow up questions. ### How we built it For the frontend, we used Reflex -- an open-source framework for building web applications in pure python. For the backend, we leveraged Python along with the OpenAI API, constructing prompts to elicit relevant responses, and dynamically updating queries based on iterative user input. ### Challenges we ran into Fine tuning GPT-4 for optimal query updating and follow-up question generation required persistent experimentation and tactical prompt engineering. Balancing the efficiency needs of a search engine with the processing time of our openai summarization tool. Integrating free-form GPT-4 responses into a highly structured frontend. ### Accomplishments we're proud of We are immensely proud of how much we were able to accomplish in such a short timeframe, especially given the context of our lack of front-end development experience. We were able to quickly pick up and leverage Reflex in order to create a fully functional product. We also put a lot of time and careful thought into our ideation process and we are very proud of the human-centered design we were able to create. ### What we learned Through our fully operational implementation of Rondo, we gained a rich and thorough understanding of sophisticated language processing tools such as ChatGPT and more specifically how to adapt and fine-tune such powerful models for our specific applications. We also gained many insights into user experience design through our design process which aimed to create a user-friendly interface with the target audience of the elderly population in mind. ### What's next Faster information retrieval and summary recall using more efficient LLMs and GPT prompts "Query Quality" bar to measure the specificity of the prompt and incentivize continued prompt specification More accessible summarization tools depending on the user's reading level Try it out: Clone and cd into the Git repository From the root directory, run pip install -r requirements.txt Replace the secret_key in summarizeWebPage.py and openai.key in follow_up_question_generation.py with your openai key ## README (from the GitHub repository) ![Alt text](https://i.ibb.co/KDcfKS9/rondo.png) ## A Search Engine for the Elderly Made Easy ### What it does Rondo is an AI-powered search-engine tool that makes the Internet truly accessible for the elderly population. Rondo takes in a query input from the user like any other regular old search engine. However, based off the specificity of the query, Rondo will prompt the user to answer a specific, well defined multiple choice question related to the original query in order to generate a more specific, better defined search query with more tailored results. ### How we built it For the frontend, we used Reflex -- an open-source framework for building web applications in pure python. For the backend, we leveraged Python along with the OpenAI API, constructing prompts to elicit relevant responses, and dynamically updating queries based on iterative user input. ## Detected evidence (automated analysis) Indexed codebase: 9 recognized source files, 28 KB. - OpenAI (technology) — detected in the code - Python (language) — detected in the code ## Codebase structure (from repository index) ### Files (120 of 198) ``` .gitignore assets/samples.json README.md requirements.txt rxconfig.py TreeHacksProject/prompt_engine/follow_up_question_generation.py TreeHacksProject/searcher_dir/__init__.py TreeHacksProject/searcher_dir/cache/aardvark.pkl TreeHacksProject/searcher_dir/cache/card_games_strategy_solitaire_improvement_game_play_interactive_learning.pkl TreeHacksProject/searcher_dir/cache/card_games_strategy_solitaire_improvement_game_play.pkl TreeHacksProject/searcher_dir/cache/card_games_strategy_solitaire_improvement.pkl TreeHacksProject/searcher_dir/cache/card_games_strategy_solitaire.pkl TreeHacksProject/searcher_dir/cache/card_games_strategy.pkl TreeHacksProject/searcher_dir/cache/card_games.pkl TreeHacksProject/searcher_dir/cache/cardiac_arrest.pkl TreeHacksProject/searcher_dir/cache/cat_exercise.pkl TreeHacksProject/searcher_dir/cache/cat_medication.pkl TreeHacksProject/searcher_dir/cache/cat.pkl TreeHacksProject/searcher_dir/cache/chess_move_pieces_possible_output_game_strategies.pkl TreeHacksProject/searcher_dir/cache/chess_move_pieces.pkl TreeHacksProject/searcher_dir/cache/chess.pkl TreeHacksProject/searcher_dir/cache/chest_p.pkl TreeHacksProject/searcher_dir/cache/chest_pain_activityrelated.pkl TreeHacksProject/searcher_dir/cache/chest_pain_and_dizziness.pkl TreeHacksProject/searcher_dir/cache/chest_pain_discomfort_or_slight_difficulty_breathing_intermittent_constant.pkl TreeHacksProject/searcher_dir/cache/chest_pain_discomfort_or_slight_difficulty_breathing.pkl TreeHacksProject/searcher_dir/cache/chest_pain_discomfort_or_slight.pkl TreeHacksProject/searcher_dir/cache/chest_pain_dizziness.pkl TreeHacksProject/searcher_dir/cache/chest_pain_eatingrelated_or_gastric_discomfort.pkl TreeHacksProject/searcher_dir/cache/chest_pain_foodrelated_spicy_cause.pkl TreeHacksProject/searcher_dir/cache/chest_pain_foodrelated.pkl TreeHacksProject/searcher_dir/cache/chest_pain_known_condition.pkl TreeHacksProject/searcher_dir/cache/chest_pain_mild_intensity.pkl TreeHacksProject/searcher_dir/cache/chest_pain_not_experiencing.pkl TreeHacksProject/searcher_dir/cache/chest_pain_occasional_discomfort.pkl TreeHacksProject/searcher_dir/cache/chest_pain.pkl TreeHacksProject/searcher_dir/cache/chest.pkl TreeHacksProject/searcher_dir/cache/deez_nuts.pkl TreeHacksProject/searcher_dir/cache/dizziness_and_nausea.pkl TreeHacksProject/searcher_dir/cache/gumbo_recipe.pkl TreeHacksProject/searcher_dir/cache/ichiro_suzuki.pkl TreeHacksProject/searcher_dir/cache/kids_diapers.pkl TreeHacksProject/searcher_dir/cache/linsanity.pkl TreeHacksProject/searcher_dir/cache/ny_yankees.pkl TreeHacksProject/searcher_dir/cache/online_casino.pkl TreeHacksProject/searcher_dir/cache/online_gambling_regulatory_concerns_australian_regulations.pkl TreeHacksProject/searcher_dir/cache/online_gambling_regulatory_concerns.pkl TreeHacksProject/searcher_dir/cache/online_gambling.pkl TreeHacksProject/searcher_dir/cache/potato_recipes_crispy_potatoes_side_dish.pkl TreeHacksProject/searcher_dir/cache/potato_recipes_crispy_potatoes.pkl TreeHacksProject/searcher_dir/cache/potato_recipes.pkl TreeHacksProject/searcher_dir/cache/rajon_rondo.pkl TreeHacksProject/searcher_dir/cache/stanford_elite_college_admissions_process.pkl TreeHacksProject/searcher_dir/cache/stanford_elite_college.pkl TreeHacksProject/searcher_dir/cache/stanford_relation_to_philosophy.pkl TreeHacksProject/searcher_dir/cache/stanford.pkl TreeHacksProject/searcher_dir/cache/television.pkl TreeHacksProject/searcher_dir/cache/treehacks_2024_application_process_eligibility_requirements.pkl TreeHacksProject/searcher_dir/cache/treehacks_2024_application_process.pkl TreeHacksProject/searcher_dir/cache/treehacks_2024.pkl TreeHacksProject/searcher_dir/search_results.py TreeHacksProject/searcher_dir/searcher.py TreeHacksProject/TreeHacksProject.py TreeHacksProject/webSummarizer/__init__.py TreeHacksProject/webSummarizer/cache/01398d5e907a8899c5b760db50d7bf36d7d4645960332bb11f4185a1097d4071.txt TreeHacksProject/webSummarizer/cache/023c715fd01a8df2d4a1991b82bbbffcc1ea86cf2a7662401841aa9ddd804981.txt TreeHacksProject/webSummarizer/cache/02c4e038d0ddce1f7f4d22cc0e4987b118b583dbd9bd33f3c97fb28093f5d65c.txt TreeHacksProject/webSummarizer/cache/062c40250715f3c1fbd0f0e9a823d3e8162bee73192e5d428067b63792b44865.txt TreeHacksProject/webSummarizer/cache/0646e39202a31003ff6d0649bc9fdd5c1ff8c443b76e21d3e454c131393510a7.txt TreeHacksProject/webSummarizer/cache/06b9d958dd0f5b67adbf535315bb2f35911bf4d8c87b8094b37e07b3497773d4.txt TreeHacksProject/webSummarizer/cache/072405949aedaeff24f937ea623bbb1a327f79ad7de1c0982064611f1083eea5.txt TreeHacksProject/webSummarizer/cache/08b721357ad75caa0939799d28fbbf0d7e64800d3d995957b936686ba0d19d84.txt TreeHacksProject/webSummarizer/cache/09470623c2df34d2300037a6818f3351ae7ff5ced13689a037986e4b9c07ccde.txt TreeHacksProject/webSummarizer/cache/0ac3d0207b44e8ccd9ed6d4b23d62e3615fadb023b4e5ae9fbd81a37710d1df5.txt TreeHacksProject/webSummarizer/cache/0d2dba70ab6421de94d31a6c92b244ba2384525f844c36c748a561324fd623a9.txt TreeHacksProject/webSummarizer/cache/15849e4371a7374585adc49d4bf3bdfdb827f77aedc07a208698a28649c024f4.txt TreeHacksProject/webSummarizer/cache/15955afd215344a0af6b9b3c7137e4ab78488e9b22e7132d97ad86bfe45f1a58.txt TreeHacksProject/webSummarizer/cache/17abb7e49b6dd17bad265aea98dc0506028d1e556d99aae2c2b715d3fd5c1f57.txt TreeHacksProject/webSummarizer/cache/1a6796fb400a60d938ca06fff64913787ac9ef0a0dc21c9fb712dc41a6670f47.txt TreeHacksProject/webSummarizer/cache/1b23bc4cd3d93a90276bbb6bb5e711e61d8f58077fc16415a8cb79cdd3e34c61.txt TreeHacksProject/webSummarizer/cache/1c0765b8a1216a6c92911cc7f1ee5c26a1bed93d7719bcbb393adff164eed01a.txt TreeHacksProject/webSummarizer/cache/1cba91906bcc50683de85c272742dac5ee9f451cabeeaf4dc381c503c18d46d5.txt TreeHacksProject/webSummarizer/cache/1cd19a2fddc4de3110a9fcdb173af6cb7e779a0d5db9b7bc371fb1b60e135277.txt TreeHacksProject/webSummarizer/cache/1d88ce6eef664990cc315a6f1eefee26696baccb8f97f1d3c42ca1346e026153.txt TreeHacksProject/webSummarizer/cache/1e631b314d2a8ad939819c87b08df7403ea336b4a37344e2515aff94df53208e.txt TreeHacksProject/webSummarizer/cache/206b7381656888d944114910b0ecec1693ce2d02107b8f409db6047b9f7b9a25.txt TreeHacksProject/webSummarizer/cache/236ab1817d3529bfc2a80b23f85808b329e32347b6524d7e706e97f498522858.txt TreeHacksProject/webSummarizer/cache/23b69ec6417d23cfba5bdf65c0659a70434646b0a06507c0260b8271b8f82019.txt TreeHacksProject/webSummarizer/cache/247ba7a1974cf5528cf8764970b73141c3cb2067930187d59b7750014ce50a68.txt TreeHacksProject/webSummarizer/cache/24caf673067da6353cd7bc4ffc16e0dbd012129c008b3448ac3fc19d904d0111.txt TreeHacksProject/webSummarizer/cache/25efa4cc5ebde46aba8265617d2e3d7719cbb9cddf798e1923cbfc505b65c66d.txt TreeHacksProject/webSummarizer/cache/281847597926071569019bca3f327aa26854f17f5fd24d28c109d1c25cf22f08.txt TreeHacksProject/webSummarizer/cache/28387ab712d8d558c61cc22e83441be90381743bcb13268b12def6db913232a3.txt TreeHacksProject/webSummarizer/cache/2d9aaaed56cfcbbc63acb257399c17f1b3dd7ef06f1c7ecd068638bb6af13fd6.txt TreeHacksProject/webSummarizer/cache/324451ef44cf967643f96d595017c015caeddb9571b6cdce8d9286171e5fec23.txt TreeHacksProject/webSummarizer/cache/36db1197f83adba11a2833aa6463d272da3eb104b3089e872b58774b9d6b4883.txt TreeHacksProject/webSummarizer/cache/370f322329ddfa939706164f9458c5ba07059a98b88d3f19b338b275e5e32234.txt TreeHacksProject/webSummarizer/cache/37c12d9012e5651ed688f4607f223ef0f5f560899d26ce70ee3beab0dd2b5157.txt TreeHacksProject/webSummarizer/cache/38a61494742c10c23e92603839df722bbe517c826050d745f1209695daf654c3.txt TreeHacksProject/webSummarizer/cache/392bf608e06ff83164f8de91dc0283c91c53c36590704fa4896b569e29215f41.txt TreeHacksProject/webSummarizer/cache/3959b7576a09f34b7f362c5aab8ebd058b972b18446e773398801c457681a2d6.txt TreeHacksProject/webSummarizer/cache/3c81eabb7145fb66298b6c95b5c0d962cfa75753b543544e8ebce09587c4033e.txt TreeHacksProject/webSummarizer/cache/3ecfc78fbb542a6cd6e24dca30548cf407ff05743826f768fcb71ffbbf7afc2d.txt TreeHacksProject/webSummarizer/cache/3fe4e047a76d28274f7e1178242c58a87ef3a6fcc5494755360ca21a640fd2b2.txt TreeHacksProject/webSummarizer/cache/43f0be739e033d685eafc43f5874a4832697a2f54bddd47c8c3353159f255060.txt TreeHacksProject/webSummarizer/cache/4581fe944293ec0384a0371cb612bfd17cc54b4397b504b11ad055ebd1917179.txt TreeHacksProject/webSummarizer/cache/45ec9dba56dbdbff873101653d06c2a87df887b41e58e278a818eb79607183f7.txt TreeHacksProject/webSummarizer/cache/4a48adbc031d184dba57f62d3c18371f65673563079b14fe920abf7818b1013c.txt TreeHacksProject/webSummarizer/cache/4d5d2bc66e3c44c642aaf8d8b5fac749651d73930dfe6a8502583466312bc880.txt TreeHacksProject/webSummarizer/cache/4e787bbd23e3b97e1ad281863205f5a888c418e85e90be8b3b29f7c96d735641.txt TreeHacksProject/webSummarizer/cache/52f01585d85d3faba49aa05eed83f586eebe49f57188421e1147fd57f8200100.txt TreeHacksProject/webSummarizer/cache/534e7abed3ccd39562a503cbd85422ba458a45642385420f7f47c688ad6531bc.txt TreeHacksProject/webSummarizer/cache/556b3a963707f039f0464af96c3b9d9a92b9c7c213451f770688ab9acfce22ea.txt TreeHacksProject/webSummarizer/cache/5825c8178a3d9332b0b910e51d34d5bd25de6043512438f00c5b14f81ac701b2.txt TreeHacksProject/webSummarizer/cache/5c650a2cee2e433413648f23acee52e8221bda249064dfaeff7cf44d976cf723.txt TreeHacksProject/webSummarizer/cache/5e5a875757f36f89415d9974673dff0e5e15d94ede33ec0162c0043ec49431bc.txt TreeHacksProject/webSummarizer/cache/5ff82c5a9f3a9c7d6b1e588495895b81b45ae561b324abc912f4155da7182c3e.txt TreeHacksProject/webSummarizer/cache/6124d1226261c49de351a6168749379b3eaa1ab809c0c50d7ee92ce77af220c5.txt TreeHacksProject/webSummarizer/cache/61af7dbad0cbe8522d89cae5555e050d1104688328f80766e49e538fb94af66b.txt TreeHacksProject/webSummarizer/cache/61c5d5e416cc8345ed18a65b887169acbb9c1f386d06977e6ccb954138f4acea.txt [78 more files omitted for size] ``` ### Dependencies - requirements.txt: beautifulsoup4@==4.12.3, openai@==1.12.0, pandas@==2.2.0, python-dotenv@==1.0.1, reflex@==0.4.0, Requests@==2.31.0 ### Recent commits (newest first) - Merge branch 'main' of github.com:ericz23/TreeHacksProject - removed hard coded keys - Update README.md - Update README.md - Update README.md - hardcoded keys - update name of api environment names - Merge pull request #1 from ericz23/reflex - remove comments and extra code - remove openAI api keys - caching - enabled multiple pages of links - update favicons - Implemented loading screens and back buttons - Added prompt response functionality - implemented caching - update webpage summarizer - enhanced summary scraping - Initialized search page on frontend - Added oscillating sample letters on index page ## Key source files (fetched from GitHub, selected and truncated for size) ### requirements.txt ``` beautifulsoup4==4.12.3 openai==1.12.0 pandas==2.2.0 python-dotenv==1.0.1 Requests==2.31.0 reflex==0.4.0 ``` ### rxconfig.py ```python import reflex as rx config = rx.Config( app_name="TreeHacksProject", ) ``` ### TreeHacksProject/TreeHacksProject.py ```python """Welcome to Reflex! This file outlines the steps to create a basic app.""" from rxconfig import config import reflex as rx import json import time import sys import os import asyncio sys.path.append(os.getcwd()) import TreeHacksProject.searcher_dir.search_results as sr import TreeHacksProject.searcher_dir.searcher as srcr import TreeHacksProject.webSummarizer.summarizeWebPage as summarizer import TreeHacksProject.prompt_engine.follow_up_question_generation as qg class searchbarState(rx.State): """The state of the homepage searchbar""" oscillating: bool = True text: str = "Chest pain" word_list: list = json.load(open("assets/samples.json"))["terms"] word_index: int = 0 def reset_states(self): self.oscillating = True self.text = "Chest pain" self.word_index = 0 self.search_page = 0 @rx.var def get_text(self) -> str: return self.text """The states of the search page""" _results: sr.SearchResults = None result_link1: str = "" result_title1: str = "" result_link2: str = "" result_title2: str = "" result_link3: str = "" result_title3: str = "" result_summary1: str = "" result_summary2: str = "" result_summary3: str = "" loading: bool = True loading_text: str = "Initializing search..." search_page: int = 0 links_loading: bool = False """The states of the question pane""" prompt_question: str = "" _previous_questions: list[str] = [] choices: list[str] = [] num_choices: int = 0 other_box: str = "" selected_answer: int = 0 prompt_answer: str = "" #functions to enable word oscillations def pause_oscillation(self, text): self.oscillating = False def pause_oscillation_(self): self.oscillating = False def resume_oscillation(self, text): self.oscillating = True @rx.background async def iterate_word(self): while True: async with self: if self.oscillating: self.text = self.word_list[self.word_index] self.word_index = (self.word_index + 1) % len(self.word_list) time.sleep(0.1) #functions to run a search async def get_results(self): self.loading = True self.links_loading = False yield # allow for other thread to launch loading screen self.loading_text = "Searching the web..." self._results = srcr.search(self.text) num_results = len(self._results.results) self.result_title1 = self._results.results[(self.search_page * 3 + 0) % num_results][0] # 10 results, 3 per page self.result_link1 = self._results.results[(self.search_page * 3 + 0) % num_results][1] self.result_title2 = self._results.results[(self.search_page * 3 + 1) % num_results][0] self.result_link2 = self._results.results[(self.search_page * 3 + 1) % num_results][1] self.result_title3 = self._results.results[(self.search_page * 3 + 2) % num_results][0] self.result_link3 = self._results.results[(self.search_page * 3 + 2) % num_results][1] self.loading_text = "Summarizing results..." self.result_summary1 = summarizer.get_summary(self.result_link1) self.result_summary2 = summarizer.get_summary(self.result_link2) self.result_summary3 = summarizer.get_summary(self.result_link3) # generate questions self.loading_text = "Generating prompts..." self.prompt_question = qg.generate_follow_up_question(self.text, self._previous_questions) self.choices, self.prompt_question = qg.get_multiple_choice(self.prompt_question) self._previous_questions.append(self.prompt_question) self.num_choices = len(self.choices) self.loading = False #function to go to next page of results def next_page(self): self.links_loading = True yield self.search_page += 1 num_results = len(self._results.results) self.result_title1 = self._results.results[(self.search_page * 3 + 0) % num_results][0] # 10 results, 3 per page self.result_link1 = self._results.results[(self.search_page * 3 + 0) % num_results][1] self.result_title2 = self._results.results[(self.search_page * 3 + 1) % num_results][0] self.result_link2 = self._results.results[(self.search_page * 3 + 1) % num_results][1] self.result_title3 = self._results.results[(self.search_page * 3 + 2) % num_results][0] self.result_link3 = self._results.results[(self.search_page * 3 + 2) % num_results][1] self.loading_text = "Summarizing results..." self.result_summary1 = summarizer.get_summary(self.result_link1) self.result_summary2 = summarizer.get_summary(self.result_link2) self.result_summary3 = summarizer.get_summary(self.result_link3) self.links_loading = False #function to answer a prompt def answer_prompt_mc(self, choice: int): self.prompt_answer = self.choices[choice] self.text = qg.refine_query(self.text, self.prompt_answer, self.prompt_question) self.loading = True self.search_page = 0 return searchbarState.get_results() def answer_prompt_open(self): self.prompt_answer = self.other_box self.text = qg.refine_query(self.text, self.prompt_answer, self.prompt_question) self.loading = True self.search_page = 0 return searchbarState.get_results() def reset_values(self): self._results = None self.result_link1 = "" self.result_title1 = "" self.result_link2 = "" self.result_title2 = "" self.result_link3 = "" self.result_title3 = "" self.result_summary1 = "" self.result_summary2 = "" self.result_summary3 = "" self.loading = True self.loading_text = "Initializing search..." self.prompt_question = "" [truncated — 9226 more characters] ``` ### TreeHacksProject/searcher_dir/search_results.py ```python ########### # SEARCH_RESULTS # Defines class to store google search results # A SearchResults object contains a: # - query string # - string number of results # - list of (title, link) tuples ########### class SearchResults: def __init__(self, query, num_results): self.query = query self.num_results = num_results self.results = [] def add_result(self, title, link): self.results.append((title, link)) def results(self): return self.results def num_results(self): return self.num_results def query(self): return self.query def display_results(self): print(f"Query: {self.query}") print(f"Number of Results: {self.num_results}") print("Search Results:") for title, link in self.results: print(f"Title: {title}") print(f"Link: {link}\n") ``` ### TreeHacksProject/searcher_dir/searcher.py ```python ########### # SEARCHER # Functions to return google search results and metadata from input queries ########### #imports import requests import json import TreeHacksProject.searcher_dir.search_results as sr import re import pickle import os #globals api_key = "AIzaSyBS0PzyXcftLs2DzJ3bHd863l0I6uF_zVo" # google programmable search api key cx = "e6fa68f5924a84919" # google search engine ID # helper functions def __send_request(query): """ Given a search query string, returns a json with search responses and metadata """ url = f"https://www.googleapis.com/customsearch/v1?q={query}&key={api_key}&cx={cx}" response = requests.get(url) if response.status_code == 200: return response.json() else: print(f"Failed to fetch search results. Status code: {response.status_code}") return None def __parse_response(results_dict): """ given a google search json, create a search_results object """ query = results_dict["queries"]["request"][0]["searchTerms"] num_results = results_dict["searchInformation"]["totalResults"] r = sr.SearchResults(query, num_results) for item in results_dict["items"]: r.add_result(item["title"], item["link"]) return r # public functions def search(input_query): """ Googles the input query and returns a SearchResults object containing results. Caches all searches, and checks the cache before running a google search """ query = input_query.lower() #make lowercase query = re.sub(r'\s+', "_", query) #replace spaces with underscores #look for cached copy of search results cache_directory = "./TreeHacksProject/searcher_dir/cache/" for filename in os.listdir(cache_directory): if os.path.isfile(os.path.join(cache_directory, filename)): if query == filename[:-4]: with open(cache_directory + filename, 'rb') as results: return pickle.load(results) #otherwise google and cache results = __parse_response(__send_request(input_query)) filename = cache_directory + query + ".pkl" with open(filename, 'wb') as file: pickle.dump(results, file, pickle.HIGHEST_PROTOCOL) return results ``` ### TreeHacksProject/webSummarizer/summarizeWebPage.py ```python import requests import pandas as pd from bs4 import BeautifulSoup from openai import OpenAI from dotenv import load_dotenv import os import hashlib secret_key = str(os.environ["OPENAI_API_KEY"]) def __getdata(url): """ Uses BeautifulSoup to extract the text data from webpage :param url: The URL of the webpage to extract data from :return: The text data from the webpage :raises ValueError: If there is no textual data that can be extracted using this method """ r = requests.get(url) soup = BeautifulSoup(r.text, 'html.parser') string = "" if soup.title is not None: string = "Title: " string += soup.title.string string += " \n Headers \n" headers = ["h1", "h2", "h3", "h4", "h5", "h6"] for header in headers: for data in soup.find_all(header): string += data.get_text() + "\n" string += "\n Body: \n" for data in soup.find_all("p"): if len(string.split()) > 600: break string += data.get_text() return string def __wordCap(string, cap): """ Caps the number of words in a string (to lessen token usage) :param string: The string to be capped :param cap: The number of words to cap the string at :return: The capped string """ return " ".join(string.split()[:cap]) def __summarize(webpage_Data): """ Uses OpenAI's GPT-3.5 model to summarize the webpage data and return a summary of the webpage :param webpage_Data: The data from the webpage to be summarized :return: A summary of the webpage data :raises ValueError: If the webpage data is too short to summarize """ if len(webpage_Data) < 100: return("Summary not available for this page.") webpage_Data = __wordCap(webpage_Data, 450) data = { "model": "gpt-3.5-turbo-0125", "messages": [ {"role": "system", "content": "You are an assistant tasked with helping elderly people utilize the internet in their daily lives. Your job is to summarize the following webpage and tell the user in a friendly way what they can find on the website so that they can figure out which site is best for what they are looking for. Keep your response under 100 words."}, {"role": "user", "content": webpage_Data}, ], "max_tokens": 200, "temperature": 0.7 } headers = { "Authorization": f"Bearer {secret_key}", "Content-Type": "application/json" } endpoint = "https://api.openai.com/v1/chat/completions" response = requests.post(endpoint, json=data, headers=headers) if response.status_code == 200: return response.json()["choices"][0]["message"]["content"] else: return "Summary not available for this page." def get_summary(link): link_hash = hashlib.sha256(link.encode('utf-8')).hexdigest() #look for cached copy of search results cache_directory = "./TreeHacksProject/webSummarizer/cache/" for filename in os.listdir(cache_directory): if os.path.isfile(os.path.join(cache_directory, filename)): if str(link_hash) == filename[:-4]: with open(cache_directory + filename, 'r') as summary: return summary.read() # cache results summary = __summarize(__getdata(link)) with open(cache_directory + str(link_hash) + ".txt", 'w') as file: file.write(summary) return summary ``` ### TreeHacksProject/prompt_engine/follow_up_question_generation.py ```python import openai import string import os # Set up your OpenAI API key openai.api_key = str(os.environ['OPENAI_API_KEY']) def generate_follow_up_question(input_query, previous_questions): # Constructing the prompt for ChatGPT prompt = f"Pretend I am a person who does not have robust technology literacy and does not know how to make specific and well defined search queries. I look up {input_query}. What is the most important follow up question you might need to ask me in order to get a more specific and well defined search query? For the follow up question, I want it to be very simple, specific, easy to answer with a 2-4 categorical answers that are provided in square brackets [(a) choice 1 * (b) choice 2 * (c) choice 3 * ...)] where each choice is delimited by a * symbol and there is ALWAYS an option that says 'Other' as the last option, but the other option also needs to have a letter in front of it and should be structured the same way as all the other options. The answers provided MUST follow this exact format and they MUST be surrounded by square brackets. I do not want it to be a compound question. I want it to be just ONE simple question not multiple. Make sure that the question is NOT repetitive. Do not ask about information already included in the input. Make sure all questions and multiple choice answers are formatted the same way. Do not ask questions that are similar to any of the previous questions. DO NOT have 'Please specify' in any of your multiple choice answers and do not use parentheses of any kind in the actual question." questions_str = ", ".join(previous_questions) prompt = prompt + "previous questions:" + questions_str # Generate response from ChatGPT response = openai.chat.completions.create( model = "gpt-4", messages = [{ "role" : "user", "content" : prompt }], #prompt = prompt, max_tokens=256, temperature=1, top_p=1, frequency_penalty=0, presence_penalty=0 ) # Extracting the follow-up questions from the response print(response.choices[0].message.content) follow_up_question = response.choices[0].message.content return follow_up_question def prompt_user(question): # Prompting the user to answer the follow-up question user_answer = input(f"{question} ") return user_answer def remove_common_words(s1, s2): # Split the strings into words words1 = s1.split() words2 = s2.split() translation_table = str.maketrans('', '', string.punctuation) # Process each word processed_words1 = [] for word in words1: # Convert to lowercase word = word.lower() # Remove punctuation word = word.translate(translation_table) # Append processed word processed_words1.append(word) processed_words2 = [] for word in words2: # Convert to lowercase word = word.lower() # Remove punctuation word = word.translate(translation_table) # Append processed word processed_words2.append(word) # Convert lists of words into sets for efficient membership testing set2 = set(processed_words2) # Remove words from s1 that are in s2 result_words = [word for word in processed_words1 if word not in set2] # Join the remaining words back into a string result = ' '.join(result_words) return result def refine_query(user_input, user_answer, follow_up_question): # Generate a more specific version of the original query prompt = f"Pretend I am a person who does not have robust technology literacy and does not know how to make specific and well defined search queries. I look up {user_input}. This is a pretty broad and not well defined search query. My friend asks me the question {follow_up_question}. I give the exact response {user_answer}. Given my response to this question and my original query, give me a 1-3 word phrase that represents the answer to the question but that does not include any of the words from what I originally looked up. I want the 1-3 word phrase to come from the answer itself so USE words from the answer! For example, if the question is 'where is the location of the chest pain' and the answer is 'on the left side' and the original query was 'chest pain’, the output should be ‘left side’. Don't include the words 'possible' or 'output'" #prompt = f"A user searches the internet search query \"{user_input}\" and responds \"{user_answer}\" to the clarification question \"{follow_up_question}\". Provide a 2-4 term extension to the original user query \"{user_input}\" that encapsulates this new response. Do not include any punctuation or special characters. DO NOT WRAP ANSWER IN QUOTES" response = openai.completions.create( model = "gpt-3.5-turbo-instruct", prompt = prompt, max_tokens=256, temperature=1, top_p=1, frequency_penalty=0, presence_penalty=0 ) add_on = response.choices[0].text.strip() add_on = remove_common_words(add_on, user_input) return user_input + " " + add_on def get_multiple_choice(question): # Find the index of the first '[' and the index of the last ']' start_index = question.find('[') end_index = question.rfind(']') # for malformed responses: if start_index == -1: start_index = question.find('(') end_index = len(question) - 1 # Extract the choices substring choices_str = question[start_index + 1:end_index] # Split the choices string by commas choices_list = choices_str.split('*') # Strip leading and trailing whitespace from each choice choices_list = [choice.strip() for choice in choices_list] # Cropped question cropped_question = question[:start_index] return choices_list, cropped_question def prompt_question_with_choices(question, choices): print(question) for i, choice in [truncated — 212 more characters] ```