# Project export: Review Raven This document was generated by HackStack to give an AI agent context about a hackathon project. Sections are labeled with their provenance; content marked as truncated was cut to keep this document small. ## Project metadata - Hackathon: CruzHacks 2025 - Tagline: Compile reviews for your small business from various websites into one place, helping you analyze customer sentiment and fight review manipulation! - Devpost: https://devpost.com/software/review-raven - GitHub: https://github.com/Ant-28/reviewtracker - Video: https://www.youtube.com/embed/Qk0BENWXc6Q?enablejsapi=1&hl=en_US&rel=0&start=&version=3&wmode=transparent - Result: winner (Justice Hacks) - Team: 3 GitHub contributor(s) — Ant (26 commits), sameerdash2 (22 commits), Neil Grover (2 commits) ## Devpost submission (written by the team) ### Inspiration Review management is quite cumbersome for a plethora of business owners. Having interacted frequently with small businesses and startups, we are attuned to the need for valuable feedback and summaries. With that in mind, we decided to design a review aggregator for small business, akin to Metacritic. Relevance to track (Justice) Our project aims to level the playing field for small businesses, especially in an era of stunted economic growth. We designed this app to bring small businesses more insight about their community and their general perception, which is crucial in our modern information-centric era. We also notice that small business are more susceptible to review bombing and other forms of review biases. For example, we noticed a discrepancy in the following figure: reviews with text tend to show businesses in a more positive light than all available reviews. This is especially true for local businesses, given the smaller numbers of review available.. ### What it does Our software collects reviews from Google Maps, Facebook and a multitude of other platforms. We combined deterministic scraping with more entropic LLM scraping using a GPT-4 powered AI agent via Langflow. We also provide an average sentiment analysis across reviews, displaying the frequency of the top 3 sentiments. Link: http://reviewraven.tech/ (Due to limitations of .tech domains, you must use http:// to view the site. It also cannot interface with the backend, unless run locally.) ### How we built it The main driver was a set of webscrapers that collect reviews for businesses, tailored to each review website. These were written in Python using Selenium. We wrote a backend in Node.js and Express, which was the client's interface to the review data. The frontend was written in HTML and JS (Vanilla) with Pico CSS as a minimal stylesheet. For the sentiment analysis, we used the NRC, in addition to Langflow's LLM features. ### Challenges we ran into (and how we solved them) Our first challenge involved consistent loading and preventing our client from being flagged as a spambot. We addressed this by using an "undectectable" driver. Since we were collecting reviews from multiple sources, we had to combine our review data with our sentiment analysis corpus. Furthermore, when using LLMs to aggregate reviews, their reliability varied significantly depending on the model used. Notably, Gemini 2.0-pro-exp sometimes found fewer reviews or threw errors, while GPT 4o-mini was more consistent as it is a production-ready LLM. ### Accomplishments we're proud of To begin with, 2/3rds of our team have never participated in a hackathon, so this has been a very enlightening experience. Additionally, we are proud of writing decent webscrapers and handling multiple failure states, ensuring all failures are as graceful as possible. Lastly, collating all of this data over a span of 2 days is a daunting task that we successfully tackled. ### What we learned Firstly, we prioritized software performance over aesthetics. In other words: "get it running!". We also discussed approaches frequently, especially when ideas weren't working. This allowed to pivot our design and implementation to match our goals more closely. Since we initially planned to use APIs, we learnt to leverage webscraping as a contingency whenever possible. ### What's next We aim to expand its scope to provide more detailed sentiment analyses and reviews. We would also let business owners create accounts and make notes on their businessess for further review or view their performance over time as graphs. Finally, our long-term goal is to expand this to a service/API that other businesses (as well as students) can use affordably. ## README (from the GitHub repository) # Review Raven A review aggregator website, collecting Google Maps, Facebook and more! Writeup name: `devpost.md` # WARNING Use http in the domain: [http://reviewraven.tech](http://reviewraven.tech) ## Setup: 1. `npm install` 2. `pip install -r requirements.txt` 3. `git submodule update --init` 4. `nodemon server.js` ## Detected evidence (automated analysis) Indexed codebase: 9 recognized source files, 31 KB. - Express (technology) — detected in the code - HTML (language) — detected in the code - JavaScript (language) — detected in the code - Python (language) — detected in the code - Node.js (technology) — claimed on Devpost, not found in the code ## Codebase structure (from repository index) ### Files (17 of 17) ``` .gitignore .gitmodules .python-version devpost.md eslint.config.mjs frontend/index.html frontend/index.js Langflow_Review_Searcher.json package.json README.md requirements.txt Scraper/facebook_reviews.py Scraper/google_reviews.py Scraper/Review Searcher.json Scraper/ta_reviews.py sentiments/google_sentiments.py server.js ``` ### Dependencies - package.json: @eslint/js@^9.24.0, acorn@^8.14.1, acorn-jsx@^5.3.2, ajv@^6.12.6, ansi-styles@^4.3.0, argparse@^2.0.1, balanced-match@^1.0.2, brace-expansion@^1.1.11, callsites@^3.1.0, chalk@^4.1.2, color-convert@^2.0.1, color-name@^1.1.4, concat-map@^0.0.1, cross-spawn@^7.0.6, debug@^4.4.0, deep-is@^0.1.4, dotenv@^16.5.0, escape-string-regexp@^4.0.0, eslint@^9.24.0, eslint-scope@^8.3.0, eslint-visitor-keys@^4.2.0, espree@^10.3.0, esquery@^1.6.0, esrecurse@^4.3.0, estraverse@^5.3.0, esutils@^2.0.3, express@^5.1.0, fast-deep-equal@^3.1.3, fast-json-stable-stringify@^2.1.0, fast-levenshtein@^2.0.6, file-entry-cache@^8.0.0, find-up@^5.0.0, flat-cache@^4.0.1, flatted@^3.3.3, glob-parent@^6.0.2, globals@^16.0.0, has-flag@^4.0.0, ignore@^5.3.2, import-fresh@^3.3.1, imurmurhash@^0.1.4, is-extglob@^2.1.1, is-glob@^4.0.3, isexe@^2.0.0, js-yaml@^4.1.0, json-buffer@^3.0.1, json-schema-traverse@^0.4.1, json-stable-stringify-without-jsonify@^1.0.1, keyv@^4.5.4, levn@^0.4.1, locate-path@^6.0.0, lodash.merge@^4.6.2, minimatch@^3.1.2, ms@^2.1.3, natural-compare@^1.4.0, nodemon@^3.1.9, optionator@^0.9.4, p-limit@^3.1.0, p-locate@^5.0.0, parent-module@^1.0.1, path-exists@^4.0.0, path-key@^3.1.1, prelude-ls@^1.2.1, punycode@^2.3.1, resolve-from@^4.0.0, shebang-command@^2.0.0, shebang-regex@^3.0.0, strip-json-comments@^3.1.1, supports-color@^7.2.0, type-check@^0.4.0, uri-js@^4.4.1, which@^2.0.2, word-wrap@^1.2.5, yocto-queue@^0.1.0 - requirements.txt: bs4, requests, selenium@==4.9.0, undetected-chromedriver@==3.5.5 ### Recent commits (newest first) - fb headless false - remove comments - Merge branch 'main' of github.com:Ant-28/reviewtracker - devpost commit 3 - add link to devpost.md, final style changes - devpost commit 2 - devpost commit - updated README re: http warning - updated README - close box fix - js click testing - Does this work: Yes (mostly) pls test - Merge branch 'facebook' - improvements (potential) to facebook reviews - add FB overall rating on page - add FB reviews, display on frontend, limit card height - Merge branch 'facebook' - implemented basic facebook version - implemented basic facebook version - Parallelize langflow reviews & google scraped reviews - implement on frontend too ## Key source files (fetched from GitHub, selected and truncated for size) ### devpost.md ```markdown ## Inspiration Review management is quite cumbersome for a plethora of business owners. Having interacted frequently with small businesses and startups, we are attuned to the need for valuable feedback and summaries. With that in mind, we decided to design a review aggregator for small business, akin to Metacritic. ### Relevance to track (Justice) Our project aims to level the playing field for small businesses, especially in an era of stunted economic growth. We designed this app to bring small businesses more insight about their community and their general perception, which is crucial in our modern information-centric era. We also notice that small business are more susceptible to review bombing and other forms of review biases. For example, we noticed a discrepancy in the following figure: reviews with text tend to show businesses in a more positive light than all available reviews. This is especially true for local businesses, given the smaller numbers of review available.. ![image](https://gist.github.com/user-attachments/assets/8be048bd-d27d-4532-ad34-45a01d17f51a) ## What it does Our software collects reviews from Google Maps, Facebook and a multitude of other platforms. We combined deterministic scraping with more entropic LLM scraping using a GPT-4 powered AI agent via [Langflow](https://www.langflow.org/). We also provide an average sentiment analysis across reviews, displaying the frequency of the top 3 sentiments. **Link**: [http://reviewraven.tech/](http://reviewraven.tech/) (Due to limitations of `.tech` domains, you must use `http://` to view the site. It also cannot interface with the backend, unless run locally.) ## How we built it The main driver was a set of webscrapers that collect reviews for businesses, tailored to each review website. These were written in Python using Selenium. We wrote a backend in Node.js and Express, which was the client's interface to the review data. The frontend was written in HTML and JS (Vanilla) with Pico CSS as a minimal stylesheet. For the sentiment analysis, we used the [NRC](https://saifmohammad.com/WebPages/NRC-Emotion-Lexicon.htm), in addition to Langflow's LLM features. ## Challenges we ran into (and how we solved them) Our first challenge involved *consistent loading* and preventing our client from being flagged as a spambot. We addressed this by using an "undectectable" driver. Since we were collecting reviews from multiple sources, we had to combine our review data with our sentiment analysis corpus. Furthermore, when using LLMs to aggregate reviews, their reliability varied significantly depending on the model used. Notably, `Gemini 2.0-pro-exp` sometimes found fewer reviews or threw errors, while `GPT 4o-mini` was more consistent as it is a production-ready LLM. ## Accomplishments that we're proud of To begin with, 2/3rds of our team have never participated in a hackathon, so this has been a very enlightening experi [truncated — 1078 more characters] ``` ### requirements.txt ``` bs4 selenium==4.9.0 requests undetected-chromedriver==3.5.5 ``` ### package.json ``` { "name": "cruzhacks25", "version": "1.0.0", "description": "bruno mars", "main": "server.js", "dependencies": { "acorn": "^8.14.1", "acorn-jsx": "^5.3.2", "ajv": "^6.12.6", "ansi-styles": "^4.3.0", "argparse": "^2.0.1", "balanced-match": "^1.0.2", "brace-expansion": "^1.1.11", "callsites": "^3.1.0", "chalk": "^4.1.2", "color-convert": "^2.0.1", "color-name": "^1.1.4", "concat-map": "^0.0.1", "cross-spawn": "^7.0.6", "debug": "^4.4.0", "deep-is": "^0.1.4", "dotenv": "^16.5.0", "escape-string-regexp": "^4.0.0", "eslint-scope": "^8.3.0", "eslint-visitor-keys": "^4.2.0", "espree": "^10.3.0", "esquery": "^1.6.0", "esrecurse": "^4.3.0", "estraverse": "^5.3.0", "esutils": "^2.0.3", "express": "^5.1.0", "fast-deep-equal": "^3.1.3", "fast-json-stable-stringify": "^2.1.0", "fast-levenshtein": "^2.0.6", "file-entry-cache": "^8.0.0", "find-up": "^5.0.0", "flat-cache": "^4.0.1", "flatted": "^3.3.3", "glob-parent": "^6.0.2", "has-flag": "^4.0.0", "ignore": "^5.3.2", "import-fresh": "^3.3.1", "imurmurhash": "^0.1.4", "is-extglob": "^2.1.1", "is-glob": "^4.0.3", "isexe": "^2.0.0", "js-yaml": "^4.1.0", "json-buffer": "^3.0.1", "json-schema-traverse": "^0.4.1", "json-stable-stringify-without-jsonify": "^1.0.1", "keyv": "^4.5.4", "levn": "^0.4.1", "locate-path": "^6.0.0", "lodash.merge": "^4.6.2", "minimatch": "^3.1.2", "ms": "^2.1.3", "natural-compare": "^1.4.0", "nodemon": "^3.1.9", "optionator": "^0.9.4", "p-limit": "^3.1.0", "p-locate": "^5.0.0", "parent-module": "^1.0.1", "path-exists": "^4.0.0", "path-key": "^3.1.1", "prelude-ls": "^1.2.1", "punycode": "^2.3.1", "resolve-from": "^4.0.0", "shebang-command": "^2.0.0", "shebang-regex": "^3.0.0", "strip-json-comments": "^3.1.1", "supports-color": "^7.2.0", "type-check": "^0.4.0", "uri-js": "^4.4.1", "which": "^2.0.2", "word-wrap": "^1.2.5", "yocto-queue": "^0.1.0" }, "devDependencies": { "@eslint/js": "^9.24.0", "eslint": "^9.24.0", "globals": "^16.0.0" }, "scripts": { "test": "echo \"Error: no test specified\" && exit 1", "start": "node server.js" }, "repository": { "type": "git", "url": "git+https://github.com/Ant-28/reviewtracker.git" }, "author": "", "license": "ISC", "bugs": { "url": "https://github.com/Ant-28/reviewtracker/issues" }, "homepage": "https://github.com/Ant-28/reviewtracker#readme" } ``` ### server.js ```javascript // const require = NodeJS.require require("dotenv").config(); const { exec } = require("node:child_process"); const express = require("express"); const app = express(); const PORT = process.env.PORT || 3000; const GOOGLE_TEXT_SEARCH = "https://places.googleapis.com/v1/places:searchText"; const GOOGLE_API_KEY = process.env.GOOG_PLACES_APIKEY; app.use(express.static("frontend")); app.use(express.json()); app.get("/locations/", async (req, res) => { let q = req.query; try { const search_text = await fetch(`${GOOGLE_TEXT_SEARCH}`, { method: "POST", body: JSON.stringify({ textQuery: q.searchText }), headers: { "X-Goog-Api-Key": GOOGLE_API_KEY, "Content-Type": "application/json", "X-Goog-FieldMask": "places.displayName,places.formattedAddress", }, }); const search_json = await search_text.json(); if (!search_json.places) { res.status(404).send(); } console.log(search_json.places[0].formattedAddress); console.log("dishplay name"); // sean connery console.log(search_json.places[0].displayName); const newRes = search_json.places.map((obj) => { return { faddr: obj.formattedAddress, fname: obj.displayName.text, }; }); res.status(200).send(newRes.slice(0, 10)); } catch (error) { console.error(error); res.status(500).send(); } }); function execAsync(command) { return new Promise((resolve, reject) => { exec(command, (error, stdout, stderr) => { if (error) { return reject(error); } resolve({ stdout, stderr }); }); }); } // /reviews/?addrText=... app.post("/reviews/", async (req, res) => { const q = req.body; const langflow_payload = { "input_value": `${q.fname} ${q.faddr}`, "output_type": "chat", "input_type": "chat" }; const options = { method: 'POST', headers: { 'Content-Type': 'application/json' }, body: JSON.stringify(langflow_payload) }; const langflowPromise = fetch('http://127.0.0.1:7860/api/v1/run/01b79eb9-710c-4d6c-ad1b-cb83f245d67c', options) .then(response => response.json()) .then(response => { // Pull JSON block out of llm response const codeblock_response = response["outputs"][0]["outputs"][0]["results"]["message"]["text"]; // Strip out leading ```json\n and trailing \n``` const json_string = codeblock_response.substring(8, codeblock_response.length - 3); const json_obj = JSON.parse(json_string); const allReviews = json_obj.flatMap(source => source.reviews.map(r => r.review) ); console.log("ALL REVIEWS", allReviews); return allReviews; }); const execPromise = execAsync(`"python3" scraper/google_reviews.py "${q.fname} ${q.faddr}"`) .then(({ stdout }) => JSON.parse(stdout)); const execPromiseFB = execAsync(`"python3" scraper/facebook_reviews.py "${q.fname} ${q.faddr}"`) .then(({ stdout }) => JSON.parse(stdout)); // Run both langflow fetch and google scraper in parallel. and facebook scraper try { const [langflowResponse, googleResponse, fbResponse] = await Promise.allSettled([langflowPromise, execPromise, execPromiseFB]); // Send back an object including keys for all three sources. // If any promise was rejected, that key's value will be empty object in response. res.status(200).json({ llmReviews: langflowResponse.status === "fulfilled" ? langflowResponse.value : [], googleReviews: googleResponse.status === "fulfilled" ? googleResponse.value : {}, fbReviews: fbResponse.status === "fulfilled" ? fbResponse.value : {} }); } catch (error) { console.error(error); res.status(500).send(); } }); app.listen(PORT, () => { console.log(`Testing on port ${PORT}`); }); ``` ### frontend/index.js ```javascript const reviews = [ [ "..." ], [ "..." ] ]; const emojiMap = { 'anger': '😡', 'anticipation': '👀', 'disgust': '🤢', 'fear': '😨', 'joy': '😂', 'sadness': '☹️', 'surprise': '🤯', 'trust': '🤝' } // Wait till full page loads to run script document.addEventListener("DOMContentLoaded", () => { // Populate Google reviews into DOM const col1 = document.getElementById("col1"); reviews[0].forEach(review => { const article = document.createElement("article"); article.textContent = review; col1.appendChild(article); }); // Populate all other reviews into DOM const col2 = document.getElementById("col2"); reviews[1].forEach(review => { const article = document.createElement("article"); article.textContent = review; col2.appendChild(article); }); }); // Handler for search box: send text query, receive places, // display options in table with accept buttons document.getElementById("search").addEventListener("keydown", async (e) => { if (e.key === "Enter") { const query = e.target.value; // Clear text box e.target.value = ""; const params = new URLSearchParams(); params.append("searchText", query); const response = await fetch(`/locations?${params}`); if (response.status === 404) { // No places found. document.getElementById('business-name').textContent = "No places found..."; } // At least 1 place found. const data = await response.json(); const table = document.getElementById("results"); table.style.display = "block"; // Clear existing rows in the table table.innerHTML = ""; // Populate table with places data.forEach(place => { const row = document.createElement("tr"); // Name column const nameCell = document.createElement("td"); nameCell.textContent = place.fname; row.appendChild(nameCell); // Address column const addressCell = document.createElement("td"); addressCell.textContent = place.faddr; row.appendChild(addressCell); // Button column const buttonCell = document.createElement("td"); const selectButton = document.createElement("button"); selectButton.textContent = "Select"; // Add event listener to the button selectButton.addEventListener("click", () => handleButtonClick(place)); buttonCell.appendChild(selectButton); row.appendChild(buttonCell); table.appendChild(row); }); } }); // Function to handle button click async function handleButtonClick(place) { document.getElementById('business-name').textContent = `${place.fname} -- ${place.faddr}`; // Hide the table document.getElementById("results").style.display = "none"; // Show indeterminate progress bar document.getElementById("indeterminate").style.display = "block"; try { const response = await fetch("/reviews", { method: "POST", headers: { "Content-Type": "application/json" }, body: JSON.stringify(place) }); const data = await response.json(); const {llmReviews, googleReviews, fbReviews} = data; // Hide progress bar document.getElementById("indeterminate").style.display = "none"; // Populate Google reviews into page const col1 = document.getElementById("col1"); col1.innerHTML = ""; // Clear existing reviews googleReviews.reviews?.forEach(review => { const article = document.createElement("article"); article.textContent = review; col1.appendChild(article); }); // Populate LLM reviews into Other column const col2 = document.getElementById("col2"); col2.innerHTML = ""; // Clear existing reviews llmReviews.forEach(review => { const article = document.createElement("article"); article.textContent = review; col2.appendChild(article); }); // Populate Facebook reviews into col3 const col3 = document.getElementById("col3"); col3.innerHTML = ""; // Clear existing reviews fbReviews.reviews.forEach(review => { const article = document.createElement("article"); article.textContent = review; col3.appendChild(article); }); // Populate overall rating amount and subtitle document.getElementById('google-progress').value = googleReviews.overall_avg_rating; document.getElementById('google-subtitle').textContent = `Google rating: ${googleReviews.overall_avg_rating} ★`; document.getElementById('facebook-progress').value = fbReviews.overall_avg_rating; document.getElementById('facebook-subtitle').textContent = `Facebook rating: ${fbReviews.overall_avg_rating} ★`; // Show sentiment on page. // Extract top 3 sentiments const topSentiments = Object.entries(googleReviews.sentiments) .sort(([, a], [, b]) => b - a) // Sort by float values in descending order .slice(0, 3) // Take the top 3 .map(([key]) => key); // Extract the keys // Display the top 3 sentiments document.getElementById("sentiment").textContent = `Top sentiments: \r\n ${emojiMap[topSentiments[0]]} ${topSentiments[0]} (${(googleReviews.sentiments[topSentiments[0]] * 100).toFixed(2)}%) \r\n ${emojiMap[topSentiments[1]]} ${topSentiments[1]} (${(googleReviews.sentiments[topSentiments[1]] * 100).toFixed(2)}%) \r\n ${emojiMap[topSentiments[2]]} ${topSentiments[2]} (${(googleReviews.sentiments[topSentiments[2]] * 100).toFixed(2)}%)`; } catch (error) { console.error("Error:", error); } } ``` ### sentiments/google_sentiments.py ```python import os import sys from typing import Dict curr_dir = os.path.dirname(os.path.realpath(__file__)) NRCPY_DIR = os.path.abspath(os.path.join(curr_dir, '..', "nrcpy")) if not NRCPY_DIR in sys.path : sys.path.append(NRCPY_DIR) import emotion_analysis EMOLEXL = emotion_analysis.get_emolex(os.path.join(NRCPY_DIR, "nrc_emolex.json")) def sentims(payload : Dict) -> Dict: reviews = payload["reviews"] all_reviews = " ".join(reviews) sentiments_dict, emotion_lvl, _, _ = emotion_analysis.emolex(all_reviews, EMOLEXL) # divide by emotion_lvl to get frequency sentiment_frequency = {k:v/emotion_lvl for k, v in sentiments_dict.items()} if emotion_lvl != 0 else sentiments_dict # Delete key 'trust' -- appears with too high of a ranking. if "trust" in sentiment_frequency: del sentiment_frequency["trust"] payload["sentiments"] = sentiment_frequency return payload ``` ### frontend/index.html ```html Review Raven
Logo

Review Raven

Business: --

Statistics

Google rating
Facebook rating

Sentiment

--

Reviews

Google

Facebook

Other Sites

``` ### Scraper/ta_reviews.py ```python from selenium import webdriver from selenium.webdriver.common.by import By from selenium.webdriver.support.ui import WebDriverWait from selenium.webdriver.support import expected_conditions as EC from selenium.common.exceptions import TimeoutException from sys import argv import undetected_chromedriver as uc def get_reviews_by_address(address: str, driver: uc.Chrome) -> dict: """ Function to scrape TripAdvisor reviews by address. """ driver.get("https://www.tripadvisor.com/") wait = WebDriverWait(driver, 15) # Set window width to 700px to hide sideabr window_size = driver.execute_script(""" return [window.outerWidth - window.innerWidth + arguments[0], window.outerHeight - window.innerHeight + arguments[1]]; """, 700, 700) driver.set_window_size(*window_size) # Wait for the search box to be present and input the address search_box = wait.until(EC.presence_of_element_located( (By.XPATH, '//input[@type="search" and @aria-label="Search" and @role="searchbox"]') )) WebDriverWait(driver, 10).until(EC.element_to_be_clickable(search_box)) search_box.click() print(search_box) search_box.send_keys(address) # Press Enter to search search_box.send_keys(u'\ue007') # Click on the first result first_result = wait.until(EC.presence_of_element_located((By.XPATH, '//a[@class="BMQDV _F Gv wSSLS SwZTJ FGwzt ukgoS"]'))) first_result.click() # Wait for the reviews button to be clickable and click it # xpath = '//div[contains(text(), "Reviews")]' # try: # reviews_button = wait.until(EC.element_to_be_clickable((By.XPATH, xpath))) # reviews_button.click() # except TimeoutException: # print("Failed to find reviews button") # return # TODO: this does not click "Read More" to load full reviews. # Scroll to load reviews review_div_class = "fIrGe _T bgMZj" for _ in range(5): try: scrollable_element = WebDriverWait(driver, 5).until( EC.presence_of_element_located((By.XPATH, f"//div[@class='{review_div_class}']")) ) # Extra check to ensure it's scrollable if driver.execute_script("return arguments[0].scrollHeight > arguments[0].clientHeight;", scrollable_element): break except: continue # Scroll whenever the height changes. previous_height = driver.execute_script("return arguments[0].scrollHeight", scrollable_element) for _ in range(7): # Scroll multiple times to load more reviews # Scroll to the bottom of the scrollable element driver.execute_script("arguments[0].scrollTop = arguments[0].scrollHeight", scrollable_element) # Wait for the scrollable element's height to increase WebDriverWait(driver, 2).until( lambda d: d.execute_script("return arguments[0].scrollHeight", scrollable_element) > previous_height ) # Update the previous height for the next iteration previous_height = driver.execute_script("return arguments[0].scrollHeight", scrollable_element) # Fetch reviews reviews = scrollable_element.find_elements(By.XPATH, '//*[@class="MyEned"]') # see_more_button = driver.find_elements(By.XPATH, './/button[@aria-label="See more"]') # for button in see_more_button: # try: # button.click() # except Exception as e: # pass print(len(reviews)) return { "reviews": [review.text for review in reviews], } def main(argv): # Set up emulated browser options driver = uc.Chrome( # headless=True, use_subprocess=False ) payload = get_reviews_by_address(argv[1], driver) print(payload) if __name__ == "__main__": main(argv) ``` ### Scraper/facebook_reviews.py ```python from selenium import webdriver from selenium.webdriver.common.by import By from selenium.webdriver.support.ui import WebDriverWait from selenium.webdriver.support import expected_conditions as EC from selenium.common.exceptions import TimeoutException from sys import argv, exit, stderr import time import os import sys import json from typing import Optional, Dict import undetected_chromedriver as uc import re # https://stackoverflow.com/questions/5925918/python-suppressing-errors-from-going-to-commandline class ourChrome(uc.Chrome): def __del__(self): try: self.service.process.kill() self.quit() except: # noqa pass # CHANGE THIS FOR FACEBOOK curr_dir = os.path.dirname(os.path.realpath(__file__)) SENTIMENTS_DIR = os.path.abspath(os.path.join(curr_dir, '..', "sentiments")) if not SENTIMENTS_DIR in sys.path : sys.path.append(SENTIMENTS_DIR) import google_sentiments as gsent MAX_ELEMS = 10 def get_facebook_site(name : str, driver: uc.Chrome) -> Optional[str]: # get name of place, return optional string driver.get("https://www.google.com") wait = WebDriverWait(driver, 15) search_box = wait.until(EC.presence_of_element_located((By.ID, "APjFqb"))) driver.execute_script("arguments[0].click();", search_box) search_box.send_keys(f"{name} site:facebook.com") search_box.send_keys(u'\ue007') # //*[@id="rso"]/div[1]/div/div/div[1]/div/div[2]/div/div/span/a xpath = "//*[@id='rso']/div[1]/div/div/div[1]/div/div[2]/div/div/span/a" site_name = None try: reviews_button = WebDriverWait(driver, 10).until(EC.element_to_be_clickable((By.XPATH, xpath))) site_name = reviews_button.get_attribute("href") # return site_name except TimeoutException: print("Failed to find website", file=stderr) exit(1) driver.get(site_name) wait = WebDriverWait(driver, 15) # Wait for the search box to be present and input the address close_box = wait.until(EC.element_to_be_clickable((By.XPATH, '//*[@aria-label="Close"]'))) driver.execute_script("arguments[0].click();", close_box) review_score_xpath = '/html/body/div[1]/div/div[1]/div/div[3]/div/div/div[1]/div[1]/div/div/div[4]/div[2]/div/div[1]/div[2]/div/div[1]/div/div/div/div/div[2]/div[2]/div/ul/div[9]/div[2]/a/div/div/span' review_score_box = wait.until(EC.presence_of_element_located((By.XPATH, review_score_xpath))) overall_score_fb = float(re.search(r"(\d+)\%.*", review_score_box.text).group(1))/100 * 5 driver.get(f"{site_name}reviews") close_box = wait.until(EC.element_to_be_clickable((By.XPATH, '//*[@aria-label="Close"]'))) driver.execute_script("arguments[0].click();", close_box) # I hate this website driver.execute_script("a = document.querySelector('html'); a.scrollTop = 0.45*a.scrollHeight; a.scrollTop = 0.45*a.scrollHeight;") reviews = [] for n in range(1, 20): try: reviews_xpath = f"/html/body/div[1]/div/div[1]/div/div[3]/div/div/div[1]/div[1]/div/div/div[4]/div/div/div/div[{n}]/div/div/div/div/div/div/div/div/div/div[2]/div/div/div[3]/div[1]/div/div/div/div/span/div" review_data = WebDriverWait(driver, 1).until(EC.presence_of_element_located((By.XPATH, reviews_xpath))) reviews.append(review_data.text) except: pass reviews_dict = { "reviews" : reviews, "text_avg_rating" : overall_score_fb, "overall_avg_rating" : overall_score_fb } reviews_with_sentiments = gsent.sentims(reviews_dict) print(json.dumps(reviews_with_sentiments)) def main(argv): # Set up headless browser options options = webdriver.ChromeOptions() options.page_load_strategy = "eager" # options.add_experimental_option( # "prefs", {"profile.managed_default_content_settings.images": 2} # ) options.add_argument("--headless=new") # preferences = { # "profile.managed_default_content_settings.images": 2, # "profile.default_content_settings.images": 2 # } options.add_argument(f'--disk-cache-dir={os.path.dirname(os.path.realpath(__file__))}') # options.add_argument("--window-size=0,0") # remove devtools listening # options.add_experimental_option('excludeSwitches', ['enable-logging']) # options.add_experimental_option("prefs", preferences) driver = ourChrome( use_subprocess=False, headless=False, version_main=112 ) # driver = webdriver.Chrome(options=options) # print(f"\033[31;1;{argv[1]}\033[0m", file=stderr) get_facebook_site(argv[1], driver) # get_reviews_by_address(linkname, driver) # payload = get_reviews_by_address(argv[1], driver) # print(json.dumps(payload)) if __name__ == "__main__": try: main(argv) except: print(json.dumps({ "reviews" : [], "text_avg_rating" : 0, "overall_avg_rating" : 0, "sentiments" : {}, })) ``` ### Scraper/google_reviews.py ```python from selenium import webdriver from selenium.webdriver.common.by import By from selenium.webdriver.support.ui import WebDriverWait from selenium.webdriver.support import expected_conditions as EC from selenium.common.exceptions import TimeoutException from sys import argv, exit, stderr import time import os import sys import json curr_dir = os.path.dirname(os.path.realpath(__file__)) SENTIMENTS_DIR = os.path.abspath(os.path.join(curr_dir, '..', "sentiments")) if not SENTIMENTS_DIR in sys.path : sys.path.append(SENTIMENTS_DIR) import google_sentiments as gsent MAX_ELEMS = 10 def get_reviews_by_address(address, driver): """ Function to scrape Google reviews by address. """ driver.get("https://www.google.com/maps") wait = WebDriverWait(driver, 30) # Wait for the search box to be present and input the address search_box = wait.until(EC.presence_of_element_located((By.ID, "searchboxinput"))) search_box.click() search_box.send_keys(address) # Press Enter to search search_box.send_keys(u'\ue007') # Wait for the reviews button to be clickable and click it xpath = '//div[contains(text(), "Reviews")]' try: reviews_button = wait.until(EC.element_to_be_clickable((By.XPATH, xpath))) reviews_button.click() except TimeoutException: print("Failed to find reviews button", file=stderr) exit(1) # Scroll to load reviews for xpath in [ '//*[@id="QA0Szd"]/div/div/div[1]/div[2]/div/div[1]/div/div/div[4]', '//*[@id="QA0Szd"]/div/div/div[1]/div[2]/div/div[1]/div/div/div[2]' ]: try: scrollable_element = WebDriverWait(driver, 10).until( EC.presence_of_element_located((By.XPATH, xpath)) ) # Extra check to ensure it's scrollable if driver.execute_script("return arguments[0].scrollHeight > arguments[0].clientHeight;", scrollable_element): break except: continue # Scroll whenever the height changes. current_height = -1 previous_height = driver.execute_script("return arguments[0].scrollHeight", scrollable_element) num_reviews = 0 while(current_height < previous_height): # Scroll multiple times to load more reviews # Update the current height for the current iteration num_curr_elems = len(scrollable_element.find_elements(By.XPATH, '//*[@class="MyEned"]')) if num_curr_elems > MAX_ELEMS: break current_height = driver.execute_script("return arguments[0].scrollHeight", scrollable_element) # Scroll to the bottom of the scrollable element driver.execute_script("arguments[0].scrollTop = arguments[0].scrollHeight", scrollable_element) # Wait for the scrollable element's height to increase try: WebDriverWait(driver, 2).until( lambda d: d.execute_script("return arguments[0].scrollHeight", scrollable_element) > previous_height ) except TimeoutException: break # Update the previous height for the next iteration previous_height = driver.execute_script("return arguments[0].scrollHeight", scrollable_element) # Fetch reviews reviews = scrollable_element.find_elements(By.XPATH, '//*[@class="MyEned"]') while (True): see_more_button = driver.find_elements(By.XPATH, './/button[@aria-label="See more"]') if not see_more_button: break for button in see_more_button: try: button.click() except Exception as e: pass # find div with role img and aria-label that contains star star_div = driver.find_element(By.XPATH, '//div[@role="img" and @aria-label]') star_rating = star_div.get_attribute("aria-label") # all elements where aria label contains star star_elements = driver.find_elements(By.XPATH, '//span[@class="kvMYJc"]') stars = [] for star in star_elements: rating = len(star.find_elements(By.XPATH, './/span[@class="hCCjke google-symbols NhBTye elGi1d"]')) stars.append(rating) average_rating = (sum(stars[:len(reviews)]) / len(reviews)) if len(reviews) > 0 else 0 # print(len(reviews)) reviews_dict = { "reviews": [review.text for review in reviews], "text_avg_rating": round(average_rating, 2), "overall_avg_rating": float(star_rating.split(" ")[0]), } reviews_with_sentiments = gsent.sentims(reviews_dict) return reviews_with_sentiments def main(argv): # Set up headless browser options options = webdriver.ChromeOptions() options.page_load_strategy = "eager" # options.add_experimental_option( # "prefs", {"profile.managed_default_content_settings.images": 2} # ) # options.add_argument("--headless") # preferences = { # "profile.managed_default_content_settings.images": 2, # "profile.default_content_settings.images": 2 # } options.add_argument(f'--disk-cache-dir={os.path.dirname(os.path.realpath(__file__))}') # remove devtools listening options.add_experimental_option('excludeSwitches', ['enable-logging']) # options.add_experimental_option("prefs", preferences) driver = webdriver.Chrome(options=options) # print(f"\033[31;1;{argv[1]}\033[0m", file=stderr) payload = get_reviews_by_address(argv[1], driver) print(json.dumps(payload)) if __name__ == "__main__": main(argv) ```