# Project export: EcoGauge This document was generated by HackStack to give an AI agent context about a hackathon project. Sections are labeled with their provenance; content marked as truncated was cut to keep this document small. ## Project metadata - Hackathon: Cal Hacks 11.0 - Tagline: EcoGauge empowers activists, policymakers, and the public with real-time sentiment insights on environmental issues, driving informed change and efficient action. - Devpost: https://devpost.com/software/ecogauge-1o0hxm - GitHub: https://github.com/aaronsongnguyen/EcoGauge - Video: https://www.youtube.com/embed/4eoO9756PdA?enablejsapi=1&hl=en_US&rel=0&start=&version=3&wmode=transparent - Team: 3 GitHub contributor(s) — aaronsongnguyen (29 commits), savagesausage04 (5 commits), ConnorCho (1 commits) ## Devpost submission (written by the team) ### Inspiration Our inspiration for EcoGauge stems from the growing concern around climate change and environmental sustainability. We recognized that while there are many discussions about sustainability happening across online platforms, it's challenging for individuals, organizations, and policymakers to easily gauge public sentiment on key environmental topics. We wanted to create a tool that empowers environmental advocates and decision-makers by providing clear, data-driven insights into how the public feels about pressing issues like renewable energy, conservation, and sustainable agriculture. ### What it does EcoGauge empowers sustainability advocates, nonprofits, and researchers by providing real-time insights into public sentiment on environmental and sustainability-related issues. The application measures whether online communities correlate the topic at hand to be positive, negative, or neutral and offers actionable steps to improve public sentiment by analyzing the constructive comments related to the sustainability issue. ### How we built it We utilized BeautifulSoup, Selenium, and Pandas to scrape and pre-process Instagram threads data to collect string comments. Simultaneously, we trained a scikit-learn SVM model using 10,000 Amazon reviews to capture the wide range of sentiments. Then, we inputted the cleaned threads into the model and used a TF-IDF vectorizer to turn strings into numerical data based on text frequency and return a classified sentiment. We used Matplotlib to display a distribution of sentiments and Gemini API to analyze constructive comments and offer constructive feedback. Lastly, we used Reflex to design the frontend with Python and present our product in a clean and organized manner. ### Challenges we ran into One of our main challenges was to web scrape information and turn it into data that we can feed into our trained sentiment analysis model. This was difficult as we had to find which social media platform both fit with goals and compatible with our program. Another challenge we ran into was linking the frontend and backend. Though we used Reflex to simplify frontend development by using Python for web development, we struggled with populating data such as the Matplotlib chart. We also struggled to convert the product in local host format to being deployed for external use. ### Accomplishments we're proud of We are proud that we were able to combine our skills and apply it to something actually impactful to the environmental community. As it is our first hackathon, we wanted to create an impactful application that helps a specific community after realizing how underrepresented the general public is in policy making and larger decisions. As the importance of sustainable action grows, we hope our product will give policymakers convenient access to insights on public opinion. ### What we learned With this being our first hackathon, we learned how to adapt in high pressure environments. While trying to create the best project in only 48 hours, we were forced to learn new skills such as Reflex. We also learned many non-coding aspects of creating a product from start to finish such as navigating git and file structure. ### What's next Though EcoGauge was designed to tackle the lack of awareness regarding sustainability, we see avenues to scale the application into other industries beyond sustainability. EcoGauge can be particularly beneficial for public figures, who can gauge public sentiment on themselves by using our tool to aggregate the most common points of feedback. Generally, any industry where the collection of public opinion is useful can be a new door for EcoGauge! ## README (from the GitHub repository) EcoGauge empowers activists, policymakers, and the public with real-time sentiment insights on environmental issues, driving informed change and efficient action. ## Detected evidence (automated analysis) Indexed codebase: 6 recognized source files, 30 KB. - Python (language) — detected in the code - Google Gemini (technology) — claimed on Devpost, not found in the code ## Codebase structure (from repository index) ### Files (8 of 8) ``` front_end/.gitignore front_end/front_end/__init__.py front_end/front_end/front_end.py front_end/main.py front_end/requirements.txt front_end/rxconfig.py front_end/webscrapyer.py README.md ``` ### Dependencies - front_end/requirements.txt: reflex@==0.6.3 ### Recent commits (newest first) - Update README.md - Update README.md - Create README.md - Delete CNAME - Create CNAME - Merge pull request #11 from aaronsongnguyen/aaron - final project - 5:40am - full implementation - Merge pull request #10 from aaronsongnguyen/aaron - full change, just need to implement into the frontend py - Merge pull request #9 from aaronsongnguyen/aaron - fitt update - Merge branch 'main' of https://github.com/aaronsongnguyen/Sentiment into aaron - lol - Merge pull request #8 from aaronsongnguyen/kyle's-branch - gpt stuff - Merge pull request #7 from aaronsongnguyen/aaron - all updated, now needed data - Merge pull request #6 from aaronsongnguyen/aaron - Merge pull request #5 from aaronsongnguyen/kyle's-branch ## Key source files (fetched from GitHub, selected and truncated for size) ### front_end/requirements.txt ``` reflex==0.6.3 ``` ### front_end/main.py ```python import json import random import numpy as np import matplotlib.pyplot as plt import seaborn from sklearn.model_selection import train_test_split from sklearn.feature_extraction.text import CountVectorizer, TfidfVectorizer from sklearn.tree import DecisionTreeClassifier from sklearn.naive_bayes import GaussianNB from sklearn import svm from sklearn.linear_model import LogisticRegression from sklearn.model_selection import GridSearchCV from sklearn.metrics import f1_score from collections import Counter from reflex_pyplot import pyplot API_KEY = 'AIzaSyA-Y9XSXMfroQnyOgzpfuW6fS0M4vlIrRI' import google.generativeai as genai import os from webscrapyer import thread_list class Sentiment: NEGATIVE = "Negative" POSITIVE = "Positive" NEUTRAL = "Neutral" class Review: def __init__(self, text, score): self.text = text self.score = score self.sentiment = self.get_sentiment() def get_sentiment(self): if self.score <= 2: return Sentiment.NEGATIVE elif self.score == 3: return Sentiment.NEUTRAL else: return Sentiment.POSITIVE class ReviewContainer: def __init__(self, reviews): self.reviews = reviews def get_text(self): return [x.text for x in self.reviews] def get_sentiment(self): return [x.sentiment for x in self.reviews] def evenly_distribute(self): negative = list(filter(lambda x: x.sentiment == Sentiment.NEGATIVE, self.reviews)) positive = list(filter(lambda x: x.sentiment == Sentiment.POSITIVE, self.reviews)) neutral = list(filter(lambda x: x.sentiment == Sentiment.NEUTRAL, self.reviews)) positive_shrunk = positive[:len(negative)] self.reviews = negative + positive_shrunk + neutral random.shuffle(self.reviews) file_name = '/Users/aaronnguyen/Desktop/Books_small_10000.json' yelp_reviews = thread_list reviews = [] with open(file_name) as f: for line in f: review = json.loads(line) reviews.append(Review(review['reviewText'], review['overall'])) #creates a list with Review objects with text and score attributes training, testing = train_test_split(reviews, test_size=0.9, random_state=42) train_container = ReviewContainer(training) test_container = ReviewContainer(testing) train_container.evenly_distribute() training_x = train_container.get_text() training_y = train_container.get_sentiment() test_container.evenly_distribute() testing_x = test_container.get_text() testing_y = test_container.get_sentiment() training_y.count(Sentiment.POSITIVE) training_y.count(Sentiment.NEGATIVE) #vectorizes training and testing text data into numerical format vectorizer = TfidfVectorizer() training_x_vectors = vectorizer.fit_transform(training_x) #testing_x_vectors = vectorizer.transform(testing_x) #scikit learn svm model clf_svm = svm.SVC(kernel='linear') clf_svm.fit(training_x_vectors, training_y) #clf_svm.predict(testing_x_vectors[0]) #scikit decision tree model ''' clf_dec = DecisionTreeClassifier() clf_dec.fit(training_x_vectors, training_y) clf_dec.predict(testing_x_vectors[0]) #scitkit gaussian naive bayes model clf_gnb = GaussianNB() training_x_vectors_dense = training_x_vectors.toarray() testing_x_vectors_dense = testing_x_vectors.toarray() clf_gnb.fit(training_x_vectors_dense, training_y) clf_gnb.predict(testing_x_vectors_dense[0].reshape(1, -1)) #scikit logistic regression model clf_log = LogisticRegression() clf_log.fit(training_x_vectors, training_y) clf_log.predict(testing_x_vectors[0]) ''' #prints the accuracy of each model ''' print(f'This is the initial svm model accuracy: {clf_svm.score(testing_x_vectors, testing_y)}') print(f'This is the initial dec model accuracy: {clf_dec.score(testing_x_vectors, testing_y)}') print(f'This is the initial gnb model accuracy: {clf_gnb.score(testing_x_vectors_dense, testing_y)}') print(f'This is the initial log model accuracy: {clf_log.score(testing_x_vectors, testing_y)}') ''' #f1_score(testing_y, clf_svm.predict(testing_x_vectors), average=None, labels=[Sentiment.POSITIVE, Sentiment.NEGATIVE]) #yelp_reviews = ['the customer service was not that great given the fact that they served me raw chicken', "wow this restaurant is really favorable", "horrible waste of time"] new_test = vectorizer.transform(yelp_reviews) print('\n') sentiment_list = clf_svm.predict(new_test) print(f'This is the prediction results for the sample test set: {sentiment_list}') ''' #parameter tuning to increase model accuracy parameters = {'kernel': ('linear', 'rbf'), 'C': (1, 4, 8, 16, 32)} svc = svm.SVC() clf = GridSearchCV(svc, parameters, cv=5) clf.fit(training_x_vectors, training_y) print('\n') #print(f'This is the svm accuracy post parameter tuning: {clf.score(testing_x_vectors, testing_y)}') ''' sentiment_counts = Counter(sentiment_list) total_sentiments = len(sentiment_list) proportions = {label: count / total_sentiments for label, count in sentiment_counts.items()} proportions_list = list(proportions.values()) keys = ["Negative", "Positive", "Neutral"] print(proportions_list) palette_color = seaborn.color_palette('bright') # plt.pie(proportions_list, labels=keys, colors=palette_color, autopct='%.0f%%') # plt.title("Distribution of Media Sentiment towards Sustainability Topic") # plt.savefig('assets/sentiment_chart.png') # plt.close() genai.configure(api_key=API_KEY) model = genai.GenerativeModel("gemini-1.5-flash") response = model.generate_content(f"Here are some of the comments from yelp reviews. Take some of the criticisms/negative comments and offer some points of future improvements. Make the feedback short, 3 improvements one line each. The text that follows are the comments: {yelp_reviews}") print(response.text) ``` ### front_end/rxconfig.py ```python import reflex as rx config = rx.Config( app_name="front_end", ) ``` ### front_end/webscrapyer.py ```python import time from selenium import webdriver from selenium.webdriver.common.by import By from selenium.webdriver.support.ui import WebDriverWait from selenium.webdriver.support import expected_conditions as EC from selenium.webdriver.chrome.options import Options from bs4 import BeautifulSoup def scrape_threads(url): # Set up Chrome options chrome_options = Options() chrome_options.add_argument("--headless") # Run in headless mode # Initialize the WebDriver driver = webdriver.Chrome(options=chrome_options) try: # Navigate to the URL driver.get(url) # Wait for the content to load (adjust the timeout and conditions as needed) WebDriverWait(driver, 10).until( EC.presence_of_element_located((By.CLASS_NAME, "x1lliihq")) ) # Allow some time for dynamic content to load time.sleep(5) # Get the page source and parse it with BeautifulSoup soup = BeautifulSoup(driver.page_source, 'html.parser') # Find all thread elements thread_elements = soup.find_all('span', class_='x1lliihq') # Extract the text from each thread element threads = [thread.get_text(strip=True) for thread in thread_elements] return threads except Exception as e: print(f"An error occurred: {e}") return [] finally: # Close the browser driver.quit() # URL of the Threads page you want to scrape topic = "Input" topics = topic.split() if len(topics) == 1: url = f"https://www.threads.net/search?q={topic}&serp_type=default" elif len(topics) == 2: url = f"https://www.threads.net/search?q={topics[0]}%20{topics[1]}&serp_type=default" # Scrape the threads thread_list = scrape_threads(url) ``` ### front_end/front_end/front_end.py ```python import reflex as rx from webscrapyer import scrape_threads import json import random import numpy as np from matplotlib import pyplot as plt import seaborn from sklearn.model_selection import train_test_split from sklearn.feature_extraction.text import CountVectorizer, TfidfVectorizer from sklearn.tree import DecisionTreeClassifier from sklearn.naive_bayes import GaussianNB from sklearn import svm from sklearn.linear_model import LogisticRegression from sklearn.model_selection import GridSearchCV from sklearn.metrics import f1_score from collections import Counter from typing import List from reflex_pyplot import pyplot API_KEY = "AIzaSyA-Y9XSXMfroQnyOgzpfuW6fS0M4vlIrRI" import google.generativeai as genai import os from webscrapyer import thread_list colors = { "background": "#E6E6FA", # Lavender purple "text": "#333333", # Dark gray for main text "accent_blue": "#007acc", # Blue for header background and accents "accent_green": "#A8E6CE", # Light green for some text and button "input_bg": "#f0f0f0", # Light gray for input background "input_text": "#333333", # Dark gray for input text "hover_teal": "#2BB673", # Darker teal for button hover "white": "#FFFFFF", "light_green": "#6cc767", } button_hover_effects = """ """ custom_css = """ @keyframes scroll { 0% { transform: translateX(0); } 100% { transform: translateX(-50%); } } """ fruits = [ "Pollution", "☘︎", "Ecosystems", "☘︎", "Biodiversity", "☘︎", "Footprint", "☘︎", "Recycle", "☘︎", "Agriculture", "☘︎", "Resources", "☘︎", "Energy", "☘︎", "Solar", "☘︎", "Geothermal", "☘︎", "Emissions", "☘︎", "Global Warming", "☘︎", "Environmental Activism", "☘︎", "Urban Planning", "☘︎", "Climate", "☘︎", "Carbon Neutral", "☘︎", "Biomass", "☘︎", "Wind Power", "☘︎", "Electricity", "☘︎", "Greenhouse Gas", "☘︎", ] class State(rx.State): business_id: str = "" show_results: bool = False next_steps: List[str] = [] sentiment_summary: str = "" plot_figure_data = [5, 5, 5] plot_figure_labels = ["Negative", "Positive", "Neutral"] saved_figure = False show_about_us: bool = False def toggle_about_us(self): self.show_about_us = not self.show_about_us # def get_pyplot(self): # data: dict = {} # pyplot. @rx.var(cache=True) def pie_maker(self) -> plt.Figure: labels = self.plot_figure_labels sizes = self.plot_figure_data fig, ax = plt.subplots() ax.pie(sizes, labels=labels) return fig def submit(self): if self.business_id: self.show_results = True topic = self.business_id topics = topic.split() if len(topics) == 1: url = f"https://www.threads.net/search?q={topic}&serp_type=default" elif len(topics) == 2: url = f"https://www.threads.net/search?q={topics[0]}%20{topics[1]}&serp_type=default" thread_list = scrape_threads(url) class Sentiment: NEGATIVE = "Negative" POSITIVE = "Positive" NEUTRAL = "Neutral" class Review: def __init__(self, text, score): self.text = text self.score = score self.sentiment = self.get_sentiment() def get_sentiment(self): if self.score <= 2: return Sentiment.NEGATIVE elif self.score == 3: return Sentiment.NEUTRAL else: return Sentiment.POSITIVE class ReviewContainer: def __init__(self, reviews): self.reviews = reviews def get_text(self): return [x.text for x in self.reviews] def get_sentiment(self): return [x.sentiment for x in self.reviews] def evenly_distribute(self): negative = list( filter( lambda x: x.sentiment == Sentiment.NEGATIVE, self.reviews ) ) positive = list( filter( lambda x: x.sentiment == Sentiment.POSITIVE, self.reviews ) ) neutral = list( filter(lambda x: x.sentiment == Sentiment.NEUTRAL, self.reviews) ) positive_shrunk = positive[: len(negative)] self.reviews = negative + positive_shrunk + neutral random.shuffle(self.reviews) file_name = "/Users/aaronnguyen/Desktop/Books_small_10000.json" yelp_reviews = thread_list reviews = [] with open(file_name) as f: for line in f: review = json.loads(line) reviews.append( Review(review["reviewText"], review["overall"]) ) # creates a list with Review objects with text and score attributes training, testing = train_test_split( reviews, test_size=0.9, random_state=42 ) train_container = ReviewContainer(training) test_container = ReviewContainer(testing) train_container.evenly_distribute() training_x = train_container.get_text() training_y = train_container.get_sentiment() tes [truncated — 17088 more characters] ```