# Project export: Article Semantic Comprehension

This document was generated by HackStack to give an AI agent context about a hackathon project. Sections are labeled with their provenance; content marked as truncated was cut to keep this document small.

## Project metadata

- Hackathon: Cal Hacks 10.0
- Tagline: Articles are often misleading, this Accessibility technology determines bias in language from articles as-well as ensure summarisation
- Devpost: https://devpost.com/software/article-semantic-comprehension
- GitHub: https://github.com/NickCoding22/article_analysis
- Video: https://www.youtube.com/embed/48itR1NgpVU?enablejsapi=1&hl=en_US&rel=0&start=&version=3&wmode=transparent
- Result: winner ([MLH - Taipy] Best Use of Taipy)
- Team: 2 GitHub contributor(s) — NAngelici (11 commits), Matthew Robillard (7 commits)

## Devpost submission (written by the team)

### Inspiration

One of our project partners came up with this idea during the summer. We decided that the hackathon is a great opportunity for us to bring the project to life.

### What it does

What our article analysis does is that it analyzes and provides a summary of the article that the person entered. It summarizes the article by giving five key bullet points that are important to help with the user's understanding of the article. The program is also able to detect misinterpretation through semantic analysis.

### How we built it

We used Hume.ai to be able to analyze semantic in text. We also used Together.ai for our LLM. We coded the project in python and Taipy is used in the backend.

### Challenges we ran into

The challenges we ran into were being able to implement Hume.ai to be able to analyze the article that we entered and being able to create the LLM.

### Accomplishments we're proud of

We were proud of the fact that we were able to fully implement the Hume.ai to work with the articles, as this took us the longest time to figure out.

### What we learned

We learned that API are very helpful in creating our programs.

### What's next

The next step is to be able to type in a concept and multiple articles popup with analysis and summaries.

## README (from the GitHub repository)

# article_analysis

## Detected evidence (automated analysis)

Indexed codebase: 9 recognized source files, 12 KB.
- CSS (language) — detected in the code
- Python (language) — detected in the code

## Codebase structure (from repository index)

### Files (31 of 31)

```
.data/data_nodes/DATANODE_input_topic_6f990866-5839-4164-b3d5-cad0b8d0407c.json
.data/data_nodes/DATANODE_sentiment_857d561a-cc2a-4564-a199-c9ecb47ed115.json
.data/data_nodes/DATANODE_Summary_07a69f23-19c0-4acd-b13c-e6dfe67edd80.json
.data/data_nodes/DATANODE_url_2c00dced-d37f-4cf4-8d7f-9a6536db7051.json
.data/jobs/JOB_build_msg_e3074ba2-7570-4cf9-8442-028d93e14976.json
.data/jobs/JOB_build_msg_faf94d60-4b5f-4ccc-880c-66044830dc70.json
.data/jobs/JOB_build_msg2_736b62fc-229e-4b61-9009-6b05e15332d2.json
.data/jobs/JOB_build_msg2_955fe545-3b03-4955-ba71-7baa56f017a1.json
.data/jobs/JOB_build_msg3_5032d834-a965-4bb2-af71-6ff012a054c4.json
.data/jobs/JOB_build_msg3_d24cdf2e-ae51-4989-8258-5912ae015541.json
.data/pickles/DATANODE_input_topic_6f990866-5839-4164-b3d5-cad0b8d0407c.p
.data/pickles/DATANODE_Summary_07a69f23-19c0-4acd-b13c-e6dfe67edd80.p
.data/scenarios/SCENARIO_scenario_bae6b247-1405-4721-87d2-0986b8eaa518.json
.data/tasks/TASK_build_msg_1c9d7600-0e56-4212-b566-216cc71d4bbb.json
.data/tasks/TASK_build_msg2_0e0d9e64-0057-4862-a73b-9c16450291f3.json
.data/tasks/TASK_build_msg3_fd36e2e8-d62a-4ecd-96fb-efc8cdae206f.json
.data/version.json
.data/version/21d4cbca-fbc2-4793-9489-99f870d8c63c.json
.gitignore
article.txt
Backend/article.txt
Backend/Backend.py
Backend/GoogleSearcher.py
Backend/index.py
Backend/Main.py
Backend/Parser.py
Backend/SentimentAnalysis.py
Backend/style.css
Backend/TestSent.py
Frontend/test.txt
README.md
```

### Dependencies

No dependency index available.

### Recent commits (newest first)

- Yes
- finalll
- FINAL
- owapdkawa
- Changes
- Stuff
- Frontend Integration
- Merge branch 'main' of https://github.com/NickCoding22/article_analysis
- Fixed Main
- Frontend imports
- Fixed Main.py
- Added the main method
- Merge branch 'main' of https://github.com/NickCoding22/article_analysis
- Fixed Sentiment Analysis
- Darren
- yes
- yes
- Updated gitignore
- Added Frontend
- Updated Structure

## Key source files (fetched from GitHub, selected and truncated for size)

### Backend/Main.py

```python
import Backend
import Parser
import SentimentAnalysis

# Returns the sentiment
def main (url):
    sent = SentimentAnalysis.get_sentiment_from_article(Parser.parse_website(url))
    final_sent = 0
    negative_total = sent["Extremely Negative"] + sent["Very Negative"] + sent["Slightly Negative"] + sent["Negative"]
    positive_total = sent["Extremely Positive"] + sent["Very Positive"] + sent["Slightly Positive"] + sent["Positive"]
    if positive_total > sent["Neutral"] or negative_total > sent["Neutral"]:
        if positive_total > negative_total:
            final_sent = "Postive: Positive language was roughly " + str(positive_total) + " of the text."
        else:
            final_sent = "Negative: Negative language was roughly " + str(negative_total) + " of the text."
    else: 
        final_sent = "Neutral."
        
    summary = Backend.analyze_website_LLM(url)["main points"]
    return [url, final_sent, summary]

# Neutral
# print(main("https://www.sfchronicle.com/bayarea/article/sf-city-college-revive-18417567.php"))
# Neutral 
# print(main("https://www.defense.gov/News/News-Stories/Article/Article/3570190/dod-announces-up-to-150m-in-aid-for-ukraine/"))
# Positive 
# print(main("https://www.washingtonpost.com/opinions/2023/10/26/biden-constraining-israel-drawbacks/"))
# print(main("https://www.nytimes.com/2023/10/27/us/san-francisco-new-slogan.html"))
```

### Backend/index.py

```python
import taipy as tp
from taipy import Config, Core, Gui
from Main import main
import SentimentAnalysis
import Parser
import Backend
from taipy import Gui

myTheme  = {
        "palette": {
            "mode": 'dark',
            "primary": {
            "main": '#00a2a2',
            },
            "secondary": {
            "main": '#f50057',
            
            },
            "background": {
            "default": '#121212',
            "paper": '#040404',
            },
        },
        };


def main0(str):
    return str
    

def main2(str1):
    sent = SentimentAnalysis.get_sentiment_from_article(Parser.parse_website(str1))
    final_sent = ''
    negative_total = sent["Extremely Negative"] + sent["Very Negative"] + sent["Slightly Negative"] + sent["Negative"]
    positive_total = sent["Extremely Positive"] + sent["Very Positive"] + sent["Slightly Positive"] + sent["Positive"]
    neg = str(negative_total)[:5]
    pos = str(positive_total)[:5]
    if positive_total > sent["Neutral"] or negative_total > sent["Neutral"]:
        if positive_total > negative_total:
            final_sent = "This article takes a POSITIVE perspective with an index of: " + pos
        else:
            final_sent = "This article takes a NEGATIVE perspective with an index of: " + neg
    else: 
        final_sent = "Neutral."
    print(final_sent,"lokawdwa")
    return final_sent

def main3(str):
    summary = Backend.analyze_website_LLM(str)["main points"]
    print(summary,"summary")
    return summary
    """return "Together API: Down"""

################################################################
# Configure application
################################################################

def build_Summary(topic):
    return main0(topic)

def build_sentiment(topic):
    return main2(topic)

def build_url(topic):
    return main3(topic)

# Data node configurations to model the input topic.
input_topic_data_node_cfg = Config.configure_data_node(id="input_topic")
# Data node configurations to model the Summarys to display.
Summary_data_node_cfg = Config.configure_data_node(id="Summary")
Summary_data_node_cfg2 = Config.configure_data_node(id="sentiment")
Summary_data_node_cfg3 = Config.configure_data_node(id="url")

# Task configurations to model the build_Summary functions.
build_msg_task_cfg = Config.configure_task("build_msg", build_Summary, input_topic_data_node_cfg, Summary_data_node_cfg)
build_msg_task_cfg2 = Config.configure_task("build_msg2", build_sentiment, input_topic_data_node_cfg, Summary_data_node_cfg2)
build_msg_task_cfg3 = Config.configure_task("build_msg3", build_url, input_topic_data_node_cfg, Summary_data_node_cfg3)

# The scenario configuration represents the whole execution graph.
scenario_cfg = Config.configure_scenario("scenario", task_configs=[build_msg_task_cfg, build_msg_task_cfg2, build_msg_task_cfg3])

################################################################
# Design graphical interface
################################################################

input_topic = "https://www.sfchronicle.com/bayarea/article/sf-city-college-revive-18417567.php"
Summary = None
sentiment = None
url = None

def submit_scenario(state):
    state.scenario.input_topic.write(state.input_topic)
    state.scenario.submit()
    state.Summary = state.scenario.Summary.read()
    state.sentiment = state.scenario.sentiment.read()
    state.url = state.scenario.url.read()

page = """
Article Sentiment Comprehenseion (ASC)

Topic:
<|{input_topic}|input|>

<|submit|button|on_action=submit_scenario|>

URL:

<|{Summary}|text|>

Sentiment:
<|{sentiment}|text|>

Summary:
<|{url}|text|>
"""
newGui = Gui(page,css_file = "style.css")


if __name__ == "__main__":
    ################################################################
    # Instantiate and run Core service
    ################################################################
    Core().run()

    ################################################################
    # Manage scenarios and data nodes
    ################################################################
    scenario = tp.create_scenario(scenario_cfg)

    ################################################################
    # Instantiate and run Gui service
    ################################################################
   

    newGui.run(theme = myTheme)

```

### Backend/style.css

```css
:root {
  font-size: 2rem;
  padding: 20px;
  display: flex;
  justify-content: center;
  align-items: center;
  height: 100vh;
  margin: 10;
}
```

### Backend/GoogleSearcher.py

```python
import requests 
import pandas as pd 
from bs4 import BeautifulSoup 
try:
    from googlesearch import search
except ImportError: 
    print("No module named 'google' found")

def get_google_results(topic, amount):
    query = topic
    website_urls = []
    for j in search(query, tld="co.in", num=amount, stop=amount, pause=2):
        website_urls.append(j)
    return website_urls

# Test Main Function
'''test_results = get_google_results("College tuition", 5)
print(test_results)
#'''


```

### Backend/Parser.py

```python
import requests 
import pandas as pd 
from bs4 import BeautifulSoup 

# Gets the website's text from url
def get_html(url): 
    headers = {"User-Agent":"Mozilla/5.0"}
    r = requests.get(url, headers=headers)
    return r.text 

# Given a website url it returns the main text content
def parse_website(website_url):
    htmldata = get_html(website_url)
    soup = BeautifulSoup(htmldata, 'html.parser') 
    data = '' 
    full_article = ''
    for data in soup.find_all("p"): 
        if len(data.get_text()) > 10 and "\n" not in data.get_text():
            full_article += data.get_text()
    return(full_article)

# Testing Main Function
'''
parsing_test = parse_website("https://www.defense.gov/News/News-Stories/Article/Article/3570190/dod-announces-up-to-150m-in-aid-for-ukraine/")
#parsing_test = parse_website("https://www.sfchronicle.com/bayarea/article/sf-city-college-revive-18417567.php")
print(parsing_test)
#'''
```

### Backend/Backend.py

```python
#https://docs.taipy.io/en/latest/knowledge_base/demos/image_classif/
import together
import Parser as parser
import Keys
together.api_key = Keys.together_api_key

def analyze_website_LLM(website_url): 
    article_paragraph = ""
    five_key_points = ""

    #article_paragraph = parser.parse_website("https://www.sfchronicle.com/bayarea/article/sf-city-college-revive-18417567.php");
    #article_paragraph = parser.parse_website("https://www.defense.gov/News/News-Stories/Article/Article/3570190/dod-announces-up-to-150m-in-aid-for-ukraine/")
    article_paragraph = parser.parse_website(website_url)
    prompt_request = "Just summarize the following article in 5 sentence paragraph: [" + article_paragraph + "]"

    output = together.Complete.create(
        prompt = prompt_request,
        model = "togethercomputer/llama-2-7b-chat", 
        max_tokens = 256,
        temperature = 0.7,
        top_k = 50,
        top_p = 0.7,
        repetition_penalty = 1
    )


    five_key_points = output['output']['choices'][0]['text']
    # print(five_key_points)
    return {"article paragraph": article_paragraph, "main points": five_key_points}
```

### Backend/TestSent.py

```python
import Keys
from typing import Any, Dict, List

def get_sentiment_map(sentiment: List[Dict[str, Any]]) -> None:
        sentiment_map = {e["name"]: e["score"] for e in sentiment}
        return sentiment_map

import asyncio
import time
import traceback

from hume import HumeStreamClient
from hume.models.config import LanguageConfig


def get_sentiment_from_article (text):
    sentiments = [0, 0, 0, 0, 0, 0, 0, 0, 0, 0]

    text_example = text
    async def main():
        try:
            client = HumeStreamClient(Keys.hume_api_key)
            config = LanguageConfig(sentiment={})
            async with client.connect([config]) as socket:
                result = await socket.send_text(text_example)
                sent_map = get_sentiment_map(result["language"]["predictions"][0]["sentiment"])
                for i in range(1, 10):
                    sentiments[i] += sent_map[str(i)]
        except Exception:
            print(traceback.format_exc())
            
    asyncio.run(main())

    return_sentiments = {"Extremely Negative": sentiments[1], 
        "Very Negative": sentiments[2],
        "Slightly Negative": sentiments[3],
        "Negative": sentiments[4],
        "Neutral": sentiments[5],
        "Slightly Positive": sentiments[6],
        "Positive": sentiments[7],
        "Very Postive": sentiments[8], 
        "Extremely Positive": sentiments[9]
    }
    return return_sentiments

#print(get_sentiment_from_article("President Obama called Wednesday on Congress to extend a tax break for students included in last year's economic stimulus package, arguing that the policy provides more generous assistance. The American Opportunity Tax Credit program, which will cost $58 billion over a decade, is due to expire at the end of this year. In a statement to reporters in the White House Rose Garden, Obama said the tax breaks help make a college education more affordable for Americans. 'I am calling on Congress to make this tax credit permanent,"))
```

### Backend/SentimentAnalysis.py

```python
import Keys
from typing import Any, Dict, List

def get_sentiment_map(sentiment: List[Dict[str, Any]]) -> None:
        sentiment_map = {e["name"]: e["score"] for e in sentiment}
        return sentiment_map

import asyncio
import time
import traceback

from hume import HumeStreamClient
from hume.models.config import LanguageConfig


def get_sentiment_from_article (text):
    sentiments = [0, 0, 0, 0, 0, 0, 0, 0, 0, 0]

    text_example = text[:1500]
    async def main():
        try:
            client = HumeStreamClient(Keys.hume_api_key)
            config = LanguageConfig(sentiment={})
            async with client.connect([config]) as socket:
                result = await socket.send_text(text_example)
                sent_map = get_sentiment_map(result["language"]["predictions"][0]["sentiment"])
                for i in range(1, 10):
                    sentiments[i] += sent_map[str(i)]
        except Exception:
            print(traceback.format_exc())
            
    asyncio.run(main())

    return_sentiments = {"Extremely Negative": sentiments[1], 
        "Very Negative": sentiments[2],
        "Slightly Negative": sentiments[3],
        "Negative": sentiments[4],
        "Neutral": sentiments[5],
        "Slightly Positive": sentiments[6],
        "Positive": sentiments[7],
        "Very Positive": sentiments[8], 
        "Extremely Positive": sentiments[9]
    }
    print(return_sentiments)
    return return_sentiments

#print(get_sentiment_from_article("President Obama called Wednesday on Congress to extend a tax break for students included in last year's economic stimulus package, arguing that the policy provides more generous assistance. The American Opportunity Tax Credit program, which will cost $58 billion over a decade, is due to expire at the end of this year. In a statement to reporters in the White House Rose Garden, Obama said the tax breaks help make a college education more affordable for Americans. 'I am calling on Congress to make this tax credit permanent,"))
#print(get_sentiment_from_article("yay I'm so happy"))
```