# Project export: MediaPilot

This document was generated by HackStack to give an AI agent context about a hackathon project. Sections are labeled with their provenance; content marked as truncated was cut to keep this document small.

## Project metadata

- Hackathon: Cal Hacks 10.0
- Tagline: Your personal assistant for NLP-powered sentiment analysis and prediction on your tweets.
- Devpost: https://devpost.com/software/mediapilot
- GitHub: https://github.com/AllenCaoo/MediaPilot
- Team: 3 GitHub contributor(s) — Allen Cao (33 commits), Annabella Chow (24 commits), Elva (9 commits)

## Devpost submission (written by the team)

### Inspiration

Twitter has become a primary channel for discovering what's happening in the world today. The way we express ourselves in those few characters can evoke different emotions that profoundly shape public discourse. Whether strategizing for your next presidential campaign or seeking to convey your next big idea, MediaPilot is your personal assistant to ensure your tweets are positively received by your intended audiences.

### What it does

After drafting your tweet in the UI, MediaPilot performs analysis of the your draft by combining sentiment analysis with topic extraction. The NLP model is trained on your personal past Twitter data and runs a personal ML model on your draft to predict the number of likes and engagement your tweet would receive.

### How we built it

React for frontend Python Flask for backend. NLTK library to conduct sentiment analysis and topic modeling. Scikit-learn for machine learning.

### Challenges we ran into

We ran into challenges working with MindsDB. Unfortunately, due to syntax and computer compatibility issues, we were unable to use this product. We also attempted to use OpenAI's API for topic modeling. In the end, we decided against using the API because it didn't group the topics properly.

### Accomplishments we're proud of

We're proud of the way we've been able to learn and experiment quickly with new technologies. For some of us, it was our first time using React and experimenting with new tools such as MindsDB and OpenAI's API. We built something that combined our interests in NLP and full-stack development, taking our software development skills to the next level. Overall, we enjoyed collaborating and bonding over creating this project.

### What we learned

We learned how much fun it can be to build something we've never tried before and just keep learning along the way.

### What's next

There are many exciting features that can be added, including recommendations of how to edit your tweet to make the words align better with your intended tone and mood. Furthermore, fine-tuning the model would improve the accuracy. Upgrading the database to cloud storage would allow for a greater training capacity of our models as well.

## README (from the GitHub repository)

# MediaPilot

## Tips:
- DO NOT PUSH `datasets/` TO GITHUB. IT'LL BE TOO MUCH TOO PUSH.
- If you are importing a new library, just let Allen know :).
- To fetch an api route, call the `api()` function.
    - `POST` methods require a second argument (request)
    - An example of `GET` method fetched from React:
        ```javascript
        fetch(api("/testget"))
            .then(
              response => response.json() // convert to json
            ).then(
              responseJSON => { 
                /* some logic */
              }
            )
        ``` 
    
    - An example of `POST` method fetched from React:
        ```javascript
        let request = 
            { 
                method: 'POST',
                headers: {
                    'Content-Type': 'application/json',
                },
                body: JSON.stringify(echo)
            }

        fetch(api("/echo"), request)
            .then(
              res => res.json()
            ).then(
              response => response.json() // convert to json
            ).then(
              responseJSON => { 
                /* some logic */
              }
            )
        ```
- Frontend will use Material UI (https://mui.com/material-ui). Find pre-made components to use. in frontend. They also provide playground for you to change up the component, and you can copy paste into codebase.
- Twitter API is way too expensive. Instead, we will down pre-existing Twitter datasets (https://github.com/shaypal5/awesome-twitter-data) into `datasets/` folder and train models from there.


## Description of folders
- **client/**: contains the react frontend
- **flask-server/**: contains the Python backend
- **nlp-model/**: ALL work pertaining to NLP predictive models
- **datasets/**: stores all our datasets for building NLP application

## Setup
0. `git clone https://github.com/AllenCaoo/MediaPilot.git` 
1. Ensure: `Python >= 3.9`
2. `pip install -r requirements.txt` (try `pip3` if `pip` doesn't work)
3. `cd client`
4. `npm install`
5. `cd ..`
6. `cd nlp-server`
7. `git clone https://github.com/mindsdb/mindsdb.git`
8. `python setup.py develop`
9. `python -m mindsdb`
10. Fin! 

### To run backend server:
- `cd flask-server`
- `python server.py` (try `python3` if `python` doesn't work)
- The Flask backend server will run on `http://localhost:5000`

### To run client
- `cd client`
- `npm start`
- The React frontend will run on `http://localhost:3000`


## Docker Setup:
- `docker run -p 47334:47334 -p 47335:47335 mindsdb/mindsdb`
- `http://127.0.0.1:47334/`
- Afterwards: 
  - powershell -> `wsl --shutdown` 

## Detected evidence (automated analysis)

Indexed codebase: 43 recognized source files, 82 KB.
- CSS (language) — detected in the code
- Flask (technology) — detected in the code
- HTML (language) — detected in the code
- JavaScript (language) — detected in the code
- Python (language) — detected in the code
- React (technology) — detected in the code

## Codebase structure (from repository index)

### Files (58 of 58)

```
client/.gitignore
client/package.json
client/public/index.html
client/public/manifest.json
client/public/robots.txt
client/README.md
client/src/api.js
client/src/App.css
client/src/App.js
client/src/components/Analysis.js
client/src/components/Button.js
client/src/components/Chart.js
client/src/components/Dashboard.js
client/src/components/Inputs.js
client/src/components/listItems.js
client/src/components/Popup.js
client/src/components/RecentRuns.js
client/src/components/Settings.js
client/src/components/Title.js
client/src/components/Upload.js
client/src/Contants/Constants.js
client/src/hooks/LocalStorage.js
client/src/index.js
client/src/pages/Home.js
client/src/pages/Index.js
client/src/pages/Results.js
client/src/reportWebVitals.js
client/src/setupProxy.js
client/src/setupTests.js
client/src/utils.js
datasets/cleaned_data.csv
datasets/followers_history.csv
datasets/README.md
datasets/realdonaldtrump_sent.csv
datasets/realdonaldtrump.csv
datasets/scores_trump.csv
datasets/tweets.csv
flask-server/server.py
nlp-model/model.py
nlp-model/README.md
openai/chatgpt.py
openai/constants.py
openai/follower_averages.py
openai/followers.py
openai/predictions.py
openai/test_predictions.py
openai/test.py
openai/topic_modeling.py
openai/topic_sentiments.py
openai/topics_and_likes.py
predictor.pkl
README.md
requirements.txt
sent_analysis/backend/pyvenv.cfg
sent_analysis/clean_data.py
sent_analysis/polarity_score_extraction.py
tm.pkl
vectorizer.pkl
```

### Dependencies

- client/package.json: @emotion/react@^11.11.1, @emotion/styled@^11.11.0, @mui/icons-material@^5.14.15, @mui/lab@^5.0.0-alpha.150, @mui/material@^5.14.15, @testing-library/jest-dom@^5.17.0, @testing-library/react@^13.4.0, @testing-library/user-event@^13.5.0, http-proxy-middleware@^2.0.6, react@^18.2.0, react-dom@^18.2.0, react-redux@^8.1.3, react-router-dom@^6.17.0, react-scripts@5.0.1, web-vitals@^2.1.4
- requirements.txt: blinker@==1.6.3, click@==8.1.7, colorama@==0.4.6, Flask@==3.0.0, itsdangerous@==2.1.2, Jinja2@==3.1.2, joblib@==1.3.2, MarkupSafe@==2.1.3, nltk@==3.8.1, numpy@==1.26.1, pandas@==2.1.2, python-dateutil@==2.8.2, pytz@==2023.3.post1, regex@==2023.10.3, six@==1.16.0, tqdm@==4.66.1, tzdata@==2023.3, Werkzeug@==3.0.1

### Recent commits (newest first)

- popup
- merge
- finally bruh
- popup message
- Merge branch 'master' of https://github.com/AllenCaoo/MediaPilot
- merge
- Merge branch 'master' of https://github.com/AllenCaoo/MediaPilot
- Final
- button
- merged
- Adding followers in csv
- Merge branch 'master' of https://github.com/AllenCaoo/MediaPilot
- stuff
- Scores_trump updated
- add followers csv
- Merge branch 'master' of https://github.com/AllenCaoo/MediaPilot
- begin incorporating followers
- A lot of infrastructure
- Working tester
- Merge branch 'master' of https://github.com/AllenCaoo/MediaPilot

## Key source files (fetched from GitHub, selected and truncated for size)

### requirements.txt

```
blinker==1.6.3
click==8.1.7
colorama==0.4.6
Flask==3.0.0
itsdangerous==2.1.2
Jinja2==3.1.2
joblib==1.3.2
MarkupSafe==2.1.3
nltk==3.8.1
numpy==1.26.1
pandas==2.1.2
python-dateutil==2.8.2
pytz==2023.3.post1
regex==2023.10.3
six==1.16.0
tqdm==4.66.1
tzdata==2023.3
Werkzeug==3.0.1

```

### client/package.json

```
{
  "name": "client",
  "version": "0.1.0",
  "private": true,
  "proxy": "http://localhost:5000/",
  "dependencies": {
    "@emotion/react": "^11.11.1",
    "@emotion/styled": "^11.11.0",
    "@mui/icons-material": "^5.14.15",
    "@mui/lab": "^5.0.0-alpha.150",
    "@mui/material": "^5.14.15",
    "@testing-library/jest-dom": "^5.17.0",
    "@testing-library/react": "^13.4.0",
    "@testing-library/user-event": "^13.5.0",
    "http-proxy-middleware": "^2.0.6",
    "react": "^18.2.0",
    "react-dom": "^18.2.0",
    "react-redux": "^8.1.3",
    "react-router-dom": "^6.17.0",
    "react-scripts": "5.0.1",
    "web-vitals": "^2.1.4"
  },
  "scripts": {
    "start": "react-scripts start --ignore client",
    "build": "react-scripts build",
    "test": "react-scripts test",
    "eject": "react-scripts eject"
  },
  "eslintConfig": {
    "extends": [
      "react-app",
      "react-app/jest"
    ]
  },
  "browserslist": {
    "production": [
      ">0.2%",
      "not dead",
      "not op_mini all"
    ],
    "development": [
      "last 1 chrome version",
      "last 1 firefox version",
      "last 1 safari version"
    ]
  }
}

```

### flask-server/server.py

```python
import sys
from flask import Flask, request, jsonify
from flask_cors import CORS
import pandas as pd
import pandas as pd
import numpy as np
import pickle
from nltk.sentiment.vader import SentimentIntensityAnalyzer
import csv

DB_PATH = 'db/saves.csv'

app = Flask(__name__)
CORS(app)

@app.route("/testget", methods=['GET'])
def test():
    return {"test": ["Test1", "Test2","Test3"]}


@app.route("/echo", methods=['POST'])
def echo():
    req = request.get_json()
    response = {
        "message": "here is your request repeated",
        "yourRequest": req
    }
    return response

@app.route("/predictLikes", methods=['POST'])
def predictLikes():
    """
    request: {
        body: {
            "content":...
        }
    }
    """
    req = request.get_json()
    tweet = req["content"]

    data_matrix = None
    with open("../vectorizer.pkl", "rb") as m:
        VECTORIZER = pickle.load(m)
        data_matrix = VECTORIZER.transform(pd.Series([tweet]))

    score = None
    with open("../tm.pkl", "rb") as m:
        TOPIC_MODEL = pickle.load(m)
        analyzer = SentimentIntensityAnalyzer()
        score = analyzer.polarity_scores(tweet)['compound']
        topOfTweet = TOPIC_MODEL.transform(data_matrix)
        topOfTweet = topOfTweet.argmax(axis=1)[0]

    topics = {}
    topics["scores"] = score
    for i in range(31):
        if i == topOfTweet:
            topics["Topic " + str(i)] = 1
        else:
            topics["Topic " + str(i)] = 0 

    res = None
    with open("../predictor.pkl", "rb") as m:
        PRED_MODEL = pickle.load(m)
        res = PRED_MODEL.predict(pd.DataFrame([topics]))[0] * 82633764

    return {"score": res, "sentimental": score}
    

@app.route("/getPastFavs", methods=['GET'])
def getPastFavs():
    """
    request: {
        body: {
            "likes": [likes1, likes2,...]
            "timestamp":
        }
    }
    """
    df = pd.read_csv('../datasets/realdonaldtrump.csv')
    response = {
        "favs": list(df["favorites"].tail(2)), 
        "timestaps": list(df["date"].tail(2))
    }
    

    return response

@app.route("/saveResults", methods=['POST'])
def saveResult():
    """
    request: {
        body: {
            
        }
    }
    """
    req = request.get_json()
        # Open the CSV file for appending
    with open(DB_PATH, 'a', newline='') as file:
        csv_writer = csv.writer(file)
        
        # Prepare the data as a list of values
        data_to_append = [req["timestamp"], req["date"], req["tweet"], req["score"]]

        # Append the data to the CSV file
        csv_writer.writerow(data_to_append)

    return {}


@app.route("/fetchRecentSaves", methods=['GET'])
def fetchRecentSaves():

    df = pd.read_csv(DB_PATH)
    df['timestamp'] = df['timestamp'].astype(int)
    df = df.sort_values(by='timestamp', ascending=False)
    df = df.head(15)
    listRows = df.values.tolist()

    print(listRows)

    return {
        "length": len(listRows),
        "rows": listRows
    }




if __name__ == "__main__":
    app.run(debug=True)
```

### client/src/index.js

```javascript
import React from 'react';
import ReactDOM from 'react-dom/client';
import App from './App';
import reportWebVitals from './reportWebVitals';

const root = ReactDOM.createRoot(document.getElementById('root'));
root.render(
  <React.StrictMode>
    <App />
  </React.StrictMode>
);

// If you want to start measuring performance in your app, pass a function
// to log results (for example: reportWebVitals(console.log))
// or send to an analytics endpoint. Learn more: https://bit.ly/CRA-vitals
reportWebVitals();

```

### client/src/App.js

```javascript
import React, { useState, useEffect, useRef, Switch } from 'react'


import { BASE_URL } from './Contants/Constants';
import { api } from './api';
import Home from './pages/Home';
import RecentRuns from './components/RecentRuns';
import Dashboard from './components/Dashboard';


import Results from "./pages/Results";

function App() {

  const [enterRecentRuns, setEnterRecentRuns] = useState([])
  const [data, setData] = useState([{"hello": ["hello"]}])
  const [echo, setEcho] = useState({"Echo": "echoooo"})
  const [page, setPage] = useState("dashboard")
  const [results, setResults] = useState(-1)
  const [sentimentalScore, setSentimentalScore] = useState(-1)
  const [enteredText, setEnteredText] = useState(''); // Define enteredText state


  const dashboard = <Dashboard setPage={setPage} 
                                enteredText={enteredText}
                                setEnteredText={setEnteredText}
                                results={results}
                                setResults={setResults}
                                sentimentalScore={sentimentalScore}
                                setSentimentalScore={setSentimentalScore}
                                enterRecentRuns={enterRecentRuns}
                                setEnterRecentRuns={setEnterRecentRuns}/>
  const recentRuns = <RecentRuns setPage={setPage}
                                enteredText={enteredText}
                                setEnteredText={setEnteredText}
                                results={results}
                                setResults={setResults}
                                sentimentalScore={sentimentalScore}
                                setSentimentalScore={setSentimentalScore}
                                enterRecentRuns={enterRecentRuns}
                                setEnterRecentRuns={setEnterRecentRuns}
                                />

  const choosePage = (pg) => {
    if (pg == "dashboard") {
      return dashboard;
    } else if (pg == "recent_runs") {
      return recentRuns;
    }
  }


  useEffect(() => {
    fetch(api("/testget"))
    .then(
      res => res.json()
    ).then(
      d => {
        setData(d);
      }
    )
  }, [])

  useEffect(() => {
    fetch(api("/fetchRecentSaves"))
    .then(
      res => res.json()
    ).then(
      d => {
        console.log(d)
      }
    )
  }, [])


  useEffect(() => {
    // Fetch data from /testget endpoint
    fetch(api("/testget"))
      .then((res) => {
        if (!res.ok) {
          throw new Error('Network response was not ok');
        }
        return res.json();
      })
      .then((data) => {
        setData(data);
      })
      .catch((error) => {
        console.error('Fetch error:', error);
      });
  }, []);
  
  useEffect(() => {
    // Fetch data from /echo endpoint
    fetch(api("/echo"), {
      method: 'POST',
      headers: {
        'Content-Type': 'application/json',
      },
      body: JSON.stringify(echo),
    })
      .then((res) => {
        if (!res.ok) {
          throw new Error('Network response was not ok');
        }
        return res.json();
      })
      .then((data) => {
        setEcho(data);
      })
      .catch((error) => {
        console.error('Fetch error:', error);
      });
  }, []); // Include 'echo' in the dependency array if it's needed to trigger this effect
  

  return (

    
    <div>
      
        {/* {JSON.stringify(data)}
        {JSON.stringify(echo)} */}
        {choosePage(page)}
        {/* <Home/>
        <Results /> */}
        
    </div>
  )
}

export default App
```

### client/src/pages/Index.js

```javascript
import React from "react";
   import ReactDOM from "react-dom";
   import { BrowserRouter, Route, Switch } from "react-router-dom";

   import Home from "./Home";
   import Results from "./Results";

    const rootElement = document.getElementById("root");
    ReactDOM.render(
      <BrowserRouter>
       <Switch>
        <Route exact path="/" component={Home} />
        <Route path="/results" component={Results} />
      </Switch>
      </BrowserRouter>,
      rootElement
    );
```

### openai/test.py

```python
import predictions
import topic_modeling


```

### openai/constants.py

```python
topics = [phrase.lower().strip() for phrase in [
    "Political Statements and Policies",
    "Elections and Campaigns",
    "International Relations",
    "Impeachment and Legal Matters",
    "COVID-19 Pandemic",
    "Twitter Usage",
    "Rallies and Events",
    "Media Coverage",
    "Fake News",
    "Protests and Demonstrations",
    "Social Issues",
    "Economy"
]]
```

### openai/topic_sentiments.py

```python
import pandas as pd

from constants import topics
import chatgpt
from nltk.sentiment.vader import SentimentIntensityAnalyzer

data = pd.read_csv('datasets/cleaned_data.csv')

#topic sentiment scores
analyzer = SentimentIntensityAnalyzer()

topic_sents = {}
for topic in topics:
    score = analyzer.polarity_scores(topic)
    topic_sents[topic] = score

sents = {
    'topic': topic_sents.keys(),
    'score': topic_sents.values()
}

csv_path = 'datasets/topic_sentiment_scores.csv'

df = pd.DataFrame(sents)

df.to_csv(csv_path, index = False)



```

### openai/follower_averages.py

```python
import pandas as pd
import csv

followers = pd.read_csv('datasets/followers_history.csv')

counts = []

for follow in followers['Followers_Count']:
    counts.append(int(follow))

followers['Followers_Count'] = counts

yearly_avg = {}
for i in range(2014, 2021):
    year = str(i)
    data = followers[followers['Date'].str.contains(year)]
    avg = sum(data['Followers_Count'])/len(data)
    yearly_avg[i] = avg

avgs = {
    'year': yearly_avg.keys(),
    'average': yearly_avg.values()
}

csv_path = 'datasets/yearly_follower_avg.csv'

df = pd.DataFrame(avgs)

df.to_csv(csv_path, index = False)
```

[30 more indexed source files omitted to keep this export small. The full file list is in the Codebase structure section above.]