# Project export: Tuition Shock

This document was generated by HackStack to give an AI agent context about a hackathon project. Sections are labeled with their provenance; content marked as truncated was cut to keep this document small.

## Project metadata

- Hackathon: CruzHacks 2024
- Tagline: Tuition Shock, a U.S.A heatmap of the shocking price of higher education across the nation. Sort by 2-year or 4-year universities, in vs out of state, and state vs private education.
- Devpost: https://devpost.com/software/tuition-shock
- GitHub: https://github.com/keithirby/CruzHacks2024/
- Demo: http://tuitionshock.us/
- Result: winner (Justice Hacks)
- Team: 4 GitHub contributor(s) — pancakesontuesday (57 commits), keithirby (25 commits), Airi Kokuryo (6 commits), dallasmeyer (3 commits)

## Devpost submission (written by the team)

### Inspiration

The inspiration for our project was the shock all our members have had learning about the cost of higher education in the United States. Two members are distressed and shocked knowing they must pay their own tuition while many of their peer's education is subsidized by their parents. The other two members as foreign exchange students shocked and disturbed by the sticker price as neither of their tuitions break $10,000 a year. From this, our goal was to create a novel way to highlight the cost of higher education across the United States and its unequal access.

### What it does

Our hack, Tuition Shock, is a web service to view a heatmap for the tuition of community colleges and universities in the United States. Our goal was to show the disparity between a 2-year education and a 4-year education while also highlighting the privilege of access to higher education in more expensive cities and states. We also added the following filtering options to our heatmap to help highlight differences: Degree length 4-year 2-year Both Degree length 4-year 2-year Both School type Private Public Both School type Private Public Both Tuition type In-state Out-of-state Tuition type In-state Out-of-state Disclaimer The dataset we used was provided by the U.S. government college scorecard online tool (linked below). As is the nature of a heatmap, schools without a definite address or failure to report their cost of attendance to the U.S. government are NOT included on the map. If the Tuition Shock is not working for you make sure at the beginning of the link you are using http:// NOT https:// link: collegescorecard.ed.gov

### How we built it

We used Python to implement our server side code to fetch college data from our databse, then pass the data to the frontend side to visualize on the US map. MongoDB Atlas was used to manage over 3.5k of the college data with each entry containing 9 associated values. Python library Folium was used to generate the heatmap as a HTML, which we used as a backbone code to attach additional JavaScript code to implement additional features. With sidebar, now users can choose degree length, school type, and tuition type to see the desired data on the heatmap. To deploy our service, we used GoDaddy to reserve the tuitionshock.us URL. Next, to make the non-static website with provided URL, we made a masked URL redirect to render.com and render our interactive python heatmap so that anyone can access our service with any device.

### Challenges we ran into

Our first major challenge was figuring out how to use Open Data Maker HTTP API for getting data from the U.S.D.E College Scorecard database. The documentation in College Scorecard API Guide is not very clear, so we had to spend a few hours to get the results we wanted so we could grab the data for our database. Luckily, Dallas was a wizard with HTTP API URL generation and made all the breakthroughs there. Entering data in the database was also expedient with Eunice's expertise in MongoDB. The next challenge we faced was filtering useful data. With over 6,000 higher education institutions in our database, we had to remove entries that lacked their tuition costs and add location data to colleges missing it. This was fixed with Eunice's MongoDB experience and Keith's Python scripting skills. Afterward, we ran into bugs generating the heatmap and connecting it to an HTML website. Airi made quick work of this with her previous experience with Flask in conjunction with HTML. She also then went on to solve how we could load the rendered heatmap using the URL.

### Accomplishments we're proud of

We are very proud of our teamwork and optimism towards a challenge. Throughout the hackathon, we all had a task at any given moment and would help each other at any chance we got. We made time to take breaks together, discuss our progress, next goals, and celebrate small victories. One major accomplishment we all smiled with glee and felt awe at was our working buttons. Seeing the heatmap change so dramatically when showing only private vs state institutions was very fascinating as you see private institutions almost disappear as you look further into the center of the United States.

### What we learned

We learned about the difficulties of building a web app that is accessible to the populace. We found out how to make a public webpage running on PorkBun with GoDaddy. Also struggled getting more than just our sidebar buttons actually on the website and the actual rendered heatmap. All of us have had experience in HTML, python, and databases in the past, but not with all of these components interconnected to such a degree. In general, We gained an immense amount of experience in building a web app as a team playing to our advantage together. What's Next for Tuition Shock Next, we would like to add more data to our database as many colleges had to be removed simply due to a lack of time to find another method of collecting tuition costs. We would also like to improve the filtering options like showing colleges in a certain state or letting users set limits for a college's cost to be shown on the heatmap.

## README (from the GitHub repository)

# CruzHacks2024 **tuitionshock.us**
# CruzHacks2024 Justice Hack 1st Place Winnner

# Group Members

* [Airi Kokuryo](https://github.com/poe125)
* [Dallas Meyer](https://github.com/dallasmeyer)
* [Eunice Hong](https://github.com/eunbeen-hong)
* [Keith Irby](https://github.com/keithirby)
 

# Summary of **[tuitionshock.us](https://tuitionshock.onrender.com/)**

Our hack, **[tuitionshock.us](https://tuitionshock.onrender.com/)**, is a heatmap of the cost for community colleges and universities in the United States. 


Our goal was to show the disparity between a 2-year education and a 4-year education while also highlighting the privilege of access to higher education in more expensive cities and states. We added the following filtering options to our heatmap to help highlight cost differences:

1. Degree length

    - 4-year
    - 2-year
    - Both

2. School type

    - Private
    - Public
    - Both

3. Tuition type
    - In-state
    - Out-of-state


## Disclaimer

The dataset we used was provided by the U.S goverment college scorecard online tool (linked below). As is the nature of a heatmap schools without a definite address or failed to report their cost of attendance to the U.S goverment are **NOT** included in the list or as locations on the map.

link: [collegescorecard.ed.gov](collegescorecard.ed.gov)

# How was data collected?

The data collected was pulled from the _U.S Department of Education College Scorecard_ database. We collected this data using the _Open Data Maker HTTP API_, which allowed us to use HTTP queries for JSON formatted packets based on a variety of fields. Some examples of fields are a specific school, state schools only, or degree level rewarded.

[Open Data Maker HTTP API](https://github.com/RTICWDT/open-data-maker/blob/master/API.md)

[College Scorecard API Guide](https://collegescorecard.ed.gov/data/documentation/)

# Tech Stack

Server: Flask 🚀\
Database: MongoDB Atlas 📊\
Frontend: HTML + CSS + JavaScript 🌐





## Detected evidence (automated analysis)

Indexed codebase: 28 recognized source files, 404 KB.
- CSS (language) — detected in the code
- HTML (language) — detected in the code
- JavaScript (language) — detected in the code
- Python (language) — detected in the code
- Flask (technology) — claimed on Devpost, not found in the code
- MongoDB (technology) — claimed on Devpost, not found in the code

## Codebase structure (from repository index)

### Files (36 of 36)

```
.gitignore
app/app.py
app/delete_data.py
app/heatmap.py
app/insert_data.py
app/models/college_model.py
app/static/css/sidebar.css
app/static/js/get_data.js
app/templates/index.html
app/tst.py
data/coordinates.py
data/getting_data.py
data/gz_2010_us_outline_500k.json
data/null_location.txt
data/null_tuition_location.txt
data/null_tuition.txt
data/output.txt
data/processing_data.py
data/whole_data.txt
heatmap_gen/colored_heatmap_geojson.html
heatmap_gen/heatmap.html
heatmap_gen/heatmap.py
heatmap_gen/us_heatmap.html
heatmap/colored_heatmap_geojson.html
heatmap/heatmap.html
heatmap/heatmap.py
heatmap/us_heatmap.html
project detail.md
README.md
requirments.txt
scripts/coordinates.py
scripts/tuition.py
static/css/sidebar.css
static/js/get_data.js
templates/index.html
tuition.py
```

### Dependencies

No dependency index available.

### Recent commits (newest first)

- updated tuition shock URL
- Update README.md
- Merge pull request #18 from keithirby/heatmap-eunice
- details in sidebars
- square radio buttons and width for legend
- text for legend
- Merge pull request #17 from keithirby/heatmap-eunice
- increase clickable area for dots
- Update README.md
- ui for sidebar
- Merge pull request #16 from keithirby/heatmap_editing
- Merge branch 'main' into heatmap_editing
- Added a legend on the left hand side
- Moving scripts to their own file
- removing CNAME
- Fixing the scroll bars and adding color to sidebar
- adding a script for finding data for creating our tuition color gradient
- Merge pull request #15 from keithirby/heatmap_editing
- Added working heatmap based on tuition values
- Update README.md

## Key source files (fetched from GitHub, selected and truncated for size)

### project detail.md

```markdown
## Inspiration

The inspiration for our project was the shock all our members have had learning about the cost of higher education in the United States. Two members are distressed and shocked knowing they must pay their own tuition while many of their peer's education is subsidized by their parents. The other two members as foreign exchange students shocked and disturbed by the sticker price as neither of their tuitions break $10,000 a year. From this, our goal was to create a novel way to highlight the cost of higher education across the United States and its unequal access.

## What it does

Our hack, **[Tuition Shock](http://tuitionshock.us)**, is a web service to view a heatmap for the tuition of community colleges and universities in the United States.

Our goal was to show the disparity between a 2-year education and a 4-year education while also highlighting the privilege of access to higher education in more expensive cities and states. We also added the following filtering options to our heatmap to help highlight differences:

1. Degree length

    - 4-year
    - 2-year
    - Both

2. School type

    - Private
    - Public
    - Both

3. Tuition type
    - In-state
    - Out-of-state

### Disclaimer

The dataset we used was provided by the U.S. government college scorecard online tool (linked below). As is the nature of a heatmap, schools without a definite address or failure to report their cost of attendance to the U.S. government are **NOT** included on the map.

link: [collegescorecard.ed.gov](collegescorecard.ed.gov)

## How we built it

We used _Python_ to implement our server side code to fetch college data from our databse, then pass the data to the frontend side to visualize on the US map. _MongoDB_ Atlas was used to manage over 3.5k of the college data with each entry containing 9 associated values. Python library _Folium_ was used to generate the heatmap as a _HTML_, which we used as a backbone code to attach additional _JavaScript_ code to implement additional features. With sidebar, now users can choose degree length, school type, and tuition type to see the desired data on the heatmap.

To deploy our service, we used GoDaddy to reserve the **[tuitionshock.us](http://tuitionshock.us)** URL. Next, to make the non-static website with provided URL, we made a masked URL redirect to [render.com](https://render.com) and render our interactive python heatmap so that anyone can access our service with any device.

## Challenges we ran into

Our first major challenge was figuring out how to use [Open Data Maker HTTP API](https://github.com/RTICWDT/open-data-maker/blob/master/API.md) for getting data from the U.S.D.E College Scorecard database. The documentation in [College Scorecard API Guide](https://collegescorecard.ed.gov/data/documentation/) is not very clear, so we had to spend a few hours to get the results we wanted so we could grab the data for our database. Luckily, Dallas was a wizar
[truncated — 2322 more characters]
```

### app/app.py

```python
from flask import Flask, render_template
from pymongo import MongoClient
from dotenv import load_dotenv
import os
from models.college_model import College

app = Flask(__name__)

# Set up MongoDB URI from environment variable
load_dotenv()

mongo_uri = os.environ.get("DB_URL")

client = MongoClient(mongo_uri)
db = client['morphius']

db_colleges = db['colleges']

# Check if the connection to MongoDB is successful
if db_colleges != None:
    print("Connected to MongoDB")
else:
    print("Failed to connect to MongoDB")


@app.route('/')
def index():
    # Fetch data from MongoDB using the data model
    colleges = db_colleges.find()
    items = [College(id=college['id'],
                     school_name=college['school.name'],
                     student_size=college['student.size'],
                     state=college['school.state'],
                     tuition_in_state=college['latest.cost.tuition.in_state'],
                     tuition_out_of_state=college['latest.cost.tuition.out_of_state'],
                     latitude=college['location.lat'],
                     longitude=college['location.lon'],
                     school_type=college['school.ownership'],
                     degree_length=college['school.degrees_awarded.highest']
                     )
             for college in colleges]
    return render_template('index.html', colleges=[i.to_dict() for i in items])


if __name__ == "__main__":
    app.run(debug=True)
```

### app/tst.py

```python
# Open the HTML file, read its content, and make changes
with open("test.html", "r") as file:
    content = file.read()

# Make changes to the content as needed (e.g., using string manipulation or regex)
modified_content = content.replace("test", "new_text")

# Save the modified content back to the HTML file
with open("your_file.html", "w") as file:
    file.write(modified_content)
```

### app/delete_data.py

```python
# # Delete ALL data from the database

# import json
# from pymongo import MongoClient
# from dotenv import load_dotenv
# import os

# load_dotenv()
# mongo_uri = os.environ.get("DB_URL")

# # Connect to MongoDB
# client = MongoClient(mongo_uri)  # Replace with your MongoDB connection string
# db = client['morphius']  # Replace with your MongoDB database name
# collection = db['colleges']  # Replace with your desired collection name

# # remove all documents from the collection
# collection.delete_many({})

```

### app/insert_data.py

```python
import json
from pymongo import MongoClient
from dotenv import load_dotenv
import os

load_dotenv()
mongo_uri = os.environ.get("DB_URL")

# Connect to MongoDB
client = MongoClient(mongo_uri)  # Replace with your MongoDB connection string
db = client['morphius']  # Replace with your MongoDB database name
collection = db['colleges']  # Replace with your desired collection name

# Read data from the file
with open('data/null_location.txt', 'r') as file:
    colleges = json.load(file)
    for college in colleges:
        try:
            # Insert collection into MongoDB
            collection.insert_one(college)
        except json.JSONDecodeError as e:
            print(f"Error decoding JSON: {e}")

```

### data/processing_data.py

```python
import json

null_tuitin_location = []
null_tuition = []
null_location = []
whole_data = []


with open('output.txt', 'r') as file:
    for line in file:
        data = json.loads(line)
        colleges = data['results']

        for college in colleges:
            if college['latest.cost.tuition.in_state'] == None and college['latest.cost.tuition.out_of_state'] == None:
                if college['location.lat'] == None and college['location.lon'] == None:
                    null_tuitin_location.append(college)
                else:
                    null_tuition.append(college)
            elif college['location.lat'] == None or college['location.lon'] == None:
                null_location.append(college)
            else:
                whole_data.append(college)

with open('null_tuition_location.txt', 'w') as file:
    json.dump(null_tuitin_location, file, indent=2)

with open('null_tuition.txt', 'w') as file:
    json.dump(null_tuition, file, indent=2)

with open('null_location.txt', 'w') as file:
    json.dump(null_location, file, indent=2)

with open('whole_data.txt', 'w') as file:
    json.dump(whole_data, file, indent=2)

```

### heatmap_gen/heatmap.py

```python
import folium
from folium.plugins import HeatMap
from branca.colormap import linear
import json
# import getting_data
from flask import Flask, render_template
from pymongo import MongoClient
from dotenv import load_dotenv
import os
from app.models.college_model import College

app = Flask(__name__)

# Set up MongoDB URI from environment variable
load_dotenv()
mongo_uri = os.environ.get("DB_URL")

client = MongoClient(mongo_uri)
db = client['morphius']

db_colleges = db['colleges']
colleges = db_colleges.find()

items = [College(id=college['id'],
                 school_name=college['school.name'],
                 student_size=college['student.size'],
                 state=college['school.state'],
                 tuition_in_state=college['latest.cost.tuition.in_state'],
                 tuition_out_of_state=college['latest.cost.tuition.out_of_state'],
                 latitude=college['location.lat'],
                 longitude=college['location.lon'],
                 school_type=college['school.ownership'],
                 degree_length=college['school.degrees_awarded.highest']
                 )
         for college in colleges]


# Create a folium map centered around the US
us_map = folium.Map(location=[37.7749, -95.4194], zoom_start=4)


# Create HeatMap data using the latitude, longitude, and tuition data
# heat_data = list(zip(items.latitude, items.longitude,
#                  items.tuition_in_state))
heat_data = [(c.latitude, c.longitude, c.tuition_in_state) for c in items]

# Create a colormap based on tuition values
# Adjust scale based on your data
colormap = linear.YlOrRd_04.scale(
    0, max(max([c.tuition_in_state for c in items]), 1))

# Add HeatMap layer
HeatMap(heat_data, radius=50, blur=50, gradient={
        0.4: 'yellow', 0.65: 'orange', 1: 'red'}).add_to(us_map)

# Display the map
us_map.save('colored_heatmap_geojson.html')

```

### heatmap/heatmap.py

```python
import folium
from folium.plugins import HeatMap
from branca.colormap import linear
import json
# import getting_data
from flask import Flask, render_template
from pymongo import MongoClient
from dotenv import load_dotenv
import os
from app.models.college_model import College

app = Flask(__name__)

# Set up MongoDB URI from environment variable
load_dotenv()
mongo_uri = os.environ.get("DB_URL")

client = MongoClient(mongo_uri)
db = client['morphius']

db_colleges = db['colleges']
colleges = db_colleges.find()

items = [College(id=college['id'],
                 school_name=college['school.name'],
                 student_size=college['student.size'],
                 state=college['school.state'],
                 tuition_in_state=college['latest.cost.tuition.in_state'],
                 tuition_out_of_state=college['latest.cost.tuition.out_of_state'],
                 latitude=college['location.lat'],
                 longitude=college['location.lon'],
                 school_type=college['school.ownership'],
                 degree_length=college['school.degrees_awarded.highest']
                 )
         for college in colleges]


# Create a folium map centered around the US
us_map = folium.Map(location=[37.7749, -95.4194], zoom_start=4)


# Create HeatMap data using the latitude, longitude, and tuition data
# heat_data = list(zip(items.latitude, items.longitude,
#                  items.tuition_in_state))
heat_data = [(c.latitude, c.longitude, c.tuition_in_state) for c in items]

# Create a colormap based on tuition values
# Adjust scale based on your data
colormap = linear.YlOrRd_04.scale(
    0, max(max([c.tuition_in_state for c in items]), 1))

# Add HeatMap layer
HeatMap(heat_data, radius=50, blur=50, gradient={
        0.4: 'yellow', 0.65: 'orange', 1: 'red'}).add_to(us_map)

# Display the map
us_map.save('colored_heatmap_geojson.html')

```

### data/getting_data.py

```python
import requests
import json
from dotenv import load_dotenv
import os
import logging

logging.basicConfig(level=logging.DEBUG,
                    format='%(asctime)s - %(levelname)s - %(message)s')

college_name = input("Enter college name: ")
school_name = []
tuition_in_state = []
tuition_out_of_state = []
latitude = []
longitude = []

load_dotenv()
api_key = os.environ.get("GOV_API")

# Define the API endpoint URL
url = "https://api.data.gov/ed/collegescorecard/v1/schools.json"
params = {
    # 'school.degrees_awarded.predominant': '3',
    # 'school.ownership': '1',
    # 'school.name': f'{collesage_name}',
    '_fields': 'id,school.name,student.size,school.state,latest.cost.tuition.in_state,latest.cost.tuition.out_of_state,location.lat,location.lon,school.ownership,school.degrees_awarded.highest',
    'api_key': api_key,
    'page': 0,
    'per_page': 100,
}
# the GET request to the API
response = requests.get(url, params=params)

# Check if the request was successful (status code 200)
if response.status_code == 200:
    ##########################
    ### send this to the db ###
    ##########################

    data = response.json()
    # Make the data type to string
    parsed_data = json.dumps(data)
    parsed_data = json.loads(parsed_data)
    # how many universities will there be?
    x = len(parsed_data[:]['school.name'])
    for i in range(x):
        school_name.append(parsed_data['results'][i]['school.name'])
        tuition_in_state.append(
            parsed_data['results'][i]['latest.cost.tuition.in_state'])
        tuition_out_of_state.append(
            parsed_data['results'][i]['latest.cost.tuition.out_of_state'])
        latitude.append(parsed_data['results'][i]['location.lat'])
        longitude.append(parsed_data['results'][i]['location.lon'])

    # the GET request to the API
    response = requests.get(url, params=params)

    # Check if the request was successful (status code 200)
    if response.status_code == 200:

        data = response.json()
        # print(data)
        # Make the data type to string
        current_data = json.dumps(data)

    else:
        # Handle the case where the request was
        print("API request failed with status code:", response.status_code)
        print("API response content:", response.text)

    # Write out the text document
    file_path = r'./output.txt'
    with open(file_path, 'a') as f:
        f.write(current_data)
        f.write('\n')

```

### data/coordinates.py

```python
# Purpose: This python file is a script to find database entries that are lacking latitude and longitude
# but have tutiton costs to increase the number of possible entries to our heat map.


# Libraries for grabbing API key(s)
import os
from dotenv import load_dotenv
#Library for finding the lat and long
from geopy.geocoders import GoogleV3
# Library for reading the data folder
import json



def grab_API_KEY():
    # Load the .env file
    load_dotenv()

    # Grab the Google API key
    GOOGLE_KEY = os.getenv('GOOGLE_API')

    # Return the key
    return GOOGLE_KEY


# Grabbing the Google API Key and making it a global variable to reduce calls to grab_API_KEY
KEY = grab_API_KEY()


def find_coordinates(address):
    geolocator = GoogleV3(KEY)
    # Try getting the coordinates from a address
    try:
        # Finding location using geocode
        location = geolocator.geocode(address)
        if location:
            # If found grab the latitude and longitude
            latitude = round(location.latitude, 6)
            longitude = round(location.longitude, 6)
            return latitude, longitude
        else:
            # If not found return None
            return None
    except Exception as e:
        # If an error, return None as well
        print(f"Error: {e}")
        return None


def read_json():
    file_path = 'data/null_location.txt'

    try:
        with open(file_path, 'r') as file:
            data = json.load(file)

            for entry in data:
                school_name = entry.get('school.name')
                if school_name:
                    try:
                        latitude, longitude = find_coordinates(school_name)
                        entry['location.lat'] = latitude
                        entry['location.lon'] = longitude
                        print(f"Updated location for {school_name}")
                    except Exception as ex:
                        print(f"Error updating location for {school_name}: {ex}")

            # Write the updated data back to the file
            with open(file_path, 'w') as file:
                json.dump(data, file, indent=2)
            
            print("Location values updated and saved to the file.")
    except FileNotFoundError:
        print(f"File not found: {file_path}")
    except json.JSONDecodeError as e:
        print(f"Error decoding JSON: {e}")
    except Exception as e:
        print(f"Error: {e}")


read_json()

#Sources
# thank you Jake Witcher for the .env help! https://dev.to/jakewitcher/using-env-files-for-environment-variables-in-python-applications-55a1
```

[11 more indexed source files omitted to keep this export small. The full file list is in the Codebase structure section above.]