# Project export: Auto-Score

This document was generated by HackStack to give an AI agent context about a hackathon project. Sections are labeled with their provenance; content marked as truncated was cut to keep this document small.

## Project metadata

- Hackathon: TreeHacks 2024
- Tagline: Enabling fast, accurate, and customizable free-response grading
- Devpost: https://devpost.com/software/auto-score
- GitHub: https://github.com/prabina-p/auto-score
- Demo: https://docs.google.com/presentation/d/1RBxTbz7ldAjXelZbIjB6Gruq3qAe4EMrT1bdpLyCLMM/edit?usp=sharing
- Video: https://www.youtube.com/embed/S7EiVUkjzv4?enablejsapi=1&hl=en_US&rel=0&start=&version=3&wmode=transparent
- Team: 5 GitHub contributor(s) — Prabina Pokharel (18 commits), tharvipop (16 commits), Yingqi Cao (11 commits), andrewpannn (6 commits), Atharva Kulkarni (2 commits)

## Devpost submission (written by the team)

### Inspiration

With a great amount of experience teaching and tutoring at the university level, we knew there was a lot to be desired in the grading experience for both students and instructors. We wished that there was a way students could receive feedback quickly and overworked instructors could focus their attention on more impactful things than grading. As a result, we decided to build a tool that would auto grade short answer response while allowing a high degree of accuracy and customization.

### What it does

Given a student response, our program analyzes the similarity to teacher provided answers. Furthermore, it uses GPT to provide quick feedback for students.

### How we built it

We used ChromaDB to handle our vector database operations and GPT4 to provide feedback for students. For our front-end, we used Reflex as our full-stack solution. pls Demo https://youtu.be/S7EiVUkjzv4

## README (from the GitHub repository)

# AutoScore

AutoScore aims to simplify the free response grading pipeline faced by instructors by utilizing ML methodologies to predict whether or not a student's answer is deemed correct.


## Inspiration

With a great amount of experience teaching and tutoring at the university level, we knew there was a lot to be desired in the grading experience for both students and instructors. We wished that there was a way students could receive feedback quickly and overworked instructors could focus their attention on more impactful things than grading. As a result, we decided to build a tool that would auto grade short answer response while allowing a high degree of accuracy and customization.


## What it does

Given a student response, our program analyzes the similarity to teacher provided answers. Furthermore, it uses GPT to provide quick feedback for students.


## How we built it

We used ChromaDB to handle our vector database operations and GPT4 to provide feedback for students. For our front-end, we used Reflex as our full-stack solution.


## Demo

* <a href="https://youtu.be/S7EiVUkjzv4" target="_blank">Click for demo video!</a>
* <a href="https://docs.google.com/presentation/d/1RBxTbz7ldAjXelZbIjB6Gruq3qAe4EMrT1bdpLyCLMM/edit?usp=sharing" target="_blank">Click for slides!</a>


## Built With

* chroma
* gpt-4
* python
* reflex
* scikit-learn



## Detected evidence (automated analysis)

Indexed codebase: 7 recognized source files, 16 KB.
- Python (language) — detected in the code

## Codebase structure (from repository index)

### Files (34 of 34)

```
.DS_Store
.gitignore
.vscode/settings.json
answer_augmentation/.DS_Store
answer_augmentation/grader_answer_gen.ipynb
auto_score/__init__.py
auto_score/auto_score.py
autoscore/.DS_Store
autoscore/autoscore.py
autoscore/data/data_queue_final.csv
autoscore/database/c8f09f14-b086-4aca-9048-fe05e443eb41/index_metadata.pickle
autoscore/database/chroma.sqlite3
autoscore/front_end/.DS_Store
autoscore/front_end/front_end.py
autoscore/requirements.txt
autoscore/rxconfig.py
chardet
front-end/.DS_Store
ML-Playground/.DS_Store
ML-Playground/combine_data.ipynb
ML-Playground/data/data_queue_final.csv
ML-Playground/data/form_responses.csv
ML-Playground/data/mohler_dataset_edited.csv
ML-Playground/data/submission_metadata.csv
ML-Playground/data/test3.csv
ML-Playground/database/c8f09f14-b086-4aca-9048-fe05e443eb41/index_metadata.pickle
ML-Playground/database/chroma.sqlite3
ML-Playground/notebook-pp.ipynb
ML-Playground/openai_api.ipynb
ML-Playground/rf_model.ipynb
ML-Playground/test_vectordb.ipynb
README.md
requirements.txt
rxconfig.py
```

### Dependencies

- autoscore/requirements.txt: reflex@==0.4.0
- requirements.txt: reflex@==0.4.0

### Recent commits (newest first)

- Update test_vectordb.ipynb
- Update openai_api.ipynb
- Update autoscore.py
- Update README.md
- Update README.md
- Update README.md
- Merge branch 'main' of github.com:prabina-p/auto-score
- Co-authored-by: andrewpannn <andrewpannn@users.noreply.github.com>
- updated gitignores
- add git ignore.
- Merge branch 'main' of https://github.com/prabina-p/auto-score
- interactivity
- added get size function header
- Merge branch 'main' of github.com:prabina-p/auto-score
- Co-authored-by: andrewpannn <andrewpannn@users.noreply.github.com>
- Merge branch 'main' of https://github.com/prabina-p/auto-score
- tuning
- Merge branch 'main' of github.com:prabina-p/auto-score
- integrated frontend and backend, gpt part
- rbf to autoscore mod

## Key source files (fetched from GitHub, selected and truncated for size)

### requirements.txt

```
reflex==0.4.0

```

### autoscore/requirements.txt

```
reflex==0.4.0

```

### rxconfig.py

```python
import reflex as rx

config = rx.Config(
    app_name="auto_score",
)
```

### autoscore/rxconfig.py

```python
import reflex as rx

config = rx.Config(
    app_name="front_end",
)
```

### auto_score/auto_score.py

```python
"""Welcome to Reflex! This file outlines the steps to create a basic app."""

from rxconfig import config

import reflex as rx

docs_url = "https://reflex.dev/docs/getting-started/introduction"
filename = f"{config.app_name}/{config.app_name}.py"


class State(rx.State):
    """The app state."""


def index() -> rx.Component:
    return rx.center(
        rx.theme_panel(),
        rx.vstack(
            rx.heading("Welcome to Reflex!", size="9"),
            rx.text("Get started by editing ", rx.code(filename)),
            rx.button(
                "Check out our docs!",
                on_click=lambda: rx.redirect(docs_url),
                size="4",
            ),
            align="center",
            spacing="7",
            font_size="2em",
        ),
        height="100vh",
    )


app = rx.App()
app.add_page(index)

```

### autoscore/autoscore.py

```python
import pandas as pd
import re
import statistics
from sklearn.ensemble import RandomForestClassifier
from sklearn.pipeline import Pipeline
from sklearn.model_selection import train_test_split
from sklearn.metrics import accuracy_score
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.metrics import f1_score, precision_score, recall_score
import nltk
from nltk.stem import WordNetLemmatizer
from nltk.corpus  import stopwords
nltk.download('punkt')
nltk.download('wordnet')
nltk.download('stopwords')
import chromadb
import pandas as pd
import numpy as np
import sklearn
from openai import OpenAI
from sklearn.feature_extraction.text import CountVectorizer
import matplotlib.pyplot as plt


### chat gpt variables::
API_KEY = "12345" # Fake API key, please use your real API key
client = OpenAI(api_key=API_KEY)

### chroma variables:
THRESHOLD = 0.65



### thresholding implementation:

def clean_data(text):
    '''
    Performs text cleaning by performing lemmatization and removing stop words.
    @param text: str sentences
    @return: cleaned text
    '''
    lowered = text.lower() 
    removed = re.sub(r'[^a-z]', ' ', lowered)  
    tokens = nltk.word_tokenize(removed)
    lemmatizer = WordNetLemmatizer()
    cleaned_tokens = [lemmatizer.lemmatize(word) for word in tokens if word not in stopwords.words('english')]
    cleaned_text = ' '.join(cleaned_tokens)
    return cleaned_text


pipeline = Pipeline([
    ('tfidf', TfidfVectorizer(preprocessor=clean_data, ngram_range=(1, 2))),
    ('classifier', RandomForestClassifier())
])

def data_load(path):
    '''
    Loads data from path
    @param path: path to the data
    @return: data
    '''
    data = pd.read_csv(path, index_col=False)
    data = data.drop(data.columns[0], axis=1)
    return data

def run_rf(data):
    '''
    Pre-processes data using TF-IDF and N-Gram and trains using Random Forest Classifier.
    @param data: data to train model on
    @return: f1-score of the model
    '''
    data['student_answer'] = data['student_answer'].apply(clean_data)

    class_0 = data[data['correct'] == 0]
    class_1 = data[data['correct'] == 1]
    class_0_undersampled = class_0.sample(n=len(class_1), random_state=42)
    new_df = pd.concat([class_0_undersampled, class_1])
    data = new_df.sample(frac=1, random_state=42).reset_index(drop=True)

    # pipeline = Pipeline([
    #     ('tfidf', TfidfVectorizer(preprocessor=clean_data, ngram_range=(1, 2))),
    #     ('classifier', RandomForestClassifier())
    # ])

    X_train, X_test, y_train, y_test = train_test_split(data['student_answer'], data['correct'], test_size=0.2, random_state=42)
    pipeline.fit(X_train, y_train)
    y_pred = pipeline.predict(X_test)
    f1 = f1_score(y_test, y_pred)
    
    return f1

def predict_answer(new_answer, model):
    '''
    Preprocesses the new answer and predicts whether it's correct or not using our model.
    @param new_answer: new input answer
    @param model: trained model
    @return: prediction of whether it's correct or not
    '''
    cleaned_answer = clean_data(new_answer)
    prediction = model.predict([cleaned_answer])[0]
    return prediction

if __name__ == "__main__":
    data = data_load("data/data_queue_final.csv")
    f1 = run_rf(data)
    print("F1 Score:", f1)

    while True:
        cur_answer = input("Enter your response (type 'quit' to exit): ")
        if cur_answer.lower() == 'quit':
            break
        prediction = predict_answer(cur_answer, pipeline)
        if prediction == 1:
            print("The answer is correct.")
        else:
            print("The answer is incorrect.")


### chroma db implementation:

# A function to get chromaDB database size
def get_dababase_size(name):
    
    return None

def create_client_collection():
    ''' 
    Create a client and collection using ChromaDB
    @return: the client created
    '''
    # return chromadb.Client('http://localhost:5000')
    try:
        client = chromadb.PersistentClient(path="./database")
        # collection = client.create_collection("BST_question", embedding_function=huggingface_ef)
        collection = client.create_collection(
            name="Queue_question",
            metadata={"hnsw:space": "cosine"} # l2 is the default
        )
    except:
        client = chromadb.PersistentClient(path="./database")
        collection = client.get_collection("Queue_question")
    return collection

def read_data(path):
    '''
    Read the data from the path
    @param path: the path to the data
    @return: the data'''
    data = pd.read_csv(path).iloc[:, 1:]
    return data

def add_to_collection(collection, df_corr, df_incorr):
    '''
    Add the student answers to the collection
    @param collection: the collection to add the student answers to
    @param df_corr: the dataframe of correct student answers
    @param df_incorr: the dataframe of incorrect student answers
    '''
    # add correct responses
    l = df_corr['student_answer'].tolist()
    ids = [f"id{i}"for i in range(len(l))]
    collection.add(
        documents=df_corr['student_answer'].tolist(),
        metadatas=[{"correct": "True"} for _ in range(len(l))],
        ids=ids,
    )
    
    # add incorrect responses
    l2 = df_incorr['student_answer'].tolist()

    ids = [f"id{i}"for i in range(len(l), len(l2)+len(l))]
    collection.add(
        documents=df_incorr['student_answer'].tolist(),
        metadatas=[{"correct": "False"} for _ in range(len(l2))],
        ids=ids,
    )

def query(collection, student_answer, k=3):
    '''
    Query the collection for the student answer
    @param collection: the collection to query
    @param student_answer: the student answer to query for
    @param k: the number of responses to return
    @return: the response from the query
    '''
    response = collection.query(
        query_texts=[student_answer],
        n_results=k
    )
    return response

def predict(response_json):
    '''
    Return the prediction based on 
[truncated — 3176 more characters]
```

### autoscore/front_end/front_end.py

```python
import reflex as rx
from autoscore import query
from autoscore import bot_compare, bot_suggests
from autoscore import create_client_collection, query, predict
from time import perf_counter

questionBank = ["Explain a binary search tree", "What is a queue in computer science?", "What is a capacitor"]
questionID = ["Queue", "Binary", "Capacitor"]
defaultSolutions = {'Queue': "A data structure that can store elements, which has the property that the last item added will be the last to be removed (or first-in-first-out)."}

class State(rx.State):
    """The app state."""
    #userInput that is passed into grading function
    userInput: str
    #input that is updated in text box
    new_item: str
    #index of which question
    question: str 
    # collection_size: int
    source: str  # source of judgement, display as result

    correctness: bool  # Correct -> True
    gpt_feedback: str  # Feedback from GPT

    execution_time: float
    collection_count: float

    def add_item(self):
        """Add a new item to the todo list."""
        self.userInput = self.new_item
    
    def display_result(self):
        # chroma api
        collection = create_client_collection()
        self.collection_count = collection.count()
        self.source = 'ChromaDB'
        start = perf_counter()
        response = query(collection, self.userInput)
        self.correctness = predict(response)[0]
        # print(f"chroma predicted: {self.correctness}")
        # print(f"chroma std-dev: {predict(response)[1]}")
        
        # chroma not amazing on correct preds, gpt api
        if self.correctness == True:
            self.source = "GPT-4"
            # gpt api
            self.correctness = bot_compare(question=self.question, solution=defaultSolutions['Queue'], student_answer=self.userInput)
            end = perf_counter()
            self.gpt_feedback = None if self.correctness else bot_suggests(question=self.question, solution="SomeSolution", student_answer=self.userInput)
        else:
            self.gpt_feedback = bot_suggests(question=self.question, solution="SomeSolution", student_answer=self.userInput)
            end = perf_counter()

        self.execution_time = end - start
        # add response to collection:
        collection.add(
            documents=[self.userInput],
            metadatas=[{"correct": f"{self.correctness}"}],
            ids=[f"id{collection.count()+1}"],
        )
        
        return

def index() -> rx.Component:
    """A view of the todo list.

    Returns:
        The index page of the todo app.
    """
    return rx.container(
        rx.hstack(
            rx.select(questionBank, default_value=questionBank[0], placeholder="Select a question",
                       radius="full", value=State.question, on_change=State.set_question, width="300px"),
            rx.button("Submit answer", on_click=State.display_result()),  
            rx.spacer(),
            # rx.box(State.collection_size, background_color="teal", width="20%"), ##add database stuff here

        ),
        rx.text_area(
            id="new_item",
            placeholder="Your answer here...",
            bg="white",
            value=State.new_item,
            on_change=State.set_new_item,
            on_blur=State.add_item(),
            box_shadow=f"{rx.color('gray', 3, alpha=True)} 0px 1px 4px",
            width="100",
            height="300px",
        ),
        rx.text_area(
            value=f"Correctness: {State.correctness},\n Suggestions: {State.gpt_feedback}, \n Time: {State.execution_time}, Size: {State.collection_count}. ",
            height="300px",
            bg="gray",
            placeholder="Feedback here..."
        ),
        size="2",
        margin_top="5em",
        margin_x="25vw",
        padding="1em",
        border_radius="0.5em",
    )

def gradedResponse() -> rx.Component:
    return rx.container(
        rx.hstack(
            rx.card(State.userInput, width="70"),
            rx.vstack(
                rx.card("Score:"),
                rx.card("100%")
            ),
        ),
        bg="white",
        height="300px",
        margin_x="25vw",
        margin_top="5em",
        padding="1em",
        border_radius="0.5em",
    )


# Create the app and add the state.
app = rx.App()

# Add the index page and set the title.
app.add_page(index, title="Enter your answer", route="/")
app.add_page(gradedResponse, title="Graded response", route="/grade")
```