# Project export: Frame

This document was generated by HackStack to give an AI agent context about a hackathon project. Sections are labeled with their provenance; content marked as truncated was cut to keep this document small.

## Project metadata

- Hackathon: Cal Hacks 11.0
- Tagline: A shared knowledge bases for research teams, that lets you see how your research, notes, and sources connect with what your peers have.
- Devpost: https://devpost.com/software/alfred-m1zx5s
- GitHub: https://github.com/abhichennupati/CalHacks24
- Video: https://www.youtube.com/embed/yNepFcg7opY?enablejsapi=1&hl=en_US&rel=0&start=&version=3&wmode=transparent
- Team: 1 GitHub contributor(s) — Abhiram Chennupati (12 commits)

## Devpost submission (written by the team)

### Inspiration

I was working at a big tech company this past summer on a research-adjacent team, and the method of storing information was mainly through google docs-style documents, where links would just be passed around via slack. This made it hard to find out what other people were working on, and moreso how that could relate to what I was working on.

### What it does

It computes similarity scores for your documents, and represents the documents you have, the documents your peers have, as well as all the sources used for each document as a weighted graph and allows you to find other people's work and research that's related to yours.

### How we built it

Used Distilbert for generating embeddings, Singlestore db with vector store, python + JS backend, with React for the frontend work. All of it is hosted on digitalocean droplets.

### Challenges we ran into

I ran into problems working with the text data from the JS editor and providing it to the bert-endpoint.

### Accomplishments we're proud of

I'm proud of the fact that it's a mostly working product, and that the similarity scoring actually works!

### What we learned

Learned a lot about devops this time and about Singlestore's DB, I was fairly familiar with the other frameworks and technologies used.

### What's next

I think it would be cool to keep building in more features, possibly ML-related.

## README (from the GitHub repository)

No README available.

## Detected evidence (automated analysis)

Indexed codebase: 3 recognized source files, 12 KB.
- Flask (technology) — detected in the code
- Python (language) — detected in the code
- React (technology) — claimed on Devpost, not found in the code

## Codebase structure (from repository index)

### Files (6 of 6)

```
.gitignore
app.py
data_helpers.py
gunicorn_config.py
Procfile
requirements.txt
```

### Dependencies

- requirements.txt: Flask@==2.3.3, gunicorn@>=20.0.4, singlestoredb@==0.10.0, urllib3@==2.0.4

### Recent commits (newest first)

- added gitignore
- added enw ones
- updated reqws
- changed to use api
- changed to using api
- updated app
- fixed gunicorn
- changed so that it won't start in debug mode
- fixed some thiings
- added gunicorm
- updated res
- added requirements.txt
- first commit

## Key source files (fetched from GitHub, selected and truncated for size)

### requirements.txt

```
singlestoredb==0.10.0  # Replace with the latest version if necessary
urllib3==2.0.4  # urllib.parse comes with urllib3 package
Flask==2.3.3  # Flask web framework
gunicorn>=20.0.4


```

### app.py

```python
from flask import Flask, request, jsonify
from data_helpers import get_similar_papers, get_source_papers, get_db_connection

app = Flask(__name__)

# API endpoint for retrieving similar papers
@app.route('/get_similar_papers', methods=['POST'])
def api_get_similar_papers():
    data = request.json
    paper_id = data.get('paper_id')

    if paper_id is None:
        return jsonify({"error": "Missing paper_id in request"}), 400

    try:
        conn = get_db_connection()
        similar_papers = get_similar_papers(conn, paper_id)
        conn.close()

        # Format the response
        papers = [
            {"id": paper["id"], "title": paper["title"], "text": paper["text"], "score": paper["score"]}
            for paper in similar_papers
        ]

        return jsonify({"papers": papers})

    except ValueError as e:
        return jsonify({"error": str(e)}), 404
    except Exception as e:
        return jsonify({"error": "An error occurred"}), 500

# API endpoint for retrieving papers linked to a source
@app.route('/get_source_paper_links', methods=['POST'])
def api_get_source_paper_links():
    data = request.json
    source_id = data.get('source_id')

    if source_id is None:
        return jsonify({"error": "Missing source_id in request"}), 400

    try:
        conn = get_db_connection()
        linked_papers = get_source_papers(conn, source_id)
        conn.close()

        # Format the response
        papers = [{"paper_id": paper["paper_id"]} for paper in linked_papers]

        return jsonify({"papers": papers})

    except Exception as e:
        return jsonify({"error": "An error occurred"}), 500

@app.route('/add_paper', methods=['POST'])
def api_add_paper():
    data = request.json
    text = data.get('text')
    title = data.get('title')
    owner = data.get('owner')

    if title is None or text is None or owner is None:
        return jsonify({'error': "missing field"}), 400

    try:
        conn = get_db_connection()
        paper_id = add_paper(conn, title, text, owner)
        conn.close()
        return jsonify({'id': paper_id})
    except Exception as e:
        conn.close()
        return jsonify({"error": "An error occurred"}), 500

@app.route('/update_paper', methods=['POST'])
def api_update_paper():
    data = request.json
    paper_id = data.get('id')
    text = data.get('text')
    title = data.get('title')

    if title is None or text is None or paper_id is None:
        return jsonify({'error': "missing field"})

    try:
        conn = get_db_connection()
        update_paper(conn, paper_id, title, text)
        conn.close()
    except Exception as e:
        conn.close()
        return jsonify({"error": "An error occurred"}), 500

@app.route('/add_source', methods=['POST'])
def api_add_source():
    data = request.json
    paper_id = data.get('paper_id')
    url = data.get('url')
    title = data.get('title')

    if title is None or url is None or paper_id is None:
        return jsonify({'error': "missing field"}), 400

    try:
        conn = get_db_connection()
        source_id = add_source(conn, title, url, paper_id)
        conn.close()
        return jsonify({'id': source_id})
    except Exception as e:
        conn.close()
        return jsonify({"error": "An error occurred"}), 500

# API endpoint for retrieving all papers owned by a specific user
@app.route('/get_user_papers', methods=['POST'])
def api_get_user_papers():
    data = request.json
    user = data.get('user')

    if user is None:
        return jsonify({'error': "missing field: user"}), 400

    try:
        conn = get_db_connection()
        user_papers = get_user_papers(conn, user)
        conn.close()

        return jsonify({"papers": user_papers})
    except Exception as e:
        conn.close()
        return jsonify({"error": "An error occurred"}), 500


# API endpoint for retrieving all sources linked to a specific paper
@app.route('/get_paper_sources', methods=['POST'])
def api_get_paper_sources():
    data = request.json
    paper_id = data.get('paper_id')

    if paper_id is None:
        return jsonify({'error': "missing field: paper_id"}), 400

    try:
        conn = get_db_connection()
        paper_sources = get_paper_sources(conn, paper_id)
        conn.close()

        return jsonify({"sources": paper_sources})
    except Exception as e:
        conn.close()
        return jsonify({"error": "An error occurred"}), 500


# API endpoint for retrieving all papers in the database
@app.route('/get_all_papers', methods=['POST'])
def api_get_all_papers():
    try:
        conn = get_db_connection()
        all_papers = get_all_papers(conn)
        conn.close()

        return jsonify({"papers": all_papers})
    except Exception as e:
        conn.close()
        return jsonify({"error": "An error occurred"}), 500


```

### gunicorn_config.py

```python
bind = "0.0.0.0:8080"
workers = 2

```

### data_helpers.py

```python
import singlestoredb as s2
from urllib.parse import urlparse, urlunparse
import json
import os
import requests

def get_db_connection():
    conn = s2.connect(os.getenv("SINGLESTORE_KEY"))
    return conn

def add_source(conn, title: str, url: str, paper_id: int):
    url = clean_url(url)
    cursor = conn.cursor()

    # Check if the URL already exists
    query = "SELECT id FROM Sources WHERE url = %s LIMIT 1"
    cursor.execute(query, (url,))
    result = cursor.fetchone()

    if result:
        source_id = result[0]
    else:
        embedding = get_embedding_from_source(url, title)

        # Convert the embedding (list of floats) to a comma-separated string
        embedding_json = json.dumps(embedding)

        insert_query = """
            INSERT INTO Sources (title, url, embedding
            VALUES (%s, %s, %s, %s)
        """
        cursor.execute(insert_query, (title, url, embedding_json))

        # Get the newly inserted source's ID
        source_id = cursor.lastrowid
        print(f"Inserted new source with ID: {source_id}")

    print(source_id)

    # Link the source to the paper
    insert_source_query = """
        INSERT INTO Papers_Sources (paper_id, source_id)
        VALUES(%s, %s)
    """
    cursor.execute(insert_source_query, (paper_id, source_id))
    conn.commit()
    cursor.close()

    return source_id




#adds paper to papers table with the given title and text, returns the id of the new entry
def add_paper(conn, title: str, text: str, owner: str):
	cursor = conn.cursor()
	embedding = get_embedding_from_paper(title, text)
	print("made embedding")
	query = """
		INSERT INTO Papers (title, text, embedding, owner)
		VALUES (%s, %s, %s, %s)
	"""

	embedding_json = json.dumps(embedding)
	cursor.execute(query, (title, text, embedding_json, owner))
	print("executed cursor")
	paper_id = cursor.lastrowid
	conn.commit()
	cursor.close()

	return paper_id

#updates paper paper_id with title, text, and embedding
def update_paper(conn, paper_id: int, title: str, text: str):
    cursor = conn.cursor()
    embedding = get_embedding_from_paper(title, text)

    query = """
        UPDATE Papers
        SET title = %s, text = %s, embedding = %s
        WHERE id = %s
    """

    embedding_json = json.dumps(embedding)
    cursor.execute(query, (title, text, embedding_json, paper_id))
    conn.commit()
    cursor.close()

#searches for 50 most similar papers, using paper embeddings, and returns the list sorted in descending order TODO need to get this to return scores as well
def get_similar_papers(conn, paper_id: int):
    cursor = conn.cursor()

    query = """
        SELECT embedding 
        FROM papers 
        WHERE id = %s
    """
    cursor.execute(query, (paper_id,))
    paper_embedding = cursor.fetchone()

    if not paper_embedding:
        cursor.close()
        raise ValueError(f"Paper with id {paper_id} not found")

    vector_search_query = """
        SET @query_vec = (?::VECTOR(768));

        SELECT id, title, text <*> @query_vec AS score
        FROM papers
        WHERE id != ?
        ORDER BY score DESC
        LIMIT 50;
    """

    # Execute the vector similarity search query
    cursor.execute(vector_search_query, (paper_embedding[0], paper_id))
    similar_papers = cursor.fetchall()

    # Close the cursor
    cursor.close()

    return similar_papers

#returns a list of paper_ids that are connected to a certain source source_id
def get_source_papers(conn, source_id: int):
	cursor = conn.cursor()

	query = """ 
		SELECT paper_id
		FROM Papers_Sources
		WHERE source_id = %s;
	"""

	cursor.execute(query, (source_id,))
	papers = cursor.fetchall()
	cursor.close()

	return papers

# Retrieve all papers from the Papers table
def get_all_papers(conn):
    cursor = conn.cursor()

    query = """
        SELECT id, title, text, owner
        FROM Papers
    """
    cursor.execute(query)
    papers = cursor.fetchall()
    cursor.close()

    # Format papers into a list of dictionaries
    all_papers = [
        {"id": paper[0], "title": paper[1], "text": paper[2], "owner": paper[3]}
        for paper in papers
    ]

    return all_papers


# Retrieve all sources linked to a specific paper
def get_paper_sources(conn, paper_id):
    cursor = conn.cursor()

    query = """
        SELECT s.id, s.title, s.url
        FROM Sources s
        JOIN Papers_Sources ps ON s.id = ps.source_id
        WHERE ps.paper_id = %s
    """
    cursor.execute(query, (paper_id,))
    sources = cursor.fetchall()
    cursor.close()

    # Format sources into a list of dictionaries
    paper_sources = [
        {"id": source[0], "title": source[1], "url": source[2]}
        for source in sources
    ]

    return paper_sources


# Retrieve all papers owned by a specific user
def get_user_papers(conn, user: str):
    cursor = conn.cursor()

    query = """
        SELECT id, title, text
        FROM Papers
        WHERE owner = %s
    """
    cursor.execute(query, (user,))
    user_papers = cursor.fetchall()
    cursor.close()

    # Format papers into a list of dictionaries
    papers = [
        {"id": paper[0], "title": paper[1], "text": paper[2]}
        for paper in user_papers
    ]

    return papers




# utility functions NEED TODO
def clean_url(url: str) -> str:
    # Parse the URL into its components
    parsed_url = urlparse(url)
    
    # Rebuild the URL without query parameters, fragments, or trailing slashes
    cleaned_url = urlunparse((
        parsed_url.scheme,       # http or https
        parsed_url.netloc.lower(),  # Domain name (lowercased for consistency)
        parsed_url.path.rstrip('/'),  # Path, with trailing slash removed
        '',                      # No params (not typically used in URLs)
        '',                      # Remove query string (parameters)
        ''                       # Remove fragment (the part after '#')
    ))
    
    return cleaned_url

def get_embedding_from_source(url: str, title: str) -> [float]:
	return get_embeddin
[truncated — 1250 more characters]
```