# Project export: Genetic Disorder Detection via cDNA Frequency Map

This document was generated by HackStack to give an AI agent context about a hackathon project. Sections are labeled with their provenance; content marked as truncated was cut to keep this document small.

## Project metadata

- Hackathon: TreeHacks 2025
- Tagline: Gain insights on fetal genetic disorders through minimally invasive means
- Devpost: https://devpost.com/software/genetic-disorder-detection-via-cdna-frequency-map
- GitHub: https://github.com/mhtruong1031/treehacks2025
- Team: 1 GitHub contributor(s) — mhtruong1031 (6 commits)

## Devpost submission (written by the team)

### Inspiration

As medical technology has progressed over the past several decades, genetic diseases have become increasingly prevalent, with many earlier treatable ailments presenting less of a concern. As such, the ability to gain early insights into genetic disorders at an early age, especially fetal, would allow medical practitioners to take preventative action and gain a better understanding of how to care for their patients.

### What it does

Current fetal genomic examinations involve dangerously invasive processes for extracting amniotic fluid, risking infection and physical hazards. Our approach facilitates minimally invasive cDNA extraction from the mother's blood, which is then put through our extensive model to reconstruct the baby's genome and reveal any concerning anomalies.

### How we built it

Datasets depicting disease-labelled cDNA fragments and genomes were datamined from FinaleDB and PGP Havard respectively. cDNA fragments and locales were compiled into a frequency distribution, normalized, and then used as a probability distribution function to sample indices and associated DNA fragments from the full human genomes. The new DNA fragments and associated disease labels were then tokenized and then put into our LLM as training data.

### Challenges we ran into

The availability of high-quality genomic data was the biggest hindrance in this project. Datasets were either protected for patient privacy and of those public, most were either very small or messy, requiring extensive data cleaning. The majority of the time hacking was spent searching for datasets, and then web scraping their sites due to poorly or nonexistent APIs.

### Accomplishments we're proud of

We're definitely proud of powering through the web scraping process, which ended up taking around 8 hours in total, considering the massive size of the sequences, alongside difficulties with data formatting.

## README (from the GitHub repository)

No README available.

## Detected evidence (automated analysis)

Indexed codebase: 8 recognized source files, 3827 KB.
- HTML (language) — detected in the code
- Python (language) — detected in the code

## Codebase structure (from repository index)

### Files (17 of 17)

```
.gitignore
Categorized_Disease_Data.csv
count.py
data_preparation.py
Diseased_Data_w_Bool.csv
filter.py
genetic_data_unfiltered.csv
identify_diseased.py
mutual_ids.txt
patient_survey_2015.csv
ProbabilityDistribution.py
resources/genetic_data_page.html
webscrape_patient_records.py
webscrape.py
wget-log
wget-log.1
wget-log.2
```

### Dependencies

No dependency index available.

### Recent commits (newest first)

- misc
- extended access to whole database
- webscraper for patient records implemented
- implemented webscraper
- balls
- grab hyperlinks
- filtered for mutual ids between databases

## Key source files (fetched from GitHub, selected and truncated for size)

### count.py

```python
import pandas as pd

pgp = pd.read_csv("Categorized_Disease_Data.csv")

print(len(pgp["Disease Category"].dropna().unique()))
```

### ProbabilityDistribution.py

```python
from random import seed, randint

index_range = int(5e8)
n = int(1e6)

sampled_indexes = []


seed(12345)

for i in range(n):
    sampled_indexes.append(randint(0, index_range))

for index in sampled_indexes:
    print(index)
```

### identify_diseased.py

```python
import pandas as pd
import matplotlib.pyplot as plt


pgp = pd.read_csv("Categorized_Disease_Data.csv")

is_diseased = []
for row, data in pgp.iterrows():
    is_diseased.append(type(data.iloc[6]) != float and 'no' not in data.iloc[6].lower())

pgp["Is Diseased"] = is_diseased

pgp.to_csv("Diseased_Data_w_Bool.csv")


```

### filter.py

```python
"""
hi chat

the main issue is that not every patient listed in the patient survey (phenotypic labels) is in the main genetic data for some reason

this just finds the participants that are present in both data tables
"""

import pandas as pd

gd = pd.read_csv("genetic_data_unfiltered.csv")
ps = pd.read_csv("patient_survey_2015.csv")

gd_participants = [id for id in gd["Participant"]]

mutual_ids = []
for id in ps["Participant"]:
    if id in gd_participants:
        mutual_ids.append(id)

with open("mutual_ids.txt", 'a') as f:
    for id in mutual_ids:
        f.write(id + "\n")  


```

### data_preparation.py

```python
from random import choice

# Fragments patient dna into num_fragments fragments
def fragment_patient_dna(probability_dist_function, full_patient_gene: str, num_fragments: int, fragment_len: int) -> list:
    sample_indices = []
    dna_fragments  = []
    
    for i in range(num_fragments):
        sample_indices.append(choice()) # TODO: add choice arugument
        pass

    for index in sample_indices:
        dna_fragments.append(full_patient_gene[index, index+fragment_len])

    # use those indices to take chunks of the full patient gene

    # get num_fragments number of chunks

    # return a list


    
    return dna_fragments
```

### webscrape.py

```python
import openpyxl, requests, os, zipfile

from bs4 import BeautifulSoup

# Configurables
UNFIILTERED_GENETIC_DATA_PATH = 'genetic_data_unfiltered.xlsx'
GENETIC_DATA_DIR_PATH         = 'resources/genetic_data'
MUTUAL_IDS_PATH               = 'mutual_ids.txt'

def main():
    # Intialize workbook
    with open(MUTUAL_IDS_PATH, 'r') as f:
        mutual_ids = f.read().splitlines()

    wb = openpyxl.load_workbook(UNFIILTERED_GENETIC_DATA_PATH)
    gd = wb["Genetic Data (unfiltered)"]

    # Scrape links
    hyperlinks = []
    for i in range(len(gd['A'])):
        if gd.cell(i+1, 1).value in mutual_ids:
            hyperlinks.append(gd.cell(i+1, 6).hyperlink.target)

    # Take links and save data into dir
    for url in hyperlinks:
        r    = requests.get(url=url)
        if r.status_code != 401:
            soup = BeautifulSoup(r.content)
            cmd  = soup.find(id = 'wget-example').contents

            os.system(cmd[0][2:] + f" -P {GENETIC_DATA_DIR_PATH}")

    # Clear zip files
    for file in os.listdir(GENETIC_DATA_DIR_PATH):
        if 'zip' in file:
            with zipfile.ZipFile(f'{GENETIC_DATA_DIR_PATH}/{file}', 'r') as zip_ref:
                zip_ref.extractall(GENETIC_DATA_DIR_PATH)
            os.system(f'rm {GENETIC_DATA_DIR_PATH}/{file}')

if __name__ == '__main__':
    main()

    
```

### webscrape_patient_records.py

```python
import requests, os

from bs4 import BeautifulSoup

# Configurables
DISPLAY_LENGTH     = 10
PROFILE_URL_PREFIX = 'https://my.pgp-hms.org/profile_public?hex='
GENOME_DIR_PATH    = 'resources/full_genome_data'


def main():
    headers = {'Content-Type': 'application/json', 'Accept': 'application/json'}

    valid_entries = []
    count         = 0

    for i in range(int(6190/DISPLAY_LENGTH)):
        print(f"{count} files validated ({int(count/6190)}%)")
        r = requests.get(
            url     = f'https://my.pgp-hms.org/users?sEcho=2&iColumns=8&sColumns=&iDisplayStart={i*10}&iDisplayLength={DISPLAY_LENGTH}&mDataProp_0=pgp_id&mDataProp_1=hex&mDataProp_2=enrolled&mDataProp_3=received_sample_materials&mDataProp_4=has_ccrs&mDataProp_5=has_relatives_enrolled&mDataProp_6=has_whole_genome_data&mDataProp_7=has_other_genetic_data&sSearch=&bRegex=false&sSearch_0=&bRegex_0=false&bSearchable_0=true&sSearch_1=&bRegex_1=false&bSearchable_1=true&sSearch_2=&bRegex_2=false&bSearchable_2=true&sSearch_3=&bRegex_3=false&bSearchable_3=true&sSearch_4=&bRegex_4=false&bSearchable_4=true&sSearch_5=&bRegex_5=false&bSearchable_5=true&sSearch_6=&bRegex_6=false&bSearchable_6=true&sSearch_7=&bRegex_7=false&bSearchable_7=true&iSortingCols=1&iSortCol_0=0&sSortDir_0=asc&bSortable_0=true&bSortable_1=true&bSortable_2=true&bSortable_3=true&bSortable_4=true&bSortable_5=true&bSortable_6=true&bSortable_7=true&_=1739673986470',
            headers = headers
            )
        
        data          = r.json().get("aaData")
        valid_entries += get_valid_entries(data)
        count += DISPLAY_LENGTH

    for ct, id in enumerate(valid_entries):
        download_data(id)
        print(f"{ct}/{len(valid_entries)} files downloaded ({int(ct/len(valid_entries))}%)")

# Data entries validated by having 1 or more complete genomes
def get_valid_entries(data: list) -> list:
    valid_entry_ids = []

    for entry in data:
        if entry['has_whole_genome_data'] != 0:
            valid_entry_ids.append(entry['hex'])

    return valid_entry_ids


def download_data(id: str) -> None:
    r = requests.get(PROFILE_URL_PREFIX + id)
    soup = BeautifulSoup(r.text)

    hyperlinks = soup.find_all('a', href=True)
    for link in hyperlinks:
        if 'genome_download' in link.get('href'):
            os.system(f'wget "{link.get("href")}" -P "{GENOME_DIR_PATH}"')


if __name__ == '__main__':
    main()
```