# Project export: Omnis
This document was generated by HackStack to give an AI agent context about a hackathon project. Sections are labeled with their provenance; content marked as truncated was cut to keep this document small.
## Project metadata
- Hackathon: TreeHacks 2025
- Tagline: AI-powered tutor and knowledge hub. Upload lecture materials, ask questions, and get answers backed by your content. Centralize learning, build knowledge, and stay organized—all in one place!
- Devpost: https://devpost.com/software/omnis-gb4zfj
- GitHub: https://github.com/adityatadimeti/omnis
- Video: https://www.youtube.com/embed/7FUOQMxGrOU?enablejsapi=1&hl=en_US&rel=0&start=&version=3&wmode=transparent
- Team: 3 GitHub contributor(s) — Aditya (23 commits), sathviknallamalli (19 commits), Sanjay Nagaraj (1 commits)
## Devpost submission (written by the team)
### Inspiration
As students ourselves who have taken a breadth of classes, we realize that each class uses its own organization system and platforms. Some classes upload materials in three different places with lecture videos and supplemental videos in another place, whereas other classes keep everything in one large folder. Being able to track how each class operates is difficult, and we dont want that to impede a student's learning process. We wanted to create a smarter way to study—one that goes beyond static notes and videos. Keeping track of lecture materials and truly understanding concepts can be overwhelming, so we built Omnis to act as an interactive AI tutor, making learning more intuitive, organized, and personalized.
### What it does
Our tool allows you to keep everything in one place and also uses LLMs to parse this information so you can actually interact with it! Omnis allows students to upload lecture slides, notes, videos, and homework into a single system where they can chat with an AI assistant trained on their own materials. Users can ask questions, get clarifications, and explore concepts with references directly from their coursework, creating a personalized and centralized learning experience.
### How we built it
We used InterSystems' IRIS Vector Search to embed the documents and information so that users can search and interact with it. We used ReactJS for our frontend, we run our own server with Python Flask, and we use Firebase for Cloud Storage and Authentication. We use OpenAI's language models for reasoning and generation. We built a pipeline to extract information from PDFs, videos, and notes into text embeddings, thus allowing the AI to generate responses grounded in user-provided content. A chat interface lets students interact seamlessly with the AI for real-time assistance. We also provide references back to the original documents that the user uploaded, specific to the timestamp and sentence so that the user can see the contextual grounding.
### Challenges we ran into
Chunking and Context limits. Ensuring AI-generated responses remain grounded in provided materials and don’t hallucinate.
### Accomplishments we're proud of
Increasing generality of our tool with different file types and platform integrations. Implementing a robust retrieval system that provides accurate, source-backed answers. Synthesizing all sources into a knowledge base and a clean and easy-to-use AI!
### What we learned
Utilizing IRIS vector search Learned about new lecture content through Omnis!
### What's next
Integrating more streams for class content, then deploying for students to use!
## README (from the GitHub repository)
# OMNIS
AI-powered tutor and knowledge hub. Upload lecture materials, ask questions, and get answers backed by your content. Centralize learning, build knowledge, and stay organized—all in one place!
DevPost: https://devpost.com/software/omnis-gb4zfj
## Detected evidence (automated analysis)
Indexed codebase: 26 recognized source files, 90 KB.
- CSS (language) — detected in the code
- Firebase (technology) — detected in the code
- HTML (language) — detected in the code
- JavaScript (language) — detected in the code
- OpenAI (technology) — detected in the code
- Python (language) — detected in the code
- React (technology) — detected in the code
## Codebase structure (from repository index)
### Files (40 of 40)
```
.DS_Store
.gitignore
assets/test_transcript.txt
data/scotch_review.csv
environment.yaml
frontend/.gitignore
frontend/package.json
frontend/public/index.html
frontend/public/manifest.json
frontend/public/robots.txt
frontend/src/api.js
frontend/src/App.css
frontend/src/App.js
frontend/src/App.test.js
frontend/src/AuthContext.js
frontend/src/AuthProvider.js
frontend/src/ChatInterface.js
frontend/src/Dashboard.js
frontend/src/firebaseConfig.js
frontend/src/FirebaseVideoPlayer.js
frontend/src/index.css
frontend/src/index.js
frontend/src/Login.js
frontend/src/ProjectsDashboard.js
frontend/src/ProtectedRoute.js
frontend/src/reportWebVitals.js
frontend/src/setupTests.js
frontend/src/test.js
frontend/src/VideoPage.js
gw.txt
README.md
scripts/app.py
scripts/backend_database.py
scripts/identification_generation.py
scripts/install/intersystems_irispython-5.0.1-8026-cp38.cp39.cp310.cp311.cp312-cp38.cp39.cp310.cp311.cp312-macosx_10_9_universal2.whl
scripts/requirements_iris.txt
scripts/requirements.txt
scripts/test_identification_generation.ipynb
scripts/test.txt
scripts/video_to_transcript.py
```
### Dependencies
- frontend/package.json: axios@^1.7.9, cra-template@1.2.0, firebase@^11.2.0, lodash@^4.17.21, lucide-react@^0.475.0, react@^18.3.1, react-dom@^18.3.1, react-router-dom@^7.1.5, react-scripts@^5.0.1, uuid@^11.0.5, web-vitals@^4.2.4
- scripts/requirements.txt: noisereduce@==3.0.3, openai@==1.61.1, pocketsphinx@==5.0.4, pydub@==0.25.1, SpeechRecognition@==3.14.0, tiktoken@==0.8.0, tk@==0.1.0, tqdm@==4.67.1
### Recent commits (newest first)
- Merge pull request #4 from adityatadimeti/sanjay_edits
- final changes
- Merge branch 'sanjay_edits' of https://github.com/adityatadimeti/omnis into sanjay_edits
- backend edits
- Merge branch 'sanjay_edits' of github.com:adityatadimeti/omnis into sanjay_edits
- update app
- update chat interface
- reworking backend / frontend
- update chat interface
- Merge branch 'sanjay_edits' of github.com:adityatadimeti/omnis into sanjay_edits
- add timestamp search
- Update README.md
- Update README.md
- Update README.md
- chat history updated
- chat history
- integrating gen changes
- fixed og url
- Merge pull request #3 from adityatadimeti/id_generation
- edits
## Key source files (fetched from GitHub, selected and truncated for size)
### scripts/requirements.txt
```
openai==1.61.1
pydub==0.25.1
SpeechRecognition==3.14.0
noisereduce==3.0.3
pocketsphinx==5.0.4
tqdm==4.67.1
tiktoken==0.8.0
tk==0.1.0
```
### frontend/package.json
```
{
"name": "omnis",
"version": "0.1.0",
"private": true,
"dependencies": {
"axios": "^1.7.9",
"cra-template": "1.2.0",
"firebase": "^11.2.0",
"lodash": "^4.17.21",
"lucide-react": "^0.475.0",
"react": "^18.3.1",
"react-dom": "^18.3.1",
"react-router-dom": "^7.1.5",
"react-scripts": "^5.0.1",
"uuid": "^11.0.5"
},
"scripts": {
"start": "react-scripts start",
"build": "react-scripts build",
"test": "react-scripts test",
"eject": "react-scripts eject"
},
"eslintConfig": {
"extends": [
"react-app",
"react-app/jest"
]
},
"browserslist": {
"production": [
">0.2%",
"not dead",
"not op_mini all"
],
"development": [
"last 1 chrome version",
"last 1 firefox version",
"last 1 safari version"
]
},
"devDependencies": {
"web-vitals": "^4.2.4"
}
}
```
### scripts/app.py
```python
# app.py
from flask import Flask, request, jsonify
from flask_cors import CORS
from backend_database import add_embeddings, search_files
import os
from video_to_transcript import AudioHandler
import pandas as pd
import numpy as np
import os
from openai import OpenAI
from dotenv import load_dotenv
from PyPDF2 import PdfReader
from typing import List
from tqdm import tqdm
import re
from difflib import SequenceMatcher
from identification_generation import setup_openai_key, parse_text_from_timestamps, parse_text_from_timestamps_original, parse_timestamps, chunk_str, get_timestamp_from_answer
from backend_database import setup_database_connection
app = Flask(__name__)
CORS(app)
@app.route('/add_embedding', methods=['POST'])
def create_embedding():
"""Add embedding for a chunk of text."""
try:
data = request.json
# Check all required fields: chunk_url, chunk_text, original_file_url, user_name
required_fields = ["chunk_url", "chunk_text", "original_file_url", "user_name", "file_type", "file_name"]
missing_fields = [f for f in required_fields if f not in data or not data[f]]
if missing_fields:
return jsonify({
"status": "error",
"message": f"Missing required fields: {', '.join(missing_fields)}"
}), 400
# Call add_embeddings with user_name included
result = add_embeddings(
chunk_url=data['chunk_url'],
chunk_text=data['chunk_text'],
original_file_url=data['original_file_url'],
user_name=data['user_name'],
file_type=data['file_type'],
file_name=data['file_name']
)
# Return 200 status regardless of whether chunk was new or existing
return jsonify(result), 200
except Exception as e:
return jsonify({
"status": "error",
"message": str(e)
}), 500
@app.route('/process_video', methods=['POST'])
def process_video():
video_file = request.files['video']
file_path = 'temp/video.mp4'
video_file.save(file_path)
ah = AudioHandler(file_path)
transcript_paths = ah.process_audio()
print("transcript(s) saved to: ", transcript_paths)
#convert the posix paths to file paths
transcript_paths = [str(path) for path in transcript_paths]
transcript_content = []
for tp in transcript_paths:
with open(tp, 'r') as file:
content = file.read()
transcript_content.append(content)
return jsonify({
"status": "success",
"message": "Video processed successfully",
"transcript_paths": transcript_paths,
"transcript_content": transcript_content
})
@app.route('/search', methods=['POST'])
def search():
"""Search for similar content in the user's chunk database."""
try:
data = request.json
# Ensure 'query' is present
if not data or 'query' not in data:
return jsonify({
"status": "error",
"message": "Missing required field: query"
}), 400
# Get user name (fallback if not provided)
user_name = data.get('user_name', 'UnknownUser')
num_results = data.get('num_results', 3)
# Pass 'user_name' to the search_files function
result = search_files(
search_phrase=data['query'],
user_name=user_name,
num_results=num_results
)
return jsonify(result)
except Exception as e:
return jsonify({
"status": "error",
"message": str(e)
}), 500
@app.route('/run_identification', methods=['POST'])
def run_identification():
"""
top_k_queries: text over either video transcript or documents (notes, slides, etc.)
top_k_types: either video, text, or image
"""
data = request.json
top_k_queries = data['top_k_queries']
top_k_types = data['top_k_types']
question = data['question']
client = setup_openai_key()
print(data)
top_k_ids = []
for idx, (chunk_text, chunk_type) in tqdm(enumerate(zip(top_k_queries, top_k_types))):
if chunk_type == "video":
# Remove whitespace after joining
chunk_text = "\n".join(parse_text_from_timestamps_original(chunk_text)).strip()
context_artifact_chunks = chunk_str(chunk_text)
ID_MODEL_SYSTEM_PROMPT = """
You are a helpful assistant specializing in identifying parts of text documents that best correspond to the answer for a query.
You specialize in thinking deeply about the answer to a given question and then returning the exact sentences word for word that
best contain the answer to the question from the given context. The context is preceded by a section header called CONTEXT.
"""
# Add context - hacky way of just adding the first chunk if context is too long
context = f"CONTEXT:\n {context_artifact_chunks[0]}"
INPUT_MSG = question + context
# Make the API call to o1-mini
id_response = client.chat.completions.create(
model="gpt-4o",
messages = [
{"role": "user", "content": f"instructions {ID_MODEL_SYSTEM_PROMPT}\n, question: {INPUT_MSG}"}],
)
id_response_content = id_response.choices[0].message.content
# Post-processing
id_response_content = id_response_content.replace("\n", " ").replace('`', "")
top_k_ids.append(id_response_content)
print(f"Finished attribution for top {idx} query")
return jsonify({
"status": "success",
"top_k_ids": top_k_ids
})
@app.route("/create_class", methods=["POST"])
def create_class():
"""
Creates a new schema for the user: user_name.class_name
Optionally create a table as well for chunk embeddings.
"""
try:
data = request.json
user_name = data.get("user_name", "UnknownUser")
class
[truncated — 7182 more characters]
```
### frontend/src/index.js
```javascript
import React from "react";
import ReactDOM from "react-dom/client";
import "./index.css";
import App from "./App";
import reportWebVitals from "./reportWebVitals";
const root = ReactDOM.createRoot(document.getElementById("root"));
root.render(
);
// If you want to start measuring performance in your app, pass a function
// to log results (for example: reportWebVitals(console.log))
// or send to an analytics endpoint. Learn more: https://bit.ly/CRA-vitals
reportWebVitals();
```
### frontend/src/App.js
```javascript
// App.js
import React from 'react';
import { BrowserRouter as Router, Routes, Route, Navigate } from 'react-router-dom';
import { AuthProvider, useAuth } from './AuthContext';
import Login from './Login';
import Dashboard from './Dashboard';
import ChatInterface from './ChatInterface';
import FirebaseVideoPlayer from './FirebaseVideoPlayer';
import VideoPage from './VideoPage';
const PrivateRoute = ({ children }) => {
const { user } = useAuth();
return user ? children : ;
};
function App() {
return (
{/* Public routes */}
} />
{/* Protected routes */}
}
/>
{/* Class name (or ID) route parameter */}
}
/>
} />
{/* Default route goes to dashboard */}
} />
);
}
export default App;
```
### environment.yaml
```yaml
name: omnis_env
channels:
- conda-forge
- defaults
dependencies:
- python=3.12
- pydub
- pip
- pip:
- annotated-types
- anyio
- certifi
- click
- distro
- h11
- httpcore
- httpx
- idna
- jiter
- openai
- pydantic
- pydantic-core
- python-dotenv
- setuptools
- sniffio
- tqdm
- typing-extensions
- wheel
```
### scripts/backend_database.py
```python
# backend_database.py
import iris
import time
import os
from dotenv import load_dotenv
from langchain.embeddings.openai import OpenAIEmbeddings
def setup_openai_key():
"""Load environment variables and set up OpenAI API key if not present."""
load_dotenv(override=True)
if not os.environ.get("OPENAI_API_KEY"):
raise ValueError("OpenAI API key not found in environment variables")
def setup_database_connection():
"""Setup and return IRIS database connection."""
username = 'demo'
password = 'demo'
hostname = os.getenv('IRIS_HOSTNAME', 'localhost')
port = '1972'
namespace = 'USER'
connection_string = f"{hostname}:{port}/{namespace}"
return iris.connect(connection_string, username, password)
def ensure_schema_exists(cursor, schema_name):
# Attempt to create the schema. IRIS supports CREATE SCHEMA in some versions:
cursor.execute(f"CREATE SCHEMA IF NOT EXISTS {schema_name}")
def ensure_table_exists(cursor, user_name):
safe_user_name = user_name.strip().replace(" ", "")
# 1) Create the schema (if IRIS allows CREATE SCHEMA)
ensure_schema_exists(cursor, safe_user_name)
# 2) Then create the table inside that schema
cursor.execute(f"""
CREATE TABLE IF NOT EXISTS {safe_user_name}.classes (
chunk_url VARCHAR(1000),
chunk_text VARCHAR(10000),
embedding VECTOR(DOUBLE, 1536),
original_file_url VARCHAR(1000),
file_type VARCHAR(1000),
file_name VARCHAR(1000)
)
""")
return True
def check_chunk_exists(cursor, user_name, chunk_url):
safe_user_name = user_name.strip().replace(" ", "")
cursor.execute(f"""
SELECT COUNT(*) FROM {safe_user_name}.classes
WHERE chunk_url = ?
""", [chunk_url])
return cursor.fetchone()[0] > 0
def add_embeddings(chunk_url, chunk_text, original_file_url, user_name, file_type, file_name):
try:
setup_openai_key()
conn = setup_database_connection()
cursor = conn.cursor()
table_name = user_name # Use user's name for the table
ensure_table_exists(cursor, table_name)
if check_chunk_exists(cursor, table_name, chunk_url):
return {"status": "success", "message": "Chunk already processed", "already_exists": True}
embeddings_model = OpenAIEmbeddings(model="text-embedding-3-small")
embedding = embeddings_model.embed_documents([chunk_text])[0]
sql = f"""
INSERT INTO {table_name}.classes
(chunk_url, chunk_text, embedding, original_file_url, file_type, file_name)
VALUES (?, ?, TO_VECTOR(?), ?, ?, ?)
"""
cursor.execute(sql, [chunk_url, chunk_text, str(embedding), original_file_url, file_type, file_name])
conn.commit()
return {"status": "success", "message": "New embeddings created and stored", "already_exists": False}
except Exception as e:
return {"status": "error", "message": str(e)}
finally:
if 'cursor' in locals():
cursor.close()
if 'conn' in locals():
conn.close()
def search_files(search_phrase, user_name, num_results=3):
"""
Search across the user's {user_name}_underscored.classes table,
retrieving the top K most similar chunks.
"""
try:
# 1) Setup keys + DB
setup_openai_key()
conn = setup_database_connection()
cursor = conn.cursor()
safe_user_name = user_name.strip().replace(" ", "")
# 3) Embed the query
embeddings_model = OpenAIEmbeddings(model="text-embedding-3-small")
search_vector = embeddings_model.embed_query(search_phrase)
# 4) Perform a top-K search in that user’s classes table only
sql = f"""
SELECT TOP ? chunk_url,
chunk_text,
original_file_url,
file_type,
file_name,
VECTOR_DOT_PRODUCT(embedding, TO_VECTOR(?)) as similarity_score
FROM {safe_user_name}.classes
ORDER BY VECTOR_DOT_PRODUCT(embedding, TO_VECTOR(?)) DESC
"""
cursor.execute(sql, [num_results, str(search_vector), str(search_vector)])
results = cursor.fetchall()
# 5) Format results
formatted_results = []
for row in results:
chunk_url, chunk_text, original_file_url, file_type, file_name, score = row
formatted_results.append({
"chunk_url": chunk_url,
"chunk_text": chunk_text,
"original_file_url": original_file_url,
"file_type": file_type,
"file_name": file_name,
"score": score
})
return {"status": "success", "results": formatted_results}
except Exception as e:
return {"status": "error", "message": str(e)}
finally:
if 'cursor' in locals():
cursor.close()
if 'conn' in locals():
conn.close()
```
### scripts/video_to_transcript.py
```python
import os
import time
import tempfile
import multiprocessing as mp
from openai import OpenAI
from tqdm import tqdm
from pydub import AudioSegment
from pathlib import Path
from dotenv import load_dotenv
load_dotenv()
class AudioHandler:
def __init__(self, filepath: str | Path):
if isinstance(filepath, Path):
filepath = str(filepath)
self.fp = str(filepath) if isinstance(filepath, Path) else filepath
def _split_audio(self, chunk_length_ms=6000) -> list:
print(f"Loading audio file: {self.fp}")
audio = AudioSegment.from_file(self.fp)
print("done loading")
chunks = []
temp_dir = tempfile.mkdtemp()
print(f"Temporary directory created at: {temp_dir}")
print("num aydios", len(audio))
for i, start in tqdm(enumerate(range(0, len(audio), chunk_length_ms))):
end = start + chunk_length_ms
chunk = audio[start:end]
if len(chunk) < 100: # Check for chunks less than 0.1 second
print(f"Skipping chunk {i} as it is too short: {len(chunk)} ms")
continue
chunk_path = os.path.join(temp_dir, f"chunk_{i}.wav")
chunk.export(chunk_path, format="wav")
chunks.append((i, start, end, chunk_path))
print(f"Split audio into {len(chunks)} chunks")
return chunks
def _transcribe_chunk(self, args: tuple) -> tuple:
chunk_index, chunk_start, chunk_end, chunk_path = args
try:
open_ai_key = os.getenv("OPENAI_APIKEY")
client = OpenAI(
api_key=open_ai_key,
)
audio_file = open(chunk_path, "rb")
transcription = client.audio.transcriptions.create(
model="whisper-1",
file=audio_file
)
return chunk_index, chunk_start, chunk_end, transcription.text
except Exception as e:
print(f"Error processing chunk {chunk_index}: {str(e)}")
return chunk_index, ""
finally:
try:
os.remove(chunk_path)
except:
pass
def process_audio(self):
try:
print("in here")
start_time = time.time()
chunks = self._split_audio()
process_args = [(i, start, end, path) for i, start, end, path in chunks]
num_processes = max(1, mp.cpu_count() - 1)
print(f"Starting parallel transcription with {num_processes} processes...")
with mp.Pool(num_processes) as pool:
results = list(tqdm(
pool.imap(self._transcribe_chunk, process_args),
total=len(process_args),
desc="Transcribing chunks"
))
def format_time(ms):
minutes, seconds = divmod(ms // 1000, 60)
return f"{minutes:02}:{seconds:02}"
results.sort(key=lambda x: x[0])
full_transcript_with_timestamps = ""
full_transcript = ""
for _, start, end, text in results:
#make it in 00:00 format
start = format_time(start)
end = format_time(end)
full_transcript_with_timestamps += f"{start} - {end}: {text}\n"
full_transcript += f"{text}\n"
print("finished transcribing")
# print(full_transcript)
print(self.fp)
raw_fp = self.fp.replace(".mp3", "")
raw_fp += "_timestamps.mp3"
output_path_timestamps = Path(raw_fp).with_suffix('.txt')
with open(output_path_timestamps, "w", encoding="utf-8") as f:
f.write(full_transcript_with_timestamps)
output_path = Path(self.fp).with_suffix('.txt')
with open(output_path, "w", encoding="utf-8") as f:
f.write(full_transcript)
print("finished writing")
end_time = time.time()
duration = end_time - start_time
#delete the temp directory now
# temp_dir = os.path.dirname(chunks[0][1])
# for _, _, files in os.walk(temp_dir):
# for file in files:
# os.remove(os.path.join(temp_dir, file))
# print("deleted the temp dir")
print(f"\nTranscription completed in {duration:.2f} seconds")
# print(f"Transcript saved to: {output_path}")
return output_path_timestamps, output_path
except Exception as e:
print(f"An error occurred: {str(e)}")
raise
finally:
for _, start, end, chunk_path in chunks:
try:
if os.path.exists(chunk_path):
os.remove(chunk_path)
except:
pass
# if __name__ == "__main__":
# fp = './audio_files/videoplayback.mp4'
# if fp[-4:] == ".mp4":
# ah = AudioHandler(fp)
# fp = ah.process_audio()
# print("transcript(s) saved to: ", fp)
```
### scripts/identification_generation.py
```python
import pandas as pd
import numpy as np
import os
from openai import OpenAI
from dotenv import load_dotenv
from PyPDF2 import PdfReader
from typing import List
from tqdm import tqdm
import re
from difflib import SequenceMatcher
def setup_openai_key():
"""
Load environment variables and set up OpenAI API key if not present.
Return client object with key
"""
load_dotenv(override=True)
if not os.environ.get("OPENAI_API_KEY"):
raise ValueError("OpenAI API key not found in environment variables")
open_ai_key = os.getenv("OPENAI_APIKEY")
client = OpenAI(
api_key=open_ai_key,
)
return client
def pdf_to_string(pdf_path):
"""
Parse pdfs: all text documents will be in pdf format
"""
# Create a PDF reader object
pdf_reader = PdfReader(pdf_path)
# Initialize an empty string to store the text
text = ""
# Iterate through all pages and extract text
for page in pdf_reader.pages:
text += page.extract_text()
# Heuristics to fix processing
processed_text = text.strip()
processed_text = processed_text.strip().replace("\n", " ").replace("\t", " ").replace(" ", " ")
processed_text = processed_text.replace("●", "")
return processed_text.strip()
def read_mp3_transcript(transcript_path):
"""
Takes in transcript path (txt) and returns string representation of transcript
"""
with open(transcript_path, 'r') as file:
content = file.read()
return content
def chunk_str(input_str, chunk_size=16384):
"""
Chunk str into pieces of approximately chunk_size words each
"""
words = input_str.split()
chunked_strs = []
for i in range(0, len(words), chunk_size):
chunk = ' '.join(words[i:i + chunk_size])
chunked_strs.append(chunk)
return chunked_strs
# Video transcript parsing
def parse_text_from_timestamps_original(data: str) -> str:
return re.sub(r'\d{2}:\d{2} - \d{2}:\d{2}: ', '', data).strip().split(": ")
def parse_text_from_timestamps(data: str) -> str:
"""
Removes timestamps from a given formatted text.
Parameters:
data (str): The input text with timestamps.
Returns:
List: The cleaned text without timestamps as a list
"""
# This regular expression will capture:
# Group 1: the timestamp range, e.g. "HH:MM - HH:MM"
# Group 2: all text that follows until the next timestamp or the end of the string
pattern = r'(\d{2}:\d{2}\s*-\s*\d{2}:\d{2}):\s*(.*?)\s*(?=\d{2}:\d{2}\s*-\s*\d{2}:\d{2}:|$)'
matches = re.findall(pattern, data, flags=re.DOTALL)
results = []
for match in matches:
timestamp_range = match[0] # e.g. "00:00 - 00:06"
text_block = match[1]
# Split into start and end times
start_time, end_time = timestamp_range.split('-')
start_time = start_time.strip() # e.g. "00:00"
end_time = end_time.strip() # e.g. "00:06"
# Create a dictionary with your desired fields
results.append({
"start_time": start_time,
"end_time": end_time,
"text": text_block.strip()
})
# results is now a list of dicts, each with start_time, end_time, and text.
return results
def parse_timestamps(chunks: list ) -> list:
"""
Extracts timestamps from a given formatted text.
Parameters:
data (str): The input text with timestamps.
Returns:
list: A list of extracted timestamps in the form of seconds since the start of the video
"""
timestamps = [chunk["start_time"] for chunk in chunks]
return [int(mm) * 60 + int(ss) for mm, ss in (ts.split(":") for ts in timestamps)]
def tokenize(text):
"""
Tokenizes the text into a set of lowercase words.
Non-alphabetic characters are used as delimiters.
"""
return set(re.findall(r'\w+', text.lower()))
def jaccard_similarity(tokens_a, tokens_b):
"""
Computes the Jaccard similarity between two sets of tokens:
Jaccard = |A ∩ B| / |A ∪ B|
"""
if not tokens_a and not tokens_b:
return 0.0
intersection = tokens_a.intersection(tokens_b)
union = tokens_a.union(tokens_b)
return len(intersection) / len(union)
def get_timestamp_from_answer(sentence, chunks):
"""
Finds the chunk with the maximum word overlap (via Jaccard similarity)
with the given sentence and returns its timestamp.
Parameters:
- sentence (str): The sentence to compare.
- chunks (dict): A dictionary of chunk_name -> chunk_id.
Each `chunk_name` is the textual content to compare,
and `chunk_id` might be a timestamp or any related info.
Returns:
- The chunk_id of the entry with the highest word-based overlap.
"""
best_score = 0.0
best_chunk = None
# Tokenize the input sentence
sentence_tokens = tokenize(sentence)
for chunk_name, chunk_id in chunks.items():
chunk_tokens = tokenize(chunk_name)
similarity = jaccard_similarity(sentence_tokens, chunk_tokens)
if similarity > best_score:
best_score = similarity
best_chunk = chunk_id
return best_chunk
def run_identification(top_k_queries: List[str], top_k_types: List[str], question: str):
"""
top_k_queries: text over either video transcript or documents (notes, slides, etc.)
top_k_types: either video, text, or image
"""
client = setup_openai_key()
top_k_ids = []
for idx, chunk_text, chunk_type in tqdm(enumerate(zip(top_k_queries, top_k_types))):
if chunk_type == "video":
# Remove whitespace after joining
chunk_text = "\n".join(parse_text_from_timestamps(chunk_text)).strip()
context_artifact_chunks = chunk_str(chunk_text)
ID_MODEL_SYSTEM_PROMPT = """
You are a helpful assistant specializing in identifying parts of text documents that best cor
[truncated — 7422 more characters]
```
### frontend/src/setupTests.js
```javascript
// jest-dom adds custom jest matchers for asserting on DOM nodes.
// allows you to do things like:
// expect(element).toHaveTextContent(/react/i)
// learn more: https://github.com/testing-library/jest-dom
import '@testing-library/jest-dom';
```
[17 more indexed source files omitted to keep this export small. The full file list is in the Codebase structure section above.]