Project Info
This project did not submit a demo video on Devpost.
Inspiration
Finding the right dataset is often harder than building the model that uses it. Search portals can return hundreds of links, but they do not always explain whether a dataset fits a project, contains the needed fields, or is suitable for a particular analysis. We wanted to make dataset discovery feel conversational without making it ungrounded: users should be able to ask for “3 NBA datasets for basketball analysis” or “Formula 1 race strategy data,” then receive real, traceable dataset recommendations rather than AI-invented answers.
What it does
Data Scout is chat-based dataset discovery assistant. A user describes what they need in plain language. The application: Interprets the request into a safe search plan using Claude Opus on Amazon Bedrock. Searches a live OpenSearch index of enriched Kaggle & Hugging Face metadata. Returns ranked dataset recommendations with canonical Kaggle & Hugging Face links, summaries, match details, files, schema fields when available, and an AI-generated summary of the result set. Falls back safely to deterministic keyword search if AI interpretation is unavailable or invalid. The catalog currently contains more than 1,600 indexed dataset records and updates automatically as the crawler adds or changes metadata.
How we built it
We built Data Scout as a serverless AWS application. A Python crawler collects public Kaggle & Hugging Face dataset metadata. Amazon Bedrock enriches metadata with structured domain/data-type labels and factual use-case summaries. DynamoDB stores the authoritative dataset records. DynamoDB Streams trigger an index worker that projects active records into an OpenSearch datasets-v1 index. API Gateway routes chat search requests to a Python Lambda. Claude Opus converts user text into a strictly validated search plan: keywords, suggested result count, formats, sources, licenses, required fields, and recency. Lambda builds a bounded OpenSearch query, returns only public result fields, and uses IAM-signed requests. A local static chat UI displays results, match details, schema/file metadata, and a short AI summary of the returned datasets. Chat UI → API Gateway → Query Lambda → Bedrock search plan ↘ OpenSearch → ranked datasets Crawler → DynamoDB → Stream → Index worker → OpenSearch
Challenges we ran into
The biggest challenge was balancing natural-language flexibility with trustworthy retrieval. Broad queries such as “housing prices” can accidentally match unrelated crypto or stock datasets because words like “price” are common. We built a read-only ranking evaluation harness to compare candidate ranking strategies before changing production behavior. We also encountered sparse metadata: not every dataset has populated file formats or schema fields, so treating AI-suggested formats as hard filters could hide otherwise useful results. We therefore keep model suggestions as soft ranking signals while preserving explicit user filters as hard constraints. Other challenges included handling pagination safely, keeping DynamoDB and OpenSearch synchronized, configuring signed OpenSearch access, avoiding leaked internal search details, and making the chat UI readable as result lists grow.
Accomplishments we're proud of
Built an end-to-end, live pipeline from crawler to DynamoDB to OpenSearch to chat UI. Indexed more than 1,600 dataset records. Made the AI grounded: Claude interprets and summarizes, while OpenSearch retrieves real dataset records. Added safe fallback behavior when Bedrock fails or returns invalid output. Kept results explainable through canonical links, match details, schema/file data, and factual summaries. Created a ranking evaluation harness so retrieval changes can be tested before deployment. Built a polished chat experience with independent scrolling and compact dataset cards.
What we learned
We learned that AI is most useful here as an interpreter and assistant—not as an unbounded search engine. Claude is good at turning a vague request into structured intent and explaining a set of real results. OpenSearch is good at fast retrieval over known metadata. Combining the two gives a more reliable experience than asking a model to generate recommendations on its own. We also learned that retrieval quality depends as much on metadata quality and evaluation as it does on model prompts. More datasets improve coverage, but better enrichment, relevance tests, synonyms, and ranking logic are necessary to avoid noisy results.
What's next
Ingest more sources beyond Kaggle & HF . Improve metadata coverage for files, schemas, licenses, and tags. Use the ranking benchmark to test synonym-aware and phrase-aware retrieval improvements. Add richer filtering controls while keeping chat as the primary interface. Host the frontend for a shareable public demo. Add budget alarms, operational dashboards, and production CORS controls.
Data Scout
Data Scout is a chat-based dataset discovery assistant. It interprets a plain-language request with Bedrock, searches an OpenSearch index of enriched dataset metadata, and returns grounded Kaggle recommendations.
Search API
POST https://lepdzanhh1.execute-api.us-east-1.amazonaws.com/v1/datasets/search
The deployed Data Scout stack provides API Gateway, Lambda, Amazon Bedrock, CloudWatch logs, and IAM-signed access to the background-owned OpenSearch index.
CLI
$env:DATA_CURATOR_API_URL = 'https://lepdzanhh1.execute-api.us-east-1.amazonaws.com/v1/datasets/search'
python search_datasets.py 'food datasets for nutrition analysis' --source kaggle --format csv
Use --json to print the full response. Run python -m unittest discover -s tests -v
for the local test suite.
Read-only ranking evaluation
Compare the current lexical query builder with a phrase-aware candidate without changing Lambda, DynamoDB, or OpenSearch:
$env:OPENSEARCH_ENDPOINT = 'https://your-domain.us-east-1.es.amazonaws.com'
$env:AWS_PROFILE = 'AdministratorAccess-958975572378'
python scripts/evaluate_ranking.py
The representative cases are in scripts/ranking_queries.json. The harness
uses deterministic keyword plans to avoid Bedrock variability and cost.
Analysis
View
Metric
- 24
- 9
Figures cover GitHub contributors during the hackathon window. A co-authored commit counts in full for each author, so per-member totals add up to more than the whole-team figures.
Technology
- CSSIn code
- HTMLIn code
- JavaScriptIn code
- PythonIn code
- AWSClaimed
4 of 5 appear in the indexed code. 1 claimed on Devpost could not be matched to code, which may simply mean the tool leaves no trace in the repository.
AI coding agents
No AI coding agent signals were found in this repository.
Detected from committed agent config files and commit authorship. Absence of a signal is not proof an agent was unused.
Codebase size
Source size
221 KB
Source files
51
Counts recognized source files only; vendored directories, binaries and lockfiles are excluded, so this is smaller than the repository on disk.
Repository
partht555/data-scout
64 files · 232 KB · @ 3752cac
Structure
Interface
2 files · 3%Screens, components and styles rendered to the user.
Application logic
27 files · 42%Domain rules, services and shared utilities.
+1 more
Supporting
Layers are inferred from where files sit in the tree, not from reading the code. A project that names its directories unconventionally will read oddly here — open the file browser to check anything the diagram implies.
Languages
- Python80%
- JavaScript7%
- YAML5%
- Markdown4%
- CSS3%
- HTML1%
Share of indexed source by file size. Binary and vendored files are excluded.
Dependencies
infra/requirements.txt
pypi · 2- aws-cdk-lib
- constructs
lambdas/huggingface_crawler/requirements.txt
pypi · 1- requests
lambdas/kaggle_crawler/requirements.txt
pypi · 1- requests
Declared in the repository’s manifests at the indexed commit. A declared package is not proof it is used, and runtime dependencies are listed first.
Export this project's context (description, README, evidence, key source files) to chat with an AI agent elsewhere.