# Project export: Market Atlas This document was generated by HackStack to give an AI agent context about a hackathon project. Sections are labeled with their provenance; content marked as truncated was cut to keep this document small. ## Project metadata - Hackathon: CruzHacks 2026 - Tagline: We measure when market narratives stop matching reality. WE ARE FIRST TIMERS! - Devpost: https://devpost.com/software/market-atlas - GitHub: https://github.com/SashaSkind/market-sentiment-analysis - Video: https://www.youtube.com/embed/ycN8J-_9bts?enablejsapi=1&hl=en_US&rel=0&start=&version=3&wmode=transparent - Team: 4 GitHub contributor(s) — SashaSkind (16 commits), nathan milton (15 commits), Idan Vaysman (10 commits), Claude Opus 4.5 (1 commits) ## Devpost submission (written by the team) ### Inspiration Markets are flooded with confident narratives — headlines that sound convincing, trend strongly on social media, and often move sentiment indicators. But as investors, we kept asking a simpler question: When does the story stop matching reality? Most sentiment tools measure how positive or negative the news is. Very few measure whether that sentiment is actually reliable when compared to real price movement. We were inspired to build a tool that doesn’t just summarize narratives, but tests them against market outcomes. ### What it does Market Atlas detects moments of false conviction — periods where headlines and sentiment are confident, but price action disagrees. For each stock, the app: Aggregates daily news sentiment Compares it to actual price returns Computes a Narrative Reliability Score over 7 / 14 / 30 days Surfaces misalignment days, where bullish narratives coincided with falling prices (or vice versa) Allows users to drill down into the actual headlines behind those failures The result is not just a score, but an auditable story: Score → Failure → Evidence ### How we built it Backend Python + FastAPI for the API layer Supabase (Postgres) for structured storage Background worker pipeline for ingestion and computation Daily jobs to: Ingest news articles Score sentiment using an ML model (with chunking to handle token limits) Aggregate daily sentiment Ingest historical and daily prices Compute rolling alignment metrics Ingest news articles Score sentiment using an ML model (with chunking to handle token limits) Aggregate daily sentiment Ingest historical and daily prices Compute rolling alignment metrics Frontend Next.js (App Router) TypeScript Material UI for fast, consistent layout A narrative-driven UI that prioritizes insight over charts Key Metric Narrative alignment is computed using: Directional agreement between sentiment and returns Strength-weighted misalignment detection Rolling windows instead of single-day noise All external data fetching happens outside the UI. The frontend only reads from the database, ensuring consistency and speed. ### Challenges we ran into Data coverage limits: Free news APIs only provide limited historical depth, forcing us to explicitly track and surface sentiment coverage. Token limits in ML models: Articles often exceeded model limits, requiring chunking and aggregation without biasing results. Avoiding noisy metrics: Simple correlation alone was misleading; we had to combine direction, magnitude, and confidence. UI clarity: The hardest part was deciding what not to show and framing the data as a clear narrative. ### Accomplishments we're proud of Turning abstract sentiment analysis into a testable, falsifiable signal Making misalignment auditable through real headlines Building a full ingestion → scoring → aggregation → visualization pipeline during a hackathon Designing a UI that tells a story instead of overwhelming users with charts Explicitly communicating uncertainty and data coverage ### What we learned Sentiment alone is cheap; sentiment reliability is hard Alignment matters more than raw positivity Clear narratives beat complex metrics Precomputing metrics is far more scalable than calculating them on every page load Strong framing can make existing data feel entirely new ### What's next Expand narrative sources (earnings calls, social platforms, analyst notes) Improve alignment math with volume- and confidence-weighted signals Add a full misalignment map across stocks and time Generate automatic explanations for why narratives failed Turn misalignment detection into alerts and weekly briefs Market Atlas isn’t about predicting markets. It’s about knowing when not to trust the story. ## README (from the GitHub repository) # Market Atlas (Sentiment Reality) **Find false conviction in financial headlines.** Market Atlas measures when financial news sentiment *diverges* from actual stock price movement — and surfaces the exact days + headlines where the narrative failed. > **YouTube Demo:** [Watch here](https://www.youtube.com/watch?v=ycN8J-_9bts) --- ## Why this is different Most "stock sentiment" tools stop at *what the news feels like*. Market Atlas asks a more practical question: > **Can you trust the story right now?** It computes a **Narrative Reliability Score** over rolling windows (7/14/30 days), then shows: - **Where the narrative failed** (specific dates where sentiment disagreed with price returns) - **Evidence** (the actual headlines that drove the sentiment signal) This makes the signal **auditable**, not just a black-box score. --- ## What it does For each tracked stock, Market Atlas: 1. Ingests relevant news articles (NewsAPI) 2. Extracts full text from each URL (newspaper3k) 3. Scores sentiment using a finance-tuned HuggingFace model 4. Fetches historical prices (yfinance) 5. Computes daily aggregates and rolling alignment metrics 6. Displays a single-page dashboard: - Reliability score (Aligned / Noisy / Misleading) - Misalignment days list (click to inspect) - Recent headlines with sentiment labels - Basic stock info + returns --- ## Architecture (high-level) ![Architecture diagram](architecture-diagram.png) **Strict separation of responsibilities:** - `jobs/` — all external API calls, all ML inference, all DB writes - `api/` — read-only queries against Supabase Postgres, returns JSON - `web/` — dashboard UI, fetches from API only (no external data) --- ## Tech stack **Backend** - Python + **FastAPI** - **Supabase Postgres** - Raw psycopg2 (no ORM) **ML / NLP** - Hugging Face Transformers `mrm8488/distilroberta-finetuned-financial-news-sentiment-analysis` **Data sources** - NewsAPI (article metadata) - newspaper3k (full text extraction) - yfinance (historical prices) **Frontend** - Next.js App Router (single-page dashboard) - TypeScript + React - Material UI (MUI) dark theme --- ## Core metric: Narrative Reliability We compute narrative alignment over a rolling window using: - **Pearson correlation** between daily sentiment and daily returns - **Directional match** = fraction of days where sentiment sign matches return sign Alignment score: $$ A = 0.5 \times \rho(s, r) + 0.5 \times (2D - 1) $$ Where *A* = alignment, *ρ(s, r)* = Pearson correlation between sentiment and returns, *D* = directional match rate. Interpretation: - $\geq 0.3$ → **Aligned** - $\leq -0.3$ → **Misleading** - otherwise → **Noisy** We also surface **misalignment days** where: - |sentiment_avg| ≥ 0.05 and |return_1d| ≥ 0.5% - sentiment sign disagrees with return sign Sorted by strength: |sentiment| · |return| --- ## Database schema (overview) | Table | Purpose | |------|---------| | `tracked_stocks` | Watchlist tickers | | `tasks` | Postgres-backed job queue | | `items` | Raw news article metadata | | `item_scores` | Per-article ML outputs | | `prices_daily` | OHLCV + `return_1d` | | `daily_agg` | Daily sentiment rollups | | `metrics_windowed` | Rolling alignment metrics | | `alignment_daily` | Optional daily alignment system | | `current_prices` | Snapshot cache for latest price | --- ## API surface (internal) | Route | Method | Description | |------|--------|-------------| | `/health` | GET | Health check | | `/api/stocks` | GET | List tracked stocks | | `/api/stocks` | POST | Add ticker → enqueue BACKFILL task | | `/api/stocks/refresh` | POST | Enqueue REFRESH task | | `/api/dashboard?ticker=X&period=N` | GET | Main dashboard payload | | `/api/headlines/by-date?ticker=X&date=YYYY-MM-DD` | GET | Drilldown headlines for a day | --- ## How "Refresh" works (end-to-end) 1) User clicks **Refresh** in the dashboard 2) Frontend calls: ```http POST /api/stocks/refresh {"ticker":"TSLA"} ``` 3) API enqueues a task in `tasks` 4) Worker claims tasks with `FOR UPDATE SKIP LOCKED` 5) Pipeline runs per ticker: - ingest news → score new items → fetch prices → aggregate → compute metrics 6) Frontend re-fetches dashboard and renders updated results > Note: the UI currently uses a fixed delay and re-fetches once. If the pipeline takes longer (e.g., heavy article extraction), the next refresh will reflect the newest state. --- ## Notable design decisions - **No ML in the request path:** the API never performs inference or external calls; heavy work runs in `jobs/`. - **Token limits handled properly:** long articles are chunked with overlap; chunk results are aggregated into a single signed score. - **Idempotent ingestion:** unique constraints + `ON CONFLICT` prevent duplicates and make re-runs safe. - **Auditable "signal":** misalignment is tied to specific dates + headlines (not just an abstract number). --- ## Known limitations - Article extraction via newspaper3k can fail on paywalls or unusual markup - NewsAPI free-tier limits historical backfill depth - No websocket/streaming updates yet (refresh is queued + later re-fetched) --- ## What's next - Wire `DAILY_UPDATE_ALL` to GitHub Actions cron for automatic freshness - Add a cross-ticker "misalignment map" view + filters - Expand narrative sources (earnings calls, social, analyst notes) - Improve alignment reliability with volume/confidence weighting and calibration --- ## About this repo This repository includes: - `jobs/` — ingestion + ML scoring + aggregation pipeline (writes to DB) - `api/` — FastAPI read-only API - `web/` — Next.js dashboard UI ## Detected evidence (automated analysis) Indexed codebase: 65 recognized source files, 196 KB. - CSS (language) — detected in the code - FastAPI (technology) — detected in the code - Hugging Face (technology) — detected in the code - Next.js (technology) — detected in the code - Python (language) — detected in the code - PyTorch (technology) — detected in the code - React (technology) — detected in the code - SQL (language) — detected in the code - Tailwind CSS (technology) — detected in the code - TypeScript (language) — detected in the code - PostgreSQL (technology) — claimed on Devpost, not found in the code - AI coding agent: Claude Code — evidence: config files committed to the repository; commit authorship or trailers ## Codebase structure (from repository index) ### Files (78 of 78) ``` .gitignore CLAUDE.md README.md sentiment-reality/.env.example sentiment-reality/.github/workflows/cron.yml sentiment-reality/.gitignore sentiment-reality/api/config.py sentiment-reality/api/db.py sentiment-reality/api/deps.py sentiment-reality/api/main.py sentiment-reality/api/models.py sentiment-reality/api/requirements.txt sentiment-reality/api/routers/__init__.py sentiment-reality/api/routers/dashboard.py sentiment-reality/api/routers/headlines.py sentiment-reality/api/routers/health.py sentiment-reality/api/routers/items.py sentiment-reality/api/routers/stocks.py sentiment-reality/api/schemas.py sentiment-reality/api/services/aggregation.py sentiment-reality/api/services/metrics.py sentiment-reality/api/sql/schema.sql sentiment-reality/jobs/__init__.py sentiment-reality/jobs/alignment.py sentiment-reality/jobs/bootstrap_watchlist.py sentiment-reality/jobs/compute/__init__.py sentiment-reality/jobs/compute/aggregate_daily.py sentiment-reality/jobs/compute/metrics.py sentiment-reality/jobs/db.py sentiment-reality/jobs/ingest_news.py sentiment-reality/jobs/ingest_to_db.py sentiment-reality/jobs/ml/__init__.py sentiment-reality/jobs/ml/sentiment.py sentiment-reality/jobs/pipeline.py sentiment-reality/jobs/providers/__init__.py sentiment-reality/jobs/providers/news.py sentiment-reality/jobs/providers/prices.py sentiment-reality/jobs/query_db.py sentiment-reality/jobs/requirements.txt sentiment-reality/jobs/run_local.py sentiment-reality/jobs/score_unscored_items.py sentiment-reality/jobs/update_current_prices.py sentiment-reality/jobs/worker.py sentiment-reality/Makefile sentiment-reality/README.md sentiment-reality/web/app/layout.tsx sentiment-reality/web/app/page.tsx sentiment-reality/web/components/charts/AlignmentChart.tsx sentiment-reality/web/components/charts/MisalignmentMiniGraph.tsx sentiment-reality/web/components/charts/PriceChart.tsx sentiment-reality/web/components/charts/SentimentChart.tsx sentiment-reality/web/components/layout/AppShell.tsx sentiment-reality/web/components/modals/AddStockModal.tsx sentiment-reality/web/components/modals/HeadlineDetailsModal.tsx sentiment-reality/web/components/modals/HeadlinesForDateModal.tsx sentiment-reality/web/components/modals/MisalignmentMapModal.tsx sentiment-reality/web/components/panels/HeadlinesPanel.tsx sentiment-reality/web/components/panels/MisalignmentDaysPanel.tsx sentiment-reality/web/components/panels/NarrativeReliabilityPanel.tsx sentiment-reality/web/components/panels/PricePanel.tsx sentiment-reality/web/components/panels/SentimentPanel.tsx sentiment-reality/web/components/panels/WatchlistPanel.tsx sentiment-reality/web/components/panels/WatchlistSelector.tsx sentiment-reality/web/components/ui/.gitkeep sentiment-reality/web/lib/api.ts sentiment-reality/web/lib/types.ts sentiment-reality/web/lib/utils.ts sentiment-reality/web/next-env.d.ts sentiment-reality/web/next.config.mjs sentiment-reality/web/package.json sentiment-reality/web/postcss.config.mjs sentiment-reality/web/public/.gitkeep sentiment-reality/web/styles/.gitkeep sentiment-reality/web/styles/globals.css sentiment-reality/web/tailwind.config.ts sentiment-reality/web/theme/MuiProviders.tsx sentiment-reality/web/theme/theme.ts sentiment-reality/web/tsconfig.json ``` ### Dependencies - sentiment-reality/api/requirements.txt: fastapi, psycopg2-binary, pydantic, python-dotenv, uvicorn - sentiment-reality/jobs/requirements.txt: feedparser, lxml, lxml_html_clean, newspaper3k, nltk, numpy, pandas, psycopg2-binary, python-dateutil, python-dotenv, requests, torch, transformers, yfinance - sentiment-reality/web/package.json: @emotion/react@^11.14.0, @emotion/styled@^11.14.1, @mui/material@^7.3.7, @mui/material-nextjs@^7.3.7, @types/node@^20.0.0, @types/react@^18.2.0, @types/react-dom@^18.2.0, autoprefixer@^10.4.0, next@^14.2.0, postcss@^8.4.0, react@^18.2.0, react-dom@^18.2.0, recharts@^2.12.0, tailwindcss@^3.4.0, typescript@^5.0.0 ### Recent commits (newest first) - added architecture-diagram.png file for README.md - Delete architecture-diagram.png - updated architecture-diagram.png file - added architecture-diagram.png for README - Revise README for Market Atlas project details - edited stock info - Merge branch 'main' of github.com:SashaSkind/market-sentiment-analysis - added aligenment logic - Merge pull request #2 from SashaSkind/ui-refactor - Refactor UI to narrative funnel layout with misalignment tracking - removed text from the bottom - Merge branch 'main' of https://github.com/SashaSkind/market-sentiment-analysis - redo the text - added alignment to daily update - add aligment to db - aligment added to db - aligenkent added - Merge branch 'main' of github.com:SashaSkind/market-sentiment-analysis - please dont delete my file - Add configurable headlines limit and clickable article links ## Key source files (fetched from GitHub, selected and truncated for size) ### CLAUDE.md ```markdown # Claude Instructions — Sentiment Reality Project This file defines how Claude should work in this repository. Follow these rules strictly. --- ## Project Overview This project analyzes when **market sentiment and public narratives diverge from actual market performance**. Core idea: - Measure sentiment from financial news and social sources - Compare historical sentiment with historical stock returns - Compute alignment / misalignment metrics (e.g. 7-day window) - Surface when narratives are confidently wrong This is a **hackathon project**. Speed, clarity, and demo reliability matter more than over-engineering. --- ## Tech Stack (Locked) - Frontend: **Next.js (TypeScript)** in `web/` - Backend API: **FastAPI (Python)** in `api/` - Background jobs / ML: **Python** in `jobs/` - Database: **Supabase Postgres (hosted)** - ML: - Hugging Face transformers for sentiment (financial model) - Gemini ONLY for explanation / topic tagging (not core sentiment) - Deployment: - Frontend: Vercel - Backend: Render / Fly.io - Jobs: GitHub Actions (cron) ❌ Docker is intentionally NOT used. --- ## Architecture Rules (Very Important) 1. **NO ML inference in API request handlers** - All sentiment scoring and heavy computation runs in `jobs/` - API endpoints only read precomputed data from the database 2. **Frontend never scrapes or computes** - Frontend only fetches from API (or read-only Supabase views) 3. **Supabase is the single source of truth** - No local databases - No in-memory state relied on for correctness 4. **Historical data = Hugging Face sentiment** - Gemini is NOT used to label historical sentiment - Gemini is allowed only for: - explanations - topic clustering - relevance filtering --- ## Repository Structure (Do Not Change Lightly) - `web/` → frontend only (TypeScript, React) - `api/` → FastAPI backend (Python) - `jobs/` → ingestion, sentiment scoring, aggregation, metrics - `.github/workflows/` → cron jobs only Claude should never mix responsibilities across these boundaries. --- ## Coding Guidelines ### General - Prefer **simple, explicit code** - Avoid premature abstractions - Avoid cleverness - Favor readability over optimization ### Python - Use type hints when reasonable - Use Pydantic models for API schemas - Keep functions small and single-purpose - Pandas is allowed for batch jobs, not API requests ### TypeScript - Use strict typing - Define shared API response types in `web/lib/types.ts` - Do not inline large fetch logic inside components --- ## Data & Schema Rules - Sentiment must be normalized to a numeric score in range **[-1, +1]** - Labels must be one of: `POSITIVE`, `NEUTRAL`, `NEGATIVE` - Rolling metrics (7d, 14d, etc.) should be **precomputed**, not computed on the fly - Database tables are append-only where possible (items, item_scores) Claude should not redesign the schema unless explicitly asked. --- ## What Claude SHOULD Do - Create files and folders w [truncated — 1004 more characters] ``` ### sentiment-reality/api/requirements.txt ``` fastapi uvicorn psycopg2-binary python-dotenv pydantic ``` ### sentiment-reality/jobs/requirements.txt ``` psycopg2-binary python-dotenv python-dateutil yfinance transformers torch pandas numpy requests newspaper3k lxml lxml_html_clean feedparser nltk ``` ### sentiment-reality/web/package.json ``` { "name": "sentiment-reality-web", "version": "0.1.0", "private": true, "scripts": { "dev": "next dev", "build": "next build", "start": "next start", "lint": "next lint" }, "dependencies": { "@emotion/react": "^11.14.0", "@emotion/styled": "^11.14.1", "@mui/material": "^7.3.7", "@mui/material-nextjs": "^7.3.7", "next": "^14.2.0", "react": "^18.2.0", "react-dom": "^18.2.0", "recharts": "^2.12.0" }, "devDependencies": { "@types/node": "^20.0.0", "@types/react": "^18.2.0", "@types/react-dom": "^18.2.0", "autoprefixer": "^10.4.0", "postcss": "^8.4.0", "tailwindcss": "^3.4.0", "typescript": "^5.0.0" } } ``` ### sentiment-reality/api/main.py ```python """Sentiment Reality API - FastAPI application.""" from fastapi import FastAPI from fastapi.middleware.cors import CORSMiddleware from routers import health, dashboard, stocks, headlines app = FastAPI(title="Sentiment Reality API") # Allow frontend to connect app.add_middleware( CORSMiddleware, allow_origins=["http://localhost:3000"], allow_methods=["*"], allow_headers=["*"], ) # Include routers app.include_router(health.router) app.include_router(dashboard.router) app.include_router(stocks.router) app.include_router(headlines.router) ``` ### sentiment-reality/web/app/layout.tsx ```typescript import type { Metadata } from 'next' import '@/styles/globals.css' import MuiProviders from '@/theme/MuiProviders' export const metadata: Metadata = { title: 'Sentiment Reality | Market Sentiment Analysis', description: 'Analyze when market sentiment diverges from actual performance', } export default function RootLayout({ children, }: { children: React.ReactNode }) { return (
{children}
 ) } ``` ### sentiment-reality/web/app/page.tsx ```typescript 'use client' import { useCallback, useEffect, useMemo, useState } from 'react' import { Alert, Box, Button, Stack, Typography } from '@mui/material' import AppShell from '@/components/layout/AppShell' import WatchlistSelector from '@/components/panels/WatchlistSelector' import NarrativeReliabilityPanel from '@/components/panels/NarrativeReliabilityPanel' import MisalignmentDaysPanel from '@/components/panels/MisalignmentDaysPanel' import WatchlistPanel from '@/components/panels/WatchlistPanel' import HeadlinesPanel from '@/components/panels/HeadlinesPanel' import AddStockModal from '@/components/modals/AddStockModal' import HeadlineDetailsModal from '@/components/modals/HeadlineDetailsModal' import MisalignmentMapModal from '@/components/modals/MisalignmentMapModal' import HeadlinesForDateModal from '@/components/modals/HeadlinesForDateModal' import { getDashboard, getStocks, refreshStock } from '@/lib/api' import type { DashboardData, NewsItem, Stock } from '@/lib/types' export default function Home() { const [stocks, setStocks] = useState([]) const [selectedTicker, setSelectedTicker] = useState(null) const [period, setPeriod] = useState(30) const [data, setData] = useState(null) const [isLoading, setIsLoading] = useState(false) const [isRefreshing, setIsRefreshing] = useState(false) const [error, setError] = useState(null) const [isAddStockOpen, setIsAddStockOpen] = useState(false) const [isMisalignmentOpen, setIsMisalignmentOpen] = useState(false) const [selectedHeadline, setSelectedHeadline] = useState(null) const [selectedMisalignmentDate, setSelectedMisalignmentDate] = useState(null) // Get active tickers from stocks const tickers = useMemo(() => { return stocks.filter((s) => s.is_active).map((s) => s.ticker).sort() }, [stocks]) const lastUpdated = useMemo(() => { const value = data?.daily_data.at(-1)?.date if (!value) return null const parsed = new Date(value) if (Number.isNaN(parsed.getTime())) return value return new Intl.DateTimeFormat('en-US', { month: 'short', day: 'numeric', year: 'numeric' }).format(parsed) }, [data]) // Fetch stocks on mount useEffect(() => { let isMounted = true const fetchStockList = async () => { try { const stockList = await getStocks() if (isMounted && stockList.length > 0) { setStocks(stockList) const activeTickers = stockList.filter((s) => s.is_active).map((s) => s.ticker) if (activeTickers.length > 0 && !selectedTicker) { setSelectedTicker(activeTickers[0]) } } } catch (err) { console.error('Failed to fetch stocks:', err) } } fetchStockList() return () => { isMounted = false } }, []) // Fetch dashboard when ticker or period changes const fetchDashboard = useCallback(async () => { if (!selectedTicker) return setIsLoading(true) setError(null) try { const response = await getDashboard(selectedTicker, period) setData(response) } catch (err) { setError(err instanceof Error ? err.message : 'Failed to load dashboard.') } finally { setIsLoading(false) } }, [selectedTicker, period]) useEffect(() => { fetchDashboard() }, [fetchDashboard]) const handleRefresh = async () => { if (!selectedTicker) return setIsRefreshing(true) setError(null) try { await refreshStock(selectedTicker) // Re-fetch after 10 seconds to allow backend to process setTimeout(() => { fetchDashboard() setIsRefreshing(false) }, 10000) } catch (err) { setError(err instanceof Error ? err.message : 'Failed to refresh ticker.') setIsRefreshing(false) } } const coverage = data?.coverage const hasCoverageGap = coverage && coverage.sentiment_period_used < coverage.sentiment_period_requested return ( <> setIsAddStockOpen(true)} lastUpdated={lastUpdated} > {error && {error}} {isRefreshing && ( Refreshing data... This may take a few seconds. )} {!data && !isLoading && !error && !isRefreshing && ( No data yet. Click Refresh. )} {hasCoverageGap && ( We measure narrative alignment only where we have narrative data. Sentiment coverage: {coverage.sentiment_days_available} / {coverage.sentiment_period_requested} days )} Narrative vs. Performance Find false conviction. Where headlines are confident and price action disagrees.