# Project export: Synthesis This document was generated by HackStack to give an AI agent context about a hackathon project. Sections are labeled with their provenance; content marked as truncated was cut to keep this document small. ## Project metadata - Hackathon: TreeHacks 2024 - Tagline: News that promotes personalization and prevents polarization. - Devpost: https://devpost.com/software/synthesis-ag1djb - GitHub: https://github.com/ronnachum11/synthesis.git - Demo: https://synthe.vercel.app/ - Video: https://www.youtube.com/embed/n6hA0DIRNjw?enablejsapi=1&hl=en_US&rel=0&start=&version=3&wmode=transparent - Result: winner (Coframe: Best AI Hack (NVIDIA RTX 4080 GPU)) - Team: 3 GitHub contributor(s) — ronnachum11 (30 commits), Pranav Ramesh (8 commits), Christopher Arraya (8 commits) ## Devpost submission (written by the team) ### Overview Political polarization, now more than ever, is moving at an alarming rate. Our team has worked in the forefront of technology, policy, political science, and sociology. Christopher Arraya is a computational political polarization researcher at UNC-Chapel Hill, studying the Bolivian coup in 2019. He also works as an AI research advisor for multiple policy-tech nonprofits. Pranav Ramesh is an emerging entrepreneur interested in applying technology to social good who has led canvassing for several political campaigns. Ron Nachum has led several emerging technology research initiatives at Harvard analyzing effects of new technologies and how to mitigate/ensure positive outcomes–from AI, to investments, tech, and even space. The current media landscape, fueled by political polarization and algorithmic echo chambers, fuels division and hinders understanding. Synthesis was born from the desire to create a news platform that prioritizes unbiased information, facilitates diverse perspectives, and puts the news more in the hands of the readers through hyper-personalization. In the Synthesis, our users don’t just want to read about the world. They want to engage in it. Think of Synthesis as a new-age of information digestion. Synthesis takes existing news articles around the internet, clusters them by topic similarity, and reshapes them into unbiased articles, equipped with enhanced readability and quality of life features that make it easier for readers to resolve any areas of confusion immediately. Search up any topic you want to learn about, and you will be presented with a list of “Gists,” which are quick blurbs. Through these readers, can decide whether or not to learn more, at which point they can dive into the “Synthesis” of the topic, or a deeper dive into the subject matter. How it works Synthesis is built on the following features: Scale: Synthesis aggregates news daily from 60K+ articles across 100+ sources. Clustering: We’ve developed the first AI News Agent that intelligently extracts and clusters using a combination of vector databases and set theory, and understands news to increase knowledge. Gists: Synthesis clusters are summarized into at-a -glance overviews of topics via our rich content extraction. Syntheses: Should the user choose to engage with a news topic past the Gist, these AI-generated analyses allow for the connection and interpretation of similar-topic articles. Recommender Algorithm: Recommendations regularize/balance against echo chambers. Info Traversal: Within-topic and global semantic search Hyper-personalization: Within syntheses, users can dynamically adjust the reading level of the content. Extensions: Users can engage with our Q&A and content explanation extensions. Rich Extraction: We adopt visuals and tooling for our high-fidelity data. ### How we built it A core part of building out this project that was particularly difficult and stimulating was our agent backbone. We reasoned about how to manage tens of thousands of articles in terms of their sources, relationships to one another, significance, and more. We came up with a truly innovative, multi-step agent framework that scraped over 60,000 news articles during this hackathon and continuously scrapes 20+ articles per minute to keep our dataset constantly active. We developed a system of clusters that related articles to one another using textual extractions, semantic similarity, and other parameters of articles. These then enabled us to extract rich data from articles and produce more complex outputs and workflows, paving the way for endless new ways to interact with news. And the most amazing thing? This agent costs almost nothing to run. Our total costs throughout this hackathon to scrape tens of thousands of data points, vectorize and store them in a vector database and a Postgresql database to query efficiently and extract powerful information lies in the single digit dollars. Due to optimizations and algorithmic ingenuity, we are able to create a truly one-of-a-kind agent system that can scrape the entire internet’s wealth of news knowledge at virtually zero cost. Creating Synthesis was a very challenging but intellectually stimulating endeavor. We sought to solve an increasingly prevalent social problem using rapidly innovating technology. In developing this project, we learned to leverage diverse web frameworks and pushed the boundaries of what was capable with generative AI. We found out very quickly, for example, that there is a tradeoff between context limit and requerying with regard to LLMs: In Lost in the Middle: How Language Models Use Long Contexts (arXiv:2307.03172), Liu et al demonstrate that when given longer contexts, LLM performance degrade when models must reference information in the middle of such contexts. Feeding multiple articles into a single context for an LLM to interpret could therefore allow them to fall prey to this trap. On the other hand, providing news articles to LLMs to read in multiple queries could prove to be costly as the state-of-the-art LLMs, such as GPT-4, can potentially be expensive. We were able to solve this issue using an ensemble of methods approach where we had our agent specialize in different contexts at different stages of our scraping pipeline. ## README (from the GitHub repository) # Synthesis: The Next Generation News App Combating Misinformation and Echo Chambers in the Age of Social Media and AI. ## We built out a universal AI agent framework to constantly scrape and categorize all the newest news, building a custom feedback loop, novel clustering algorithm, and a new approach to recommendations that prevent polarization. We analyze over 60,000 articles (scraping an additional 20+ every single minute) and cluster algorithms using a novel topic relationship model using embedding relationships and a Union-Find structure for relationships. We then build out key features off of this foundation to change the way people interact with news with the goal of preventing spread of misinformation, falling into echo chambers or extreme polarization. Some cool key features include: * Scale: News aggregated from 60K+ articles daily (100+ sources) * Clustering: Novel system for grouping to increase knowledge * Gists: At a glance overviews of topics via rich content extraction * Syntheses: Unbiased aggregation of many similar-topic articles * Algorithm: Recommendations regularize/balance against echo * Info Traversal: Within-topic and global semantic search * Hyperpersonalization: Dynamic reading level adjustment * Extensions: Question and answering, content explanations * Rich Extraction: High-fidelity data for visuals and tooling ![agent-flow-chart](flowchart.png) Built by Christopher Arraya, Ron Nachum, & Pranav Ramesh. ## Detected evidence (automated analysis) Indexed codebase: 95 recognized source files, 446 KB. - CSS (language) — detected in the code - JavaScript (language) — detected in the code - Next.js (technology) — detected in the code - OpenAI (technology) — detected in the code - Python (language) — detected in the code - React (technology) — detected in the code - Supabase (technology) — detected in the code - Tailwind CSS (technology) — detected in the code - TypeScript (language) — detected in the code - Vercel AI SDK (technology) — detected in the code ## Codebase structure (from repository index) ### Files (108 of 108) ``` .DS_Store .gitignore advertisements.json api.py clustering.py example_structured_data.json frontend/.eslintrc.json frontend/.gitignore frontend/app/api/chat/route.ts frontend/app/auth/confirm/route.ts frontend/app/error/page.tsx frontend/app/globals.css frontend/app/layout.tsx frontend/app/login/page.tsx frontend/app/news/[currentClusterID]/page.tsx frontend/app/news/page.tsx frontend/app/page.tsx frontend/app/private/page.tsx frontend/app/results/page.tsx frontend/app/signup/page.tsx frontend/components.json frontend/components/actor-network.tsx frontend/components/answer-box.tsx frontend/components/gist.tsx frontend/components/globe.tsx frontend/components/login-form.tsx frontend/components/logo.tsx frontend/components/logout.tsx frontend/components/mode-toggle.tsx frontend/components/navbar.tsx frontend/components/reaction-buttons.tsx frontend/components/search-bar.tsx frontend/components/signup-form.tsx frontend/components/sources.tsx frontend/components/synthesis.tsx frontend/components/theme-provider.tsx frontend/components/ui/accordion.tsx frontend/components/ui/alert-dialog.tsx frontend/components/ui/alert.tsx frontend/components/ui/aspect-ratio.tsx frontend/components/ui/avatar.tsx frontend/components/ui/badge.tsx frontend/components/ui/button.tsx frontend/components/ui/calendar.tsx frontend/components/ui/card.tsx frontend/components/ui/carousel.tsx frontend/components/ui/checkbox.tsx frontend/components/ui/collapsible.tsx frontend/components/ui/command.tsx frontend/components/ui/context-menu.tsx frontend/components/ui/dialog.tsx frontend/components/ui/drawer.tsx frontend/components/ui/dropdown-menu.tsx frontend/components/ui/form.tsx frontend/components/ui/hover-card.tsx frontend/components/ui/input.tsx frontend/components/ui/label.tsx frontend/components/ui/menubar.tsx frontend/components/ui/navigation-menu.tsx frontend/components/ui/pagination.tsx frontend/components/ui/popover.tsx frontend/components/ui/progress.tsx frontend/components/ui/radio-group.tsx frontend/components/ui/resizable.tsx frontend/components/ui/scroll-area.tsx frontend/components/ui/select.tsx frontend/components/ui/separator.tsx frontend/components/ui/sheet.tsx frontend/components/ui/skeleton.tsx frontend/components/ui/slider.tsx frontend/components/ui/sonner.tsx frontend/components/ui/switch.tsx frontend/components/ui/table.tsx frontend/components/ui/tabs.tsx frontend/components/ui/textarea.tsx frontend/components/ui/toast.tsx frontend/components/ui/toaster.tsx frontend/components/ui/toggle-group.tsx frontend/components/ui/toggle.tsx frontend/components/ui/tooltip.tsx frontend/components/ui/use-toast.ts frontend/lib/actions/articles.ts frontend/lib/actions/auth.ts frontend/lib/actions/clusters.ts frontend/lib/actions/search.ts frontend/lib/actions/synthesis.ts frontend/lib/utils.ts frontend/middleware.ts frontend/next.config.mjs frontend/package.json frontend/postcss.config.js frontend/README.md frontend/tailwind.config.ts frontend/tsconfig.json frontend/utils/supabase/client.ts frontend/utils/supabase/server.ts links.txt models.py official_api.py push_to_db.py README.md scrape.py semantic.py sites.py sites.txt small_vectors.json summarize.py websites.txt ``` ### Dependencies - frontend/package.json: @hookform/resolvers@^3.3.4, @radix-ui/react-accordion@^1.1.2, @radix-ui/react-alert-dialog@^1.0.5, @radix-ui/react-aspect-ratio@^1.0.3, @radix-ui/react-avatar@^1.0.4, @radix-ui/react-checkbox@^1.0.4, @radix-ui/react-collapsible@^1.0.3, @radix-ui/react-context-menu@^2.1.5, @radix-ui/react-dialog@^1.0.5, @radix-ui/react-dropdown-menu@^2.0.6, @radix-ui/react-hover-card@^1.0.7, @radix-ui/react-icons@^1.3.0, @radix-ui/react-label@^2.0.2, @radix-ui/react-menubar@^1.0.4, @radix-ui/react-navigation-menu@^1.1.4, @radix-ui/react-popover@^1.0.7, @radix-ui/react-progress@^1.0.3, @radix-ui/react-radio-group@^1.1.3, @radix-ui/react-scroll-area@^1.0.5, @radix-ui/react-select@^2.0.0, @radix-ui/react-separator@^1.0.3, @radix-ui/react-slider@^1.1.2, @radix-ui/react-slot@^1.0.2, @radix-ui/react-switch@^1.0.3, @radix-ui/react-tabs@^1.0.4, @radix-ui/react-toast@^1.1.5, @radix-ui/react-toggle@^1.0.3, @radix-ui/react-toggle-group@^1.0.4, @radix-ui/react-tooltip@^1.0.7, @supabase/ssr@^0.1.0, @supabase/supabase-js@^2.39.6, @types/node@^20, @types/node-fetch@^2.6.11, @types/react@^18, @types/react-dom@^18, ai@^2.2.35, autoprefixer@^10.0.1, class-variance-authority@^0.7.0, clsx@^2.1.0, cmdk@^0.2.1, cobe@^0.6.3, date-fns@^3.3.1, embla-carousel-react@8.0.0-rc22, eslint@^8, eslint-config-next@14.1.0, fetch@^1.1.0, framer-motion@^11.0.5, graphology@^0.25.4, lucide-react@^0.331.0, next@14.1.0, next-themes@^0.2.1, node-fetch@^3.3.2, openai@^4.28.0, pinecone-client@^2.0.0, postcss@^8, react@^18, react-day-picker@^8.10.0, react-dom@^18, react-hook-form@^7.50.1, react-resizable-panels@^2.0.9, sigma@3.0.0-beta.6, sonner@^1.4.0, tailwind-merge@^2.2.1, tailwindcss@^3.3.0, tailwindcss-animate@^1.0.7, typescript@^5, vaul@^0.9.0, zod@^3.22.4 ### Recent commits (newest first) - Clean up sources cards - Final production adjustment - Fixing deployment bug - Fix linting errors, final push! - Try to fix eslint... again!? - Remove eslint - Remove gpt4all from pnpm lock - Remove gpt4all from package.json - Add access code - Fix search message - Merge pull request #3 from ronnachum11/rerefactored-frontend - Fix repeat articles - Merge pull request #2 from ronnachum11/rerefactored-frontend - fix merge conflicts - Clean up linking, home page, etc. - Merge branch 'rerefactored-frontend' of https://github.com/ronnachum11/treehacks into rerefactored-frontend - Fix title and icon - Merge branch 'rerefactored-frontend' of github.com:ronnachum11/treehacks into rerefactored-frontend - [PR] Landing page - Add source, need to check specifics and improve styling ## Key source files (fetched from GitHub, selected and truncated for size) ### frontend/package.json ``` { "name": "frontend", "version": "0.1.0", "private": true, "scripts": { "dev": "next dev", "build": "next build", "start": "next start", "lint": "next lint" }, "dependencies": { "@hookform/resolvers": "^3.3.4", "@radix-ui/react-accordion": "^1.1.2", "@radix-ui/react-alert-dialog": "^1.0.5", "@radix-ui/react-aspect-ratio": "^1.0.3", "@radix-ui/react-avatar": "^1.0.4", "@radix-ui/react-checkbox": "^1.0.4", "@radix-ui/react-collapsible": "^1.0.3", "@radix-ui/react-context-menu": "^2.1.5", "@radix-ui/react-dialog": "^1.0.5", "@radix-ui/react-dropdown-menu": "^2.0.6", "@radix-ui/react-hover-card": "^1.0.7", "@radix-ui/react-icons": "^1.3.0", "@radix-ui/react-label": "^2.0.2", "@radix-ui/react-menubar": "^1.0.4", "@radix-ui/react-navigation-menu": "^1.1.4", "@radix-ui/react-popover": "^1.0.7", "@radix-ui/react-progress": "^1.0.3", "@radix-ui/react-radio-group": "^1.1.3", "@radix-ui/react-scroll-area": "^1.0.5", "@radix-ui/react-select": "^2.0.0", "@radix-ui/react-separator": "^1.0.3", "@radix-ui/react-slider": "^1.1.2", "@radix-ui/react-slot": "^1.0.2", "@radix-ui/react-switch": "^1.0.3", "@radix-ui/react-tabs": "^1.0.4", "@radix-ui/react-toast": "^1.1.5", "@radix-ui/react-toggle": "^1.0.3", "@radix-ui/react-toggle-group": "^1.0.4", "@radix-ui/react-tooltip": "^1.0.7", "@supabase/ssr": "^0.1.0", "@supabase/supabase-js": "^2.39.6", "ai": "^2.2.35", "class-variance-authority": "^0.7.0", "clsx": "^2.1.0", "cmdk": "^0.2.1", "cobe": "^0.6.3", "date-fns": "^3.3.1", "embla-carousel-react": "8.0.0-rc22", "fetch": "^1.1.0", "framer-motion": "^11.0.5", "graphology": "^0.25.4", "lucide-react": "^0.331.0", "next": "14.1.0", "next-themes": "^0.2.1", "node-fetch": "^3.3.2", "openai": "^4.28.0", "pinecone-client": "^2.0.0", "react": "^18", "react-day-picker": "^8.10.0", "react-dom": "^18", "react-hook-form": "^7.50.1", "react-resizable-panels": "^2.0.9", "sigma": "3.0.0-beta.6", "sonner": "^1.4.0", "tailwind-merge": "^2.2.1", "tailwindcss-animate": "^1.0.7", "vaul": "^0.9.0", "zod": "^3.22.4" }, "devDependencies": { "@types/node": "^20", "@types/node-fetch": "^2.6.11", "@types/react": "^18", "@types/react-dom": "^18", "autoprefixer": "^10.0.1", "eslint": "^8", "eslint-config-next": "14.1.0", "postcss": "^8", "tailwindcss": "^3.3.0", "typescript": "^5" } } ``` ### frontend/app/layout.tsx ```typescript import "./globals.css"; import { Inter as FontSans } from "next/font/google"; import { cn } from "@/lib/utils"; import { Navbar } from "@/components/navbar"; import { Toaster } from "@/components/ui/sonner"; import { ThemeProvider } from "@/components/theme-provider"; export const fontSans = FontSans({ subsets: ["latin"], variable: "--font-sans", }); export default function RootLayout({ children, }: Readonly<{ children: React.ReactNode; }>) { return ( Synthesis - News Reimagined {children} ); } ``` ### frontend/app/page.tsx ```typescript import Globe from "@/components/globe"; import Image from "next/image"; import Link from "next/link"; export default function Home() { return (
Synthesis
The Truly-Everything News Platform.
Landing page
Enjoy your news on your own terms.
Get the facts quickly.
Read the news at your own pace. Like something? Swipe right. Need something explained? Get the answer quickly with the help of our chat model.
Phone
Avoid Misinformation.
Our AI agent is trained to curate in-demand content for you in a reliable, unpolarized manner. Don't be fooled by fake news anymore.
Phone
); } ``` ### frontend/app/error/page.tsx ```typescript export default function ErrorPage() { return

Sorry, something went wrong

; } ``` ### frontend/app/news/page.tsx ```typescript 'use client' import { useEffect } from 'react'; import { useRouter } from 'next/navigation'; export default function Page() { const router = useRouter(); useEffect(() => { router.push('/news/15db7a48-eaca-42b8-b3f9-1e11cef77792'); }, [router]); return null; // Render nothing or a loading indicator while redirecting } ``` ### frontend/app/private/page.tsx ```typescript import { cookies } from "next/headers"; import { redirect } from "next/navigation"; import { createClient } from "@/utils/supabase/server"; export default async function PrivatePage() { const cookieStore = cookies(); const supabase = createClient(cookieStore); const { data, error } = await supabase.auth.getUser(); if (error || !data?.user) { redirect("/"); } return

Hello {data.user.email}

; } ``` ### frontend/utils/supabase/server.ts ```typescript import { createServerClient, type CookieOptions } from "@supabase/ssr"; import { type cookies } from "next/headers"; export function createClient(cookieStore: ReturnType) { return createServerClient( process.env.NEXT_PUBLIC_SUPABASE_URL!, process.env.NEXT_PUBLIC_SUPABASE_ANON_KEY!, { cookies: { get(name: string) { return cookieStore.get(name)?.value; }, set(name: string, value: string, options: CookieOptions) { try { cookieStore.set({ name, value, ...options }); } catch (error) { // The `set` method was called from a Server Component. // This can be ignored if you have middleware refreshing // user sessions. } }, remove(name: string, options: CookieOptions) { try { cookieStore.set({ name, value: "", ...options }); } catch (error) { // The `delete` method was called from a Server Component. // This can be ignored if you have middleware refreshing // user sessions. } }, }, } ); } ``` ### frontend/app/login/page.tsx ```typescript import Link from "next/link"; import { LoginForm } from "@/components/login-form"; export default function AuthenticationPage() { return (

Login to you're account.

Don' have an account?{" "} Signup here

By clicking continue, you agree to our{" "} Terms of Service {" "} and{" "} Privacy Policy .

); } ``` ### frontend/app/signup/page.tsx ```typescript import { Metadata } from "next"; import Image from "next/image"; import Link from "next/link"; import { cn } from "@/lib/utils"; import { buttonVariants } from "@/components/ui/button"; import { SignupForm } from "@/components/signup-form"; export default function AuthenticationPage() { return (

Create an account

Already have an account?{" "} Login here

By clicking continue, you agree to our{" "} Terms of Service {" "} and{" "} Privacy Policy .

); } ``` ### frontend/app/results/page.tsx ```typescript "use client"; import { Card, CardContent, CardHeader, CardTitle } from "@/components/ui/card"; import { useSearchParams } from "next/navigation"; import { useEffect, useState } from "react"; import { globalSearch } from "@/lib/actions/search"; import { getClusterById } from "@/lib/actions/clusters"; import Link from "next/link"; export default function SearchResults() { const searchParams = useSearchParams(); const [searchResults, setSearchResults] = useState([]); const [loading, setLoading] = useState(true); useEffect(() => { setLoading(true); async function performSearch(query: string) { const res = await globalSearch(query); return res; } const res = performSearch(String(searchParams.get("query"))); // map through res and then get the cluster by id, but still keep all the other data res.then((data) => { const clusterIds = data.map((result: any) => result.cluster_id); const clusters = clusterIds.map((id: string) => getClusterById(id)); Promise.all(clusters).then((clusterData) => { const results = data.map((result: any, index: number) => { return { ...result, title: clusterData[index].title, cluster_id: clusterData[index].id, }; }); setSearchResults(results); setLoading(false); }); }); }, [searchParams]); return (
{loading ? (
Loading...

Synthesizing semantic search results...

) : (
{searchResults.map((result: any) => { return (

{result.title}

{result.excerpt}

 ); })}
)}
); } ``` [83 more indexed source files omitted to keep this export small. The full file list is in the Codebase structure section above.]