# Project export: Kaleido This document was generated by HackStack to give an AI agent context about a hackathon project. Sections are labeled with their provenance; content marked as truncated was cut to keep this document small. ## Project metadata - Hackathon: TreeHacks 2024 - Tagline: Kaleido illuminates the colorful spectrum of bias in the news, like peering through a kaleidoscope. - Devpost: https://devpost.com/software/insight-critter - GitHub: https://github.com/austinchen12/insight - Video: https://www.youtube.com/embed/6J37C_T-aOg?enablejsapi=1&hl=en_US&rel=0&start=&version=3&wmode=transparent - Result: winner (InterSystems: Best Use of GenAI using InterSystems IRIS Vector Search ($2k Cash [1st] & $1.5k Cash [2nd] & $1k Cash [3rd])) - Team: 3 GitHub contributor(s) — Braden Wong (121 commits), Joe Zakielarz (27 commits), Austin Chen (25 commits) ## Devpost submission (written by the team) ### Inspiration In a world saturated with information and misinformation, the upcoming 2024 presidential election stands as a testament to our need for clarity and truth. This is where Kaleido steps in. Inspired by the vision of bringing light to the unseen, our mission goes beyond merely summarizing news content. We dive deeper, revealing not just what you're reading, but also what you're missing out on. In addition, by assessing bias and sentiment across related articles, Kaleido provides a fuller, more balanced view of every story. Our approach is grounded in the belief that understanding the full spectrum of information is crucial in navigating the complexities of today's world. Kaleido is not just a tool; it's a movement towards informed, critical thinking and a beacon for those who seek to understand beyond the surface. What Kaleido Does Kaleido is a Chrome extension designed to seamlessly integrate into your browsing and reading experience. Kaleido works in the background as you explore news articles, leveraging vector embedding search technology to analyze the content of your current article. After identifying similar articles discovered by other users through embeddings, Kaleido offers a unique comparative analysis of the article at hand. The core functionality of Kaleido is twofold: Comparative Analysis: Kaleido enables users to compare the bias and sentiment of the current article with a wide array of other similar articles identified by its vector embedding search. This feature allows for an in-depth understanding of where the article stands, in bias and sentiment, within a broader spectrum of perspectives and analyses. Comparative Analysis: Kaleido enables users to compare the bias and sentiment of the current article with a wide array of other similar articles identified by its vector embedding search. This feature allows for an in-depth understanding of where the article stands, in bias and sentiment, within a broader spectrum of perspectives and analyses. Idea Aggregation and Analysis: The second key feature distills the essence of an article into multiple points or ideas, and each point is then embedded in vector space. This process constructs a vast network or a "superset" of ideas shared across articles, identified as clusters of similar thoughts in the vector-embedded space. Through this approach of embedding points across articles, Kaleido surfaces significant, overarching takeaways from these groups of articles, offering a comprehensive view that goes beyond the surface level. Idea Aggregation and Analysis: The second key feature distills the essence of an article into multiple points or ideas, and each point is then embedded in vector space. This process constructs a vast network or a "superset" of ideas shared across articles, identified as clusters of similar thoughts in the vector-embedded space. Through this approach of embedding points across articles, Kaleido surfaces significant, overarching takeaways from these groups of articles, offering a comprehensive view that goes beyond the surface level. Additionally, Kaleido aggregates crucial insights by comparing the focal points of an article against others, enriching the user's understanding with enhanced data on embeddings, sentiment, and bias. This not only broadens the perspective of readers but also deepens their engagement with content, fostering a more informed and critical approach to information consumption. Why We Built Kaleido Since our team's convergence at TreeHacks, the driving force behind Kaleido has been a deep-seated interest in education and politics, with a spotlight on the impending 2024 election. This particular moment in time underscores the paramount importance of clarity, truth, and context in media consumption. Our decision to bring Kaleido to life stemmed from a desire to make a meaningful impact within the media space, guided by the principle that the quality of input fundamentally shapes the quality of output. In crafting Kaleido, our ambition was to elevate the caliber and context of the information we consume. We recognized that in an era where information is both weapon and tool, enhancing the input quality—by providing a more nuanced, comprehensive understanding of the news—can significantly influence our perceptions, decisions, and, ultimately, our societal output. Our hope with Kaleido was to optimize this input process, fostering a more informed, discerning, and engaged populace, especially in the context of critical events like elections, where the stakes are incredibly high. How We Built Kaleido From the start, we knew that building Kaleido would be a complex adventure with a lot of moving parts. In fact, the project ended up being particularly complicated, especially with the employment of multiple embeddings and clustering. However, although Kaleido embeddings seem particularly scary, we were very lucky to have sponsored technology that significantly streamlined our development process. Unified Database Solution: Perhaps one of the most significant achievements in our development process was the integration of a vector database that allowed us to consolidate our data storage needs into a single, versatile database, provided by InterSystems IRIS Database. This eliminated the necessity for multiple databases, streamlining our data management and enhancing the efficiency of our operations. InterSystem's multi-paradigm database capabilities allowed us to not only store vector information efficiently, but also to leverage SQL for our database needs. This dual use case was crucial for managing the vector data for our embedding and clustering functionalities. The ability for us to conduct vector embedding search and store data in the same database was an advantage whose importance could not be understated. Front-End and Back-End Synergy: The architectural foundation of Kaleido was a harmonious blend of front-end and back-end technologies. Our Chrome extension, developed using Plasma and React, was intricately designed to offer a user-friendly interface and a responsive user experience. The backend, powered by Bun, facilitated a REST API in a monorepo setup, enabling seamless type sharing with the front-end and ensuring a cohesive development environment. API Development: To facilitate seamless communication between our Chrome extension and the backend, we opted for Bun with Elysia. Elysia is a highly ergonomic web framework, with an API like express, for building backend servers, but specifically designed to run in the Bun runtime. While Bun, unfortunately, does not fully support a Turborepo monorepo setup, we actually were able to still use Bun for everything else (because Bun is both a package manager and runtime). By using pnpm as a package manager, we realized we could still use Bun as a runtime. This choice proved instrumental in developing a high-performance, type-safe API. Elysia, being a drop-in replacement for Express, offered us the ease of building our API with enhanced efficiency and reliability, ensuring our backend infrastructure was robust and scalable. ### Challenges we ran into From our initial ideation, we realized quickly that our application was nowhere near as simple as we initially thought and we brainstormed a lot to think and address these complexities before coding. One significant hurdle was the process of acquiring and scraping articles, a task made increasingly difficult by the recent rise of anti-scraping measures implemented by many websites. However, the true test of our resolve was the spontaneous decision to incorporate clustering into Kaleido. This idea, born from the brainstorming sessions aimed at addressing our project's growing complexity, introduced a new level of challenge since nobody on our team had any familiarity with how to do it. Achieving proficiency required not just technical acumen but a willingness to dive into uncharted waters. ### Accomplishments we're proud of Similar to the previous sections about the challenges we ran into, we're quite proud of the challenges that we have overcome. Particularly, during our brainstorming session, we went from a half-baked idea with very little development into one which had many moving parts and complexity, and all thanks to a few hours of brainstorming that we set aside at the very beginning of the weekend. In addition, we are particularly happy about how we had spontaneously stumbled upon the idea of clustering—until then, nobody in our group had done it before, but it just felt like the right word describing what we wanted to do. Finally, we're very happy with how the logo and the user interface turned out. It turned out to be far better than we ever would have imagined. ### What we learned The importance of brainstorming and thinking things through before implementation. Although our team was formed relatively late, we were able to avoid a lot of bumps by taking an hour at the beginning to think and brainstorm and concretely write down our implementations before coding them. Additionally, we really enjoyed engaging with new technologies and learning how to cluster! Conclusion Developing Kaleido was a pleasure, and our team really enjoyed working together, learning new technologies, and making new friends. Now, we hope you will join us in our quest to illuminate the unseen, enhance your reading experience, and equip you with the insights needed to face the misinformation challenge head-on. Together, let's create a more informed society ready to make educated decisions for the future. ## README (from the GitHub repository) # Kaleido 🦎 ## Inspiration ✨ In a world saturated with information and misinformation, the upcoming 2024 presidential election stands as a testament to our need for clarity and truth. This is where Kaleido steps in. Inspired by the vision of bringing light to the unseen, our mission goes beyond merely summarizing news content. We dive deeper, revealing not just what you're reading, but also what you're missing out on. Additionally, by assessing bias and sentiment across related articles, Kaleido provides a fuller, more balanced view of every story 📚. Our approach is grounded in the belief that understanding the full spectrum of information is crucial in navigating the complexities of today's world. Kaleido is not just a tool; it's a movement towards informed, critical thinking and a beacon for those who seek to understand beyond the surface 🚀. ## What Kaleido Does 🛠️ Kaleido is a Chrome extension designed to seamlessly integrate into your browsing and reading experience. It works in the background as you explore news articles, leveraging vector embedding search technology to analyze the content of your current article. After identifying similar articles discovered by other users through embeddings, Kaleido offers a unique comparative analysis of the article at hand. The core functionality of Kaleido is twofold: 1. **Comparative Analysis**: Kaleido enables users to compare the bias and sentiment of the current article with a wide array of other similar articles identified by its vector embedding search. This feature allows for an in-depth understanding of where the article stands, in bias and sentiment, within a broader spectrum of perspectives and analyses 📊. 2. **Idea Aggregation and Analysis**: The second key feature distills the essence of an article into multiple points or ideas, and each point is then embedded in vector space. This process constructs a vast network or a "superset" of ideas shared across articles, identified as clusters of similar thoughts in the vector-embedded space. Through this approach, Kaleido surfaces significant, overarching takeaways from these groups of articles, offering a comprehensive view that goes beyond the surface level 🔍. Additionally, Kaleido aggregates crucial insights by comparing the focal points of an article against others, enriching the user's understanding with enhanced data on embeddings, sentiment, and bias. This not only broadens the perspective of readers but also deepens their engagement with content, fostering a more informed and critical approach to information consumption 🌐. ## Why We Built Kaleido 🏗️ Since our team's convergence at TreeHacks, the driving force behind Kaleido has been a deep-seated interest in education and politics, with a spotlight on the impending 2024 election. This particular moment in time underscores the paramount importance of clarity, truth, and context in media consumption. Our decision to bring Kaleido to life stemmed from a desire to make a meaningful impact within the media space, guided by the principle that the quality of input fundamentally shapes the quality of output 🌟. In crafting Kaleido, our ambition was to elevate the caliber and context of the information we consume. We recognized that in an era where information is both weapon and tool, enhancing the input quality—by providing a more nuanced, comprehensive understanding of the news—can significantly influence our perceptions, decisions, and, ultimately, our societal output. Our hope with Kaleido was to optimize this input process, fostering a more informed, discerning, and engaged populace, especially in the context of critical events like elections, where the stakes are incredibly high 🗳️. ## How We Built Kaleido 🛠️ From the start, we knew that building Kaleido would be a complex adventure with a lot of moving parts. The project ended up being particularly complicated, especially with the employment of multiple embeddings and clustering. However, although Kaleido embeddings seem particularly scary, we were very lucky to have sponsored technology that significantly streamlined our development process 🤖. **Unified Database Solution**: Perhaps one of the most significant achievements in our development process was the integration of a vector database that allowed us to consolidate our data storage needs into a single, versatile database, provided by InterSystems IRIS Database. This eliminated the necessity for multiple databases, streamlining our data management and enhancing the efficiency of our operations 📈. **Front-End and Back-End Synergy**: The architectural foundation of Kaleido was a harmonious blend of front-end and back-end technologies. Our Chrome extension, developed using Plasma and React, was intricately designed to offer a user-friendly interface and a responsive user experience. The backend, powered by Bun, facilitated a REST API in a monorepo setup, enabling seamless type sharing with the front-end and ensuring a cohesive development environment 💻. **API Development**: To facilitate seamless communication between our Chrome extension and the backend, we opted for Bun with Elysia. Elysia is a highly ergonomic web framework, with an API like express, for building backend servers, but specifically designed to run in the Bun runtime 🌍. ## Challenges we ran into 🚧 From our initial ideation, we realized quickly that our application was nowhere near as simple as we initially thought and we brainstormed a lot to think and address these complexities before coding. One significant hurdle was the process of acquiring and scraping articles, a task made increasingly difficult by the recent rise of anti-scraping measures implemented by many websites 🛑. However, the true test of our resolve was the spontaneous decision to incorporate clustering into Kaleido. This idea, born from the brainstorming sessions aimed at addressing our project's growing complexity, introduced a new level of challenge since nobody on our team had any familiarity with how to do it. Achieving proficiency required not just technical acumen but a willingness to dive into uncharted waters 🌊. ## Accomplishments that we're proud of 🏆 Similar to the previous sections about the challenges we ran into, we're quite proud of the challenges that we have overcome. Particularly, during our brainstorming session, we went from a half-baked idea with very little development into one which had many moving parts and complexity, and all thanks to a few hours of brainstorming that we set aside at the very beginning of the weekend. In addition, we are particularly happy about how we had spontaneously stumbled upon the idea of clustering—until then, nobody in our group had done it before, but it just felt like the right word describing what we wanted to do. Finally, we're very happy with how the logo and the user interface turned out. It turned out to be far better than we ever would have imagined 🎨. ## What we learned 📚 The importance of brainstorming and thinking things through before implementation. Although our team was formed relatively late, we were able to avoid a lot of bumps by taking an hour at the beginning to think and brainstorm and concretely write down our implementations before coding them. Additionally, we really enjoyed engaging with new technologies and learning how to cluster! 🌟 ## Conclusion 🎉 Developing Kaleido was a pleasure, and our team really enjoyed working together, learning new technologies, and making new friends. Now, we hope you will join us in our quest to illuminate the unseen, enhance your reading experience, and equip you with the insights needed to face the misinformation challenge head-on. Together, let's create a more informed society ready to make educated decisions for the future 🌍. ## Creators 👨‍💻 - **Austin Chen** - **Braden Wong** - **Joe Zakielarz** ## Detected evidence (automated analysis) Indexed codebase: 79 recognized source files, 653 KB. - CSS (language) — detected in the code - HTML (language) — detected in the code - JavaScript (language) — detected in the code - Python (language) — detected in the code - React (technology) — detected in the code - SQL (language) — detected in the code - Tailwind CSS (technology) — detected in the code - TypeScript (language) — detected in the code ## Codebase structure (from repository index) ### Files (94 of 94) ``` .gitignore .prettierrc apps/backend/.gitignore apps/backend/drizzle.config.ts apps/backend/drizzle/0000_gorgeous_wolf_cub.sql apps/backend/drizzle/meta/_journal.json apps/backend/drizzle/meta/0000_snapshot.json apps/backend/package.json apps/backend/README.md apps/backend/src/db/index.ts apps/backend/src/db/schema.ts apps/backend/src/index.ts apps/backend/tsconfig.json apps/extension/.github/workflows/submit.yml apps/extension/.gitignore apps/extension/.prettierrc.mjs apps/extension/components.json apps/extension/components/Summary.tsx apps/extension/components/Topics.tsx apps/extension/components/ui/accordion.tsx apps/extension/components/ui/button.tsx apps/extension/components/ui/card.tsx apps/extension/components/ui/collapsible.tsx apps/extension/components/ui/progress.tsx apps/extension/components/ui/skeleton.tsx apps/extension/lib/eden.ts apps/extension/lib/utils.ts apps/extension/package.json apps/extension/popup.tsx apps/extension/postcss.config.js apps/extension/README.md apps/extension/style.css apps/extension/tailwind.config.js apps/extension/tsconfig.json articles/Carlson Putin/Alexei Navalny’s death underlines the horrors of Tucker Carlson’s Putin interview..md articles/Carlson Putin/Analysis After Tucker Carlson’s softball interview, Putin walks away with propaganda victory CNN Business.md articles/Carlson Putin/BORIS JOHNSON Putin's interview with his fawning stooge Tucker Carlson was straight out of Hitler's playbook. I pray Americans see through this unholy charade Daily Mail Online.md articles/Carlson Putin/Putin complains about lack of piercing questions from Tucker Carlson Reuters.md articles/Carlson Putin/Putin Roasts Tucker Carlson Even More After That Pathetic Interview The New Republic.md articles/Carlson Putin/Senate Republican rips Tucker Carlson over Russia grocery store trip The Hill.md articles/Carlson Putin/Tucker Carlson ‘No decent person would defend’ what happened to Navalny The Hill.md articles/Carlson Putin/Tucker Carlson criticised for praising Putin before Navalny death ‘Leadership requires killing people’ The Independent.md articles/Carlson Putin/Tucker Carlson interview Fact-checking Putin's 'nonsense' history.md articles/Carlson Putin/Tucker Carlson Promised an Unedited Putin. The Result Was Boring The New Yorker.md articles/Carlson Putin/Tucker Carlson releases video of interview with Russian leader Putin - The Washington Post.md articles/Carlson Putin/Tucker Carlson’s Lesson in the Perils of Giving Airtime to Vladimir Putin - The New York Times.md articles/George Santos Lawsuit/Former Rep. George Santos sues Jimmy Kimmel, accuses him of misusing Cameo videos.md articles/George Santos Lawsuit/George Santos Files Federal Lawsuit Against Jimmy Kimmel.md articles/George Santos Lawsuit/George Santos Is Suing His Cameo Customer Jimmy Kimmel.md articles/George Santos Lawsuit/George Santos is suing Jimmy Kimmel for buying Cameos from him under false pretenses.md articles/George Santos Lawsuit/George Santos Sues Jimmy Kimmel For Buying Cameo Prank Videos.md articles/George Santos Lawsuit/George Santos Sues Jimmy Kimmel for Pranking Him on Cameo.md articles/George Santos Lawsuit/George Santos sues Jimmy Kimmel for soliciting, broadcasting Cameo videos Fox News.md articles/George Santos Lawsuit/George Santos Sues Jimmy Kimmel For Using Cameo Videos On Show.md articles/George Santos Lawsuit/George Santos sues Jimmy Kimmel over Cameo video pranks AP News.md articles/George Santos Lawsuit/George Santos sues Jimmy Kimmel, alleges fraud over Cameo clips.md articles/George Santos Lawsuit/George Santos Sues Late-Night Host Jimmy Kimmel for Tricking Him Into Making Videos to Ridicule Him.md articles/George Santos Lawsuit/Lyin' expelled ex-Rep. George Santos makes Jimmy Kimmel's 'wishes come true' by suing host over misusing Cameo clips.md articles/George Santos Lawsuit/Ousted Rep. George Santos Sues Jimmy Kimmel Over Cameo Video Prank HuffPost Entertainment.md articles/https:__www.cnn.com_2024_02_16_us_kansas-city-chiefs-rally-shooting-saturday_index.html articles/https:__www.foxnews.com_us_kansas-city-shooting-chiefs-super-bowl-parade-leaves-dead-wounded articles/https:__www.nytimes.com_2024_02_17_us_super-bowl-kansas-city-shooting.html articles/Sam Altman 7 Trillion/Nvidia Founder Jensen Huang Dismisses $7 Trillion AI Investment Figure Floated by OpenAI's Sam Altman Inc.com.md articles/Sam Altman 7 Trillion/OpenAI CEO Sam Altman Reportedly Pitches $7 Trillion AI Funding Project PYMNTS.com.md articles/Sam Altman 7 Trillion/OpenAI CEO Sam Altman reportedly seeks trillions of dollars for AI chip project.md articles/Sam Altman 7 Trillion/OpenAI CEO Sam Altman seeks $5 to $7 trillion to build a network of fabs for AI chips Tom's Hardware.md articles/Sam Altman 7 Trillion/Report Sam Altman seeking trillions for AI chip fabrication from UAE, others Ars Technica.md articles/Sam Altman 7 Trillion/Sam Altman Sam Altman eyes semiconductor industry, in talks to raise nearly $7 trillion to boost AI chip production - The Economic Times.md articles/Sam Altman 7 Trillion/Sam Altman Wants US Backing for His $7 Trillion AI Chip Venture Report.md articles/Sam Altman 7 Trillion/Sam Altman's $7 Trillion 'Moonshot' - The Journal. - WSJ Podcasts.md articles/Sam Altman 7 Trillion/Sam Altman's $7 trillion AI chip dream has him rounding on critics 'You can grind to help secure our collective future or you can write Substacks about why we are going to fail'.md articles/Sam Altman 7 Trillion/Sam Altman's $7 Trillion Chip Dreams Way Off Mark Nvidia's Jensen Huang.md articles/Sam Altman 7 Trillion/The latest rumor about Sam Altman’s AI chip-building dream could require up to $7 trillion. - The Verge.md articles/Sam Altman 7 Trillion/What Sam Altman's chimerical trillions say about AI hype.md articles/Trump Election Fraud/‘Literally off his rocker’ Why Trump is fixated on Indiana - POLITICO.md articles/Trump Election Fraud/Challenges filed to keep Biden, Trump off Indiana ballot wthr.com.md articles/Trump Election Fraud/Election 2024 Trump blasts decision not to charge Biden in documents case, citing ‘selective persecution.’ - The New York Times.md articles/Trump Election Fraud/Former President Trump faces ballot challenge in Indiana.md articles/Trump Election Fraud/Indiana presidential election Jan. 6 cited in Trump ballot challenge.md articles/Trump Election Fraud/Nikki Haley makes Indiana ballot, but others at risk - State Affairs.md articles/Trump Election Fraud/Presidential candidate Nikki Haley to appear on Indiana Republican primary ballot.md articles/Trump Election Fraud/Supreme Court skeptical of efforts to kick Trump off the ballot AP News.md articles/Trump Election Fraud/Supreme Court will decide if Trump can be kept off 2024 presidential ballots AP News.md articles/Trump Election Fraud/Tracking State Efforts to Remove Trump From the 2024 Ballot - The New York Times.md articles/Trump Election Fraud/Trump tells supporters his $355 million fraud fine is election interference Reuters.md clustering.py iris_backend.py LICENSE.txt load_article.py package.json pnpm-workspace.yaml README.md turbo.json utils.py ``` ### Dependencies - apps/backend/package.json: @elysiajs/cors@^0.8.0, @elysiajs/swagger@^0.8.5, @libsql/client@^0.5.0, @sinclair/typebox@^0.32.14, bun-types@latest, drizzle-kit@^0.20.14, drizzle-orm@^0.29.3, drizzle-typebox@^0.1.1, elysia@latest, nanoid@^5.0.5, typescript@^5.3.3, zod@^3.22.4 - apps/extension/package.json: @elysiajs/eden@^0.8.1, @ianvs/prettier-plugin-sort-imports@^4.1.1, @radix-ui/react-accordion@^1.1.2, @radix-ui/react-collapsible@^1.0.3, @radix-ui/react-progress@^1.0.3, @radix-ui/react-slot@^1.0.2, @repo/backend@workspace:*, @types/chrome@^0.0.258, @types/node@^20.11.5, @types/react@^18.2.48, @types/react-dom@^18.2.18, axios@^1.6.7, class-variance-authority@^0.7.0, clsx@^2.1.0, lucide-react@^0.331.0, plasmo@^0.84.2, postcss@^8.4.35, prettier@^3.2.4, react@^18.2.0, react-dom@^18.2.0, react-icons@^5.0.1, recharts@^2.12.0, tailwind-merge@^2.2.1, tailwindcss@^3.4.1, tailwindcss-animate@^1.0.7, typescript@^5.3.3 - package.json: prettier@^3.2.5, turbo@^1.12.4 ### Recent commits (newest first) - Merge branch 'dev' of https://github.com/austinchen12/insight into dev - Create LICENSE.txt - chore: types - feat: Topic and such - fix: clicking and such - fix: bias and suhc - refactor: readability - fix: numerator and denominators - fix: coalescence - chore: temp fix on topics - Update README.md - feat: skeleton loaders - Create README.md - chore: layout contents - Merge pull request #17 from austinchen12/share-button - share button - feat: Skeleton - chore: try catch and errros - fix: ScannedArticle - Merge remote-tracking branch 'origin/dev' into dev ## Key source files (fetched from GitHub, selected and truncated for size) ### articles/Sam Altman 7 Trillion/The latest rumor about Sam Altman’s AI chip-building dream could require up to $7 trillion. - The Verge.md ```markdown The latest rumor about Sam Altman’s AI chip-building dream could require up to $7 trillion. For context, here’s how the _Wall Street Journal_ describes [OpenAI’s once-again leader’s](https://www.theverge.com/23966325/openai-sam-altman-fired-turmoil-chatgpt) trillion dollar effort to “reshape the global semiconductor industry:” > Such a sum of investment would dwarf the current size of the global semiconductor industry. Global sales of chips were $527 billion last year and are expected to rise to $1 trillion annually by 2030. The money is needed to fuel AI’s growth and solve the scarcity of expensive AI chips required to train the large language models that underpin systems like ChatGPT. According to the _WSJ_, Altman is pitching a chip-making partnership to investors from the UAE, SoftBank CEO Masayoshi Son ([again](https://www.theverge.com/2023/9/28/23893939/jony-ive-openai-sam-altman-iphone-of-artificial-intelligence-device)), and TSMC. ``` ### articles/Trump Election Fraud/Nikki Haley makes Indiana ballot, but others at risk - State Affairs.md ```markdown ![](https://stateaffairs.com/wp-content/uploads/2024/02/Rust-Haley-Mahant.png) Candidates John Rust, left; Nikki Haley, center; and Sid Mahant. (Credit: State Affairs, Haley and Mahant campaigns) At least 10 candidates for Congress or the Legislature face being taken off Indiana’s May primary ballot in the wake of a state Supreme Court order allowing enforcement of the [state’s two-primary voting standard](https://stateaffairs.com/indiana/elections/indiana-supreme-court-john-rust/) for Republican or Democratic candidates. The most prominent of those at risk is John Rust, the Republican U.S. Senate candidate who challenged the constitutionality of the law. Rust is the subject of several challenges filed with state election officials by Friday’s deadline. Also facing likely removal when the state Election Commission meets next week is a Democratic candidate for governor. Republican presidential candidate Nikki Haley will be listed on the Indiana ballot as no challenges were filed against her despite recent taunting from Donald Trump’s campaign that she wouldn’t gather the required petition signatures to qualify. ##### Read this story for free. Create Account ### Read this story for free Create your free account or login to continue reading. ``` ### package.json ``` { "name": "kaleido", "version": "1.0.0", "description": "", "scripts": { "build": "turbo build", "dev": "turbo dev", "lint": "turbo lint", "format": "prettier --write \"**/*.{ts,tsx,md}\"" }, "packageManager": "pnpm@8.15.3", "keywords": [], "author": "The Critters", "devDependencies": { "prettier": "^3.2.5", "turbo": "^1.12.4" } } ``` ### apps/backend/package.json ``` { "name": "@repo/backend", "version": "1.0.50", "exports": { ".": "./src/index.ts" }, "scripts": { "test": "echo \"Error: no test specified\" && exit 1", "dev": "bun run --watch src/index.ts" }, "dependencies": { "@elysiajs/cors": "^0.8.0", "@elysiajs/swagger": "^0.8.5", "@libsql/client": "^0.5.0", "@sinclair/typebox": "^0.32.14", "drizzle-orm": "^0.29.3", "drizzle-typebox": "^0.1.1", "elysia": "latest", "nanoid": "^5.0.5", "zod": "^3.22.4" }, "devDependencies": { "bun-types": "latest", "drizzle-kit": "^0.20.14", "typescript": "^5.3.3" }, "module": "src/index.js" } ``` ### apps/extension/package.json ``` { "name": "@repo/extension", "displayName": "Kaleido", "version": "0.0.1", "description": "The colorful spectrum of stories.", "author": "The Critters", "scripts": { "dev": "plasmo dev", "build": "plasmo build", "package": "plasmo package" }, "dependencies": { "@elysiajs/eden": "^0.8.1", "@radix-ui/react-accordion": "^1.1.2", "@radix-ui/react-collapsible": "^1.0.3", "@radix-ui/react-progress": "^1.0.3", "@radix-ui/react-slot": "^1.0.2", "@repo/backend": "workspace:*", "axios": "^1.6.7", "class-variance-authority": "^0.7.0", "clsx": "^2.1.0", "lucide-react": "^0.331.0", "plasmo": "^0.84.2", "react": "^18.2.0", "react-dom": "^18.2.0", "react-icons": "^5.0.1", "recharts": "^2.12.0", "tailwind-merge": "^2.2.1", "tailwindcss-animate": "^1.0.7" }, "devDependencies": { "@ianvs/prettier-plugin-sort-imports": "^4.1.1", "@types/chrome": "^0.0.258", "@types/node": "^20.11.5", "@types/react": "^18.2.48", "@types/react-dom": "^18.2.18", "postcss": "^8.4.35", "prettier": "^3.2.4", "tailwindcss": "^3.4.1", "typescript": "^5.3.3" }, "manifest": { "host_permissions": [ "https://*/*" ] } } ``` ### apps/backend/src/index.ts ```typescript import { cors } from "@elysiajs/cors"; import { swagger } from "@elysiajs/swagger"; import { TypeCompiler } from "@sinclair/typebox/compiler"; import { Elysia, Static, TSchema, t } from "elysia"; import { SelectArticle, SelectSpecificPoint, SelectSupersetPoint, insertArticles, insertSpecificPoints, insertSupersetPointsSchema, nanoid, selectArticlesSchema, selectSpecificPointsSchema, selectSupersetPointsSchema, } from "./db/schema"; export type SelectArticleJoinedSpecificPoints = SelectArticle & { specificPoints: SelectSpecificPoint[]; }; export type SelectSupersetPointJoinedSpecificPoints = SelectSupersetPoint & { specificPoints: SelectSpecificPoint[]; }; export type GlobalData = { thisArticle: SelectArticleJoinedSpecificPoints; relevantArticles: SelectArticleJoinedSpecificPoints[]; supersetPoints: SelectSupersetPoint[]; }; export function parse(schema: T, data: unknown) { try { const C = TypeCompiler.Compile(schema); const isValid = C.Check(data); if (!isValid) { const errors = [...C.Errors(data)]; throw errors; } return C.Decode(data); } catch (error) { console.log("🚀 ~ data:", data); console.error(error); } } const BASE_URL = "https://3b2d-68-65-175-38.ngrok-free.app"; const EXECUTE_DATABASE_URL = `${BASE_URL}/execute_sql`; async function execute({ sql, params, schema, }: { sql: string; params: Record; schema: T; }): Promise> { const result = await fetch(EXECUTE_DATABASE_URL, { method: "POST", headers: { "Content-Type": "application/json", }, body: JSON.stringify({ sql, params }), }) .then((res) => { if (!res.ok) throw new Error("Failed to execute SQL"); return res.json(); }) .then((data) => data.result) .then((rows) => { return rows.map((row: any) => Object.entries(row).reduce((acc, [key, value]) => { if (key === "sentiment") { return { ...acc, [key]: JSON.parse(value.replaceAll("'", '"')) }; } return { ...acc, [key]: value }; }, {}) ); }); return parse(schema, result); } // const t = initTRPC.create(); // // const router = t.router({ // findSimilarArticles: t.procedure // .input(z.object({ url: z.string() })) // .query(({ input: { url } }) => { // // ... // }), // insertArticles: t.procedure.input(insertArticles).mutation(({ input }) => { // return execute({ // sql: "INSERT INTO articles (title, url, bias, sentiment, embedding) VALUES (:title, :url, :bias, :sentiment, :embedding)", // params: input, // }); // }), // insertSpecificPoints: t.procedure // .input(insertSpecificPoints) // .mutation(async ({ input }) => { // return execute({ // sql: "INSERT INTO specific_points (article_id, original_excerpt, embedding, bias, sentiment, superset_point_id) VALUES (:article_id, :original_excerpt, :embedding, :bias, :sentiment, :superset_point_id)", // params: input, // }); // await db.insert(specificPoints).values(input); // }), // updateSpecificPoints: t.procedure // .input(insertSpecificPoints) // .mutation(async ({ input }) => { // // await db.update(specificPoints).set({ // // superset_point_id: input.superset_point_id, // // }); // }), // insertSupersetPoints: t.procedure // .input(insertSupersetPoints) // .mutation(async ({ input }) => { // return execute({ // sql: "INSERT INTO superset_points (title_generated, embedding) VALUES (:title_generated, :embedding)", // params: input, // }); // // await db.insert(supersetPoints).values(input); // }), // }); // export type Router = typeof router; // const app = new Elysia().use(swagger()).use(trpc(router)).listen(3000); const app = new Elysia() .use(swagger()) .use( cors({ allowedHeaders: ["*"], }) ) .get("/", () => "Hello, world!") .get( "/getGlobalData", async ({ query }): Promise => { const { url } = query; console.log("🚀 ~ url:", url); const returnedFetch = await fetch(`${BASE_URL}/find_similar_articles`, { method: "POST", headers: { "Content-Type": "application/json", }, body: JSON.stringify({ url }), }) .then((response) => response.json()) .then((data) => { data.article.sentiment = JSON.parse(data.article.sentiment); data.relevantArticles = data.relevantArticles.map((article: any) => { article.sentiment = JSON.parse(article.sentiment); return article; }); return parse( t.Object({ article: t.Nullable(selectArticlesSchema), relevantArticles: t.Array(selectArticlesSchema), }), data ); }); if (!returnedFetch) throw new Error("Failed to fetch"); const { article: thisArticle, relevantArticles } = returnedFetch; console.log("🚀 ~ thisArticle:", thisArticle); console.log("🚀 ~ relevantArticles:", relevantArticles); if (!thisArticle) throw new Error("Article not found"); const thisArticleSpecificPoints = await execute({ sql: "SELECT * FROM specific_points WHERE article_id = :id", params: { id: thisArticle.id }, schema: t.Array(selectSpecificPointsSchema), }); console.log("🚀 ~ thisArticleSpecificPoints:", thisArticleSpecificPoints); const supersetPoints = await execute({ sql: "SELECT * FROM superset_points", params: {}, schema: t.Array(selectSupersetPointsSchema), }); console.log("🚀 ~ supersetPoints:", supersetPoints); return { thisArticle: { ...thisArticle, specificPoints: thisArticleSpecificPoints, }, relevantArticles: await Promise.all( relevantArticles.map(async (article) => { const specificPoints = await execute({ sql: "SELECT * FROM specific_points WHERE article_id = :id", params: { id: article.id }, schema: t.Array(selectSpecificPointsSchema), }); return { ...article, specificPoints, }; }) ), supersetPoints, }; }, { query: t.Object({ url: t.String() }), } ) .po [truncated — 1729 more characters] ``` ### apps/backend/src/db/index.ts ```typescript import { drizzle } from "drizzle-orm/libsql"; import { createClient } from "@libsql/client"; const client = createClient({ url: "http://127.0.0.1:8080" }); export const db = drizzle(client); ``` ### pnpm-workspace.yaml ```yaml packages: - "packages/*" - "apps/*" ``` ### clustering.py ```python import hdbscan import numpy as np def get_clusters(embeddings): clusterer = hdbscan.HDBSCAN() print("clusterer", clusterer) clusterer.fit(embeddings) cluster_labels = clusterer.labels_ print("cluster labels", cluster_labels) num_clusters = cluster_labels.max() + 1 print(f"Number of clusters: {num_clusters}") def generate_cluster(center, num_points, deviation=0.1): return np.random.normal(loc=center, scale=deviation, size=(num_points, len(center))) def main(): cluster1_center = np.array([0.5, 0.5]) cluster2_center = np.array([-0.5, -0.5]) cluster1_vectors = generate_cluster(cluster1_center, 50, deviation=0.05) cluster2_vectors = generate_cluster(cluster2_center, 50, deviation=0.05) embedding_vectors = np.vstack([cluster1_vectors, cluster2_vectors]) get_clusters(embedding_vectors) if __name__ == "__main__": main() ``` ### iris_backend.py ```python from utils import resolver from flask import Flask, request, jsonify from sqlalchemy import create_engine, text from utils import embed_text, fetch_article app = Flask(__name__) username = 'SUPERUSER' password = 'SYS2' hostname = 'localhost' port = '1972' namespace = 'USER' CONNECTION_STRING = f"iris://{username}:{password}@{hostname}:{port}/{namespace}" engine = create_engine(CONNECTION_STRING) @app.route('/execute_sql', methods=['POST']) def execute_sql(): data = request.json sql = data.get('sql') params = data.get('params', {}) print('SQL: ' + sql) print(f'Params: {params}') print() if not sql: return jsonify({'error': 'SQL statement is required.'}), 400 try: with engine.connect() as connection: result = connection.execute(text(sql), params) connection.commit() if result.returns_rows: return jsonify({'result': [{key: value for key, value in row.items()} for row in result.mappings()]}) else: return jsonify({'result': f'{result.rowcount} row(s) affected.'}) except Exception as e: return jsonify({'error': str(e)}), 500 @app.route('/find_similar_articles', methods=['POST']) def find_similar_articles(): data = request.json url = data.get('url') path = resolver[url] # "articles/Carlson Putin/Alexei Navalny’s death underlines the horrors of Tucker Carlson’s Putin interview..md" # need map from url to path article = fetch_article(path) search_vector = embed_text(article) sql = text(""" SELECT *, VECTOR_DOT_PRODUCT(embedding, TO_VECTOR(:search_vector)) AS dot_product_result FROM articles WHERE VECTOR_DOT_PRODUCT(embedding, TO_VECTOR(:search_vector)) > 0.75 ORDER BY dot_product_result DESC """) try: with engine.connect() as connection: result = connection.execute(sql, {'search_vector': str(search_vector)}) article = None relevantArticles = [] for row in result.mappings(): if row['url'] == path: article = dict(row) else: relevantArticles.append(dict(row)) return jsonify({'article': article, 'relevantArticles': relevantArticles}) except Exception as e: return jsonify({'error': str(e)}), 500 if __name__ == '__main__': app.run(debug=True, port=5001) ``` [67 more indexed source files omitted to keep this export small. The full file list is in the Codebase structure section above.]