# Project export: Recollect: Leave No Book Behind This document was generated by HackStack to give an AI agent context about a hackathon project. Sections are labeled with their provenance; content marked as truncated was cut to keep this document small. ## Project metadata - Hackathon: TreeHacks 2024 - Tagline: Our robot scans, digitizes, and analyzes books with zero intervention. - Devpost: https://devpost.com/software/recollect-ytn0bk - GitHub: https://github.com/Scott-Hickmann/Book-Scanner - Demo: https://recollect-knowledge.vercel.app/ - Video: https://www.youtube.com/embed/r6JPr8ceKi0?enablejsapi=1&hl=en_US&rel=0&start=&version=3&wmode=transparent - Result: winner (Best Hardware Hack (4x Prusa 3D Printers)) - Team: 3 GitHub contributor(s) — Jason Lin (48 commits), Scott Hickmann (16 commits), Kaien Yang (7 commits) ## Devpost submission (written by the team) ### Overview Recollect is a $150 robot that scans, digitizes, and analyzes books with zero intervention. Once a book is placed on the stand, Recollect's robotic arm delicately flips the pages while a high-resolution camera captures each page. The images are sent to a website which merges the images into a PDF and creates AI summaries and insights of the document. Why build this? Only 12% of all published books have been digitized. Historical records, ancient manuscripts, rare collections. Family photo albums, old journal entries, science field notes. Without digitization, centuries of accumulated wisdom, cultural treasures, personal narratives, and family histories threaten to be forever lost to time. Large-scale digitization currently requires highly specialized equipment and physical personnel to manually flip, scan, and process each page. Oftentimes, this is simply not practical, resulting in many books remaining in undigitized form, which necessitates careful, expensive, and unsustainable transportation across various locations for analysis. ### How we built it Hardware: Recollect was made with easy-to-fabricate material including 3D printed plastic parts, laser-cut acrylic and wood, and cheap, and off-the-shelf electronics. A book rests at a 160-degree angle, optimal to hold the book naturally open while minimizing distortions. The page presser drops onto the book, flattening it to further minimize distortions. After the photo is taken, the page presser is raised, then a two-degree-of-freedom robotic arm flips the page. A lightly adhesive pad attaches to the page, and then one of the joints rotates the page. The second joint separates the page from the adhesive pad, and the arm returns to rest. The scanner was designed to be adaptable to a wide range of books, up to 400mm tall and 250 mm page width, with easy adjustments to the arm joints and range of motion to accommodate for a variety of books. Software: Image processing: On the backend, we leverage OpenCV to identify page corners, rescale images, and sharpen colors to produce clear images. These images are processed with pre-trained Google Cloud Vision API models to enable optical character recognition of handwriting and unstructured text. The data are saved into a Supabase database to allow users to access their digital library from anywhere. Webpage and cloud storage: The front end is a Vercel-deployed web app built with Bun, Typescript, Chakra, Next.js, and React.js. ### Challenges we ran into We ran into challenges involving getting the perfect angle for the robotic arm to properly stick to the page. To fix this, we had to modify the pivot point of the arm’s base to be in line with the book’s spine and add a calibration step to make it perfectly set up for the book to be scanned. Our first version also used servo motors with linkages to raise the acrylic page presser up and down, but we realized these motors did not have enough torque. As a result, we replaced them with DC motors and a basic string and pulley system which turned out to work surprisingly well. ### Accomplishments we're proud of This project was a perfect blend of each team member’s unique skill sets: Lawton, a mechanical engineering major, Scott, an electrical and systems engineer, Kaien, an AI developer, and Jason, a full-stack developer. Being able to combine our skills in this project was amazing, and we were truly impressed by how much we were able to accomplish in just 24 hours. Seeing this idea turn into a physical reality was insane, and we were able to go beyond what we initially planned on building (such as adding summarization, quotation, and word cloud features as post-processing steps on your diary scans). We’re happy to say that we’ve already digitized over 100 pages of our diaries through testing. ### What we learned We learned how to effectively divide up the project into several tasks and assign it based on area of expertise. We also learned to parallelize our work—while parts were being 3D-printed, we would focus on software, design, and electronics. ### What's next for Recollect We plan to improve the reliability of our system to work with all types of diaries, books, and notebooks, no matter how stiff or large the pages are. We also want to focus on recreating PDFs from these books in a fully digital format (i.e. not just the images arranged in a PDF document but actual text boxes following the formatting of the original document). We also plan to release all of the specifications and software publicly so that anyone can build their own Recollect scanner at home to scan their own diaries and family books. We will design parts kits to make this process even easier. We will also explore collaborating with Stanford libraries and our close communities (friends and family). Thanks to Recollect, we hope no book is left behind. ## README (from the GitHub repository) # Recollect Book Scanner: Leave No Book Left Behind Winner of Treehacks 2024 Best Hardware Prize https://devpost.com/software/recollect-ytn0bk ## Inspiration A mere 12% of all books ever published have been digitized. Historical records, ancient manuscripts, rare collections. Family photo albums, old journal entries, science field notes. Without digitization, centuries of accumulated wisdom, cultural treasures, personal narratives, and family histories threaten to be forever lost to time. Large-scale digitization currently requires highly specialized equipment and physical personnel to manually flip, scan, and process each page. Oftentimes, this is simply not practical, resulting in many books remaining in undigitized form, which necessitates careful, expensive, and unsustainable transportation across various locations for analysis. In the modern era of dextrous robots and powerful AI algorithms, the status quo is unacceptable. ## What it does Introducing Recollect: a $150 robot-powered book scanner that addresses both major barriers to large-scale digitization: expensive equipment and manual labor. Recollect employs state-of-the-art robotics and AI to automate the digitization process. Once a book is placed on the stand, Recollect's robotic arm delicately flips the pages while a high-resolution camera captures each page. The web interface allows users to organize, analyze, and share digitized content easily. Recollect democratizes digitization and ensures that no book, manuscript, or document is left behind. ## How we built it *Hardware:* Recollect was made with easy-to-fabricate material including 3D printed plastic parts, laser-cut acrylic and wood, and cheap, and off-the-shelf electronics. A book rests at a 160-degree angle, optimal to hold the book naturally open while minimizing distortions. The page presser drops onto the book, flattening it to further minimize distortions. After the photo is taken, the page presser is raised, then a two-degree-of-freedom robotic arm flips the page. A lightly adhesive pad attaches to the page, and then one of the joints rotates the page. The second joint separates the page from the adhesive pad, and the arm returns to rest. The scanner was designed to be adaptable to a wide range of books, up to 400mm tall and 250 mm page width, with easy adjustments to the arm joints and range of motion to accommodate for a variety of books. *Software:* Image processing: On the backend, we leverage OpenCV to identify page corners, rescale images, and sharpen colors to produce clear images. These images are processed with pre-trained Google Cloud Vision API models to enable optical character recognition of handwriting and unstructured text. The data are saved into a Supabase database to allow users to access their digital library from anywhere. Webpage and cloud storage: The front end is a Vercel-deployed web app built with Bun, Typescript, and Next.js/React.js. ## Challenges we ran into We ran into challenges involving getting the perfect angle for the robotic arm to properly stick to the page. To fix this, we had to modify the pivot point of the arm’s base to be in line with the book’s spine and add a calibration step to make it perfectly set up for the book to be scanned. Our first version also used servo motors with linkages to raise the acrylic page presser up and down, but we realized these motors did not have enough torque. As a result, we replaced them with DC motors and a basic string and pulley system which turned out to work surprisingly well. ## Accomplishments that we're proud of This project was a perfect blend of each team member’s unique skill sets: Lawton, a mechanical engineering major, Scott, an electrical and systems engineer, Kaien, an AI developer, and Jason, a full-stack developer. Being able to combine our skills in this project was amazing, and we were truly impressed by how much we were able to accomplish in just 24 hours. Seeing this idea turn into a physical reality was insane, and we were able to go beyond what we initially planned on building (such as adding summarization, quotation, and word cloud features as post-processing steps on your diary scans). We’re happy to say that we’ve already digitized over 100 pages of our diaries through testing. ## What we learned We learned how to effectively divide up the project into several tasks and assign it based on area of expertise. We also learned to parallelize our work—while parts were being 3D-printed, we would focus on software, design, and electronics. ## What's next for Recollect We plan to improve the reliability of our system to work with all types of diaries, books, and notebooks, no matter how stiff or large the pages are. We also want to focus on recreating PDFs from these books in a fully digital format (i.e. not just the images arranged in a PDF document but actual text boxes following the formatting of the original document). We also plan to release all of the specifications and software publicly so that anyone can build their own Recollect scanner at home to scan their own diaries and family books. We will design parts kits to make this process even easier. As soon as this is ready, we plan to quickly scale Recollect’s design to Stanford libraries and our close communities (friends and family) as well as expand to the wider public. Thanks to Recollect, we hope no book will ever be left behind. ## Detected evidence (automated analysis) Indexed codebase: 23 recognized source files, 60 KB. - C++ (language) — detected in the code - Next.js (technology) — detected in the code - OpenAI (technology) — detected in the code - Python (language) — detected in the code - React (technology) — detected in the code - Supabase (technology) — detected in the code - TypeScript (language) — detected in the code - Vercel (technology) — claimed on Devpost, not found in the code ## Codebase structure (from repository index) ### Files (41 of 41) ``` .gitignore Arduino/.gitignore Arduino/.vscode/extensions.json Arduino/include/README Arduino/lib/README Arduino/platformio.ini Arduino/src/main.cpp Arduino/test/README book-scan-frontend/.eslintrc.json book-scan-frontend/.gitignore book-scan-frontend/app/analysis/[doc_id]/page.tsx book-scan-frontend/app/analysis/page.tsx book-scan-frontend/app/api/quote/route.ts book-scan-frontend/app/api/summarize/route.ts book-scan-frontend/app/components/docStream.tsx book-scan-frontend/app/layout.tsx book-scan-frontend/app/page.tsx book-scan-frontend/app/providers.tsx book-scan-frontend/bun.lockb book-scan-frontend/next.config.mjs book-scan-frontend/package.json book-scan-frontend/public/book-flip.json book-scan-frontend/README.md book-scan-frontend/tsconfig.json Controller/.gitignore Controller/.vscode/settings.json Python-Backend/.gitignore Python-Backend/controller.py Python-Backend/doc.py Python-Backend/environment.yml Python-Backend/joined_motor.py Python-Backend/joined_servo.py Python-Backend/keyboard.py Python-Backend/motor.py Python-Backend/pictureProcessing.py Python-Backend/pictureStreaming.py Python-Backend/README.md Python-Backend/requirements.txt Python-Backend/servo_manager.py Python-Backend/servo.py README.md ``` ### Dependencies - book-scan-frontend/package.json: @chakra-ui/icons@^2.1.1, @chakra-ui/next-js@^2.2.0, @chakra-ui/react@^2.8.2, @emotion/react@^11.11.3, @emotion/styled@^11.11.0, @million/lint@latest, @supabase/supabase-js@^2.39.6, @types/node@^20, @types/react@^18, @types/react-dom@^18, date-fns@^3.3.1, eslint@^8, eslint-config-next@14.1.0, framer-motion@^11.0.5, lottie-react@^2.4.0, next@14.1.0, openai@^4.28.0, prop-types@^15.8.1, react@^18, react-dom@^18, react-lottie@^1.2.4, react-parallax-tilt@^1.7.211, react-wordcloud@^1.2.7, stopword@^3.0.1, typescript@^5 - Python-Backend/requirements.txt: google-cloud-vision, img2pdf, openai, rembg, scipy, supabase ### Recent commits (newest first) - Delete Arduino/.DS_Store - Update page.tsx - Update README.md - Create README.md - Remove sensitive files - Combine with controller - Better motor logic - fix fonts - back button - change url - colormode fixes - fix numpages - frontend changes to work with new backend - threading - DC motor - darkmode - Merge pull request #6 from Scott-Hickmann/list-entries - add date - word cloud - fix crashing out of index, frontend ## Key source files (fetched from GitHub, selected and truncated for size) ### Python-Backend/requirements.txt ``` supabase google-cloud-vision img2pdf openai scipy rembg ``` ### book-scan-frontend/package.json ``` { "name": "book-scan-frontend", "version": "0.1.0", "private": true, "scripts": { "dev": "next dev", "build": "next build", "start": "next start", "lint": "next lint" }, "dependencies": { "@chakra-ui/icons": "^2.1.1", "@chakra-ui/next-js": "^2.2.0", "@chakra-ui/react": "^2.8.2", "@emotion/react": "^11.11.3", "@emotion/styled": "^11.11.0", "@million/lint": "latest", "@supabase/supabase-js": "^2.39.6", "date-fns": "^3.3.1", "framer-motion": "^11.0.5", "lottie-react": "^2.4.0", "next": "14.1.0", "openai": "^4.28.0", "prop-types": "^15.8.1", "react": "^18", "react-dom": "^18", "react-lottie": "^1.2.4", "react-parallax-tilt": "^1.7.211", "react-wordcloud": "^1.2.7", "stopword": "^3.0.1" }, "devDependencies": { "typescript": "^5", "@types/node": "^20", "@types/react": "^18", "@types/react-dom": "^18", "eslint": "^8", "eslint-config-next": "14.1.0" } } ``` ### book-scan-frontend/app/layout.tsx ```typescript // app/layout.tsx import { Providers } from './providers' export default function RootLayout({ children, }: { children: React.ReactNode, }) { return ( {children} ) } ``` ### book-scan-frontend/app/page.tsx ```typescript "use client"; import { Heading, VStack, Text, Box, Button, Icon, Link, HStack, useColorMode, IconButton, Spacer, } from "@chakra-ui/react"; import { SunIcon } from "@chakra-ui/icons"; import DocStream from "./components/docStream"; export default function Page() { const { toggleColorMode } = useColorMode(); return ( Recollect Your Memories, Preserved Forever } onClick={toggleColorMode} /> Made with 💚 by Lawton, Scott, Kaien, Jason ); } ``` ### book-scan-frontend/app/analysis/page.tsx ```typescript "use client"; import { Button, HStack, Heading, VStack, UnorderedList, ListItem, Link, IconButton, } from "@chakra-ui/react"; import { createClient } from "@supabase/supabase-js"; import { useEffect, useState } from "react"; import NextLink from "next/link"; // Import Next.js Link component for client-side transitions import { format } from 'date-fns'; // Import format function from date-fns for formatting timestamps import { ArrowLeftIcon } from "@chakra-ui/icons"; const supabase = createClient( process.env.NEXT_PUBLIC_SUPABASE_URL!, process.env.NEXT_PUBLIC_SUPABASE_ANON_KEY! ); // Define a TypeScript interface for document data interface DocData { doc_id: string; timestamp: string; // Assuming timestamp is a string, adjust based on your actual data type } export default function Page() { const [docList, setDocList] = useState([]); useEffect(() => { const fetchDocs = async () => { let { data: pdfs, error } = await supabase .from("pdfs") .select("doc_id, timestamp"); // Select both doc_id and timestamp if (error) { console.error("Error fetching documents:", error); return; } if (pdfs) { // Map pdfs to include both doc_id and timestamp const docData = pdfs.map((pdf) => ({ doc_id: pdf.doc_id, timestamp: pdf.timestamp, })); setDocList(docData); } }; fetchDocs(); }, []); return ( Your Documents } as={Link} href="/"> {docList.map((doc, index) => ( ))} ); } ``` ### book-scan-frontend/app/api/summarize/route.ts ```typescript import OpenAI from "openai"; const openai = new OpenAI({ apiKey: process.env.OPENAI_API_KEY! }); export async function POST(request: Request) { const body = await request.json(); const { text } = body; const summary = text /*(0, 1000) + "..."*/; // only copy beginning 1000 const chatCompletion = await openai.chat.completions.create({ messages: [ { role: "system", content: "You are a summarizer for documents. Summarize this document in 2 sentences.", }, { role: "user", content: `Here is the summary: ${summary}`, }, ], model: "gpt-4-turbo-preview", }); const response = chatCompletion.choices[0].message.content; return new Response(JSON.stringify({ response }), { headers: { "Content-Type": "application/json" }, }); } ``` ### book-scan-frontend/app/api/quote/route.ts ```typescript import OpenAI from "openai"; const openai = new OpenAI({ apiKey: process.env.OPENAI_API_KEY! }); export async function POST(request: Request) { const body = await request.json(); const { text } = body; const summary = text /*.slice(0, 1000) + "..."*/; // only copy beginning 1000 const chatCompletion = await openai.chat.completions.create({ messages: [ { role: "system", content: "Extract a quote from this document. Return it verbatim. DO NOT mention anything else.", }, { role: "user", content: `Here is the document: ${summary}`, }, ], model: "gpt-4-turbo-preview", }); const response = chatCompletion.choices[0].message.content; return new Response(JSON.stringify({ response }), { headers: { "Content-Type": "application/json" }, }); } ``` ### book-scan-frontend/app/analysis/[doc_id]/page.tsx ```typescript "use client"; import { Box, Button, Image, Grid, GridItem, HStack, Heading, Link, Text, VStack, IconButton, Spinner, } from "@chakra-ui/react"; import { ArrowBackIcon } from "@chakra-ui/icons"; import { useEffect, useState } from "react"; import { createClient } from "@supabase/supabase-js"; import ReactWordcloud from "react-wordcloud"; const supabase = createClient( process.env.NEXT_PUBLIC_SUPABASE_URL!, process.env.NEXT_PUBLIC_SUPABASE_ANON_KEY! ); // don't roast me lmao 🥵 const stopwords = [ "a", "a's", "able", "about", "above", "according", "accordingly", "across", "actually", "after", "afterwards", "again", "against", "ain't", "all", "allow", "allows", "almost", "alone", "along", "already", "also", "although", "always", "am", "among", "amongst", "an", "and", "another", "any", "anybody", "anyhow", "anyone", "anything", "anyway", "anyways", "anywhere", "apart", "appear", "appreciate", "appropriate", "are", "aren't", "around", "as", "aside", "ask", "asking", "associated", "at", "available", "away", "awfully", "b", "be", "became", "because", "become", "becomes", "becoming", "been", "before", "beforehand", "behind", "being", "believe", "below", "beside", "besides", "best", "better", "between", "beyond", "both", "brief", "but", "by", "c", "c'mon", "c's", "came", "can", "can't", "cannot", "cant", "cause", "causes", "certain", "certainly", "changes", "clearly", "co", "com", "come", "comes", "concerning", "consequently", "consider", "considering", "contain", "containing", "contains", "corresponding", "could", "couldn't", "course", "currently", "d", "definitely", "described", "despite", "did", "didn't", "different", "do", "does", "doesn't", "doing", "don't", "done", "down", "downwards", "during", "e", "each", "edu", "eg", "eight", "either", "else", "elsewhere", "enough", "entirely", "especially", "et", "etc", "even", "ever", "every", "everybody", "everyone", "everything", "everywhere", "ex", "exactly", "example", "except", "f", "far", "few", "fifth", "first", "five", "followed", "following", "follows", "for", "former", "formerly", "forth", "four", "from", "further", "furthermore", "g", "get", "gets", "getting", "given", "gives", "go", "goes", "going", "gone", "got", "gotten", "greetings", "h", "had", "hadn't", "happens", "hardly", "has", "hasn't", "have", "haven't", "having", "he", "he's", "hello", "help", "hence", "her", "here", "here's", "hereafter", "hereby", "herein", "hereupon", "hers", "herself", "hi", "him", "himself", "his", "hither", "hopefully", "how", "howbeit", "however", "i", "i'd", "i'll", "i'm", "i've", "ie", "if", "ignored", "immediate", "in", "inasmuch", "inc", "indeed", "indicate", "indicated", "indicates", "inner", "insofar", "instead", "into", "inward", "is", "isn't", "it", "it'd", "it'll", "it's", "its", "itself", "j", "just", "k", "keep", "keeps", "kept", "know", "known", "knows", "l", "last", "lately", "later", "latter", "latterly", "least", "less", "lest", "let", "let's", "like", "liked", "likely", "little", "look", "looking", "looks", "ltd", "m", "mainly", "many", "may", "maybe", "me", "mean", "meanwhile", "merely", "might", "more", "moreover", "most", "mostly", "much", "must", "my", "myself", "n", "name", "namely", "nd", "near", "nearly", "necessary", "need", "needs", "neither", "never", "nevertheless", "new", "next", "nine", "no", "nobody", "non", "none", "noone", "nor", "normally", "not", "nothing", "novel", "now", "nowhere", "o", "obviously", "of", "off", "often", "oh", "ok", "okay", "old", "on", "once", "one", "ones", "only", "onto", "or", "other", "others", "otherwise", "ought", "our", "ours", "ourselves", "out", "outside", "over", "overall", "own", "p", "particular", "particularly", "per", "perhaps", "placed", "please", "plus", "possible", "presumably", "probably", "provides", "q", "que", "quite", "qv", "r", "rather", "rd", "re", "really", "reasonably", "regarding", "regardless", "regards", "relatively", "respectively", "right", "s", "said", "same", "saw", "say", "saying", "says", "second", "secondly", "see", "seeing", "seem", "seemed", "seeming", "seems", "seen", "self", "selves", "sensible", "sent", "serious", "seriously", "seven", "several", "shall", "she", "should", "shouldn't", "since", "six", "so", "some", "somebody", "somehow", "someone", "something", "sometime", "sometimes", "somewhat", "somewhere", "soon", "sorry", "specified", "specify", "specifying", "still", "sub", "such", "sup", "sure", "t", "t's", "take", "taken", "tell", "tends", "th", "than", "thank", "thanks", "thanx", "that", "that's", "thats", "the", "their", "theirs", "them", "themselves", "then", "thence", "there", "there's", "thereafter", "thereby", "therefore", "therein", "theres", "thereupon", "these", "they", "they'd", "they'll", "they're", "they've", "think", "third", "this", "thorough", "thoroughly", "those", "though", "three", "through", "throughout", "thru", "thus", "to", "together", "too", "took", "toward", "towards", "tried", "tries", "truly", "try", "trying", "twice", "two", "u", "un", "under", "unfor [truncated — 7617 more characters] ``` ### Python-Backend/joined_motor.py ```python import time from motor import Motor class JoinedMotor(Motor): def __init__(self, motors): self.motors = motors def set_speed(self, target): for servo in self.servos: servo.set_speed(target, wait=False) def move_for(self, seconds, speed): for motor in self.motors: motor.set_speed(speed) time.sleep(seconds) for motor in self.motors: motor.set_speed(0) time.sleep(0.1) ``` ### Python-Backend/motor.py ```python import time class Motor: def __init__(self, id, servo_manager): self.id = id self.servo_manager = servo_manager self.set_speed(0) def set_speed(self, target): self.servo_manager.set_position(self.id, target) def move_for(self, seconds, speed): self.set_speed(speed) time.sleep(seconds) self.set_speed(0) class InvertedMotor(Motor): def set_speed(self, target): target = 180 - target super().set_speed(target) ``` [12 more indexed source files omitted to keep this export small. The full file list is in the Codebase structure section above.]