# Project export: Fudu This document was generated by HackStack to give an AI agent context about a hackathon project. Sections are labeled with their provenance; content marked as truncated was cut to keep this document small. ## Project metadata - Hackathon: TreeHacks 2024 - Tagline: Fudu is a browser utility that helps seniors interact with the internet in the way they know best - through their voice. - Devpost: https://devpost.com/software/test-l5xbjp - GitHub: https://github.com/wikiPika/treehacks-2024 - Video: https://www.youtube.com/embed/eMUUmhLwvSE?enablejsapi=1&hl=en_US&rel=0&start=&version=3&wmode=transparent - Team: 2 GitHub contributor(s) — wikiPika (4 commits), Dylan Epstein-Gross (2 commits) ## Devpost submission (written by the team) ### Inspiration Everyone's been there -- on the phone with a grandparent, trying for hours to help them sign up for that new service they need, figure out social media, scroll their camera roll, etc. It's not their fault -- they didn't grow up with this stuff like we did -- but it's still a serious challenge for both them and whoever's trying to help them. But what if that didn't need to be the case? The four of us all care deeply for the seniors in our lives, and we realized the potential for an easy yet surprisingly universal fix to internet usage woes: a browser plugin that converts forms and navigation into something as simple as having a conversation. ### What it does Fudu enables seniors and others with accessibility challenges the freedom to navigate forms on the internet through a more streamlined verbal interface. It takes advantage of state-of-the-art large language models to decipher spoken inputs into the exact fields needed for forms, and has been built with the utmost care to ensure that mistakes aren't made in the communication. ### How we built it To build Fudu, we developed a frontend with React and Next.js, calling the WebSpeech API for speech recognition and generation. The GPT-3.5 LLM is used to parse spoken inputs into the form fields. ### Challenges we ran into We struggled to narrow down the scope of our project after ambitious initial plans. We had hoped at first to build off of Terra API nutrition data and use fetch.ai agents to plan out further health tasks for seniors from the vocal interface. However, we ultimately realized these ideas were incoherent with our vision and decided to focus on our most important goal: making the internet more accessible for our loved ones. ### Accomplishments we're proud of And we are proud to say that we handily accomplished our goal! We believe the app has a great feel and is the starting point for a product that could be implemented on devices across the world. Some of our members also hadn’t hacked at an event like this before, so we were really proud of being able to put something together at all, especially something so eminently usable. We learned a lot and are really glad we came all this way to Stanford! ### What we learned Our frontend-focused teammates learned a great deal about UI design through React and Next.js, as they iteratively built a product which looked and felt as great as possible. But the lion’s share of our learning came from the AI in the backend. Besides the WebSpeech API and LLM ultimately in use, we explored many other sponsor possibilities such as fetch.ai agents, Terra API webhooks, Intel PredictionGuard models, and more. The vestiges of these explorations can be seen in our repository – we learned and experimented a lot, experiences which will stick with us long after this hackathon. ### What's next We feel that Fudu has a ton of potential to help the health and wellbeing of seniors across the world! More testing with real seniors must be done to ensure that it is ready for public use, and additional frontend work must be integrated to ensure that it actually “plugs in” to forms across the internet, but we’ve made great first steps here! We can’t wait to build Fudu out further and see where it goes. ## README (from the GitHub repository) This is a [Next.js](https://nextjs.org/) project bootstrapped with [`create-next-app`](https://github.com/vercel/next.js/tree/canary/packages/create-next-app). ## Getting Started First, run the development server: ```bash npm run dev # or yarn dev # or pnpm dev # or bun dev ``` Open [http://localhost:3000](http://localhost:3000) with your browser to see the result. You can start editing the page by modifying `app/page.tsx`. The page auto-updates as you edit the file. This project uses [`next/font`](https://nextjs.org/docs/basic-features/font-optimization) to automatically optimize and load Inter, a custom Google Font. ## Learn More To learn more about Next.js, take a look at the following resources: - [Next.js Documentation](https://nextjs.org/docs) - learn about Next.js features and API. - [Learn Next.js](https://nextjs.org/learn) - an interactive Next.js tutorial. You can check out [the Next.js GitHub repository](https://github.com/vercel/next.js/) - your feedback and contributions are welcome! ## Deploy on Vercel The easiest way to deploy your Next.js app is to use the [Vercel Platform](https://vercel.com/new?utm_medium=default-template&filter=next.js&utm_source=create-next-app&utm_campaign=create-next-app-readme) from the creators of Next.js. Check out our [Next.js deployment documentation](https://nextjs.org/docs/deployment) for more details. ## Detected evidence (automated analysis) Indexed codebase: 47 recognized source files, 72 KB. - CSS (language) — detected in the code - JavaScript (language) — detected in the code - Next.js (technology) — detected in the code - OpenAI (technology) — detected in the code - Python (language) — detected in the code - React (technology) — detected in the code - Tailwind CSS (technology) — detected in the code - TypeScript (language) — detected in the code ## Codebase structure (from repository index) ### Files (105 of 105) ``` .eslintrc.json .gitignore agent.py agentsExperiment/agent1qdy3k24res_data.json agentsExperiment/agent1qt8xgt2hdu_data.json agentsExperiment/agent1qw50wcs4nd_data.json agentsExperiment/book.py agentsExperiment/food_query.py agentsExperiment/langer.py agentsExperiment/market.py agentsExperiment/query.py agentsExperiment/restaurant.py agentsExperiment/shop.py agentsExperiment/shopper.py agentsExperiment/short_text.txt agentsExperiment/test_agents.py agentsExperiment/user.py components.json data.csv data.txt data/data0.txt data/data1.txt data/data10.txt data/data2.txt data/data3.txt data/data4.txt data/data5.txt data/data6.txt data/data7.txt data/data8.txt data/data9.txt fetch_experiment.ipynb FoodData_Central_foundation_food_csv_2023-10-26/acquisition_samples.csv FoodData_Central_foundation_food_csv_2023-10-26/agricultural_samples.csv FoodData_Central_foundation_food_csv_2023-10-26/food_attribute_type.csv FoodData_Central_foundation_food_csv_2023-10-26/food_attribute.csv FoodData_Central_foundation_food_csv_2023-10-26/food_calorie_conversion_factor.csv FoodData_Central_foundation_food_csv_2023-10-26/food_component.csv FoodData_Central_foundation_food_csv_2023-10-26/food_nutrient_conversion_factor.csv FoodData_Central_foundation_food_csv_2023-10-26/food_nutrient.csv FoodData_Central_foundation_food_csv_2023-10-26/food_portion.csv FoodData_Central_foundation_food_csv_2023-10-26/food_protein_conversion_factor.csv FoodData_Central_foundation_food_csv_2023-10-26/food_update_log_entry.csv FoodData_Central_foundation_food_csv_2023-10-26/food.csv FoodData_Central_foundation_food_csv_2023-10-26/foundation_food.csv FoodData_Central_foundation_food_csv_2023-10-26/input_food.csv FoodData_Central_foundation_food_csv_2023-10-26/lab_method_code.csv FoodData_Central_foundation_food_csv_2023-10-26/lab_method_nutrient.csv FoodData_Central_foundation_food_csv_2023-10-26/lab_method.csv FoodData_Central_foundation_food_csv_2023-10-26/market_acquisition.csv FoodData_Central_foundation_food_csv_2023-10-26/measure_unit.csv FoodData_Central_foundation_food_csv_2023-10-26/nutrient.csv FoodData_Central_foundation_food_csv_2023-10-26/sample_food.csv FoodData_Central_foundation_food_csv_2023-10-26/sub_sample_food.csv FoodData_Central_foundation_food_csv_2023-10-26/sub_sample_result.csv llm_experiment.ipynb middleware.ts model/adapter_config.json model/adapter_model.safetensors model/README.md model/special_tokens_map.json model/tokenizer_config.json model/tokenizer.json model/tokenizer.model next.config.mjs package.json pg-war_and_peace.txt postcss.config.js process_foods.ipynb process_form.ipynb public/index.ts README.md src/app/(home)/layout.tsx src/app/(home)/page.tsx src/app/api/consumeTerraWebhook/route.ts src/app/api/generateWidgetSession/route.ts src/app/demo/layout.tsx src/app/demo/page.tsx src/app/globals.css src/app/layout.tsx src/components/ui/button.tsx src/components/ui/calendar.tsx src/components/ui/card.tsx src/components/ui/checkbox.tsx src/components/ui/form.tsx src/components/ui/input.tsx src/components/ui/label.tsx src/components/ui/popover.tsx src/lib/utils.ts src/ui/Animation.ts src/ui/demo/Dashboard.tsx src/ui/demo/DataContext.tsx src/ui/demo/Onboarding.tsx src/ui/demo/OpenAIContext.tsx src/ui/demo/TerraApi.tsx src/ui/demo/TTSContext.tsx src/ui/home/Blurb.tsx src/ui/home/Footer.tsx src/ui/home/Hero.tsx src/ui/home/Instructions.tsx src/ui/home/Logo.tsx src/ui/home/Navbar.tsx tailwind.config.ts terra_experiment.ipynb tsconfig.json ``` ### Dependencies - package.json: @clerk/nextjs@^4.29.7, @hookform/resolvers@^3.3.4, @radix-ui/react-checkbox@^1.0.4, @radix-ui/react-label@^2.0.2, @radix-ui/react-popover@^1.0.7, @radix-ui/react-slot@^1.0.2, @tryterra/terra-ui@^0.2.0, @types/node@^20, @types/react@^18, @types/react-dom@^18, @types/react-speech-recognition@^3.9.5, autoprefixer@^10.0.1, axios@^1.6.7, class-variance-authority@^0.7.0, clsx@^2.1.0, date-fns@^3.3.1, eslint@^8, eslint-config-next@14.1.0, framer-motion@^11.0.5, lucide-react@^0.331.0, next@14.1.0, openai@^4.28.0, postcss@^8, react@^18, react-day-picker@^8.10.0, react-dom@^18, react-hook-form@^7.50.1, react-joyride@^2.7.2, react-speech-recognition@^3.10.0, tailwind-merge@^2.2.1, tailwindcss@^3.3.0, tailwindcss-animate@^1.0.7, terra-api@^1.4.5, typescript@^5, zod@^3.22.4 ### Recent commits (newest first) - Merge branch 'main' of https://github.com/wikiPika/treehacks-2024 into main - Co-authored-by: hyunerinkim - Merge branch 'main' of https://github.com/wikiPika/treehacks-2024 - Backend AI - layout - Initial commit from Create Next App ## Key source files (fetched from GitHub, selected and truncated for size) ### package.json ``` { "name": "treehacks-2024", "version": "0.1.0", "private": true, "scripts": { "dev": "next dev", "build": "next build", "start": "next start", "lint": "next lint" }, "dependencies": { "@clerk/nextjs": "^4.29.7", "@hookform/resolvers": "^3.3.4", "@radix-ui/react-checkbox": "^1.0.4", "@radix-ui/react-label": "^2.0.2", "@radix-ui/react-popover": "^1.0.7", "@radix-ui/react-slot": "^1.0.2", "@tryterra/terra-ui": "^0.2.0", "@types/react-speech-recognition": "^3.9.5", "axios": "^1.6.7", "class-variance-authority": "^0.7.0", "clsx": "^2.1.0", "date-fns": "^3.3.1", "framer-motion": "^11.0.5", "lucide-react": "^0.331.0", "next": "14.1.0", "openai": "^4.28.0", "react": "^18", "react-day-picker": "^8.10.0", "react-dom": "^18", "react-hook-form": "^7.50.1", "react-joyride": "^2.7.2", "react-speech-recognition": "^3.10.0", "tailwind-merge": "^2.2.1", "tailwindcss-animate": "^1.0.7", "terra-api": "^1.4.5", "zod": "^3.22.4" }, "devDependencies": { "@types/node": "^20", "@types/react": "^18", "@types/react-dom": "^18", "autoprefixer": "^10.0.1", "eslint": "^8", "eslint-config-next": "14.1.0", "postcss": "^8", "tailwindcss": "^3.3.0", "typescript": "^5" } } ``` ### public/index.ts ```typescript import HomeImage from './elderLanding.png' import LogoImage from './Design.png' export { HomeImage, LogoImage } ``` ### src/app/layout.tsx ```typescript import { ClerkProvider } from '@clerk/nextjs' import type { Metadata } from "next"; import { Inter } from "next/font/google"; import "./globals.css"; const inter = Inter({ subsets: ["latin"] }); export const metadata: Metadata = { title: "Create Next App", description: "Generated by create next app", }; export default function RootLayout({ children, }: Readonly<{ children: React.ReactNode; }>) { return ( {children} ); } ``` ### src/app/(home)/page.tsx ```typescript import Hero from "@/ui/home/Hero"; export default function Home() { return (
); } ``` ### src/app/demo/layout.tsx ```typescript "use client"; import DataProvider from "@/ui/demo/DataContext"; import { ReactNode } from "react"; export default function Layout(props: { children: ReactNode }) { return {props.children} } ``` ### src/app/(home)/layout.tsx ```typescript import Navbar from "@/ui/home/Navbar"; import Footer from "@/ui/home/Footer"; import "../globals.css"; import { ReactNode } from "react"; export default function Layout({children}: { children: ReactNode, }) { return
{children}
} ``` ### src/app/demo/page.tsx ```typescript "use client"; import { useData } from "@/ui/demo/DataContext"; import Onboarding from "@/ui/demo/Onboarding"; import Dashboard from "@/ui/demo/Dashboard"; import TTSProvider from "@/ui/demo/TTSContext"; import { SignIn, SignedIn, SignedOut } from "@clerk/nextjs"; import OpenAIProvider from "@/ui/demo/OpenAIContext"; /* TODO: onboarding process (food, restaurant) suggestions to complement diet quick links to purchase items */ export default function Page() { const { onboarded, setOnboarded } = useData(); return (
{!onboarded && (
)}
); } ``` ### src/app/api/generateWidgetSession/route.ts ```typescript import { NextRequest, NextResponse } from "next/server"; import Terra from "terra-api"; const terra = new Terra(process.env.TERRAAPI_DEVELOPER_ID ?? "", process.env.TERRAAPI_API_KEY ?? "", process.env.TERRA_WEBHOOK_SECRET ?? ""); export async function GET(request: NextRequest) { const resp = await terra.generateWidgetSession({ referenceID: "HelloStanford", language: "en", authSuccessRedirectUrl: "http://localhost:3000/demo", authFailureRedirectUrl: "http://localhost:3000/demo" }) return NextResponse.json({ data: resp }, { status: 200}); } ``` ### src/app/api/consumeTerraWebhook/route.ts ```typescript import { NextRequest, NextResponse } from "next/server"; import axios from "axios"; import Terra from "terra-api"; const terra = new Terra(process.env.TERRAAPI_DEVELOPER_ID ?? "", process.env.TERRAAPI_API_KEY ?? "", process.env.TERRA_WEBHOOK_SECRET ?? ""); export async function POST(request: NextRequest) { const res = await request.json(); console.log(res); return NextResponse.json({ }, { status: 200}); } async function streamToString(stream: NodeJS.ReadableStream): Promise { const chunks: Array = []; for await (let chunk of stream) { chunks.push(chunk) } const buffer = Buffer.concat(chunks); return buffer.toString("utf-8") } ``` ### postcss.config.js ```javascript module.exports = { plugins: { tailwindcss: {}, autoprefixer: {}, }, }; ``` [35 more indexed source files omitted to keep this export small. The full file list is in the Codebase structure section above.]