# Project export: Ancestory This document was generated by HackStack to give an AI agent context about a hackathon project. Sections are labeled with their provenance; content marked as truncated was cut to keep this document small. ## Project metadata - Hackathon: TreeHacks 2024 - Tagline: Preserving your familial heritage through an immersive experience - Devpost: https://devpost.com/software/ancestory - GitHub: https://github.com/bztravis/ancestory - Demo: https://app.pitch.com/app/presentation/65a55a4a-ae80-4581-a394-a6fca9d43fc3/6b3186e7-22e1-4b64-a11f-3dc36dd26015 - Team: 1 GitHub contributor(s) — Brian Travis (23 commits) ## Devpost submission (written by the team) ### Inspiration Our team members were fortunate to have gotten the chance to foster their relationships with their grandparents, especially during the pandemic. Learning more about the rich - oft-forgotten - stories of our grandparents' lives filled us with warmth. We recognized two main categories of benefits from these stories - one for the senior members of a family and one for the younger members. Sharing meaningful stories helps senior members feel more connected to their families at an age marked by increased loneliness and a gradual loss of memories due to conditions. Similarly, sharing meaningful stories helps younger members appreciate their family's legacy even more. The experience provided by Ancestory also helps everyone look back on their life stories with nostalgic gratitude. ### What it does Ancestory is a web application where you can record a conversation that gets passed through AI to glean important information about the characters featured in the story, the place and time of the story, a summary of the story, etc. This important information is featured as a "node" on the knowledge graph of the senior member's life stories. One can explore the different stories and click on each to read more about the story that was narrated. ### How we built it We built it using React with TypeScript for the front end, NextJS, DeepGram for Speach to Text, and Together AI to run the LLMs on the transcript. ### Challenges we ran into One challenge we ran into was drawing an engineering design of how the different features and functionalities across the stack interact with each other. Another challenge involved some technical difficulties in using some technologies. ### Accomplishments we're proud of We're also proud of learning new technologies and learning how to learn new technologies through leveraging the support of mentors and sponsor representatives. ### What we learned We learned how to brainstorm a product idea. We had to adopt a lot of product thinking in identifying pain points from our observations, market research data to support our hypothesis, and prioritizing features in the 36-hour sprint. We also learned how to design engineering projects thoroughly before we begin building. ### What's next We're very excited about our idea and want to continue to develop it. We want to make the knowledge graph more immersive by including the ability to filter stories based on the people featured in it and the ability to play snippets of the conversation where the individuals seemed the happiest in an attempt to capture the vibe of the story for posterity. We're planning on continuing to build this out and potentially expand its application to a VR app! ## README (from the GitHub repository) This is a [Next.js](https://nextjs.org/) project bootstrapped with [`create-next-app`](https://github.com/vercel/next.js/tree/canary/packages/create-next-app). ## Getting Started First, run the development server: ```bash npm run dev # or yarn dev # or pnpm dev # or bun dev ``` Open [http://localhost:3000](http://localhost:3000) with your browser to see the result. You can start editing the page by modifying `app/page.tsx`. The page auto-updates as you edit the file. This project uses [`next/font`](https://nextjs.org/docs/basic-features/font-optimization) to automatically optimize and load Inter, a custom Google Font. ## Learn More To learn more about Next.js, take a look at the following resources: - [Next.js Documentation](https://nextjs.org/docs) - learn about Next.js features and API. - [Learn Next.js](https://nextjs.org/learn) - an interactive Next.js tutorial. You can check out [the Next.js GitHub repository](https://github.com/vercel/next.js/) - your feedback and contributions are welcome! ## Deploy on Vercel The easiest way to deploy your Next.js app is to use the [Vercel Platform](https://vercel.com/new?utm_medium=default-template&filter=next.js&utm_source=create-next-app&utm_campaign=create-next-app-readme) from the creators of Next.js. Check out our [Next.js deployment documentation](https://nextjs.org/docs/deployment) for more details. ## Detected evidence (automated analysis) Indexed codebase: 28 recognized source files, 44 KB. - CSS (language) — detected in the code - JavaScript (language) — detected in the code - Next.js (technology) — detected in the code - OpenAI (technology) — detected in the code - React (technology) — detected in the code - Tailwind CSS (technology) — detected in the code - TypeScript (language) — detected in the code ## Codebase structure (from repository index) ### Files (36 of 36) ``` .env .eslintrc.json .gitignore app/[relativeId]/page.tsx app/[relativeId]/record/page.tsx app/api/parse/route.ts app/api/transcription/route.ts app/globals.css app/layout.tsx app/page.tsx components.json components/ancestree2D.tsx components/ancestree3D-highlight.tsx components/ancestree3D.tsx components/audio-reaction-2.tsx components/audio-reaction.tsx components/focus-graph-wrapper.tsx components/knowledge-graph-reagraph.tsx components/my-force-graph.tsx components/node-drawer.tsx components/record-button.tsx components/story-drawer.tsx components/ui/button.tsx components/ui/card.tsx components/ui/drawer.tsx components/ui/input.tsx components/ui/label.tsx hooks/useLocalStorage.ts lib/utils.ts next.config.mjs package.json postcss.config.js public/test.json README.md tailwind.config.ts tsconfig.json ``` ### Dependencies - package.json: @deepgram/sdk@^3.1.9, @instructor-ai/instructor@^0.0.6, @radix-ui/react-dialog@^1.0.5, @radix-ui/react-icons@^1.3.0, @radix-ui/react-label@^2.0.2, @radix-ui/react-slot@^1.0.2, @types/node@^20, @types/react@^18, @types/react-dom@^18, @typescript-eslint/eslint-plugin@^7.0.1, @typescript-eslint/parser@^7.0.1, autoprefixer@^10.0.1, class-variance-authority@^0.7.0, clsx@^2.1.0, eslint@^8, eslint-config-next@14.1.0, graphology@^0.25.4, lucide-react@^0.331.0, next@14.1.0, openai@^4.28.0, postcss@^8, react@^18, react-dom@^18, react-force-graph@^1.44.3, react-force-graph-3d@^1.24.2, react-graph-vis@^1.0.7, reagraph@^4.15.10, shadcn-ui@^0.8.0, sigma@^3.0.0-beta.6, tailwind-merge@^2.2.1, tailwindcss@^3.3.0, tailwindcss-animate@^1.0.7, three-spritetext@^1.8.2, typescript@^5, vaul@^0.9.0, zod@^3.22.4 ### Recent commits (newest first) - Add post parseData to local storage - Using dynamic transcript SLDKJFPSOIDJFPSOI - Add loading lSKDJFLKSJDFPHIUHSPDOFIJ - Merge branch 'transcription' - Merge branch 'main' of https://github.com/bztravis/ancestory - drawer - Add parseData - Add parse support - Add working transcription parse - Add working flattened transcription - Fix transcription to use latest recording - OMGGGGGGGG GRAPH THAT FOCUSES AND TEXT - Add basic transcription of previous message - working 3D node viewer - added comments - Modified prompt to include self in every character and put N/A for fields not caught - AI actually works - starting graph - starting graph - Add random stuff ## Key source files (fetched from GitHub, selected and truncated for size) ### package.json ``` { "name": "ancestory", "version": "0.1.0", "private": true, "scripts": { "dev": "next dev", "build": "next build", "start": "next start", "lint": "next lint" }, "dependencies": { "@deepgram/sdk": "^3.1.9", "@instructor-ai/instructor": "^0.0.6", "@radix-ui/react-dialog": "^1.0.5", "@radix-ui/react-icons": "^1.3.0", "@radix-ui/react-label": "^2.0.2", "@radix-ui/react-slot": "^1.0.2", "class-variance-authority": "^0.7.0", "clsx": "^2.1.0", "graphology": "^0.25.4", "lucide-react": "^0.331.0", "next": "14.1.0", "openai": "^4.28.0", "react": "^18", "react-dom": "^18", "react-force-graph": "^1.44.3", "react-force-graph-3d": "^1.24.2", "react-graph-vis": "^1.0.7", "reagraph": "^4.15.10", "shadcn-ui": "^0.8.0", "sigma": "^3.0.0-beta.6", "tailwind-merge": "^2.2.1", "tailwindcss-animate": "^1.0.7", "three-spritetext": "^1.8.2", "vaul": "^0.9.0", "zod": "^3.22.4" }, "devDependencies": { "@types/node": "^20", "@types/react": "^18", "@types/react-dom": "^18", "@typescript-eslint/eslint-plugin": "^7.0.1", "@typescript-eslint/parser": "^7.0.1", "autoprefixer": "^10.0.1", "eslint": "^8", "eslint-config-next": "14.1.0", "postcss": "^8", "tailwindcss": "^3.3.0", "typescript": "^5" } } ``` ### app/page.tsx ```typescript 'use client'; export default function Home() { return (
Home page: select a family member off the family tree.
); } ``` ### app/layout.tsx ```typescript import type { Metadata } from "next"; import { Inter } from "next/font/google"; import "./globals.css"; const inter = Inter({ subsets: ["latin"] }); export const metadata: Metadata = { title: "Create Next App", description: "Generated by create next app", }; export default function RootLayout({ children, }: Readonly<{ children: React.ReactNode; }>) { return ( {children} ); } ``` ### app/[relativeId]/page.tsx ```typescript "use client"; import { useCallback, useEffect, useRef } from 'react'; import dynamic from 'next/dynamic'; import Head from 'next/head'; //import Graph from '@/components/ancestree3D'; //import Graph from '@/components/ancestree3D-highlight'; //import Graph from '@/components/my-force-graph'; import FocusGraph from '@/components/focus-graph-wrapper'; export default function Page() { return (
); } // Since `react-force-graph-3d` might use browser-specific APIs, we dynamically import it to avoid SSR issues ``` ### app/api/transcription/route.ts ```typescript import { NextResponse, type NextRequest } from 'next/server'; const { createClient } = require('@deepgram/sdk'); const deepgram = createClient(process.env.DEEPGRAM_API_KEY); export const runtime = 'experimental-edge'; export async function POST(request: NextRequest) { const searchParams = request.nextUrl.searchParams; // query is "hello" for /api/search?query=hello // const audio; const formData = await request.formData(); const audio = formData.get('audio'); console.log('audio', audio); const arrayBuffer = await audio.arrayBuffer(); const buffer = Buffer.from(arrayBuffer); // console.log('buffer', buffer); const { result, error } = await deepgram.listen.prerecorded.transcribeFile( // path to the audio file buffer, // STEP 3: Configure Deepgram options for audio analysis { model: 'nova-2', smart_format: true, } ); const transcript = JSON.stringify( result.results.channels[0].alternatives[0].paragraphs.paragraphs .map((paragraph) => paragraph.sentences.map((sentence) => sentence.text).join(' ') ) .join(' ') ); return NextResponse.json({ transcript }); } ``` ### app/api/parse/route.ts ```typescript import { z } from 'zod'; import OpenAI from 'openai'; import Instructor from '@instructor-ai/instructor'; import { NextRequest, NextResponse } from 'next/server'; // client const togetherai = new OpenAI({ apiKey: process.env.TOGETHERAPI_KEY, baseURL: 'https://api.together.xyz/v1', }); // Instructor to get structured JSON response const client = Instructor({ client: togetherai, mode: 'JSON_SCHEMA', }); // Schema for information retrieval from transcript // TODO: character limit on title depending on the css of the graph const StorySchema = z.object({ title: z.string().describe('A short descriptive title of the story.'), summary: z .string() .describe('A short summary of the story narrated') .max(500), year: z.string().describe('The age at which this story took place').max(100), place: z.string().describe('The place the story is situated in'), characters: z .array(z.string()) .describe('A list of unique names of characters mentioned in the story'), }); // GET genAI response on information extracted from the story export async function GET(request: NextRequest) { // transcript dummy data const transcript = request.nextUrl.searchParams.get('transcript'); // Make the request to the AI try { const response = await client.chat.completions.create({ messages: [ { role: 'system', content: "The following is a transcript of an audio recording between an individual and their older family member. This audio recording is a story being shared by the older family member. Extract a title, a summary, and a list of characters (names of people) mentioned in the story. Return it as a JSON object in this format: {title: string, summary: string, characters: [string, string, string, ...]} For any field you didn't get data for from the audio, put the response as 'N/A' in a string. Make sure the 'speaker' is featured as a character in every story narrated. ", }, { role: 'user', content: transcript, }, ], model: 'mistralai/Mixtral-8x7B-Instruct-v0.1', response_model: { schema: StorySchema, name: 'AnalyzeStory', }, max_tokens: 1000, temperature: 0.6, }); // print the response // console.log(response); return NextResponse.json({ ...response }); } catch (error) { console.log('Failed to extract information from the transcript:', error); throw error; } } ``` ### app/[relativeId]/record/page.tsx ```typescript 'use client'; import AudioReactiveElement from '@/components/audio-reaction-2'; import { Card } from '@/components/ui/card'; import { useState, useRef } from 'react'; import { useRouter } from 'next/navigation'; import useLocalStorage from '@/hooks/useLocalStorage'; type Props = { params: { relativeId: string }; }; const Page = ({ params }: Props) => { const [recording, setRecording] = useState(false); const mediaRecorder = useRef(null); const [audioChunks, setAudioChunks] = useState([]); const [loading, setLoading] = useState(false); const [parseData, setParseData] = useState(null); const [setItem, getItem, removeItem] = useLocalStorage('stories', { nodes: [], links: [], }); const router = useRouter(); const getStream = async () => { console.log('perm'); if ('MediaRecorder' in window) { try { const streamData = await window.navigator.mediaDevices.getUserMedia({ audio: true, video: false, }); return streamData; } catch (err) { console.error(err); } } else { console.error('Microphone is not supported in your browser.'); } }; const startRecording = async () => { const streamData = await getStream(); if (!streamData) return; setRecording(true); const media = new MediaRecorder(streamData, { mimeType: 'audio/webm' }); mediaRecorder.current = media; mediaRecorder.current.start(); const localAudioChunks: Blob[] = []; mediaRecorder.current.ondataavailable = (event) => { if (typeof event.data === 'undefined') return; if (event.data.size === 0) return; localAudioChunks.push(event.data); }; setAudioChunks(localAudioChunks); }; const stopRecording = async () => { setRecording(false); if (!mediaRecorder.current) return; let audioUrl = null; mediaRecorder.current.onstop = async () => { const audioBlob1 = new Blob(audioChunks, { type: 'audio/webm' }); audioUrl = URL.createObjectURL(audioBlob1); setAudioChunks([]); console.log('audioUrl', audioUrl); const response = await fetch(`${audioUrl}`); const audioBlob = await response.blob(); const formData = new FormData(); formData.append('audio', audioBlob); formData.append('url', `blob:${audioUrl}`); console.log('audioBlob', audioBlob); const res = await fetch('/api/transcription', { method: 'POST', body: formData, }); const { transcript } = await res.json(); console.log('transcript', transcript); setLoading(true); const url = new URL('http://localhost:3000/api/parse'); url.searchParams.append('transcript', transcript); const parseRes = await fetch(url); const parseData = await parseRes.json(); console.log('parseData', parseData); setParseData(parseData); setLoading(false); }; mediaRecorder.current.stop(); }; return (
{loading ? (

Loading... {/* eslint-disable-next-line @next/next/no-img-element */} loading

) : !parseData ? ( <>

Record {params.relativeId}

) : (

{`${params.relativeId}'s Story:`}

{`${parseData.title}`}

{`${parseData.time}`}

{`${parseData.place}`}

{`Summary: ${parseData.summary}`}

)}
); function handleRecordButtonClick() { if (recording) stopRecording(); else startRecording(); } }; function getRandomIdsInRange(min, max, count) { const randomIds = []; while (randomIds.length < count) { const randomId = Math.floor(Math.random() * (max - min + 1)) + min; if (randomIds.indexOf(randomId) === -1) { randomIds.push(randomId); } } return randomIds; } export default Page; ``` ### postcss.config.js ```javascript module.exports = { plugins: { tailwindcss: {}, autoprefixer: {}, }, }; ``` ### tailwind.config.ts ```typescript import type { Config } from "tailwindcss"; const config: Config = { content: [ "./pages/**/*.{js,ts,jsx,tsx,mdx}", "./components/**/*.{js,ts,jsx,tsx,mdx}", "./app/**/*.{js,ts,jsx,tsx,mdx}", ], theme: { extend: { backgroundImage: { "gradient-radial": "radial-gradient(var(--tw-gradient-stops))", "gradient-conic": "conic-gradient(from 180deg at 50% 50%, var(--tw-gradient-stops))", }, }, }, plugins: [require("tailwindcss-animate")], }; export default config; ``` ### components/ancestree2D.tsx ```typescript import React from 'react' const Graph2D = () => { return (
Graph2D
) } export default Graph2D ``` [18 more indexed source files omitted to keep this export small. The full file list is in the Codebase structure section above.]