# Project export: KronAI This document was generated by HackStack to give an AI agent context about a hackathon project. Sections are labeled with their provenance; content marked as truncated was cut to keep this document small. ## Project metadata - Hackathon: Cal Hacks 11.0 - Tagline: Snap to Schedule: Your Personal AI Scheduling Assistant. For students, young professionals, parents, and more. - Devpost: https://devpost.com/software/kronai - GitHub: https://github.com/Preellis/CalHacks11 - Demo: http://kronai.co/ - Video: https://www.youtube.com/embed/G_J-KsUNKqE?enablejsapi=1&hl=en_US&rel=0&start=&version=3&wmode=transparent - Team: 4 GitHub contributor(s) — Preston Ellis (42 commits), thisismyamazingusername (19 commits), Michael Li (16 commits), Edwardcg17 (14 commits) ## Devpost submission (written by the team) ### Inspiration Have you ever received a concert flyer or a graphical timeline, hoping to add it to your calendar, but finding it too tedious to add them event to event. This is the exact problem we are inspired by and want to solve. ### What it does What KronAI does is it drastically simplifies this workflow by combining every action into one click. All you need to do is take a picture, screenshot, or upload from gallery, and KronAI will take care of the scheduling for you. But KronAI is more than that. Utilizing Perplexity and ChromaDB, KronAI's scheduling assistant can capture the nuances of your schedule just like your own personal secretary! ### How we built it We built the software using Next.js, integrating with Gemini, Perplexity, Google Calendar, and Chromadb. Oauth is used for login. ### Challenges we ran into We ran into challenges when we tried to integrate APIs such as Google's OAuth and calendar. ### Accomplishments we're proud of We are proud that when we talk to people about our idea, many people say they experience the exact problem we are solving and say that they would love to use a product like this. This means we are working in the right direction, and we are proud to work on something that solve the little inconveniences in life. ### What we learned We learned a lot about the process of how to integrate the APIs we used into our application. ### What's next The benefit of speed and easy calendar integration can be leveraged in the healthcare space as patients who are trying new medications like during the adhd process or for diseases or antibiotics. By scanning their prescription box and officially adding health to their productivity calendar, they can see when to take their medications alongside other priorities. More technically, integration with G-mail to autodetect events and convert into bite sized todo lists or boost action over information via push notifs can be helpful. Potentially turning into a personal executive assistant for students and for you page with events near you in a tinder like fashion based on what can fit or requires adjusting your schedule. ## README (from the GitHub repository) # KronAI (CalHacks 11.0) ![gallery](https://github.com/user-attachments/assets/a242bd05-266c-4547-841c-c8a74ab24da5) ## Inspiration Have you ever received a concert flyer or a graphical timeline, hoping to add it to your calendar, but finding it too tedious to add them event to event. This is the exact problem we are inspired by and want to solve. ## What it does What KronAI does is it drastically simplifies this workflow by combining every action into one click. All you need to do is take a picture, screenshot, or upload from gallery, and KronAI will take care of the scheduling for you. But KronAI is more than that. Utilizing Perplexity and ChromaDB, KronAI's scheduling assistant can capture the nuances of your schedule just like your own personal secretary! ## How we built it We built the software using Next.js, integrating with Gemini, Perplexity, Google Calendar, and Chromadb. Oauth is used for login. ## Challenges we ran into We ran into challenges when we tried to integrate APIs such as Google's OAuth and calendar. ## Accomplishments that we're proud of We are proud that when we talk to people about our idea, many people say they experience the exact problem we are solving and say that they would love to use a product like this. This means we are working in the right direction, and we are proud to work on something that solve the little inconveniences in life. ## What we learned We learned a lot about the process of how to integrate the APIs we used into our application. ## What's next for KronAI Integration with G-mail to autodetect events and schedule events automatically. ## Detected evidence (automated analysis) Indexed codebase: 20 recognized source files, 875 KB. - Google Gemini (technology) — detected in the code - JavaScript (language) — detected in the code - Next.js (technology) — detected in the code - OpenAI (technology) — detected in the code - React (technology) — detected in the code - Tailwind CSS (technology) — detected in the code - TypeScript (language) — detected in the code ## Codebase structure (from repository index) ### Files (35 of 35) ``` README.md schedulr/.env.example schedulr/.eslintrc.json schedulr/.gitignore schedulr/next.config.mjs schedulr/package.json schedulr/postcss.config.cjs schedulr/postcss.config.mjs schedulr/server.webpack.lock schedulr/server.webpack.lock.data/https_unpkg.com/xenova_transformers_2.13.2_dist_transformers.min_08e533521b68610f4b72.js schedulr/src/app/_colorscheme.scss schedulr/src/app/api/chroma/add-memory/route.ts schedulr/src/app/api/chroma/get-context/route.ts schedulr/src/app/api/gemini/image-to-calendar/route.ts schedulr/src/app/api/gemini/route.ts schedulr/src/app/api/perplexity/route.ts schedulr/src/app/chat/page.tsx schedulr/src/app/globals.scss schedulr/src/app/home/page.tsx schedulr/src/app/layout.tsx schedulr/src/app/mainFunctionScreen/page.tsx schedulr/src/app/mainFunctionScreen/styles.module.scss schedulr/src/app/page.tsx schedulr/src/app/styles.module.scss schedulr/src/atoms/index.ts schedulr/src/commons/context/menuContext.tsx schedulr/src/components/footer/Footer.jsx schedulr/src/components/footer/styles.module.scss schedulr/src/components/navbar/Navbar.tsx schedulr/src/components/navbar/styles.module.scss schedulr/src/components/test-deepgram/page.tsx schedulr/src/components/upload/UploadModal.tsx schedulr/src/utils/eventEnhancer.ts schedulr/tailwind.config.ts schedulr/tsconfig.json ``` ### Dependencies - schedulr/package.json: @deepgram/sdk@^3.8.1, @google-cloud/storage@^7.13.0, @google/generative-ai@^0.21.0, @mantine/carousel@^7.13.3, @mantine/core@^7.13.3, @mantine/dates@^7.13.3, @mantine/dropzone@^7.13.3, @mantine/hooks@^7.13.3, @mantine/modals@^7.13.3, @mantine/nprogress@^7.13.3, @mantine/spotlight@^7.13.3, @react-oauth/google@^0.12.1, @tabler/icons-react@^3.19.0, @types/node@20.16.13, @types/react@^18, @types/react-dom@^18, @types/ws@^8.5.12, @xenova/transformers@^2.13.2, axios@^1.7.7, base64-arraybuffer@^1.0.2, chromadb@^1.9.2, chromadb-default-embed@^2.13.2, dayjs@^1.11.13, deepgram@^1.0.3, dotenv@^16.4.5, embla-carousel-react@^7.1.0, eslint@^8, eslint-config-next@14.2.15, fs@^0.0.1-security, googleapis@^144.0.0, jotai@^2.10.1, jwt-decode@^4.0.0, next@14.2.15, next-auth@^4.24.8, openai@^4.68.1, postcss@^8.4.47, postcss-preset-mantine@^1.17.0, postcss-simple-vars@^7.0.1, react@^18, react-camera-pro@^1.4.0, react-dom@^18, sass@^1.80.3, socket.io-client@^4.8.0, styled-components@^6.1.13, tailwindcss@^3.4.1, typescript@5.6.3, webpack@^5.95.0, ws@^8.18.0 ### Recent commits (newest first) - Update README.md - Update README.md - Update README.md - add perplexity - lint - lint - lint - lint - lint - fix chat - chat is chatty asf - responds w 3rd context but preston said not to show it - Merge branch 'main' of github.com:Preellis/CalHacks11 - almost there for chat - Delete page.tsx - Update page.tsx - lint - lint - lint - fix footer functionality upload ## Key source files (fetched from GitHub, selected and truncated for size) ### schedulr/package.json ``` { "name": "schedulr", "version": "0.1.0", "private": true, "scripts": { "dev": "next dev", "build": "next build", "start": "next start", "lint": "next lint" }, "dependencies": { "@deepgram/sdk": "^3.8.1", "@google-cloud/storage": "^7.13.0", "@google/generative-ai": "^0.21.0", "@mantine/carousel": "^7.13.3", "@mantine/core": "^7.13.3", "@mantine/dates": "^7.13.3", "@mantine/dropzone": "^7.13.3", "@mantine/hooks": "^7.13.3", "@mantine/modals": "^7.13.3", "@mantine/nprogress": "^7.13.3", "@mantine/spotlight": "^7.13.3", "@react-oauth/google": "^0.12.1", "@tabler/icons-react": "^3.19.0", "@xenova/transformers": "^2.13.2", "axios": "^1.7.7", "base64-arraybuffer": "^1.0.2", "chromadb": "^1.9.2", "chromadb-default-embed": "^2.13.2", "dayjs": "^1.11.13", "deepgram": "^1.0.3", "dotenv": "^16.4.5", "embla-carousel-react": "^7.1.0", "fs": "^0.0.1-security", "googleapis": "^144.0.0", "jotai": "^2.10.1", "jwt-decode": "^4.0.0", "next": "14.2.15", "next-auth": "^4.24.8", "openai": "^4.68.1", "react": "^18", "react-camera-pro": "^1.4.0", "react-dom": "^18", "sass": "^1.80.3", "socket.io-client": "^4.8.0", "styled-components": "^6.1.13", "webpack": "^5.95.0", "ws": "^8.18.0" }, "devDependencies": { "@types/node": "20.16.13", "@types/react": "^18", "@types/react-dom": "^18", "@types/ws": "^8.5.12", "eslint": "^8", "eslint-config-next": "14.2.15", "postcss": "^8.4.47", "postcss-preset-mantine": "^1.17.0", "postcss-simple-vars": "^7.0.1", "tailwindcss": "^3.4.1", "typescript": "5.6.3" } } ``` ### schedulr/src/atoms/index.ts ```typescript import { atom } from 'jotai'; import { TokenResponse } from '@react-oauth/google'; export const userAtom = atom(null); export const userIdAtom = atom('test'); export const imageAtom = atom(null); ``` ### schedulr/src/app/layout.tsx ```typescript "use client"; import "./globals.scss"; import { GoogleOAuthProvider } from '@react-oauth/google'; // export const metadata: Metadata = { // title: "Create Next App", // description: "Generated by create next app", // }; export default function RootLayout({ children, }: Readonly<{ children: React.ReactNode; }>) { return ( {children} ); } ``` ### schedulr/src/app/page.tsx ```typescript "use client"; import { useState, useEffect } from 'react'; import { useRouter } from 'next/navigation'; import { useGoogleLogin } from '@react-oauth/google'; import axios from 'axios'; import styles from "./styles.module.scss"; import { useAtom, useSetAtom } from 'jotai'; import { userAtom, userIdAtom } from '@/atoms'; export default function Home() { const [user, setUser] = useAtom(userAtom); const setUserId = useSetAtom(userIdAtom); const [loggedIn, setLoggedIn] = useState(false); const router = useRouter(); const login = useGoogleLogin({ onSuccess: async (tokenResponse) => { console.log(tokenResponse); setUser(tokenResponse); setLoggedIn(true); // Fetch user info to get the unique ID try { const userInfo = await axios.get('https://www.googleapis.com/oauth2/v3/userinfo', { headers: { Authorization: `Bearer ${tokenResponse.access_token}` } }); setUserId(userInfo.data.sub); // 'sub' is the unique Google user ID console.log('User ID:', userInfo.data.sub); } catch (error) { console.error('Error fetching user info:', error); } }, scope: 'https://www.googleapis.com/auth/calendar.events https://www.googleapis.com/auth/userinfo.profile', onError: () => console.log('Login Failed'), }); // Redirect to another page when eventAdded becomes true useEffect(() => { if (loggedIn) { router.push('/mainFunctionScreen'); // Redirect to the desired page } }, [loggedIn, router]); return (

KronAI

Plan Smarter,

Schedule Faster,

Powered by AI.

{user ? ( ) : ( <> )}
); } ``` ### schedulr/src/app/home/page.tsx ```typescript import React from 'react' export default function page() { return ( <> ) } ``` ### schedulr/src/app/mainFunctionScreen/page.tsx ```typescript "use client" import React, { useState, useRef } from 'react'; import { userAtom, userIdAtom } from '@/atoms'; import { useAtomValue } from 'jotai'; import Navbar from '@/components/navbar/Navbar'; import Footer from '@/components/footer/Footer'; import styles from './styles.module.scss'; import { Camera, CameraType } from 'react-camera-pro'; import { MenuContext } from '@/commons/context/menuContext'; import axios from 'axios'; import { eventEnhancer } from '@/utils/eventEnhancer'; export default function MainFunctionScreen() { const camera = useRef(null); const user = useAtomValue(userAtom); const userId = useAtomValue(userIdAtom); const [menuStatus, setMenuStatus] = useState(false); const errorMessages = { noCamera: "No camera detected. Please check your device.", permissionDenied: "Camera permission denied. Please enable it.", }; if (typeof window !== "undefined" && window?.navigator?.mediaDevices) { console.log("Browser supports camera access"); } else { console.log("Browser does not support camera access"); } //Function that handles entire functionality of camera button const HandleCamBtn = async () => { if (camera.current) { const photo = camera.current.takePhoto(); const res = await axios.post("/api/gemini/image-to-calendar", { base64Image: photo }) const initialEvent = JSON.parse(res.data.response.replace(/```json/g, '').replace(/```/g, '')) const enhancedEvent = await eventEnhancer(initialEvent, user, userId) console.log(enhancedEvent) } }; return (

Previously Added Events

Tropical Purple Party
The Back Room
Tangerine Tumble
Flamingo Night
setMenuStatus(false)} className={menuStatus ? `${styles.darkBackground} ${styles.active}` : `${styles.darkBackground}`}/>
); }; ``` ### schedulr/src/app/chat/page.tsx ```typescript "use client"; import { useState } from 'react'; import { useAtomValue } from 'jotai'; import axios from 'axios'; import { userAtom, userIdAtom } from '@/atoms'; import { CalendarEvent, eventEnhancer } from '@/utils/eventEnhancer'; const ChatPage = () => { const [messages, setMessages] = useState([]); const [userInput, setUserInput] = useState(''); const [loading, setLoading] = useState(false); const userId = useAtomValue(userIdAtom); const user = useAtomValue(userAtom); const handleSendMessage = async () => { if (!userInput.trim()) return; setLoading(true); setMessages((prevMessages) => [...prevMessages, `You: ${userInput}`]); try { // Step 1: Save user input to Chroma DB await axios.post('/api/chroma/add-memory', { userId, documentText: userInput, }); // Step 2: Fetch context from Chroma DB const contextResponse = await axios.post('/api/chroma/get-context', { userId, query: userInput, }); const contextResult = contextResponse.data; // Step 3: Ask Gemini if it's an event or not const geminiResponse = await axios.post('/api/gemini', { prompt: `Is "${userInput}" a public event? If yes, The .json file should be in the format of 'eventname', 'description', 'start', 'end', 'location', 'context'. The 'start' and 'end' should be in ISO string timestamp format. 'context' should contain anything that may help you figure out more about the event in searches. If any values are not found, return null for that category in .json. If not an event, return an invalid json`, // Using userInput directly }); console.log(geminiResponse.data.response); let jsonEvent: CalendarEvent | string = "" try { jsonEvent = JSON.parse(geminiResponse.data.response.split("```json")[1].split("```")[0]); } catch (e) { console.log(e); jsonEvent = ""; } // Step 4: Check if an event is detected if (!jsonEvent) { // If no event detected, use the userInput as a normal conversation prompt const followUpResponse = await axios.post('/api/gemini', { prompt: `Based on the message: "${userInput}", how would you respond in a conversational way? Do not give multiple possible responses, only 1. Always try to help the users schedule something. If they give you an event, possibly search it up and suggest adding it to gcal. Use this context: ${contextResult}`, }); const followUpChatResponse = followUpResponse.data.response; setMessages((prevMessages) => [ ...prevMessages, `KronAI: ${followUpChatResponse || "I'm not sure about that."}`, ]); } else { // Event detected, proceed to add to Google Calendar const enhancedEvent = await eventEnhancer(jsonEvent as CalendarEvent, user, userId); console.log(enhancedEvent); setMessages((prevMessages) => [ ...prevMessages, 'KronAI: The event has been added to your Google Calendar.', ]); } } catch (error) { console.error('Error:', error); setMessages((prevMessages) => [ ...prevMessages, 'System: Something went wrong, please try again.', ]); } finally { setLoading(false); setUserInput(''); } }; return (
{messages.map((message, index) => (
{message}
))}
setUserInput(e.target.value)} placeholder="Type your message here..." disabled={loading} />
); }; export default ChatPage; ``` ### schedulr/src/app/api/gemini/route.ts ```typescript import { NextRequest, NextResponse } from "next/server"; import { GoogleGenerativeAI } from "@google/generative-ai"; export async function POST(request: NextRequest): Promise { const apiKey = process.env.GEMINI_API!; const genAI = new GoogleGenerativeAI(apiKey); const model = genAI.getGenerativeModel({ model: "gemini-1.5-flash" }); try { const { prompt } = await request.json(); const result = await model.generateContent(prompt); const response = result.response; const text = response.text(); return NextResponse.json({ response: text }); } catch (error) { console.error("Error generating content:", error); return NextResponse.json({ error: "An error occurred when parsing text" }, { status: 500 }); } } ``` ### schedulr/src/app/api/perplexity/route.ts ```typescript import { NextRequest, NextResponse } from "next/server"; import OpenAI from "openai"; import { ChatCompletionMessageParam } from "openai/resources/index.mjs"; export async function POST(request: NextRequest): Promise { const { prompt } = await request.json(); const apiKey = process.env.PERPLEXITY_API!; const messages = [ { role: "user", content: `You are supposed to provide context from the web for a user's query about an event. Try to find the time, location, and other details about the event but do not make up any information or make assumptions. Prompt: ${prompt}` }, ] const client = new OpenAI({ apiKey: apiKey, baseURL: "https://api.perplexity.ai", }); try { const response = await client.chat.completions.create({ model: "llama-3.1-sonar-small-128k-online", messages: messages as ChatCompletionMessageParam[], }); console.log(response) return NextResponse.json({ response: response.choices[0].message.content }); } catch (error) { console.error("Error generating content:", error); return NextResponse.json({ error: "An error occurred when parsing text" }, { status: 500 }); } } ``` ### schedulr/src/app/api/chroma/add-memory/route.ts ```typescript import { ChromaClient, GoogleGenerativeAiEmbeddingFunction } from "chromadb"; import { randomUUID } from "crypto"; import { NextRequest, NextResponse } from "next/server"; export async function POST(request: NextRequest): Promise { const { userId, documentText } = await request.json(); const embedder = new GoogleGenerativeAiEmbeddingFunction({ googleApiKey: process.env.GEMINI_API!, }); const chromaClient = new ChromaClient({ path: "http://34.102.41.119:8000" }); const collection = await chromaClient.getOrCreateCollection({ name: userId, embeddingFunction: embedder, }); await collection.add({ ids: [randomUUID()], documents: [documentText], metadatas: [{ createdAt: new Date().toISOString(), type: "memory" }], }); return NextResponse.json({ success: true }); } ``` [9 more indexed source files omitted to keep this export small. The full file list is in the Codebase structure section above.]