# Project export: Zina

This document was generated by HackStack to give an AI agent context about a hackathon project. Sections are labeled with their provenance; content marked as truncated was cut to keep this document small.

## Project metadata

- Hackathon: Cal Hacks 11.0
- Tagline: A collaborative learning environment led by AI, Zina offers inclusive, personalized education, adhering to best academic practices and providing quality education to underserved communities.
- Devpost: https://devpost.com/software/zina-0q5aws
- GitHub: https://github.com/noahk004/zina
- Team: 4 GitHub contributor(s) — Neet (34 commits), FishReddy (26 commits), Noah Kim (15 commits), Rohan Kumar (10 commits)

## Devpost submission (written by the team)

### Inspiration

Through meeting hundreds and hundreds of learners from all over the world in our time as university students, we have slowly realized over time the variability that exists in education systems around the world. After many conversations, it is apparent that in many areas, teenagers and young adults do not have anywhere near the quality of education that one may find at an accredited four-year university-- an experience that requires thousands of dollars to have. With such a boom for artificial intelligence and research in educational best practices, we came up with a way to allow for all those who yearn for this learning to have it. As such, we sought to build an AI-powered tutoring with a human touch to create an engaging, community-based, learning experience. The goal is to help students, no matter where they are, to get real-time guidance through a platform that is not just instructional, but also one that is engaging and that pushes students to want to learn more. We believed that our combined experiences in machine learning and fullstack development was fit for the job.

### What it does

Zina is a personalized group instructor that uses video and audio sessions to guide student learning. Students can generate course topics of their choice and be in a classroom with peers of their choice, putting the power of learning into the hands of the students. Our application supports features such as lesson plan generation, live video streaming, questions and answers.

### How we built it

Tools: Next.js + React.js Tailwind CSS Flask Socket.IO WebRTC Deepgram Agora Video API Chroma DB Multimodal Data Langchain

### Challenges we ran into

One of the biggest challenges was ensuring speech to text and text to speech communication is effective and engaging. Handling real-time communication across various network speeds was tricky, and balancing performance with AI processing for live transcription and analysis was another technical hurdle. Integrating the Deepgram API and ensuring the seamless operation of AI services without lag was the main point of challenge here. Additionally, ensuring the platform worked equally well across different devices and operating systems required extensive testing and optimization.

### Accomplishments we're proud of

We are proud of the seamless integration between AI-powered transcription, real-time feedback, and live tutoring sessions. The combination of voice recognition and real-time analysis to help tutors adjust lessons based on students' needs was a significant accomplishment.

### What we learned

The importance of scalability in real-time applications. Building for low latency while incorporating AI services that require heavy computational resources taught us how to optimize both frontend and backend processes in short time frames. We also gained insights into how students and tutors interact in virtual settings, helping us design a more intuitive user experience. This project also reinforced our knowledge of WebRTC and how to leverage AI to enhance human interaction in education.

### What's next

Next, we hope to expand on our philosophy of quality education by engaging in continuous research in education practices to ensure that students' needs are met. Due to the limited time constraints, we were forced to cut corners, which made us all the more eager to go the full lengths to make this project complete.

## README (from the GitHub repository)

No README available.

## Detected evidence (automated analysis)

Indexed codebase: 34 recognized source files, 79 KB.
- CSS (language) — detected in the code
- Flask (technology) — detected in the code
- HTML (language) — detected in the code
- JavaScript (language) — detected in the code
- LangChain (technology) — detected in the code
- Next.js (technology) — detected in the code
- Python (language) — detected in the code
- React (technology) — detected in the code
- Tailwind CSS (technology) — detected in the code

## Codebase structure (from repository index)

### Files (42 of 42)

```
.gitignore
backend/client.html
backend/readme.md
backend/requirements.txt
backend/server.py
frontend/.gitignore
frontend/jsconfig.json
frontend/next.config.mjs
frontend/package.json
frontend/postcss.config.mjs
frontend/README.md
frontend/src/app/client.html
frontend/src/app/context/DeepgramContextProvider.js
frontend/src/app/courses/[id]/classroom/loading.js
frontend/src/app/courses/[id]/classroom/page.js
frontend/src/app/courses/[id]/classroom/realpage.js
frontend/src/app/courses/[id]/classroom/temppage.js
frontend/src/app/courses/[id]/classroom/Visualizer.js
frontend/src/app/courses/[id]/classroom/VoiceToText.js
frontend/src/app/courses/[id]/loading.js
frontend/src/app/courses/[id]/lobby/loading.js
frontend/src/app/courses/[id]/lobby/LocalUserComponent.js
frontend/src/app/courses/[id]/lobby/page.js
frontend/src/app/courses/[id]/lobby/RemoteUserComponent.js
frontend/src/app/courses/[id]/page.js
frontend/src/app/courses/create/loading.js
frontend/src/app/courses/create/page.js
frontend/src/app/courses/loading.js
frontend/src/app/courses/page.js
frontend/src/app/globals.css
frontend/src/app/layout.js
frontend/src/app/loading.js
frontend/src/app/page.js
frontend/src/components/agora/AccessToken2.js
frontend/src/components/agora/data.js
frontend/src/components/agora/RtcTokenBuilder2.js
frontend/src/components/agora/tokenGenerator.js
frontend/src/components/agora/uidGenerator.js
frontend/src/components/loading/loading.js
frontend/src/components/navigation/Navigation.js
frontend/tailwind.config.js
package.json
```

### Dependencies

- backend/requirements.txt: beautifulsoup4, chromadb, deepgram-sdk, eventlet, flask, langchain, langchain_community, langchain-chroma, langchain-mistralai, open-clip-torch, pillow, Pillow, pydub, python-socketio, requests
- frontend/package.json: @deepgram/sdk@^3.8.1, @radix-ui/react-icons@^1.3.0, agora-rtc-react@^2.3.0, cross-fetch@^4.0.0, dotenv@^16.4.5, framer-motion@^11.11.9, next@14.2.14, postcss@^8, react@^18, react-dom@^18, react-speakup@^1.1.2, react-speech-recognition@^3.10.0, react-spinners@^0.14.1, regenerator-runtime@^0.14.1, socket.io-client@^4.8.0, tailwindcss@^3.4.1, typewriter-effect@^2.21.0
- package.json: framer-motion@^11.11.9

### Recent commits (newest first)

- final
- adding final changes
- Merge branch 'Neet'
- Neet page issue fix
- Merge remote-tracking branch 'origin/Vishwa' into Neet
- Added Overflow
- Merge remote-tracking branch 'origin/noah/readme' into Neet
- buttons
- Merge remote-tracking branch 'origin/noah/readme' into Vishwa
- Merge remote-tracking branch 'origin/Neet' into Vishwa
- Merge branch 'main' into Vishwa
- changes to backend
- add gif
- Merge branch 'main' into Neet
- Merge branch 'main' into Neet
- Merge pull request #9 from noahk004/noah/update-ui
- Merge branch 'main' into noah/update-ui
- changes
- Merge remote-tracking branch 'origin/noah/update-ui' into Neet
- Merge pull request #8 from noahk004/rkumar/text-to-speech

## Key source files (fetched from GitHub, selected and truncated for size)

### package.json

```
{
  "dependencies": {
    "framer-motion": "^11.11.9"
  }
}

```

### backend/requirements.txt

```
flask
python-socketio
eventlet
pillow
pydub
beautifulsoup4
requests
Pillow
chromadb
langchain
langchain-chroma
langchain-mistralai
open-clip-torch
langchain_community
deepgram-sdk
```

### frontend/package.json

```
{
  "name": "frontend",
  "version": "0.1.0",
  "private": true,
  "scripts": {
    "dev": "next dev",
    "build": "next build",
    "start": "next start",
    "lint": "next lint"
  },
  "dependencies": {
    "@deepgram/sdk": "^3.8.1",
    "@radix-ui/react-icons": "^1.3.0",
    "agora-rtc-react": "^2.3.0",
    "cross-fetch": "^4.0.0",
    "dotenv": "^16.4.5",
    "framer-motion": "^11.11.9",
    "next": "14.2.14",
    "react": "^18",
    "react-dom": "^18",
    "react-speakup": "^1.1.2",
    "react-speech-recognition": "^3.10.0",
    "react-spinners": "^0.14.1",
    "regenerator-runtime": "^0.14.1",
    "socket.io-client": "^4.8.0",
    "typewriter-effect": "^2.21.0"
  },
  "devDependencies": {
    "postcss": "^8",
    "tailwindcss": "^3.4.1"
  }
}

```

### backend/server.py

```python
# server.py
import socketio
import chromadb
import requests
import getpass
import os
import httpx
from dotenv import load_dotenv
import threading

from deepgram import (
    DeepgramClient,
    LiveTranscriptionEvents,
    LiveOptions,
    SpeakOptions
)

from bs4 import BeautifulSoup
from urllib.parse import urljoin
from flask import Flask, render_template
from chromadb.utils.embedding_functions import OpenCLIPEmbeddingFunction
from chromadb.utils.data_loaders import ImageLoader
from langchain import hub
from langchain_chroma import Chroma
from langchain_core.output_parsers import StrOutputParser
from langchain_core.runnables import RunnablePassthrough, RunnableLambda
from langchain_text_splitters import RecursiveCharacterTextSplitter
from langchain.embeddings.base import Embeddings
from chromadb.utils.embedding_functions import OpenCLIPEmbeddingFunction
from langchain.docstore.document import Document
from langchain.memory import ConversationBufferMemory
from langchain_community.chat_message_histories import ChatMessageHistory#from langchain.memory import ChatMessageHistory
from langchain_core.prompts import ChatPromptTemplate, MessagesPlaceholder
from langchain.prompts import PromptTemplate
from langchain_mistralai import ChatMistralAI

load_dotenv()

URL = "http://localhost:3000/courses/data-structures/classroom"

API_KEY = os.getenv("5eea543264e20ce240e0496ee8ed7abebab66d1d")



course_outline = {
    'data_structures': ['Classification of Data Structures', 'Introducing Queue', 'Introducing Stack']
}
data_loader = ImageLoader()
client = chromadb.Client()
os.environ["MISTRAL_API_KEY"] = 'dmyWQ4LGbojOTdrlvWtwujkmLSScUQGo'
llm = ChatMistralAI(model="mistral-large-latest")

def main():
    try:
        # STEP 1: Create a Deepgram client using the API key
        deepgram = DeepgramClient(API_KEY)

        # STEP 2: Create a websocket connection to Deepgram
        dg_connection = deepgram.listen.live.v("1")

        # STEP 3: Define the event handlers for the connection
        def on_message(self, result, **kwargs):
            sentence = result.channel.alternatives[0].transcript
            if len(sentence) == 0:
                return
            print(f"speaker: {sentence}")

        def on_metadata(self, metadata, **kwargs):
            print(f"\n\n{metadata}\n\n")

        def on_error(self, error, **kwargs):
            print(f"\n\n{error}\n\n")

        # STEP 4: Register the event handlers
        dg_connection.on(LiveTranscriptionEvents.Transcript, on_message)
        dg_connection.on(LiveTranscriptionEvents.Metadata, on_metadata)
        dg_connection.on(LiveTranscriptionEvents.Error, on_error)

        # STEP 5: Configure Deepgram options for live transcription
        options = LiveOptions(
            model="nova-2", 
            language="en-US", 
            smart_format=True,
            )
        
        # STEP 6: Start the connection
        dg_connection.start(options)

        # STEP 7: Create a lock and a flag for thread synchronization
        lock_exit = threading.Lock()
        exit = False

        # STEP 8: Define a thread that streams the audio and sends it to Deepgram
        def myThread():
            with httpx.stream("GET", URL) as r:
                for data in r.iter_bytes():
                    lock_exit.acquire()
                    if exit:
                        break
                    lock_exit.release()

                    dg_connection.send(data)

        # STEP 9: Start the thread
        myHttp = threading.Thread(target=myThread)
        myHttp.start()

        # STEP 10: Wait for user input to stop recording
        input("Press Enter to stop recording...\n\n")

        # STEP 11: Set the exit flag to True to stop the thread
        lock_exit.acquire()
        exit = True
        lock_exit.release()

        # STEP 12: Wait for the thread to finish
        myHttp.join()

        # STEP 13: Close the connection to Deepgram
        dg_connection.finish()

        print("Finished")

    except Exception as e:
        print(f"Could not open socket: {e}")
        return

def scrape_text_and_images(url):
    # Fetch the content of the URL
    response = requests.get(url)
    html_content = response.content

    # Parse the HTML content
    soup = BeautifulSoup(html_content, 'html.parser')

    # # Remove <script> and <style> elements
    # for script_or_style in soup(['script', 'style']):
    #     script_or_style.decompose()

    # Extract text
    text = soup.get_text(separator=' ', strip=True)

    # Extract image URLs
    image_urls = []
    for img in soup.find_all('img'):
        img_src = img.get('src')
        if img_src:
            # Handle relative URLs
            img_url = urljoin(url, img_src)
            image_urls.append(img_url)

    return text, image_urls

class ChromaOpenCLIPEmbeddings(Embeddings):
    def __init__(self):
        self.embedding_function = OpenCLIPEmbeddingFunction()

    def embed_documents(self, texts):
        # Chroma's embedding function can handle lists of texts
        return self.embedding_function(texts)

    def embed_query(self, text):
        # For a single query, wrap it in a list and extract the first result
        return self.embedding_function([text])[0]

embeddings = ChromaOpenCLIPEmbeddings()
vectorstore = Chroma(
    collection_name='text_collection',
    embedding_function=embeddings,
)
image_collection = client.get_or_create_collection(
    name="image_collection",
    embedding_function=OpenCLIPEmbeddingFunction()
)
urls = ['https://www.programiz.com/dsa/queue', 'https://www.programiz.com/dsa/stack', 'https://www.geeksforgeeks.org/introduction-to-data-structures/']
for url in urls:
    text, image_urls = scrape_text_and_images(url)
    doc = Document(page_content=text, metadata={'source': url})
    text_splitter = RecursiveCharacterTextSplitter(chunk_size=2000, chunk_overlap=200)
    splits = text_splitter.split_documents([doc])
    text_chunks = [split.page_content for split in splits
[truncated — 6362 more characters]
```

### frontend/src/app/layout.js

```javascript
import { Afacad } from 'next/font/google'
import "./globals.css";

const font = Afacad({
  subsets: ['latin'],
  display: 'swap',
})

export const metadata = {
  title: "Zina",
  description: "Generated by create next app",
};

export default function RootLayout({
  children,
}) {
  return (
    <html lang="en" className={font.className}>
      <body>{children}</body>
    </html>
  )
}

```

### frontend/src/app/page.js

```javascript
"use client";

import { motion } from "framer-motion";

import Typewriter from "typewriter-effect";

import Image from "next/image";
import Link from "next/link";

import { useState } from "react";

import Navigation from "@/components/navigation/Navigation.js";

import landing from "@/media/landing.png";
import logoWhite from "@/media/logo-white.png";

export default function Page() {
  const [prompt, setPrompt] = useState("");

  return (
    <div>
      <Navigation />
      <div className="w-screen translate-y-[25vh] flex flex-col justify-center items-center">
        <div className="font-semibold text-6xl text-purpleLight mb-7 flex items-end">
          &nbsp;
          <Typewriter
            onInit={(typewriter) => {
              typewriter
                .pauseFor(1000)
                .typeString("I want to learn about the human mind.")
                .pauseFor(2500)
                .deleteChars(16)
                .pauseFor(200)
                .typeString(" creative writing.")
                .pauseFor(2500)
                .deleteChars(18)
                .pauseFor(200)
                .typeString(" music history.")
                .pauseFor(2500)
                .deleteAll()
                .pauseFor(200)

                .typeString("Give me a lesson on linked lists.")
                .pauseFor(2500)
                .deleteChars(14)
                .pauseFor(200)
                .typeString(" protein synthesis.")
                .pauseFor(2500)
                .deleteChars(19)
                .pauseFor(200)
                .typeString(" thermodynamics.")
                .pauseFor(2500)
                .deleteAll()
                .pauseFor(200)

                .typeString("Show me the process of launching a new product.")
                .pauseFor(2500)
                .deleteChars(25)
                .pauseFor(200)
                .typeString(" deploying an EC2 instance.")
                .pauseFor(2500)
                .deleteChars(27)
                .pauseFor(200)
                .typeString(" training a neural network.")
                .pauseFor(2500)
                .deleteAll()
                .start();
            }}
            options={{
              loop: true,
              delay: 50,
              deleteSpeed: 30,
            }}
          />&nbsp;
        </div>
        <div className="flex gap-4 mb-4">
          <input
            value={prompt}
            onChange={(e) => setPrompt(e.target.value)}
            placeholder="What would you like to learn today?"
            className="w-[600px] p-3 rounded-full text-lg px-6"
          />
          <button
            onClick={() => {}}
            className="text-lg p-3 bg-purpleLight rounded-full duration-200 hover:bg-white hover:shadow-xl px-5 disabled:bg-gray-200 disabled:text-gray-400 disabled:hover:shadow-none"
            disabled={!prompt}
          >
            Let's go!
          </button>
        </div>
        <div>
          <Image src={logoWhite} alt="logo" className="w-20" />
        </div>
      </div>
      <motion.div initial={{ opacity: 0 }} animate={{ opacity: 1 }}>
        <Image
          src={landing}
          alt="landing design"
          className="absolute w-screen bottom-0"
        />
      </motion.div>
    </div>
  );
}

```

### frontend/src/app/courses/page.js

```javascript
import Navigation from "@/components/navigation/Navigation"

import Link from "next/link"
import { PlusIcon } from "@radix-ui/react-icons"

export default function Page() {
    return (
        <div>
            <Navigation />
            <div className="mt-[100px] mx-[150px]">
                <div className="flex justify-between">
                <h2 className="font-semibold text-3xl text-white">Active Courses</h2>
                <Link href="/courses/create" className="flex items-center text-purple2 gap-1 rounded-full bg-white px-4 py-2 hover:bg-purpleLight duration-200">
                    <PlusIcon className="w-6 h-6" />
                    <span className="font-medium text-lg">New Course</span>
                </Link>
                </div>
            </div>

        </div>
    )
}
```

### frontend/src/app/courses/[id]/page.js

```javascript
"use client";

import { useParams } from "next/navigation";

import Link from "next/link";

import Navigation from "@/components/navigation/Navigation";

export default function Page() {
  const { id } = useParams();

  return (
    <div>
      <Navigation />
      <div className="w-screen flex justify-center py-[100px]">
        <div className="rounded-lg w-[1000px] bg-white px-7 py-7">
          <h1 className="text-3xl font-bold mb-4">Course {id}</h1>

            <div className="mb-3">
                <div>
                Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat. Duis aute irure dolor in reprehenderit in voluptate velit esse cillum dolore eu fugiat nulla pariatur. Excepteur sint occaecat cupidatat non proident, sunt in culpa qui officia deserunt mollit anim id est laborum.
                </div>
                <div>
                Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat. Duis aute irure dolor in reprehenderit in voluptate velit esse cillum dolore eu fugiat nulla pariatur. Excepteur sint occaecat cupidatat non proident, sunt in culpa qui officia deserunt mollit anim id est laborum.
                </div>
            </div>
          <div className="flex gap-2 text-lg justify-end">
            <Link
              href="/courses"
              className="bg-white rounded-lg px-4 py-1 border-2 border-purpleLight hover:bg-purpleLight duration-200"
            >
              Back
            </Link>
            <Link
              href={`/courses/${id}/lobby`}
              className="bg-purple3 text-white rounded-lg px-4 py-1 hover:bg-purple1 duration-200"
            >
              Continue
            </Link>
          </div>
        </div>
      </div>
    </div>
  );
}

```

### frontend/src/app/courses/create/page.js

```javascript
"use client";

import { useState } from "react";

import { useRouter } from "next/navigation";

import Navigation from "@/components/navigation/Navigation";

export default function Page() {
  const [prompt, setPrompt] = useState("");

    const getParam = (str) => {
        str = str.toLowerCase()
        let param = str;
        if (str.includes("data".toLowerCase()) && str.includes("structures".toLowerCase())) {
          param = "data_structures"
        } else if (str.includes("stacks".toLowerCase())) {
            param = "stacks"
        } else if (str.includes("queues".toLowerCase())) {
            param = "queues"
        }
        return param
    }


  return (
    <div>
      <Navigation />
      <div className="mx-[150px] mt-[100px]">
        <h2 className="font-semibold text-3xl text-white">Create new course</h2>
        <p className="text-lg font-light text-purpleLight mb-4">
          Enter a topic and generate your course within minutes!
        </p>
        <div>
          <textarea
            value={prompt}
            onChange={(e) => setPrompt(e.target.value)}
            placeholder="What would you like to learn today?"
            className=" mb-6 rounded-lg text-lg px-4 py-3 w-[600px] h-[200px] bg-white"
          />
        </div>
        {
            prompt ? <a
            href={`/courses/${getParam(prompt)}/lobby`}
            className="transition-all duration-300 bg-gradient-to-r to-purple1 via-purple3 from-purple2 bg-size-200 disabled:border-purple2 hover:bg-pos-100 px-4 py-2 rounded-full border-2 border-white text-white disabled:text-gray-400 disabled:pointer-events-none hover:drop-shadow-[0_0_10px_rgba(255,255,255,.75)]"
          >
            Create
          </a>
          : <button
          href={`/courses/${getParam(prompt)}/lobby`}
          className="transition-all duration-300 bg-gradient-to-r to-purple1 via-purple3 from-purple2 bg-size-200 disabled:border-purple2 hover:bg-pos-100 px-4 py-2 rounded-full border-2 border-white text-white disabled:text-gray-400 disabled:pointer-events-none hover:drop-shadow-[0_0_10px_rgba(255,255,255,.75)]"
            disabled
        >
          Create
        </button>
        }
        
      </div>
    </div>
  );
}

```

### frontend/src/app/courses/[id]/classroom/realpage.js

```javascript
"use client";

import { useState } from "react";
import { appId, channelName, token } from "@/components/agora/data";

import { useJoin, useRemoteUsers } from "agora-rtc-react";

import LocalUserComponent from "../lobby/LocalUserComponent";
import RemoteUserComponent from "../lobby/RemoteUserComponent";

import AgoraRTC, { AgoraRTCProvider } from "agora-rtc-react";

import { CaretDownIcon, CaretRightIcon } from "@radix-ui/react-icons";

export default function Page() {
  const client = AgoraRTC.createClient({ mode: "rtc", codec: "vp8" });

  return (
    <AgoraRTCProvider client={client}>
      <Classroom />
    </AgoraRTCProvider>
  );
}

function Classroom() {
  const [stageIndex, setStateIndex] = useState(0);
  const [sectionList, setSectionList] = useState(null);

  // ------ AGORA -------
  const [calling, setCalling] = useState(true);
  useJoin(
    { appid: appId, channel: channelName, token: token ? token : null },
    calling
  );
  const remoteUsers = useRemoteUsers();
  // ------ AGORA -------

  // ------ COMPONENTS -------
  const [camerasVisible, setCamerasVisible] = useState(true);
  // ------ COMPONENTS -------


  return (
    <div className="w-screen flex justify-between">
      <div></div>
      <div className="absolute bomax-w-[600px] bg-black text-white text-sm bottom-[30px]">
        Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod
        tempor incididunt ut labore et dolore magna aliqua.
      </div>

      <div
        className="py-[30px] pr-[30px]"
      >
        <button onClick={() => setCamerasVisible(!camerasVisible)} className="w-fit">
          {camerasVisible ? (
            <CaretRightIcon className="w-5 h-5 text-purple3" />
          ) : (
            <CaretDownIcon
              className="w-5 h-5 text-purple3"
            />
          )}
        </button>
        <div className={`flex flex-col gap-2 j-fit ${camerasVisible ? "visible" : "invisible"}`}>
        <LocalUserComponent />
        {remoteUsers.map((user) => (
          <RemoteUserComponent key={user.uid} name={user.uid} />
        ))}
        </div>
        
      </div>
    </div>
  );
}

```

[25 more indexed source files omitted to keep this export small. The full file list is in the Codebase structure section above.]