# Project export: wonderlands. This document was generated by HackStack to give an AI agent context about a hackathon project. Sections are labeled with their provenance; content marked as truncated was cut to keep this document small. ## Project metadata - Hackathon: Cal Hacks 11.0 - Tagline: Build the world of your wildest dreams with your favorite ebooks, one adventure at a time. - Devpost: https://devpost.com/software/wonderland-ai - GitHub: https://github.com/Calhacks11/BookChat - Video: https://www.youtube.com/embed/yRPenS-slso?enablejsapi=1&hl=en_US&rel=0&start=&version=3&wmode=transparent - Team: 1 GitHub contributor(s) — Amit Thakur (7 commits) ## Devpost submission (written by the team) ### How we built it Python 3 Fast API+ + PyQT, React/NextJS + Google gemini + Deepgram SDK + Hume AI SDK ### Challenges we ran into Finalizing an initial roadmap Internet connection disrupted our flow NextJS integrations and endpoints ### Accomplishments we're proud of We made reading an ebook file (EPUB) into engaging live-speaking characters with their traits (Gender, Tone etc) with whom the listener can interact using their voice. We wrote a cross-platform app (Desktop and Web) and utilized multiple services: Google Gemini for the intelligence, Deepgram Voice API for Text to Speech, and HumeAI for Live Interaction with the characters. We used multimodality of text and speech (with tone and expressions) to generate an engaging experience. Live highlighting of live speech with our own code. ### What we learned Got familiar with the potential of modern voice agents. We understood how we can leverage the intelligence of LLMs and expressiveness capturing the power of the various voice APIs. A ton about Web Application architectures, UI framework, and integration of speech APIs with modern apps. Future of entertainment and education technologies. ### What's next The goal for wonderlands is to become a multimodal, cross platform and engaging service leveraging GENAI for entertainment, knowledge and human prosperity, bringing you art from the abstract. ## README (from the GitHub repository) # BookApp ## Develop Environment Setup - Install conda. Follow instructions [here](https://conda.io/projects/conda/en/latest/user-guide/install/index.html). - Create conda environment: ```commandline conda create --name book-chat python=3.11 conda activate book-chat pip install -r requirements.txt ``` ## Run Backend ```commandline cd BookChat fastapi dev main.py ``` ## Detected evidence (automated analysis) Indexed codebase: 18 recognized source files, 46 KB. - HTML (language) — detected in the code - JavaScript (language) — detected in the code - Python (language) — detected in the code - FastAPI (technology) — claimed on Devpost, not found in the code - Google Gemini (technology) — claimed on Devpost, not found in the code - Next.js (technology) — claimed on Devpost, not found in the code ## Codebase structure (from repository index) ### Files (30 of 30) ``` .gitignore .idea/.gitignore .idea/BookChat.iml .idea/inspectionProfiles/profiles_settings.xml .idea/inspectionProfiles/Project_Default.xml .idea/misc.xml .idea/modules.xml .idea/vcs.xml app.py data/conversation.epub data/conversation/ch-1.txt epub_parser/conversation.epub epub_parser/epub_parser.py epub_parser/epub_to_text.py epub_parser/output.txt epub_parser/romeo-and-juliet.epub epub_parser/romeo-juliet.epub epub_parser/shaw-caesar-and-cleopatra.epub llm/gemini.py llm/llm_queries.py main.py prompts/conversation-1.txt prompts/conversation.txt README.md voice/deepgram_util.py voice/dialog_utils.py voice/hume_util.py web/deepgram.html web/index.html web/script.js ``` ### Dependencies No dependency index available. ### Recent commits (newest first) - Updated the character names and few cosmetic changes - Added Qt GUI - Added JSONIzed prompt responses - Added Gemini - Added web page with chapter fetch - Added chapter fetch - Added conda env and README - Initial commit ## Key source files (fetched from GitHub, selected and truncated for size) ### main.py ```python from fastapi import FastAPI, File, UploadFile import shutil from epub_parser.epub_to_text import epub_to_chapters, get_chapter from fastapi.middleware.cors import CORSMiddleware from pydantic import BaseModel from typing import Union from llm.gemini import query_prompt app = FastAPI() origins = [ "http://localhost", "http://localhost:8080", "http://localhost:4200", "http://127.0.0.1:4200" ] app.add_middleware( CORSMiddleware, allow_origins=["*"], allow_credentials=True, allow_methods=["*"], allow_headers=["*"], ) class LLMQuery(BaseModel): query: Union[str, None] = None @app.get("/") def read_root(): return {"Hello": "World"} @app.post("/uploadbook") async def create_upload_file(file: UploadFile = File(...)): # Specify the path to save the file save_path = f'./data/{file.filename}' # Write the uploaded file to the local disk with open(save_path, "wb") as buffer: shutil.copyfileobj(file.file, buffer) epub_to_chapters(file.filename) return {"filename": file.filename, "status": "File uploaded successfully"} @app.get("/book/{bookname}/chapter/{ch_no}") async def read_chapter(bookname: str, ch_no: int): content = get_chapter(bookname, ch_no) return { "bookname": bookname, "ch_no": ch_no, "content": content } @app.post("/llm-query") async def llm_query(llm_query: LLMQuery): llm_response = query_prompt(llm_query.query) return { "response": llm_response } ``` ### app.py ```python import sys import time from PyQt6.QtWidgets import QApplication, QLabel, QWidget, QGridLayout, QPushButton, QFileDialog, QVBoxLayout, QLineEdit import os from epub_parser.epub_to_text import epub_to_chapters, get_chapter from llm.llm_queries import query_characters from voice.dialog_utils import rotate_character_narration from voice.deepgram_util import speak class ChapterWindow(QWidget): def __init__(self, bookname): super().__init__() self.bookname = bookname self.initUI() def initUI(self): self.layout = QVBoxLayout() content = get_chapter(self.bookname, 1) status = QLabel('---', self) self.layout.addWidget(status) self.setLayout(self.layout) self.setWindowTitle('Chapter Window') self.setGeometry(300, 300, 400, 200) status.setText('Understanding the characters...') self.characters = query_characters(content) print(self.characters) status.setText('Characters are ready!') self.n_dialogs = sum([len(character['dialogues']) for character in self.characters]) self.n_characters = len(self.characters) self.labels = [] for i in range(self.n_dialogs): i_character = i % self.n_characters character = self.characters[i_character] jth_dialog = int(i / self.n_characters) label = QLabel(f"{character['character_name']}: {character['dialogues'][jth_dialog]}") self.labels.append(label) self.layout.addWidget(label) # rotate_character_narration(characters) self.play_char_btn = QPushButton(f"Play Characters") self.play_char_btn.clicked.connect(self.play_btn_click) self.layout.addWidget(self.play_char_btn) # self.close() # print(prompt_response) def play_btn_click(self): self.rotate_character_narration() self.play_char_btn.setEnabled(False) def rotate_character_narration(self): for i in range(self.n_dialogs): i_character = i % self.n_characters character = self.characters[i_character] jth_dialog = int(i / self.n_characters) self.labels[i].setStyleSheet("background-color: yellow; color: black;") app.processEvents() speak(character['dialogues'][jth_dialog], character['gender']) class FileUploadApp(QWidget): def __init__(self): super().__init__() self.initUI() def initUI(self): # Set the layout layout = QVBoxLayout() # Create a button to open the file dialog self.upload_button = QPushButton('Upload epub File', self) self.upload_button.clicked.connect(self.openFileDialog) # Create a label to display the selected file path self.label = QLabel('No file selected', self) # Create a button to save the file to the specified path self.save_button = QPushButton('Save File', self) self.save_button.clicked.connect(self.saveFile) self.save_button.setEnabled(False) # Disable save button until a file is selected # Add widgets to layout layout.addWidget(self.upload_button) layout.addWidget(self.label) layout.addWidget(self.save_button) # Set the main layout self.setLayout(layout) # Set window properties self.setWindowTitle('File Upload') self.setGeometry(300, 300, 400, 200) self.file_bytes = None # Variable to store the file bytes def openFileDialog(self): # Open the file dialog file_name, _ = QFileDialog.getOpenFileName(self, 'Open File', '', 'All Files (*)') # If a file is selected if file_name: self.label.setText(f'Selected .epub file: {file_name}') self.selected_file_path = file_name # Read the file in binary mode and get the bytes with open(file_name, 'rb') as file: self.file_bytes = file.read() self.save_button.setEnabled(True) # Enable save button after file is selected else: self.label.setText('No file selected') self.save_button.setEnabled(False) # Disable save button if no file is selected def saveFile(self): # Get the new file path from the input field file_name = os.path.splitext(os.path.basename(self.selected_file_path))[0] new_file_path = f'data/{file_name}.epub' self.label.setText(f'Uploading and parsing the .epub file...') if new_file_path and self.file_bytes: try: # Write the file bytes to the new path with open(new_file_path, 'wb') as new_file: new_file.write(self.file_bytes) self.label.setText(f'File saved to: {new_file_path}') epub_to_chapters(f'{file_name}.epub') time.sleep(1) self.chapter_window = ChapterWindow(file_name) self.chapter_window.show() self.close() except Exception as e: self.label.setText(f'Error saving file: {str(e)}') else: self.label.setText('No file selected or no file path specified') # Create an instance of QApplication app = QApplication([]) # window = QWidget() # window.setWindowTitle("Ebook Reader App") # window.setGeometry(300, 200, 300, 200) # # layout = QGridLayout() # # helloMsg = QLabel("

Upload the .epub file

", parent=window) # helloMsg.move(60, 15) # # window.show() # Run the application's event loop ex = FileUploadApp() ex.show() sys.exit(app.exec()) ``` ### .idea/vcs.xml ```xml ``` ### .idea/modules.xml ```xml ``` ### .idea/misc.xml ```xml ``` ### epub_parser/epub_parser.py ```python from epub2txt import epub2txt # from a url to epub # url = "https://github.com/ffreemt/tmx2epub/raw/master/tests/1.tmx.epub" # res = epub2txt(url, outputlist=True) # # from a local epub file filepath = 'shaw-caesar-and-cleopatra.epub' res = epub2txt(filepath) # output as a list of chapters # ch_list = epub2txt(filepath, outputlist=True) # chapter titles will be available as epub2txt.content_titles if available print(res) ``` ### voice/dialog_utils.py ```python from voice.deepgram_util import speak def rotate_character_narration(characters): n_dialogs = sum([len(character['dialogues']) for character in characters]) n_characters = len(characters) for i in range(n_dialogs): i_character = i % n_characters character = characters[i_character] jth_dialog = int(i / n_characters) speak(character['dialogues'][jth_dialog], character['gender']) ``` ### llm/llm_queries.py ```python from llm.gemini import query_prompt import ast def query_characters(content): query = content + ( '\nTell me the unique characters, their possible genders, and personality attributes in the above conversation.' '\nGive me the response in the following python dictionary schema:\n' 'Character = {"character_name" : str, "gender": "male/female", "personality": list[str], "dialogues": list[str]}\n' 'Return: list[Character]\n' 'where dialogues is a list of dialogues spoken by respective characters.' ) prompt_response = query_prompt(query) first_ind = prompt_response.find('[') last_ind = prompt_response.rfind(']') prompt_response = prompt_response[first_ind:last_ind + 1] prompt_response = ast.literal_eval(prompt_response) return prompt_response ``` ### llm/gemini.py ```python import google.generativeai as genai from dotenv import load_dotenv import os load_dotenv() GOOGLE_API_KEY = os.getenv("GOOGLE_API_KEY") genai.configure(api_key=GOOGLE_API_KEY) prompt_file = '/Users/amitthakur/PycharmProjects/BookChat/prompts/conversation-1.txt' def query_prompt(query_lines): model = genai.GenerativeModel('gemini-1.5-flash') chat = model.start_chat(history=[]) # lines = read_prompt(prompt_file) response = chat.send_message(query_lines) return response.text def read_prompt(file_path): with open(file_path, 'r') as file: # Read all lines into a list lines = file.readlines() return lines def main(): model = genai.GenerativeModel('gemini-1.5-flash') chat = model.start_chat(history=[]) lines = read_prompt(prompt_file) response = chat.send_message(lines) print(response.text) if __name__ == '__main__': main() ``` ### epub_parser/epub_to_text.py ```python import ebooklib from ebooklib import epub from bs4 import BeautifulSoup import os def get_chapter(bookname: str, ch_no: int): chapter_file = f'data/{bookname}/ch-{ch_no}.txt' with open(chapter_file, 'r') as file: content = file.read() return content def epub_to_chapters(epub_file_name): # Read the EPUB file epub_file = f'data/{epub_file_name}' book = epub.read_epub(epub_file) book_dir = epub_file.removesuffix(".epub") os.makedirs(book_dir, exist_ok=True) # Initialize an empty list to store the text chapters = [] # Loop through the items in the book count = 0 for i, item in enumerate(book.get_items()): # print('-----type-------------', item.get_type()) if item.get_type() == ebooklib.ITEM_DOCUMENT: # count += 1 # Parse the document content with BeautifulSoup soup = BeautifulSoup(item.get_body_content(), 'html.parser') # Append the text to the list, stripped of HTML tags chapter = soup.get_text() # print('-------text length--------', len(text)) chapters.append(chapter) # print(chapter) with open(f'{book_dir}/ch-{i + 1}.txt', 'w', encoding='utf-8') as output_file: output_file.write(chapter) # Join all the text content into a single string # return '\n'.join(chapters) return chapters ``` [7 more indexed source files omitted to keep this export small. The full file list is in the Codebase structure section above.]