https://drive.google.com/drive/u/0/folders/1XKlzUc1PADYSHSm9W1Nnx540w5vlVJGt

Inspiration

We are visual learners. Purely text-based learning leaves us quickly feeling sleepy and disinterested. Textbooks provide as much utility as a rough pillow. Whether through pestering our professors, coworkers, and friends, the only way we can grasp concepts is through interactive learning. Thus, we build a system where given a codebase, documentation, textbook, or novel can produce visual elements as well as talking us through the concepts. Our system can also pause, answer questions, and modify the presentation on the fly giving endless opportunities for long fascinating tangents.

What it does

Given access to your code base, documentation materials, and a prompt. Pitcher can create a powerpoint presentation that contains relevant images/information from documentation you gave and also custom data visualization charts. Pitcher allows you to showcasing your project in whichever angle you want. Moreover, Pitcher will present to you in its own voice the project that you included. When interrupted verbally Pitcher will stop to answer questions while simultaneously generating more slides to answer your question as well as directly citing information from the codebase/documentation. Key features: generates slides from codebase and documents generates and includes contextual images and data visualizations auto-presents with automatic voiceover Allows user to interrupt and ask questions Generates additional slides for clarification after questions are asked.

How we built it

To extract the essential information from the input sources (github, google drive) we developed a chunking system that allows us to take full advantage of the context length of LLMs. This chunking system allows us to extract essential statistics and central ideas from multiple documentation files concurrently. We also built tools to extract and analyze images within pdfs as well as summarization tools to understand the functionality of entire code bases. All this information is vectorized into a IRIS database and is stored for later RAG. The summarized input sources are then used to design and build powerpoint presentations. To automate this process we employ MARP, a tool that allows us to create powerpoint presentations with markdown files. The ideas are passed through multiple sets of AI agents. The first set finds the ideal location for text making sure to leave enough space for future images while brainstorming ideas of what visual graphs or photos are best for the situation. Before the next pass this graphics’s information is autonomously retrieved and built by another array of agents based on the framework Mermaid. The presentations are then passed through the build agents that add the images and modify the presentation to be more visually appealing. This presentations are then rendered out and displayed to the user via an iframe. The presentation as well as speaker notes generated during the last round of agentic building are then passed to a reactive Text to speech system which responds to interruptions with relevant information thanks to the RAG system of the entire project.

Challenges we ran into

One of the challenges we ran into was the sheer scale of the entire project. With multiple sets of agent agents, an immense amount of computational logic needed to extract essential detail and summaries of code bases, RAG system and database management software as well as interactive interfaces needed to provide a solid user experience. This is a huge project. What helped us overcome this challenge was deciding on our project vision early with extremely productive ideation sessions. We also effectively delegated our work with each member taking up a clear piece of the project.

What's next

We plan to scale this project to service thousands of visual learners. We also plan to improve the capabilities of our agentic system to produce more accurate presentation information as well as give more precise verbal responses.