# Project export: Chopsticks - AI powered Video Editor This document was generated by HackStack to give an AI agent context about a hackathon project. Sections are labeled with their provenance; content marked as truncated was cut to keep this document small. ## Project metadata - Hackathon: TreeHacks 2024 - Tagline: Streamline video editing for content creators with AI-powered software armed with text-based edit queries and automatic identification and compilation of engaging livestream moments. - Devpost: https://devpost.com/software/clipgenius-ai-powered-streamedit - GitHub: https://github.com/Shrey1306/chopsticks - Demo: https://chopsticks.streamlit.app/ - Video: https://www.youtube.com/embed/R-r3VN00dOs?enablejsapi=1&hl=en_US&rel=0&start=&version=3&wmode=transparent - Team: 1 GitHub contributor(s) — Abhishek Pillai (3 commits) ## Devpost submission (written by the team) ### Inspiration We live in a digital world fueled and filled with more content than ever. After conducting extensive market research with numerous Twitch and YouTube creators, we stumbled upon a rather niche issue. Content creators face tremendous difficulty when having to edit lengthy videos; Their tools are often designed to be used by experts. To account for this, many have to waste a lot of time learning tools or resort to outsourcing their work. In addition to this, the user experience with certain editing software often feels archaic with a disproportionate amount of tools provided to the user. ### What it does In comes Chopsticks, the premier AI-powered editing software that utilizes deep learning to improve efficiency, enhance user experience, and (amazingly) increase creator profits. Our platform consists of a dual-use system: one chat-powered end where we take user text queries and perform video manipulation, and another where we analyze the most entertaining and important parts of a video (based on developed metrics) and present the user with many different clips of short-form content. With that being said, Chopsticks is a first-of-its-kind software coming to market, and here is how it works: ### How we built it Retrieves voice transcription using Whisper, chat logs using OCR/Web Scraping, and creator expressions using OpenCV. Uses Roberta's fine-tuned model to analyze the viewership engagement (chats) with the creator, and its sentiment. Uses a fine-tuned T5 model to transcribe the stream and run a text-text analysis to gauge the streamer's key moments. Uses a DeepFace model to read the streamer's reaction and weigh this metric with the models used above to produce valuable insights into key moments of a stream. 5 We normalize these metrics and generate "spikes" that occur at certain time intervals, representing high levels of engagement between the streamer and the viewer. This data is fed into our LLM-based video clipping tool, to autogenerate or chop clips into short-form content. Using Reflex, we created a simple user interface that allows users of any level to be able to edit their videos seamlessly. ### Challenges we ran into One of our most complex challenges was regarding how to determine whether something was "entertaining" or not. When dealing with human emotions, classifying data in a meaningful way becomes less boolean and harder to quantify. To overcome some of this friction, we spent a lot of time identifying relevant factors that contribute to this metric. We decided to give custom weights to certain inputs (chats being the highest since we have more consistent data to rely on), leading to an overall better model. Another big problem we faced technically was the memory and time needed to classify our inputs. For our first run with a 30-minute video, our combined time to gather transcription data, chat logs, and facial emotion recognition data took us well over an hour. Thinking about the consumer, we realized this wouldn't be sustainable in the long run so we cleaned up our algorithms, ignoring certain data to significantly reduce overhead. Recently, we were able to classify this large video in less than 20 minutes by running scripts at the same time and using better hardware. Coming into TreeHacks we initially were on track to pursue a project that analyzes research papers for beginner researchers. When we talked to a mentor here (shoutout to Luke), he asked us the hard, but important questions. When we discussed who our consumers would be and the real use case of our product, we realized that maybe research was a track we didn't want to pursue. 4 hours into hacking we got back to the drawing board and went about choosing a project a different way. ### Accomplishments we're proud of Although coming up with an idea on the spot started very difficult, we approached the idea by first conducting heavy market research in many different fields which then led us down the content creation path. Coming across this hump in our journey was not only a breath of relief, but it also provided us with a newfound motivation to put all our effort into a singular goal that we all believed had potential. On top of this, we all as a team have grown tremendously in the technical space. Being introduced to new sponsor technologies like Reflex, we were able to create a compelling web app using only python. Lastly, our proudest moment was when our first output was generated. We had selected a random Pewdiepie Minecraft stream and when we saw the quality of the short format videos generated, we knew that all the work we had put in was not in vain and our project indeed had a future. ### What we learned Coming in with a diverse range of skill sets, a quick thing we failed to grasp on our first night was splitting work efficiently. When we had our first team meeting the day after, we split up work better, allowing members who are proficient at doing something to create quality work in that area. This reduced our workload (still 2 all-nighters) and allowed us to get significantly more work done. ### What's next for Chopsticks? As a potentially (very) successful startup, our goal for Chopsticks is to push directly into the market. One big constraint we had on us during TreeHacks was simply time. Our models were efficient but sometimes didn't classify our inputs perfectly. By having the time to fine-tune our custom models, generate better metrics for clips, and reduce overhead, we will be able to scale our company quickly and efficiently beat everyone to market. We hope to launch initially as open-source software to gain traction in the industry but then transition to a subscription-based model, which will allow us to pay for new hardware required to run our algorithm as fast as possible. In terms of pure concept, our company has the potential to do good in our community. Not only is our product significantly cheaper than our direct customers, but our software has limitless applications to do social good, especially in the education space. By being able to quickly extract important bits of lectures into viewable content, students with short attention spans could easily learn content without being bored to death. We hope to secure funding for this idea so we can keep spending time on a project we are all so passionate about. ## README (from the GitHub repository) # TreeHacks2024 tbh: cool ## Detected evidence (automated analysis) Indexed codebase: 37 recognized source files, 2686 KB. - CSS (language) — detected in the code - JavaScript (language) — detected in the code - Next.js (technology) — detected in the code - OpenAI (technology) — detected in the code - Python (language) — detected in the code - React (technology) — detected in the code - Tailwind CSS (technology) — detected in the code - Hugging Face (technology) — claimed on Devpost, not found in the code ## Codebase structure (from repository index) ### Files (47 of 47) ``` .devcontainer/devcontainer.json .gitignore .web/.gitignore .web/components/reflex/chakra_color_mode_provider.js .web/components/reflex/radix_themes_color_mode_provider.js .web/env.json .web/jsconfig.json .web/next.config.js .web/package.json .web/pages/_app.js .web/pages/_document.js .web/pages/404.js .web/pages/index.js .web/postcss.config.js .web/public/custom_video_controls.js .web/reflex.install_frontend_packages.cached .web/reflex.json .web/styles/styles.css .web/styles/tailwind.css .web/tailwind.config.js .web/utils/client_side_routing.js .web/utils/components.js .web/utils/context.js .web/utils/helpers/dataeditor.js .web/utils/helpers/range.js .web/utils/state.js .web/utils/stateful_components.js .web/utils/theme.js assets/custom_video_controls.js editing_functions.py Final_whisperaitranscribe.ipynb get-pip.py README.md requirements.txt rxconfig.py streamlit_app.py webui/__init__.py webui/components/__init__.py webui/components/chat.py webui/components/loading_icon.py webui/components/modal.py webui/components/navbar.py webui/components/sidebar.py webui/components/videodisplay.py webui/state.py webui/styles.py webui/webui.py ``` ### Dependencies - .web/package.json: @chakra-ui/icons@^2.0.19, @chakra-ui/react@^2.6.1, @chakra-ui/system@^2.5.7, @emotion/react@11.11.1, @radix-ui/themes@^2.0.0, autoprefixer@10.4.14, axios@1.4.0, framer-motion@^10.16.4, json5@2.2.3, next@14.0.1, next-sitemap@4.1.8, next-themes@0.2.0, postcss@8.4.31, react@18.2.0, react-dom@18.2.0, react-dropzone@^14.2.3, react-loading-icons@^1.1.0, socket.io-client@4.6.1, tailwindcss@^3.3.2, universal-cookie@4.0.4 - requirements.txt: openai@==0.28, reflex@>=0.2.0 ### Recent commits (newest first) - changes - working - Initial commit ## Key source files (fetched from GitHub, selected and truncated for size) ### requirements.txt ``` reflex>=0.2.0 openai==0.28 ``` ### .web/package.json ``` { "name": "reflex", "scripts": { "dev": "next dev", "export": "next build", "export-sitemap": "next build && next-sitemap", "prod": "next start" }, "dependencies": { "@chakra-ui/icons": "^2.0.19", "@chakra-ui/react": "^2.6.1", "@chakra-ui/system": "^2.5.7", "@emotion/react": "11.11.1", "@radix-ui/themes": "^2.0.0", "axios": "1.4.0", "framer-motion": "^10.16.4", "json5": "2.2.3", "next": "14.0.1", "next-sitemap": "4.1.8", "next-themes": "0.2.0", "react": "18.2.0", "react-dom": "18.2.0", "react-dropzone": "^14.2.3", "react-loading-icons": "^1.1.0", "socket.io-client": "4.6.1", "tailwindcss": "^3.3.2", "universal-cookie": "4.0.4" }, "devDependencies": { "autoprefixer": "10.4.14", "postcss": "8.4.31" } } ``` ### .web/pages/index.js ```javascript /** @jsxImportSource @emotion/react */ import { Fragment, useCallback, useContext, useRef } from "react" import { EventLoopContext, StateContexts, UploadFilesContext } from "/utils/context" import { Event, getBackendURL, getRefValue, getRefValues, isTrue, refs, set_val } from "/utils/state" import { Button as RadixThemesButton, Dialog as RadixThemesDialog, Flex as RadixThemesFlex, Heading as RadixThemesHeading, ScrollArea as RadixThemesScrollArea, Text as RadixThemesText, Theme as RadixThemesTheme } from "@radix-ui/themes" import env from "/env.json" import { Avatar, Box, Breadcrumb, BreadcrumbItem, Button, Drawer, DrawerBody, DrawerContent, DrawerHeader, DrawerOverlay, Flex, FormControl, Heading, HStack, Image as ChakraImage, Input, Link, Menu, MenuButton, MenuDivider, MenuItem, MenuList, Modal, ModalBody, ModalContent, ModalFooter, ModalHeader, ModalOverlay, Text, VStack } from "@chakra-ui/react" import Script from "next/script" import { CloseIcon, DeleteIcon, HamburgerIcon } from "@chakra-ui/icons" import NextLink from "next/link" import ReactDropzone from "react-dropzone" import "@radix-ui/themes/styles.css" import theme from "/utils/theme.js" import { SpinningCircles } from "react-loading-icons" import NextHead from "next/head" export function Closeicon_11ed883525187cdaad471aef955f22dd () { const [addEvents, connectError] = useContext(EventLoopContext); const on_click_2905983f8758758258aab6a80fcc9a4c = useCallback((_e) => addEvents([Event("state.state.toggle_drawer", {})], (_e), {}), [addEvents, Event]) return ( ) } export function Flex_90098c846fefa7a78ae981fcb13da1e4 () { const [filesById, setFilesById] = useContext(UploadFilesContext); return ( {(filesById.default ? filesById.default.map((f) => (f.path || f.name)) : []).map((children, props) => ( {children} ))} ) } export function Box_97d1f8abdc2c15e2135cd714633c9ae3 () { const handleSubmit_a6fb427b7adf0f7db4d9b28324ef7e6f = useCallback((ev) => { const $form = ev.target ev.preventDefault() const form_data = {...Object.fromEntries(new FormData($form).entries()), ...{"question": getRefValue(refs['ref_question'])}} addEvents([Event("state.state.process_question", {form_data:form_data})]) if (true) { $form.reset() } }) const [addEvents, connectError] = useContext(EventLoopContext); return ( ) } export function Formcontrol_939eaeffc75d989a7e4dacd3ee8319c9 () { const state__state = useContext(StateContexts.state__state) const ref_question = useRef(null); refs['ref_question'] = ref_question; return ( ) } export function Button_38c35384972e8b7c4fe387f6d5088b12 () { const [addEvents, connectError] = useContext(EventLoopContext); const on_click_44ea287a5fe43aff33d47baeab2ad68c = useCallback((_e) => addEvents([Event("_call_script", {javascript_code:`refs['__clear_selected_files']('default')`})], (_e), {}), [addEvents, Event]) return ( {`Clear`} ) } export function Button_8ae269be87b4d8ff7e1648aeec904542 () { const [addEvents, connectError] = useContext(EventLoopContext); const [filesById, setFilesById] = useContext(UploadFilesContext); const on_click_4e2d8f20ed65f854cae117923adec0c7 = useCallback((_e) => addEvents([Event("state.state.handle_upload", {files:filesById.default,upload_id:`default`}, "uploadFiles")], (_e), {}), [addEvents, Event, filesById, setFilesById]) return ( {`Upload`} ) } export function Box_ac1bd614b21a3bd657f85f0ce95694e7 () { const state__state = useContext(StateContexts.state__state) return ( {state__state.chats[state__state.current_chat].map((qa, index_2f0915031b59136cfd0724124f2efcb1) => ( {qa.question} {qa.answer} ))} ) } export function Button_3d311a7261a303c7b1a9953fd4706e2e () { const [addEvents, connectError] = useContext(EventLoopContext); const on_click_65775bd3c3ca6de4793090251b518aa6 [truncated — 13339 more characters] ``` ### streamlit_app.py ```python ``` ### rxconfig.py ```python import reflex as rx config = rx.Config( app_name="webui", ) ``` ### editing_functions.py ```python import sys from PyQt5.QtWidgets import QApplication, QWidget, QVBoxLayout, QLabel, QLineEdit, QMessageBox, QPushButton, QFileDialog from PyQt5.QtGui import QImage, QPixmap from PyQt5.QtCore import Qt, QThread, pyqtSignal import cv2 import numpy as np import tempfile import os import ffmpeg class VideoProcessor(QThread): update_signal = pyqtSignal(QImage) finished = pyqtSignal() def __init__(self, video_path): super().__init__() self.original_video_path = video_path self.video_path = video_path self.video_history = [video_path] # Initialize history with the original path self.running = True self.trim_required = False self.start_sec = 0 self.end_sec = 0 def run(self): while self.running: if self.trim_required: self.trim_video(self.start_sec, self.end_sec) self.trim_required = False self.finished.emit() self.play_video() #Trimming feature def trim_video(self, trim_start_sec, trim_end_sec): probe = ffmpeg.probe(self.video_path) total_duration = float(probe['format']['duration']) # Adjust the start and end times based on the trimming requirements adjusted_start_sec = trim_start_sec # Start trimming 1 second into the video adjusted_end_sec = total_duration - trim_end_sec # End trimming 1 second before the video ends # Calculate the new duration to keep after trimming from both ends duration_to_keep = adjusted_end_sec - adjusted_start_sec if duration_to_keep <= 0: print("Error: The resulting duration is non-positive after trimming.") return temp_video_path = tempfile.mktemp(suffix='.mp4') try: # Use 'ss' for the adjusted start time and 't' for the duration to keep ( ffmpeg .input(self.video_path, ss=adjusted_start_sec, t=duration_to_keep) .output(temp_video_path, c='copy') # Use 'copy' to avoid re-encoding .run(overwrite_output=True) ) self.video_path = temp_video_path # Update self.video_path to use the trimmed video self.video_history.append(self.video_path) except ffmpeg.Error as e: print(f"Failed to trim video: {e.stderr.decode() if e.stderr else 'Unknown FFmpeg error'}") #Cropping feature by a specific scale def crop_video(self, scale): # Fetch video dimensions using ffprobe probe = ffmpeg.probe(self.video_path) video_stream = next((stream for stream in probe['streams'] if stream['codec_type'] == 'video'), None) original_width = int(video_stream['width']) original_height = int(video_stream['height']) # Calculate new width to make the video vertical with the same height new_width = original_height*scale # For a mobile-friendly vertical aspect ratio # Ensure new width is an even number (required by some codecs) if new_width % 2 != 0: new_width -= 1 # Calculate the horizontal offset to crop the video from the center x_offset = (original_width - new_width) // 2 # Prepare the crop filter dimensions crop_filter = f"{new_width}:{original_height}:{x_offset}:0" # Generate a temporary path for the cropped video temp_video_path = tempfile.mktemp(suffix='.mp4') try: # Apply the crop filter and save the output to a temporary file ( ffmpeg .input(self.video_path) .filter('crop', *crop_filter.split(':')) .output(temp_video_path, vcodec='libx264', crf=22) # Re-encode for compatibility .overwrite_output() .run() ) self.video_path = temp_video_path # Update to use the cropped video self.video_history.append(self.video_path) except ffmpeg.Error as e: print(f"Failed to crop video: {e.stderr.decode('utf-8')}") #Zooming in only (out is undo) def zoom_video(self, zoom_scale): # Fetch video dimensions using ffprobe probe = ffmpeg.probe(self.video_path) video_stream = next((stream for stream in probe['streams'] if stream['codec_type'] == 'video'), None) original_width = int(video_stream['width']) original_height = int(video_stream['height']) # Calculate the new dimensions based on the zoom scale new_width = int(original_width * zoom_scale) new_height = int(original_height * zoom_scale) # Ensure the new dimensions are even new_width += (new_width % 2) new_height += (new_height % 2) # Calculate the offsets to keep the crop centered x_offset = (original_width - new_width) // 2 y_offset = (original_height - new_height) // 2 # Generate a temporary path for the zoomed video temp_video_path = tempfile.mktemp(suffix='.mp4') try: # Apply the crop filter with calculated dimensions and offsets ( ffmpeg .input(self.video_path) .filter('crop', w=new_width, h=new_height, x=x_offset, y=y_offset) .output(temp_video_path, vcodec='libx264', crf=22) # Re-encode for compatibility .overwrite_output() .run() ) self.video_path = temp_video_path # Update to use the zoomed video self.video_history.append(self.video_path) # Add to history for undo functionality except ffmpeg.Error as e: print(f"Failed to zoom video: {e.stderr.decode('utf-8')}") #Speed changing def change_speed(self, speed_factor): """ Changes the speed of the video. A speed_f [truncated — 8907 more characters] ``` ### .web/postcss.config.js ```javascript module.exports = { plugins: { tailwindcss: {}, autoprefixer: {}, }, } ``` ### .web/next.config.js ```javascript module.exports = {basePath: "", compress: true, reactStrictMode: true, trailingSlash: true}; ``` ### .web/tailwind.config.js ```javascript /** @type {import('tailwindcss').Config} */ module.exports = { content: ["./pages/**/*.{js,ts,jsx,tsx}", "./utils/**/*.{js,ts,jsx,tsx}"], theme: null, plugins: [ ], }; ``` ### webui/webui.py ```python """The main Chat app.""" import reflex as rx from webui import styles from webui.components import chat, modal, navbar, sidebar, videodisplay from webui.state import State @rx.page( title="Chopsticks", ) def index() -> rx.Component: """The main app.""" return rx.chakra.vstack( rx.script(src='/custom_video_controls.js'), navbar(), rx.chakra.hstack( rx.scroll_area( # Scrollable area for the videodisplay rx.chakra.flex( # Flex container for the videodisplay content videodisplay(), # Assuming videodisplay() returns the content to display direction="column", spacing="4", ), type="always", scrollbars="vertical", style={"height": "calc(100vh - 20vh)", "width": "80%", "borderRightWidth": '1px', 'borderColor': 'white'}, ), rx.scroll_area( # Scrollable area for the chat rx.chakra.flex( # Flex container for the chat content chat.chat(), # Assuming chat.chat() returns the chat component direction="column", spacing="4", ), type="always", scrollbars="vertical", style={"height": "calc(100vh - 20vh)", "width": "20%"}, ), align="stretch", spacing="0", ), chat.action_bar(), sidebar(), modal(), bg=styles.bg_dark_color, color=styles.text_light_color, min_h="100vh", align_items="stretch", spacing="0", ) # Add state and page to the app. app = rx.App(style=styles.base_style) app.add_page(index) ``` [26 more indexed source files omitted to keep this export small. The full file list is in the Codebase structure section above.]