# Project export: Sapheneia
This document was generated by HackStack to give an AI agent context about a hackathon project. Sections are labeled with their provenance; content marked as truncated was cut to keep this document small.
## Project metadata
- Hackathon: TreeHacks 2024
- Tagline: Sapheneia brings revolutionary AI video enhancement to everyone. Our free tools upscale and enhance videos in real-time on all platforms. Accessible technology that transcends socioeconomic barriers.
- Devpost: https://devpost.com/software/sapheneia
- GitHub: https://github.com/nairvishnumail/Sharp-ai-ly
- Team: 2 GitHub contributor(s) — Vishnu Nair (4 commits), shishir610 (3 commits)
## Devpost submission (written by the team)
### Overview
Problem There are two sides to technological progress: a brighter, progressive future where new technology becomes further integrated into our lives and the thousands of old devices that fall into disuse to welcome the new change. As we approach another technological frontier with the advancement of AI, it is often easy to look at the bright future ahead, but it is far more important to see what and more importantly, who we leave behind. Low-quality videos and pictures often fall to the wayside despite the significant amount of childhood memories they hold. We aim to bring back these memories through our software. However, even more importantly, the advancement of technology can leave people behind. Limits on the availability of good graphics can worsen education, restrict communication, and widen disparities for people of lower socioeconomic status. We aim to bridge these disparities by bringing video enhancement to all.
### Inspiration
: Our journey to creating this suite of video enhancement tools stemmed from a critical realization: video content is not one-size-fits-all. By recognizing the diverse needs of audiences including live streamers, individuals who are colorblind, the elderly, and film preservationists, in addition to the general population, we set out to dismantle the barriers to digital content accessibility. We aim to harness the latest advancements in AI to elevate video quality and ensure that everyone, regardless of their age, socioeconomic status, or disability, can fully engage with and appreciate the richness of digital media.
### What it does
: Sapheneia utilizes open-source video enhancement technology, we have developed a straightforward web interface that significantly simplifies the process of improving video quality. This platform represents a leap forward in our mission to make high-quality video content accessible to everyone. Centralized Video Enhancement Hub: Our platform provides a seamless experience for users looking to enhance their video content. By uploading a video directly to our website, users can witness the transformation of their graphics in real-time. The interface thoughtfully displays the enhanced video alongside the original, enabling immediate comparison and showcasing the dramatic improvements our technology delivers. How it works: The Bread and Butter: Our core technology revolves around an AI-driven approach specifically designed for enhancing animated visuals. This model is adept at analyzing low-resolution footage and predicting a high-quality output by filling in missing details, reducing noise, and sharpening lines without compromising the animated original style. Trained on a comprehensive dataset of animated content, it ensures that each frame is not just clearer but also in line with the intent of the original creation. At Sapheneia, our infrastructure is built around a powerful pipeline hosted on AWS Cloud, which serves as the foundation for our vision enhancement model. This setup enables our web interface to interact seamlessly with our backend. Here, videos are spliced and sent to the server, where they undergo real-time enhancement. Once processed, the enhanced video is promptly returned to the interface, ready to be displayed. What’s Next: Dim Scene Enhancement: A challenge our currently implemented model faces is that with darker scenes, it ranks poorly in video enhancement because the model finds it difficult to find the contrast lines of buildings, faces, etc, and can’t accurately differentiate and enhance different artifacts in the frame. This algorithm will isolate these darker scenes, temporarily increase their brightness for more effective enhancement, and then restore the original brightness level before presenting the final, enhanced video to the user. This method aims to improve the model's ability to enhance details in low-light conditions, ensuring better video quality regardless of scene brightness. Content-Type Enhancement: Understanding the distinct characteristics of animated versus live-action footage, we're introducing a user-friendly feature that allows for the selection between enhancement modes tailored to each type. This approach enhances the viewing experience by applying model adjustments that are most appropriate for the content's nature. Our Market: Our suite of tools is aimed at a broad user base, from individuals looking for a better viewing experience to live streamers and film restorationists seeking to enhance their video quality. Due to the accessibility and affordability of our product, we bring an attractive alternative to a new market of people that has been blocked off by paywalls and hardware limitations.We envision our largest market opportunity in licensing collaborations with leading live content creation platforms, scaling our reach and impact post-beta testing with companies such as Twitch, Tiktok, & Youtube Gaming. For an exploration of additional applications, including the transformative use of our real-time vision enhancement technology in various settings, please scroll down to the Proof of Concept section. Differences from our Competitors: The biggest advantage we have over our competitors is accessibility. We unlock the technology for the public - developers and non-developers alike. Our software program is downloadable by anyone with access to a computer and is completely free. In comparison, people choosing to opt-into services like RTX or Topaz Labs are locked behind a paywall that can scale as high as $300 per tool and hardware components such as a graphics card or large amounts of RAM to be used. Making our product more inclusive and available to a broader audience, enabling individuals from various economic backgrounds to access and utilize it. Proof of Concept @ TreeHacks: Reazon Holdings: At Sapheneia, we found a unique opportunity to leverage our video enhancement technology in collaboration with Reazon's venture into anime production. This partnership not only aims to elevate the quality of new anime but also rejuvenates older titles, sparking renewed interest and enabling game development around these refreshed IPs. The growing demand for revitalizing early 2000s anime aligns perfectly with our capabilities, positioning us to significantly impact Reazon's diverse portfolio by enhancing viewer engagement and expanding their market reach. Verdaka: Known for their innovative use of generative AI to anonymize individuals in video footage by blurring faces, is set to enhance its capabilities further through our collaboration. Building on the momentum of our successful hackathon project, we are developing a real-time plugin designed to significantly upscale video quality for both security footage and playback scenarios. This initiative aims to refine visual clarity without compromising privacy, ensuring that security surveillance and video playback are both sharp and secure. Our willingness to collaborate represents a leap forward in combining privacy with high-definition video technology, setting a new standard for security and playback solutions. Parrot Drones: A leader in creating drones operable from any global location with cellular data, offers an exciting opportunity through its open SDK. By which we aim to leverage this by introducing a real-time video quality enhancement plugin that boosts the clarity of footage captured by these drones. This innovation is not just a leap in video technology; it has significant social implications. Enhanced video quality becomes a crucial asset in missions like searching for survivors in disaster-stricken or conflict-affected zones. Moreover, our approach ingeniously navigates the challenges of low-bandwidth areas by allowing the capture of lower-quality video, which we then upscale on the receiver's end where better internet connectivity is present. This ensures that critical visual data remains accessible and clear, maximizing the drones' utility in vital humanitarian and surveillance efforts. Business Plan: Our approach centers on offering a range of free, user-friendly tools that democratize access to cutting-edge video enhancement technology, ensuring minimal to no barriers for the average user. Through this approach, we can reach out to a wider and more interested audience that is looking for video-enhancement software because of the equitable and accessible nature of our software. This means we can bridge the gaps brought about by socioeconomic differences and bring video enhancement to the ones who need and want it most at a much more affordable price. We plan to access this market through an extensive advertising campaign and a two-month free trial period where all of our premium features will be available to the wider market. This would include the live-streaming feature as well as more specific models that are tailored for the consumption of certain forms of media like anime. Our general video enhancement software, however, would be available to the wider public. Our objective is to attract a significant volume of online traffic to our extensions and plugins, leveraging this momentum as detailed in our "Next Steps" section. This pivotal phase will enable us to initiate outreach and forge connections with the companies highlighted in our Proof of Concept and with leading live content platforms. Our goal is to cultivate partnerships that facilitate the seamless integration of our video enhancement technology, thereby elevating the quality and accessibility of video content across the digital landscape.
## README (from the GitHub repository)
# Sharp-AI-ly
Engineered a dockerized environment of an open-source [REAL ESRGAN model ↗](https://github.com/the-database/mpv-upscale-2x_animejanai) interfaced with a FlaskAPI on an AWS EC2 p2.xlarge instance, ensuring cost-effective scalability for GPU-intensive tasks through NVIDIA’s TensorRT SDK, leading to video enhancement in 2x time. Submitted to [TreeHacks 2024 ↗](https://devpost.com/software/sapheneia)
## Demo
https://github.com/nairvishnumail/Sharp-ai-ly/assets/27198773/c8b4ea97-988f-40ba-a6bb-5c9e5e5a19df
## AWS Deploy Steps
In order to deploy on AWS, you need to make sure that the AWS account has at least 4 vCPUs available. If you do not have at least 4 vCPUs, you can request more [here](https://support.console.aws.amazon.com/support/home?region=us-east-2#/case/create). Provide a reasonable explanation as to why you need it and within 24-48 hours you should have access to it.
### EC2 launch steps:
1. Launch an EC2 instance
2. Choose an appropriate AMI (important) - I chose Deep Learning OSS Nvidia Driver AMI GPU PyTorch 2.0.1 (Amazon Linux 2) 20240206
3. Choose p2.xlarge instance type. G instances do not work with the above AMI. But, feel free to try different types.
4. Create a security group to allow SSH traffic ("anywhere" is good for testing, but not recommended due to low security)
5. Choose 45 GB gp3 storage
> **_WARN:_** AWS EC2 instances that need NVIDIA GPUs can be quite costly, so make sure you terminate the instance once you're done working with it.
### SSH & Setup NVIDIA drivers
1. ssh into the instance by clicking on "connect" and following the ssh steps
2. Follow steps [here](https://docs.nvidia.com/datacenter/tesla/tesla-installation-notes/index.html#ubuntu-lts) to setup NVIDIA SMI on your EC2 instance. This step is hard to get right since there are many different guides.
### Setting up server
Follow the self-deploy steps below on the instance to have the server up and working.
## Self-deploy steps
> **_NOTE:_** To self-deploy, your machine needs to have an NVIDIA GPU running with the correct drivers. Make sure `nvidia-smi` works correctly.
1. `git clone https://github.com/nairvishnumail/Sharp-ai-ly.git`
2. Make sure you have docker and docker-compose: Get it [here](https://www.docker.com/get-started/)
3. Dockerize input and output directories `docker run -v "/Sharpr-ai-ly/src/backend/ai/input:/input" -v "/Sharpr-ai-ly/src/backend/ai/out:/out" -it `
4. `docker-compose run --rm vsgan_tensorrt`
5. `python app.py`
You now have the server running on at [localhost:8080](http://localhost:8080)
Send a POST request with video_url in the body to `http://localhost:8080/process-video` to get back a S3 object URL for the processed video
For tunnelling a localhost environment, I used [ngrok](https://ngrok.com/download) that helped me hit the endpoint from different PCs
## Detected evidence (automated analysis)
Indexed codebase: 17 recognized source files, 18 KB.
- CSS (language) — detected in the code
- HTML (language) — detected in the code
- JavaScript (language) — detected in the code
- Python (language) — detected in the code
- React (technology) — detected in the code
- Tailwind CSS (technology) — detected in the code
- C++ (language) — claimed on Devpost, not found in the code
- Docker (technology) — claimed on Devpost, not found in the code
- Node.js (technology) — claimed on Devpost, not found in the code
## Codebase structure (from repository index)
### Files (27 of 27)
```
.gitignore
2x_AnimeJaNai_V2_Compact_36k_op18_fp16_clamp copy.engine
app.py
compose_mpv.yaml
compose.yaml
Dockerfile
Dockerfile_ffmpeg_trt
frontend/.gitignore
frontend/package.json
frontend/public/index.html
frontend/public/manifest.json
frontend/public/robots.txt
frontend/README.md
frontend/src/App.css
frontend/src/App.js
frontend/src/App.test.js
frontend/src/index.css
frontend/src/index.js
frontend/src/reportWebVitals.js
frontend/src/setupTests.js
frontend/tailwind.config.js
inference_batch.py
inference_config.py
LICENSE.txt
main.py
nvidia_icd.json
README.md
```
### Dependencies
- frontend/package.json: @testing-library/jest-dom@^5.17.0, @testing-library/react@^13.4.0, @testing-library/user-event@^13.5.0, react@^18.2.0, react-dom@^18.2.0, react-scripts@5.0.1, tailwindcss@^3.4.1, web-vitals@^2.1.4
### Recent commits (newest first)
- Demo video
- ngrok
- SSH & NVIDIA setup
- Added EC2 launch steps
- Create README.md
- Simple react frontend for quick testing
- added missing yaml files for docker-compose
- changed main.py for running as service
- Add files via upload
- Add files via upload
- Add files via upload
- Add files via upload
- Initial commit
## Key source files (fetched from GitHub, selected and truncated for size)
### frontend/package.json
```
{
"name": "frontend",
"version": "0.1.0",
"private": true,
"dependencies": {
"@testing-library/jest-dom": "^5.17.0",
"@testing-library/react": "^13.4.0",
"@testing-library/user-event": "^13.5.0",
"react": "^18.2.0",
"react-dom": "^18.2.0",
"react-scripts": "5.0.1",
"web-vitals": "^2.1.4"
},
"scripts": {
"start": "react-scripts start",
"build": "react-scripts build",
"test": "react-scripts test",
"eject": "react-scripts eject"
},
"eslintConfig": {
"extends": [
"react-app",
"react-app/jest"
]
},
"browserslist": {
"production": [
">0.2%",
"not dead",
"not op_mini all"
],
"development": [
"last 1 chrome version",
"last 1 firefox version",
"last 1 safari version"
]
},
"devDependencies": {
"tailwindcss": "^3.4.1"
}
}
```
### Dockerfile
```
############################
# FFMPEG
############################
FROM archlinux as ffmpeg-arch
RUN --mount=type=cache,sharing=locked,target=/var/cache/pacman \
pacman -Syu --noconfirm --needed base base-devel cuda git
ENV NVIDIA_VISIBLE_DEVICES all
ENV NVIDIA_DRIVER_CAPABILITIES compute,utility
ARG user=makepkg
RUN useradd --system --create-home $user && \
echo "$user ALL=(ALL:ALL) NOPASSWD:ALL" >/etc/sudoers.d/$user
USER $user
WORKDIR /home/$user
RUN git clone https://aur.archlinux.org/yay.git && \
cd yay && \
makepkg -sri --needed --noconfirm && \
cd && \
rm -rf .cache yay
RUN yay -Syu rust tcl nasm cmake jq libtool wget fribidi fontconfig libsoxr meson pod2man nvidia-utils base-devel --noconfirm --ask 4
USER root
RUN mkdir -p "/home/makepkg/python311"
RUN wget https://github.com/python/cpython/archive/refs/tags/v3.11.3.tar.gz && tar xf v3.11.3.tar.gz && cd cpython-3.11.3 && \
mkdir debug && cd debug && ../configure --enable-optimizations --disable-shared --prefix="/home/makepkg/python311" && make -j$(nproc) && make install && \
/home/makepkg/python311/bin/python3.11 -m ensurepip --upgrade
RUN cp /home/makepkg/python311/bin/python3.11 /usr/bin/python
ENV PYTHONPATH /home/makepkg/python311/bin/
ENV PATH "/home/makepkg/python311/bin/:$PATH"
RUN pip3 install cython meson
ENV PATH "$PATH:/opt/cuda/bin/nvcc"
ENV PATH "$PATH:/opt/cuda/bin"
ENV LD_LIBRARY_PATH "/opt/cuda/lib64"
# -O3 makes sure we compile with optimization. setting CFLAGS/CXXFLAGS seems to override
# default automake cflags.
# -static-libgcc is needed to make gcc not include gcc_s as "as-needed" shared library which
# cmake will include as a implicit library.
# other options to get hardened build (same as ffmpeg hardened)
ARG CFLAGS="-O3 -static-libgcc -fno-strict-overflow -fstack-protector-all -fPIE"
ARG CXXFLAGS="-O3 -static-libgcc -fno-strict-overflow -fstack-protector-all -fPIE"
ARG LDFLAGS="-Wl,-z,relro,-z,now"
# master is broken https://github.com/sekrit-twc/zimg/issues/181
# No rule to make target 'graphengine/graphengine/cpuinfo.cpp', needed by 'graphengine/graphengine/libzimg_internal_la-cpuinfo.lo'. Stop.
RUN git clone https://github.com/sekrit-twc/zimg --depth 1 --recurse-submodules --shallow-submodules && cd zimg && \
./autogen.sh && CFLAGS=-fPIC CXXFLAGS=-fPIC ./configure --enable-static --disable-shared && make -j$(nproc) && make install
ENV PATH /usr/local/bin:$PATH
RUN wget https://github.com/vapoursynth/vapoursynth/archive/refs/tags/R65.tar.gz && \
tar -zxvf R65.tar.gz && cd vapoursynth-R65 && ./autogen.sh && \
PKG_CONFIG_PATH="/usr/lib/pkgconfig:/usr/local/lib/pkgconfig" ./configure --enable-static --disable-shared && \
make && make install && cd .. && ldconfig
RUN git clone https://github.com/gypified/libmp3lame && cd libmp3lame && ./configure --enable-static --enable-nasm --disable-shared && make -j$(nproc) install
RUN git clone https://github.com/mstorsjo/fdk-aac/ && \
cd fdk-aac && ./autogen.sh && ./configure --enable-static --disable-shared && make -j$(nproc) install
RUN git clone https://github.com/xiph/ogg && cd ogg && ./autogen.sh && ./configure --enable-static --disable-shared && make -j$(nproc) install
RUN git clone https://github.com/xiph/vorbis && cd vorbis && ./autogen.sh && ./configure --enable-static --disable-shared && make -j$(nproc) install
RUN git clone https://github.com/xiph/opus && cd opus && ./autogen.sh && ./configure --enable-static --disable-shared && make -j$(nproc) install
RUN git clone https://github.com/xiph/theora && cd theora && ./autogen.sh && ./configure --disable-examples --enable-static --disable-shared && make -j$(nproc) install
RUN git clone https://github.com/webmproject/libvpx/ && \
cd libvpx && ./configure --enable-static --enable-vp9-highbitdepth --disable-shared --disable-unit-tests --disable-examples && \
make -j$(nproc) install
RUN git clone https://code.videolan.org/videolan/x264.git && \
cd x264 && ./configure --enable-pic --enable-static && make -j$(nproc) install
# -w-macro-params-legacy to not log lots of asm warnings
# https://bitbucket.org/multicoreware/x265_git/issues/559/warnings-when-assembling-with-nasm-215
RUN git clone https://bitbucket.org/multicoreware/x265_git/ && cd x265_git/build/linux && \
cmake -G "Unix Makefiles" -DENABLE_SHARED=OFF -D HIGH_BIT_DEPTH:BOOL=ON -DENABLE_AGGRESSIVE_CHECKS=ON ../../source -DCMAKE_ASM_NASM_FLAGS=-w-macro-params-legacy && \
make -j$(nproc) install
RUN git clone https://github.com/webmproject/libwebp/ && \
cd libwebp && ./autogen.sh && ./configure --enable-static --disable-shared && make -j$(nproc) install
RUN git clone https://github.com/xiph/speex/ && \
cd speex && ./autogen.sh && ./configure --enable-static --disable-shared && make -j$(nproc) install
RUN git clone --depth 1 https://aomedia.googlesource.com/aom && \
cd aom && \
mkdir build_tmp && cd build_tmp && cmake -DBUILD_SHARED_LIBS=0 -DENABLE_TESTS=0 -DENABLE_NASM=on -DCMAKE_INSTALL_LIBDIR=lib .. && make -j$(nproc) install
RUN git clone https://github.com/georgmartius/vid.stab/ && \
cd vid.stab && cmake -DBUILD_SHARED_LIBS=OFF . && make -j$(nproc) install
RUN git clone https://github.com/ultravideo/kvazaar/ && \
cd kvazaar && ./autogen.sh && ./configure --enable-static --disable-shared && make -j$(nproc) install
RUN git clone https://github.com/libass/libass/ && \
cd libass && ./autogen.sh && ./configure --enable-static --disable-shared && make -j$(nproc) && make install
RUN git clone https://github.com/uclouvain/openjpeg/ && \
cd openjpeg && cmake -G "Unix Makefiles" -DBUILD_SHARED_LIBS=OFF && make -j$(nproc) install
RUN git clone https://code.videolan.org/videolan/dav1d/ && \
cd dav1d && meson build --buildtype release -Ddefault_library=static && ninja -C build install
# add extra CFLAGS that are not enabled by -O3
# http://websvn.xvid.org/cvs/viewvc.cgi/trunk/xvidcore/bu
[truncated — 38876 more characters]
```
### main.py
```python
# example usage: python main.py
# vapoursynth does not have audio support and processing multiple files is not really possible
# hacky script to make batch processing with audio and subtitle support
# make sure tmp_dir is also set in inference.py
# maybe should pass arguments instead of a text file instead
# main.py
import glob
import os
import shutil
# Define your directories
input_dir = "/workspace/tensorrt/input/"
tmp_dir = "tmp/"
output_dir = "/workspace/tensorrt/output/"
# Refactor the code to make it callable
def process_videos():
files = glob.glob(input_dir + "**/*.mp4", recursive=True)
files.sort()
for f in files:
# creating folders if they dont exist
if not os.path.exists(tmp_dir):
os.mkdir(tmp_dir)
if not os.path.exists(output_dir):
os.mkdir(output_dir)
if os.path.exists(tmp_dir):
shutil.rmtree(tmp_dir)
os.mkdir(tmp_dir)
# paths
out_render_path = os.path.join(
output_dir, os.path.splitext(os.path.basename(f))[0] + "_rendered.mkv"
)
mux_path = os.path.join(
output_dir, os.path.splitext(os.path.basename(f))[0] + "_mux.mkv"
)
# x264 crf10 preset slow [31fps]
os.system(
f"vspipe -c y4m inference_batch.py --arg source='{f}' - | ffmpeg -y -i '{f}' -thread_queue_size 100 -i pipe: -map 1 -map 0 -map -0:v -max_interleave_delta 0 -scodec copy -crf 10 -preset slow '{mux_path}'"
)
if __name__ == '__main__':
process_videos()
```
### app.py
```python
from flask import Flask, request, jsonify, url_for, send_from_directory
from pytube import YouTube
import os
from celery import Celery
from main import process_videos
from urllib.parse import urlparse, parse_qs
app = Flask(__name__)
app.config['CELERY_BROKER_URL'] = 'redis://localhost:6379/0'
app.config['CELERY_RESULT_BACKEND'] = 'redis://localhost:6379/0'
input = "/input"
output = "/output"
celery = Celery(app.name, broker=app.config['CELERY_BROKER_URL'])
celery.conf.update(app.config)
# Define your REST endpoint
@app.route('/process-video', methods=['POST'])
def process_video():
video_url = request.json.get('video_url')
parsed_url = urlparse(video_url)
query_params = parse_qs(parsed_url.query)
video_id = query_params.get("v")[0] if "v" in query_params else None
download_video(video_url)
process_downloaded_video()
processed_video_url = generate_video_url(video_id + "_mux.mkv")
return jsonify({ "processed_video_url" : processed_video_url })
def download_video(video_url):
if not os.path.exists(input):
os.makedirs(input)
yt = YouTube(video_url)
video_stream = yt.streams.get_highest_resolution()
video_stream.download(output_path=input)
print(f"Video '{yt.title}' downloaded successfully to {input}")
def process_downloaded_video():
process_videos()
def serve_video(filename):
return send_from_directory(output, filename)
def generate_video_url(filename):
if not filename:
return jsonify({'error': 'Missing filename'}), 400
video_url = url_for('serve_video', filename=filename, _external=True)
return video_url
if __name__ == "__main__":
app.run(debug=True, host='0.0.0.0', port=8080)
```
### frontend/src/index.js
```javascript
import React from 'react';
import ReactDOM from 'react-dom/client';
import './index.css';
import App from './App';
import reportWebVitals from './reportWebVitals';
const root = ReactDOM.createRoot(document.getElementById('root'));
root.render(
);
// If you want to start measuring performance in your app, pass a function
// to log results (for example: reportWebVitals(console.log))
// or send to an analytics endpoint. Learn more: https://bit.ly/CRA-vitals
reportWebVitals();
```
### frontend/src/App.js
```javascript
import logo from "./logo.svg"
import "./App.css"
import { useEffect } from "react"
import { useState } from "react"
function App() {
const [sendURL, setSendURL] = useState("")
const [receiveURL, setReceiveURL] = useState("")
const [loading, setLoading] = useState(false)
const handleEnhance = () => {
console.log("Fetching")
fetch("https://412b-128-84-124-184.ngrok-free.app/process-video", {
method: "POST",
headers: {
"Content-Type": "application/json",
},
body: JSON.stringify({ video_url: sendURL }),
})
.then((response) => response.json())
.then((data) => {
setReceiveURL(data.processed_video_url)
})
.catch((error) => {
console.error("Error:", error)
})
}
console.log(sendURL)
console.log(receiveURL)
return (