Project Info

Media transcode

Inspiration

During OpenAI Build Week, I wanted to explore the boundaries of AI-assisted software engineering in a highly technical and performance-critical domain. I have long been interested in media processing pipelines that power modern video platforms. At the same time, I was fascinated by Codex harness engineering — the systematic use of structured prompting, iteration, and validation to let AI models build complex systems. This led to the question: Can Codex, guided by a well-designed harness, create a media transcoding engine that meets or even exceeds industrial standards? The combination of deep media systems engineering and autonomous AI coding made this an exciting and ambitious experiment.

What it does

The project is an AI-native media transcoding engine that converts and optimizes video and audio files. It supports: Format conversion (H.264, H.265, AV1, VP9, etc.) Resolution scaling and bitrate control Audio encoding and stream multiplexing Basic filtering and post-processing The engine aims to deliver high perceptual quality while maintaining strong performance characteristics, with the long-term goal of reaching or surpassing traditional industrial-grade transcoding solutions.

How we built it

I built the entire project using Codex harness engineering, with zero manual code writing. The development followed a structured iterative loop: Defining project functionality, coding constraints, validation requirements, quality scoring criteria, and entropy/cleanup protocols via AGENTS.md. Establishing the project's overall framework via ARCHITECTURE.md. Tracking scores via QUALITY_SCORE.md, a critical component of the project's autonomous iteration loop. Assigning distinct sub-agents to handle planning, coding, and code reviews. Codex generated individual modules (parsers, encoders, pipelines, optimizers) in small, testable units. Automated tests and quality benchmarks were run, with failures fed back to Codex for refactoring. I performed only subjective human evaluation (visual and audio A/B testing) and provided feedback through the harness. This created a closed-loop system where the AI handled implementation and iteration, while I focused on direction and quality validation.

Challenges we ran into

Zero manual coding constraint: Every line of code had to come from Codex, making debugging and optimization significantly harder. Context and codebase management: Maintaining coherence as the project grew required advanced prompt engineering and summarization. Performance optimization: Achieving low-level efficiency (memory layout, SIMD, GPU acceleration) through natural language guidance demanded many refinement cycles. Perceptual quality consistency: Ensuring output matched industry standards across edge cases relied heavily on repeated subjective testing. Long-term complexity: Building a truly competitive engine is a multi-month effort, not a one-week sprint.

Accomplishments we're proud of

Successfully built a functional media transcoding engine without writing a single line of code manually. Established a reusable Codex harness framework for complex systems engineering. Through AI-driven iterative optimization, impressive results have been achieved in both local and real-time video transcoding. Demonstrated that AI can tackle deep technical domains when properly scaffolded.

What we learned

The real skill in AI-assisted development shifts from writing code to designing excellent requirements, feedback loops, and evaluation systems. Media transcoding is an outstanding benchmark for AI coding capabilities due to its mix of algorithmic complexity and human-perceptible results. Disciplined, iterative prompting with strong validation is essential to prevent drift and maintain quality. Autonomous AI engineering shows tremendous potential but still requires thoughtful human oversight and patience.

What's next

This is a long-term project. Future plans include: Expanding codec support and advanced features ( Multiple forms of hardware acceleration, adaptive streaming, AI-enhanced upscaling) Improving automation and self-optimization capabilities within the harness Rigorous benchmarking against industry leaders (FFmpeg, commercial encoders) Exploring distributed transcoding and cloud-native architectures Continuing to push the boundaries of what fully AI-generated systems can achieve in performance-critical software I’m excited to keep iterating and evolving this engine well beyond Build Week.

MediaTranscode

MediaTranscode is a C++20 and FFmpeg-backed media transcode framework centered on the graph DAG architecture.

The active code path is the graph runtime:

src/internal/graph/        # graph model, planner, builder, runtime, and runtime nodes
tools/local_video_cli/     # local-file video transcode CLI
tools/realtime_video_cli/  # realtime video transcode CLI
include/media_transcode/
    Result.h               # shared Result<T> and ErrorInfo type

Only the graph architecture and Result.h are documented as active project surfaces.

Build

Use the existing CMake flow:

cmake -S . -B out/build/x64-debug
cmake --build out/build/x64-debug --target media_transcode_core
cmake --build out/build/x64-debug --target media_transcode_local_video_cli
cmake --build out/build/x64-debug --target media_transcode_realtime_video_cli

Useful CMake options:

MEDIA_TRANSCODE_BUILD_GRAPH_TOOLS=ON

Realtime DAG Path

The realtime CLI accepts URL/RTSP, separate H.264/AAC RTP, or MPEG-TS/UDP input. Input clock selection and output protocol selection are independent; every A/V input reaches the same canonical scheduler before exactly one output adapter is assembled.

URL/RTSP, separate RTP, or MPEG-TS/UDP input
    -> planner-owned input clock
    -> shared A/V startup, drift, recovery, and scheduler
    -> separate RTP | MPEG-TS/UDP | MPEG-TS/RTP

--output-layout and the mandatory --output-transport select one of three exact modes:

--output-layout separate --output-transport rtp  per-media RTP/RTCP plus SDP
--output-layout mpegts  --output-transport udp  Project MPEG-TS in UDP datagrams
--output-layout mpegts  --output-transport rtp  Project MPEG-TS over RTP/AVP plus SDP

separate + udp is rejected. MPEG-TS/RTP uses the static MP2T payload type 33, a 90 kHz RTP clock, adjacent RTP/RTCP ports, and one generated SDP media description. Open separate dynamic-payload RTP through its generated SDP; open MPEG-TS/RTP through its production rtp://@host:port URL.

For visible MPEG-TS acceptance, VLC opens the production URL directly:

& 'D:\VideoLAN\VLC\vlc.exe' 'rtp://@192.168.96.122:60000'
& 'D:\VideoLAN\VLC\vlc.exe' 'udp://@192.168.96.122:60000'

Do not insert FFmpeg, an observer remux, or SDP playback between the production MPEG-TS output and VLC. Separate dynamic-payload RTP still writes SDP as the signaling artifact selected by the caller, but it is not the MPEG-TS/RTP receiver URL.

--max-duration SECONDS is optional realtime CLI monitoring policy. When it is present, it must be positive and stops a still-running CLI after that duration. When it is absent, startup output, progress, and lifecycle failures retain their existing checks, but the CLI has no duration deadline and reports max_duration=source_driven. This value is not planned or passed into the media graph runtime.

Raw RTP input requires explicit video and audio endpoint metadata. For H.264 or HEVC video only, omitting --video-rtp-fmtp enables preflight in-band parameter-set detection. Codec, payload type, 90 kHz clock rate, URL, and all timeout/capacity limits remain mandatory. The planner derives canonical fmtp only after complete, unambiguous evidence; the runtime receives the same bound UDP transport and the original pre-read RTP/RTCP queue. Supplying video fmtp keeps strict manual mode and performs no probe I/O. AAC always requires explicit fmtp; Opus keeps its existing no-fmtp contract.

The following VideoOnly example demonstrates automatic H.264 fmtp detection. Add --video-rtp-fmtp only when authoritative signaling is available and manual mode is required:

$inputRtp = @(
    '--input-type','rtp',
    '--video-rtp-url','rtp://127.0.0.1:5004',
    '--video-rtp-codec','h264','--video-rtp-payload-type','96',
    '--video-rtp-clock-rate','90000',
    '--open-timeout-ms','30000','--read-timeout-ms','2000',
    '--analyze-duration-us','5000000','--probe-size','5000000'
)
$common = @(
    '--egress-capacity-bps','50000000','--maximum-wire-residence-ms','100',
    '--video-codec','hevc','--rc','cbr',
    '--width','1920','--height','1080','--fps','25',
    '--bitrate','6000','--gop','50','--no-audio'
)
$cli = 'out/build/x64-release/media_transcode_realtime_video_cli.exe'

& $cli @inputRtp @common --media-id rtp-to-tsudp `
    --output-layout mpegts --output-transport udp `
    --output 'udp://127.0.0.1:5010'

& $cli @inputRtp @common --media-id rtp-to-tsrtp `
    --output-layout mpegts --output-transport rtp `
    --rtp-host 127.0.0.1 --rtp-port 5020 `
    --sdp out/build/x64-debug/rtp-to-tsrtp.sdp

Probe limits have exact meanings: --open-timeout-ms is the target deadline for controllable startup work, --analyze-duration-us is the maximum interval for obtaining complete unambiguous facts after the first matching RTP packet, --probe-size caps all bytes received during preflight, and --read-timeout-ms bounds each socket wait. Detection ends when the required parameter sets are complete; conflicts observed before completion fail. FFmpeg/driver capability calls are checked for deadline overrun immediately after return because those APIs are not cancellable. Timeout, wrong PT, incomplete/conflicting parameter sets, unsupported packetization, or capacity exhaustion fails preflight before the DAG starts.

Add --max-duration SECONDS only when the caller wants the explicit CLI stop gate; do not use 0, a sentinel, or a large replacement duration.

Hardware backend and realtime low-latency behavior are planner-owned and have no caller override; the highest-scoring capability-admitted chain is selected or the request fails before runtime. --quiet-graph only controls diagnostics. udp://host:port remains valid for UDP-carried RTP input in RTP-port mode; MPEG-TS input uses --input-type mpegts-udp and requires an explicit --mpegts-max-pcr-gap-ms. URL input uses --input-type url; rtsp:// URLs require --rtsp-transport, while non-RTSP URLs reject it.

Production acceptance

Project acceptance is based on the real local and realtime CLIs, real media streams, FFmpeg/VLC observation, runtime memory metrics, and continuous A/V drift telemetry. The current absolute-path PowerShell commands and evidence are recorded under docs/completed/.

Input termination is strict: only a true FFmpeg EOF completes successfully. HTTP, UDP, and RTP source disappearance returns one preserved source-loss failure; coordinated cancellation of the remaining workers does not replace that primary error. EOF drains codec state and all sinks before the realtime CLI performs its normal stop path.

Analysis

Compare with all teams

View

Metric

tangmingcheng
89

Figures cover GitHub contributors during the hackathon window. A co-authored commit counts in full for each author, so per-member totals add up to more than the whole-team figures.

Technology

Found in codeClaimed only

CIn code
C++In code

2 of 2 appear in the indexed code.

AI coding agents

CodexConfig

Detected from committed agent config files and commit authorship. Absence of a signal is not proof an agent was unused.

Codebase size

Source size

7.1 MB

Source files

1,443

Counts recognized source files only; vendored directories, binaries and lockfiles are excluded, so this is smaller than the repository on disk.