One filter graph: how ReelForge renders captions, B-roll and audio in a single FFmpeg pass

A ReelForge export has a lot going on: a talking-head clip, captions that fade and slide and bounce, B-roll cutting in over specific lines, background music, sound effects, sometimes a watermark. The naive way to render that is a chain of passes, each writing an intermediate file. We do it in one.

The unit of editing is three words#

Everything starts with the transcript. Speech-to-text gives us every word with a start and end time, and the editor groups them three at a time into caption segments. That number wasn’t arbitrary: three words read punchy on a phone screen, and they’re long enough to hang other things on.

That’s the trick. When you add B-roll or a sound, you don’t drag it onto a timeline and fiddle with its edges — you pin it to one or more caption segments. Its timing comes from the speech, so it lines up by construction.

Two renderers, one format#

In the browser, captions are previewed on a <canvas> redrawn every animation frame, with presets — Classic, Bold, Minimal, Neon — and enter and exit animations. On the server, the final video is rendered by FFmpeg with libass. The bridge between them is the ASS subtitle format, which supports positioning, fonts, colours and animation tags natively.

Each caption becomes one dialogue line with override tags:

Dialogue: 0,0:00:01.20,0:00:02.05,Default,,0,0,0,,{\pos(960,918)\fnInter\fs64\c&H00FFFFFF&\fad(300,300)}nobody tells you

1. 1

   Fade

   \fad(in, out) — a straight mapping from the editor’s fade durations.
2. 2

   Slide

   \move(x1, y1, x2, y2) between an offset start and the caption’s resting position.
3. 3

   Bounce

   \t(…\move…) — a timed transform that overshoots and settles.
4. 4

   Colour

   The editor’s RGBA becomes ASS &HAABBGGRR — byte order reversed, and alpha inverted, because in ASS 00 is opaque.

The filter graph#

The render worker downloads every input, then builds a single filter_complex. In order:

# each B-roll clip: fill the frame, shift to its start, overlay only while it's on
[1:v]scale=W:H:force_original_aspect_ratio=increase,crop=W:H,
     setpts=PTS-STARTPTS+START/TB[b1];
[0:v][b1]overlay=enable='between(t,START,END)'[v1];

# audio layers: normalise, delay to their start, pad, then mix pairwise
[2:a]aformat=…,adelay=…,apad[a1];
[0:a][a1]amix=inputs=2:normalize=0[a];

# free tier: a blurred patch with the watermark on top, bottom-right

# captions last, so nothing ever covers them
[v1]subtitles=filename=captions.ass[out]

One decode, one encode. With no audio layers, the original audio is stream-copied instead of re-encoded. The Docker image refuses to build unless its FFmpeg was compiled with libass — we learned that one the hard way.

Small details that took a day each#

• Phones lie about orientation. They record portrait video as landscape with a rotation tag. We read both the tag and the side data, and swap width and height before anything else.
• Caches are too good. Re-exporting a project used to return the previous video from the CDN. Every export now writes to a unique path.
• Failures shouldn’t cost you. If an upload fails or contains no speech, it doesn’t count against your plan’s video allowance.

Moving the render off the request#

Early on, exports ran inside the API request. In April 2026 they moved to a Celery worker on Redis, with a hard time limit, three retries and named progress steps — “Downloading assets”, “Running FFmpeg”, “Uploading result” — that the editor polls every few seconds. When it’s done, you get an email with the video. Uploads got a safety net too: a project stuck in processing for 15 minutes after a restart is marked failed instead of spinning forever.