Every rendering comparison on the internet has the same shape: two bars, one taller, a confident caption. What they never show is the workload, the hardware, or who ran the test — the three things that decide whether the bars mean anything.
This piece contains no benchmark numbers on purpose. What it contains is the method: how to run a rendering comparison yourself, on your workload, so the numbers that come out are ones you can act on. The design below works for any two pipelines — it was built for Remotion vs Meishe Web SDK, but the discipline is universal.
Why Most Rendering Benchmarks Mislead
Three traps, in order of frequency:
- Synthetic workloads. Rendering a spinning logo proves nothing about rendering a 3-minute multi-track edit with transitions and audio. Real products render real content; benchmarks should too.
- Different output settings. Pipeline A renders 720p/30, Pipeline B renders 1080p/60 — the comparison is over before it starts. Output must be identical, verified, not assumed.
- Cold/warm mixing. First render on a fresh instance (cold: image pulls, JIT warmup, cache misses) vs fifth render on a warm one can differ by multiples. Mix them silently and the data is noise.
Step 1: Design the Workload
Build three representative templates from your actual product mix — the honest sample beats the impressive demo:
| Workload | Represents | Design |
|---|---|---|
| W1: Light | Template cards, title animations | 15s, 1 track, text + images, 2 transitions |
| W2: Typical | Your median product video | 60s, 2–3 tracks, clips + text + music, 5–8 transitions |
| W3: Heavy | Your worst real case | 180s+, multi-track, effects (keyframes/filters), mixed audio |
Freeze the assets: same source clips, same fonts, same audio files for both pipelines. Host them identically (same storage, same region) so asset fetching doesn’t pollute render time.
Step 2: Match the Outputs
Before timing anything, verify both pipelines produce the same output spec — resolution, fps, bitrate/quality preset, codec — and eyeball that the visual results match (a pipeline that quietly drops frames isn’t faster, it’s wrong). Render duration comparisons are meaningless between different outputs.
Step 3: Control the Environment
- Same hardware class per run. Identical instance types (or the same machine), same region, same time-of-day window to dodge cloud noise
- For server pipelines (Remotion Lambda, Meishe server rendering): same concurrency (start with 1, then test 10 and 50), same warmup policy — discard the first 2 runs per workload, time from the third
- For client-side (Meishe WASM in-browser): fixed hardware matrix instead — one modern laptop, one mid-range, one low-end — same browser versions, power plugged in, hardware acceleration enabled and verified
Step 4: Measure What Matters
Wall-clock time is the headline, but these are the numbers that decide production:
- Render time per job (median + p95 — tails break SLAs, medians flatter)
- Cost per 1,000 renders — compute metered (instance-seconds × price), plus licensing metered (per-render fees where they apply — Remotion’s metered model compounds here; Meishe’s platform license amortizes instead)
- Failure rate at concurrency — jobs errored / jobs attempted
- Queue behavior — time-in-queue as you push from 1 → 50 concurrent, where saturation begins
- Output quality parity — file size and spot-check frames; confirm equal specs actually produced equal video
Step 5: Run It Honestly
Per workload, per pipeline, per concurrency level: 10 timed runs after warmup. Record everything — timestamps, instance IDs, ambient metrics. Then swap the execution order (A-first morning, B-first afternoon) to kill ordering bias. If a run looks anomalous, note it and keep it — cherry-picking runs is how comparisons lie.
A results table worth trusting looks like:
Workload | Pipeline | Concurrency | Median | p95 | Fail% | Cost/1k
W2 | A | 10 | ... | ... | ... | ...
W2 | B | 10 | ... | ... | ... | ...What to Expect, Structurally
Without predicting your numbers, the physics to keep in mind while reading them:
- Browser-frame pipelines (Remotion) spend per-frame CPU regardless of content simplicity — light workloads carry fixed browser overhead, heavy ones multiply it by frames
- Engine-native pipelines (Meishe) spend compute closer to actual content complexity, and can shift interactive loads client-side entirely — a cost category the server benchmark won’t even see, worth measuring separately (in-browser export on your hardware matrix)
- Concurrency changes the story twice — serverless absorbs bursts elastically; fixed pipelines trade burst headroom for per-unit economics
Publish It (Even Internally)
Write the method and the raw data next to the conclusions — workload files, run logs, cost calculations. Six months from now, when someone asks “why did we pick this engine,” the answer is a folder, not a memory.
And if you run this comparison against both pipelines with real product workloads, you’ll have something rarer than a benchmark chart: a decision. Same workloads, same outputs, honest meters — may the better invoice win.