Face AR SDK for Web & Mobile

Ask a team what they want from a Face AR SDK and you usually get one description: a sticker that stays on the face, controls that react to a gesture, a camera view that looks the same whether the user opens a browser tab or installs an app. Then two builds start, and that single description quietly becomes two sets of behaviour that have to be reconciled before every release.

The drift is rarely a decision anyone makes. On mobile, frames arrive with predictable timing and the GPU is close to the surface. On the web, the same frames arrive through standard browser facilities, with permissions, compositing, and device quirks negotiated by the host. Teams patch each difference locally, and after a year the patches are the product.

Meishe builds its face pipeline so there is less to patch. One cross-platform audio/video processing engine evaluates the same effect definitions everywhere, and AR runs at three points — capture, timeline editing, and live streaming — through a single effect stack. What follows is what genuinely stays consistent, and what stays yours to build.

1. Product Positioning

Meishe Face AR SDK is a module of the cross-platform audio/video processing engine used across the product family. It performs face detection and tracking frame by frame, reads gestures, anchors avatar-driven props and stickers, and carries beauty and filters on the same render path. Because capture, timeline editing, and live streaming share one effect stack, an effect authored once behaves the same whether the user is recording, assembling a clip, or going out live.

The SDK does not ship a finished camera UI. Recording interfaces, permission handling, and upload flows belong to your team. It suits groups that already own their product surface and want the face layer underneath it.

2. Where web and mobile implementations actually diverge

Three forces push the two builds apart, and none of them are specific to AR.

The first is frame delivery. A native pipeline gives you frames on a schedule you control; a browser gives you frames subject to tab visibility, background throttling, and whatever the device’s own media stack decides. Any code that assumes a fixed cadence behaves differently on the two targets.

The second is effect economics. If an effect must be recompiled or re-reviewed to reach a second target, every seasonal drop becomes two tickets and one release train. Where effects ship as packages of materials rather than code, a new look is a content release everywhere at once.

The third is UI ownership. Teams that build their own camera experience tend to shape it per platform — different controls, different permission copy, different review screens. That is legitimate product differentiation. The risk is letting the platform-specific interface also define effect semantics, because then your interface becomes the source of truth for something the engine should own.

Materials as content, not code: a prop or sticker package describes what the effect does, and nothing in that description changes when it reaches another target.

3. One detection pass, several consumers

The most expensive mistake in a face pipeline is running the same analysis more than once per frame. A sticker layer wants to know where the face is. Skin smoothing wants that same answer to bound its work. Face reshaping wants it again. If each module issues its own request, cost multiplies with catalogue size, and a routine prop update silently changes the performance profile of the whole app.

Meishe shares that work. One face analysis per frame feeds every downstream consumer: tracking for prop anchoring, the beauty path for skin smoothing and reshaping, and the gesture path for interaction. Adding a sticker to the library therefore does not add a second analysis to the render loop.

That property matters more than any individual capability row, because it is what separates a catalogue you can grow from a catalogue you have to ration. It also keeps tuning coherent — when every consumer reads the same result, they cannot disagree about where the face is.

4. Capability parity you can plan around

This is what integration teams usually ask for first: not a feature list, but an answer to whether each row behaves differently in the browser.

Capability	Web	Mobile (iOS / Android)	Desktop (Windows)	Notes
Face detection and tracking	Yes	Yes	Yes	One tracker, one tuning model
Gesture detection	Yes	Yes	Yes	Hand raise, prop switching, template steps
Avatar-driven props and stickers	Yes	Yes	Yes	Same material packages per target
Skin smoothing, reshaping, makeup	Yes	Yes	Yes	Per-element intensity carried in the material
Portrait and background segmentation	Yes	Yes	Yes	Loadable as an optional module
Intelligent color correction, AI subtitles	Yes	Yes	Yes	Part of the optional AI set
Up to 8K editing, 16K image processing	Yes	Yes	Yes	Upper bound also set by host device
HDR-aware pipeline, HLG / ST 2084 export	Yes	Yes	Yes	Mixed HDR/SDR material supported
On-demand module loading	Yes	Yes	Yes	Core footprint around 3 MB

Parity here means the effect definition and its semantics travel. It does not mean every device offers identical headroom: a mid-range phone in bright sunlight and a laptop running a busy tab are different machines, and the expectation is that the engine degrades in a controlled way rather than dropping effects abruptly.

5. AR at three points with one stack

The part teams underestimate is how rarely AR lives in only one place.

In capture, the preview shows the effect while the user frames themselves, and the file being recorded carries the same rendering rather than a lookalike.

In timeline editing, an effect lands on a segment of a clip that already exists, which means the renderer evaluates the same stack against decoded frames instead of camera frames.

In live streaming, the effect renders on frames going out in real time, with no second pipeline to keep in sync.

Running all three through one effect stack removes an entire category of bug: the prop that looks right in the viewfinder but shifts once applied to a recorded segment. It also removes a category of meeting — the one where you decide which of the three looks is canonical.

6. Embedding the face layer

The snippet below is schematic. Module names, method names, and signatures are illustrative and are not the shipped API surface.

// Illustrative: mounting a face-effect session in a browser host.
import { createFaceArSession } from "@meishe/face-ar"; // placeholder name

const session = await createFaceArSession({
  surface: document.querySelector("#camera-surface")!,
  modules: ["detect", "props", "beauty"],        // requested on demand
  effects: { stack: ["skin.soft", "prop.avatar.fox"] },
  onFrame: (frame) => telemetry.sample(frame),
});

await session.props.apply("prop.avatar.fox", {
  anchor: "face",
  intensity: 0.7,
  followGesture: true,
});

const output = await session.finish({ width: 1080, height: 1920, hdr: "hlg" });

Two properties matter here. The effect list is data, so a browser host and a native host can read the same configuration without translation. And module loading is explicit: leaving out the beauty and AI modules leaves their cost out too, which is how a build that needs only props stays light.

7. Honest limits worth reading before you scope

A face tracker is not an identity system and not a scene-understanding system. It finds faces, follows them frame by frame, and infers gestures from what it can see. Behaviour degrades when a face is strongly backlit, partially hidden behind hands or hair, very small in frame, or when the subject is a photograph of a face rather than a face.

There is also a boundary that has nothing to do with recognition quality: the SDK does not ship a finished camera UI. Your team owns the recording interface, the permission flow, the asset picker, and the upload path. Teams that budget for this early treat it as product work; teams that do not tend to discover it during their first round of user testing.

FAQ

Do the same effect packages run in the browser and in the mobile app?
Yes. Props, stickers, filters, and beauty parameters are packaged as materials rather than code, so one authored package is read by every target. What differs is host capability — device decode limits, GPU headroom, browser version — not the definition itself.

Does adding more stickers slow down the render loop?
Not per prop. One face analysis per frame feeds tracking, beauty, reshaping, and prop anchoring, so a larger catalogue does not multiply detection cost. Content can grow without reopening the performance question.

Can effects used during capture be reused in recorded clips and live output?
Yes. Capture, timeline editing, and live streaming evaluate the same effect stack, so a look authored for the viewfinder can be applied to a timeline segment or rendered onto frames leaving for a stream.

What do we have to build ourselves?
The camera experience. Recording controls, permission prompts, asset pickers, and upload or publishing flows are yours. The SDK supplies detection, tracking, effects, and rendering beneath that interface.

How does the SDK behave on older browsers or low-cost devices?
Acceleration on the web happens through standard browser facilities, with no plug-in and no driver install. That keeps deployment simple, but available headroom depends on the browser and device in front of the user. What you tune are parameters that express intent — smoothing strength, prop count, module selection — and the engine degrades along those lines rather than failing all at once.

Ready to see it in your own stack? Request a free trial or talk to our integration team — we’ll help you scope the modules you actually need.