Integrate a virtual character with video: signal, transparency, and timing

Bring in the camera and choose the compositing stage

1. Camera / format
2. Documented input
3. Engine / composite
4. Output / surface

A camera may arrive as SDI/HDMI through a capture device, as a USB video device, or through a documented IP route. A USB connector alone does not establish UVC output. Record what the source produces and the application accepts, alongside cables/adapters.

• Source: model, selected output, resolution, exact frame rate, and scan; include embedded audio. Identify who can change these values.
• Input: card/device, driver, OS, and application build. Confirm format support and availability to that process.
• Composite: decide whether the character is added inside the engine, another application, or a switcher. Mark cropping, scaling, and audio sync stages.
• Destination: canvas, physical output, processor, and surface. Keep a preview distinguishing received source from delivered composite.

Unreal’s Media Framework separates media source, texture, and audio; capture-card sources use specific plugins. A source working in one player does not imply equivalent support in every engine. Use the capture guide matching that route to check input and synchronization.

Proposed rehearsal: start with a stable pattern/source, then camera, then composite. Hold the format fixed and check orientation, aspect ratio, color, levels, and voice. If source switching fails, record before/after modes and the affected stage. Preview picture does not confirm final delivery. Prepare local content to remove a lost camera without leaving the character over an incorrect background.

Transparency: alpha, keying, and key/fill

A black background is not automatically transparent. Alpha describes the image’s contribution when compositing; chroma key calculates a mask from a color. A figure already composited over the final background may not need transported transparency.

Key/fill separates color picture and key signal. vMix 29 documents separate outputs and Straight/Premultiplied modes for selected devices. This example does not confirm other engines, cards, or ports. The signal pair needs compatible format and timing at the receiver.

• Agree delivery file/texture/signals, receiving application/build, color space, and alpha handling. Container extension alone is insufficient.
• Test semitransparent edges over light and dark backgrounds: hair, shadows, and highlights expose halos or clipping. Compare against the approved render.
• Check whether color is premultiplied and what the receiver expects. A mismatch may darken edges; do not blindly alter the entire character.
• For two signals, identify key versus fill, frame rate, resolution, and alignment. Rehearse loss of one signal and transition to local precomposited content.

Proposed test: a fictional figure with opaque, transparent, and gradual areas, first still and then moving. Inspect the final composite; an editor’s transparency checkerboard does not validate end-to-end transport. Document results per route/version. If alpha is not preserved, assess earlier compositing or another approved method and test it with the selected receiver.

Spout or Syphon: pass pictures between applications

Spout targets video sharing between Windows applications; Syphon shares frames between macOS applications. They are candidates for connecting a visual engine with compositing/playback on the same system. They do not by themselves replace a venue-to-venue link or video network design.

• Identify sending/receiving applications, versions, plugin/integration, and OS. A program appearing in a list does not confirm every build.
• Name the output and select the correct sender. Record resolution, frame rate, alpha, and behavior when the sender restarts.
• Review active GPU and interprocess route against both applications’ documentation. With multiple GPUs, test the real combination before assuring continuity.
• Draw voice as a separate path and confirm transport. Receiving a texture does not prove audio or synchronization.

Proposed rehearsal: send a pattern with frame number and transparent edge, receive, composite, and inspect the destination. Open processes in the intended order, close/reopen the test sender, and check whether the receiver recovers or needs source reselection. Record load and smoothness with the complete project.

If the picture vanishes, the operator switches to local content while sender, receiver, and state are identified. Measure the complete delay through both applications and the final display, even if the sharing mechanism is described as low latency. Physical output and screen processing add stages. Document this interface alongside both application projects so it can be restored.

Latency: measure the complete reaction

Separate data/image capture, filtering, processing, rendering, transport, and presentation. Someone moving a hand cares when the surface changes, not only the engine’s reported frames per second. Conversation also depends on the relationship between mouth and voice.

Arithmetic example, not a measurement: at 60 fps, one frame period is 1000/60 ≈ 16.67 ms; at 30 fps, ≈ 33.33 ms. This does not mean every stage takes one frame or total delay equals that period. Buffers, evaluation, and display can add waiting while other tasks run in parallel.

• Agree the start event and observed response: fictional button press→visual change, authorized gesture→reaction, or voice→output. Do not mix measurements of different scopes.
• Use a common observation method for start/end where possible; record temporal resolution and uncertainty. For separate clocks, verify their relationship before subtracting timestamps.
• Repeat with intended project/load, recording resolution, exact rates, sensor mode, network, and output. Record variation, not just the best value.
• Distinguish estimated, documented, and measured. A manufacturer’s component figure is not an end-to-end measurement.

Proposed test: compare fictional stimulus and graphic response first, then add real input and the surface. Measure voice and picture paths separately and check perceived conversational sync. If the result cannot follow the action, adjust filters, content, or rules and repeat the measurement to check whether the response suits the action. Keep prior settings so changes remain reversible.