Realtime Compositing with Realtime Pipeline™ for Unreal Engine

Realtime Compositing with Realtime Pipeline™ for Unreal Engine

Realtime Pipeline™

Asa Bailey Director

WAKE UP 2
Director Asa Bailey
Production MR Factory
Agency Growthland
Client NTT Data

Realtime Compositing in UE4 with Realtime Pipeline™

Take a look at the future of realtime content production. Live composited on-set, the production made a saving in post-production costs while at the same time giving the Director, actors and crew the ability to work together with immediate feedback.

Realtime Compositing and VFX Octane Render and Unreal Engine

Director and Virtual Production Supervisor Asa Bailey directed WAKE UP 2 in realtime using the OSF Realtime Pipeline™ the now available end-to-end realtime virtual production solution, developed with Partners Brainstorm, Octane Render, Black Magic, Mo-Sys and Unreal Engine (UE). The project was produced by MR Factory on their 500m green screen stage in Madrid.

Octane Render Unreal Engine Realtime Compositing

Octane Render Unreal Engine Realtime Compositing.

The key is to give your teams enough time to pre-produce all the assets you’ll need on set, but if you need to change anything post shooting, it’s OK as you can recreate all the Unreal backgrounds in post using the systems saved camera FBX data.

“The ability to shoot VFX and composite in realtime is a game changer for me as a director, Im really looking forward to sharing these techniques with other directors and producers, better on-set story telling and better creative decisions are just two immediate benefits” says Bailey “shooting in realtime also gives crew and talent a more tangible world to perform in.”

All the virtual sets in Wake Up 2 were rendered in Octane Render and then baked into UE for realtime compositing on-set. But if the sets needed changing after the shoot, due to client changes or creative considerations, that was OK as the system saved the FBX data from the on-set shoot, meaning backgrounds in UE could be re-run after the shoot to recreate any backgrounds that still needed work in post, and then composited in a traditional sense.

Realtime Pipeline™ System Outputs

Realtime Pipeline™ is an open modular virtual production system that works with any tracking technology, game engine or configuration. This means that productions are not limited to shooting in realtime, Producers and Directors can decide to go with shots composited in realtime and saved on-set, or the saved passes can be opened later by post production artists for post processing.

What you take away from set when you shoot with Realtime Pipeline™

  • ForeGround in 4:2:2 10bit in ProRes, Uncompressed or Avid format.
  • BackGround in 4:2:2 10bit in ProRes, Uncompressed or Avid format.
  • Matte in 4:2:2 10bit in ProRes, Uncompressed or Avid format.
  • Composite in 4:2:2 in ProRes, Uncompressed or Avid format.
  • FBX Data of the camera movement: Position (x,y,z) Rotation (x,y,z) Zoom (data) Focus (soon)
  • RAW camera files in camera native format (Alexa Wide Gamut or Rec 709 and LOG Curve)
On-Set Facilities

On the set of Wake Up 2 Directed by Asa Bailey.

With monitors and big screen projectors on-set, everyone had a great view of the realtime production and when action was called both the cameras and the virtual production system jumped into life recorded everything in sync, locked by timecode. With on-set colour grades applied and matched lighting, everyone on-set could also see the mood and feeling of the films they were making. The films were written by the creative team at brand agency Growthland and the scripts covered future business cases for all sectors of NTT’s digital framework. You can see all the films of the NTT youtube channel.

Everything Comes Together in Unreal Engine

In Wake Up 2 3D animation was composited in later in post, but in time Jon Gress, Chief Pipeline Officer at OSF said “animations and particle effects will all make it into the OSF Realtime Pipeline™ on-set system. With Unreal Engine’s Chaos High-Performance Physics and Destruction System and realtime rendering capabilities, soon we’ll be bringing pre-vis and final pixel effects to set” said Gress.

Chaos High-Performance Physics and Destruction System Real-Time Tech Demo | GDC 2019 | Unreal Engine

Bailey goes on “realtime production is about creative possibilities, working and communicating faster on-set and being more in control. I like how working with Realtime Pipeline™ it puts the Director back in control of decisions on-set, it enables me to communicate complex ideas in a more immediate and visual way. Then I like how virtual production in Realtime Pipeline gives me options. I have the option to go for final pixel on-set or to take the shots in to post to complete the look using additional CG and VFX.”

Realtime Pipeline™ is available as a service or as a total turnkey solution for broadcasters, VFX facilities, studios and stages worldwide.