Virtual Production in Unreal Engine

Virtual Production in Unreal Engine

Virtual Production
VFX Breakdown of virtual production in unreal engine by on-set facilities.

To round off our year, we set our team the challenge of producing a music video in just 12 hours using OSF Virtual Production Systems, from first shot to uploading to the OSF youtube channel. With no post-production.

Created entirely using OSF virtual production systems in Unreal Engine, this test demonstrates high end mixed reality, with real-time compositing of 3D virtual sets, with real-time character, foreground and chroma key layers, with real-time rendering and on-set colour grading.

The test video was all filmed on the OSF MR Factory Stage, a 500m virtual production green-screen stage at On-set Facilities in Madrid.

Real-time compositing of live action actors in virtual sets powered by Unreal Engine. With realtime VFX and on-set colour correction, the only post production required for this video was editing and mastering the 4k file in Adobe Premier. Directed by Asa Bailey, Virtual Production Supervisor at On-set Facilities.

The Virtual Production

Shot by a small crew of just myself on the camera (Alexa Mini), 3 system operators and 2 willing actors, it took little more than 3 hours to shoot the test and a previous 5 hours in preproduction to find our props, texture, light and then bake the 3D set. We used C4D to model the set and we rendered in Octane before baking into Unreal Engine (plugin on the way).

The idea of this test was to show VFX Supervisors, Directors and Producers that it is possible to produce quick cost effective “ready to edit” footage with the OSF realtime virtual production system.

Virtual Production Spectrum

The rules of this test said no post allowed. But, do you remember 8bit and then 16bit computer games? No, well once upon-a-time computer games looked well dodgy, and its a bit like virtual production methods today (written Dec 2018). To really use the technology for high end film productions you’ll still want to open up the files and give them a polish in post. But the more effort you put into pre-production (creating 3D sets and realtime VFX) the better.

16 bit game graphics

Remember 16bit game graphics, virtual production requires 32bit and 64bit.

But what we are proving is that for content production, today virtual production methods can greatly reduce the post-production process, freeing up production budgets to spend more on-set and not in post.

We had just 12 hours. It was a test in discipline for the crew and also an annual quality bench mark of the realtime footage from our systems. What you are seeing in the test are 4k video files from our systems, as recorded on-set.

Virtual Production is a Tech Wave

Virtual production is an emerging space and it involves realtime visual effects, live audio (3D), realtime character animation and realtime motion capture. With developments moving fast, the amount of post-production required to deliver final shots in realtime is reducing rapidly, month to month.

In OSF film work, 60% of close and mid shots are good to go directly to the edit, with just 40% needing to be opened up in post and given a polish. Wide shots are more tricky, but we are making big improvements in colour matching optical and virtual layers and in developing AI shadows and reflections.

Virtual Production Workstations.

OSF Realtime Machines Custom Built Virtual Production Workstations.

 

This year we’ve seen a massive increase in quality, as OSF push towards 32bit and even 64bit colour and graphics in Unreal Engine. Realtime Virtual Production takes serious on-set power. In response OSF have started a line of OSF Realtime Machines, Intel and Nvidia based workstations that are optimised to power realtime virtual productions and AI driven realtime VFX.

The Technology Used in This Test

OSF realtime virtual production system integrations are based on OSF Realtime Machines, running Unreal Engine with realtime rendering and virtual camera cinematic capabilities.

Realtime camera tracking is taken care of by Mo-sys Engineering on top of  the optical camera (Alexa Mini). Blackmagic hardware also plays a large part in OSF solutions, incorporate 4K and 8K broadcast standards.

On-Set Facilities Madrid Studios

OSF Green Screen Stage and Realtime Virtual Production Systems.

But what about Post Production Options?

In this test case, the only post production allowed was to pull the footage from our data recorders and onto a timeline in Adobe Premier. We then cut the rushes to a library track from Audio Networks and that was it.

But, we could have opened the recorded layers and data – Matte – Foreground – Background and RAW optical layers, into a long list of post production software and clean and tweak until our hearts (and clients) content.

For anything more than a test, we’d still prefer to open up the files in post as all assets from the system are editable in post production applications.

For instance we’d have liked to clean up the chroma, improving the look and feel of the virtual sets. But that was not the idea in this test – it was strictly no post allowed, what you see is what you get, in realtime.

Have faith.
Asa Bailey
Director and Virtual Production Supervisor
linkedin.com/in/asabailey