The arrival of Realtime Virtual Production comes thanks to the advances in realtime technologies, a shift to GPU enhanced hardware and software and the ever increasing convergence of digital technologies. To better understand the potential scope for the development of Realtime Virtual Production methods, OSF created the Virtual Production Spectrum.
The Virtual Production Spectrum is a tool that shows the scope of virtual production as a way to creating final-image-quality 2d and 3D content in realtime. All shots in RVP fall somewhere on the Virtual Production Spectrum. OSF
At one end of the spectrum you have 100% physical production, when shooting 100% live action. In the middle you have Augmented or Mixed Reality, this is where virtual production methods combines live action elements with virtually created elements (realtime VFX, virtual sets, realtime animation etc.) On the far right of the The Virtual Production Spectrum is Virtual Reality, this is where the final image is created as a 100% digital image, in realtime.
Since the birth of the film, filmmakers have sought out the most fantastic filming locations and used mechanical and practical effects to tell their stories. On a virtual sound stage, any real-world location can be pre-captured and then digitally recreated, as a photorealistic virtual set.
By combining live action with mixed reality and virtual reality on a virtual sound stage, Virtual Production methods and realtime graphic systems are today capable of producing final-image-quality footage for a number media and entertainment applications.
Realtime Virtual Production combines the best of live action, mixed reality and virtual reality to enable limitless story telling opportunities.
Deep compositing with no green screen requires a volumetric, depth view of the scene. The Intel Realsense range of cameras are a consumer level mass product that allows creators to combine VR sets with real world images based on depth, so that elements like background and foreground can be easily separated from the camera image, or more precisely layered in the correct order so that your picture makes sense.
VFX Breakdown of virtual production in unreal engine by on-set facilities.
To round off our year, we set our team the challenge of producing a music video in just 12 hours using OSF Virtual Production Systems, from first shot to uploading to the OSF youtube channel. With no post-production.
Created entirely using OSF virtual production systems in Unreal Engine, this test demonstrates high end mixed reality, with real-time compositing of 3D virtual sets, with real-time character, foreground and chroma key layers, with real-time rendering and on-set colour grading.
The test video was all filmed on the OSF MR Factory Stage, a 500m virtual production green-screen stage at On-set Facilities in Madrid.
Real-time compositing of live action actors in virtual sets powered by Unreal Engine. With realtime VFX and on-set colour correction, the only post production required for this video was editing and mastering the 4k file in Adobe Premier. Directed by Asa Bailey, Virtual Production Supervisor at On-set Facilities.
The Virtual Production
Shot by a small crew of just myself on the camera (Alexa Mini), 3 system operators and 2 willing actors, it took little more than 3 hours to shoot the test and a previous 5 hours in preproduction to find our props, texture, light and then bake the 3D set. We used C4D to model the set and we rendered in Octane before baking into Unreal Engine (plugin on the way).
The idea of this test was to show VFX Supervisors, Directors and Producers that it is possible to produce quick cost effective “ready to edit” footage with the OSF realtime virtual production system.
The rules of this test said no post allowed. But, do you remember 8bit and then 16bit computer games? No, well once upon-a-time computer games looked well dodgy, and its a bit like virtual production methods today (written Dec 2018). To really use the technology for high end film productions you’ll still want to open up the files and give them a polish in post. But the more effort you put into pre-production (creating 3D sets and realtime VFX) the better.
But what we are proving is that for content production, today virtual production methods can greatly reduce the post-production process, freeing up production budgets to spend more on-set and not in post.
We had just 12 hours. It was a test in discipline for the crew and also an annual quality bench mark of the realtime footage from our systems. What you are seeing in the test are 4k video files from our systems, as recorded on-set.
Virtual Production is a Tech Wave
Virtual production is an emerging space and it involves realtime visual effects, live audio (3D), realtime character animation and realtime motion capture. With developments moving fast, the amount of post-production required to deliver final shots in realtime is reducing rapidly, month to month.
In OSF film work, 60% of close and mid shots are good to go directly to the edit, with just 40% needing to be opened up in post and given a polish. Wide shots are more tricky, but we are making big improvements in colour matching optical and virtual layers and in developing AI shadows and reflections.
This year we’ve seen a massive increase in quality, as OSF push towards 32bit and even 64bit colour and graphics in Unreal Engine. Realtime Virtual Production takes serious on-set power. In response OSF have started a line of OSF Realtime Machines, Intel and Nvidia based workstations that are optimised to power realtime virtual productions and AI driven realtime VFX.
The Technology Used in This Test
OSF realtime virtual production system integrations are based on OSF Realtime Machines, running Unreal Engine with realtime rendering and virtual camera cinematic capabilities.
Realtime camera tracking is taken care of by Mo-sys Engineering on top of the optical camera (Alexa Mini). Blackmagic hardware also plays a large part in OSF solutions,incorporate 4K and 8K broadcast standards.
But what about Post Production Options?
In this test case, the only post production allowed was to pull the footage from our data recorders and onto a timeline in Adobe Premier. We then cut the rushes to a library track from Audio Networks and that was it.
But, we could have opened the recorded layers and data – Matte – Foreground – Background and RAW optical layers, into a long list of post production software and clean and tweak until our hearts (and clients) content.
For anything more than a test, we’d still prefer to open up the files in post as all assets from the system are editable in post production applications.
For instance we’d have liked to clean up the chroma, improving the look and feel of the virtual sets. But that was not the idea in this test – it was strictly no post allowed, what you see is what you get, in realtime.
Here’s a quick introduction and round up of what’s on offer Tax Break or as what’s called ‘soft money’ to Producers who are looking to produce films and television in the UK.
UK Film Tax Relief: Up to 25% cash rebate of qualifying expenditure. Must pass a cultural test or qualify as an official co-production (that is, made under UK’s co-production treaties). Project cap: 80% of qualifying expenditure. Minimum spend: 10% of qualifying production expenditure
Google – National Funding: National funding incentives available to international producers include BBC Films, BFI Production & Development Funding, Film4 and Pinewood Pictures. Also see Regional Funding: Regional incentives include the following:
(1) the Yorkshire Content Fund (will invest the lower of up to 10% of the total production budget or up to £500,000);
(2) Ffilm Cymru Wales Production Funding (provides grants of up to £300,000 for writers, directors and producers who are either born or currently residing in Wales working in English or Welsh);
(3) Creative Scotland Screen Funding (provides single project development Funding of £3,000-£50,000, Slate Development Funding of £50,000-£100,000, production funding of £50,000-£500,000, and distribution and exhibition funding of £5,000 – £15,000);
(4) Northern Ireland Production Funding of a maximum of £800,000 for feature film and television production and £500,000 for interactive content production, up to a ceiling of 25% of the overall project budget. Funding is in the form of a recoupable loan with profit participation or in limited circumstances a grant. Available to productions with more than 65% of funding already in place.