Anything’s possible with virtual production in Unreal Engine
The possibilities of virtual production are endless and the many in the industry starting to recognise just what virtual production can actually offer their productions. With the use of virtual production on the new film ‘Welcome to Marwen’ the abilities of the new wave of production methods is being heavily showcased.
“What I love is how virtual production is now being used on-set to increase creative possibilities; opening up new ways for Directors to think, beyond what’s been possible before. Directors are now using realtime virtual methods and technique, on-set” Asa Bailey Virtual Production Supervisor.
Welcome to Marwen is based on the true story of artist Mark Hogancamp who, after suffering a brutal assault, turned to photography as a means of healing. The subject of his photos was a 1:6 scale WWII-era village that he assembled in his backyard, including dozens of carefully-posed dolls representing his friends, family, and attackers as well as Hogancamp himself.
Zemeckis the director wanted the dolls to come to life in Hogancamp’s imagination and walk, talk, and act just like their real-life counterparts. Using groundbreaking virtual production techniques Zemeckis was able to transfer live actor performances to the digital doll characters while retaining each actor’s unique performance.
The production had a staggering 46 minutes of doll animation and the problem of getting the animation to be believable human-like performances, at this scale and also have them mesh completely with the real physical world at the same time. Mo-cap was the obvious option for the team but the director was concerned with the limitations of such systems and wanted the VFX team to come up with something new.
Kevin Baillie the visual effects supervisor on the project overcome this problem through vigorous testing and tried augmenting the actors with doll features which gave a hideous out of proportion look. Billie then had the idea to reverse engineer the process and augment a doll with the actors features and he quickly realised he was on to a winner.
This approach required the team to record the actors’ facial performances on a mocap stage, but with an unusual twist. Due to them using face footage they couldn’t have any facial markers in any of the ares and not allowed to wear head cams.
Lighting was also a challenge for the production due to the set having to be lit the exact same as how the final shot wold look. To overcome this challenge they built out all the doll scenes in Unreal Engine before shooting began. This way, director of photography was able to design all the lighting long before the mo-cap shoot and try out different looks to see which suited the shoot best.
The lighting also involved a lot of virtual production techniques to give the director of photography an insight into how the decisions he made on the mocap stage would actually affect the entire environment in the end result.
if Zemeckis wanted to try out different lighting, blocking, or camera angles, the UE4 team was on hand on-set to update the corresponding digital scene and give a preview of the changes in realtime. (see http://onsetfacilities.com)
Due to the actors performances being so important to the director it was essential to keep them engaged. Thanks to the Unreal Engine realtime feedback it was possible to demonstrate what the changes were actually doing by just checking the monitor which helped the actors by showing evidence instead of just a “Trust me, its gonna work out”. The actors have an instant preview of the finished product there and then.
Once all the performances were recorded, the VFX team combined the footage with hand-keyframed animation on the doll head rigs, going back and forth between the 3D scan of the actor’s head and the doll model to find the right mix of face footage and keyframing. Through this process, the team applied believable, recognisable actor performances to the dolls’ faces.
Beyond providing a means to record and transfer actors’ performances to animated characters, virtual production gives filmmaking crews opportunities for on-set creativity that just weren’t possible before.