On-set lighting and reflections for Virtual Production in Unreal Engine.
In this post Director of Virtual Production Asa Bailey gives his views and a comparison between various methods for creating realtime reflections on set and why he favours those methods that will still give him options in post.
At this years NAB there was a bit of a battle brewing with some folks claiming that their new “final pixel” virtual screen walls would make green screen shooting and in fact post production a thing of the past. I don’t think so, but I have used both on set in virtual productions, I’d say they are both valid options for shooting. It’s about using the right tool for the shot.
In my view, the big problem with virtual wall technologies is that the footage is “baked” in camera and sold as “final pixel” just as they guys say in this video, and this leaves a lot less, if any options for layered work (post VFX passes) in post.
But to their credit Virtual Walls do give amazing reflections, to faces, eye’s and highly reflective surfaces (glass, paint, mirrors etc) as show in this demo from UE and Stargate Studios who’s done work on some big projects, so they do know a thing or two about VFX.
But they sing the praises of ‘final pixel” in camera is king for both creative and financial reasons. But come on guys, you know as well as I do that in the real world, in the reality of working in a studio pipeline, with studio Producers, Agencies, Talent and god knows who else who wants to have an input in to the final product, loosing the option to post process shots is too big a risk for Producers.
So would I shoot against a virtual wall, if the scene needed it and it would give the best result yes, would I use them to generate a realistic reflection on someones face who was say looking out of a window – yes I totally would and do. But on my sets they would be out of shot, so I could really use any sort of screen, it would not necessarily have to be connected to my virtual production rig and offer realtime prospective, a nice bonus but not essential.
But as a Virtual Production Supervisor or as a Director either way it’s my job to make sure that we leave the set with options, and thats why I shoot realtime in layers with a composite as an option. 80% of my shots are ready to grade and edit with minimum clean up in post. The other 20% need critical changes as often driven by other stakeholders (Agencies, Producers etc).
We shoot the talent layer (what the optical camera sees) and then we record the UE4 generated backgrounds as plates along with other realtime VFX layers on separate data layers, so that we can open them up later and VFX passes as required, indeed if required. We have the choice we can go to grade an edit with the composite files that are recorded on 10bit 4.2.2 with audio, ready to drop onto any edit timeline, or we can open them up and add more VFX passes, we can even regenerate UE backgrounds using our tracking (FBX) data, so if we decide that we don’t want that tree in the background anymore, thats ok, we simply remove it and then run the BG layer again shooting with virtual cameras in UE and export the shot and drop it back in our layer stack.
Realtime reflections are a challenge, in fact lighting in virtual production is a challenge full stop and you have to have done it many times to know what works, as they say practice makes perfect. My Gaffers and DOP’s have been lighting green screen sets for for years and they know where to put the light and how to spill just the right amount of light on a subject (optical) to make it match in the final realtime composite, but the point is if we need to, we can open up our recorded layers (optical, background, matt, foreground, composite) in post and fix any lighting issues. You may think this defeats the object of realtime production, but realistically realtime is a turbo boost to any production, it is not a golden egg laying goose. 9 out of 10 times you’ll still want to go into post production options especially if you are shooting for high end streaming shows, feature films or big brand commercials. For me, I want to shoot in realtime virtual production methods, but I also want the options to be able to take advantage of 100 year old VFX industry and all the amazing talent it holds.
Methods For Creating On-Set Lighting for Virtual Productions
Back to creating reflections in realtime, I prefer real lights rigged up on set to mimic the light that would be there in the virtual set. As you can see in the images below we had an actor walk down a neon lit corridor (the UE background) on-set we set up a number of lights to mimic the virtual set lights, so that our optical layer had inherited the scenes lighting. Having some physical lighting on-set that matches as best as possible the light you’d get if you was on the virtual stage is important. In post it really helps if there is something close to work with.
If the reflections need to move like in a driving scene, I use 3 methods.
1 – Moving practical lights
We rig up stage lighting with bells and temperatures set to match the virtual world fixed to some sort of mechanical rig that will move the lights. Or we set up moving flags infant of the lights to mimic passing buildings etc.
2 – Screens
We’ve used multiple large LCD walls to generate a reflective images. Usually positioned out of shot (so we can still shoot for chroma) the screen projects a large layer of realtime light onto the scene, we’ll run the background on the screen from within UE or as a simple video file playing while we do the take.
3 – Projectors
We also use projectors projecting moving backgrounds, in fact this is my favourite way to cast realtime moving reflections on my cast. You can use scrims and diffusion to soften the projection, the light from the projection on it’s own is often very hard, but this gives you another creative tool to tweak and get just right for your look.
I hope that helps, as I say practice makes perfect and there is no one right way, it’s about the shot, the vision, the methods and obviously your budget, but any of the above approaches can be done on any budget, its just a matter of scale and complexity.
See the shot, craft the light, shoot.