On-set Facilities Founder and inventor of the GODBOX real-time computer architecture, explains why Virtual Production is the human interface of the Metaverse.
Written by Studio VP Supervisor Asa Bailey
CEO On-Set Facilities.
Before we get going I have to say I am still concerned for the Metaverse its very much still WIP. I worry most about the timing of its true arrival. I have been wanting to put these thoughts on the page for a long time now. And as companies begin to gear up and make plans for adopting the concept of persistent 3D worlds, that time is now. Virtual Production can come out of stealth mode and stand up for what it truly is, the place where we are telling stories in the Metaverse.
On the VP stage we tell stories, design and play together, but having seen many technology waves in my time, my concern is not so much if the Metaverse is coming, it is more about it’s adoption, timing, and the media burn out as a concept handle. In a way I wish the hype machine would stop talking about it and give it a chance to fully form – hopefully around us.
Technology is like a child, it needs nurturing and understanding and that takes – time. Yet truthfully, in my view, the Metaverse is already here, we are already surrounded by persistent data, we have interoperable 3D (USD), and we have all the component parts (Blockchain, Compute, Networks etc), it’s just that as yet they are not yet all interconnected, or as yet truly human friendly interactive. At the moment it’s mostly code, developments, and strategy, all running in different tracks, or lanes on the Metaverse Highway. Here’s the story of how Virtual Production is the human gateway to the Metaverse.
Virtual Production is our Gateway to the Metaverse.
Virtual Production has become a big thing in the movie and content creation business over the past 5 years, with a boom during the pandemic as crews looked for new ways to make content. One LED manufacturer has now sold over 100,000 of its LED panels that are specifically designed for VP and that number quadrupoles when you count in the other VP LED manufactures.
Why am I going on about LED? Well, it’s the current and best way for humans to feel like they are inside the Metaverse, inside a Massive User Interface (Mui). You don’t need a VR headset on a Virtual Production Stage, although they have their uses, instead we walk into what is in fact a massive user interface. We stand inside these massive systems with hundreds of synchronised displays. Remember the holo-deck on the SS Enterprise, well now that is our reality, real-time rendered phot-real graphics and visual FX, on display all around us on a VP set.
It will be Stories that bring people into the Metaverse.
Today, on what are now hundreds of VP stages being built around the world, actors and crew shoot using real-time game engines and virtual production methods, to create content for SVOD Services, Broadcasters, Movie Studios, and Brands. Filming takes place in front of these massive LED walls, or sometimes green screens, and all of them use at least one of the most common virtual production pipelines.
On the LED walls appear augmented photoreal 3D virtual worlds that are used as virtual sets. The Metaverse is on the walls and it’s the first time humans have ever seen it at scale, with their own eyes. With the advances in camera tracking and motion capture technology it’s also possible to track people on the set and combine live action with AR VFX and animation, this creates a mixed up reality when viewed through a camera. A combined image of the real and virtual world interacting in real-time, together.
VR headsets make you feel sick, unless you are 7 years old it seems, they give adults headaches and we are not built (as humans) to put a part of our body, especially our head into a machine, again this is hardwired. It’s like holding your head underwater for a long time, it feels like you are scuba diving in a graphic reality. And like diving, fully immersed VR (which is not fully immersed as its just your head that’s immersed) is ok for a short period of time, it’s fun, but it sucks.
Another thing so far lacking from any current so called Metaverse app is graphic quality. Have you seen a VR game or any virtual world application (from the big 5 tech co’s) that actually looks any good, that actually looks like something our brain will accept as reality? No, neither have I. But when we look at the work done in many VFX houses and on VP stages, they have it nailed. Now VFX artists are turning their skills to VAD (virtual art departments) creating real-time, photo-real environments for the VP stage.
AR looks very promising but as yet graphically underwhelming. AR puts a graphic overlay onto the physical world around you. They call it passthrough when you can see through a device such as a VR or AR headset to see graphics that appear to be married with the world around you. This far from sucks as our brains still think they are in our world yet they can see the virtual world.
Back to virtual production…
VP is the gateway to the metaverse, it’s a bold statement but let me explain. Humans are hardwired to tell and listen to stories, and its not surprising that storytellers are finding the attraction creating stories in real-time on VP stages. To pull millions of user into any iteration of the Metaverse stories are the key.
Stories are how we learn, it’s even how we fall in love, we listen (hopefully) to others and we decide if we like or believe them, or not. For the Metaverse to reach mass adoption it needs to be more than just a VR immersed head, for financial gain or even technological for technical sake, The Metaverse first an foremost needs to enable people to come together and create, tell stories, stories about our past, or future, our hopes, and even our products and services.
It is no surprise that the first folks on to the Metaverse hype train were the money folks and indeed crypto is a fundamental lane on the Meta Highway. While in other lanes, quietly, with less buzz, storytellers have been entering a version of the Metaverse to capture and tell stores in real-time. Admittedly, these VP created stories are in parts, shots, or scenes, and often inserted in to final 2D formats for consumption.
But this 2D capture inside the Metaverse is just the start for VP, we are in a period of change, of re-skilling. We are now using volumetric capture, see the video above, to construct real-time 3D meshes of actors. These are immediately re-lit and rigged to enable the actor on the set to push around 3D objects in the virtual world. Step on-to a VP stage and you will see people telling stories in multiple virtual worlds (or sets). People dressed up as any imaginable character, performing on an endless opportunity of locations, sound familiar? It’s the promise we are sold of life in the Metaverse.
We are all on this journey together, but we are still in different lanes, lanes on a Metaverse Highway. In the storyteller lane, we are already going inside the Metaverse, and we are making it very real right there in front of us, right now.
We are learning how to cross lanes too, plug in 3D assets that are open and interoperable, we are reusing 3D assets in film, in games, and even in products. The standards that will enable you to switch from lane to another are on the way. USD is the html of 3D, a 3D scene description file format that holds all the information of a 3D asset in much the same way as CSS style sheets describe the layout and design of a webpage.
The blockchain enabled developers to write ownership and rights into an open ledger at first for NFTs, as these development tracks cross over to creative storytelling, we can own our 3D assets and we can share them across multiple Metaverses. All these lanes of the Metaverse Highway are slowly, but surly coming together, and it’s virtual production that will make the Meta user experience look good and feel right for users. Virtual Productions are the first fully human interactive environments where humans interact with 3D worlds without the need for combersom devices. This ‘around us’ approach to interactive design will soon break out of the film set and onto the highstreets, in our stores, theme parks, and as more and more display technologies come on line, even homes, bringing Metaverse Mui applications to the masses. Unlike most of the ‘they will’ buzz headlines currently going around about Metaverse strategy, labs, and corporate adoption, in the world of VP the headlines already read as ‘we are’ as the first humans to truly know what it like to go inside the Metaverse.
Try the VP Glossary.
Studio Virtual Production Supervisor