Creating human NPCs with Accelerated Volumetric Video
How the 7thGuest was reimagined for VR
In a groundbreaking leap, Vertigo Games pioneered the use of volumetric video for NPCs - an industry first. Three decades after its initial release, The 7thGuest was resurrected in virtual reality (VR), and hailed for delivering “the best human performance ever seen” - (The Gamer). This technological feat garnered recognition within the gaming community, particularly for its transformative impact on human character portrayal. As we explore the details behind this achievement, we’ll examine how volumetric video and gaming came together, and consider the future possibilities for these technologies.
The video game market today
The 347 billion dollar gaming industry is still growing with recent technological advancements paving the way for gamemakers to attract new users through increasingly immersive experiences. The metaverse boom, albeit subject to debate and still in its infancy, has played a pivotal role in drawing attention to the profound impacts of game engines like Unreal and Unity. These software frameworks both accelerate the game development process as well as empower studios to create larger and more ambitious projects by providing pre-built systems for rendering, physics, and audio. Despite the gaming market being driven by the big three - Nintendo, Microsoft, Sony - smaller studios are carving a distinct identity by boldly challenging the industry through innovation and technology.
Driving market demand: Technological advancements in gaming
The success of virtual games depends largely on the quality of content, so it's not surprising that most advancements in graphics and resolution quickly become the industry standard. Both high-resolution screens and real-time 3D graphics rendering can significantly elevate the overall player experience; while realism is often contingent on creative choices made by individual gamemakers while designing the virtual environment and characters. To meet the changing expectations of the growing gaming community, technological advancements have to empower creative decision making and elevate the game experience - not only serve as novelty.
Vertigo Games pioneers first use of Accelerated Volumetric Video in gaming
A technical look at how 3D FMV (full motion video) was achieved
Given the ties between high-quality graphics and immersion, it was inevitable for gamemakers to start leveraging mediums like volumetric video in virtual reality gaming. Volumetric video, the process of creating a 3D asset (or digidouble) from a real human performance, has successfully been leveraged across any use case where maintaining human authenticity is paramount - training simulations, celebrity ambassadors, fan engagement, and more. While volumetric video has been compatible in game engines for years, the heaviness of the medium restricted gamemakers until Arcturus announced its new format in 2023: Accelerated Volumetric Video (AVV).
The weight of graphics in gaming is often a determining factor, essentially meaning how heavy a component is and whether it risks slowing down the game or decreasing the user experience will affect its inclusion in games. Virtual worlds in and of themselves are heavy, so additional elements need to be lightweight to avoid placing additional stressors on the user's hardware or causing playback issues. Prior to Arcturus’ advancement, there was no solution to deliver volumetric characters that could playback in real-time while retaining graphic quality. In fact, most stress tests began to fail with one, or sometimes two volumetric characters and often suffered from unwanted artifacts or texture desyncs as the hardware struggled to keep up with the demand of the scene and the character. Accelerated Volumetric Video was a significant technological advancement because it eliminated these barriers by attacking the problem at the source. Watch our CTO, Devin Horsman, explain how Arcturus developed a new way to compress volumetric video at the 2023 AutoDesk Vision Series.
Arcturus re-imagined how the medium was being rendered and compressed, leading to designing a new codec, specifically for volumetric video. With rendering speeds faster than conventional codecs and at least 5x faster than current encoding solutions, AVV doesn’t only deliver performance but also impressive detail support. Progressive Level of Detail (LoD) management and selective decimation allow for rendering to happen in proportion with the user's view, lessening the load on hardware and improving graphic quality control. Motion is often where the immersion wall is broken, but AVV enables motion blur and temporal anti-aliasing, so human performance remains human throughout the entirety of performance.
While Vertigo Games pioneered the first use case, they won’t be the last. With technical barriers now removed, it's only a matter of time before this medium becomes an industry standard in delivering human non playable characters (NPCs).
Behind the scenes: The making of human NPCs
Computer-generated (CG) graphics have advanced leaps and bounds in the last twenty years, but there’s something about an authentic human performance that's hard to compete with. Add to it the length of time to create a believable CG human character, and one that doesn’t tip the uncanny scales in motion, it's not surprising that volumetric video is emerging as a viable rival to CG characters in gaming and extended reality.
“I can still remember being mesmerized when I played the original. It all felt so real to me, and that's what we wanted to achieve with this resurrection of 7th Guest. Every interaction had to be authentic to the experience, it was all about having that sensation of being in the same place with the actors, which made volumetric video the only option for us.” - Paul van der Meer, Head Gamemaker, Vertigo Games
Let's take a look at what it takes to develop a volumetric character.
1. Capturing human performance
The super power of volumetric video is both its authenticity of human capture, and of capturing motion. In games like The 7th Guest, where individuals were carefully cast, dressed, and rehearsed, volumetric video was able to retain the authenticity of the entire performance. The process starts with an individual entering a volumetric capture facility where numerous cameras record that actor’s every action. Below, the actors of the 7thGuest show the volumetric capture studio at 4DR Studios in The Netherlands.
2. Post-production of volumetric content
Like any other type of video production, the post-production of volumetric video is just as critical to the capture to achieve high quality results. In this stage, every camera's feed is synched together to create a volumetric asset which is compressed for seamless playback. Depending on final output, performing other editing actions like blending the best takes together, or optimizing textures might be necessary to achieve specific results. Vertigo Games performed the post-production using HoloSuite’s designated path for Accelerated Volumetric Video.
3. Integrating volumetric characters into game engines
The final step in delivering volumetric characters as NPCs is integrating them into whatever game engine is powering your experience, and then rigorously testing the characters in the scene and on the target hardware.
“It's amazing to see how the original performance played out throughout the game. An eye roll, a smirk, these subtleties we captured and delivered through volumetric video that made the narrative that much more engaging.” - Paul van der Meer. Head Gamemaker, Vertigo Games
The volumetric advantage: Community response to AVV
Vertigo Games’ commitment to authenticity didn’t go unnoticed, with both the gaming community and media recognizing the value of authentic human characters in the game.
“Most realistic human characters I’ve ever seen in VR” - TheGamer
“Fantastic Resurrection of a Puzzle Classic” - Gaming Trend
“A spooky puzzle treat with way more to offer than nostalgia” - Eurogamer
The positive reaction was far reaching, with The 7th Guest also taking home the Best Immersive Game 2023 Award at the Raindance Film Festival.
Not just for VR: Volumetric video a viable NPC in gaming
Though its first gaming use case came in the form of VR, volumetric video has longer legs than just extended reality. In fact any game leveraging 3D or built in a game engine can implement the medium today. 2023 had its first use case, but that number will only climb as the medium increases in fidelity and becomes more widely accessible.
“Accessibility has always been paramount to us at Arcturus - we know the medium is cutting edge, but we’ve been at its helm since its inception, and we’re here as partners to help you achieve human performance wherever you need it. Just because you haven’t seen it doesn’t mean it's not possible, we really encourage open dialogue with potential partners and are motivated to expand the use of the medium.” - Devin Horsman, Chief Technology Officer, Arcturus
Step into the future of storytelling
Arcturus is an end-to-end partner for volumetric experiences. Whether it's delivering technology, servicing a project, or filling in the gaps on your team, we’re here to deliver human authenticity in any project. Learn more about our volumetric production solutions and how we work with game studios. Contact us today.
Download The 7th Guest today!
The 7th Guest is available for purchase on PlayStation VR2, MetaQuest and Steam VR. Learn more about the game and download options here.
