Get in touch with our experts

Thank you for submitting the form we'll be in touch soon

How NASA is using mixed-reality for astronaut medical training

Did you know NASA astronauts receive only 40 hours of medical training before space exploration missions? Deep space missions can last between 1-3 years, meaning those forty hours of training may have to be recalled over 1000 days later.

To combat this issue, the Translational Research Institute for Space Health (“TRISH”) sought to create an entirely new training experience for NASA; one that was hyper-real, scalable and interactive, so astronauts could further their medical training while in-flight. Led by Panogramma, Radiant Images, and Dr. Roger Dias , MD, PhD, MBA at the Brigham and Women's Hospital and Harvard Medical school, in collaboration with several other companies, a vision for a mixed reality (MR) medical training experience took shape.

Building a human experience

Knowing that the medical scenarios needed to be as human as possible, the team chose to explore volumetric video. Panogramma brought in Metastage, Sony, and Arcturus to handle the delivery of this medium. Volumetric video creates a digital double of the recorded human. And while it would lend the project the human authenticity it demanded, there were several other pieces that would need to come together. The experience would also need to:

Support Q&A interaction: Present an issue and a variety of solutions from which astronauts could select from and learn.

Combine photogrammetry and multiple volumetric video captures in a single experience: This required integrating volumetric video captures across different capture technology systems and having characters interact with 3D objects captured with photogrammetry.

Reach small enough file sizes for a seamless experience: Volumetric video produces very large files. Video by itself is already large; adding the ability to tumble around the video and view it from any angle a viewer chooses, magnifies that file size and complexity when it comes to editing that footage into a story.

Support multiple platforms: The experience would need to be viewable on several different platforms, each with their own set of requirements and required optimizations for playback. The experience would need to be available on a PC-powered virtual reality headset (e.g. Vive), mobile virtual reality (Oculus) and augmented reality devices (iOS/Android).

Bridging the gap between idea and execution

With Metastage and Sony performing the volumetric capture, it was up to Panogramma to edit and deliver the final volumetric video. They chose to work in HoloSuite, which enabled artists their artists to:

Build Q&A interaction via a branching narrative: Using blending tools and workflows with common artist software like Maya, artists created a branching narrative where depending on the answer the viewer provides, the video seamlessly blends with the selected path.

Combine footage from two different capture systems: In today's market, without the purchase of a post-production tool like HoloSuite, the tools you have available to edit your volumetric capture are largely dependent on where you performed your capture. HoloSuite is the only capture-agnostic software that can easily combine footage from various systems in the same experience and stream them together in the final project.

Increase visual quality with texture clean up: Digital artifacts can appear as floating geo surrounding a capture. With custom workflows to remove these artifacts, artists were able to drastically increase the final visual quality of the experience.

Compress the experience for optimal playback and engagement: HoloSuites’ open standard allows significant size reduction of the volumetric video while retaining visual quality, thereby enabling the volumetric video to playback seamlessly on any device.

Deliver across multiple platforms: The HoloSuite player works with wired and wireless VR headsets, AR devices (iOS and Android), and WebXR.

The future of training

Due to the nature of working with NASA and the Department of Defense, it is unlikely that we will receive results of their internal testing. Though we can anticipate excellent results for memory retention and viewer experience, one thing we’re certain of today is this is only the beginning - volumetric video will revolutionize training experiences in the years to come.