Vrgineers and ManoMotion are set to unveil an innovative breakthrough in pilot training at the AWE USA 2023 event in Santa Clara, CA. This groundbreaking development introduces an AI hand-tracking algorithm designed to enhance the immersive experience of pilot training scenarios in virtual and mixed reality environments. The remarkable feature, exclusive to XTAL Mixed Reality headsets, enables real-time hand-tracking functionality specifically optimized for cockpit settings.
The key differentiator of this solution lies in its integration of pass-through RGB cameras and an advanced AI algorithm. By harnessing this unique combination, the new hand visualization algorithm effectively eliminates the background and allows for a focused display of the hands within virtual environments. Using the RGB camera stream, ManoMotion leverages this data to visualize the hands in both virtual reality (VR) and mixed reality (MR) simulations. It is important to note that this development was directly tailored to meet the specific requirements of Vrgineers’ pilot cockpit tracking needs.
The overarching goal of the mixed reality cockpit hand tracking project is to create a proof-of-concept, AI-driven hand visualization functionality for pilot cockpit training. By utilizing the real-time RGB camera stream, the system detects and isolates the pilot’s hands, seamlessly overlaying them onto the virtual reality scene. The development process involved the collaboration of Vrgineers and ManoMotion, who worked together to gather training data in the form of video feeds capturing users interacting with aircraft cockpit instrument panels. Subsequently, the real-time data stream undergoes processing, leveraging a pre-trained hand-tracking model to separate the hands from the live video stream and render them in a visual representation.
“We are thrilled about our cooperation with Vrgineers,” says Lars Österberg, CEO of ManoMotion, and adds: “Enabling pilots to see and interact with their real hands in VR helps make the cockpit pilot training solution even more immersive.”
This pioneering integration of AI hand-tracking technology and the cockpit environment marks a significant advancement in the field of pilot training. The resulting immersive experience offers enhanced realism and interactivity, further bridging the gap between virtual simulations and real-world applications. Vrgineers and ManoMotion’s collaboration exemplifies their dedication to pushing the boundaries of technological innovation and their commitment to meeting the specific needs of pilot training in today’s rapidly evolving landscape.
“As a virtual and mixed reality simulation specialist, we always try to find new innovative ways to increase the immersion of our solutions. Vrgineers was the first company to fully integrate infrared hand-tracking into a virtual reality headset. And now, together with ManoMotion AI recognition using visible light, we are doing the same for the mixed reality,” says Marek Polcak, CEO of Vrgineers, and adds: “By supporting both technologies, we can combine their strengths to deliver the very best existing hand-tracking available in any headset on the market.”
