Recently, Meta and BMW are teaming up to explore the integration of augmented reality (AR) and virtual reality (VR) technologies into smart vehicles, aiming to revolutionize the way we travel.
The partnership between Meta and BMW was announced in 2021 and focuses on researching how AR and VR could enrich the passenger venture inside moving vehicles. The foremost goal is to form a hands-free and immersive environment that delivers communication, entertainment, and utility beyond the traditional screens and instruments seen in today’s vehicles.
Claus Dorrer, Head of BMW Group Technology Office USA in Mountain View, expresses that, while it is still early to provide exact details on when and how this technology will reach customers, there are myriad exciting possibilities for XR (Extended Reality) devices in vehicles. These conceivable applications include assisting drivers in finding their cars in congested parking lots, cautioning them about road hazards, and displaying crucial vehicle information.
He further added that the future implications of AR glasses and VR devices hold great promise for both drivers and passengers. Through their research partnership with Meta, BMW aims to explore and define immersive XR experiences that can seamlessly integrate into vehicles, paving the way for the incorporation of these devices into cars.
Overcoming Challenges: Tracking Movement in Moving Vehicles for Meta Quest 2
One of the challenges in achieving this vision is the tracking of movement in a moving vehicle as the Meta Quest 2, is equipped with various sensors which include inertial motion sensors (IMUs) and cameras. These sensors work together to determine the location and motion of the headset. Nonetheless, when it comes to utilizing VR headsets inside moving vehicles, a challenge arises.
The tracking technology utilized in VR headsets, like Oculus Insight, leans on both IMUs and cameras to estimate the headset’s position accurately. IMUs measure acceleration and rotational velocity relative to the world, while cameras observe the motion relative to the inside of the car. In a moving environment, specifically in a non-inertial reference frame like a vehicle, these two sensor modalities contradict each other.
Thus, the conflicting nature of these two sensor modalities presents a challenge for VR headsets like the Meta Quest 2 in maintaining stable virtual content while the vehicle is undergoing acceleration or making turns.
Meta and BMW Collaborate to Overcome VR Headset Challenge in Moving Vehicles
To tackle this challenge, Meta and BMW have collaborated to integrate the IMU data from BMW’s car sensor array in real-time with Meta’s Project Aria research glasses’ tracking system. This integration authorizes the system to calculate the glasses’ location relative to the car, providing the necessary data for stable and accurate virtual content display.
The integration of BMW’s IMU data into Meta’s tracking system marks a significant milestone in their collaboration. This breakthrough allowed the researchers to seamlessly attach virtual objects to a moving car using a digital representation of the vehicle. By utilizing the Meta Quest Pro headset with the enhanced tracking system, they have already showcased captivating Virtual as well as Mixed reality experiences for passengers inside moving cars.
Meta and BMW’s to Enable World-Locked Rendering
The next phase will concentrate on incorporating the car’s location proximate to the external world, which will facilitate world-locked rendering and open up further possibilities for immersive content.
Meta and BMW furthermore aim to utilize the car’s precise 6DOF positioning system to render world-locked virtual content beyond the confines of the vehicle. This may comprise identifying landmarks, points of interest, and other relevant information, delivering highly valuable capabilities for personalized AI assistants and future AR glasses.
Meta and BMW’s groundbreaking research collaboration has opened new opportunities for integrating AR and VR technology into moving vehicles, promising enhanced passenger experiences.
