There\u2019s nothing quite like the call of the open road. Whether it\u2019s quality conversation on an epic road trip, enjoying a podcast or your favorite album, just about everyone has some fond memories in cars. But what if we could make our time spent in cars more productive, social and entertaining than ever before?<\/span><\/p>\n That question is at the heart of <\/span>Meta\u2019s partnership with BMW<\/span><\/a>. Announced in 2021, the goal of this joint research project is to explore how augmented and virtual reality (AR and VR) could one day be integrated into smart vehicles to enhance the passenger experience.<\/span><\/p>\n If we get it right, this technology could revolutionize travel in cars, trains, planes and beyond, unlocking new forms of hands-free communication, entertainment and utility \u2014 giving us far more value than the screens and instruments we\u2019re used to seeing in vehicles today.<\/span><\/p>\n VR headsets are equipped with a number of sensors but moving vehicles pose a tricky challenge, because tracking technology like Oculus Insight use both inertial motion sensors (IMUs) and cameras to precisely estimate the headset\u2019s location and motion.<\/span><\/p>\n In a moving environment \u2014 more precisely, in a non-inertial reference frame \u2014 these two modalities are in conflict as the cameras observe motion relative to the inside of the car, while the IMUs measure acceleration and rotational velocity relative to the world. That mismatch means a VR headset like <\/span>Meta Quest 2<\/span><\/a> can\u2019t currently display stable virtual content when traveling inside a vehicle as it turns and\/or accelerates.<\/span><\/p>\n At least, that is, until now.<\/span><\/p>\n To solve this problem, we collaborated with BMW to incorporate IMU data from a BMW car\u2019s sensor array in real time into the tracking system of our <\/span>Project Aria research glasses<\/span><\/a>. This additional information allows the system to calculate the glasses\u2019 location relative to the car.<\/span><\/p>\n That was a huge feat because after transferring the tracking system to a Meta Quest Pro, it allowed us to accurately anchor virtual objects to a moving car using a digital twin of the car. We\u2019ve been able to demo some compelling virtual and mixed reality passenger experiences in moving cars using this new tracking system and <\/span>Meta Quest Pro<\/span><\/a>. The next step will be to add the car\u2019s location relative to the world, which would enable world-locked rendering.<\/span><\/p>\n Going forward, we hope to continue working with BMW to further leverage the growing machine perception capabilities of modern cars to enable future use cases. Access to the car\u2019s precise 6DOF positioning system could allow us to render world-locked virtual content outside of the vehicle, like identifying landmarks and other points of interest. We expect this capability to be invaluable for future AR glasses and personalized AI assistants.<\/span><\/p>\n It\u2019s an exciting vision, and it\u2019s a road we\u2019re committed to traveling.<\/span><\/p>\n<\/p><\/div>\n![\"A](\"https:\/\/about.fb.com\/wp-content\/uploads\/2023\/05\/02_Plane-Demo.gif?resize=640%2C360\")
<\/a><\/p>\nOculus Insight vs. the Open Road<\/h2>\n
Challenge Accepted<\/h2>\n
![\"\"](\"https:\/\/about.fb.com\/wp-content\/uploads\/2023\/05\/01_Car-Demo-Research.jpg?resize=960%2C540\")
<\/a><\/p>\n