I rode with Nissan’s AR and 5G-powered virtual passengers

Nissan’s vision for the future of connected cars is about as buzzword-laden as possible. Making use of 5G, AR, and VR to power an advanced in-vehicle experience, the company’s “Invisible-to-Visible” (I2V) technology not only allows for HUD-style overlays but also humanoid virtual assistants that ride alongside you to provide conversation and locally relevant information.

Here’s a handy diagram to show how all of this works:

Got it? Yeah, probably not — so I recently visited Nissan’s Grandrive proving ground in Yokosuka, Japan to see for myself how this would work. The car maker has just begun testing the system in a moving vehicle with 5G technology provided by NTT Docomo, Japan’s largest carrier. The test vehicle is based on an NV350 Caravan.

Nissan’s I2V system makes use of what the company calls “the Metaverse,” which is not a reference to Persona 5’s alternate dimension but describes how virtual characters can appear in real-world vehicles through augmented reality. In my demos, I wore a Meta 2 AR headset which, despite being made by a company that no longer exists, is very comfortable and offers a much wider field of view than something like the Microsoft HoloLens.

I could see various assistants through the headset, from a PS2-level realistic Nissan PR rep to a cute cuboid bear called Nissan Bear to the extremely anime mascot of Unity Japan, Unity-chan. (Nissan says they used Unity-chan because she’s free to license and the underlying software was written with Unity.)

The avatars sat across from me in the back of the caravan, or next to me in a separate, static demo where I was in the driver’s seat. The effect was reasonably convincing, though the alignment with each seat wasn’t perfect. Combined with the brightness of the Meta 2’s image, it was hard to avoid the sense that the images had been laid over my surroundings rather than being part of them.

The twist is that the virtual avatars I was seeing were actually real people, in a sense. While I was on the road, the avatars’ speech and movements were being “acted” out back at the Grandrive pavilion by two women wearing HTC Vive Pro VR headsets and — I swear I’m not making this up — Crocs with Vive trackers attached. They could see a virtual representation of me thanks to cameras inside the car, and were able to interact with my environment while talking live to me. It is equally weird and cool to see a glowing augmented-reality person press a real button to open a real window.

All of this happens in real time without much perceptible latency, which is one of the benefits of 5G. Just in general, you don’t want any risk of lag when moving vehicles are involved. But the technology is still in testing, a fact that was brought into acute focus when Unity-chan attempted to give me a donut, then froze into place before my eyes as I heard a Windows error sound. She never revived. Nissan told me it was a problem with Docomo’s 5G station, which was mounted in a van parked next to the pavilion.

The AR assistant use case is clearly not a scalable concept, unless we head for a strange future where cars on the road are matched one-to-one with Crocs-wearing employees in VR cubicles pretending to be your friend. As a potential model for in-car chat, though, it’s pretty cool. On a long enough time scale, I don’t think it’s that far-fetched to imagine someone using VR to call a family member while they’re driving — if nothing else, this is the closest thing we’ll have to Star Wars-style hologram phone calls for a while.

Photography by Sam Byford

Go to Source