Passthrough camera access is the talk of the town in the XR world these days. We’ve all seen what Meta, Apple, and Pico are doing, but the big question remains: what’s Google’s plan for Android XR? After having a chat with them, I can say they’re aiming for something similar to what we see on phones. Stick around to find out more!
The Camera Access Dilemma
For those a bit out of the loop, here’s a quick refresher. The latest VR devices are more than just that; they’re MR headsets offering users an RGB passthrough view of their surroundings via front cameras. This opens up a magical realm of mixed reality apps like Cubism, Starship Home, and Pencil.
Imagine tapping into those camera frames to give users an experience where AI and computer vision algorithms enrich reality itself. As I’ve pointed out before, getting access to these camera frames is essential for unleashing the full potential of mixed reality. With camera access, I was able to tinker with an AI+MR application for home interior design on Quest, something which wouldn’t be feasible without it.
But there’s a flip side: privacy concerns. Should a malicious developer gain access, they could potentially snoop on personal details like IDs or bank cards left lying around, even capturing images of faces or bodies for nefarious purposes.
We walk a tightrope between preserving user privacy and unlocking the possibilities of mixed reality.
The Stance of XR Companies
Initially, everyone had open access to camera frames. If you’ve been following me since the early days, you’ll recall the wild experiments with camera textures on the Vive Focus we did back in 2019: diminished reality, Aruco marker tracking, sound reactivity, etc.
But as mixed reality began gaining traction, companies started restricting camera access. Meta, Pico, HTC, Apple—all took the same route, citing privacy concerns.
Yet the XR development community wasn’t having it. Pioneers like Cix Liv, Michael Gschwandtner, and I pushed back, demanding transparent and user-consent-based camera access to allow for innovative applications. It seemed ironic that our pocketsized devices offered such access easily with just a permission request.
These efforts bore fruit, leading companies like Meta to promise a "Passthrough API" this year. But what about Google and Android XR?
Android XR to Mirror Android Phones
Android, the bread and butter of countless phones worldwide, allows developers to access camera streams with a simple user permission. Google’s game plan for Android XR? Adopt a similar stance. After a detailed email exchange with a Google spokesperson, they confirmed this approach for Android XR.
Developers can request camera streams just like on phones: ask for permission, specify the camera ID—0 for world-facing, 1 for selfie—and you’re good to grab frames using standard Camera APIs, Camera2 and CameraX.
Great news, right? While the forward-facing camera streams in a realistic manner, the selfie camera streams as a reconstructed avatar, similar to Apple’s Vision Pro, making it coherent with how Android phones function by showing either the user’s view or their avatar.
Google appears keen on ensuring Android apps are instantly compatible with Android XR, even those using camera access. It’s a clever move that enhances consistency between XR devices and phones.
Yet there’s a lingering question—what about accessing all camera streams, including raw ones? Unfortunately, God’s still keeping the cards close:
“At this moment, we aren’t enabling apps to dig into non-standard (forward or reconstructed inward) camera data.”
While frustrating, it might change in the future, especially for enterprise users.
Unity developers wondering how they fit into this picture might find solace in WebcamTexture or possibly a JNI workaround to bridge CameraX functionality into Unity—a topic for another day but a hopeful prospect nonetheless.
A Word of Caution on Android XR
Do remember, Android XR is still in its preview phase, with no devices currently live. So, this could shift before an official release, even though I doubt it.
The Future of Camera Access
As Google and Meta open the gates to camera access, expect other XR giants to follow in their footsteps. It looks like 2025 might become the year of groundbreaking mixed reality innovations. I’m on the edge of my seat, eagerly awaiting what the developer community will conjure up!
Disclaimer: This blog contains ads and affiliate links that help keep it running. Clicking an affiliate link might mean a small commission for me on any purchase you make. Check out my full disclosure here.
Share this with fellow visionaries!
Related Content:
