The Apple Vision Pro is both more and less than what I expected. We all knew Apple was making an “AR” headset, but what came out wasn’t really AR. To see what I mean, let’s compare what’s available right now on the market, that we can see, feel and touch.

The True AR crowd

First up is the Microsoft Hololens. This was an early entrant in the AR market, and wowed us all with some very promising videos. Hololens is a true AR device, projecting virtual objects onto the real world. Unfortunately, it suffered from a fatal flaw: the completely lackluster field of view of 30 degrees. Since these words make little sense, I will encourage watching this video instead.

Note how when displaying the Hololens 1, there’s this little square box in the middle of the video. That is the “FOV window” and when wearing a Hololens, only that small little rectangle showed the AR view. The content was visible and of a nice resolution, but any content, windows, etc, outside of that tiny little rectangle were invisible. With 180 degree horizontal, and 130 degree vertical FOV of the see through area, this tiny center square was covering less than 10% of the viewing area.

Later revisions of hardware have increased the AR surface area, but none of them cover 100% of the world. Apple Vision Pro doesn’t even compete in this space; it is not see through. No, it is very, very opaque.

The True Virtual crowd

Many comparisons will be made between the Vision Pro and the Meta Quest, as well as the Quest Pro, and to a lesser degree the Valve Index or even Playstation VR. All of them are unwarranted.

For one, all of the true VR headsets don’t try to project the real world in a recognizable way. The video passthrough modes, where present (such as the Quest and Index) are laughable at best, making a blurry simulacrum of the real world. Enough to roughly navigate, but nowhere near enough to actually use in a reasonable way.

There’s a reason these devices are in the $1000 or less market; they can be used to play games, and they’re pretty good at that, but games are capable of papering over lacking resolution by creating virtual, fictional worlds. By not giving us a frame of reference, these worlds can exaggerate proportions, make larger text, and in general cover over the failings of VR.

But VR has a huge failing. Screen door effect. Blurry text. VR Fatigue. Trying to use a VR headset for an extended period of time is an exercise in endurance and headaches when the virtual world simply does not have the level of detail that eyes expect.

Enter Varjo

The Varjo XR3 is a headset that regular people won’t buy. At more than €6,000, this headset is purely in the class of industry/enterprise customers. It’s also packed with hardware to justify the price, from multiple cameras to LiDAR. Crucially different from pure-VR or even pure-AR, this is a mixed reality (or eXtended reality/XR) headset.

So whereas the Meta Quest projects a virtual world first and foremost, the Varjo XR3 spends immense camera and processing power, and has high resolution internal displays, all aimed first and foremost at reproducing a fast, lifelike and detailed rendering of the exact same room that the headset is being used in. And blending in virtual objects on top of that.

It may seem counter-intuitive that cheap headsets can produce a virtual world, but a very expensive headset is needed to reproduce the real world. Unfortunately for us, the real world is difficult to reproduce. Much has been written about the uncanny valley, but the challenges for an XR headset are of a different nature.

Latency is an obvious one. An XR display must be insanely quick to show the true real world, otherwise we’ll easily get disoriented and confused. This requires using fast refresh cameras, fast refresh displays and fast processing chips. The latter becomes even more important because the AR headset also needs to process the video feeds before displaying.

Naively this seems odd; the room is right there so just pipe the video feed from two stereo cameras. Unfortunately, the effect of this creates a terrible effect; the video cameras are not at the same position as the eyes, so you get a disconnect between what the brain expects to see when the head moves, and what the feeds show. So an XR headset needs to quickly scan the world, analyze it onto a virtual 3D world, then render it back again before displaying. This processing is what very often lowers the resolution of the real world in a headset and makes it blurry.

This is where the massive hardware budget of the Varjo comes in, both in processing, and also LiDAR to map out the real world into a 3D world before rendering it back.

Enter Apple

It’s in the Varjo space that Apple is entering. A headset that continuously renders the real world, presumably in a full surround display. This is why Apple markets “23 million pixels” (which roughly equates to one 4k display per eye). Notably, this type of resolution is very similar to the Varjo’s dual displays (with a high resolution center display and blurrier surround.

This is also where the measure of success for Apple comes in. At a 4k display per eye, this is still incredibly low resolution compared to the real world, making their claims of “looks good from any angle” suspect at best. Do they use uniform resolution throughout (at the expense of center clarity)? Or do they copy Varjo and introduce a blurry halo instead?

The vision is okay-ish. But the proof is in the hardware, and Apple has really yet to show that off. Much like Microsoft’s Hololens announcement, all we saw was a third-person rendering of what one “might” see in the Vision Pro. But they’re not letting us see the actual first person perspective.