Apple Vision Pro: Our Spatial Computing Future
Originally Posted: June 29th, 2023
Apple Vision Pro: Our Spatial Computing Future
A few weeks ago, Apple revealed the future of computing. As it turns out the future is expensive, heavily patented, currently unavailable, and available only in the US at launch.
While many others have covered this product and every minor detail we’ve learned since, I find it more fun to reflect on what this means and where we are. In my view, and according to the written gospel of Apple’s marketing department, we are at the dawn of a new era.
According to Wikipedia, Simon Greenwold defined the term spatial computing in 2003 as “human interaction with a machine in which the machine retains and manipulates referents to real objects and spaces”. His original paper (direct PDF link) is worth a read. As with many new technologies and computing paradigms, Apple didn’t invent it. They were, however, arguably the first to use it to deliver a complete and compelling product.
The Spatial Computing Era will start in early 2024, in the form of a $3499 mixed reality headset named the Apple Vision Pro. Yes, it is easy to mock the price or get lost in the details we do or do not have. That is not what I’m trying to do. I want to zoom out, and project my mind to the future that Apple is leading us towards. Their vision of what’s next is worth focusing on.
The front of the Apple Vision Pro, showing some of the cameras and sensors, and the Digital Crown in the upper left and single button in the upper right.
The Apple Vision Pro and the battery pack.
What is the Apple Vision Pro?
The simplest question is hard to answer: What is the Apple Vision Pro?
The Apple Vision Pro is a mixed reality headset. It is a computer you strap to your face and look through, rather than look at. You use your eyes, hands, Bluetooth accessories, and voice as controls. It uses a complicated mix of displays, lenses, cameras, sensors, and speakers to augment, or replace, what you see and hear around you. It is a thousand enormous engineering challenges condensed into one object.
It tracks your eye movement, so you just look around to change your focus. You select objects by looking at them and pinching your fingers. You scroll, zoom, and control everything else with hand and body gestures. You can also use a mouse, trackpad, or a physical or virtual keyboard.
The unique pattern of your eye’s retina authenticates you, then the headset presents both traditional two-dimensional application windows and three-dimensional objects placed in the physical space around you. You can move them around in this virtual space, but they stay planted as you move around the room. You can also block out your surroundings, and transition between these two modes by scrolling on an Apple Watch style Digital Crown.
It can do many other things. It can capture 3D photos and videos and replay them. It can act as a display for your Mac. It allows you to interact with people surrounding you by blending them into your reality, albeit it in a potentially socially awkward way. Time will tell how EyeSight and that stigma plays out. It is not anything like Google Glass though, or HTC Vive, or Microsoft HoloLens, or Meta/Oculus Quest. It is something new.
Physical peripherals like a trackpad can be used in addition to eye and hand controls.
EyeSight in action. ¯\_(ツ)_/¯
Detail Devils
The devil is always in the details. Right now, most of the details we have are in this 40 minute section of the WWDC 2023 keynote, a product page, second-hand accounts of a scripted 30 minute demo, the SDK, and some developer-focused documentation. Despite all that we know so far, Apple’s story isn’t finished. They still have a lot to reveal.
We don’t have a comprehensive list of specs, and we don’t know everything it will do. In fact we don’t know much about the physical headset or the software, outside of what they said in the keynote or in documentation. It is clear visionOS and the Vision Pro are still under active development, and I’m sure there is an incomprehensible amount of work left to deliver 1.0. We’re not sure exactly when that will be, and it sounds like Apple isn’t either.
What Apple did show us was the core experience, and the basic interaction model that will soon become ubiquitous. Eye tracking, simple hand gestures, and voice input will be the basis for a new form of computer. The rest of the details will be historical footnotes.
While we wait it is easy to get wrapped up in the hype, as many have for years about this rumored AR/VR/XR headset. I was one of them. In the lead up to WWDC 2023 I preferred the term mixed reality, but in their initial announcement Apple framed it a different way. I think that “spatial computing” is a good definition of what they’ve shown. It is not a new concept, but soon it will be our reality. I can’t stop thinking about that framework and imagining how this spatial era of computing is going to play out.
The Inevitable iPhone Comparison
What we have seen of Vision Pro and visionOS so far is compelling, but this is like judging the iPhone by the 2007 MacWorld keynote. Simple interactions like slide-to-unlock and scrolling a list generated amazement and applause. Syncing to iTunes and playing MP3 files were a core part of the product. The camera stunk, you couldn’t record video, you couldn’t even change your wallpaper or rearrange the home screen icons. Perhaps most damningly — at launch in June 2007, there was no App Store.
Looking back, the first iPhone was primitive, but it was announced at the perfect time. It was a miraculous achievement. It was clear after January 9th, 2007 the iPhone was what mobile computing was going to be. The first model lacked many things but it delivered the correct fundamental interaction model. Direct manipulation and multi-touch made T9 keyboards, Nokia’s Symbian OS, BlackBerry, and Microsoft’s Windows CE look like tools for cavemen in comparison.
The best designs are obvious after you see them, and there is a refreshing clarity that comes from seeing the correct design. Multi-touch and direct manipulation was one, just like the graphical user interface was in 1984. These kind of events are rare, but I’d argue the Vision Pro and visionOS are in that same category.
I’m biased towards Apple and their products, I’ll gladly admit that. I grew up with Macs in my school and my home. After my 2nd Rio Karma bit the dust, I wanted an iPod in my pocket. After our Dell Inspiron needed a 4th replacement power supply, we replaced it with a white polycarbonate MacBook. I watched the iPhone announcement as it happened, and it was a long wait until I was able to use one later that year. It was one of my most informative experiences, and it accelerated my interest in technology.
This is the first time I’ve felt that way since.
Application windows can be arranged and overlayed in your physical space, using a Stage Manager style multitasking UI. Also shown is the floating virtual keyboard, which is one option for text input.
I’m biased towards Apple, but I’m not blinded by that. There are no shortage of legendary companies, teams, and individuals that have pushed computing forward. In the most recent decades though, Apple has become one of the best at a specific skill — packaging new technologies into compelling products that define a category.
Lately, Apple is releasing new generations and minor updates often. These steady improvements are always welcomed, even if they aren’t exciting. They have become a Goliath, efficient at churning out incremental updates and making money. Occasionally, they can still dazzle us with a new paradigm. We just witnessed one of those events. No one can predict the future or what the Vision Pro will become. It’s clear it’s not going to be like what came before.
Nothing will replace the iPhone, and nothing needs to replace the iPhone. At least not yet. If anything, the Vision Pro is competing with the Mac.
Personal Computing Eras and Revolutions
What defines an era of computing? Ask a hundred people and you’ll get a hundred different answers. There’s many ways to view it, but I think in terms of devices and what they are capable of. To make these leaps, something fundamental has to change about what a computer is or what it can do.
We’ve gone from mainframes and vacuum tubes, to microprocessors and personal computers. From command line to graphical user interfaces. From isolation to local area networks, to wide area networks, and eventually the world wide web. Now, we’re going from mobile and physical, into virtual and spatial.
The original Macintosh in 1984 became the foundation of modern graphical interfaces. Apple didn’t invent it from scratch, they stood on Engelbart, Xerox PARC, and countless other giant shoulders to do so. The way history played out most of those became footnotes, while the original Macintosh made a dent in the universe.
That doesn’t diminish earlier work, and it doesn’t mean the original Macintosh was perfect. The lack of storage, puny amount of RAM, high price, and limited sales shouldn’t distract from the core innovation it delivered. Despite many flaws, it brought us the graphical user interface, desktop publishing revolution, and laid the groundwork for modern computing.
The first Macintosh and the first iPhone are useful frames to view the first Vision Pro through. It is too expensive and it will inevitably be eclipsed by better and cheaper revisions. It won’t include features we’ll soon take for granted. It’s flawed, and that is OK. That’s the price of being the foundation, and being the first mover in a new era. Well that, and three thousand five hundred dollars, plus tax.
Apple are master marketers, but underneath the marketing is an innate sense of how technology can be applied to solve a problem. Many others fall into the trap of starting with the technology and working backwards to the product, usually with middling results. When Apple is at their best, they focus on the product first, then work backwards to develop the technology to do that. They’ve proven this many times. Their highest achievements are testament to their relentless desire for the fundamental truth found in great design.
Following that great leap of the Macintosh, Apple lost their way. Microsoft became the Goliath, until NeXTSTEP and Mac OS X saved Apple from the brink. Then, the iTunes and iPod ecosystem changed the music industry, and further accelerated Apple’s growth and interest in personal electronics. The culmination of all of this heritage, along with the good and bad personality traits of Steve Jobs as a leader and Apple as a company, delivered us the iPhone in 2007. Apple Computer, Inc became Apple, Inc. That was the last time the computing world truly shifted. The last time we entered a new era, the mobile computing era.
Innovation Isn’t Enough
Over the past several years we’ve seen a lot of innovation and steady improvement to existing products and paradigms. Some of those were incredible on their own like the iPhone X, but none of them made me feel the way Vision Pro did. Most updates are formulaic. Every September we know to expect a new iPhone that is 20% better, and they think we’re going to love it. Tim Cook has done a fantastic job at making Apple the most profitable, valuable, yet predictable and occasionally boring company on the planet.
Occasionally, they have stepped away from their monotonous iteration to draw on the old Apple magic. Tim Cook has occasionally abused the mythology surrounding “One More Thing…”, but this is not one of those times. Cook isn’t the marketer or visionary that Jobs was, but when you’re surrounded by great people and a corporate culture of innovation, precision, and tastefulness, it turns out you don’t need that.
Next year may be the start of a new era, but that doesn’t mean smartphones and other mobile computers are going away. Just like how the iPhone didn’t replace desktop and laptop computers, or make the graphical user interface or the mouse and keyboard obsolete. They are additive and complementary, and increasingly work together. Another movement we’re apart of now is ambient computing, where more and more objects are becoming computers.
The bad examples of ambient computing are cheap Android tablets slapped onto home appliances like fridges, which are usually horrifyingly awful. The better examples of ambient computing are devices like smart speakers or AirPods. They constantly work in the background, processing and delivering data through natural interactions. As we move forward ambient, mobile, and traditional two-dimensional computing will grow and improve, and continue to enhance our lives. Spatial computers like the Vision Pro build on top of these, but they are a fundamental shift in how we interact with computers and what they allow us to do.
Viewing the App Store in a floating window.
I’m not saying that Microsoft, Oculus/Meta, Valve, HTC, Sony, HTC, and other companies haven’t taken great strides in AR/VR/XR computing. That would be a worthless argument. There is a reason why they’ve mostly remained a curiosity or an expensive toy, though. Apple made it clear that the Vision Pro isn’t a toy.
The Vision Pro is designed to fit into your life and get real work done. Sure, it will be able to entertain you with movies, shows, games, and other forms of content consumption. Apple made it clear that isn’t all the Vision Pro was designed for. It’s a display and a computer first, it’s just a new kind of computer. It is not like those other headsets you may have tried before, or played a few games on.
Speaking of games, a lot of AR/VR development has been focused around that. I think one of the biggest omissions in the Vision Pro announcement was any real focus on gaming. We haven’t seen much of what gaming will look like, especially given the lack of dedicated hand controllers. We only saw a floating two-dimensional window of a console game, being controlled by a PlayStation controller. I doubt that is the end of the story for gaming on the Vision Pro, there’s too much money in it for Apple to ignore. I think we’re going to see a lot more, hopefully before launch.
Hopefully that comes with expansion in other areas, and a continued focus on productivity and usefulness. That is what will push this category forward into the mainstream.
Looking Forward
While there is a lot left to learn about visionOS, the Vision Pro, and what Apple have planned, what we know is exciting. There is a lot left for Apple and millions of developers to create, improve, and perfect. Applications don’t have to live inside windows anymore, and they don’t need to be two-dimensional. That alone will spark a wave of creativity, new applications and new experiences.
Eventually prices will come down, competitors will copy and race ahead, and the relentless march of technology will carry on. Soon, we’ll be bored by the latest minor improvements in this years headset, and we’ll be yearning for something new again. Right now, we should be excited for what’s ahead.
At first, it is going to take $3500 to buy a portal into this world, but we’re at the dawn of a new era. I can’t wait to see where it goes.
