Face the Future
Computing in an Augmented World
[Transcript from my keynote talk at AWE 2017.]
We live in a 3D world. People move, think and experience in three dimensions.
But for decades, human-computer interface has been… less than that.
As computing continues to get woven deeper into our daily lives, in many ways, interface hasn’t caught up. We still operate our computing devices — or perhaps more accurately, they operate us — at some remove from how we work within the real world.
From punch cards to command lines to GUIs to touch screens, the computer industry has seen a steady progression in usability; but there remain levels of abstraction compared to how we interact with our surroundings and each other on the physical plane.
Today, we peer into an endless sea of information via tiny portholes… flip through pages and tabs linearly while trying maintain a mental map of things and their connections of much higher than one dimension … and remember and share moments from the world around us by capturing only the photons in front of us.
Tomorrow, information will be displayed where and when we need it, its essence and connections laid bare to see… and whole experiences will be captured and shared. We are about to escape the tyranny of the rectangle… and information itself will achieve six degrees of freedom, in a virtual sensorium that maps the web onto the real world and brings the real world into the web.
Over the past several decades, every time people made computers work more like we do — every time we removed a layer of abstraction between us and them — computers became more broadly accessible, useful, and valuable to us.
— Clay Bavor, VP of Virtual and Augmented Reality, Google
With the advent of immersive technology we are approaching the next step change in human-computer interaction. As HCI goes 3D, some truly staggering things are going to happen. Doubtless this transformation to an immersive computing landscape is going to make digital technology, in Clay Bavor’s words, “more broadly accessible, useful, and valuable to us.”
We’re here today because we all know that immersive computing represents the next platform. But most of us can’t really imagine the ways it is going to disrupt our daily lives, any more than back in the day we could have imagined the Internet disrupting brick and mortar retail, or smart phones upending the transportation and hospitality industries.
We’ll explore some of the possibilities momentarily. But first, let’s ask ourselves why now? Real time 3D is nothing new, and we’ve been tinkering with virtual and augmented reality hardware for decades. Why do we think now is the time to collectively invest so much energy as an industry?
3D graphics is nearly as old as the computer itself, tracing its roots back to the 1960s. It has been used in applications spanning engineering, education, training, architecture, finance, sales and marketing, as well as gaming and entertainment. Even the first immersive VR systems were demonstrated over fifty years ago.
Historically, 3D applications relied on expensive high-end computer systems. But that has changed in the last decade. 3D processing hardware is now shipped in every computer and mobile device, with the consumer smartphone of today possessing more graphics power than the professional workstation of a decade ago.
Look around: a lot of our media is already 3D — though until recently it’s been presented on flat screens. Animated films are created from computer-generated 3D images. Video games, whether running on dedicated consoles or mobile phones, are typically rendered in 3D. And even the news has gone 3D: the sight of a CNN analyst meandering through a virtual set, comically awkward a few years ago, has become an accepted part of the broadcast milieu as cable channels vie for increasing attention in a 24-hour news cycle.
But until very recently, 3D was still lurking at the margins. Other than professional design software, the only widespread use of 3D has been in video games.
In my opinion, that’s because historically, 3D has been a luxury for most applications. The benefits of real time interfaces on flat screens were simply never worth it, other than for niches in design, engineering and filmmaking. Gaming is the one exception — the only segment of the computer industry that has been able to solve simultaneously for the high cost of producing killer content, and monetizing it at scale.
Well, all of this changes with XR.
Inexpensive stereo displays, holographic optics, binaural audio, 360 cameras, volumetric capture, and computer vision, combined with the previous innovations in mobility, low-latency networking and location services, are enabling applications even the wildest of us 3D dreamers couldn’t have even dreamt about a decade before.
And with immersive hardware, we have no choice but to render in 3D. The entire graphics pipeline is set up for it. We have a 360 degree space in which to work. Even if we only display textured rectangles in that space, we need to lay them out using 3D coordinates. But of course we can do so much better that that: we can create virtual spaces and display virtual objects. We can organize information the way we want, using spatial memory and spatial reasoning to get the most out of it. And all this content can be blended seamlessly with the real world around us.
Beyond cheap hardware, we also now have millions of content creators savvy in 3D development– thanks to the video games, design and VFX industries– and most importantly, an interactive generation who expects — nay demands — this. Put a print magazine in front of a three-year-old. What’s the first thing she’ll do? Tap it; pinch to zoom. Today’s young kids are tomorrow’s XR natives.
With XR, real time 3D is not a sideshow; it the main attraction.
With XR, there’s nowhere to go but in.
While much of the focus today is on the innovations on the hardware side, software, content and services are where most of the value will be created over time.
I’m sure you’re familiar with my company, Unity. We occupy a unique position in this new industry.
For those that aren’t fully aware of us: Unity is the creator of a flexible and high-performance end-to-end development platform used to create rich interactive 2D, 3D, VR and AR experiences. Our graphics engine and full-featured editor enable the development of beautiful games or apps and easily bring them to multiple platforms: mobile devices, home entertainment systems, personal computers, and embedded systems. We also offer additional solutions and services including Unity Ads, Unity Analytics, Unity Asset Store, Unity Cloud Build, Unity Collaborate, Unity Connect and Unity Certification, all focused on helping our to developers succeed.
By the numbers, here are some amazing stats: 38% of the top 1,000 free mobile games are Made With Unity; Unity games triggered 5B app installs last quarter alone, across 2B unique mobile devices. When it comes to VR and AR, around 58% of developers surveyed are Unity users, and over two thirds of all VR/AR content released through known channels is Made With Unity.
You probably know many of the top Made with Unity VR games like Job Simulator and I Expect You to Die, and popular VR experiences like TiltBrush and new Cinematic fare like Baobab’s ASTEROIDS. And Facebook Spaces, the company’s new VR social application, recently shown at F8 and Unity’s Vision Summit.
Add to this that over 80% of Vuforia AR applications, and 91% of all Hololens mixed reality is made with our software… you can see that Unity has become the Foundation of VR/AR development.
So, how did we get here? Over the last couple of years, we invested heavily in supporting VR and AR hardware platforms, forging deep partnerships with the major HMD providers.
But we couldn’t have even gotten into that position without first having built a solid foundation for our business. Unity was founded one three core principles, to which we remain committed to this day:
•Democratize development — everyone can bring their creations to life, including a new wave of creators in AR and VR — many of whom do not have game development experience
•Solving hard problems — supporting almost every platform on the planet, and continuing to invest in the best graphics and performance
•Make developers successful — making a profit is hard… but we’re proud to be able to say that in 2016, we paid out more to developers than we’ve made ourselves
Understanding Unity’s position in the ecosystem, now let’s look at some of the great XR being made with Unity, in particular augmented and mixed reality that is pushing the envelope on user interface.
For brands: Rewind Studios created Flight Deck, an application that combines HoloLens with 2D screens to create a whole new type of viewing experience for the Red Bull Air Race. In the HoloLens, the user can actually put the race course on a coffee table or likely even the floor — for a real bird’s eye view — and watch in real time as the pilots make their maneuvers. This piece envisions both brand experience and live sports in a whole new way.
For work: ScopeAR “puts KNOWLEDGE WHERE YOU NEED IT”… they created Remote AR, a telepresence collaboration tool that allows remote field technicians to connect to experts via live video, audio, and a powerful annotation toolset that is augmented to ‘lock’ onto the real world with augmented reality.
Scope used Unity to create and manage 3D objects, while being cross platform and supporting many different variations of devices, from phones and tablets to AR glasses to desktops. And taking advantage of Unity’s robust 3D rendering and plugin architecture they built an entire real time audio-video stack along with the capability to remotely synchronize 3D data (meshes from Tango or Hololens for example, or 3D drawings using line renderers, or complicated CAD models).
Nexus Studios took outdoor adventure to a new level, by creating an AR experience for The Gruffalo, the best-selling picture book. Families follow clues on an interactive trail in the English forest, tracking signs of their favourite characters. These clues lead to different Augmented Reality markers — footprint signposts,specific to each one of the characters. When the visitor aims their device at the footprint marker, a short animation of that character is triggered, blending in with its natural surroundings. The Gruffalo experience shows how we can use AR to turn the world into a canvas for art and storytelling.
SwapBots are collectable & customisable toys that are brought to life by a smartphone or tablet using augmented reality. There are hundreds of combinations of the physical toy, as you swap parts, and endless single- and multi-user game play possibilities.
Mekamon are multi-functional, connected battlebots with augmented reality capabilities. Control them with your smartphone and see power ups and explosions using AR. Fight hostile aliens in AR, play arcade games or engage in warfare with other bots.
Finally, I think we’re all excited about the possibilities for education. MyLab provides augmented reality educational content for Hololens…
And MergeVR’s Merge Cube is a physical cube that, when viewed through a standard smartphone and the MergeVR headset, displays animated, interactive education content. Wild.
Software is Eating the World
— Marc Andreesen
Even if with all this great stuff going on, you’re still skeptical, well I’m here to say it’s time to suspend that disbelief.
It’s fair to think of the last few years of XR as one big, expensive, global proof of concept. We’re still experimenting with hardware and tools, formats and form factors. We’re still learning the right way to do a lot of things, and the best applications of immersive computing. In fact, we’re just getting started. But the pace of development doesn’t seem to be letting up. Massive, ongoing investments are being made in this space — Vuforia, FB, Snap, Google, MS, and startups like Meta, Magic Leap, Improbable.
So this is happening. And more innovation is right around the corner.
A few years back, Marc Andreessen famously said that software is eating the world. At the time, he didn’t intend it to mean the actual world — he meant information, communications, business processes. But the actual world — well, that comes next.
Google Tango is about to hit the mainstream.
Asus and Lenovo are now showing their new Tango-powered phones, with built-in fisheye camera, depth sensor and infrared sensor. Combine this with powerful software from Vuforia and Unity, and we can blend the real and virtual worlds seamlessly.
And mapping the interior — what Google Maps and StreetView did for the outside, SLAM (Simultaneous Localization and Mapping) does for the inside. Facebook made a big deal about this at their F8 conference, talking about SLAM and object recognition. Google has branded their SLAM-based interior mapping software WorldSense, and Vuforia has its Smart Terrain toolkit for environment and object recognition.
We can use technologies like this to find our way around within a store, museum, or other property. And not just physical objects, but we can heat map wifi coverage, airflow within a room, and more…
But not just mapping the world — ingesting it. In just a few seconds, this Tango-enabled phone scanned a room interior in crude form. With several more minutes’ effort, you can get a high quality scan, bring it into Unity and publish a Daydream app based on real physical environments. All from a consumer-grade smartphone.
With the help of some nifty new smartphone hardware, software is now poised to — literally — eat the world.
Of course it’s not only about getting the right tools in the hands of both the consumers and pros; it’s also about infrastructure: a global information system that allows anyone to publish and share content and applications. Also known as the World Wide Web.
At Google I/O, the company demonstrated a development version of Chrome that extends WebVR — the API that connects web browsers to VR hardware — into augmented reality. WebAR brings the phone’s camera and Tango-style sensing capabilities into the browser, which means that for many use cases, you would no longer need to download a specialized app to recognize markers, scan the environment and overlay information: the world becomes your QR code. And you can easily find cool virtual stuff nearby with one click, because someone shared a link in their social feed.
The digital world will soon be enmeshed enough with the physical world that our ‘reality’ will be the transparency mode that we choose.
— Monika Bielsktye, Creative Strategist/Future Prototyper
This is all the great stuff that is happening right now.
Now let’s play it forward a few years.
Take it as a given that immersive computing becomes the dominant paradigm: virtual worlds, virtual places, virtual objects commingling with physical space, all presented in familiar, intuitive real-world metaphors, accessible with simple commands and discoverable online. Gesture input and voice, of course. Untethered and fully mobile. Lightweight glasses or even contact lenses. These all seem obvious… once we’ve taken the plunge into immersion.
What’s really interesting is what comes after that, when other computing innovations converge with XR. AI for endlessly variant NPCs in large scale worlds and never-ending stories; deep learning to enable a malleable environment that responds to our every whim. Integrate IoT and we can control stuff in the real world — with our very minds, or as close as we can get.
Last year, designer and filmmaker Keiichi Matsuda released a thought-provoking short film called Hyper-Reality, depicting some of the possibilities of an immersive, augmented digital world.
As computing goes immersive, every inch of available virtual real estate will be up for sale for manufacturers and marketers to hawk their wares. The history of advertising is, after all, the history of a rise in production value. And with XR we so much more production value at our disposal: 3D graphics, animation, 3D sound, and haptic inputs, to name a few. This could make our future world a brand-rich candyland with myriad choices…
Or a hot mess — swipe left to reset your life, your credit history, your social status… as these technologies that promise to bring us closer to our world and each other instead trap us in a Super Sad True Love Story of our own making.
Of course it’s too early to tell how this will all shake out. I’m hopeful, if only because we’ve been here before with other disruptive technology.
One of the safeguards against some of these more negative outcomes is democratization of the platform. By enabling the independent creator, we can level the playing field, and not everything has to be paid for by big companies. A 15 year old kid could become a future YouTube star in XR… or develop cancer detection tools… or create the next XR artistic masterpiece.
Though the early returns are incredibly promising, they’re just that: early. We don’t know which devices and form factors will win; which will be the dominant formats or killer apps. But we do know that the world is going immersive.
Investments made now in immersive computing — in learning the tools of the trade like Unity, in understanding immersive design, in adding meaningful layers to the real world, in short, in building interfaces for people — will pay off many times over in the future. Because that future is 3D.
Thank you — and enjoy AWE!
