The Real-Time Revolution

The transition of the computer interface from 2D to 3D is upon us. Here are some thoughts on what’s driving this change now.

The BMW 8 Series Coupe: one is real; one is rendered real-time in the Unity engine. Which is which?

Real-time 3D has emerged as the next interactive media type. Forged in gaming, it’s now being used in nearly every industry. Soon, 3D graphics will be as pervasive as images, videos and web pages. Why is this happening? And why now?

Decades in the Making

3D graphics is nearly as old as the computer itself, tracing its roots back to the 1960s. It’s been used in applications spanning engineering, architecture, sales and marketing, defense, gaming and animation. But until recently, 3D was still lurking at the margins, generally not in the mainstream of digital media.

Historically, 3D has been a luxury; the benefits of displaying it on flat screens were not worth the costs. Content creation was laborious, hardware was expensive, and consumer reach was low. 3D was also typically passive, pre-rendered to still images or linear video, with no ability for the user to change the view or manipulate objects in the scene interactively. Video games are the one exception to this — the only segment of the computer industry that has been able to solve simultaneously for the high cost of producing killer interactive 3D content and the ability to distribute and monetize it at scale.

But this situation is changing as we speak. Real-time 3D is going mainstream, driven largely by immersive technologies like virtual, augmented and mixed reality; but other factors are playing a part as well.

Converging Developments

Broad adoption of real-time 3D is happening due to several technology innovations coming together. These include:

• Dedicated Graphics Hardware. For years now, all desktop and mobile devices have included one or more graphics processing units (GPUs) — hardware designed to support the real-time rendering of 3D objects and visual effects. And each upgrade cycle for PCs and phones comes with a new generation of GPU that powers faster, more realistic-looking graphics.
• Game Production Software. Game production software, such as the Unity engine and editor, offers creators a tool set to develop and distribute real-time 3D content across a range of hardware systems. The cost of ownership of these software platforms continues to trend downward, even as they offer ever more powerful capabilities with each release.
• WebGL. Since 2012, all web browsers have supported WebGL, a programming layer that connects standard web technology like HTML5 and JavaScript to the GPU, avoiding the need to build and distribute native executables and apps, and providing consumers with no-download, one-click access to 3D content in web pages.
• Content Standards. Soon after the introduction of WebGL, the industry banded together to create a common file format for real-time interactive 3D models and scenes that can be displayed in web pages or in mobile and desktop applications. This format, glTF, is now an established Khronos Group standard being used in a wide range of tools and applications. Think of it like JPEG or PNG, but for 3D. (Last year, Apple introduced a similar format, USDZ, but USDZ is only standardized across Apple’s line of products.)
• Virtual, Augmented and Mixed Reality (VR, AR, MR; collectively: XR). In 2014 Facebook purchased VR hardware maker Oculus and kicked off a boom in a field that was languishing. Consumer-grade VR, and mixed reality systems like Microsoft Hololens or Magic Leap One, can cost in the small hundreds to small thousands of dollars, depending on the setup. On the AR side, Apple and Google released ARKit and ARCore, respectively, in 2017, unleashing the power of augmented reality on standard consumer mobile phones numbering at a billion handsets and counting. The low cost of ownership of these systems is making it viable for businesses across industries to experiment with immersive technology.

The Next Computing Paradigm

As businesses have leaned into XR, they’ve begun to understand the benefits of real-time 3D across the board — even when rendered on flat screens. Real-time 3D pipelines can produce cinematic-quality graphics, save production time and cost, and unleash creativity by freeing creators from mundane and repetitious tasks. My company Unity is finding this to be true in multiple industries outside of gaming, including automotive, architecture and film. As I like to say about Unity’s non-gaming customers: they come for the XR, but they stay for the rest.

On the consumer side, there are many use cases being driven by tech industry heavyweights: 3D posts in the Facebook feed; Snapchat augmented reality lenses and face filters; Google 3D search and shopping (more on this in an upcoming post); and 3D animated objects in Microsoft PowerPoint. Most of these cool innovations are powered by glTF, WebGL and the other enabling technologies listed above.

These twin pillars of useful enterprise solutions and accessible consumer experiences reinforce each other, echoing the way that the personal computer market was established first in the office and then at home. As employees became comfortable using PCs at work, they began to shop for them for personal use in specialty stores like CompUSA; soon after, the dynamic reversed, with consumers comfortable with PCs at home finding jobs that required computer skills.

Analogies aside, there is one more thing to consider. Two full generations of consumers have grown up on video games, CG movies and interactive devices. They expect their computer interfaces to work like their games, or like the holograms they’ve seen in Iron Man and Minority Report. I’ve previously written about the Immersive Generation: the twenty-somethings who are the XR natives about to be in charge of everything. To them, this is all inevitable. As for the rest of us… we can only hope to pave the way for them.

What this all adds up to is that we are on the cusp of a step change. The computer interface is moving from 2D to 3D — the culmination of decades of innovation. Sooner rather than later, 3D will become the dominant paradigm, with legacy media types like images and video embedded in spatial interfaces that immerse us in information, imbue the physical world with digital magic, and transport us to fantastical other places. Technology providers are laying the foundation for it; creators are mastering new skills in preparation for it; businesses are driving investments in it. And our digital lives are about to become more friendly, fun, and useful as a result.