Deploying Brand AR at Scale: Dispatches from the Front Lines

18 min readJun 4, 2019

Lessons learned on the way to creating the next global advertising format

[Adapted from my recent talk at Augmented World Expo 2019. https://www.awexr.com/usa-2019/agenda/470-deploying-brand-ar-at-scale-dispatches-from-the-fr]

Summer, 2017: the ink wasn’t dry on Apple’s ARKit announcement, and people were already thinking about how to use AR to advertise at mass scale. In the world of mobile gaming, a mainstay for my employer Unity, advertising in free-to-play games is a dominant business model, so we got right on the case to figure out if AR could help our publishers make more money, by presenting short-form AR experiences in addition to our video and 2D interactive ad formats.

In this post I’d like to share some of Unity’s experiences in what it’s taken to get a new advertising medium off the ground. It’s still early days, so this article is lean on quantitative case study data; but it does go into some detail about the lessons we’re learning in how to design, develop and deliver content for this brave new world.

The Most Powerful Storytelling Medium. Ever.

Let’s begin with a premise: immersive technologies like VR and AR arguably represent the most powerful storytelling medium we have ever had. Good VR can take us to other places and make us forget about the outside world for a bit; AR brings digital magic into the real world and blurs the lines between the real and the virtual.

And think about it: what do marketers do all day, every day? They tell stories.

Immersive tech promises to be a brand marketer’s dream, with greater production value, higher engagement and more responsive, personalized interaction than any media types we’ve tried before. This could the next step in a succession of print, radio/TV, 2D digital, and now immersive: real-time 3D ads that bring the viewer into a brand’s story and put them at the center of the action in a way that makes sense in their world.

VR: The Ultimate in Immersion

With this in mind, in early 2017 Unity began experimenting with advertising in VR. We worked with the IAB (Interactive Advertising Bureau) to develop a new ad unit format called the “Virtual Room”- a 30 to 60 second VR brand mini-experience that you opt into from a VR title such as a game or cinematic experience. You’re effectively traveling into another world via a portal, and then back out again.

Unity created the first-ever Virtual Room ad in partnership with Lionsgate: a short VR experience to promote Jigsaw, the fifth movie in the Saw horror franchise. In classic Saw fashion, this experience is an escape room puzzle you have to solve before you die, by pulling the right sequence of levers. The only catch is, when you solve the puzzle your friend dies instead… accompanied by horrific screams and virtual blood spatter.

We ran the Jigsaw pilot in Spiruloid’s Nanite Fulcrum virtual comic book app on the Rift, and Samsung Internet Browser for Gear VR. In total, we reached a few hundred thousand people — pretty impressive given VR’s limited reach in terms of number of headsets — and measured their emotional reactions.

We partnered with digital agency Isobar to run a study across both users who experienced the ad on their own, as well as users who came into a lab and had their biometrics measured to more deeply understand the emotional and physical impact that the ad had on their bodies. Here’s what we learned.

VR ad experiences elicit greater emotional response, including elevated heart rate (24%), increased skin response (44%), and 3x more muscle activation (e.g. smiling) than linear video movie trailers.
At the end of the VR experience, we showed the linear 2D trailer within the VR environment. Even though the trailer in VR had the same immersive atmosphere in headset, people reacted almost the same way as if they were watching a trailer just on their mobile phone. This led us to understand that the immersive, non-linear storytelling where users are able to forge their own way elicited greater emotional response from them than merely passively watching the trailer.

That was a great experiment and we learned a lot. We knew that it was early and against a very limited sample size, but it was very encouraging.

Outside of ad units, there have been some amazing VR marketing pieces done with the Unity engine, such as a virtual Lexus created by TiltShift in LA. Using real-time 3D and some new features of Unity such as the High Definition Render Pipeline and advanced shading to make the paint finishes, gloss coat and interiors look amazing. Check out the video:

It’s important to note that, while you’re seeing this as a video capture on Youtube, this is not a video! It was captured from an application running in real time on a PC. This is 100% computer graphics. Even more amazing than that, the TiltShift team used recent advanced features of Unity to bring this same level of visual quality into an interactive experience in VR, an interactive car configurator.

Auto makers are super picky about how theirs cars look in digital. Paint has look like paint; the interior has to look so good that you can practically smell the leather seats. Imagine bringing this level of cinematic quality visuals together with the deep immersion and interaction of VR. It could be a game-changer.

Jigsaw and Lexus are awesome examples that show the power of the Unity engine for creating amazing marketing content. But while VR can take us deep, unfortunately we need to acknowledge a few facts:

VR headsets have still not achieved the mass scale required to reach large audiences. There is growth in the home, and location-based venues provide an additional way for the public to experience VR. But generously, VR headsets comprise reach in the small tens of millions of consumers.
VR is awkward and expensive. To date, the best VR has required connecting to a reasonably powerful PC, require cumbersome setups, and have a total cost of ownership of a few thousand dollars US (counting the PC)… all factors currently limiting scale.
VR scene content is heavy. The Lexus in particular is comprised of gigabytes of content to achieve that level of photorealism.
Delivering VR requires developing apps distributed through an app store. So-called “WebXR” browsing and other methods of casual discovery and consumption are still a work in progress.
Most of these points can also be made about headworn AR devices like Hololens and Magic Leap: slightly cumbersome, a bit pricy, and not in mass distribution yet.

All of the above add up to friction. And marketers don’t like friction. Advertising is a numbers game. While the use cases are incredible and take consumers deep into a brand’s story, the reach simply isn’t there. *

Smartphone AR: Immersion at Scale

Thankfully, there’s another way that immersive media can be used to deliver marketing content: via the billion-plus AR-capable devices people have in their pockets today thanks to ARKit and ARCore.

Across social apps, games, shopping and more, AR has exploded. We’re moving to a camera-first world. First-party applications that feature AR now reach into the hundreds of millions of active users. From social apps like Snap and Instagram, to games like Jurassic World Alive and Angry Birds AR, to utility shopping experiences from Amazon, IKEA, Houzz and Lowe’s, consumers are becoming increasingly comfortable with the camera as an integral part of their digital lives far beyond taking pictures and video.

IKEA Place: view furniture virtually and try it in your home with AR

Additionally, a good portion of the content folks see through these apps is brand-based. I think brand AR is here to stay, and it just keeps getting better. My favorite is the recent HBO promotion for the final season of Game of Thrones, with ice dragon Viserion menacing the populace at the Flatiron building in New York. This uses Snapchat’s “landmarkers” feature, which brings famous landmarks around the world to life with Snapchat lenses.

Winter came to the Flatiron District in Manhattan

Bringing AR Ads to Unity’s Developer Community

Most people are familiar with Unity as a creation platform. What you may not know is that we also provide one of the world’s largest mobile advertising platforms, in the form of delivering in-app ads. Because we are committed to helping our developers succeed, it’s not just about helping them make the games; it’s also about helping them monetize.

By way of stats, the breadth of our platform allows us to reach 1.7M devices globally, with 11.5B global monthly ad impressions, and over 9B minutes spent in Unity-powered games and apps. In addition to Unity Ads’ reach, there are several other aspects of in-game advertising that should be attractive to marketers:

Mobile time spent has now outpaced TV time spent [Source: eMarketer, April 2018]
When it comes to mobile, 91% of time is spent in-app vs. in a web browser. [Source: eMarketer, April 2018]
70% of gamers don’t multitask. [Source: Activate analysis, Activate 2016 Consumer Tech & Media Research Study] They’re focused on the screen in front of them, so that they can get back to the game.
In terms of demographics, who is the gamer? You may find the answer surprising: it’s a 54/46% female/male split, with a broad age range, and over 81% have a household income of over $100k US. In other words, it’s not the sweaty dude in his mom’s basement; it’s everyone.

Until recently, Unity-based AR brand applications and experiences have been relegated to experiments and bespoke activations, trapped within the confines of purpose-built applications. Even though AR apps are lighter than VR, and can reach a much larger addressable market, if it’s built into an app there is going to be friction. Unless there is obvious utility, say for buying plane tickets or shopping for clothes, brands have a hard time reaching users with bespoke apps.

But what if, instead, brand AR could be delivered in a more scalable, low-friction way via Unity’s ads service?

This was the question we set out to answer after ARKit was released and on its heels, ARCore — promising a billion mobile handsets capable of powering great immersive content. We wanted to offer the same immersive advertising power that Snap, Facebook and other first-party apps provide, but to Unity creators across our network of hundreds of thousands of titles.

From Concept to Beta to Reality

In September of 2017, I pulled down an open source code library that connects HTML5 with the ARKit and ARCore features built into the native operating system. This library, created by the brilliant Iker Jamardo at Google, enables JavaScript developers to create mobile AR experiences with tracking, plane detection and light estimation, and render the graphics in WebGL using AR extensions to Three.js.

A little prototype I built — a Star Wars stormtrooper dancing around in my office — worked well enough that my colleague Agatha Bochenek got really excited about the idea. The prototype proved that we were going to be able to deliver HTML5-powered AR into Unity’s advertiser SDK. (Our SDK uses HTML5 to deliver video and 2D interactive “playable” ads; this was just an extension of that same stack.) Agatha and I pitched it to our management and, not long after that, were greenlit to add AR support to the ads SDK. Fast-forward to May of 2018, and we had released a beta of AR ads to a limited supply of Unity developers.

Pilot Campaign: Disney Duck Tales

Check out the first campaign we ran last year with Disney for their DuckTales animated TV series. We call this type of ad unit a portal: enter a virtual world by walking through the real world into a magic doorway. The ad lets you jump into Scrooge McDuck’s vault, swim around in the coins, and meet Donald’s family and friends.

This first project was very instructive. In addition to standing up the tech stack to convert 3D assets and package and deliver dynamic HTML5 AR ad content, we learned important design and production lessons:

A small-sized creative can pack a big punch. Snap lenses and Facebook’s Spark AR platform have very restrictive size limitations, approximately 3-4Mb. This is for good reason, as brand content needs to load quickly so as not to interrupt the user flow and perform well across a range of devices, including older phones. Our AR ads are focused on the later-generation phones with ARKit and ARCore support (this is still a big number of devices), so we can be a bit more lenient on file sizes. Still, we try to keep our creative under 10Mb, ideally under 5Mb. Luckily for the execution of this first unit, Duck Tales is a cartoon, so we didn’t have to worry about really big, photorealistic textures but were able to deliver an experience true to the brand.
Be prepared to deliver across multiple channels. We were on point to conceive and develop the creative, as we often are, because AR ads are still a bleeding edge offering. Given that none of these AR ad platforms yet have huge reach, the client asked us if they could have the assets to run in other places, like Snap lenses. Our stack is web-friendly in a similar way to Snap’s, and the source assets for the 3D files were all in FBX format, so it was relatively easy to accomplish this technically. This first project conditioned us to always be ready to create the assets in a way that they can run in other channels.
The UI around the AR experience is just as important as the AR itself. How you message to the player what is going to happen, e.g. why they need to grant camera access to get this awesome experience, and what happens at the end like a call to action to watch the show, can make a big difference. Small things like having a Replay button on the end card can help with re-engagement, and get players interacting with the content multiple times in a single ad session.

Duck Tales was a fun way to get us off the ground. We learned how to design, produce and deliver our first experience, and got our first glimpse into what brand advertisers care about when exploring AR storytelling.

3D Commerce with Fossil Smartwatch

AR is not only about fun, it’s about utility. For our second campaign during beta, we ran a campaign with fashion with ad agency 360i and watchmaker Fossil to show the new features of their line of smart watches. In the ad unit, you select from several watch styles, swipe left and right to rotate the watch, explore the smart features of the watch face, and then, using your camera, look through your phone and try it on your wrist. Here we see what that user flow looks like, from first being presented with the ad, to the call to action.

The combined capabilities of 3D for exploring products and AR for bringing them into a personal, real-world context have tremendous potential for e-commerce. From discovery to engagement to purchase, XR can be an amazing tool for retailers. Someday, all products will be advertised this way!

Along the way to building this unit, we learned a few things:

We were able to deliver high photorealism within our small file size budget. We stuck to our 5Mb limitation and were able to deliver a great result, including multiple styles of face and wristband. The brand was very happy with how faithfully the watch was represented.
Camera access was important for the watch try-on, but we didn’t need ARKit scene understanding (positional tracking) to show the watch. Rather than having the watch “float in space” in front of you, in this case it would have been better to lock the camera view of the watch so it was always centered in the phone view. The experience worked, but that one tweak would have improved usability. This was a great lesson in when not to use a certain AR feature.
Requiring camera access introduces friction. Thus far, we have been delivering our AR ads into apps that require the camera anyway, e.g. AR-first games. But, to offer the strongest privacy measures and respect app store and user concerns, we show an additional prompt that tells the user, essentially, “this is an AR ad, it’s going to access your camera.” This extra privacy measure requires an additional click, which of course, introduces user friction and lowers conversion of users seeing AR ads vs. a non-camera enabled experience. We don’t believe there is anything to be done for it; but over time, we think this situation will get better as consumers become more comfortable with having the camera on for such experiences. We’re also experimenting with how to ease users into this better with different permission flows. I can’t share the details yet, but we will be talking about this soon and in the coming months.

Beverage Marketing with Miller Lite

Now that we are out of beta and supply is ramping up, we’re starting to work with other segments, including the beverage industry. For St. Patrick’s Day 2019, we worked with ad agency Spark Foundry, and creative agencies TiltShift and Trigger Global to put together a campaign for Miller Lite. The campaign featured multiple creatives, including 2 ad units running in Unity Ads, and a WebAR pub crawl experience running in-browser.

Each campaign teaches us new things about design and production, and what works well in this new medium. Miller Lite was no exception:

Develop multiple AR ad creatives to deliver a range of experiences, instead of trying to jam every experience into a single ad unit. This campaign employed 2 in-app units and a WebAR experience, all featuring a Miller Lite leprechaun character. The ad units were a portal which brought you into virtual pub to watch the leprechaun play games and drink beer, and a “dunking game” where you dunk the little guy into a vat of gold coins and beer. The WebAR was a real-world pub crawl that used image recognition, triggering virtual experiences by recognizing Miller Lite labels on actual cans of beer.
Develop different types of AR for different types of brand experience. While ad units are great to achieve scale because they can run anywhere and anytime, sometimes you want to limit an experience to a certain place at a certain time. Take, for example, the pub crawl. That happened in real-world bars only on St. Patrick’s day. It would be hard to deliver an in-game ad within these time and place restrictions: the player would have to be playing the game on St. Patrick’s day and only at one of a few designated pub locations. By providing ad units that run anywhere/anytime and a unique onsite activation the day of, we were able to get the best of both worlds.
WebAR is a great complement to Unity in-app AR ads. We wanted to deliver the pub crawl experience, which is effectively application functionality, with the lowest amount of friction. So rather than build an app and distribute it in the app stores, we turned to WebAR. The user still needs to grant camera permission, but at least there’s no download and install of an app. We are able used the same brand assets across the various units, including in-app and WebAR. This is in part because Unity AR ads are based on a web stack with extensions, so technically it is a (somewhat) straightforward effort to repurpose the creative. As far as the WebAR goes, even though mobile Safari and Chrome, frustratingly, still don’t ship with ARKit/ARCore features yet, there is an awesome set of workarounds. 8th Wall provides environment understanding and tracking, and Genee does the image recognition. These are awesome JavaScript-based technologies that work far better than I ever imagined possible.

On the Horizon

The lessons we’ve learned thus far have primarily been qualitative. We only came out of beta at the end of last year, and are still testing before we turn AR ads on in a much broader supply of apps. We’re getting ready to do that soon, and we hope to have meaningful numbers to share in addition to our other great learnings before the end of the summer.

Meanwhile, there will be more experiments to run and a few cool new things on the horizon.

AR Performance Ads?

Unity’s historical, bread-and-butter ad business is actually in game ads, what we call performance ads, which are ads intended to drive installs of apps. When a player sees an ad for a game and installs that game, the developer of the game that hosted the ad gets a user acquisition fee. About three years ago, Unity opened our brand advertising business to provide brands access to our great audience of free-to-play game players. With brand ads, the goal is not always conversion; it’s often simply exposure or engagement.

Thus far our AR ads have only been for the brand side of the business. But we’ve been thinking about what it might mean to run AR performance ads, too. The video below is a capture of a creative test we did for Rovio: an ad for Angry Birds 2 running in AR (on Unity’s San Francisco office rooftop).

Angry Birds 2 AR Ad Test Creative

A few things of note:

This is NOT an ad for Angry Birds AR. That game hadn’t even come out yet when we produced the creative test. It’s an ad for Angry Birds 2, with the assets translated into AR.
This is also not an official Rovio ad. Rovio has seen the creative and likes it. They have given Unity permission to show it as a proof of concept. That’s it. But it is fun; it shows the concept nicely.
We still don’t know what to do with this. Are performance AR ads even a thing? With brand AR ads, the key is engagement: you’re getting consumers to experience your brand in a new way. It’s becoming clear that brands really want AR and are starting to value it for this purpose. In game ads, however, the key is to get players to quickly discover and install the game. If engagement is key to that conversion, then by all means, create as engaging an ad as possible. Sometimes, video is enough; but other times a 2D “playable” mini-game, such as a demonstration level or a demo of key game assets, helps convert players better. So the burning question is, does AR add value for obtaining new players? Maybe if the game itself is AR and therefore the ad can demonstrate it better… more work to be done here.

Mobile AR Not Standing Still

We have several enhancements that we’re contemplating for the Unity AR ads core tech, including face tracking and image recognition. These have both been around in the major mobile platforms for a bit now; however, how these would surface in ad units, and what are the best use cases for each, remain to be seen.

Also, the tech is not standing still. At WWDC today, Apple announced ARKit 3, with new features including advanced lighting and rendering, camera effects, body tracking and body segmentation/occlusion. This was shortly after Google showed off amazing new ARCore features at Google IO, including advanced markers and superior light estimation.

It’s game on for mobile AR, and advertising platforms like Unity will continue to track these developments and include those that make sense for our SDK.

Industry Collaboration and Standardization

The final frontier for AR advertising is standardization. The current state of affairs is that each major AR ad channel — currently including Facebook, Snap, Google, Unity, Verizon, and the New York Times — employs a similar tech stack, but none of it works quite the same. It’s like the early days of Internet banners and video: there were a handful of formats and proprietary players. Eventually it all shook out, and the web standardized on just a few of them. A few years after that, groups like the Interactive Advertising Bureau (IAB) settled on how those technologies would be used to deliver ads.

Eventually, AR will go through the same progression toward standardization. The IAB is about to publish a playbook on using AR for marketing. This is an educational documented intended to inform brands and agencies on how the technology is being used today. (I’m delighted an honored to be a co-chair of the IAB working group that wrote this report.) But the next step after that is going to have to be standardizing on the tech itself. As I described above, Unity builds AR ad creative in a way that the 3D assets can be easily repurposed for delivery to other channels such as Snap lenses. The approach works but requires undue amounts of gruntwork and isn’t scalable.

Ultimately, AR ads have to be write-once, run-anywhere, just like video. It sounds like a tall order, but the industry could move in that direction quickly. Under the covers, everyone is using file format and language standards like glTF and JavaScript; so most of the hard technical work is already done. Recent developments from immersive web innovators like Google demonstrate a commitment to the shared vision of low-friction creation and delivery of AR content for education, brand advertising and retail. Discover/engage/purchase via search, your camera, or ad units. This future is upon us. Unity is in this mix, and I’m thrilled and feel privileged to be able to work at the frontier and learn what this exciting medium has to offer, every day.

* Note on VR scale. VR is still climbing out of the “trough of disillusionment,” up the “slope of enlightenment,” and so on; therefore the number of headsets shipped globally still aren’t great, generously in the small tens of millions if you don’t count Google Cardboard (and I don’t). But I’m super psyched about the Oculus Quest, released just a few weeks ago. The early returns are that this untethered, mobile, six-degree-of-freedom headset is the first consumer-ready VR device that will achieve massive scale. But we will need to wait out the better part of 2019 to see just how big it is going to get.