The New Pioneers
Sundance New Frontier once again showed that XR is not only alive and well; it continues to break new ground.
Tears streamed down his face, leaked out of the HMD. From time to time, he would wipe his face with his shirtsleeve. Occasionally, he would be distracted by his thoughts, wondering who in the cozy ski condo was watching him heaving sobs and turning his head wildly about from time to time. Or who was responsible for cleaning the Oculus Go after each viewing — and would it actually ever get clean? Those were a lot of tears. But mostly, this white boy spent the better part of twenty minutes Traveling While Black… and it was devastating.
They did it to me again. They do it to me every time. Felix and Paul Studios’ latest production, in collaboration with Ayesha Nadarajah and first-time VR director Roger Ross Williams, is a heart-rending trip through black America that had me bawling uncontrollably throughout. Traveling While Black recalls first-person stories from a time when black Americans needed a guide on which places were safe and accepting of blacks on the road (the “Green Guide”). It’s set in Ben’s Chili Bowl, one of the Green Guide safe spaces, and features multiple conversations with people who lived through that time, as well as reenactments set in the early days of the civil rights movement. It puts you right in the action, at the table with the others. By the time it’s finished, you see how far we’ve come as a nation with respect to race relations. (Spoiler alert: not very far.)
Traveling While Black is 360 video, not interactive VR. That distinction matters to some — not me — and for those who say that 360 isn’t VR, maybe you should skip ahead; there’s plenty of talk about interactive content below. Still with me? Good. To be honest, even though I spend most of my time with interactive XR content, I have not yet seen a CG-based piece of cinematic XR that can instantly put me in a state of empathy the way that a well-done 360 film can. And Felix and Paul are the masters of the craft. This piece was the best XR coming out of a crop of amazing new stuff showcased at the Sundance Film Festival’s New Frontier program for 2019.
New Frontier is about the cutting edge of cinema. From immaculately produced 360 like Traveling While Black that tears your heart out, to mind-bending experiments in scale and time, to gorgeous art direction and promising new hardware platforms, this year’s Sundance showed, yet again, that the medium continues to evolve, driven by the pioneering storytellers of tomorrow.
New Year, New Tech
New Frontier 2019 featured an early look at great new hardware and big steps forward in production tech, powering some of the most intense experiences imaginable.
The Oculus Go is a VR device for the rest of us: $200, no wires, and fits easily in your bag. The Go really shines with video, where through some combination of optics and battery management, you get much better video quality than older portable VR rigs like the Gear VR. I’m sure this contributed to the sublime experience I had watching Traveling While Black. This 3DOF device has been out for about a year, and I think it’s finally starting to show its true value as a self-contained portable immersive video device.
After I wiped off my tears and white guilt, I jumped to another experience hosted at Oculus’ private suite, and got my first taste of their newest device, the Oculus Quest. This is an awesome HMD, with full 6DOF movement and controls. Based on the demo I experienced, the Quest has enough power to create convincing reality. The Under Presents is part immersive theater and part multiplayer VR, set in a virtual nightclub (“The Under”) with live performers onstage prompting the 3 other players to interact and participate in the fun. Created by Tender Claws, the piece occupies an absurdist VR niche the team first staked out with Virtual Virtual Reality. While still in development, The Under Presents shows the promise of bringing these two highly engaging entertainment media together. And I was thrilled to finally experience lightweight, untethered 6DOF VR. I can’t wait for the Quest to come out!
A surprise hardware standout was the Magic Leap One. Since the company de-cloaked last year, we’ve seen a steadily improving stream of compelling content, but mostly tech demos until now. Sundance was something of a milestone for the device, as it was used to power three of the stronger experiences at New Frontier. I’m still frustrated by the narrow field of view, but unfortunately this is just where we’re at today with headworn mixed reality. For the most part, it didn’t get in the way of the experiences.
He sat in the child’s room, nervously anticipating what was to come. Points appeared in space, gathered, formed into a nebulous shape. The shape talked to him — more of an interview, really. The shape asked; he answered. After several questions, a child appeared, reclined on the bed. Then his mother appeared and told a bedtime story. Then: rats. Talking, feral, politically woke rats. Beating back fear, he persevered, hoping that this murine horror would end shortly. It did, replaced by something worse. The AI asked another series of questions, ending in a grave moral test. He failed.
A Jester’s Tale, the latest product from the fertile mind of Asad J. Malik, is a disturbing journey into fable and imagination. Combining the alienated angst of Stanley Kubrik’s AI with the spine-tingling terror of the 1970’s B-movie classic Willard, A Jester’s Tale uses the Magic Leap One to transform a child’s room into a test of sentience, will and, ultimately, morality. While it’s a bit of a departure from his earlier politically-focused work, it continues Malik’s style of hyper-personal storytelling, with you as a central figure that comes face to face with the other major players to help the story along.
A Jester’s Tale was created in partnership with RYOT’s production team, and used the Metastage volumetric capture system to record performances by live actors in full 3D. Metastage has been steadily improving on their production process built on Microsoft’s “mixed reality capture” tech; while it wasn’t perfect (hands in particular are still hard to clean up), it’s a step change from the captures seen in pieces like Aeronaut VR. And that one won a Cannes Lions innovation award last year! (Just sayin’…)
It was so nice to have all these shiny new toys to play with, and have my mind blown in the process.
Studies in Space, Time and Scale
If Sundance 2017 was about professionalizing the craft of XR storytelling, and 2018 about commercializing it, 2019 marked a trend back toward experimentation. Several of the pieces explored new form factors and narrative formats. The ways in which creators toyed with physical scale and time made a particular impression on me. With XR you can be giant-sized or microscopically small, travel to other eras, or move fluidly back and forth through time. Many of the great pieces at Sundance played with these concepts. In no particular order:
Melissa Painter’s Embody, done in collaboration with LuluLemon’s innovation head Siân Slawson, brings movement and generative art together in VR to create a space that embodies you fully but transcends physics, as you dance about a world tree and float through space. (It also taught me that it’s impossible to do tree pose wearing a headset with no visible frame of reference in the room around you…)
Emergence (by Matt Pyke and Within) is a great tech demo of crowd behavior AI. Presumably intentionally, this piece invoked for me a godlike feeling. You control a human figure that is swarmed and followed by crowds, first at close proximity, then eventually from a distance, above an endless landscape. Emergence gave me the sense that I was a messianic, all-powerful, Dr. Manhattan-like being beyond space and time.
In Interlooped (Maria Guta, Robin Mange, Javier Bello Ruiz et al), you enter a closed room, don a VR headset and start moving around. Look down and you can see your arms and body using some crazy cool voxel rendering tech. You have a live companion (Guta). After cavorting with her for a while, you realize that your movements have been recorded as VR snapshots and are being played back for you, as multiple versions of yourself filling the space. You are both content and star.
The branching storylines and live immersive theater of Mechanical Souls (Gaëlle Mourre, L.P. Lee et al) take what would otherwise be a straightforward 360 video story into a different realm. Even though it’s primarily made of captured video, behind the scenes there’s a Unity-powered gaze-tracking system driving an AI that decides which part of the storyline each person experiences. I partook with four other audience members, and none of us saw quite the same thing. What we did experience collectively was a fun mashup of Chinese soap opera tropes and Westworld/Black Mirror near-future dystopia.
These are just some of the ways in which the New Frontier creators messed with four dimensions. My favorite technique involved virtual dioramas: small-scale, self-contained worlds with the viewer as a giant-sized observer.
Winter. Bare trees, falling snow, gloomy skies. Gives way to warm spring, full bloom. Gives way to sunny summer. Then autumn… but somehow it’s a year earlier. The little people scurry about. Gunshots. Screams. Shouts from inside the tiny house. We have no idea what’s happening, but we’re drawn in. Each of us leans forward to get a closer look as the miniature drama unfolds. It’s hard to follow; one of the audience has the controls and the timeline keeps shifting. I slowly begin to piece it together. Ah…!
THE DIAL, created by Peter Flaherty in collaboration with RYOT, is a multiplayer, room-sized augmented reality experience. Housed in a slick white designer cube, the installation combines projection mapping and an ARKit-based experience to create a dollhouse-sized virtual diorama of family drama and skeletons in the closet that takes place over several years’ time. One of the players assumes the role of “navigator,” where they get to control the flow of the story. This made it challenging for the other viewers to follow the narrative, but at the same time I think it forced us to go with the flow and just experience the world in which the story was set. I was fully immersed in this tiny place, even though I was a giant, physically outside of the action.
For you Philip K. Dick fans out there: THE DIAL is full-on Perky Pat, minus the narcotics. I loved it.
Art, Above All
I think we are still defining what XR is at at its core. Some would say that it’s storytelling; others, world-building; still others, experience design. Many of the pieces at New Frontier combine these ideas, but rather than try to be any one specific thing, they just are. Just art.
Gloomy Eyes is an animated three-part series about forbidden zombie-human love, created by the team at Atlas V. Episode one premiered at New Frontier. There are so many fantastic elements to this piece, including a creepy but somehow adorable protagonist; a silky voiceover narration by Colin Farrell; a carefully constructed diorama world (you’ll find yourself leaning in really close to see all the detail); and a gorgeously rendered environment powered by (cough cough) the Unity engine.
Runnin’, an incredible interactive music experience, is based on the song of the same name by Wajatta (Reggie Watts and John Tejada) and directed by Kiira Benzing. Runnin’ makes use of many of the cool tech and techniques I’ve already touched on, including an untethered Vive with the wireless adapter, volumetrically captured people using Intel technology (the band is singing, dancing and spinning in there with you!), and otherworldly physics to create an a Escher-like rave world powered by this relentless retro funk/soul tune. It’s sooooo good! Wave them glow sticks!
Finally, there was… Grisaille.
You are guided down a long corridor. There’s a podium. You walk up to the podium. You put on the headset. You touch the podium, lean on the podium, look around; emptiness. Then she enters. She is. It blooms all around you. You move through it, the art. Space. Nebulae. At the center of it all… her eyes. You continue. Gray. Color. Light. The infinite self. Where are you? Where is here? IRL, someone takes your arm. You remove the headset. You are still in the art, but in this other dimension. She is there, too, the artist. She paints you into the picture. But there is no you; no her; no art. It is all one. The art is all.
How to even talk about Grisaille? It’s a visually stunning work created by an amazing new talent, Teek Mach. Teek has spent thousands of hours painting in TiltBrush. She brought her formidable skills to bear to create a transcendent virtual environment, based on a fifteenth-century painting technique that mimics sculpture with shades of gray (hence, “grisaille”). Supporting the scene but never eclipsing it are soundscapes composed by the prolific Joel Douek. For a handful of minutes, Grisaille pulls you through Teek’s vision and wraps its arms fully around you.
But this wasn’t the half of it. After I exited the VR headset, I was in a projection environment with the visuals around me. Then Teek took my hand, and in one of the most intimate experiences I have ever had, painted me into the world. That’s the only way I can describe it.
With a few strokes of the Vive controller — in one fell swoop — Teek Mach did something that artists have been trying to do for millennia with little success: she shattered the barrier between artist, art, and observer. And it wouldn’t have been possible without VR.
See you all on the Frontier.
This article relates a sliver of what was on display at New Frontier. I couldn’t possibly write up all the amazing experiences I had — and I wasn’t able to see all of the work. Also, given limited space, not all creators and collaborators have been credited. You can find a more complete list here. And do try to catch these experiences, maybe at an upcoming festival. You won’t be sorry.
