William Uricchio is one of the world's top media scholars, a professor of comparative media studies at MIT, and professor of comparative media history at Utrecht University. At MIT, Uricchio is the founder and principal investigator of the MIT Open Documentary Lab, a close collaborator of the IDFA DocLab Research & Development Program. We talked to Uricchio about his research with DocLab, the nature and development of new digital media as well as their relationship with traditional forms of expression, and the cultural and technological trends in the field that we can expect in the coming years.
This year we're looking very closely at user experience particularly as it pertains to story. How do people connect the dots of their experience? Do they find the same story that the author told? Even the most abstract media experiences create distinctive impressions, sometimes even stories. So I'm interested in what user experiences are, what forms they take, and how they are expressed. And I’m especially interested in variations in how people make sense of their experiences, even the most clearly labeled and defined projects.
In Event of Moon Disaster [an immersive installation which includes a deepfake of a Nixon speech supposedly recorded in the event of the Apollo 11 crew getting stranded on the Moon] is a clearly labeled project, and I can't tell you how many people we heard say, "I didn't know they filmed contingency tapes." The installation says "deepfake" everywhere you look, yet some visitors didn't get the point. How is that possible? So I'm interested in interpretative variability, in why different people find different things, and in the patterns of that variation.
We’re still in the early days of artificial intelligence and media like virtual reality. These forms are still "soft" and makers are exploring their possibilities. Users are also orienting themselves, which partially explains their variable interpretations. I feel as though we're in the first decade of cinema, when filmmakers were trying to copy theater, with its proscenium composition and language of the stage, and slowly discovered that they could take close-ups, and cut time and space… It’s amazing to watch new media acquire cultural form. It’s like watching a new generation of makers "invent" their own language, just as film pioneers such as D.W. Griffith and Edwin S. Porter did over one hundred years ago.
Just as some early filmmakers took their lead from older theater conventions, we can see some VR makers taking their ideas of storytelling from older media such as film and games. I'm trying to make visible this tendency to retro-fit VR into earlier media forms. It's fine if people do it, as long as they know that's what they are doing. But meanwhile, let's find a vocabulary, a conception that's more specific to VR. This does not necessarily entail a complete break from the past. I suspect that there are many good sources of inspiration to be found in discovery-based forms like games and immersive theater. We need to pick our historical precedents carefully, since some seem to address VR’s affordances better than others.
I think we're reaching that moment where VR will find its own way, its own specificity as a medium. It may take a few jabs with a sharp elbow to keep things moving, and that's the self-appointed task of critics!
VR and developments such as AR offer a great way to shift the balance of power between authors and users, letting users find their own stories rather than simply listening to the author tell them one. In this scenario, authors become more like the designers of environments loaded with narrative potential. Think Disneyland (a theme park where we can wander and connect whatever dots we would like to) rather than The Lion King (where we are told a particular tale). There’s nothing wrong with well-told stories, as literature and film prove every day; but VR has a distinctive potential. It can put the user in an authored world where she can follow her interests, discover things, and find her own story.
Embodiment is also a pretty important issue in VR, and happens in good ways and bad ways. The bad way is when users feel physically ill or dizzy and have to stop the experience. The good way is when users feel out of body. This can take many forms, such as feeling that we are in another place, or physically identifying with a character we see, or even literally leaving our own bodies. Sometimes embodiment is an attribution that we give to the characters we encounter. We react as though they are really there, even if we know differently. As with story-finding, we don’t have a great way to talk about this. We still need to better understand how this works, and develop an appropriate vocabulary.
For example, Ayahuasca - The Shamanic Exhibition [a VR installation that represents the psychedelic experience of using Ayahuasca, expanded with a special exhibition at IDFA] created an out of body experience for me, but I think the chanting on the soundtrack created this effect rather than the images; or perhaps it was the combination of both. I say this because I’ve encountered a similar feeling with several other Indigenous VR pieces, all of which were built around mesmerizing chants in the soundtrack. We need to better understand how these experiences work in order to deploy them predictably and effectively.
We are about to experience a radically new form of VR with systems that track our eye movements, and with them, potentially our pupil dilation and heart rate as well. This matters for three reasons.
First, with systems that use foveated rendering, the computer generates only that part of the world where we are looking. This contrasts sharply with our current strategy of creating the whole world, only a small portion of which we can see. Foveated rendering is relatively efficient, meaning less drain on the computer and higher resolution images where we are actually looking.
Second, this technology coupled with AI will enable us to build much more responsive worlds, including with characters and worlds that acknowledge and respond to us. This will be a game changer, and mark the end of the Madame Tussaud effect, where characters seem unaware of our presence.
And finally, we need to think proactively about data privacy, because where we look and how we physiologically react will surely be seen as a treasure trove by marketers. Just as we have to invent a new vocabulary and set of techniques to work with these new possibilities, we need to think about ways to ensure our privacy.
There will certainly be other technological changes that will enable us to develop a new language for VR. But VR is different from most of our classic media technologies in the sense that it has not yet stabilized. Consider, say, film, which was pretty much in its current form of 35 mm way back in the Lumières' day. Sure, we added color and sound, but the basic medium was fixed. Compare this to the various forms of VR such as 360, real-time capture, and foveated rendering, together with the ever-changing technologies coming from companies like Oculus. VR remains unstable. And VR’s story is complicated by the pressure exerted by huge corporations, which is arguably deforming the medium in its still nascent days.
The tech giants need to maximize their return. Innovation means churn, and churn confuses the marketplace. So the large corporations exert a conservative impulse, putting the brakes on the development of the medium.
What we see here at IDFA DocLab is mostly work done by artists who are pushing the boundaries and exploring the potentials of the medium. Their work is often difficult to classify: where would it be located in the Oculus store? Artists are the medium’s research and development department. They often seek out constraints and push them to the breaking point; they take assumptions and so-called best practices, and turn them on their heads. Artists are developing the VR language of the future, or at least they are provoking critics to think hard about that language and to discover what the medium can really do.
Constraints can be a creative incentive. Artists are really good at finding a wall and breaking through. And that's what a festival like this does—it's terrific as a place where new ideas, new applications, new possibilities for the medium take form and are tested. So despite the pressure from tech giants to simply sell more stuff, what happens here is that a lot of new ideas are born and hopefully the medium will keep up and learn from this and grow.
We're at a very interesting junction right now. In the film world everything is now pretty much digital, even though much of it emulates the work of the celluloid past. I mean by this things like genres, and the language of film (establishing shots, parallel editing, and the rest). The idea of cinema has been remarkably persistent, even if the technology of film has been largely replaced by digital video alternatives designed to mimic film. Given this disjunction between familiar film form, and quite different video technique, it’s rather remarkable that so much of the digital video technical pipeline has been retrofitted to fit with the old celluloid world.
In the next five or so years, I think we are going to see some big changes in that pipeline in the form of the engines we know from VR, AR, and games such as Unity and Unreal. A new generation of these engines is going to be used for prefiguring and editing film and various related texts. This is as much an acknowledgement of the cross-platform circulation of texts (tie-ins in the form of promos, phone versions, VR, games, and more), as it is the simple efficiency of these digitally-native technologies.
I don’t mean to say that cinema is ending any time soon! When I say the word "cinema" in the classroom, students immediately think "fiction," "two hours," and as mentioned, a whole battery of techniques. But if we are lucky, we will also see some exploration of new possibilities. Imagine the Bandersnatch episode of Black Mirror, except that instead of clicking your way through the branched narrative, your eye movements alone will seamlessly navigate you through the ecosystem for a personalized story. That would be quite a chore to produce with today’s systems, but with game engines like Unity (plus eye-trackers), one can imagine quite interesting possibilities.
Cinema’s conventions and assumptions will stay with us, I think, and that’s fine. There’s nothing wrong with cultural stability! And media industries will certainly do their best to make their products predictable—both for us as consumers, and for themselves as revenue producers. But this should not preclude an expansion of cultural possibility, and an exploration of the affordances of new technologies. When it comes to culture, I think that we can have our cake, and eat it too. And as I said earlier, that’s where the artists and festivals like IDFA come in.
Of course, it's a business model and it works. Remember the cinema’s key operating principle: first you pay and then you walk into the dark room. The customer has to be convinced that the product is good even before seeing it. This means that predictability in the form of famous directors and actors, genres, remakes and the rest are important in order to reassure the audience that they are facing a low-risk investment. This contrasts with how we buy clothes (we try them on first or we can return them), or purchase meals in a restaurant (first we eat, then we pay, but not if we are dissatisfied). Cinema’s economic and cultural logics are tightly twisted together, and this accounts for the default conservatism of industrially-produced media products.
And that’s why festivals are so important. They provide a place for artists working outside the industry to show their work, to demonstrate the importance of their vision, and to push the medium to the breaking point in order to show what it is capable of. And IDFA stands out because it does all of this not just for art’s sake, but as part of a larger effort to use the documentary arts to help us see the world more clearly and critically.
