The immersive sector has become synonymous with cutting-edge technology innovation, and the UK is now internationally renowned for this kind of creative production. According to the Culture is Digital report commissioned by the Department of Digital, Culture, Media and Sport, ‘immersive and augmented reality are changing the way in which we are able to experience the world around us, offering a particular opportunity as international demand is increasing’ (2018).
The likes of virtual reality (VR), augmented reality (AR), spatial audio and artificial intelligence (AI) all present rich avenues for the creative industries, capable of transforming our cultural and media experiences. But they also pose new kinds of methodological challenges for researchers and makers alike, and the central question at the heart of this special issue is:
What role does immersive media play in the design of new research practices?
Academia and industry alike are now working towards developing an innovative set of aesthetic categories, terms, concepts, practices and methodologies to make sense of the uniqueness of immersive media technologies in and across the creative landscape. But with much research across the disciplines of media, performance, art, computing and beyond all increasingly exploring the creative potential of immersive technologies and platforms, it is key that we also better understand the necessary approaches to researching these kinds of technologies and platforms, and how they feed into our creative research practices.
How, for example, can we best engage with the innate hybridity of immersive media, given the way that VR and AR experiences often incorporate elements from performance, games, film, gallery installations and even theme parks? What is the impact of this hybridity on our ability to define immersive media as an object of study? If AI is in some way capable of re-imagining our relationships with daily life, then how are these kinds of technologies reshaping research practices too? How might immersive media impact our interpretation of audience or user responses across the creative industry landscape? And is now the time to develop new kinds of research practices based on the affordances of immersive media?
This special issue engages with these questions by exploring the ways in which different disciplines and different corners of the creative industries are approaching the task of researching (with) immersive media technologies and their audiences. Articles span a range of cutting-edge topics, comprising VR film, AR games, immersive sound, mixed-reality performance and art installations, immersive marketing, and uses of immersive media as forms of audience engagement, cultural heritage, and academia-industry R&D.
Characterising Immersive Media
Immersion can refer to being immersed in a space (spatial immersion) or being mentally immersed (strategic immersion, narrative immersion and tactical immersion). According to the International Society for Presence Research, tactical immersion is experienced when performing tactical operations that involve skill; strategic immersion is more cerebral, and is associated with mental challenge, like a chess game; and narrative immersion, finally, describes the way that an audience may become invested in a story, like when reading a book. What, then, does this mean to our understanding of creative research? Understanding what role immersive media can play in the design of new research practices must first start with an understanding of what characterises immersive media, both in terms of aesthetics and engagement. In other words, what are the 'promises' that this kind of media can offer?
Promise No. 1: Immersive Media as a New Way of Seeing the World
Artist and academic Duncan Speakman believes that we should rethink our idea of immersive media by shifting the emphasis away from being a creator of or a participant within an immersive experience, and instead focus on the ways that immersive technologies allow us to become more aware of what we are already immersed in: ‘Immersive media is a way of exposing how we exist deep inside the tangled ecologies of [the world] rather than external viewers’ (2020) Explains Speakman:
Immersion … might in one instance conjure images of being underwater and in another reading a book. It might mean being immersed in a task or immersed in the invisible microwaves of digital networks. Yet when we talk about immersive media it often feels there is a lean towards describing a kind of cocooning. Whether that’s a darkened room filled with sound or the forward-looking and body-forgetting embrace of a VR headset, it’s often a totality – immersive media meaning something where the media itself is all-encompassing in some way, where the only “thing” you are immersed in is the work (2020).
If we were take Speakman’s definition of immersive technologies and apply it through the lens of research practice, it opens up fascinating avenues of investigation: for instance, according to Speakman, the fundamental power of immersive technologies is that they ‘highlight, reveal or create one or more of the multiple layers of things we are already immersed in’ (2020). This echoes Jaron Lanier, one of VR’s most recognisable figures, who claims that through VR we are able to experience a far broader range of identities and are capable of seeing the world in a more profound way. So, in other words, immersive media technologies have the potential to help us see the world in a whole new way, which raises fascinating questions about how the use of such technologies within or as research might lead to new - or at least very different - forms of knowledge-creation and understanding.
For instance, in her contribution to this special issue, Celia Quico’s article ‘Nazaré and Immersive Media: New Approaches to Cultural Heritage through Mixing Old and New Media’ analyses the Nazaré Immersive project from Lusofona University, which is based on the use of stereographic images that are remediated and repurposed for immersive media technologies. Located in Portugal, the coastal town of Nazaré has inspired many visual representations by painters, photographers and filmmakers. The project aims to contribute towards the promotion of Nazaré’s cultural and natural heritage in an innovative way, mixing traditional and contemporary representation technologies. Furthermore, the Nazaré Immersive project addresses current environmental issues such as over-pollution and the impact of touristification, challenging its participants to speculate about possible futures for Nazaré – and uses immersive media like 360º videos as a creative approach to communication.
Equally, Matthew Freeman considers the relationship between augmented reality’s potentially world-altering mode of engagement and the way that such technology should be marketed. In his article, ‘“I am not really sure how to explain”: Marketing Augmented Reality Experiences to New Audiences’, Freeman argues that if the immersive sector is to fully move beyond engaging early technology adoptors and gamers, and is to truly capture mass audiences from different corners of the cultural landscape, then more gateway promotional content must be developed. That is, marketing that engages those new to immersive technologies. How, though, can the magic of immersive 3D worlds be communicated via 2D media? Addressing this current industry challenge, this final article discusses research funded by StoryFutures Academy in 2020, which aims to identify and experiment with new promotional strategies for how virtual and augmented reality experiences can be better marketed to today’s audiences. Freeman’s article outlines his audience-informed conceptualisation of immersive media in terms of how it can provide (i) a new way of seeing the world, (ii) a new way of connecting with other people, and (iii) a unique personal experience. He then outlines his approach to applying this research to a set of promotional materials created for Studio McGuire’s The Invited, a 2020 CreativeXR-funded immersive prototype that reimagines the story of Dracula with a pop-up book and augmented reality technology. He also discusses the results of evaluating these research-informed promotional materials on audiences, outlining some key learnings about how best to communicate the magic of immersive media to new audiences.
Promise No. 2: Immersive Media as Connection with Other People
Just as important to understandings of immersive media as that which promises new ways of seeing the world is its potential to develop new connections with other people. Any form of media is likely to show us the world, or at least a representation of a world, but can it always show us as part of that world, shaped by our embodied interactions with other people? The idea that immersive media represents a paradigm shift from the mechanics of storytelling to ‘story-living’ has been well-discussed. But there remains merit in understanding the likes of VR and AR not in cinematic or even strictly visual terms, but as something far more spatial. By thinking of immersive media as 3D architectural space, in fact, it lends itself to questions of intersubjective relationships and joint meaning-making processes. With AR experiences, for example, sometimes this means that performers and audiences can share a personal and intimate moment of interaction together, while in some VR experiences, the technology is capable of mediating relationships between multiple audiences and non-live characters.
A perfect example of this kind of relationship is provided by Rik Lander’s article, ‘Audience as Co-Writers: Using Conversational AI to Deliver Audience Agency in a Participatory Drama’. Lander offers a glimpse of the elusive holy grail for participatory dramatists: a way to offer audience members a role within the immersive narrative and to give them genuine agency over events and even the outcome. Lander’s article describes the use of a ‘conversational artificial intelligence’ as both a character in, and the co-writer with the audience of, a live theatrical drama, I am Echoborg. This approach represents a novel and powerful means of delivering to audience members both narrative agency and the ability to take on a role in the drama. It also demonstrates how an AI can be a compelling dramatic character. Lander explores the psychological mechanisms exploited in the creation of this immersive event, such as breaching environments and projection of theory of mind, and considers the role of structure as a means of balancing authorial voice and audience agency.
Equally, in their contribution to this special issue, Carina Westling, Elliott Hall and Mary Krell's article ‘Inverting the ‘Black Box’ of Technology: The Digital Ghost Hunt’ provides an account of her own project, The Digital Ghost Hunt, which was a 2017-2019 collaboration between the University of Sussex, King’s Digital Lab and KIT Theatre. The project was funded by the Arts and Humanities Research Centre in the 2018 ‘Immersive Experience’ funding stream and was awarded a full grant for the pilot and an extension to build and widen audiences in the second phase in 2019. Westling et al explore the role of an immersive experience in deepening social and historical engagement with the physical world in contrast to the ‘black box’ design paradigm and frames the interior of technology as an opaque domain that is only open to initiates. This work allows for the comparison of the engagement of participants invited to engage with technology-aided experience as builder-makers and as naïve audiences to an already designed experience. Westling considers the interactions between design and technology as a means to bring people of different ages and attitudes to technology together through shared learning and problem-solving, also with a nod to the popularity of escape room experiences.
Taking this notion of mediating relationships between multiple audiences and non-live characters and exploring it in more pragmatic terms is the central aim of Helen Davies and Bronwin Patrickson’s article, ‘Is This Even Possible? Co-Creating University and Industry Learnings for Immersive Experiences’. This article discusses an ambitious university-industry collaboration based at the University of South Wales that was designed to further develop digital storytelling practice within the UK Creative Industries. During this UKRI-funded, Audience of the Future R&D demonstrator project, Davies and Patrickson collaborated with Bristol-based animation studio Aardman, who were both licensors and content contributors for an immersive transmedia experience called The Big Fix Up based on the globally loved Wallace and Gromit intellectual property. Working closely with Aardman, the consortium drew upon their combined skills in games production, animation, creative marketing and new technology development to create a mobile application that marries Wallace and Gromit with mobile storytelling and the latest augmented reality and mixed reality technologies. One of the key aims of this larger industry-led demonstrator programme was to propel new immersive storytelling insights informed by extensive audience research. In their article, Davies and Patrickson examine this university-industry collaboration in methodological terms, outlining the factors that enabled them to negotiate cross-sector issues whilst reflecting on the multi-faceted approach to audience research that emerged from those negotiations.
Promise No. 3: Immersive Media as Personal Experience
In effect, we can argue quite convincingly that immersive media, by definition, places its audiences in the middle of a complex virtual-physical entanglement, and this entanglement cannot help but inspire a viscerally personal response or mode of engagement, in turn forming a liminal experience around audiences that captures the sense of leaving one space behind but not yet fully entering another. In short, a VR/AR experience cares where it is. For example, seeing a painting inside a VR recreation of, let’s say, a historic chapel, will no doubt create a particular emotional effect on those who see it. Meanwhile, seeing the same painting in a contemporary art gallery, this time via AR, will likely create a different emotional effect, as would seeing the same painting, again via AR, but this time in your own living room.
This relationship between place and immersive media is at the heart of Izabela Derda’s, Tom Feustel's and Zoi Popoli's article, ‘(In-)Between Spaces: Challenges in Defining the Experience of Space in Mixed Reality Art Exhibitions’. Derda et al explore three conceptual challenges in defining the experience of space within the context of mixed-reality art experiences, specifically based on the accounts of her interviewees. In the article’s conceptual mapping, the authors identify the issues of in-betweenness, inseparability, and (un)realness, which she argues can be used to describe the problem of the elusiveness of space in mixed-reality environments. They conclude that the relationship between mixed-reality technology and its immediate spatial context is highly interdependent and has profound implications for the purpose of virtual augmentation, the user experience, and its ability to mediate one’s own sense of place.
Exploring the question of immersive media’s impact of personal engagement from a very different disciplinary perspective, Steve Whitford’s article, ‘The Truth of Sound: Exploring Immersive Location Sound Recording in Realist Filmmaking’, focuses on the somewhat neglected (at least within scholarly circles) area of location-based sound recording, drawing much-needed critical attention to the intricacies and skills involved in location sound recording within realist filmmaking – both scripted and unscripted. Through his own practice-as-research, Whitford aims to reimagine an ontological definition of location sound recording by proposing that a reinvigoration of the ‘realist’ genre can be achieved by connecting the storytelling skills of recording for single camera with the new opportunities afforded by immersive audio technologies – ambisonics here being a vital part of that development process. Whitford demonstrates how use of such immersive audio technologies offer new creative opportunities for realist makers and audiences, based on the unique experience of geographical place and physical event that immersive audio delivers.
Finally, in her article, ‘Dis_place: Reflections on Creating Mixed Reality Performance Using Virtual Reality Technologies’, Kerryn Wise discusses the Dis_place project, a mixed reality performance that takes audiences on a personal journey using a range of virtual reality technologies, immersive sound, and live dance performance. Through close analysis of her own practice-as-research, this article presents reflections on the developing creative strategies and approaches to making VR-based mixed reality performance. Wise traces the creative process in the making of the work, combining links to the VR artwork, video footage of the live performance, and images from the project. This is combined with her observations and analysis of audience feedback. Through this analysis, the writing assesses the affordances of using VR technologies within immersive performance practices, addressing some of the technological, practical, choreographic and conceptual concerns. She concludes that immersive media has huge potential for offering audiences new embodied encounters that can shift perspectives and produce transformational, intimate, emotive and unsettling experiences.
So then, immersive media sews people into a world in a dynamic way, be it through the use of sound, performance, installation, film, or even marketing content. It is also a wrap-around sensory experience that is capable of changing mood and state of mind, and is constructed with the aid of knowledge bases from disciplines as wide-ranging as computing, art, media, performance, psychology, and beyond. In terms of emerging research practices, immersive media opens the door to not only a fascinatingly interdisciplinary set of approaches, but also encourages us to rethink how the implementation of such media within our research projects might elicit a similarly fascinating range of interdisciplinary outcomes and findings.
This special issue is sponsored by the Bristol+Bath Creative R+D Partnership, a £6.8 million AHRC-funded collaboration between the University of the West of England, Bath Spa University, University of Bath, University of Bristol and Watershed, Bristol’s digital creativity centre. The partnership aims to connect the worlds of university research and creative business to collectively imagine and develop the future of the creative industries.
Department for Digital, Culture, Media and Sport (2018) ‘Culture is Digital’ (March). Available at: https://www.gov.uk/government/publications/culture-is-digital.
Speakman, Duncan (2020) ‘No Vantage Point’, Immerse (May 1). Available at: https://immerse.news/no-vantage-pointb6a4da415584.