Emergent Storytelling: Interactive Transmedia Installation for Digital Cultural Heritage

Paper
Vibeke Sorensen, Singapore

Keywords: immersive, multimodal, installation, digital multimedia

1. Introduction

A story may be about an event that engaged people in their entirety – so to tell it with words only would be an approximation. Each person who experienced the event, and telling the story of it, will tell it slightly differently because they were at a different place and time when it took place and had a different point of view. Many things would be the same, but some things would be different. And each moment when the story is being remembered, the current condition of the storyteller and the listener would affect the emphasis, perception, and the selection of related memories that might be invoked. There may even be some embellishment.

The number of variations in the story depends on the number of elements and the number of people involved in telling and hearing it. The more elements there are means that more and more variations are possible. At a certain point there are so many elements that the possible variations are endless. This is an emergent system.

With a digital real-time system and a large database, we can emulate this process and even try to predict future trends. With sensors and physical computing, we can connect to the real-world itself. There is a network of things, and soon, living things. How we connect with people, animals, and plants through technology, and for what purpose, are ethical questions that artists, technologists, and users will increasingly face. With a sense of responsibility for the wellbeing of others, and with compassion, respect, and sensitivity, it is also an opportunity for imagination, creativity and invention, and for critical reflection.

The author’s approach has been to work with language, music, moving images, and sensors, to develop a new kind of narrative that works with database memory fragments, including texts, and live gesture. She uses a computer with human interaction to re-member a story into a new and dynamic whole that is constantly changing. She develops new systems that are positioned in physical environments and work with objects and concepts already familiar to and with meaning in global traditional cultures, such as houses, tables, chairs, bowls, boxes, folding screens, and musical instruments. Calling the familiar objects “implicit systems,” she employs a wide range of media such as texts, images, sounds and movies, and uses memory triggers such as physical movement, touch, and smell (or aroma) to link and navigate them and their higher order associations, what she calls “transmedia.” The resulting works engage and immerse the entire body, and groups of people, as much as possible inside a story and provide cooperative rather than competitive strategies, so that the story will be more alive, and never exactly the same twice. To do this, she makes use of approaches, concepts and models from neuroscience, psychology, psychiatry, visual art, cinema, mathematics, computer science, music, and language. This paper will review some of the projects.

Mindshipmind and Art-Science

For the installation entitled Mindshipmind, the author gathered texts from 30 artists and scientists at Mindship, a conference on art and science directed by the Danish writer and philosopher of science Tor Nørrentranders, focused on the theme of Order, Complexity and Beauty, in Copenhagen, Denmark in 1996. Sørensen asked that they write position statements about their work and provide them together with images, movies, sounds, and any other data they wished to submit. She deconstructed their texts linguistically and grammatically, into files containing dependent clauses, independent clauses, prepositions, subjects, objects, and so forth, and then developed a computer algorithm incorporating weighted randomness and emergence to generate new texts. The result was a kind of on-going stream of consciousness, a never ending sentence, or infinite book. She collaborated with Austrian composer Karlheinz Essl to realize it, and presented it as a four computer multimedia installation along with found objects from the installation site. The following year, they worked with Florian Kramer to transform it into a web-based work that in addition to having an on-going narrative, also generated new layouts, produced or added thematically related links, new music, voices, and animation on the fly. The result was a collective mind of the Mindship, or Mindshipmind.

Open Books and Global Visual Music

Another project was based on Sørensen’s journals from the time she was an architecture student in Denmark in the early 1970s, when she was also a jazz violinist. She went on a study trip to Morocco, and combining jazz improvisation with journal entries to create “open books” – people she met could draw or paint or write in the books. This became part of her personal memory, where their narratives and memories were richly recorded and became a stronger part of her narratives and memories. In order to perform spaces and build a dynamic experience, she added architecture to the mix, putting the different artforms together through a common technology as an updated Gesamtkunstwerk. (Gesamtkunstwerk is a German term coined in 1827 by K. F. E Trandorff that literally means “the total work of art”. Originally a Romantic ideal associated with Opera, artists and theorists conceived it as a synthesis of all artforms where the synergy between them would yield a much larger, complete new form that would make the individual artforms disappear.) She then began working with a spectrum of media including documentary and experimental film, photography, animation, and electronics, with the goal of creating a global multimodal artform that would synthesize them, that would span the next several decades. Her degrees in mathematics and English Literature, her work with historic preservation in architecture, and her skills in music, shaped her vision and served as tools for realizing it.

Calling it Global Visual Music (GVM), her idea was to incorporate music and visual images, as in abstract musical films produced by artists such as Oskar Fischinger, but it would go beyond the flat screen to incorporate multiple dimensions and more of the senses. It would integrate tradition and memory, multiculturalism, and contemporary global experience, using polyrhythmic traditional musical forms, and jazz with its characteristics of structured improvisation as models. For GVM to work, it would need a global network of satellites and computing so that all the elements could be put into digital bits, which meant inventing techniques to create and transmit 2D and 3D models of architectural spaces, send live images and text, include sound and live music in an open form for participation and improvisation. This was 1972, so computers were not easily available on the market, nor was authoring software. She began by developing techniques using electronics and synthesizers for sound and image, wired networks, and early digital systems. She used content originating from her own personal and cultural memories and artifacts and those of people she knew or met to drive the technological development. Over the next 25 years, she worked with the various component media that would ultimately be joined together, and wrote her own programming and worked closely with computer scientists and engineers to develop new computer graphics and multimedia systems for this purpose, much of which was put into the public domain or disseminated freely.

Morocco Journal 1972/1995

More than 20 years later, in 1995, when reflecting on how she told stories about her memories of Morocco, which included a written collection of short stories, Sørensen noticed that she had ordered their telling according to particular details in the story at the time she decided to focus on at the time of the storytelling, and this guided subsequent associations and connections to related stories. Chronology became of secondary importance thus allowing the order to be non-linear, it freed the stories from stasis, keeping them fresh as new paths and associations were discovered each time and resulted in new insights. All of the stories were told, but not in the same order. Their telling was more interesting for the storyteller and the reader/listener. Inspired by this, she decided to use hyperlinked media to emulate the way memory and the mind actually work to associate, correlate, logically explore, and reinforce elements, changing the choice of elements and thus sequence of connections each time the story was told. The chronology was not a problem as the stories had many common elements that could be associated, and all of the stories were subsets of larger stories and categories. The author used both Sigmund Freud’s theories of free association and Carl Jung’s ideas of direct association, developing a system of diagrams depicting connections and branchings of ideas, texts, images, sounds, movies, and other elements, as a kind of roadmap for the mind, and a program for creating interactive narratives that would be equally flexible.

Sørensen compared her memories as written from the distance of 23 years in 1995 with the original recorded memories in journals and in photographs from 1972. When she found a connection or association, she recorded and included it as a media element and link in the digital version. This allowed her to tell stories as a cluster of smaller stories that could be ordered according to key associations, similar to metadata. The resulting structure of the links was similar to gears or spokes on three dimensional wheels or better, a dynamic conch shell where stories radiated and referred back to other stories as connections were made between them. The piece was realized using Macromedia Director, a multimedia authoring program available at the time. This allowed people to read the stories and view the images in whatever order they wished, and follow their curiosity and associations. When installed at the University of California, Riverside in 1997, the audience reported enjoying the stories and discovering new associations.

This approach had its limitations. The database was large enough to engage a reader or audience for 15-30 minutes, appropriate for a gallery but too short when compared to the time it takes to read a book or view a feature film. The project’s content needed to be more varied, larger, contain more data and media, and be networked to allow for more meaningful input and interaction from the audience. It also needed to be more immersive. Sørensen felt that by reducing interaction to mouse clicks there was a distancing of the reader that limited rather than enhanced engagement. Interaction with computers, she felt, should be as rich and intuitive as interaction with the real world, or the setting in which the memories and stories originated or took place.

With funding from Intel, Sørensen worked on the development of a new visual music system, the Global Visual Music Project (GVM), in collaboration with Miller S. Puckette, Rand Steiger, and Mark Danks of the University of California, San Diego, called Pure Data/GEM. Pure Data was originally written by Puckette as an open-source programming language for computer music, and for GVM the idea was to extend it to 2 and 3 D graphics, text, animation, video, networking, the web, and physical computing to interface sensors and wireless devices for use in installations. They initially developed it for networked visual-music performance with improvising musicians at multiple locations worldwide.

Morocco Memory II

Sørensen decided to develop further the PureData/GEM software to solve the problems that arose in the first Morocco piece. She used her ‘open book’ idea, and collected many more memory fragments or lexia and short stories from other people, expanding the work to engage all of the senses. (Lexia is a term coined by Dr. Marsha Kinder of the University of Southern California to describe text elements that would be used in database narratives such as Morocco Memory II.) She thus incorporated smell, taste, touch, light on the skin, movement of the body in space, visual images, sound and text. Given the powerful effect of aroma on memory, the guiding metaphor was “the mixing of smells is like the mixing of memories.”

For Interactive Frictions, an exhibit held at the Fisher Gallery at the University of Southern California in 1999, Sørensen designed and built a small house of wood and satin, where the entire back wall from floor to ceiling was a large screen for rear-projection. Images projected onto it varied from textures that reinforced the screen’s wall-ness, to street scenes that were carefully composed and proportioned so that the wall disappeared, becoming a large opening or doorway into another location such as a street market scene in Marrakesh. There was spatial sound, and lights that changed color and brightness in response to the real-time interaction. Six Moroccan wooden boxes with Moroccan spices in them were used as implicit interaction devices. People could open and close and carry them around, and smell and even taste the spices. The boxes had miniature custom circuits embedded inside them, fitted with wireless transmitters to send interaction data to remote computers holding the multimedia data, stories and memory fragments, for combining in real-time. There were no visible wires or parts, so as to background the technology and foreground the cultural content. The computers were placed behind a barrier outside the house for this reason.

Upon entering the installation for the first time, one saw a blank screen, and in front of it were two stools, and the table with the six boxes on it. A light at placed at the apex of the ceiling warmly lit them. A wireless receiver/transmitter was placed on the underside of the table so that when people entered, sat down on the stools and opened a box, everything changed. The lights dimmed, beautiful aromas emerged, music played, and the screen came alive with colors, photographs, texts, or movies. The number of boxes that opened and closed, and the order of openings and closings, directed the navigation of the elements. Lateral interaction strategies accommodated up to six users. In one case, three people shared the two benches, and three others sat on pillows on the carpeted floor. All of them held a box and interacted. People cooperated rather than competed to alter the resulting narratives. Short stories were told in their entirety and all of the stories were related and had multiple connections. Memories and stories came from personal and cultural memory sources, including private archives and popular media. Dr. Marsha Kinder, a colleague of Sørensen’s at the University of Southern California where both were professors, contributed a number of personal photographs and stories of her own experiences in Morocco. When all six boxes were open, theoretical texts about memory and culture appeared on the screen.

Using familiar objects such as boxes, fabric, and spices, together with smell and touch, people felt comfortable and knew intuitively how to interact. They did not need to read instructions. And when asked for feedback on whether they felt their cultures were respected, audience members from Morocco and related cultures commented that they felt respected, welcomed, and included, and were not offended by this piece. In fact people stayed for hours, were enthusiastic, and some even asked if they could stay overnight and sleep in there, as apparently the fluctuating waves of music, light, voices, and aromas intrigued and comforted them. One audience member told Sørensen during the opening that she had made a “sanctuary.”

Sanctuary

Sørensen’s next large interactive architectural installation, produced over the next five year period was inspired by this comment. Entitled “Sanctuary” its theme was cultural and natural sanctuaries around the world, working with the idea of meditation on nature and the universe. It included documentary and animation, photography and sound, and was activated by physical computing, similar to but extending beyond Morocco Memory II. Approximately ten thousand media recordings made in natural and spiritual sanctuaries in Bolivia, Brazil (the Amazon region), Canada, China, Denmark, France, Indonesia (Bali), Italy, Japan, the U. S., and other countries were incorporated into this project. This included recordings of ancient Incan rituals at Tiwanaku, Bolivia on the Winter Solstice (June 21) where the author received the name Urina from an Incan Shaman.

Sanctuary premiered at Gallery One-One-One at the University of Manitoba, Canada in 2005. The design consisted of four large projection surfaces (three when installed in Canada due to ceiling height restrictions), computer controlled lights, and spatial sound. The wooden structure was a small house inspired by Japanese temples in the countryside. This piece was intended in part to be a critical comment on, and an alternative to, the Virtual Reality CAVE that isolates people from nature and the physical world. Japanese temples on the other hand are situated inside of nature so that people can be in direct contact with it, while contemplating it. A roof may protect people from rain and snow, but the sides are often left open to allow air, light, and sound to flow through, and in this way connect interior and exterior, and people with nature. It is meant to facilitate contemplation on nature and the universe. Therefore, in Sanctuary, walls of the simple house were made of screens, also an homage to Japanese houses, but they were rotated open so that people could pass between them and engage other living things, including living plants. Images from sanctuaries around the world were projected onto the screens, and as in Morocco Memory, they transitioned from textures and thus between wall-ness and space-ness, emptiness and fullness.

The wooden frames holding the screens were made of actual tree trunks, and branches clustered together at the top served as a roof. The screens were bound to the frames with a traditional technique used by First Nations Peoples of Canada for attaching hides to frames for tents and dwellings. This was suggested by Dakotah/Anishinaabe/Metis artist and scholar, Leah Fontaine (B.A. Theatre, B.F.A. Drawing, M.A. Native Studies), who worked in Gallery 1-1-1 in the School of Art, University of Manitoba, Winnipeg. She contributed to the design and content in Sanctuary as it related to her culture.

Sanctuary employed an emergent system using the same approach for layering and associating the digital memory fragments as initially developed for Morocco Memory II, but went further by using environmental data, proximity sensors, and living plants to alter and navigate the system. The plants were incorporated so as to remind people that they exist within the local and global natural ecology and are dependent on plants, and that their actions affect them. So when human beings gently touched the plants in this piece, the plants sensed it and that altered the state of the system, and thus also the selection, processing and display of music and sound, images, texts, and movies. As all of the content was based on nature, the living plants and narrative/textual content reinforced each other in an expansive meditative relationship. This also served as a contrast to a computer game or entirely virtual environment. Instead, it was intended to be the opposite, and literally ‘bring people back to nature’ through engagement of actual, physical nature as part of the interactive digital media system. This kind of mixed media approach is today referred to as Augmented or Mixed Reality.

In order to respect the living plants, it was important not to encumber or damage them, and instead listen to them by detecting the actual signals emanating from them. Therefore, the solution was to use capacitive sensor technology attached to a simple wire placed in the dirt near the roots. Sørensen discovered through her research that touching leaves naturally caused a signal to be sent through the leaf’s cells and stem to the roots in the dirt. (She presented her results in the Department of Medical Biochemistry at the Federal University of Rio de Janeiro in 2005). The signal was detected by a custom circuit containing a capacitance sensor that was connected to the wire in the dirt, and that signal was sent on to a computer running Pd/GEM via a wireless transmitter.

A metal bowl was placed on a table located about four meters in front of the small house, as a way to prepare the audience to enter it, like a ritual for entering a temple. A spotlight over the bowl cast soft golden light into it, attracting the audience. The same capacitance technology sensed proximity so that when someone approached it, their presence was detected and this caused the installation to ‘wake up’ from its ‘dreaming’ state. This concept was inspired by Australian Aboriginal Dreamtime. The content of the projections and sounds in the ‘dreaming’ state were free-associative, subjective, lateral, and collaged. When the system awoke, the interaction metaphor changed to direct association, objective, linear, and graphically composed. Thereby the layering of images and sounds followed accordingly.

There was a wooden pathway that connected the table and bowl to the house-temple, and when the audience walked along it and entered, they could sit down on one of two benches next to a lower table, surrounded by screens which were the size of walls. The benches were covered with copper and were active in the system. There were plants on the table in front of the benches, and when touching them, as already described above, the system responded. If people were active, the media materials became more objective, and the more quiet the people became or engaged in meditation, the system became more dreamlike. It was “tuned” to subtle and implicit interaction with the environment, and the resulting experience was therefore also programmed to be subtle, to achieve a sensitive and poetic experience. It was not like a game that required speed or hard impact to make it work ‘better’. In fact, the opposite was true. If a user/audience was too aggressive, the system was less likely to respond.

Music for Sanctuary was composed by Shahrokh Yadegari of the University of California, San Diego (UCSD). Miller Puckette also of UCSD, and Heitor Capuzzo from the Federal University of Minas Gerais (UFMG), Brazil, offered technical assistance. Custom hardware was provided by James Snook of the Neurosciences Institute of La Jolla, California. Sorensen received a Fellowship from the Rockefeller Foundation for this project, and additional support was provided by the University of Southern California (USC) Zumberge Fund for Innovation in Research, Intel Corporation, The Canada Council, and the University of Manitoba, Winnipeg, Canada.

Asia

Sørensen moved to Asia in 2009 to become Professor and Chair of the School of Art, Design and Media at Nanyang Technological University, and where she was inspired by the living history, heritage, and cosmologies of the people of the region.

Illuminations

Illuminations was a large scale, illuminated folding screen that incorporated living plants, three Tibetan Singing Bowls, and embedded systems in Indian ottomans. The folding screen was designed to traverse the length of a 30 meter long, curved gallery space in the School of Art, Design and Media that had walls of mirrors placed on either end. As a result of the reflections, the length of the folding screen was effectively tripled up to 90 meters. The plants, singing bowls, and furniture had embedded systems or sensors, similar to previous works, and interacted with the Pure Data/GEM real-time multimedia system. When the audience or player is singing the bowls, a giant rainbow was produced that reached from one end to the other of the folding screen enveloping the audience. The folding screens put the images into a giant, arched panorama and the motion of images and sounds was likened by the audience to a constantly flowing river or clouds that take many different forms, similar but never repeating. All of the sounds and images used waveforms and spirals, shapes inspired by the Ying Yang symbol, and the Chinese concept of qi, where spirals are considered symbols of the life energy that exist everywhere in the universe, and connect the human being to the cosmos. It was also a combination of Eastern and Western conceptions of space, because it used visual strategies associated with the panorama of the folding screen, as well as perspective used in the three-dimensional computer animation. Also a meditative piece, the audience understood and responded intuitively, given the selection and use of implicit interfaces, music, shapes, and colors.

The plants responded to and interacted with the colored light, moving images, and music as well. However, unlike in Sanctuary where capacitance sensors were used to detect human touch of plants, in Illuminations an oxygen sensor was used to listen to the plants’ daily cycles of photosynthesis and incorporate their continuous stream of natural data directly into the real-time system. All life on Earth, including human beings, is affected by rhythmic changes in light caused by the daily rotation of the Earth on its axis and its yearly orbit around the Sun, and so have similar naturally occurring cycles (Chronobiology). The entire work was conceived as a large polyrhythm of constantly transforming cycles, spirals, and memory elements represented in musical notes, sounds, rhythms, and changing light, colors, textures, and shapes.

Illuminations used thirteen computers, one for each screen and one for music and interaction (including the plants), twelve data projectors, twelve custom screens, three Tibetan singing bowls, custom furniture and circuits, and two walls of mirrors. Sørensen designed the installation as site specific, composed the music and the animation, for which she did the Pure Data/GEM computer programming. Custom hardware and software were made by Marilia Bergamo, Nagarau Thummanapalli, Miller S. Puckette, and Biju Dhanapalan. Nagaraju Thummanapalli also programmed the networking. Custom furniture was made by Fabrizio Galli and Ong Kee Sing. A stereoscopic version was shown at the Beyond 3D Festival at the Zentrum fur Kunst und Medientechnologie (ZKM) in Karlsruhe, Germany in October 2013. Audiences included children, adults, students, faculty, the public, and people from many religious and cultural backgrounds. Both children and adults especially enjoyed playing the Singing Bowls and making the rainbow, and being enveloped in the illuminated folding screen, that was like a river of light.

Borubudur and Prambanan

Borubudur and Prambanan are UNESCO World Heritage sites in central Java, Indonesia. Borubudur is the largest Buddhist temple in the world, built between 750 and 842 AD. Its beautiful design is a representation of the cosmos meant to inspire meditation on nature. Prambanan, the largest Hindu Temple in Indonesia, was built in the 9th century. The author is developing a project that incorporates photography, historical documents, narratives, animation, and traditional Javanese Gamelan music in a real-time interactive system, pushing further the techniques developed for the projects mentioned above for emergent storytelling and cultural heritage preservation. It is intended to be performed or navigated by musicians or the public using traditional musical instruments and artifacts interfaced to the system, presented in a performance venue, museum or gallery, and connected to the web and mobile devices. This will allow audiences to explore and engage the content in playful, intuitive, or didactic ways, thus providing people of different ages and backgrounds various ways to intuitively explore the data, engage and enjoy the stories, and through multiple sensory and conceptual approaches.

The author has also used motion capture for cultural heritage preservation, producing data visualizations, animations, and documentary video projects, including Chinese Fan Dance (2011) and Wushu (2014).

2. Conclusion

The ability to creatively bridge all physical and digital media, tangible and intangible heritage, including historical and contemporary artifacts and narrative, through a common representation or shared form (ie digital encoding) that would not only store but dynamically connect them, was for many years a dream requiring a great deal of technological development. Artists working in installation, performance, and digital art, including Sørensen, participated in this development, making new works while collaborating on the invention of new devices and systems that emphasized the principles of diversity and inclusion. Notions of flexibility, openness, synthesis, and syncretism, with sources ranging from personal and cultural memories of traditional and contemporary global cultures, including storytelling, would drive this development. The processes developed to integrate a wide span of media in real-time, especially emergent systems, would not only help to engage the ideas and forms of tangible and intangible media and memory across cultures, but bring living cultural traditions into creative dialogue with each other, thus directly affecting new technology development. This would help provide greater access to and improved participation in digital culture among ethnic global populations, as well as new possibilities for preserving cultural heritage and simultaneously extend it through expanded forms of communication and creativity.

Acknowledgements

Thanks to Dr. Cristina Venegas, Chair and Associate Professor, Department of Film and Media Studies, University of California, Santa Barbara

Copyright to all photographs in this paper belongs to the author, Vibeke Sørensen

---

Cite as:
V. Sorensen, Emergent Storytelling: Interactive Transmedia Installation for Digital Cultural Heritage. In Museums and the Web 2013, N. Proctor & R. Cherry (eds). Silver Spring, MD: Museums and the Web. Published May 31, 2014. Consulted .
https://mwf2014.museumsandtheweb.com/paper/emergent-storytelling-interactive-transmedia-installation-for-digital-cultural-heritage/

---