The Gustave Baumann Marionettes- Surfacing a Rare Collection through Interaction

Paper
Miriam Langer, USA , Jonathan Lee, USA, Daniela De Angeli, UK

The New Mexico Museum of Art in Santa Fe owns a collection of over seventy marionettes carved by artist Gustave Baumann during the 1920s and 30s. Their age and fragility means that they are rarely displayed and never used as designed. Figuring out a way to surface this collection and to encourage an interactive experience has long been a goal for the collection’s curators.

The museum approached the development team from Media Arts & Technology at New Mexico Highlands University. After visiting the marionettes in storage we got to know a few of them (plus their caretakers). We decided that the Kinect (and now the Leap Motion as well) would help us design the best interactive experience.

First, we had to model the collection. The marionettes were scanned using photogrammetry and turned into 3D models using a stitching application. The scans were cleaned up in Blender and programmed in Unity 3D to model the articulation of the actual marionettes.

Once the models were functional and able to move as the original marionettes, the Microsoft Kinect was used to detect the user’s movements to match the marionette. The virtual marionette is limited by its designed joint structure, so it can only follow the user to the best of its abilities. This allows the user to engage as the marionette, while learning about the complexity and limitations of these artworks.

The visitors see themselves embodied as Nambe Nell, Pecos Bill, Gustave Baumann, or Lord L on a projected or screen-based stage set with Baumann’s props and painted backdrops. Up to two users and two marionettes can be on the stage at any time. Users can act independently or perform a puppet show, using their own script or one of Gustave Baumann’s original New Mexico folklore stories.

The Baumann Marionettes Interactive will be included in the exhibit “Gustave Baumann: A Santa Fe Legend”, opening at  the Las Cruces Museum of Art in February 2014. Baumann was born in Germany and emigrated to the United States at the age of ten. While he is known as an American artist, primarily as a printmaker, his marionette skills were deeply rooted in his German heritage.

Our presentation includes a walkthrough of the process of creating the 3D models from rare objects, the clean-up, skeletal rigging, and programming. We will then present the final project in its current iteration using the Kinect in the Unity environment. We will show a sneak preview of phase two, involving the Leap Motion controller, as a more fine-tuned natural user interface.

The code for this project is open, and the documentation is designed to be shared with other museums with an interest in surfacing a fragile collection of this nature.