Virtual Reality Music pt 2: Interface and Instruments

This post is just mopping up some more interesting research and stimulus used in the development of a VR music interface. The items below range from literal transpositions of composition environments to more experimental concepts in music composition in immersive environments. 

The Pensato Fissure project has achieved wide recognition within dance music communities and internet publications for its immersive VR approach to performing and composing using Ableton live. The project has undergone many developments in interface and input methods. Project formed a Masters project in Design and is still currently active, the authors Showtime github is particularly useful for syncing with Ableton, it effectively reveals all possible Live Objects to another system, which enables rapid application development of music interfaces utilising the underlying power and flexibility of the Ableton Live API.

Of the research conducted in VR music and interaction, topics fall into some definable categories, though not comprehensive account of all research just some good exemplars: 

+ Controlling musical characteristics of pre-existing composition [1]

+ Mixer style control of spatial audio characteristics for pre-recorded sound sources, with optional control of effects [2] 

+ Virtual audio environment, multi-process 3D instruments [4, 5] 

+ Virtual instruments, virtual representations of instruments and synthesis control [3,6,7]

+ Virtual object manipulation with parameterised sound output [8,9]

Many of these implementations offer novel interaction methods coupled with creative feedback and visualisation. Many systems require considerable training and learning to be able to perform with it, though reportedly the basics of user control can be learned quite quickly. This presents a problem for the target audience of the Objects project, where more immediate control and enjoyment is required. Therefore a combination of musical composition control and spatial audio will be explored, using simplified musical interaction that can allow novice users to learn within the experience. Though the control method and interaction metaphors differ considerably from the work presented in [1] and [2].

[1]  Xavier Rodet, Jean-philippe Lambert, Thomas Gaudy, and Florian Gosselin. Study of haptic and visual interaction for sound and music control in the Phase project. International Conference on New Interfaces for Musical Expression, pages 109–114, 2005.

[2]  Wozniewski, Mike, Zack Settel, and J Cooperstock. A spatial interface for audio and music production. Digital Audio Effects (DAFx), pages 18–21, 2006.

[3]  Teemu Maki-patola, Juha Laitinen, Aki Kanerva, and Takala Takala. Experiments with virtual reality instruments. Proceedings of the 2005 international Conference on New Interfaces for Musical Expression, pages 11–16, 2005.

[4]  Leonel Valbom and Aderito Marcos. Wave: Sound and music in an immersive environment. Computers & Graphics, 29(6):871–881, 2005.

[5]  F. Berthaut, M. Desainte-Catherine, and Martin Hachet. DRILE: an immersive environment for hierarchical live-looping. NIME ’10 Proceedings of the 2010 conference on New interfaces for musical expression, (Nime):192–197, 2010. 

[6] S. Gelineck, “Virtual Reality Instruments capable of changing Dimensions in Real-time,” 2005.

[7] A. Mulder, S. Fels, and K. Mase, “Design of Virtual 3D Instruments for Musical Interaction,” Graph. Interface, 1999.

[8] Mulder, A. (1996). Getting a GRIP on alternate controllers: Addressing the variability of gestural expression in musical instrument design. Leonardo music journal, 33-40.

[9] Mulder, A., Fels, S. S., & Mase, K. (1997). Mapping virtual object manipulation to sound variation. IPSJ Sig Notes97(122), 63-68.

Tangible Music Interfaces pt 2: Tables

Tangible User Interfaces (TUIs) combine control and representation within a physical artefact [1]. Interactive interfaces are based on tables, fiducials, tokens, computer vision, custom hardware and other bits. 

The AudioPad is a early musical interface table reportedly the first musical table interface. It uses proximity to control various actions so similar markers have multiple use cases. This is best understood by watching the video. 

Their blurb summarises it like so: 

"Audiopad is a composition and performance instrument for electronic music which tracks the positions of objects on a tabletop surface and converts their motion into music. One can pull sounds from a giant set of samples, juxtapose archived recordings against warm synthetic melodies, cut between drum loops to create new beats, and apply digital processing all at the same time on the same table. Audiopad not only allows for spontaneous reinterpretation of musical compositions, but also creates a visual and tactile dialogue between itself, the performer, and the audience."

Since then they have gone onto establishing a creative technology studio, doing alot of very interesting work 

Though somewhat stating the obvious, tangible interaction in music is refreshing addition to the spectrum of sonic control possibilities. Just the act of not sitting in front of a traditional screen can allow you to sway and react to the music more freely. This then increases your engagement while also focussing yourself on the task of fresh manipulations. A key feature of this abstracted objectified interface is the process of engaged learning. As objects have intrinsic physical affordances, the control space can be explored in a natural trial and error method. While engaged activity is important for learning so too are periods of disengaged reflection. A table allows this as you can simply step back and observe.

One of the most widely known tangible music tables is that of reactable. The Reactable is built upon a tabletop interface, which is controlled by manipulating tangible acrylic pucks on its surface. By putting these pucks on the Reactable’s translucent and luminous round surface, by rotating them and connecting them to each other, performers can combine different elements like synthesizers, effects, sample loops or control elements in order to create a unique and flexible composition [2]. An interesting revelation came from the creators of reactable, namely that  music performance and control can constitute an ideal source of inspiration and test bed for exploring novel ways of interaction, especially in highly complex, multidimensional and continuous interaction spaces such as the ones present when browsing huge multimedia databases. This type of interaction involves searching in complex, hyperpopuladed and multi-dimensional spaces, often looking for unknown and probably not single targets. In these types of “fuzzy interaction” environments, exploration can follow infinite paths, results can hardly be totally right or wrong, and the evaluation metrics can become so unclear, that joyful and creative use may become one of the essential assets [3].

Key design criteria are essential for not turning these interfaces into pretty messes. Sheridan et al offer the following advice in the design of performative tangible interaction [4], systems should be:

1 Intuitive – allow people to quickly grasp an understanding of the basic elements of the interaction, rather than being aimed at expert performers.
2 Unobtrusive – allow the public to carry on their normal activities if they choose to.
3 Enticing – encourage spontaneous interaction by passers-by without any, or very little, instruction.
4 Portable – are lightweight and low power, and easily transported, set up and taken down.

5 Robust – can withstand, and recover from, a range of environmental conditions such as adverse weather and changeable lighting, and different forms of interaction.
6 Flexible – can be dynamically tailored to the environment in which they are deployed.

Key design criteria include: visibility, controllability, robustness and responsiveness. It is theorised that if guidelines are followed and systems are engineered correctly such interfaces allow novice users to quickly learn the performance frame and be able to enjoy creative experiences with the device. Which is nice. Posing design questions in terms of the performance frame (technical skills, wittingness, interpretive abilities) and witting transitions can drive design decisions and maintain a balance between the technology and performance. 

Though working in pure VR for the current project, such design studies and artefacts, inform the nature of how enjoyable ‘fun’ interaction could be shaped. Within a VR environment every array of possibilities are conceivable, but when interacting with music, many of these possibilities must be culled. The ability to create new custom environments based around tangible interaction but free from certain physical restraints (just annoying ones like gravity) allow for creation of seemingly tangible interactions, tangible in terms of interacting with objects rather than a physical interaction. Though a major problem still exists, namely that the physical link presented in tangible interaction creates a sensory flow of information that can guide decisions for the user. Without a haptic feedback channel, will VR ‘tangible’ music interfaces just fall down? I hope not.


[1] B. Ullmer and H. Ishii, “Emerging frameworks for tangible user interfaces,” IBM Syst. J., vol. 39, no. 3.4, pp. 915–931, 2000.

[2] S. Jordà, G. Geiger, M. Alonso, and M. Kaltenbrunner, “The reacTable,” Proc. 1st Int. Conf. Tangible Embed. Interact. – TEI ’07, p. 139, 2007.

[3] S. Jordà and G. Geiger, “The reacTable: exploring the synergy between live music performance and tabletop tangible interfaces,” … Embed. Interact., 2007.

[4] J. Sheridan and N. Bryan-Kinns, “Designing for Performative Tangible Interaction,” vol. 1, pp. 288–308, 2008.

Tangible Music Interface pt 1: Cubes!

Nice cubes eh, menger

I really like cubes, no really. This first post is just a statement really, the future posts in this title thread will cover a variety of tangible musical interfaces, such as Reactable. Though working in VR for the current project the user research and design considerations in tangible interfaces share alot of crossovers. In this series of posts I will hope to determine how these areas can be mutually informed. But for now I want to look at some cubes.

Cube Illusions

First up is a Rubiks cube puzzle sequencer that uses computer vision based markers as a means of controlling the underlying musical structures. I really like this playful and mysterious approach to discovering musical elements. Check out his other work and talks, really interesting chap.

Roli Seaboard Grand

Just in case you haven’t seen it, its awesome. Part of the new breed of expressive musical controllers that allow new dimensions of expression in electronic music. The company based in Dalston, London are pushing forward with beautifully designed products that don’t compromise on key functionality. Go visit their products page for a better look!

The continuous pitch control mechanism allows a performer to introduce musical “systematic deviations”. These deviations are the things missing from standard electronic music controllers unless all kinds of other controllerism is exploited. This simple yet elegant approach can be used to faithfully recreate original acoustic instruments or explore new depths by providing tactile micro control to an array of synthesis parameters. Plus its got squishy keys which is always nice.