Virtual Reality Music pt 2: Interface and Instruments

This post is just mopping up some more interesting research and stimulus used in the development of a VR music interface. The items below range from literal transpositions of composition environments to more experimental concepts in music composition in immersive environments. 

The Pensato Fissure project has achieved wide recognition within dance music communities and internet publications for its immersive VR approach to performing and composing using Ableton live. The project has undergone many developments in interface and input methods. Project formed a Masters project in Design and is still currently active, the authors Showtime github is particularly useful for syncing with Ableton, it effectively reveals all possible Live Objects to another system, which enables rapid application development of music interfaces utilising the underlying power and flexibility of the Ableton Live API.

Of the research conducted in VR music and interaction, topics fall into some definable categories, though not comprehensive account of all research just some good exemplars: 


+ Controlling musical characteristics of pre-existing composition [1]

+ Mixer style control of spatial audio characteristics for pre-recorded sound sources, with optional control of effects [2] 

+ Virtual audio environment, multi-process 3D instruments [4, 5] 

+ Virtual instruments, virtual representations of instruments and synthesis control [3,6,7]

+ Virtual object manipulation with parameterised sound output [8,9]


Many of these implementations offer novel interaction methods coupled with creative feedback and visualisation. Many systems require considerable training and learning to be able to perform with it, though reportedly the basics of user control can be learned quite quickly. This presents a problem for the target audience of the Objects project, where more immediate control and enjoyment is required. Therefore a combination of musical composition control and spatial audio will be explored, using simplified musical interaction that can allow novice users to learn within the experience. Though the control method and interaction metaphors differ considerably from the work presented in [1] and [2].

[1]  Xavier Rodet, Jean-philippe Lambert, Thomas Gaudy, and Florian Gosselin. Study of haptic and visual interaction for sound and music control in the Phase project. International Conference on New Interfaces for Musical Expression, pages 109–114, 2005.

[2]  Wozniewski, Mike, Zack Settel, and J Cooperstock. A spatial interface for audio and music production. Digital Audio Effects (DAFx), pages 18–21, 2006.

[3]  Teemu Maki-patola, Juha Laitinen, Aki Kanerva, and Takala Takala. Experiments with virtual reality instruments. Proceedings of the 2005 international Conference on New Interfaces for Musical Expression, pages 11–16, 2005.

[4]  Leonel Valbom and Aderito Marcos. Wave: Sound and music in an immersive environment. Computers & Graphics, 29(6):871–881, 2005.

[5]  F. Berthaut, M. Desainte-Catherine, and Martin Hachet. DRILE: an immersive environment for hierarchical live-looping. NIME ’10 Proceedings of the 2010 conference on New interfaces for musical expression, (Nime):192–197, 2010. 

[6] S. Gelineck, “Virtual Reality Instruments capable of changing Dimensions in Real-time,” 2005.

[7] A. Mulder, S. Fels, and K. Mase, “Design of Virtual 3D Instruments for Musical Interaction,” Graph. Interface, 1999.

[8] Mulder, A. (1996). Getting a GRIP on alternate controllers: Addressing the variability of gestural expression in musical instrument design. Leonardo music journal, 33-40.

[9] Mulder, A., Fels, S. S., & Mase, K. (1997). Mapping virtual object manipulation to sound variation. IPSJ Sig Notes97(122), 63-68.

Tangible Music Interfaces pt 2: Tables

Tangible User Interfaces (TUIs) combine control and representation within a physical artefact [1]. Interactive interfaces are based on tables, fiducials, tokens, computer vision, custom hardware and other bits. 

The AudioPad is a early musical interface table reportedly the first musical table interface. It uses proximity to control various actions so similar markers have multiple use cases. This is best understood by watching the video. 

Their blurb summarises it like so: 

"Audiopad is a composition and performance instrument for electronic music which tracks the positions of objects on a tabletop surface and converts their motion into music. One can pull sounds from a giant set of samples, juxtapose archived recordings against warm synthetic melodies, cut between drum loops to create new beats, and apply digital processing all at the same time on the same table. Audiopad not only allows for spontaneous reinterpretation of musical compositions, but also creates a visual and tactile dialogue between itself, the performer, and the audience."

Since then they have gone onto establishing a creative technology studio, doing alot of very interesting work 

Though somewhat stating the obvious, tangible interaction in music is refreshing addition to the spectrum of sonic control possibilities. Just the act of not sitting in front of a traditional screen can allow you to sway and react to the music more freely. This then increases your engagement while also focussing yourself on the task of fresh manipulations. A key feature of this abstracted objectified interface is the process of engaged learning. As objects have intrinsic physical affordances, the control space can be explored in a natural trial and error method. While engaged activity is important for learning so too are periods of disengaged reflection. A table allows this as you can simply step back and observe.

One of the most widely known tangible music tables is that of reactable. The Reactable is built upon a tabletop interface, which is controlled by manipulating tangible acrylic pucks on its surface. By putting these pucks on the Reactable’s translucent and luminous round surface, by rotating them and connecting them to each other, performers can combine different elements like synthesizers, effects, sample loops or control elements in order to create a unique and flexible composition [2]. An interesting revelation came from the creators of reactable, namely that  music performance and control can constitute an ideal source of inspiration and test bed for exploring novel ways of interaction, especially in highly complex, multidimensional and continuous interaction spaces such as the ones present when browsing huge multimedia databases. This type of interaction involves searching in complex, hyperpopuladed and multi-dimensional spaces, often looking for unknown and probably not single targets. In these types of “fuzzy interaction” environments, exploration can follow infinite paths, results can hardly be totally right or wrong, and the evaluation metrics can become so unclear, that joyful and creative use may become one of the essential assets [3].

Key design criteria are essential for not turning these interfaces into pretty messes. Sheridan et al offer the following advice in the design of performative tangible interaction [4], systems should be:

1 Intuitive – allow people to quickly grasp an understanding of the basic elements of the interaction, rather than being aimed at expert performers.
2 Unobtrusive – allow the public to carry on their normal activities if they choose to.
3 Enticing – encourage spontaneous interaction by passers-by without any, or very little, instruction.
4 Portable – are lightweight and low power, and easily transported, set up and taken down.

5 Robust – can withstand, and recover from, a range of environmental conditions such as adverse weather and changeable lighting, and different forms of interaction.
6 Flexible – can be dynamically tailored to the environment in which they are deployed.


Key design criteria include: visibility, controllability, robustness and responsiveness. It is theorised that if guidelines are followed and systems are engineered correctly such interfaces allow novice users to quickly learn the performance frame and be able to enjoy creative experiences with the device. Which is nice. Posing design questions in terms of the performance frame (technical skills, wittingness, interpretive abilities) and witting transitions can drive design decisions and maintain a balance between the technology and performance. 

Though working in pure VR for the current project, such design studies and artefacts, inform the nature of how enjoyable ‘fun’ interaction could be shaped. Within a VR environment every array of possibilities are conceivable, but when interacting with music, many of these possibilities must be culled. The ability to create new custom environments based around tangible interaction but free from certain physical restraints (just annoying ones like gravity) allow for creation of seemingly tangible interactions, tangible in terms of interacting with objects rather than a physical interaction. Though a major problem still exists, namely that the physical link presented in tangible interaction creates a sensory flow of information that can guide decisions for the user. Without a haptic feedback channel, will VR ‘tangible’ music interfaces just fall down? I hope not.

References

[1] B. Ullmer and H. Ishii, “Emerging frameworks for tangible user interfaces,” IBM Syst. J., vol. 39, no. 3.4, pp. 915–931, 2000.

[2] S. Jordà, G. Geiger, M. Alonso, and M. Kaltenbrunner, “The reacTable,” Proc. 1st Int. Conf. Tangible Embed. Interact. – TEI ’07, p. 139, 2007.

[3] S. Jordà and G. Geiger, “The reacTable: exploring the synergy between live music performance and tabletop tangible interfaces,” … Embed. Interact., 2007.

[4] J. Sheridan and N. Bryan-Kinns, “Designing for Performative Tangible Interaction,” vol. 1, pp. 288–308, 2008.

Deciphering Space in Virtual Reality

Space, and its perception, is fundamental to navigation through the world. Whether through sight, sound or touch we build mental images of how we relate to the world around us. So, to better understand the possibilities and pitfalls of design for virtual reality interfaces a thorough understanding of spatial cognition is advised. The following article is a literature review, many sources have been copied as is so beware of plagiarism if you use these words. Plus why rephrase something that is already very clear?

William Gibson Visions of future worlds

In Neuromancer, Gibson envisioned highly complex structures like a digital hallucination, overwhelming and alienating. Gibson coined certain phrases that describe certain interactions with game systems, how a space is ‘consensually hallucinated’ in a ‘non-space of the mind’.  Given user centered developments in HCI and transparent natural user interfaces (NUI), graphic representations now would conceal this Gibson’s complexity and to visualise the underling systems no standard geometry and dimensional space could accurately grasp this vision. Why would we want to see everything? As  Aarseth (2007) describes in Allegories of Space, Gibson’s vision is highly perceptive, bleak and ironic but this irony is missed by many futurists that it inspired. Additionally in this essay, Aarseth laid out how structurally different, games are as a new medium. In this capacity they must engage a player through new structures of interaction, by breaking with familiar forms of representation. But to what end does this enhance experience, and on a subtler point what should be the goal of these new mediums? How can they address shortcomings in previous experiences, and the consequences of their success; such as addiction and excessive emphasis on violent interactions.  Aarseth goes on to define spatiality in games: 

"The defining element in computer games is spatiality. Computer games are essentially concerned with spatial representation and negotiation, and therefore the classification of a computer game can be based on how it represents or, perhaps, implements space. More than time (which in most games can be stopped), more than actions, events and goals (which are tediously similar from game to game) and unquestionably more than characterization (which is usually nonexistent), games celebrate and explore spatial representation as a central motif and raison d’être. In terms of playability, themes, tasks, subgenres and dramatic structure, nothing much has changed over the last two decades. The innovation takes place in spatial representation, and the genre’s more slowly evolving complexity in the other areas (such as physical simulation) can be seen as a result of the increasing complexity of the spatial representation." 

– Aarseth, E. (2007). Allegories of space.   Space Time Play , 44-47.

Together this proposes how games can be used as mediums of experience in themselves, what knowledge can they convey? This then poses the question of how we define space, beyond a physical abstraction of geometry. What makes a social space? How do we generate regions of space for different purposes. How do computer mediated spaces differ?

“Rilke wrote: ‘These trees are magnificent, but even more magnificent is the sublime and moving space between them, as though with their growth it too increased.”

– Bachelard,   Poetics of Space,   Beaccon, 1958, p. 201

In the Poetics of Space Bachelard is, in part, inviting designers to re-consider how users’ experience by encouraging reference to formative, intimate spaces rather than formal abstractions. The quotation of a quotation above is useful as it prompts us to consider game spaces on two fronts – poetically (‘the sublime’: what does this space invoke? ), and kinetically (‘moving space’: reminding us that, above all, we are concerned with dynamics and locomotion ). Both these aspects are useful, as they allow us to consider and assess qualities of virtual-spaces that side-step ‘newer must be better perspectives’.

Ecological Perspective

[space is] neither a mere ‘frame’, after the fashion of the frame of a painting, nor a form or container of a virtually neutral kind, designed simply to receive whatever is poured into it. Space is a social morphology: it is to lived experience what form itself is to the living organism, and just as intimately bound up with function and structure. 

– Lefebvre, The Production of Space

This poetic quote summarises the nature of human relationship to space, a view that was built upon by James J. Gibson. James Gibson’s theory of direct perception states that the environment contains all of the information needed to specify its properties. Hence,perceiving these properties is a matter of detecting the information available in the environment.  This concept is significant precisely because it allows James Gibson to locate the moving, perceiving body at the heart of meaningful perceptual experience; an experience which emerges in the dynamical structures that cross the body and the world.  His theory of environmental affordances shows that perception is more than a means of passively representing the intrinsic physical organization of objects. Perception is inherently active and exploratory, we seek out alterations in the vast flow of information enveloping it. These alterations are detected when the perceiver moves through the environment and probes it with a pair of glancing eyes.  Locomotion opens up new possibilities for the pick-up of information specific to the perceiver’s environment. This information can then be used to guide subsequent movements, as in a perception-action loop (the two being inseparable). As David Morris puts it, “we do not … perceive naked properties of the environment, rather we perceive what the environment affords to our bodies, what we can do with, or in the environment.” The theory of affordances demonstrates that the structures of information are intrinsically meaningful for perceptually-guided action.  Affordances are not subjective valuations superimposed on sensations (as theories of perception typically presuppose). Rather, the affordances of the environment are directly perceived as structural information in the environment. 

– Braun, The Structures of Perception: An ecological perspective, 2008

Of all the terms we use to describe the world we inhabit, [space] is the most abstract, the most empty, the most detached from the realities of life and experience. Biologists say that living organisms inhabit environments   not space, and whatever else they may be, human beings are certainly organisms.

– Tim Ingold, Against Space: Place, Knowledge, Movement

Acoustic Spaces

“perception must be understood as a relationship between environmentally available information and the capacities,sensitivities, and interests of a perceiver.” 

– Eric Clarke, Ways of Listening: An Ecological Approach to the Perception of Musical Meaning

To use ecological perception theory to determine musical meaning, Clarke identifies properties of musical sounds that “afford” certain meanings. Affordance is a term coined by Gibson to describe the purposes or uses of various objects as communicated by stimulus. As an example, a chair affords sitting,taming lions, and knocking down bad guys. Clarke uses this concept to show how listeners can glean meaning from a musical performance. Note that he either stipulates a specific listener and a specific performance, or gives ranges of possible meanings dependent upon the context of the performance and the listener.

Interesting Links not discussed in detail:

Leap Motion Blog post about VR OS

Harrison and Dourish’s Work on Collabritive spaces

Virtual Reality Music pt 1: A wee taster…

In the start of a likely lengthy series of posts, I’ll be presenting some of the internets occurrences in VR music making/experiences. This set of posts will run in parallel with a "Tangible Music Interfaces" set of posts that will cover alternative interaction methods with audio and music applications. For this first one I’m being generous and sharing two things that caught my attention lately.

Squarepushers 360 music video for Stor Eiglass, is a journey through a bubblegum world of bizarre 8 bit escapism. It highlights the possible dangers of VR addication and is also hilarious/scary. Reminds me abit of Osamu Sato’s early psychadelic computer game Eastern Mind: The Lost Souls of Tong-Nou.

Next up is a bit of research form the NIME community on a hierarchical live looping VR music app, DrilleThis technique consists in creating musical trees whose nodes are composed of sound effects applied to a musical content. The system presented requires considerable training and learning to be able to perform with it though the basics of user control can be learned quite quickly. The first video shows a good overview of the system while the second video explores a more structured performance. Further research indicates how metaphors of interaction are utilised in the creation of user control and system design. 

Sound Ideas

Virtual reality and interactive media applications are a great testing ground for novel audio applications. Bizarre control dynamics and immersive feedback stimulus make for an engaging method of challenge and reward. But a question of what areas of sound are to be manipulated remains open, normal synthesis algorithms are build on the morphological relationships related to their early construction… they were made for keyboards and circuits. With natural user interfaces it is possible to explore more physical relationships to sound control. Along this line I found some nice visuals and sounds of physical acoustic relationships. Whether these can be implemented in a controllable and rewarding fashion is another question…

Tangible Music Interface pt 1: Cubes!

Nice cubes eh, menger

I really like cubes, no really. This first post is just a statement really, the future posts in this title thread will cover a variety of tangible musical interfaces, such as Reactable. Though working in VR for the current project the user research and design considerations in tangible interfaces share alot of crossovers. In this series of posts I will hope to determine how these areas can be mutually informed. But for now I want to look at some cubes.

Cube Illusions

First up is a Rubiks cube puzzle sequencer that uses computer vision based markers as a means of controlling the underlying musical structures. I really like this playful and mysterious approach to discovering musical elements. Check out his other work and talks, really interesting chap.

This is pretty fly

Some WebGL graphics stuff

AUTOMATParticularly like the exposing of the mesh underneath in this piece,  use your gyroscope or mouse to interact with it. Giving me ideas for objects to spontaneously appear in VR experience, representing a metaverse of acoustic objects, as the become used in a context: making a phone call, streaming a tune, playing with something.

Waves Wave simulator that made me think again about using basic acoustic formula for control of 3D graphics for reactive audio. Fished out my Kuttruff Room Acoustics book and subsequently melted my brain.

JavaScript: If you fancy playing with this sort of stuff go to PlayGnd (based on three.js), or just look through the archives with slinky code overlay. If thats abit softcore for you head over to WebGL Fundamentals and get all detailed. Also a fully speced WebGL and js library is that of Babylon.js that supports super easy Oculus Rift integration, check out their playground for examples and browser based IDE.