Speculative musical interface design

What is the current state of Speculative Design in musical instrument design communities, and what is its value? This article serves as a series of reviews and critiques to approach answering these questions, written in more of a polemic style than I actually would practise myself. The work is the write-up of a talk presented at Queen Mary University of London’s qualitative research methods group. Unfortunately it does not include definitions for speculative, critical or fictional design; as other speakers covered these topics before me. In a future post I will present a brief overview of these concepts, or link to co-presenters writings. For now you can check out Tobias Revell’s great lecture on the topic.  The article covers the following:

Current Practise: what value does fictional, speculative or critical design approaches receive in current digital musical instruments (DMIs)  design communities?

Approaches: what are some current design fiction approaches to DMI design? What is the work being done currently in other fields that could bridge towards more speculative practises in music technology design? 

Codetta: why do we want design fiction in DMIs?

Current practise and its discontents.

Accurately describing or conceptualising current DMI design and lutherie practise is a nebulous endeavour,  see NIME for a taster, and as such will not be attempted here.

While music practise often revels in abstract and impressionistic realms of the imaginary, crudely, I would posit many of the design practises found in research and arts still take a distinctly concrete approach to conceptualisation, prototyping, and realisation.

And yes, what use is the unheard sound from an imaginary gesture on machine that has not been built yet?

Image sourced from 3dmin.orgEven in more adventurous research projects that challenge conservative music educational practise and research, such as Berlin based Development and Dissemination of New Musical Instruments (3DMIN); there is a distinct emphasis on developing entirely useable prototypes or corporeal music experiences. With this scope, really questioning what digital augmentation can offer as a new form of  expression may not be addressed by the bias of requiring formal realisation. This is where speculative approaches can offer different avenues of discourse around how we want technology to augment musical experience as a performer, participant, or listener.

Before moving on, I would like to position that the majority sound and music interface design research is digital crafts. Echoing Dunne and Raby’s account of new media art:

“device art usually focused on aesthetic, communicative, and functional possibilities for new media rather than visions of how life could be, and mainly takes the form of digital craft rather than future speculations.” – Dunne and Raby, Speculative Everything, pg26

Though often adventurous, groundbreaking even, the limit of such research practise is in looking at iterative expansion, focussed around recent technology developments. So, what could creativity and expression be, through speculative eyes? How can we look towards ideas of things yet discovered as expressive? What new bodies or entities do we allow into the  process of expression: AI, big data, mass interaction, computational agents?  

Why critique digital crafts for not looking beyond the boundaries of current possibility? 

With its parental links to the formal HCI community, DMI design has certain traits or tropes of perspective that have been adopted and reinforced. The development of the “evaluation” theme in current literature, while useful, evaluation metrics and standard setting does bias consumable technology focused results. This does not address what the perspective or philosophical traditions that the chosen developments reinforce. Many design disciplines with extensive histories have embraced of fictional accounts, see the book Speculative Everything by Dunne and Raby for a in depth review. Such accounts nullify market forces from decisions and can remove conventional functionality. This allows space for different kinds of questions to be asked about the role of music, sound and technology for creative expression.

How is fiction used in DMI design?

After conducting a very brief literature synthesis of the NIME community, the role of fiction, speculation, or critical design loosely occurred around three themes:

  1. Urban fiction: use of fictional music or sound based overlays to reality through ubiquitous computing technologies such as locative media.
  2. Fiction as inspiration: use of fictional design or science fiction as inspiration for beyond what is possible now.
  3. Expanding experience: using fiction as a concept for different interaction experiences, how to stimulate new feelings or ways of doing.

Looking across these themes, very few papers or descriptions employ fiction as the goal. It appears as a composite part of the project but not the purpose. While encouraging that it appears, I would ask that its role is expanded.

So, if robust prototypes are not exciting enough, what is?

By means of example, I would like to put forward the work of Choi Ka Fai whose project questions embodied knowledge and possibly new expressive meanings of agency. 

Central to the project is the study of body movement in dance. Using electronic muscle stimulation, you can remotely control another’s actions  or be controlled by previously recorded muscle memories. While entirely based in the now of technology, this work addresses complex issues of our bodies including memory, synchronicity, expressivity, and agency; that can be extended to musical interaction quite easily.

westminster2

In City Symphonies, this project looks at the consequences of very near technological developments when electric cars will change the soundscape of cities, Mark McKeague asks whether a city can become a symphony. This work alters the boundaries of what we classify as music, though it could dismissed as simply as sonification.

“From a street level perspective the motions of traffic combine the sounds to create soundscapes that are unique to the place and time. The roadside becomes a new context for sound – the city is the score.”

photo2_2_8001

Aloïs Yang presents The Star Light Collection, primarily a fictional sonification project, based in a design object, that draws parallels between the interaction of stellar bodies to musical performance.

Approaches

How can we practice at the unknown, talk about and act around that which is yet-to-be-imagined?!

Before moving into HCI methods and approaches to engaging fiction, it is worth describing the Dunne and Raby Method of framing Speculative Design.

Overview: Using design to ask questions rather than providing answers or solving problems.

Process: A highly simplified process for adapting conceptual design into different technology areas is describe as follows (Dunne and Raby, Speculative Everything, pg57)
+ identify a specific area of science research, eg biotechnology and music

+imagine issues that might arise once the research moves from lab to everyday life, eg sonically adaptive skin layer to feel music, then people use music as weapon to `scar’ others

+ embody these issues in a design proposal aimed at sparking debate or discussion. eg looks at sadistic tendency but also imaginative possibility of feeling sounds in new ways.

Framing the process

research_types

The figure above maps a large variety of design practises, critical design occupies the top-left as design-led expert-mindset. Adopting a opportunist research and design led agenda is something that Speculative design is amenable to. By framing the design strategy loosely, but not focussing on problem solving too explicitly we can cherry pick the useful methods required for asking the right questions. We could ask how we open the circle of what is a musical performer or collaborator can be e.g. nature, society, AI? Considering such new collaborators, how do we address the potential design and solution spaces? Each methodology in the figure above would offer slightly different versions in answer to those questions.

From a previous blog post, a series of techniques used in speculative design were outlined, they include: Fictional worlds; Cautionary tales; What if… scenarios; Counterfactual histories; Thought experiments; Reductio ad absurdum; Artefacts from the future; Pre-figurative futures; Small things big issues; Tell worlds rather than tell stories. From my perspective, those in bold are the key areas that musical interfaces design can benefit from. These areas can be explored through critical design, generative techniques, and experiential prototyping. 

Speculative design deals with imagination and fiction. Adapting it to musical interaction could lead to new user experience understandings. Its focus on new extensions of expressive capability using technology (yet to be determined) could result in novel research directions. For instance, using experiential prototyping to explore agency in musical control, we utilise imaginative expression that can infer self-initiated prediction cycles, where we exhibit our expectations, by making visible through play our expressive relationship to the world. Music design fiction is embodied knowledge at play.

screen-shot-2016-11-21-at-22-50-52

Context mapping is a methodology that is commensurate with fictional design as UCD. Context mapping designs for experience, and generative tools could build towards new notions of music interface dynamics, by avoiding preset assumptions based on simple observation or explicit questioning. The methodology acknowledges the environment where HCI takes place, it’s not just the link between user and object. But what is context, everything else? Visser et al (2005) define is as ‘all factors that influence the experience of a product use’. These can be social, physical and internal emotional worlds. To access information around these factors context mapping utilises generative techniques to gain knowledge about what people know, feel and dream. In describing why it is important to focus on such goals in design Visser et al (2005) indicates that we can separate out tacit and latent knowledge, using generative (projective) techniques:

“The use of these projective techniques provides a view to reveal future states of people. These techniques can reveal tacit knowledge and expose latent needs (Sanders, 2001). Tacit knowledge is knowledge that people can act upon, but cannot readily express in words (Polanyi, 1964). Latent needs are those that people are not yet aware of. They are needs that become realised in the future.” – Visser et al (2005)

screen-shot-2016-11-21-at-21-32-32

Continuing with work on generative techniques, research conducted by Kristina Andersen at STEIM explores playfulness and strangeness through experience design, design fiction and participatory workshop methodology; to source inspiration for music technology design. Particularly, in the GiantSteps project she focuses on creating new and disruptive interfaces for creating electronic music. While the goal of the overall project is integrating findings into marketable toolchains, the methodology targets desires, future scenarios [KneesAndersenTkalcic2015], and design fictions [Andersen2014]. Andersen’s approaches design exploration through workshops and artist engagement [AndersenGrote2015] that look to temporarily suspend the history embedded in everyday objects and music instruments, to look at what we want expressivity to be [AndersenGibson2015]. Andersen’s perspective remains open to new relationships of sound and musical context, but involves users in a participatory process via generative techniques such as non-functional prototypes and fictional technological objects. These processes create “machines that might make that sound” act as props and explanatory objects that allow discussion and acting out of complex understanding, prospective modalities, and how expressive abilities are to be embodied in future interface ideas. This process is an attempt to tap into lived experience and engage with an essentially imaginary future object.

Codetta

Why is it interesting: Speculative design can pull new technological developments into imaginary but believable everyday situations so that we can explore possible consequences before they happen. We can make ideas tangible well before the are made  or even possible to be made.

Why are we interested in it for music technology: As the boundaries of what we could consider collaborators in musical process changes we need to engage with alternative aesthetics that inspire us in different ways, by questioning technology, ideology, and technological vs social imagination; we make a space to propose different visions of musical expression calling on more ludic experiences of music rather than notions of perfectly engineered autonomous control. To imagine new modes of expression that challenge traditional fetishism of historical music practise. While useful they immediately limit what we think musical expression is and what its function is to us.

Link bucket

Some RCA Design Interactions graduates that utilise sound and music as medium for speculative design:

+ http://samconran.com

+ http://aloisyang.com/

+ https://markmckeague.com/

Value of playful experience in technology: http://www.ted.com/talks/steven_johnson_how_play_leads_to_great_inventions

References

Froukje Sleeswijk Visser, Pieter Jan Stappers, Remko Van Der Lugt, et al. 2005. Contextmapping: experiences from practice. CoDesign 1, 2: 119–149. http://doi.org/10.1080/15710880500135987

Kristina Andersen and Dan Gibson. 2015. The Instrument as the Source of new in new Music. Research Through Design, March: 25–27. http://doi.org/10.6084/m9.figshare.1327992.

Kristina Andersen.2014. Using Props to Explore Design Futures: Making New Instruments. In CHI workshop on Alternate Endings: Using Fiction to Explore Design Futures.

Kristina Andersen and Florian Grote. 2015. GiantSteps: Semi-Structured Conversations with Musicians. Extended Abstracts of the ACM CHI’15 Conference on Human Factors in Computing Systems 2: 2295–2300. http://doi.org/10.1145/2702613.2732868

Peter Knees, Kristina Andersen, and Marko Tkalcic. 2015. “ I ’ d like it to do the opposite ”: Music-Making Between Recommendation and Obstruction. Proceedings of the 2nd International Workshop on Decision Making and Recommender Systems (DMRS)(CEUR-WS): 1–7.

P. Knees, K. Andersen, S. Jorda, et al. 2015. Giantsteps-progress towards developing intelligent and collaborative interfaces for music production and performance. 2015 IEEE International Conference on Multimedia and Expo Workshops, ICMEW 2015: 4–7. http://doi.org/10.1109/ICMEW.2015.7169826

Lessons learned in VR dev

Following is some pastoral advice gained from doing a project in a new field when the brief is quite open. As with all advice, it depends on your personality!

Define the concept as simply as possible – if communication of the underlying concept isn’t clear, how will the implementation not be?

If its a good idea, follow it – When populating a design space with early concepts, tangents and ideas abound. These may diverge significantly from the original concept you thought of, but in the creative process this is perhaps the nature of ideas. As when balancing all the elements, hidden parameters and approaches appear. These are things you couldn’t perceive in your original constructs and perhaps hold a grain of something truly novel. If you don’t have a strict brief, let go and see where it leads.

Domain knowledge – when coming from a specialist field, such as audio, be wary of the perceived knowledge in users. Your domain knowledge and intellectual predispositions will guide your design space decisions. If you are not careful your ability to communicate to a wider audience will be doomed from the start, due to relying on existing interface metaphors that do not communicate effectively to new users. But if you focus the application to specific domains, these nuances can make it through a design process and be of use to the field more generally.

Concept Development: Possible Futures

A important concept in early development was the use of VR as a lense into imagined worlds. The work of Dunne and Raby on Speculative design was particularly persuasive. Their techniques include:

+ Fictional worlds

+ Cautionary tales

+ What if… scenarios

+ Counterfactual histories

+ Thought experiments

+ Reductio ad absurdum

+ Artefacts from the future

+ Pre-figurative futures

+ Small things big issues

+ Tell worlds rather than tell stories

These aspects are employed as alternative aesthetics that engage us in different ways, questioning technology, ideology, and technological vs social imagination

Metaverse

The Metaverse, as traditionally imagined, would be an unfiltered firehose of humanity. The Metaverse that people are actually trying to build would be, in a meaningful sense, a social network.  Most of its value is bringing people together socially, and letting them communicate with their friends and make new ones.  Putting everyone together into the same chaotic chatroom has less value than intelligently providing spaces where friends can hang out, as web-based social networks have proven. This concept would be engaged with a speculative frame in VR by posing the question of how algorithms would mediate our interaction and communications, with people and machines that are sharing the space.

http://lucidscape.com/ found this data visualisation quite stimulating, the debug view is quite attractive too

Jaap Drupsteen’s music visualisations are fantastic, particularly the one below was of interest at around 3:40 where the concrete structure is morphed into a twitching jittering mass of nodes. This was to be imagined as a transition of aesthetic to be employed in a VR experience to draw the users attention to the concepts of the experience.

Michael Chorost’s book World Wide Mind, increasing emotional communication. 

Virtual Reality Music pt 2: Interface and Instruments

This post is just mopping up some more interesting research and stimulus used in the development of a VR music interface. The items below range from literal transpositions of composition environments to more experimental concepts in music composition in immersive environments. 

The Pensato Fissure project has achieved wide recognition within dance music communities and internet publications for its immersive VR approach to performing and composing using Ableton live. The project has undergone many developments in interface and input methods. Project formed a Masters project in Design and is still currently active, the authors Showtime github is particularly useful for syncing with Ableton, it effectively reveals all possible Live Objects to another system, which enables rapid application development of music interfaces utilising the underlying power and flexibility of the Ableton Live API.

Of the research conducted in VR music and interaction, topics fall into some definable categories, though not comprehensive account of all research just some good exemplars: 


+ Controlling musical characteristics of pre-existing composition [1]

+ Mixer style control of spatial audio characteristics for pre-recorded sound sources, with optional control of effects [2] 

+ Virtual audio environment, multi-process 3D instruments [4, 5] 

+ Virtual instruments, virtual representations of instruments and synthesis control [3,6,7]

+ Virtual object manipulation with parameterised sound output [8,9]


Many of these implementations offer novel interaction methods coupled with creative feedback and visualisation. Many systems require considerable training and learning to be able to perform with it, though reportedly the basics of user control can be learned quite quickly. This presents a problem for the target audience of the Objects project, where more immediate control and enjoyment is required. Therefore a combination of musical composition control and spatial audio will be explored, using simplified musical interaction that can allow novice users to learn within the experience. Though the control method and interaction metaphors differ considerably from the work presented in [1] and [2].

[1]  Xavier Rodet, Jean-philippe Lambert, Thomas Gaudy, and Florian Gosselin. Study of haptic and visual interaction for sound and music control in the Phase project. International Conference on New Interfaces for Musical Expression, pages 109–114, 2005.

[2]  Wozniewski, Mike, Zack Settel, and J Cooperstock. A spatial interface for audio and music production. Digital Audio Effects (DAFx), pages 18–21, 2006.

[3]  Teemu Maki-patola, Juha Laitinen, Aki Kanerva, and Takala Takala. Experiments with virtual reality instruments. Proceedings of the 2005 international Conference on New Interfaces for Musical Expression, pages 11–16, 2005.

[4]  Leonel Valbom and Aderito Marcos. Wave: Sound and music in an immersive environment. Computers & Graphics, 29(6):871–881, 2005.

[5]  F. Berthaut, M. Desainte-Catherine, and Martin Hachet. DRILE: an immersive environment for hierarchical live-looping. NIME ’10 Proceedings of the 2010 conference on New interfaces for musical expression, (Nime):192–197, 2010. 

[6] S. Gelineck, “Virtual Reality Instruments capable of changing Dimensions in Real-time,” 2005.

[7] A. Mulder, S. Fels, and K. Mase, “Design of Virtual 3D Instruments for Musical Interaction,” Graph. Interface, 1999.

[8] Mulder, A. (1996). Getting a GRIP on alternate controllers: Addressing the variability of gestural expression in musical instrument design. Leonardo music journal, 33-40.

[9] Mulder, A., Fels, S. S., & Mase, K. (1997). Mapping virtual object manipulation to sound variation. IPSJ Sig Notes97(122), 63-68.

Tangible Music Interfaces pt 2: Tables

Tangible User Interfaces (TUIs) combine control and representation within a physical artefact [1]. Interactive interfaces are based on tables, fiducials, tokens, computer vision, custom hardware and other bits. 

The AudioPad is a early musical interface table reportedly the first musical table interface. It uses proximity to control various actions so similar markers have multiple use cases. This is best understood by watching the video. 

Their blurb summarises it like so: 

"Audiopad is a composition and performance instrument for electronic music which tracks the positions of objects on a tabletop surface and converts their motion into music. One can pull sounds from a giant set of samples, juxtapose archived recordings against warm synthetic melodies, cut between drum loops to create new beats, and apply digital processing all at the same time on the same table. Audiopad not only allows for spontaneous reinterpretation of musical compositions, but also creates a visual and tactile dialogue between itself, the performer, and the audience."

Since then they have gone onto establishing a creative technology studio, doing alot of very interesting work 

Though somewhat stating the obvious, tangible interaction in music is refreshing addition to the spectrum of sonic control possibilities. Just the act of not sitting in front of a traditional screen can allow you to sway and react to the music more freely. This then increases your engagement while also focussing yourself on the task of fresh manipulations. A key feature of this abstracted objectified interface is the process of engaged learning. As objects have intrinsic physical affordances, the control space can be explored in a natural trial and error method. While engaged activity is important for learning so too are periods of disengaged reflection. A table allows this as you can simply step back and observe.

One of the most widely known tangible music tables is that of reactable. The Reactable is built upon a tabletop interface, which is controlled by manipulating tangible acrylic pucks on its surface. By putting these pucks on the Reactable’s translucent and luminous round surface, by rotating them and connecting them to each other, performers can combine different elements like synthesizers, effects, sample loops or control elements in order to create a unique and flexible composition [2]. An interesting revelation came from the creators of reactable, namely that  music performance and control can constitute an ideal source of inspiration and test bed for exploring novel ways of interaction, especially in highly complex, multidimensional and continuous interaction spaces such as the ones present when browsing huge multimedia databases. This type of interaction involves searching in complex, hyperpopuladed and multi-dimensional spaces, often looking for unknown and probably not single targets. In these types of “fuzzy interaction” environments, exploration can follow infinite paths, results can hardly be totally right or wrong, and the evaluation metrics can become so unclear, that joyful and creative use may become one of the essential assets [3].

Key design criteria are essential for not turning these interfaces into pretty messes. Sheridan et al offer the following advice in the design of performative tangible interaction [4], systems should be:

1 Intuitive – allow people to quickly grasp an understanding of the basic elements of the interaction, rather than being aimed at expert performers.
2 Unobtrusive – allow the public to carry on their normal activities if they choose to.
3 Enticing – encourage spontaneous interaction by passers-by without any, or very little, instruction.
4 Portable – are lightweight and low power, and easily transported, set up and taken down.

5 Robust – can withstand, and recover from, a range of environmental conditions such as adverse weather and changeable lighting, and different forms of interaction.
6 Flexible – can be dynamically tailored to the environment in which they are deployed.


Key design criteria include: visibility, controllability, robustness and responsiveness. It is theorised that if guidelines are followed and systems are engineered correctly such interfaces allow novice users to quickly learn the performance frame and be able to enjoy creative experiences with the device. Which is nice. Posing design questions in terms of the performance frame (technical skills, wittingness, interpretive abilities) and witting transitions can drive design decisions and maintain a balance between the technology and performance. 

Though working in pure VR for the current project, such design studies and artefacts, inform the nature of how enjoyable ‘fun’ interaction could be shaped. Within a VR environment every array of possibilities are conceivable, but when interacting with music, many of these possibilities must be culled. The ability to create new custom environments based around tangible interaction but free from certain physical restraints (just annoying ones like gravity) allow for creation of seemingly tangible interactions, tangible in terms of interacting with objects rather than a physical interaction. Though a major problem still exists, namely that the physical link presented in tangible interaction creates a sensory flow of information that can guide decisions for the user. Without a haptic feedback channel, will VR ‘tangible’ music interfaces just fall down? I hope not.

References

[1] B. Ullmer and H. Ishii, “Emerging frameworks for tangible user interfaces,” IBM Syst. J., vol. 39, no. 3.4, pp. 915–931, 2000.

[2] S. Jordà, G. Geiger, M. Alonso, and M. Kaltenbrunner, “The reacTable,” Proc. 1st Int. Conf. Tangible Embed. Interact. – TEI ’07, p. 139, 2007.

[3] S. Jordà and G. Geiger, “The reacTable: exploring the synergy between live music performance and tabletop tangible interfaces,” … Embed. Interact., 2007.

[4] J. Sheridan and N. Bryan-Kinns, “Designing for Performative Tangible Interaction,” vol. 1, pp. 288–308, 2008.

Deciphering Space in Virtual Reality

Space, and its perception, is fundamental to navigation through the world. Whether through sight, sound or touch we build mental images of how we relate to the world around us. So, to better understand the possibilities and pitfalls of design for virtual reality interfaces a thorough understanding of spatial cognition is advised. The following article is a literature review, many sources have been copied as is so beware of plagiarism if you use these words. Plus why rephrase something that is already very clear?

William Gibson Visions of future worlds

In Neuromancer, Gibson envisioned highly complex structures like a digital hallucination, overwhelming and alienating. Gibson coined certain phrases that describe certain interactions with game systems, how a space is ‘consensually hallucinated’ in a ‘non-space of the mind’.  Given user centered developments in HCI and transparent natural user interfaces (NUI), graphic representations now would conceal this Gibson’s complexity and to visualise the underling systems no standard geometry and dimensional space could accurately grasp this vision. Why would we want to see everything? As  Aarseth (2007) describes in Allegories of Space, Gibson’s vision is highly perceptive, bleak and ironic but this irony is missed by many futurists that it inspired. Additionally in this essay, Aarseth laid out how structurally different, games are as a new medium. In this capacity they must engage a player through new structures of interaction, by breaking with familiar forms of representation. But to what end does this enhance experience, and on a subtler point what should be the goal of these new mediums? How can they address shortcomings in previous experiences, and the consequences of their success; such as addiction and excessive emphasis on violent interactions.  Aarseth goes on to define spatiality in games: 

"The defining element in computer games is spatiality. Computer games are essentially concerned with spatial representation and negotiation, and therefore the classification of a computer game can be based on how it represents or, perhaps, implements space. More than time (which in most games can be stopped), more than actions, events and goals (which are tediously similar from game to game) and unquestionably more than characterization (which is usually nonexistent), games celebrate and explore spatial representation as a central motif and raison d’être. In terms of playability, themes, tasks, subgenres and dramatic structure, nothing much has changed over the last two decades. The innovation takes place in spatial representation, and the genre’s more slowly evolving complexity in the other areas (such as physical simulation) can be seen as a result of the increasing complexity of the spatial representation." 

– Aarseth, E. (2007). Allegories of space.   Space Time Play , 44-47.

Together this proposes how games can be used as mediums of experience in themselves, what knowledge can they convey? This then poses the question of how we define space, beyond a physical abstraction of geometry. What makes a social space? How do we generate regions of space for different purposes. How do computer mediated spaces differ?

“Rilke wrote: ‘These trees are magnificent, but even more magnificent is the sublime and moving space between them, as though with their growth it too increased.”

– Bachelard,   Poetics of Space,   Beaccon, 1958, p. 201

In the Poetics of Space Bachelard is, in part, inviting designers to re-consider how users’ experience by encouraging reference to formative, intimate spaces rather than formal abstractions. The quotation of a quotation above is useful as it prompts us to consider game spaces on two fronts – poetically (‘the sublime’: what does this space invoke? ), and kinetically (‘moving space’: reminding us that, above all, we are concerned with dynamics and locomotion ). Both these aspects are useful, as they allow us to consider and assess qualities of virtual-spaces that side-step ‘newer must be better perspectives’.

Ecological Perspective

[space is] neither a mere ‘frame’, after the fashion of the frame of a painting, nor a form or container of a virtually neutral kind, designed simply to receive whatever is poured into it. Space is a social morphology: it is to lived experience what form itself is to the living organism, and just as intimately bound up with function and structure. 

– Lefebvre, The Production of Space

This poetic quote summarises the nature of human relationship to space, a view that was built upon by James J. Gibson. James Gibson’s theory of direct perception states that the environment contains all of the information needed to specify its properties. Hence,perceiving these properties is a matter of detecting the information available in the environment.  This concept is significant precisely because it allows James Gibson to locate the moving, perceiving body at the heart of meaningful perceptual experience; an experience which emerges in the dynamical structures that cross the body and the world.  His theory of environmental affordances shows that perception is more than a means of passively representing the intrinsic physical organization of objects. Perception is inherently active and exploratory, we seek out alterations in the vast flow of information enveloping it. These alterations are detected when the perceiver moves through the environment and probes it with a pair of glancing eyes.  Locomotion opens up new possibilities for the pick-up of information specific to the perceiver’s environment. This information can then be used to guide subsequent movements, as in a perception-action loop (the two being inseparable). As David Morris puts it, “we do not … perceive naked properties of the environment, rather we perceive what the environment affords to our bodies, what we can do with, or in the environment.” The theory of affordances demonstrates that the structures of information are intrinsically meaningful for perceptually-guided action.  Affordances are not subjective valuations superimposed on sensations (as theories of perception typically presuppose). Rather, the affordances of the environment are directly perceived as structural information in the environment. 

– Braun, The Structures of Perception: An ecological perspective, 2008

Of all the terms we use to describe the world we inhabit, [space] is the most abstract, the most empty, the most detached from the realities of life and experience. Biologists say that living organisms inhabit environments   not space, and whatever else they may be, human beings are certainly organisms.

– Tim Ingold, Against Space: Place, Knowledge, Movement

Acoustic Spaces

“perception must be understood as a relationship between environmentally available information and the capacities,sensitivities, and interests of a perceiver.” 

– Eric Clarke, Ways of Listening: An Ecological Approach to the Perception of Musical Meaning

To use ecological perception theory to determine musical meaning, Clarke identifies properties of musical sounds that “afford” certain meanings. Affordance is a term coined by Gibson to describe the purposes or uses of various objects as communicated by stimulus. As an example, a chair affords sitting,taming lions, and knocking down bad guys. Clarke uses this concept to show how listeners can glean meaning from a musical performance. Note that he either stipulates a specific listener and a specific performance, or gives ranges of possible meanings dependent upon the context of the performance and the listener.

Interesting Links not discussed in detail:

Leap Motion Blog post about VR OS

Harrison and Dourish’s Work on Collabritive spaces

Virtual Reality Music pt 1: A wee taster…

In the start of a likely lengthy series of posts, I’ll be presenting some of the internets occurrences in VR music making/experiences. This set of posts will run in parallel with a "Tangible Music Interfaces" set of posts that will cover alternative interaction methods with audio and music applications. For this first one I’m being generous and sharing two things that caught my attention lately.

Squarepushers 360 music video for Stor Eiglass, is a journey through a bubblegum world of bizarre 8 bit escapism. It highlights the possible dangers of VR addication and is also hilarious/scary. Reminds me abit of Osamu Sato’s early psychadelic computer game Eastern Mind: The Lost Souls of Tong-Nou.

Next up is a bit of research form the NIME community on a hierarchical live looping VR music app, DrilleThis technique consists in creating musical trees whose nodes are composed of sound effects applied to a musical content. The system presented requires considerable training and learning to be able to perform with it though the basics of user control can be learned quite quickly. The first video shows a good overview of the system while the second video explores a more structured performance. Further research indicates how metaphors of interaction are utilised in the creation of user control and system design.