Speculative musical interface design

What is the current state of Speculative Design in musical instrument design communities, and what is its value? This article serves as a series of reviews and critiques to approach answering these questions, written in more of a polemic style than I actually would practise myself. The work is the write-up of a talk presented at Queen Mary University of London’s qualitative research methods group. Unfortunately it does not include definitions for speculative, critical or fictional design; as other speakers covered these topics before me. In a future post I will present a brief overview of these concepts, or link to co-presenters writings. For now you can check out Tobias Revell’s great lecture on the topic.  The article covers the following:

Current Practise: what value does fictional, speculative or critical design approaches receive in current digital musical instruments (DMIs)  design communities?

Approaches: what are some current design fiction approaches to DMI design? What is the work being done currently in other fields that could bridge towards more speculative practises in music technology design? 

Codetta: why do we want design fiction in DMIs?

Current practise and its discontents.

Accurately describing or conceptualising current DMI design and lutherie practise is a nebulous endeavour,  see NIME for a taster, and as such will not be attempted here.

While music practise often revels in abstract and impressionistic realms of the imaginary, crudely, I would posit many of the design practises found in research and arts still take a distinctly concrete approach to conceptualisation, prototyping, and realisation.

And yes, what use is the unheard sound from an imaginary gesture on machine that has not been built yet?

Image sourced from 3dmin.orgEven in more adventurous research projects that challenge conservative music educational practise and research, such as Berlin based Development and Dissemination of New Musical Instruments (3DMIN); there is a distinct emphasis on developing entirely useable prototypes or corporeal music experiences. With this scope, really questioning what digital augmentation can offer as a new form of  expression may not be addressed by the bias of requiring formal realisation. This is where speculative approaches can offer different avenues of discourse around how we want technology to augment musical experience as a performer, participant, or listener.

Before moving on, I would like to position that the majority sound and music interface design research is digital crafts. Echoing Dunne and Raby’s account of new media art:

“device art usually focused on aesthetic, communicative, and functional possibilities for new media rather than visions of how life could be, and mainly takes the form of digital craft rather than future speculations.” – Dunne and Raby, Speculative Everything, pg26

Though often adventurous, groundbreaking even, the limit of such research practise is in looking at iterative expansion, focussed around recent technology developments. So, what could creativity and expression be, through speculative eyes? How can we look towards ideas of things yet discovered as expressive? What new bodies or entities do we allow into the  process of expression: AI, big data, mass interaction, computational agents?  

Why critique digital crafts for not looking beyond the boundaries of current possibility? 

With its parental links to the formal HCI community, DMI design has certain traits or tropes of perspective that have been adopted and reinforced. The development of the “evaluation” theme in current literature, while useful, evaluation metrics and standard setting does bias consumable technology focused results. This does not address what the perspective or philosophical traditions that the chosen developments reinforce. Many design disciplines with extensive histories have embraced of fictional accounts, see the book Speculative Everything by Dunne and Raby for a in depth review. Such accounts nullify market forces from decisions and can remove conventional functionality. This allows space for different kinds of questions to be asked about the role of music, sound and technology for creative expression.

How is fiction used in DMI design?

After conducting a very brief literature synthesis of the NIME community, the role of fiction, speculation, or critical design loosely occurred around three themes:

  1. Urban fiction: use of fictional music or sound based overlays to reality through ubiquitous computing technologies such as locative media.
  2. Fiction as inspiration: use of fictional design or science fiction as inspiration for beyond what is possible now.
  3. Expanding experience: using fiction as a concept for different interaction experiences, how to stimulate new feelings or ways of doing.

Looking across these themes, very few papers or descriptions employ fiction as the goal. It appears as a composite part of the project but not the purpose. While encouraging that it appears, I would ask that its role is expanded.

So, if robust prototypes are not exciting enough, what is?

By means of example, I would like to put forward the work of Choi Ka Fai whose project questions embodied knowledge and possibly new expressive meanings of agency. 

Central to the project is the study of body movement in dance. Using electronic muscle stimulation, you can remotely control another’s actions  or be controlled by previously recorded muscle memories. While entirely based in the now of technology, this work addresses complex issues of our bodies including memory, synchronicity, expressivity, and agency; that can be extended to musical interaction quite easily.


In City Symphonies, this project looks at the consequences of very near technological developments when electric cars will change the soundscape of cities, Mark McKeague asks whether a city can become a symphony. This work alters the boundaries of what we classify as music, though it could dismissed as simply as sonification.

“From a street level perspective the motions of traffic combine the sounds to create soundscapes that are unique to the place and time. The roadside becomes a new context for sound – the city is the score.”


Aloïs Yang presents The Star Light Collection, primarily a fictional sonification project, based in a design object, that draws parallels between the interaction of stellar bodies to musical performance.


How can we practice at the unknown, talk about and act around that which is yet-to-be-imagined?!

Before moving into HCI methods and approaches to engaging fiction, it is worth describing the Dunne and Raby Method of framing Speculative Design.

Overview: Using design to ask questions rather than providing answers or solving problems.

Process: A highly simplified process for adapting conceptual design into different technology areas is describe as follows (Dunne and Raby, Speculative Everything, pg57)
+ identify a specific area of science research, eg biotechnology and music

+imagine issues that might arise once the research moves from lab to everyday life, eg sonically adaptive skin layer to feel music, then people use music as weapon to `scar’ others

+ embody these issues in a design proposal aimed at sparking debate or discussion. eg looks at sadistic tendency but also imaginative possibility of feeling sounds in new ways.

Framing the process


The figure above maps a large variety of design practises, critical design occupies the top-left as design-led expert-mindset. Adopting a opportunist research and design led agenda is something that Speculative design is amenable to. By framing the design strategy loosely, but not focussing on problem solving too explicitly we can cherry pick the useful methods required for asking the right questions. We could ask how we open the circle of what is a musical performer or collaborator can be e.g. nature, society, AI? Considering such new collaborators, how do we address the potential design and solution spaces? Each methodology in the figure above would offer slightly different versions in answer to those questions.

From a previous blog post, a series of techniques used in speculative design were outlined, they include: Fictional worlds; Cautionary tales; What if… scenarios; Counterfactual histories; Thought experiments; Reductio ad absurdum; Artefacts from the future; Pre-figurative futures; Small things big issues; Tell worlds rather than tell stories. From my perspective, those in bold are the key areas that musical interfaces design can benefit from. These areas can be explored through critical design, generative techniques, and experiential prototyping. 

Speculative design deals with imagination and fiction. Adapting it to musical interaction could lead to new user experience understandings. Its focus on new extensions of expressive capability using technology (yet to be determined) could result in novel research directions. For instance, using experiential prototyping to explore agency in musical control, we utilise imaginative expression that can infer self-initiated prediction cycles, where we exhibit our expectations, by making visible through play our expressive relationship to the world. Music design fiction is embodied knowledge at play.


Context mapping is a methodology that is commensurate with fictional design as UCD. Context mapping designs for experience, and generative tools could build towards new notions of music interface dynamics, by avoiding preset assumptions based on simple observation or explicit questioning. The methodology acknowledges the environment where HCI takes place, it’s not just the link between user and object. But what is context, everything else? Visser et al (2005) define is as ‘all factors that influence the experience of a product use’. These can be social, physical and internal emotional worlds. To access information around these factors context mapping utilises generative techniques to gain knowledge about what people know, feel and dream. In describing why it is important to focus on such goals in design Visser et al (2005) indicates that we can separate out tacit and latent knowledge, using generative (projective) techniques:

“The use of these projective techniques provides a view to reveal future states of people. These techniques can reveal tacit knowledge and expose latent needs (Sanders, 2001). Tacit knowledge is knowledge that people can act upon, but cannot readily express in words (Polanyi, 1964). Latent needs are those that people are not yet aware of. They are needs that become realised in the future.” – Visser et al (2005)


Continuing with work on generative techniques, research conducted by Kristina Andersen at STEIM explores playfulness and strangeness through experience design, design fiction and participatory workshop methodology; to source inspiration for music technology design. Particularly, in the GiantSteps project she focuses on creating new and disruptive interfaces for creating electronic music. While the goal of the overall project is integrating findings into marketable toolchains, the methodology targets desires, future scenarios [KneesAndersenTkalcic2015], and design fictions [Andersen2014]. Andersen’s approaches design exploration through workshops and artist engagement [AndersenGrote2015] that look to temporarily suspend the history embedded in everyday objects and music instruments, to look at what we want expressivity to be [AndersenGibson2015]. Andersen’s perspective remains open to new relationships of sound and musical context, but involves users in a participatory process via generative techniques such as non-functional prototypes and fictional technological objects. These processes create “machines that might make that sound” act as props and explanatory objects that allow discussion and acting out of complex understanding, prospective modalities, and how expressive abilities are to be embodied in future interface ideas. This process is an attempt to tap into lived experience and engage with an essentially imaginary future object.


Why is it interesting: Speculative design can pull new technological developments into imaginary but believable everyday situations so that we can explore possible consequences before they happen. We can make ideas tangible well before the are made  or even possible to be made.

Why are we interested in it for music technology: As the boundaries of what we could consider collaborators in musical process changes we need to engage with alternative aesthetics that inspire us in different ways, by questioning technology, ideology, and technological vs social imagination; we make a space to propose different visions of musical expression calling on more ludic experiences of music rather than notions of perfectly engineered autonomous control. To imagine new modes of expression that challenge traditional fetishism of historical music practise. While useful they immediately limit what we think musical expression is and what its function is to us.

Link bucket

Some RCA Design Interactions graduates that utilise sound and music as medium for speculative design:

+ http://samconran.com

+ http://aloisyang.com/

+ https://markmckeague.com/

Value of playful experience in technology: http://www.ted.com/talks/steven_johnson_how_play_leads_to_great_inventions


Froukje Sleeswijk Visser, Pieter Jan Stappers, Remko Van Der Lugt, et al. 2005. Contextmapping: experiences from practice. CoDesign 1, 2: 119–149. http://doi.org/10.1080/15710880500135987

Kristina Andersen and Dan Gibson. 2015. The Instrument as the Source of new in new Music. Research Through Design, March: 25–27. http://doi.org/10.6084/m9.figshare.1327992.

Kristina Andersen.2014. Using Props to Explore Design Futures: Making New Instruments. In CHI workshop on Alternate Endings: Using Fiction to Explore Design Futures.

Kristina Andersen and Florian Grote. 2015. GiantSteps: Semi-Structured Conversations with Musicians. Extended Abstracts of the ACM CHI’15 Conference on Human Factors in Computing Systems 2: 2295–2300. http://doi.org/10.1145/2702613.2732868

Peter Knees, Kristina Andersen, and Marko Tkalcic. 2015. “ I ’ d like it to do the opposite ”: Music-Making Between Recommendation and Obstruction. Proceedings of the 2nd International Workshop on Decision Making and Recommender Systems (DMRS)(CEUR-WS): 1–7.

P. Knees, K. Andersen, S. Jorda, et al. 2015. Giantsteps-progress towards developing intelligent and collaborative interfaces for music production and performance. 2015 IEEE International Conference on Multimedia and Expo Workshops, ICMEW 2015: 4–7. http://doi.org/10.1109/ICMEW.2015.7169826

Augmented Audio

In this rather text heavy sprawling post I’ll be briefly covering the present and future possibilities of augmented reality audio (ARA for the purposes of speed). For the uninitiated the augmented audio website provides a comprehensive overview:

“Augmented Audio is defined as live audio being modified and/or enhanced by computer-generated sensory input. By introduction of another layer of information or media on top of an input feed, it is possible to produce highly enriched audio feedback. The concept belongs in the Augmented Reality (AR) family, but has started to distinguish itself as an important independent application area due to recent media attention. New innovative implementations have shown that the concept is applicable in many different contexts, including tourism, location-oriented communication and information sharing, gaming and entertainment and even as a distinctive new interaction technique.”

Also this website provided a link the audio gaming company RjDj (Reality Jockey Ltd). This company was an early developer in the area pushing beyond the confines of the research based approach to fully fledged consumer applications. Their apps won critical acclaim, and are still developing new concepts in audio gaming and general entertainment. Their new situ app is particularly useful for playlist junkies such as myself, whereby it learns your activity cycle’s and related music choices to automatically recommends music. Like a automated, situational, related artist search. Seeing as I’ve spent hours doing this on Spotify its very welcome. Semantic music analysis straight in your lugs, and its free!

To pick up on a article in designing sound (fantastic sound design blog), in which they covered Audio within VR environments. The article mentioned the 3Dception toolset for game design. Surround sound has existed within the gaming community for a long time. But, where 3Dception makes its mark is the ease by which a developer can integrate binaural audio into their workflow with tools for the Unity and Wwise platforms. So, where does fit with ARA? Glad you asked. They are currently developing toolsets for NativeSDK, Pure data (and libpd…hence android and iOS) and Max/MSP. This then allows for construction of augmented environments using sensor data from the respective platform, ta da, augmented reality! This coupled with their interest in integrating procedural audio could make some amazing possibilities for dynamic reactive audio environments.

(Additionally from the article on designing sound I was pleased to find this chap in the comments. Frank the terrorist is a homemade dummy head used for binaural recording, and it works pretty well, the recent thunder storm clip is very detailed)

So now we have some ARA options, what else are we going to need? In steps the intelligent headset with its uninspiring promotional videos and prohibitively high cost except for affluent early adopters. Not that prohibitively high cost has stopped many things in audio geekery. This being said dedicated hardware for ARA is a good idea, as positioning of sensors is not optimal when they all stored within a handset. By adapting headphones with accurate positional indicators the HRTF algorithms can happily match head position to aural environment. Also indluded in the headset is GPS, compass, gyro and accelerometer. One key element missing, in my humble opinion, that is accurate stereo microphones so that augmented audio can be layered on top of existing real world audio. Though quoted as a ARA solution I think it fundamentally misses the mark. In truth this is just a well optimised binaural solution.

Also Dolby recently launched their Atmos mobile venture inside the new Amazon kindle. This lab note is a great technical description and well worth a look. But whether this technology can be adapted for ARA remains to be seen. Dolby haven’t yet created any tools beyond that of professional services. Though this maintains brand integrity it does stifle the development of the technology in the public sphere. This has the benefit of separating professional products, such as Hollywood films, from other content. But innovation in a medium does not always come from the establishment. So maybe it would be worth Dolby developing accessible production tools to allow smaller ventures to endorse and enhance the technology in a organic sense. A Dolby mobile API maybe?


 To finish, if you are still reading, I would like to point you in the direction of a fantastic tech blog praxtime. In the article highlighted a series of possible wearable augmented reality solutions are presented. These predictions and concepts are based on the trends and possibilities in wearable tech. 

Thanks for reading.


Interference: Journal of Audio Culture

Courtesy of UCLA

Listen too much you have trouble listening? Working in the audio field can often sap you of your basic ability to listen, as you constantly have to produce to deadlines, assess quality and generally conform a product to external conventions. But the most fundamental requirement of sound to a human is to inform them of a position in space, as before verbal communication the auditory system prevented you from getting eaten by bigger predators. This being said the importance of communication cannot be overlooked, as it is tied with our gradual evolutionary supremacy.

In steps Interference. A peer reviewed journal, supported by Trinity College Dublin, that is entirely free access. They describe themselves as follows:

“Interference is an open access forum on the role of sound in cultural practices, providing a trans-disciplinary platform for the presentation of research and practice in areas such as acoustic ecology, sensory anthropology, sonic arts, musicology, technology studies and philosophy. The journal seeks to balance its content between scholarly writing, accounts of creative practice, and an active engagement with current research topics in audio culture.”

Of special note is one article in the current issue Leandra Lambert called “Experienced Sonic Fictions“, which I shall very superficially contextualise for this article. Throughout the introduction of the piece Lambert mentions the founders in the field that established the ‘deep listening’ and ‘sonic awareness’ disciplines, which can be approximated to a form of listening meditation. Proceeding this she describes the process of free-form sound walks, and the associated imagery that is stimulated. As by letting her imagination guide her through these walks the stimulation is less and less guided by any conscious purpose, and in reaction the ideas and concepts imagined become more lucid and fantastic. Though rather random and quite time consuming it does reaffirm the idea that we need to listen to our environments and not try to block them out or classify them to swiftly. Though this capacity for ordering reality is essential in modern life, for a sound designer the ability to stop and actually listen to a scene for all its richness is worth remembering. In many respects it is reminiscent of the John Cage works on silence and of how evocative the absence of direct stimulus is, paradox or contradiction I’m not sure?

Coming back to the opening gambit, though sound walks may not be for you, the idea that to truly assess and recreate a sound scape one must remember how to listen is a very important skill. How you choose to do this can come in many forms, as with all creative processes, but it is a important principle as audio technology reproduction methods approach the means to reproduce true soundscapes to a mass market.

Another journal of note is that of SoundEffects, also open access and very stimulating.