In 1995, Heath, Luff, & Sellen lamented the uptake of video conferencing indicating that it had not at the time reached its promise. But looking back at this projection, the ubiquity of video systems for social and work communication can be seen. And subsequently, research has gone about understanding it further in a variety of HCI paradigms (CHI2010, CSCW2010, CHI2018). So, for my research, making projections on the use of VR for music collaboration, it might be that findings and insights do not reach fruition, either, in a timely fashion, or in the domain of interest that they were investigated in, or ever! Though this could be touching on a form of hindsight bias.
Going back to the article that speculated on the unobtained promise of video conferencing technologies, Heath Luff, and Sellen (1995), provide a piece of insight that can still be placed into perspective on design interventions for collaboration:
It becomes increasingly apparent, when you examine work and collaboration in more conventional environments, that the inflexible and restrictive views characteristic of even the most sophisticated media spaces, provide impoverished settings in which to work together. This is not to suggest that media space research should simply attempt to ‘replace’ co-present working environments, such ambitions are way beyond our current thinking and capabilities. Rather, we can learn a great deal concerning the requirements for the virtual office by considering how people work together and collaborate in more conventional settings. A more rigorous understanding of more conventional collaborative work, can not only provide resources with which to recognise how, in building technologies we are (inadvertently) changing the ways in which people work together, but also with ways in which demarcate what needs to be supported and what can be left to one side (at least for time being). Such understanding might also help us deploy these advanced technologies.
The bold section highlights the nub of what I’m interested in; for VR music collaboration systems. I break this down into how I’ve tackled framing collaboration in my research:
conventional collaborative work – ethnographies of current and developing practice. Even if you pitch a radical agenda of VR workspace, basic features of the domain of interest need to be understood for their contextual and technical practices.
building technology is changing practice – observing the impact of design interventions on how people collaborate in media production. Not only does a technology suggest new ways of working, it can enforce them! Observing and understanding this in domain-specific ways is important.
what needs to be supported – basic interactional requirements, we have to be able to make sense of each other, and the work, together, in an efficient manner.
what can be left to one side – the exact models and metaphors of how work is constructed in reality, in VR we can create work setups and perspectives that cannot exist in reality. For instance, shared spatial perspectives i.e. seeing the same thing from the same perspective is impossible in reality as we have to occupy a separate physical space. In repositioning basic features of spatial collaboration, the effects need to be understood in terms of interaction and domain requirement. But the value is in finding new ways of doing things not possible in face to face collaboration.
Overall, the key theme that should be taken away is that of humans’ need to communicate and collaborate. In this sense, any research that looks to make collaboration easier is provisioning for basic human understanding. That is quite nice to be a part of.
The Polyadic interface enables collaborative composition of 16 step drum loops to accompany backing tracks in 4 different genres of electronic music for two or more co-located participants utilising two user interface media, Virtual Reality (VR) and Desktop (DT).
To accommodate the cross-platform development of the system an overview of programming paradigms was made, to determine an architecture for quick prototyping. The architecture design goals were:
Ease of feature development for testing multiple approaches.
Deterministic network interaction with interfaces.
Modular code structure to allow a Git-Flow style of development with parallel feature development not causing merge nightmares.
The final architectures included:
Entity-Component-System – the final winner, talk more about this later.
Dependency Injection / Inversion of Control – lots of supporters, but it all seemed a bit weird to set up and work with for this application. Initial tests were positive, but the style of the structure started to annoy me.
Model-view-controller – classic, solid design pattern. But scaling it to maintain a tidy feature set for the cross-platform network application felt dangerous. I saw it turning into a pseudo pattern, where best intentions are kept but the flexibility and my laziness would make me turn it into illogical spaghetti.
Hack away – what I have done previously, a lot. The speed of just bringing functions together however you like is always appealing in the short term, like a really fatty burger, but it will shorten your life somehow.
Entity-Component-System
Here, here, and here are some good introductions/discussions by Maxim Zaks, a major contributor to the Entitas ECS framework. To summarise, ECS, and specifically Entitas, reduce everything down to data, groups of data, and systems that act on data. This is very different from classic OOP design and required a little retraining of my process and thinking. I made many mistakes and introduced a lot of pseudo-dependencies during this process. Now, in the process of refactoring, after doing some other projects with it, I am rooting out these pseudo-dependencies and reducing the reliance on wasteful LINQ operations. In the end, it mostly produced decoupled code that allowed very feature-driven development. As I’m working by myself on the project, I haven’t got into the unit testing possibilities, but these are said to be great.
So when not to use ECS?
Building frameworks for others or purely computational systems, see this for discussion. Though I am toying with a fully ECS driven audio signal processing idea, might be folly though…
Managed to get the VR version of Polyadic scaled down, instead of a massive panel you have to stretch across to operate on, the scaled down version is roughly the width of an old MPC. This is important for visual pattern recognition in the music making process, but also the sizing allows for alternate workspace configurations, that are more ergonomic and can handle more toys being added!
To get the scaled down features to work a tool morphing process has been designed. The problem is the Oculus Rift and HTC Vice controllers are quite large, especially in comparison to a mouse pointer. So by using smaller hand models when you are in the proximity of the drum machine you can have a higher ratio of control to display, with respect to less of the hand model being able to physically touch features in the interface.
Control-display (C-D) ratio adaptation is an approach for facilitating target acquisition, for a mouse the C-D ratio is a coefficient that maps the physical displacement of the pointing device to the resulting on-screen cursor movement (Blanch, 2004), for VR it is the ratio between the amplitude of movements of the user’s real hand and the amplitude of movements of the virtual hand model. Low C-D ratio (high sensitivity) could save time when users are approaching a target, while high C-D ratio (low sensitivity) could help a user with fine adjustment when they reach the target area. Adaptive Control-Display ratios such as non-linear mappings have been shown to benefit 3D rotation and 3D navigation tasks.
But the consequence of this mapping change will be an expressive difference. In the original prototype with the oversized wall of buttons and sliders, the experience of physical exertion might have been quite enjoyable? By reducing this down, a very different body space will be created, the effects of this remain to be tested. Subjectively it did feel more precise and coherent as a VR interface, less toy-like and comical. As mentioned in the introduction, the sizing can have implications for pattern recognition. The smaller size allows you to overview the whole pattern while working on it, whereas previously the size meant stepping back or craning your neck to take it all in. It would be interesting to know how much effect the gestalt principles of pattern recognition have on cognitive load in music making situations, given the time-critical nature of the audiovisual interaction.
Blanch, R., Guiard, Y. & Beaudouin-Lafon, M., 2004. Semantic Pointing – Improving Target Acquisition with Control-display Ratio Adaptation. Proceedings of the International Conference on Human Factors in Computing Systems (CHI’04), 6(1), pp.519–526. Available at: http://doi.acm.org/10.1145/985692.985758.
What is the current state of Speculative Design in musical instrument design communities, and what is its value? This article serves as a series of reviews and critiques to approach answering these questions, written in more of a polemic style than I actually would practise myself. The work is the write-up of a talk presented at Queen Mary University of London’s qualitative research methods group. Unfortunately it does not include definitions for speculative, critical or fictional design; as other speakers covered these topics before me. In a future post I will present a brief overview of these concepts, or link to co-presenters writings. For now you can check out Tobias Revell’s great lecture on the topic. The article covers the following:
Current Practise: what value does fictional, speculative or critical design approaches receive in current digital musical instruments (DMIs) design communities?
Approaches: what are some current design fiction approaches to DMI design? What is the work being done currently in other fields that could bridge towards more speculative practises in music technology design?
Codetta: why do we want design fiction in DMIs?
Current practise and its discontents.
Accurately describing or conceptualising current DMI design and lutherie practise is a nebulous endeavour, see NIME for a taster, and as such will not be attempted here.
While music practise often revels in abstract and impressionistic realms of the imaginary, crudely, I would posit many of the design practises found in research and arts still take a distinctly concrete approach to conceptualisation, prototyping, and realisation.
And yes, what use is the unheard sound from an imaginary gesture on machine that has not been built yet?
Even in more adventurous research projects that challenge conservative music educational practise and research, such as Berlin based Development and Dissemination of New Musical Instruments (3DMIN); there is a distinct emphasis on developing entirely useable prototypes or corporeal music experiences. With this scope, really questioning what digital augmentation can offer as a new form of expression may not be addressed by the bias of requiring formal realisation. This is where speculative approaches can offer different avenues of discourse around how we want technology to augment musical experience as a performer, participant, or listener.
Before moving on, I would like to position that the majority sound and music interface design research is digital crafts. Echoing Dunne and Raby’s account of new media art:
“device art usually focused on aesthetic, communicative, and functional possibilities for new media rather than visions of how life could be, and mainly takes the form of digital craft rather than future speculations.” – Dunne and Raby, Speculative Everything, pg26
Though often adventurous, groundbreaking even, the limit of such research practise is in looking at iterative expansion, focussed around recent technology developments. So, what could creativity and expression be, through speculative eyes? How can we look towards ideas of things yet discovered as expressive? What new bodies or entities do we allow into the process of expression: AI, big data, mass interaction, computational agents?
Why critique digital crafts for not looking beyond the boundaries of current possibility?
With its parental links to the formal HCI community, DMI design has certain traits or tropes of perspective that have been adopted and reinforced. The development of the “evaluation” theme in current literature, while useful, evaluation metrics and standard setting does bias consumable technology focused results. This does not address what the perspective or philosophical traditions that the chosen developments reinforce. Many design disciplines with extensive histories have embraced of fictional accounts, see the book Speculative Everything by Dunne and Raby for a in depth review. Such accounts nullify market forces from decisions and can remove conventional functionality. This allows space for different kinds of questions to be asked about the role of music, sound and technology for creative expression.
How is fiction used in DMI design?
After conducting a very brief literature synthesis of the NIME community, the role of fiction, speculation, or critical design loosely occurred around three themes:
Urban fiction: use of fictional music or sound based overlays to reality through ubiquitous computing technologies such as locative media.
Fiction as inspiration: use of fictional design or science fiction as inspiration for beyond what is possible now.
Expanding experience: using fiction as a concept for different interaction experiences, how to stimulate new feelings or ways of doing.
Looking across these themes, very few papers or descriptions employ fiction as the goal. It appears as a composite part of the project but not the purpose. While encouraging that it appears, I would ask that its role is expanded.
So, if robust prototypes are not exciting enough, what is?
By means of example, I would like to put forward the work of Choi Ka Fai whose project questions embodied knowledge and possibly new expressive meanings of agency.
Central to the project is the study of body movement in dance. Using electronic muscle stimulation, you can remotely control another’s actions or be controlled by previously recorded muscle memories. While entirely based in the now of technology, this work addresses complex issues of our bodies including memory, synchronicity, expressivity, and agency; that can be extended to musical interaction quite easily.
In City Symphonies, this project looks at the consequences of very near technological developments when electric cars will change the soundscape of cities, Mark McKeague asks whether a city can become a symphony. This work alters the boundaries of what we classify as music, though it could dismissed as simply as sonification.
“From a street level perspective the motions of traffic combine the sounds to create soundscapes that are unique to the place and time. The roadside becomes a new context for sound – the city is the score.”
Aloïs Yang presents The Star Light Collection, primarily a fictional sonification project, based in a design object, that draws parallels between the interaction of stellar bodies to musical performance.
Approaches
How can we practice at the unknown, talk about and act around that which is yet-to-be-imagined?!
Before moving into HCI methods and approaches to engaging fiction, it is worth describing the Dunne and Raby Method of framing Speculative Design.
Overview: Using design to ask questions rather than providing answers or solving problems.
Process: A highly simplified process for adapting conceptual design into different technology areas is describe as follows (Dunne and Raby, Speculative Everything, pg57)
+ identify a specific area of science research, eg biotechnology and music
+imagine issues that might arise once the research moves from lab to everyday life, eg sonically adaptive skin layer to feel music, then people use music as weapon to `scar’ others
+ embody these issues in a design proposal aimed at sparking debate or discussion. eg looks at sadistic tendency but also imaginative possibility of feeling sounds in new ways.
Framing the process
The figure above maps a large variety of design practises, critical design occupies the top-left as design-led expert-mindset. Adopting a opportunist research and design led agenda is something that Speculative design is amenable to. By framing the design strategy loosely, but not focussing on problem solving too explicitly we can cherry pick the useful methods required for asking the right questions. We could ask how we open the circle of what is a musical performer or collaborator can be e.g. nature, society, AI? Considering such new collaborators, how do we address the potential design and solution spaces? Each methodology in the figure above would offer slightly different versions in answer to those questions.
From a previous blog post, a series of techniques used in speculative design were outlined, they include: Fictional worlds; Cautionary tales; What if… scenarios; Counterfactual histories; Thought experiments;Reductio ad absurdum;Artefacts from the future; Pre-figurative futures; Small things big issues; Tell worlds rather than tell stories. From my perspective, those in bold are the key areas that musical interfaces design can benefit from. These areas can be explored through critical design, generative techniques, and experiential prototyping.
Speculative design deals with imagination and fiction. Adapting it to musical interaction could lead to new user experience understandings. Its focus on new extensions of expressive capability using technology (yet to be determined) could result in novel research directions. For instance, using experiential prototyping to explore agency in musical control, we utilise imaginative expression that can infer self-initiated prediction cycles, where we exhibit our expectations, by making visible through play our expressive relationship to the world. Music design fiction is embodied knowledge at play.
Context mapping is a methodology that is commensurate with fictional design as UCD. Context mapping designs for experience, and generative tools could build towards new notions of music interface dynamics, by avoiding preset assumptions based on simple observation or explicit questioning. The methodology acknowledges the environment where HCI takes place, it’s not just the link between user and object. But what is context, everything else? Visser et al (2005) define is as ‘all factors that influence the experience of a product use’. These can be social, physical and internal emotional worlds. To access information around these factors context mapping utilises generative techniques to gain knowledge about what people know, feel and dream. In describing why it is important to focus on such goals in design Visser et al (2005) indicates that we can separate out tacit and latent knowledge, using generative (projective) techniques:
“The use of these projective techniques provides a view to reveal future states of people. These techniques can reveal tacit knowledge and expose latent needs (Sanders, 2001). Tacit knowledge is knowledge that people can act upon, but cannot readily express in words (Polanyi, 1964). Latent needs are those that people are not yet aware of. They are needs that become realised in the future.” – Visser et al (2005)
Continuing with work on generative techniques, research conducted by Kristina Andersen at STEIM explores playfulness and strangeness through experience design, design fiction and participatory workshop methodology; to source inspiration for music technology design. Particularly, in the GiantSteps project she focuses on creating new and disruptive interfaces for creating electronic music. While the goal of the overall project is integrating findings into marketable toolchains, the methodology targets desires, future scenarios [KneesAndersenTkalcic2015], and design fictions [Andersen2014]. Andersen’s approaches design exploration through workshops and artist engagement [AndersenGrote2015] that look to temporarily suspend the history embedded in everyday objects and music instruments, to look at what we want expressivity to be [AndersenGibson2015]. Andersen’s perspective remains open to new relationships of sound and musical context, but involves users in a participatory process via generative techniques such as non-functional prototypes and fictional technological objects. These processes create “machines that might make that sound” act as props and explanatory objects that allow discussion and acting out of complex understanding, prospective modalities, and how expressive abilities are to be embodied in future interface ideas. This process is an attempt to tap into lived experience and engage with an essentially imaginary future object.
Codetta
Why is it interesting: Speculative design can pull new technological developments into imaginary but believable everyday situations so that we can explore possible consequences before they happen. We can make ideas tangible well before the are made or even possible to be made.
Why are we interested in it for music technology: As the boundaries of what we could consider collaborators in musical process changes we need to engage with alternative aesthetics that inspire us in different ways, by questioning technology, ideology, and technological vs social imagination; we make a space to propose different visions of musical expression calling on more ludic experiences of music rather than notions of perfectly engineered autonomous control. To imagine new modes of expression that challenge traditional fetishism of historical music practise. While useful they immediately limit what we think musical expression is and what its function is to us.
Link bucket
Some RCA Design Interactions graduates that utilise sound and music as medium for speculative design:
Froukje Sleeswijk Visser, Pieter Jan Stappers, Remko Van Der Lugt, et al. 2005. Contextmapping: experiences from practice. CoDesign 1, 2: 119–149. http://doi.org/10.1080/15710880500135987
Kristina Andersen and Dan Gibson. 2015. The Instrument as the Source of new in new Music. Research Through Design, March: 25–27. http://doi.org/10.6084/m9.figshare.1327992.
Kristina Andersen.2014. Using Props to Explore Design Futures: Making New Instruments. In CHI workshop on Alternate Endings: Using Fiction to Explore Design Futures.
Kristina Andersen and Florian Grote. 2015. GiantSteps: Semi-Structured Conversations with Musicians. Extended Abstracts of the ACM CHI’15 Conference on Human Factors in Computing Systems 2: 2295–2300. http://doi.org/10.1145/2702613.2732868
Peter Knees, Kristina Andersen, and Marko Tkalcic. 2015. “ I ’ d like it to do the opposite ”: Music-Making Between Recommendation and Obstruction. Proceedings of the 2nd International Workshop on Decision Making and Recommender Systems (DMRS)(CEUR-WS): 1–7.
P. Knees, K. Andersen, S. Jorda, et al. 2015. Giantsteps-progress towards developing intelligent and collaborative interfaces for music production and performance. 2015 IEEE International Conference on Multimedia and Expo Workshops, ICMEW 2015: 4–7. http://doi.org/10.1109/ICMEW.2015.7169826
Following is some pastoral advice gained from doing a project in a new field when the brief is quite open. As with all advice, it depends on your personality!
Define the concept as simply as possible – if communication of the underlying concept isn’t clear, how will the implementation not be?
If its a good idea, follow it – When populating a design space with early concepts, tangents and ideas abound. These may diverge significantly from the original concept you thought of, but in the creative process this is perhaps the nature of ideas. As when balancing all the elements, hidden parameters and approaches appear. These are things you couldn’t perceive in your original constructs and perhaps hold a grain of something truly novel. If you don’t have a strict brief, let go and see where it leads.
Domain knowledge – when coming from a specialist field, such as audio, be wary of the perceived knowledge in users. Your domain knowledge and intellectual predispositions will guide your design space decisions. If you are not careful your ability to communicate to a wider audience will be doomed from the start, due to relying on existing interface metaphors that do not communicate effectively to new users. But if you focus the application to specific domains, these nuances can make it through a design process and be of use to the field more generally.
A important concept in early development was the use of VR as a lense into imagined worlds. The work of Dunne and Raby on Speculative design was particularly persuasive. Their techniques include:
+ Fictional worlds
+ Cautionary tales
+ What if… scenarios
+ Counterfactual histories
+ Thought experiments
+ Reductio ad absurdum
+ Artefacts from the future
+ Pre-figurative futures
+ Small things big issues
+ Tell worlds rather than tell stories
These aspects are employed as alternative aesthetics that engage us in different ways, questioning technology, ideology, and technological vs social imagination
The Metaverse, as traditionally imagined, would be an unfiltered firehose of humanity. The Metaverse that people are actually trying to build would be, in a meaningful sense, a social network. Most of its value is bringing people together socially, and letting them communicate with their friends and make new ones. Putting everyone together into the same chaotic chatroom has less value than intelligently providing spaces where friends can hang out, as web-based social networks have proven. This concept would be engaged with a speculative frame in VR by posing the question of how algorithms would mediate our interaction and communications, with people and machines that are sharing the space.
http://lucidscape.com/ found this data visualisation quite stimulating, the debug view is quite attractive too
Jaap Drupsteen’s music visualisations are fantastic, particularly the one below was of interest at around 3:40 where the concrete structure is morphed into a twitching jittering mass of nodes. This was to be imagined as a transition of aesthetic to be employed in a VR experience to draw the users attention to the concepts of the experience.
Even in more adventurous research projects that challenge conservative music educational practise and research, such as Berlin based 
layer audio 