Deciphering Space in Virtual Reality

Space, and its perception, is fundamental to navigation through the world. Whether through sight, sound or touch we build mental images of how we relate to the world around us. So, to better understand the possibilities and pitfalls of design for virtual reality interfaces a thorough understanding of spatial cognition is advised. The following article is a literature review, many sources have been copied as is so beware of plagiarism if you use these words. Plus why rephrase something that is already very clear?

William Gibson Visions of future worlds

In Neuromancer, Gibson envisioned highly complex structures like a digital hallucination, overwhelming and alienating. Gibson coined certain phrases that describe certain interactions with game systems, how a space is ‘consensually hallucinated’ in a ‘non-space of the mind’.  Given user centered developments in HCI and transparent natural user interfaces (NUI), graphic representations now would conceal this Gibson’s complexity and to visualise the underling systems no standard geometry and dimensional space could accurately grasp this vision. Why would we want to see everything? As  Aarseth (2007) describes in Allegories of Space, Gibson’s vision is highly perceptive, bleak and ironic but this irony is missed by many futurists that it inspired. Additionally in this essay, Aarseth laid out how structurally different, games are as a new medium. In this capacity they must engage a player through new structures of interaction, by breaking with familiar forms of representation. But to what end does this enhance experience, and on a subtler point what should be the goal of these new mediums? How can they address shortcomings in previous experiences, and the consequences of their success; such as addiction and excessive emphasis on violent interactions.  Aarseth goes on to define spatiality in games: 

"The defining element in computer games is spatiality. Computer games are essentially concerned with spatial representation and negotiation, and therefore the classification of a computer game can be based on how it represents or, perhaps, implements space. More than time (which in most games can be stopped), more than actions, events and goals (which are tediously similar from game to game) and unquestionably more than characterization (which is usually nonexistent), games celebrate and explore spatial representation as a central motif and raison d’être. In terms of playability, themes, tasks, subgenres and dramatic structure, nothing much has changed over the last two decades. The innovation takes place in spatial representation, and the genre’s more slowly evolving complexity in the other areas (such as physical simulation) can be seen as a result of the increasing complexity of the spatial representation." 

– Aarseth, E. (2007). Allegories of space.   Space Time Play , 44-47.

Together this proposes how games can be used as mediums of experience in themselves, what knowledge can they convey? This then poses the question of how we define space, beyond a physical abstraction of geometry. What makes a social space? How do we generate regions of space for different purposes. How do computer mediated spaces differ?

“Rilke wrote: ‘These trees are magnificent, but even more magnificent is the sublime and moving space between them, as though with their growth it too increased.”

– Bachelard,   Poetics of Space,   Beaccon, 1958, p. 201

In the Poetics of Space Bachelard is, in part, inviting designers to re-consider how users’ experience by encouraging reference to formative, intimate spaces rather than formal abstractions. The quotation of a quotation above is useful as it prompts us to consider game spaces on two fronts – poetically (‘the sublime’: what does this space invoke? ), and kinetically (‘moving space’: reminding us that, above all, we are concerned with dynamics and locomotion ). Both these aspects are useful, as they allow us to consider and assess qualities of virtual-spaces that side-step ‘newer must be better perspectives’.

Ecological Perspective

[space is] neither a mere ‘frame’, after the fashion of the frame of a painting, nor a form or container of a virtually neutral kind, designed simply to receive whatever is poured into it. Space is a social morphology: it is to lived experience what form itself is to the living organism, and just as intimately bound up with function and structure. 

– Lefebvre, The Production of Space

This poetic quote summarises the nature of human relationship to space, a view that was built upon by James J. Gibson. James Gibson’s theory of direct perception states that the environment contains all of the information needed to specify its properties. Hence,perceiving these properties is a matter of detecting the information available in the environment.  This concept is significant precisely because it allows James Gibson to locate the moving, perceiving body at the heart of meaningful perceptual experience; an experience which emerges in the dynamical structures that cross the body and the world.  His theory of environmental affordances shows that perception is more than a means of passively representing the intrinsic physical organization of objects. Perception is inherently active and exploratory, we seek out alterations in the vast flow of information enveloping it. These alterations are detected when the perceiver moves through the environment and probes it with a pair of glancing eyes.  Locomotion opens up new possibilities for the pick-up of information specific to the perceiver’s environment. This information can then be used to guide subsequent movements, as in a perception-action loop (the two being inseparable). As David Morris puts it, “we do not … perceive naked properties of the environment, rather we perceive what the environment affords to our bodies, what we can do with, or in the environment.” The theory of affordances demonstrates that the structures of information are intrinsically meaningful for perceptually-guided action.  Affordances are not subjective valuations superimposed on sensations (as theories of perception typically presuppose). Rather, the affordances of the environment are directly perceived as structural information in the environment. 

– Braun, The Structures of Perception: An ecological perspective, 2008

Of all the terms we use to describe the world we inhabit, [space] is the most abstract, the most empty, the most detached from the realities of life and experience. Biologists say that living organisms inhabit environments   not space, and whatever else they may be, human beings are certainly organisms.

– Tim Ingold, Against Space: Place, Knowledge, Movement

Acoustic Spaces

“perception must be understood as a relationship between environmentally available information and the capacities,sensitivities, and interests of a perceiver.” 

– Eric Clarke, Ways of Listening: An Ecological Approach to the Perception of Musical Meaning

To use ecological perception theory to determine musical meaning, Clarke identifies properties of musical sounds that “afford” certain meanings. Affordance is a term coined by Gibson to describe the purposes or uses of various objects as communicated by stimulus. As an example, a chair affords sitting,taming lions, and knocking down bad guys. Clarke uses this concept to show how listeners can glean meaning from a musical performance. Note that he either stipulates a specific listener and a specific performance, or gives ranges of possible meanings dependent upon the context of the performance and the listener.

Interesting Links not discussed in detail:

Leap Motion Blog post about VR OS

Harrison and Dourish’s Work on Collabritive spaces

Virtual Reality Music pt 1: A wee taster…

In the start of a likely lengthy series of posts, I’ll be presenting some of the internets occurrences in VR music making/experiences. This set of posts will run in parallel with a "Tangible Music Interfaces" set of posts that will cover alternative interaction methods with audio and music applications. For this first one I’m being generous and sharing two things that caught my attention lately.

Squarepushers 360 music video for Stor Eiglass, is a journey through a bubblegum world of bizarre 8 bit escapism. It highlights the possible dangers of VR addication and is also hilarious/scary. Reminds me abit of Osamu Sato’s early psychadelic computer game Eastern Mind: The Lost Souls of Tong-Nou.

Next up is a bit of research form the NIME community on a hierarchical live looping VR music app, DrilleThis technique consists in creating musical trees whose nodes are composed of sound effects applied to a musical content. The system presented requires considerable training and learning to be able to perform with it though the basics of user control can be learned quite quickly. The first video shows a good overview of the system while the second video explores a more structured performance. Further research indicates how metaphors of interaction are utilised in the creation of user control and system design. 

Sound Ideas

Virtual reality and interactive media applications are a great testing ground for novel audio applications. Bizarre control dynamics and immersive feedback stimulus make for an engaging method of challenge and reward. But a question of what areas of sound are to be manipulated remains open, normal synthesis algorithms are build on the morphological relationships related to their early construction… they were made for keyboards and circuits. With natural user interfaces it is possible to explore more physical relationships to sound control. Along this line I found some nice visuals and sounds of physical acoustic relationships. Whether these can be implemented in a controllable and rewarding fashion is another question…

Tangible Music Interface pt 1: Cubes!

Nice cubes eh, menger

I really like cubes, no really. This first post is just a statement really, the future posts in this title thread will cover a variety of tangible musical interfaces, such as Reactable. Though working in VR for the current project the user research and design considerations in tangible interfaces share alot of crossovers. In this series of posts I will hope to determine how these areas can be mutually informed. But for now I want to look at some cubes.

Cube Illusions

First up is a Rubiks cube puzzle sequencer that uses computer vision based markers as a means of controlling the underlying musical structures. I really like this playful and mysterious approach to discovering musical elements. Check out his other work and talks, really interesting chap.

This is pretty fly

Some WebGL graphics stuff

AUTOMATParticularly like the exposing of the mesh underneath in this piece,  use your gyroscope or mouse to interact with it. Giving me ideas for objects to spontaneously appear in VR experience, representing a metaverse of acoustic objects, as the become used in a context: making a phone call, streaming a tune, playing with something.

Waves Wave simulator that made me think again about using basic acoustic formula for control of 3D graphics for reactive audio. Fished out my Kuttruff Room Acoustics book and subsequently melted my brain.

JavaScript: If you fancy playing with this sort of stuff go to PlayGnd (based on three.js), or just look through the archives with slinky code overlay. If thats abit softcore for you head over to WebGL Fundamentals and get all detailed. Also a fully speced WebGL and js library is that of Babylon.js that supports super easy Oculus Rift integration, check out their playground for examples and browser based IDE. 

Data Visualisation

More trawling results! Here is a few interesting data visualisation techniques and representations. Finding these quite stimulating for the idea of how a human would have to navigate extremely complex data structures. By simplifying the complexity through visual algorithms a form of order can emerge that is almost tangible and readily usable. Hope to integrate some of this work as an influence to the virtual reality interface in the project.  

The example above is particularly mathematics heavy once you inspect the linked paper, imagine this one will just have to be used as influence rather than direct transcribing…  The visual designer’s interpretation and implementation for this project is phenomenal, after inspecting the paper and watching again I am in awe.

This rhizome navigation example reminds me alot of minority report style interfaces with circular information modules utilising layered detail and selectable depth. Unfortunately the source code for the project is no longer available, but in principle a similar node architecture could be mocked up using Prefuse or Processing, trouble is migrating this into Unity. Music is proper hootenanny aswell!