Sonification sketchbook: Audio Portrait — “REHEARSING SILENCE” (2018)

Speculative prototype for a binaurally immersive medical portraiture

(9 minute audio loop)

For best results, listen on noise-cancelling headphones.

Rehearsing Silence is part audio essay, part medical portraiture, part data sonification, part prosthetic design sketch. It proposes a binaurally-encoded, audio-based approach to portraiture that frames and compresses the gradual and inevitable diminishment of auditory perception as a consequence of aging and neurologically collapsing bodies. This design sketch stems, in part, from ongoing research focused on developing instruments and tools to support multi-sensory (non-visual) data analytics, and a continuing interest in how the effects of aging and sensory impairment manifest themselves as perceptual artifacts within an artistic practice (Claude Monet painted through cataracts, Beethoven composed through tinnitus, etc.).

It also proposes an alternative approach to data sonification in which data is represented as absences, mutations, disfigurements or erasures of a previously whole or intact sonorous entity.

This audio portrait contains simulations of high frequency tinnitus tones and frequency-based hearing loss which are different in each ear. If you currently suffer from tinnitus, listening to this portrait may exacerbate your symptoms if listened to at high volume levels.**

Further Reading:

Begault, Durand R. “The Virtual Reality of 3-D Sound.” In Cyberarts, edited by Linda Jacobson, 79-87. San Francisco: Miller Freeman, 1992.

Eggermont, Jos J. Tinnitus. Springer, 2012.

Gruener, Anna. “The Effect of Cataracts and Cataract Surgery on Claude Monet.” The British Journal of General Practice 65.634 (2015): 254–255. PMC. Web. 1 Feb. 2018.

Lupton, Deborah. The Quantified Self : A Sociology of Self-Tracking. Polity Press, 2016.

Marmor MF. “Ophthalmology and Art: Simulation of Monet’s Cataracts and Degas’ Retinal Disease.” Arch Ophthalmol. 2006;124(12):1764–1769. doi:10.1001/archopht. 124.12.1764

Mermikides, Alex et al. Performance and the Medical Body. Bloomsbury Publishing, 2016.

van Beethoven, Ludwig, and Paul Lewis. “Beethoven No. 3: Sonatas Op. 2, 7, 26, 27 ‘Moonlight’, 54, 57 ‘Appassionata’” (2005), Harmonia Mundi.

Additional field recordings by

Related post: Sonification sketchbook: A sonification model based on variations or mutations of single sound objects?

Sonification sketchbook: A sonification model based on variations or mutations of single sound objects?

What if data were expressed, not as individual and discrete sound events, but as a sequence of variations, mutations, erasures, or distortions applied to iterations of a single ‘sound object‘ (in the Schaeffer-ian sense of ‘musique concrete’)? The medical practice of auscultation could serve as an existing model for this approachThis idea was inspired, in part, by the following data visualization:

‘Giorgia — week fifty-two’:‘Giorgia — week fifty-two'

This is an example from ‘Dear Data’ ( (Links to an external site.)Links to an external site.) — “…a year-long, analog data drawing project by Giorgia Lupi and Stefanie Posavec, two information designers living on different sides of the Atlantic.” For each week of an entire year, they chose a different aspect of their daily lives to track and render as a ‘data drawing’ on blank postcards, and then mail them to each other.
Below is a brief explanation of Giorgia’s data visualization.

In one or two sentences, what story does it tell?

It chronicles one week of “good byes,” “bye byes,” and “goodnights” spoken by the author in chronological order.

Identify the data. What type of data is it?

The data is a combination of quantitative and qualitative data.

Identify the visual variables used.

According to legend on back of postcard:

  • Each element is a goodbye spoken by the author that week, arranged in chronological order.
  • Each element is comprised of the following:
    • A primary shape
    • A secondary downward pointing triangle beneath the primary shape
    • The occasional presence of a small dot at the top right-hand corner of the element.
  • Shape:
    • A set of 2-dimensional shapes are used to represent an array of variations pertaining to how each goodbye was articulated. Here, particular attention is paid to the communicational medium/context/situation in which each goodbye was spoken — i.e., ‘in public’, ‘over Skype/(Google)hangout’, ‘over the phone’, ‘in Real Life’
    • The addition of a downward- pointing triangle at the bottom of each shape indicates that the goodbye contained additional words, such as “good luck!”, “have fun!”, “thanks!” etc.
    • The presence of a small circle at the top right-hand corner of each goodbye shape indicates that physical contact was a part of the goodbye gesture.
    • The asterisk at the end of the 3rd row indicates a ‘missed goodbye’ — i.e. she feel asleep before her boyfriend that night.
  • Colour:
    • The colour of each ‘goodbye’ shape distinguishes the person to whom the goodbye was spoken — i.e. mother, boyfriend, friend, stranger, etc.
    • Each variation of ‘additional words spoken’ (the downward-pointing triangle) is identified by a different colour fill. NB: in some instances, distinctions between variations is hampered by the use of similar hues. For example, ‘have a nice day’ and ‘love you!’ have nearly identical colour assignments.
    • The colour hue of each ‘physical contact’ dot indicates whether the contact was a kiss, a hug or a handshake.
    • Position, size, orientation and texture are not utilized.

How many dimensions being visually mapped?

  1. The number of goodbyes spoken in one week
  2. The location/context in which each goodbye was spoken
  3. The relationship of the person to whom each goodbye was spoken
  4. Variations in the message content of each goodbye.
  5. Textual variations that were appended to each goodbye
  6. The occurrence of physical contact as part of each goodbye
  7. The type of physical contact engaged in as part of each goodbye.

Identify the type of visualization, or methods used.

I think this qualifies as a compound visualization.

Referring to the Venn Diagram for information design, comment on this visualization’s

  • Interestingness
    • Representation of communicational exchanges between people through hand-drawn shapes (rather than computer-generated). As a result, the shapes posses a man-made, artifact-like quality.
  • Integrity
    • The highly personal nature of the visualization is intriguing. On the one hand, this is a personal communication to her collaborator, so accuracy and integrity in the reporting of data is assumed. Yet because the data is of a personal nature, one can’t help but wonder if there was some degree of self-censorship involved.
  • Form:
  • Function:
    • Intriguing to explore how the colours and shapes evolve over through time.
    • Provides clear indication of prevailing and evolving trends in the author’s social exchanges through the week.
    • The entirely graphical nature of this visualization tends to encourage a period of prolonged perceptual engagement.
  • Where does it succeed and where does it fall short?
    • One absent (and potentially insightful) element is some time-stamped indication as to what day and/or time of day within the week each ‘goodbye’ occurred.
    • I wonder what effect the use of position, proximity/grouping would have (as akin to a social network array) in representing the ‘who’ of the goodbyes would have on the visualization’s effectiveness.
    • As mentioned above, a few of the colour/hue choices are too similar  and, at first glance, could lead to mis-interpretation.
    • I could also imagine a more rigorous colour scheme being used — i.e., the colour spectrum (warm <> cool) could be mapped to the degree of social familiarity or intimacy (i.e. strangers assigned cool cool colours, family/boyfriend assigned to warmer hues).
Edited by Richard Windeyer on Jan 31, 2017 at 8:36am

The difference between science and art — a potentially useful explanation

IMG_1666 (1)
Perspectival photo of the spiral staircase at the Fields Institute for Research in Mathematical Sciences, Toronto, Canada 

This explanation comes from a recent conversation between artist/musician Brian Eno and architect Bjarke Ingels (see partial transcription below). It’s an explanation that I’ve been contemplating while devising possible experiments in the field of information/data art and performance (i.e., my dissertation research). Eno’s explanation seems (to me) to be not only insightful, but also approachable and — perhaps more importantly — useful. Not only do I think this because my own research is forcing me to oscillate between the sciences and the arts at ever-increasing degrees of intensity, but because so many artists (myself included), tend to do a poor job of explaining why everyone else should care about what they (we) do. 


“Why don’t we all do science? Science obviously produces results — we all live with those results; they’re all around us. Science is practical; makes good sense, and it’s very inspiring too, it’s not unimaginative or anything like that; it’s not dull. So I started thinking about this question of what do artists do and why do we want artists? Why do we want art? […] I started to think that the difference between science and art was that science was really interested in trying to find out how this world works. If you ask any scientist what they’re doing they’ll say, “Well I’m just fascinated by spectacles, so I just really want to see how they worked,” or “I’m just fascinated by water; I really want to understand water.” If you ask artists what they’re doing they generally tell you to ‘bugger off’ or they don’t really have an answer to the question. They say, “I just like doing it.” So you say, “But why do you like doing it?” And what you realize is that what artists like is to build new worlds of some kind; unfamiliar worlds. They want to build other possible realities and in doing that understand something about this reality; reflect back on this reality […] That was really where things like Music For Airports came from. They came from projecting a future that I hoped to create by building some of the objects that belonged in that future. I thought, if you put the objects there, even though the rest of the future didn’t exist, it would start to grow around those objects…. And it seemed to work.” 

Nevertheless, I remain curious as to other possible distinctions between what science and art do for us. I’m especially interested in explanations which emphasize the complimentary nature of science and art rather than those that continue to promote opposition or divisiveness.    

You can view the entire conversation below. Eno’s explanation begins at 2min:58sec in.

Brian Eno & Bjarke Ingels — ‘On Instruments Of Change’ (Heartland Festival, 2016)

Selected links:

Chalk line futures

Talking about dissertation cycles


The next two+ years of (my) academic life, cast in chalk (courtesy of Dr. Peter Coppin, OCADU).


On interdisciplinary studies in knowledge media and the performing arts

Here is a brief profile interview I did for the University of Toronto’s Knowledge Media Design Institute, one which I think (hopefully) begins to explain my research focus as a doctoral student with the KMD collaborative program.

What department are you part of and what is your research focus?

I come from the Centre for Drama, Theatre and Performance Studies (CDTPS) with a background in music technology and sound design for theatre and performance. My research focuses on how our ability to capture and process certain forms of data (i.e. ’big data’, biodata, etc.), are transforming how artists create immersive and/or participatory theatre and performative experiences.

Why did you decide to join the KMD collaborative program?

I’ve always tried to position myself at the intersections between disciplines. Working this way is not easy, but it’s always rewarding and ripe with opportunities for innovation and cross-fertilization. I first heard about the KMD/CDTPS collaborative program through Dr. Bruce Barton (a former CDTPS faculty member now at the University of Calgary), just after it had been established. Being one of the first KMD/CDTPS students was a little strange at first — in my KMD classes, I usually ended up being referred to as ‘the drama guy’, which I think was actually a blessing in disguise. It motivated me to clarify (for myself) and to articulate (to others) what I thought the most important connections between knowledge media design and theatre/performance are — or should be — and how these disciplines might benefit most from collaborative cross-pollination.

How has KMD impacted your research?

The prerequisite KMD courses helped me develop a solid grounding in the contexts and practices surrounding ‘big data’ and ‘open data’ while providing me with an opportunity to begin researching the methods and practices of contemporary artists specializing in data visualization and/or sonification. KMDI’s Human Centred Design course was particularly valuable in expanding my understanding of the methods and practices employed to cultivate meaningful user participation in the design of products, services or systems and — by extension — theatre/performance experiences.

How is your research related to the KMD collaborative program?

People ask me what contemporary knowledge media design has in common with the traditions of theatre and performance. It’s a great question and while I am not a specialist in the theatre of Ancient Greece, scholars such as Derrick de Kerckhove[1] (McLuhan Program, University of Toronto) and others have attempted to demonstrate how a primary function of the ancient Greek theatre was to provide the predominantly illiterate spectator with models for processing, integrating and communicating their perceptions and experiences of the world as knowledge. Theatre has always served as a form of knowledge media. More specifically though, I tend to view theatre as a kind of ‘reality simulator’ — a place where complex or chaotic situations or scenarios can be conceived, enacted and rehearsed and new personas, behaviours and perceptual models can be prototyped and tested, all in a comparatively safe and controlled environment.

How did you find the transition to KMD and how compatible is it with the goals of collaboration and innovation within Performing Arts studies?

I found it was relatively easy. Sound design and interactive technologies have always been essential components of my work in theatre and performance. Yet, I’ve always been equally interested in exploring how this particular design knowledge could be transferred to other fields, such as the design of sounds produced through a user’s interaction with a physical object, system or environment.

What words of advice would you give a future Theatre and Performance Arts Student who thinks about pursuing further studies with KMD?

If you focus on the places where KMD and theatre/performance overlap and intersect meaningfully, opportunities for cross-pollination and trans-disciplinary insight can be discovered and explored. Part of this involves thinking beyond the more popular definitions of ‘theatre’ (i.e., plays, musicals, dramatic texts, etc.) until you begin to see how most people engage some form of ‘performance’ as part of their daily lives.

[1] for example,see the article Theatre as Information-Processing in Western Cultures by Derrick de Kerckhove

The Digital Voice Lab in Paris

Bodies On Stage: Acting Confronting Technologies conferenceAs June busts out across the European Union, I have just joined fellow Digital Dramaturgy Lab member Sebastian Samur in a hot and humid Paris, France to co-present a paper at the Bodies On Stage: Acting Confronting Technologies conference held at the Université Sorbonne Nouvelle Paris 3. Our presentation — Moi qui parle à moi-même dans le numérique / Auditory Alienation and Liberation for the Intermedial Performer — documents the results of our recent Digital Voice Lab experiments — a component of Sebastian’s ongoing Voice Exchange forum at The Centre For Drama, Theatre and Performance Studies (University of Toronto), in which we explored the creative and dramaturgical potential of actors working with electronic voice modulation (a.k.a. ‘voice mask’) tools — such as pitch-shifting and time delay —to develop new characters or (alternatively) new approaches to performing well-known characters, such as Shakespeare’s Hamlet.

Here is Sebastian explaining our digital ‘voice mask’ research (in French) during a conference break.

Practice-as-Research in the Arts

Practice-as-research book piles
Week 4, year 1 of doctoral studies. You will know me by my piles of books!