First in Foundations
When Jordan Wirfs-Brock and I started exploring multi-sensory data representations earlier this year (i.e. using the 5 senses to encode data), we revisited Observe, Collect, Draw, a data visualization sketchbook by Giorgia Lupi and Stefanie Posavec, and expanded the exercises to include the other 4 senses. Being gourmand and all, I immediately took to taste. With each prompt, I started by asking myself: how can I translate this data into taste? I’ll present you with my first foray with four tasty portraits from my favorite artist, Nakajima Tsuzen.
Each subsequent exercise led to more questions, though unlike the show, Lost, it led to insights as well.
I chose to keep the visual cues associated with the prompt not just for the purposes of this exercise, but as I believe in multi-sensory data representations, one sense is not enough. In this sketch, I chose sight & taste.
Let me transcribe and clean up my explanations more.
Much better.
The artist's whose work and colors I analyzed is Nakajima Tsuzen, a Japanese hangiga (woodblock) artist, and the legend is right of the dotted line and. If those constraints weren’t enough, I decided all meals have to be Japanese. Luckily I know a lot about Japanese home cooking, so it was more like a fun memory and creative game.
Here's some questions that came up, and that Jordan and I plan on exploring:
How many steps can we (or should we) take as the translator? How many liberties?
What other strategies can be used to invoke taste?
My translation involved steps. For example, I took the color blue to be synonymous with the ocean and thus the ingredient would have to come from the ocean.
The senses at last!
Data visualization isn’t just limited to graphs and images. Data “visualization” has a history of expanding to senses such as in sonification, where data is represented by sound and to the physical realm, such as The DataPhys project by Evandro Damião, allowing blind people to interact with data through sound (narrating the data and subject) and touch (being able to feel the ups and downs of the a plastic area chart). Throughout this newsletter I will default to the term data visceralization to describe projects such as the latter: “Visual things are for the eyes, but visceralizations are representations of data that the whole body can experience, emotionally as well as physically,” as Klein and D'Ignazio explain it in Data Feminism.
Translations of Threatening Tastes
A data visualization designer makes choices, accentuating one aspect of the data set or variable that stuck out to them in their analysis. We like to use the analogy of translation, but it’s not without its problems. While translating between languages clings to the same medium, translating data poses different problems: these numbers have been simplified from their original form and numbers. Translating to visuals thus captures one perspective of the data, but visuals don’t tell the whole story, nor are they accessible to all (and neither is focusing on one other sense, for that matter). In fact, even when you “just” use visuals, it is not the only sense at play - other senses are invoked, albeit at smaller ratios. No sense exists in isolation and I believe that evening out the sensory ratio to express data can create a more immersive and impactful experience. A multi-sensory approach is the best we can do in order to be inclusive, and to make for a richer experience. And so we get to the heart of the newsletter: How does considering data visualization -- or data representation for a more encompassing term -- as translation bring new dimensions, ideas, and experience to the work?
Jordan and I embarked on a collaboration exploring multisensory data representations in order to explore this theme: how can we sonify motion or taste a timeline? Although we don’t have solutions to these questions (and no one solution rules them all!), we are forging our own path with our curiosity and passion for this topic.
In the week where we set out to develop sensory encodings of AQI colors, our taste approaches were widely different. While Jordan took inspiration from her pantry choosing fruits and vegetables based on the AQI color scale and arranging them appropriately then ate them, I thought of an analogy of burnt toast: as you move down the color AQI scale the toast gets more and more burnt until it’s “cancerous,” as my cousin often deems toast too burnt to eat. As Jordan noted, my answer was made for taste, imagining the experience and memory of burnt toast, and Jordan’s was in taste, engaging in the sensory experience during the exercise. Similar to choosing a line chart over a bar chart, each of our submissions represent a different translation of the data, and a different story.
Merlin Missions
This newsletter is meant to narrow in and expand on some of our explorations, research, and prior work on sensory translations and will end around the time where we hope to launch our first multi-sensory data representation workshop (late spring/early summer of 2022). I like to have phases, as they give me structure, so I’m planning on 2-3 phases for this newsletter:
Phase 1: Exploring sensory data
We’ll dive into the overwhelmingly broad category of the senses, which could include one to all of the following: my experimentations, data and non-data related examples, thoughts, how to collect sensory data, and exercises to follow along. This phase will last 5 articles, covering the five senses, and is meant to provide necessary background and get those creative juices going.
Phase 2: Representing multi-sensory data
We have our building blocks and we are ready for action. Working with a dataset, I will go into unknown territory and build a multi-sensory experience using a dataset (that’s the goal, at least). You’re welcome to work alongside me.
Phase 3: Translating sensory data
If we’re still alive and kicking, we’ll take inspiration from Douglas Hofstadter to explore translating between the senses.
If you can’t wait two weeks and are eager to learn more, here’s two prompts to get you started. Feel free to answer one or both.
Have you collected data on your senses before? Food you ate, number of birds you saw etc. What were you hoping to learn from this data?
Would you say that you have a dominant sense? If so, which one?
Feel free to respond with your answer to this email, or on twitter with @max_graze and #datathesenses
Until then, fellow data sensory explorers!
Note: I am a nerd and I couldn’t resist taking inspiration by the Magic Treehouse to name the sections. No, there isn’t a direct link to multi-sensory experiences although I’m sure anyone with a creative bone could forge one.
A newsletter about sensory sketching, and representing data with all our senses.