Simulating Synesthesia: A Project That Turns Sound Into Colour

The ITP 30 Show gathers experimental work made by people connected with NYU's Interactive Telecommunications Program, and this year's online showcase includes a piece that goes further than most interactive installations in terms of what it borrows from the body. Rather than reacting to movement or sound with visual effects, the project borrows the wiring of a synesthetic nervous system, where hearing and sight share the same channels.

The work, presented with project descriptions, author credits and event details, invites visitors to speak, hum or tap a surface. Their voice is analysed for pitch, breath and rhythm, then translated into shifting colour fields and layered drones that seem to answer back. Nothing about the mapping is one-to-one, which is precisely the point: a genuine synesthete experiences colour as an involuntary echo of a sound, not as a label attached to it.

The idea behind the piece

Synesthesia is estimated to affect a small but real proportion of the population, most commonly in the grapheme-colour form, where letters, days of the week or musical notes reliably trigger hues that no one else shares. The project takes that private consistency as its starting point and asks what happens when the wiring is externalised into a room full of strangers.

Visitors discover quickly that the system is not a mood lighting rig. It responds to texture, not only volume: a whispered phrase produces a thin, translucent wash, while a shouted vowel sends dense saturated bands across the wall. Two people talking at once create interference patterns that neither triggered alone.

The conceptual wager is that shared perception can be staged without becoming a novelty trick. If a hundred people contribute a single syllable across a day, the installation accumulates a memory of the room in colour, which is then folded back into the next visitor's response.

How the simulation is built

Technically, the piece runs on a microphone array paired with a touch surface, feeding a Max/MSP and WebGL pipeline. Frequency data is mapped onto hue, amplitude onto saturation and density, and phase differences between two channels drive the horizontal drift of the image. The soundscape is synthesised rather than sampled, so every visitor generates a tone that never existed before.

Latency is the design constraint that matters most. Above roughly thirty milliseconds, the colour stops feeling caused by the voice and starts feeling like a parallel event happening nearby, which collapses the illusion. The developers tuned the chain on the actual room rather than on a laptop, since a gallery's hard surfaces and air conditioning change everything about how a voice arrives at the microphone.

A second layer builds on the first: each session is recorded as a short score and queued for the following hour, so the sound you hear may be a visitor from the morning session. The installation performs its own past.

Why simulated synesthesia is difficult

Real cross-modal perception is involuntary and stable across decades. Any simulation has to choose rules, and the choices are never neutral. A piece that maps loud sounds to red because it reads well on a projector has quietly become a light show, and the difference between spectacle and sensation is easy to lose.

The project's documentation is unusually careful here, listing which behavioural research informed each mapping and where it deliberately departs from clinical description. Where the mapping is arbitrary, it says so.

Voice data raises the other hard question. Visitors are asked to surrender the one biometric nobody can change, and the team had to decide whether to store anything at all. The current build keeps audio in memory for the length of a session and writes nothing to disk.

Showing the work in a physical room

Digital work in a gallery inherits every practical problem of the medium: daylight, projectors that read as plastic, phones whose torches betray the image. Sydney and Melbourne venues often fill with visitors who expect their own screens to be the exhibition, and a piece asking for sustained attention competes with the ambient photography reflex.

The journal thread on exhibiting digital art physically covers the unglamorous logistics that shaped the installation's final form, from calibrating a projector against a curved wall to deciding where a queue should stand so it does not block the sound.

Where sound and code meet locally

Australian venues are unusually opinionated about volume. Inner-city residents in places like Newtown and Marrickville have complained about amplified bass for years, so a piece built on drones and low frequencies has to negotiate a local tolerance that European gallery spaces often do not.

That constraint produced unexpected choices: the piece runs quieter than a club and more legibly than a museum, with directional speakers aimed at the interaction zone. Festival programmers in Melbourne and Sydney have seen similar compromises at events such as Vivid and AURORA, where visual work and sound work are forced into the same brief.

The wider argument about craft sits in the journal piece on sound meets code, which treats the composer-engineer relationship as the real content of a sonic installation rather than a production detail.

Everyday rhythms, weather and local rules

Daily life in Sydney or Brisbane supplies the baseline the installation has to out-compete: air conditioners, thunderstorms rolling in off the coast, the low murmur of a cafe at eight in the morning, tram doors and ticket taps on the way to work. Australian summers also mean humidity, which softens projection and shifts how high frequencies travel through a room.

Regulation shapes the visitor experience more than visitors expect. The Australian Classification Board handles ratings for interactive material, and the recent restrictions on social media access for under-sixteens matter for any installation that invites people to upload their own voice or images to a shared server.

For producers, the funding picture runs through Creative Australia and the Australia Council, while the local market for boutique sound design and small-scale AV hire in Melbourne and Sydney remains competitive enough that technical teams are usually assembled from freelancers rather than in-house staff.

Building your own cross-sensory piece

The show's most transferable lesson is that fidelity to the phenomenon matters more than visual polish, and that a small number of carefully tuned rules beats a large number of spectacular ones.

The showcase runs online with full project descriptions, credits, event details and RSVP information, and the programme also publishes a scholarship and a census of participants, which makes it easy to see how others approached the same problems. Reading the entries, visiting the installation if it travels to a venue near you, and registering for the online programme are the quickest ways to see how much a room can be persuaded to feel.