How an Artist Uses Voice Recognition to Generate Poetry
At the ITP 30 Show, the online showcase from NYU’s Interactive Telecommunications Program, experimental projects invite audiences to speak, move, play or reshape digital material. Voice-responsive poetry makes a visitor’s utterance the source of a poem that appears on screen.
The creative process begins with a chain of translation: speech becomes a transcript, the transcript becomes a set of words, and software arranges those words into lines shaped by sound, timing, emphasis or chance. The result is more than a recording with captions or a chatbot reply; it is a generative poem assembled from live human voice.
From voice to poetic material
A microphone captures more than language. It records room tone, breathing, laughter, accents and the length of a pause. An artist can preserve these features as data, letting the poem react to volume, rhythm or silence. A clipped word might trigger one line, while a pause might open a space between lines.
Automatic speech recognition, often shortened to ASR, turns captured audio into text and can attach timing information to each word. The artist can treat that transcript as a word pool, removing common terms, repeating sounds or grouping words by grammatical role. Some systems preserve the speaker’s sequence; others rearrange it so the voice contributes texture rather than a straightforward message.
This becomes especially vivid in Australian cities. On trains through Sydney and Melbourne, or in a busy Melbourne laneway café, a phone microphone hears announcements, traffic, cups and conversation. Voice input is familiar through dictated messages and notes, yet a public artwork must account for ambient noise that a personal phone partly controls.
Designing the recognition system
Recognition is only one layer. The artist may select words through sound similarity, emotional tone, syllable count or a language model trained on poetry. Curated rules can create formal patterns, while generative models can surprise the performer. A well-designed system can make each choice visible: a line changes because a word was too loud, recognised with low confidence or spoken at an unexpected moment.
Explaining that chain is part of the experience, not backstage machinery. The ITP journal’s account of participatory data visualisation shows how interface framing changes what an audience thinks its contribution means. A waveform, highlighted words or a confidence score can make an invisible process legible without turning the poem into a dashboard.
Automatic speech recognition can also perform unevenly across Australian accents, code-switching and multilingual speech. Testing with local voices matters. Participants, performers and community advisers can expose errors before an installation opens, allowing the artist to adjust the microphone, vocabulary or fallback text.
The role of restraint
A voice poem needs silence as much as spectacle. If every word arrives with animation, sound effects and a crowded interface, the spoken contribution loses presence. A restrained screen may show one line at a time in plain type, with a recording waveform receding into the background. The poem then feels discovered rather than packaged.
The ITP journal’s reflection on minimalism in digital art offers a useful lens for this choice. Minimalism can direct attention to timing, scale and the physical display, but restraint must suit the material. A near-empty frame can make a breath feel enormous; a dense field of text can turn overlapping voices into a chorus of fragments.
The presentation also needs an accessible counterpart for an online audience. Captions, transcripts, audio descriptions and a plain-text version preserve the work for viewers who cannot hear the room or read small type. These alternatives should feel authored rather than appended because each changes the pace and emotional emphasis of the piece.
Consent, copyright and public space
When the work collects voices in Australia, data protection belongs in the creative process. For a covered organisation, the Privacy Act 1988 and Australian Privacy Principles govern the collection, use, storage and disclosure of personal information. A recording can identify someone even when no name is requested, so an “anonymous” label may not describe the data accurately. Clear notice, informed agreement and a retention policy reduce that uncertainty.
Copyright raises separate questions. Under the Copyright Act 1968, a captured performance can attract protection as a sound recording, and a written transcript may qualify as a literary work. Participants may also expect their voices to be recognised as creative contributions. Credits, permissions and reuse terms should distinguish the artist’s software, the participant’s words and the recorded performance.
An installation can minimise exposure by recording only when a participant activates the microphone, deleting raw audio after processing and storing only the poem. Australian festivals add another layer, as queues, children, workers and bystanders may enter a marked area. Clear signage and an option to contribute without preserving audio help keep participation voluntary.
Showing a voice poem online
The online showcase gives the process a durable public life. Its project page can explain the artist’s concept, list technical and creative credits, and pair the work with event information and RSVP details. Viewers can understand whether the poem changes by speaker, encounter related programme resources such as the scholarship and census, and plan a demonstration without attending the live installation.
That format suits Australia’s cultural market, where independent galleries, public museums, festivals and residencies connect experimental digital practices with physical audiences. Melbourne’s café culture and Sydney’s public spaces already make ambient sound part of urban experience, while livestreams and headphones can carry a voice work into domestic settings. The tension between a communal room and a personal screen becomes part of the artwork’s meaning.
Documentation should preserve how the piece behaves over time. A recorded example, source-code notes and a short statement about recognition errors let researchers, curators and future collaborators study the system. The archive should distinguish a final generated poem from the audio and personal data used to make it.
Explore the ITP 30 Show project pages, follow the listed events and RSVP openings, and experience a live voice becoming a lasting digital poem.