Using machine learning to generate abstract art from text

Machine learning can turn language into colour, shape, movement and texture without treating words as simple captions. A prompt about a humid Sydney afternoon, a tram passing through Melbourne fog or waves breaking along the Great Ocean Road can become a visual system shaped by tone, rhythm and association. The result is less an illustration of a sentence than an interpretation of its atmosphere.

This approach sits comfortably within the experimental spirit of NYU’s Interactive Telecommunications Program, where artists and technologists test unfamiliar relationships between people, media and machines. For a showcase such as ITP 30 Show, an AI-generated image can be presented alongside the prompt, model choices and transformations that reveal how an idea became an artwork.

Turning language into visual signals

A text-to-image model converts words into numerical representations, often called embeddings. These representations position concepts near related ideas in a vast mathematical space. “Rust”, “industrial”, “summer heat” and “orange” may influence one another even when the prompt never explains exactly how they should appear.

Artists can exploit this ambiguity by writing prompts with sensory detail rather than relying on generic labels. “A pale, restless coastline under fluorescent cloud” gives a model more emotional and material direction than “abstract beach art”. Repeated generation then exposes variations: one image may favour line, another may emphasise grain, while a third may produce a surprising balance of empty space.

An experimental project can document this process through a visual diary. A showcase journal can help audiences see how a concept develops through tests, failures and unexpected outcomes rather than presenting the final image as a magical, finished object.

Choosing data and model

Different machine-learning systems produce different visual languages. Diffusion models are widely used for text-to-image generation because they begin with visual noise and gradually refine it into a coherent result. Some are trained on broad image collections, while others are fine-tuned on specific aesthetics such as collage, ink, photography or 3D forms.

The training material matters ethically and artistically. A creator should consider whether a model reproduces recognisable styles, appropriates cultural imagery or reflects limited ideas about Australian places and communities. A prompt mentioning a remote Aboriginal community, for example, needs cultural care rather than decorative treatment. Clear attribution and consultation are essential when a project draws on living traditions or community knowledge.

Local references can also make a work more specific. A model might respond to the sharp geometry of Brisbane’s subtropical architecture, the saturated colour of Vivid Sydney installations or the dusty palette associated with an inland road trip. These details create a stronger relationship with an Australian audience than interchangeable images of “the future”.

Directing randomness with prompts

Prompt writing is a form of composition. Artists can combine nouns, verbs, colour relationships and spatial instructions to control the image while leaving room for chance. Useful terms include “layered”, “fragmented”, “low contrast”, “cellular”, “topographic” and “slow movement”, although the meaning of each term depends on the model.

Negative prompts can remove unwanted features, while seed values and aspect ratios help compare variations. A creator might generate a square series for an online gallery, then use a tall format for a projection in a narrow Melbourne laneway. Changing a single phrase at a time makes the visual effects easier to understand and gives the final series a coherent structure.

Practical prompt habits for Australian projects

Making process visible

The strongest machine-learning artworks often show their construction. A screen-based piece could display a prompt as it mutates, allowing viewers to watch language become colour fields and visual noise. Another work might compare a human sketch, a generated interpretation and a final image edited by the artist.

This transparency gives authorship a more nuanced shape. The artist may be responsible for the concept, selection, sequencing, editing and presentation, while the model contributes statistical variation. Listing these roles clearly helps audiences understand that generative art involves many decisions before and after the image is produced. The author directory provides a useful model for connecting projects with the people and practices behind them.

For Australian creators, exhibition conditions are part of the design. Large files need to load reliably beyond inner-city fibre networks, projected colour can shift in bright gallery spaces, and public venues may need captions or transcripts. A work designed for an ITP event should function as an encounter, not just a downloadable image.

Showing work responsibly

Abstract output can still carry bias. A model may associate certain occupations, bodies, landscapes or moods with narrow stereotypes. Reviewing multiple generations, documenting problematic results and explaining editorial choices can turn these limitations into a meaningful part of the project.

Copyright and consent also require attention, especially when using photographs, private archives or identifiable people as source material. Commercial platforms may retain prompts or uploads, so creators should check their terms before submitting unpublished work. In Australia, privacy expectations, Indigenous cultural intellectual property and the local market for commissioned digital art all influence how a project should be developed and sold.

An online archive can preserve more than polished images. Store prompt versions, source sketches, process notes, credits and technical specifications alongside the final work. The project archive offers the kind of durable context that helps experimental pieces remain discoverable after an exhibition or live event has ended.

Use machine learning as a collaborator in a clearly framed artistic process: write precise prompts, test widely, edit with intention and keep a record of each transformation. Publish the resulting series with its methods and credits so viewers can experience both the abstract image and the human decisions that made it meaningful.