A team from UVM gets help from the Northeast Cybterteam to develop an AI-enabled tool for creating digital artworks.
Reporting by Helen Hill for MGHPCC
Jenn Karson is an artist who works with concepts of mapping —the organization of space—as well as scoring—the organization of time. Her artworks take the form of intermedia, sculpture, sound works, programming, and installation. She is also a lecturer at the University of Vermont's Department of Art and Art History and oversees the College of Engineering and Mathematical Sciences digital fabrication lab (the UVM CEMS FabLab).
In 2018 Karson found herself creating a series of collages exploring ideas of visual hierarchy. Originally inspired by the architecture of the Old Mill tower on the UVM campus - the octagon forming its base and the eight triangles forming the facets of its roof - Karson began to realize the images she was creating, and the decisions that led to them, felt a lot like some kind of algorithm at work in her head.
Dancing Stars: Old Mill Tower
Jenn Karson, 2018
24″ x 18″
mixed media
Building on this idea, Karson began musing on what machine learning might bring to her process and so her UVM Art and AI Initiative was born.
"We are exploring approaches to artistic image production, comparing the results of StyleGAN and Genetic Algorithms," explains Karson. "More broadly, we are exploring emerging artistic practices with Machine Learning and AI while referencing an artistic lineage to artists such as Wassily Kandinsky, John Cage, and Yoko Ono; artists who employ(ed) instructions and systems in their non-digital artworks. For example, Kandinsky distinguished systems and developed a science of aesthetics with the basic elements of point, line, and plane; Cage used the oracle 'I Ching' like a computer to inform his compositional decisions; Ono writes poetic scores that turn her audience into active participants when they follow a series of imaginative instructions. Through this ongoing research and practice, we intend to join the larger conversation about art and AI and design new curricula for UVM undergraduate students."
Work began in February 2020 with a team that has so far included five UVM students: two graduate students in data science, an undergraduate mechanical engineering student, and two undergraduate art minors, as well a UVM alumna who did her undergraduate degree in Studio Art and is now at Parsons School of Design.
StyleGAN is a generative adversarial network (GAN), an unsupervised machine learning tool able to automatically discover and learn patterns in input data in order to generate new examples with characteristics indistinguishable from those of the original. Developed by NVIDIA it is notable for its disturbing effectiveness when used to generate lifelike artificial faces
Recognizing it as a fascinating tool for communicating the otherwise intangible, latent aesthetic information or "style" encoded in a piece of artwork, the team, after first digitizing the collection of physical "source" artworks Karson had created, initially turned to a genetic algorithm as a tool to expand the number of training images.
Based on Processing, an open-source language, and development environment built on top of the Java programming language, the team tailored their genetic algorithm (based on one built by Daniel Shiffman) to evolve the arrangement of individual shapes, pulled randomly from the original dataset, allowing a user to flag "pleasing" combinations according to their own preference. The manually chosen selections are then added to the initial training set allowing the algorithm to continuously learn the user's taste while also growing the collection of artworks informing the algorithm.
Once armed with a sufficiently large dataset, the expanded image library can be used to train the GAN to create more and more compositions.
The genetic algorithm is small enough to run on a laptop but evolving hundreds of images, through thousands of steps required by the StyleGAN2 experiments takes a lot more computing muscle and for this, the team uses UVM's Vermont Advanced Computing Core (VACC) and RunwayML but look forward to writing their own bespoke GANs in the future. "Our focus is not necessarily on conventional imitation of the original artworks but on the creation of novel images: Trying to find a compromise between physical and digital practice in art," explained Ethan Davis, a computer science masters student who has a background in fine art .
StyleGAN2 has yielded some delightful surprises for the team in terms of their response to many of the images that emerge.
"Something really interesting happens with images starting to look almost lifelike, like vegetation or even cellular structures," Karson says. "Other things which, as an artist, are really engaging and interesting visually are the latent spacewalks which seem to be suggestive of something almost multi-dimensional."
She says, "I am delighting in how machine learning has helped me to move into this more organic space."
To date, the Art and AI project has been featured in the Art, Robot, and Virtual Creature Gallery at ALife 2020 and the 2020 Mind Candy Small File Media Festival.
The team is currently in conversation with the Fleming Museum of Art at UVM about some kind of exhibition and is also collaborating with choreographer Paula Higa to film a short film this month (to be presented at dance festivals) that incorporates the images made over the summer as projections and digitally fabricated objects.
Finally, the team will be sharing their work as part of a public online series with the Vermont Studio Center on December 17
In the meantime, you can hear Jenn (and the students she has been working with) speak in more depth about this project in their ALife video in which she highlights the "artificial life" emergent in the AI-generated images.
The team is particularly grateful for advice and support from Northeast Cyberteams mentor Adrian Del Maestro a professor at the University of Tennessee, Knoxville.
Jennifer Karson
Jenn Karson is a lecturer at the University of Vermont’s Department of Art and Art History and the director of the College of Engineering and Mathematical Sciences digital fabrication lab (the UVM CEMS FabLab). Additionally, she is a curator of exhibitions, programs, and curricula that cross the disciplines of the arts, technology, fashion, and architecture. She received her MFA in Design + Technology from the San Francisco Art Institute and has artworks exhibited in the United States, Canada, and Europe.
