- People
- Administration & Staff
- Research faculty
- Gabriela Aceves-Sepúlveda
- Alissa N. Antle
- Sheelagh Carpendale
- Parmit Chilana
- Jon Corbett
- Steve DiPaola
- Halil Erhan
- Brian Fisher
- Diane Gromala
- Marek Hatala
- Kate Hennessy
- Alireza Karduni
- Sylvain Moreno
- Carman Neustaedter
- Will Odom
- Philippe Pasquier
- Niranjan Rajah
- Bernhard Riecke
- Gillian Russell
- Thecla Schiphorst
- Chris Shaw
- Wolfgang Stuerzlinger
- Ron Wakkary
- Ö. Nilay Yalçin
- Teaching faculty
- Emeritus
- Adjunct Faculty
- Alumni
- Work at SIAT
- Opportunities
- Research
- Programs
- News & Events
- Spaces & Equipment
- StudioSIAT
- Media
- Showcase
- Contact
- Staff & faculty resources
Research
Researchers present groundbreaking works in creative AI
School of Interactive Arts & Technology (SIAT) faculty, students, and alumni are revolutionizing the use of generative AI and computational creativity, demonstrated by their significant presence at this weekend’s prestigious Conference and Workshop on Neural Information Processing Systems (NeurIPS).
The conference is held this year from December 10th-15th at the Vancouver Convention Centre and SIAT faculty, students, and alumni are presenting numerous papers, demos, and artworks at the conference.
Researchers from SIAT’s iViz Lab were well represented at the conference with projects and papers by lab director and SIAT professor Steve DiPaola, alum and SIAT instructor Rafael Arias Gonzalez, and PhD student Vanessa Utz.
At a juried booth this week,the iViz lab researchers showcased their aesthetic movement transfer project, cognitive memory large language model (LLM), and research on the implications of AI and climate change.
Professor Philippe Pasquier and researchers at the Metacreation Lab for Creative AI, including PhD students Keon Ju Lee, Arshia Sobhan, and Ahmed M. Abuzuraiq, also present projects on AI in music and several papers at this weekend's Workshop on Creativity and Generative AI.
Explore the artworks, projects, and papers below.
Workshops and demos
Forrest Dance
By Steve DiPaola
Forrest Dance is an art video that uses advanced AI movement transfer techniques. The art piece speaks to how trees wake each morning and dance, becoming pure light dancers by nightfall. At its core, Forrest Dance is a piece about the environment and nature—how all beings have consciousness in their own way.
The video uses state-of-the-art video techniques that transfer movement from live dance performances to new art forms (dancing trees, leaves, light spirals) using IP-adapters to transfer movement from one object to another.
DiPaola and his collaborators are interested in using works like this in school classrooms to educate students, in a fun way, about both AI and nature and the climate.
“We are looking to setup a school program where young students can watch art videos like this and both learn about the world of nature and seeing nature through art, even this new for of art,” says DiPaola.
After discussing issues like climate change, students would be asked to get up and, seeing themselves as trees, move and video one another.
“We would supply our software tools so they could select a few videos to then process themselves during class with our AI movement transference software into moving trees similar to our art video.”
AI Modeling Thought & Language
By Steve DiPaola, Rafael Arias Gonzalez, Nilay Yalçın, Maryam Ahmadzadeh
Rafael Arias Gonzalez, an alumnus from SIAT's master’s program, demonstrated his cognitive memory large language model (LLM), a project that he collaborated on with DiPaola, Assistant Profesor Nilay Yalçın, and PhD student Maryam Ahmadzadeh.
Interacting with the chat bot, individuals can talk as faithfully as possible to historical figures like Vincent Van Gogh. Gonzalez and his collaborators used Van Gogh’s life events, as well as 800+ letters that he wrote to his brother Theo, to bring his memories to life.
Users can talk with the fully 3D-animated Van Gogh chat bot while he recounts real events and thoughts from his life.
eWaste/Climate Change
By Vanessa Utz, Steve DiPaola
Utz and DiPaola collaborated on research examining the ethical implications of AI, particularly its effects on climate change.
Last week, Utz presented their work on the climate implications of rapidly developing digital technologies, such as blockchains and the associated crypto mining and NFT minting. In their work, they report on the growth of diffusion-based visual AI systems, their patterns of use, growth, and the implications on the climate.
Revival: Collaborative Artistic Creation through Human-AI Interactions in Musical Creativity
By Keon Ju Lee and Philippe Pasquier
Keon Lee and Philippe Pasquier presented their work on musical agent systems that collaborate with human musicians in real-time. In Revival, a live audiovisual performance created in collaboration with VJ Amagi, they demonstrate how AI can seamlessly blend with human artistry, offering a glimpse into the future of collaborative music and visuals.
The innovative live audiovisual performance and music improvisation blends human and AI musicianship to create electronic music with audio-reactive visuals. The performance features real-time co-creative improvisation between a percussionist, an electronic music artist, and AI musical agents.
Papers
Autolume 2.0: A GAN-based No-Coding Small Data and Model Crafting Visual Synthesizer
By Arshia Sobhan, Ahmed M. Abuzuraiq, and Philippe Pasquier
Abstract: Autolume is a no-code, generative AI tool that leverages the artistic potential of Generative Adversarial Networks (GANs). It empowers artists to train and manipulate their own models using small datasets, thereby fostering unique aesthetic exploration and personal creative control. By streamlining the full creative workflow—covering data preprocessing, model training, real-time latent space navigation, and output upscaling—Autolume makes the artistic potential of generative AI accessible to non-technical users. It also supports interactive applications, such as audio-reactive visuals, through OSC integration. As a free and open-source platform, Autolume expands creative possibilities for artists, enabling them to blend and customize aesthetic styles in a flexible and efficient manner.
Musical Agent Systems: MACAT and MACataRT
By Keon Ju M. Lee and Philippe Pasquier
Abstract: Our research explores the development and application of musical agents—humanin-the-loop generative AI systems designed to support music performance and improvisation within co-creative spaces. We introduce MACAT and MACataRT, two distinct musical agent systems crafted to enhance interactive music-making between human musicians and AI. MACAT is optimized for agent-led performance, employing real-time synthesis and self-listening to shape its output autonomously, while MACataRT provides a flexible environment for collaborative improvisation through audio mosaicing and sequence-based learning. Both systems emphasize training on personalized, small datasets, fostering ethical and transparent AI engagement that respects artistic integrity. This research highlights how interactive, artist-centred generative AI can expand creative possibilities, empowering musicians to explore new forms of artistic expression in real-time, performance-driven and music improvisation contexts.
Rethinking Artificial Intelligence creativity and ideation systems
By Steve DiPaola
Abstract: The visual and media arts community faces a significant challenge from the rise of generative visual AI, which poses a threat to their livelihoods. The current prompt-based generative AI (genAI) systems, such as Stable Diffusion, Midjourney, and DallE, are limited in their ability to provide sustainable financial opportunities for artists and designers, while also producing output that homogenizes unique cultural perspectives and styles. The importance of this topic lies in the cultural value that artists and designers bring to society, serving as a medium for expressing and preserving diverse traditions, beliefs, and values. We discuss a new approach to genAI and revised technological direction, as the current trajectory of AI development has the potential to displace human creators and devalue their skills.
DreamLLM-3D: Affective Dream Reliving using Large Language Model and 3D Generative AI
By Pinyao Liu , Keon Ju Lee, Alexander Steinmaurer, Claudia Picard-Deland, Michelle Carr, and Alexandra Kitson
We present DreamLLM-3D, a composite multimodal AI system behind an immersive art installation for dream re-experiencing. It enables automated dream content analysis for immersive dream-reliving, by integrating a Large Language Model (LLM) with text-to-3D Generative AI. The LLM processes voiced dream reports to identify key dream entities (characters and objects), social interaction, and dream sentiment. The extracted entities are visualized as dynamic 3D point clouds, with emotional data influencing the color and soundscapes of the virtual dream environment. Additionally, we propose an experiential AI-Dreamworker Hybrid paradigm. Our system and paradigm could potentially facilitate a more emotionally engaging dream-reliving experience, enhancing personal insights and creativity.