Kyungja, by Eunsu Kang
Collaboratively Designing Metrics to Evaluate Creative Machines
Sunday, October 18
15:00pm - 19:00pm (Montreal Time, GMT-4, EST)
ISEA2020 - Why Sentience
| Speakers | Schedule | Questions | Participants | Call for Participation | Metrics | Organizers |
How do we make a creative machine? This workshop follows a series of studies conducted in the classroom setting at CMU and UC San Diego. For ISEA, participants will collectively find a definition of creativity specific to machines, addressing follow up questions: Can we computationally model ambiguity? Would a novelty search result in valuable discoveries? Where is the threshold between randomness and creativity? How do we evaluate the creativity of an algorithm? To answer these questions, participants develop sets of criteria to assess their own and peer groups’ creative AI and ML projects. Although such a human-centered method is subjective, we anticipate discovering ways to describe and interpret dimensions of algorithmic creativity. This half-day workshop engages a broader arts and machine learning community to collaboratively define these quantitative criteria, in a first attempt to collectively establish evaluation metrics for the area of creative AI.
ISEA 2020 is a virtual event and this workshop will be conducted online. Our invited speakers will provide prerecorded video talks and participate in live panel discussions.
NOTE: This page will be updated frequently as we confirm workshop participants.
Introduction
Introductory slides from Eunsu Kang: Measuring Computational Creativity (pdf)
Speakers
Allison Parrish (New York University, decontextualize.com)
Devi Parikh (Georgia Tech/Facebook AI Research)
Aaron Hertzmann (Adobe Research)
Roger Dannenberg (Carnegie Mellon University)
Fabrizio Poltronieri (Institute of Creative Technologies at De Montfort University, fabriziopoltronieri.com)
Haru Ji (OCAD University, artificialnature.net/) and Graham Wakefield (York University, artificialnature.net)
Jun-Yan Zhu (Carnegie Mellon University)
Ahmed Elgammal (Rutgers University)
Schedule
tentative schedule (online videos: invited speaker talks, participants will watch it beforehand)
| Time | Activity | Location |
|---|---|---|
| 3:00 - 3:20 | Introduction (20 min) | main room |
| 3:20 - 3:50 | Discussion 1 - Elements of Creative AI (30 min) | breakout rooms |
| 3:50 - 4:05 | Q & A (15 min) | main room |
| 4:05 - 4:40 | Guest Speaker Panel 1 (35min) | main room |
| 4:40 - 5:10 | Discussion 2 - Evaluating ML/Art Projects (30 min) | breakout rooms |
| 5:10 - 5:25 | Q & A (15 min) | main room |
| 5:25 - 6:10 | Guest Speaker Panel 2 (35 min) | main room |
| 6:10 - 6:30 | Discussion 3 - Revising Metrics, Evaluation 2 (20 min) | breakout rooms |
| 6:30 - 7:00 | Presentation of Results and Q&A | main room |
Questions
Panel 1: Graham Wakefield, Haru Ji, Fabrizio Poltronieri, Allison Parrish, Aaron Hertzmann
- What are your methods and metrics for evaluating the creative AI system that you and your colleagues have developed?
- How does the study of creative AI systems inform human creative practice?
- How do we attribute (who is responsible for) the creativity in collaborative creative systems?
- Do you think there is a difference between creative AI and human creativity? What would be the difference? If not, why not.
Panel 2: Ahmed Elgammal, Devi Parikh, Jun-Yan Zhu, Roger Dannenberg
- What are your methods and metrics for evaluating the creative AI system that you and your colleagues have developed?
- Do human creative methods provide valuable models for computational creativity? How does human creativity influence computational creativity?
- How do we attribute (who is responsible for) the creativity in collaborative creative systems?
- How would you define creativity in generative models? If these models are specifically trained to generate new content, does this mean that they are necessarily creative?
Additional Questions
- Are all generative systems “creative”? or… Is there a meaningful distinction between generativity and creativity?
- How can we contribute to a better definition of “creativity” (which is still fuzzy) using measures that capture computational creativity?
- Evaluating a creative work can be considered subjective. Can we (and should we) account for this subjectivity in our artificially intelligent evaluation methods, or should we strive to make objective measures?
Participants
This session is an actual working session: together we will collaboratively define metrics for Creative AI.
Tuomas Sandholm Superhuman AI for heads-up no-limit poker: Libratus beats top professionals
Carlos Castellanos PlanetConnect
Austin Lee mongyudowondo.com/
Nick Fox-Gieg https://www.instagram.com/p/BxffQ0RADZ0/
Koosilja https://koosil-jadance.com/
Peter Schaldenbrand https://pschaldenbrand.github.io/
Kat Mustatea What is the Value of Art in the Age of Intelligent Machines?
Dana Sperry https://vimeo.com/460211868
Kim Baraka https://www.youtube.com/watch?v=PNzeT8ZsyfM
Rodrigo F. Cádiz https://creativai-uc.github.io/
Jie-Eun Hwang https://openarchive.uosarch.ac.kr/
Joel Ong Terra et Venti
Alex MacLean Switch Jockey
Claire Jervert https://m.youtube.com/watch?v=UKHgOupljmA
Aaron Oldenburg Desert Mothers
Amalia Foka The Invisible Structures of the Artworld
Weihua Zhao 404 Not Found
Vít Růžička GAN Explorer
Patricia Alves-Oliveira https://www.youtube.com/watch?v=e-K3J5UZ9M4&feature=emb_logo
Violaine Lafortune Organisme No. 17
Donald Craig https://www.youtube.com/watch?v=hWTwmDpVtdk
Andrew Brown https://vimeo.com/142107969
Johnny Diblasi https://www.design.iastate.edu/news/2020/04/johnny-diblasi-fulbright/
Kiel Howe Baby’s First Image Classification Training Data Set - An Opensource Children’s Book
Jon Paden Dadum
Pablo Sotres Uruguay25
Participants will be updated as they are confirmed.
Call for Participation
Participants in this workshop, as a group, will examine a number of Artificial Intelligence (AI) Art projects, and articulate metrics to evaluate dimensions of creativity in those works. Workshop participants are key contributors to this research in Measuring Creative AI, and we plan for every participant to be named on future mCreativeAI publications (website, papers, etc.) as contributors. This is a novel research project with no prior examples as far as we know, and this workshop will be the inaugural event for this effort as this exercise has only been conducted with students previously.
How to participate: If you are interested to participate, please fill out the following google form by Friday, September 25th, midnight Eastern.
Form: https://forms.gle/ZshgTgHti4Q7aJVz8
We will send out an acceptance notification by Friday, October 2nd.
Metrics
Here is our metrics worksheet: google docs
We will publish our results after the workshop.
Organizers
Eunsu Kang is an artist, a researcher, and an educator who explores the intersection of art and machine learning, one of the core methods for building AI. Her works have been presented at conferences including ACM, ICMC, ISEA, SIGGRAPH Asia and NeurIPS. Kang earned her Ph.D. from DXARTS of the University of Washington, an MA from MAT, UCSB, and an MFA from Ewha Womans University. She was a tenured Associate Professor of New Media Art at the University of Akron and currently is a Visiting Professor of Art and Machine Learning at Carnegie Mellon University’s School of Computer Science. kangeunsu.com
Jean Oh is a faculty member at the Robotics Institute at Carnegie Mellon University. Her research addresses the creativity in decision making in the problem domains including self-driving cars, disaster response, healthcare, and arts. Her team’s work won the Best Paper Award at the 2018 IEEE International Conference on Robotics and Automation, and also the Best Cognitive Robotics Paper Award at the IEEE International Conference on Robotics and Automation in 2015. Jean received her Ph.D. in Language and Information Technologies at Carnegie Mellon University, M.S. in Computer Science at Columbia University, and B.S. in Biotechnology at Yonsei University.
Robert Twomey is an artist and engineer exploring the poetic intersection of human and machine perception. He has presented his work at SIGGRAPH, the Museum of Contemporary Art San Diego, and his research has been supported by the National Science Foundation, Microsoft, Amazon, and NVIDIA. Twomey received his BS from Yale with majors in Art and Biomedical Engineering, his MFA from UC San Diego, and his PhD in DXARTS from the University of Washington. He is an Assistant Professor with the Johnny Carson Center for Emerging Media Arts at the University of Nebraska-Lincoln, and a Visiting Scholar with the Arthur C. Clarke Center for Human Imagination at UC San Diego.
Support
This workshop is made possible, in part, by generous support from the Johnny Carson Center for Emerging Media Arts.