The First Workshop on the Evaluation of Generative Foundation Models at CVPR 2024


Workshop Time, June 18 2024, 1pm - 6:30pm PDT

Workshop location, Seattle Convention Center | Summit 433

The landscape of artificial intelligence is being transformed by the advent of Generative Foundation Models (GenFMs), such as Large Language Models (LLMs) and diffusion models. GenFMs offer unprecedented opportunities to enrich human lives and transform industries. However, they also pose significant challenges, including the generation of factually incorrect or biased information, which might be potentially harmful or misleading. With the emergence of multimodal GenFMs, which leverage and generate content in an increasing number of modalities, these challenges are set to become even more complex. This emphasizes the urgent need for rigorous and effective evaluation methodologies.

The 1st Workshop on Evaluation for Generative Foundation Models at CVPR 2024 aims to build a forum to discuss ongoing efforts in industry and academia, share best practices, and engage the community in working towards more reliable and scalable approaches for GenFMs evaluation.

Workshop Schedule

Gathering + Posters Arrangement

Jun 18, 13:00 - 13:20 PDT

Opening Remarks

Jun 18, 13:20 - 13:30 PDT

Keynote, Ludwig Schmidt

DataComp: Evaluating Training Sets for Multimodal Models
Jun 18, 13:30 - 14:00 PDT

Keynote, Ranjay Krishna

The past, Present, and Future of Vision-Language Evaluation
Abstract Jun 18, 14:00 - 14:30 PDT

Keynote, Hanwang Zhang

Road to Level 5 MM Generalist
Abstract Jun 18, 14:30 - 15:00 PDT

Spotlight Presentations

Jun 18, 15:00 - 15:15 PDT

Poster Session, Coffee Break

Jun 18, 15:15 - 15:45 PDT

Keynote, Leonid Karlinsky

Analyzing and Improving Compositional Reasoning in Multi-Modal Foundation Models
Abstract Jun 18, 15:45 - 16:15 PDT

Oral Presentation, Linjie Li

Diagnostic Benchmark and Iterative Inpainting for Layout-Guided Image Generation
Abstract Jun 18, 16:15 - 16:30 PDT

Panel Discussion

Jun 18, 16:30 - 17:00 PDT

Keynote, Sadeep Jayasumana

Rethinking FID: Towards a Better Evaluation Metric for Image Generation
Abstract Jun 18, 17:00 - 17:25 PDT

Keynote, Jungo Kasai

Dramatic Five Years of AI and NLP: Evaluation and the Future of Foundation Models
Abstract Jun 18, 17:25 - 17:50 PDT

Keynote, Bo Li

Risk Assessment, Safety Alignment, and Guardrails for Generative Models
Abstract Jun 18, 17:50 - 18:15 PDT

Closing Remarks

Jun 18, 18:15 - 18:30 PDT

Invited Speakers and Panelists

Ludwig Schmidt

Assistant Professor at University of Washington

Bo Li

Associate Professor at University of Chicago

Ranjay Krishna

Associate Professor at University of Washington

Hanwang Zhang

Associate Professor at Nanyang Technological University

Leonid Karlinsky

Principal Research Scientist at IBM Research

Jungo Kasai

Co-founder & CTO at Kotoba Technologies, Inc.

Sadeep Jayasumana

Staff Research Scientist at Google Research

Besmira Nushi

Researcher at Microsoft Research

Lucas Beyer

Staff Research Engineer at Google Brain

Accepted Papers

  • Diagnostic Benchmark and Iterative Inpainting for Layout-Guided Image Generation; Jaemin Cho, Linjie Li, Zhengyuan Yang, Zhe Gan, Lijuan Wang, Mohit Bansal.

  • Evaluating and Improving Compositional Text-to-Visual Generation; Baiqi Li, Deepak Pathak, Jiayao Li, Yixin Fei, Kewen Wu, Xide Xia, Graham Neubig, Pengchuan Zhang, Zhiqiu Lin, Deva Ramanan.

  • Evaluating Multimodal Large Language Models across Distribution Shifts and Augmentations; Aayush Atul Verma, Amir Saeidi, Shamanthak Hegde, Ajay Therala, Fenil Denish Bardoliya, Nagaraju Machavarapu, Shri Ajay Kumar Ravindhiran, Srija Malyala, Agneet Chatterjee, Yezhou Yang, Chitta Baral.

  • Towards Quantitative Evaluation Metrics for Image Editing Approaches; Dana Cohen Hochberg, Oron Anschel, Alon Shoshan, Igor Kviatkovsky, Manoj Aggarwal, Gerard G Medioni.

  • ReMOVE: A Reference-free Metric for Object Erasure; Aditya Chandrasekar, Goirik Chakrabarty, Jai Bardhan, Ramya Hebbalaguppe, Prathosh A P.

Program Committee

Alex C Williams


Amir Tavanaei


Angel Martinez-Gonzalez


Bardiya Akhbari


Ebrahim Safadi


Fatemeh Ghezloo

University of Washington

Gauthier Guinet


Gokhan Kirlik


Gozde Sahin


Hammad Ayyubi

Columbia University

Hengameh M Dastjerdi


Hengkang Wang

University of Minnesota

Huitian Lei

Lyft INC

Jiawei Ma

Columbia University

Junyi Hu

UMass at Lowell

Kee Kiat Koo


Mahtab Bigverdi

University of Washington

Meng Han


Michael Opitz


Nadine Behrmann

Bosch Center for AI

Nicholas Dronen


Pradeep Yarlagadda


Prateek Singhi


Qi Li


Rahul Sharma


Rustin Soraki

University of Washington

Thomas Cilloni


Tristan McKinney


Wentai Zhang


Wenyi Wu


Wisdom Ikezogwo

University of Washington

Xiao Zhang


Xin Li


Yiming Qian


Yuguang Li

Zillow Group

Special thanks: we thank Ece Kamar from Microsoft Research for her support of our workshop.


Maria Zontak

Amazon (Primary Contact)

Xu Zhang


Mehmet Saygin Seyfioglu

University of Washington

Erran Li


Bahar Erar Hood


Suren Kumar


Karim Bouyarmane


© 2024 EVGENFM2024

We thank Jalpc for the jekyll template