The First Workshop on the Evaluation of Generative Foundation Models at CVPR 2024

EVGENFM2024

Workshop Time, June 18 2024, 1pm - 6:30pm PDT

Workshop location, Seattle Convention Center | Summit 433

The landscape of artificial intelligence is being transformed by the advent of Generative Foundation Models (GenFMs), such as Large Language Models (LLMs) and diffusion models. GenFMs offer unprecedented opportunities to enrich human lives and transform industries. However, they also pose significant challenges, including the generation of factually incorrect or biased information, which might be potentially harmful or misleading. With the emergence of multimodal GenFMs, which leverage and generate content in an increasing number of modalities, these challenges are set to become even more complex. This emphasizes the urgent need for rigorous and effective evaluation methodologies.

The 1st Workshop on Evaluation for Generative Foundation Models at CVPR 2024 aims to build a forum to discuss ongoing efforts in industry and academia, share best practices, and engage the community in working towards more reliable and scalable approaches for GenFMs evaluation.

Workshop Schedule

Gathering + Posters Arrangement

Jun 18, 13:00 - 13:20 PDT

Opening Remarks

Jun 18, 13:20 - 13:30 PDT

Keynote, Ludwig Schmidt

DataComp: Evaluating Training Sets for Multimodal Models
Jun 18, 13:30 - 14:00 PDT

Keynote, Ranjay Krishna

The past, Present, and Future of Vision-Language Evaluation
Abstract Jun 18, 14:00 - 14:30 PDT

Keynote, Hanwang Zhang

Road to Level 5 MM Generalist
Abstract Jun 18, 14:30 - 15:00 PDT

Spotlight Presentations

Jun 18, 15:00 - 15:15 PDT

Poster Session, Coffee Break

Jun 18, 15:15 - 15:45 PDT

Keynote, Leonid Karlinsky

Analyzing and Improving Compositional Reasoning in Multi-Modal Foundation Models
Abstract Jun 18, 15:45 - 16:15 PDT

Oral Presentation, Linjie Li

Diagnostic Benchmark and Iterative Inpainting for Layout-Guided Image Generation
Abstract Jun 18, 16:15 - 16:30 PDT

Panel Discussion

Jun 18, 16:30 - 17:00 PDT

Keynote, Sadeep Jayasumana

Rethinking FID: Towards a Better Evaluation Metric for Image Generation
Abstract Jun 18, 17:00 - 17:25 PDT

Keynote, Jungo Kasai

Dramatic Five Years of AI and NLP: Evaluation and the Future of Foundation Models
Abstract Jun 18, 17:25 - 17:50 PDT

Keynote, Bo Li

Risk Assessment, Safety Alignment, and Guardrails for Generative Models
Abstract Jun 18, 17:50 - 18:15 PDT

Closing Remarks

Jun 18, 18:15 - 18:30 PDT


Invited Speakers and Panelists

Ludwig Schmidt

Assistant Professor at University of Washington

Bo Li

Associate Professor at University of Chicago

Ranjay Krishna

Associate Professor at University of Washington

Hanwang Zhang

Associate Professor at Nanyang Technological University

Leonid Karlinsky

Principal Research Scientist at IBM Research

Jungo Kasai

Co-founder & CTO at Kotoba Technologies, Inc.

Sadeep Jayasumana

Staff Research Scientist at Google Research

Besmira Nushi

Researcher at Microsoft Research

Lucas Beyer

Staff Research Engineer at Google Brain


Accepted Papers

  • Diagnostic Benchmark and Iterative Inpainting for Layout-Guided Image Generation; Jaemin Cho, Linjie Li, Zhengyuan Yang, Zhe Gan, Lijuan Wang, Mohit Bansal.

  • Evaluating and Improving Compositional Text-to-Visual Generation; Baiqi Li, Deepak Pathak, Jiayao Li, Yixin Fei, Kewen Wu, Xide Xia, Graham Neubig, Pengchuan Zhang, Zhiqiu Lin, Deva Ramanan.

  • Evaluating Multimodal Large Language Models across Distribution Shifts and Augmentations; Aayush Atul Verma, Amir Saeidi, Shamanthak Hegde, Ajay Therala, Fenil Denish Bardoliya, Nagaraju Machavarapu, Shri Ajay Kumar Ravindhiran, Srija Malyala, Agneet Chatterjee, Yezhou Yang, Chitta Baral.

  • Towards Quantitative Evaluation Metrics for Image Editing Approaches; Dana Cohen Hochberg, Oron Anschel, Alon Shoshan, Igor Kviatkovsky, Manoj Aggarwal, Gerard G Medioni.

  • ReMOVE: A Reference-free Metric for Object Erasure; Aditya Chandrasekar, Goirik Chakrabarty, Jai Bardhan, Ramya Hebbalaguppe, Prathosh A P.


Program Committee

Alex C Williams

Amazon

Amir Tavanaei

Amazon

Angel Martinez-Gonzalez

Amazon

Bardiya Akhbari

Amazon

Ebrahim Safadi

Amazon

Fatemeh Ghezloo

University of Washington

Gauthier Guinet

Amazon

Gokhan Kirlik

Amazon

Gozde Sahin

Amazon

Hammad Ayyubi

Columbia University

Hengameh M Dastjerdi

Amazon

Hengkang Wang

University of Minnesota

Huitian Lei

Lyft INC

Jiawei Ma

Columbia University

Junyi Hu

UMass at Lowell

Kee Kiat Koo

Amazon

Mahtab Bigverdi

University of Washington

Meng Han

Amazon

Michael Opitz

Amazon

Nadine Behrmann

Bosch Center for AI

Nicholas Dronen

Amazon

Pradeep Yarlagadda

Amazon

Prateek Singhi

Amazon

Qi Li

Amazon

Rahul Sharma

Amazon

Rustin Soraki

University of Washington

Thomas Cilloni

Amazon

Tristan McKinney

Amazon

Wentai Zhang

Amazon

Wenyi Wu

Amazon

Wisdom Ikezogwo

University of Washington

Xiao Zhang

Amazon

Xin Li

Amazon

Yiming Qian

Amazon

Yuguang Li

Zillow Group


Special thanks: we thank Ece Kamar from Microsoft Research for her support of our workshop.


Organizers

Maria Zontak

Amazon (Primary Contact)

Xu Zhang

Amazon

Mehmet Saygin Seyfioglu

University of Washington

Erran Li

Amazon

Bahar Erar Hood

Amazon

Suren Kumar

Samsung

Karim Bouyarmane

Amazon

© 2024 EVGENFM2024

We thank Jalpc for the jekyll template