We are pleased to announce that Florian Holldack, Leonardo Banh and Gero Strobel have had their paper „Synthetic Conversations, Real Insights: Towards Realistic User Simulations with Generative Agents“ accepted for presentation at ICIS 2025.

The International Conference on Information Systems (ICIS 2025) is the world’s leading conference in the field of Information Systems and will take place this year in Nashville, Tennessee, USA. The paper was published in the IS Design, Development, and Project Management track.

The study addresses a central challenge in research on Conversational Recommender Systems (CRS): traditional user studies are costly and time-consuming, while existing simulations fail to adequately capture realistic user interactions. The paper demonstrates how generative agents based on Large Language Models can be used to conduct user simulations. In an experiment, 200 generative agents and 50 human participants interacted with a prototypical CRS and were subsequently evaluated using standardized questionnaires.

The results show that, on a macro level, generative agents behave similarly to human users and provide subjective evaluations that can serve as an alternative to labor-intensive studies. Thus, the work makes an important contribution to advancing research methods in the field of Information Systems—particularly for user-centered design and evaluation of socio-technical systems in the age of generative AI.

You can find the paper here.

Abstract

The rise of generative AI and large language models (LLMs) has sparked interest in applying generative agents to simulate users in conversational recommender system (CRS) evaluations. While CRS rely on user perception for success, traditional user studies are costly and time-intensive. Existing simulation approaches lack interaction depth and user-centric evaluation. This study addresses these gaps by leveraging LLM-based generative agents to conduct scalable, subjective CRS assessments. We present a comparative study where 200 generative agents and 50 human participants interact with a prototypical agentic CRS and evaluate their experience using a structured questionnaire. Results indicate that generative agents approximate human-like behavior and subjective assessments at the macro-level, despite granular precision limitations, offering an alternative to traditional user studies. Our findings advance research on CRS evaluation by demonstrating how agentic simulations can support human-aligned assessments of socio-technical systems and open new avenues for applying Generative AI in user-centered decision support.

We warmly congratulate the authors on this successful publication and are delighted about the strong presence of our chair at ICIS 2025!