An Introduction to Simulating Human Survey Responses with Large Language Models: Potentials and Pitfalls

A tutorial co-located with the 12th International Conference on Computational Social Science (IC²S²) in Burlington, Vermont.

Program Overview

The tutorial is designed as a 3-hour interactive session, combining short lectures with hands-on exercises and group discussion.

Schedule




Session Time Contents
First Part  13:15–14:45 (90 min) — see ic2s2-2026.org/program
Session 0 Opening Presenter introductions
Session 1 Overview Lecture
What are social simulations? Why should we simulate society?
Traditional non-LLM simulations
What is new about LLM simulations?
What is the role of LLMs in these social simulations?
Methodological Challenges, Validation, and Open Questions
Pipeline overview / practical contribution to surveys
Session 2 Survey Simulation Overview Lecture
– Persona populations, role-playing & instruction prompts
– Response generation & parsing
– QSTN Oerview
Hands-on
– Setting up the QSTN framework for questionnaire inference with LLMs
– Design choices in personas, prompts, response generation and comparison of outcomes
Break  14:45–15:00 (15 min)
Second Part  15:00–16:30 (90 min)
Session 3 Hybrid Samples Lecture
Combining human & simulated survey responses: fine-tuning, prompting, PPI and adaptive sampling
Session 4 Evaluation and Validation Lecture
– Introduction to existing benchmarks and evaluation metrics
– Special focus: Evaluation of imputation methods
Hands-on
– Evaluation of previously generated survey data: individual-level F1, selected subgroups TVD
– Downstream estimand: regr. coef. evaluated with imputation metrics: abs. error, interval width, coverage
– Rectification of the estimand with PPI
Session 5 Risks, Methodological Limitations, and Ethical Reflections Lecture
– Methodological risks, ethical trade-offs, and appropriate use cases
– Impact of AI-led surveys on survey methodology and its perceptions by the public
Group Discussion
– What kind of disciplines need to contribute to this?
– Think-pair-share

Equipment & Format

Participants will work with Python-based tools (QSTN framework) in Jupyter notebooks. All materials will be provided in advance. Slides will use clear visual structure, code examples will be heavily commented on, and all notebooks will be runnable end-to-end.

Requirements: