Large language models can predict the results of social science experiments

Abstract

There is growing interest in how large language models (LLMs) can advance social and behavioural science1."),2."),3."),4."),5."). Previous work has assessed LLMs’ ability to predict survey responses6."),7."),8."),9."), but less is known about whether they can predict the outcomes of social science experiments10."), particularly those absent from training data. Here we built an archive of 70 preregistered, nationally representative survey experiments in the USA involving 469 experimental effects and 119,330 participants. We prompted an LLM to simulate how representative samples from American individuals would respond to experimental stimuli, and then we inferred treatment effects by comparing simulated responses across conditions. Predictions derived from GPT-4, whose training-data cutoff predated the publication of many studies in our archive, were strongly correlated with actual treatment effects, achieving accuracy similar to pooled human forecasts. Correlations remained high for studies not published or publicly posted by the model’s training-data cutoff date and for predictions from prominent open-weight models. Despite high correlations, predictions systematically overestimated effect sizes. In a secondary archive of 15 megastudies featuring 606 effects, correlations were lower but comparable to those of pooled expert forecasters. To assess implications for scientific practice, we surveyed 460 social scientists about probable uses and perceived risks and used our archives to assess several applications (pilot testing, intervention selection, identifying effects needing replication) and risks (bias, misuse). Together, these results indicate that LLMs can augment experimental methods in science and practice while raising important considerations for responsible use.

This is a preview of subscription content, access via your institution

Access options

Access through your institution

Access Nature and 54 other Nature Portfolio journals

Get Nature+, our best-value online-access subscription

27,99 € / 30 days

cancel any time

Learn more

Subscribe to this journal

Receive 52 print issues and online access

185,98 € per year

only 3,58 € per issue

Learn more

Buy this article

• Purchase on SpringerLink
• Instant access to the full article PDF.

39,95 €

Prices may be subject to local taxes which are calculated during checkout

Fig. 1: Correlations of predicted treatment effects with original treatment effects in social science survey experiments.



Fig. 2: Correlations of predicted treatment effects with original treatment effects in megastudies.



Fig. 3: Scientific uses of LLM-derived predictions of experimental results.



Similar content being viewed by others



Evaluating the ability of large Language models to predict human social decisions

Article Open access 02 September 2025



A large-scale replication of scenario-based experiments in psychology and management using large language models

Article 09 July 2025



Large language models robustness against perturbation

Article Open access 29 November 2025

Subjects

• Human behaviour
• Psychology

Data availability

All data used in this paper are available at <https://codeocean.com/capsule/9843791/tree/v1>. All study materials are available in the Supplementary Information. Source data are provided with this paper.

Code availability

Code to reproduce the results is available at <https://codeocean.com/capsule/9843791/tree/v1>. We also invite researchers to try the model in our web demo (<https://treatmenteffect.app/>), which generates AI-based forecasts of experimental effects using the approach detailed in this paper.

References

1. Bail, C. A. Can generative AI improve social science? Proc. Natl Acad. Sci. USA 121, e2314021121 (2024). Article CAS PubMed PubMed Central Google Scholar 2. Grossmann, I. et al. AI and the transformation of social science research. Science 380, 1108–1109 (2023). Article CAS PubMed ADS Google Scholar 3. Luo, X. et al. Large language models surpass human experts in predicting neuroscience results. Nat. Hum. Behav. 9, 305–315 (2025). Article PubMed Google Scholar 4. Crockett, M. & Messeri, L. Should large language models replace human participants? Preprint at PsyArXiv <https://doi.org/10.31234/osf.io/4zdx9> (2024). 5. Abdurahman, S. et al. Perils and opportunities in using large language models in psychological research. PNAS Nexus 3, pgae245 (2024). Article PubMed PubMed Central Google Scholar 6. Argyle, L. P. et al. Out of one, many: using language models to simulate human samples. Polit. Anal. 31, 337–351 (2023). Article Google Scholar 7. Bisbee, J., Clinton, J. D., Dorff, C., Kenkel, B. & Larson, J. M. Synthetic replacements for human survey data? The perils of large language models. Polit. Anal. 32, 401–416 (2024). Article Google Scholar 8. Atari, M., Xue, M. J., Park, P. S., Blasi, D. & Henrich, J. Which humans? Preprint at PsyArXiv <https://doi.org/10.31234/osf.io/5b26t> (2023). 9. Dillion, D., Tandon, N., Gu, Y. & Gray, K. Can AI language models replace human participants? Trends Cogn. Sci. 27, 597–600 (2023). Article PubMed Google Scholar 10. Filippas, A., Horton, J. J. & Manning, B. S. Large language models as simulated economic agents: what can we learn from homo silicus? In Proc. 25th ACM Conference on Economics and Computation 614–615 (ACM, 2024). 11. Binz, M. & Schulz, E. Using cognitive psychology to understand GPT-3. Proc. Natl Acad. Sci. USA 120, e2218523120 (2023). Article CAS PubMed PubMed Central Google Scholar 12. Bai, H., Voelkel, J. G., Muldowney, S., Eichstaedt, J. C. & Willer, R. LLM-generated messages can persuade humans on policy issues. Nat. Commun. 16, 6037 (2025). Article PubMed PubMed Central ADS Google Scholar 13. Hackenburg, K. et al. The levers of political persuasion with conversational artificial intelligence. Science 390, eaea3884 (2025). Article CAS PubMed Google Scholar 14. Meta Fundamental AI Research Diplomacy Team (FAIR) et al. Human-level play in the game of diplomacy by combining language models with strategic reasoning. Science 378, 1067–1074 (2022). 15. Park, J. S. et al. Generative agents: interactive simulacra of human behavior. In Proc. 36th Annual ACM Symposium on User Interface Software and Technology (eds Follmer, S. et al.) 1–22 (ACM, 2023). 16. Schramowski, P., Turan, C., Andersen, N., Rothkopf, C. A. & Kersting, K. Large pre-trained language models contain human-like biases of what is right and wrong to do. Nat. Mach. Intell. 4, 258–268 (2022). Article Google Scholar 17. Yin, Y., Jia, N. & Wakslak, C. J. AI can help people feel heard, but an AI label diminishes this impact. Proc. Natl Acad. Sci. USA 121, e2319112121 (2024). Article CAS PubMed PubMed Central Google Scholar 18. Lehr, S. A., Caliskan, A., Liyanage, S. & Banaji, M. R. ChatGPT as research scientist: probing GPT’s capabilities as a research librarian, research ethicist, data generator, and data predictor. Proc. Natl Acad. Sci. USA 121, e2404328121 (2024). Article CAS PubMed PubMed Central Google Scholar 19. Time-sharing Experiments for the Social Sciences. TESS www.tessexperiments.org (2026). 20. Coppock, A., Leeper, T. J. & Mullinix, K. J. Generalizability of heterogeneous treatment effect estimates across samples. Proc. Natl Acad. Sci. USA 115, 12441–12446 (2018). Article CAS PubMed PubMed Central ADS Google Scholar 21. Chang, S., Kennedy, A., Leonard, A. & List, J. A. 12 Best Practices for Leveraging Generative AI in Experimental Research (National Bureau of Economic Research, 2024). 22. Broska, D., Howes, M. & van Loon, A. The mixed subjects design: treating large language models as potentially informative observations. Sociol. Methods Res. 54, 1074–1109 (2025). Article MathSciNet Google Scholar 23. Chu, J. Y. et al. Academics are more specific, and practitioners more sensitive, in forecasting interventions to strengthen democratic attitudes. Proc. Natl Acad. Sci. USA 121, e2307008121 (2024). Article CAS PubMed PubMed Central Google Scholar 24. DellaVigna, S. & Linos, E. RCTs to scale: comprehensive evidence from two nudge units. Econometrica 90, 81–116 (2022). Article Google Scholar 25. Dreber, A. et al. Using prediction markets to estimate the reproducibility of scientific research. Proc. Natl Acad. Sci. USA 112, 15343–15347 (2015). Article CAS PubMed PubMed Central ADS Google Scholar 26. Milkman, K. L. et al. Megastudies improve the impact of applied behavioural science. Nature 600, 478–483 (2021). Article CAS PubMed PubMed Central ADS Google Scholar 27. Harding, J., D’Alessandro, W., Laskowski, N. G. & Long, R. AI language models cannot replace human research participants. AI Soc. 39, 2603–2605 (2024). 28. Messeri, L. & Crockett, M. J. Artificial intelligence and illusions of understanding in scientific research. Nature 627, 49–58 (2024). Article CAS PubMed ADS Google Scholar 29. Binz, M. et al. A foundation model to predict and capture human cognition. Nature 644, 1002–1009 (2025). 30. Petrov, N. B., Serapio-García, G. & Rentfrow, J. Limited ability of LLMs to simulate human psychological behaviours: a psychometric analysis. Preprint at <https://arxiv.org/abs/2405.07248> (2024). 31. Tjuatja, L., Chen, V., Wu, T., Talwalkwar, A. & Neubig, G. Do LLMs exhibit human-like response biases? A case study in survey design. Trans. Assoc. Comput. Linguist. 12, 1011–1026 (2024). Article Google Scholar 32. Chen, Y., Hu, Y. & Lu, Y. Predicting field experiments with large language models. Preprint at <https://arxiv.org/abs/2504.01167> (2025). 33. Wang, X. et al. Self-consistency improves chain of thought reasoning in language models. In Proc. 11th International Conference on Learning Representations (poster) (ICLR, 2023). 34. Arora, S. et al. Ask me anything: a simple strategy for prompting language models. In Proc. 11th International Conference on Learning Representations (ICLR, 2023). 35. Voronov, A., Wolf, L. & Ryabinin, M. Mind your format: towards consistent evaluation of in-context learning improvements. In Findings of the Association for Computational Linguistics: ACL 2024 (eds Ku, L.-W. et al.) 6287–6310 (ACL, 2024). 36. Hou, B., O’Connor, J., Andreas, J., Chang, S. & Zhang, Y. Promptboosting: black-box text classification with ten forward passes. In Proc. International Conference on Machine Learning (eds Krause, A. et al.) 13309–13324 (PMLR, 2023). 37. Open Science Collaboration. Estimating the reproducibility of psychological science. Science 349, aac4716 (2015). Article Google Scholar 38. Milkman, K. L. et al. Megastudy shows that reminders boost vaccination but adding free rides does not. Nature 631, 179–188 (2024). Article CAS PubMed PubMed Central ADS Google Scholar 39. Milkman, K. L. et al. A 680,000-person megastudy of nudges to encourage vaccination in pharmacies. Proc. Natl Acad. Sci. USA 119, e2115126119 (2022). Article CAS PubMed PubMed Central Google Scholar 40. Zickfeld, J. H. et al. Effectiveness of ex ante honesty oaths in reducing dishonesty depends on content. Nat. Hum. Behav. 9, 169–187 (2025). Article PubMed Google Scholar 41. Broockman, D. E., Kalla, J. L., Caballero, C. & Easton, M. Political practitioners poorly predict which messages persuade the public. Proc. Natl Acad. Sci. USA 121, e2400076121 (2024). Article CAS PubMed PubMed Central Google Scholar 42. DellaVigna, S. & Vivalt, E. Forecasting social science: evidence from 100 projects. National Bureau of Economic Research <https://doi.org/10.3386/w34493> (2025). 43. Allen, J., Watts, D. J. & Rand, D. G. Quantifying the impact of misinformation and vaccine-skeptical content on Facebook. Science 384, eadk3451 (2024). Article CAS PubMed ADS Google Scholar 44. DellaVigna, S. & Pope, D. What motivates effort? Evidence and expert forecasts. Rev. Econ. Stud. 85, 1029–1069 (2018). Article Google Scholar 45. Vlasceanu, M. et al. Addressing climate change with behavioral science: a global intervention tournament in 63 countries. Sci. Adv. 10, eadj5778 (2024). Article CAS PubMed PubMed Central Google Scholar 46. Tappin, B. M., Wittenberg, C., Hewitt, L. B., Berinsky, A. J. & Rand, D. G. Quantifying the potential persuasive returns to political microtargeting. Proc. Natl Acad. Sci. USA 120, e2216261120 (2023). Article PubMed PubMed Central Google Scholar 47. Voelkel, J. G. et al. Megastudy testing 25 treatments to reduce antidemocratic attitudes and partisan animosity. Science 386, eadh4764 (2024). Article CAS PubMed Google Scholar 48. Voelkel, J. G. et al. A registered report megastudy on the persuasiveness of the most-cited climate messages. Nat. Clim. Chang. 16, 214–225 (2026). 49. Saccardo, S. et al. Field testing the transferability of behavioural science knowledge on promoting vaccinations. Nat. Hum. Behav. 8, 878–890 (2024). Article PubMed PubMed Central Google Scholar 50. Mason, K. et al. A megastudy of behavioral interventions to increase voter registration ahead of the 2024 US presidential election. Preprint at PsyArXiv <https://doi.org/10.31234/osf.io/p6b2e%5Fv3> (2025). 51. Goldwert, D. et al. A megastudy of behavioral interventions to catalyze public, political, and financial climate advocacy. PNAS Nexus 5, pgaf400 (2026). Article PubMed PubMed Central Google Scholar 52. Druckman, J. N. Experimental Thinking (Cambridge Univ. Press, 2022). 53. Freese, J. & Peterson, D. Replication in social science. Annu. Rev. Sociol. 43, 147–165 (2017). Article Google Scholar 54. Bai, X., Wang, A., Sucholutsky, I. & Griffiths, T. L. Explicitly unbiased large language models still form biased associations. Proc. Natl Acad. Sci. USA 122, e2416228122 (2025). Article CAS PubMed PubMed Central Google Scholar 55. Wang, A., Morgenstern, J. & Dickerson, J. P. Large language models that replace human participants can harmfully misportray and flatten identity groups. Nat. Mach. Intell. 7, 400–411 (2025). 56. Park, P. S., Schoenegger, P. & Zhu, C. Diminished diversity-of-thought in a standard large language model. Behav. Res. Methods 56, 5754–5770 (2024). 57. Frank, M. C. Openly accessible LLMs can help us to understand human cognition. Nat. Hum. Behav. 7, 1825–1827 (2023). Article PubMed Google Scholar 58. Wack, M., Ehrett, C., Linvill, D. & Warren, P. Generative propaganda: evidence of AI’s impact from a state-backed disinformation campaign. PNAS Nexus 4, pgaf083 (2025). Article PubMed PubMed Central Google Scholar 59. Response retrieved from ChatGPT-4 on 22 May. ChatGPT <https://chatgpt.com/share/2a2b66d0-e9fd-421f-bedd-bb6f9354742a> (2024). 60. Almaatouq, A. et al. Beyond playing 20 questions with nature: integrative experiment design in the social and behavioral sciences. Behav. Brain Sci. 47, e33 (2024). Article Google Scholar 61. Zhu, J.-Q., Xie, H., Arumugam, D., Wilson, R. C. & Griffiths, T. L. Using reinforcement learning to train large language models to explain human decisions. Preprint at <https://arxiv.org/abs/2505.11614> (2025). 62. Tyner, A. H. et al. Investigating the replicability of the social and behavioural sciences. Nature 652, 143–150 (2026). 63. Cremer, D. D. & Kasparov, G. AI should augment human intelligence, not replace it. Harvard Business Review <https://hbr.org/2021/03/ai-should-augment-human-intelligence-not-replace-it> (2021). 64. Cummins, J. The threat of analytic flexibility in using large language models to simulate human data: a call to attention. Preprint at <https://arxiv.org/abs/2509.13397> (2025). 65. Obermeyer, Z., Powers, B., Vogeli, C. & Mullainathan, S. Dissecting racial bias in an algorithm used to manage the health of populations. Science 366, 447–453 (2019). Article CAS PubMed ADS Google Scholar 66. Coppock, A. Generalizing from survey experiments conducted on mechanical turk: a replication approach. Political Sci. Res. Methods 7, 613–628 (2019). Article Google Scholar 67. Mullinix, K. J., Leeper, T. J., Druckman, J. N. & Freese, J. The generalizability of survey experiments. J. Exp. Political Sci. 2, 109–138 (2015). Article Google Scholar 68. Achiam, J. et al. GPT-4 technical report. Preprint at <https://arxiv.org/abs/2303.08774> (2023). 69. Viechtbauer, W. metafor: Meta-Analysis Package for R v.4.8-0 (2015). 70. Rios, K., Roth, Z. C. & Coleman, T. J. III. The importance of scientists’ intellectual humility for communicating effectively across ideological and identity-based divides. Proc. Natl Acad. Sci. USA 122, e2400930121 (2025).

Download references

Acknowledgements

We thank N. Egan, K. Fuller, V. Isarraras, S. Kane, B. Kinder, A. Kissi, K. McPeek, S. Muldowney, C. Redekopp and T. Ren for research assistance. We are grateful to researchers who shared data, including J. Chu, S. DellaVigna, J. Druckman, A. Duckworth, D. Goldwert, N. Malhotra, K. Mason, K. Milkman, K. Okst, S. Saccardo, M. Vlasceanu and J. Voelkel. We also thank J. Allen, M. Atari, C. Bail, M. Bernstein, M. Dehghani, D. Broockman, K. Clayton, M. Crockett, D. Ganguli, D. Dillon, J. Eichstaedt, M. Frank, N. Goodman, D. Green, J. Kalla, P. Liang, J. McClelland, D. Rand, R. Orchinik, N. Otis, S. Rathje, B. Tappin, A. Van Loon, A. Waytz and D. Willner for invaluable feedback on the project.

Funding

We acknowledge funding from the Stanford Center on Philanthropy and Civil Society.

Author information

Author notes 1. These authors contributed equally: Ashwini Ashokkumar, Luke Hewitt

Authors and Affiliations

1. Department of Psychology, Harvard University, Cambridge, MA, USA Ashwini Ashokkumar & Isaias Ghezae 2. Department of Sociology, Stanford University, Stanford, CA, USA Luke Hewitt & Robb Willer 3. Transluce, San Francisco, CA, USA Luke Hewitt

Authors 1. Ashwini Ashokkumar View author publications Search author on:PubMed Google Scholar 2. Luke Hewitt View author publications Search author on:PubMed Google Scholar 3. Isaias Ghezae View author publications Search author on:PubMed Google Scholar 4. Robb Willer View author publications Search author on:PubMed Google Scholar

Contributions

L.H., A.A. and R.W. designed the research. A.A., L.H., I.G. and R.W. assembled the experimental archives. L.H. and A.A. analysed the data. A.A., L.H. and R.W. wrote the paper. L.H. built the web tool.

Corresponding authors

Correspondence to Ashwini Ashokkumar, Luke Hewitt or Robb Willer.

Ethics declarations

Competing interests

The authors declare no competing interests.

Peer review

Peer review information

Nature thanks Jamie Cummins, James Druckman, John Protzko and Youyou Wu for their contribution to the peer review of this work. Peer reviewer reports are available.

Additional information

Publisher’s note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Extended data figures and tables

Extended Data Fig. 1 Method.

We re-analyze raw data from 70 nationally-representative U.S. studies and estimate treatment effects in a consistent manner. We use a Large Language Model to predict results of those same experiments, providing the original study materials and demographically-diverse participant profiles, and then calculate the average predicted response for each condition. We evaluate the accuracy of the model in terms of the correspondence between observed- and predicted- treatment effects.

Extended Data Fig. 2 Averaging across more LLM prompts improves predictive accuracy.

Error bars indicate 95% confidence intervals for the mean Pearson correlation across 32 simulation runs. Each run uses a different random subset of prompts (drawn without replacement from 120). The CI reflects uncertainty in the mean across simulation runs, but does not include uncertainty over the sample of 70 experiments or the sample of 120 prompts.

Source data

Extended Data Fig. 3 The accuracy of LLM-derived predictions for demographic subgroups in the US.

Error bars indicate 95% confidence intervals.

Source data

Supplementary information

Supplementary Information (download PDF )

Supplementary Figs. 1–10, Tables 1–6 and text on the construction of the primary and secondary experimental archives, prompting strategy, layperson and expert forecasts, survey recruitment, analytical methods for both archives and the scientific applications, robustness checks and further applications and risk analyses. Supplementary Figs. 1–10 and Tables 1–6 provide additional detail on the study archives and methods, assess the robustness of LLM-based predictions, present further analyses of pilot-testing and intervention-selection use cases, and report detailed survey results from social scientists.

Reporting Summary (download PDF )

Peer Review File (download PDF )

Source data

Source Data Fig. 1 (download XLSX )

Source Data Fig. 2 (download XLSX )

Source Data Fig. 3 (download XLSX )

Source Data Extended Data Fig. 2 (download XLSX )

Source Data Extended Data Fig. 3 (download XLSX )

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

[](https://crossmark.crossref.org/dialog/?doi=10.1038/s41586-026-10742-x)

Cite this article

Ashokkumar, A., Hewitt, L., Ghezae, I. et al. Large language models can predict the results of social science experiments. Nature (2026). https://doi.org/10.1038/s41586-026-10742-x

Download citation

• Received: 13 July 2024
• Accepted: 01 June 2026
• Published: 08 July 2026
• Version of record: 08 July 2026
• DOI: https://doi.org/10.1038/s41586-026-10742-x

Share this article

Anyone you share the following link with will be able to read this content:

Get shareable link

Sorry, a shareable link is not currently available for this article.

Copy shareable link to clipboard

Provided by the Springer Nature SharedIt content-sharing initiative