Tools
Tools
Çelebi, Can and Penczynski, Stefan 
ORCID: https://orcid.org/0000-0001-5477-6830
(2026)
Much Ado about Prompting: LLM Classification of Text Messages from Experiments.
PLoS One, 21 (8).
e0354757.
ISSN 1932-6203
Preview |
PDF (journal.pone.0354757)
- Published Version
Available under License Creative Commons Attribution. Download (1MB) | Preview |
Abstract
Researchers who classify text with large language models often possess codebooks written for human annotators. We ask how these codebooks can serve as prompts. Using three codebooks from experimental economics (one on promise classification, two on strategic thinking), we vary the level of information in the prompt, then vary its formatting, framing and wording at the lowest and highest information levels, across two proprietary and two open-weight models. Used as prompts, the codebooks reach 82–88% agreement with human annotators across the three tasks. On the recognition-heavy task (promise classification), model choice accounts for most of the variation in accuracy; on the learning-heavy tasks (strategic thinking classification), the level of detail in the classification instructions carries comparable weight. These information components partly substitute for one another, whereas model reasoning does not reliably compensate for missing content and yields little or no improvement once the content is present. Larger models make better use of additional information and are more robust to formatting, framing, and wording of the prompt, while smaller models can be hurt by extra information and are more sensitive to how the information is presented. Our results advocate for a shift in focus from prompt engineering techniques (formatting, framing, reasoning, etc.) to the content of the prompt: preparing instructions as one would for human annotators, with detailed context, category definitions, and examples.
| Item Type: | Article |
|---|---|
| Additional Information: | Data Availability: All relevant data for this study are publicly available from the Zenodo repository (https://doi.org/10.5281/zenodo.21473191). |
| Faculty \ School: | Faculty of Social Sciences > School of Economics |
| UEA Research Groups: | Faculty of Social Sciences > Research Centres > Centre for Behavioural and Experimental Social Sciences Faculty of Social Sciences > Research Groups > Behavioural Economics |
| Related URLs: | |
| Depositing User: | LivePure Connector |
| Date Deposited: | 13 Aug 2026 15:14 |
| Last Modified: | 17 Aug 2026 07:43 |
| URI: | https://ueaeprints.uea.ac.uk/id/eprint/104126 |
| DOI: | 10.1371/journal.pone.0354757 |
Downloads
Downloads per month over past year
Actions (login required)
 |
View Item |