Research software · Charles Crabtree
Does the treatment do
what I think it does?
I built repllm to generate materials for survey experiments and check them before putting them in front of respondents.
A language model can write a set of experimental vignettes quickly. I’d still want to know whether those vignettes vary in the way I intended. An economic frame should make people think about economic consequences. It might also produce a longer text, or one that’s harder to read. Those differences could matter for the experiment.
I use repllm to generate materials from a factorial design, then inspect what changed across conditions. Its text checks and rating tools help me decide which materials need revision before I field a study.
Installation
I’m preparing repllm for CRAN. For now, the source repository is private; email me if you’d like access. Once you have it:
# install.packages("remotes")
remotes::install_github("lobsterbush/repllm")If I’ve sent you a source archive, you can install it and its dependencies with remotes::install_local("repllm_0.4.0.tar.gz", dependencies = TRUE).
You’ll need R 4.1 or later. You can run the local checks and bundled examples without an API key. Generating materials or collecting model ratings needs a provider supported by ellmer, your own credentials, and any fees the provider charges.
The documentation is public.
Checking the materials
Check the text
I start with the things I can check locally: length, reading level, vocabulary, and words that give away the condition.
Ask a model to rate it
Model ratings help screen the materials. The raters get the text without its condition label or the conversation that produced it.
Ask people, too
I want to know whether people read the materials the same way. The package prepares blinded rating sheets for a sample.
The workflow
design_conditions() -> generate_materials()
|
+------------------+------------------+
| | |
validate_auto() synthetic_ratings() sample_for_human_validation()
(tier 1, free) synthetic_check() export_rating_task()
(tier 2) import_human_ratings()
human_check()
(tier 3)
Tiers 2 and 3 ask the same question of different raters and report it the same
way, so you set the two results side by side.| Function | What it does |
|---|---|
design_conditions() |
Cross the factors in your experiment |
replicate_design() / randomize_design()
|
Repeat conditions or shuffle their order |
generate_materials() |
Ask a model to write materials for each condition |
generation_sensitivity() |
Generate materials with several models |
validate_auto() |
Run the local text checks |
check_length_balance() |
Compare word and character counts |
check_readability() |
Compare Flesch-Kincaid reading levels |
check_manipulation_leakage() |
Find words that give away the condition |
check_lexical_overlap() |
Compare vocabulary across conditions |
synthetic_ratings() |
Collect model ratings without showing condition labels |
synthetic_check() |
Compare conditions on the dimensions you’ve rated |
rater_reliability() |
Check agreement among raters |
sample_for_human_validation() |
Sample materials for people to rate |
export_rating_task() |
Write shuffled rating sheets and a separate key |
import_human_ratings() |
Read completed sheets and restore condition labels |
human_check() / human_reliability()
|
Analyse human ratings and coder agreement |
I’ve included 24 carbon-tax vignettes and simulated ratings so you can try the package without making a model call. repllm_materials holds the texts; repllm_synthetic and repllm_human hold ratings representing three model raters and two human coders.
Quick start
Start with the bundled example. These calls run the local checks and compare the simulated model and human ratings:
library(repllm)
validate_auto(repllm_materials, condition_col = "frame")
targets <- c(economic = "economic", moral = "moral", scientific = "scientific")
synthetic_check(repllm_synthetic, target = targets)
human_check(repllm_human, target = targets)These ratings are teaching data. They show how the functions behave. They don’t tell us how well a model would agree with people in your study.
Generate your own materials
This example crosses frame with source and asks for three versions of each condition. You’ll need your provider credentials to run it.
library(repllm)
design <- design_conditions(
frame = c("economic", "moral"),
source = c("expert", "layperson")
)
chat <- ellmer::chat_openai(
"You write short, neutral survey vignettes. Return only the vignette text.",
echo = "none"
)
materials <- generate_materials(
design,
template = "Write a 60-word quote about a carbon tax from a {source}
emphasising the {frame} perspective.",
chat = chat,
n_versions = 3
)Use a fresh chat object. Its system prompt applies to every condition, so put condition-specific instructions in the template. The function warns if it finds a factor level in the system prompt and rejects a chat with earlier conversation turns.
The use of several versions follows Porter and Velez (2022). I want the design to depend less on which particular wording I happened to choose.
Tier 1: the local checks
These checks run on your machine. Here’s what they find in the bundled texts:
data(repllm_materials)
validate_auto(repllm_materials, condition_col = "frame")
#> -- Automatic validation --
#> v Length balance: max deviation 4.8%
#> v Readability: grade-level spread 0.4
#> ! Manipulation leakage: 2/24 materials name a condition term.
#> v Distinctiveness: Jaccard overlap in [0.1, 0.11]
#> ! Needs attention: "leakage"Two vignettes contain “economic” or “scientific”, the label of their own condition. I’d inspect those texts. A rater might respond to the label without reading the intended argument.
The other checks pass at the default thresholds. That tells us what these checks found; it doesn’t rule out confounding. The readability formula is for English, so I’d use a language-appropriate measure for other texts.
Tier 2: synthetic raters
Here I ask model raters to score economic and moral appeals. The analysis below uses the bundled ratings, which also include scientific appeals.
ratings <- synthetic_ratings(
repllm_materials,
dimensions = c(
economic = "how strongly the text appeals to economic consequences",
moral = "how strongly the text appeals to moral duty"
),
chat = ellmer::chat_openai(echo = "none"),
condition_col = "frame",
n_raters = 3,
personas = c("a general survey respondent", "a policy analyst",
"an undergraduate student")
)
targets <- c(economic = "economic", moral = "moral", scientific = "scientific")
synthetic_check(repllm_synthetic, target = targets)
#> Synthetic validation
#> Materials: 24 | raters: 3 | dimensions: 3
#> Ratings: 216
#> Standard errors: cluster-robust by material (24 clusters)
#> Manipulation recovery:
#> ok economic on economic margin +3.71 over scientific (d = +6.23)
#> ok moral on moral margin +3.37 over scientific (d = +6.67)
#> ok scientific on scientific margin +2.75 over economic (d = +3.23)
#> Recovered 3/3 intended contrastsThe recovery table names the strongest competing condition and reports the gap in rating points and Cohen’s d. I find that easier to interpret than a comparison with the average of all the other conditions.
Each rater sees the material text. The package removes earlier chat turns, replaces the system prompt with the rating instructions, and shuffles the texts. It asks ellmer for structured ratings and records failed or out-of-range responses as missing.
I’d give the raters different personas, while keeping in mind that they’re still using the same model. High agreement among them can reflect shared model behavior. I wouldn’t take it as evidence that people will agree.
Tier 3: human coders
Here I sample 12 materials and prepare a separate sheet for each coder:
dims <- c(economic = "economic appeal",
moral = "moral appeal",
scientific = "scientific appeal")
sub <- sample_for_human_validation(repllm_materials, n = 12,
stratify_by = "frame")
task <- export_rating_task(sub, file.path(tempdir(), "sheet.csv"),
dimensions = dims, condition_col = "frame",
n_raters = 3)
# ... your coders fill in the sheets ...
human <- import_human_ratings(task$sheets, task$key_path, scale = c(1, 7))
human_check(human, target = targets)The sheets use random IDs and shuffled rows. The separate key links those IDs to the original materials and conditions; keep it away from the coders. Choose new file paths for a new task, since existing sheets won’t be overwritten.
Reading the two tiers together
I use the same summaries for both sets of ratings. That makes it easier to see where they differ. Here’s the bundled example:
targets <- c(economic = "economic", moral = "moral", scientific = "scientific")
synthetic_check(repllm_synthetic, target = targets)
#> ok economic on economic margin +3.71 over scientific (d = +6.23)
#> ok moral on moral margin +3.37 over scientific (d = +6.67)
#> ok scientific on scientific margin +2.75 over economic (d = +3.23)
human_check(repllm_human, target = targets)
#> ok economic on economic margin +2.44 over scientific (d = +2.82)
#> ok moral on moral margin +2.62 over scientific (d = +3.24)
#> ok scientific on scientific margin +2.62 over economic (d = +3.16)Each intended condition ranks highest on its target dimension in both sets of ratings. The model ratings show wider economic and moral gaps than the human ratings do. The scientific gaps are similar. I’d look at those differences before deciding how much to rely on the model ratings.
The ranking is descriptive. It doesn’t test whether the two sets of ratings are equivalent, and it doesn’t establish that the manipulation will work with respondents.
I’d also report rater_reliability() and specify which ICC I’m using. In these data, the reliability of one synthetic rater ranges from .72 to .94 across dimensions. For the mean of three raters, it ranges from .88 to .98. The choice depends on whether I’m analysing individual ratings or their average.
What this owes to Porter and Velez
The package grew out of Porter and Velez (2022). Their work motivates using several stimuli per condition and averaging over them. I’ve carried that idea into replicate_design() and the n_versions argument.
I also want evidence about what the generated materials convey. That’s why I’ve put generation and validation in the same package.
Porter, E., & Velez, Y. R. (2022). Placebo selection in survey experiments: An agnostic approach. Political Analysis, 30(4), 481-494.
Contact
I’d be glad to hear how you’re using the package, especially where a check misses something. Open an issue or email me.
Charles Crabtree, Monash University and Korea University. MIT License.
Development record
I maintain the package and edit its documentation. I used OpenAI Codex for the September 2026 audit and revisions.
Generated materials carry a separate record of the model and prompt used to produce them.
Development and replication
To work on the package, install devtools, here, and pkgdown from CRAN. In your local checkout, run remotes::install_deps(dependencies = TRUE), followed by:
The tests use simulated data and mock model calls, so they don’t need API keys. The getting-started vignette walks through the bundled example.