Package index • repllm

Design and generation

Start with the factors you want to vary, then ask a model to write several materials per condition.

design_conditions(): Build the experimental conditions
replicate_design(): Add versions of each condition
randomize_design(): Shuffle the order of the conditions
generate_materials(): Generate materials for each condition
generation_sensitivity(): Generate materials with several models

Tier 1, automatic validation

These checks run on your machine. I’d use them to find problems in the text before collecting ratings.

validate_auto(): Run the local text checks
check_length_balance(): Compare text length across conditions
check_readability(): Compare reading levels across conditions
check_manipulation_leakage(): Find words that could give away the condition
check_lexical_overlap(): Compare vocabulary across conditions

Tier 2, synthetic validation

Ask a model to score the texts on the dimensions you care about, without showing it the condition labels.

synthetic_ratings(): Collect model ratings of the materials
synthetic_check(): Compare conditions using model ratings
rater_reliability(): Check agreement among raters

Tier 3, human validation

Sample materials for people to rate, prepare blinded sheets, and compare their ratings with the model’s.

sample_for_human_validation(): Sample materials for people to rate
export_rating_task(): Prepare blinded rating sheets
import_human_ratings(): Read the completed rating sheets
human_check(): Compare conditions using human ratings
human_reliability(): Check agreement among human coders

Example data

I’ve included a worked example with 24 texts and simulated ratings. You can run the analysis without an API key.

repllm_materials: Example experimental materials
repllm_synthetic: Example model ratings
repllm_human: Example human ratings