Simulated ratings in the format returned by synthetic_ratings.
Three model raters score all 24 materials in repllm_materials on three
dimensions.
Format
A tibble with 216 rows and 5 columns:
- material_id
Identifier matching repllm_materials.
- condition
The framing condition of that material.
- rater
"synthetic_1"to"synthetic_3".- dimension
Rated dimension:
"economic","moral", or"scientific".- rating
Integer rating on a 1 to 7 scale.
Details
I've set up the example so both model and human ratings rank each intended
condition highest. The model ratings show larger economic and moral gaps
than the simulated human ratings do; the scientific gaps are similar. Run
synthetic_check and human_check to compare them.
Examples
data(repllm_synthetic)
synthetic_check(
repllm_synthetic,
target = c(economic = "economic", moral = "moral",
scientific = "scientific")
)
#> Synthetic validation
#> Materials: 24 | raters: 3 | dimensions: 3
#> Ratings: 216
#> Standard errors: cluster-robust by material (24 clusters)
#> Manipulation recovery:
#> ok economic on economic margin +3.71 over scientific (d = +6.23)
#> ok moral on moral margin +3.37 over scientific (d = +6.67)
#> ok scientific on scientific margin +2.75 over economic (d = +3.23)
#> Recovered 3/3 intended contrasts
rater_reliability(repllm_synthetic)
#> # A tibble: 3 × 8
#> dimension n_raters n_materials n_complete n_failed icc_single icc_average
#> <chr> <int> <int> <int> <int> <dbl> <dbl>
#> 1 economic 3 24 24 0 0.935 0.977
#> 2 moral 3 24 24 0 0.930 0.975
#> 3 scientific 3 24 24 0 0.716 0.883
#> # ℹ 1 more variable: krippendorff_alpha <dbl>