À propos
July 2026 paper introducing ESPP, an evaluation method that replaces one LLM judge with a panel of evidence-grounded, psychologically diverse personas. Panelists independently rate generated-interface screenshots, exchange opinions through a bounded-confidence mechanism, and are combined with social weighting.
Résumé
Why ESPP Adds New Signal:
1. Plural evaluation: Models how different user groups perceive the same generated interface instead of collapsing judgment into one implicit viewpoint
2. Human-alignment evidence: Improves Pearson correlation with human ratings from 0.716 to 0.922
3. Controlled comparison: Shows that a prompt ensemble recovers only about one third of the improvement
4. Dimension-level disagreement: Preserves where personas agree on rankings but diverge on particular interface qualities
5. Reproducible implementation: Releases a runnable three-stage scoring pipeline, sample screenshots, and worked cases
1. Plural evaluation: Models how different user groups perceive the same generated interface instead of collapsing judgment into one implicit viewpoint
2. Human-alignment evidence: Improves Pearson correlation with human ratings from 0.716 to 0.922
3. Controlled comparison: Shows that a prompt ensemble recovers only about one third of the improvement
4. Dimension-level disagreement: Preserves where personas agree on rankings but diverge on particular interface qualities
5. Reproducible implementation: Releases a runnable three-stage scoring pipeline, sample screenshots, and worked cases
Tags
genuiui-evaluationhuman-preferencesllm-as-a-judgepersonas