Entity · benchmark

PoPE (Popperian Placebo-controlled Evaluation)

benchmarkactiveprovisionalpope-popperian-placebo-controlled-evaluation--57d250c4·1 events·first seen 4d ago

Aliases: PoPE (Popperian Placebo-controlled Evaluation)

Co-occurring entities

Form, Not Content? A Preregistered, Placebo-Controlled Evaluation of Learned Error-Conditioned Self-Repair Through Prompts and Weights in Frozen Small Code Models

More like this (12)

Cranfield evaluation paradigm ICO (Intervention, Comparator, Outcome) framework Form, Not Content? A Preregistered, Placebo-Controlled Evaluation of Learned Error-Conditioned Self-Repair Through Prompts and Weights in Frozen Small Code Models Plausibility Evaluation STAGE-Eval olmo-eval G-Eval Always-On Evaluation Protocol (AOEP-v0)Evaluation on the Hub political bias evaluation Personalized Evaluation as Learning L-Eval

Recent events (1)

5arXiv · cs.LG·4d ago·source ↗

PoPE: Placebo-controlled evaluation of error-conditioned self-repair in frozen small code LLMs

A preregistered arXiv preprint introduces PoPE (Popperian Placebo-controlled Evaluation), a methodology for rigorously measuring whether execution error feedback actually improves code generation in frozen small LLMs (0.5–1.5B parameters). The study uses channel-specific placebos—ablating error content while preserving scaffold form—to test both prompt-based and adapter-based self-repair. Results across both channels failed to confirm content-attributable superiority over placebos or baselines, suggesting that apparent self-repair gains in prior work may reflect form rather than semantic error content. The paper argues that writing oracle-derived representations back into generation state replaces testing with conditioning.

Evaluation and Benchmarking Agent and Tool Ecosystem PoPE (Popperian Placebo-controlled Evaluation)Form, Not Content? A Preregistered, Placebo-Controlled Evaluation of Learned Error-Conditioned Self-Repair Through Prompts and Weights in Frozen Small Code Models