technique
zero-shot systematizer
techniqueactiveprovisional
zero-shot-systematizer-04777547·1 events·first seen 22d agoAliases: zero-shot systematizer
Co-occurring entities
More like this (12)
zero-shot evaluationzero-shot learningmulti-agent systematizerOne-Shot Imitation Learningfew-shot learningLanguage Models are Few-Shot Learnersfew-shot promptingSystem 0AI-Assisted Systematization for Evaluating GenAI SystemsFodor and Pylyshyn's Systematicity Challenge Still StandsRevisiting the Systematicity in Negation in the Era of In-Context Learningquantization-aware training
Recent events (1)
AI-Assisted Systematization for Evaluating GenAI Systems
This paper addresses a foundational gap in GenAI evaluation: the underspecification of broad, contested concepts like 'reasoning,' 'fairness,' or 'creativity.' The authors introduce a structured artifact called a 'concept spec' and a validation worksheet, then build two AI-assisted systematizers—a zero-shot approach and a multi-agent approach—to convert vague evaluation targets into measurable, structured accounts. They apply these tools to hate-based rhetoric and digital empathy, assessing the resulting specs on content validity and information recoverability. The work positions AI assistance as a scalable aid for the cognitively demanding process of evaluation design.