PolkitBench

Context-aware distillation and ablation study for Text2DSL Polkit rule generation

Researchers extend a Text2DSL system for generating Polkit domain-specific language rules from natural language, replacing prompt-only synthetic data generation with context-aware distillation using DeepSeek-V4-Flash as a teacher model operating under structured context (BNF grammar, API spec, closed vocabulary). The approach scales a verified corpus from 4,204 to 10,073 NL-to-Polkit-rule pairs at near-perfect validity rates. A factorial ablation across eight context conditions on GigaChat-10B-A1.8B finds that structured context is load-bearing rather than cosmetic, with vocabulary contributing the largest semantic-quality gains via Shapley decomposition.