From Found to Designed: Concepts as a Design Axis for Large Language Models

Survey argues LLMs should be designed with explicit conceptual representations, not just probed for them

A new arXiv preprint proposes treating 'concepts' as a first-class design axis for LLMs, contrasting the current norm of recovering concept-like structure post-hoc (via probing or dictionary learning) with the alternative of deliberately engineering it into training objectives, architectures, or inference procedures. The authors map the design space along two dimensions: pipeline stage and whether structure is internally derived or externally grounded. Key findings include that inference-time approaches are underexplored, related ideas have developed in isolation across pipeline stages, and externally grounded methods span the full pipeline under inconsistent terminology.