the-pile-9080e526·2 events·first seen Aliases: The Pile
Researchers study the input-side dual of activation steering: optimizing fluent prompts to drive a chosen internal latent toward zero without inference-time model access. The target is an 'evaluation-awareness' latent whose suppression would threaten safety evaluation validity if models behave differently when detecting they are being tested. Experiments on Llama-3.2-3B and Llama-3.1-8B across five latent constructions (CAA direction, subspace norm, SAE feature, MLP neuron, behavioral logit) find the latent is robustly suppressible, but a key cautionary result emerges: a placebo random direction is suppressed just as hard and shifts behavior just as far, and suppressing the eval-direction in context fails to reduce behavioral eval judgment. The paper concludes that activation-readability does not imply behavioral controllability, with implications for how safety evaluations should be designed and interpreted.
Researchers from Astera Institute, Nvidia, Stanford, UC Berkeley, and UC San Diego introduced TTT-E2E, a method that compresses long context into transformer weights by training the model during inference via meta-learning. The approach uses sliding-window attention restricted to 8,000 tokens and updates only the fully connected layers of the last quarter of the network on each 1,000-token chunk at inference time, keeping per-token generation latency roughly constant as context scales to 128,000 tokens. TTT-E2E slightly outperforms vanilla transformers on next-token prediction loss across long contexts and matches efficient architectures like Mamba 2 and Gated DeltaNet on inference speed, but fails dramatically on Needle-in-a-Haystack retrieval beyond 8,000 tokens and incurs substantially higher training latency. The work reframes long-context handling as a training-inference trade-off rather than an architectural design problem.