Simon Willison uses DSPy to evaluate and optimize Datasette Agent SQL system prompts
Simon Willison documents an experiment using DSPy to systematically evaluate and improve the SQL system prompts used by Datasette Agent. The post covers applying DSPy's prompt optimization framework to a real-world agentic tool, demonstrating a practical workflow for automated prompt engineering. This is a hands-on practitioner account of using DSPy for prompt evaluation in a production-adjacent context.
Related guides (3)
Related events (8)
Datasette Agent
Simon Willison describes a Datasette Agent, an AI agent built on top of the Datasette data exploration tool. The post appears to demonstrate an agent capable of querying and reasoning over SQLite databases via natural language. This represents a practical deployment of LLM-powered tooling for data analysis workflows.
Simon Willison releases datasette-agent-edit 0.1a0
Simon Willison published an early alpha release (0.1a0) of datasette-agent-edit, a new plugin or tool for the Datasette ecosystem. The release appears to add agent-based editing capabilities to Datasette, a tool for exploring and publishing data. As an alpha release, details are sparse but it signals active development at the intersection of AI agents and data tooling.
Simon Willison releases datasette-agent 0.2a0
Simon Willison published datasette-agent 0.2a0, an early alpha release of an agent tool built around Datasette. The release is part of the growing ecosystem of AI agent tooling for data exploration and querying. Minimal detail is available from the body, but the versioning indicates active development.
MAS-PromptBench: Systematic study of prompt optimization in multi-agent LLM systems
A new arXiv preprint introduces MAS-PromptBench, a benchmark and study examining when and how much system-prompt optimization improves multi-agent LLM systems (MAS). The authors evaluate two prompt optimizers across diverse MAS configurations varying in task, workflow, communication protocol, and team size. Results show prompt optimization can unlock significant gains but also expose open challenges, particularly around the exponentially growing search space as agent count increases.
Simon Willison releases datasette-agent 0.3a0
Simon Willison published datasette-agent 0.3a0, an alpha release of an agent tooling package for Datasette. The release appears to be a tooling update in the agent/data exploration space. Datasette is an open-source tool for exploring and publishing data, and this agent layer extends it with AI-driven capabilities.
datasette-agent 0.1a4
Simon Willison releases datasette-agent 0.1a4, an early alpha of an AI agent plugin for Datasette, the open-source data exploration tool. The release represents ongoing development of agentic tooling that allows LLMs to interact with SQLite databases through the Datasette interface. As an alpha release from a prominent AI tooling developer, it signals continued growth in the agent-tool ecosystem for data querying use cases.
datasette-agent 0.1a3
Simon Willison releases datasette-agent 0.1a3, an early alpha of an AI agent plugin for Datasette, the open-source data exploration tool. The release represents ongoing development of agentic tooling that allows LLMs to interact with SQLite databases through Datasette's interface. As an alpha release from a prominent AI tooling developer, it signals continued growth in the agent-tool ecosystem for data workflows.
Simon Willison reports on prompt injection and adversarial attacks after 2,000 people tried to hack his AI assistant
Simon Willison documents the results of a public experiment in which approximately 2,000 people attempted to compromise or manipulate his personal AI assistant. The post covers the attack patterns observed, what succeeded or failed, and lessons learned about prompt injection and adversarial robustness in deployed AI systems. This is a practical, first-hand account of real-world AI security challenges from a respected practitioner.