Advancing independent research on AI alignment
OpenAI is committing $7.5 million to The Alignment Project to fund independent AI alignment research. The grant is framed as part of broader efforts to address AGI safety and security risks. This represents a notable external funding move by OpenAI to support alignment work outside its own walls.
Related guides (3)
Related events (8)
OpenAI Superalignment Fast Grants: $10M for Superhuman AI Safety Research
OpenAI is launching $10M in fast grants to fund external technical research on aligning and ensuring the safety of superhuman AI systems. Priority research areas include weak-to-strong generalization, interpretability, and scalable oversight. The program is part of OpenAI's broader Superalignment initiative, which aims to solve the alignment problem for superintelligent systems within four years.
Our approach to alignment research
OpenAI outlines its alignment research strategy, centered on improving AI systems' ability to learn from human feedback and to assist humans in evaluating AI outputs. The stated long-term goal is to build a sufficiently aligned AI system capable of helping solve remaining alignment problems. This represents OpenAI's public framing of its scalable oversight and RLHF-centric research agenda as of mid-2022.
Announcing the OpenAI Safety Fellowship
OpenAI has announced a Safety Fellowship, described as a pilot program aimed at supporting independent safety and alignment research while developing the next generation of AI safety talent. The announcement is sparse on details but signals a structured investment in external safety research capacity. This follows broader industry trends of labs funding independent safety work to build the research ecosystem.
OpenAI Awards Up to $2M in Grants for AI and Mental Health Research
OpenAI is launching a grant program of up to $2 million to fund research at the intersection of AI and mental health. The program targets studies examining real-world risks, benefits, and applications of AI with the goal of improving safety and well-being. No specific grantees or research directions are named in the announcement.
OpenAI Launches $1M Grant Program for Democratic AI Governance Experiments
OpenAI's nonprofit arm is awarding ten grants of $100,000 each to fund experiments in designing democratic processes for determining the rules AI systems should follow. The program aims to explore how collective human input can shape AI behavior within legal boundaries. This represents an early institutional effort to operationalize participatory governance for AI alignment decisions.
OpenAI co-founds Agentic AI Foundation, donates AGENTS.md
OpenAI has co-founded the Agentic AI Foundation, a new organization housed under the Linux Foundation, aimed at developing open and interoperable standards for safe agentic AI systems. As part of this initiative, OpenAI is donating AGENTS.md, a specification document, to the foundation. The move signals OpenAI's intent to shape industry-wide norms around agentic AI through open governance structures rather than proprietary control.
Anthropic launches initiative to fund third-party AI safety evaluations
Anthropic announced a funded initiative to source third-party evaluations measuring advanced AI capabilities and safety risks, with priority areas including cybersecurity, CBRN threats, model autonomy, national security risks, social manipulation, and misalignment. The initiative is tied to Anthropic's Responsible Scaling Policy and AI Safety Level (ASL) framework, aiming to address a gap between demand and supply of high-quality safety-relevant evals. Proposals are solicited via an application form, with Anthropic framing the effort as benefiting the broader AI safety ecosystem rather than just internal use.
AI Safety Needs Social Scientists
OpenAI published a paper arguing that long-term AI safety research requires social scientists to address uncertainties in human psychology, rationality, emotion, and biases that affect alignment algorithms. The paper contends that aligning advanced AI with human values cannot be solved by machine learning alone. OpenAI announced plans to hire social scientists full-time to work on these problems.