AI and Compute: OpenAI Analysis of Exponential Growth in Training Compute Since 2012
OpenAI published an analysis in May 2018 showing that compute used in the largest AI training runs has been doubling every 3.4 months since 2012, far outpacing Moore's Law's 2-year doubling period. Over the 2012–2018 period, this metric grew by more than 300,000x. The analysis frames compute scaling as a key driver of AI progress and argues for preparing for systems with capabilities well beyond those of the time.
Related guides (3)
Related events (8)
AI and Efficiency: Algorithmic Progress Halving Training Compute Every 16 Months Since 2012
OpenAI released an analysis showing that compute required to match AlexNet-level ImageNet performance has decreased 44x since 2012, with algorithmic efficiency doubling every 16 months. This outpaces Moore's Law, which would have yielded only an 11x improvement over the same period. The findings suggest that for heavily-invested AI tasks, algorithmic progress is a larger driver of efficiency gains than hardware improvements alone.
Building the compute infrastructure for the Intelligence Age
OpenAI is scaling its Stargate initiative to expand compute infrastructure aimed at supporting AGI development. The announcement describes new data center capacity additions to meet growing AI demand. This represents a continuation of OpenAI's large-scale infrastructure buildout strategy under the Stargate program.
Accelerating the next phase of AI
OpenAI has raised $122 billion in new funding, marking one of the largest capital raises in AI history. The funds are earmarked for expanding frontier AI development globally, investing in next-generation compute infrastructure, and scaling to meet growing demand for ChatGPT, Codex, and enterprise AI products. The announcement signals continued aggressive investment in AI infrastructure and model development at the frontier.
Scaling Kubernetes to 7,500 Nodes
OpenAI describes scaling Kubernetes clusters to 7,500 nodes to support large-scale AI training workloads including GPT-3, CLIP, and DALL·E. The post details infrastructure challenges and solutions enabling both massive model training and rapid small-scale research iteration. This represents a significant engineering milestone in ML training infrastructure at the time of publication (January 2021).
Anthropic Growing 10x/Year While Competitors Cut Workforce
A Latent Space newsletter item highlights a notable divergence in the AI industry: Anthropic is reportedly growing at roughly 10x per year while other AI/tech companies are conducting layoffs exceeding 10% of their workforces. The piece frames this as a significant economic dichotomy within the AI sector. The body is brief and reflective, offering limited technical detail.
AI Scaling Myths
A commentary piece from normaltech.ai argues that AI scaling will eventually hit limits, framing the debate as a question of timing rather than whether limits exist. The piece appears to challenge prevailing optimism around continued scaling returns. Given the minimal body text, the depth of argument is unclear, but the topic directly engages the scaling laws debate central to frontier AI development.
OpenAI and NVIDIA Announce Strategic Partnership to Deploy 10 Gigawatts of AI Datacenters
OpenAI and NVIDIA have announced a strategic partnership targeting deployment of 10 gigawatts of AI datacenter capacity powered by NVIDIA systems. The first phase of the buildout is scheduled to launch in 2026. This represents a major infrastructure commitment between two of the most prominent organizations in AI compute and model development.
AMD and OpenAI Announce Strategic Partnership to Deploy 6 Gigawatts of AMD GPUs
AMD and OpenAI have entered a multi-year strategic partnership to deploy 6 gigawatts of AMD Instinct GPUs for OpenAI's AI infrastructure, with 1 gigawatt planned for 2026. The deal represents a significant diversification of OpenAI's compute supply beyond its existing NVIDIA dependency. This is one of the largest publicly announced GPU deployment commitments in the industry.