Entity · company
Artificial Analysis
companyactive
artificial-analysis-9232f433·12 events·first seen Aliases: Artificial Analysis, Artificial Analysis Arena, Artificial Analysis Video Arena
Co-occurring entities
More like this (12)
Artificial Analysis Conversational DynamicsArtificial Analysis Big Bench AudioArtificial Analysis Intelligence IndexArtificial Analysis Intelligence LeaderboardArtificial Analysis Text to Image LeaderboardArtificial Analysis Coding Agent IndexArtificial Analysis Text to ImageArtificial Analysis LLM Performance LeaderboardAI for ScienceAllen Institute for AIautomated AI researchAllenAI
Recent events (12)
Artificial Analysis publishes DeepSeek V4 Flash 0731 intelligence, performance, and price analysis
Artificial Analysis has published a benchmarking and pricing analysis of DeepSeek V4 Flash 0731, a new model variant from DeepSeek. The post is gaining significant traction on Hacker News with 343 points and 172 comments, suggesting notable community interest. The analysis covers intelligence benchmarks, performance metrics, and cost positioning for this flash-tier model.
Meta Superintelligence Labs launches Muse Image and previews Muse Video with agentic generation capabilities
Meta Superintelligence Labs (MSL) has launched Muse Image, its most advanced image generation model, and previewed Muse Video, both representing the first media generation models from the newly formed lab. Muse Image operates as an agent with tool use (web search, code execution), emergent self-refinement, and test-time compute scaling, achieving a No. 2 Arena Elo ranking for text-to-image and editing tasks at launch. The model integrates with Muse Spark for joint agentic planning and is deploying across Meta AI, Instagram Stories, and WhatsApp. Muse Video, built on the same pretraining base, adds native audio support and is coming soon to creators.
The Batch digest: U.S. chatbot adoption tops 50%, AA-Briefcase benchmark, ARD spec, North Mini Code, Fable/Mythos export controls
A weekly digest from DeepLearning.AI covers five AI developments: a Pew Research Center survey showing nearly half of U.S. adults now use AI chatbots (ChatGPT at 44% adoption); Artificial Analysis releasing AA-Briefcase, a new benchmark for complex knowledge-work tasks where Claude Opus 4.8 is a top performer; Hugging Face publishing a reference implementation of the Agentic Resource Discovery (ARD) open spec co-developed with Microsoft, Google, and others for runtime tool discovery by agents; Cohere releasing North Mini Code, a 30B-parameter open-weight MoE coding model under Apache 2.0; and over 100 cybersecurity professionals signing an open letter urging the U.S. government to reverse export controls on Anthropic's Claude Fable 5 and Claude Mythos 5. The ARD and export-control items are the highest-signal stories, touching agent infrastructure standards and AI regulatory policy respectively.
Independent evaluators struggle to benchmark Claude Fable 5 due to Anthropic's safety classifiers and data retention policies
Multiple independent organizations found they could not fully evaluate Claude Fable 5 (the public-facing safeguarded version of Claude Mythos 5) because Anthropic's classifiers silently rerouted flagged prompts to the weaker Claude Opus 4.8 or refused them outright. Evaluators including Artificial Analysis, Vals AI, and ARC Prize Foundation each adopted different scoring strategies — blended, pure, or abstaining entirely — producing widely divergent rankings depending on how refusals were handled. On GPQA Diamond, Claude Fable 5's score swung from 93.18% (2nd place) to 55.56% (94th place) depending on whether refusals were counted as failures. The episode surfaces a structural tension between safety-oriented deployment constraints and the ability of the field to independently measure frontier model capabilities.
DeepSWE, ProgramBench, and ITBench-AA emerge as harder successors to SWE-bench for agent evaluation
Three new benchmarks — DeepSWE (by Datacurve), ProgramBench (Meta/Stanford/Harvard), and ITBench-AA (IBM/Artificial Analysis) — are positioned as more rigorous replacements for the SWE-bench family, which models have largely saturated. DeepSWE tests feature implementation using private codebases and human-written problems; ProgramBench evaluates agents' ability to recreate functional programs from scratch; ITBench-AA measures root-cause diagnosis in real-world IT incident scenarios. Current top performers include GPT-5.5 (70% on DeepSWE), Claude Opus 4.7 (46.7% on ITBench-AA), and Claude Opus 4.7 (3% on ProgramBench at the 95% pass threshold), illustrating that even frontier models have substantial headroom.
Grok Imagine 1.0 Sharply Cuts Costs for High-Quality Video Generation
xAI launched Grok Imagine 1.0, a text-and-image-to-video model that topped the Artificial Analysis Video Arena leaderboard in both text-to-video and image-to-video categories at launch. The model generates up to 15-second clips with audio at $4.20 per minute of output, significantly undercutting Google Veo 3.1 ($12/min) and OpenAI Sora 2 Pro ($30/min). It is integrated with the X social network, enabling direct generation and sharing, though xAI disclosed no technical details about the model's architecture. The launch highlights continued rapid cost compression in video generation, with a seven-fold price gap between Grok Imagine 1.0 and Sora 2 Pro.
OpenAI Shuts Down Sora Video Generation Model, Redirects Team to World Models and Robotics
OpenAI is discontinuing its Sora video generation model, with web/app access ending April 26 and API access closing September 24, 2026. The model was losing roughly $1 million per day, with daily active users falling below 500,000 after peaking at 1 million post-mobile launch. The Sora team will be redirected to longer-term projects including world models and robotics, while compute resources have already been diverted to a new coding/enterprise model codenamed Spud. The shutdown also effectively ends OpenAI's high-profile partnership with Disney, which had planned to invest up to $1 billion contingent on Sora integration.
ByteDance Launches Seedance 2.0 Video Generation Model Globally via CapCut
ByteDance has deployed Seedance 2.0, a multimodal video generation model, to hundreds of millions of CapCut users across multiple global regions. The model supports text, image, audio, and video inputs with synchronized audio-video output, lip-synced dialogue, and camera control via prompts. It ranks within the top two on Arena AI and Artificial Analysis video leaderboards, and is available via API at $0.30 per second of output. The issue also features Andrew Ng's editorial arguing against the 'AI jobpocalypse' narrative, attributing it to incentive structures at labs and companies.
ByteDance Deploys Seedance 2.0 Video Model to CapCut's 736M Users as OpenAI Shutters Sora
ByteDance has integrated Seedance 2.0, its multimodal video generation model, into CapCut for paying users across multiple global regions, reaching a platform with approximately 736 million monthly active users. The model supports text, image, audio, and video inputs, generates synchronized audio-video output in a single pass including multi-shot sequences, and ranks in the top two on Arena AI and Artificial Analysis video leaderboards, with Alibaba's HappyHorse-1.0 as its closest competitor. Simultaneously, OpenAI is discontinuing the Sora app and API after daily active users fell below 500,000 and operating costs reached an estimated $1 million per day. The contrast illustrates a broader market shift where Chinese developers are accelerating video model releases while U.S. consumer video products retreat.
ITBench-AA: Frontier Models Score Below 50% on the First Benchmark for Agentic Enterprise IT Tasks
IBM Research and Artificial Analysis have released ITBench-AA, a benchmark targeting agentic AI performance on enterprise IT operations tasks. Frontier models evaluated on the benchmark score below 50%, indicating significant capability gaps in real-world IT automation scenarios. The benchmark appears to be the first of its kind focused specifically on agentic enterprise IT workflows, covering tasks relevant to site reliability engineering and IT operations.
Bringing the Artificial Analysis LLM Performance Leaderboard to Hugging Face
Hugging Face is hosting the Artificial Analysis LLM Performance Leaderboard, which tracks inference performance metrics such as latency, throughput, and cost across multiple LLM providers. The leaderboard provides a standardized comparison of how different models perform in production deployment contexts rather than purely capability benchmarks. This collaboration brings infrastructure and deployment performance data into the Hugging Face ecosystem.
Launching the Artificial Analysis Text to Image Leaderboard & Arena
Hugging Face and Artificial Analysis are launching a combined leaderboard and arena for evaluating text-to-image models. The leaderboard tracks quality, speed, and cost metrics across leading image generation models, while the arena component collects human preference votes for side-by-side comparisons. This provides a structured benchmark for comparing commercial and open-weight image generation systems.