The Archive

LQM hierarchical taxonomy diagnoses MT errors in diglossic languages beyond surface-form mistakes.

MS-RCGR reversible multi-scale encoding transforms biological sequences into interpretable scale-invariant geometric features.

Extends Semantic Tube Prediction by sampling at reasoning step boundaries to improve geometric structure and efficiency in LLM fine-tuning.

Commentary on AI conference culture prioritizing acceptance metrics over reproducibility and lasting research impact.

Framework for calibrating model-based summarization metrics without reference summaries or human annotations, addressing reliability in automatic evaluation.

Reward-based optimization framework combining reasoning traces with preference alignment for faithful multi-role dialogue summarization.

Method for learning low-dimensional manifold structure in stochastic dynamical systems via geometric regularization of autoencoders using ambient covariance.

VEFX-Bench: large-scale video editing benchmark with human annotations, complete examples, quality labels, and specialized evaluation metrics for instruction-guided editing.

Theoretical framework for Hebbian plasticity using Wasserstein geometry; neuroscience formalism without ML applications.

In the AI era, power is the ultimate constraint, and every AI factory operates within a hard limit. This makes performance per watt—the rate at which power is... In the AI era, power is the ultimate constraint, and every AI factory operates within a hard limit. This makes…

Computer-aided engineering (CAE) is shifting from human-driven workflows toward AI-driven ones, including physics foundation models that generalize across... Computer-aided engineering (CAE) is shifting from human-driven workflows toward AI-driven ones, including physics…

Today, Shreya Shankar and I are publishing evals skills, a set of skills for AI product evals1. Eval tools often get in the way. They nudge you toward generic off-the-shelf metrics and fully automated evals before you’ve looked at your data. These skills help you avoid common…

Summary Snowflake’s Vino Duraisamy and Replit’s Manny Bernabe return to turn simple dashboards into working data tools. Live forecasting and drilldowns explain why metrics move, not just what changed on your Snowflake dashboard. Custom Slack alerts fire when revenue crosses key…

Anthropic Economic Index report reveals AI usage primitives and interaction metrics from November 2025.

Import AI 435 reports on training scale milestones, labor displacement metrics, and compute efficiency trends.

GPT-5.1 system card addendum reports updated safety metrics for Instant and Thinking variants, including mental health evaluations.

Mistral AI contributes to development of global environmental standards and metrics for AI systems.

Evaluation metrics, how to build eval datasets, eval methodology, and a review of several benchmarks.

Earlier this year, I wrote Your AI product needs evals. Many of you asked, “How do I get started with LLM-as-a-judge?” This guide shares what I’ve learned after helping over 30 companies set up their evaluation systems. The Problem: AI Teams Are Drowning in Data Ever spend…

Reference, context, and preference-based metrics, self-consistency, and catching hallucinations.

We’ve developed a method to assess whether a neural network classifier can reliably defend against adversarial attacks not seen during training. Our method yields a new metric, UAR (Unforeseen Attack Robustness), which evaluates the robustness of a single model against an…

We’ve discovered that the gradient noise scale, a simple statistical metric, predicts the parallelizability of neural network training on a wide range of tasks. Since complex tasks tend to have noisier gradients, increasingly large batch sizes are likely to become useful in the…

We’re releasing CoinRun, a training environment which provides a metric for an agent’s ability to transfer its experience to novel situations and has already helped clarify a longstanding puzzle in reinforcement learning. CoinRun strikes a desirable balance in complexity: the…

We’re releasing an analysis showing that since 2012, the amount of compute used in the largest AI training runs has been increasing exponentially with a 3.4-month doubling time (by comparison, Moore’s Law had a 2-year doubling period)[^footnote-correction]. Since 2012, this…