§ Benchmarks

Analysis of emerging patterns in AI mathematical reasoning capabilities and predictability of model performance on unseen problems.

Proposes Sensorimotor Alignment as offline proxy for evaluating robotic foundation models by testing convergence between vision, language, and motion representations.

Introduces EpiSEM-Bench, executable benchmark with 300 task pairs measuring unsafe execution when tool-using LLM agents face missing safety-relevant facts.

Evidence-aware evaluation protocol for clinical RAG system on breast lesion corpus tests retrieval chunk size, overlap, and reranking.

Evaluates four open LLMs (8-27B) on Spanish MIR medical exam for uncertainty calibration, reasoning soundness, and multimodal grounding beyond accuracy.

Empirical benchmark of recent LLMs vs. fine-tuned encoder-decoder models on knowledge-graph-to-text generation, narrowing performance gap.

User simulators balancing complex intent adherence with natural conversational behavior for multi-turn AI assistant evaluation.

Overcooked-inspired benchmark for evaluating vision-language agent coordination on multi-player task planning in Unreal Engine.

Empirical evaluation of LLM inference optimizations (KV-cache pruning, speculative decoding, quantization) on Apple Silicon vs NVIDIA GPUs.

Study reveals ASR systems miss 22-42% of rare words in human speech, inflating measured TTS error rates by 2-3x on evaluation benchmarks.

Evaluation of foundation models for particle track reconstruction and identification in high-energy physics experiments.

Multilingual benchmark of 9 prompting strategies across 4 judges, 27 languages, 13 datasets reveals LLM-as-Judge robustness varies by model.

New benchmark for video-grounded research agents that synthesize multimodal evidence and generate interactive reports linked to video moments.

Study on prompt sensitivity in tool-augmented LLM comparisons, proposing matched protocol to isolate reasoning, tool-calling, and readout steps.

CNN feature aggregation method using Fisher Vectors for scene retrieval with improved accuracy on standard image benchmarks.

Comparative analysis of position bias across Transformer, state-space, and Mamba architectures in long-context QA tasks.

Evaluation protocol for personalized federated learning that reveals generalization gaps by testing global, rare-class, and unseen-class performance.

ProductWebBench: benchmark for frontend coding agents measuring both task completion and website continuity across 400 real-repo tasks.

DeepOpen: intent-classification system achieving 94.12% (BANKING77) and 98.02% (CLINC150) via SAM and ensemble methods.

MQAScore evaluates text-to-music alignment via large audio language model on five atomic dimensions (genre, instrument, mood, vocal, usage).

Dual Exam benchmark isolates multimodal agent action-selection from world-model state-evolution via fixed mechanisms across 15 domains and 1,488 test items.

Study reveals Theory of Mind benchmarks conflate explicit belief recognition with applied behavior prediction; Llama-3.1-8B shows 51.5% aggregate but uneven competencies.

Proposes trajectory-based benchmark framework for artificial wisdom, evaluating whether LLM recommendations adapt appropriately as decision conditions change.

SecLog-Bench introduces scalable cybersecurity benchmark with 29 attack scenarios, multi-source evidence, and dense reasoning annotations for LLM log analysis evaluation.

EEG-Captioner dataset with 160K signal-grounded QA pairs evaluates whether EEG-to-language models learn actual signal properties vs. generating plausible text.

Environment Duels benchmark evaluates frontier LLMs on creating and solving interactive environments; reveals distinct authoring vs. solving abilities.

Quant-Arena benchmark evaluates LLM agents on end-to-end quantitative trading research with A-share market data and out-of-sample testing.

BatchBench leaderboard measures AI robustness on biomedical images across unseen experimental batches using JUMP-CP dataset.

P2NPCBench pairs NP-complete and polynomial-time problems via AgentGAN adversarial rewriting to test LLM reasoning beyond recall on complexity boundaries.

Evaluates frontier AI models (Claude, GPT-4, others) on LIBERO-10 robot manipulation via closed-loop visual control without task-specific training.

SWE-DIVE benchmarks coding agents via multi-turn interaction with dynamic user simulation beyond single-prompt SWE-bench evaluation.

Agent Behavior Maps profile LLM-harness combinations via action log-likelihoods on reference trajectories, enabling performance prediction across 100 LLMs.

Open-source telecom alarm root-cause generator using real Internet topologies and RFC 8632 standards; dataset benchmarking contribution for infrastructure diagnostics.

TIDE introduces behavior-driven decision-tree induction for fine-grained LLM capability diagnosis beyond aggregate benchmark scores.

CulturalRiddles, a benchmark of 62 culturally-grounded riddles across 45 countries and 51 languages, evaluates LLM understanding of non-factual cultural reasoning.

System One decision models for legal authority verification with abstention, tested on CaseHOLD dataset with missing holdings.

First English-Mizo parallel corpus of Indian court judgments (13,168 pairs) from Gauhati High Court and Supreme Court.

FrontierCS 2 benchmark of 48 open-ended CS R&D tasks measures LLM agent contributions in working research environments.

Framework formalizing benchmark failure modes: measure mismatch and coverage gaps in difficulty distributions.

MindView benchmark evaluates generative multi-view spatial intelligence across 1,500 samples with tasks spanning geometry and semantic reasoning.

DAR-MGSM benchmark evaluates math reasoning across five Arabic dialects and Modern Standard Arabic.

TS2Vec adapted as single-encoder foundation model for time series classification without per-dataset retraining.

RTLRepairDataset and RTLRepairBench identify measurement confounds in HDL repair benchmarks using simulation and formal equivalence.

Empirical study comparing LLM-based and classical feature selection on tabular data with GRPO-adapted Qwen and Llama models.

CSFRBench measures gradient-based learning of compositionally sparse functions to reduce sample complexity via structure discovery.

DecisionPivot benchmark evaluates routing, guardrail, and verification models in LLM systems across 48 tasks with 40K test items.

OptionWorld benchmark reveals LLMs fail at multi-day task completion, with 35.5% of failures due to lost context before deadline.

ACE-Compression framework evaluates which core LLM abilities survive compression beyond task performance metrics.

MMEnterprise-Bench benchmark with 1,613 questions evaluates enterprise agents on reasoning over evolving multimodal workplace records.

Benchmark study of associative creativity in MLLMs vs. humans on abstract visual stimuli with controlled naming difficulty; evaluates originality and effectiveness.