The Archive

Transformer parameterization with 280 parameters achieves perfect length generalization on Boolean algebra circuits via depth-1 reduction.

Hybrid CNN-Transformer for Antarctic sea ice forecasting using seasonal factorization; domain-specific climate modeling, not AI capability research.

AcrossVAM1.0 factorizes robot video prediction into object-centric motion and appearance using text-assisted 0.28M-parameter Transformer on frozen SAM3-DLP codec.

Study showing Vision Transformer attention heads in multimodal LLMs specialize into object/background roles; SHS-Index metric correlates specialization with downstream performance.

SCG: adaptive Vision Transformer width/depth expansion for JEPA world models, starting minimal and growing task-driven.

SCIT (Suffix Cache Interchange Test): causal protocol for testing which transformer components carry latent reasoning in chain-of-thought models.

Study on architectural choices for latent-space transition prediction in robot World Action Models, comparing Transformer-based approaches to operator-structured designs.

Importance-scoring metric applied to Transformer attention heads for tabular data, demonstrating head-drop robustness across 40 datasets.

Theoretical rank-error bounds for LoRA on Transformer attention prove KL divergence scales with distance between target and source attention distributions.

RACE: Statistical framework for evaluating functional consistency of Transformer neurons at scale via forward-pass perturbation analysis.

ChebBooster: training-free Chebyshev polynomial extrapolation for stable diffusion transformer inference acceleration without costly cache reuse schemes.

Next-scale transformer enables photorealistic multi-view human face synthesis at high resolution without 2D pre-training; narrow multimodal application.

Transformer model for predicting acute COPD exacerbation from home ventilator data with temporal awareness.

Zeroth-order optimization for spiking neural networks on in-memory computing accelerators; hardware-specific fine-tuning technique.

Tydra hybrid Transformer-SSM architecture reduces TabPFN inference time 30% on tabular data while maintaining accuracy.

OAttention replaces binary attention masks with token presence coefficients to enable graduated token participation in Transformer layers.

BERT-LER pretrained on 75M-patient EHR dataset uses discrete laboratory tokens with percentile binning and Integrated Gradients for interpretable clinical prediction.

Sparse Autoencoders reveal feature evolution and migration dynamics across Vision Transformer depth and training time via CLS-token representation analysis.

Recirculation inference-time technique adds learned recurrence to transformers enabling stateful belief tracking, reducing perplexity with minimal latency overhead.

A language model's output does not by itself provide verifiable evidence about the internal computation that produced it. We study computational provenance: whether generated text can carry detectable evidence of which causally relevant internal state occurred. We test a bounded form of this idea in two controlled architectures: a modular feed-forward neural network and a transformer-based model. Both architectures are trained on the same arithmetic task with a mandatory pathway through two discrete intermediate states, allowing different internal paths to produce the same answer. We delibera...

Reliable optimization is central to neural network (NN) training, yet Adam, the default optimizer for modern LLMs, rests on a fragile foundation. This thesis develops a principled grounding for Adam and motivates new designs. First, we revisit Adam's divergence--convergence debate and show the existence of a problem-dependent phase transition: with properly chosen, batch-size-dependent hyperparameters, Adam converges, whereas under small-$β_2$ regimes it can diverge. Second, we investigate why Adam substantially outperforms SGD on Transformers through Hessian structure. We find that the Hessi...

The language-model head maps a hidden state of width D to a vocabulary of size V, so its transpose can return at most D independent directions to the Transformer. Godey and Artzi argue that this severe projection is a harmful optimization bottleneck. We separate the geometry from the causal claim. Our backward-only intervention keeps the ordinary logits and the exact LM-head parameter update while reducing only the rank of the gradient sent into the Transformer. Across five paired seeds on byte-level and BPE-8192 WikiText-2 models, reducing backward rank increases validation loss. An equally ...

We present QuantumPhaseNet, a gauge-covariant geometric and quantum-spectral extension of Transformer representations. Context-dependent semantic states are modeled as complex amplitudes; a covariant phase rate induces a semantic wavelength used as a proxy for conceptual scale; and low-frequency graph modes define a document-level discourse direction. The theoretical part establishes local gauge invariance, unitarity of the quantum block, boundedness and conditional stability of WavePhase Attention, and a calibratable hallucination-risk formulation. We also implemented a fully offline Validat...

We address a fundamental gap in 3D-LLMs: existing models focus on single-object/scene description, struggling with detailed, inter-object comparison. We propose a framework for detailed object-level reasoning across multiple objects with three components: (1) MO3D (Multi-Object in 3D), an instruction dataset requiring fine-grained multi-object comparison; (2) Multi-3DLLM, using a minimal Patch-Interaction Transformer (PIT) that models inter-/intra-object relationships while preserving local geometry; (3) Mini-apps, two application-driven benchmarks (Shape Mating, Change Captioning) that probe...

High-order multiple-input multiple-output (MIMO) detection requires efficient search over a large discrete symbol space while producing reliable soft information for channel decoding. This paper develops a learning-to-transition (L2T) framework that formulates MIMO detection as a stochastic sequence of complete-vector transitions. At each transition, a channel-coupled Transformer updates both the instance embedding and the sampling policy, while a blockwise autoregressive factorization captures inter-stream dependence with moderate sequential complexity. For hard-output detection, a transitio...

Recipe data arises in domains such as materials synthesis, pharmaceutical formulation, and industrial manufacturing, where procedures are represented as ordered sequences of steps containing heterogeneous structured fields. Existing tabular learning methods typically flatten this structure into fixed-schema representations, limiting their ability to capture hierarchical field interactions and procedural dependencies. We propose RecipeNet, a hierarchical Transformer architecture that encodes field-level interactions within each step and sequential dependencies across steps through stacked Tran...

Cross-Tabular Data Generation (CTDG) seeks to learn a generative model from multiple heterogeneous tables and produce new synthetic tabular datasets. However, existing synthetic tabular data generation methods are largely restricted to single-input-table scenarios and struggle to effectively handle multiple heterogeneous tables with diverse feature sets. To address this limitation, we propose a two-stage framework for cross-tabular data generation. In the first stage, each heterogeneous raw table is transformed into a standardized statistical table with the same set of columns across all tabl...

Neural Architecture Search (NAS) is naturally formulated as a bilevel optimization problem, where the upper-level optimizes the architecture using validation performance and the lower-level trains network parameters using training loss. However, NAS is computationally expensive due to discrete architectural decisions, exponentially growing search spaces, and the high cost of training candidate architectures. This work develops a general bilevel optimization framework for NAS across diverse architectures, including MLPs, CNNs, RNNs, and Transformers, to identify compact architectures with stro...