The Archive

Transformer architecture using Born measurements for attention and exact word-level evidence attribution via quantum-inspired mechanics.

Circuit analysis reveals in-context learning in 3-4B transformers implements key-value memory over label tokens enabling few-shot classification.

CATF uses Transformer manager-worker routing for context-aware time-series forecasting across diverse regimes.

LoopVQ achieves sub-1-bit KV cache quantization for looped transformers, reducing memory bottleneck via loop-axis redundancy.

CRF-based de novo peptide sequencing model with global normalization over mass-feasible candidates, replacing end-to-end transformer approaches.

ARLoop: parameter-efficient looped transformer with attention-based residuals allowing selective reuse of intermediate representations across recurrent depth.

Best-of-Depth training method for looped Transformers optimizes intermediate latent states as candidate set without depth-specific labels; inference-time scaling technique.

Vision transformer attention exhibits token-level biases independent of queries; reformulated attention with input-dependent bias term makes this explicit.

DPPG uses Transformer autoencoder and task-conditioned diffusion to generate RL policy network parameters with generalization capabilities.

LoopKV optimizes KV cache compression for looped transformers like Ouro by adapting quantization precision per (loop, layer) cell.

Exemplar-free continual learning on pre-trained Vision Transformers increasingly relies on Low-Rank Adaptation (LoRA), yet prevailing methods govern the low-rank update space through strict subspace isolation. By projecting each incoming task into the approximate null space of historical token activations, isolation-based learners equate any cross-task feature overlap with catastrophic interference. We show that this orthogonality premise induces a structural bottleneck we term \emph{spectral starvation}: when down-projection bases are locked before task optimization, the gradient norm reachi...

Skin cancer diagnosis, particularly differentiating melanoma from benign nevi, is challenging due to the visual similarity between lesions. Although convolutional neural networks (CNNs) and vision transformers (ViTs) have shown promising performance, their effectiveness often deteriorates on data from heterogeneous clinical sources. Conventional domain adaptation methods address domain shift but require access to source data during adaptation, which is often infeasible due to privacy regulations. Multi-source-free unsupervised domain adaptation (MSFDA) addresses this limitation by leveraging ...

Computer programmers chain together the same reusable computational primitives, for example, conditional statements or loops, but in different orders to implement different algorithms. In contrast, existing language model architectures always execute their layers in the same order. We introduce Library of Layers (LoL), an architecture that dynamically orders its layers during inference to solve the problem at hand. LoL has a single hyperparameter, the size of its global library of transformer blocks, that allows it to transition seamlessly between parameter reuse and sparsity, generalizing th...

Parameter-efficient continual learning on pre-trained Vision Transformers largely relies on a costly premise: preventing catastrophic forgetting requires either spawning a dedicated LoRA branch per session---causing adapter parameters to scale linearly with task horizon---or applying sparse Mixture-of-Experts gating symmetrically to both forward and backward passes, which sacrifices representational rank at test time. We argue that reading consolidated knowledge and writing new task credit are fundamentally asymmetric operations. Because visual features in a shared low-rank space distribute a...

Generative models increasingly simulate users at different skill levels in education and AI evaluation. Yet it remains unclear whether lower-skill behavior reflects less internal knowledge or a different use of retained knowledge. This distinction is difficult to study in natural language, where proficiency and relevant knowledge are hard to measure. In this work, we study Maia-2, a chess transformer that predicts human moves conditioned on player skill, using quantitative skill ratings and deterministically labeled chess concepts. We hold board positions and model weights fixed and analyze p...

RouteREPA adapts representation alignment for vision transformers via routed learning, resolving conflicts between contrastive and discriminative objectives.

InfoNCE regularization on diffusion transformers yields marginal gains due to noise shortcuts; proposes refined contrastive approach.

Study compares quantization methods for recurrent language models versus Transformer softmax architectures on inference properties.

Detection Transformer directly predicts T1-T2 MRI compartments, outperforming inverse Laplace transforms for microstructure analysis.

Conic programming framework certifies transformer attention robustness bounds; proves NP-hardness under input perturbations.

UpLoop pretraining schedule trains looped Transformers via weight reuse matching dense baseline FLOPs for improved reasoning depth.

Hierarchical Residuals decouples residual width from compute width in Transformers using block-level outer residuals with learned projections.

SpecSlot decouples pixel granularity from sequence length in pixel-space generative transformers via role-coupled tokenization.

Multi-view MoE uses multiple routers per token to improve expert selection; dense ranking shows weak router agreement in mid/deep transformer layers.

Lineage inpainting combines tree-structured transformer with stochastic interpolant generative model for cell dynamics using lineage context in embryogenesis.

TACFormer applies transformers to target-conditioned PROTAC design for in-silico protein degradation molecule discovery.

Formal semantics framework for Transformer parameterized coordinated transformations with intensional interpretations.

Autoregressive transformer for antenna design optimization; domain-specific CAD/physics application without general ML infrastructure relevance.

HotPOT scales PDE pre-training using heterogeneous physics experts with learned routing to mitigate cross-task interference.