The Archive

Linear Vision Transformers initialized from Softmax ViT weights via attention transfer outperform from-scratch training, reducing efficiency-accuracy trade-off.

Mechanistic analysis reveals read-write asymmetry in Transformer attention/FFN layers: blocks ignore extreme-value features while reading but not writing.

Attention Graphons applies graph limit theory to study whether Graph Transformer attention matrices converge to stable kernel structures.

ADPTNet proposes neuromorphic sequence model with adaptive prescriptive timescales and non-linear recurrence as energy-efficient Transformer alternative for long-range dependencies.

Mathematics of Arrays formally derives memory-optimal transformer kernels (attention, decode, RMSNorm, MLP) validated across Purdue Anvil and NCSA Delta HPC clusters.

Associative algebra layers replace standard matrix multiplication in Transformers with sparser operations, achieving quadratic arithmetic complexity for fixed block sizes.

Extender introduces log-structured Transformer variant with separate residual and concatenation channels, reducing key-value projection dimensionality.

Trust Guided Decision Transformer uses next-state prediction error to detect context drift in offline RL.

Theoretical analysis of transformer in-context learning under heterogeneous geometric data with minimax-optimal bounds and geometry adaptation.

As language models grow, scaling dense architectures becomes increasingly expensive. In a dense transformer, every token passes through every layer, so adding... As language models grow, scaling dense architectures becomes increasingly expensive. In a dense transformer, every token passes through every layer, so adding capabilities increases computation for both training and inference. Mixture-of-experts (MoE) architectures take a different approach to scaling by using many subnetworks, or experts, while activating only a small subset for each token. Source

Studies nonequilibrium dynamics of minimal recurrent transformer with repulsive attention; discovers bifurcations, chaos, and emergent locality.

Dual-encoder transformer estimates planetary boundary layer height from satellite radiances using MetOp–ERA5 dataset; multimodal spatially incomplete data regression.

COMPASS: decentralized multi-robot architecture using LLM reasoning with spatial transformers for coordinated control of robot collectives.

Transformer KV caching architecture with shared global keys/values plus layer-specific local history reduces perplexity 1.4% vs. current-token baselines.

Task-induced Riemannian metrics improve Vision Transformer feature spaces by computing task-sensitive geometry via decoder Jacobian pullback.

Newton-Schulz spectral normalization applied to Vision Transformer attention heads improves training efficiency without adding parameters.

Autonomy-1 will have a small, transformer-based AI model taking charge of a space probe.

Comparison of CNNs and vision transformers via EEG encoding models across network depth, testing token representations.

EMERGE applies equivariant graph diffusion to resolution-agnostic 3D point cloud generation, addressing topological structure absent in Transformer/VAE approaches.

Vision Transformer saliency maps on breast MRI show visually plausible but unfaithful explanations, exposing evaluation pitfalls.

GeoPair: training-free Transformer compression via cross-layer factorization preserving activation geometry.

SLICEChat integrates progressive token pruning with Mamba-Transformer encoder for scalable gigapixel whole-slide pathology image processing.

XSQ-AST framework localizes synthetic speech artifacts via saliency maps and phoneme alignment, validated on perceptual quality dimensions.

Transformer architecture enhancement for symbolic ODE expressions preserving commutative properties via improved positional embeddings.

Mechanistic analysis reveals transformer trained on navigation task maintains coherent spatial world model; behavioral failures traced to feature interference.

Latent Recurrent Transformers trade temporal depth vs. physical depth during decoding via latent thought tokens between vocabulary tokens.

Deep Noir automatically discovers LLM activation steering parameters via Logit Lens and causal attribution, improving task performance 16–42 percentage points.

SETTer applies sparse-encoder Transformers to long-horizon multivariate time-series forecasting with oversmoothing mitigation.

Architectural interventions like looped transformers modify scaling exponents and enable exponential performance gains independent of compute alone.