The Archive

Framework for building 2D-constrained minimal transformers enabling full visualization of internal representations and inference of learned algorithms from geometry.

Smaller transformer architecture executes Lloyd's k-means clustering in-context with reduced embedding size versus prior work.

Flow matching models with modified training objectives improve multi-step reasoning; proves two-layer Transformer solves graph reachability with steps proportional to target distance.

EntroPrefill uses Renyi entropy to prune mid-prefill context in transformers for RAG with stability guarantees on attention mass.

Framework interprets Multimodal Diffusion Transformers' contextual tokens by mapping them into LLM input space via lightweight bottleneck.

SeqSmoother uses transformers to estimate sleep heart rate from wrist accelerometers without optical sensors.

MC-Sparse reduces latency in diffusion transformers via training-free sparse attention, addressing quality degradation from token grouping and interaction selection errors.

MatrixFormer pre-trained transformer predicts full distributions for missing matrix entries in single forward pass, trained on synthetic low-rank matrices.

SAGA-CDR pairs books with mood-matched music via transformer sentiment embeddings and conditional GANs for cross-domain recommendation.

Hybrid Linear Attention (HLA) enables query-dependent chunk-level access for long-context transformers via dynamic mixing on Gated DeltaNet.

WNet replaces Transformer self-attention with discrete wavelet transform for linear-cost token mixing on long sequences.

Survey of numerical linear algebra techniques foundational to LLM computation, from eigenvalue solvers to modern transformer implementations.

Formal verification bridge from transformers to weighted automata enables rigorous reasoning about LLM behavior in safety-critical deployments.

Solver-aware target selection for caching in few-step diffusion transformers optimizes tensor reuse post-distillation.

Probing video Diffusion Transformers reveals kinematic and rigid-body physical quantities are linearly decodable from internal activations despite poor physics reasoning benchmarks.

NEPA improves diffusion transformers by predicting next embeddings as conditioning instead of static embeddings.

LoopCD training-free contrastive decoding for Looped Transformers exploits intermediate recurrent states as weak predictions to improve token selection without auxiliary models.

Faynt 10M/75M Transformer policies master Super Smash Bros. Melee across 26 characters, winning 98.4% same-character matchups vs. specialist releases.

Loop Scaling Laws jointly model recurrence and sparsity to characterize effective-parameter gains in looped MoE transformers.

Looped Diffusion Transformer scales computation by reusing Transformer blocks within denoising steps to improve text-to-image generation without larger models.

Learnable subspace projection for transformer compression via error propagation analysis, improving on local closed-form criteria.

Mixture of Function Attention (MoFA) parameter-free modification to Transformer attention explores softmax/sigmoid ratios as intrinsic inductive bias.

Detection Transformer approach for fiber orientation and microstructure quantification from multi-shell diffusion MRI without Monte-Carlo inversion.

Association Profile Set-Temporal transformer adapts intracortical motor decoders across sessions with frozen weights using association profiles.

Hybrid transformer-CNN model (PAtteRNS) applying separate self-attention to temporal, spectral, and spatial dimensions for crop segmentation in Sentinel-2 satellite imagery.

LIFT architecture enabling recurrent state propagation across transformer layers during pretraining, eliminating feed-forward-only bottleneck for information reuse during generation.

Study explains how local mixing layers encode positional information in NoPE transformer attention without explicit position encodings.

Telescopic Language Models train nested-capacity Transformers via stochastic prefix supervision, enabling single model to serve multiple compute budgets.

NHMO solves elliptic PDEs on variable domains using transformer-based boundary kernels supervised by Walk-on-Spheres sampling.

Study finds looped transformers achieve steeper accuracy-compute scaling slopes than non-looped baselines during test-time decoding, though with suboptimal absolute performance.