The Archive

Weakening neurons in GLU-based LLMs suppress residual stream directions via negative input-output cosine similarity, concentrated in late layers across models.

NISQ-compatible quantum transformer with variational quantum blocks replaces attention layers for four-token sequence modeling.

Systematic study of pruning effects on LLM tool-calling across Transformer, hybrid, and MoE architectures; evaluates 19,500+ smart-home instances.

Persistent recurrent memory module between transformer layers reduces language model evaluation loss 28.5% with 3.7% parameter overhead.

Transformer-based limit order book forecasting for counterfactual market impact; financial ML, not frontier AI.

Directional decomposition analysis reveals parallel components in transformer representation evolution beyond residual paths, with space-dependent asymmetry in attention/MLP updates.

Mixture of experts (MoE) has become one of the defining architectural trends in large-scale AI model training. DeepSeek, Qwen, and Mixtral are examples of MoE... Mixture of experts (MoE) has become one of the defining architectural trends in large-scale AI model training. DeepSeek, Qwen, and Mixtral are examples of MoE models that match or exceed the performance of dense model counterparts at a fraction of the training compute. MoE models provide efficient training through conditional computation. Instead of one dense feed-forward network (FFN) shared… Source

Type diversity in training data, not Transformer architecture, explains why structural generalization lags lexical generalization in compositional tasks.

UniPart enables zero-shot language-grounded 3D part segmentation for robotic manipulation using cross-modal Transformer conditioning on CLIP embeddings and LangPart-1M dataset.

Empirical study showing Mixture-of-Experts models overfit more severely to repeated training data than dense Transformers across 80M-1B scale.

Studies distance generalization in Transformers—whether positional encoding helps when inter-token distances differ between training and inference.

Annealable soft-prior Transformers: smooth annealing preserves retrieval circuits after positional prior removal; hard switching fails.

Analysis of transformer interpretability: 53 components drive 90% of token predictions; 13 critical; 8 sufficient alone. Mass of off-predicted-token contributions exceeds predicted by 7x median.

Transformer-based emotion classifier applied to 1.7M Congressional speeches to correlate emotional expression with legislative outcomes.

Theoretical proof that frozen transformers perform data generation via in-context learning without parameter updates.

Theoretical analysis of transformer length generalization via C-RASP language; resolves contradictions in prior length-generalization bounds and empirics.

Linear algebra unification of 14 works on transformer attention rank, compression, and KV-cache efficiency via SVD and eigendecomposition.

UniMate: diffusion transformer for text-to-motion synthesis across arbitrary skeleton topologies without per-skeleton retraining.

RegionFed: personalized federated learning for retail search across heterogeneous regions, fixing transformer collapse in parameter-level FL.

Study diagnoses visual grounding failures in visuomotor imitation policies under distribution shift using Action Chunking with Transformers and controlled distractor objects.

Combines pruning, quantization, and distillation to compress Vision Transformers for crop disease detection on mobile agricultural devices.

Open toolkit for measuring context-dependent word sense representation in transformer embedding layers using controlled bridge-form constructs.

Text-AB unified diffusion-transformer framework for voice dubbing and full-duplex dialogue via alignment-free latent synthesis with DAC-VAE compression.

Trace as State reformulates long-context Transformer reasoning by caching reasoning traces as conditional state, reducing memory exponentially in worst case.

Open-weight transformers show logical validity representations remain decodable from hidden states despite near-chance behavioral performance.

AGI Maze Prediction Datasets benchmark evaluates Transformer world modeling through procedurally generated grid-world transition and state prediction tasks.

TREVIS: Tree Transformer VAE approach jointly optimizes decision tree predictive performance and structural sparsity.

Vision Transformer method for cancer grading combines nuclei classification with histopathology images via semantic-guided multimodal preprocessing.

Analysis of Transformer attention heads reveals scaled idempotence pattern in OV operators across 2.8B–235B parameter models via principal-coordinate factorization.

ViTAMINS integrates synthetic hard negatives into vision transformer pretraining, improving ImageNet and transfer learning with +11.3% gains.