The Archive

OrionMoE-Small: compact tabular foundation model using distribution-aware dual-axis Transformer with dataset-conditioned sparse MoE for in-context learning.

SPARO neural operator upsamples Head-Related Transfer Functions from sparse spatial measurements using Transformer-based latent mode processing for personalized binaural audio.

Analysis of transformer tabular foundation models reveals early layer predictive accessibility but quantifies per-input variation in depth benefits across layers.

Analysis of looped transformers reveals failure modes beyond computation: recurrent reasoners lose retention or state reusability when iterating past training depth.

ProtModernBERT applies modern architectural refinements to encoder-only transformers for low-compute protein language modeling on amino acid sequences.

Shows Mamba's retrieval deficit vs Transformers persists even when state capacity is unconstrained; selection mechanism insufficient for context utilization.

Scenario-Actor framework adaptively selects LLM decoding temperature based on inferred situational context from Transformer embeddings.

Byte-sequence transformer for malware detection operates on raw executable bytes to capture long-range dependencies without handcrafted features.

Transformers trained on sequences with context-dependent Markov sources learn source structure comparable to informed Bayesian predictors with ground-truth priors.

Graph Metanetworks extends weight-space learning to transformers by accounting for attention-induced continuous symmetries beyond neuron permutations.

Thinker-WM unifies RGB, depth, and action modeling in a shared Unified Dynamics Transformer for embodied AI.

Diffusion language models reconstruct masked tokens in parallel, offering an alternative to autoregressive generation. Yet their Transformer backbones typically carry visible evidence and evolving masked predictions through a single residual stream. We equip masked diffusion models with multiple residual streams that persist through depth and adaptive routing between them, then investigate how they support these complementary roles. On the pretraining task, two streams reduce reconstruction perplexity by 5.71% on validation data and 5.13% on average across external corpora compared with stand...

RELAGRAM applies latent recursion and compact transformers to combinatorial graph optimization, specifically the capacitated vehicle routing problem using edge token encoding.

HodgePhaseNet decomposes semantic flows in Transformer attention via discrete Hodge decomposition and phase topology.

T-PCOT uses Transformer-based partial optimal transport for cross-material magnetic core loss prediction with limited data.

Role-aware transformer for detecting sentiment, emotion, politeness, formality in dialogue via semantic role labeling.

SmolWAM compresses Cosmos Policy to 2B parameters for edge robotic manipulation using latent world-action models with cached text features and reduced transformer blocks.

Orthros attention dynamically switches computation patterns across sequence processing phases to maintain in-context recall while reducing quadratic complexity bottleneck.

CNN-Transformer with sigmoid label function for direction-of-arrival estimation; signal processing application.

Explores attention sink phenomena in text-to-image Transformers across diffusion and autoregressive models, extending LLM findings to vision tasks.

Ben Affleck is going viral for his deep knowledge of AI, from neural networks and transformers to open weights. The actor, who sold his AI filmmaking startup to Netflix earlier this year, is proving he's more than just a Hollywood star.

reViT achieves full-depth vision encoder accuracy using one recurrent Transformer block with depth-programmed expert mixtures, reducing inference FLOPs.

LeWAM introduces bidirectional transformer JEPA world model for robots supporting forward, backward, inverse dynamics and policy prediction end-to-end.

Marformer predicts conditional marginal distributions from partial observations for Bayes risk and Value of Information computation under incomplete data.

Layer-selective fine-tuning reduces LLM capability degradation by identifying which Transformer layers should adapt to specialized tasks.

Loop-native attention residuals stabilize looped Transformers across tens of thousands test-time iterations preventing performance degradation.

EngramEdit enables factual knowledge updates in LLMs via conditional n-gram memory architectures without retraining the Transformer backbone.

Transformers can execute causal discovery algorithms via fixed-weight architectures that reproduce optimization trajectories for learning acyclic causal graphs.

ResidualQuant enables 2-bit KV cache quantization for looped transformers by exploiting correlation across recurrent iterations, reducing memory bottlenecks.

QCATS enables sparse Transformer inference in databases by exposing query context and relational metadata to neural execution planning.