The Archive

ITGPT: generative pretraining framework for irregular timeseries data with missing values via transformer architecture.

User reports optimized inference performance for DeepSeek V4 Pro using ktransformers on AMD EPYC + RTX PRO hardware with tuned NUMA settings.

Comparison of LSTM, GNN, and Transformer architectures for NBA player movement forecasting in signal processing.

Transformer-based smartwatch framework for early psychotic relapse detection using uncertainty-driven anomaly detection on cardiac and motion signals.

Stateful transformer inference engine cuts streaming query latency from O(n) to O(|q|) via persistent KV cache and Flash Queries.

Tiny-scale study compares dense vs. sparse (MoE) transformers under matched parameter budgets; sparse outperforms with top-2 routing.

Wanted to share our latest paper on an alternative building block for Vision Transformers. [Illustration of our model's accuracy and dense features](https://preview.redd.it/x4acnx478w0h1.png?width=2457&format=png&auto=webp&s=3ce49caf2b0cdea5d35141aebb7297862fdc6a7d)…

TinyStories-260K transformer running natively on Game Boy Color via INT8 quantization and bank-switched ROM, no external compute.

VECA (Visual Elastic Core Attention) reduces Vision Transformer computational cost by eliminating direct patch-to-patch interactions while maintaining representation quality.

Attractor Models stabilize recurrent Transformers via fixed-point refinement with implicit differentiation, maintaining constant training memory across variable depths.

Theoretical analysis of geometric memorization in transformers showing embeddings encode relational structure vs. linear parameter scaling.

Paper formalizes positional encoding requirements for Transformer-based neural combinatorial optimization on vehicle routing, accounting for spatial structure unlike NLP.

Layer-wise relevance propagation extends attribution methods to EEG-based foundation models for interpretability verification.

FIS-DiT: training-free frame interleaving acceleration for video diffusion transformers in few-step inference regimes.

Shift-invariant transformer autoencoder for calibration-robust spectral unmixing in pharmaceutical and biomedical analysis.

DECO: sparse MoE architecture matching dense Transformer performance with reduced storage/memory for edge device deployment.

Mean-field analysis of Transformer token concentration in low-temperature regime via multi-particle system convergence.

Discussion of BDH architecture exploring memory in network weights vs. KV cache as alternative to transformer design.

EditMGT applies Masked Generative Transformers to localized image editing, outperforming diffusion-based approaches.

CLEF, a Transformer-based EEG foundation model using multitaper spectrograms, aligns clinical signals with neurologist reports via contrastive learning on 234-task benchmark.

RigidFormer applies Transformers to mesh-free rigid-body dynamics simulation with contact handling.

Semantic role understanding emerges during transformer pre-training without task-specific fine-tuning via linear probes.

Looped-MoE transformers scale better than dense looped models; sparse layers enable routing diversity across repeated passes.

AlphaExploitem extends AlphaHoldem for poker by learning to exploit suboptimal play beyond Nash equilibrium using hierarchical transformer reasoning over hand history.

Character-level Transformer for Tajik-to-Persian transliteration with 52K-word parallel corpus from verified lexicographic sources.

Engineer critiques transformer architecture limitations for exact reasoning tasks, argues prompt engineering cannot overcome fundamental probabilistic design constraints.

Byte Latent Transformer (BLT) accelerates byte-level LM inference via diffusion-based multi-byte parallel generation without subword vocabularies.

Proves convergence guarantees for stochastic training of Transformer attention layers with LoRA parameterization.

Novel logical characterization of encoder-decoder transformers via temporal logic and distributed automata provides theoretical foundation for cross-attention architectures.

Causal Energy Minimization framework derives Transformer block parameterization (MHA, gated MLPs) from energy function optimization.