Foundation models by year¶

简体中文 · English

Browse by first public release year, newest first within each year.

2026¶

2026-10 · XLA from the Trenches: Correctness on TPU Workloads at Scale (xla-trenches): Isolation proceeds from a large HLO graph to a fusion, then inspects intermediate VREG values without splitting the fusion. This is compiler-correctness infrastructure, not a new recommender or an industrial recommendation paper passing an A/B gate. Not reproduced due to cost (review complete); research material only, no executable entry.
2026-10 · GAVAGAI: Referential Alignment for Infant-Scale Vision-Language Models via Balanced Optimal Transport (gavagai): GAVAGAI adds both an uncapped Nothing column and upper capacities on visual regions. It does not require every visual column to use all its capacity. Bounded component only; full experiments are not reproduced.
2026-10 · ProLand: A New Infeasible Manifold Optimization Method for LoRA Fine-Tuning (proland): It first forms coordinated spectral directions in the product space of \(BA\), then independently restores feasibility with Landing. Running Muon separately on A and B is not an equivalent algorithm. Bounded component only; full experiments are not reproduced.
2026-10 · Folding Attention Biases into Keys Makes Compacted KV Caches Run on Standard Kernels (fold-bias): This is an approximation, not an exact replacement for all future queries: a dot product is linear in the query, so it cannot represent a nonzero constant over the entire query space. The fit constrains the mean projection to one on the calibration queries and limits its variation. Bounded component only; full experiments are not reproduced.
2026-10 · GUI-JEPA: Candidate-Conditioned JEPA Latents as a Fast Action Policy for Mobile GUI Agents (gui-jepa): The policy does not equate latent proximity to a goal screenshot with action value. Its predictor is trained on actual candidate sets with a listwise decision loss; the structural head sees candidate visual information through the conditioned latent. Bounded component only; full experiments are not reproduced.
2026-10 · Multi-State Linear Attention with Generalized Second-Order Routing (merlin): This changes segment aggregation; it does not replace the recurrent memory with the moment statistics or recover exact full softmax attention. The second-order cumulant analysis assumes equal-length segments. Bounded component only; full experiments are not reproduced.
2026-10 · Select-TTT: Query-Conditioned Selective Fast Weight Updates for In-Place Test-Time Training (select-ttt): The answer-chunk experiment motivates the method; identifying gold answer chunks is not part of the deployment rule. The scoring interface takes the question, not answer annotations. Bounded component only; full experiments are not reproduced.
2026-10 · SWIM: Learning What Matters from Behaviors for JEPA World Models (swim): Stage 1 gates only the future displacement, leaving the entire current state visible. Stage 2 scales previously standardized states by the square root of the frozen weights and matches latent and state distances for the same frame pairs. Bounded component only; full experiments are not reproduced.
2026-10 · ResCom: Activation Residual Compensation for Large Language Model Quantization (rescom): Ranking channels by residual magnitude is not equivalent: even a small residual can be amplified by the main projection. ResCom uses cross-statistics of residuals and output errors to allocate its rank budget for output reconstruction. Bounded component only; full experiments are not reproduced.
2026-10 · DT-JEPA: Learning to Predict Future Representations for Decision-Making (dt-jepa): A reparameterized sample conditions the Decision Transformer via AdaLN with separate scale and shift MLPs. Offline training combines behavior cloning and NLL; online adaptation uses the smaller of two target critics for a multi-step target and a Q1-guided actor objective. Bounded component only; full experiments are not reproduced.
2026-10 · EditJev: System One Routing for Image Editing Models (editjev): The router takes the last non-padding hidden state of one image/instruction forward pass and predicts a utility vector in fixed candidate order. At serving time, add a fixed normalized speed prior weighted by beta and select once; candidate editors are not evaluated before choosing. Bounded component only; full experiments are not reproduced.
2026-10 · PathJev: Decision-Native Vision–Language Modelling for Histopathology (pathjev): The head and two-stage training follow OmniMed-Jev. The pathology contribution is the 40K decision states, typed labeling, patient/slide and overlap audit, unseen-organ tests and generative readout comparisons. Not reproduced due to cost (review complete); research material only, no executable entry.
2026-10 · Inside-JEPA: From Prediction to Relational Grounding in Latent World Models (inside-jepa): Entity latents are width 32, ordered-pair channels eight or sixteen. Four slots, padded and diagonal terms fixed at zero, give 256 dimensions for eight channels, not 224. Bounded component only; full experiments are not reproduced. Method reference 1
2026-10 · Jev-DLC: Decision-Level Continual Learning from Typed Feedback (jev-dlc): Feedback can affect only later requests. A decision must be committed before its feedback is revealed. Bounded component only; full experiments are not reproduced.
2026-10 · LazyFlash: Forward-Dense, Backward-Sparse Attention for Accelerated Long-Context Fine-Tuning (lazyflash): The retained blocks are inverse-probability weighted. Speed depends on skipping tile loads and computation inside the FlashAttention backward kernel. Not reproduced due to cost (review complete); research material only, no executable entry.
2026-10 · OpenAttention: One Percent Is Enough with Tail Proxies (openattention): Geometry uses prefill-visible queries, not future answers. ErrorGate estimates proxy error per cluster and allocates exact reads by capped water filling. Bounded component only; full experiments are not reproduced.
2026-10 · Rethinking Sequence Compression with Test-time Regression (ttr-compressor): The outer task trains projection and regularizer through the exact solve. Test-time refitting changes input-dependent coefficients, not model weights; the base has \(2d\) trainable parameters. Bounded component only; full experiments are not reproduced.
2026-10 · Reverse-KL Policy Distillation: Statistical Guarantees and Limits (rkl-policy-distillation): Original algorithm excerpt Bounded component only; full experiments are not reproduced.
2026-10 · ALPS: Visual State Encoders for World Models (alps): A four-layer predictor uses geometry to predict clean crop B teacher tokens from enhanced crop A. Supervise only B centers in A; discard the predictor and retain the student and language interface. Bounded component only; full experiments are not reproduced.
2026-10 · Building Auditable AI Text Detectors with Sparse Autoencoders (detector-scope): A sparse linear head yields logit contributions verifiable by zeroing features. Neither a text description nor a rank score certifies concept semantics, human review or low false-positive risk. Bounded component only; full experiments are not reproduced.
2026-10 · MoEZip: Routing-Aware KV Cache Compression for Sparse Mixture-of-Experts LLMs (moezip): Use Fisher trace from prefill instruction-token gates as a sensitivity proxy and propagate it to context locations with headwise maximum attention. Retain global routing anchors in all layer/heads and fill remaining budget using external base-compressor scores, without future answer tokens. Bounded component only; full experiments are not reproduced.
2026-10 · Right On Q: Recovering Reasoning under Quantization via Self-Generated Trajectories (right-on-q): Calibrate each quantized block against the full-precision output, then have the quantized student generate and let the frozen BF16 teacher score those same prefixes. A token-specific probability ratio adjusts the balance of forward and reverse KL rather than using one coefficient for a whole response. Not reproduced due to cost (review complete); research material only, no executable entry.
2026-10 · When Multiple Choices Are Correct: Asymmetric Probability Allocation in Jev and Other Decision Models (jev-multicorrect): Distinguish normalized categorical shares from independently queried binary correctness probabilities; the former are not calibrated binary confidence. The paper studies Jev, Luna and Gemini. Bounded component only; full experiments are not reproduced.
2026-10 · LocaLeJEPA: Maximum Entropy Representations under Geometric Priors (locale-jepa): Retain clusters of size at least four; if none qualify, use the whole batch and record fallback. Cluster centers need not approach one common origin. Bounded component only; full experiments are not reproduced.
2026-10 · RACE: RoPE-Aligned Attention Compression Based on Query-Key Activation Energies (race-rope-compression): Fold both into Q/K/V/O. Preserve the original frequencies rather than renumbering pairs. Bounded component only; full experiments are not reproduced.
2026-10 · Stratified Sparse Attention: Reducing the Sparse-to-Dense Gap at Extreme Sparsity (stratified-sparse-attention): Keep \(\lambda q_i\ge1\) exactly; renormalize residual proxy probabilities, draw the remaining budget with replacement and deduplicate. The paper's optimized fixed-heavy-fraction kernel differs from adaptive Algorithm 1. Bounded component only; full experiments are not reproduced.
2026-10 · DeepSEA: A Multilingual Evaluation Benchmark for Southeast Asia (deepsea): Compare by resource tier and analyze English pivoting, errors and tokenizer efficiency. This is a benchmark, not a parameter update. Bounded component only; full experiments are not reproduced.
2026-10 · FLINT: Fixed-budget Late Interaction (flint): FLINT uses separate learned query/document latent sets after a shared backbone, with shared cross-attention and FFN to emit fixed \(V\) vectors per side. This is neither first-token truncation nor averaging. Bounded component only; full experiments are not reproduced.
2026-10 · Kronecker Delta Rule for Long Context Linear Attention (kronlstm): In vector form \(a=v\otimes k\), the step is \(s'=(I-\beta aa^T)s+\beta ca\). Overlap depends on both key and value inner products; orthogonality of either preserves that association for the current update, not arbitrary long-sequence retention. Bounded component only; full experiments are not reproduced.
2026-10 · T-MUON: Tensorizing Muon via t-SVD for Large Language Model Pre-training (t-muon): Grouping and transform are separate choices. The paper compares adjacent layers, QKV, heads of a single matrix, and a stack of QKV heads. Bounded component only; full experiments are not reproduced.
2026-10 · Muse Glimmer 30B: An On-Device Agentic Model (muse-glimmer-30b): Training starts with distillation from Muse Spark logits. Near the end of pretraining, it switches to cross-entropy to address top-\(k\) truncation bias. Not reproduced due to cost (review complete); research material only, no executable entry.
2026-10 · BMuon: Budgeting Muon's Spectral Geometry for Structured Compressibility (bmuon): Muon's spectral-norm update need not produce weights that are easy to prune by hidden channel. BMuon neither adds a forward-pass gate nor merely prunes a pretrained Muon model. Hidden channels from all MLP blocks share one allocation budget, and the update geometry changes jointly on the input-row and output-column sides of each channel. Bounded component only; full experiments are not reproduced.
2026-10 · Jev-as-a-Judge for Brazilian Portuguese Information Retrieval: Relevance Assessment and Reranking (jev-portuguese-ir): On Quati it separately uses an independent Noul probability that each document directly answers the query, gathers every batch, and sorts the results. Candidates do not compete in a single Choice, probabilities are not mixed with BM25 scores, and there is no trained calibrator. Evaluation protocol interface; not a model capability result.
2026-10 · Persistent Planar Memory for Video World Models (ppmem): Each local plane has seven parameters: center depth, two rectangle scales, and a quaternion. Planes are merged after transformation to world coordinates. Not reproduced due to cost (review complete); research material only, no executable entry.
2026-10 · Sink-Sparse Hybrid Attention for Efficient Inference in T2I/T2V Diffusion Transformers (sink-sparse): One offline calibration can therefore select a static sparse support. A lower-resolution global KV branch recovers distributed context without changing weights or the sampling schedule. Bounded component only; full experiments are not reproduced.
2026-10 · AdaLoop: Thinking with the Vision Encoder (adaloop-vision): AdaLoop freezes pretrained visual weights, pools question tokens to modulate shared visual blocks, and repeatedly updates the visual state. The question can change how many passes the model uses rather than leaving all reasoning to the language decoder. Bounded component only; full experiments are not reproduced.
2026-10 · Architecting for Hardware Diversity: Meta's End-to-End Co-Design for High-Performance AI Training on AMD GPUs (hardware-diversity): Using recommendation training as its case, the paper jointly changes the model, parallelism, and memory policy, uses an analytical model to choose GEMM configurations, and uses agents for compilation, debugging, and kernel porting. This is systems research, not one new recommender model. Not reproduced due to cost (review complete); research material only, no executable entry.
2026-10 · BitSparse: Compact Activation Storage for ReLU Feed-Forward Networks (bit-sparse): BitSparse stores bitmasks, positive values, and tile prefixes and reconstructs them losslessly in backward, rather than trading lossy quantization or extra forward passes for space. Section 3 defines the format; Eqs. 1–2 and Appendix C specify ReLU/ReLU² linear backward. Bounded component only; full experiments are not reproduced.
2026-10 · Compression Does Not End at Selection: Post-Selection Positional Geometry for Sparse KV Caches (rope-relay): It sorts retained states by their original timestamps, compacts the position coordinates, and synchronizes question and generation positions. The method follows §3, Algorithm 1 in Appendix A, and the numerical conventions in that appendix. Bounded component only; full experiments are not reproduced.
2026-10 · Designing Transformers for Continuous Diffusion Language Models (langdit): LangFlow representation, cross-entropy, and sampler stay unchanged. Plaid only adopts the compatible subset; the gradient rules are not interchangeable. Bounded component only; full experiments are not reproduced.
2026-10 · DropKV: Decoupling Residual-Output Perturbation for Near-Optimal KV-Cache Eviction (drop-kv): Section 3 and Algorithm 1 define scoring. Section 4's guarantees are conditional, not global optimality for arbitrary inputs. Bounded component only; full experiments are not reproduced.
2026-10 · EgoP²C: Internalizing Plan-Perceive-Compose for Spatial Reasoning in Egocentric Video (ego-p2c): EgoP²C first trains separate plan/compose (PC) and perception (V) LoRA teachers on a frozen VLM. It copies selected FFNs into PC, V, and general direct-answer (G) experts. Bounded component only; full experiments are not reproduced.
2026-10 · Fractional Power-of-Two Quantization for Efficient and Effective Multiplier-Free LLM Inference (fpot): FPoT halves the exponent spacing for 4-bit weights, then approximates the square-root-of-two coefficient for odd exponents with 1.5 for shift-add execution. Calibration searches the actual approximate grid, avoiding the extra error from changing coefficients only after calibration. Bounded component only; full experiments are not reproduced.
2026-10 · Gen4P: Video Generators are 4D Perceivers (gen4p): Gen4P does not recover geometry through repeated denoising: it encodes clean video latents with one DiT pass and uses on-demand cross-attention queries to output 3D tracks, point clouds, depth, and camera parameters. Continuous-time queries support different lengths, frozen depth LoRA supplies an additional geometry prior, and separate 4D LoRA learns the new tasks. Not reproduced due to cost (review complete); research material only, no executable entry.
2026-10 · LAMP: Layerwise Adaptive Mixing of Predictors for Training-Free Few-Shot Vision-Language Adaptation (lamp): Regional kernels calibrate only on the support bank. When LAMP-F adapts text prototypes, it also updates residual predictions so the support residuals do not go stale. Bounded component only; full experiments are not reproduced.
2026-10 · LCT: Losslessly Compressed Tensors with GPU friendly execution for Training (lct): See Section 4 and Appendices B/C. Generic zlib is not a substitute. Bounded component only; full experiments are not reproduced.
2026-10 · MJev-Spatial: System One Spatial Intelligence via Native Attention Matching (mjev-spatial): The decision marker supplies the query, and candidate-end markers supply keys using the backbone's native final attention projections. Continuous values use bins or three-stage regional refinement, without a separate classification head. Bounded component only; full experiments are not reproduced.
2026-10 · Patch where the text is new: a compression view of where byte-level models should spend compute (novelty-patching): It computes match labels offline and looks up only the two preceding bytes online. Sources are Section 3 for method, Sections 4–7 for results and negatives, Section 8 for limits, and Code and data. Bounded component only; full experiments are not reproduced.
2026-10 · Prism-1: Position- and Rival-Independent Scoring for System One Models (prism-1): A softmax combines independent scores. Softmax alone cannot guarantee this without attention isolation and position reset. Bounded component only; full experiments are not reproduced.
2026-10 · Rethinking Routing Exploration in Modern Mixture-of-Experts (mirror-moe): Rather than simply adding noise, MirrorMoE temporarily suppresses some experts through mirrored selection and anneals back to normal routing. The main source is Section 3, Eqs. 3–6. Bounded component only; full experiments are not reproduced.
2026-10 · Retrofitting Recurrent Vision-Language Models from Pretrained Backbones (rr-vlm): RR-VLM fuses prelude context and recurrent state with a linear adapter, then performs text recovery, projector alignment, and full-model instruction recovery. Text and final recovery also distill from the original unpruned teacher. Bounded component only; full experiments are not reproduced.
2026-10 · RoleQ: Role-Aligned Scales for Converting Language Models to Ternary (roleq): Full RoleQ-T additionally needs rotation, GPTQ, distillation, and an optional Fisher two-plane method. A standalone quantization function does not replace that pipeline. Bounded component only; full experiments are not reproduced.
2026-10 · SoftCache: Adaptive Step Size Caching for Diffusion Transformers (softcache): SoftCache converts the headroom between accumulated staleness and its threshold into a continuous step size: a more reliable cache takes a longer step. It extrapolates from two fresh residuals and rearranges the remaining grid so skipped nodes are not visited later. Bounded component only; full experiments are not reproduced.
2026-10 · When Does Muon Actually Win? A Paired Data-, Time-, and Tuning-Budget Benchmark (muon-paired-budget): Its main contribution is the separation of these comparison protocols, not a new Muon update equation. The test set is evaluated only once, after tuning and trajectory analysis are frozen. Evaluation protocol interface; not a model capability result.
2026-10 · When the KV Cache Is Slower Than the GPU: Scaling Remote KV Tiers for Context-Parallel Prefill (remote-kv-tiers): The work first calculates the break-even bandwidth and fixed overhead for loading versus recomputing. It then shards the MLA KV cache by PCP rank and uses a separate remote instance for each rank plus asynchronous loading to eliminate head-of-line blocking. Not reproduced due to cost (review complete); research material only, no executable entry.
2026-10 · Dataset-Level Regularization for Joint-Embedding Self-Supervised Learning (oob-regularization): The Gaussian variant projects onto random unit directions, sorts the examples along each direction, and matches standard-normal quantiles. The contrastive variant optimizes uniformity on the unit sphere while preserving the correspondence with each current representation. Bounded component only; full experiments are not reproduced.
2026-10 · EgoJEPA: Beyond Retargeting for Dexterous Manipulation from Egocentric Videos (egojepa): Hand movement in human first-person video need not be mapped with hand-written rules to each robot's actions. EgoJEPA first conditions JEPA on hand-motion differences to predict future visual representations, then jointly models vision and latent actions, and finally adapts only the target robot's input and output action interfaces. Not reproduced due to cost (review complete); research material only, no executable entry.
2026-10 · ESC: Escaping 2D Diffusion Ambiguity with Geometric Constraints for Consistent Novel View Synthesis (esc): ESC jointly generates RGB and inverse depth, conditions Qwen MM-DiT on Plücker ray maps, and uses a bidirectional depth-reprojection constraint, PCC, to identify missing and hallucinated geometry. At inference it selects from repeated stochastic generations using PCC. Not reproduced due to cost (review complete); research material only, no executable entry.
2026-10 · Memory-Efficient Subspace Optimization with Residual Correction: From Momentum SGD to Muon (smrc-pmrc): SMRC uses EMA momentum; PMRC takes the matrix sign of low-dimensional momentum. They are not interchangeable instances of projected SGD. Bounded component only; full experiments are not reproduced.
2026-10 · One-Step Prediction Fidelity Is a Poor Proxy for Planning Performance in JEPA World Models (jepa-prediction-planning): Planning is evaluated separately with matching checkpoints, tasks and paired seeds. Success rates cannot be inferred from prediction metrics. Evaluation protocol interface; not a model capability result.
2026-10 · Open-world AI: a Tale of Three Requirements (memory-mosaics-plus): The two-stage variant estimates current and related tasks separately. A single mixed pool could make accumulating irrelevant history harmful. Bounded component only; full experiments are not reproduced.
2026-10 · PN-Mix: Pair-Norm Guided Mixed-Precision Key Cache Quantization for Efficient Long-Output LLM Inference (pn-mix): Each KV head selects its high-precision pairs once. Later quantization pages reuse the choice without looking at future decoding states. Bounded component only; full experiments are not reproduced.
2026-10 · Selective Reasoning for Typed Decision Models: When to Start, Read, and Stop (selective-reasoning): The controller separates three decisions: a benefit gate decides whether to begin reasoning; a trigger from the natural generation state decides when to request explicit answer readout; a normalized-entropy threshold decides whether to accept a candidate and cancel further generation. The original online system also has concurrent generation and reading, request cancellation, exact prefix restoration, and cost accounting. Evaluation protocol interface; not a model capability result.
2026-10 · Time-JEPA: Grounding Video World Models with Physical Time Series (time-jepa): During training, the same online encoder processes masked context and unmasked video separately. Evaluation removes the physical-state head and still takes video only. Bounded component only; full experiments are not reproduced.
2026-10 · The Company a Question Keeps: Auditing Open Jev-Style Models for Retrieval (jev-request-audit): The paper uses 92 queries from TREC DL2019/2020 to construct 276 three-passage states, balancing scenarios with zero, one, or two positives. This is not a production distribution.
2026-10 · Does System One Transfer? Calibrated Typed-Decision Models, Label Budgets, and Translation for Indonesian Regional Languages (jev-regional-transfer): A low-cost decision model answers every example; a local classifier or LLM handles a low-confidence subset. The authors use validation macro F1 to pick the smallest routing fraction, then freeze it for test evaluation rather than choosing the best test point.
2026-10 · DualQuant: Dual-scale Calibration-free Weight-and-Activation Quantization for LLMs (dualquant): This is not a search over heuristic exponents. Variable splitting separates inner scales used to assign quantization codes from outer scales used in reconstruction.
2026-10 · EV-JEPA: Elastic V-JEPA for Any Device and Cross-Device Inference (ev-jepa): EV-JEPA shares parameters across nested widths, jointly training narrow, wide, and narrow-to-wide paths at every step. A frozen video teacher constrains their common representation space, and adjustable token compression supports transfer. Not reproduced for now due to cost; validation pending, no executable entry.
2026-10 · Multi-Stream Attention Residuals (msar): Block MSAR persists only the embedding, single-stream summaries of completed blocks, and the current block's partial sum. Views are expanded when consumed, not written back persistently.
2026-10 · Not All Dimensions Are Equal: Outlier-Aware Attention Linearization with Grouped Quadratic Kernels (oal): After calibration, early and late softmax layers remain in place, intermediate layers are converted, and q/k/v/o LoRA in all layers trains jointly with kernel coefficients. Groups stay fixed once selected.
2026-10 · On the Design Space of Continuous Diffusion Language Models (doublecat-semicat): High-dimensional Gaussian noise has norm about \(\sqrt{|V|}\), while the one-hot signal has norm 1. Existing tokens can therefore be hard to identify.
2026-10 · Orthros: Phase-Aware Heterogeneous Attention for Efficient Transformers (orthros): “Linear prefill” here means fixed-window \(O(Tw)\), not kernelized linear attention. Prefill copies only the last prompt token; decoding copies one new token per step, with the two streams respectively updating KV and retrieving globally.
2026-10 · ReplaySSM: Cache SSM Inputs, Not State (replayssm): The acceptance length advances only the committed-input pointer; rejected drafts never enter the recurrence. Separating a low-precision copy used for output reads from a high-precision recurrence anchor prevents repeated feedback of read quantization error.
2026-10 · SmolWAM: A Latent World-Action Model for Robotic Manipulation on Edge Devices (smolwam): A single-frame Wan VAE and 0.5M adapter map back to the clip latent space. Cached BERT features are padded to 512 positions, retaining a zero-bias attention sink.
2026-10 · CoTS: Competitive Token Selection for Latent Reasoning in Large Language Models (cots): CoTS uses top-two probability competition to control soft mixing, retains at least half the sampled token's weight, and irreversibly switches to discrete output after reasoning. See Sections 3.1–3.3, Equations 1–6 and Section 4 for experiments and ablations. Bounded component only; full experiments are not reproduced.
2026-10 · Few-Step Generation via Data-Space Iteration (data-space-iteration): Prefix rollouts do not build a gradient graph. Method and appendix Method reference 1
2026-10 · LeWAM: End-to-End World and Action Modeling with JEPAs (lewam): The visual encoder combines a ResNet-18 trained from scratch, spatial softmax, and a two-layer projection MLP. The simulation model has about 16.9M parameters, latent/MoT width 192, 4 layers and 4 heads.
2026-10 · MetaEncoder: Exploring the Limit of Bi-Encoders for Multimodal System One Decision Making with Natural Language Interface (metaencoder): Closed-set prompts may list options; partial option dropout discourages literal matching. Section 3 Not reproduced due to cost (review complete); research material only, no executable entry. Method reference 1
2026-10 · Rounding in Preconditioner Space: Redesigning 4-bit AdamW Optimizer-State Quantization (adamw4bit): Both store the first moment in NF4, with stochastic rounding on the LM head over the final 10%. Method Method reference 1
2026-10 · Smoothing the Top-k Exposure Boundary for Sparse Mixture-of-Experts (elastic-expert-routing): This exposes near-boundary experts to actual sparse computation at unchanged expected budget. Deploy fixed k.
2026-10 · SparseDecoding: Decoding-Aware Pruning for Accurate and Efficient LLM Inference (sparse-decoding): For batch-one decoding matrix–vector multiplication, each group of 32 consecutive columns uses a 32-bit word marking retained positions. At 50% sparsity, exactly 16 bits are set.
2026-10 · Specialized Decision Models vs. General-Purpose LLMs: Benchmarking Jev Across Knowledge, Reasoning, and Multilingual Tasks (jev-capability): Math is weaker than knowledge in the paper. Keep vendor-reported and author-measured scores separate because prompting protocols differ; this is not a new architecture or objective.
2026-10 · VFold: Symmetry-Aware Cross-Layer Value Cache Compression (vfold): VFold uses full-rank CCA and Hungarian head matching, folds transformations into Value/output weights offline, and uses each current token's full Value before averaging it into shared adjacent-layer storage. Keys are not merged; important prefix/recent tokens remain separate.
2026-10 · Video Encoders Built on Image Representations (video-encoder-selection): The paper freezes the frame encoder and selects original tokens under a fixed budget using question relevance, within-frame diversity, and cross-frame correspondences. An optional learned refiner accesses extra neighboring-frame context after selection and updates retained anchors without increasing output length. Bounded component only; full experiments are not reproduced.
2026-10 · AF-Muon: An AdamW-Free Muon Optimizer for Tied-Embedding Models (af-muon): The vocabulary matrix shared by sparse input lookups and the dense output classifier uses a support-aware capped linear minimization oracle (LMO). One-dimensional parameters use RMS updates.
2026-10 · IrekoGPT: Turning Structured Pruning into Post-Hoc Slimmable LLMs (irekogpt): Keep SliceGPT's projection basis rather than discarding it after pruning, so one checkpoint exposes nested widths. Calibrate at multiple compression ratios and use ridge corrections for downstream linear layers.
2026-10 · LLM2Jev: LLMs Are Already Jev-Style Decision Models -- When and How to Fine-Tune Them (llm2jev): Use next-token probabilities over parenthesized numeric options to form a distribution over a finite candidate set. Fine-tune with a listwise decision loss and a KL term that anchors auxiliary outputs to the base model.
2026-10 · MWOP: Modality-aware Width-wise Operation Pruning for Efficient MLLMs (mwop): Prune V2V, T2V, and T2T paths separately inside the same attention head and select distinct FFN channels for visual and text processing. Re-estimate FFN importance after attention pruning.
2026-10 · Omni-Embed-Mini: Binding Modalities Without Forgetting via Dense Distillation (omni-embed-mini): Freeze the text backbone and encode dense cascaded descriptions of each media example into teacher targets using that same backbone. Train lightweight projectors and staged LoRA updates to bring six modalities into its geometry.
2026-10 · Role-aware Heuristic Episodic Attention for Conversational LLMs (rea): Put global instructions in a persistent prefix and multi-turn exchanges in episodic memory. When retrieving, use the role to decide whether to keep original text, a compressed representation or nothing.
2026-10 · TACO: Ternary Absolute-max Column-wise One-sparse Optimizer for LLM Fine-Tuning (taco-optimizer): In each matrix column keep only the gradient component with largest magnitude, retaining its sign. Use low-precision column momentum rather than dense per-parameter optimizer state.
2026-10 · TESS: Scalable, Transferable Meta-network for Data Selection Requires a Different Loss (and Why the Obvious Choice is Problematic) (tess): Use their per-example loss gap as a value pseudo-label. Pointwise value matching avoids the poorly behaved direct objective and can transfer the small teachers' selection rules to an unseen pool.
2026-10 · VETO: Video Efficient Token Optimization for Vision Language Models (veto): Merge semantically neighboring tokens within frames first, then remove cross-frame redundancy from compressed frame representations. Spatial-first processing reduces global temporal matching cost.
2026-09 · HAWK: Rethinking Multimodal Drafting for Speculative Decoding (hawk): Learn a mixture of hidden states from multiple target layers to supply compressed visual features to a shallow drafter. Train on the shifted trajectory after the drafter's own proposals.
2026-09 · Chinese-Jev: Bringing System One Model to Chinese-Language Tasks (chinese-jev): Heterogeneous labels become candidate-probability distributions. Pretraining uses 10 million general examples before separate medical, legal and finance fine-tuning.
2026-09 · Cross-Entropy Guided Routing in Mixture-of-Experts Large Language Models (ce-guided-moe): Use a shifted logarithm of predicted error to attenuate routing logits before Top-K, downweighting experts with high predicted error. Actual next-token cross-entropy supervises the error head.
2026-09 · LeapQuant: Efficient Linear Attention with Accurate Recurrent State Quantization (leapquant): LeapQuant retains a low-bit base state, buffers within-window updates at high precision, and requantizes once at the window boundary. It separates the largest outliers into a few high-precision compensator tokens, then smooths and quantizes the residual.
2026-09 · On Trajectory-Aware Training for Masked Diffusion Language Models (pumba): During training, unroll consecutive denoising steps along the model's own progressive-unmasking trajectory. Pass a hidden carry to the next step and backpropagate through time within a fixed window, so an earlier step learns to produce a useful carry for later steps.
2026-09 · Pretraining Latent Information Feedback Transformers with Teacher Supervision (lift-feedback): Inference feeds the model's own state back. The objective is token cross-entropy plus forward KL to the teacher's top-k distribution.
2026-09 · SPLASH: Switching Parallel Layouts of Attention with Seamless Handoff for LLM Serving (splash): Unify attention-weight and KV-cache ownership. DOP shards weights while giving requests exclusive KV; schedule among TP, DP, CP, and DOP online, amortizing transition cost over remaining steps.
2026-09 · STEPQuant: When and Where Errors Matter in Delta-Rule Recurrent State Quantization (stepquant): Each concurrent request retains a fixed-size delta-rule state, and quantization errors compound. Estimate remaining state lifetime and key-row impact, allocate more precision to important blocks under a fixed average budget, and fit row/column scales.
2026-09 · Time-Anchored Diffusion Language Models: Latent-Space Caching for Fast Generation (tadm): Cache an expensive anchor network's latent representation across reverse-diffusion steps and refresh it periodically. A small fusion module uses the current state to gate a correction before the lightweight denoiser.
2026-09 · Triadic Linear Attention: Three-Dimensional Recurrent States for Long-Context Sequence Modeling (triadic-linear-attention): Triadic attention adds another key to write \(k\otimes k'\otimes v\), contracting the two key axes with \(q,q'\) during reads. A second key dimension of \(E\) multiplies state capacity by \(E\) while adding only two projection sets. \(E=1\) reduces exactly to ordinary linear attention.
2026-09 · V-JEPA Policy: Building Effective World-Action Models on Predictive Visual Latents (vjepa-policy): Freeze V-JEPA 2.1 vision encoding and jointly train an instruction-conditioned future-latent predictor and flow-matching action expert. Action generation reads the predicted future context.
2026-09 · Fractional State Space Transition for Long Sequence Modeling (frac-ssm): Approximate fractional power-law memory with finitely many logarithmically spaced exponential modes, combining long-tailed memory with bounded recurrent state and parallel training.
2026-09 · MS-GLA: Multi-Scale Gated Linear Attention for Addressing Representational Bottlenecks via Multi-Temporal Resolution (ms-gla): One time resolution can bottleneck linear attention. Allocate heads over scales, run nonoverlapping masked-average pooling and separate GLA branches, then align back with causal hold and fuse through input-dependent softmax routing.
2026-09 · Telescopic Language Models (telescopic-lm): Sample a network-depth prefix each step for next-token supervision while preserving a full-depth anchor. The goal is to make every supported depth of one checkpoint usable as a language model.
2026-09 · Words Speak Louder Than Order: A Behavioral Evaluation of Gemma 4 (gemma-priority-protocol): The scope is one model, short context, and English single-word choices. This is neither a new training algorithm nor evidence about neural circuits. Bounded component only; full experiments are not reproduced.
2026-09 · dQwen3.5: Adapting Hybrid Language Models into Bidirectional Diffusion Language Models (dqwen35): Convert Qwen3.5's attention/RNN hybrid backbone to bidirectional state propagation, then train a diffusion language model with a masked denoising objective.
2026-09 · On-Demand Attention: Efficient Long-Context Decoding with Learned Recall (oda): A lightweight recall head predicts whether global attention would help before it runs, and only high-benefit tokens trigger the full read. The full KV store is retained.
2026-09 · ASPIRE: Asynchronous Batch Self-Speculative Decoding (aspire): Choose each request's draft length from online acceptance and current batch cost. Periodic full-attention refresh corrects the self-speculative state.
2026-09 · ECHO: Early-layer Collaborative Hierarchical Orchestration with Bonus Logits in Speculative Decoding (echo): A fast early-layer loop explores while a lower-frequency final-layer loop verifies. Bonus logits on both sides fill out the tree while retaining exact rejection-sampling correction.
2026-09 · How Calibration Content Shapes Attention-Based Reranking (interpolated-null-calibration): Select λ only on validation. The paper's oracle tunes over dataset queries and is not an independent test result. Bounded component only; full experiments are not reproduced.
2026-09 · LoopSpec: Pipelined Self-Speculative Decoding for Looped Transformers (loopspec): Use intermediate predictions across recurrent depth for self-speculation. Pipeline residual proposals to hide verification latency.
2026-09 · OPEN-1B: A Fully Auditable Training Run (open-1b-audit): Fix the order of kernel reductions, data batches, and collective communication. Per-step state hashes enable single-step replay audits across heterogeneous hardware.
2026-09 · Persistent Recurrent Memory Between Transformer Layers Improves Language Model Generalization (persistent-recurrent-memory): Insert an observe–GRU update–gated influence state path between the lower and upper Transformer halves, so recurrent memory affects later computation.
2026-09 · AgentKV: Phase-Aware KV Eviction for Agentic LLMs (agentkv): Keep a representative query buffer for each agent phase, and use relevance across phase queries to decide KV eviction.
2026-09 · Register Tokens for Bounded-State Reasoning in Diffusion Language Models (register-tokens-dllm): Discard previous chunk text and carry a fixed number of continuous register hidden states into the next diffusion chunk, supporting cross-chunk reasoning without retaining all text.
2026-09 · SAS: Simple Attention Sparsification via End-to-End Optimization of Context Ranking (sas-attention): A shared softmax normalizes history gates. The current block always has gate 1, preventing drift in the aggregate weight of past context.
2026-09 · Caption-once, Frames-on-Demand: Visual-Need Routing for Budget-Aware Agentic Long Video Understanding (frames-on-demand): Build caption memory once for each video. The visual-need router reads relevant frames within a fixed budget only when text evidence is insufficient.
2026-09 · Data Scarcity and Model Sparsity: Mixtures-of-Experts Overfit More to Repeated Data (repeat-aware-moe): MoE can overfit repeated data sooner than dense models. Stronger masking and dropout suppress expert memorization; adjust regularization according to repetition and sparsity.
2026-09 · Musec: MomentUm SpEctral Clipping for Stable Muon-type Training (musec): Smoothly clip singular values of the Muon momentum. This avoids the discontinuity of hard clipping while constraining large unstable directions.
2026-09 · OmniKVQuant: KV Cache Quantization for Omni-LLMs (omnikvquant): Short time windows determine Key quantization ranges; rotate Values separately by modality to handle temporal drift and heterogeneous geometry.
2026-09 · SenseNova-U1.5: Towards Native Unified Visual Intelligence (sensenova-u1-5): Use spatial patch reconstruction instead of tokenizer/VAE, and multi-expert OPD to align understanding and generation post-training.
2026-09 · SWRouter: Similarity-Contractive Window Routing for Multi-Turn Large Language Model Conversations (swrouter): Contract multi-turn windows by adjacent semantic similarity, then use a contrastive router to select a model. Evaluate context construction separately from routing correctness.
2026-09 · BeaconKV: Key-Value Cache Compression Guided by Beacon Queries for Efficient Large Reasoning Model Inference (beaconkv): Representative queries vote on key importance while a recent window is always retained. Under cache budget, rank historical key \(k_j\) by \(I_j=\max_{b\in B}\langle b,k_j\rangle\).
2026-09 · KVMem: Virtualizing Million-Token Agent Workspaces on a Consumer GPU (kvmem): KVMEM pages precomputed KV across GPU, host memory, and NVMe. An index in the model's attention space selects relevant blocks and materializes an execution view within the native window.
2026-09 · Lngram v2: Latent N-Gram Memory with Interpretable Discrete Representations (lngram-v2): Grouped-query attention reads multiple route tokens back into the backbone. Hard addresses are interpretable, neighboring counterfactual addresses provide gradients, and a zero sink allows the model to ignore unhelpful memory.
2026-09 · Random Attention: Rethinking KV Cache Eviction for Efficient Reasoning (random-attention): Random Attention protects the prompt and retains a fixed budget from generated tokens with independent uniform samples per head. It avoids the scoring pass and does not force all heads to share a ranking rule.
2026-08 · CritICL: Inference-Time Weak-to-Strong Generalization from Small Language Model Failure Modes (criticl): Static retrieval uses globally common failures; dynamic retrieval selects critiques relevant to the current input. The strong model then generates an answer once.
2026-08 · PACE: A Unified Condense-and-Extract Paradigm for Fast VLM Inference (pace-vlm): APC uses a shallow ViT preview to estimate global semantic density and local detail and rescale by image difficulty. DDAE fuses LLM and vision-encoder attention, weighting them by confidence rather than always trusting one attention map.
2026-08 · TwinKV: A Composable Repair Pass for KV Cache Eviction via Pairwise Key Redundancy (twinkv): TwinKV repairs a wrapped eviction policy such as StreamingLLM or H2O. At exactly the same budget it pairs dropped orphans without similar retained keys with retained donors that have a near twin.
2026-08 · MLLMCLIP: Feature-Level Distillation of MLLM for Robust Vision-Language Representations (mllmclip): Rich visual semantics from MLLMs are hard to transfer directly to lightweight CLIP. The paper selects teacher tokens adaptively by layer attention and aligns student image/text features with CKA, retaining dual-tower efficiency at deployment.
2026-08 · VBVR-Pro: A Scalable and Verifiable Suite for Native Visual Reasoning (vbvr-pro): A fluent response can mislead a general VLM judge. Per-task executable scorers check intermediate states, constraints, and final states to provide rewards for multimodal generators.
2026-08 · WeMM-Embedding: WeChat Multi-Modal Embedding Technical Report (wemm-embedding): Stage 1 aligns hundreds of millions of pairs; stage 2 adds curated relevance data, fine-grained supervision, and cross-scale knowledge transfer. Matryoshka representations let users choose an output dimension that fits the budget.
2026-08 · RARE: Decoupling Representation Steering from Expert Routing in Mixture-of-Experts Language Models (rare): RARE projects an arbitrary steering direction into the router's null space, then removes router-visible components introduced during propagation at later protected layers. This changes behavior representations without changing the original expert paths.
2026-08 · Fast A/B/n Testing: Exact Multi-Policy Comparison via Tree-Coupled Feedback Sharing (tcab): Use maximal coupling and a minimum spanning tree to share feedback for matching actions while preserving each adaptive policy's marginal trajectory distribution.
2026-08 · Generation as Auxiliary Supervision: Enhancing Visual Understanding at Zero Inference Overhead via Decoupled Embedding Prediction (gas): Generation gradients improve shared visual features without directly changing the understanding-only upper layers. Generation can be removed entirely at deployment.
2026-08 · Cracks in the Foundation: Seemingly Minor Architectural Choices Impact Long Context Extension (olmpool-long-context): Use a controlled 7B model pool to isolate compound effects of normalization, GQA, pretraining length, and sliding-window attention on long-context extension.
2026-08 · DistillCache: KL-Guided Adaptive KV-Cache Eviction for Memory-Efficient LLM Inference (distillcache): Treat KV eviction as sequential decision-making. Stepwise KL rewards train a lightweight policy to preserve future predictive distributions.
2026-08 · Autonomy-of-Heads: Data-Free Sparse Attention from Frozen Query-Key Geometry (autonomy-heads): Classify retrieval and streaming heads from the spectral effective rank of frozen QK projections, without calibration data or runtime gating.
2026-08 · BaKron: Efficient Quantization with Kronecker-Factored Hessians (bakron): GPTQ usually uses input-side curvature. Bilateral Kronecker Hessians are richer but direct vectorized optimization is expensive; use anti-diagonal parallelism and recursive divide-and-conquer for adaptive rounding.
2026-08 · Detecting Safety Training Modification in Language Models via Activation Analysis (activation-model-scanner): This structural signal has blind spots and does not replace behavioral safety evaluation. See §IV; implementation conventions were checked against the pinned upstream extractor.py. Bounded component only; full experiments are not reproduced. Method reference 1
2026-08 · Hierarchical Latent Prediction for Language Models (hilp): Topic: hierarchical latent pretraining. Stepwise NextLat rollouts accumulate error. HiLP adds a coarser abstract latent target so long-timescale structure also constrains local states.
2026-08 · MACRO: Markov Chain Routing of Transformer Layers (macro): MACRO represents skips, repeats, and residual additions with a context-conditioned Markov policy. Feedback updates the routing distribution and top-k Viterbi decodes candidate programs without changing base-model weights.
2026-08 · DBLast: Dependent Block Drafting for Stochastic Speculative Decoding (dblast): Topic: Speculative decoding. Parallel block drafters often assume positions are conditionally independent and struggle to match the joint distribution at high sampling entropy. DBLast uses a shared low-rank latent mixture to model cross-position dependence and trains for expected verification length.
2026-08 · QEvict: Recoverable Quantized KV Eviction for Attention-Drift-Robust Long-Context Decoding (qevict): Topic: Long-context KV cache. Binary keep/drop cannot handle attention drift: a low-priority window can later become relevant. QEvict uses full-precision, quantized-recoverable and deleted tiers, and dequantizes and promotes windows as accumulated attention changes.
2026-08 · Towards Physics of Multimodal Pretraining: Knowledge Flow, Modality Synergy, Early Unification, and Recipes (physics-mm-pretraining): Controlled experiments characterize knowledge flow across modalities, synergy versus competition, early unification, and a recipe with shared attention and normalization but modality-specific FFNs.
2026-08 · TaskPress: Query-Agnostic KV Cache Compression via Task-Guided Pruning (taskpress): Lower-index tie-breaking is a local convention. The method/resources follow v2 Section 3 Eqs. 1–3; no assertion of line-by-line identity to v1. Bounded component only; full experiments are not reproduced.
2026-08 · DART: Decoded Attention over Recurrent States for Efficient Long-Context Sequence Modeling (dart): Keep Mamba-2 chunk-state contributions, decode token-conditioned K/V, and run state-memory attention.
2026-08 · Learning What to Remember: Test-Time Training via Context Distillation (ttcd): A long-window teacher supervises short-window student fast weights through hidden-state differences, retaining information useful later.
2026-08 · Role-Decoupled Attention Residuals (rd-attnres): QK handles matching while V carries content, so their depth preferences can differ. RD-AttnRes adds only one model-width route vector for V without changing residual sources or attention itself.
2026-07 · TransMem: Transforming Hidden States into Memory for Large Language Models (transmem): Convert sparse historical hidden states from a frozen backbone to reusable parametric memory. Learn a gate by evidence-conditioned self-distillation.
2026-07 · ReToken: One Token to Improve Vision–Language Models for Visual Retrieval (retoken): Only the token and one projection matrix are trained, using class-balanced BCE on relevant/irrelevant images. The VLM is frozen by default.
2026-07 · WIDE: Boosting Adaptive LLM Inference via Token-level Dynamic Width Pruning (wide): Static pruning cannot allocate compute by token difficulty, and dynamic depth is too coarse. WIDE separately routes attention-head groups and FFN-channel groups for each token, co-designing mask reordering, block skipping, and device-side skipping.
2026-07 · Penelope: Localized Latent Recurrence for Efficient Structured Reasoning (penelope): Run shared-weight latent recurrence only at one decoder boundary. Gated state repeatedly refines representations without rerunning the full decoder.
2026-07 · DataOrchestra: Learning to Orchestrate Per-Example Curation of Pretraining Data (data-orchestra): A strong teacher drafts a plan. Tool models execute it and a verifier checks the result.
2026-07 · Adaptive Depth Sparse Framework: Similarity-Driven Resource Allocation for Pre-Trained LLMs (adadsf): An MLP router sends only Top-K tokens through the original Transformer block; other tokens take a residual bypass. Hidden states and output distributions are aligned jointly.
2026-07 · Anti-Periodic Positional Encoding: Möbius Boundary Conditions Make In-Context Retrieval Reliable (mobius-rope): The paper applies antiperiodic frequencies to some heads so the rotation across the full training window is always \(-I\). Other heads retain standard RoPE for language modeling.
2026-07 · Hilbert Operator for Progressive Encoding (HOPE): A Mathematical Framework for Deconstructing Learned Representations in Deep Networks (hope-hilbert): The paper uses batch-normalization statistics to constrain a Gaussian surrogate, computes a functional ReLU kernel analytically, and compares PRUNE, MERGE, and residual-block EVICT at a common distortion scale. Each greedy decision divides distortion by the parameter savings measured on the initial model to avoid selection bias as adjacent layers shrink. Bounded component only; full experiments are not reproduced.
2026-07 · Naju: A Native Discrete State-Space Model with Independent Retention and Writing for Long-Sequence Memory (naju): A single coupled gate can also make strong retention incompatible with writing. Naju parameterizes discrete poles directly, separates retain and write gates, and keeps token-dependent \(B/C\) directions, short-range causal convolution, direct feedthrough, and output modulation.
2026-07 · Parameter-free Adaptive Sparse Attention via Compression-Based Content Selection (gzip-sparse-attention): Above-mean literal blocks connect at long range, all blocks retain a local window, and no fixed global token is used. Of the heads, 50% are local only, 25% literal-long-range only, and 25% use both.
2026-07 · Windowed-MTP: Removing the Full-Context Draft-KV Tax at Million-Token Context (windowed-mtp): The target still reads full context and verifies every candidate. The window can affect acceptance and speed, but not the output distribution determined by the target.
2026-07 · GaugeQuant: Online Learning of Quantization-Optimal Bases from LLM Symmetries (gaugequant): Function-preserving channel transformations can have very different quantization errors. Learn an orthogonal gauge online and suppress activation outliers with LogSumExp without a separate calibration corpus.
2026-07 · Convolution for Large Language Models (conv-llm): The paper compares 17 convolution placements on a fixed Qwen3 backbone and selects depthwise 1D convolution with kernel=3 after linear Q/K/V projection and before attention aggregation. A residual keeps the original projection; no normalization or activation is added, and additional parameters are below 0.01%.
2026-07 · C²KV: Compressed and Composable KV Cache Reuse for Efficient LLM Inference (c2kv): Learn position-independent, composable KV representations with compression tokens and structured attention, jointly training compression and reuse.
2026-07 · PPL-Factory: Task-Aware and Budget-Aware Data Selection from Language Modeling to Reasoning (ppl-factory): PPL-Factory uses a frozen base model to compute task-aware NLL: packed blocks for language modeling, and only reasoning/answer tokens for reasoning SFT. High budgets favor easy examples, lower budgets select middle examples, and very low budgets sample randomly from the middle pool to preserve coverage.
2026-07 · OpenLanguageModel: Readable and Composable Small-Language-Model Pretraining for Education and Research (open-language-model): OLM keeps components as ordinary PyTorch modules and expresses wiring with Block, Residual, Repeat, and Parallel. The same model can move from a notebook to CPU, one GPU, or multiple GPUs on one host.
2026-07 · Looped Latent Attention: Cross-Loop KV Compression for Looped Transformers (looped-latent-attention): Looped Transformers reuse weights but still duplicate KV caches across loops. LLA learns shared low-rank K/V latents and reconstructs loop-specific K/V during serving, exploiting recurrence redundancy for near-lossless compression.
2026-06 · Cross-Lingual Exploration for Parametric Knowledge (cross-lingual-exploration): Gemini judges semantic correctness and cross-language consistency. A counter or string translator would break the defining mechanism, so no heuristic component is invented. Not reproduced due to cost (review complete); research material only, no executable entry.
2026-06 · MiniMax Sparse Attention (minimax-sparse-attention): Add a lightweight index per GQA group to select historical blocks before exact attention over their tokens. Train and infer with the same path.
2026-06 · Language Models Need Sleep: Learning to Self-Modify and Consolidate Memories (sleep-lti): Knowledge seeding combines teacher data and distillation on student-generated prefixes. LTI has the student complete teacher dreams and rewards a frozen semantic judgment and thresholded token edit similarity. Bounded component only; full experiments are not reproduced.
2026-05 · XLGoBench: Detecting cross-lingual skill gaps with algorithmic tasks (xlgobench): Graph-SP requires a unique shortest path, and T-Rank uses a Hamiltonian DAG with transitive edges and a 1:3 node-to-edge ratio. The local callback receives deep copies of allowlisted task/problem fields; exact JSON scoring needs no LLM judge. Bounded component only; full experiments are not reproduced.
2026-05 · Memory Grafting: Scaling Language Model Pre-training via Offline Conditional Memory (memory-grafting): The recipient performs longest exact suffix lookup in expected \(O(1)\) time. Misses fall back to trainable hash Engram; hits use separate projection, context gating, and short convolution before being written to the residual stream.
2026-05 · Customizing an LLM for Enterprise Software Engineering (gemini-for-google): Deduplication, quality filtering, and contamination controls produce mid-training data. General-data replay limits catastrophic forgetting, followed by post-training for internal tools and deployment evaluation.
2026-04 · Optimized Deferral for Imbalanced Settings (mild): The local code implements the logistic surrogate, analytical gradient, and bounded linear-router training. Inference accepts only features and trained parameters, not answers or actual expert costs. Bounded component only; full experiments are not reproduced. Method reference 1
2026-04 · ProEval: Proactive Failure Discovery and Efficient Performance Estimation for Generative AI Evaluation (proeval): A target score enters the update only after its item is selected. Primary source: v1 §§2.1–2.4, Eqs. 2, 5, 7, 10, and 12. Bounded component only; full experiments are not reproduced. Method reference 1
2026-04 · FUSE: Ensembling Verifiers with Zero Labeled Data (fuse-verifiers): FUSE reads only candidate score matrices, estimates query-dependent verifier quality from second and third moments, selects transforms to reduce dependence, constructs correctness posteriors, and optimizes an ensemble. True labels are used only for independent evaluation. Not reproduced due to cost (review complete); research material only, no executable entry.
2026-03 · Switch Attention: Towards Dynamic and Fine-grained Hybrid Transformers (switch-attention): Static hybrid attention fixes the pattern for every token. Learn a fine-grained router that selects full attention only for tokens needing global context.
2026-03 · Efficient, Property-Aligned Fan-Out Retrieval via RL-Compiled Diffusion (r4t): Define alignment with a composite set-level reward, then compile that target into a single diffusion fan-out retrieval pass.
2026-03 · Theoretical Perspectives on Data Quality and Synergistic Effects in Pre- and Post-Training Reasoning Models (data-quality-lsa): The paper defines OS as outcome-supervised regression, not full RL with sampling, advantage estimation, and policy gradients. See §§3 and 4 for the linear attention and closed-form solution. Bounded component only; full experiments are not reproduced. Method reference 1
2026-02 · A Theoretical Framework for Modular Learning of Robust Generative Models (modular-robust-generative): Noncausal teacher samples are also distilled into a causal router that reads only prefixes. See §§5.2, 6, and 7.3. Bounded component only; full experiments are not reproduced. Method reference 1
2026-01 · ECO: Quantized Training without Full-Precision Master Weights (eco-optimizer): ECO updates quantized weights directly and feeds requantization error into existing momentum, without master weights or a persistent error buffer. Adam uses an elementwise effective step size; reusing SGDM's scalar coefficient is incorrect. Not reproduced due to cost (review complete); research material only, no executable entry.
2026-01 · Mobility-Embedded POIs: Learning What A Place Is and How It Is Used from Human Movement (me-pois): After pretraining, frozen representations feed map-attribute probes. See §§3.2–3.6. Bounded component only; full experiments are not reproduced. Method reference 1
2026-01 · Conditional Memory via Scalable Lookup: A New Axis of Sparsity for Large Language Models (engram): MoE adds conditional compute, but layers still reconstruct static local patterns. Hash normalized n-grams into a large embedding table for deterministic \(O(1)\) lookup, then gate memory into early layers so attention/FFNs can focus on compositional reasoning.

2025¶

2025-12 · mHC: Manifold-Constrained Hyper-Connections (mhc): This retains cross-stream exchange while keeping each layer's residual map nonexpansive. The paper also supplies fused kernels, activation recomputation, and DualPipe overlap to control overhead.
2025-10 · Detecting Post-generation Edits to Watermarked LLM Outputs via Combinatorial Watermarking (combinatorial-watermark): Use calibration to control type I error. Deleting one complete binary period can be undetectable, so localization is not perfect. Bounded component only; full experiments are not reproduced.
2025-08 · GLiClass: Generalist Lightweight Model for Sequence Classification Tasks (gliclass): This avoids unstable parsing of generative LLM outputs and separate cross-encoder inference on each text–label pair. The repository retains the defining mechanisms: candidate content affects scores, and one normalization produces a distribution over all candidates.
2025-07 · Model Directions, Not Words: Mechanistic Topic Models Using Sparse Autoencoders (mechanistic-etm): Eqs. 16–19 use independent SAE decoder columns only for steering: combine and normalize those directions, remove the existing parallel component, then inject the specified strength. A negative control with different encoders and decoders catches implicit weight tying. Bounded component only; full experiments are not reproduced.
2025-05 · Gated Attention for Large Language Models: Non-linearity, Sparsity, and Attention-Sink-Free (gated-attention): Of 30 variants, the paper finds that a query-dependent sigmoid gate after each head's SDPA is simple and stable. It adds nonlinearity and can sparsely suppress unhelpful heads.
2025-04 · SmolVLM: Redefining small and efficient multimodal models (smolvlm): Move neighboring spatial tokens into channels with pixel shuffle, then project into LM space with an MLP. The paper also studies long contexts, image tiling, learned position tokens and data mixtures for 256M/500M/2.2B image/video models.
2025-02 · Muon is Scalable for LLM Training (muon): The paper adds weight decay and shape-aware scaling for large-scale training. Parameters outside hidden matrices still use AdamW.
2025-02 · SigLIP 2: Multilingual Vision-Language Encoders with Improved Semantic Understanding, Localization, and Dense Features (siglip2): SigLIP 2 extends SigLIP's pairwise sigmoid loss with captioning pretraining, global-local self-distillation, masked prediction, online data curation, and NaFlex dynamic resolution. This improves semantics, localization, dense features, and multilingual fairness.
2025-02 · MoBA: Mixture of Block Attention for Long-Context LLMs (moba): Split sequences into blocks. A router chooses a few relevant blocks for each query while always retaining the current causal block.
2025-02 · Native Sparse Attention: Hardware-Aligned and Natively Trainable Sparse Attention (native-sparse-attention): From pretraining onward it learns three parallel paths: compressed history blocks for global structure, query-dependent block selection for important detail, and a sliding window for recent precise information. Learned gates blend the three outputs.

2024¶

2024-12 · Byte Latent Transformer: Patches Scale Better Than Tokens (blt): Process bytes directly, segmenting dynamic patches by next-byte entropy. The global Transformer operates on patches, while local encoders and decoders recover bytes.
2024-11 · Hymba: A Hybrid-head Architecture for Small Language Models (hymba): Run attention and a state-space branch in parallel within one layer. An input-dependent gate combines exact local retrieval with linear long-range state.
2024-03 · Data Mixing Laws: Optimizing Data Mixtures by Predicting Language Modeling Performance (data-mixing-laws): Train small pilot mixtures, fit per-evaluation-domain mixing laws and search previously unseen mixtures.
2024-01 · Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Heads (medusa): Add future-token heads to a frozen or jointly fine-tuned backbone. Verify candidate branches together with tree attention to reduce serial decoding steps.

2023¶

2023-12 · Mamba: Linear-Time Sequence Modeling with Selective State Spaces (mamba): Mamba makes the SSM step size and read/write vectors depend on the current token. This selectively retains information while keeping computation linear in sequence length.
2023-06 · AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration (awq): Use calibration activations to find salient channels. Equivalent channel scaling protects roughly 1% of key weights before hardware-friendly uniform weight-only quantization.
2023-05 · GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints (gqa): Share fewer KV heads among more query heads, balancing MHA quality and MQA decoding bandwidth.
2023-05 · DoReMi: Optimizing Data Mixtures Speeds Up Language Model Pretraining (doremi): Use excess loss from a small proxy for group DRO, increasing underfit domain weights before training the target mixture.
2023-04 · Visual Instruction Tuning (llava): Freeze a vision encoder and project its features to the LLM token space, then tune with GPT-4-generated multimodal instructions.
2023-01 · BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models (blip2): Freeze the image encoder and LLM; train a lightweight Q-Former with learnable queries attending to vision tokens. First learn image–text representations, then project queries as a soft visual prompt for the frozen LLM.

2022¶

2022-11 · Fast Inference from Transformers via Speculative Decoding (speculative-decoding): A smaller draft model proposes multiple tokens; the target verifies a whole block. On rejection, sample from the corrected residual distribution, preserving the target distribution.

2021¶

2021-08 · Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolation (alibi): Replace learned positional vectors with a head-specific linear distance penalty on attention logits, enabling length extrapolation.
2021-04 · RoFormer: Enhanced Transformer with Rotary Position Embedding (rope): Rotate two-dimensional Q/K subspaces of each attention head by position, so their dot product depends naturally on relative offsets.
2021-02 · Learning Transferable Visual Models From Natural Language Supervision (clip): Independent image and text encoders project paired samples to the same unit sphere. A symmetric in-batch contrastive objective learns reusable zero-shot representations.
2021-01 · Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity (switch-transformer): Replace dense FFNs with sparse experts. Each token activates one expert, expanding parameter capacity at approximately fixed FLOPs.