News / #long-context Tag Long Context 227 articles archived under #long-context · RSS Sign in to follow arXiv — Machine Learning research 2h ago MosaicKV: Serving Long-Context LLM with Dynamic Two-D KV Cache Compression arXiv:2607.00760v1 Announce Type: new Abstract: Long-context LLM services now sustain prompts with hundreds of thousands to millions of tokens, making the key-value (KV) cache a first-order serving cost. Because the cache grows linearly with context length, it can exhaust GPU… 9 arXiv — Machine Learning research 2h ago GSRQ: Gain-Shape Residual Quantization for Sub-1-bit KV Cache arXiv:2607.01065v1 Announce Type: new Abstract: The deployment of Large Language Models (LLMs) with extended context windows is increasingly constrained by the linear growth of Key-Value (KV) cache memory. Vector Quantization (VQ), particularly Residual Quantization (RQ), is a… 36 arXiv — NLP / Computation & Language research 2h ago Logit-Contribution Scoring Identifies Non-Literal Retrieval Heads arXiv:2607.01002v1 Announce Type: new Abstract: In long-context use, large language models frequently synthesize answers from the meaning of a relevant context span rather than literally copy-pasting them. Identifying which attention heads perform this synthesis matters for… 5 r/LocalLLaMA community 19h ago Thinking about grabbing 4x Ascend GX10s Some in this sub have tested GLM5.2 on 4x DGX Sparks (or Ascend GX10) with 400-500 tok/s prompt processing and ~15 tok/s output at 128k context. Not blazing fast, but usable imo, especially with quantization. My thinking: If there's an open-source fable 5 sometime in december or… 20 arXiv — Machine Learning research 1d ago Hierarchical Global Attention (HGA) arXiv:2606.30709v1 Announce Type: new Abstract: Hierarchical Global Attention (HGA) is a drop-in replacement for dense causal attention in pretrained long-context transformers. HGA preserves the original checkpoint parameters: the pretrained $W_Q$, $W_K$, $W_V$, and $W_O$… 23 arXiv — NLP / Computation & Language research 1d ago RaBitQCache: Rotated Binary Quantization for KVCache in Long Context LLM Inference arXiv:2606.31519v1 Announce Type: cross Abstract: Long-context Large Language Model inference is severely bottlenecked by the massive Key-Value (KV) cache, yet existing sparse attention methods often suffer from static fixed-budget (Top-k) retrieval or rely on proxy scores that… 5 arXiv — Machine Learning research 1d ago ECHO: Prune to act, trace to learn with selective turn memory in agentic RL arXiv:2606.31650v1 Announce Type: new Abstract: Long-horizon language agents must repeatedly interact with tools, accumulate evidence, and make decisions under bounded context windows. Existing context-management methods make such rollouts feasible by truncating distant history,… 12 arXiv — NLP / Computation & Language research 1d ago SeKV: Resolution-Adaptive KV Cache with Hierarchical Semantic Memory for Long-Context LLM Inference arXiv:2606.31145v1 Announce Type: new Abstract: Large language models increasingly operate over long contexts, where the KV cache becomes a dominant memory bottleneck: its size grows linearly with sequence length and must be retained throughout decoding, making full GPU caching… 11 r/LocalLLaMA community 1d ago Devs - you have 64gb of VRAM - which model do you use for coding? I've currently settled on an unsloth version of Qwen 3.5 122b-a10b model (UD-IQ4_NL). With 100k bf16 context window, I only had to load a few layers into CPU/RAM, it runs around 30 tok/sec which is fine for me. I've tested many models, hours of testing but I am currently deeply… 32 r/LocalLLaMA community 1d ago HydraHead: From Head-Level Functional Heterogeneity to Specialized Attention Hybridization (from the Qwen team) The quadratic complexity of attention poses a critical bottleneck for long-context processing, spurring interest in hybrid attention designs. Most open-source hybrid models adopt a layer-wise strategy. Yet, prior work has noted the inherent difficulty of integrating Linear… 9 arXiv — Machine Learning research 2d ago HARD-KV: Head-Adaptive Regularization for Decoding-time KV Compression arXiv:2606.28831v1 Announce Type: new Abstract: Long-context LLM inference faces a fundamental conflict: head-adaptive compression algorithms (e.g., Top-$p$ nucleus sampling) offer superior accuracy by dynamically fluctuating memory budgets, yet modern inference engines (e.g.,… 29 arXiv — NLP / Computation & Language research 2d ago Turn-Averaged SAEs for Feature Discovery and Long-Context Attribution arXiv:2606.28548v1 Announce Type: new Abstract: Sparse autoencoders (SAEs) have become a useful tool for extracting interpretable features in language models. However, standard SAE architectures operate on individual token activations, meaning that the number of active features… 25 arXiv — NLP / Computation & Language research 2d ago Memory-Managed Long-Context Attention: A Preliminary Study of Editable Request-Local Memory arXiv:2606.28876v1 Announce Type: new Abstract: Long-context language models often conflate two different goals: compressing history into an efficient state, and maintaining reliable long-term memory. Linear, recurrent, and sparse attention reduce the cost of processing long… 14 arXiv — NLP / Computation & Language research 2d ago Coverage-Driven KV Cache Eviction for Efficient and Improved Inference of LLM arXiv:2606.29563v1 Announce Type: new Abstract: Large language models (LLMs) excel at complex tasks like question answering and summarization, thanks to their ability to handle long-context inputs. However, deploying LLMs is costly, not only due to the high computational demands… 7 arXiv — NLP / Computation & Language research 2d ago MATCH: Modulating Attention via In-Context Retrieval for Long-Context Transformers arXiv:2606.29844v1 Announce Type: new Abstract: The quadratic computational cost of traditional attention mechanisms poses a major bottleneck to the scalability and practical deployment of large language models (LLMs), particularly in long-context scenarios. To improve… 15 arXiv — NLP / Computation & Language research 2d ago LLM Agents Are Latent Context Managers: Eliciting Self-Managed Context via a Proprioceptive Dashboard arXiv:2606.30005v1 Announce Type: new Abstract: Long-horizon tool agents are bottlenecked by how their context grows toward the limits of the context window. Recent systems make context management agent- or system-controlled, but they either learn a compression policy that… 34 arXiv — NLP / Computation & Language research 3d ago Mitigating Position Bias in Transformers via Layer-Specific Positional Embedding Scaling arXiv:2606.27705v1 Announce Type: new Abstract: Large Language Models (LLMs) still struggle with the ``lost-in-the-middle'' problem, where critical information located in the middle of long-context inputs is often underrepresented or lost. While existing methods attempt to… 4 arXiv — NLP / Computation & Language research 3d ago NLL-Guided Full-Attention Layer Selection for Training-Free Sliding-Window Adaptation arXiv:2606.27791v1 Announce Type: new Abstract: Hybrid attention models that mix full and sliding-window attention across layers offer a promising approach to efficient long-context inference, but the critical question of \emph{which layers} should retain full attention remains… 19 arXiv — NLP / Computation & Language research 3d ago Position Bias Correction is Insufficient for One-Pass Attention Sorting arXiv:2606.27793v1 Announce Type: new Abstract: Long-context language models suffer from position bias, where information in middle positions is underutilized. Attention Sorting addresses this by iteratively reordering documents based on attention patterns, but its multiple… 9 r/LocalLLaMA community 3d ago High-quality GLM-5.2 Quant on 4x DGX Spark - Guide, Results, and Comps I got GLM-5.2 NVFP4 running on four DGX Sparks at 128K context. This is still a niche/hacky setup, but it is now a real serving point rather than just a proof of life. Objective : A high quality 4-bit quant running on 4x spark. Model: https://huggingface.co/Mapika/GLM-5.2-NVFP4… 9 r/LocalLLaMA community 3d ago Ornith-1.0-35B GGUF update: native MTP speculative-decode graft + full serving/TTFT/long-context numbers (llama.cpp, tp=1) Follow-up to my previous Ornith-1.0-35B Q3_K_M post. I grafted a native MTP draft head onto the IQ4_XS body (head at Q6) for self-speculative decode, single GPU, llama.cpp: 1.3-1.35x single-stream decode (172.6 -> 233.8 tok/s). Next-token distribution is byte-identical to… 11 NVIDIA Developer Blog official-blog 5d ago Creating the NVIDIA Nemotron 3 Ultra NVFP4 Checkpoint with NVIDIA Model Optimizer As context windows grow longer, moving large model weights efficiently becomes critical to performance. A common way to address this is quantization, an... 37 Hugging Face Daily Papers research 5d ago Information-Aware KV Cache Compression for Long Reasoning Abstract InfoKV is an entropy-aware KV cache compression framework that enhances long-context reasoning in LLMs by incorporating information-theoretic signals alongside attention weights. Generated by Qwen/Qwen2.5-Coder-32B-Instruct Reasoning capability has advanced rapidly in… 10 r/MachineLearning community 5d ago What if context compression is a diffusion noise function? Proposal + honest results from untrained-model experiments [R] I'm proposing a way to handle massive context longer than a model's context window by treating semantic compression as the noise function of a diffusion-like process. Instead of denoising masked tokens into coherent text (like DiffusionGemma or Nemotron-Diffusion do for… 20 arXiv — Machine Learning research 6d ago SSM Adapters via Hankel Reduced-order Modeling: Injection Site Determines Task Suitability in Long-Context Fine-Tuning arXiv:2606.26290v1 Announce Type: new Abstract: While parameter-efficient fine-tuning (PEFT) typically targets attention projectors, its efficacy for tasks requiring sequential state accumulation remains under-explored. We examine if PEFT for such tasks can benefit from state… 18 arXiv — Machine Learning research 6d ago PersistentKV: Page-Aware Decode Scheduling for Long-Context LLM Serving on Commodity GPUs arXiv:2606.26666v1 Announce Type: new Abstract: Autoregressive large language model (LLM) serving is increasingly limited by key-value (KV) cache movement rather than dense matrix multiplication. Modern paged-attention systems reduce KV-cache fragmentation and mature kernels… 20 arXiv — NLP / Computation & Language research 6d ago Context Recycling for Long-Horizon LLM Inference arXiv:2606.26105v1 Announce Type: new Abstract: Large language models (LLMs) exhibit strong capabilities in short-context reasoning but degrade in performance over long conversational horizons due to context window limitations and inefficient token usage. We introduce… 27 arXiv — NLP / Computation & Language research 7d ago Dustin: Draft-Augmented Sparse Verification for Efficient Long-Context Generation with Speculative Decoding arXiv:2606.24957v1 Announce Type: new Abstract: While speculative decoding improves inference throughput for multi-batch long-context Large Language Models (LLMs), its efficiency is often limited by a verification bottleneck where Key-Value (KV) cache loading dominates latency.… 19 arXiv — Machine Learning research 8d ago Forget Without Compromise: Nexus Sampling for Streaming KV-Cache Eviction Under Fixed Budgets arXiv:2606.23961v1 Announce Type: new Abstract: Long-context and agentic LLM workloads push the KV cache past any fixed memory budget, forcing the inference stack to permanently evict tokens at every step of a continuous-inference stream. Existing methods all share the same… 20 arXiv — NLP / Computation & Language research 8d ago AVOC: Enhancing Hour-Level Audio-Video Understanding in Omni-Modal LLMs via Retrieval-Inspired Token Compression arXiv:2606.24286v1 Announce Type: new Abstract: Multimodal Large Language Models have achieved remarkable progress in short-form audio-video understanding, yet long-form audio-video comprehension remains challenged by limited context windows and severe information redundancy. To… 15 arXiv — NLP / Computation & Language research 8d ago Harmonic: Hierarchical State Space Models for Efficient Long-Context Language Modeling arXiv:2606.24650v1 Announce Type: new Abstract: We present Harmonic, a hierarchical state space model (SSM) for language modeling. The architecture stacks three recurrent levels at progressively slower timescales; each level receives the prediction error of the level below as… 21 arXiv — NLP / Computation & Language research 8d ago The Sparse Frontier: Sparse Attention Trade-offs in Transformer LLMs arXiv:2504.17768v3 Announce Type: replace Abstract: Sparse attention offers a promising strategy to extend long-context capabilities in Transformer LLMs, yet its efficiency-accuracy trade-offs remain unclear due to the lack of comprehensive evaluation. We address this gap with… 29 Hugging Face Daily Papers research 8d ago Beyond Reward Engineering: A Data Recipe for Long-Context Reinforcement Learning Abstract Data-centric approach using curated datasets and minimal GRPO setup significantly improves long-context reasoning in large language models, outperforming prior reinforcement learning methods. Generated by Qwen/Qwen2.5-Coder-32B-Instruct Long-context reasoning is an… 15 Hugging Face Daily Papers research 9d ago HydraHead: From Head-Level Functional Heterogeneity to Specialized Attention Hybridization Abstract HydraHead is a novel attention hybridization architecture that combines Full Attention and Linear Attention at the head level, achieving superior long-context performance with reduced training overhead through interpretability-driven selection and scale-normalized… 34 Hugging Face Daily Papers research 9d ago EvoEmbedding: Evolvable Representations for Long-Context Retrieval and Agentic Memory Abstract EvoEmbedding is a dynamic embedding model that generates adaptive representations by maintaining a continuously updated latent memory, enabling improved retrieval performance in long-context scenarios. Generated by Qwen/Qwen2.5-Coder-32B-Instruct Existing embedding… 32 r/LocalLLaMA community 9d ago Is Gemma 4 going to be the next Mistral (or Qwen3.6) one day? Concerning the lack of finetunes https://eqbench.com/creative_writing.html#:~:text=gemma%2D4%2D31B,Sample From what I've seen Gemma 4 has better everything (especially long-context adherence) EXCEPT for the raw prosing performance of Mistral... finetunes . Comparing bases only, Mistral Small 3.2 (the… 5 r/LocalLLaMA community 9d ago GLM-5.2 UD-IQ1_M on llama.cpp — 5090 + 3090 Ti speed test (~ 579 t/s prefill @ 8k ctx, ~324 t/s prefill @ 57k ctx, ~10.6 t/s decode) Just sharing some speed test numbers for GLM-5.2 running on llama.cpp. Setup: Model: unsloth/GLM-5.2-GGUF, UD-IQ1_M quant GPUs: RTX 5090 + RTX 3090 Ti 186 GB DDR5 used Debian 13 CUDA 13.3 128k context, q8_0 KV cache Prefill (prompt processing): n_tokens tokens/s 8,201 579.75… 4 r/LocalLLaMA community 10d ago Not a new model, just a Happy Father's Day and a thank you. I know this isn't our usual discussion about context windows, quantization, or the latest model drop, but I just wanted to take a quick moment to say thank you. As a dad myself, I really appreciate this great community. Between the daily grind and family life, diving into this… 12 r/MachineLearning community 10d ago I released a softmax-free attention model at GPT-2 Medium scale (~354M params, 11.5B tokens): structural sparsity + tile-skipping kernels for long-context VRAM savings. Open weights + custom Triton kernels [R]   submitted by   /u/NonGameCatharsis [link]   [comments] 29 Simon Willison community 12d ago Quoting Sean Lynch The real valuable capability MCP offers over skills/CLI is isolating the auth flow outside of the agent’s context window, and potentially out of the harness completely. [...] Maybe the idealized form of MCP is just an auth gateway for the API and nothing else. That’d still be a… 8 arXiv — NLP / Computation & Language research 13d ago AtomMem: Building Simple and Effective Memory System for LLM Agents via Atomic Facts arXiv:2606.19847v1 Announce Type: new Abstract: Large language models (LLMs) demonstrate strong reasoning and generation abilities, but their fixed context windows limit long-term information accumulation and reuse across multi-session interactions. Existing memory-augmented… 32 arXiv — NLP / Computation & Language research 13d ago HydraHead: From Head-Level Functional Heterogeneity to Specialized Attention Hybridization arXiv:2606.20097v1 Announce Type: new Abstract: The quadratic complexity of attention poses a critical bottleneck for long-context processing, spurring interest in hybrid attention designs. Most open-source hybrid models adopt a layer-wise strategy. Yet, prior work has noted the… 13 arXiv — NLP / Computation & Language research 13d ago MedRLM: Recursive Multimodal Health Intelligence for Long-Context Clinical Reasoning, Sensor-Guided Screening, Evidence-Grounded Decision Support, and Community-to-Tertiary Referral Optimization arXiv:2606.20164v1 Announce Type: new Abstract: Real-world clinical decision support requires reasoning over heterogeneous and longitudinal patient information rather than answering isolated medical questions. However, current medical large language models and… 29 r/LocalLLaMA community 13d ago 2 weeks since the release of Gemma 4 12b Unified, how are we feeling about it? I'm looking for a good model to run on a 5090 and have ample context ~128k. This model looks good for me, it seems to have good performance in the 12b range, almost comparable to Gemma 4 26B A4B. Building a custom harness for it and have ~300m of tokens to fine tune on. Do you… 6 arXiv — Machine Learning research 14d ago Gaussian Mixture Attention: Linear-Time Sequence Mixing via Probabilistic Latent Routing arXiv:2606.18283v1 Announce Type: new Abstract: The dense token-to-token interaction pattern of standard dot-product attention remains a central bottleneck in scaling Transformer architectures to long contexts. We introduce \textbf{Gaussian Mixture Attention (GMA)}, a… 26 arXiv — NLP / Computation & Language research 14d ago Beyond Reward Engineering: A Data Recipe for Long-Context Reinforcement Learning arXiv:2606.18831v1 Announce Type: new Abstract: Long-context reasoning is an essential capability for large language models, particularly when they are deployed as autonomous agents that must reason over lengthy trajectories. Reinforcement learning (RL) has recently emerged as a… 36 arXiv — NLP / Computation & Language research 14d ago Human-AI Coevolution Dynamics: A Formal Theory of Social Intelligence Emergence Through Long-Term Interaction arXiv:2606.19144v1 Announce Type: cross Abstract: Current conversational AI systems have made significant progress in language generation, personalization, and long-context interaction. However, most existing methods model social behavior through isolated components such as… 33 Hugging Face Daily Papers research 14d ago Rethinking the Role of Efficient Attention in Hybrid Architectures Abstract Hybrid architectures combining full attention with efficient attention modules like sliding-window attention exhibit distinct scaling behaviors and optimization trajectories, with efficient attention primarily affecting the emergence speed of long-context capabilities… 29 arXiv — Machine Learning research 15d ago AnchorKV: Safety-Aware KV Cache Compression via Soft Penalty with a Refusal Anchor arXiv:2606.17872v1 Announce Type: new Abstract: Large language models (LLMs) outperform earlier architectures on generative inference and long-context tasks, but their large size introduces significant challenges in memory usage, energy cost, and on-device deployment. Since… 27 r/LocalLLaMA community 15d ago GLM-5.2 just dropped open weights and it already looks weirdly strong for coding GLM-5.2 just released and the early numbers look pretty insane. 1M context window, open weights, MIT license, two reasoning effort modes, and it is already showing up near the top of coding arenas. I know every new model gets hyped for 24 hours, but this one actually looks worth… 28 Page 1 of 5 · 227 articles Older →
