News / #long-context Tag Long Context 433 articles archived under #long-context · RSS Sign in to follow Simon Willison community 8h ago Introducing Hy4 Preview Introducing Hy4 Preview New open weight text input (no vision) LLM from Chinese company Tencent today: 770B total parameters, 49B active parameters, 1M token context window, 1.56TB on Hugging Face . This is a big size increase from their previous Hy3 in July, which was 295B, 21B… 9 r/LocalLLaMA community 14h ago (NInfer Fork) I wanted to have a 1M context Qwen-3.8 27B, tp2, dual 5090s Hey! I forked NInfer (a from-scratch C++20/CUDA inference engine for Qwen models) and added two things: tensor-parallel across two GPUs, and YaRN ×4 rope scaling. Together they let Qwen3.8-27B NVFP4 run a 1,048,576-token context on two consumer 5090s — 27.4 GB per card, no… 21 r/LocalLLaMA community 19h ago Qwen 3.8 27B at 50 tok/s with 100k Context on a 16GB GPU! (beellama.cpp) I wanted to share my successful setup for running a Qwen 3.8 27B model with a massive context window on a consumer 16GB GPU (RTX 4070 Ti SUPER). The goal was to fit everything into VRAM without sacrificing quality or speed. 🧠 Key Components Model:… 38 arXiv — NLP / Computation & Language research 2d ago TwinKV: A Composable Repair Pass for KV Cache Eviction via Pairwise Key Redundancy arXiv:2608.27128v1 Announce Type: new Abstract: Long-context inference is bottlenecked by the memory footprint of the key-value (KV) cache, especially for small models under tight resource budgets. Existing KV cache eviction methods score tokens using the model's attention… 14 r/LocalLLaMA community 2d ago yall are sleeping on qwen 3.8 27b q2 + q2 dflash + q5 kv ok bit more context: it's actually a QAT Q2 for Qwen 3.8 27 B: https://huggingface.co/sdkyuan/qwen3.8-27B-qat-q2_0-gguf QAT Q2 for DFlash model: https://huggingface.co/HermiHg/Qwen3.8-27B-DFlash2-Q2_K_S-MIX-GGUF Q5 KV seems to cause 0 problems for me; I've used it up to 200K… 36 Vercel — AI dev-tools 2d ago Hy4 Preview now available on AI Gateway Hy4 Preview from Tencent is now available on AI Gateway. Hy4 Preview is an open-source Mixture-of-Experts model with 770B total parameters aimed at long-horizon coding, document analysis, game development, and scientific reasoning. It serves a context window of 1M tokens. To use… 32 r/LocalLLaMA community 2d ago Over 200k context on 16GB VRAM with Qwen 3.8 27B UD-IQ3_XXS I was using UD-Q3_K_XL until now with more than 140000 context. Quality wise it's very good, very few erroneous tool calls. Then I saw many others here reporting good results with IQ3_XXS, so I gave it a try. The downside is prompt processing speed went down from 700-800 tk/s to… 7 r/LocalLLaMA community 2d ago GLM-5.3-Flash @ DGX Station GB300: ~206 tok/s (single stream), 1M context Hey all! I'm finally doing some cool stuff with my "thinking heater" (h/t u/-TV-Stand- ). I'm still experimenting with GLM-5.2 (in anticipation of 5.3 coming tomorrow, I hope!) and things are very cool so far. With the release of GLM-5.3-flash, I decided to play with it on the… 25 arXiv — NLP / Computation & Language research 3d ago A Storage-Retrieval Gap in Parametric Knowledge Graph Memory arXiv:2608.25489v1 Announce Type: cross Abstract: Graph retrieval-augmented generation places retrieved subgraphs into the model's context window at query time, paying a recurring token cost and exposing source data on every call. We study an alternative: compiling a knowledge… 21 arXiv — Machine Learning research 3d ago Beyond Scaling: Self-Evolving LLM Agents for Hardware Kernel Optimization via an Experience-Driven Workflow and Experience Graph Memory arXiv:2608.25570v1 Announce Type: new Abstract: Hardware kernel optimization requires repeated compilation, correctness testing, profiling, and revision. LLM agents can automate parts of this process, and stronger foundation models, longer context windows, and longer execution… 19 arXiv — NLP / Computation & Language research 3d ago ClueWeaver: Reward-Guided Dual-Agent Evidence Reasoning for Compact LLMs on Literary Long Narratives arXiv:2608.25531v1 Announce Type: new Abstract: Humanities and social science research requires close reading of long narrative materials such as novels, scripts, archives, and case reports, yet many users have limited access to costly proprietary long-context models. Compact,… 29 arXiv — NLP / Computation & Language research 3d ago Reconstructing the Right Episode: Evaluating Interleaved Conversational Memory Beyond Long Context arXiv:2608.25655v1 Announce Type: new Abstract: Conversations with chat assistants increasingly span many topics in a single long-running thread, challenging memory systems. Existing long-context and memory benchmarks often expose session or topic boundaries, or probe direct… 25 Vercel — AI dev-tools 3d ago Ling 3.0 Flash Fin now available on AI Gateway for free Ling 3.0 Flash Fin from Inclusion AI is now available on AI Gateway, free to use through September 25. Ling 3.0 Flash Fin is a finance-focused version of Ling 3.0 Flash . It has a 256K token context window, produces up to 32K output tokens, and supports reasoning and function… 29 r/LocalLLaMA community 4d ago First serious confirmation. Ox Alpha is GLM-5.3-Flash https://x.com/romanchernin/status/2092488160680751437?s=20 - Multimodal (Vision) - 1M Tokens Context Window - DeepSWE ~63%   submitted by   /u/MrWidmoreHK [link]   [comments] 14 Smol AI News news-outlet 4d ago not much happened today **Z.ai** launched **GLM-5.3-Flash**, a natively multimodal model with a **1M-token context window**, **320B total parameters / 18B active parameters**, under the **MIT License**. It is positioned as a price-competitive successor to GLM-5.2 and claims performance on par with… 29 arXiv — Machine Learning research 4d ago PuzzleKV: Page-Wise Low-Rank Decomposition for KV Cache Compression arXiv:2608.23843v1 Announce Type: new Abstract: Long-context inference in large language models (LLMs) is increasingly limited by the memory required for the key-value (KV) cache. KV cache compression addresses this problem by reducing the storage cost of previous tokens. Among… 9 Vercel — AI dev-tools 4d ago GLM 5.3 Flash now available on AI Gateway GLM 5.3 Flash from Z.ai is now available on AI Gateway. The model is a faster, cheaper sibling of GLM 5.3 built for coding and agent tasks that run across many steps. GLM-5.3 Flash is a multimodal model that supports text and vision input, with a 1M token context window and a… 4 Vercel — AI dev-tools 4d ago Qwen 3.8 Flash now available on AI Gateway Qwen3.8-Flash from Alibaba is now available on AI Gateway. It takes text and images as input, serves a context window of 1 million tokens, and can return up to 65k tokens in a response. Alibaba points it at coding, tool use, and multi-step agent work. To use Qwen3.8-Flash, set… 24 r/LocalLLaMA community 4d ago Peak Portable Personal Datacenter Portable rig for Qwen3.8-27B-BF16 200K+ token prompts. My work Panasonic Toughbook + the T1 + power brick + headphones all fit in my lunchbox. Need the BF16 for huge context highly sensitive document OCR, image analysis, aggregation and summarization. I've done a ton of testing… 35 Hugging Face Daily Papers research 5d ago RIBOSPAN: A Long-Context RNA Foundation Model for Versatile RNA Modeling Abstract RIBOSPAN is a large bidirectional RNA foundation model pretrained on up to 10,240 nucleotides that enables high-resolution full-transcript modeling, strong long-context representations, and discrete-diffusion-based mRNA generation and redesign. Generated by… 29 Hugging Face Daily Papers research 5d ago TileMix: Tile-Centric Mixed-Precision Attention for LLM Inference Acceleration Abstract TileMix routes attention score tiles to mixed FP16 or INT8 precision within fused dense attention, recovering long-context accuracy while improving prefill throughput without retraining. Generated by thinkingmachines/Inkling-Small Long-context prefill in large language… 5 NVIDIA Developer Blog official-blog 5d ago How NVIDIA Groq 3 LPX Unlocks Ultrafast Interactivity at Long Context on NVIDIA Vera Rubin NVIDIA Groq 3 LPX is the interactive AI inference accelerator for the NVIDIA Vera Rubin platform. At the core of the platform is NVIDIA Vera Rubin NVL72, the... 15 Smol AI News news-outlet 6d ago not much happened today **Z.ai** released the **GLM-5.3** open-weight model family, optimized for **agentic coding** and **cyber defense**, with impressive specs like **744B total / 40B active parameters**, **1M context window**, and a **239GB 2-bit** variant retaining **81% accuracy**. **Tencent**… 28 arXiv — Machine Learning research 6d ago BF1: A Causal Dyadic Sparse-Attention Retrofit for Efficient Long-Context Transformers arXiv:2608.20427v1 Announce Type: new Abstract: Dense causal attention remains expensive at long context even when implemented with highly optimized exact kernels. We study BF1, a deterministic block-aligned dyadic sparse-attention route that combines a small exact local… 16 arXiv — Machine Learning research 6d ago Rethinking Expressivity and Efficiency in Test-Time Training arXiv:2608.21308v1 Announce Type: new Abstract: Test-Time Training (TTT) enables long-context processing via continuous weight updates during inference, but current methods struggle to balance the expressivity of per-token update dynamics with the hardware efficiency of… 19 arXiv — NLP / Computation & Language research 6d ago Inhibitory Attention for Clinical Long-Context Reasoning: Characterizing and Mitigating Lost-in-the-Middle Effects in EHR Processing arXiv:2608.20348v1 Announce Type: new Abstract: Electronic health records now routinely exceed 100,000 tokens per patient. Yet large language models exhibit the lost-in-the-middle (LitM) effect: information near the center of a long context is retrieved less reliably than… 19 arXiv — NLP / Computation & Language research 6d ago SCOPE: A Generative Approach for LLM Prompt Compression arXiv:2508.15813v2 Announce Type: replace Abstract: A big issue in modern LLM applications is they tend to feed long context to LLM, which results in high inference cost and latency, and may exceed the context limit. Prompt compression addresses this issue by reducing the length… 11 r/LocalLLaMA community 7d ago Single RTX 5090: Qwen3.8-27B NVFP4 at a real 262K context in vLLM — 77 tok/s short-context, 64.7 tok/s at 128K This is the Qwen3.8-27B setup I actually use every day on one RTX 5090. I wanted to write it down with enough detail that another 5090 owner can reproduce it instead of guessing which memory knobs I used. The short version: the full 262,144-token window fits together with… 25 r/MachineLearning community 8d ago I developed my own quantized LLM from scratch, trained on 30B tokens, deploys in 60 MB [R] I trained a 250M parameter model from scratch on 30B tokens of fineweb. It’s quantized to under 2 bits so the whole deployment is 60 MB and it needs about 80 MB of RAM to run. Runs around 400 tok/s on a normal laptop CPU, no GPU needed. How the long context works: the most… 10 Hugging Face Daily Papers research 9d ago FlashPrefill V2: Block-Sparse Prefill Attention for Long-Context LLM Serving Abstract FlashPrefill V2 improves long-context serving via mean-corrected sparse attention, optimized GPU operators, and framework integration, achieving large speedups over dense baselines. Generated by thinkingmachines/Inkling-Small Long-context modeling is a pivotal… 9 arXiv — Machine Learning research 9d ago Inadvertent Context Leakage in Language Models arXiv:2608.19857v1 Announce Type: new Abstract: For AI agents to be useful beyond simple chat, they must hold sensitive user context such as calendars, credentials, health records, and financial data. We study whether the mere presence of such secrets in a model's context window… 13 arXiv — Machine Learning research 9d ago HYDRA: A Heterogeneous Chiplet DSE Framework for Serving Dynamic Hybrid LLM Workloads arXiv:2608.19395v1 Announce Type: cross Abstract: Hybrid Transformer-Mamba large language models (LLMs) enhance long-context efficiency, but their heterogeneous computation and communication patterns complicate efficient hardware acceleration. Chiplet-based architectures offer a… 7 arXiv — NLP / Computation & Language research 9d ago FlashPrefill V2: Block-Sparse Prefill Attention for Long-Context LLM Serving arXiv:2608.19758v1 Announce Type: new Abstract: Long-context modeling is a pivotal capability for Large Language Models, yet the quadratic complexity of attention remains a critical bottleneck, particularly during the compute-intensive prefilling phase. Our previous work,… 33 arXiv — NLP / Computation & Language research 9d ago Learning how to Forget: Fine-tuning for Long-Context Sparse Attention arXiv:2608.19920v1 Announce Type: new Abstract: A lot of prior work addressed key-value (KV) cache selection and compression by sparse attention to enable long-context inference for transformer language models without excessive hardware budgets. We provide a new method for… 18 arXiv — NLP / Computation & Language research 10d ago Beyond Teacher Likelihood: Group-Calibrated On-Policy Distillation for Long-Context Reasoning arXiv:2608.19181v1 Announce Type: cross Abstract: On-policy distillation (OPD) trains a student on its own responses using dense token-level guidance from a stronger teacher. In long-context tasks, however, token-level teacher support can favor locally plausible responses that… 36 arXiv — NLP / Computation & Language research 10d ago LongNovel: A Multi-Scale Benchmark for Hallucination Detection in Long-Context Novel Summarization arXiv:2608.18082v1 Announce Type: new Abstract: Although context windows have expanded significantly in recent years, hallucinations in long-context summarization remain a challenge. Long novels are better suited than news or papers for researching these hallucinations, due to… 25 arXiv — NLP / Computation & Language research 10d ago Institutional Books - Enriched Text: A customizable multilingual open-source pipeline for denoising, deduplicating, and annotating OCR text at scale arXiv:2608.19026v1 Announce Type: new Abstract: Released in 2025, Institutional Books: Harvard Library (IB-HL) is a collection of 983,004 volumes (242B o200k_base tokens), originally digitized through Harvard Library's participation in the Google Books Library project. As… 17 Hugging Face Daily Papers research 11d ago CoinVE-200K: A Large-Scale High-Quality Dataset for Compositional Instruction-Guided Video Editing Abstract A new dataset, benchmark, and 22B model enable compositional instruction-guided video editing with multi-region attention and temporal coherence. Generated by thinkingmachines/Inkling-Small The quality and diversity of instruction-based video editing datasets are… 11 arXiv — Machine Learning research 11d ago Dynamic Compression in Recurrent Networks arXiv:2608.17896v1 Announce Type: new Abstract: Recurrent models process long contexts efficiently by compressing their history into a fixed-size state, but modern architectures typically do so in a single causal pass over the sequence. Each input must therefore be compressed… 13 arXiv — NLP / Computation & Language research 11d ago Token Optimization and Context Window Management in Multi-Agent AI Workflows arXiv:2608.17188v1 Announce Type: new Abstract: Multi-agent AI workflows are limited not only by model quality but by token cost, latency, and context-window quality. This paper presents a practitioner framework for token optimization and context-window management, grounded in… 23 arXiv — NLP / Computation & Language research 11d ago Do Large Language Models Play Six Degrees of Separation? Measuring Topological Compression in Long-Context Manifolds arXiv:2608.17950v1 Announce Type: new Abstract: Large Language Models (LLMs) demonstrate remarkable multi-hop reasoning capabilities over long contexts, yet the internal mechanisms enabling these distant cognitive leaps remain poorly understood. Traditional attention-based… 12 arXiv — NLP / Computation & Language research 11d ago MoNe: Modular Neural Memory for Efficient Long Context Inference arXiv:2608.17616v1 Announce Type: cross Abstract: We present MoNe, a lightweight modular neural memory that attaches to any frozen pretrained Transformer to enable long-context inference without retraining. MoNe reads context in fixed-size segments via test-time learning of… 8 r/LocalLLaMA community 11d ago Ling-3.0-tiny is a very interesting model. Run on NVIDIA Orin Nano Super 8GB at 128K context with IQ4_NL quant. I have been searching for suitable model to run on my 8GB RAM toy, NVIDIA Orin Nano Super 8GB. This little toy was priced at $249 earlier this year (not any more), and pulls very little power when idle. It was an interesting device that suitable for an agent to host on. It is… 31 r/LocalLLaMA community 11d ago Running DeepSeek V4 Flash Q4_K_XL at ~100 tok/s prompt processing on 4× RTX 3060 12GB I managed to run the 143–144 GiB DeepSeek-V4-Flash-0731 UD-Q4_K_XL GGUF on four RTX 3060 12GB cards while keeping a 360k–376k context window. Hardware: CPU: Intel Core i9-10920X, 12C/24T RAM: 128 GB DDR4-3200, quad-channel GPU: 4× NVIDIA RTX 3060 12GB Total VRAM: 48 GB Storage:… 25 arXiv — NLP / Computation & Language research 12d ago SEER: Long-Context Reasoning via Selective Visual-Text Compression arXiv:2608.15962v1 Announce Type: new Abstract: Long-context reasoning remains computationally expensive for large language models due to the quadratic complexity of attention over text tokens. Visual-text compression offers a promising alternative by rendering text into images… 12 Hugging Face Daily Papers research 12d ago MegaParts: Scaling Part-Aware 3D Object Generation to 300 Parts via Token-Efficient Autoregressive Modeling Abstract MegaParts scales part-aware 3D generation via token-efficient vector-quantized part tokens and structured autoregressive sequence modeling with long-context training. Generated by thinkingmachines/Inkling-Small Part-aware 3D object generation is essential for graphics… 21 r/LocalLLaMA community 12d ago Made this game in two prompts with Q4, Qwen 3.8 is amazing This took one prompt to build, and another follow up prompt to fix two issues (player got stuck with the bomb and broken enemies path-finding), this is only html, css and js, no external assets, all done by Qwen. Using UD-Q4_K_XL in llama.cpp with 128k context and k5_0/v4_1… 36 Hugging Face Daily Papers research 13d ago SimpleOPD: Simple Tokenizer-Agnostic On-Policy Distillation for Long-Context Reasoning Abstract On-policy distillation from a long-context reasoning teacher to short-context students improves mathematical proof reasoning and generalizes to science benchmarks by aligning token spans, constraining length growth, and stabilizing training. Generated by… 26 arXiv — Machine Learning research 13d ago The Query Knows What to Forget: A Second Erase Direction for Linear Attention arXiv:2608.13668v1 Announce Type: new Abstract: Linear attention keeps a state of fixed size. At long context, many stored items share this state, and interference between them degrades retrieval. Gated DeltaNet-2 (GDN-2), like every delta-rule model before it, derives its erase… 28 arXiv — NLP / Computation & Language research 13d ago KV Cache Compression Through the Lens of Transform Coding arXiv:2608.14191v1 Announce Type: cross Abstract: The key-value (KV) cache stores information from past tokens and is a major memory bottleneck in long-context inference. Existing quantization methods address this bottleneck by representing the KV cache uniformly with… 29 Page 1 of 9 · 433 articles Older →
