News / #benchmark Tag Benchmark 500 articles archived under #benchmark · RSS Sign in to follow r/LocalLLaMA community 6d ago Qwen 3.8 27B Aider score I ran the Aider benchmark on Qwen 3.8 27B FP8 with FP8 KV cache 256K context vLLM. The score: 72.9 This matches Gemini 2.5 Pro from 2025-04-12 which also scored 72.9. Beats Claude Opus 4 from 2025-05-25 which scored 72.0. DeepSeek R1 2025-06-06 scored 71.4. It may just be a… 24 Hugging Face Daily Papers research 6d ago Daedalus-150M: A Convolution-Attention Hybrid Designed for CPU Inference Abstract A small hybrid language model using sparse attention and short convolutions achieves faster CPU inference and better benchmark scores than larger conventional models despite training on far less data. Generated by thinkingmachines/Inkling-Small Small language models are… 26 Smol AI News news-outlet 6d ago not much happened today **OpenAI** announced benchmark results for its custom inference chip **Jalapeño**, showing **1.5–1.9×** better efficiency and **1.7–3.6×** lower latency compared to NVIDIA **GB200/GB300**. Deployment starts by year-end with **Gen 2** and **Gen 3** in development. The chip runs… 9 Hugging Face Daily Papers research 6d ago OmniAssistBench: Assistant-style Interaction Benchmark for Omni-LLMs Abstract OmniAssistBench evaluates real-time interactive video assistants by reverse-engineering multi-turn interaction videos, revealing that current omni-modal models struggle with visual prompts, context retention, and timely responses. Generated by… 33 arXiv — Machine Learning research 6d ago RiskTraf: Risk-Extrapolated Residual Learning for Multi-Variate Traffic Flow Prediction arXiv:2608.20656v1 Announce Type: new Abstract: Traffic sensors commonly record flow, speed, and occupancy, but standard traffic flow forecasting benchmarks and models rarely exploit all three raw measurements reliably. Although speed and occupancy provide sensor-native… 28 arXiv — Machine Learning research 6d ago A Critical Audit of Spatiotemporal Forecasting Benchmark Datasets and Baselines arXiv:2608.20980v1 Announce Type: new Abstract: Graph neural networks (GNNs) are routinely employed for short-range forecasting on multivariate time series with a spatial graph structure. Despite the availability of many alternative datasets, method innovations within this… 14 arXiv — Machine Learning research 6d ago BackDFL: A Unified Benchmark For Backdoor Attacks and Defenses In Decentralized Federated Learning arXiv:2608.21137v1 Announce Type: new Abstract: Decentralized Federated Learning (DFL) promises trust-free collaborative learning by replacing the centralized parameter server with peer-to-peer model exchange. However, this architectural shift fundamentally reshapes the threat… 7 arXiv — Machine Learning research 6d ago ConceptTS: LLM-Guided Concept Bottlenecks for Interpretable Multivariate Time-Series Forecasting arXiv:2608.21277v1 Announce Type: new Abstract: State-of-the-art multivariate time-series forecasters can model complex temporal and cross-variable dependencies, yet their opaque representations provide limited insight into why a particular forecast is produced. This lack of… 11 arXiv — NLP / Computation & Language research 6d ago Multilingual Verifier Bias in RLVR: Benchmark, Rollout Diagnosis, and the Cross-Lingual Selection Bottleneck arXiv:2608.20362v1 Announce Type: new Abstract: Reinforcement learning with verifiable rewards (RLVR) is a standard recipe for training large language models on mathematical reasoning, where an answer verifier serves as a language-neutral reward function. We show that this… 33 arXiv — Machine Learning research 6d ago FlavourBench: Ranking Frontier Language Models with Executable Culinary Ground Truth arXiv:2608.20574v1 Announce Type: cross Abstract: Open-ended language-model benchmarks usually inherit a judge: a human preference panel, another model, or a brittle exact-match key. We introduce FlavourBench, an automated benchmark in which a versioned culinary system supplies… 35 arXiv — NLP / Computation & Language research 6d ago Evaluation-as-Search: Adaptive Discovery of Grounding Failures in Meeting Assistants arXiv:2608.20392v1 Announce Type: new Abstract: LLM-powered meeting assistants are deployed at scale, yet systematic evaluation of their grounding fidelity remains limited to static benchmarks that miss failure modes tied to specific discourse structures or reasoning demands. We… 14 arXiv — NLP / Computation & Language research 6d ago AgentMercury: Your Agent Can Synthesize Verifiable Environments for Business Scenarios at scale arXiv:2608.20634v1 Announce Type: new Abstract: Agents learn to act through interaction with environments, yet the environments used for training are often manually constructed or synthesized around predefined tasks and benchmarks. This task-centric paradigm makes it difficult… 15 arXiv — NLP / Computation & Language research 6d ago Benchmarking Patent Drafting from Inventor-Style Disclosures arXiv:2608.21249v1 Announce Type: new Abstract: While recent large language models (LLMs) have achieved promising results on individual patent drafting tasks, they fundamentally fail to investigate the core challenge of real-world patent drafting: generating a complete and… 37 Hugging Face Daily Papers research 6d ago Beyond Correctness: Benchmarking and Aligning Response Behaviors in Hybrid-Thinking MLLMs Abstract Hybrid-thinking multimodal language models suffer from response-pattern misalignment between thinking and non-thinking modes, which is addressed by a diagnostic benchmark and pattern-specific reinforcement learning penalties. Generated by thinkingmachines/Inkling-Small… 4 r/LocalLLaMA community 6d ago Benchmark results: what is the best and fastest engine to run Qwen3.8-27B on macOS WARNING: I think I made a mistake with omlx, and its results below seem to be without any kind of acceleration. I am rerunning them with Lightning MTP and ANE support for prefill. Will update the post once done. The new Qwen 3.8 27B is fantastic for local agentic use. The… 5 r/LocalLLaMA community 6d ago Qwen 3.8 27B for actual local programming Most YouTube benchmarks only show trivial tasks like generating landing pages or simple Three.js games. Is a local model like Qwen 3.8 27B actually capable of real-world systems programming—such as building GTK4 or Qt 6 applications in Rust or C++ with external libraries?… 10 r/MachineLearning community 6d ago 28 TPS on Qwen2.5-7B across two separate cloud regions over public WAN using speculative decoding + CUDA Graphs [P] been building ShardFlow for the past few months, a distributed LLM inference framework that splits any HuggingFace transformer across N GPU machines and uses neural speculative decoding to deal with WAN latency. the setup for the benchmark: two T4 nodes in separate GCP regions… 7 r/LocalLLaMA community 7d ago # Qwen3.8-27B — One Week Later: The r/LocalLLaMA + r/LocalLLM Verdict Companion to the Qwen 3.8 Release Megathread . Compiled from ~2,000 posts scanned across both subs, with deep reads of the 45 highest-signal threads (560 posts and comments), Aug 15–22, 2026, plus independent X benchmarks. Every number is attributed to the poster's stated… 16 r/LocalLLaMA community 7d ago I benchmark DFlash 2 (PR build) in llama.cpp on Qwen 3.8 27B against all speculative methods for 3 days. 2.26x on 100 real coding prompts, 4.68x with one n-gram drafter on top. Up to 8x on specific cases. Hey guys, Inco AI shipped DFlash 2 a few days ago with a drafter for Qwen 3.8 27B and a llama.cpp PR. I built the PR and ran it against plain decoding, MTP, the n-gram lookup drafters, and my July DFlash 1 numbers on Qwen 3.6 27B for 3 days. One RTX PRO 6000, concurrency 1,… 26 r/LocalLLaMA community 8d ago Artificial Analysis "Intelligence": A meaningless benchmark https://preview.redd.it/84zi5nsdawkh1.png?width=2368&format=png&auto=webp&s=1109e69db807b153064b1f5b61d22cf1e9fbca05 Another user posted the benchmarks for Qwen 3.8 27B today, and while I think Qwen 27B is a really powerful model, I can't help but notice just how meaningless… 34 Hugging Face Daily Papers research 8d ago FlowEvo: Self-Evolving Agents through the Co-Evolution of Workflows and Executable Skills Abstract FlowEvo enables large language model agents to co-evolve reusable skills and workflows during inference, improving accuracy and efficiency across diverse benchmarks. Generated by thinkingmachines/Inkling-Small Large language model agents can adapt to complex tasks by… 15 Vercel — AI dev-tools 8d ago How Ora benchmarks every major AI agent on Vercel Ora on Vercel Front end, back end, and agent runtime on one platform Every major agent tested side by side on live sites Hundreds of commits a day from a 16-person engineering team Ora sends agents onto live websites with instructions to sign up for a product, integrate with it,… 18 r/LocalLLaMA community 8d ago Qwen 3.8 Low and Medium are goated Artificial Analysis just benchmarked them and the scores are crazy good, proving the earlier success wasn't only enabled by overthinking.   submitted by   /u/Eyelbee [link]   [comments] 37 r/LocalLLaMA community 8d ago Qwen3.8-27B on an RTX 5060 Ti 16GB: IQ4 vs Q8, 64K context, MTP, vision, and agent benchmarks I’ve been testing Qwen3.8-27B as a possible replacement for the Qwen3.5-9B that I have been running on RTX 5060Ti 16G. The goal was not just maximum tokens/sec, but useful context capacity, reliable tool calling, multi-turn behavior, and vision on a single 16GB GPU for true… 34 The Information — AI news-outlet 8d ago Dragoneer Founder Stad to Buy Timberwolves Controlling Stake Marc Stad, the founder of Dragoneer Investment Group, is buying a controlling stake in the Minnesota Timberwolves and Minnesota Lynx professional basketball teams, according to The New York Times’ Athletic publication. Stad is buying the interest at a $4.5 billion valuation from… 15 r/MachineLearning community 8d ago Safety critical systems (SCS) are the only real benchmark for ML systems. Thoughts? [D] What are real-world safety critical systems (SCS)? A flight controller for a commercial airplane carrying 300 passengers. A braking system for a bullet train that operates at 320km/hour. A reactor protection system for nuclear power plant that serves millions of people. A piece… 31 r/LocalLLaMA community 8d ago I benchmarked Ox Alpha on SWE-bench Verified Mini (50 tasks): 96% resolved. Now I’m skeptical of myself. TL;DR: The result looks TOO GOOD TO BE TRUE — that's exactly why I'm posting it. I ran Ox alpha on the entire SWE-bench Verified-Mini set with the official mini-swe-agent scaffold — the same agent used for the swebench.com Bash-Only leaderboard. Result: 48/50 = 96% resolved,… 11 r/LocalLLaMA community 8d ago NVIDIA AVO got 100% on ARC-AGI-3. It completed all 183 levels across all 25 public environments, figuring out what to do with no instructions, explicit rules, or stated goals.   submitted by   /u/theologi [link]   [comments] 35 NVIDIA Developer Blog official-blog 8d ago NVIDIA AVO Reaches 100% on ARC-AGI-3, Demonstrating a Frontier-Level General-Purpose Architecture for Long-Horizon Autonomous Agents A frontier language model is only one component of an AI agent. The surrounding agent system—often called a harness—determines how the model receives... 18 r/LocalLLaMA community 9d ago Fastest NVFP4 quant of Qwen3.8 27B out there Here's a brand new Blackwell-native, prefill-optimized 4-bit quant that runs 50% faster on compatible hardware than a Q4 quant of the same memory footprint. And it runs 4-7% faster than other NVFP4 quants as benchmarked on RTX 5090 32GB. Quant Benchmark Speed NVFP4 pp2048 6250… 27 Hugging Face Daily Papers research 9d ago NARU: A Benchmark for NARrative Evolution and Cultural Nuance Understanding in Japanese Extreme Long Video Abstract NARU is a Japanese long-form video benchmark evaluating narrative evolution and cultural reasoning through a hierarchical annotation pipeline and extensive native-speaker verification. Generated by thinkingmachines/Inkling-Small Long-form video understanding encompasses… 35 r/LocalLLaMA community 9d ago Qwen3.8-27B at 262K context on a Strix Halo + RTX 3090 Ti: 9.5 -> 153 tok/s, and it beats a dual-3090 vLLM box on HumanEval Spent a while treating layer placement, KV format and llama.cpp itself as experimental variables. 159 logged experiments. Numbers first, caveats after. Hardware: AMD Ryzen AI MAX+ 395 (Strix Halo, 128 GB unified) + RTX 3090 Ti on an eGPU link. One llama.cpp process, AMD on… 37 Hugging Face Daily Papers research 9d ago Towards Quantifying Benchmark Optimization in ASR Models Abstract High-performing speech recognition models reproduce benchmark transcripts despite contradictory audio, revealing benchmark-optimized behaviors that inflate scores without improving real-world transcription. Generated by thinkingmachines/Inkling-Small Public benchmarks… 13 Hugging Face Daily Papers research 9d ago SWE-bench Science: Can Coding Agents Resolve Engineering Tasks in Science? Abstract SWE-bench Science benchmarks coding agents on scientific software repair, revealing failure mechanisms and mixed effects of scientific guidance. Generated by thinkingmachines/Inkling-Small Software increasingly functions as part of the scientific instrument itself,… 17 arXiv — Machine Learning research 9d ago Holtercare-Bench: A Multimodal Benchmark for Evaluating Long-Term Dynamic ECG Analysis arXiv:2608.19297v1 Announce Type: new Abstract: While multimodal large language models (MLLMs) excel in medical applications, most of them favor static images or short-term signals. In the critical field of dynamic electrocardiograms (ECG), models struggle with complex temporal… 19 arXiv — Machine Learning research 9d ago Empirical Characterization of Learning Geometry in Hybrid Quantum Forecasting Models arXiv:2608.19497v1 Announce Type: new Abstract: We characterize the learning dynamics of a compact hybrid quantum forecasting model through comparison with a structurally aligned classical baseline. Using stationary harmonic-mixture and nonstationary chirp benchmarks with… 4 arXiv — Machine Learning research 9d ago DeltaML-Bench: Evaluating Machine Learning Agents on Real-World Research Repositories arXiv:2608.19653v1 Announce Type: new Abstract: Autonomous agents for machine learning experimentation must navigate heterogeneous repositories, repair training pipelines, and evaluate candidate improvements under realistic compute constraints. Existing benchmarks only partially… 6 arXiv — Machine Learning research 9d ago Answer-Level Trust Selection for Physical Vision-Language Reasoning arXiv:2608.19807v1 Announce Type: new Abstract: Vision-language models (VLMs) can estimate physical quantities such as duration, speed, and acceleration from visual observations, but existing benchmarks primarily assess overall model performance against annotated ground truth.… 4 arXiv — Machine Learning research 9d ago Fine-Tuning VLAs with Self-Demonstrated Generative Control for Multi-Task Manipulation arXiv:2608.19490v1 Announce Type: cross Abstract: State-of-the-art vision-language-action (VLA) models such as $\pi_{0.5}$ exhibit strong semantic understanding, instruction following and task behavior. However, when deployed on new robots, even minor mismatches in hardware… 5 arXiv — NLP / Computation & Language research 9d ago Compliance, Capability, and Conflict: Benchmarking Multimodal LLMs under System Messages arXiv:2608.19207v1 Announce Type: new Abstract: Production deployments of Multimodal Large Language Models (MLLMs) increasingly rely on system messages to govern model behavior. Yet existing benchmarks either evaluate constraints in text only or embed them into the user turn,… 30 arXiv — NLP / Computation & Language research 9d ago NepOOC-M: Bilingual Nepali-English Benchmark and Comparative Analysis of Multimodal Architectures for OOC Detection arXiv:2608.19212v1 Announce Type: new Abstract: Out-of-context (OOC) misinformation pairs authentic images with misleading captions to construct false narratives without image manipulation, making detection a problem of multimodal alignment rather than image forensics. Despite… 20 arXiv — NLP / Computation & Language research 9d ago Are LLMs becoming similarly creative? Evidence from three years of models arXiv:2608.19437v1 Announce Type: new Abstract: Many benchmarks track Large Language Model (LLM) performance on tasks with verifiable answers, but less is known about how LLM performance is evolving on open-ended tasks, where creativity, originality and diversity may matter as… 8 arXiv — NLP / Computation & Language research 9d ago Hear2Act: Benchmarking When Prosody Should Change What an Assistant Does arXiv:2608.19515v1 Announce Type: new Abstract: Prosodic cues can convey task-relevant information that alters the trajectory and outcome of a task-oriented dialogue, even when the words themselves remain unchanged. Yet existing benchmarks typically evaluate prosodic perception,… 36 arXiv — NLP / Computation & Language research 9d ago One Success Isn't Reliability: Thinkingbox, a Sandbox and Benchmark for Agents in Stateful Business Workflows arXiv:2608.19741v1 Announce Type: new Abstract: Recent agent benchmarks increasingly ground evaluation in executable environments, from code repair to web navigation, app APIs, and function calling. Yet completing consequential work beyond code requires more than producing a… 23 arXiv — NLP / Computation & Language research 9d ago PersonalBench: Measuring the Authorship Gap in LLM Personalization arXiv:2608.19746v1 Announce Type: new Abstract: Personalized text generation aims to make LLMs write in a specific individual's style, yet existing benchmarks measure task accuracy or preference alignment rather than whether the model's output actually resembles the target… 19 arXiv — NLP / Computation & Language research 9d ago SWE-bench Science: Can Coding Agents Resolve Engineering Tasks in Science? arXiv:2608.19799v1 Announce Type: new Abstract: Software increasingly functions as part of the scientific instrument itself, making failures in scientific code capable of compromising not only program behavior but also the evidence underlying scientific conclusions. Yet existing… 11 arXiv — NLP / Computation & Language research 9d ago Natural Language Code Retrieval for 1C:Enterprise: An Open Benchmark and Efficient Bi-Encoder arXiv:2608.19957v1 Announce Type: new Abstract: Natural language code retrieval is a rapidly evolving task in computer science. However, the 1C:Enterprise ecosystem combines Russian syntax with highly domain-specific terminology, for which open datasets and specialized models… 12 arXiv — NLP / Computation & Language research 9d ago HealMed: Multilingual Evaluation of Large Language Models in Medicine arXiv:2608.19981v1 Announce Type: new Abstract: We present HealMed, an expert-reviewed benchmark for multilingual evaluation of large language models in medicine. HealMed contains 1,000 examples in each of nine languages, drawn from nine datasets and covering three task formats:… 27 arXiv — NLP / Computation & Language research 9d ago OenoBench: A Wine-Domain Benchmark for Knowledge-Grounded Evaluation of Large Language Models arXiv:2608.20106v1 Announce Type: new Abstract: We introduce OenoBench, a wine-domain knowledge benchmark of 3,266 multiple-choice questions across six pillars (regions, grape varieties, viticulture, winemaking, producers, business) and four difficulty tiers. The corpus is built… 12 arXiv — NLP / Computation & Language research 9d ago FormalTCS: Benchmarking End-to-End Frontier Formal Theoretical Computer Science Research of Large Language Models arXiv:2608.20153v1 Announce Type: new Abstract: Large language models (LLMs) have shown growing potential for automated theoretical computer science (TCS) research, yet existing benchmarks remain far from realistic research settings. We introduce \ourbenchmark, an… 38 Page 3 of 10 · 500 articles ← Newer Older →
