News / #benchmark Tag Benchmark 500 articles archived under #benchmark · RSS Sign in to follow arXiv — Machine Learning research 14d ago CADENCE: A Confidence-Adaptive Dual-Expert Network for Fast and Accurate Time Series Classification arXiv:2609.35929v1 Announce Type: new Abstract: Time series classification (TSC) exhibits a sharp trade-off between accuracy and computational scalability. Meta-ensembles like HIVE-COTE 2.0 reach state-of-the-art accuracy but require extensive compute, whereas ultra-fast random… 24 arXiv — Machine Learning research 14d ago ThinQuant: Scalable Rotation Learning for Weight and Activation Quantization of LLMs arXiv:2609.36120v1 Announce Type: new Abstract: Learned rotations play an important role in enabling low-bit weight and activation quantization of large language models by smoothing outliers in the activation distribution. State-of-the-art approaches include gradient-based… 6 arXiv — NLP / Computation & Language research 14d ago Constructing Challenging Browser-Use Tasks by Controlled Environment Interventions arXiv:2609.35814v1 Announce Type: new Abstract: As browser-use agents improve, benchmarks keep pace by collecting new tasks, websites, and applications, often making tasks longer or more novel. This makes difficulty expensive to refresh and difficult to control: when many… 17 arXiv — NLP / Computation & Language research 14d ago $\tau$-Multilingual: Benchmarking Voice Agents Across Languages arXiv:2609.35820v1 Announce Type: new Abstract: English-only benchmarks expose only a narrow slice of voice-agent behavior. We introduce $\tau$-Multilingual, extending $\tau$-Voice to Spanish, Brazilian Portuguese, Hindi, Korean, and Mandarin with native-speaker review and… 26 arXiv — NLP / Computation & Language research 14d ago CruxBench: A Benchmark of Information Discovery arXiv:2609.35879v1 Announce Type: new Abstract: Benchmarks for large language models (LLMs) typically evaluate the accuracy of answers against fixed reference labels. But a central step in many complex real-world tasks is identifying which questions are worth asking in the first… 27 arXiv — NLP / Computation & Language research 14d ago Benchmarking Automatic Speech Recognition Tools for Iberian Languages arXiv:2609.36920v1 Announce Type: new Abstract: Comprehensive evaluations of automatic speech recognition (ASR) for Iberian languages remain limited, and low-resource languages, biases, and efficiency trade-offs are underexplored. We benchmark eleven systems, ten open-weight… 38 arXiv — NLP / Computation & Language research 14d ago CypherTurn: A Multi-Turn Benchmark for Conversational Text-to-Cypher Evaluation and the Autonomy Divergence arXiv:2609.36987v1 Announce Type: new Abstract: Graph databases are increasingly queried through natural language, yet every existing benchmark evaluates isolated single-turn queries rather than the multi-turn sessions through which analysts actually work. We introduce… 25 Hugging Face official-blog 14d ago Open TTS Leaderboard: Scalable Evaluation for Multilingual Text-to-Speech and Voice Cloning Back to Articles a]:hidden"> Open TTS Leaderboard: Scalable Evaluation for Multilingual Text-to-Speech and Voice Cloning Published September 30, 2026 Update on GitHub Upvote - Eric Bezzam bezzam Steven Zheng Steveeeeeeen Eustache Le Bihan eustlb mrfakename mrfakename guest… 4 Simon Willison community 15d ago Quoting Anthropic Frontier Red Team We evaluate several models on 100 tasks from the [internal Binary Exploitation benchmark] (selected at random), and find that GLM-5.3 develops full control flow hijacks in 4% of the trials; Claude Mythos Preview did so in 6%. Although GLM-5.3 performs below Claude Mythos Preview… 15 arXiv — Machine Learning research 15d ago Auditing System-1 Models on Biosecurity-Relevant Benchmarks: Calibration, Selective Prediction, and Permutation Instability in a Non-Generative Model arXiv:2609.30454v1 Announce Type: new Abstract: Non-generative "System-1" models return structured probabilistic decisions in a single forward pass, without autoregressive decoding, at a small fraction of the inference cost of a generative model. This makes them of interest as… 14 arXiv — Machine Learning research 15d ago Benchmarking the Connectomes of Caenorhabditis elegans within the Reservoir Computing Framework arXiv:2609.30508v1 Announce Type: new Abstract: The aim of this work is to examine the connectomes of Caenorhabditis elegans through a computational lens using the reservoir computing framework. Connectomes are mappings of biological neural networks; C. elegans is the first… 8 arXiv — Machine Learning research 15d ago SAGE: A sampling-aware global evaluation benchmark for species distribution modeling arXiv:2609.31082v1 Announce Type: new Abstract: Knowing where species occur is fundamental for biodiversity research and conservation. Species distribution models (SDMs) link species observations to environmental conditions to estimate their spatial distribution. However,… 30 Simon Willison community 16d ago Claude Sonnet 5.5 Claude Sonnet 5.5 New Sonnet model from Anthropic today. They say it "runs 30%+ faster, and costs up to 30% less for most work" - it's priced the same as Sonnet 5 but appears to beat it on every benchmark, and should be cheaper to run as well. Here are some pelicans riding… 28 r/LocalLLaMA community 16d ago Swift 1.5 + HyperQwen = 37% less task completion time at 100+ tps w/ 150k context on RTX 3090 Hi everyone :) The amazing Swift finetunes of Qwen3.8 27B generate much fewer tokens at mostly similar benchmark performance to the original model, while repos like HyperQwen (formerly syv-ai/qwen38-27b-rtx3090) deliver insane TPS on an RTX 3090. To get the best of both worlds,… 15 r/LocalLLaMA community 16d ago Trained locally: ultra-fast 0.8B/2B System 1 decision models that match Jev on benchmarks and Doom, ~30 ms per decision (open weights) TL;DR: The Jeff models are a set of Qwen3.5 and Gemma fine-tunes for zero-shot classification: small, efficient, open-weight models with respectable out-of-the-box performance that can be slotted right into code or fine-tuned/LoRA-trained as needed. Give them a situation and a… 12 r/MachineLearning community 16d ago Qwen3-VL 8B on a laptop vs Opus 5.5 / Sonnet 5 / GPT-5.6 on 137 messy documents: beat GPT-5.6 on tax forms, lost badly on Indian date formats[R] I benchmarked Qwen3-VL 8B Instruct (Q4_K_M, Ollama, M5 24GB, ~30s/doc) against Claude Opus 5.5, Sonnet 5 and GPT-5.6 Terra on: - receipts: CORD (Indonesia) and SROIE (Malaysia), 30 each - 20 scanned 1980s-90s invoices , answer keys human-verified - 32 real IRS forms, 4 damage… 18 r/LocalLLaMA community 16d ago If you are running Qwen 3.8 Flash Next on Strix Halo, use this software for inference. It's so much faster than llama.cpp especially at high context. Here's the project. I have nothing to do with it. I'm just an amazed user. https://github.com/gufo-org/gufo/blob/main/docs/models/qwen3.8-flash-next/BENCHMARKS.md Those benchmark numbers hold up on real work loads. Here are some numbers I got during a chat. "[6204 chunks in… 33 arXiv — NLP / Computation & Language research 16d ago A Mechanistic Study of AI-Text Detection Neurons in Frozen BERT: Sparse Probing and Activation Patching on RAID arXiv:2609.30287v1 Announce Type: new Abstract: AI-generated text detectors achieve high accuracy on standard benchmarks, yet the internal representations that drive these predictions remain poorly understood. We study which neurons in a frozen BERT-base-uncased encoder support… 27 arXiv — NLP / Computation & Language research 16d ago A Benchmark Framework for Screening Automation in Systematic Reviews arXiv:2609.30298v1 Announce Type: new Abstract: Systematic reviews (SR) are essential for evidence-based research, but their screening phase is highly time-consuming and labor-intensive. Large language models (LLMs) offer a promising opportunity to reduce this workload by… 20 arXiv — NLP / Computation & Language research 16d ago The Hard Part Comes After Search: Benchmarking Web Agents on Synthesizing, Organizing, and Displaying Knowledge arXiv:2609.30604v1 Announce Type: new Abstract: Existing computer-use agent benchmarks do not fully evaluate agents acting as assistants. A useful assistant retrieves information across complex, multi-step workflows, synthesizes it into artifacts (documents, presentations,… 31 arXiv — NLP / Computation & Language research 16d ago From annotation to reasoning: Culture in language models arXiv:2609.30897v1 Announce Type: new Abstract: How should we evaluate language models when more than one interpretation can be right? Cultural benchmarks often test factual knowledge, agreement with survey responses, or recognition of a predefined meaning. These tasks leave… 30 arXiv — NLP / Computation & Language research 16d ago AcoustiClaim: A Numeric Claim Benchmark with Instrument Ground Truth arXiv:2609.30483v1 Announce Type: cross Abstract: Audio language models state numbers for acoustic quantities, and neither human opinion nor a judge model says whether such a number is true of the signal. AcoustiClaim extracts each numeric claim from free text, scores it against… 4 arXiv — NLP / Computation & Language research 16d ago JevAdvBench: A Benchmark and Black-Box Attacks for Reinforcement Learning for Calibrated Decisions Models arXiv:2609.31142v1 Announce Type: cross Abstract: Models trained with reinforcement learning for calibrated decisions (RLCD), such as Jev, answer a typed question about an input, the state, with a probability, a choice, or a score, and software acts on the answer without a… 33 arXiv — NLP / Computation & Language research 16d ago PriceBench: A Diagnostic Benchmark for Price, Quality, and Brand Preferences in LLM Booking Agents arXiv:2609.31468v1 Announce Type: cross Abstract: LLMs increasingly act as purchasing agents, which makes the LLM, not the user, the one choosing among the options that satisfy a request; its preferences quietly fix what gets bought and what it costs. Hotel booking is a clean… 19 arXiv — NLP / Computation & Language research 16d ago Compact Documentation for Coding Agents: A Benchmark, an Optimizer, and Why It Does Not Transfer arXiv:2609.31587v1 Announce Type: cross Abstract: We investigate whether natural-language documentation helps coding agents resolve software issues, and we build the tools to construct and evaluate it. We introduce a roundtrip benchmark that scores code descriptions by whether… 14 r/MachineLearning community 16d ago Reduced my Jev judge’s calibration error [D] I benchmarked Jev on TRIVIA+ dataset using an untouched 645-example test set. Before calibration: ECE: 0.0982 After learning from human-labelled examples: ECE: 0.0313 That’s a 68.1% reduction in calibration error . But hallucination-detection F1 only moved: 0.5833 → 0.5877 So… 35 r/LocalLLaMA community 16d ago It would be really cool to have an official 3D-print engineering benchmark/leaderboard like this Someone prompted different LLMs to generate CAD code for a bridge under fixed constraints (2-foot span, under 500g filament, 18-hour print limit), printed them, and load-tested them to failure. The results were quite varied: some models couldn't even design parts that fit… 32 r/LocalLLaMA community 17d ago We released VeriLoop E2 (27B, Apache-2.0). The design question behind it: should an LLM be allowed to commit its own state? Disclosure: I’m one of the authors of VeriLoop E2, a 27B model post-trained from Qwen3.8-27B. The weights are Apache-2.0; the harness we evaluate it in is not open source (details at the bottom). This is a release post, but rather than a benchmark dump I want to talk about the… 28 r/LocalLLaMA community 17d ago Nonobench v1.2: 43 LLMs on nonogram puzzles. Open-weight DeepSeek V4 Pro ties for 4th, and no open model solves the new 20×20 Hard mode Follow-up to my January post: https://www.reddit.com/r/LocalLLaMA/comments/1q4i19c/benchmarking_23_llms_on_nonogram_logic_puzzle/ . That thread shaped v1.2: Reasoning effort is explicit per run Every prompt and output is public. All current top ranking private and open weight… 7 r/MachineLearning community 18d ago Publication potential [D] Hi all, I may be being silly but I have just finished my masters thesis which benchmarked three deep learning architectures (one of which is novel), across varying preprocessing pipelines for the purpose of EEG motor imagery task classification SPECIFICALLY on a consumer grade… 17 r/LocalLLaMA community 18d ago For the longest time I’ve felt this sub should have a pinned section where a detailed post about each model should get featured. For instance whenever a model comes out, what’s the best engine to run it, the best harness and absolute minimum you need to get same or near same re results that the benchmark of that model claims. And whenever a quant from Unsloth guys comes out the guide can either be updated… 30 r/LocalLLaMA community 18d ago Qwen3.8-27B IQ3_XXS vs Qwen3.6-35B-A3B Q4_K_M Which one is better for difficult tasks like web scrapping, coding, using tools? Looking for any benchmarks because i couldn't actually find one after quite some digging   submitted by   /u/Loose_Doubt367 [link]   [comments] 30 r/LocalLLaMA community 18d ago My Reading Library: Evaluating LLMs on Android Tasks Can LLM agents actually get through a day in the life of a normal user? That question got me reading papers on Android agents and mobile benchmarks over the past few months. A few patterns kept showing up: Most benchmarks run on emulators, making real-device metrics difficult to… 9 r/LocalLLaMA community 19d ago Has anyone benchmarked AI agents against the SOLIDWORKS CSWA exam? Would be interesting right? Models are starting to score higher and higher on benchmarks like Parametric CAD Bench , but can they pass an actual exam? The Certified SOLIDWORKS Associate (CSWA) exam might be an interesting place to start. They have an sample exam on their… 6 r/LocalLLaMA community 19d ago Did anyone do a full bench of e.g. Qwen Flash Next IQ4 and Qwen 27b FP8? Here are some I let Codex do some eval on Qwen 3.8 Flash Next IQ4_XS (served via vllm and r9v) and Qwen 3.8 FP8 (served via vllm and radiance). Here are the results: Benchmark Flash-Next IQ4_XS Qwen3.8 27B FP8 Result MMLU-Pro 83.8% 75.0% Flash-Next GPQA Diamond 42.5% 27.5% Flash-Next GSM8K… 8 arXiv — Machine Learning research 19d ago fable.intermittent: benchmarking probabilistic forecasting methods for intermittent time series arXiv:2609.28607v1 Announce Type: new Abstract: Intermittent time series are common in spare-parts demand and retail sales. Since the cost of forecast errors is typically asymmetric, decisions such as inventory control require the full predictive distribution rather than a point… 17 arXiv — Machine Learning research 19d ago Language Specificity vs. Domain Diversity: Benchmarking Transformers for Bangla Medical NER arXiv:2609.29101v1 Announce Type: new Abstract: Medical Named Entity Recognition (NER) for low-resource languages remains a challenging task due to high linguistic variability and a scarcity of domain-specific annotated corpora. This work presents a comprehensive empirical… 31 arXiv — Machine Learning research 19d ago When Identical Rows Disagree: From Benchmark Identifiability to Replication-Robust Anomaly Detection arXiv:2609.29580v1 Announce Type: new Abstract: A released table is often treated as an i.i.d. sample, although its repeated rows may encode business frequency, repeated entities, joins, resampling, or extraction errors. We show that this ambiguity creates a hidden measurement… 35 arXiv — Machine Learning research 19d ago Limited Structural Reliability in Public Educational Prediction Benchmarks: A Four-Dimension Audit of Seven Datasets arXiv:2609.29625v1 Announce Type: new Abstract: Across seven public educational prediction datasets, three passed all four pre-modeling reliability checks; the remaining four either failed group-aware generalization tests or lacked the provenance metadata needed to run them. One… 28 arXiv — Machine Learning research 19d ago TopU-LBVS: A Realistic Multi Target Benchmark for Ligand Based Virtual Screening arXiv:2609.29740v1 Announce Type: new Abstract: Ligand-based virtual screening (LBVS) is a practical first-pass tool in early-stage drug discovery, but existing benchmarks can overestimate performance through random negatives, easy decoys, limited target coverage, and… 25 arXiv — NLP / Computation & Language research 19d ago Benchmarking Argumentative Behaviour of LLMs: A Study of Defences Against Character Attacks arXiv:2609.28673v1 Announce Type: new Abstract: Large Language Models (LLMs) are increasingly deployed as argumentative agents in persuasive dialogues, necessitating rigorous evaluation of their debating competence relative to human interlocutors. In this study, we focus on… 23 arXiv — NLP / Computation & Language research 19d ago COILD: An Indic-Centric Parallel Corpus and Benchmark for Machine Translation Across Indian Languages arXiv:2609.28826v1 Announce Type: new Abstract: Machine translation (MT) for Indian languages remains constrained by the limited availability of high-quality, Indic-centric parallel corpora and evaluation benchmarks. Existing multilingual resources are largely constructed from… 24 arXiv — NLP / Computation & Language research 19d ago Polite but Misaligned: Evaluating LLM Politeness Judgments Against Human Pragmatic Norms arXiv:2609.29001v1 Announce Type: new Abstract: Despite strong performance on standard benchmarks, it remains unclear whether large language models (LLMs) evaluate social pragmatics in ways that align with human judgments. We evaluate LLM politeness judgments using two… 8 arXiv — NLP / Computation & Language research 19d ago BanglaTurn: A Benchmark and Whisper-Based Model for End-of-Turn Detection in Bangla Speech arXiv:2609.29371v1 Announce Type: new Abstract: This paper presents BanglaTurn, a corpus for end-of-turn detection in Bangla conversational speech, and a model trained on it. The corpus holds 35,374 samples of 3 to 15 s of podcast speech, labelled for turn state by combining… 8 arXiv — NLP / Computation & Language research 19d ago Two Emojis of Difference: What Multilingual Affective Generation Benchmarks Actually Measure arXiv:2609.29445v1 Announce Type: new Abstract: We audit a multilingual affective generation benchmark eight instruction-tuned LLMs producing emoji summaries for 17,100 Bangla, English and Hindi sentences, with 6,960 human judgements and find its headline conclusions to be… 9 arXiv — NLP / Computation & Language research 19d ago Clinical Intent Extraction: A FHIR-Aligned Representation and the CIRCA Benchmark arXiv:2609.29479v1 Announce Type: new Abstract: Prospective clinical actions, the follow-ups, orders, referrals, and instructions that deter-mine what happens to a patient next, are annotated today in thin fragments across incom-patible corpora: each records a text span and one… 11 arXiv — NLP / Computation & Language research 19d ago PROOF: Profiling Reliability of Object-Level Facts in Large Language Models arXiv:2609.29504v1 Announce Type: new Abstract: Aggregate factuality scores hide where a language model succeeds, which relations it confuses, and whether an answer survives innocuous changes to the question or decoder. We introduce PROOF, a profile-oriented benchmark for… 14 arXiv — NLP / Computation & Language research 19d ago EnSiTa - A Trilingual Multi-Domain Parallel Dataset and Benchmark for Domain-Specific Machine Translation arXiv:2609.29511v1 Announce Type: new Abstract: Machine Translation (MT) for low-resource languages remains far behind that of high-resource languages, and the gap is widest in specialised domains, where parallel data is scarce or entirely absent. We present EnSiTa, a trilingual… 27 arXiv — NLP / Computation & Language research 19d ago Benchmarking Arabic--Russian Machine Translation: A Comparison of Fine-tuned NMT and Few-shot LLMs under Rich Morphology and Low Lexical Overlap arXiv:2609.29559v1 Announce Type: new Abstract: Arabic-Russian machine translation (MT) remains under-explored due to the rich morphology of Arabic and low lexical overlap between the two languages. We benchmark seven fine-tuned neural machine translation (NMT) models against… 11 arXiv — NLP / Computation & Language research 19d ago ModularSQL: A Runtime Guardrail for the Multiplicity Blind Spot in Text-to-SQL arXiv:2609.29573v1 Announce Type: new Abstract: Text-to-SQL systems are increasingly deployed on production databases, where queries that pass benchmark evaluation can still produce results that distort downstream workflows. Standard set-based execution accuracy (Set-EX)… 36 Page 4 of 10 · 500 articles ← Newer Older →
