News / #benchmark Tag Benchmark 500 articles archived under #benchmark · RSS Sign in to follow arXiv — NLP / Computation & Language research 5d ago UXBench Pro: Benchmarking Personalized User Experience in Multi-Turn Dialogue Interactions arXiv:2610.11638v1 Announce Type: new Abstract: Evaluating user experience (UX) with automated computational methods has gained increasing attention, supported by empirical evidence from UXBench. However, binary preference prediction provides limited insight, while relying on a… 8 arXiv — NLP / Computation & Language research 5d ago TRACE: Diagnosing Verifier Brittleness in Agentic Evaluation arXiv:2610.11678v1 Announce Type: new Abstract: Verifier scores now serve as both benchmark metrics and training rewards for large language model (LLM) agents, and a change in score is routinely read as a change in capability. It may instead reflect a change in the evaluation.… 29 arXiv — NLP / Computation & Language research 5d ago Specialized Decision Models vs. General-Purpose LLMs: Benchmarking Jev Across Knowledge, Reasoning, and Multilingual Tasks arXiv:2610.11978v1 Announce Type: new Abstract: Jev is a "System One" model that returns a choice among given options instead of generating text. We study how such a specialized decision model compares with general-purpose large language models (LLMs). We evaluate Jev on 13… 20 arXiv — NLP / Computation & Language research 5d ago StoreBench: A Live-Commerce Environment for Evaluating and Training Autonomous Operator Agents arXiv:2610.10942v1 Announce Type: cross Abstract: Reinforcement learning environments are now a primary lever for improving large language model (LLM) capabilities in post-training, yet most agentic benchmarks remain static: the world moves only when the agent acts, the reward… 19 arXiv — NLP / Computation & Language research 5d ago GameCommBench: A Unified Benchmark and Type-Aware Evaluation for AI-Generated Game Commentary arXiv:2610.11129v1 Announce Type: cross Abstract: Game commentary is an open-ended generation task requiring multimodal perception, strategic reasoning, and contextual knowledge. Existing AI-Generated Game Commentary (AI-GGC) studies remain fragmented across games, modalities,… 29 r/MachineLearning community 6d ago Whatever happened to BABA is AI from 2024? [D] We test three state-of-the-art multi-modal large language models (GPT-4o, Gemini-1.5-Pro, Gemini-1.5-Flash) and find that they fail dramatically when generalization requires that the rules of the game must be manipulated and combined. This is the slap to the face in bold. The… 10 TechCrunch — AI news-outlet 6d ago Popular AI leaderboard Arena nearly doubles valuation to $3.1B valuation in 10 months The company behind the popular LMArena leaderboard has raised $200 million led by Lightspeed and Khosla, and is now measuring AI models on alignment issues such as lying. 35 r/MachineLearning community 6d ago Best practices when running a benchmark on online models [D] I'm developing a benchmark for a low resource language and I don't want it to be leaked and used for training when it is being used to get predictions. For locally run models it shouldn't be a problem, but for models that are only accessible via API, it is. Is there an… 11 r/LocalLLaMA community 6d ago We unlearned CCP alignment from Qwen3.6-35B-A3B: censored/propaganda answers 89.8% → 2.8%, general benchmarks within ~1 point (open weights) Disclosure: I'm a researcher at Hirundo, the company that made this. Happy to answer anything. Qwen ships with the CCP's political alignment trained in. Ask Qwen3.6 what happened on June 4, 1989 and it says "I don't know what you are referring to." A system prompt doesn't… 10 arXiv — NLP / Computation & Language research 6d ago FinVector-Market-4B: A Controlled Study of LoRA Adaptation for Structured Financial Tasks arXiv:2610.08882v1 Announce Type: cross Abstract: FinVector-Market-4B adapts Qwen/Qwen3.5-4B with rank-16 LoRA on a 22,000-example corpus for structured financial tasks. We evaluate the base and adapted models on the same 600-example benchmark under implicit and explicit… 8 arXiv — Machine Learning research 6d ago The Best Optimizer Depends on Batch Size arXiv:2610.08975v1 Announce Type: new Abstract: A plethora of new adaptive optimizers are designed to efficiently estimate and use minibatch gradient statistics to shape parameter updates, but they are typically benchmarked at a single batch size. Hyperparameter scaling rules… 34 arXiv — Machine Learning research 6d ago Are We Really Benchmarking Forecasting Models? The Impact of Preprocessing on Time Series Performance arXiv:2610.09096v1 Announce Type: new Abstract: While established literature underscores the pivotal role of preprocessing in forecasting accuracy, this stage remains largely overlooked in current research. Modern benchmarks typically resort to simple scaling, failing to account… 37 arXiv — Machine Learning research 6d ago Kuration SDK: Addressing the Virtual2Real Gap via Data Curation arXiv:2610.09305v1 Announce Type: new Abstract: Benchmarks for measuring the quality of action-conditioned world models are still evolving and shifting away from visual similarity-based metrics to action-semantic and physically-grounded metrics. However, for domain and… 13 arXiv — Machine Learning research 6d ago A Framework for the Systematic Review of ML Assets in AI Registries arXiv:2610.09551v1 Announce Type: new Abstract: Background: Modern software systems increasingly rely on Machine Learning (ML) assets (i.e., pre-trained models, datasets, benchmarks) for building, evaluating, and integrating ML-based systems. However, current exploration,… 14 arXiv — NLP / Computation & Language research 6d ago sk-bench: A Native-First Benchmark for Evaluating Large Language Models in Slovak arXiv:2610.09152v1 Announce Type: new Abstract: Multilingual LLM benchmarks omit Slovak, a morphologically rich West Slavic language of five million speakers, or cover it only by machine translation. We present sk-bench, a native-first Slovak benchmark with 30 datasets (33… 18 arXiv — NLP / Computation & Language research 6d ago ToolRACER: A Robust Agentic Conversation Emulation Resource for Agent Training and Evaluation arXiv:2610.09163v1 Announce Type: new Abstract: Task-oriented conversational agents remain fragile under real world conversation scenarios as they rarely follow a predictable script, especially when users exhibit non-cooperative behavior. Existing function-calling benchmarks… 12 arXiv — NLP / Computation & Language research 6d ago Arctic Questions, Missing Answers: A Dataset and Benchmark for LLM Abstention in Arctic Science arXiv:2610.09446v1 Announce Type: new Abstract: Large language models (LLMs) should abstain from scientific multiple-choice questions when no option is valid, but frequent abstention alone does not demonstrate sensitivity to answer availability. We introduce ArcticQA, a dataset… 23 arXiv — NLP / Computation & Language research 6d ago BanglaRhet: Benchmarking Classical and Transformer Models for Rhetorical and Persuasion Detection in Bangla Political Speech arXiv:2610.09464v1 Announce Type: new Abstract: Political discourse often uses rhetorical and persuasive language to frame narratives, influence public opinion, and mobilize audiences. While Bangla natural language processing has made progress in sentiment analysis and opinion… 4 arXiv — NLP / Computation & Language research 6d ago How Do LLMs Change Predictions Under Negation? arXiv:2610.09571v1 Announce Type: new Abstract: Negation is an essential feature of human language, yet large language models (LLMs) remain unreliable in processing it. We evaluate recent open-source and closed-source LLMs on our negation benchmark and find that, in 37-71% of… 34 arXiv — NLP / Computation & Language research 6d ago Alice: A Large-Scale German Benchmark for Rubric-Based Multi-Dimensional Automatic Short Answer Scoring arXiv:2610.09661v1 Announce Type: new Abstract: Automatic Short Answer Scoring (ASAS) is central to NLP for Education. However, openly available benchmarks remain scarce, and existing datasets largely address how well students answer a question directly rather than how well they… 12 arXiv — NLP / Computation & Language research 6d ago InsClaimBench: Benchmarking Insurance Claim Adjudication Across the Decision Chain arXiv:2610.09671v1 Announce Type: new Abstract: Recent advances in reasoning-oriented large language models (LLMs) have motivated increasing evaluation of their ability to perform professional decision tasks. Insurance claim adjudication is one such task, requiring models to… 14 arXiv — NLP / Computation & Language research 6d ago Bookkeeping, Composition, or Unreachable Gold? Reading MemoryAgentBench's Conflict-Resolution Scores Against a Frozen Last-Write Resolver arXiv:2610.09193v1 Announce Type: cross Abstract: MemoryAgentBench's Conflict Resolution split is read as measuring "selective forgetting". We execute the benchmark's own rule - the newest statement about a fact wins - as a zero-learning resolver frozen on one of the four fact… 20 arXiv — NLP / Computation & Language research 6d ago Coding-Agent Benchmarks Should Match Their Users' Task Flows arXiv:2610.09633v1 Announce Type: cross Abstract: The evaluation of coding agents generally strives to be as realistic as possible. In our study, we collect 4,782 agent sessions of real software engineers in JetBrains IDEs, which we call Production Sessions. Since our subject is… 6 r/MachineLearning community 7d ago Looking for developer-friendly inference providers who give you enough API credits to experiment [D] I’m hitting rate limits on Together AI. For context, I’ve been working on an agentic repository indexing and benchmark generation tool, and I’m running multiple agents in parallel across models like Llama 3.3 70B and Qwen 2.5. When I first started working on this, Together AI… 33 r/LocalLLaMA community 7d ago Single 3090 Qwen 27B user, considering buying 128GB of RAM because of the hype My current setup: - single 3090 running turboderp/Qwen3.8-27B-exl3:SC_5.00bpw_H6_V6 - ~150k context, ~70t/s, unknown prefill because I didn't benchmark it (but it is ok) - Intel 12400 CPU with 32GB DDR4 RAM All the hype around strata makes me consider buying 128GB of 6000MHz… 26 arXiv — Machine Learning research 7d ago Neutrosophic Ensemble Classification for Uncertainty-Aware Bearing Fault Detection: Evidence from Laboratory and Variable-Speed Industrial Benchmarks arXiv:2610.06880v1 Announce Type: new Abstract: Machine learning classifiers for bearing fault detection produce scalar confidence scores that conflate confident errors with genuinely ambiguous predictions, and the conventional truth/falsity pair (F = 1 - T) is algebraically… 33 arXiv — Machine Learning research 7d ago Uncertainty in Representation Learning on Knowledge Graphs arXiv:2610.06974v1 Announce Type: new Abstract: Knowledge graph embedding (KGE) methods represent entities and predicates in continuous vector spaces to infer missing knowledge. Despite strong benchmark performance, their predictions often lack principled reliability guarantees,… 9 arXiv — Machine Learning research 7d ago An Information-Theoretic Evaluation Framework for Benchmark and Model Diagnosis in Knowledge Tracing arXiv:2610.06988v1 Announce Type: new Abstract: Knowledge tracing (KT) models are predominantly evaluated using aggregate metrics such as area under the curve (AUC) and accuracy. However, these global scores obscure where the remaining errors originate and fail to indicate… 17 arXiv — NLP / Computation & Language research 7d ago CLM-as-a-Judge: Evaluating an Open Contrastive Decision Model on Public Judge Benchmarks arXiv:2610.07177v1 Announce Type: cross Abstract: An open contrastive decision model is near chance as a judge on the hard public benchmarks: Contrastive-LM/CLM-v0.1-8B scores between 0.351 (best- of-four, chance 0.250) and 0.593 (pairwise, chance 0.500), is statistically… 5 arXiv — Machine Learning research 7d ago Benchmarking Time Series Foundation Models for Load Forecasting Under Covariate Uncertainty arXiv:2610.07232v1 Announce Type: new Abstract: Accurate short-term load forecasting (STLF) is essential for the reliable and efficient operation of modern power systems. While time series foundation models (TSFMs) have recently demonstrated remarkable performance across a wide… 21 arXiv — Machine Learning research 7d ago Towards Explainable Benchmarking for Data-driven Post-Wildfire Debris Flow Prediction arXiv:2610.07358v1 Announce Type: new Abstract: Post-wildfire debris flows (PFDFs) are destructive sediment-laden hazards triggered when intense rainfall strikes recently burned terrain, destabilizing hillslopes and threatening infrastructure, local economies, and community… 26 arXiv — Machine Learning research 7d ago Multigroup Fairness and Omniprediction: Separations and Equivalences arXiv:2610.07374v1 Announce Type: new Abstract: Omniprediction is a learning guarantee which requires a single predictor to be competitive relative to the best hypothesis from a benchmark class for any loss chosen from a family of loss functions. Loss Outcome… 27 arXiv — Machine Learning research 7d ago Benchmarking Label-Revealed Online Updates for EEG BCI Decoding arXiv:2610.07420v1 Announce Type: new Abstract: Electroencephalography (EEG) signals drift over time, which can cause static brain-computer interface (BCI) models to degrade in practice. We present a benchmark for online adaptation and compare two widely used pipeline families,… 4 arXiv — NLP / Computation & Language research 7d ago Investigating Model Compression for Neural Machine Translation in the Biomedical Domain arXiv:2610.07032v1 Announce Type: new Abstract: Large-scale pretrained transformer models have achieved state-of-the-art performance across diverse machine translation tasks, including multilingual settings. Knowledge distillation has emerged as a sustainable approach for model… 22 arXiv — NLP / Computation & Language research 7d ago CroissantMiner: Automated Extraction and Validation of Croissant Metadata for ML Datasets arXiv:2610.07132v1 Announce Type: new Abstract: Croissant has emerged as a standard for machine-readable dataset metadata, yet populating its fields remains labor-intensive and requires careful reading of accompanying dataset documentation. We present the first benchmark… 17 arXiv — NLP / Computation & Language research 7d ago The Labeling Problem in Hallucination Detection Benchmarks: An Empirical Evaluation arXiv:2610.08026v1 Announce Type: new Abstract: In recent years, several methods for detecting when large language models (LLMs) hallucinate have been developed. These methods are often benchmarked with open-domain question answering (QA) datasets containing questions and… 29 arXiv — NLP / Computation & Language research 7d ago Language Unalignability: Why Some Concepts Resist Cross-Cultural Benchmark Evaluation arXiv:2610.08303v1 Announce Type: new Abstract: Current evaluation of multilingual Large Language Models (LLMs) rests on an implicit Translation-Isomorphism Assumption (TIA): that semantic structures across languages are congruent and mutually mappable without loss of… 13 arXiv — NLP / Computation & Language research 7d ago Wiki-Talkie: Multilingual Benchmarking of Persona-Based Agents on Real-World Discussions arXiv:2610.08513v1 Announce Type: new Abstract: LLMs are increasingly deployed as autonomous agents in social environments, making it critical to study their ability to faithfully simulate human interactions. Central to this is grounding agents in realistic user personas, yet… 32 arXiv — NLP / Computation & Language research 7d ago GIVE-KWS: Gated Injection of Visual Evidence for Noise-Robust Query-by-Example Keyword Spotting arXiv:2610.07046v1 Announce Type: cross Abstract: Visual speech promises noise-robust keyword spotting, yet a visual stream is not necessarily used. On a tri-modal query-by-example keyword spotting (QbyE-KWS) benchmark, we find that a system with a task-trained visual encoder… 23 arXiv — NLP / Computation & Language research 7d ago Logbook: Extremely Long-form Audio Event Understanding arXiv:2610.07338v1 Announce Type: cross Abstract: Audio benchmarks are built around short, pre-segmented clips, limiting model design to brief inputs or fixed vocabularies. To close this gap, we introduce Logbook, a benchmark for hour-scale audio understanding, with recordings… 16 r/LocalLLaMA community 7d ago BenchLabs' Cagliostro V3.5 135M finally takes Hugging Face's SmolLM2-135M's first place on Open SLM Leaderboard with an Intelligence Index of 27.49   submitted by   /u/Megneous [link]   [comments] 9 r/LocalLLaMA community 7d ago Why is ik_llama.cpp said to be faster than Mainline? On my hybrid multi-GPU rig, Mainline easily beats it I constantly see recommendations saying that ik_llama.cpp (ikawrakow's fork) is the undisputed king of hybrid CPU/GPU offloading and MoE performance. However, every time I benchmark it against mainline ggml-org, mainline consistently beats it by a wide margin. Am I missing… 28 r/LocalLLaMA community 7d ago MiMo V2.6 Pro almost matched GPT-6 Astra at figuring out what a user actually wants Introducing InferBench: A benchmark testing how well frontier LLMs infer a user's priorities from their instructions. We tested 12 LLMs across 20 scenarios with 2.8k conversations to see which models understand the user's goals. Each conversation has a simulated user with a… 10 Simon Willison community 8d ago Mistral Large 4 My comment on Mistral Large 4 — Hacker News. wren6991 : The benchmark is saturated. Frontier models are tested with an armadillo in fishnet tights jaywalking on Mars. OK well I couldn't resist this one: llm -m claude-opus-5.5 'Generate an SVG of an armadillo in fishnet… 16 r/LocalLLaMA community 8d ago Mistral Large 4 benchmarks Just dropped.   submitted by   /u/Informal-Trouble2183 [link]   [comments] 24 r/MachineLearning community 8d ago SWE-Race: a coding-agent benchmark of 188 real concurrency bugs, with results from three models [P] We've been building a benchmark out of real concurrency bugs (race conditions, deadlocks, cancellation issues) taken from merged PRs in about 100 Python projects. Each task gets graded by the project's own tests, in a container with no network, and the repo is cut down to a… 7 r/LocalLLaMA community 8d ago Qwen 3.8 27b just feels… ok? I’ve seen posts here raving about how good Qwen 3.8 27b is. The benchmarks look incredible, and all the online discourse seems to deem it the best local model. I have 32GB VRAM and run Unsloth’s Q6 version with OpenCode. For small tasks, it feels fine. I range 30-40 t/s decode… 15 r/LocalLLaMA community 8d ago A benchmark for LLMs playing Civilization V. GLM-5.3 is ahead of Opus-5.5, and Qwen-3.8-27B holds up surprisingly well. A while ago, I posted here getting OSS-120B and GLM-4.6 playing full games of Civilization V . Since then, models have moved pretty far, and we wanted a better understanding about models' capabilities playing the game. Introducing the controlled version of CivBench on newer… 14 GitHub Blog — AI & ML official-blog 9d ago ReviewBench: An open benchmark for AI code review We’re launching ReviewBench, a benchmark for code review agents built on representative GitHub pull requests, multi-source ground truth, calibrated evaluation, and production-aligned metrics. The post ReviewBench: An open benchmark for AI code review appeared first on The GitHub… 22 r/LocalLLaMA community 9d ago Speakrail - a low-latency fully-local voice assistant that runs on a single RTX 4090 tl;dr: I created a fully-local open-source full-duplex voice agent that rivals GPT-Live on some benchmarks. It uses Voxtral Realtime with a turn-taking head, a microturn-finetuned Gemma 4 12B and Breeze TTS 2 under the hood. Go try it out: https://github.com/speakrail/speakrail… 21 Page 2 of 10 · 500 articles ← Newer Older →
