News / #benchmark Tag Benchmark 500 articles archived under #benchmark · RSS Sign in to follow arXiv — NLP / Computation & Language research 28d ago Are We Grading Properly? Understanding Failure Modes in Medical Benchmarks arXiv:2609.16023v1 Announce Type: new Abstract: Medical evaluation is shifting from static option-based questioning to realistic clinical scenarios with open-ended output modes. Grading these at scale naively, however, is expensive, and rubric-based evaluation has become the… 25 arXiv — NLP / Computation & Language research 28d ago ParsHate: A Benchmark Dataset for Hate and Target Detection in Persian arXiv:2609.16393v1 Announce Type: new Abstract: We introduce ParsHate, a manually annotated dataset of 10,000 Persian tweets spanning 2013-2022, representing the first decade-long benchmark for hate speech detection in Persian. The dataset contains 31% hateful content and… 19 arXiv — NLP / Computation & Language research 28d ago RoleBreak: Benchmarking Long-Horizon Role-Playing Robustness in Spoken Dialogue arXiv:2609.16614v1 Announce Type: new Abstract: Speech-to-speech dialogue models increasingly support persona control, yet existing spoken role-playing benchmarks remain largely character-centric and short-horizon. This leaves open whether spoken dialogue models can sustain… 15 arXiv — NLP / Computation & Language research 28d ago Japanese Stroke LLM Evaluation: A Conversational Benchmark for Safe Stroke Care in Japanese Using Large Language Models arXiv:2609.16739v1 Announce Type: new Abstract: Background: Large language models (LLMs) have achieved physician-comparable performance on multiple-choice medical knowledge examinations, but their capabilities in clinical history taking, urgency assessment, and safety remain… 18 arXiv — NLP / Computation & Language research 28d ago Benchmarking Factual Robustness of LLMs via Multi-conversation Persuasion arXiv:2609.16777v1 Announce Type: new Abstract: As Large Language Models (LLMs) increasingly serve as primary knowledge retrieval interfaces, their robustness against \textit{persuasion attacks}---attempts to inject misinformation or enforce counterfactuals---has become a… 13 arXiv — NLP / Computation & Language research 28d ago RiskChainBench: A Benchmark for Obfuscated Platform Message Restoration and Evidence-Grounded Web Investigation arXiv:2609.16900v1 Announce Type: new Abstract: Platform abuse campaigns conceal redirection instructions with emojis, homophones, character decomposition, and redundant symbols, then route users through disguised links to services associated with pornography, fraud, gambling,… 23 arXiv — NLP / Computation & Language research 28d ago Towards Detecting AI-Assisted Responses in Online Surveys arXiv:2609.17317v1 Announce Type: new Abstract: The use of LLMs to complete online surveys impacts the validity of survey-based research, but detecting such usage remains underexplored. We introduce an initial benchmark dataset, namely ASURRE, for AI-assisted survey… 17 arXiv — NLP / Computation & Language research 28d ago ECHO: A Matched-Contrast Benchmark for Context-Sensitive Turn-Taking in Full-Duplex Dialogue arXiv:2609.17360v1 Announce Type: new Abstract: Full-duplex spoken dialogue systems must distinguish interruptions that require yielding the floor from backchannels that permit continued speaking. Existing benchmarks typically evaluate events independently and may therefore… 28 arXiv — NLP / Computation & Language research 28d ago Right Tool, Right Job: Native-Language Evaluation, Tokenizer Sensitivity, and Methodological Findings from a French-Only BabyLM arXiv:2609.17435v1 Announce Type: new Abstract: We submit M\'eTRON-FR, a 125M GPT-2 pretrained on 92.47M words of French, to the BabyLM 2026 Strict track. It scores 85.97 +/- 0.17% on QFrBLiMP (a native Quebec-French benchmark of grammatical minimal pairs) and 62.80% on the… 25 arXiv — NLP / Computation & Language research 28d ago BLINDSPOT: A Benchmark for Safety and Refusal Calibration in Long-Horizon Tool-Using Agents arXiv:2609.16305v1 Announce Type: cross Abstract: Large language model (LLM) agents increasingly operate over long-horizon interactions involving tool use, persistent state, evolving authorization, and external environment feedback. In such settings, safety failures may emerge… 16 arXiv — NLP / Computation & Language research 28d ago Audio-Visual Turn-taking Prediction in Cocktail Party Scenarios arXiv:2609.17056v1 Announce Type: cross Abstract: Current predictive turn-taking models (PTTMs) achieve strong performance on benchmarks with controlled acoustic conditions and clean audio signals. Their generalisation to conversations with overlapping speech and background… 28 r/LocalLLaMA community 29d ago I benchmarked IFM/K2-Horizon-7B on 16GB VRAM After two days of fighting with benchmarking infrastructure, I finally benchmarked IFM/K2-Horizon-7B on 16GB VRAM. TL/DR: it works, but far behind Qwen-3.8-27B. Setup I compared 3 models - Qwen3.8-27B, Ornith-1.5-9B and new IFM/K2-Horizon-7B - on the following setup: All models… 5 r/LocalLLaMA community 29d ago Cut Qwen3.8-27B Reasoning Tokens by 40% -- 3.8 'ThinkingCap' benchmarked! I doubt I'm in the minority here when I say I love Qwen models, but the overthinking is a major timekiller. It was bad in 3.6-27B, and it's worse in 3.8. I know there are some who say, "well that's how it achieves such a good performance/size ratio"... But now there's some… 36 r/MachineLearning community 29d ago TabPFN-3.5 is released as the next SOTA tabular foundation model [N] Prior Labs released their latest tabular foundation model, TabPFN-3.5 today. The model is top of both TabArena and BeyondArena and SOTA for 1M rows and up to 20k features It comes with: - TabPFN-3.5-Fast (in alpha): This one goes 6x faster than the base model -… 11 r/LocalLLaMA community 29d ago ByteShape Qwen 3.8 27B: To KL Diverge or Not to KL Diverge, Part 2: Metric Boogaloo Hey r/LocalLLaMA , We’ve released our full ShapeLearn GGUFs for Qwen 3.8 27B. Blog / Download models TL;DR 3.84 bpw (GPU-5) reaches 99.63% of BF16’s aggregate score of 8 benchmarks, being the most accurate quant we’ve evaluated; 3.23 bpw (GPU-4) reaches 98.72%. These average… 6 r/LocalLLaMA community 29d ago Voodoo Dynamic Quant - Now MIT Licensed Two months ago I announced I had found a new dynamic quant method called Voodoo Quant which was SOTA for the most aggressive quant levels on some smaller Qwen3.5 GGUF models. I kept the methodology private at the time, but I've seen too many requests for dyn quants for various… 6 arXiv — Machine Learning research 29d ago A derivative-fidelity failure mode in physics-informed neural networks: strengthened benchmark evidence from function-value training arXiv:2609.13171v1 Announce Type: new Abstract: Physics-informed neural networks (PINNs) use automatic differentiation to impose differential-equation residuals, but good agreement in function values does not necessarily imply accurate derivatives. This paper formulates… 16 arXiv — Machine Learning research 29d ago LLMs or Naive Bayes? Old Gems or New Ways arXiv:2609.13185v1 Announce Type: new Abstract: Large language models (LLMs) prompt a recurring question in research computing: should classical methods like Naive Bayes (NB) be retired? We benchmark Complement Naive Bayes against zero-shot and few-shot LLMs spanning four model… 23 arXiv — Machine Learning research 29d ago A Three-Axis Stress Test of LLM vs Classical ML for Network Intrusion Detection under Distribution Shift and Adversarial Evasion arXiv:2609.13511v1 Announce Type: new Abstract: Large language models are increasingly benchmarked against classical machine learning for network intrusion detection (NIDS), almost always using same-dataset evaluation, and that protocol turns out to be incomplete. Evaluating… 19 arXiv — Machine Learning research 29d ago Benchmarking Optimizers to Solve Inverse Problems with Differentiable Physics Simulators arXiv:2609.13819v1 Announce Type: new Abstract: Solving inverse problems with differentiable physics simulators holds the potential to revolutionize scientific discovery and engineering design, as it enjoys both the strict physical correctness from rigorous numerical physics… 6 arXiv — Machine Learning research 29d ago Accuracy Is Not Service: A Decision-Aware Benchmark for Intermittent-Demand Forecasting arXiv:2609.13840v1 Announce Type: new Abstract: A contract-logistics spare-parts operator is paid on order-level service: an order counts only if every requested line is fulfilled, yet forecasters are selected based on line-level forecast accuracy. This disconnect matters when… 19 arXiv — Machine Learning research 29d ago CoArena: Evaluating Computer-Use and Multi-Agent Systems in Real Time arXiv:2609.14239v1 Announce Type: new Abstract: Static benchmarks for computer-use agents fix a task set at release and score every system against it once. That makes them reproducible, and it lets them drift from what they should measure: a fixed task set ages, leaks into… 24 arXiv — NLP / Computation & Language research 29d ago PhysMent: An Interactive Approach For LLM Reasoning In Physics Problems arXiv:2609.13152v1 Announce Type: new Abstract: Large language models (LLMs) perform strongly on static science benchmarks, yet their ability to reason about the physical world through active experimentation remains poorly understood. We introduce PhysMent, a benchmark that… 18 arXiv — NLP / Computation & Language research 29d ago TestHallVQA: Exploring LVLMs' Document-Level Reasoning under Redundant Contexts from Scientific Exams arXiv:2609.13158v1 Announce Type: new Abstract: Large Vision--Language Models (LVLMs) are increasingly expected to perform visual question answering (VQA) over planar media. However, existing planar VQA benchmarks typically emphasize isolated challenges: some emphasize… 13 arXiv — NLP / Computation & Language research 29d ago Same Patient, Different Order: Action-Level Reliability of Clinical LLM Agents Under Repeated Runs arXiv:2609.13582v1 Announce Type: new Abstract: A clinical agent benchmark can report the same verdict on identical inputs while the agent files a materially different order on each run. Such agents order tests, request medications and place referrals, yet benchmarks typically… 22 arXiv — NLP / Computation & Language research 29d ago Understanding the Limits of Agentic ICD Coding arXiv:2609.13806v1 Announce Type: new Abstract: ICD-10-CM codes are alphanumeric codes used in the US to classify diagnoses and injuries for medical billing and epidemiological reporting. Standard ICD-10-CM benchmarks report aggregate metrics that obscure performance on complex… 9 arXiv — NLP / Computation & Language research 29d ago Bangla Sentence Function Classification: Corpus Development, Model Benchmarking, and Interpretability arXiv:2609.13869v1 Announce Type: new Abstract: Automatic sentence function identification is important for many downstream natural language processing (NLP) applications such as dialogue systems, text-to-speech synthesis, and machine translation. However, benchmark resources… 4 arXiv — NLP / Computation & Language research 29d ago Mizan: A National Benchmark for Evaluating Large Language Models on Iraqi Arabic and the Iraqi Civic Context arXiv:2609.13980v1 Announce Type: new Abstract: Arabic large-language-model (LLM) evaluation has matured around Modern Standard Arabic (MSA): aggregated leaderboards such as the Open Arabic LLM Leaderboard (OALL), HELM Arabic, and BALSAM rank models across dozens of MSA tasks,… 37 arXiv — NLP / Computation & Language research 29d ago E2A-Bench: Benchmarking Evidence-to-Action Reliability in Financial Chart Reasoning arXiv:2609.14302v1 Announce Type: new Abstract: Can financial vision-language models (VLMs) turn chart evidence into reliable action recommendations? Existing hallucination evaluations are mostly claim-centric; they assess whether generated statements are supported, but not… 6 arXiv — NLP / Computation & Language research 29d ago Policy Loopholes in Agent Evaluation: When Policy Ambiguity Masquerades as Agent Error arXiv:2609.14400v1 Announce Type: new Abstract: Agent benchmarks evaluate policy compliance but assume each policy determines a unique correct action. Natural-language policies can violate this assumption through silence, ambiguity, or contradiction, admitting multiple… 12 arXiv — NLP / Computation & Language research 29d ago One Example Is Enough to Pass Fairness Benchmarks: Rethinking Fairness Evaluation for Aligned LLMs arXiv:2609.14860v1 Announce Type: new Abstract: Warning: This submission studies stereotypes and biases, and contains toxic and offensive examples, used for illustration purposes only. Fairness benchmarks such as BBQ have become the de facto standard for fairness evaluation… 34 arXiv — NLP / Computation & Language research 29d ago MTAC-IFBench: Benchmarking Instruction-Following in Multi-Turn Agentic Coding arXiv:2609.14992v1 Announce Type: new Abstract: Recently, the rapid development of large language models (LLMs) has reshaped software engineering by enabling autonomous code agents that plan, execute, and utilize external tools iteratively to tackle complex tasks. Beyond… 8 arXiv — NLP / Computation & Language research 29d ago SALUTE: Benchmarking and Adapting LLMs for the Defense Domain arXiv:2609.15022v1 Announce Type: new Abstract: Defense is a knowledge-intensive domain that requires precise understanding of specialized terminology, doctrinal concepts, operational procedures, and evolving military events. Although recent work has explored language… 13 Hugging Face Daily Papers research 29d ago Atria Dawn: The Dawn of Agentic Superintelligence Abstract Atria Dawn Preview is a foundation agentic language model trained through verified tool interactions that achieves strong benchmark results and demonstrates a shift toward human-AI project-level collaboration in scientific research. Generated by… 9 Hugging Face Daily Papers research 1mo ago PhysBrain 1.5: From Vision-Language Models to Physical Foundation Models Abstract PhysBrain 1.5 unifies physical environment understanding, action generation, and future state prediction via joint autoregressive training on discrete vision-language, motion, and visual target sequences, achieving state-of-the-art open-source embodied performance.… 29 Hugging Face Daily Papers research 1mo ago BVB: Benchmarking Agentic Video Understanding via Programmatic Reconstruction in Blender Abstract A benchmark requiring agents to programmatically reconstruct real-world videos in Blender reveals that current models achieve high perceptual similarity but struggle to retain spatiotemporal facts. Generated by thinkingmachines/Inkling-Small Multimodal agents can create… 16 r/LocalLLaMA community 1mo ago K2 Horizon lineup is out on AA, and once again AA plots are misleading. The full K2 Horizon lineup is out on Artificial Analysis. The AA intelligence vs. parameters plots show that - 0.9B and 375B are bad - 3.7B and 7B are SOTA - 36B A4B is SOTA for hardware with poor memory bandwidth (spilled experts, Strix Halo, DGX Spark). I'm going to take the… 21 Hugging Face Daily Papers research 1mo ago DataFlex-RL: An Evaluation Platform for RLVR Data Policies Abstract DataFlex-RL evaluates reinforcement learning data policies and finds that uniform sampling matches or exceeds adaptive rollout selection, reweighting, and domain mixing across math, logic, and science benchmarks. Generated by thinkingmachines/Inkling-Small Data policies… 8 r/MachineLearning community 1mo ago Duplicating baseline benchmarks [D] Suppose I create two machine learning models suppose tree and neural network for a task let's suppose regression problem, now suppose I am sending both of this paper to two different journals, now the thing is the baseline models I need to only run once because I have reported… 35 r/LocalLLaMA community 1mo ago Running Qwen 3.8 next on 16vram+32ram - A useful/fun post for the gpu poors Hello Reddit. Posting this for fun. I thought it was a lonely and silly journey to set up Qwen 3.8 Next on a system that doesn't really run it properly—it was a challenge that might help the community. I have yet to benchmark this specific REAP version versus Qwen 3.8 27B QK4,… 7 Hugging Face Daily Papers research 1mo ago Feyospace-v1: How the Cyber Mercury Seven Trained Frontier Cyber Models Abstract A data-centric framework with specialized systems for reasoning analysis, cost reduction, and execution verification enables small teams to train open-weight cyber agents that achieve top-tier performance on benchmark suites. Generated by thinkingmachines/Inkling-Small… 17 arXiv — Machine Learning research 1mo ago FINESSE: An Agent-Based Simulator and Benchmark Dataset for Multimodal Financial Event Sequences arXiv:2609.11993v1 Announce Type: new Abstract: Machine learning research in financial services is limited by the scarcity of representative open-source datasets. Existing resources are often narrowly focused on a single modality or task and fail to reflect the structured,… 11 arXiv — Machine Learning research 1mo ago ParaRecover: A Process-Level Benchmark for Error Localization and Recovery in Parallel Tool-Use Agents arXiv:2609.12345v1 Announce Type: new Abstract: Existing agent benchmarks mainly evaluate final task success or tool-call correctness, providing limited insight into whether agents can reliably diagnose and recover from intermediate execution failures. This limitation becomes… 26 arXiv — Machine Learning research 1mo ago Certified AI Triage of ICU Alarms arXiv:2609.12365v1 Announce Type: new Abstract: In the VTaC benchmark 71% of ventricular-tachycardia alarms are false, but silencing a real one can delay recognition of a dangerous arrhythmia. We reframe alarm reduction as three-way triage (retain, suppress, or defer) and bound… 12 arXiv — NLP / Computation & Language research 1mo ago MAxBench: A Multinomial Concept Recovery Benchmark arXiv:2609.13072v1 Announce Type: cross Abstract: Fine-grained control of language model behaviors (e.g., steering) is among the more actionable outcomes of interpretability research. For binary concepts such as refusal, a single direction in activation space often suffices for… 31 arXiv — NLP / Computation & Language research 1mo ago Zipbench: Low-Cost Framework for Compressing Comprehensive Benchmarks of Large Language Models arXiv:2609.12475v1 Announce Type: new Abstract: Comprehensive benchmark suites are essential for improving large language models (LLMs), but many widely used benchmarks are redundant, making evaluation unnecessarily expensive. Although recent benchmark compression methods (BCMs)… 13 arXiv — NLP / Computation & Language research 1mo ago MedSNIP: Building and Benchmarking Snippet-Level Granularity for Medical Fact Verification arXiv:2609.12884v1 Announce Type: new Abstract: A medical claim's correctness often depends not on the claim alone, but on the clinical structure around it. A claim may require a lab reference range, a causal or conditional link, or patient-specific details to be judged… 14 arXiv — NLP / Computation & Language research 1mo ago Judging by the Cover: Cleaning LLM Truthfulness Benchmarks to Avoid Surface-Level Feature Leakage arXiv:2609.13003v1 Announce Type: new Abstract: Binary-choice truth benchmarks ask models to choose between a correct and an incorrect answer, but if the two answers differ systematically in surface-level features, models can exceed chance without performing the intended… 35 arXiv — NLP / Computation & Language research 1mo ago Tasks over Application Manuals: Revealing Gaps in Long-Horizon Procedural Reasoning for Language Models arXiv:2609.13005v1 Announce Type: new Abstract: Large language models (LLMs) have achieved strong performance on a wide range of natural language tasks, and recent benchmarks suggest that they are increasingly adept at multi-hop reasoning. However, these benchmarks are typically… 26 arXiv — NLP / Computation & Language research 1mo ago Do LLMs Trust the Accuser or the Accusation? Measuring Belief Shifts in Werewolf arXiv:2609.12446v1 Announce Type: cross Abstract: Social-deduction games such as Werewolf are increasingly used to evaluate LLM agents, but existing evaluations often rely on final game outcomes. We propose a belief-shift evaluation benchmark in Werewolf for analyzing… 17 Page 8 of 10 · 500 articles ← Newer Older →
