News / #benchmark Tag Benchmark 500 articles archived under #benchmark · RSS Sign in to follow arXiv — NLP / Computation & Language research 19d ago TTLab at StanceEval-2026: A Cloze-Style Prompting Approach for Arabic-Language Stance Detection (CLASP-Ar) arXiv:2609.29733v1 Announce Type: new Abstract: Arabic-language stance detection remains challenging, and previous shared-task systems have largely relied on multitask learning and ensembles. While these systems achieve state-of-the-art performance, their applicability and… 32 arXiv — NLP / Computation & Language research 19d ago Benchmarking and Domain Adaptation of Automatic Speech Recognition (ASR) for Adolescent Health Communication in Ghanaian Languages arXiv:2609.29798v1 Announce Type: new Abstract: This paper presents an end-to-end study of automatic speech recognition (ASR) for adolescent health communication in three Ghanaian languages (Twi, Dagbani, and Ewe). The work proceeds in three connected stages; First, we benchmark… 33 arXiv — NLP / Computation & Language research 19d ago Multi-Task Learning by using Contextualized Word Representations for Syntactic Parsing of a Morphologically Rich Language arXiv:2609.29855v1 Announce Type: new Abstract: We address the challenge of syntactic parsing for Urdu, a morphologically rich language, and present state-of-the-art results for both constituency and dependency parsing. This paper offers four major contributions: 1) the… 9 arXiv — NLP / Computation & Language research 19d ago Artificial Societies Benchmark: A Validation Framework for Synthetic Research arXiv:2609.30030v1 Announce Type: new Abstract: A synthetic survey can reproduce the average answer while misrepresenting how people differ, how their answers relate to one another, or how they respond to changes in conditions. We introduce the Artificial Societies Benchmark to… 20 arXiv — NLP / Computation & Language research 19d ago Personalized Korean Lipreading as Visual Speech Recognition: Transfer, Census and Adaptation on OLKAVS arXiv:2609.28988v1 Announce Type: cross Abstract: We present a personalized Korean visual speech recognition (VSR) system and quantify, on the nine-camera OLKAVS corpus, the gap between the population-level benchmark score and an individual user's error. A video-only Conformer… 27 arXiv — NLP / Computation & Language research 19d ago Design and Evaluation of LLM Chaining-Based Task Planning for General Purpose Service Robots arXiv:2609.29043v1 Announce Type: cross Abstract: General Purpose Service Robot (GPSR) tasks, as defined in the RoboCup@Home benchmark, require robots to interpret diverse natural language commands and generate multi-step action sequences in real home environments. Conventional… 6 r/LocalLLaMA community 19d ago normalize benchmarks from different time period LiveBench has benchmark snapshots from different points in time. Could someone run an agent to normalize the values across these snapshots so we can compare model strength consistently from 2024 through 2026? Right now, it’s difficult to make meaningful comparisons across the… 18 r/LocalLLaMA community 20d ago 7900 XTX — two "low-thinking" Qwen 3.8 27B quants (Swift + ThinkingCap) vs the regular quant First, do they actually produce less tokens? Yes. Total tokens per benchmark run (4 scenarios): base quant ~66k, ThinkingCap ~49k (−26%), Swift ~45k (−33%). So the "less thinking" is real — and Swift cuts the most. Then the cost: and this is where it got interesting. The two… 29 r/LocalLLaMA community 20d ago ThinkingCap 3.8-27B vs. Swift 3.8-27B vs. Qwen 3.8-27B Benchmarks With the release of ThinkingCap-Qwen3.8-27B , I thought it would be worthwhile to do a comparison between the original Qwen3.8-27B, the new ThinkingCap, and Swift-Qwen3.8-27B . Both Swift which I already reviewed , and ThinkingCap do exactly the same thing: they reduce the… 8 r/LocalLLaMA community 20d ago Lesson learned. Don't blindly trust repos and make sure everything is stable for a long running (multi weeks) benchmark. I posted previously my swe-verified django 100 tasks benchmark comparing different local models and quantization. No new models for now, but a fix in my evaluation workflow that was unfortunately not stable during the weeks/months of me using it. I redid the evaluation on all… 24 r/LocalLLaMA community 20d ago Folks, have you purchased the Mac M5 Ultra with 256GB yet? We need serious benchmarks, because we only get YouTube clowns influencers results Ok, so the Mac M5 Ultra (256GB) hit the market, but the only publicly available benchmarks material are flashy YouTube "clown influencers" videos. We need serious numbers to evaluate whether Apple’s silicon can actually compete with Nvidia’s current GPU‑centric workflows or not.… 11 arXiv — Machine Learning research 20d ago What Makes a Terminal-Bench Task Hard? Separating Genuine Hardness from Fake-Hardness on an Adjudicated Agentic Corpus arXiv:2609.26826v1 Announce Type: new Abstract: Frontier benchmarks need tasks that current models cannot solve. But a task that no model solves is not automatically a hard task. The same zero pass rate can come from a real capability gap, but it can also come from missing… 13 arXiv — Machine Learning research 20d ago COPE: Continual Personalization of LLMs under Sparse User Feedback via User Embeddings and Self-Evaluation arXiv:2609.26853v1 Announce Type: new Abstract: While Large Language Models (LLMs) have achieved remarkable results across various benchmarks, their alignment with normative values often results in homogenized responses that fail to address diverse user preferences. Existing… 5 arXiv — Machine Learning research 20d ago QUARTET: Quad-branch cross-Attention and Random-walk Traces for Enhancing Transformers on Relational Graphs arXiv:2609.26855v1 Announce Type: new Abstract: Relational Deep Learning (RDL) models multi-table databases as heterogeneous temporal graphs, and graph transformers currently achieve state-of-the-art performance on benchmarks like RelBench. However, the current leading model,… 20 arXiv — Machine Learning research 20d ago An open benchmark for machine learning-based polymer property prediction arXiv:2609.27036v1 Announce Type: new Abstract: Polymer property prediction lacks open, standardized benchmarks that enable rigorous comparison of machine-learning methods, with existing resources covering only a narrow fraction of polymer architectures, such as homopolymers. We… 16 arXiv — Machine Learning research 20d ago A Systematic Benchmark of Explainable Methods for Temporal Attribution in Sequential Recommendation Systems arXiv:2609.27201v1 Announce Type: new Abstract: Sequential RecSys are central to modern personalization, exploiting user's historical interaction sequences to drive next-step decisions. Deep learning models, particularly CNN and Transformer-based architectures, have proven… 32 arXiv — Machine Learning research 20d ago Evaluation Choices Decide the Forecasting Leaderboard: Evidence from a Production Marketplace Panel arXiv:2609.27867v1 Announce Type: new Abstract: A forecasting benchmark reports which method won. We show that the answer is set by the evaluator's choices before any model is fitted. We benchmark 24 forecasting methods and one textbook reference, including six 2025-era time… 26 arXiv — NLP / Computation & Language research 20d ago Recognized but Not Produced: A Generation Benchmark for Culturally Specific Kinship Terms arXiv:2609.26942v1 Announce Type: new Abstract: Current literature evaluates large language models (LLMs) on multilingual kinship understanding using multiple choice benchmarks, treating it as a recognition problem. We instead prompt five open weight LLMs to generate kinship… 7 arXiv — NLP / Computation & Language research 20d ago Classifying Interpretive Canons at the Sentence Level: A Benchmark from the German Federal Constitutional Court arXiv:2609.26945v1 Announce Type: new Abstract: Judicial reasoning remains challenging for large language models (LLMs) to analyze. This paper contributes a sentence-level benchmark for evaluating the ability of LLMs to classify interpretive canons as articulated by Larenz in… 19 arXiv — NLP / Computation & Language research 20d ago Beyond Overlap: Estimating the Causal Effect of Benchmark Exposure arXiv:2609.27176v1 Announce Type: new Abstract: Evidence that evaluation material entered training does not reveal how much it affected evaluation. This distinction leaves a contaminated benchmark score difficult to interpret: provenance can establish contact, but only a… 28 arXiv — NLP / Computation & Language research 20d ago Phonemizing User-Generated Text: A Benchmark, Taxonomy, and Compositional Approach arXiv:2609.27205v1 Announce Type: new Abstract: Text-to-speech systems increasingly process user-generated text (UGT) such as ppl and imo, whose pronunciation must be inferred from the canonical rather than surface form. We introduce UGTPhon, the first grapheme-to-phoneme (G2P)… 17 arXiv — NLP / Computation & Language research 20d ago Neither Silence nor Overlap Is Failure: Intent-Conditioned Evaluation of Turn-Taking in Full-Duplex Spoken Dialogue Models arXiv:2609.27372v1 Announce Type: new Abstract: Benchmarks for full-duplex spoken dialogue models score turn-taking with binary fixed-window rules that reward immediate response or silence by completeness of the prior turn. We argue that the appropriateness of a response offset,… 28 arXiv — NLP / Computation & Language research 20d ago PRISM-VLM: A Multi-Axis Discriminative Benchmark for Compact Vision-Language Models arXiv:2609.27395v1 Announce Type: new Abstract: Compact vision-language models (VLMs) now power a growing share of multimodal applications. The benchmarks used to compare them, however, inherit a frontier-centric design: each model is reduced to a single accuracy number,… 15 arXiv — NLP / Computation & Language research 20d ago Uncheatable Eval: Dynamic Compression-Based Evaluation of Language Models arXiv:2609.27510v1 Announce Type: new Abstract: Modern large language models are pretrained on massive datasets, making it difficult to prevent benchmark data from entering their training sets and undermining the reliability of evaluation results. Reliable evaluation is… 30 arXiv — NLP / Computation & Language research 20d ago Evaluating Open-Weight LLMs for Turkish Domain Documents Under Retrieval and Hardware Constraints arXiv:2609.28007v1 Announce Type: new Abstract: Most Turkish-capable large language models (LLMs) are evaluated using general-purpose benchmarks rather than long, structurally complex domain documents. This paper evaluates five open-weight 7B-8B models for Turkish document… 33 arXiv — NLP / Computation & Language research 20d ago Evaluating Feedback Focus and Pedagogical Adaptivity in LLM-Generated Feedback on Student Writing arXiv:2609.28026v1 Announce Type: new Abstract: We investigate whether state-of-the-art large language models (LLMs) generate feedback that reflects the pedagogical practices of expert teachers in terms of feedback focus and adaptivity. Previous evaluation efforts have examined… 18 arXiv — NLP / Computation & Language research 20d ago Can LLMs Catch a Rigged Backtest? A Clean-Control Calibration Benchmark arXiv:2609.28090v1 Announce Type: new Abstract: Backtest auditing is a calibration problem: high flaw recall is not useful when the model falsely flags matched clean strategies. We build a 96-item paired benchmark in which every flawed backtest has a clean control that holds… 4 arXiv — NLP / Computation & Language research 20d ago Fine-Tuning LLMs for Translation: General Forgetting Mitigation Does Not Preserve MT-Specific Instruction Following arXiv:2609.28395v1 Announce Type: new Abstract: Fine-tuning large language models on parallel data improves translation quality but can cause catastrophic forgetting. Mitigation methods are generally evaluated by retention on general benchmarks. We ask whether these findings… 34 arXiv — NLP / Computation & Language research 20d ago What Looks Like a Capability Limit in Vision-Language Models Is a Readout Limit arXiv:2609.27408v1 Announce Type: cross Abstract: Benchmarks for vision-language models offer their answer choices in some convention: a letter, a color name, a pixel coordinate. That convention is treated as neutral. We find it is not, and that the limits a benchmark reports… 26 Don't Worry About the Vase community 21d ago Claude Opus 5.5: The System Card Introducing the world’s most powerful model, at least by some measures like Artificial Analysis or any standard benchmark list, which is now Claude Opus 5.5. 7 r/LocalLLaMA community 21d ago M5U base 96GB inference numbers for Q3.8FN after 112M tokens TLDR; Base M5 Ultra 96 GB ran Q3.8 FN aggregate 3.2k PP and ~170 TG in 4 concurrency Alert: Numbers and custom server details at end are AI assisted So the good news is that I got the base model on launch day with only 64 core GPU. All benchmarks are for current maxed out model,… 34 r/LocalLLaMA community 21d ago MiMo-V2.6 (both Pro and Flash) is a benchmaxxed scam MiMo-V2.6-Pro has an insanely high score of 46 on AA, putting it at the head of the opensource models available. It also costs pennies. Flash is not out on AA yet, but it costs less than half on datacenter and is slightly below on Xiaomi's own benchmarks. It also fits in 192GB,… 12 OpenAI official-blog 21d ago Introducing MentalHealthBench MentalHealthBench is an expert-informed benchmark for evaluating helpful and safe AI responses across realistic mental health conversations. 7 The Information — AI news-outlet 21d ago Shares of Chinese AI Model Firms Fall After Report of Regulatory Probe Hong Kong-listed shares of Z.ai and MiniMax plunged on Wednesday, after The Information reported that China’s internet regulator is probing potential data leaks to Anthropic. Z.ai, also known as Zhipu, dropped 12.4%, MiniMax fell 4% and Alibaba declined 4.4%. The benchmark Hang… 37 TechCrunch — AI news-outlet 21d ago “We’re already fighting yesterday’s battle”: Greece’s prime minister gets candid about AI Most leaders on a trade mission stick to the pitch, but when I interviewed Greek Prime Minister Kyriakos Mitsotakis this week, he also admitted that no government is ready for what AI is about to do. 7 arXiv — NLP / Computation & Language research 21d ago Trains but Doesn't Learn: A Post-Training Delivery Benchmark for LLM Agents as Forward-Deployed Engineers arXiv:2609.25237v1 Announce Type: cross Abstract: Post-training is becoming a service (PTaaS): a customer hands an operator data and a goal, and a forward-deployed engineer (FDE) returns a fine-tuned, evaluated, and deployed model under a budget, a human-approval gate, and… 32 arXiv — NLP / Computation & Language research 21d ago Efficient Cost-Aware LLM Evaluation via Bayesian Bandit Gittins Indices arXiv:2609.25645v1 Announce Type: cross Abstract: Exhaustively evaluating every candidate LLM configuration on every benchmark item to identify a high-performing one is costly. We formulate configuration selection as a cost-aware Bayesian bandit problem and propose GittinsEval,… 32 arXiv — Machine Learning research 21d ago Can You Delete a Year of Market Data? Machine Unlearning Against Exact Retraining Oracles arXiv:2609.26242v1 Announce Type: new Abstract: When a data license expires, deleting stored records does not remove influence encoded in a trained forecaster. Machine unlearning seeks to remove this influence without retraining. We benchmark temporal unlearning with 3,200… 18 arXiv — NLP / Computation & Language research 21d ago Peerify: Benchmarking Peer-Review Claim Verification arXiv:2609.25046v1 Announce Type: new Abstract: Peer review plays a central role in scholarly publishing, yet verifying whether reviewer claims are supported by manuscript evidence remains a largely manual and time-consuming process. We present Peerify, a pipeline for… 36 arXiv — NLP / Computation & Language research 21d ago FrontierMath Erd\H{o}s arXiv:2609.25050v1 Announce Type: new Abstract: We introduce FrontierMath Erd\H{o}s (FME), a benchmark of 68 Erd\H{o}s problems that are open as of August 2026. To solve a task in FME, AI systems must resolve (prove or disprove) one of the 68 conjectures in the proof assistant… 34 arXiv — NLP / Computation & Language research 21d ago FinFIRST: Benchmarking Search Agents for Financial Information Retrieval, Sourcing and Traceability arXiv:2609.25192v1 Announce Type: new Abstract: Financial search is a highly demanding task for LLM agents, requiring not only a correct final answer but also temporally valid information retrieval, authoritative source selection, entity and period alignment, unit and definition… 35 arXiv — NLP / Computation & Language research 21d ago FineWeb-CLaR: Culture, Language, and Region Annotations for Benchmark-Aligned Corpus Auditing arXiv:2609.25298v1 Announce Type: new Abstract: Cultural evaluation coverage and robustness in language models are difficult to diagnose because pretraining corpora and cultural benchmarks are rarely indexed with comparable metadata. Benchmarks increasingly target culturally… 32 arXiv — NLP / Computation & Language research 21d ago Passes Alone, Fails Together: Benchmarking Semantic Coordination in Parallel LLM-Agent Development arXiv:2609.25396v1 Announce Type: new Abstract: Parallel coding agents can produce patches that work alone but fail when merged. This happens when one agent changes an interface or rule that another agent still relies on. We study these failures with stale, a benchmark for… 11 arXiv — NLP / Computation & Language research 21d ago SpecialEduBench: Benchmarking Vision-Language Models on Knowledge, Skill, and Attitude in Language Intervention for Autistic Children arXiv:2609.26090v1 Announce Type: new Abstract: Language is the target of most early intervention for autistic children. Because the goal and the method change from child to child, the work falls to a teacher who takes one child at a time and judges each scene as it unfolds.… 18 arXiv — NLP / Computation & Language research 21d ago Differentiable Fuzzy Inference Layer: A Monotone, Compositional Ordinal Reasoning Head for Large Language Models arXiv:2609.26113v1 Announce Type: new Abstract: A state-of-the-art language model asked to interpret "most of most students passed" typically answers "most," though composing two instances of "most" yields a proportion closer to "some." We trace this failure to an architectural… 36 arXiv — NLP / Computation & Language research 21d ago Modality-Gated Deep Adapters: Adding a Modality to a Frozen Embedding Model with Exact Preservation arXiv:2609.26182v1 Announce Type: new Abstract: Multimodal embedding models are deployed at scale: retrieval indices, benchmark results, and behavioral audits all depend on the base model's exact outputs. Extending such a model to a new modality with existing parameter-efficient… 29 arXiv — NLP / Computation & Language research 21d ago Beyond Short Segments : Expanding Speaker Embeddings with Vector Archives arXiv:2609.25007v1 Announce Type: cross Abstract: The performance of state-of-the-art speaker verification (SV) systems severely degrades on short utterances due to insufficient speaker-specific information. To address this critical challenge, we propose the Vector Archive… 18 r/LocalLLaMA community 22d ago New 6B image model coming, AntLing just open sourced the Ming-Image-0.1-Design family • Ming-Image-0.1-Design, 6B • Ming-Image-0.1-Design-Layer, 6B • Two open-source Agent Skills: the Ling UI Design Skill and the Image-to-Editable-PPT Skill Ming-Image-0.1-Design ranks #1 among open-weight models on Artificial Analysis’s UI/UX Design leaderboard.… 12 r/MachineLearning community 22d ago LinearSolveBench: new benchmark for linear solvers [P] LinearSolverBench measures the ability of a model or harness to write fast, accurate, and general numerical solvers for large sparse linear systems in C. The goal is to encourage algorithmic advances in numerical methods for solving linear systems of equations.… 27 r/MachineLearning community 22d ago QontoFAQ: A better Information Retrieval Benchmark [R] Retrieval benchmarks sometimes feel benchmaxxed by models, so we wanted to find a way to tie it as close as possible to my objective: finding the article that answers a product question right . We worked on a new metric which seems more proportional to document relevance, and… 8 Page 5 of 10 · 500 articles ← Newer Older →
