News / #benchmark Tag Benchmark 500 articles archived under #benchmark · RSS Sign in to follow Hugging Face Daily Papers research 1mo ago SAEScientist-Bench: Can AI Agents Conduct Autonomous SAE Interpretability Research? Abstract The study introduces a benchmark to evaluate AI agents using sparse autoencoders for autonomous mechanistic interpretability and feature discovery, revealing progress but significant gaps versus expert baselines. Generated by thinkingmachines/Inkling-Small While… 4 arXiv — Machine Learning research 1mo ago AhaBench: Do Agents Learn from Prior Experience? A Benchmark for Long-Horizon Continual Learning arXiv:2609.05435v1 Announce Type: new Abstract: Modern language agents are expected to operate over long horizons: they ask follow-up questions, reuse worked examples, handle tool feedback, and adapt to delayed consequences. Most evaluations still reset the agent after a prompt… 19 arXiv — Machine Learning research 1mo ago MetaRSI / RSI2: A Meta-Recursive Self-Improving System for Recursive Self-Improving Systems Themselves arXiv:2609.06396v2 Announce Type: new Abstract: Recursive self-improvement (RSI) lets a system improve the model-building machinery from its own failures, so every later model inherits the gain. Yet RSI has been validated almost exclusively on coding and formal benchmarks such… 38 arXiv — NLP / Computation & Language research 1mo ago StochBench: A Domain-Specific Benchmark for Stochastic Processes in Lean arXiv:2609.09264v1 Announce Type: new Abstract: Leading benchmarks for formal theorem proving with large language models are small collections drawn from competition math, such as the IMO and Putnam, that poorly represent field-specific applications. We introduce StochBench, a… 7 arXiv — NLP / Computation & Language research 1mo ago SWORD: Wikidata-based Distortions Reveal Hidden Cross-Lingual Inconsistencies in LLM Factual Error Rejection arXiv:2609.09349v1 Announce Type: new Abstract: Modern LLMs demonstrate impressive multilingual performance, yet standard benchmarks primarily reward selecting correct answers rather than evaluating genuine factual understanding. We introduce Systematic Wikidata-based… 8 arXiv — NLP / Computation & Language research 1mo ago Benchmarking Hybrid Deep Research Across Database Querying and Web Search arXiv:2609.09410v1 Announce Type: new Abstract: While autonomous agents have made significant strides in "deep research" by iteratively navigating the open web to synthesize information, real-world problem-solving is rarely confined to a single environment. Complex analytical… 17 arXiv — NLP / Computation & Language research 1mo ago SEA-SpeechBench: A Large-Scale Multitask Benchmark for Speech Understanding Across Southeast Asia arXiv:2609.09672v1 Announce Type: new Abstract: The rapid advancement of audio and multimodal large language models has unlocked transformative speech understanding capabilities, yet evaluation frameworks remain predominantly English-centric, leaving Southeast Asian (SEA)… 22 arXiv — NLP / Computation & Language research 1mo ago YallaMorph: A Benchmark for Evaluating Arabic Morphological Generation in Large Language Models arXiv:2609.10153v1 Announce Type: new Abstract: Arabic morphology remains challenging for large language models, since fluent generation does not guarantee accurate morphosyntactic control. Existing Arabic evaluations mainly target downstream tasks and do not directly test… 14 arXiv — NLP / Computation & Language research 1mo ago IBIB: A Protocol for Measuring Enterprise AI Systems by Serving Route, Not Model Identifier arXiv:2609.10494v1 Announce Type: new Abstract: Enterprises deploy systems, not checkpoints. Usable capability depends jointly on weights, serving route, precision, output contract, and harness, yet all 18 audited benchmarks score advertised model identifiers. We treat this as… 36 arXiv — NLP / Computation & Language research 1mo ago IdeaAMBIG: Benchmarking Implementation-Critical Gaps in Research-Idea Specifications arXiv:2609.10539v1 Announce Type: new Abstract: A research idea may be novel, coherent, and scientifically plausible, yet its proposed method may remain insufficiently specified for faithful implementation. We study the codification readiness of implementation-facing… 7 arXiv — NLP / Computation & Language research 1mo ago What Does MMLU Actually Measure? A Psychometric Audit of Difficulty Structure in Aggregate Benchmark Scores arXiv:2609.09372v1 Announce Type: cross Abstract: Although MMLU is widely adopted as a benchmark for calibrating general AI capabilities, we psychometrically demonstrate that its aggregate score primarily evaluates a model's factual retrieval capacity rather than its reasoning… 35 arXiv — NLP / Computation & Language research 1mo ago LogiScope-VQA: Benchmarking Vision-Language Models for Logistics Hazard Identification in Industrial Scenarios arXiv:2609.09790v1 Announce Type: cross Abstract: Large Multimodal Models (LMMs) large-scale deployment in industrial warehouse settings specifically necessitates that models exhibit human-expert-level hazard-oriented perception, understanding, and reasoning capabilities.… 18 arXiv — NLP / Computation & Language research 1mo ago MetroLLM-Bench: Evaluating Language Models as Transit Kiosk Runtimes arXiv:2609.10016v1 Announce Type: cross Abstract: We introduce MetroLLM-Bench, a 955-case benchmark for testing language models as the policy layer of a transit kiosk. It covers six real metro systems, ranging from 37 to 414 stations, and eleven categories that include routing,… 34 arXiv — NLP / Computation & Language research 1mo ago Deterministic Prompting for Speaker-Stable Low-Resource Greek TTS arXiv:2609.10022v1 Announce Type: cross Abstract: Modern TTS systems approach human quality for high-resource languages but degrade when clean speech data is scarce. Modern Greek exemplifies this, lacking the curated corpora behind state-of-the-art synthesis. We propose a data… 19 Hugging Face Daily Papers research 1mo ago RESCUE-BENCH: Towards Relation-Aware Multi-Party Emotional Support Conversation Systems Abstract This work introduces a benchmark for evaluating whether large language models can understand evolving interpersonal dynamics to provide effective emotional support in multi-party settings. Generated by thinkingmachines/Inkling-Small Existing emotional support… 17 Hugging Face Daily Papers research 1mo ago VDiff-Bench: A Challenging Benchmark for Fine-Grained Image Difference Identification Abstract VDiff-Bench evaluates multimodal language models on fine-grained image difference identification, revealing major weaknesses in detecting subtle low-level visual changes. Generated by thinkingmachines/Inkling-Small Multimodal Large Language Models (MLLMs) perform… 7 Hugging Face official-blog 1mo ago IBM releases SOTA Granite Time Series PatchTST-FM-r2 model with commercial-friendly license Back to Articles a]:hidden"> IBM releases SOTA Granite Time Series PatchTST-FM-r2 model with commercial-friendly license Enterprise Article Published September 9, 2026 Upvote 2 Roman Vaculin vaculin ibm-research Ayhan Sebin ayhansebin ibm-research Wesley M. Gifford wmgifford… 38 r/LocalLLaMA community 1mo ago SOTA ImageGen Locally NVIDIA Cosmos3(64B) INT4 quants CUDA/MLX Cosmos3 INT4 T2I + I2V on Apple Silicon — code, weights and a Grok comparison GitHub - https://github.com/gtrg55/cosmos3-quant-mlx-cuda HF weights - https://huggingface.co/JuliaML/Cosmos3-Super-Text2Image-4Step-INT4-G64-BF16 Single clip took approximately 5m on M4 MAX 128 GB Mac… 26 Hugging Face Daily Papers research 1mo ago SynthGait-19K: A Physically Grounded Synthetic Video Dataset for Gait Parameter Estimation Abstract SynthGait-19k is a large synthetic video dataset for gait analysis that enables benchmarking of video-based gait estimation and shows synthetic supervision transfers to real data. Generated by thinkingmachines/Inkling-Small Accurate estimation of clinically meaningful… 9 r/LocalLLaMA community 1mo ago new Nex model new model of nex benchmark seems like good https://huggingface.co/nex-agi/Nex-N2.5-Max   submitted by   /u/Lordaizen639 [link]   [comments] 32 Hugging Face Daily Papers research 1mo ago RoboSPA: Can VLA Models Go Beyond Simple Scenes and Short-Horizon Tasks? Abstract RoboSPA is a large-scale robotic manipulation benchmark that evaluates vision-language-action models on fine-grained spatial reasoning and long-horizon procedural planning across progressively harder task variants. Generated by thinkingmachines/Inkling-Small… 25 r/LocalLLaMA community 1mo ago What OpenBMB 1B version is this? This is from the https://huggingface.co/IFM/K2-Horizon-0.9B benchmark image Tried searching for it but couldn't find it, I am guessing is https://huggingface.co/openbmb/MiniCPM5-1B but the numbers do not match   submitted by   /u/FerLuisxd [link]   [comments] 7 Hugging Face Daily Papers research 1mo ago NeoHorse-1: Towards Recursive Self-Improvement via Agentic Post-Training with Routing Harness Abstract NeoHorse-1 uses agentic post-training with intelligent routing, structured feedback loops, and curriculum-based distillation to improve model capabilities across agent benchmarks. Generated by thinkingmachines/Inkling-Small Recursive self-improvement (RSI) requires a… 33 Hugging Face Daily Papers research 1mo ago MOLE: Detecting Insider Threats in AI Agents Abstract MOLE is a benchmark for evaluating defenses that detect harmful actions by AI agents operating shared services under limited review budgets. Generated by thinkingmachines/Inkling-Small Model misalignment, prompt injection, or operator misuse could lead AI agents… 33 r/LocalLLaMA community 1mo ago Are there any (small, ~10B) models that you would say are a good collaborator? Most of the new ~30B (and now ~10B, thankfully for my GPU) models we see score really high on benchmarks, but I feel like they don't push back on dumb ideas enough. I think most people don't being like told by an LLM that the premise is flawed but I certainly do. In my opinion… 32 Hacker News — AI on Front Page community 1mo ago Benchmarking Qwen3.8 27B quantizations: 4-bit holds up, 1-bit collapses Article URL: https://quesma.com/blog/qwen38-27b-quantizations-benchmarked/ Comments URL: https://news.ycombinator.com/item?id=49611128 Points: 216 # Comments: 107 11 r/LocalLLaMA community 1mo ago XHToken/Spark-X2.5-4B VS inclusionAI/Ling-3.0-tiny VS Nanbeige/Nanbeige4.2-3B Which small model are you ppl finding the most useful rn? They all seem to compete for the same size class while actively avoiding each other in benchmark tables. Edit: These three in the title and also MiniCPM5 2b, Spark 1.7b, and LFM2.5 2.6b and LFM2.5 8b   submitted by… 15 r/MachineLearning community 1mo ago I reduced image-processing token usage by ~95% compared with GPT-4o direct vision, while maintaining roughly the same accuracy.How significant is that?[P] I'm testing a new approach for reducing the cost of image-based LLM inference. I evaluated it on the MOMA Graph benchmark , using 1,315 questions . Compared with using GPT-4o to process the original images directly, I observed approximately: ~95% lower token usage roughly the… 18 r/LocalLLaMA community 1mo ago Can Some Generous Soul Grant Me My Wish? Please, Oh gentle soul, please beautiful soul, please kindhearted soul, with serious hardware and time to kill, please benchmark Unsloth Qwen3.8-27B-Q8_K_XL against Qwen3.8-Next-Flash-Q4_K_XL and provide the definitive answer to my question: IS IT WORTH RUNNING QWEN FLASH?… 14 r/LocalLLaMA community 1mo ago Higher acceptance length, slower prose: Ling’s n=1/2/3 MTP test on one Spark The missing control is visible in sudoingX’s Ling-3.0-flash benchmark graphics. The earlier table leaves Ling’s no-speculation baseline as “not measured.” The later code/prose graphic fills it in: about 23 tok/s without the drafter, against 40.9 on code and 38.7 on prose with… 31 r/LocalLLaMA community 1mo ago Why are the SOTA open-weight models scoring (relatively) low scores on AA-Omniscience Index https://preview.redd.it/qwmh23nfr3oh1.png?width=1062&format=png&auto=webp&s=4741852dc3f5e7d86ae85281b089b2a325a1804a I mean they aren't that low but seeing them much lower than Gemini-3.* flash surprises me   submitted by   /u/the-grand-finale [link]   [comments] 24 r/LocalLLaMA community 1mo ago tencent/EVIE-8B and EVIE-4.5B (High-Capacity Visual Document Retrieval) https://preview.redd.it/3p6234jzk2oh1.png?width=900&format=png&auto=webp&s=b9eab3351d6c6dbd4d5b3fd677a6c40b57f18167 https://huggingface.co/tencent/EVIE-8B 🌟 Highlights SOTA Retrieval Performance : 66.75 nDCG@10 on ViDoRe V3, delivering industry-leading visual document retrieval… 15 Hugging Face Daily Papers research 1mo ago Bilevel Coordinated Reflection: A Game-Theoretic Approach to Multi-Agent LLM Systems Abstract The study formalizes multi-agent LLM coordination via bilevel games and stochastic memory reflection, introducing a grounded evaluation gate and SRMA algorithm with convergence guarantees, validated on SWE-bench. Generated by thinkingmachines/Inkling-Small Multi-agent… 4 r/MachineLearning community 1mo ago Measuring LLM performance drift: observations and methodology from 31,352 repeated benchmark measurements [D] One thing that has bothered me about LLM benchmarks for a while is that most of them are essentially snapshots. A model is evaluated, a score is published, and we tend to talk about that score as if it describes a relatively stable object. But with API-served models, the thing… 21 Hugging Face Daily Papers research 1mo ago τ^τ-Bench: An Environment for End-To-End, Realistic Agent Construction Abstract The τ^τ-bench benchmark evaluates coding agents on building real-world customer-service agents from business records, client requirements, and production APIs, revealing substantial gaps versus expert performance. Generated by thinkingmachines/Inkling-Small LLM agents… 10 r/LocalLLaMA community 1mo ago Lit Review on Benchmarking LLMs Running in your phone!: MobileWorld: Benchmarking Autonomous Mobile Agents in Agent-User Interactive and MCP-Augmented Environments Back to reading about LLMs as agents on your phone doing GUI tasks! This time I read about MobileWorld: Benchmarking Autonomous Mobile Agents in Agent-User Interactive and MCP-Augmented Environments and this paper forms the basis of the benchmark I am currently making because it… 24 r/LocalLLaMA community 1mo ago Benchmarking calories evaluation with LLMs I wanted a quick calories counter for myself, using LLMs to evaluate the calories from pictures of meals + descriptions. I needed to pick a model so I made a quick benchmark. The setup was: - Nutrition5k photos for photo + calories:… 29 Hugging Face Daily Papers research 1mo ago MaxKernel: Agentic Kernel Generation for TPUs Abstract MaxKernel is a multi-agent system that automates TPU kernel development through collaborative, autonomous, and graph-based search paradigms, achieving expert-level performance on diverse benchmarks. Generated by thinkingmachines/Inkling-Small Designing and authoring… 38 arXiv — Machine Learning research 1mo ago Mitra-v2 Technical Report arXiv:2609.04540v1 Announce Type: new Abstract: We introduce Mitra-v2, a tabular foundation model that delivers state-of-the-art performance on real-world classification and regression problems, from credit-risk scoring and clinical prediction to equipment-failure detection and… 9 arXiv — Machine Learning research 1mo ago Physics-Aware Random Walk Fingerprints for Scalable Power Grid Graph Classification arXiv:2609.04943v1 Announce Type: new Abstract: Recent benchmarks such as PowerGraph provide large collections of power-grid graphs for cascading-failure classification. Graph neural networks (GNNs) achieve strong predictive performance on this task, but typically require… 28 arXiv — Machine Learning research 1mo ago Phase Transition Frequency as a Training Time Predictor of Test Accuracy in ResNets arXiv:2609.05194v1 Announce Type: new Abstract: The number of discrete class-separability jumps observed during ResNet finetuning is examined empirically as a predictor of final test accuracy. Across 75 experiments spanning four benchmarks (CIFAR-10, CIFAR-100, TinyImageNet, and… 18 arXiv — Machine Learning research 1mo ago Corporate-Family Resolution Is Not a String-Matching Problem: A Public Benchmark Stratified by Name Visibility arXiv:2609.04269v1 Announce Type: cross Abstract: Deciding whether two supplier records belong to the same corporate family is a prerequisite for spend consolidation, credit exposure aggregation and sanctions screening. It is usually treated as entity matching, but the tasks… 22 arXiv — NLP / Computation & Language research 1mo ago You Really Didn't Get That? Benchmarking Social Pragmatic Inference for Indirect and Playful Chinese Online Comments arXiv:2609.04384v1 Announce Type: new Abstract: Chinese online comments often convey social meaning through indirect and playful language that is hard to interpret without context. Existing evaluations largely organize items around predefined phenomena or controlled pragmatic… 12 arXiv — NLP / Computation & Language research 1mo ago TRILOGUE: A Trilingual Spoken Dialogue Fact-Checking Benchmark with Evidence and Paired Audio arXiv:2609.04452v1 Announce Type: new Abstract: Modern misinformation is often heard before it is read, yet fact-checking systems are still evaluated mainly on clean written claims. Spoken dialogue remains different even when systems operate on transcripts: claims may be… 11 arXiv — NLP / Computation & Language research 1mo ago PetQA: Benchmarking Veterinary Knowledge and Clinical Reasoning arXiv:2609.04598v1 Announce Type: new Abstract: We introduce PetQA, a Korean long-form question-answering (QA) benchmark for evaluating veterinary knowledge and clinical reasoning in large language models (LLMs) and large vision-language models (LVLMs). PetQA contains 10,076… 5 arXiv — NLP / Computation & Language research 1mo ago ConsensusBench: Benchmark of Consensus Nodes for LLM Reasoning via Outcome Reward Densifying arXiv:2609.04648v1 Announce Type: new Abstract: Reinforcement learning (RL) has become one of the primary paradigms for reasoning enhancement of large language models (LLMs). In particular, Group Relative Policy Optimization (GRPO) and related algorithms have demonstrated strong… 13 arXiv — NLP / Computation & Language research 1mo ago Knowing What Not to Answer: Selective Non-Compliance in Vision-Language Models arXiv:2609.04720v1 Announce Type: new Abstract: Vision-language models (VLMs) are expected to respond helpfully to appropriate requests while withholding compliance with requests that are incorrect, unsafe, infeasible, or unanswerable. However, existing benchmarks predominantly… 6 arXiv — NLP / Computation & Language research 1mo ago MMTClinic: Multimodal, Multilingual Time Series Question Answering and Reasoning Benchmark for Clinical Domain arXiv:2609.04842v1 Announce Type: new Abstract: Time-series data in clinical settings is crucial for capturing dynamic changes in a patient's health over time, enabling timely diagnosis, personalized treatment, and early detection of critical events. However, the development of… 27 arXiv — NLP / Computation & Language research 1mo ago Discourse Dependency: A Continuous Criterion for Translation Difficulty arXiv:2609.04959v1 Announce Type: new Abstract: Recent calls for harder machine translation benchmarks have not clarified what difficulty should mean. We argue that one meaningful and currently unmeasured axis is referential reach, the distance a segment must look back into its… 21 arXiv — NLP / Computation & Language research 1mo ago WearableQA: A Benchmark for Health Reasoning over Real-World Wearable Data arXiv:2609.05405v1 Announce Type: new Abstract: Recent advances in wearable sensing enable continuous monitoring of physiological and behavioral signals, yet existing benchmarks rarely evaluate whether AI systems can reason over a real user's longitudinal wearable record. We… 6 Page 10 of 10 · 500 articles ← Newer
