News / #benchmark Tag Benchmark 500 articles archived under #benchmark · RSS Sign in to follow TechCrunch — AI news-outlet 25d ago Vals, backed by Andreessen Horowitz, is looking to become the gold standard for AI benchmarking Vals AI is hoping to make AI benchmarking a more neutral and trustworthy resource in a world increasingly inundated by AI models. 32 r/LocalLLaMA community 25d ago Prefil of local models vs opus and astra Why does no one talk about what the prefil speeds of these API providers are vs running locally. People with sparks or strix halos only seem to focus on decode without considering how much slower it is because of slow pp. Are there any benchmarks / figures of how fast the APIs… 8 r/LocalLLaMA community 26d ago M5 Ultra and M6 Chip Benchmark Results Reveal Graphics Performance First benchmarks   submitted by   /u/DustNearby2848 [link]   [comments] 18 NVIDIA Developer Blog official-blog 26d ago Benchmarking LLM Inference at Scale with AIPerf You’re deploying a model on a system. It starts up, prompts are getting responses. Now the hard question: Is this fast? Your instincts might lead you to send... 24 r/LocalLLaMA community 26d ago We benchmarked 24 LLMs against human writers on 475 creative writing prompts We just released the first version of our Creative Writing benchmark, comparing 24 LLMs against human writers across 475 writing prompts. Creative writing is subjective, so the rankings aren't meant to predict what any one person will prefer. Instead, they predict what a large… 26 r/LocalLLaMA community 26d ago Prism-ML Bonsai 2 Joins Our Qwen3.8 Quantization Comparison Hey r/LocalLLaMA , Prism-LM recently released its Bonsai 2 QAT models based on Qwen3.8, and they quickly gained traction. In our evaluation, the models strike a strong balance between throughput and quality, reaching roughly 91.5% on our composite benchmark . We wanted to see… 24 r/LocalLLaMA community 26d ago bonsai's document reveal how much cherry picked their headlines are bonsai claim 98.2% intelligent retained, but their own documents show Ternary Bonsai 2 27B reaches 52.8 and 60.8, respectively, compared with 69.7 and 80.6 for Qwen3.5-27B, retaining roughly three quarters of the full-precision performance on both benchmarks. that qwen3.5 is a… 19 r/LocalLLaMA community 26d ago Made the horizontal open-source model for Jev with RLCD, and it surpasses all the Jev benchmarks. HF space, benchmark, model, repo Thanks for the exceptional support ( https://www.reddit.com/r/LocalLLaMA/comments/1wijo3e/i_literally_built_the_jev_architecture_one_year/ ) and for the dozens of requests to make a generic model, run benchmarks, and create an HF space so anyone can test it. So here you go,… 30 arXiv — Machine Learning research 26d ago Is It Still Worth Training a Classical Model in the Era of LLMs? A Crossover Benchmark on Tabular Data arXiv:2609.20218v1 Announce Type: new Abstract: Large language models can label a tabular row from a plain-English description with no training - a capability now shipping in mainstream spreadsheet tools such as Microsoft Copilot in Excel and Anthropic's Claude for Excel -… 27 arXiv — NLP / Computation & Language research 26d ago Neo-Classic: A Benchmark for Evaluating Linguistic-Aesthetic Reasoning in Classical Chinese Poetry arXiv:2609.19154v1 Announce Type: new Abstract: While Large Language Models (LLMs) achieve high accuracy on established Classical Chinese Poetry benchmarks, it remains challenging to distinguish transferable Linguistic-Aesthetic Reasoning from reliance on familiar pre-training… 4 arXiv — NLP / Computation & Language research 26d ago V\={a}kQA: A Benchmark and Evaluation Study for Telugu Spoken Factoid Question Answering arXiv:2609.19879v1 Announce Type: new Abstract: Question answering has advanced rapidly with large language models, but predominantly for high-resource languages, in both text and spoken settings. Spoken question answering (SQA) benchmark for Telugu remains unexplored, and the… 11 arXiv — NLP / Computation & Language research 26d ago PetriBench: Benchmarking LLM Reasoning over Dynamic State Spaces arXiv:2609.19883v1 Announce Type: new Abstract: Characterizing LLM reasoning remains an open challenge, as many existing benchmarks isolate specific reasoning skills, rely on external knowledge, or are costly to extend. We introduce PetriBench, a compact, fully self-contained,… 8 arXiv — NLP / Computation & Language research 26d ago KoNeoBench: A Curated Evaluation Dataset for LLM Understanding of Korean Neologisms arXiv:2609.19916v1 Announce Type: new Abstract: Large language models (LLMs) are typically evaluated on static benchmarks, even though natural language constantly evolves through newly emerging words and meanings. Existing Korean benchmarks are centered on established vocabulary… 32 arXiv — NLP / Computation & Language research 26d ago Before the Arrest: Benchmarking LLMs on Criminal Profiling from Incomplete Evidence arXiv:2609.19965v1 Announce Type: new Abstract: Large Language Models (LLMs) are increasingly applied to legal and criminal justice tasks, yet existing work focuses almost exclusively on post-arrest scenarios where the suspect's identity is already known, leaving the critical… 28 arXiv — NLP / Computation & Language research 26d ago Benchmarking LLM Compliance with China AI Generated Content Regulations arXiv:2609.19989v1 Announce Type: new Abstract: The widespread adoption of LLMs has led to escalating content compliance risks. Prior works have contributed to addressing these risks in the English context, downplaying the complexity of Chinese language content. This paper… 36 arXiv — NLP / Computation & Language research 26d ago SAFARI: An Industrial Benchmark for LLM-Assisted Hazard Analysis and Risk Assessment arXiv:2609.20584v1 Announce Type: new Abstract: Large language models (LLMs) are increasingly considered for safety-critical engineering, yet their reliability in regulated functional-safety workflows remains underexplored. We introduce SAFARI (Safety-Aware Functional Automotive… 22 arXiv — NLP / Computation & Language research 26d ago What Do We Expect from LLMs? Mapping the Design of LLM Benchmarks arXiv:2609.19182v1 Announce Type: cross Abstract: Benchmarks are central to how progress in large language models (LLMs) is assessed and communicated. Yet model rankings alone reveal little about how evaluation requirements themselves are changing. The expanding variety of… 38 arXiv — NLP / Computation & Language research 26d ago From Intent to Action: Benchmarking LLM Safety in Vehicle Voice Command Authorization arXiv:2609.19630v1 Announce Type: cross Abstract: Large language models (LLMs) are increasingly integrated into vehicle voice assistants. But linking natural-language requests to vehicle functions creates a safety-critical authorization problem. Before executing a command, the… 25 Vercel — AI dev-tools 27d ago Run Terminal-Bench and other Harbor evals on Vercel Sandbox You can now run Harbor evals on Vercel Sandbox. Harbor is the open-source harness behind Terminal-Bench , whose registry includes many other benchmarks such as SWE-bench, tau3-bench and OSWorld. Pass --env vercel to harbor run and each trial executes in its own isolated… 18 r/LocalLLaMA community 27d ago First M5 Ultra benchmarks just saw some benchmarks on the omlx website for the m5 ultra (don’t know how official they are but they seem reasonable): Link For Qwen 3.8 27B q4 it gets 50 tok/s th and 1800 tok/s pp at8k context and without mtp. Seems very promising!   submitted by   /u/Ashefromapex… 8 r/LocalLLaMA community 27d ago AndroidLife: Can an AI agent survive a day in the life of a real user? Qwen3.8-27b run: 56.7% SR I let AI run my phone 60 real tasks, back to back, on the OnePlus I use every day Best text model still failed 43% of them Peak chip temp 98.2 C 69% of the battery gone The benchmark is AndroidLife, and this is the first of 11 models, qwen3.8-27b from Alibaba, running in text… 24 arXiv — Machine Learning research 27d ago EdgeReMIND: A Scalable, Top-Ranked Memorization Baseline for Temporal Multi-Relational Link Prediction arXiv:2609.17916v1 Announce Type: new Abstract: Temporal link prediction on the Temporal Graph Benchmark 2.0 (TGB 2.0) faces a scalability ceiling: on the benchmark's three largest datasets, every existing embedding method runs out of memory or exceeds the time budget. These… 13 arXiv — Machine Learning research 27d ago iMINDBench: iEEG Multi-Institution Neural Decoding Benchmark arXiv:2609.18104v1 Announce Type: new Abstract: Intracranial electroencephalography (iEEG) is widely used to record electrical activity directly from electrodes inside the human brain, making it an attractive modality for neural decoding. However, progress in iEEG decoding,… 6 arXiv — Machine Learning research 27d ago Reliable Virtual Sensing: A Multi-Domain Benchmark for Robustness Under Sensor Failures arXiv:2609.18396v1 Announce Type: new Abstract: Virtual sensing, the estimation of hard-to-measure quantities from available sensor measurements, is a critical enabler for control and monitoring in cyber-physical systems. However, when sensors fail, learning-based predictors can… 15 arXiv — Machine Learning research 27d ago StableEval Arena: A Cost-Aware Agentic Benchmark for Stablecoin Price Stability Prediction arXiv:2609.18949v1 Announce Type: new Abstract: We introduce StableEval Arena, a cost-aware benchmark framework for evaluating agentic AI systems on stablecoin peg-risk prediction. StableEval Arena evaluates LLM-backed agentic systems on diagnosing peg stress and forecasting… 14 arXiv — NLP / Computation & Language research 27d ago From Pixels to Pairs: A Comprehensive Benchmark of LLM-Based Key-Value Extraction in Noisy Document Settings arXiv:2609.17538v1 Announce Type: new Abstract: Large language models (LLMs) are increasingly used for structured information extraction from documents, yet their behavior under realistic OCR noise remains poorly understood. We present a systematic benchmark of open-source… 7 arXiv — NLP / Computation & Language research 27d ago English Word Sense Disambiguation in 2026: When the Labels Become the Bottleneck arXiv:2609.17554v1 Announce Type: new Abstract: In English all-words word sense disambiguation (WSD), the labels, not the models, have become the bottleneck: frontier LLMs are accurate enough that the errors surviving in the gold standard decide benchmark rankings -- in the test… 31 arXiv — NLP / Computation & Language research 27d ago DualSQL: Text-to-SQL with Multi-Agent Reinforcement Learning arXiv:2609.18135v1 Announce Type: new Abstract: State-of-the-art Text-to-SQL systems are typically multi-agent pipelines centered around two fundamental tasks: schema linking and SQL generation. However, existing work trains separate models for each task, failing to leverage the… 24 arXiv — NLP / Computation & Language research 27d ago TeochewBench: A Human-Reviewed Benchmark for Teochew Hanzi Translation arXiv:2609.18156v1 Announce Type: new Abstract: Teochew has a substantial speaker community and exhibits distinctive lexical, syntactic, and pragmatic features, yet textual resources for evaluating large language models remain limited. We present TeochewBench, a human-reviewed… 28 arXiv — NLP / Computation & Language research 27d ago Behavior2Value: Benchmarking and Empowering LLMs for Consumer Value Measurement from E-commerce Behaviors arXiv:2609.18203v1 Announce Type: new Abstract: Human values are deep motivational orientations that shape human behaviors. In e-commerce, they reveal the stable drivers behind users' purchase decisions. Compared with short-term interests, consumer values better explain how… 26 arXiv — NLP / Computation & Language research 27d ago I code or AI code: A comparative evaluation of AI-rated scores in classroom observations arXiv:2609.18274v1 Announce Type: new Abstract: Classroom observations are widely recognized as a key tool for establishing benchmarks of education quality and guiding pedagogical improvement, yet they remain resource-intensive and dependent on trained observers. This study… 26 arXiv — NLP / Computation & Language research 27d ago Fallacy Benchmarks Measure Scheme Recognition, Not Fallacy Detection arXiv:2609.18644v1 Announce Type: new Abstract: Fallacy-detection benchmarks pair fallacy classes with a single "valid" or "none" class that takes everything data collection did not label as a fallacy. This construction is misleading: a classifier can learn cues that do well on… 24 arXiv — NLP / Computation & Language research 27d ago HearInContext: A Benchmark for Implicit Context in Speech Recognition arXiv:2609.18680v1 Announce Type: new Abstract: Contextual ASR can benefit from semantic cues or from target words explicitly provided in the context. We introduce HearInContext, a Mandarin--English benchmark that pairs shared synthetic speech with assistant replies supporting… 13 arXiv — NLP / Computation & Language research 27d ago A Scalable Framework for Automated NER Annotation Correction in Low-Resource Languages arXiv:2609.18739v1 Announce Type: new Abstract: Poor quality or noisy annotations in Named Entity Recognition (NER), as in any other NLP task, make it challenging to achieve state-of-the-art performance. In this paper, we present a multi-step framework to enhance the annotation… 29 arXiv — NLP / Computation & Language research 27d ago ReFigBench: Benchmarking Scientific Figure Reconstruction as Editable PowerPoint Artifacts arXiv:2609.18844v1 Announce Type: new Abstract: Multimodal coding agents are expected to turn visual inputs into usable artifacts, and they act through a harness, the layer of tools, context management, and execution environment around the model. Existing evaluations often… 21 arXiv — NLP / Computation & Language research 27d ago How Much is a Human Right Worth? ECtHR-NPD: A Benchmark for Predicting Non-Pecuniary Damage Awards arXiv:2609.18908v1 Announce Type: new Abstract: Existing legal benchmarks cover diverse tasks, while continuous monetary remedies remain comparatively underexplored. We introduce ECtHR-NPD, to the best of our knowledge, the first benchmark for predicting non-pecuniary damage… 37 arXiv — NLP / Computation & Language research 27d ago Beyond Outcomes: Dual-View Relational Learning for Efficient Agent Benchmarking arXiv:2609.18909v1 Announce Type: new Abstract: Agent benchmarks are substantially more costly to evaluate than conventional LLM benchmarks. Benchmark compression is therefore a natural solution, yet existing methods primarily model redundancy in task--model final-score… 32 arXiv — NLP / Computation & Language research 27d ago A Benchmark Suite and Ground-Truth Methodology for Formal Verification of IEC 61131-3 Ladder Diagram Programs arXiv:2609.18994v1 Announce Type: new Abstract: We present the first benchmark suite for formal verification of Programmable Logic Controller (PLC) programs that combines controlled ground truth with coverage of both textual (Structured Text, ST) and graphical (Ladder Diagram,… 30 arXiv — NLP / Computation & Language research 27d ago WordPolo: Evaluating Language Models Through Iterative Semantic Feedback arXiv:2609.19006v1 Announce Type: new Abstract: Large Language Models (LLMs) and Large Reasoning Models (LRMs) are typically evaluated on challenging benchmarks through dataset accuracy alone, providing no insight into the quality or faithfulness of their reasoning processes. We… 23 arXiv — NLP / Computation & Language research 27d ago Benchmarking Large Language Models for Biomedical Relation Extraction arXiv:2609.19071v1 Announce Type: new Abstract: Extracting SNP-phenotype associations from biomedical literature is vital but challenging. We benchmarked diverse NLP models, including MLMs, hybrid architectures, and state-of-the-art LLMs (Gemini 2.0, OpenAI O-series, Qwen,… 33 arXiv — NLP / Computation & Language research 27d ago Safety-Flag: A Unified Benchmark for the Reliability and Calibration of LLM Content Moderators arXiv:2609.19072v1 Announce Type: new Abstract: Large language models are increasingly used for content moderation, but most evaluations still report aggregate accuracy on individual benchmarks. We introduce Safety-Flag, which places seven widely used safety benchmarks… 31 arXiv — NLP / Computation & Language research 27d ago GraphEcho: Structural Redundancy and Evidence Provenance in LLM Graph Agents arXiv:2609.17695v1 Announce Type: cross Abstract: A large language model (LLM) agent can follow more graph paths without acquiring more independent evidence. GraphEcho tests whether agents mistake these repeated encounters for additional corroboration. The benchmark varies path… 13 Hugging Face Daily Papers research 27d ago VC-Attention: Value Smoothing and Softmax Casting for Low-bit Attention Abstract Diffusion Transformers deliver state-of-the-art video generation, but their long spatiotemporal sequences make attention the dominant deployment cost, and a deployable low-bit kernel must be accurate and fast. Accuracy is limited by outliers: a block's quantization… 4 NVIDIA Developer Blog official-blog 28d ago TensorRT Edge-LLM Completes the MLPerf Edge Agentic Benchmark 6.4x Faster on Jetson AGX Thor AI agents are moving from cloud data centers to vehicles, robots, and other edge devices. Unlike a chatbot that answers a single prompt, an agent works through... 14 r/MachineLearning community 28d ago GoBench: Evaluating LLMs on the game of Go [R] GoBench evaluates LLMs on 9x9 Go games against a ladder of KataGo opponents, from random to superhuman. It measures general reasoning ability, strongly correlates with ARC-AGI 2 (r=0.83 correlation), and remains highly unsaturated. GPT-6 Astra max achieves 2500 Elo, much lower… 7 r/LocalLLaMA community 28d ago Can current MiniCPM5-2B/any best SOTA under 10B + modern harness beat pre-March 2025 frontier models like Grok 3/GPT-4o? Hey experts! I genuinely have this question and would love a general consensus from people actually using these models. LLMs have advanced a lot in benchmarks and in practical usage. Even GPT-4o and Grok 3 were already enough for general chatting,search lookup, RP, etc. So if… 38 r/LocalLLaMA community 28d ago China's open-weight AI models are now just 4 months behind frontier US offerings, Mozilla report claims — models still lag in some benchmarks but are drastically cheaper to use   submitted by   /u/DustNearby2848 [link]   [comments] 38 r/LocalLLaMA community 28d ago Qwen3.8 Max (0902) scores 45 on the Artificial Analysis Intelligence Index, up 5 points in a month and back on top of China's leaderboard, nosing out GLM-5.3 (44.9) and Kimi K3 (43.8) https://preview.redd.it/17s4r20sgwph1.png?width=1265&format=png&auto=webp&s=1166261eb76a5675c7ff01b673edfe473cda08cc A 30-day upgrade on the 2.4T MoE takes the crown back from GLM-5.3, can't wait for Qwen 4.0   submitted by   /u/UmpireBorn3719 [link]   [comments] 23 r/LocalLLaMA community 28d ago [Release] SOTA GGUFs for Qwen3.8-Flash-Next: GSQ-RCO Providing Near Baseline Performance https://preview.redd.it/e5wn8eyh7vph1.png?width=1080&format=png&auto=webp&s=5faeec866acc9e8eca8dc9e84a6661bee7bcf954 https://preview.redd.it/8ov5gl8j7vph1.png?width=1080&format=png&auto=webp&s=440320ca4646ff7dceaf2aa5ac2ef76faeeba525 New Qwen3.8-Flash-Next quantization using… 23 r/LocalLLaMA community 28d ago qwen4exp: add hc ops by am17an · Pull Request #28901 · ggml-org/llama.cpp time to re-benchmark Qwen Flash Next again   submitted by   /u/jacek2023 [link]   [comments] 24 Page 7 of 10 · 500 articles ← Newer Older →
