News / #benchmark Tag Benchmark 500 articles archived under #benchmark · RSS Sign in to follow r/LocalLLaMA community 8h ago The AI race: China's open-weight models trail US closed models by just 7 months Source: whatmodel.sh , an open source AI leaderboard I built (and launched here yesterday). Every point comes from published benchmark results: 22,113 of them across 987 benchmarks, from 1,011 sources, each one linked to where it was published. This chart uses data for the text… 35 Latent.Space news-outlet 15h ago [AINews] AI Engineer NYC: Gemini 4 Argon, Vals AI Benchmarks Gemini 4 Argon went GA today, but we already gave it title story 2 weeks ago and most benchmarks, like CursorBench, simply independently confirm their claims. 25 TechCrunch — AI news-outlet 1d ago The final Disrupt Stage lineup: Three days of conversations you won’t hear anywhere outside of TechCrunch Disrupt 2026 The full TechCrunch Disrupt Stage lineup revealed, featuring Max Hodak, Mark Wahlberg, Benchmark partners, and more. Register now to save up to $100 on your pass and get a second at 50% off. 7 r/LocalLLaMA community 1d ago NVIDIA’s 9-Year-Old Tesla V100 Still Packs Some Inference Muscle; 4 GPUs On An NVLink Board Can Outperform M5 Ultra, DGX Spark In Various AI Benchmarks By Up To 230% Didn't realize the V100 were still so fast, anyone have first hand experience on these?   submitted by   /u/chemist_slime [link]   [comments] 14 The Information — AI news-outlet 1d ago Meta Reaches AI’s Crowded Frontier—Will Users Care? If it’s Monday, it must be time for another AI model release. This time, it was Meta Platforms’ turn, unveiling Muse Wonder, which it claims can “deliver state-of-the-art performance” on certain tasks. It was a triumphant moment for the company, whose AI efforts previously went… 36 The Information — AI news-outlet 1d ago Meta Reaches AI’s Crowded Frontier—Will Users Care? If it’s Monday, it must be time for another AI model release. This time, it was Meta Platforms’ turn, unveiling Muse Wonder, which it claims can “deliver state-of-the-art performance” on certain tasks. It was a triumphant moment for the company, whose AI efforts previously went… 16 r/LocalLLaMA community 1d ago Qwen 3.8 Flash Next Q3 S - Benchmark creation After running a benchmark test on Exo in Opencode and being amazed by the result, I thought I'd try other models on Opencode. Many were surprisingly terrible. Especially some mid level models. Earlier posts:… 14 TechCrunch — AI news-outlet 2d ago Meta releases its first frontier model, Muse Wonder Meta's Superintelligence Lab released a pair of new state-of-the-art models on Monday, Muse Wonder and Muse Wonder Full, the company's first attempt at a frontier model. 11 r/LocalLLaMA community 2d ago BrokenML: can agents fix broken ML jobs? Write-up: https://zhebrak.io/posts/brokenml/ Repo: https://github.com/zhebrak/brokenml I built a benchmark to determine the access level agents need to successfully fix broken ML workloads (PyTorch over Gloo on a two-node Slurm cluster or a llama.cpp server running in Docker… 30 r/LocalLLaMA community 2d ago I made an open-source AI leaderboard that shows the best models your machine can run Hey everyone, I made whatmodel.sh To use it, Enter your RAM and GPU See what models you can run locally with its rank and the command to run it Compare them with API models on the same score Building it made me realise a 32GB Mac can now run a model that scores on-par with… 11 r/LocalLLaMA community 2d ago Decision 3.0: open, multimodal decision models from 0.6B to 27B. Ranks #1 on both Jev Decision Index boards With the vLLM Semantic Router team we released Decision 3.0 (d3) , our new SOTA decision model family ranging from 0.6B to 27B, Apache-2.0, all reading text, images and video. How it works - Built on Qwen3.5 (0.8B / 2B / 4B / 9B) and Qwen3.8-27B. The parameter counts include the… 38 arXiv — Machine Learning research 2d ago A Categorical Atlas of Parameter-Efficient Fine-Tuning: Cones, Orbits, and Gauges arXiv:2610.12728v1 Announce Type: new Abstract: Parameter-efficient fine-tuning has hundreds of methods, usually compared by parameter count and benchmark score. We compare them as maps. A pretrained checkpoint is a base point in weight space, and a weight-space method is a… 21 arXiv — Machine Learning research 2d ago In-Context Learning for System Identification: Probabilistic Formulation, Long-Sequence Handling, and Benchmark Validation arXiv:2610.13411v1 Announce Type: new Abstract: Traditional system identification typically involves two steps: initially, training data is processed to estimate a model of the system of interest; then, the model is used to predict outputs for new input queries. In contrast,… 16 arXiv — Machine Learning research 2d ago PPG-Bench: A Unified Benchmark for PPG Foundation Models arXiv:2610.13435v1 Announce Type: new Abstract: Photoplethysmography (PPG) is increasingly recorded by wearable and clinical devices at enormous scale, and a growing number of PPG foundation models (FMs) have been developed to compute reusable representations for various… 27 arXiv — NLP / Computation & Language research 2d ago SWE-Duel: Towards a Self-Scaling Arena for Coding Agents arXiv:2610.12569v1 Announce Type: new Abstract: Static coding benchmarks are fixed at release. Over time their tasks leak into training data, and stronger agents eventually saturate them. We argue that a benchmark should instead aim to be self-scaling: its tasks and oracles are… 19 arXiv — NLP / Computation & Language research 2d ago GeopoliticalQA: Benchmarking Retrieval and Generation over Intelligence Reports arXiv:2610.12617v1 Announce Type: new Abstract: Geopolitical and strategic intelligence reports constitute a rich source of expert knowledge and have increasing value in the current political climate, but their heterogeneity and analytical complexity pose challenges for… 35 arXiv — NLP / Computation & Language research 2d ago perturber: A Stateless Service for Semantics-Preserving Text Perturbations arXiv:2610.12667v1 Announce Type: new Abstract: Large language model (LLM) evaluations on clean benchmarks overstate model robustness: real inputs arrive perturbed, and these meaning-preserving variants routinely move model answers. Studying this systematically requires applying… 21 arXiv — NLP / Computation & Language research 2d ago GeoFitBench-LLM: A Benchmark for Assessing Large Language Models' Capabilities in Geospatial Data Requirements Elicitation for Geo-Analytic Tasks arXiv:2610.12669v1 Announce Type: new Abstract: Large language models are becoming central to GeoAI systems, yet their evaluation rarely considers the elicitation of geospatial data requirements. This omission is critical as retrieval oriented geo-analytic tasks cannot be… 19 arXiv — NLP / Computation & Language research 2d ago Unsupervised Multimodal Reward Modeling arXiv:2610.12707v1 Announce Type: new Abstract: Vision-language reward models are trained on annotated preference pairs, which are expensive to collect, can disagree, and are ranked by benchmark accuracy. We build one with no human preference label: a linear head on an… 22 arXiv — NLP / Computation & Language research 2d ago MADS: Multi-Agent Dialog Simulation for Robust Conversational Agents arXiv:2610.12757v1 Announce Type: new Abstract: Task-oriented conversational agents remain fragile under realistic conditions, especially when users exhibit non-cooperative behavior. Existing function-calling benchmarks underrepresent adversarial conversation trajectories,… 4 arXiv — NLP / Computation & Language research 2d ago When Charts Deceive Web Agents: A Paired Benchmark for Misleading Visualization-Grounded Tasks arXiv:2610.12823v1 Announce Type: new Abstract: Web agents increasingly browse, interpret, and act on web content on behalf of users. In many realistic workflows, however, the visual evidence they consume is not neutral: charts and dashboards can be misleading, shaping judgments… 18 arXiv — NLP / Computation & Language research 2d ago Evaluating LLM Creativity through Long-Tail Performance arXiv:2610.12998v1 Announce Type: new Abstract: Creativity benchmarks for large language models (LLMs) vary widely in task and metric design. This raises the question of whether they capture a common notion of creativity. We propose that this shared target can be formalized as… 14 arXiv — NLP / Computation & Language research 2d ago CoBe: A Benchmark For Conversational Counterfactual Text Editing arXiv:2610.13057v1 Announce Type: new Abstract: Counterfactual text editing is the ability to revise a given text in light of a hypothetical change while preserving facts not causally affected by that change. When asked what would have happened if a doctor had not recommended… 32 arXiv — NLP / Computation & Language research 2d ago Measuring Artifact Reliance in Machine-Generated Text Detectors arXiv:2610.13583v1 Announce Type: new Abstract: Detectors of machine-generated text score almost perfectly on scientific-literature benchmarks, implying we can reliably distinguish with these detectors human and machine-generated documents. We looked at ten scientific-document… 30 arXiv — NLP / Computation & Language research 2d ago BLiMPPI: Benchmark of Linguistic Minimal Pairs for Polarity Items arXiv:2610.13607v1 Announce Type: new Abstract: Negative polarity items (NPIs), such as "anymore" and "ever", are grammatical only in select environments, predominantly in the context of negation. Because their grammaticality depends on both syntax and semantics, they have a… 18 arXiv — NLP / Computation & Language research 2d ago SpireBench: Evaluating Long-Horizon Strategic Play of Agentic Models with Slay the Spire arXiv:2610.12596v1 Announce Type: cross Abstract: Agentic large language models now tackle complex tasks, yet strategic reasoning over long horizons is under-studied: most agentic benchmarks are short and episodic, and performance on long tasks is hard to dissect into stored… 25 arXiv — NLP / Computation & Language research 2d ago MVGraph: Benchmarking LVLMs on Evidence Composition across Multi-View Visual Graph arXiv:2610.12961v1 Announce Type: cross Abstract: We introduce MVGraph, a benchmark for evaluating large vision-language models (LVLMs) on multi-graph reasoning where the evidence required by a query is distributed across multiple interrelated visual graph views. Although such… 31 arXiv — NLP / Computation & Language research 2d ago SmartSalesBench: A Resistance-Aware Multi-Turn Benchmark for Evaluating LLM Sales Agents arXiv:2610.12975v1 Announce Type: cross Abstract: We present SmartSalesBench, the first multi-turn conversational benchmark explicitly designed to evaluate large language model (LLM) sales agents against psychologically grounded, resistant buyer personas. Existing task-oriented… 14 arXiv — NLP / Computation & Language research 2d ago Benchmarking Embedding Geometries and Clustering Algorithms for Unsupervised Patent Claim Classification arXiv:2610.12997v1 Announce Type: cross Abstract: Prior-art retrieval systems are structurally organized around classification codes, making cross-code discovery difficult. This paper asks if an unsupervised representation trained on patent text can recover relevant connections… 37 arXiv — NLP / Computation & Language research 2d ago Les Mis\'erables Dialogue: A film-based network representation of social communication arXiv:2610.13148v1 Announce Type: cross Abstract: In this work, we revisit the classic Les Mis\'erables benchmark network from the perspective of social communication. Using the 1998 film adaptation of the novel, we construct the publicly available Les Mis\'erables Dialogue… 25 r/MachineLearning community 3d ago [P] Pecision models that score every allowed label from the logits: Jebadiah v2.1 (27B, 9B), open weights and self-run benchmark results [P] I've been building open models that treat a decision as a closed-set scoring problem rather than text generation. The input is structured context plus a typed question with a fixed set of options. The output is a probability for each option, taken from the candidate-label… 23 r/LocalLLaMA community 3d ago Qwen3.8-Flash-Next Strata pi beat Claude-code Opus 5.5 on a 5090 EDIT to address that the title is misleading: Qwen3.8-Flash-Next Strata pi beat Claude-code Opus 5.5 on a 5090 on a benchmark dedicated to compare model for everyday tasks. not to discriminate frontier model. --- Original post: I'm using Qwen3.8-Flash-Next on Strata with pi as… 16 r/LocalLLaMA community 4d ago Strata with Qwen3.8 Flash Next UD-Q4_K_XL Been seeing quite a few posts about Strata lately, so I figured I'd give it a shot on my RTX PRO 6000 and I am very impressed. Most of the benchmarks I've seen are using IQ2 or IQ3 quants, so I wanted to see how Unsloth's UD-Q4_K_XL performs instead. Setup: GPU: 1x RTX PRO 6000… 35 llama.cpp releases dev-tools 5d ago b11535 llama-bench: respect -fitc if bigger than required benchmark size ( #28331 ) Assisted-By: opencode,llama.cpp,Qwen3.6-35B-A3B,Qwen3.8-27B Website: https://llama.app Attestations: https://github.com/ggml-org/llama.cpp/attestations/54430110 macOS/iOS: macOS Apple Silicon (arm64)… 30 r/LocalLLaMA community 5d ago Just another purely open-weight models benchmark Live results, coding benchmarks included, agentic benchmarks included, domains and other classifications filters included. (Spoiler: DeepSeek-V4-Vision-Exp rules, but other models have their rule areas): https://beta.locallm.top Evaluated by domain experts (my friends mostly;… 14 r/LocalLLaMA community 5d ago GLM 5.3 Flash opensource @ the top of Artificial Analysis Cyber Index over Claude. It seems Dario's "too powerful for you users" strategy is paying off: we have two open models at the top of the leaderboard, surpassing every single model from Anthropic. Open source prevails. Even Mistral Large 4 is better!   submitted by   /u/LegacyRemaster [link]… 8 r/LocalLLaMA community 5d ago Stepping away from Benchmarks and Code, what models are you using for Creating writing projects All anyone ever talks in this sub is about x new model having x benchmark scores and how well it can code. Or some new tech to increase t/s for qwen. So I wanted to go back to the roots of this sub and talk about how well local models can write. I am not going to mention closed… 35 r/MachineLearning community 5d ago I built MaRN: a PyTorch library for training neural networks through low-dimensional parameter mappings [P] I built MaRN (Mapping Networks) , a PyTorch library that lets you optimize a compact latent representation instead of directly training every model parameter. Some results from my current benchmarks: MNIST CNN: 107,998 → 1,872 trainable parameters ( 57.7× reduction ), with… 29 arXiv — NLP / Computation & Language research 5d ago Conversational Task Disambiguation over Tabular Data: Leakage-Aware Formulation, Benchmark Suite, and Training arXiv:2610.10740v1 Announce Type: cross Abstract: Conversational task disambiguation over tabular data uses dialogue to resolve missing information about a user's intended task before producing a solution over tables or databases. Existing evaluation and training lack a… 18 arXiv — Machine Learning research 5d ago Controlled Acquisition and Abstention in Three-Channel Score Conflicts arXiv:2610.10808v1 Announce Type: new Abstract: When audio, video, and text disagree, accuracy alone does not show whether to acquire another source or abstain. We study these choices in a controlled three-score benchmark: a policy observes two signed scores, may request the… 5 arXiv — Machine Learning research 5d ago Evaluating Rubric Generation with Interventional Transfer arXiv:2610.10809v1 Announce Type: new Abstract: Instance-specific rubrics are common in AI benchmarks where reliable evaluation requires specific expert knowledge. This approach is difficult to scale, prompting research into the generation of rubrics with large language models… 4 arXiv — Machine Learning research 5d ago How Hackable Is Your Speech Quality Metric? A Corrected Protocol, a Benchmark, and What Patching Buys arXiv:2610.10899v1 Announce Type: new Abstract: Speech quality predictors are increasingly used as rewards, yet no agreed measure of their hackability exists. The usual measurement has two flaws. First, the perturbation reaches the predictor through a processing chain -- here a… 31 arXiv — Machine Learning research 5d ago RH-Detect: A Unified Benchmark for Reward Hacking Detection arXiv:2610.10947v1 Announce Type: new Abstract: Reward hacking, where a model exploits an evaluation signal without completing the intended task, threatens the reliability of deployed language model systems. Existing datasets use different labels, response formats, and metadata… 16 arXiv — Machine Learning research 5d ago CityDeploy-Bench: Benchmarking Physics-Grounded Spatial Set Planning for Multi-Transmitter Network Deployment arXiv:2610.11065v1 Announce Type: new Abstract: Automating city-scale wireless deployment remains challenging under complex urban propagation and network-wide interference. We introduce \textbf{CityDeploy-Bench}, a benchmark that reframes multi-transmitter deployment as… 7 arXiv — Machine Learning research 5d ago RideBench: A Large-Scale Exogenous-Aware Benchmark for Ride-Hailing Time Series Forecasting arXiv:2610.11164v1 Announce Type: new Abstract: We release Ride-Hailing, a large-scale ride-hailing time series dataset synthesized from DiDi's marketplace data across 200 spatial areas. Ride-Hailing spans four consecutive years at half-hourly granularity and covers three… 36 arXiv — Machine Learning research 5d ago Generative Adversarial Loops arXiv:2610.11458v1 Announce Type: new Abstract: AI research progress can be viewed as the interaction between two processes: benchmark creation and method discovery. Historically, both were driven by human intelligence. However, recent advances in AI have accelerated automated… 27 arXiv — NLP / Computation & Language research 5d ago When Citations Mislead? A Claim-Level Benchmark for Legal Hallucination Detection arXiv:2610.10971v1 Announce Type: new Abstract: Large language models are increasingly used in legal research and drafting, but they can still produce claims that sound convincing without being supported by the cited source. We introduce PARCEL, a benchmark for checking whether… 23 arXiv — NLP / Computation & Language research 5d ago Clinician use of language models diverges from how the models are evaluated arXiv:2610.11069v1 Announce Type: new Abstract: Large language model (LLM) assistants are being deployed to clinicians across health systems, and judgments about their readiness rest largely on benchmark scores, most of them derived from examination questions or curated cases. A… 23 arXiv — NLP / Computation & Language research 5d ago SWE-Journey: Towards More Realistic Evaluation of Coding Assistants through Long-Horizon, Multi-Turn Interaction arXiv:2610.11559v1 Announce Type: new Abstract: Coding assistants such as Claude Code and Codex have become a major application of LLM agents, yet existing benchmarks remain far from real-world use, particularly in task horizon and interaction length. Code assistants require… 35 arXiv — NLP / Computation & Language research 5d ago Large Language Model Turnover Undermines Screening for Artificial Intelligence-Assisted Scientific Writing arXiv:2610.11599v1 Announce Type: new Abstract: Journals and conferences have begun to screen submitted manuscripts for text written using large language models (LLMs). The reliability of this screening rests on benchmark evaluations against a fixed set of LLM versions, while… 21 Page 1 of 10 · 500 articles Older →
