News / #benchmark Tag Benchmark 500 articles archived under #benchmark · RSS Sign in to follow r/LocalLLaMA community 9d ago Smallest Jev-like model TinyDecide is 10M Jev-like mode with 10M parameters and fits in just ~6MB. Smaller than every model on the Decision Index leaderboard and it punches way above its size . It runs almost anywhere: in the browser, Node.js, Python, Rust, and even on an ESP32.… 7 TechCrunch — AI news-outlet 9d ago The final Disrupt Stage lineup: Three days of conversations you won’t hear anywhere outside of TechCrunch Disrupt 2026 The full TechCrunch Disrupt Stage lineup revealed, featuring Max Hodak, Mark Wahlberg, Benchmark partners, and more. Register now to save up to $100 on your pass and get a second at 50% off. 34 r/LocalLLaMA community 9d ago M5 Ultra 256 running GLM 5.3 Flash 68.8 tok/s Running oQ4e+MTP on oMLX 0.7.0, with still more to optimize. Prefill is 1,878 toks. I saw some other benchmarks below what id expect so i figured I would share.   submitted by   /u/cryotic [link]   [comments] 37 r/LocalLLaMA community 9d ago Which model, which harness? I have data for you. I keep testing harness & model setups on real tasks: basic math, vision, computer use (reading emails, navigating stores) and coding. I created a benchmark and a leaderboard. You can see it here: https://airbench.ai/leaderboard?k=poL It measure capabilities (a % of sucess on the… 9 arXiv — Machine Learning research 9d ago TRACE: A Reproducible Benchmark for Electricity Price Forecasting with Official Operational Text arXiv:2610.02256v1 Announce Type: new Abstract: Electricity price forecasting (EPF) supports scheduling, bidding, and risk management in electricity markets, yet existing benchmarks focus mainly on numerical inputs, leaving the forecasting value of forecast-time textual context… 6 arXiv — Machine Learning research 9d ago ArrivalBench: Agent-Generated Data Pipelines Are Correct Once and Wrong Under Time arXiv:2610.02363v1 Announce Type: new Abstract: Benchmarks for agent-generated data work grade a pipeline by running it once against a fixed snapshot. ArrivalBench instead re-executes the pipeline an agent leaves behind under adversarial but replayable delivery schedules (late,… 34 arXiv — Machine Learning research 9d ago Co-design Gym: A Unified Benchmark for Embodiment-Policy Co-optimization arXiv:2610.02366v1 Announce Type: new Abstract: Finding an optimal behaviour policy within a given environment is a widely studied problem in domains as diverse as games, robotics, energy infrastructure, communication networks, and multi-agent systems. Numerous benchmarks have… 9 arXiv — Machine Learning research 9d ago OpenGameEval: Benchmarking Agentic Programming and Exploration in a Stateful Game Engine arXiv:2610.02563v1 Announce Type: new Abstract: We present OpenGameEval, a benchmark and evaluation framework for agentic game development inside Roblox Studio. It runs language models as agents in reproducible, stateful game-engine sessions and scores each run with executable… 8 arXiv — NLP / Computation & Language research 9d ago HakemBench: A Turkish Benchmark of Typed Decisions arXiv:2610.02293v1 Announce Type: new Abstract: HakemBench is a Turkish benchmark of typed decisions, in which the model under test reads a text, a question and a fixed set of options and returns a probability for every option. Version 1.0 is released fully open under CC BY 4.0,… 38 arXiv — NLP / Computation & Language research 9d ago Finding the Move Is Not Winning the Game: XiangqiBench for Closed-Loop Evaluation of LLM Agents arXiv:2610.02425v1 Announce Type: new Abstract: Static evaluations credit a language model for naming the right move, but an agent must carry a plan through to a verified outcome while an opponent responds. We introduce XiangqiBench, an executable benchmark that measures this… 34 arXiv — NLP / Computation & Language research 9d ago CUEing User Simulators: Calibrated User Embeddings for Multi-Turn Benchmarking arXiv:2610.02460v1 Announce Type: new Abstract: Recent benchmarks rely on user simulators to evaluate AI agents in multi-turn interaction. While existing simulation techniques demonstrate surface fidelity to human style and behavior, ecologically valid interactive benchmarking… 12 arXiv — NLP / Computation & Language research 9d ago TPBench: A Turning-Point Benchmark for Dialogue Compression arXiv:2610.02736v1 Announce Type: new Abstract: A compressor can keep the facts of a dialogue and still drop the turn that changed them. A user corrects a price, reverses a choice, or adds a constraint. We call this failure turning-point eviction. One overall retention score… 35 arXiv — NLP / Computation & Language research 9d ago Output Language Confusion under Multilingual Prompt Contamination arXiv:2610.02926v1 Announce Type: new Abstract: Standard factual benchmarks assume clean monolingual prompts and exact-match scoring, two assumptions that break simultaneously in real-world multilingual deployment, from retrieval-augmented generation pipelines returning… 36 arXiv — NLP / Computation & Language research 9d ago OLMo-Detect: A Multi-Stage, Confounder-Controlled Benchmark for Membership Inference on Large Language Models arXiv:2610.02986v1 Announce Type: new Abstract: Membership inference on large language models (LLMs) aims to determine whether a given text sample was included in an LLM's training data, without access to its training corpus. Despite recent progress, existing benchmarks suffer… 38 arXiv — NLP / Computation & Language research 9d ago Ontological Instability and Statistical Amplification: The Paradox of "Humanizing" LLM-Generated Text arXiv:2610.03110v1 Announce Type: new Abstract: Supervised AI-text detectors report high benchmark accuracy, but it is not clear what their decisions are based on. We analyze a RoBERTa-based detector under semantic, structural, and tokenizer-level perturbations, using the M4… 4 arXiv — NLP / Computation & Language research 9d ago MintEval: Do LLMs Implement the Trading Strategy You Asked For? A Behavioural-Equivalence Benchmark for Natural-Language-to-Strategy Code arXiv:2610.03080v1 Announce Type: cross Abstract: Large language models are moving from producing trading signals to writing the code that executes them. The failure mode of the second role is silent: generated code runs, a backtest plots, yet the risk logic that the trader… 21 r/LocalLLaMA community 10d ago Benchmarking decision models is fun - Clef Q8 vs Jev   submitted by   /u/3VITAERC [link]   [comments] 36 r/MachineLearning community 10d ago Top ARC-ΑGI-3 scores on Kaggle just went from 7% to 56% [N] https://preview.redd.it/gkjgii48gfth1.png?width=575&format=png&auto=webp&s=4e1c35bdacf41d18a6409baedcddc9432ec79814 This happened over the past 30 days. So smallish local models (Kagglers can only use those), in a harness, just started beating average humans at a benchmark… 25 r/MachineLearning community 10d ago Nonobench: an open benchmark of 49 LLMs on nonogram puzzles, public and open source [P] Nonobench measures how well LLMs solve nonograms (picross). Each model gets the row and column clues once and returns the full grid. No tools, one attempt per puzzle. Method: - Standard mode: 30 puzzles from 5x5 to 15x15 (from the Nonograms dataset by Moyà-Alcover, CC BY 4.0). -… 22 r/MachineLearning community 10d ago Here are some pictures of a robot costume wearing high-specularity edge-case mirror suit, a dataset (425 RAW/JPEGs) for benchmarking CV & depth-estimation algorithms against extreme mirror reflections [D] Purpose-built to stress-test computer vision models, depth cameras, and spatial AI against severe specular glare and geometric reflections. This 425-asset production archive features a custom faceted mirror suit captured in high-contrast outdoor environments to trigger… 23 r/MachineLearning community 10d ago Jev: Not Frontier, But Still Worth Your Attention [R] "TypeSafe AI sells Jev as a frontier-class reasoner that cannot hallucinate, built by the co-inventor of ChatGPT - fast, and almost free. We ran it live on 16,379 benchmark requests, measured its latency and billing, and probed what it is underneath. The result is a smaller,… 15 r/LocalLLaMA community 11d ago Dual Radeon MI50 benchmarks Still don't have a good cooling solution, but here are few benchmarks. I lowered the power limit (TDP) to 145 watts each. I changed the firmware on one MI50 to activate the miniDP port. Did have to use xrandr to create a new mode so I could get 1920x1080 output. Each GPU has… 25 r/LocalLLaMA community 11d ago Qwen3.8-Flash-Next 177B running at 11–15 tok/s on a single RTX 5070 12GB + 32GB RAM DDR4 Benchmarking an LLM here with a NVIDIA RTX 5070 12 GB VRAM here I had been working on a llama.cpp based expert streaming setup for Qwen3.8-Flash-Next 177B (UD-IQ3_XXS) on Windows. Benchmark is about 11.5 tok/s, up from roughly 7 tok/s on the inherited setup. In normal… 35 arXiv — Machine Learning research 11d ago The Hidden Costs of 99% Accuracy: A Trustworthiness Audit of the Telco Customer Churn Benchmark arXiv:2610.00118v1 Announce Type: new Abstract: Customer churn prediction on the IBM Telco Customer Churn benchmark (n = 7,043) routinely reports test accuracies above 95%, with the most cited published study reporting 99.01%. We audit this benchmark for four trustworthiness… 11 arXiv — Machine Learning research 11d ago The Null Is the Hard Part: Exact Tests for Memorization in Generative Models arXiv:2610.00251v1 Announce Type: new Abstract: Memorization audits of generative models read similarity scores against thresholds, with no null distribution, and the conclusions they support can be wrong. By MemBench's rule, the benchmark's mitigations roughly halve Stable… 7 arXiv — Machine Learning research 11d ago Benchmarking System One decision models against trained classifiers and language models for automated decision gates arXiv:2610.00346v1 Announce Type: new Abstract: Software that hands branching decisions to a model needs a declared option and a probability it can threshold. Typed decision models, also called System One models, return such probabilities without generating text, while… 16 arXiv — Machine Learning research 11d ago M$^2$Weather: A Benchmark for Joint Multi-Station and Multi-Variable Weather Forecasting arXiv:2610.00370v1 Announce Type: new Abstract: Station weather forecasting is fundamentally shaped by both complex spatial dependencies across stations and strong physical coupling among weather variables. However, existing studies often consider these relationships separately… 28 arXiv — Machine Learning research 11d ago SimplexUQ: An Evaluation Framework and Benchmark for Conformal Uncertainty on Simplex-Valued Predictions arXiv:2610.00523v1 Announce Type: new Abstract: Conformal prediction guarantees marginal coverage, but a single calibration threshold can still spread that coverage unevenly, over-covering easy regions and under-covering hard ones. SimplexUQ is, to our knowledge, the first… 23 arXiv — Machine Learning research 11d ago ALER: Adaptive Learnable Experience Rewriting for Reinforcement Learning arXiv:2610.00592v1 Announce Type: new Abstract: In partially observable reinforcement learning (RL), a later observation can make stored information obsolete or change what it implies for the next decision. Memory architectures and benchmarks for RL mostly test retention, the… 27 arXiv — Machine Learning research 11d ago MIKASA-Robo-VLA: Benchmarking Memory in VLA Models for Long-Horizon Manipulation arXiv:2610.00604v1 Announce Type: new Abstract: Vision-language-action policies often see only one or a few recent frames, which makes it difficult to evaluate how they use information that disappears during a task. We introduce MIKASA-Robo-VLA, a benchmark of 90… 36 arXiv — Machine Learning research 11d ago WOMBAT: Whitebox Oracle for Molecular Benchmarking and Attribution Testing arXiv:2610.00713v1 Announce Type: new Abstract: When a graph neural network (GNN) explainer produces an unexpected attribution on a molecule, the attribution alone cannot reveal whether the explainer has failed or the model has learned a shortcut. We introduce WOMBAT, a… 14 arXiv — Machine Learning research 11d ago Benchmarking Generative Models for Weather Data Assimilation on Real Station Observations arXiv:2610.00728v1 Announce Type: new Abstract: Weather reanalysis products rely on computationally intensive numerical weather predictions followed by data assimilation that corrects the forecast toward observations. Deep generative models offer a cheaper alternative that… 18 r/LocalLLaMA community 12d ago AxiomicLabs' new benchmark, Tiny Theory of Mind, meant to gauge small models' theory of mind capabilities, has made it to the front page of Hugging Face's Datasets   submitted by   /u/Megneous [link]   [comments] 17 r/LocalLLaMA community 12d ago Two 96 GB Ascend cards crun Qwen3.8-flash-next hardware notes, vLLM work, benchmarks, and what is next I have been building a somewhat unusual local inference machine around two Huawei Atlas 300I Duo cards. They are relatively inexpensive, passive, dual-accelerator PCIe cards with 96 GB of device memory apiece. They are also absolutely not drop-in CUDA replacements. When I first… 36 r/LocalLLaMA community 12d ago I made my own cybersecurity benchmark and ran Qwen3.8 27B, here's how a local model actually does at hacking Hey local AI community, I've been working on this for a while and finally feel ok sharing it. It's a cyber benchmark where the model gets a shell in an isolated docker box and has to find the exact flag. Pwn, web, crypto, rev, forensics, a few real CVEs and some multi-stage… 4 arXiv — Machine Learning research 12d ago MM-FinEval: A Multi-Task Multimodal Benchmark for Real-World Financial Forecasting arXiv:2609.38523v1 Announce Type: new Abstract: Financial forecasting from earnings conference calls requires models to reason over complex corporate disclosures, market expectations, and subtle communication signals. However, existing financial benchmarks are often limited to… 11 arXiv — Machine Learning research 12d ago ChartDensity-Bench: Benchmarking MLLMs for Numerical Data Reconstruction under Visual Density arXiv:2609.38781v1 Announce Type: new Abstract: Multimodal large language models (MLLMs) offer a promising approach for recovering numerical data from scientific charts, but their ability to reconstruct chart data from visually dense figures remains poorly understood. Existing… 23 r/LocalLLaMA community 13d ago DDR4/PCIe4 vs DDR5/PCIe5 for LLMs- I benchmarked them for pre-training. What are your thoughts? Hello everyone. I've recently ran some experiments comparing DDR4/PCIe4 and DDR5/PCIe5 for AI workstations on a pre-training run, and would like to hear yours thoughts. First of all, and as a summary of my results ( code here:… 8 r/LocalLLaMA community 13d ago We benchmarked 18 RAG pipelines against an agent loop on Google's FRAMES. The best pipeline hit 78.9%. The agent loop hit 92.7%. Hybrid search, reranking, query decomposition, and query expansion are often treated as must-haves for good RAG. We wanted to see how much each actually helped, so we tested them. Same model, same embeddings, same documents, across all 824 multi-hop questions in FRAMES. We built… 32 r/MachineLearning community 13d ago Gemini 4 Argon - 1 Million Output Headroom. Hype or a Leap? [D] I rarely write about benchmarks; a competitor always beats them next week. But I care about 'Leaps'. Gemini 4 Argon feels like one to me. While Opus 5.5 and Astra cap output at 128-300K tokens (~90-180 pages), Argon hits 1 Million (~1400 pages). "Context glue" ruins agentic… 4 arXiv — NLP / Computation & Language research 13d ago The Backdrop Exposes What the World Around an Agent Costs It arXiv:2609.38469v1 Announce Type: new Abstract: Agent benchmarks test agents in worlds that stay still. Deployed agents work in worlds that other people also change. Someone texts the agent to send the money elsewhere or an order confirmation asks it to reply with a door code.… 16 arXiv — NLP / Computation & Language research 13d ago KlinikeBench: Evaluating Language Models Beyond Diagnostic Accuracy arXiv:2609.38480v1 Announce Type: new Abstract: Most clinical benchmarks evaluate language models (LMs) on diagnosis using complete case descriptions. In clinical practice, however, patients present information in different ways, and clinicians must obtain relevant history and… 4 arXiv — NLP / Computation & Language research 13d ago Beyond Oracle Communication: Benchmarking Interactive Intent Alignment Under Miscommunication and Evolving User Intent arXiv:2609.38604v1 Announce Type: new Abstract: Modern LLM agents increasingly tackle complex tasks through interactive, long-horizon exchanges with users, while existing benchmarks generally assume that users always accurately and sufficiently communicate a fixed intent.… 10 arXiv — NLP / Computation & Language research 13d ago BARRAC: Adaptation of an English Aspect-based Sentiment Analysis Approach for Classification Tasks in Arabic Dialects arXiv:2609.38820v1 Announce Type: new Abstract: With the rapid growth of Arabic NLP, several models, datasets and benchmarks have been reported. This paper asks whether approaches developed for majority languages like English can be adapted to Arabic tasks. We adapt an English… 10 arXiv — NLP / Computation & Language research 13d ago A Missing Piece for Trustworthy AI Reviewers: From Benchmarking Rhetorical Robustness to SciCore Review arXiv:2609.39027v1 Announce Type: new Abstract: AI reviewers can assign different judgments to manuscripts that report the same science in different wording, potentially rewarding rhetorical optimization over scientific improvement. We formulate Rhetorical Robustness as the… 16 arXiv — NLP / Computation & Language research 13d ago ViLegalExpert: A Large-Scale Benchmark for Vietnamese Legal Retrieval and Question Answering from Real-World Consultations arXiv:2609.39189v1 Announce Type: new Abstract: Trustworthy Legal AI requires systems that can answer legal questions while grounding their responses in authoritative sources. However, existing Vietnamese legal benchmarks provide limited coverage of real-world legal… 13 arXiv — NLP / Computation & Language research 13d ago DuplexAct-Bench: Broadening Full-Duplex Speech Evaluation toward Proactive Interaction across Diverse Behavioral Requirements arXiv:2609.39446v1 Announce Type: new Abstract: Existing full-duplex speech benchmarks cover only subsets of real-time interaction behaviors, often under limited contextual conditions. We introduce DuplexAct-Bench, a bilingual benchmark that systematically covers six… 28 arXiv — NLP / Computation & Language research 13d ago When a Kindergartener Solves Calculus: Measuring Capability Leakage in Role-Prompted Reasoning Models arXiv:2609.39846v1 Announce Type: new Abstract: We investigate the problem of role-capability leakage (RCL), in which a role-prompted reasoning model generates convincing in-role text while continuing to exhibit capabilities on benchmarks that exceed those implied by the… 11 r/MachineLearning community 14d ago Open-sourcing RightWayUp - a 360-degree image rotation model, and a JPEG shortcut we found in a common benchmark [P] We are open-sourcing RightWayUp - a new image rotation detection model. I work at ORTUS AI (we develop video analytics). We needed to tell from a single CCTV frame whether a camera had been rotated or installed at an angle (or upside-down), so we tried different models for that,… 15 r/LocalLLaMA community 14d ago AA-AgentPerf-Local: Benchmarking local AI agents on laptops and workstations   submitted by   /u/nuclearbananana [link]   [comments] 16 Page 3 of 10 · 500 articles ← Newer Older →
