News / #benchmark Tag Benchmark 500 articles archived under #benchmark · RSS Sign in to follow Hugging Face Daily Papers research 3d ago Are Android GUI Agents Robust Against Runtime Anomalies? AnTrap: Evaluating Agents in Dynamic Adversarial Environments Abstract AnTrap benchmarks GUI agent robustness by injecting dynamic anomalies into execution trajectories, revealing universal vulnerabilities and distinguishing learnable traps from intrinsic reasoning limits. Generated by thinkingmachines/Inkling-Small GUI agents often… 19 The Information — AI news-outlet 3d ago Four-month-old AI Assistant Startup Instinct Raises at a $2.5 Billion Valuation Instinct, a startup whose AI assistant connects to users’ applications and performs tasks on their behalf, is raising a Series B at a $2.5 billion valuation in a round led by Index Ventures and Benchmark, said a person with direct knowledge. The round will close in the next… 7 r/MachineLearning community 3d ago A dataset with 52 Text to image model evaluation [P] I created a simple text to image benchmark. I curated 192 prompts that are difficult for T2I models in various ways: text rendering, spatial reasoning, human realism, negations, etc... I then asked a VLM to judge every output against a pre-specified binary question with the… 30 r/MachineLearning community 3d ago HNSW from scratch, benchmarked against FAISS: brute force still wins at 5,183 documents. [P] I built a retrieval engine with no retrieval libraries in the core — BM25 over a hand-rolled inverted index, HNSW from the Malkov & Yashunin paper, RRF to fuse them — mainly to understand HNSW rather than treat it as a black box inside a vector DB. FAISS, bm25s and rank_bm25… 35 r/LocalLLaMA community 3d ago Self-hosting LLMs on budget hardware: general principles, hardware, benchmarks and frontends Hello, I've been self-hosting LLMs on various budget hardware for a while (6x RTX 3060 12 GB, Intel Arc Pro B60 24 GB, RX 9070 XT, etc). Over the last few months, I wrote about it in 4 articles: General principles Hardware and inference optimization CPU+RAM offloading, MoE,… 36 r/LocalLLaMA community 3d ago Benchmarking Qwen3.8 27B quantizations: 4-bit holds up, 1-bit collapses I measured various Qwen3.8 27B quantizations by Unsloth on popular benchmarks: FPQA Diamond, IFBench, and Terminal-Bench-2.1. Q4_K_M is all you need.   submitted by   /u/pmigdal [link]   [comments] 4 r/LocalLLaMA community 3d ago [Megathread] GLM-5.3-Flash - former ox-alpha Megathread for discussing the release of GLM-5.3-Flash. Quants Fine-Tunes & Abliterations Chat Templates Inference Server Support & Configuration Experiences, Benchmarks & Model Comparisons We'll try to clean up future duplicates around the release and point them here.… 35 r/LocalLLaMA community 3d ago What's your most reliable model, even if it's "outdated"? What's a model you keep coming back to even though newer ones have technically surpassed it? I've noticed I default to the same one for daily tasks despite downloading every shiny new release. Curious if others have a reliable workhorse they trust over benchmark leaders  … 31 r/LocalLLaMA community 3d ago Gemma4 31B vs Qwen3.8 27B - why the huge difference in benchmarks? Hi all, I'm looking for the best model for a hobby project and trying to make sense of the various data I came across. I know benchmarks do not often translate to the real world, especially to your particular use case (whatever it may be). But this is truly baffling: AA says… 28 TechCrunch — AI news-outlet 3d ago Surprise: Z.ai is the AI lab behind the mysterious Ox Alpha model Z.ai confirms it is behind Ox Alpha, the mysterious open AI model topping benchmarks and leaderboards, and its weights are set to be released soon. 19 r/LocalLLaMA community 3d ago Forget the Pelican, it's Weevil-Time! / Benchmaxxing-Proof SVG and Vision Benchmark The Artist: Qwen3.8-27B-UD-Q3\ K_XL, q8_0 caches, xhigh, temp 1.0, image-min-tokens 1024, froggeric template) I was screwing around with different Qwen3.8-27B quants and thought of this very simplistic but seemingly bechmaxxing resistant combined SVG and vision test. Just let… 4 Hugging Face Daily Papers research 4d ago CAFE: Self-Improving Search Agents Need Co-Evolving Feedback Abstract CAFE couples a search agent and critic via shared parameters to learn in-trajectory corrective feedback, improving search performance and reducing hallucinations across benchmarks. Generated by thinkingmachines/Inkling-Small Outcome-supervised search agents learn when… 19 Hugging Face Daily Papers research 4d ago LAION-BVD: A 10-Million-Hour Open Video Dataset for Multimodal Pre-training Abstract LAION-BVD is a large-scale open video dataset enabling multimodal pre-training across video, audio, and image modalities with synthetic captions and strong benchmark performance. Generated by thinkingmachines/Inkling-Small We present LAION-BVD, a large-scale open video… 22 r/LocalLLaMA community 4d ago [Megathread] Qwen3.8-Flash-Next - Release Day Megathread for discussing the (impending) release of Qwen 3.8 Flash Next. Quants Fine-Tunes & Abliterations Chat Templates Inference Server Support & Configuration Experiences, Benchmarks & Model Comparisons We'll try to clean up future duplicates around the release and point… 37 r/LocalLLaMA community 4d ago Underrated Muse Glimmer Benchmarked qwen3.8 xhigh, medium and muse glimmer. Xhigh effort mode with qwen3.8 took almost 30hrs. (And still failed on 16 cases because of the 32K output token limit) Medium effort mode and muse glimmer were 3-4 hours each. But I'm actually surprised by the muse glimmer… 31 arXiv — Machine Learning research 4d ago UHI-Bench: Benchmarking Dual-Source Urban Heat Island Modeling Across Cities in Diverse Climate Regimes arXiv:2608.23857v1 Announce Type: new Abstract: Urban heat islands (UHIs) are intensifying under climate change, exacerbating thermal exposure risks. Their two primary observations, land surface temperature UHI (LST-UHI) and near-surface air temperature UHI (AirT-UHI), capture… 21 arXiv — Machine Learning research 4d ago PhysicsBench: A Unified Leaderboard for Generative and Predictive Models in Engineering Design and Simulation arXiv:2608.24056v1 Announce Type: new Abstract: Generative and predictive artificial intelligence models are increasingly used to generate geometry and to predict physical fields and scalar quantities in engineering design and simulation. Yet these models are typically evaluated… 7 arXiv — Machine Learning research 4d ago WarpSAC: Towards the Pinnacle of Scalable Off-policy RL by Rethinking Exploration and Exploitation arXiv:2608.24479v1 Announce Type: new Abstract: Massively parallel simulation changes the data regime in which off-policy reinforcement learning (RL) is trained, challenging stabilizers designed for data-limited replay. Through controlled experiments across eight benchmark… 35 arXiv — Machine Learning research 4d ago FraudBench: Protocol-Sensitive Benchmarking of Adversarial Robustness for Financial Risk Assessment arXiv:2608.24551v1 Announce Type: new Abstract: Machine learning models are widely used in financial fraud and credit-risk detection, yet their adversarial robustness remains difficult to evaluate because financial tabular data involve domain-specific constraints, severe class… 25 Hugging Face Daily Papers research 4d ago WeMM-Embedding: WeChat Multi-Modal Embedding Technical Report Abstract WeMM-Embedding is a family of universal multimodal embedding models that align text, images, videos, and interleaved inputs in a shared space, achieving state-of-the-art retrieval and recommendation performance across public benchmarks and large-scale WeChat… 15 Hugging Face Daily Papers research 4d ago AutoSaddler: Automatic Harness Optimization with Durable Updates from Agent Execution Traces Abstract AutoSaddler automatically improves LLM agent harnesses via offline failure-driven optimization, boosting performance on long-horizon benchmarks. Generated by thinkingmachines/Inkling-Small LLM agents remain unreliable on long-horizon tasks, where small local failures… 26 r/LocalLLaMA community 4d ago 35B-A3B tool calling benchmark: Original Qwen vs. KAT Coder, Ornith and Tiel-Coder With hopes of a Qwen3.8-35B-A3B release now mostly dashed, many people including myself are looking at fine-tunes and other variants of Qwen3.6-35B-A3B to run on VRAM-limited hardware. I decided to try to benchmark some of the top contenders: KAT-Coder, Ornith 1.5 and the very… 35 r/LocalLLaMA community 4d ago NInfer 4090 Windows update is out with 1.5-2k t/s prefill, extended MTP, disk caching with DirectStorage, built-in llama.cpp WebUI and more I've made a few changes here and there to get nearly 2.1k tokens/sec prefill, ~210-230 tokens/sec decode with MTP7 (configurable, extended up to 15) on benchmarks. Also added disk caching options, up to 30GB per config by default for near-instant loads after server restart,… 10 TechCrunch — AI news-outlet 4d ago OpenAI’s Jalapeño chip is built for fast inference at scale, benchmarks show Tested on SemiAnalysis’ InferenceX benchmark, Jalapeño registered both more tokens per user and more throughput per kilowatt than the currently available state-of-the art. 6 r/MachineLearning community 4d ago What would a fair benchmark for agent architecture look like? [D] I am working on an evaluation design and would appreciate criticism before running it. Most coding-agent benchmarks collapse the model and its harness into one score. If a run fails, it is difficult to tell whether the cause was model capability, context assembly, task… 20 r/MachineLearning community 4d ago How we built a SOTA search engine using PostgreSQL, pgvector, and Qwen3 embeddings [P] I wrote a technical breakdown of how search works on Papers with Code. The system combines keyword and semantic search, which produced better results than either approach alone. The stack includes: PostgreSQL with pgvector Qwen3-Embedding-0.6B for text embeddings Hugging Face… 31 Hugging Face Daily Papers research 5d ago One Success Isn't Reliability: Thinkingbox, a Sandbox and Benchmark for Agents in Stateful Business Workflows Abstract Recent agent benchmarks increasingly ground evaluation in executable environments, from code repair to web navigation, app APIs, and function calling. Yet completing consequential work beyond code requires more than producing a plausible response or valid tool call:… 33 Hugging Face Daily Papers research 5d ago MobilePA-Bench: Benchmarking Mobile Planner Agents on Complex Real-World Tasks Abstract MobilePA-Bench is an interactive sandbox benchmark that evaluates mobile planning agents on tool-calling, sub-agent collaboration, memory usage, and composite skill invocation under real runtime constraints. Generated by thinkingmachines/Inkling-Small As on-device LLM… 13 arXiv — NLP / Computation & Language research 5d ago Wazobia Eval: A Benchmark for Nigerian Pidgin Emotion Understanding, Sarcasm Detection, and Cultural Reasoning arXiv:2608.21369v1 Announce Type: new Abstract: Nigerian Pidgin is one of Africa's most widely spoken languages, yet remains severely underrepresented in language model evaluation. Existing benchmarks primarily focus on translation, transcription, or generic sentiment analysis,… 21 arXiv — NLP / Computation & Language research 5d ago L\"etzCross: A Cross-Lingual Page-Level Benchmark for Multimodal Retrieval over Luxembourgish Documents arXiv:2608.21714v1 Announce Type: new Abstract: Recent page-image retrievers such as ColPali have improved retrieval over visually rich documents, yet little is known about how they behave in cross-lingual, low-resource settings. We introduce L\"etzCross, a benchmark for… 9 arXiv — NLP / Computation & Language research 5d ago Evaluation Awareness in Language Models: Representation, Verbalization, and Control arXiv:2608.21766v1 Announce Type: new Abstract: Both capability and safety benchmarks rest upon the assumption that the behavior of language models undergoing a test is informative about their behavior in deployment. This assumption can fail, should models infer that they are… 16 arXiv — NLP / Computation & Language research 5d ago No One Model Catches Every Harm: Benchmarking Content Moderation Across Safety Scenarios arXiv:2608.21775v1 Announce Type: new Abstract: Large Language Models (LLMs) are increasingly deployed in real-world applications, yet they remain vulnerable to generating harmful content. From adversarial jailbreaks that bypass safety filters to implicit hate that evades… 30 arXiv — NLP / Computation & Language research 5d ago Lexical Coupling in GUI Element Grounding: Sentence Embeddings Track Labels across Mobile and Web arXiv:2608.21794v1 Announce Type: new Abstract: GUI grounding evaluations that expose UI elements as text metadata often treat high instruction-element embedding similarity as evidence of semantic grounding. Across three mobile and web benchmarks, we show that this… 28 arXiv — NLP / Computation & Language research 5d ago GUI-Primitives: Diagnosing Spatial Reasoning Failures in Vision-Language GUI Grounding arXiv:2608.21832v1 Announce Type: new Abstract: Computer-use agents ground natural-language instructions in screenshots to locate interface elements, yet existing benchmarks do not isolate whether models bind relational language to the correct element. We introduce… 37 arXiv — NLP / Computation & Language research 5d ago PUMA: A Polish Benchmark for Culturally Grounded Multimodal Understanding arXiv:2608.21853v1 Announce Type: new Abstract: Large language models are increasingly moving beyond text processing, adding support for other modalities such as images and audio. While text understanding and generation have been extensively studied, multimodal data processing… 12 arXiv — NLP / Computation & Language research 5d ago BanglaVeilGuard: Cross-Script Safety Benchmarking and Lightweight Guardrails for Bangla Large Language Models arXiv:2608.21880v1 Announce Type: new Abstract: Bangla large language model (LLM) safety is difficult to evaluate with English-centric or standard-script benchmarks because Bangla users routinely write across scripts, spellings, code-mixed forms, and regional registers. This… 4 arXiv — NLP / Computation & Language research 5d ago Machine learning and digital pragmatics: Which word category influences emoji use most? arXiv:2608.21975v1 Announce Type: new Abstract: This study examines the performance of the state-of-the-art MARBERT model in identifying the lexical/pragmatic category associated with emoji use on X within a digital pragmatics approach (DPA). A net corpus of 15856 Colloquial… 28 arXiv — NLP / Computation & Language research 5d ago Grounded Normative Rule Generation with Structured Search arXiv:2608.22229v1 Announce Type: new Abstract: Normative rules like institutional charters and workplace policies must be both human-readable and operationally verifiable against actual environment records. However, current language generation and structured-output benchmarks… 14 arXiv — NLP / Computation & Language research 5d ago Noise Floor Audit for Agent Benchmarks arXiv:2608.22331v1 Announce Type: new Abstract: We audit measurement variability for 3 native tool-calling endpoints across 2 providers on the official BFCL multiple and parallel categories, using matched AST grading. At temperature 0, reruns are nearly deterministic across Groq… 16 arXiv — NLP / Computation & Language research 5d ago Register Shifts Break LLM Safety: A Bengali Benchmark with Culturally Grounded Harms arXiv:2608.22335v1 Announce Type: new Abstract: Bengali is the seventh-most-spoken language globally, yet LLM safety evaluation remains overwhelmingly English-centric. We introduce BanglaSafe, a benchmark of 879 Bengali prompts combining 309 natively authored prompts with 570… 12 arXiv — NLP / Computation & Language research 5d ago SchemaGUI: A Schema-Driven Benchmark for Controllable GUI Generation Evaluation arXiv:2608.22390v1 Announce Type: new Abstract: Large language models (LLMs) have demonstrated strong potential in graphical user interface (GUI) generation, but reliable evaluation remains challenging due to uncontrolled data distributions, noisy annotations, and limited layout… 35 arXiv — NLP / Computation & Language research 5d ago Rank Reversal in Multilingual LLM Judges: A Label-Free Double-Centering Calibrator arXiv:2608.22432v1 Announce Type: new Abstract: Multilingual LLM judges produce different evaluator-backbone rankings depending on the prompt language: on an eight-language Agent-as-a-Judge benchmark, the top-ranked backbone alternates across English, Arabic, Chinese, Hindi,… 33 arXiv — NLP / Computation & Language research 5d ago A Source-Grounded Framework for Constructing and Evaluating Progressive Multimodal Diagnostic Dialogues from Clinical Case Reports arXiv:2608.22713v1 Announce Type: new Abstract: Clinical diagnosis requires progressive integration of patient history, physical examination, laboratory findings, medical images, and diagnostic-informative tests. However, most multimodal medical benchmarks evaluate fixed inputs… 38 arXiv — NLP / Computation & Language research 5d ago Beyond Factual Knowledge: Benchmarking and Learning Step-Level Procedural Rule Reasoning in Large Language Models arXiv:2608.22753v1 Announce Type: new Abstract: Large language models (LLMs) excel at text understanding and generation, yet still struggle to reliably understand and apply externally provided procedural rules at scale. To evaluate this capability, we introduce RuleWorld, a… 24 arXiv — NLP / Computation & Language research 5d ago Industrial-Instruction: An End-to-End Framework for Building Instruction-Tuning and Benchmark Datasets from Industrial Technical Reports arXiv:2608.22817v1 Announce Type: new Abstract: Industrial technical reports contain high-value knowledge for maintenance, troubleshooting, and product engineering, but their heterogeneous structure (dense prose, specifications, tables) makes them difficult to index and reason… 30 arXiv — NLP / Computation & Language research 5d ago What Proves You Wrong: Benchmarking Language Models on Falsifiable Research Ideation arXiv:2608.22948v1 Announce Type: new Abstract: Large language models are increasingly used to propose research ideas, yet the prevailing ways of judging such ideas supply no shared decision rule: free-form judging sways with style and position, and scoring against a later paper… 27 r/LocalLLaMA community 5d ago Qwen-3.8-27B, Nemotron-3.5-Lightning-30B-A3B, Ornith-1.5-35B-A3B, Muse-Glimmer-30B oQ8e comparison Ornith does really well. TielCoder ( https://llm-bench.io/benchmarks/cmt7kp2zj002r01lcmpchvlko ) might be even a bit better in coding. Will give it a try soon. Details of the comparison see here:… 35 r/LocalLLaMA community 5d ago Planning to spend ~$100 benchmarking differnet Qwen3.8-27B quants and kv cache and looking for input before I start TL;DR: I'm planning to spend around $100 on cloud GPUs to benchmark Qwen3.8-27B with a focus on questions that actually matter when running it locally: different quant levels/providers, 8-bit vs 16-bit KV cache, GGUF vs EXL3, context length tradeoffs, and token efficiency on… 29 r/LocalLLaMA community 5d ago TielCoder's 22 GB 4-bit quant matches Opus4.6 medium on recent real life coding issues, surpassing KAT-Coder and Nail as strongest and fastest MoE picks. Qwen3.8-27B is amazing, but it’s slow. A stronger 35B-A3B Mixture of Experts-coder that can run and solve real codebase issues fast (even on constrained hardware) is a valuable addition to the arsenal. This one is the strongest and most consistent 35B-A3B I’ve benchmarked, on… 38 Hugging Face Daily Papers research 5d ago FlavourBench: Ranking Frontier Language Models with Executable Culinary Ground Truth Abstract FlavourBench evaluates language models on culinary portfolio tasks using executable ground truth, statistical rigor, and fully reproducible verification. Generated by thinkingmachines/Inkling-Small Open-ended language-model benchmarks usually inherit a judge: a human… 9 Page 2 of 10 · 500 articles ← Newer Older →
