News / #funding Tag Funding 500 articles archived under #funding · RSS Sign in to follow arXiv — NLP / Computation & Language research 1mo ago Contrastive ESA: Human Evaluation of Multiple Translations at Once arXiv:2607.26640v1 Announce Type: new Abstract: Current human evaluation of machine translation typically assesses single outputs in isolation, a paradigm that suffers from high annotator noise and cost. We introduce Contrastive Error Span Annotation (cESA), a protocol that… 29 arXiv — NLP / Computation & Language research 1mo ago TREK: A Travel Reasoning and Evaluation Kit for LLM Agents in Complex Trip Planning arXiv:2607.26977v1 Announce Type: new Abstract: Travel planning is a demanding stress test for tool-using LLM agents: a usable itinerary is a single artifact that must be right along many axes at once - every flight, hotel, and attraction must exist and be bookable, the days… 29 arXiv — NLP / Computation & Language research 1mo ago ML2B: Benchmarking LLMs on Cross-Lingual ML Pipeline Generation arXiv:2509.22768v3 Announce Type: replace Abstract: We introduce ML2B, the first benchmark for evaluating cross-lingual task comprehension in end-to-end ML pipeline generation by large language models. Despite growing global AI adoption, no systematic evaluation exists for ML… 30 r/MachineLearning community 1mo ago Open-source tabular model validation toolkit TanML needs feedback [D] We’re developing TanML, an MIT-licensed automated model-validation toolkit for tabular machine-learning models. TanML runs locally and provides an end-to-end workflow covering data profiling, preprocessing, feature-power ranking, model development, evaluation, drift analysis,… 5 TechCrunch — AI news-outlet 1mo ago As AI content floods the internet, Pangram raises $9M to detect it Pangram has raised $9 million to scale its AI detection software. The startup has also released a new AI text detection model, Pangram 4, and an AI image detection model in research preview. 15 Hugging Face Daily Papers research 1mo ago PerceptionBench: Evaluating Atomic Visual Perception in Multimodal Large Language Models Abstract We introduce PerceptionBench, a benchmark specifically designed to evaluate the atomic visual perception capabilities of Multimodal Large Language Models (MLLMs). Existing benchmarks often fail to isolate perception: holistic evaluations conflate perceptual errors with… 34 Hugging Face Daily Papers research 1mo ago Novel Claim or Déjà Vu? Rethinking "Contamination-Free'' Dynamic Evaluation for Multimodal Automated Fact-Checking Abstract Multimodal automated fact-checking (MAFC) verifies claims by retrieving and reasoning over external evidence. However, most existing static benchmarks risk contamination: they primarily consist of outdated claims verifiable using an LLM's internal knowledge without… 28 arXiv — NLP / Computation & Language research 1mo ago CogArena: A Multimethod Evaluation of Cognitive Ability Structure in Large Language Models arXiv:2607.24999v1 Announce Type: new Abstract: LLM cognitive scores are increasingly summarized as per-ability profiles whose dimensions should converge across tasks, respond selectively to matched interventions, and generalize beyond the models used to define them. We… 31 arXiv — NLP / Computation & Language research 1mo ago Evaluation of forced alignment of code-mixed speech: the case of Hindi-English arXiv:2607.25581v1 Announce Type: new Abstract: Code-mixed speech poses unique challenges to forced alignment: expanded inventories, orthographic errors, and speaker variation. We evaluate forced alignment of Hindi-English code-mixed speech using the Montreal Forced Aligner. We… 35 arXiv — NLP / Computation & Language research 1mo ago Evaluation of Adversarial Robustness in Arabic Language Models arXiv:2607.25814v1 Announce Type: new Abstract: The emergence of the recent outstanding capabilities of Arabic Language Models has opened doors for exposing their vulnerabilities. One of the major security risks associated with such Natural Language Processing models is… 18 arXiv — NLP / Computation & Language research 1mo ago AI's Capability in Assisting Scientific Research in Physics, Astrophysics, and Cosmology II: Project Planning and Proposal Evaluation arXiv:2607.25881v1 Announce Type: new Abstract: We investigate how well large language models (LLMs) can assist scientific project planning and proposal evaluation. One-page project plans were independently generated for eight expert-conceived research projects in physics,… 29 arXiv — NLP / Computation & Language research 1mo ago Evaluating Multi-Turn Multimodal Diagnostic Reasoning on Challenging Real-World Clinical Cases arXiv:2607.25933v1 Announce Type: new Abstract: Clinical diagnostic evaluation should not only assess whether models can provide correct diagnoses, but also reflect the realities of clinical practice, including progressive disclosure of multimodal information, dynamic updating… 32 arXiv — NLP / Computation & Language research 1mo ago RSMeM: Knowledge-Enhanced Memory Evolution for Remote Sensing Agents with Systematic Evaluation arXiv:2607.24772v1 Announce Type: cross Abstract: Geoscience research requires complex analysis and domain expertise, with remote sensing (RS) observations as a key foundation. However, existing RS agents built on general-purpose LLMs remain largely domain-agnostic, resulting in… 24 arXiv — NLP / Computation & Language research 1mo ago CLBench-V: Evaluating Multimodal Context Learning from Grounding to Knowledge Acquisition arXiv:2607.25294v1 Announce Type: cross Abstract: Real-world tasks often require models to learn from task-specific context rather than relying only on pre-trained knowledge. While recent work has highlighted this capability as context learning, existing evaluations mainly focus… 4 arXiv — NLP / Computation & Language research 1mo ago Instruction-based Image Editing: A Survey on Data, Models, Evaluation, and Applications arXiv:2607.25642v1 Announce Type: cross Abstract: Instruction-based Image Editing (IIE) aims to transform a given image into a new one based on textual instructions. Advances in Large Language Models (LLMs) and Vision-Language Models (VLMs) have accelerated progress toward… 32 arXiv — NLP / Computation & Language research 1mo ago Minimizing Targeted Activations: Input-Only Suppression of Evaluation-Awareness Latents in Large Language Models arXiv:2607.25907v1 Announce Type: cross Abstract: Activation steering controls model behavior by editing internal activations at inference time. We study its input-side dual: optimizing a fluent prompt so that a chosen internal latent is driven toward zero, with no… 22 arXiv — NLP / Computation & Language research 1mo ago A Cost-Effective Multimodal LLM Reasoning Framework for Question Answering over Irregular Clinical Time Series arXiv:2607.25947v1 Announce Type: cross Abstract: Question answering (QA) over irregular clinical time series (ICTS) plays a pivotal role in a wide range of healthcare applications. Although recent multimodal time-series large language models (LLMs) have shown considerable… 26 arXiv — NLP / Computation & Language research 1mo ago Eye Tracking Based Cognitive Evaluation of Automatic Readability Assessment Methods arXiv:2502.11150v5 Announce Type: replace Abstract: Automatic methods for scoring text readability have been studied for over a century, and are widely used in research and in user-facing applications in many domains. Thus far, the development and evaluation of such methods have… 22 arXiv — NLP / Computation & Language research 1mo ago Beyond Factual Accuracy: Evaluating Global Reasoning Integrity in RAG Systems with LogicScore arXiv:2601.15050v5 Announce Type: replace Abstract: Current evaluation methods for Retrieval Augmented Generation (RAG) suffer from \textit{factual myopia}: they relentlessly emphasize factual accuracy yet neglect global logical integrity in long-form answer generation. This… 28 Hugging Face Daily Papers research 1mo ago Codifying the Judge: Scalable Evaluation via Program Distillation Abstract LLM-as-a-judge has become the standard for automated evaluation, but it suffers from high cost, significant latency, and opaque decisions -- limitations that undermine its scalability and reliability. We address these with a simple, efficient alternative: program… 16 arXiv — Machine Learning research 1mo ago Beyond Shapley: An Influence-Based Data Auditing Pipeline for LLM Alignment and Evaluation arXiv:2607.22766v1 Announce Type: new Abstract: The alignment of Large Language Models (LLMs) is increasingly bottlenecked by data quality. As datasets scale, massive preference and instruction-tuning corpora inevitably accumulate hidden structural contradictions, safety risks,… 25 arXiv — Machine Learning research 1mo ago CC-AOS: Cost- and Horizon-Conditioned Amortized Backward Induction for Finite-Horizon Optimal Stopping arXiv:2607.22774v1 Announce Type: new Abstract: Finite-horizon optimal stopping is a central problem in early time-series classification, where a system must decide at each sequence prefix whether the expected benefit of another observation justifies its acquisition cost.… 14 arXiv — Machine Learning research 1mo ago Optimizing Transformer Neural Network for Real-Time Outlier Detection on FPGAs arXiv:2607.22786v1 Announce Type: new Abstract: In this work, we explore how the inference time of a Transformer Neural Network can be efficiently optimized with applications to real-time anomaly detection in financial time series. The financial time series are price series such… 38 arXiv — Machine Learning research 1mo ago Online Policy Evaluation for MDPs with Dynamic UBSR Measures arXiv:2607.23030v1 Announce Type: new Abstract: Developing efficient function-approximation methods for policy evaluation is a fundamental challenge in risk-aware reinforcement learning. Existing approaches either focus on restrictive classes of risk measures or rely on access… 10 arXiv — Machine Learning research 1mo ago In-Context Learning as Implicit Policy Gradient arXiv:2607.23153v1 Announce Type: new Abstract: Recent work has shown that large language models (LLMs) can iteratively improve their outputs by incorporating generated samples and their corresponding evaluation scores as in-context examples. Despite these empirical findings,… 35 arXiv — Machine Learning research 1mo ago Transfer Learning Architectures for Scalable Multi-Fidelity Bayesian Optimization arXiv:2607.23404v1 Announce Type: new Abstract: Self-driving laboratories increasingly rely on multi-fidelity Bayesian optimization (MFBO) to balance cheap, approximate evaluations against scarce, expensive ones, with a predictive surrogate at its core. Gaussian processes (GPs)… 7 arXiv — NLP / Computation & Language research 1mo ago ADAGE: A Language-Agnostic Pipeline for Analogical Reasoning Evaluation arXiv:2607.23058v1 Announce Type: new Abstract: Multilingual reasoning evaluation overwhelmingly relies on translating English benchmarks, a practice that introduces linguistic artifacts and fails to test culturally-grounded reasoning. We introduce ADAGE (Analogical… 33 arXiv — NLP / Computation & Language research 1mo ago Beyond a Global Norm: Personalizing Toxicity Sensitivity in Language Models Without Retraining arXiv:2607.23175v1 Announce Type: new Abstract: Reducing toxicity is often framed as a global alignment problem, yet perceptions of harmful language are subjective and context-dependent. We present the first comparative evaluation of training-free methods for aligning language… 35 arXiv — NLP / Computation & Language research 1mo ago Novel Claim or D\'ej\`a Vu? Rethinking "Contamination-Free'' Dynamic Evaluation for Multimodal Automated Fact-Checking arXiv:2607.23514v1 Announce Type: new Abstract: Multimodal automated fact-checking (MAFC) verifies claims by retrieving and reasoning over external evidence. However, most existing static benchmarks risk contamination: they primarily consist of outdated claims verifiable using… 28 arXiv — NLP / Computation & Language research 1mo ago BioSentinel at EXIST 2026: Soft-Label Optimization with XLM-RoBERTa for Sexism Intent Classification in Memes arXiv:2607.24137v1 Announce Type: new Abstract: This paper describes the BioSentinel team's participation in EXIST 2026 Task 2.2: Source Intention in Memes, part of the CLEF 2026 evaluation campaign. The task requires classifying the communicative intent behind memes as direct,… 8 arXiv — NLP / Computation & Language research 1mo ago Accuracy Hides How Language Models Fail: Measuring Failure States Under Matched Output Budgets arXiv:2607.24268v1 Announce Type: new Abstract: Language-model benchmarks collapse two distinct measurement questions into a single accuracy score: whether a response reached an evaluable state, and whether its answer was judged correct. We introduce a two-layer evaluation… 6 r/MachineLearning community 1mo ago Evaluated 6 frontier LLMs (GPT-5.4, Claude Sonnet 4.6, Claude Opus 4.7, Gemini Pro/Flash, Grok 4.3) on political, gender, and racial bias across 8 benchmarks (~20,600 examples) [R] I ran a solo evaluation project benchmarking six current frontier models: GPT-5.4, Claude Sonnet 4.6, Claude Opus 4.7, Gemini Pro, Gemini Flash, and Grok 4.3. I tested tham across 8 established bias/fairness datasets (WinoBias, BBQ Race/Ethnicity, SeeGULL, OpinionsQA, cajcodes… 26 TechCrunch — AI news-outlet 1mo ago Enigma raises $70M to make controlling a robot as easy as adjusting the volume The massive seed round was led by Index Ventures and Ribbit Capital, with participation from Sarah Guo's Conviction Partners. 32 arXiv — Machine Learning research 1mo ago Cloud-Native Evaluation-as-a-Service: A Microservices Architecture for Scalable AI Monitoring with Conformal Guarantees arXiv:2607.21623v1 Announce Type: new Abstract: We present EaaS, a cloud-native reference architecture that operationalizes AI evaluation methods as six stateless Kubernetes microservices: conformal prediction with finite-sample-corrected Adaptive Prediction Sets, calibration… 11 arXiv — Machine Learning research 1mo ago Toward User-Conditioned Evaluation of Personal LLM Agents under Temporal Interventions arXiv:2607.21635v1 Announce Type: new Abstract: Personal agents maintain memories, learned skills, tool configurations, and policy state that evolve with each user. Existing agent benchmarks often evaluate these capabilities in isolation: tool benchmarks test invocation under… 32 arXiv — Machine Learning research 1mo ago MissHyper: Restoring Clinical Synchronicity in Missingness-Guided Hypergraph Forecasting arXiv:2607.21922v1 Announce Type: new Abstract: Clinical irregular multivariate time series are shaped not only by physiological dynamics but also by the measurement process that determines when and what to observe. In event-centric models, however, co-timestamp structure can be… 29 arXiv — Machine Learning research 1mo ago Cross-Domain Off-Policy Evaluation and Learning for Contextual Bandits arXiv:2607.22012v1 Announce Type: new Abstract: Off-Policy Evaluation and Learning (OPE/L) in contextual bandits is rapidly gaining popularity in real systems because new policies can be evaluated and learned securely using only historical logged data. However, existing methods… 32 arXiv — Machine Learning research 1mo ago An Insight on Evaluation Metrics Under the Imbalanced Case of Anomaly Detection arXiv:2607.22286v1 Announce Type: new Abstract: Anomaly detection is inherently characterised by severe class imbalance, making the interpretation of evaluation metrics challenging. Although metrics such as AUROC, AUPR, F1-score, and MCC are widely used, their values convey… 4 arXiv — NLP / Computation & Language research 1mo ago A Consensus-Based Framework for Relative Preference Evaluation of Large Language Models arXiv:2607.21632v1 Announce Type: new Abstract: Traditional benchmarks for LLMs primarily rely on static datasets and objective scoring metrics, which often fail to capture differences in response quality when multiple answers are acceptable. In such settings, correctness alone… 25 arXiv — NLP / Computation & Language research 1mo ago Evaluation design conditions the expert-vs-auto MeSH gap: a controlled comparison of bag-of-words and BiomedBERT on the Cohen benchmark arXiv:2607.21685v1 Announce Type: new Abstract: A systematic review begins with someone reading thousands of abstracts to identify the few that are relevant, and classifiers are used to prioritise that reading. Their inputs are often augmented with Medical Subject Headings… 13 arXiv — NLP / Computation & Language research 1mo ago Agentic Evaluation of Copyright Law Compliance arXiv:2607.21799v1 Announce Type: new Abstract: Large language model (LLM) agents increasingly perform commercial tasks that involve retrieving external content such as images and, where appropriate, reproducing that content. LLM agents should comply with the law, including… 6 arXiv — NLP / Computation & Language research 1mo ago Ground Truth First: A Longitudinal Evaluation Instrument for Agent Memory, and the Tenure Crossover in Memory-Architecture Rankings arXiv:2607.21962v1 Announce Type: new Abstract: Benchmarks for LLM-agent memory typically generate conversations first and extract answer keys afterwards -- with documented label-error and contamination problems -- and they overwhelmingly measure short interaction histories. We… 18 arXiv — NLP / Computation & Language research 1mo ago From Isolated Tasks to Structured Capabilities: A Multilayer Taxonomy for Large Language Models arXiv:2607.22182v1 Announce Type: new Abstract: Large language model (LLM) evaluation spans diverse tasks and benchmarks, yet evidence remains organized around tasks rather than the capabilities they probe. This fragmentation limits cross-study comparison, obscures capabilities… 30 arXiv — NLP / Computation & Language research 1mo ago Why Large Language Models and Humans Converge and Diverge in Evaluating Creativity arXiv:2607.22218v1 Announce Type: new Abstract: Despite the growing use of large language models (LLMs) as creativity evaluators, evidence of their alignment with human evaluations remains mixed, raising the question of when and why their judgments converge with or diverge from… 37 arXiv — NLP / Computation & Language research 1mo ago grapheme-kit: Grapheme-Level Metrics and Text Processing for Multilingual NLP arXiv:2607.22456v1 Announce Type: new Abstract: Existing lexical distance, similarity, and evaluation metrics operate on Unicode code points, which can misrepresent errors in writing systems where a single grapheme is represented by multiple Unicode code points. We introduce… 37 arXiv — NLP / Computation & Language research 1mo ago Zero-Shot Mission-Level Evaluation for Aerial MLLM Agents arXiv:2607.22014v1 Announce Type: cross Abstract: Multimodal Large Language Models (MLLMs) are emerging as core reasoning modules for embodied agents, yet it remains unclear how well general-purpose models can solve long-horizon embodied tasks from a single high-level… 15 arXiv — NLP / Computation & Language research 1mo ago DBA-Bench: A Production-Fidelity Benchmark for LLM-Based Database Operations Agents arXiv:2607.22165v1 Announce Type: cross Abstract: LLM-based database agents show promise, but differing task scopes, testbeds, and metrics hinder comparison. We identify four gaps between evaluation and production operations: live-environment fidelity (multi-turn read-write… 17 arXiv — NLP / Computation & Language research 1mo ago LMEB: Long-horizon Memory Embedding Benchmark arXiv:2603.12572v5 Announce Type: replace Abstract: Memory embeddings are crucial for memory-augmented systems, such as OpenClaw, but their evaluation is underexplored in current text embedding benchmarks, which narrowly focus on traditional passage retrieval and fail to assess… 27 arXiv — NLP / Computation & Language research 1mo ago WHBench: Evaluating Frontier LLMs with Expert-in-the-Loop Validation on Women's Health Topics arXiv:2604.00024v2 Announce Type: replace Abstract: Large language models are increasingly used for medical guidance, but women's health remains under-evaluated in benchmark design. We present the Women's Health Benchmark (WHBench), a targeted evaluation suite of 47… 25 r/LocalLLaMA community 1mo ago CachyLLama: llama.cpp fork with persistent SSD-backed KV caching for local agent workflows If you run local agentic coding harnesses (Aider, Claude Code, etc.), prompt evaluation usually eats up most of your execution time. Every turn re-evaluates thousands of identical prefix tokens_system prompts, tool schemas, and conversation history. CachyLLama is a llama.cpp… 19 Page 9 of 10 · 500 articles ← Newer Older →
