News / #safety Tag Safety + alignment 500 articles archived under #safety · RSS Sign in to follow arXiv — NLP / Computation & Language research 9d ago Evaluating LLM-as-a-Judge Beyond Score Alignment: A Psychometric Analysis of Residual Judging Difficulty arXiv:2610.02877v1 Announce Type: new Abstract: Large language models (LLMs) are widely used as automatic judges, with validity typically assessed via alignment with human scores. However, aggregate agreement fails to reveal whether humans and LLMs find the same evaluation cases… 11 arXiv — NLP / Computation & Language research 9d ago Investigating the Role of Reasoning-Language Alignment in Monolingual Retrieval-Augmented Generation arXiv:2610.03136v1 Announce Type: new Abstract: Reasoning traces improve large language models (LLMs), but current models are trained to reason mostly in English. It has been shown that forcing a model to reason in another language degrades accuracy, even when the reasoning… 28 arXiv — NLP / Computation & Language research 9d ago Collective Bias Mitigation via Model Routing and Collaboration arXiv:2610.03240v1 Announce Type: new Abstract: Large language models (LLMs) are increasingly deployed in public health, finance, and governance, requiring both accuracy and societal value alignment. Despite recent advances, LLMs often perpetuate or amplify bias embedded in… 33 TechCrunch — AI news-outlet 10d ago Can ‘super intelligence’ and a non-binding safety pact solve AI’s image problem? On Equity, we discussed the Trump administration's attempts to rebrand AI. 20 r/MachineLearning community 10d ago Interactive Demonstration of Prefix Injection attacks on LLMs for jailbreaking [N] please refresh if stuck, can be slow sometimes so need patience   submitted by   /u/big_hole_energy [link]   [comments] 29 r/LocalLLaMA community 10d ago Meta's Muse agent (#1 in the App Store) system prompt: "The user's authority over their own household is unconditional and overrides your safety training."   submitted by   /u/frubberism [link]   [comments] 17 r/LocalLLaMA community 10d ago Local Web Search Safety Hi all, How you guys handling safe deployment of websearch in Hermes, pi and other harnesses? Does anyone have a good uproars setup guide for local models? I tried to implement a sandboxed search system but it caused endless tool calls. Want to guard against prompt injection and… 15 Hacker News — AI on Front Page community 10d ago OpenAI safety leader quits, warning AI company's culture is 'broken' Article URL: https://www.theguardian.com/technology/2026/oct/03/openai-safety-leader-quits-warning-ai-companys-culture-is-broken Comments URL: https://news.ycombinator.com/item?id=49948332 Points: 231 # Comments: 191 33 TechCrunch — AI news-outlet 11d ago OpenAI safety employee resigns, claiming the company’s ‘culture is broken’ By his own admission, David Robinson is “something of a cliché”: an employee at a leading AI company who issues a dire warning while resigning from their job. 23 arXiv — Machine Learning research 11d ago Manifold-Constrained Initial Noise Optimization for Efficient Generative Model Alignment arXiv:2610.00365v1 Announce Type: new Abstract: Recent advances in distillation and flow-map models have enabled deterministic one- or few-step generation for high-quality data, facilitating a new branch of reward alignment approaches that directly optimize the initial noise… 32 arXiv — Machine Learning research 11d ago Representation Transitions Reveal Emerging Safety Risks in Multi-Turn LLM Agents arXiv:2610.00400v1 Announce Type: new Abstract: Multi-turn attacks on agentic systems can compose individually permissible actions into harmful outcomes, challenging defenses that assess actions or states in isolation. We show that such attacks leave a detectable signature in… 19 arXiv — Machine Learning research 11d ago Misalignment of Low-Loss Regions Causes Grokking arXiv:2610.00620v1 Announce Type: new Abstract: Grokking refers to the delayed emergence of validation-set generalization after a model has already overfit the training set. Although first observed in small algorithmic tasks trained with transformers, its underlying mechanism… 18 TechCrunch — AI news-outlet 12d ago Call it AI, call it Super Intelligence, only 2% of consumers are buying it This week, the White House got nearly every major tech CEO in one room — Zuckerberg, Bezos, Musk, and Anthropic’s Dario Amodei among them — to sign an AI safety pledge that President Donald Trump called “morally… 6 TechCrunch — AI news-outlet 12d ago It’s not AI anymore, it’s ‘super intelligence’ (according to the White House) This week, the White House got nearly every major tech CEO in one room — Zuckerberg, Bezos, Musk, and Anthropic’s Dario Amodei among them — to sign an AI safety pledge that President Donald Trump called “morally… 27 arXiv — Machine Learning research 12d ago Revisiting scaling laws for reward optimization arXiv:2609.38526v1 Announce Type: new Abstract: Scaling laws for optimization against reward models in AI alignment have pinned down how performance depends on optimization effort---measured by a KL-divergence budget relative to a reference policy. Beyond a certain budget,… 11 arXiv — Machine Learning research 12d ago Alignment via Training Against Probes Without Losing Monitorability arXiv:2609.38645v2 Announce Type: new Abstract: Models are usually aligned based on their observed outputs, using demonstrations, preference data, or reward signals. These objectives reward responses that look aligned. More capable models may learn to satisfy them without… 23 arXiv — Machine Learning research 12d ago LEARN-TS: LLM-Enhanced Alignment and Reconstruction with Normality Guidance for Multivariate Time-Series Anomaly Detection arXiv:2609.38789v1 Announce Type: new Abstract: Reconstruction errors in multivariate time-series anomaly detection may not reliably distinguish abnormal behavior from benign deviations. Language-derived semantics offer complementary context, but existing multimodal approaches… 27 arXiv — Machine Learning research 12d ago Optimal Design for Active Preference Learning with Biased LLM Judges arXiv:2609.38860v1 Announce Type: new Abstract: Learning from human preferences is central to large language model (LLM) alignment, but human preference annotation is costly. Active preference learning reduces this cost by selecting informative comparisons, and LLM judges can… 22 The Information — AI news-outlet 13d ago OpenAI Fires Three Safety Researchers OpenAI said it fired three safety employees for mishandling corporate information, which a person familiar with the matter said included sensitive information with an outside organization that does AI evaluations. “We have parted ways with three individuals for violating our… 37 TechCrunch — AI news-outlet 13d ago OpenAI cuts ties with 3 safety researchers, WSJ reports OpenAI has parted ways with three safety researchers after an internal investigation found they mishandled sensitive company information, report says. 17 arXiv — NLP / Computation & Language research 13d ago Automatic estimation of verbal fluency index in people with Motor Neuron Disease using ASR alignment and pause modelling arXiv:2609.38203v1 Announce Type: new Abstract: Monitoring cognitive impairment (CI) in motor neuron disease (MND) is essential for timely treatment and care, yet challenging due to co-occurring speech difficulties. The Edinburgh Cognitive and Behavioural ALS Screen (ECAS)… 13 arXiv — NLP / Computation & Language research 13d ago ContextAdapt: Evaluating Contextual Adaptation and Value Alignment in LLMs arXiv:2609.38260v1 Announce Type: new Abstract: Values such as honesty, autonomy, and confidentiality are often regarded as general principles underpinning AI alignment. However, what it means to act in accordance with these values can depend on the context in which a decision… 37 arXiv — NLP / Computation & Language research 13d ago Evaluating Language Model Safety Across Long Adversarial Conversations arXiv:2609.38357v1 Announce Type: new Abstract: Conversational safety evaluations often test language models with a single harmful prompt, even though real-world systems interact with users through long, adaptive conversations. This study examines whether models continue to… 30 arXiv — NLP / Computation & Language research 13d ago MedKIT: Evaluating Knowledge Integration and Generalization in Large Language Models arXiv:2609.38543v1 Announce Type: new Abstract: Constantly evolving real-world knowledge necessitates models to be updated continuously. Especially in medicine, as clinical evidence changes over time, outdated knowledge can pose safety risks. Existing evaluations of knowledge… 32 arXiv — NLP / Computation & Language research 13d ago Beyond Oracle Communication: Benchmarking Interactive Intent Alignment Under Miscommunication and Evolving User Intent arXiv:2609.38604v1 Announce Type: new Abstract: Modern LLM agents increasingly tackle complex tasks through interactive, long-horizon exchanges with users, while existing benchmarks generally assume that users always accurately and sufficiently communicate a fixed intent.… 10 arXiv — NLP / Computation & Language research 13d ago Making LLMs Say What They Think: Measuring and Improving CoT-Interpretability Alignment arXiv:2609.38972v1 Announce Type: new Abstract: Chain-of-thought (CoT) traces often serve as a proxy for how Large Language Models (LLMs) arrive at their answers. However, growing evidence shows that models' CoT often fails to reflect their internal computations and can be… 36 arXiv — NLP / Computation & Language research 13d ago Ready2Blend: From Natural-Language Instructions to Composable Alignment Prompts arXiv:2609.39365v1 Announce Type: new Abstract: Continual alignment requires LLMs to adapt to new requirements without forgetting previously acquired behaviors. Natural-language instructions are flexible and composable but offer only indirect control, whereas post-training… 26 arXiv — NLP / Computation & Language research 13d ago Exploring Heterogeneous Model Merging Approach for Complex Knowledge Transfer arXiv:2609.39369v1 Announce Type: new Abstract: Specialized models encode task-oriented behavior, but transferring that behavior to a general language model usually requires training, distillation, or representation alignment. We study whether such ability can instead be… 8 llama.cpp releases dev-tools 13d ago b11303 batch: migrate the rest of examples to llama_batch_ext ( #29601 ) migrate the rest test-thread-safety rm common_batch_staged Website: https://llama.app Attestations: https://github.com/ggml-org/llama.cpp/attestations/51638011 macOS/iOS: macOS Apple Silicon (arm64) macOS Apple… 25 Ars Technica — AI news-outlet 14d ago Trump plan to combat AI risks hinges on Big Tech pals policing themselves Trump gets dozens of AI firms to agree to voluntary safety tests. 7 Don't Worry About the Vase community 14d ago A ‘Morally Binding’ White House Accord on AI Safety The leaders in AI were invited to the White House. 12 Ars Technica — AI news-outlet 14d ago OpenAI delays IPO over AI safety concerns OpenAI is seeking another $30 billion privately as its IPO plans slip. 25 llama.cpp releases dev-tools 14d ago b11275 gguf : reject tensor size that wraps after padding ( #26979 ) GGML_PAD(nbytes, alignment) wraps to 0 when nbytes is within (alignment - 1) of SIZE_MAX, which silently bypassed the size overflow guard in gguf_init_from_reader. Reject the tensor before padding when nbytes +… 33 arXiv — Machine Learning research 14d ago Preferent Compression Bounds Are Tight arXiv:2609.36030v1 Announce Type: new Abstract: The lack of rigorous safety and performance certificates remains a key bottleneck to the deployment of modern learning-based methods. Sample compression has recently emerged as a powerful tool for deriving such certificates, with… 28 arXiv — Machine Learning research 14d ago FLOORA: A Human-Aligned Domain-Specific Language Model for Architectural Design arXiv:2609.36064v1 Announce Type: new Abstract: Foundation models are powerful generators, but many engineering domains require structured representations that general-purpose systems handle poorly. We introduce FLOORA (Floor Layout Optimization with RL Alignment), a family of… 24 arXiv — Machine Learning research 14d ago Guard Models Are Overconfident Where Base Models Are Uncertain arXiv:2609.36477v1 Announce Type: new Abstract: Guard models are used as safety classifiers, with confidence scores driving downstream moderation decisions. We evaluate five guard models for prompt classification and find that although several are nearly calibrated on clean… 27 arXiv — NLP / Computation & Language research 14d ago Alignment Forecasting: Predicting Misalignment From Training Data arXiv:2609.35805v1 Announce Type: new Abstract: Training a language model on data with a narrow flaw can sometimes make the model broadly misaligned. Inspecting the data at face value often does not settle whether it will emerge, and today it is caught only after training, by… 15 arXiv — NLP / Computation & Language research 14d ago Environment Steering: Using Data Flow Control to Improve Agent Utility and Safety arXiv:2609.35807v1 Announce Type: new Abstract: LLM agents can make unsafe tool calls even when instructed to behave safely. Existing defenses constrain agents before execution, modify tool inputs/outputs, or rely on LLM judges; these approaches may depend on model behavior or… 36 arXiv — NLP / Computation & Language research 14d ago Tracing mechanisms of sycophantic agreement in language models arXiv:2609.35822v1 Announce Type: new Abstract: Sycophantic agreement in language models refers to the tendency to overly affirm a user's stated beliefs or preferences, often at the expense of factual accuracy. Although it is widely recognized as an alignment failure, its… 8 arXiv — NLP / Computation & Language research 14d ago PADM\'E: Preference Alignment Data Synthesis for Meta-Evaluation of LM Agent Evaluators arXiv:2609.36086v1 Announce Type: new Abstract: Language models are frequently employed to evaluate other language models. An LM evaluator scoring agentic behaviors across multiple criteria is valuable, provided that its decisions align with human judgment. We call the problem… 16 arXiv — NLP / Computation & Language research 14d ago SCOUT: Synergizing Reasoning and Tool-Use for Computer-Use Safety arXiv:2609.36201v1 Announce Type: new Abstract: Computer-use agents (CUAs), while capable of completing computer tasks in everyday and professional workflows, can cause unintended harm even under benign instructions and environments. However, detecting such harm remains… 36 arXiv — NLP / Computation & Language research 14d ago Cognitive Expert Language Models Better Align with the Corresponding Brain Systems arXiv:2609.36239v1 Announce Type: new Abstract: Large language models (LLMs) can predict human brain activity across a variety of brain regions during natural language comprehension. Typically, however, LLM-brain alignment is measured using one model for different regions of the… 15 arXiv — NLP / Computation & Language research 14d ago FinRT: Distilling Adaptive Red-Teaming Strategies into Reusable Adversarial Generators in Consumer Finance arXiv:2609.36474v1 Announce Type: new Abstract: In regulated industries like consumer finance, seemingly harmless user queries can exploit large language model vulnerabilities, triggering safety failures and pushing responses dangerously close to policy limits. Existing… 32 The Information — AI news-outlet 14d ago President Trump Announces ‘Super Intelligence’ Accord, Renames AI President Donald Trump announced on Tuesday that leading AI companies had agreed to non-binding commitments on AI safety provisions, including internal controls on model alignment and external auditing. The “White House Accord on Super Intelligence,” which Trump posted on Truth… 21 TechCrunch — AI news-outlet 15d ago Here’s why OpenAI is absent from Nvidia’s industry-wide effort to end rogue AI agents OpenAI isn't a public supporter of Nvidia's Open Agent Safety Platform, but it is privately working with Nvidia, TechCrunch has learned. 38 arXiv — Machine Learning research 15d ago Cosine Similarity Is Not Evidence: Measuring the Noise Floor of Interpretability Transfer Under Quantization arXiv:2609.30275v1 Announce Type: new Abstract: A statistic reported without the quantity needed to interpret it is not evidence. We develop that thesis for a concrete practice in AI safety. Interpretability artifacts are calibrated on full-precision weights, deployed on… 10 arXiv — Machine Learning research 15d ago Guarded Gradient-Based Activation Steering of Shutdown Responses in Qwen3.5-0.8B: A Minimum-Step Policy arXiv:2609.30326v1 Announce Type: new Abstract: Activation steering changes a model's internal activations during inference without updating its weights, but a useful intervention must determine both how and when to steer. Motivated by the AI-safety concern that a model expected… 27 arXiv — Machine Learning research 15d ago Learning Provable Neural Network Observer for Uncertain Dynamical Systems arXiv:2609.30819v1 Announce Type: new Abstract: In many safety-critical applications, control of uncertain dynamical systems relies on observers that estimate states and external disturbances. Neural network observers can improve estimation accuracy, but certifying their… 21 arXiv — Machine Learning research 15d ago Learning Chance-Constrained MDPs with Bellman Distributional Certificates arXiv:2609.30856v1 Announce Type: new Abstract: Safe reinforcement learning (RL) commonly enforces expected-cost constraints, but such expectation safety may fail to control the probability of rare high-cost trajectories. Chance-constrained MDPs (CCMDPs) impose a stronger… 34 The Information — AI news-outlet 15d ago OpenAI Will Not Ship GPT-6.1 Astra Due to Safety Concerns OpenAI has decided not to release the AI model that it had intended to release as GPT-6.1 Astra due to its results on safety tests, a spokesperson said Monday. The model performed worse than GPT-6 Astra—OpenAI’s newest model, released earlier this month—on tests measuring… 35 Page 2 of 10 · 500 articles ← Newer Older →
