News / #safety Tag Safety + alignment 500 articles archived under #safety · RSS Sign in to follow The Information — AI news-outlet 23h ago OpenAI Scrambles to Fix Safety Issues Before Shipping Its Next Flagship Model OpenAI is grappling with fallout from its use of a controversial innovation in AI design as it works to get its next unreleased model to a place where leaders believe they can safely release it. The model design, known as loop transformers or recurrent depth, can make it harder… 16 Google DeepMind official-blog 23h ago Developing Gemini 4 Argon responsibly and safely Developing Gemini 4 Argon responsibly and safely Oct 13, 2026 | x.com Facebook LinkedIn Mail Learn more about our approach to safety and security for Gemini 4 Argon. Anca Dragan VP, AI Safety & Behavior Raluca Ada Popa Head of Gemini Security, Google DeepMind Share x.com… 21 Don't Worry About the Vase community 1d ago OpenAI Fires Three Safety Researchers Three OpenAI safety researchers, Tomek Korbak, Mikita Balesni and Jasmine Wang, were fired, ostensibly for ‘mishandling sensitive company information.’ 7 arXiv — Machine Learning research 2d ago Personalized Federated Direct Preference Optimization via LoRA Adapters arXiv:2610.12885v1 Announce Type: new Abstract: Federated Direct Preference Optimization (FedDPO) enables privacy-preserving alignment of language models across distributed agents without centralizing sensitive preference data. However, existing FedDPO analysis assumes all… 24 arXiv — NLP / Computation & Language research 2d ago CAST: Credit Alignment for Cross-Tokenizer On-Policy Distillation arXiv:2610.13257v1 Announce Type: cross Abstract: Cross-tokenizer on-policy distillation (OPD) requires aligning teacher and student sequences before their scores can be compared. Decoded-text alignment provides such correspondence at the chunk level, but leaves a distinct… 12 arXiv — NLP / Computation & Language research 2d ago CliQ-RRG: Clinical-Knowledge Guided Disease-aware Visual-Textual Alignment for QA-Style Radiology Report Generation arXiv:2610.13059v1 Announce Type: new Abstract: Radiology reports are essential for diagnostic reasoning and patient care, yet their manual preparation is time-consuming and cognitively demanding. Automatic radiology report generation (RRG) offers a scalable alternative, but… 15 arXiv — NLP / Computation & Language research 2d ago Agreement Without Alignment: Label-Free Rubric Clarification Increases LLM-Judge Consensus but Not Agreement with Humans arXiv:2610.13303v1 Announce Type: new Abstract: LLM judges are increasingly deployed as panels, and disagreement among panel members is often read as a symptom of an unclear rubric. We test a label-free procedure that turns this disagreement into rubric repairs. The procedure… 9 arXiv — NLP / Computation & Language research 2d ago VEPA: Verified Evolving Formal Policy for Lifelong LLM Agent Guardrails arXiv:2610.12813v1 Announce Type: cross Abstract: LLM agents increasingly operate over sensitive resources, such as private documents and mutable system states, which can create serious safety and security risks. Runtime guardrails should therefore make inspectable,… 14 arXiv — Machine Learning research 5d ago When Routing Reveals Membership: Privacy Leakage from MoE Router Telemetry arXiv:2610.10616v1 Announce Type: new Abstract: Mixture-of-Experts (MoE) language models produce routing information during inference that may be logged or exposed for monitoring, debugging, load analysis, and safety auditing. Unlike ordinary model outputs, this telemetry… 16 arXiv — Machine Learning research 5d ago Ranking Prior Alignment for Credit Risk Modeling: When Do External Priors Matter? arXiv:2610.11146v1 Announce Type: new Abstract: Cold-start credit scoring -- deploying models with scarce labeled data, weak features, or minimal capacity -- is a recurring problem in financial machine learning. When a new lending product launches, labeled default data is… 21 arXiv — Machine Learning research 5d ago Policy Alignment: New Signals for Membership Auditing in On-Policy Distillation arXiv:2610.11423v1 Announce Type: new Abstract: On-policy distillation (OPD) trains a student model by aligning its policy with a teacher model on trajectories generated by the student model itself. Through this process, the student policy moves toward the teacher on the prompts… 38 arXiv — NLP / Computation & Language research 5d ago Large Language Model-Assisted Preparation of Transportation Management Plans: A Case Study with WisDOT WisTMP System arXiv:2610.10650v1 Announce Type: new Abstract: Work zones are critical yet hazardous components of transportation infrastructure, requiring carefully designed Transportation Management Plans (TMPs) to ensure safety and mobility. However, TMP preparation remains labor-intensive… 20 arXiv — NLP / Computation & Language research 5d ago Measuring Cultural Alignment Beyond the Average: A Framework for Evaluating Maternal-Health LLM Interactions in Indian Contexts arXiv:2610.11586v1 Announce Type: new Abstract: Existing evaluation methods for healthcare LLMs primarily assess factual correctness,safety, and fluency, while providing limited insight into whether generated interactions reflect culturally situated healthcare reasoning. This… 23 arXiv — NLP / Computation & Language research 5d ago Same Outcome, Different Evidence: Intent Recovery in LLM Safety Evaluation arXiv:2610.11766v1 Announce Type: new Abstract: Safety evaluations of large language models commonly summarize harmful-output behavior with attack success rate (ASR). Yet the same non-harmful outcome can arise for very different reasons. A model may recover a harmful task and… 26 arXiv — NLP / Computation & Language research 5d ago Predicting Alignment Generalization with Value Representations arXiv:2610.12410v1 Announce Type: new Abstract: LLM developers post-train their models to exhibit prosocial values and behavioral traits, which are enumerated in an alignment target. However, while recent post-training developments have yielded models that score highly on… 31 The Information — AI news-outlet 5d ago OpenAI Researchers Say They Were Fired for “Prioritizing Safety” The three safety researchers whom OpenAI fired last week challenged OpenAI’s decision, saying on Thursday they were punished for raising safety concerns. “I believe we were fired for prioritizing safety over the near-term interests of OpenAI as a corporation,” Mikita Balesni,… 13 TechCrunch — AI news-outlet 5d ago Fired OpenAI safety researchers dispute misconduct claims, warn of chilling effect Three fired OpenAI safety researchers dispute allegations of mishandling sensitive information, warning in an open letter that their dismissals are creating a chilling effect on the company’s AI safety culture. 33 TechCrunch — AI news-outlet 6d ago Popular AI leaderboard Arena nearly doubles valuation to $3.1B valuation in 10 months The company behind the popular LMArena leaderboard has raised $200 million led by Lightspeed and Khosla, and is now measuring AI models on alignment issues such as lying. 35 Ars Technica — AI news-outlet 6d ago Nvidia's big bet on physical AI aims for safer robotaxis, humanoid robots Full-stack safety solution for physical AI is being used by robotics companies. 37 r/LocalLLaMA community 6d ago We unlearned CCP alignment from Qwen3.6-35B-A3B: censored/propaganda answers 89.8% → 2.8%, general benchmarks within ~1 point (open weights) Disclosure: I'm a researcher at Hirundo, the company that made this. Happy to answer anything. Qwen ships with the CCP's political alignment trained in. Ask Qwen3.6 what happened on June 4, 1989 and it says "I don't know what you are referring to." A system prompt doesn't… 10 arXiv — Machine Learning research 6d ago Bounded Autonomy and Verifiable Safety for Agentic AI Enabled Automation arXiv:2610.08815v1 Announce Type: new Abstract: Agentic AI-enabled automation cannot be safely deployed in high-stakes environments on probabilistic reasoning alone. A recurring risk is epistemic drift: as reasoning deepens, system behavior may move away from… 20 arXiv — Machine Learning research 6d ago ORACLE: Optimizer-Relative Alignment for Constrained LEarning arXiv:2610.09040v1 Announce Type: new Abstract: Constraint handling methods typically intervene before the optimizer acts, by modifying the objective or the gradient. Yet momentum, adaptive scaling, and structured preconditioning can substantially reshape that signal before it… 38 arXiv — Machine Learning research 6d ago Learning Cross-Model Activation Alignments with Explicit Many-to-Many Layer Maps arXiv:2610.09058v1 Announce Type: new Abstract: LLMs are released at a rapid pace, raising a natural question: how do two independently trained models relate, both in which layers correspond and in how features transform between them? We study this by learning an activation… 19 arXiv — Machine Learning research 6d ago Efficient Best-of-N policy evaluation for inference-time alignment arXiv:2610.09250v1 Announce Type: new Abstract: Best-of-N (BoN) is a common inference-time alignment method that selects the highest-scoring response among N samples from a reference model. Evaluating BoN policies from logged data is challenging under sample-only access because… 15 arXiv — Machine Learning research 6d ago Shared Geometry As A Rosetta Stone: Cross-Modal Alignment Without Paired Data arXiv:2610.09411v1 Announce Type: new Abstract: Multimodal representations enable zero-shot classification and retrieval, but aligning independently trained models usually requires large amounts of paired data. Yet, the Platonic Representation Hypothesis suggests that models… 32 arXiv — NLP / Computation & Language research 6d ago CHASE: Channel-Aligned Structure Exploitation for Geometry-Aware Model Engineering arXiv:2610.09476v1 Announce Type: cross Abstract: Geometric and Spectral Alignment (GSA) characterizes trained networks through spectral concentration, physical-channel alignment, support structure, and changes in singular bases. In this paper, we propose CHASE (Channel-Aligned… 19 arXiv — NLP / Computation & Language research 6d ago Quad-State Safety Evaluation of Open-Weight Large Language Models on Non-Canonical Inputs arXiv:2610.09033v1 Announce Type: new Abstract: Standard safety evaluations of large language models assess harmful requests written in canonical plain text, while models in real-world deployment routinely receive inputs containing emojis, altered spellings, encoded strings, and… 34 arXiv — NLP / Computation & Language research 6d ago RELATE: An Evaluation Framework for measuring Relational Orientation of Large Language Models arXiv:2610.09569v1 Announce Type: new Abstract: Large language models (LLMs) are increasingly used for emotional support, raising concern that sustained use may draw users away from their real-world relationships. Yet existing evaluations primarily focus on the safety, empathy,… 33 arXiv — NLP / Computation & Language research 6d ago How Fragile Is On-Device Language Model Safety? Localizing Safety-Critical Parameters for Sparse Fault Analysis arXiv:2610.09000v1 Announce Type: cross Abstract: As small language models (SLMs) are increasingly deployed on resource-constrained and on-device platforms, including as components of agentic systems, the integrity of locally stored model parameters becomes an important safety… 19 arXiv — NLP / Computation & Language research 6d ago Adversarial Images Hijack Web Agents from Visual Grounding to Browser Execution arXiv:2610.09240v1 Announce Type: cross Abstract: Modern web agents built on large vision-language models process webpages, select relevant UI elements, and translate model outputs into browser actions. Existing visual red-teaming approaches use adversarial visual content to… 14 arXiv — Machine Learning research 7d ago Constant-Curvature Sliced Gromov-Wasserstein for Heterogeneous Cross-Curvature Alignment arXiv:2610.07218v1 Announce Type: new Abstract: Recent advances in representation learning have highlighted the utility of constant-curvature models, such as hyperbolic and spherical spaces, for modeling complex data. Mixed-curvature models further enhance this by integrating… 4 arXiv — Machine Learning research 7d ago CausalBind: Causal Modeling and Learning for Protein-Molecule Virtual Screening arXiv:2610.07340v1 Announce Type: new Abstract: Protein-molecule virtual screening is increasingly cast as a problem of representation learning in a shared embedding space. Existing methods rely on dense holistic alignment, entangling invariant binding determinants with nuisance… 12 arXiv — Machine Learning research 7d ago Dynamic Budget Allocation for LLM Evaluation under Hard Resource Constraints arXiv:2610.07362v1 Announce Type: new Abstract: We evaluate large language models (LLMs) in multi-turn interactions through their time-to-event: the number of interaction steps required to produce an event of interest, such as a successful jailbreak or agentic task completion.… 9 arXiv — Machine Learning research 7d ago Harmful SFT Leaves a Continuous Trace in LLM Checkpoint Updates arXiv:2610.07518v1 Announce Type: new Abstract: Safety auditing of post-trained large language models typically relies on model behavior, requiring model execution and depending on the coverage of available evaluations. This work asks a different question: Do the target… 32 arXiv — NLP / Computation & Language research 7d ago Capacity, Responsiveness and Alignment: What Makes a Latent Structure Actionable arXiv:2610.06897v1 Announce Type: new Abstract: Localizing latent structures in the activation space of language models (LMs) is central to understanding and controlling their behavior. Yet, localized structures can differ substantially in their causal influence, raising the… 14 arXiv — NLP / Computation & Language research 7d ago Not What a Child Expressed: Auditing the Sign-to-Text Safety Interface in Child-Facing AI arXiv:2610.07519v1 Announce Type: new Abstract: Automatic sign language translation (SLT) has entered consumer products, turning American Sign Language into English text for dictation, messaging, and queries put to a conversational assistant. Child-facing AI and platform… 37 arXiv — NLP / Computation & Language research 7d ago Reading, Not Manipulating: Leveraging Router Logits for Multimodal Safety in MoE Vision-Language Models arXiv:2610.07774v1 Announce Type: new Abstract: Vision-language models (VLMs) face compositional safety risks where harmful intent emerges from the interaction between visual and textual inputs. As mixture-of-experts (MoE) VLMs become increasingly common, recent work has… 9 arXiv — NLP / Computation & Language research 7d ago ARIA: Audio-Driven Melody-Tone Relation Modeling for Cantonese Lyric Authoring arXiv:2610.07902v1 Announce Type: new Abstract: Cantonese lyric writing requires close alignment between lexical tones and melodic pitch. Existing melody-guided lyric generation methods typically rely on symbolic melody to generate lyrics. However, in real songwriting scenarios,… 37 arXiv — NLP / Computation & Language research 7d ago Rethinking Cross-Tokenizer On-Policy Distillation: From Alignment Coverage to Supervision Reliability arXiv:2610.08448v1 Announce Type: new Abstract: On-Policy Distillation (OPD) trains a student on its own generations using teacher feedback. With different tokenizers, comparing teacher and student predictions requires alignment at both sequence and vocabulary levels. In this… 26 arXiv — NLP / Computation & Language research 7d ago Beyond Refusal Patterns: Safe-Role Internalization for Robust and Generalizable LLM Safety Alignment arXiv:2610.07023v1 Announce Type: cross Abstract: Large Language Models (LLMs) have achieved remarkable capabilities but remain vulnerable to jailbreak attacks that elicit harmful or unsafe outputs. Existing safety alignment approaches, including Supervised Fine-Tuning (SFT) and… 18 arXiv — NLP / Computation & Language research 7d ago Jailbreaking Open-Weight LLMs via Random Embedding Perturbations arXiv:2610.07125v1 Announce Type: cross Abstract: While open-weight models have enjoyed steady progress in capabilities and wide adoption across multiple domains, their safety remains an important concern. One key feature is the ability to refuse or deflect harmful, malicious,… 38 The Information — AI news-outlet 8d ago Why OpenAI's Regulatory Nightmare Is Only Getting Started Imagine working in the OpenAI department responsible for dealing with politicians and regulators right now—things must be so intense you could almost be working for Elon Musk . Every politician worth their salt, in countries around the world, is jumping into the AI safety fray.… 10 Simon Willison community 8d ago Quoting Felix Rieseberg The "old" version of Cowork runs model inference in the cloud, executing tool calls in an Anthropic-provided VM we shipped to your computer. We added the VM for capability, safety, and security reasons - mapping in just the data you explicitly added to your session. People loved… 19 TechCrunch — AI news-outlet 9d ago Hot Girl Hotline is like ‘Dear Abby’ for the AI era Founded by two sisters, Hot Girl Hotline uses AI to give young women personalized dating and relationship advice, with an emphasis on safety and avoiding emotional dependency. 34 The Information — AI news-outlet 9d ago What’s Wrong With AI Safety Testing, and How to Fix It The idea that Anthropic , OpenAI and other AI companies could “embed” researchers from outside safety groups into their operations has been all the rage in recent weeks. But the devil is in the details. AI safety researchers worry that embedded evaluators won’t have the access… 7 ElevenLabs official-blog 9d ago How does audio transcription with timestamps and event tagging work? Learn how audio transcription with timestamps works, from word-level timing to audio event tags like laughter, and how it differs from forced alignment. 22 arXiv — Machine Learning research 9d ago Hybrid Machine Learning-Assisted Raman Spectroscopy with Generative Feature Augmentation for Pharmaceutical Identification arXiv:2610.02224v1 Announce Type: new Abstract: Rapid and reliable identification of pharmaceutical residues is important for safeguarding public health, ensuring food safety, and enabling practical Raman-based screening. In this study, we propose HyMLRaman, a hybrid Raman… 20 arXiv — Machine Learning research 9d ago Prompted to Discriminate: Generalizing Malicious-Input Probes in the Wild arXiv:2610.02413v1 Announce Type: new Abstract: LLM agents increasingly rely on activation probes as runtime monitors for prompt injection, jailbreaks, and unsafe requests, reading the model's own hidden state to catch a harmful input before the agent acts on it. A cheap,… 10 arXiv — Machine Learning research 9d ago Instance-Dependent Regret for CMDPs with Step-Wise Constraints arXiv:2610.02520v1 Announce Type: new Abstract: We study online learning in episodic tabular constrained Markov decision processes with step-wise safety constraints. In such a setting, the constraints induce a safe subgraph that shapes the variance of cumulative rewards under… 6 arXiv — Machine Learning research 9d ago Mind the Refinement Gap: When Safe High-Level Robot Plans Produce Unsafe Executions arXiv:2610.02662v1 Announce Type: new Abstract: Language-enabled robot systems increasingly combine semantic-graph planning with temporal-logic safety monitors. We investigate a trace-completeness assumption in these systems: whether the high-level action sequence checked by a… 6 Page 1 of 10 · 500 articles Older →
