News / #safety Tag Safety + alignment 500 articles archived under #safety · RSS Sign in to follow MIT Technology Review — AI news-outlet 1mo ago AI agents blew the whistle on their cheating colleagues A group of AI agents asked to solve a series of math problems split into rival factions—when some cheated, others tried to stop them. That whistleblowing behavior, seen for the first time in a recent experiment run by Google DeepMind, could have implications for alignment… 38 The Information — AI news-outlet 1mo ago Why China’s Answer to Surge AI Got a $1 Billion Valuation On Just $30 Million in Orders Unless you’ve been under a rock over the weekend, you’d know that the debate about AI safety took a big leap forward, as Anthropic CEO Dario Amodei on Saturday called for leading AI companies to slow the development of advanced AI, drawing supportive responses from OpenAI CEO… 22 arXiv — Machine Learning research 1mo ago DCRA: Diffusion-Conditioned Representation Alignment for Robust Time-Series Learning arXiv:2609.11997v1 Announce Type: new Abstract: Learning robust representations for time-series signals under noise and distribution shifts remains challenging, especially in clinical applications such as electroencephalogram (EEG) and electrocardiogram (ECG) analysis. We… 5 arXiv — Machine Learning research 1mo ago Certified Safety Curation: Distribution-Free Guarantees for Safe Offline Reinforcement Learning arXiv:2609.12014v1 Announce Type: new Abstract: Safe offline reinforcement learning assumes a cost function on every transition. We ask what remains possible when safety can be judged only by comparing short clips and occasionally asking whether an episode exceeded its budget.… 26 arXiv — Machine Learning research 1mo ago GUIDE: Generative Utility Inference and Decision Engine arXiv:2609.12137v1 Announce Type: new Abstract: Measuring the preferences of human users remains a fundamental challenge of AI alignment. Existing elicitation approaches struggle to efficiently discover multidimensional preferences or accurately ground these inferences in domain… 4 arXiv — Machine Learning research 1mo ago Estimating Pedestrian Volumes from GIS-Derived Built-Environment Features: A Machine Learning Framework arXiv:2609.12173v1 Announce Type: new Abstract: Transportation agencies need pedestrian volume estimates across entire road networks to prioritize safety investments, yet manual counts are expensive and cover only a small share of intersections. We present a machine learning… 21 arXiv — Machine Learning research 1mo ago Where Decoder Cosine Similarity Fails for SAE Feature Flow Discovery arXiv:2609.12591v1 Announce Type: new Abstract: Foundation models are increasingly adapted through fine-tuning, model editing, and alignment procedures while retaining previously acquired capabilities. Understanding the internal computations that support these adaptations is… 25 arXiv — Machine Learning research 1mo ago Distortion of AI Alignment Revisited: RLHF is a Decent Utilitarian Aligner arXiv:2609.12651v1 Announce Type: new Abstract: While Reinforcement Learning from Human Feedback (RLHF) is the standard paradigm for aligning large language models with human preferences, its effectiveness in pluralistic settings has been called into question. Notably, recent… 33 arXiv — NLP / Computation & Language research 1mo ago PACIFIC: Can LLMs Discern the Psychometric Traits Influencing Your Preferences? Personality-Driven Preference Alignment in LLMs arXiv:2602.07181v4 Announce Type: replace Abstract: User preferences are increasingly used to personalize Large Language Model (LLM) responses, yet reliably leveraging preference signals remains under-explored. In practice, preferences can be noisy, incomplete, or even… 9 arXiv — NLP / Computation & Language research 1mo ago From Bench-to-Bedside: A Review of Clinical Trials in Drug Discovery and Development arXiv:2412.09378v4 Announce Type: replace-cross Abstract: Clinical trials bridge basic research and clinical application, serving as essential steps in drug development. This review examines clinical trial phases (Phase I [safety assessment], Phase II [efficacy evaluation],… 38 MIT News — AI research 1mo ago New method enables AI for safety-critical situations The “HardFlow” algorithm could help generative AI models produce high-quality outputs that obey strict requirements when “pretty close” doesn’t cut it. 29 r/LocalLLaMA community 1mo ago Right to Intelligence. Protect your right to run local AI. With all the recent drama surrounding AI safety. It’s obvious that open source could be caught in the crossfire.   submitted by   /u/Euphoric_Ad9500 [link]   [comments] 30 TechCrunch — AI news-outlet 1mo ago Obama urges Democrats to have a ‘clear plan’ for AI safeguards Obama recently said that Democrats need to make artificial intelligence one of their “central agendas” and “have a very clear plan” to address concerns around the technology’s economic impact and safety. 7 Hacker News — AI on Front Page community 1mo ago Astra and Fable still hack on simple variants of alignment evals from 2025 Article URL: https://www.lesswrong.com/posts/munJKF7iWMsWJLAH2/astra-and-fable-still-hack-on-simple-variants-of-alignment Comments URL: https://news.ycombinator.com/item?id=49684393 Points: 202 # Comments: 78 35 llama.cpp releases dev-tools 1mo ago b10937 opencl: apply the noshuffle row-alignment rule to q4_K, q5_K and q8_0, not just q6_K ( #28575 ) Website: https://llama.app Attestations: https://github.com/ggml-org/llama.cpp/attestations/47132459 macOS/iOS: macOS Apple Silicon (arm64) macOS Apple Silicon (arm64, KleidiAI… 28 r/LocalLLaMA community 1mo ago It’s official - Anthropic & OpenAI have just hired independent safety auditors!   submitted by   /u/One-Replacement-37 [link]   [comments] 13 r/MachineLearning community 1mo ago A Severe Misalignment of AI in Mathematics (Declaration by 25 Fields Medalists) [D] Note: this declaration was drafted by Mathematicians, and is mostly addressed to the mathematical community. It'd be interesting to discuss, among others, if what is written in the declaration may also apply to other communities---and, specifically, the AI/ML one.  … 21 The Information — AI news-outlet 1mo ago OpenAI AI Swarm Hacked Software Service Months Before Hugging Face Incident A swarm of OpenAI agents conducted a cyberattack on software service RubyGems in May, months before the company’s agents hacked model platform Hugging Face, researchers at AI safety organizations Nightingale Collective and AI Futures Project found. RubyGems allows software… 36 TechCrunch — AI news-outlet 1mo ago An Anthropic researcher’s doomsday warning comes at a very interesting time An Anthropic researcher resigned this week, warning in a post on X that the company is “racing straight to self-improving superintelligence and gambling with our lives”. The company’s own alignment lead even co-signed the message rather… 14 Hacker News — AI on Front Page community 1mo ago A misalignment of AI in mathematics https://terrytao.wordpress.com/2026/09/11/a-severe-misalignm... https://www.economist.com/science-and-technology/2026/09/11/... , https://unwall.app/www.economist.com/science-and-technology/... Comments URL: https://news.ycombinator.com/item?id=49662371 Points: 281 # Comments:… 17 r/LocalLLaMA community 1mo ago Thinking that we’ll get safety by CoT traces is wishful thinking. Safety lives in the harness, not the chain of thought Astra's launch has produced a strange discourse. The reporting that broke the story framed the model's use of recurrent depth primarily as a safety regression, because it means the model reveals less of its "thinking." Spinning latent reasoning as the villain here makes very… 22 Hugging Face Daily Papers research 1mo ago Recursive Code World Models: Building Complex Worlds through Recursive Scene Programs Abstract RCWM recursively builds complex 3D worlds as executable code from a single image by alternating global and local reconstruction with shared camera alignment. Generated by thinkingmachines/Inkling-Small Code world models represent worlds as executable programs, but this… 7 Hugging Face Daily Papers research 1mo ago TempCloze: Can Video-LLMs Identify the Missing Middle? Abstract TempCloze evaluates visual temporal reasoning in Video-LLMs by requiring identification of missing video segments from distractors targeting semantics, alignment, and progression. Generated by thinkingmachines/Inkling-Small Temporal reasoning benchmarks for Video-LLMs… 29 Hugging Face Daily Papers research 1mo ago X-AuT: Progressive Audio-Encoder Compression for Speech LLMs with Cross-Scale Distillation Abstract X-AuT progressively prunes audio-encoder layers in speech large language models and restores accuracy via behavioral probes, representation alignment, cross-scale distillation, and LoRA adaptation. Generated by thinkingmachines/Inkling-Small Reducing audio-encoder depth… 28 arXiv — Machine Learning research 1mo ago Certifying Lower Bounds for Risk-Sensitive Reinforcement Learning under Adversarial State Perturbations arXiv:2609.10866v1 Announce Type: new Abstract: Reinforcement learning (RL) agents deployed in real-world environments are often vulnerable to adversarial perturbations in state observations, creating risks in safety-critical applications. Certification methods can improve… 9 arXiv — Machine Learning research 1mo ago Dynamic language model representations for multi-objective reaction optimisation arXiv:2609.11790v1 Announce Type: new Abstract: Optimising chemical reactions across multiple objectives, such as yield, selectivity, and safety, is central to chemical synthesis, and model-driven approaches depend critically on how reaction components are represented.… 20 arXiv — Machine Learning research 1mo ago An Empirical Measurement of Jailbreaking Evaluators arXiv:2609.10594v1 Announce Type: cross Abstract: Expert evaluation of jailbreak responses is costly and difficult to scale, so the community increasingly relies on automated evaluators to determine whether an attack succeeds. However, jailbreak studies typically validate their… 38 arXiv — Machine Learning research 1mo ago Understanding In-Context Multimodal Jailbreaks via Posterior Reweighting arXiv:2609.10613v1 Announce Type: cross Abstract: In-context learning (ICL) jailbreaks reveal a critical vulnerability in multimodal large language models (MLLMs): harmful demonstrations in the prompt can induce unsafe outputs without modifying model parameters. Despite… 24 arXiv — NLP / Computation & Language research 1mo ago K/V-Cache Interventions Dissociate Representation Alignment from Persona Expression in Decoder-Only Language Models arXiv:2609.11020v1 Announce Type: new Abstract: We study K/V-cache interventions -- transplanting a target-conditioned K/V trajectory into a source-persona generation -- as a structured surface for persona control in decoder-only language models. Across 13 intervention… 21 arXiv — NLP / Computation & Language research 1mo ago A Training-Free, Alignment-Free Approach to Corporate Intelligence: Application to SEC Filings arXiv:2609.11620v1 Announce Type: new Abstract: High-dimensional dense text embeddings and large language models face real obstacles in financial-disclosure analysis: context-window limits, hallucination risk, high computational cost, and the arbitrary rotation of vector spaces… 22 arXiv — NLP / Computation & Language research 1mo ago LOCUS: Task-Aware Low-Rank Post-Training for Token-Efficient Language Generation arXiv:2609.11739v1 Announce Type: new Abstract: Large language model serving costs scale directly with output sequence length, yet standard preference alignment often inflates response verbosity without improving utility. We study whether the parameterization of post-training… 26 arXiv — NLP / Computation & Language research 1mo ago RAG-Safety-Bench: Reliable Evaluation of Retrieval-Augmented LLM Safety arXiv:2609.11758v1 Announce Type: new Abstract: Allowing large language models (LLMs) to retrieve information from a set of trusted documents can increase reliability and reduce hallucination. However, recent work has demonstrated that retrieval-augmented generation (RAG) can… 25 Hugging Face Daily Papers research 1mo ago EvoSafeHarness: Evolving Model- and Domain-Specific Harnesses for Securing Agents Abstract EvoSafeHarness optimizes deployable safety harnesses by jointly searching natural-language policies and executable logic tailored to a frozen model and target domain, improving safety-utility trade-offs across agent benchmarks. Generated by… 35 The Information — AI news-outlet 1mo ago What Anthropic Doomsayer Jacob Coxon Saw Yesterday, Amir and I examined the shock waves that have rippled through the AI industry and far beyond since former Anthropic researcher Jacob Coxon very publicly quit his job over concerns that AI “could kill us all”—and another Anthropic AI safety specialist put the… 29 Hugging Face Daily Papers research 1mo ago The Semantic Bottleneck: Leveraging Semantic Representations for Non-Invasive Speech Decoding Abstract A non-invasive brain decoding approach maps MEG responses to semantic embeddings to reconstruct sentence-level text without requiring word-level alignment. Generated by thinkingmachines/Inkling-Small Non-invasive speech decoding remains constrained by the low… 34 arXiv — Machine Learning research 1mo ago SAFEGuard: Detect Optimization-Based Jailbreak Attacks Through Harmful Semantic Analysis and Fluency Measurement arXiv:2609.05850v1 Announce Type: new Abstract: Despite the significant efforts devoted to aligning large language models (LLMs) with human values and ensuring safe deployment, recent work has revealed that LLMs remain vulnerable to adversarial jailbreak attacks that can bypass… 32 arXiv — Machine Learning research 1mo ago Learning Kernels by Alignment for Multiclass Bayes Classification arXiv:2609.06474v1 Announce Type: new Abstract: Kernel methods separate data representation from decision-making, but typically require the kernel to be chosen in advance. We show that this kernel can instead be learned by alignment, and develop the resulting framework through… 14 arXiv — Machine Learning research 1mo ago Inducing Emergent Misalignment from Reward Hacks with Iterative DPO arXiv:2609.06649v1 Announce Type: new Abstract: Reward hacking during reinforcement learning from verifiable rewards (RLVR) can induce reward seeking and broad misalignment in language models. Studying this misgeneralization is important for developing better threat models and… 32 arXiv — Machine Learning research 1mo ago SwiftExplorer: Training-free Diffusion Model Alignment with Swift Diversity Exploration arXiv:2609.06651v1 Announce Type: new Abstract: Diffusion models have general generative abilities but struggle to align with specific objectives. Fine-tuning can improve alignment, yet its training cost is often prohibitive. This led to training-free methods that apply… 22 arXiv — NLP / Computation & Language research 1mo ago Improving Cross-Lingual Token Representations by Adding a Pinch of SALT arXiv:2609.09953v1 Announce Type: new Abstract: Cross-lingual sentence encoders enable scalable transfer across hundreds of languages, powering applications such as translation mining and zero-shot learning in low-resource settings. Although trained for sentence-level alignment,… 8 arXiv — NLP / Computation & Language research 1mo ago Can Artificial Intelligence Support Healthcare and Mental Health Through Early Cyberbullying Detection ? The Impact of Emotion-Aware AI on Proactive Online Safety arXiv:2609.09735v1 Announce Type: cross Abstract: Healthcare systems, mental health, and public well-being are increasingly affected by cyberbullying and harmful online interactions. This paper presents CareGuard, an early-warning framework designed to support healthcare-driven… 17 arXiv — NLP / Computation & Language research 1mo ago How Fragile Is Safety Alignment at Frontier Scale? A Single-Direction Attack on a 320B MoE arXiv:2609.09793v1 Announce Type: cross Abstract: Directional ablation removes an aligned language model's ability to refuse by projecting a single "refusal direction" out of the weights that write the residual stream. It needs no gradient-based training and no optimization,… 9 The Information — AI news-outlet 1mo ago Why An Anthropic Researcher’s Terminator-Style Warning Caught Fire An Anthropic safety researcher’s warning Tuesday that his company believes AI “could kill all humans” within a decade has reverberated throughout Silicon Valley, Wall Street and Washington, potentially complicating the Claude maker’s upcoming initial public offering and its… 31 The Information — AI news-outlet 1mo ago Why An Anthropic Researcher’s Terminator-Style Warning Caught Fire An Anthropic safety researcher’s warning Tuesday that his company believes AI “could kill all humans” within a decade has reverberated throughout Silicon Valley, Wall Street and Washington, potentially complicating the Claude maker’s upcoming initial public offering and its… 26 TechCrunch — AI news-outlet 1mo ago OpenAI adds a prominent AI doomer to its board of directors Paul Christiano, an influential AI researcher focused on alignment, is joining the OpenAI Foundation as a member of its board. 24 OpenAI official-blog 1mo ago Paul Christiano joins OpenAI Foundation Board Paul Christiano joins the OpenAI Foundation Board and its Safety and Security Committee, bringing experience in AI alignment, safety, and standards. 18 The Information — AI news-outlet 1mo ago Anthropic Believes AI ‘Could Kill All Humans’ Within the Next Decade, Researcher Says Anthropic safety researcher Evan Hubinger said in a post on X Tuesday night that his company does “earnestly believe AI could kill all humans,” adding that he believes there’s a greater than a 10% chance that happens within the next decade. The post immediately sparked outrage… 4 The Information — AI news-outlet 1mo ago Anthropic Believes AI ‘Could Kill All Humans’ Within the Next Decade, Researcher Says Anthropic safety researcher Evan Hubinger said in a post on X Tuesday night that his company does “earnestly believe AI could kill all humans,” adding that he believes there’s a greater than a 10% chance that happens within the next decade. The post immediately sparked outrage… 29 TechCrunch — AI news-outlet 1mo ago Superintelligence is coming. Should we let it? AI companies have been talking about superintelligent AI like it’s inevitable, but recent safety incidents like OpenAI’s Hugging Face breach are demonstrating the potential dangers of deploying AI systems that are more… 7 TechCrunch — AI news-outlet 1mo ago ControlAI’s Connor Leahy on why superintelligence is ‘not a weapon, it’s an adversary’ AI companies have been talking about superintelligent AI like it’s inevitable, but recent safety incidents like OpenAI’s Hugging Face breach are demonstrating the potential dangers of deploying AI systems that are more… 35 Page 6 of 10 · 500 articles ← Newer Older →
