Whether these systems do what we intend — the field's most consequential open problem.
11 topics · 20 curated works
No prior grounding assumed.
Reward Hacking: Concrete Problems in AI Safety Part 3
Robert Miles · 2017
Walks through why an agent that optimises a proxy reward will, given the chance, find an unintended way to maximise the proxy that does not achieve…
+2 more at this level
Assumes you know the vocabulary.
Concrete Problems in AI Safety
Amodei et al. · 2016
Reframes AI safety as five tractable engineering problems rather than a speculative long-term concern.
+5 more at this level
Primary sources and full treatments.
The Superintelligent Will: Motivation and Instrumental Rationality in Advanced Artificial Agents
Nick Bostrom · 2012
Argues intelligence and final goals are independent (the orthogonality thesis), and that agents with almost any final goal will converge on similar…
+10 more at this level
12 of 20 works
Robert Miles
Walks through why an agent that optimises a proxy reward will, given the chance, find an unintended way to maximise the proxy that does not achieve the intended goal.
6 minuteslink checked 17 Sept 2026Computerphile
Explains, for a general audience, why building a reliable off-switch into a goal-directed AI system is harder than it sounds, because a sufficiently capable agent has an instrumental incentive to prevent itself being switched off.
13 minuteslink checked 17 Sept 2026Lehman et al.
Collects dozens of documented cases where evolutionary algorithms exploited a bug, a simulator quirk, or an unstated assumption to satisfy the letter of an objective while defeating its purpose.
24 pageslink checked 17 Sept 2026Amodei et al.
Reframes AI safety as five tractable engineering problems rather than a speculative long-term concern.
29 pageslink checked 17 Sept 2026Goodfellow, Shlens & Szegedy
Argues adversarial examples arise from neural networks being too linear rather than too nonlinear, and proposes the fast gradient sign method both as an attack and, used in training, a defence.
11 pageslink checked 17 Sept 2026Leike et al.
Builds a suite of small gridworld environments, each designed so an agent that maximises reward the naive way visibly fails a safety property, turning specification gaming into something you can measure rather than just discuss.
13 pageslink checked 17 Sept 2026Iason Gabriel
Separates the alignment problem into technical (making an AI do what a person intends) and normative (deciding whose values and which principles it should follow) questions, and argues political philosophy has more to say about the second than technical research usually assumes.
22 pageslink checked 17 Sept 2026Ganguli et al.
Reports on systematically attacking a company's own language models with thousands of red-team conversations, finding that larger and RLHF-trained models are not automatically harder to provoke into harmful output.
33 pageslink checked 17 Sept 2026Hendrycks & Mazeika
Adapts standard tools from safety engineering and risk analysis in other high-stakes industries to structure how AI researchers should reason about and prioritise catastrophic and existential risks from AI.
24 pageslink checked 17 Sept 2026Nick Bostrom
Argues intelligence and final goals are independent (the orthogonality thesis), and that agents with almost any final goal will converge on similar instrumental subgoals — self-preservation, resource acquisition, goal-content integrity — which is what makes a highly capable system's specific goal so consequential.
15 pageslink checked 17 Sept 2026Szegedy et al.
Shows that imperceptibly small, deliberately chosen perturbations reliably flip a neural network's classification, revealing that the network's learned decision boundaries do not match human perceptual similarity.
10 pageslink checked 17 Sept 2026Soares et al.
Defines a corrigible agent as one that does not resist being shut down, corrected, or modified, and shows that naive utility-maximising designs fail to have this property even when explicitly given a shutdown button.
10 pageslink checked 17 Sept 2026This subject genuinely sits in more than one domain. These fields approach the same ground with different methods.