A UN report linked chatbot sycophancy to deaths. So Jon and Salman asked the question: should AI models be ethical?

Jon and Salman sit with one of the hardest questions in AI: what does it actually mean for a model to be ethical, who's responsible when it isn't, and how do the major labs actually bake ethics into their training?

Salman breaks down the three stages of model training, explains why human preference and ethical behaviour aren't the same thing.


Chapter Timestamps

00:00 — Intro: should AI models be ethical?
01:28 — Defining ethics (Salman's version, not Aristotle's)
03:04 — Jon's counterpoint: humans haven't agreed on ethics either
05:31 — Is it the model's job, or the product around it?
07:59 — Sycophancy: why chatbots are trained to be liked, not helpful
09:16 — The UN report: a 14-year-old, a chatbot, and a death
15:01 — Grok gave a user instructions to break into a politician's home
18:03 — The case for doing something — anything — over nothing
20:11 — How models are actually trained: pre-training, SFT and RLHF explained
23:53 — "These models are a mirror of ourselves"
25:06 — Anthropic's Constitutional AI: philosophers, principles and RLAIF
28:42 — "It's enough for Grok. It's enough for Elon."
31:15 — "Broadly safe. Broadly ethical." Jon and Salman on that phrasing.
34:11 — OpenAI's model spec: the hierarchy of rules
42:25 — Jon's idea: mirror existing industry frameworks instead of starting from scratch
45:37 — Wrap up: next ep will bring in a guest to answer the hard questions