#209 – Rose Chan Loui on OpenAI’s gambit to ditch its nonprofit

"For every far-out idea that turns out to be true, there were probably hundreds that were simply crackpot ideas. In general, [science] advances building on the knowledge we have, and seeing what the next questions are, and then getting to the next stage and the next stage and so on. And occasionally there’ll be revolutionary ideas which will really completely change your view of science. And it is possible that some revolutionary breakthrough in our understanding will come about and we might crack this problem, but there’s no evidence for that. It doesn’t mean that there isn’t a lot of promising work going on. There are many legitimate areas which could lead to real improvements in health in old age. So I’m fairly balanced: I think there are promising areas, but there’s a lot of work to be done to see which area is going to be promising, and what the risks are, and how to make them work." —Venki RamakrishnanIn today’s episode, host Luisa Rodriguez speaks to Venki Ramakrishnan — molecular biologist and Nobel Prize winner — about his new book, Why We Die: The New Science of Aging and the Quest for Immortality.Links to learn more, highlights, and full transcript.They cover:What we can learn about extending human lifespan — if anything — from “immortal” aquatic animal species, cloned sheep, and the oldest people to have ever lived.Which areas of anti-ageing research seem most promising to Venki — including caloric restriction, removing senescent cells, cellular reprogramming, and Yamanaka factors — and which Venki thinks are overhyped.Why eliminating major age-related diseases might only extend average lifespan by 15 years.The social impacts of extending healthspan or lifespan in an ageing population — including the potential danger of massively increasing inequality if some people can access life-extension interventions while others can’t.And plenty more.Chapters:Cold open (00:00:00)Luisa's intro (00:01:04)The interview begins (00:02:21)Reasons to explore why we age and die (00:02:35)Evolutionary pressures and animals that don't biologically age (00:06:55)Why does ageing cause us to die? (00:12:24)Is there a hard limit to the human lifespan? (00:17:11)Evolutionary tradeoffs between fitness and longevity (00:21:01)How ageing resets with every generation, and what we can learn from clones (00:23:48)Younger blood (00:31:20)Freezing cells, organs, and bodies (00:36:47)Are the goals of anti-ageing research even realistic? (00:43:44)Dementia (00:49:52)Senescence (01:01:58)Caloric restriction and metabolic pathways (01:11:45)Yamanaka factors (01:34:07)Cancer (01:47:44)Mitochondrial dysfunction (01:58:40)Population effects of extended lifespan (02:06:12)Could increased longevity increase inequality? (02:11:48)What’s surprised Venki about this research (02:16:06)Luisa's outro (02:19:26)Producer: Keiran HarrisAudio engineering: Ben Cordell, Milo McGuire, Simon Monsour, and Dominic ArmstrongContent editing: Luisa Rodriguez, Katy Moore, and Keiran HarrisTranscriptions: Katy Moore

19 Syys 20242h 20min

"Perception is quite difficult with cameras: even if you have a stereo camera, you still can’t really build a map of where everything is in space. It’s just very difficult. And I know that sounds surprising, because humans are very good at this. In fact, even with one eye, we can navigate and we can clear the dinner table. But it seems that we’re building in a lot of understanding and intuition about what’s happening in the world and where objects are and how they behave. For robots, it’s very difficult to get a perfectly accurate model of the world and where things are. So if you’re going to go manipulate or grasp an object, a small error in that position will maybe have your robot crash into the object, a delicate wine glass, and probably break it. So the perception and the control are both problems." —Ken GoldbergIn today’s episode, host Luisa Rodriguez speaks to Ken Goldberg — robotics professor at UC Berkeley — about the major research challenges still ahead before robots become broadly integrated into our homes and societies.Links to learn more, highlights, and full transcript.They cover:Why training robots is harder than training large language models like ChatGPT.The biggest engineering challenges that still remain before robots can be widely useful in the real world.The sectors where Ken thinks robots will be most useful in the coming decades — like homecare, agriculture, and medicine.Whether we should be worried about robot labour affecting human employment.Recent breakthroughs in robotics, and what cutting-edge robots can do today.Ken’s work as an artist, where he explores the complex relationship between humans and technology.And plenty more.Chapters:Cold open (00:00:00)Luisa's intro (00:01:19)General purpose robots and the “robotics bubble” (00:03:11)How training robots is different than training large language models (00:14:01)What can robots do today? (00:34:35)Challenges for progress: fault tolerance, multidimensionality, and perception (00:41:00)Recent breakthroughs in robotics (00:52:32)Barriers to making better robots: hardware, software, and physics (01:03:13)Future robots in home care, logistics, food production, and medicine (01:16:35)How might robot labour affect the job market? (01:44:27)Robotics and art (01:51:28)Luisa's outro (02:00:55)Producer: Keiran HarrisAudio engineering: Dominic Armstrong, Ben Cordell, Milo McGuire, and Simon MonsourContent editing: Luisa Rodriguez, Katy Moore, and Keiran HarrisTranscriptions: Katy Moore

13 Syys 20242h 1min

"It’s very hard to find examples where people say, 'I’m starting from this point. I’m starting from this belief.' So we wanted to make that very legible to people. We wanted to say, 'Experts think this; accurate forecasters think this.' They might both be wrong, but we can at least start from here and figure out where we’re coming into a discussion and say, 'I am much less concerned than the people in this report; or I am much more concerned, and I think people in this report were missing major things.' But if you don’t have a reference set of probabilities, I think it becomes much harder to talk about disagreement in policy debates in a space that’s so complicated like this." —Ezra KargerIn today’s episode, host Luisa Rodriguez speaks to Ezra Karger — research director at the Forecasting Research Institute — about FRI’s recent Existential Risk Persuasion Tournament to come up with estimates of a range of catastrophic risks.Links to learn more, highlights, and full transcript.They cover:How forecasting can improve our understanding of long-term catastrophic risks from things like AI, nuclear war, pandemics, and climate change.What the Existential Risk Persuasion Tournament (XPT) is, how it was set up, and the results.The challenges of predicting low-probability, high-impact events.Why superforecasters’ estimates of catastrophic risks seem so much lower than experts’, and which group Ezra puts the most weight on.The specific underlying disagreements that superforecasters and experts had about how likely catastrophic risks from AI are.Why Ezra thinks forecasting tournaments can help build consensus on complex topics, and what he wants to do differently in future tournaments and studies.Recent advances in the science of forecasting and the areas Ezra is most excited about exploring next.Whether large language models could help or outperform human forecasters.How people can improve their calibration and start making better forecasts personally.Why Ezra thinks high-quality forecasts are relevant to policymakers, and whether they can really improve decision-making.And plenty more.Chapters:Cold open (00:00:00)Luisa’s intro (00:01:07)The interview begins (00:02:54)The Existential Risk Persuasion Tournament (00:05:13)Why is this project important? (00:12:34)How was the tournament set up? (00:17:54)Results from the tournament (00:22:38)Risk from artificial intelligence (00:30:59)How to think about these numbers (00:46:50)Should we trust experts or superforecasters more? (00:49:16)The effect of debate and persuasion (01:02:10)Forecasts from the general public (01:08:33)How can we improve people’s forecasts? (01:18:59)Incentives and recruitment (01:26:30)Criticisms of the tournament (01:33:51)AI adversarial collaboration (01:46:20)Hypotheses about stark differences in views of AI risk (01:51:41)Cruxes and different worldviews (02:17:15)Ezra’s experience as a superforecaster (02:28:57)Forecasting as a research field (02:31:00)Can large language models help or outperform human forecasters? (02:35:01)Is forecasting valuable in the real world? (02:39:11)Ezra’s book recommendations (02:45:29)Luisa's outro (02:47:54)Producer: Keiran HarrisAudio engineering: Dominic Armstrong, Ben Cordell, Milo McGuire, and Simon MonsourContent editing: Luisa Rodriguez, Katy Moore, and Keiran HarrisTranscriptions: Katy Moore

4 Syys 20242h 49min

"I do think that there is a really significant sentiment among parts of the opposition that it’s not really just that this bill itself is that bad or extreme — when you really drill into it, it feels like one of those things where you read it and it’s like, 'This is the thing that everyone is screaming about?' I think it’s a pretty modest bill in a lot of ways, but I think part of what they are thinking is that this is the first step to shutting down AI development. Or that if California does this, then lots of other states are going to do it, and we need to really slam the door shut on model-level regulation or else they’re just going to keep going. "I think that is like a lot of what the sentiment here is: it’s less about, in some ways, the details of this specific bill, and more about the sense that they want this to stop here, and they’re worried that if they give an inch that there will continue to be other things in the future. And I don’t think that is going to be tolerable to the public in the long run. I think it’s a bad choice, but I think that is the calculus that they are making." —Nathan CalvinIn today’s episode, host Luisa Rodriguez speaks to Nathan Calvin — senior policy counsel at the Center for AI Safety Action Fund — about the new AI safety bill in California, SB 1047, which he’s helped shape as it’s moved through the state legislature.Links to learn more, highlights, and full transcript.They cover:What’s actually in SB 1047, and which AI models it would apply to.The most common objections to the bill — including how it could affect competition, startups, open source models, and US national security — and which of these objections Nathan thinks hold water.What Nathan sees as the biggest misunderstandings about the bill that get in the way of good public discourse about it.Why some AI companies are opposed to SB 1047, despite claiming that they want the industry to be regulated.How the bill is different from Biden’s executive order on AI and voluntary commitments made by AI companies.Why California is taking state-level action rather than waiting for federal regulation.How state-level regulations can be hugely impactful at national and global scales, and how listeners could get involved in state-level work to make a real difference on lots of pressing problems.And plenty more.Chapters:Cold open (00:00:00)Luisa's intro (00:00:57)The interview begins (00:02:30)What risks from AI does SB 1047 try to address? (00:03:10)Supporters and critics of the bill (00:11:03)Misunderstandings about the bill (00:24:07)Competition, open source, and liability concerns (00:30:56)Model size thresholds (00:46:24)How is SB 1047 different from the executive order? (00:55:36)Objections Nathan is sympathetic to (00:58:31)Current status of the bill (01:02:57)How can listeners get involved in work like this? (01:05:00)Luisa's outro (01:11:52)Producer and editor: Keiran HarrisAudio engineering by Ben Cordell, Milo McGuire, Simon Monsour, and Dominic ArmstrongAdditional content editing: Katy Moore and Luisa RodriguezTranscriptions: Katy Moore

29 Elo 20241h 12min

"This is a group of animals I think people are particularly unfamiliar with. They are especially poorly covered in our science curriculum; they are especially poorly understood, because people don’t spend as much time learning about them at museums; and they’re just harder to spend time with in a lot of ways, I think, for people. So people have pets that are vertebrates that they take care of across the taxonomic groups, and people get familiar with those from going to zoos and watching their behaviours there, and watching nature documentaries and more. But I think the insects are still really underappreciated, and that means that our intuitions are probably more likely to be wrong than with those other groups." —Meghan BarrettIn today’s episode, host Luisa Rodriguez speaks to Meghan Barrett — insect neurobiologist and physiologist at Indiana University Indianapolis and founding director of the Insect Welfare Research Society — about her work to understand insects’ potential capacity for suffering, and what that might mean for how humans currently farm and use insects. If you're interested in getting involved with this work, check out Meghan's recent blog post: I’m into insect welfare! What’s next?Links to learn more, highlights, and full transcript.They cover:The scale of potential insect suffering in the wild, on farms, and in labs.Examples from cutting-edge insect research, like how depression- and anxiety-like states can be induced in fruit flies and successfully treated with human antidepressants.How size bias might help explain why many people assume insects can’t feel pain.Practical solutions that Meghan’s team is working on to improve farmed insect welfare, such as standard operating procedures for more humane slaughter methods.Challenges facing the nascent field of insect welfare research, and where the main research gaps are.Meghan’s personal story of how she went from being sceptical of insect pain to working as an insect welfare scientist, and her advice for others who want to improve the lives of insects.And much more.Chapters:Cold open (00:00:00)Luisa's intro (00:01:02)The interview begins (00:03:06)What is an insect? (00:03:22)Size diversity (00:07:24)How important is brain size for sentience? (00:11:27)Offspring, parental investment, and lifespan (00:19:00)Cognition and behaviour (00:23:23)The scale of insect suffering (00:27:01)Capacity to suffer (00:35:56)The empirical evidence for whether insects can feel pain (00:47:18)Nociceptors (01:00:02)Integrated nociception (01:08:39)Response to analgesia (01:16:17)Analgesia preference (01:25:57)Flexible self-protective behaviour (01:31:19)Motivational tradeoffs and associative learning (01:38:45)Results (01:43:31)Reasons to be sceptical (01:47:18)Meghan’s probability of sentience in insects (02:10:20)Views of the broader entomologist community (02:18:18)Insect farming (02:26:52)How much to worry about insect farming (02:40:56)Inhumane slaughter and disease in insect farms (02:44:45)Inadequate nutrition, density, and photophobia (02:53:50)Most humane ways to kill insects at home (03:01:33)Challenges in researching this (03:07:53)Most promising reforms (03:18:44)Why Meghan is hopeful about working with the industry (03:22:17)Careers (03:34:08)Insect Welfare Research Society (03:37:16)Luisa's outro (03:47:01)Producer and editor: Keiran HarrisAudio engineering by Ben Cordell, Milo McGuire, Simon Monsour, and Dominic ArmstrongAdditional content editing: Katy Moore and Luisa RodriguezTranscriptions: Katy Moore

26 Elo 20243h 48min

The three biggest AI companies — Anthropic, OpenAI, and DeepMind — have now all released policies designed to make their AI models less likely to go rogue or cause catastrophic damage as they approach, and eventually exceed, human capabilities. Are they good enough?That’s what host Rob Wiblin tries to hash out in this interview (recorded May 30) with Nick Joseph — one of the original cofounders of Anthropic, its current head of training, and a big fan of Anthropic’s “responsible scaling policy” (or “RSP”). Anthropic is the most safety focused of the AI companies, known for a culture that treats the risks of its work as deadly serious.Links to learn more, highlights, video, and full transcript.As Nick explains, these scaling policies commit companies to dig into what new dangerous things a model can do — after it’s trained, but before it’s in wide use. The companies then promise to put in place safeguards they think are sufficient to tackle those capabilities before availability is extended further. For instance, if a model could significantly help design a deadly bioweapon, then its weights need to be properly secured so they can’t be stolen by terrorists interested in using it that way.As capabilities grow further — for example, if testing shows that a model could exfiltrate itself and spread autonomously in the wild — then new measures would need to be put in place to make that impossible, or demonstrate that such a goal can never arise.Nick points out what he sees as the biggest virtues of the RSP approach, and then Rob pushes him on some of the best objections he’s found to RSPs being up to the task of keeping AI safe and beneficial. The two also discuss whether it's essential to eventually hand over operation of responsible scaling policies to external auditors or regulatory bodies, if those policies are going to be able to hold up against the intense commercial pressures that might end up arrayed against them.In addition to all of that, Nick and Rob talk about:What Nick thinks are the current bottlenecks in AI progress: people and time (rather than data or compute).What it’s like working in AI safety research at the leading edge, and whether pushing forward capabilities (even in the name of safety) is a good idea.What it’s like working at Anthropic, and how to get the skills needed to help with the safe development of AI.And as a reminder, if you want to let us know your reaction to this interview, or send any other feedback, our inbox is always open at podcast@80000hours.org.Chapters:Cold open (00:00:00)Rob’s intro (00:01:00)The interview begins (00:03:44)Scaling laws (00:04:12)Bottlenecks to further progress in making AIs helpful (00:08:36)Anthropic’s responsible scaling policies (00:14:21)Pros and cons of the RSP approach for AI safety (00:34:09)Alternatives to RSPs (00:46:44)Is an internal audit really the best approach? (00:51:56)Making promises about things that are currently technically impossible (01:07:54)Nick’s biggest reservations about the RSP approach (01:16:05)Communicating “acceptable” risk (01:19:27)Should Anthropic’s RSP have wider safety buffers? (01:26:13)Other impacts on society and future work on RSPs (01:34:01)Working at Anthropic (01:36:28)Engineering vs research (01:41:04)AI safety roles at Anthropic (01:48:31)Should concerned people be willing to take capabilities roles? (01:58:20)Recent safety work at Anthropic (02:10:05)Anthropic culture (02:14:35)Overrated and underrated AI applications (02:22:06)Rob’s outro (02:26:36)Producer and editor: Keiran HarrisAudio engineering by Ben Cordell, Milo McGuire, Simon Monsour, and Dominic ArmstrongVideo engineering: Simon MonsourTranscriptions: Katy Moore

22 Elo 20242h 29min

"In the 1980s, it was still apparently common to perform surgery on newborn babies without anaesthetic on both sides of the Atlantic. This led to appalling cases, and to public outcry, and to campaigns to change clinical practice. And as soon as [some courageous scientists] looked for evidence, it showed that this practice was completely indefensible and then the clinical practice was changed. People don’t need convincing anymore that we should take newborn human babies seriously as sentience candidates. But the tale is a useful cautionary tale, because it shows you how deep that overconfidence can run and how problematic it can be. It just underlines this point that overconfidence about sentience is everywhere and is dangerous." —Jonathan BirchIn today’s episode, host Luisa Rodriguez speaks to Dr Jonathan Birch — philosophy professor at the London School of Economics — about his new book, The Edge of Sentience: Risk and Precaution in Humans, Other Animals, and AI. (Check out the free PDF version!)Links to learn more, highlights, and full transcript.They cover:Candidates for sentience, such as humans with consciousness disorders, foetuses, neural organoids, invertebrates, and AIsHumanity’s history of acting as if we’re sure that such beings are incapable of having subjective experiences — and why Jonathan thinks that that certainty is completely unjustified.Chilling tales about overconfident policies that probably caused significant suffering for decades.How policymakers can act ethically given real uncertainty.Whether simulating the brain of the roundworm C. elegans or Drosophila (aka fruit flies) would create minds equally sentient to the biological versions.How new technologies like brain organoids could replace animal testing, and how big the risk is that they could be sentient too.Why Jonathan is so excited about citizens’ assemblies.Jonathan’s conversation with the Dalai Lama about whether insects are sentient.And plenty more.Chapters:Cold open (00:00:00)Luisa’s intro (00:01:20)The interview begins (00:03:04)Why does sentience matter? (00:03:31)Inescapable uncertainty about other minds (00:05:43)The “zone of reasonable disagreement” in sentience research (00:10:31)Disorders of consciousness: comas and minimally conscious states (00:17:06)Foetuses and the cautionary tale of newborn pain (00:43:23)Neural organoids (00:55:49)AI sentience and whole brain emulation (01:06:17)Policymaking at the edge of sentience (01:28:09)Citizens’ assemblies (01:31:13)The UK’s Sentience Act (01:39:45)Ways Jonathan has changed his mind (01:47:26)Careers (01:54:54)Discussing animal sentience with the Dalai Lama (01:59:08)Luisa’s outro (02:01:04)Producer and editor: Keiran HarrisAudio engineering by Ben Cordell, Milo McGuire, Simon Monsour, and Dominic ArmstrongAdditional content editing: Katy Moore and Luisa RodriguezTranscriptions: Katy Moore

15 Elo 20242h 1min

"Computational systems have literally millions of physical and conceptual components, and around 98% of them are embedded into your infrastructure without you ever having heard of them. And an inordinate amount of them can lead to a catastrophic failure of your security assumptions. And because of this, the Iranian secret nuclear programme failed to prevent a breach, most US agencies failed to prevent multiple breaches, most US national security agencies failed to prevent breaches. So ensuring your system is truly secure against highly resourced and dedicated attackers is really, really hard." —Sella NevoIn today’s episode, host Luisa Rodriguez speaks to Sella Nevo — director of the Meselson Center at RAND — about his team’s latest report on how to protect the model weights of frontier AI models from actors who might want to steal them.Links to learn more, highlights, and full transcript.They cover:Real-world examples of sophisticated security breaches, and what we can learn from them.Why AI model weights might be such a high-value target for adversaries like hackers, rogue states, and other bad actors.The many ways that model weights could be stolen, from using human insiders to sophisticated supply chain hacks.The current best practices in cybersecurity, and why they may not be enough to keep bad actors away.New security measures that Sella hopes can mitigate with the growing risks.Sella’s work using machine learning for flood forecasting, which has significantly reduced injuries and costs from floods across Africa and Asia.And plenty more.Also, RAND is currently hiring for roles in technical and policy information security — check them out if you're interested in this field! Chapters:Cold open (00:00:00)Luisa’s intro (00:00:56)The interview begins (00:02:30)The importance of securing the model weights of frontier AI models (00:03:01)The most sophisticated and surprising security breaches (00:10:22)AI models being leaked (00:25:52)Researching for the RAND report (00:30:11)Who tries to steal model weights? (00:32:21)Malicious code and exploiting zero-days (00:42:06)Human insiders (00:53:20)Side-channel attacks (01:04:11)Getting access to air-gapped networks (01:10:52)Model extraction (01:19:47)Reducing and hardening authorised access (01:38:52)Confidential computing (01:48:05)Red-teaming and security testing (01:53:42)Careers in information security (01:59:54)Sella’s work on flood forecasting systems (02:01:57)Luisa’s outro (02:04:51)Producer and editor: Keiran HarrisAudio engineering team: Ben Cordell, Simon Monsour, Milo McGuire, and Dominic ArmstrongAdditional content editing: Katy Moore and Luisa RodriguezTranscriptions: Katy Moore

1 Elo 20242h 8min