#191 (Part 1) – Carl Shulman on the economy and national security after AGI

This is the first part of our marathon interview with Carl Shulman. The second episode is on government and society after AGI. You can listen to them in either order!

The human brain does what it does with a shockingly low energy supply: just 20 watts — a fraction of a cent worth of electricity per hour. What would happen if AI technology merely matched what evolution has already managed, and could accomplish the work of top human professionals given a 20-watt power supply?

Many people sort of consider that hypothetical, but maybe nobody has followed through and considered all the implications as much as Carl Shulman. Behind the scenes, his work has greatly influenced how leaders in artificial general intelligence (AGI) picture the world they’re creating.

Carl simply follows the logic to its natural conclusion. This is a world where 1 cent of electricity can be turned into medical advice, company management, or scientific research that would today cost $100s, resulting in a scramble to manufacture chips and apply them to the most lucrative forms of intellectual labour.

It’s a world where, given their incredible hourly salaries, the supply of outstanding AI researchers quickly goes from 10,000 to 10 million or more, enormously accelerating progress in the field.

It’s a world where companies operated entirely by AIs working together are much faster and more cost-effective than those that lean on humans for decision making, and the latter are progressively driven out of business.

It’s a world where the technical challenges around control of robots are rapidly overcome, leading to robots into strong, fast, precise, and tireless workers able to accomplish any physical work the economy requires, and a rush to build billions of them and cash in.

It’s a world where, overnight, the number of human beings becomes irrelevant to rates of economic growth, which is now driven by how quickly the entire machine economy can copy all its components. Looking at how long it takes complex biological systems to replicate themselves (some of which can do so in days) that occurring every few months could be a conservative estimate.

It’s a world where any country that delays participating in this economic explosion risks being outpaced and ultimately disempowered by rivals whose economies grow to be 10-fold, 100-fold, and then 1,000-fold as large as their own.

As the economy grows, each person could effectively afford the practical equivalent of a team of hundreds of machine ‘people’ to help them with every aspect of their lives.

And with growth rates this high, it doesn’t take long to run up against Earth’s physical limits — in this case, the toughest to engineer your way out of is the Earth’s ability to release waste heat. If this machine economy and its insatiable demand for power generates more heat than the Earth radiates into space, then it will rapidly heat up and become uninhabitable for humans and other animals.

This eventually creates pressure to move economic activity off-planet. There’s little need for computer chips to be on Earth, and solar energy and minerals are more abundant in space. So you could develop effective populations of billions of scientific researchers operating on computer chips orbiting in space, sending the results of their work, such as drug designs, back to Earth for use.

These are just some of the wild implications that could follow naturally from truly embracing the hypothetical: what if we develop artificial general intelligence that could accomplish everything that the most productive humans can, using the same energy supply?

In today’s episode, Carl explains the above, and then host Rob Wiblin pushes back on whether that’s realistic or just a cool story, asking:

• If we’re heading towards the above, how come economic growth remains slow now and not really increasing?
• Why have computers and computer chips had so little effect on economic productivity so far?
• Are self-replicating biological systems a good comparison for self-replicating machine systems?
• Isn’t this just too crazy and weird to be plausible?
• What bottlenecks would be encountered in supplying energy and natural resources to this growing economy?
• Might there not be severely declining returns to bigger brains and more training?
• Wouldn’t humanity get scared and pull the brakes if such a transformation kicked off?
• If this is right, how come economists don’t agree and think all sorts of bottlenecks would hold back explosive growth?

Finally, Carl addresses the moral status of machine minds themselves. Would they be conscious or otherwise have a claim to moral or rights? And how might humans and machines coexist with neither side dominating or exploiting the other?

Learn more and read the full transcript on the 80,000 Hours website.

This episode was originally released in June 2024.

Chapters:

• Cold open (00:00:00)
• Rob’s intro (00:01:00)
• Transitioning to a world where AI systems do almost all the work (00:05:21)
• Economics after an AI explosion (00:14:25)
• Objection: Shouldn’t we be seeing economic growth rates increasing today? (00:59:12)
• Objection: Speed of doubling time (01:07:33)
• Objection: Declining returns to increases in intelligence? (01:11:59)
• Objection: Physical transformation of the environment (01:17:39)
• Objection: Should we expect an increased demand for safety and security? (01:29:14)
• Objection: “This sounds completely whack” (01:36:10)
• Income and wealth distribution (01:48:02)
• Economists and the intelligence explosion (02:13:31)
• Baumol effect arguments (02:19:12)
• Denying that robots can exist (02:27:18)
• Classic economic growth models (02:36:12)
• Robot nannies (02:48:27)
• Slow integration of decision-making and authority power (02:57:39)
• Economists’ mistaken heuristics (03:01:07)
• Moral status of AIs (03:11:45)
• Rob’s outro (04:11:47)

Producer and editor: Keiran Harris
Audio engineering lead: Ben Cordell
Technical editing: Simon Monsour, Milo McGuire, and Dominic Armstrong
Transcriptions: Katy Moore