Winamp Logo
The Lunar Society Cover
The Lunar Society Profile

The Lunar Society

English, Social, 1 season, 64 episodes, 4 days, 5 hours, 33 minutes
About
Host Dwarkesh Patel interviews intellectuals, scientists, and founders about their big ideas. YouTube: https://www.youtube.com/DwarkeshPatel Apple Podcasts: https://apple.co/3oBack9 Spotify: https://spoti.fi/3S5g2YK www.dwarkeshpatel.com (https://www.dwarkeshpatel.com?utm_medium=podcast)
Episode Artwork

Jung Chang - Living through Cultural Revolution and the Crimes of Mao

A true honor to speak with Jung Chang.She is the author of Wild Swans: Three Daughters of China (sold 15+ million copies worldwide) and Mao: The Unknown Story.We discuss:- what it was like growing up during the Cultural Revolution as the daughter of a denounced official- why the CCP continues to worship the biggest mass murderer in human history.- how exactly Communist totalitarianism was able to subjugate a billion people- why Chinese leaders like Xi and Deng who suffered from the Cultural Revolution don't condemn Mao- how Mao starved and killed 40 million people during The Great Leap Forward in order to exchange food for Soviet weaponsWild Swans is the most moving book I've ever read. It was a real privilege to speak with its author.Watch on YouTube. Listen on Apple Podcasts, Spotify, or any other podcast platform. Read the full transcript here. Follow me on Twitter for updates on future episodes.Timestamps(00:00:00) - Growing up during Cultural Revolution(00:15:58) - Could officials have overthrown Mao?(00:34:09) - Great Leap Forward(00:48:12) - Modern support of Mao(01:03:24) - Life as peasant(01:21:30) - Psychology of communist society This is a public episode. If you would like to discuss this with other subscribers or get access to bonus episodes, visit www.dwarkeshpatel.com
11/29/20231 hour, 31 minutes, 15 seconds
Episode Artwork

Andrew Roberts - Leading Historian on Warfare from Napoleon to Ukraine

Andrew Roberts is the world's best biographer and one of the leading historians of our time.We discussed* Churchill the applied historian,* Napoleon the startup founder,* why Nazi ideology cost Hitler WW2,* drones, reconnaissance, and other aspects of the future of war,* Iraq, Afghanistan, Korea, Ukraine, & Taiwan.Watch on YouTube. Listen on Apple Podcasts, Spotify, or any other podcast platform. Read the full transcript here. Follow me on Twitter for updates on future episodes.Timestamps(00:00:00) - Post WW2 conflicts(00:10:57) - Ukraine(00:16:33) - How Truman Prevented Nuclear War(00:22:49) - Taiwan(00:27:15) - Churchill(00:35:11) - Gaza & future wars(00:39:05) - Could Hitler have won WW2?(00:48:00) - Surprise attacks(00:59:33) - Napoleon and startup founders(01:14:06) - Robert’s insane productivity This is a public episode. If you would like to discuss this with other subscribers or get access to bonus episodes, visit www.dwarkeshpatel.com
11/22/20231 hour, 18 minutes, 49 seconds
Episode Artwork

Dominic Cummings - COVID, Brexit, & Fixing Western Governance

Here is my interview with Dominic Cummings on why Western governments are so dangerously broken, and how to fix them before an even more catastrophic crisis.Dominic was Chief Advisor to the Prime Minister during COVID, and before that, director of Vote Leave (which masterminded the 2016 Brexit referendum).Watch on YouTube. Listen on Apple Podcasts, Spotify, or any other podcast platform. Read the full transcript here. Follow me on Twitter for updates on future episodes.Timestamps(00:00:00) - One day in COVID…(00:08:26) - Why is government broken?(00:29:10) - Civil service(00:38:27) - Opportunity wasted?(00:49:35) - Rishi Sunak and Number 10 vs 11(00:55:13) - Cyber, nuclear, bio risks(01:02:04) - Intelligence & defense agencies(01:23:32) - Bismarck & Lee Kuan Yew(01:37:46) - How to fix the government?(01:56:43) - Taiwan(02:00:10) - Russia(02:07:12) - Bismarck’s career as an example of AI (mis)alignment(02:17:37) - Odyssean education This is a public episode. If you would like to discuss this with other subscribers or get access to bonus episodes, visit www.dwarkeshpatel.com
11/15/20232 hours, 34 minutes, 13 seconds
Episode Artwork

Paul Christiano - Preventing an AI Takeover

Paul Christiano is the world’s leading AI safety researcher. My full episode with him is out!We discuss:- Does he regret inventing RLHF, and is alignment necessarily dual-use?- Why he has relatively modest timelines (40% by 2040, 15% by 2030),- What do we want post-AGI world to look like (do we want to keep gods enslaved forever)?- Why he’s leading the push to get to labs develop responsible scaling policies, and what it would take to prevent an AI coup or bioweapon,- His current research into a new proof system, and how this could solve alignment by explaining model's behavior- and much more.Watch on YouTube. Listen on Apple Podcasts, Spotify, or any other podcast platform. Read the full transcript here. Follow me on Twitter for updates on future episodes.Open PhilanthropyOpen Philanthropy is currently hiring for twenty-two different roles to reduce catastrophic risks from fast-moving advances in AI and biotechnology, including grantmaking, research, and operations.For more information and to apply, please see the application: https://www.openphilanthropy.org/research/new-roles-on-our-gcr-team/The deadline to apply is November 9th; make sure to check out those roles before they close.Timestamps(00:00:00) - What do we want post-AGI world to look like?(00:24:25) - Timelines(00:45:28) - Evolution vs gradient descent(00:54:53) - Misalignment and takeover(01:17:23) - Is alignment dual-use?(01:31:38) - Responsible scaling policies(01:58:25) - Paul’s alignment research(02:35:01) - Will this revolutionize theoretical CS and math?(02:46:11) - How Paul invented RLHF(02:55:10) - Disagreements with Carl Shulman(03:01:53) - Long TSMC but not NVIDIA This is a public episode. If you would like to discuss this with other subscribers or get access to bonus episodes, visit www.dwarkeshpatel.com
10/31/20233 hours, 7 minutes, 1 second
Episode Artwork

Shane Legg (DeepMind Founder) - 2028 AGI, New Architectures, Aligning Superhuman Models

I had a lot of fun chatting with Shane Legg - Founder and Chief AGI Scientist, Google DeepMind!We discuss:* Why he expects AGI around 2028* How to align superhuman models* What new architectures needed for AGI* Has Deepmind sped up capabilities or safety more?* Why multimodality will be next big landmark* and much moreWatch full episode on YouTube, Apple Podcasts, Spotify, or any other podcast platform. Read full transcript here.Timestamps(0:00:00) - Measuring AGI(0:11:41) - Do we need new architectures?(0:16:26) - Is search needed for creativity?(0:19:19) - Superhuman alignment(0:29:58) - Impact of Deepmind on safety vs capabilities(0:34:03) - Timelines(0:41:24) - Multimodality This is a public episode. If you would like to discuss this with other subscribers or get access to bonus episodes, visit www.dwarkeshpatel.com
10/26/202344 minutes, 19 seconds
Episode Artwork

Grant Sanderson (3Blue1Brown) - Past, Present, & Future of Mathematics

I had a lot of fun chatting with Grant Sanderson (who runs the excellent 3Blue1Brown YouTube channel) about:- Whether advanced math requires AGI- What careers should mathematically talented students pursue- Why Grant plans on doing a stint as a high school teacher- Tips for self teaching- Does Godel’s incompleteness theorem actually matter- Why are good explanations so hard to find?- And much moreWatch on YouTube. Listen on Spotify, Apple Podcasts, or any other podcast platform. Full transcript here.Timestamps(0:00:00) - Does winning math competitions require AGI?(0:08:24) - Where to allocate mathematical talent?(0:17:34) - Grant’s miracle year(0:26:44) - Prehistoric humans and math(0:33:33) - Why is a lot of math so new?(0:44:44) - Future of education(0:56:28) - Math helped me realize I wasn’t that smart(0:59:25) - Does Godel’s incompleteness theorem matter?(1:05:12) - How Grant makes videos(1:10:13) - Grant’s math exposition competition(1:20:44) - Self teaching This is a public episode. If you would like to discuss this with other subscribers or get access to bonus episodes, visit www.dwarkeshpatel.com
10/12/20231 hour, 31 minutes, 20 seconds
Episode Artwork

Sarah C. M. Paine - How Xi & Putin Think: Maritime vs Continental Powers & The Wars of Asia

I learned so much from Sarah Paine, Professor of History and Strategy at the Naval War College.We discuss:- how continental vs maritime powers think and how this explains Xi & Putin's decisions- how a war with China over Taiwan would shake out and whether it could go nuclear- why the British Empire fell apart, why China went communist, how Hitler and Japan could have coordinated to win WW2, and whether Japanese occupation was good for Korea, Taiwan and Manchuria- plus other lessons from WW2, Cold War, and Sino-Japanese War- how to study history properly, and why leaders keep making the same mistakesIf you want to learn more, check out her books - they’re some of the best military history I’ve ever read.Watch on YouTube, listen on Apple Podcasts, Spotify, or any other podcast platform. Read the full transcript.Timestamps(0:00:00) - Grand strategy(0:11:59) - Death ground(0:23:19) - WW1(0:39:23) - Writing history(0:50:25) - Japan in WW2(0:59:58) - Ukraine(1:10:50) - Japan/Germany vs Iraq/Afghanistan occupation(1:21:25) - Chinese invasion of Taiwan(1:51:26) - Communists & Axis(2:08:34) - Continental vs maritime powers This is a public episode. If you would like to discuss this with other subscribers or get access to bonus episodes, visit www.dwarkeshpatel.com
10/4/20232 hours, 24 minutes, 33 seconds
Episode Artwork

George Hotz vs Eliezer Yudkowsky AI Safety Debate

George Hotz and Eliezer Yudkowsky hashed out their positions on AI safety.It was a really fun debate. No promises but there might be a round 2 where we better hone in on the cruxes that we began to identify here.Watch the livestreamed YouTube version (high quality video will be up next week).Catch the Twitter stream.Listen on Apple Podcasts, Spotify, or any other podcast platform. Check back here in about 24 hours for the full transcript. This is a public episode. If you would like to discuss this with other subscribers or get access to bonus episodes, visit www.dwarkeshpatel.com
8/17/20231 hour, 28 minutes, 22 seconds
Episode Artwork

Dario Amodei (Anthropic CEO) - $10 Billion Models, OpenAI, Scaling, & AGI in 2 years

Here is my conversation with Dario Amodei, CEO of Anthropic.We discuss:- why human level AI is 2-3 years away- race dynamics with OpenAI and China- $10 billion training runs, bioterrorism, alignment, cyberattacks, scaling, ...Dario is hilarious and has fascinating takes on what these models are doing, why they scale so well, and what it will take to align them.---I’m running an experiment on this episode.I’m not doing an ad.Instead, I’m just going to ask you to pay for whatever value you feel you personally got out of this conversation.Pay here: https://bit.ly/3ONINtp---Watch on YouTube. Listen on Apple Podcasts, Spotify, or any other podcast platform. Read the full transcript here. Follow me on Twitter for updates on future episodes.Timestamps(00:02:03) - Scaling(00:16:49) - Language(00:24:01) - Economic Usefulness(00:39:08) - Bioterrorism(00:44:38) - Cybersecurity(00:48:22) - Alignment & mechanistic interpretability(00:58:46) - Does alignment research require scale?(01:06:33) - Misuse vs misalignment(01:10:09) - What if AI goes well?(01:12:08) - China(01:16:14) - How to think about alignment(01:30:21) - Manhattan Project(01:32:34) - Is modern security good enough?(01:37:12) - Inefficiencies in training(01:46:56) - Anthropic’s Long Term Benefit Trust(01:52:21) - Is Claude conscious?(01:57:17) - Keeping a low profile This is a public episode. If you would like to discuss this with other subscribers or get access to bonus episodes, visit www.dwarkeshpatel.com
8/8/20231 hour, 59 minutes, 46 seconds
Episode Artwork

Andy Matuschak - Self-Teaching, Spaced Repetition, & Why Books Don’t Work

A few weeks ago, I sat beside Andy Matuschak to record how he reads a textbook.Even though my own job is to learn things, I was shocked with how much more intense, painstaking, and effective his learning process was.So I asked if we could record a conversation about how he learns and a bunch of other topics:* How he identifies and interrogates his confusion (much harder than it seems, and requires an extremely effortful and slow pace)* Why memorization is essential to understanding and decision-making* How come some people (like Tyler Cowen) can integrate so much information without an explicit note taking or spaced repetition system.* How LLMs and video games will change education* How independent researchers and writers can make money* The balance of freedom and discipline in education* Why we produce fewer von Neumann-like prodigies nowadays* How multi-trillion dollar companies like Apple (where he was previously responsible for bedrock iOS features) manage to coordinate millions of different considerations (from the cost of different components to the needs of users, etc) into new products designed by 10s of 1000s of people.Watch on YouTube. Listen on Apple Podcasts, Spotify, or any other podcast platform. Read the full transcript here. Follow me on Twitter for updates on future episodes.To see Andy’s process in action, check out the video where we record him studying a quantum physics textbook, talking aloud about his thought process, and using his memory system prototype to internalize the material.You can check out his website and personal notes, and follow him on Twitter.CometeerVisit cometeer.com/lunar for $20 off your first order on the best coffee of your life!If you want to sponsor an episode, contact me at [email protected](00:02:32) - Skillful reading(00:04:10) - Do people care about understanding?(00:08:32) - Structuring effective self-teaching(00:18:17) - Memory and forgetting(00:34:50) - Andy’s memory practice(00:41:47) - Intellectual stamina(00:46:07) - New media for learning (video, games, streaming)(01:00:31) - Schools are designed for the median student(01:06:52) - Is learning inherently miserable?(01:13:37) - How Andy would structure his kids’ education(01:31:40) - The usefulness of hypertext(01:43:02) - How computer tools enable iteration(01:52:24) - Monetizing public work(02:10:16) - Spaced repetition(02:11:56) - Andy’s personal website and notes(02:14:24) - Working at Apple(02:21:05) - Spaced repetition 2 Get full access to The Lunar Society at www.dwarkeshpatel.com/subscribe
7/12/20232 hours, 25 minutes, 20 seconds
Episode Artwork

Carl Shulman (Pt 2) - AI Takeover, Bio & Cyber Attacks, Detecting Deception, & Humanity's Far Future

The second half of my 7 hour conversation with Carl Shulman is out!My favorite part! And the one that had the biggest impact on my worldview.Here, Carl lays out how an AI takeover might happen:* AI can threaten mutually assured destruction from bioweapons,* use cyber attacks to take over physical infrastructure,* build mechanical armies,* spread seed AIs we can never exterminate,* offer tech and other advantages to collaborating countries, etcPlus we talk about a whole bunch of weird and interesting topics which Carl has thought about:* what is the far future best case scenario for humanity* what it would look like to have AI make thousands of years of intellectual progress in a month* how do we detect deception in superhuman models* does space warfare favor defense or offense* is a Malthusian state inevitable in the long run* why markets haven't priced in explosive economic growth* & much moreCarl also explains how he developed such a rigorous, thoughtful, and interdisciplinary model of the biggest problems in the world.Watch on YouTube. Listen on Apple Podcasts, Spotify, or any other podcast platform. Read the full transcript here. Follow me on Twitter for updates on future episodes.Catch part 1 here80,000 hoursThis episode is sponsored by 80,000 hours. To get their free career guide (and to help out this podcast), please visit 80000hours.org/lunar.80,000 hours is without any close second the best resource to learn about the world’s most pressing problems and how you can solve them.If this conversation has got you concerned, and you want to get involved, then check out the excellent 80,000 hours guide on how to help with AI risk.To advertise on The Lunar Society, contact me at [email protected](00:02:50) - AI takeover via cyber or bio(00:34:30) - Can we coordinate against AI?(00:55:52) - Human vs AI colonizers(01:06:58) - Probability of AI takeover(01:23:59) - Can we detect deception?(01:49:28) - Using AI to solve coordination problems(01:58:04) - Partial alignment(02:13:44) - AI far future(02:25:07) - Markets & other evidence(02:35:29) - Day in the life of Carl Shulman(02:49:08) - Space warfare, Malthusian long run, & other rapid fireTranscript Get full access to The Lunar Society at www.dwarkeshpatel.com/subscribe
6/26/20233 hours, 9 minutes, 10 seconds
Episode Artwork

Carl Shulman - Intelligence Explosion, Primate Evolution, Robot Doublings, & Alignment

In terms of the depth and range of topics, this episode is the best I’ve done.No part of my worldview is the same after talking with Carl Shulman. He's the most interesting intellectual you've never heard of.We ended up talking for 8 hours, so I'm splitting this episode into 2 parts.This part is about Carl’s model of an intelligence explosion, which integrates everything from:* how fast algorithmic progress & hardware improvements in AI are happening,* what primate evolution suggests about the scaling hypothesis,* how soon before AIs could do large parts of AI research themselves, and whether there would be faster and faster doublings of AI researchers,* how quickly robots produced from existing factories could take over the economy.We also discuss the odds of a takeover based on whether the AI is aligned before the intelligence explosion happens, and Carl explains why he’s more optimistic than Eliezer.The next part, which I’ll release next week, is about all the specific mechanisms of an AI takeover, plus a whole bunch of other galaxy brain stuff.Maybe 3 people in the world have thought as rigorously as Carl about so many interesting topics. This was a huge pleasure.Watch on YouTube. Listen on Apple Podcasts, Spotify, or any other podcast platform. Read the full transcript here. Follow me on Twitter for updates on future episodes.Timestamps(00:00:00) - Intro(00:01:32) - Intelligence Explosion(00:18:03) - Can AIs do AI research?(00:39:00) - Primate evolution(01:03:30) - Forecasting AI progress(01:34:20) - After human-level AGI(02:08:39) - AI takeover scenarios Get full access to The Lunar Society at www.dwarkeshpatel.com/subscribe
6/14/20232 hours, 44 minutes, 16 seconds
Episode Artwork

Richard Rhodes - Making of Atomic Bomb, AI, WW2, Oppenheimer, & Abolishing Nukes

It was a tremendous honor & pleasure to interview Richard Rhodes, Pulitzer Prize winning author of The Making of the Atomic BombWe discuss* similarities between AI progress & Manhattan Project (developing a powerful, unprecedented, & potentially apocalyptic technology within an uncertain arms-race situation)* visiting starving former Soviet scientists during fall of Soviet Union* whether Oppenheimer was a spy, & consulting on the Nolan movie* living through WW2 as a child* odds of nuclear war in Ukraine, Taiwan, Pakistan, & North Korea* how the US pulled of such a massive secret wartime scientific & industrial projectWatch on YouTube. Listen on Apple Podcasts, Spotify, or any other podcast platform. Read the full transcript here. Follow me on Twitter for updates on future episodes.Timestamps(0:00:00) - Oppenheimer movie(0:06:22) - Was the bomb inevitable?(0:29:10) - Firebombing vs nuclear vs hydrogen bombs(0:49:44) - Stalin & the Soviet program(1:08:24) - Deterrence, disarmament, North Korea, Taiwan(1:33:12) - Oppenheimer as lab director(1:53:40) - AI progress vs Manhattan Project(1:59:50) - Living through WW2(2:16:45) - Secrecy(2:26:34) - Wisdom & warTranscript(0:00:00) - Oppenheimer movieDwarkesh Patel 0:00:51Today I have the great honor of interviewing Richard Rhodes, who is the Pulitzer Prize-winning author of The Making of the Atomic Bomb, and most recently, the author of Energy, A Human History. I'm really excited about this one. Let's jump in at a current event, which is the fact that there's a new movie about Oppenheimer coming out, which I understand you've been consulted about. What did you think of the trailer? What are your impressions? Richard Rhodes 0:01:22They've really done a good job of things like the Trinity test device, which was the sphere covered with cables of various kinds. I had watched Peaky Blinders, where the actor who's playing Oppenheimer also appeared, and he looked so much like Oppenheimer to start with. Oppenheimer was about six feet tall, he was rail thin, not simply in terms of weight, but in terms of structure. Someone said he could sit in a children's high chair comfortably. But he never weighed more than about 140 pounds and that quality is there in the actor. So who knows? It all depends on how the director decided to tell the story. There are so many aspects of the story that you could never possibly squeeze them into one 2-hour movie. I think that we're waiting for the multi-part series that would really tell a lot more of the story, if not the whole story. But it looks exciting. We'll see. There have been some terrible depictions of Oppenheimer, there've been some terrible depictions of the bomb program. And maybe they'll get this one right. Dwarkesh Patel 0:02:42Yeah, hopefully. It is always great when you get an actor who resembles their role so well. For example, Bryan Cranston who played LBJ, and they have the same physical characteristics of the beady eyes, the big ears. Since we're talking about Oppenheimer, I had one question about him. I understand that there's evidence that's come out that he wasn't directly a communist spy. But is there any possibility that he was leaking information to the Soviets or in some way helping the Soviet program? He was a communist sympathizer, right? Richard Rhodes 0:03:15He had been during the 1930s. But less for the theory than for the practical business of helping Jews escape from Nazi Germany. One of the loves of his life, Jean Tatlock, was also busy working on extracting Jews from Europe during the 30. She was a member of the Communist Party and she, I think, encouraged him to come to meetings. But I don't think there's any possibility whatsoever that he shared information. In fact, he said he read Marx on a train trip between Berkeley and Washington one time and thought it was a bunch of hooey, just ridiculous. He was a very smart man, and he read the book with an eye to its logic, and he didn't think there was much there. He really didn't know anything about human beings and their struggles. He was born into considerable wealth. There were impressionist paintings all over his family apartments in New York City. His father had made a great deal of money cornering the markets on uniform linings for military uniforms during and before the First World War so there was a lot of wealth. I think his income during the war years and before was somewhere around $100,000 a month. And that's a lot of money in the 1930s. So he just lived in his head for most of his early years until he got to Berkeley and discovered that prime students of his were living on cans of god-awful cat food, because they couldn't afford anything else. And once he understood that there was great suffering in the world, he jumped in on it, as he always did when he became interested in something. So all of those things come together. His brother Frank was a member of the party, as was Frank's wife. I think the whole question of Oppenheimer lying to the security people during the Second World War about who approached him and who was trying to get him to sign on to some espionage was primarily an effort to cover up his brother's involvement. Not that his brothers gave away any secrets, I don't think they did. But if the army's security had really understood Frank Oppenheimer's involvement, he probably would have been shipped off to the Aleutians or some other distant place for the duration of the war. And Oppenheimer quite correctly wanted Frank around. He was someone he trusted.(0:06:22) - Was the bomb inevitable?Dwarkesh Patel 0:06:22Let's start talking about The Making of the Bomb. One question I have is — if World War II doesn't happen, is there any possibility that the bomb just never gets developed? Nobody bothers.Richard Rhodes 0:06:34That's really a good question and I've wondered over the years. But the more I look at the sequence of events, the more I think it would have been essentially inevitable, though perhaps not such an accelerated program. The bomb was pushed so hard during the Second World War because we thought the Germans had already started working on one. Nuclear fission had been discovered in Nazi Germany, in Berlin, in 1938, nine months before the beginning of the Second World War in Europe. Technological surveillance was not available during the war. The only way you could find out something was to send in a spy or have a mole or something human. And we didn't have that. So we didn't know where the Germans were, but we knew that the basic physics reaction that could lead to a bomb had been discovered there a year or more before anybody else in the West got started thinking about it. There was that most of all to push the urgency. In your hypothetical there would not have been that urgency. However, as soon as good physicists thought about the reaction that leads to nuclear fission — where a slow room temperature neutron, very little energy, bumps into the nucleus of a uranium-235 atom it would lead to a massive response. Isidore Rabi, one of the great physicists of this era, said it would have been like the moon struck the earth. The reaction was, as physicists say, fiercely exothermic. It puts out a lot more energy than you have to use to get it started. Once they did the numbers on that, and once they figured out how much uranium you would need to have in one place to make a bomb or to make fission get going, and once they were sure that there would be a chain reaction, meaning a couple of neutrons would come out of the reaction from one atom, and those two or three would go on and bump into other Uranium atoms, which would then fission them, and you'd get a geometric exponential. You'd get 1, 2, 4, 8, 16, 32, and off of there. For most of our bombs today the initial fission, in 80 generations, leads to a city-busting explosion. And then they had to figure out how much material they would need, and that's something the Germans never really figured out, fortunately for the rest of us. They were still working on the idea that somehow a reactor would be what you would build. When Niels Bohr, the great Danish physicist, escaped from Denmark in 1943 and came to England and then United States, he brought with him a rough sketch that Werner Heisenberg, the leading scientist in the German program, had handed him in the course of trying to find out what Bohr knew about what America was doing. And he showed it to the guys at Los Alamos and Hans Bethe, one of the great Nobel laureate physicists in the group, said — “Are the Germans trying to throw a reactor down on us?” You can make a reactor blow up, we saw that at Chernobyl, but it's not a nuclear explosion on the scale that we're talking about with the bomb. So when a couple of these emigres Jewish physicists from Nazi Germany were whiling away their time in England after they escaped, because they were still technically enemy aliens and therefore could not be introduced to top secret discussions, one of them asked the other — “How much would we need of pure uranium-235, this rare isotope of uranium that chain reacts? How much would we need to make a bomb?” And they did the numbers and they came up with one pound, which was startling to them. Of course, it is more than that. It's about 125 pounds, but that's just a softball. That's not that much material. And then they did the numbers about what it would cost to build a factory to pull this one rare isotope of uranium out of the natural metal, which has several isotopes mixed together. And they figured it wouldn't cost more than it would cost to build a battleship, which is not that much money for a country at war. Certainly the British had plenty of battleships at that point in time. So they put all this together and they wrote a report which they handed through their superior physicists at Manchester University where they were based, who quickly realized how important this was. The United States lagged behind because we were not yet at war, but the British were. London was being bombed in the blitz. So they saw the urgency, first of all, of eating Germany to the punch, second of all of the possibility of building a bomb. In this report, these two scientists wrote that no physical structure came to their minds which could offer protection against a bomb of such ferocious explosive power. This report was from 1940 long before the Manhattan Project even got started. They said in this report, the only way we could think of to protect you against a bomb would be to have a bomb of similar destructive force that could be threatened for use if the other side attacked you. That's deterrence. That's a concept that was developed even before the war began in the United States. You put all those pieces together and you have a situation where you have to build a bomb because whoever builds the first bomb theoretically could prevent you from building more or prevent another country from building any and could dominate the world. And the notion of Adolf Hitler dominating the world, the Third Reich with nuclear weapons, was horrifying. Put all that together and the answer is every country that had the technological infrastructure to even remotely have the possibility of building everything you'd have to build to get the material for a bomb started work on thinking about it as soon as nuclear fusion was announced to the world. France, the Soviet Union, Great Britain, the United States, even Japan. So I think the bomb would have been developed but maybe not so quickly. Dwarkesh Patel 0:14:10In the book you talk that for some reason the Germans thought that the critical mass was something like 10 tons, they had done some miscalculation.Richard Rhodes 0:14:18A reactor. Dwarkesh Patel 0:14:19You also have some interesting stories in the book about how different countries found out the Americans were working on the bomb. For example, the Russians saw that all the top physicists, chemists, and metallurgists were no longer publishing. They had just gone offline and so they figured that something must be going on. I'm not sure if you're aware that while the subject of the Making of the Atomic Bomb in and of itself is incredibly fascinating, this book has become a cult classic in AI. Are you familiar with this? Richard Rhodes 0:14:52No. Dwarkesh Patel 0:14:53The people who are working on AI right now are huge fans of yours. They're the ones who initially recommended the book to me because the way they see the progress in the field reminded them of this book. Because you start off with these initial scientific hints. With deep learning, for example, here's something that can teach itself any function is similar to Szilárd noticing the nuclear chain reaction. In AI there's these scaling laws that say that if you make the model this much bigger, it gets much better at reasoning, at predicting text, and so on. And then you can extrapolate this curve. And you can see we get two more orders of magnitude, and we get to something that looks like human level intelligence. Anyway, a lot of the people who are working in AI have become huge fans of your book because of this reason. They see a lot of analogies in the next few years. They must be at page 400 in their minds of where the Manhattan Project was.Richard Rhodes 0:15:55We must later on talk about unintended consequences. I find the subject absolutely fascinating. I think my next book might be called Unintended Consequences. Dwarkesh Patel 0:16:10You mentioned that a big reason why many of the scientists wanted to work on the bomb, especially the Jewish emigres, was because they're worried about Hitler getting it first. As you mentioned at some point, 1943, 1944, it was becoming obvious that Hitler, the Nazis were not close to the bomb. And I believe that almost none of the scientists quit after they found out that the Nazis weren't close. So why didn’t more of them say — “Oh, I guess we were wrong. The Nazis aren't going to get it. We don't need to be working on it.”?Richard Rhodes 0:16:45There was only one who did that, Joseph Rotblat. In May of 1945 when he heard that Germany had been defeated, he packed up and left. General Groves, the imperious Army Corps of Engineers General who ran the entire Manhattan Project, was really upset. He was afraid he'd spill the beans. So he threatened to have him arrested and put in jail. But Rotblat was quite determined not to stay any longer. He was not interested in building bombs to aggrandize the national power of the United States of America, which is perfectly understandable. But why was no one else? Let me tell it in terms of Victor Weisskopf. He was an Austrian theoretical physicist, who, like the others, escaped when the Nazis took over Germany and then Austria and ended up at Los Alamos. Weisskopf wrote later — “There we were in Los Alamos in the midst of the darkest part of our science.” They were working on a weapon of mass destruction, that's pretty dark. He said “Before it had almost seemed like a spiritual quest.” And it's really interesting how different physics was considered before and after the Second World War. Before the war, one of the physicists in America named Louis Alvarez told me when he got his PhD in physics at Berkeley in 1937 and went to cocktail parties, people would ask, “What's your degree in?” He would tell them “Chemistry.” I said, “Louis, why?” He said, “because I don't really have to explain what physics was.” That's how little known this kind of science was at that time. There were only about 1,000 physicists in the whole world in 1900. By the mid-30s, there were a lot more, of course. There'd been a lot of nuclear physics and other kinds of physics done by them. But it was still arcane. And they didn't feel as if they were doing anything mean or dirty or warlike at all. They were just doing pure science. Then nuclear fission came along. It was publicized worldwide. People who've been born since after the Second World War don't realize that it was not a secret at first. The news was published first in a German chemistry journal, Die Naturwissenschaften, and then in the British journal Nature and then in American journals. And there were headlines in the New York Times, the Los Angeles Times, the Chicago Tribune, and all over the world. People had been reading about and thinking about how to get energy out of the atomic nucleus for a long time. It was clear there was a lot there. All you had to do was get a piece of radium and see that it glowed in the dark. This chunk of material just sat there, you didn't plug it into a wall. And if you held it in your hand, it would burn you. So where did that energy come from? The physicists realized it all came from the nucleus of the atom, which is a very small part of the whole thing. The nucleus is 1/100,000th the diameter of the whole atom. Someone in England described it as about the size of a fly in a cathedral. All of the energy that's involved in chemical reactions, comes from the electron cloud that's around the nucleus. But  it was clear that the nucleus was the center of powerful forces. But the question was, how do you get them out? The only way that the nucleus had been studied up to 1938 was by bombarding it with protons, which have the same electric charge as the nucleus, positive charge, which means they were repelled by it. So you had to accelerate them to high speeds with various versions of the big machines that we've all become aware of since then. The cyclotron most obviously built in the 30s, but there were others as well. And even then, at best, you could chip a little piece off. You could change an atom one step up or one step down the periodic table. This was the classic transmutation of medieval alchemy sure but it wasn't much, you didn't get much out. So everyone came to think of the nucleus of the atom like a little rock that you really had to hammer hard to get anything to happen with it because it was so small and dense. That's why nuclear fission, with this slow neutron drifting and then the whole thing just goes bang, was so startling to everybody. So startling that when it happened, most of the physicists who would later work on the bomb and others as well, realized that they had missed the reaction that was something they could have staged on a lab bench with the equipment on the shelf. Didn't have to invent anything new. And Louis Alvarez again, this physicist at Berkeley, he said — “I was getting my hair cut. When I read the newspaper, I pulled off the robe and half with my hair cut, ran to my lab, pulled some equipment off the shelf, set it up and there it was.” So he said, “I discovered nuclear fission, but it was two days too late.” And that happened all over. People were just hitting themselves on the head and saying, well, Niels Bohr said, “What fools we've all been.” So this is a good example of how in science, if your model you’re working with is wrong it doesn't lead you down the right path. There was only one physicist who really was thinking the right way about the uranium atom and that was Niels Bohr. He wondered, sometime during the 30s, why uranium was the last natural element in the periodic table? What is different about the others that would come later? He visualized the nucleus as a liquid drop. I always like to visualize it as a water-filled balloon. It's wobbly, it's not very stable. The protons in the nucleus are held together by something called the strong force, but they still have the repellent positive electric charge that's trying to push them apart when you get enough of them into a nucleus. It's almost a standoff between the strong force and all the electrical charge. So it is like a wobbly balloon of water. And then you see why a neutron just falling into the nucleus would make it wobble around even more and in one of its configurations, it might take a dumbbell shape. And then you'd have basically two charged atoms just barely connected, trying to push each other apart. And often enough, they went the whole way. When they did that, these two new elements, half the weight of uranium, way down the periodic table, would reconfigure themselves into two separate nuclei. And in doing so, they would release some energy. And that was the energy that came out of the reaction and there was a lot of energy. So Bohr thought about the model in the right way. The chemists who actually discovered nuclear fusion didn't know what they were gonna get. They were just bombarding a solution of uranium nitrate with neutrons thinking, well, maybe we can make a new element, maybe a first man-made element will come out of our work. So when they analyzed the solution after they bombarded it, they found elements halfway down the periodic table. They shouldn't have been there. And they were totally baffled. What is this doing here? Do we contaminate our solution? No. They had been working with a physicist named Lisa Meitner who was a theoretical physicist, an Austrian Jew. She had gotten out of Nazi Germany not long before. But they were still in correspondence with her. So they wrote her a letter. I held that letter in my hand when I visited Berlin and I was in tears. You don't hold history of that scale in your hands very often. And it said in German — “We found this strange reaction in our solution. What are these elements doing there that don't belong there?” And she went for a walk in a little village in Western Sweden with her nephew, Otto Frisch, who was also a nuclear physicist. And they thought about it for a while and they remembered Bohr's model, the wobbly water-filled balloon. And they suddenly saw what could happen. And that's where the news came from, the physics news as opposed to the chemistry news from the guys in Germany that was published in all the Western journals and all the newspapers. And everybody had been talking about, for years, what you could do if you had that kind of energy. A glass of this material would drive the Queen Mary back and forth from New York to London 20 times and so forth, your automobile could run for months. People were thinking about what would be possible if you had that much available energy. And of course, people had thought about reactors. Robert Oppenheimer was a professor at Berkeley and within a week of the news reaching Berkeley, one of his students told me that he had a drawing on the blackboard, a rather bad drawing of both a reactor and a bomb. So again, because the energy was so great, the physics was pretty obvious. Whether it would actually happen depended on some other things like could you make it chain react? But fundamentally, the idea was all there at the very beginning and everybody jumped on it. Dwarkesh Patel 0:27:54The book is actually the best history of World War II I've ever read. It's about the atomic bomb, but it's interspersed with the events that are happening in World War II, which motivate the creation of the bomb or the release of it, why it had to be dropped on Japan given the Japanese response. The first third is about the scientific roots of the physics and it's also the best book I've read about the history of science in the early 20th century and the organization of it. There's some really interesting stuff in there. For example, there was a passage where you talk about how there's a real master apprentice model in early science where if you wanted to learn to do this kind of experimentation, you will go to Amsterdam where the master of it is residing. It's much more individual focused. Richard Rhodes 0:28:58Yeah, the whole European model of graduate study, which is basically the wandering scholar. You could go wherever you wanted to and sign up with whoever was willing to have you sign up. (0:29:10) - Firebombing vs nuclear vs hydrogen bombsDwarkesh Patel 0:29:10But the question I wanted to ask regarding the history you made of World War II in general is — there's one way you can think about the atom bomb which is that it is completely different from any sort of weaponry that has been developed before it. Another way you can think of it is there's a spectrum where on one end you have the thermonuclear bomb, in the middle you have the atom bomb, and on this end you have the firebombing of cities like Hamburg and Dresden and Tokyo. Do you think of these as completely different categories or does it seem like an escalating gradient to you? Richard Rhodes 0:29:47I think until you get to the hydrogen bomb, it's really an escalating gradient. The hydrogen bomb can be made arbitrarily large. The biggest one ever tested was 56 megatons of TNT equivalent. The Soviet tested that. That had a fireball more than five miles in diameter, just the fireball. So that's really an order of magnitude change. But the other one's no and in fact, I think one of the real problems, this has not been much discussed and it should be, when American officials went to Hiroshima and Nagasaki after the war, one of them said later — “I got on a plane in Tokyo. We flew down the long green archipelago of the Japanese home island. When I left Tokyo, it was all gray broken roof tiles from the fire bombing and the other bombings. And then all this greenery. And then when we flew over Hiroshima, it was just gray broken roof tiles again.” So the scale of the bombing with one bomb, in the case of Hiroshima, was not that different from the scale of the fire bombings that had preceded it with tens of thousands of bombs. The difference was it was just one plane. In fact, the people in Hiroshima didn't even bother to go into their bomb shelters because one plane had always just been a weather plane. Coming over to check the weather before the bombers took off. So they didn't see any reason to hide or protect themselves, which was one of the reasons so many people were killed. The guys at Los Alamos had planned on the Japanese being in their bomb shelters. They did everything they could think of to make the bomb as much like ordinary bombing as they could. And for example, it was exploded high enough above ground, roughly 1,800 yards, so that the fireball that would form from this really very small nuclear weapon — by modern standards — 15 kilotons of TNT equivalent, wouldn't touch the ground and stir up dirt and irradiate it and cause massive radioactive fallout. It never did that. They weren't sure there would be any fallout. They thought the plutonium and the bomb over Nagasaki now would just kind of turn into a gas and blow away. That's not exactly what happened. But people don't seem to realize, and it's never been emphasized enough, these first bombs, like all nuclear weapons, were firebombs. Their job was to start mass fires, just exactly like all the six-pound incendiaries that had been destroying every major city in Japan by then. Every major city above 50,000 population had already been burned out. The only reason Hiroshima and Nagasaki were around to be atomic bombed is because they'd been set aside from the target list, because General Groves wanted to know what the damage effects would be. The bomb that was tested in the desert didn't tell you anything. It killed a lot of rabbits, knocked down a lot of cactus, melted some sand, but you couldn't see its effect on buildings and on people. So the bomb was deliberately intended to be as much not like poison gas, for example, because we didn't want the reputation for being like people in the war in Europe during the First World War, where people were killing each other with horrible gasses. We just wanted people to think this was another bombing. So in that sense, it was. Of course, there was radioactivity. And of course, some people were killed by it. But they calculated that the people who would be killed by the irradiation, the neutron radiation from the original fireball, would be close enough to the epicenter of the explosion that they would be killed by the blast or the flash of light, which was 10,000 degrees. The world's worst sunburn. You've seen stories of people walking around with their skin hanging off their arms. I've had sunburns almost that bad, but not over my whole body, obviously, where the skin actually peeled blisters and peels off. That was a sunburn from a 10,000 degree artificial sun. Dwarkesh Patel 0:34:29So that's not the heat, that's just the light? Richard Rhodes 0:34:32Radiant light, radiant heat. 10,000 degrees. But the blast itself only extended out a certain distance, it was fire. And all the nuclear weapons that have ever been designed are basically firebombs. That's important because the military in the United States after the war was not able to figure out how to calculate the effects of this weapon in a reliable way that matched their previous experience. They would only calculate the blast effects of a nuclear weapon when they figured their targets. That's why we had what came to be called overkill. We wanted redundancy, of course, but 60 nuclear weapons on Moscow was way beyond what would be necessary to destroy even that big a city because they were only calculating the blast. But in fact, if you exploded a 300 kiloton nuclear warhead over the Pentagon at 3,000 feet, it would blast all the way out to the capital, which isn't all that far. But if you counted the fire, it would start a mass-fire and then it would reach all the way out to the Beltway and burn everything between the epicenter of the weapon and the Beltway. All organic matter would be totally burned out, leaving nothing but mineral matter, basically. Dwarkesh Patel 0:36:08I want to emphasize two things you said because they really hit me in reading the book and I'm not sure if the audience has fully integrated them. The first is, in the book, the military planners and Groves, they talk about needing to use the bomb sooner rather than later, because they were running out of cities in Japan where there are enough buildings left that it would be worth bombing in the first place, which is insane. An entire country is almost already destroyed from fire bombing alone. And the second thing about the category difference between thermonuclear and atomic bombs. Daniel Ellsberg, the nuclear planner who wrote the Doomsday machine, he talks about, people don't understand that the atom bomb that resulted in the pictures we see of Nagasaki and Hiroshima, that is simply the detonator of a modern nuclear bomb, which is an insane thing to think about. So for example, 10 and 15 kilotons is the Hiroshima Nagasaki and the Tsar Bomba, which was 50 megatons. So more than 1,000 times as much. And that wasn't even as big as they could make it. They kept the uranium tamper off, because they didn't want to destroy all of Siberia. So you could get more than 10,000 times as powerful. Richard Rhodes 0:37:31When Edward Teller, co-inventor of the hydrogen bomb and one of the dark forces in the story, was consulting with our military, just for his own sake, he sat down and calculated, how big could you make a hydrogen bomb? He came up with 1,000 megatons. And then he looked at the effects. 1,000 megatons would be a fireball 10 miles in diameter. And the atmosphere is only 10 miles deep. He figured that it would just be a waste of energy, because it would all blow out into space. Some of it would go laterally, of course, but most of it would just go out into space. So a bomb more than 100 megatons would just be totally a waste of time. Of course, a 100 megatons bomb is also a total waste, because there's no target on Earth big enough to justify that from a military point of view. Robert Oppenheimer, when he had his security clearance questioned and then lifted when he was being punished for having resisted the development of the hydrogen bomb, was asked by the interrogator at this security hearing — “Well, Dr. Oppenheimer, if you'd had a hydrogen bomb for Hiroshima, wouldn't you have used it?” And Oppenheimer said, “No.” The interrogator asked, “Why is that?” He said because the target was too small. I hope that scene is in the film, I'm sure it will be. So after the war, when our bomb planners and some of our scientists went into Hiroshima and Nagasaki, just about as soon as the surrender was signed, what they were interested in was the scale of destruction, of course. And those two cities didn't look that different from the other cities that had been firebombed with small incendiaries and ordinary high explosives. They went home to Washington, the policy makers, with the thought that — “Oh, these bombs are not so destructive after all.” They had been touted as city busters, basically, and they weren't. They didn't completely burn out cities. They were not certainly more destructive than the firebombing campaign, when everything of more than 50,000 population had already been destroyed. That, in turn, influenced the judgment about what we needed to do vis-a-vis the Soviet Union when the Soviets got the bomb in 1949. There was a general sense that, when you could fight a war with nuclear weapons, deterrence or not, you would need quite a few of them to do it right. And the Air Force, once it realized that it could aggrandize its own share of the federal budget by cornering the market and delivering nuclear weapons, very quickly decided that they would only look at the blast effect and not the fire effect. It's like tying one hand behind your back. Most of it was a fire effect. So that's where they came up with numbers like we need 60 of these to take out Moscow. And what the Air Force figured out by the late 1940s is that the more targets, the more bombs. The more bombs, the more planes. The more planes, the biggest share of the budget. So by the mid 1950s, the Air Force commanded 47% of the federal defense budget. And the other branches of services, which had not gone nuclear by then, woke up and said, we'd better find some use for these weapons in our branches of service. So the Army discovered that it needed nuclear weapons, tactical weapons for field use, fired out of cannons. There was even one that was fired out of a shoulder mounted rifle. There was a satchel charge that two men could carry, weighed about 150 pounds, that could be used to dig a ditch so that Soviet tanks couldn't cross into Germany. And of course the Navy by then had been working hard with General Rickover on building a nuclear submarine that could carry ballistic missiles underwater in total security. No way anybody could trace those submarines once they were quiet enough. And a nuclear reactor is very quiet. It just sits there with neutrons running around, making heat. So the other services jumped in and this famous triad, we must have these three different kinds of nuclear weapons, baloney. We would be perfectly safe if we only had our nuclear submarines. And only one or two of those. One nuclear submarine can take out all of Europe or all of the Soviet Union.Dwarkesh Patel 0:42:50Because it has multiple nukes on it? Richard Rhodes 0:42:53Because they have 16 intercontinental ballistic missiles with MIRV warheads, at least three per missile. Dwarkesh Patel 0:43:02Wow. I had a former guest, Richard Hanania, who has a book about foreign policy where he points out that our model of thinking about why countries do the things they do, especially in foreign affairs, is wrong because we think of them as individual rational actors, when in fact it's these competing factions within the government. And in fact, you see this especially in the case of Japan in World War II, there was a great book of Japan leading up to World War II, where they talk about how a branch of the Japanese military, I forget which, needed more oil to continue their campaign in Manchuria so they forced these other branches to escalate. But it’s so interesting that the reason we have so many nukes is that the different branches are competing for funding. Richard Rhodes 0:43:50Douhet, the theorist of air power, had been in the trenches in the First World War. Somebody (John Masefield) called the trenches of the First World War, the long grave already dug, because millions of men were killed and the trenches never moved, a foot this way, a foot that way, all this horror. And Douhet came up with the idea that if you could fly over the battlefield to the homeland of the enemy and destroy his capacity to make war, then the people of that country, he theorized, would rise up in rebellion and throw out their leaders and sue for peace. And this became the dream of all the Air Forces of the world, but particularly ours. Until around 1943, it was called the US Army Air Force. The dream of every officer in the Air Force was to get out from under the Army, not just be something that delivers ground support or air support to the Army as it advances, but a power that could actually win wars. And the missing piece had always been the scale of the weaponry they carried. So when the bomb came along, you can see why Curtis LeMay, who ran the strategic air command during the prime years of that force, was pushing for bigger and bigger bombs. Because if a plane got shot down, but the one behind it had a hydrogen bomb, then it would be just almost as effective as the two planes together. So they wanted big bombs. And they went after Oppenheimer because he thought that was a terrible way to go, that there was really no military use for these huge weapons. Furthermore, the United States had more cities than Russia did, than the Soviet Union did. And we were making ourselves a better target by introducing a weapon that could destroy a whole state. I used to live in Connecticut and I saw a map that showed the air pollution that blew up from New York City to Boston. And I thought, well, now if that was fallout, we'd be dead up here in green, lovely Connecticut. That was the scale that it was going to be with these big new weapons. So on the one hand, you had some of the important leaders in the government thinking that these weapons were not the war-winning weapons that the Air Force wanted them and realized they could be. And on the other hand, you had the Air Force cornering the market on nuclear solutions to battles. All because some guy in a trench in World War I was sufficiently horrified and sufficiently theoretical about what was possible with air power. Remember, they were still flying biplanes. When H.G. Wells wrote his novel, The World Set Free in 1913, predicting an atomic war that would lead to world government, he had Air Forces delivering atomic bombs, but he forgot to update his planes. The guys in the back seat, the bombardiers, were sitting in a biplane, open cockpit. And when the pilots had dropped the bomb, they would reach down and pick up H.G. Wells' idea of an atomic bomb and throw it over the side. Which is kind of what was happening in Washington after the war. And it led us to a terribly misleading and unfortunate perspective on how many weapons we needed, which in turn fermented the arms race with the Soviets and just chased off. In the Soviet Union, they had a practical perspective on factories. Every factory was supposed to produce 120% of its target every year. That was considered good Soviet realism. And they did that with their nuclear war weapons. So by the height of the Cold War, they had 75,000 nuclear weapons, and nobody had heard yet of nuclear winter. So if both sides had set off this string of mass traps that we had in our arsenals, it would have been the end of the human world without question. Dwarkesh Patel 0:48:27It raises an interesting question, if the military planners thought that the conventional nuclear weapon was like the fire bombing, would it have been the case that if there wasn't a thermonuclear weapon, that there actually would have been a nuclear war by now because people wouldn't have been thinking of it as this hard red line? Richard Rhodes 0:48:47I don't think so because we're talking about one bomb versus 400, and one plane versus 400 planes and thousands of bombs. That scale was clear. Deterrence was the more important business. Everyone seemed to understand even the spies that the Soviets had connected up to were wholesaling information back to the Soviet Union. There's this comic moment when Truman is sitting with Joseph Stalin at Potsdam, and he tells Stalin, we have a powerful new weapon. And that's as much as he's ready to say about it. And Stalin licks at him and says, “Good, I hope you put it to good use with the Japanese.” Stalin knows exactly what he's talking about. He's seen the design of the fat man type Nagasaki plutonium bomb. He has held it in his hands because they had spies all over the place. (0:49:44) - Stalin & the Soviet programDwarkesh Patel 0:49:44How much longer would it have taken the Soviets to develop the bomb if they didn't have any spies? Richard Rhodes 0:49:49Probably not any longer. Dwarkesh Patel 0:49:51Really? Richard Rhodes 0:49:51When the Soviet Union collapsed in the winter of ‘92, I ran over there as quickly as I could get over there. In this limbo between forming a new kind of government and some of the countries pulling out and becoming independent and so forth, their nuclear scientists, the ones who'd worked on their bombs were free to talk. And I found that out through Yelena Bonner, Andrei Sakharov's widow, who was connected to people I knew. And she said, yeah, come on over. Her secretary, Sasha, who was a geologist about 35 years old became my guide around the country. We went to various apartments. They were retired guys from the bomb program and were living on, as far as I could tell, sac-and-potatoes and some salt. They had government pensions and the money was worth a salt, all of a sudden. I was buying photographs from them, partly because I needed the photographs and partly because 20 bucks was two months' income at that point. So it was easy for me and it helped them. They had first class physicists in the Soviet Union, they do in Russian today. They told me that by 1947, they had a design for a bomb that they said was half the weight and twice the yield of the Fat Man bomb. The Fat Man bomb was the plutonium implosion, right? And it weighed about 9,000 pounds. They had a much smaller and much more deliverable bomb with a yield of about 44 kilotons. Dwarkesh Patel 0:51:41Why was Soviet physics so good?Richard Rhodes 0:51:49The Russian mind? I don't know. They learned all their technology from the French in the 19th century, which is why there's so many French words in Russian. So they got good teachers, the French are superb technicians, they aren't so good at building things, but they're very good at designing things. There's something about Russia, I don't know if it's the language or the education. They do have good education, they did. But I remember asking them when they were working, I said — On the hydrogen bomb, you didn't have any computers yet. We only had really early primitive computers to do the complicated calculations of the hydrodynamics of that explosion. I said, “What did you do?” They said, “Oh, we just used nuclear. We just used theoretical physics.” Which is what we did at Los Alamos. We had guys come in who really knew their math and they would sit there and work it out by hand. And women with old Marchant calculators running numbers. So basically they were just good scientists and they had this new design. Kurchatov who ran the program took Lavrentiy Beria, who ran the NKVD who was put in charge of the program and said — “Look, we can build you a better bomb. You really wanna waste the time to make that much more uranium and plutonium?” And Beria said, “Comrade, I want the American bomb. Give me the American bomb or you and all your families will be camp dust.” I talked to one of the leading scientists in the group and he said, we valued our lives, we valued our families. So we gave them a copy of the plutonium implosion bomb. Dwarkesh Patel 0:53:37Now that you explain this, when the Soviet Union fell, why didn’t North Korea, Iran or another country, send a few people to the fallen Soviet Union to recruit a few of the scientists to start their own program? Or buy off their stockpiles or something. Or did they?Richard Rhodes 0:53:59There was some effort by countries in the Middle East to get all the enriched uranium, which they wouldn't sell them. These were responsible scientists. They told me — we worked on the bomb because you had it and we didn't want there to be a monopoly on the part of any country in the world. So patriotically, even though Stalin was in charge of our country, he was a monster. We felt that it was our responsibility to work on these things, even Sakharov. There was a great rush at the end of the Second World War to get hold of German scientists. And about an equal number were grabbed by the Soviets. All of the leading German scientists, like Heisenberg and Hans and others, went west as fast as they could. They didn't want to be captured by the Soviets. But there were some who were. And they helped them work. People have the idea that Los Alamos was where the bomb happened. And it's true that at Los Alamos, we had the team that designed, developed, and built the first actual weapons. But the truth is, the important material for weapons is the uranium or plutonium. One of the scientists in the Manhattan Project told me years later, you can make a pretty high-level nuclear explosion just by taking two subcritical pieces of uranium, putting one on the floor and dropping the other by hand from a height of about six feet. If that's true, then all this business about secret designs and so forth is hogwash. What you really need for a weapon is the critical mass of highly enriched uranium, 90% of uranium-235. If you've got that, there are lots of different ways to make the bomb. We had two totally different ways that we used. The gun on the one hand for uranium, and then because plutonium was so reactive that if you fired up the barrel of a cannon at 3,000 feet per second, it would still melt down before the two pieces made it up. So for that reason, they had to invent an entirely new technology, which was an amazing piece of work. From the Soviet point of view, and I think this is something people don't know either, but it puts the Russian experience into a better context. All the way back in the 30s, since the beginning of the Soviet Union after the First World War, they had been sending over espionage agents connected up to Americans who were willing to work for them to collect industrial technology. They didn't have it when they began their country. It was very much an agricultural country. And in that regard, people still talk about all those damn spies stealing our secrets, we did the same thing with the British back in colonial days. We didn't know how to make a canal that wouldn't drain out through the soil. The British had a certain kind of clay that they would line their canals with, and there were canals all over England, even in the 18th century, that were impervious to the flow of water. And we brought a British engineer at great expense to teach us how to make the lining for the canals that opened up the Middle West and then the West. So they were doing the same thing. And one of those spies was a guy named Harry Gold, who was working all the time for them. He gave them some of the basic technology of Kodak filmmaking, for example. Harry Gold was the connection between David Greenglass and one of the American spies at Los Alamos and the Soviet Union. So it was not different. The model was — never give us something that someone dreamed of that hasn't been tested and you know works. So it would actually be blueprints for factories, not just a patent. And therefore when Beria after the war said, give us the bomb, he meant give me the American bomb because we know that works. I don't trust you guys. Who knows what you'll do. You're probably too stupid anyway. He was that kind of man. So for all of those reasons, they built the second bomb they tested was twice the yield and half the way to the first bomb. In other words, it was their new design. And so it was ours because the technology was something that we knew during the war, but it was too theoretical still to use. You just had to put the core and have a little air gap between the core and the explosives so that the blast wave would have a chance to accelerate through an open gap. And Alvarez couldn’t tell me what it was but he said, you can get a lot more destructive force with a hammer if you hit something with it, rather than if you put the head on the hammer and push. And it took me several years before I figured out what he meant. I finally understood he was talking about what's called levitation.Dwarkesh Patel 0:59:41On the topic that the major difficulty in developing a bomb is either the refinement of uranium into U-235 or its transmutation into plutonium, I was actually talking to a physicist in preparation for this conversation. He explained the same thing that if you get two subcritical masses of uranium together, you wouldn't have the full bomb because it would start to tear itself apart without the tamper, but you would still have more than one megaton.Richard Rhodes 1:00:12It would be a few kilotons. Alvarez's model would be a few kilotons, but that's a lot. Dwarkesh Patel 1:00:20Yeah, sorry I meant kiloton. He claimed that one of the reasons why we talk so much about Los Alamos is that at the time the government didn't want other countries to know that if you refine uranium, you've got it. So they were like, oh, we did all this fancy physics work in Los Alamos that you're not gonna get to, so don't even worry about it. I don't know what you make of that theory. That basically it was sort of a way to convince people that Los Alamos was important. Richard Rhodes 1:00:49I think all the physics had been checked out by a lot of different countries by then. It was pretty clear to everybody what you needed to do to get to a bomb. That there was a fast fusion reaction, not a slow fusion reaction, like a reactor. They'd worked that out. So I don't think that's really the problem. But to this day, no one ever talks about the fact that the real problem isn't the design of the weapon. You could make one with wooden boxes if you wanted to. The problem is getting the material. And that's good because it's damned hard to make that stuff. And it's something you can protect. Dwarkesh Patel 1:01:30We also have gotten very lucky, if lucky is the word you want to use. I think you mentioned this in the book at some point, but the laws of physics could have been such that unrefined uranium ore was enough to build a nuclear weapon, right? In some sense, we got lucky that it takes a nation-state level actor to really refine and produce the raw substance. Richard Rhodes 1:01:56Yeah, I was thinking about that this morning on the way over. And all the uranium in the world would already have destroyed itself. Most people have never heard of the living reactors that developed on their own in a bed of uranium ore in Africa about two billion years ago, right? When there was more U-235 in a mass of uranium ore than there is today, because it decays like all radioactive elements. And the French discovered it when they were mining the ore and found this bed that had a totally different set of nuclear characteristics. They were like, what happened? But there were natural reactors in Gabon once upon a time. And they started up because some water, a moderator to make the neutrons slow down, washed its way down through a bed of much more highly enriched uranium ore than we still have today. Maybe 5-10% instead of 3.5 or 1.5, whatever it is now. And they ran for about 100,000 years and then shut themselves down because they had accumulated enough fusion products that the U-235 had been used up. Interestingly, this material never migrated out of the bed of ore. People today who are anti-nuclear say, well, what are we gonna do about the waste? Where are we gonna put all that waste? It's silly. Dwarkesh Patel 1:03:35Shove it in a hole. Richard Rhodes 1:03:36Yeah, basically. That's exactly what we're planning to do. Holes that are deep enough and in beds of material that will hold them long enough for everything to decay back to the original ore. It's not a big problem except politically because nobody wants it in their backyard.Dwarkesh Patel 1:03:53On the topic of the Soviets, one question I had while reading the book was — we negotiated with Stalin at Yalta and we surrendered a large part of Eastern Europe to him under his sphere of influence. And obviously we saw 50 years of immiseration there as a result. Given the fact that only we had the bomb, would it have been possible that we could have just knocked out the Soviet Union or at least prevented so much of the world from succumbing to communism in the aftermath of World War II? Is that a possibility? Richard Rhodes 1:04:30When we say we had the bomb, we had a few partly assembled handmade bombs. It took almost as long to assemble one as the battery life of the batteries that would drive the original charge that would set off the explosion. It was a big bluff. You know, when they closed Berlin in 1948 and we had to supply Berlin by air with coal and food for a whole winter, we moved some B-29s to England. The B-29 being the bomber that had carried the bombs. They were not outfitted for nuclear weapons. They didn't have the same kind of bomb-based structure. The weapons that were dropped in Japan had a single hook that held the entire bomb. So when the bay opened and the hook was released, the thing dropped. And that's very different from dropping whole rows of small bombs that you've seen in the photographs and the film footage. So it was a big bluff on our part. We took some time after the war inevitably to pull everything together. Here was a brand new technology. Here was a brand new weapon. Who was gonna be in charge of it? The military wanted control, Truman wasn't about to give the military control. He'd been an artillery officer in the First World War. He used to say — “No, damn artillery captain is gonna start World War III when I'm president.” I grew up in the same town he lived in so I know his accent. Independence, Missouri. Used to see him at his front steps taking pictures with tourists while he was still president. He used to step out on the porch and let the tourists take photographs. About a half a block from my Methodist church where I went to church. It was interesting. Interestingly, his wife was considered much more socially acceptable than he was. She was from an old family in independence, Missouri. And he was some farmer from way out in Grandview, Missouri, South of Kansas City. Values. Anyway, at the end of the war, there was a great rush from the Soviet side of what was already a zone. There was a Soviet zone, a French zone, British zone and an American zone. Germany was divided up into those zones to grab what's left of the uranium ore that the Germans had stockpiled. And there was evidence that there was a number of barrels of the stuff in a warehouse somewhere in the middle of all of this. And there's a very funny story about how the Russians ran in and grabbed off one site full of uranium ore, this yellow black stuff in what were basically wine barrels. And we at the same night, just before the wall came down between the zones, were running in from the other side, grabbing some other ore and then taking it back to our side. But there was also a good deal of requisitioning of German scientists. And the ones who had gotten away early came West, but there were others who didn't and ended up helping the Soviets. And they were told, look, you help us build the reactors and the uranium separation systems that we need. And we'll let you go home and back to your family, which they did. Early 50s by then, the German scientists who had helped the Russians went home. And I think our people stayed here and brought their families over, I don't know. (1:08:24) - Deterrence, disarmament, North Korea, TaiwanDwarkesh Patel 1:08:24Was there an opportunity after the end of World War II, before the Soviets developed the bomb, for the US to do something where either it somehow enforced a monopoly on having the bomb, or if that wasn't possible, make some sort of credible gesture that, we're eliminating this knowledge, you guys don't work on this, we're all just gonna step back from this. Richard Rhodes 1:08:50We tried both before the war. General Groves, who had the mistaken impression that there was a limited amount of high-grade uranium ore in the world, put together a company that tried to corner the market on all the available supply. For some reason, he didn't realize that a country the size of the Soviet Union is going to have some uranium ore somewhere. And of course it did, in Kazakhstan, rich uranium ore, enough for all the bombs they wanted to build. But he didn't know that, and I frankly don't know why he didn't know that, but I guess uranium's use before the Second World War was basically as a glazing agent for pottery, that famous yellow pottery and orange pottery that people owned in the 1930s, those colors came from uranium, and they're sufficiently radioactive, even to this day, that if you wave a Geiger counter over them, you get some clicks. In fact, there have been places where they've gone in with masks and suits on, grabbed the Mexican pottery and taken it out in a lead-lined case. People have been so worried about it but that was the only use for uranium, to make a particular kind of glass. So once it became clear that there was another use for uranium, a much more important one, Groves tried to corner the world market, and he thought he had. So that was one effort to limit what the Soviet Union could do. Another was to negotiate some kind of agreement between the parties. That was something that really never got off the ground, because the German Secretary of State was an old Southern politician and he didn't trust the Soviets. He went to the first meeting, in Geneva in ‘45 after the war was over, and strutted around and said, well, I got the bomb in my pocket, so let's sit down and talk here. And the Soviet basically said, screw you. We don't care. We're not worried about your bomb. Go home. So that didn't work. Then there was the effort to get the United Nations to start to develop some program of international control. And the program was proposed originally by a committee put together by our State Department that included Robert Oppenheimer, rightly so, because the other members of the committee were industrialists, engineers, government officials, people with various kinds of expertise around the very complicated problems of technology and the science and, of course, the politics, the diplomacy. In a couple of weeks, Oppenheimer taught them the basics of the nuclear physics involved and what he knew about bomb design, which was everything, actually, since he'd run Los Alamos. He was a scientist during the war. And they came up with a plan. People have scoffed ever since at what came to be called the Acheson-Lilienthal plan named after the State Department people. But it's the only plan I think anyone has ever devised that makes real sense as to how you could have international control without a world government. Every country would be open to inspection by any agency that was set up. And the inspections would not be at the convenience of the country. But whenever the inspectors felt they needed to inspect. So what Oppenheimer called an open world. And if you had that, and then if each country then developed its own nuclear industries, nuclear power, medical uses, whatever, then if one country tried clandestinely to begin to build bombs, you would know about it at the time of the next inspection. And then you could try diplomacy. If that didn't work, you could try conventional war. If that wasn't sufficient, then you could start building your bombs too. And at the end of this sequence, which would be long enough, assuming that there were no bombs existing in the world, and the ore was stored in a warehouse somewhere, six months maybe, maybe a year, it would be time for everyone to scale up to deterrence with weapons rather than deterrence without weapons, with only the knowledge. That to me is the answer to the whole thing. And it might have worked. But there were two big problems. One, no country is going to allow a monopoly on a nuclear weapon, at least no major power. So the Russians were not willing to sign on from the beginning. They just couldn't. How could they? We would not have. Two, Sherman assigned a kind of a loudmouth, a wise old Wall Street guy to present this program to the United Nations. And he sat down with Oppenheimer after he and his people had studied and said, where's your army? Somebody starts working on a bomb over there. You've got to go in and take that out, don't you? He said, what would happen if one country started building a bomb? Oppenheimer said, well, that would be an act of war. Meaning then the other countries could begin to escalate as they needed to to protect themselves against one power, trying to overwhelm the rest. Well, Bernard Baruch was the name of the man. He didn't get it. So when he presented his revised version of the Acheson–Lilienthal Plan, which was called the Baruch Plan to the United Nations, he included his army. And he insisted that the United States would not give up its nuclear monopoly until everyone else had signed on. So of course, who's going to sign on to that deal? Dwarkesh Patel 1:15:24I feel he has a point in the sense that — World War II took five years or more. If we find that the Soviets are starting to develop a bomb, it's not like within the six months or a year or whatever, it would take them to start refining the ore. And to the point we found out that they've been refining ore to when we start a war and engage in it, and doing all the diplomacy. By that point, they might already have the bomb. And so we're behind because we dismantled our weapons. We are only starting to develop our weapons once we've exhausted these other avenues. Richard Rhodes 1:16:00Not to develop. Presumably we would have developed. And everybody would have developed anyway. Another way to think of this is as delayed delivery times. Takes about 30 minutes to get an ICBM from Central Missouri to Moscow. That's the time window for doing anything other than starting a nuclear war. So take the warhead off those missiles and move it down the road 10 miles. So then it takes three hours. You've got to put the warhead back on the missiles. If the other side is willing to do this too. And you both can watch and see. We require openness. A word Bohr introduced to this whole thing. In order to make this happen, you can't have secrets. And of course, as time passed on, we developed elaborate surveillance from space, surveillance from planes, and so forth. It would not have worked in 1946 for sure. The surveillance wasn’t there. But that system is in place today. The International Atomic Energy Agency has detected systems in air, in space, underwater. They can detect 50 pounds of dynamite exploded in England from Australia with the systems that we have in place. It's technical rather than human resources. But it's there. So it's theoretically possible today to get started on such a program. Except, of course, now, in like 1950, the world is awash in nuclear weapons. Despite the reductions that have occurred since the end of the Cold War, there's still 30,000-40,000 nuclear weapons in the world. Way too many. Dwarkesh Patel 1:18:01Yeah. That's really interesting. What percentage of warheads do you think are accounted for by this organization? If there's 30,000 warheads, what percentage are accounted for? Richard Rhodes 1:18:12All.Dwarkesh Patel 1:18:12Oh. Really?  North Korea doesn't have secrets? Richard Rhodes 1:18:13They're allowed to inspect anywhere without having to ask the government for permission. Dwarkesh Patel 1:18:18But presumably not North Korea or something, right? Richard Rhodes 1:18:21North Korea is an exception. But we keep pretty good track of North Korea needless to say. Dwarkesh Patel 1:18:27Are you surprised with how successful non-proliferation has been? The number of countries with nuclear weapons has not gone up for decades. Given the fact, as you were talking about earlier, it's simply a matter of refining or transmuting uranium. Is it surprising that there aren’t more countries that have it?Richard Rhodes 1:18:42That's really an interesting part. Again, a part of the story that most people have never really heard. In the 50s, before the development and signing of the Nuclear Non-Proliferation Treaty, which was 1968 and it took effect in 1970, a lot of countries that you would never have imagined were working on nuclear weapons. Sweden, Norway, Japan, South Korea. They had the technology. They just didn't have the materials. It was kind of dicey about what you should do. But I interviewed some of the Swedish scientists who worked on their bomb and they said, well, we were just talking about making some tactical nukes that would slow down a Russian tank advance on our country long enough for us to mount a defense. I said, so why did you give it up? And they said, well, when the Soviets developed hydrogen bombs, it would only take two to destroy Sweden. So we didn't see much point. And we then signed the Non-Proliferation Treaty. And our knowledge of how to build nuclear weapons helped us deal with countries like South Africa, which did build a few bombs in the late 1980s. Six World War II-type gun bombs fueled with enriched uranium, because South Africa is awash in uranium ore and makes a lot of uranium for various purposes. So efforts were starting up. And that's where John Kennedy got the numbers in a famous speech he delivered, where he said, I lose sleep at night over the real prospect of there being 10 countries with nuclear weapons by 1970 and 30 by 1980. And of course, that would have been a nightmare world, because the risk of somebody using them would have gone up accordingly. But after the Cuban Missile Crisis, we and the Soviet Union basically said, we've got to slow this thing down for us and for others as well. And the treaty that was then put together and negotiated offered a good deal. It said, if you don't build nuclear weapons, we will give you the knowledge to build nuclear energy technology that will allow you to forge ahead very successfully with that. There was a belief in the early years of nuclear weapons that as soon as the technology was learned by a country, they would immediately proceed to build the bomb. And no one really thought it through. It seemed sort of self-evident. But it wasn't self-evident. There are dangers to building a nuclear weapon and having an arsenal. If you're a little country and you have a nuclear arsenal, you have the potential to destroy a large country, or at least disable a large country, because you have these terribly destructive weapons. That makes you a target. That means that a large country is going to look at you and worry about you, which they never would have before. That kind of logic dawned on everybody at that point. And they were getting a good deal. And the other part of the deal was the part that the nuclear powers never kept to this day, which was an agreement that we would work seriously and vigorously toward nuclear disarmament. We didn't do that. We just told them we would. And then kind of snuck around on the sides. So much so that by this treaty, because no one was quite trusting of the whole deal, treaties are usually signed and they exist in perpetuity. They don't have any end date. They go on until somebody breaks the rules. But this treaty was given a 25-year review period, which would have been 1995, at which point if the countries had chosen to abrogate the treaty, it would have been set aside. And everybody could have gone back to making nuclear weapons. It almost came to that for the very reason that the main nuclear powers had not fulfilled their agreement to start reducing arsenals. We didn't start reducing our nuclear arsenal until the end of the Cold War, until the collapse of the Soviet Union. That's when we began cutting back, as did the former Soviet Union. A diplomat who's a friend of mine, Tom Graham, was assigned the task by our State Department of going around to the countries that were going to be voting on this renewal or not of the treaty and convincing their leaderships around the world. It wasn't in their best interest to abrogate the treaty at that point. Tom spent two years on the road. The only place he thought he should go is not the UN, where there's a second-level diplomat he could have talked to, but back to the home countries. And he convinced enough countries around the world. He's another hero who's never been properly celebrated. He convinced enough countries around the world that they did agree to extend the treaty in perpetuity. With the proviso that the goddamn nine nuclear powers get busy eliminating their nukes. And of course, George H.W. Bush, bless him, I didn't like his politics otherwise, but he stepped forward and split the nuclear arsenal in half right away. We dropped our numbers way, way lower than we had been. He pulled the amount of South Korea, which was a great bugaboo for both the Soviets and the North Koreans and China, and did a lot of good work toward moving toward a real reduction in nuclear arsenal. And the Russians agreed at that time. It was before Putin took power. So there was a change for the better, but there are still too many around, unfortunately. So that's why there are only nine nuclear powers to this day. Dwarkesh Patel 1:25:16How worried are you about a proxy war between great powers turning nuclear? For example, people have been worried about the Ukraine conflict for this reason. In the future, if we're facing an invasion of Taiwan by China, that's another thing to worry about. I had a friend who understands these things really well, and we were arguing because I thought, listen, if there's like a war, if there's a risk of nuclear war, let them take Taiwan. We'll build semiconductor factories in Arkansas. Who cares, right? And he explains, no, you don't understand, because if we don't protect Taiwan, then Japan and South Korea decide to go nuclear because they're like America won't protect us. And if they go nuclear, then the risk of nuclear war actually goes up, not down. Richard Rhodes 1:26:02Or they just decide to align with China. Yeah, because we didn't protect them with our nuclear umbrella the way we promised. Dwarkesh Patel 1:26:10Oh, I guess we haven't promised Taiwan that, but it's implied, I guess. Richard Rhodes 1:26:14I think it's implied. Yeah. If we said we're going to help defend them, that's what that means. Dwarkesh Patel 1:26:19Yeah. But anyway, the question was, how worried are you about proxy wars turning nuclear? Richard Rhodes 1:26:26There's been a lot of argument about whether nuclear deterrence actually works or not. The best evidence is that the United States fought a number of wars on the periphery, beginning with Korea and then Vietnam, and some other smaller things in between, where we were willing to accept defeat. We accepted defeat in Vietnam for sure, rather than use our nuclear arsenal, always because behind those peripheral countries, was a major nuclear power. China, Soviet Union, whatever. And we didn't want to risk a nuclear war. So at that level, deterrence really seemed to work for a long time. But there's been a change lately. And I find it kind of terrifying. The first manifestation was with India and Pakistan. They both went nuclear full scale in the late 1990s. India had tested one bomb in 1974, which they claimed was a peaceful explosion, whatever that is. But they hadn't proceeded anywhere from there. And Pakistan had tested their first bomb in China when they got it from AQ Khan, the same guy who was trying to proliferate to Iran a little later in Iraq. But they didn't build a lot of warheads either. And then their conflict or the personalities involved in the governments got sideways with each other. And both sides tested a quick flurry of four or five bombs each around 1997 or 1998. Now they were full fledged nuclear powers on the scale of two regional countries. But then in 1999, there was a border conflict between the two countries. And Pakistan came up with a whole new argument about nuclear deterrence. Not only could you prevent a nuclear escalation, but if you kept your deterrent in place, you could have a conventional war with the other side not willing to escalate to nuclear because you still had your nuclear arsenal. And that, which came very close to a nuclear exchange, we jumped in with both feet, believe me, and we're all over both countries about, don't do this, don't go this far, and they backed off. But Putin or someone in the Russian complex picked up on that new approach. And it's the one Putin is using now. He's basically saying, I'm having a conventional war here. And don't you dare introduce nuclear weapons or I will. In fact, if you defeat me, I may introduce nuclear weapons. Screw you. I'm going to use my nukes as a backup. That's new. And it's terrifying because it's a totally different kind of deterrence that risks going too far, too fast to be able to know what's going on. Dwarkesh Patel 1:29:47And in some sense, he is calling our bluff or at least I hope he is. Obviously you shouldn’t say this but hopefully the government would not respond to a tactical nuke used in Ukraine by getting the US annihilated.Richard Rhodes 1:30:04I don't think we would respond to it with a full scale nuclear attack. But I do think we would respond with some level of nuclear exchange with Russia or maybe just with that part of Russia, I don't know. We've long had a policy called decapitation. We long ago could the individual apartments and [unclear] of this Russian leadership with individual warheads. In the window at high noon, because they are very accurate now and they're totally stealthy. If you're thinking about cruise missiles, we can put one in someone's window and it's a nuclear warhead and not just a high explosive. They've known that for a long time. That doesn't give anybody time to get into the bomb shelter. This has gotten really very hairy. Used to be pretty straightforward. Don't bomb us, or we will bomb you. Attack in some peripheral country and we won't bomb you because you might bomb us. We'll lose that little war, but we'll come in somewhere else. All of these things, that's complicated enough. But now we're talking about this other level. Dwarkesh Patel 1:31:23So in some sense, this idea that we can backstop conventional power with nuclear weapons worked. After World War Two, the Soviets had millions of men in the Red Army stationed in Eastern Europe and we didn't have troops remaining in Western Europe but we said, listen, if you invade Western Europe, we'll respond with a nuclear attack. I guess that worked.Richard Rhodes 1:31:51It worked until August 1949, when the Soviets tested their first atomic bomb. And that's when panic hit Washington. And the whole debate went on about, do we build hydrogen bombs? And ultimately, the military prevailed and the chairman signed on. And plus, Fuchs was outed and there was this whole knowledge that the Russians knew a lot because Fuchs knew our early work on hydrogen weapons before the end of the war. We were exploring the possibility. All of that came together too with a kind of a terrible moment when the Teller side prevailed. And we said, let's build bigger bombs, as if that would somehow help. But there had been a balance of forces, you're quite right. They had two million men on the ground in Europe. We had the bomb. And then the balance of force was disrupted when they got the bomb. So then how do we rebalance? And the rebalance was the hydrogen bomb. And that's how you get to Armageddon ultimately, unfortunately. Dwarkesh Patel 1:32:57I was reading the acknowledgements and you talked to Teller in writing the book. Is that right? Richard Rhodes 1:33:02I did. Dwarkesh Patel 1:33:03And obviously, he was a big inspiration for the US pursuing the hydrogen bomb. What were his feelings when you talked to him about this?(1:33:12) - Oppenheimer as lab directorRichard Rhodes 1:33:12I made the mistake of going to see Teller at his house on the grounds of Stanford University early on in my research when I really didn't have as clear a grasp of who everyone was and what I should ask and so forth. I sent him a copy of an essay I'd written about Robert Oppenheimer for a magazine and that set him off. He had reached the point where he was telling TV interviewers, asking them how much time he would actually be on the air, and when they said three minutes or whatever, he would say, all right, then I will answer three questions no more. Trying to control because he was convinced that everyone was cutting the story to make him look bad. He really had quite a lot of paranoia at that point in his life. So when he read my essay on Oppenheimer, he used that as the basis for basically shouting at me, waving my big, heavy book, one of my big, heavy books at him. I remember thinking, oh, my God, he's going to hit me with my book. Then I thought, wait, this guy's 80 years old. I can take him. But he finally said, all right, I will answer three questions. And we sat down and I asked him one and he didn't give me an interesting answer. I asked him the second question and it was worth the whole interview. I said, was Robert Oppenheimer a good lab director? And I thought, well, here's the chance where he'll slice him. But Oppenheimer's worst enemy said to me, “Robert Oppenheimer was the best lab director I ever knew.” And I thought, bingo. And then he chased me out of the house. And I went up the road to a friend's house and got very drunk. Because I was really shaken. It was so new to me, all of this. But that quote was worth the whole thing because Eisenhower in one of his memoirs says, I always liked Hannibal best of all the classical figures in the military of the Roman Empire, because he comes down to us only in the written memoirs of his enemies. And if they thought he was such a good leader, he must have been a hell of a leader. Dwarkesh Patel 1:35:35The way the Manhattan Project is organized, it's interesting because if you think of a startup in Silicon Valley, you usually have a technical founder and a non-technical founder. The non-technical founder is in charge of talking to investors and customers and so on. And the technical founders in charge of organizing the technology and the engineers. And in Oppenheimer, you had the guy who understood all the chemistry, the metallurgy, the obviously the nuclear physics, and then Groves is getting appropriations and it's an interesting organization that you see. But why was Oppenheimer such a great lab director? Richard Rhodes 1:36:13Oppenheimer was a man with a very divided self and an insecure self. One of his closest friends was I.I. Robbie, a really profound and interesting human being. I'm going to be writing about him in my next book. I spent some time with Robbie just before he died and he said once of Oppenheimer, “I always felt as if he could never decide whether he wanted to be president of the [unclear] of Columbus or [unclear].” He said he was a certain kind of American Jew. The German Jews who came over before and after the First World War were westernized, they were not from the shtetls of the paler settlement of the Eastern European Jews. They were sophisticated, they were well educated. They were a totally different group and as such, they were able to assimilate fairly easily. Oppenheimer wasn't sent to a Jewish school. He was sent to the famous school that was opened in New York, it was called the ethical culture school. It was based on the idea of an ethical, moral education, but not a religious education. So his parents found this niche for him. He never quite pulled himself together as a human being. And as is true with many people with that kind of personality structure, he was a superb actor. He could play lots of different roles and he did. Women adored him. He was one of the most lavish courtiers of women. He would bring a bouquet of flowers to their first date, which was apparently shocking to people in those days. He was just a lovely, courtly man. But in the classroom, if somebody made a stupid mistake, he would just chew them out. And he taught a course in physics that was so advanced that most of the best students took it twice because they couldn't store it all the first time. He was nasty to people all the time in a way that bothered a lot of people. Louis Alvarez, who was someone I got to know pretty well because I helped him write his memoirs, he was one of the important scientists at Los Alamos who didn't get along with Oppenheimer at all because Oppenheimer was so condescending to everyone. Louis was kind of a hothead and he didn't like people being condescending to him. Oppenheimer never won a Nobel, Louis did. There was this layer of Oppenheimer being waspish all the time, which was his insecurity and his insecurity extended to physics. Robbie said later he couldn't sit down and focus on one problem because he always wanted to be someone who always knew everything that was going on in physics. You could call that someone who was a very sophisticated, knowledgeable scientist or you could call him someone who was superficial and he was superficial. He knew broadly rather than deeply. He and a graduate student of his developed the basic idea of the black hole long before it came up after the war. They published a paper on what was essentially the physics of the black hole in 1929. But it wasn’t called the black hole yet. John Wheeler invented that term many years later but the idea that a big enough sun, if it collapsed, would just keep on collapsing until nothing could come out of it including light, came from Oppenheimer. And if the black hole had been discovered out in space before he died, he certainly would have had a Nobel for the theory. That being said, he still was someone who was broad rather than deep. And he was someone who was really good at playing roles. General Groves, who himself had two or three degrees from MIT in engineering, he was no slouch. But his was more the industrial side of everything. How do you build a factory when you don't know what you're going to put in it? He built the factories at Oak Ridge, Tennessee to enrich uranium and start building the early piles that would later lead to the big reactors in Washington before they even knew what was going to go in the factory. He got orders of magnitude numbers and he said, start laying the concrete, we want it this big, we want this attached to it. We’re going to need power, going to need water, going to need gas, whatever they needed. He was that kind of really great engineer but he needed someone to help explain the physics to it. And he saw pretty quickly at the meetings he was holding at the University of Chicago, where they were building the first reactor, little one, that Oppenheimer was really good at explaining things. So he grabbed him and Groves spent the war riding in trains back and forth among all these various sites. He'd have the advisors from one site like Chicago jump on the train while he was taking the train from Chicago to Santa Fe. Then they'd get off and take the next train back to Chicago. Then he'd pick up the guys who were going to go with him to Tennessee from Santa Fe and they'd ride with him round and round and round. He got a plane later in the war. But most of the war, he just had people riding with him. And Oppenheimer later said, well, I became his idiot savant. And Oppenheimer did. He explained all the physics to Groves because Groves was a little insecure around six Nobel Laureates around the table and would say things like, well, you each have a PhD, but I think my work at MIT probably was the equivalent of about six PhDs, wouldn't you think? And they would think, who is this guy? Dwarkesh Patel 1:42:31Was he joking? Was that sarcasm or? Richard Rhodes 1:42:33No, he wasn’t. He had multiple degrees. You know how the military works when there's no war. They send their guys to school to get them better trained. So when the time came to find someone to run, Oppenheimer had been pushing for a separate place where the scientists could get together and do what scientists must do if they're going to advance their science. And that is, talk to each other. The system that Groves had installed around the country at all these factories and so forth, was called compartmentalization for secrecy. And basically it was — you're only allowed to know just enough to do your job and not the overall picture of what your job might be for. So, for example, there were women at Oak Ridge running big magnetic machines that would separate uranium 235 from uranium 238 with magnetic coils of various kinds, taking advantage of the very slight difference in mass between these two otherwise identical materials. The women who were doing this work were set in front of a board with a big dial on it, a big arrow that went from zero to ten or whatever and told keep the arrow about between right here on this. They didn't know what they were making. They really got good at spinning their dials and maintaining what was basically the level of whatever electric process was going on in this machine. So compartmentalization worked. But Oppenheimer said, if we are compartmentalized as scientists, we're not going to get anywhere. Science works by gift exchange. I discover something, I publish the results. All the other scientists in my field can read it or be told of it at a meeting. And then they can take that information and use that to move a little farther. And that's the way it's always been done. And that's the only way it works. As soon as you lock people up and tell them they can't talk to each other, it stops because the discovery over here doesn't get applied to a need over here. Simple. Groves reluctantly agreed to let the place have openness, as it was called. You see the parallel with the open world about the bomb. Same sort of thing. How can you know what's going on if you can't let people talk to each other? See what they're doing. But he insisted that the whole crew be put behind barbed wire in a faraway place when no one else was around. So they did. Groves had worked with Oppenheimer. Oppenheimer was now playing the lab director. And he was superb at it, as Teller’s remark about a good lab director would let you know. For the period of the war, Hans Bethe told me this, “Before the war, Robert really could be cruel. He would pounce on a mistake you made.” And Bethe, a Nobel Laureate, discovered what makes the sun work. That's how important Bethe's work was and how significant. Bethe told me, “everyone makes mistakes. I made mistakes. Oppenheimer would charge me with a mistake if I spoke wrong. But before the war, after the war, but not during the war.” During the war, he was a superb wise lab director. Because unlike most scientists, he was not only a physicist of high class, but he really was psychologically astute as a human being, as I think insecure people often are because got to scope out what's going on. Oppenheimer wrote pretty good poetry. He was interested in art. He wanted to read the Bhagavad Gita Gita in the original Sanskrit so he learned Sanskrit. He was very smart, needless to say, and had a very high IQ. Not all the physicists who did first class work did have high IQs. They took some other qualities as well. It took the [unclear] sitting down in a chair and focusing on one thing until you got through to it. That's what Robbie said. And he said, that's why Oppenheimer never won a Nobel Prize. All in all, Oppenheimer became in this place that some of these people later were calling [unclear]. Remember they were working on a bomb that was going to kill hundreds of thousands of people but it was the most curious collection of people who had felt like theirs was a spiritual field before the war. And here they were in the war and they began to think, well, maybe this isn't so spiritual, maybe we're doing something truly horrendous and when Bohr comes along and says, wait a minute. Oppenheimer by then was kind of a student of Bohr's. Oppenheimer had the job of recruiting everyone for Los Alamos without telling them what they would be doing because it was secret. So he would go to a university campus where there was a young physicist he wanted to recruit and they would go out for a walk to get away from any hidden microphones. And Oppenheimer would say, I can't tell you what we're going to be doing. I can tell you that it will certainly end this war and it may end all war. And that was quite enough. I mean, most of them figured out what they'd be doing anyway, because it was sort of obvious when you start looking at the list of people who are going there. They're all nuclear physicists. So Oppenheimer and Bohr together brought this idea to Los Alamos and later to the world that there was a paradox. The bomb had two sides and they were in a sense complementary because although it's certainly true that this was going to be a very destructive weapon, it would also maybe be true if all worked out and they tried to make it work out, that it would put an end to large scale war. If you go to the numbers and graph the number of man-made deaths from war starting in the 18th century, it's almost exponential up to 1943, when 15 million people died between the war itself and the Holocaust. And then it starts to decline as the war begins to come to an end. The war is really 1945. It drops down to about one to two million deaths and it stays there ever after. And although one to two million deaths a year from war is nothing to be proud of, we lose six to seven million people in the world every year from smoking. So in a curious way, the introduction of how to control nuclear energy changed the nature of nation states such that they could no longer at the limit defend their borders with war. They had to find some other way, at least when the scale goes up, they had to find another way. I think that's very important because people somehow don't really understand what a millennial change, the introduction of the release of nuclear energy into the world, really was. As we've been talking, I've been thinking over and over again about your question about AI and the whole larger interesting question that you can see how it fits into the bomb story of unintended consequences. All the countries that worked on the bomb at some level were thinking, oh my god, we're going to have a weapon that will surpass them all. One ring to bind them all like Lord of the Rings. They thought it would aggrandize national power, but what it did was put a limit to national power. For the first time in the history of the world, war became something that was historical, rather than universal. It was something that would no longer be possible. And who did that? Scientists going about their quiet work of talking to each other and exploring the way the universe works. Bohr, who's one of my favorite people in the world, he liked to say, science is not about power over nature, as people seem to think. Science is about the gradual removal of prejudice. By that, he meant when Galileo and Copernicus changed the way everyone looked at the position of the earth in the universe, not the center of the universe anymore, but just a planet revolving around a third-rate star. It changed the way everyone thought about the whole world. When Darwin definitively identified our place in the natural world as a brainy ape, it's still taking time for a lot of people to swallow that one. But inch by inch, these prejudices about where we are in the world and how powerful we are and what our purpose is and so forth are being drained away by science. The science of nuclear fission and nuclear energy is draining away and has drained away the theory that we are sort of universally capable of destroying each other and getting away with it. But the dark side is, the unintended consequence is, it's only by having a Damocles sword over our heads, the potential for destruction of the human world, that it's possible to limit war in this way. That's the threat. And when people start saying, well, look, we can have a conventional war if we've got nuclear weapons, because you're not going to attack us. You don't dare. We'll use our nuclear weapons on you. Something's changed most recently in all of this. It's outside the range of all the books I've written. It's a whole new thing. I guess you have to work through all the combinations, just as evolution does, before you come up with the one that actually fits the reality of the world.(1:53:40) - AI progress vs Manhattan ProjectDwarkesh Patel 1:53:40There's at least 10 different things and I'm trying to think which branch of the tree I want to explore first. On the AI question, yeah, I'm trying not explicitly connect the dots too much. Every time I read something in history, I think oh, this is exactly like so and so. First of all, the broader outline of the super eclectic group of people who are engaged in this historical pursuit, they understand it's a historical pursuit, they see the inklings of it. My second to last guest was Ilya Sutskever, who is the chief scientist at OpenAI which is the big lab building this. He was basically the Szliard of AI, how Szilard discovered nuclear chain reaction, Sutskever was the first person to train a neural network called ImageNet. Anyway, from that moment on, he was one of these scientists who sees that you could build a nuclear bomb immediately as soon as the news hits the floor. He saw this coming 10 years ago, scale this up and you've got something that's human level intelligent. I was reading through the book and so many things like this came up. One was a good friend of mine who works at one of these companies. And they train these models on GPUs, which are computers that can perform all these matrix calculations in parallel. And I was thinking about these engineers who were in short supply during the making of the bomb, let's say they're working on the electromagnetic separation of the isotopes. And it's this really esoteric thing, you're a chemist or a metallurgist or something that you're really needed for this specific thing. And he's in super high demand right now, he's like the one guy who can make these very particular machines run as efficiently as possible. You start looking and there's so many analogies there. Richard Rhodes 1:55:41I don't think there's much question that AI is going to be at least as transformative as nuclear weapons and nuclear energy. And it's scaring the hell out of a lot of people and a lot of other people are putting their heads in the sand. And others are saying, let's live it around with laws, which certainly we should do. We've tried to do that with nuclear weapons and have had some success. But people have no idea what's coming, do they? Dwarkesh Patel 1:56:10Yeah. One thing I wanted to ask is — Some of these scientists didn't see this coming. I think Fermi said you could never refine uranium and get 235 but then some of these other scientists saw it coming. And I was actually impressed by a few of them who accurately predicted the year that we’ll have the atomic bomb. Another one was, Russia is five to 10 years behind. So I'm curious, what made some of these scientists really good at forecasting the development of this technology and its maturity and what made some either too pessimistic or too optimistic? Is there some pattern you noticed among the ones who could see the progression of the technology? Richard Rhodes 1:56:57That's a good question. Well, the experience that I've had in working with scientists, physicists, is that they really are not very interested in history or in the future. They're interested in the Now, where the work is, where the cutting edge is. They'd have to devote quite a bit of energy to projecting in the future. Of course there have been a few. One thinks of some of the guys who wrote science fiction, some of the guys who wrote essays and so forth about where we were going. And if you ask them, particularly later on in their careers, when their basic work is already done, and I remember talking to a Nobel Laureate in another line of science, but he said, I would never be a graduate student connecting up with the Nobel Prize winner because they've already done their best work. And he was one so he was talking about himself, too. It takes a certain mentality to do that. And maybe scientists aren't the best ones to do it. Alvarez told me, he said, you know, I was always a snoop and I would poke around Berkeley in the various drawers of the benches in the laboratory. He said, one day I was poking around and I found this little glass cylinder about the size of a Petri dish with some wires inside. He said, and I realized it was the first cyclotron. They just put it in a drawer. I asked, so where is it now? He said, it's in the Smithsonian, of course. Where else would it be? I talked to the guy who invented the first laser and actually held one in my hand. He was an engineer at one of the big telephone companies. He said, the first laser is supposedly in the Smithsonian but they don't have it. They got one in the lab and I took my first one home. You want to see it? I said, God, of course. We went to the credenza in his dining room and he opened the drawer and pulled out a little box. Inside it was basically a cylinder of aluminum about the size of a little film can, which he opened up and took out a man-made ruby cylinder that was half-silvered on each end, surrounded by a very high-intensity flash bulb. That was it. It was this beautiful, simple machine. He said they didn't get the right one. I said, why didn't you give them the right one? He said, they didn't ask me. He was angry all his life because he wasn't included in the Nobel prizes. It went to the theoretician to first theorize the laser, but he built the first laser and there it was.(1:59:50) - Living through WW2Dwarkesh Patel 1:59:50When you were interviewing the scientists, how many of them did you feel regretted their work? Richard Rhodes 2:00:04They'd been down the road so far, they really didn't think that way anymore. What they did think about was they regretted the way governments had handled their work. There were some who were hawks and patriots, Alvarez was one of those. But most of them had tried in the years since the war to move in the direction of reducing nuclear weapons, eliminating nuclear weapons. It was a problem for them. When the war was over, these were mostly graduate students or new PhDs who had been recruited for this program. The average age at Los Alamos was 27. Oppenheimer was an old guy, he was 39. These were young guys who had been pulled out of their lives, pulled out of their careers. They wanted to get back to the university and do the work they had started out to do. By and large, they did and Los Alamos, just emptied out. Teller was horrified, he wanted the bomb program to continue because he wanted to build his hydrogen bomb. It was going to be his bid for history just as Oppenheimer's bid for history was the fission bomb. Teller’s was going to be the hydrogen bomb. Over the years, after that work was done, he systematically and meanly tried to exclude one by one anyone else who'd helped him work on the hydrogen bomb. Originally, he said it was a Polish mathematician named Stanislaw Ulam, whom I interviewed also, who really came up with the basic idea for the hydrogen bomb. He took it to Teller and Teller came up with an improvement. Then together, they wrote a paper which was signed by both of them. But by the 1980s, Teller was saying, “Ulam did nothing. Nothing.” Which wasn't true. It was his piece of history because he was scattered too, and he, number one, did Nobel-level work. They didn't talk so much about their personal guilt. I was a child in the Second World War. I was eight years old in 1945. So many young men had been killed on all sides in the war. It was a kind of a strange, peaceful time for children. Cars couldn't get tires, they were rationed. Cars couldn't get gasoline, it was rationed. So the streets were empty. We played in the streets. Meat was rationed, so we lived on macaroni and cheese. You got four ounces of meat a week per person during the Second World War. That was kind of wonderful and peaceful, and kids running around in gangs and so forth. But in at least one house in almost every block in the city, there was a black velvet flag or drape hanging with a gold star on it. That meant that someone in that family, a father, a brother, a son, had been killed. And I was a smart little kid, I understood what all that meant. I was reading newspapers by the time I was six following the war. It was the strangest time of darkness and terror. We didn't know until 1943 if we were going to win the war. It was scary for a while up front, the war in Europe. We sort of set Japan aside. Our government did until the war in Europe was done before we finished that other war. But it took a while for the United States to get its industrial plant up and cranking out planes at the rate of one a day or more. Churchill famously said when Pearl Harbor occurred, “The Lord hath delivered them into my hands.” And he explained later what he meant was that America's going to join the war. And I know we can win now because America's just one vast factory, much more so than the British could have put together. So it was a peaceful time, but it was a very dark time even for a child. My mother died when I was an infant so I understood what the death of a family member was about.Dwarkesh Patel 2:04:29Do you remember when the bombs dropped? Richard Rhodes 2:04:34By 1945, we were so pissed off at the Japanese. We had destroyed their air force, we had destroyed their navy, we had destroyed their army. The population of Japan was down to about a thousand calories per person of the worst kind of stuff, buckwheat and weeds and whatever they could find. And yet they wouldn't surrender. And they still had a million men on the ground in Western Manchuria. They only had about a year's worth of bullets left. We knew that much, but that's a long time with a million men. With that in mind, and because they felt that the Soviet Union was still neutral in the Eastern Front, because it had basically fought the war in Europe. We didn't win the war in Europe, the Russians did. We didn't enter the war on the ground until June 1944, by which time they were already moving forward against the German forces that attacked them in 1941. But the Japanese just wouldn't surrender. You could read the documents about the bombs. General George Marshall, who was leading the war, was in charge of all the forces, had this idea that maybe if we could use these bombs on the beaches before we landed, to kill any Japanese defense that was there, maybe they would get the message and be shocked and surrender. But from the Japanese point of view, as it turned out later, it's a myth that the bombs won the war. They contributed to winning the war, but they didn't win the war. What won the war was when Stalin finally was convinced that there were such things as atomic bombs. He was half convinced these were American pieces of disinformation. We were feeding the espionage to the Soviet Union to make him spend a lot of money and waste a lot of time on something that didn't exist. When the news came back from Hiroshima and then from Nagasaki that these things existed, he called Igor Kurchatov in and said famously, “Comrade, give us the bomb. You have all the resources of the state at your disposal. Give us the bomb as soon as you can.” But up until then, he wasn't so sure. He had told Truman at Potsdam that they would invade Manchuria with fresh Soviet forces on the 15th of August. Truman was trying to get him to agree to invade at all. Then when word came from New Mexico that the bomb had worked, which it did right in the middle of the Potsdam conference, Truman then was trying to get Stalin to come in as late as possible because he figured the bombs would end the war before Stalin could take over enough of Japan to divide the country the way Europe was divided. He didn't want a Soviet zone, an American zone, and a British zone. He knew we could do better with the Japanese than the Soviets would do. But Stalin, having heard that the bombs really worked, moved up the date of the Soviet invasion of Manchuria to the 8th of August between Hiroshima and Nagasaki and invaded early. I found it very interesting that the conventional air forces on our side staged the largest firebombing raid of the war, on the 14th of August, after the Japanese were in the middle of their surrender negotiations. The air force wanted to get credit for winning the war and they wanted to hold back the Soviets who were advancing down from Sakhalin Island to the northern islands of Japan as well as inward from Manchuria. So our bombing was in northern Japan. It was a way of telling the Soviets, back off buddy, we're not going to let you in here. Then the Japanese military leadership, which had been adamant that they would fight to the last Japanese, the 100 million, they called it, turned and finally realized that it was futile. With the fresh Soviet army coming into Manchuria, with the United States and the British coming in from the west to the south, they were surrounded and there was no reason to continue. But the bombs had their effect. The Japanese emperor used the bombings of Hiroshima and Nagasaki as a reason for entering politics for the first time in Japanese history. He had always been a spiritual figure, but he wasn't allowed to vote or veto the political arrangements. He stepped forth and said, we must do it for peace. And in his final imperial rescript on the 15th of August, recorded and played out to the people by radio, he said a new and most terrible weapon of war has led us to think the unthinkable and we must now lay down our arms. So the bomb had its effect, but it wasn't what we thought at the time. A lot of Americans said, thank God for the atomic bomb because our boys came home. The actor Paul Newman was a friend of mine and Paul was an 18-year-old bombardier on a two-man Navy fighter bomber training for the invasion of Japan. He said to me once, “Rhodes, I'm one of those guys who said, thank God for the atomic bomb because I probably wouldn't have come home if we'd invaded the Japanese home islands.” And a million men said that. And the truth is, there were so many Japanese who would have been killed if we had invaded that even the killings of the two bombs would have been dwarfed by the killing that happened. For the first time in the history of war, more Japanese civilians were killed in World War II than had ever been killed in a war before. War was beginning to become what it has since become, which is a war against civilians. Dwarkesh Patel 2:11:06We were talking near the beginning about whether it was possible that the bomb could not have been built at all and in the case of the nuclear physics involved here, it seems like it was sufficiently obvious. But one question I have, seeing the history of that science and whether it was plausible or not for some conspiracy to just hold it off, how plausible do you think it is that there's some group of scientists somewhere else who have discovered some other destructive phenomenon or technology that decides that we can't tell anyone about this. One area I think this might be plausible is bioweapons where they discover something and they just shut up about it. Given the study of this history, do you think that's more or less plausible?Richard Rhodes 2:11:51I don't think it's very likely to take bioweapons as an example. I remember talking to a biologist, one of the early DNA researchers who had been a physicist until DNA came along, and I asked “How'd you switch over to biology?” He said, “Well, it's molecules.” So from his perspective, it wasn't very different. But we were talking about the question of just the one you asked, but about biological agents. He said, nature has had millions of years to work all the combinations on the worst things you can imagine. The odds of anybody in a laboratory coming up with anything worse are really vanishingly small. I took that with great comfort. I went home and slept that night. And I think he's probably right. Evolution has done such a job. We're still digging stuff out. I mean, it's just amazing how much of our technology started out as something in the natural world that we adapted it and simplified it and engineered it so we could make it too. That's still going on in all sorts of ways. Dwarkesh Patel 2:13:04I hope that's the case. I was talking to a biologist and he was explaining to me, if you've seen things like AlphaFold, which is this AI program that models that can predict how a protein will fold, you can run billions of different permutations of a protein. And you can find smallpox, but it binds 100 times better with human receptors or something. Richard Rhodes 2:13:35I'll tell you a story, which I don't think is well known, I wish it were. Back in the 60s the Russians proposed a world-scale program of public health to eradicate smallpox to the UN. And they said, we'll contribute a vaccine and other countries can contribute whatever they can contribute. This program got going. It was run out of Geneva by the World Health Organization, by a wonderful American public health doctor, D.A. Henderson, a big old burly country boy looking guy whom I followed around for several months once in Geneva and elsewhere. And by the late 1970s, the last case of smallpox, which happened to be a disease that's only spread among humans and therefore was of all the diseases the most obvious one to try to get rid of. Because if there are reservoirs in the natural world outside of the human world, then there will be a problem. If it's also something that's carried around by rabbits or deer or whatever then it's harder to deal with. But if it's just humans, then all you have to do is to identify people who start showing signs of smallpox. In fact, you need everyone around them to make sure they don't go anywhere for a while and the disease can't spread. And that's the method that they use to eliminate smallpox everywhere in the world. And then in the 90s, when the Soviet Union collapsed, D.A. learned, as we all did, although it wasn't terribly well publicized, that there was a secret lab still operating that the Russian plan had been to eliminate smallpox vaccination from the world so that everybody, except people in their country who had been vaccinated for this purpose, would not be immune and a bacteriological agent like smallpox could be used as a weapon of war. D.A. was so incensed. He took this story to our government and we started a whole new section of the public health business to work on biological warfare. And he did for the last part of his life to try to get past this and that lab was eventually, we hope, closed down. So that scenario is not outside the bounds of possibility. But generally speaking, biological warfare isn't very effective because your own people could be infected as well, if not your war people, at least your civilian population. Much like poison gas, which used to blow back over the lines of the people who'd launched it to their horror. It never was a terribly good weapon of war. Plus Hitler's experience of using gas in the first world war that all sides decided not to use it in the second. So that's part of it.(2:16:45) - SecrecyDwarkesh Patel 2:16:45Speaking of secret labs, you've written so many books about not only the atomic bomb, the hydrogen bomb, and the Cold War and so on. Is there a question you have about any of those stories that you were really interested in but you haven't been able to get the answer to because the information is classified?Richard Rhodes 2:17:05Over the years, it's slowly leaked out. The latest thing I discovered is that from early on, our hydrogen bombs were shaped sort of like a dumbbell, more spread out. There's a picture of Kim Jong-un looking at a hydrogen bomb of North Korean make and it's perfectly obvious that it's a real bomb because that's its configuration. I didn't know that until just a year or so ago. But sooner or later, everyone tells you at least a little bit. Dwarkesh Patel 2:17:46And then is there anything you've learned that you can't put in your books because it's... Richard Rhodes 2:17:50The only thing I didn't put in the book, rightly so, was the dimensions of the Fat Man bomb that were on the original Fuchs documents that he passed to the Russians. When the Soviet Union collapsed and the scientists became available, I learned that the KGB in the interest of promoting its service to the state in this new world they were going into, had published a historical piece about their good work in stealing the secret to the bomb. And they included a picture of the sketch that Fuchs did showing the dimensions of each of the shells of this multi-shelled implosion weapon with the numbers there in millimeters. And when the scientists realized that the KGB had published this stuff, they raised a great hue and cry and said, that's in violation of the Nuclear Non-Puller Variant Treaty. They said, we have to pull all the issues of that journal. And they did. But I had a very enterprising assistant in Moscow, this geologist I mentioned before. And he said, I think I know where I can find a copy of the journal. And he jumped on the night train from Moscow to St. Petersburg and went to a science library there. And they said, no, of course not. We pulled that. And then he thought, wait a minute, where's the public library? It was across the street. And he went across the public library. And they said, yeah, we have the journal. And handed it to me. He made a copy and gave me one. But when I realized that I had this, I never published that information. That's the only one, though. Dwarkesh Patel 2:19:35That is a wise thing to do. One of the final questions. A lot of times people use the phrase, we need a Manhattan project for X. If it's some new technology or something. When you hear that, do you think that is a naive comparison? Or does it make sense to use that terminology? Richard Rhodes 2:19:57No. It’s been used so many times over the years for cancer, for this, for that. But Manhattan was a very special time and it was a very special problem. The basic science was in hand. Oppenheimer once said in his funny way, we didn't do any basic science between 1939 and 1945. In a sense, it was a vast engineering program. There was some basic physics, but very little. It was mostly using Nobel laureate level physicists as engineers to develop a whole new weapon with the precision of a Swiss watch that weighed 9,000 pounds. And they did a beautiful job considering the time and place and the rationale. They solved some problems that I don't know how anybody else could have solved. How do you make plutonium explode without a gun? All of that. But most of these problems aren't like that. Nobody was starting a startup with some investment from an investment company to build the bomb. This was a government project, and it was secret. And if you divulged the secret, you'd go to jail. A lot of the parameters of what they were doing were carefully kept secret. It's remarkable that 600,000 people worked on the bomb and the secret never got out. Dwarkesh Patel 2:21:28Do you say 600,000?Richard Rhodes 2:21:29YeahDwarkesh Patel 2:21:33This is actually one of the questions I want to ask you. Truman, when he became president, he had to be told about it. He didn't know about it, right? I don't know how many people were working on it when he became president, but hundreds of thousands of people are working on it. The vice president doesn't know about it. He only learns about it as president. How was it possible that with so many people working on it, the vice president doesn't know that the atom bomb is in the works? How did they keep that so under wraps? Richard Rhodes 2:21:57One of Roosevelt’s several vice presidents famously said, the vice presidency isn't worth a bucket of warm piss. Dwarkesh Patel 2:22:05Can you note, Kennedy had a saying, I'm against [unclear] in all its forms. Richard Rhodes 2:22:11Yes, well, interesting for me. Roosevelt wanted to keep it a secret and so did Groves. They didn't want the word to get out. They didn’t want the Germans to get ahead of us. And what was the vice president for? He was to sit around in a waiting room in case the president died, which in Truman's case, he hit the jackpot. He was just at the right time because Roosevelt had several vice presidents in his long reign. So from their point of view, he didn't need to know. It was the need-to-know thing again, it was the compartmentalization. And of course, as soon as he became president, Groves and Stimson and several others got together with Truman and filled him in on it. Truman had some inkling because he was a crusading senator who had taken on the job of preventing waste in war. And if he heard about some industry that was sloughing off and putting money in people's pockets or whatever, he would go visit their factories, call them out. And he was ready to go to Hanford, Washington, and Oak Ridge, and find out what these people were doing. But Stimson, the Secretary of War at the time, whom he greatly admired and one of the great elder statesmen of the era, said, “Mr. Vice President, please don't go. I guarantee you that what is happening is being well managed. Please don't go.” And he said, “If you say so, Mr. Stimson, I will believe you.” So he knew a little bit, but he didn't know very much. And I don't think that helped. I don't know how much Roosevelt understood either. He was not a scientist. Truman was not even a college graduate, he was self-educated. A well self-educated guy. One of the senator’s once said, every time I go to the Library of Congress to pull out a book, Truman's name is always there. The senator said I think he read everything in the Library of Congress. It's partly that there wasn't any way to communicate except by letter or telephone and you didn't call long distance unless somebody died, basically. Tell someone they had a long distance call and their woman would start crying. Or a man, for that matter. You thought your son was probably dead in Iwo Jima. So the communication was more limited to be sure. But even so, it's extraordinary. Dwarkesh Patel 2:24:45But was the culture different in that the idea of leaking something would be much more frowned upon than it is now? Richard Rhodes 2:24:52I can't remember in my entire life a more patriotic time than the Second World War. We children gathered together pieces of aluminum foil from the lining of cigarette packages, about the only place you could get aluminum foil in those days, watered it up into balls and took them to school so they could use it to make bombers. We collected this bacon grease from cooking meat in the kitchen and took it to school in cans because it had some application to making bullets. We collected newspapers for the same reason. The Boy Scouts during the Second World War took it as their special responsibility to collect the fibers that come off milkweed with this little ball that blows away because it was used in place of kapok to line life vests for sailors. They collected 500,000 tons of milkweed fluff in the course of the war. We were all consumed with winning this thing, which didn't seem to be a certain thing at all, as I said earlier, before around 1943. Of course, there was a black market and people got to see a farmer and pick up some steaks, so they didn't have to live on some more macaroni and cheese for the next month as we all did. But despite those changes, people were very, very patriotic and fought in every way we could to win the war. (2:26:34) - Wisdom & warDwarkesh Patel 2:26:34Speaking of elder statesmen, by the way. Who, since the development of the nuclear bomb, has been the wisest political leader, not necessarily the U.S. and not even necessarily as a leader of state, but contributed most to the stability and peace? Richard Rhodes 2:26:52It depends on what period you're talking about. There's no question that Oppenheimer's advice to the government after the war was really good. I don't think anyone except the Oppenheimer committee, the Acheson-Lilienthal Committee, has ever found a better way of thinking about eliminating nuclear weapons in a world that understands how to build them. And that really was Oppenheimer. I don't mean he deluded anybody. He just led them straight down the path. All these engineers and industrialists who were on the committee with him, skeptical men, men who wouldn't have been easily convinced of anything, but he convinced them this was the right way to go. So he, up until he was forced out of government because he made the mistake of not supporting the Air Force's favorite bomb, and they found a way to destroy him basically, and they did destroy him. I talked to one of the scientists who was his closest friend, and he said Robert was never the same after the security hearing in 1954. He was one of [unclear]’s smiling public men after that. It really devastated him. That basic insecurity he had as a Jew in America all the way from childhood, haunted him, and he became the director of the Institute for Advanced Study, and he wrote a lot of nice essays. But I don't know. The leader that I've fallen into more and more is Niels Bohr. He tried to figure out a way to bring the openness of science into a world without nuclear weapons. He and Robbie taught Oppenheimer the ideas that ended up in the Acheson-Lilienthal plan, and he and Robbie later, were the ones who started up the scientific laboratory in Geneva that is now CERN where they built these new giant colliders. With the idea in mind that at the very least if the scientists have devastated Europe, we remember what things were like for the former Soviet Union. People were living on crusts of bread and bags of old potatoes. They really were. And particularly people who worked for the government because their pensions weren't worth anything. That's what I saw when I went back there in the spring of ‘92 after the place collapsed. The money wasn't worth anything. Their salaries weren't being paid. In the midst of all of that, Europe needed something to sustain it. And of course, there was the Marshall Plan, and that was absolutely amazing. And the help it gave to Europe when it needed it most. But Bohr wanted to make sure, and Robbie wanted to make sure, that the scientists of Europe were tapped to go off somewhere and build nuclear weapons. And they invented this international laboratory in Geneva, which is still a thriving enterprise, where they could do basic physics and where they'd be paid for their work and could do the kind of exciting thing that good science is without having to drift over into the dark side of the whole thing. Let's face it, it more or less worked. The French had to have the bomb, you know, because they're French. Dwarkesh Patel 2:30:26They need their lovers, they need their bombs. Richard Rhodes 2:30:29The British had to have the bomb because they knew how to build one. They'd worked with us all during the war, and then we'd cut them off from a supply of uranium and from any new developments along the way. And most of all, because the British Empire was bankrupt by the end of the Second World War, and Churchill was determined to get the bomb because without it, his country would fall away and wouldn't get to sit at the table with the big boys. So they had the typical reasons for getting bombs because you don't want to be left out because of the prestige, or because you have a mortal enemy who's got the bomb or is getting the bomb. That's North Korea, that's Iraq. Its attempt,  it didn't get there. That's Pakistan and India. We, because of Germany. The Soviets, because of us. By and large, the countries that did go toward the bomb, finally built bombs because they were afraid of another country having the bomb. And everyone else stood back and said, well, if you'll protect us with atomic bombs, when somebody comes calling here in South Korea or Germany or wherever, we won't build them and share your knowledge of how to make energy out of nuclear power. Dwarkesh Patel 2:31:54There's easily another three hours of questions I could ask you. I can't say I want to be respectful of your time because I haven't been with the extra hour I've taken, but I want to be less than totally disrespectful of your time. So I guess the final question that we can go out on is — In the next 50 years, what odds do you put on a non-test nuke going off? Richard Rhodes 2:32:22A nuke going off in anger? That's the way I usually put it. I think the odds are high.Dwarkesh Patel 2:32:35Over 50% in the next 50 years? Richard Rhodes 2:32:26I wouldn’t put a number on it, but it's certainly higher than zero and it's probably higher than 10%. And that's high enough if we're talking about millions of people dying. There was a period when people in the field were talking about, well, maybe we'll have a little regional nuclear war between India and Pakistan and that'll scare everybody to the point where they realize you've got to get rid of these things. The same guys who did the nuclear winter studies back in the 80s decided in 2007 first to look at nuclear winter world scale war using the much better computers of today. And they found out that that would be even worse than they thought it would when they had only one dimensional atmospheric models. Then they said, well, what would a small regional nuclear war look like? So they simulated a war between India and Pakistan where each country explodes 50 Hiroshima sized, 15 kiloton nuclear weapons over the enemy cities. And what would follow from that? And it turned out as the model develops, you can see it online, you can watch the graph develop that even that small in exchange, less than some of our individual hydrogen bombs, about a megaton between the two countries, would be enough to cause enough fire from burning cities to spread smoke around the world and reduce the amount of sunlight. They figured in the end that there would be 20 million prompt deaths from the explosions themselves and the fires. But then over the course of the next several years, up to two billion people would die of starvation because you would have the same phenomenon that the world had in the 18th century when there was an interim of rather cold, the sun was pulling back a bit and it was freezing hard in July in New England and the crops failed. And a mass of people died worldwide during that period of time. Like the flu epidemic of 1918, everybody seems to have forgotten. I don't know where our memory goes with these things. Therefore even a small so-called nuclear war must engage the whole world because it's going to engage us if it ever goes off. So we're still in a very precarious place. And as long as any country in the world has nuclear weapons, we're going to continue to be. There is a sword of Damocles over our heads. That has been the price of nuclear deterrence. It isn't necessary that that be the price. It's possible to have nuclear deterrence without actual weapons, but it's damned hard to convince leaders, particularly in totalitarian and authoritarian countries, that that's the case. So the odds, I don't know, but there's no such thing as a machine that doesn't malfunction sooner or later. And these machines, these weapons that we make us such supernatural powers of, are just machines. We built, put them together. We can take them apart. We can put the ore back in the ground if we want to. We don't have to live with this. I think we're in a big, wide, long transition. Maybe the world scale problem of solving global warming will help with this, will help people see that if they work together. Here I am saying there could be a good outcome from this technology. Well, there was a good outcome from the telephone. There was a good outcome from television, but there was also a dark side and we're going to have to learn to handle dark sides better. Maybe that's the last thing to say on this subject. Dwarkesh Patel 2:36:39Yeah, that's a sound note to close on. The book is The Making of the Atomic Bomb. Unfortunately, we didn't get a chance to talk as much about your new book on energy, but when your next book comes out, we'll get a chance to talk about the remainder that we didn't get a chance to talk about. It's been a true honor, and an incredible pleasure. The stories, the insight. It was really wonderful. Thank you so much for your time.Richard Rhodes 2:37:02My pleasure.  Get full access to The Lunar Society at www.dwarkeshpatel.com/subscribe
5/23/20232 hours, 37 minutes, 36 seconds
Episode Artwork

Eliezer Yudkowsky - Why AI Will Kill Us, Aligning LLMs, Nature of Intelligence, SciFi, & Rationality

For 4 hours, I tried to come up reasons for why AI might not kill us all, and Eliezer Yudkowsky explained why I was wrong.We also discuss his call to halt AI, why LLMs make alignment harder, what it would take to save humanity, his millions of words of sci-fi, and much more.If you want to get to the crux of the conversation, fast forward to 2:35:00 through 3:43:54. Here we go through and debate the main reasons I still think doom is unlikely.Watch on YouTube. Listen on Apple Podcasts, Spotify, or any other podcast platform. Read the full transcript here. Follow me on Twitter for updates on future episodes.As always, the most helpful thing you can do is just to share the podcast - send it to friends, group chats, Twitter, Reddit, forums, and wherever else men and women of fine taste congregate.If you have the means and have enjoyed my podcast, I would appreciate your support via a paid subscriptions on Substack.Timestamps(0:00:00) - TIME article(0:09:06) - Are humans aligned?(0:37:35) - Large language models(1:07:15) - Can AIs help with alignment?(1:30:17) - Society’s response to AI(1:44:42) - Predictions (or lack thereof)(1:56:55) - Being Eliezer(2:13:06) - Othogonality(2:35:00) - Could alignment be easier than we think?(3:02:15) - What will AIs want?(3:43:54) - Writing fiction & whether rationality helps you winTranscriptTIME articleDwarkesh Patel 0:00:51Today I have the pleasure of speaking with Eliezer Yudkowsky. Eliezer, thank you so much for coming out to the Lunar Society.Eliezer Yudkowsky 0:01:00You’re welcome.Dwarkesh Patel 0:01:01Yesterday, when we’re recording this, you had an article in Time calling for a moratorium on further AI training runs. My first question is — It’s probably not likely that governments are going to adopt some sort of treaty that restricts AI right now. So what was the goal with writing it?Eliezer Yudkowsky 0:01:25I thought that this was something very unlikely for governments to adopt and then all of my friends kept on telling me — “No, no, actually, if you talk to anyone outside of the tech industry, they think maybe we shouldn’t do that.” And I was like — All right, then. I assumed that this concept had no popular support. Maybe I assumed incorrectly. It seems foolish and to lack dignity to not even try to say what ought to be done. There wasn’t a galaxy-brained purpose behind it. I think that over the last 22 years or so, we’ve seen a great lack of galaxy brained ideas playing out successfully.Dwarkesh Patel 0:02:05Has anybody in the government reached out to you, not necessarily after the article but just in general, in a way that makes you think that they have the broad contours of the problem correct?Eliezer Yudkowsky 0:02:15No. I’m going on reports that normal people are more willing than the people I’ve been previously talking to, to entertain calls that this is a bad idea and maybe you should just not do that.Dwarkesh Patel 0:02:30That’s surprising to hear, because I would have assumed that the people in Silicon Valley who are weirdos would be more likely to find this sort of message. They could kind of rocket the whole idea that AI will make nanomachines that take over. It’s surprising to hear that normal people got the message first.Eliezer Yudkowsky 0:02:47Well, I hesitate to use the term midwit but maybe this was all just a midwit thing.Dwarkesh Patel 0:02:54All right. So my concern with either the 6 month moratorium or forever moratorium until we solve alignment is that at this point, it could make it seem to people like we’re crying wolf. And it would be like crying wolf because these systems aren’t yet at a point at which they’re dangerous. Eliezer Yudkowsky 0:03:13And nobody is saying they are. I’m not saying they are. The open letter signatories aren’t saying they are.Dwarkesh Patel 0:03:20So if there is a point at which we can get the public momentum to do some sort of stop, wouldn’t it be useful to exercise it when we get a GPT-6? And who knows what it’s capable of. Why do it now?Eliezer Yudkowsky 0:03:32Because allegedly, and we will see, people right now are able to appreciate that things are storming ahead a bit faster than the ability to ensure any sort of good outcome for them. And you could be like — “Ah, yes. We will play the galaxy-brained clever political move of trying to time when the popular support will be there.” But again, I heard rumors that people were actually completely open to the concept of  let’s stop. So again, I’m just trying to say it. And it’s not clear to me what happens if we wait for GPT-5 to say it. I don’t actually know what GPT-5 is going to be like. It has been very hard to call the rate at which these systems acquire capability as they are trained to larger and larger sizes and more and more tokens. GPT-4 is a bit beyond in some ways where I thought this paradigm was going to scale. So I don’t actually know what happens if GPT-5 is built. And even if GPT-5 doesn’t end the world, which I agree is like more than 50% of where my probability mass lies, maybe that’s enough time for GPT-4.5 to get ensconced everywhere and in everything, and for it actually to be harder to call a stop, both politically and technically. There’s also the point that training algorithms keep improving. If we put a hard limit on the total computes and training runs right now, these systems would still get more capable over time as the algorithms improved and got more efficient. More oomph per floating point operation, and things would still improve, but slower. And if you start that process off at the GPT-5 level, where I don’t actually know how capable that is exactly, you may have a bunch less lifeline left before you get into dangerous territory.Dwarkesh Patel 0:05:46The concern is then that — there’s millions of GPUs out there in the world. The actors who would be willing to cooperate or who could even be identified in order to get the government to make them cooperate, would potentially be the ones that are most on the message. And so what you’re left with is a system where they stagnate for six months or a year or however long this lasts. And then what is the game plan? Is there some plan by which if we wait a few years, then alignment will be solved? Do we have some sort of timeline like that?Eliezer Yudkowsky 0:06:18Alignment will not be solved in a few years. I would hope for something along the lines of human intelligence enhancement works. I do not think they’re going to have the timeline for genetically engineered humans to work but maybe? This is why I mentioned in the Time letter that if I had infinite capability to dictate the laws that there would be a carve-out on biology, AI that is just for biology and not trained on text from the internet. Human intelligence enhancement, make people smarter. Making people smarter has a chance of going right in a way that making an extremely smart AI does not have a realistic chance of going right at this point. If we were on a sane planet, what the sane planet does at this point is shut it all down and work on human intelligence enhancement. I don’t think we’re going to live in that sane world. I think we are all going to die. But having heard that people are more open to this outside of California, it makes sense to me to just try saying out loud what it is that you do on a saner planet and not just assume that people are not going to do that.Dwarkesh Patel 0:07:30In what percentage of the worlds where humanity survives is there human enhancement? Like even if there’s 1% chance humanity survives, is that entire branch dominated by the worlds where there’s some sort of human intelligence enhancement?Eliezer Yudkowsky 0:07:39I think we’re just mainly in the territory of Hail Mary passes at this point, and human intelligence enhancement is one Hail Mary pass. Maybe you can put people in MRIs and train them using neurofeedback to be a little saner, to not rationalize so much. Maybe you can figure out how to have something light up every time somebody is working backwards from what they want to be true to what they take as their premises. Maybe you can just fire off little lights and teach people not to do that so much. Maybe the GPT-4 level systems can be RLHF’d (reinforcement learning from human feedback) into being consistently smart, nice and charitable in conversation and just unleash a billion of them on Twitter and just have them spread sanity everywhere. I do worry that this is not going to be the most profitable use of the technology, but you’re asking me to list out Hail Mary passes and that’s what I’m doing. Maybe you can actually figure out how to take a brain, slice it, scan it, simulate it, run uploads and upgrade the uploads, or run the uploads faster. These are also quite dangerous things, but they do not have the utter lethality of artificial intelligence.Are humans aligned?Dwarkesh Patel 0:09:06All right, that’s actually a great jumping point into the next topic I want to talk to you about. Orthogonality. And here’s my first question — Speaking of human enhancement, suppose you bred human beings to be friendly and cooperative, but also more intelligent. I claim that over many generations you would just have really smart humans who are also really friendly and cooperative. Would you disagree with that analogy? I’m sure you’re going to disagree with this analogy, but I just want to understand why?Eliezer Yudkowsky 0:09:31The main thing is that you’re starting from minds that are already very, very similar to yours. You’re starting from minds, many of which already exhibit the characteristics that you want. There are already many people in the world, I hope, who are nice in the way that you want them to be nice. Of course, it depends on how nice you want exactly. I think that if you actually go start trying to run a project of selectively encouraging some marriages between particular people and encouraging them to have children, you will rapidly find, as one does in any such process that when you select on the stuff you want, it turns out there’s a bunch of stuff correlated with it and that you’re not changing just one thing. If you try to make people who are inhumanly nice, who are nicer than anyone has ever been before, you’re going outside the space that human psychology has previously evolved and adapted to deal with, and weird stuff will happen to those people. None of this is very analogous to AI. I’m just pointing out something along the lines of — well, taking your analogy at face value, what would happen exactly? It’s the sort of thing where you could maybe do it, but there’s all kinds of pitfalls that you’d probably find out about if you cracked open a textbook on animal breeding.Dwarkesh Patel 0:11:13The thing you mentioned initially, which is that we are starting off with basic human psychology, that we are fine tuning with breeding. Luckily, the current paradigm of AI is  — you have these models that are trained on human text and I would assume that this would give you a starting point of something like human psychology.Eliezer Yudkowsky 0:11:31Why do you assume that?Dwarkesh Patel 0:11:33Because they’re trained on human text.Eliezer Yudkowsky 0:11:34And what does that do?Dwarkesh Patel 0:11:36Whatever thoughts and emotions that lead to the production of human text need to be simulated in the AI in order to produce those results.Eliezer Yudkowsky 0:11:44I see. So if you take an actor and tell them to play a character, they just become that person. You can tell that because you see somebody on screen playing Buffy the Vampire Slayer, and that’s probably just actually Buffy in there. That’s who that is.Dwarkesh Patel 0:12:05I think a better analogy is if you have a child and you tell him — Hey, be this way. They’re more likely to just be that way instead of putting on an act for 20 years or something.Eliezer Yudkowsky 0:12:18It depends on what you’re telling them to be exactly. Dwarkesh Patel 0:12:20You’re telling them to be nice.Eliezer Yudkowsky 0:12:22Yeah, but that’s not what you’re telling them to do. You’re telling them to play the part of an alien, something with a completely inhuman psychology as extrapolated by science fiction authors, and in many cases done by computers because humans can’t quite think that way. And your child eventually manages to learn to act that way. What exactly is going on in there now? Are they just the alien or did they pick up the rhythm of what you’re asking them to imitate and be like — “Ah yes, I see who I’m supposed to pretend to be.” Are they actually a person or are they pretending? That’s true even if you’re not asking them to be an alien. My parents tried to raise me Orthodox Jewish and that did not take at all. I learned to pretend. I learned to comply. I hated every minute of it. Okay, not literally every minute of it. I should avoid saying untrue things. I hated most minutes of it. Because they were trying to show me a way to be that was alien to my own psychology and the religion that I actually picked up was from the science fiction books instead, as it were. I’m using religion very metaphorically here, more like ethos, you might say. I was raised with science fiction books I was reading from my parents library and Orthodox Judaism. The ethos of the science fiction books rang truer in my soul and so that took in, the Orthodox Judaism didn't. But the Orthodox Judaism was what I had to imitate, was what I had to pretend to be, was the answers I had to give whether I believed them or not. Because otherwise you get punished.Dwarkesh Patel 0:14:01But on that point itself, the rates of apostasy are probably below 50% in any religion. Some people do leave but often they just become the thing they’re imitating as a child.Eliezer Yudkowsky 0:14:12Yes, because the religions are selected to not have that many apostates. If aliens came in and introduced their religion, you’d get a lot more apostates.Dwarkesh Patel 0:14:19Right. But I think we’re probably in a more virtuous situation with ML because these systems are regularized through stochastic gradient descent. So the system that is pretending to be something where there’s multiple layers of interpretation is going to be more complex than the one that is just being the thing. And over time, the system that is just being the thing will be optimized, right? It’ll just be simpler.Eliezer Yudkowsky 0:14:42This seems like an ordinate cope. For one thing, you’re not training it to be any one particular person. You’re training it to switch masks to anyone on the Internet as soon as they figure out who that person on the internet is. If I put the internet in front of you and I was like — learn to predict the next word over and over. You do not just turn into a random human because the random human is not what’s best at predicting the next word of everyone who’s ever been on the internet. You learn to very rapidly pick up on the cues of what sort of person is talking, what will they say next? You memorize so many facts just because they’re helpful in predicting the next word. You learn all kinds of patterns, you learn all the languages. You learn to switch rapidly from being one kind of person or another as the conversation that you are predicting changes who is speaking. This is not a human we’re describing. You are not training a human there.Dwarkesh Patel 0:15:43Would you at least say that we are living in a better situation than one in which we have some sort of black box where you have a machiavellian fittest survive simulation that produces AI? This situation is at least more likely to produce alignment than one in which something that is completely untouched by human psychology would produce?Eliezer Yudkowsky 0:16:06More likely? Yes. Maybe you’re an order of magnitude likelier. 0% instead of 0%. Getting stuff to be more likely does not help you if the baseline is nearly zero. The whole training set up there is producing an actress, a predictor. It’s not actually being put into the kind of ancestral situation that evolved humans, nor the kind of modern situation that raises humans. Though to be clear, raising it like a human wouldn’t help, But you’re giving it a very alien problem that is not what humans solve and it is solving that problem not in the way a human would.Dwarkesh Patel 0:16:44Okay, so how about this. I can see that I certainly don’t know for sure what is going on in these systems. In fact, obviously nobody does. But that also goes through you. Could it not just be that reinforcement learning works and all these other things we’re trying somehow work and actually just being an actor produces some sort of benign outcome where there isn’t that level of simulation and conniving?Eliezer Yudkowsky 0:17:15I think it predictably breaks down as you try to make the system smarter, as you try to derive sufficiently useful work from it. And in particular, the sort of work where some other AI doesn’t just kill you off six months later. Yeah, I think the present system is not smart enough to have a deep conniving actress thinking long strings of coherent thoughts about how to predict the next word. But as the mask that it wears, as the people it is pretending to be get smarter and smarter, I think that at some point the thing in there that is predicting how humans plan, predicting how humans talk, predicting how humans think, and needing to be at least as smart as the human it is predicting in order to do that, I suspect at some point there is a new coherence born within the system and something strange starts happening. I think that if you have something that can accurately predict Eliezer Yudkowsky, to use a particular example I know quite well, you’ve got to be able to do the kind of thinking where you are reflecting on yourself and that in order to simulate Eliezer Yudkowsky reflecting on himself, you need to be able to do that kind of thinking. This is not airtight logic but I expect there to be a discount factor. If you ask me to play a part of somebody who’s quite unlike me, I think there’s some amount of penalty that the character I’m playing gets to his intelligence because I’m secretly back there simulating him. That’s even if we’re quite similar and the stranger they are, the more unfamiliar the situation, the less the person I’m playing is as smart as I am and the more they are dumber than I am. So similarly, I think that if you get an AI that’s very, very good at predicting what Eliezer says, I think that there’s a quite alien mind doing that, and it actually has to be to some degree smarter than me in order to play the role of something that thinks differently from how it does very, very accurately. And I reflect on myself, I think about how my thoughts are not good enough by my own standards and how I want to rearrange my own thought processes. I look at the world and see it going the way I did not want it to go, and asking myself how could I change this world? I look around at other humans and I model them, and sometimes I try to persuade them of things. These are all capabilities that the system would then be somewhere in there. And I just don’t trust the blind hope that all of that capability is pointed entirely at pretending to be Eliezer and only exists insofar as it’s the mirror and isomorph of Eliezer. That all the prediction is by being something exactly like me and not thinking about me while not being me.Dwarkesh Patel 0:20:55I certainly don’t want to claim that it is guaranteed that there isn’t something super alien and something against our aims happening within the shoggoth. But you made an earlier claim which seemed much stronger than the idea that you don’t want blind hope, which is that we’re going from 0% probability to an order of magnitude greater at 0% probability. There’s a difference between saying that we should be wary and that there’s no hope, right? I could imagine so many things that could be happening in the shoggoth’s brain, especially in our level of confusion and mysticism over what is happening. One example is, let’s say that it kind of just becomes the average of all human psychology and motives.Eliezer Yudkowsky 0:21:41But it’s not the average. It is able to be every one of those people. That’s very different from being the average. It’s very different from being an average chess player versus being able to predict every chess player in the database. These are very different things.Dwarkesh Patel 0:21:56Yeah, no, I meant in terms of motives that it is the average where it can simulate any given human. I’m not saying that’s the most likely one, I’m just saying it’s one possibility.Eliezer Yudkowsky 0:22:08What.. Why? It just seems 0% probable to me. Like the motive is going to be like some weird funhouse mirror thing of — I want to predict very accurately.Dwarkesh Patel 0:22:19Right. Why then are we so sure that whatever drives that come about because of this motive are going to be incompatible with the survival and flourishing with humanity?Eliezer Yudkowsky 0:22:30Most drives when you take a loss function and splinter it into things correlated with it and then amp up intelligence until some kind of strange coherence is born within the thing and then ask it how it would want to self modify or what kind of successor system it would build. Things that alien ultimately end up wanting the universe to be some particular way such that humans are not a solution to the question of how to make the universe most that way. The thing that very strongly wants to predict text, even if you got that goal into the system exactly which is not what would happen, The universe with the most predictable text is not a universe that has humans in it. Dwarkesh Patel 0:23:19Okay. I’m not saying this is the most likely outcome. Here’s an example of one of many ways in which humans stay around despite this motive. Let’s say that in order to predict human output really well, it needs humans around to give it the raw data from which to improve its predictions or something like that. This is not something I think individually is likely…Eliezer Yudkowsky 0:23:40If the humans are no longer around, you no longer need to predict them. Right, so you don’t need the data required to predict themDwarkesh Patel 0:23:46Because you are starting off with that motivation you want to just maximize along that loss function or have that drive that came about because of the loss function.Eliezer Yudkowsky 0:23:57I’m confused. So look, you can always develop arbitrary fanciful scenarios in which the AI has some contrived motive that it can only possibly satisfy by keeping humans alive in good health and comfort and turning all the nearby galaxies into happy, cheerful places full of high functioning galactic civilizations. But as soon as your sentence has more than like five words in it, its probability has dropped to basically zero because of all the extra details you’re padding in.Dwarkesh Patel 0:24:31Maybe let’s return to this. Another train of thought I want to follow is — I claim that humans have not become orthogonal to the sort of evolutionary process that produced them.Eliezer Yudkowsky 0:24:46Great. I claim humans are increasingly orthogonal and the further they go out of distribution and the smarter they get, the more orthogonal they get to inclusive genetic fitness, the sole loss function on which humans were optimized.Dwarkesh Patel 0:25:03Most humans still want kids and have kids and care for their kin. Certainly there’s some angle between how humans operate today. Evolution would prefer us to use less condoms and more sperm banks. But there’s like 10 billion of us and there’s going to be more in the future. We haven’t divorced that far from what our alleles would want.Eliezer Yudkowsky 0:25:28It’s a question of how far out of distribution are you? And the smarter you are, the more out of distribution you get. Because as you get smarter, you get new options that are further from the options that you are faced with in the ancestral environment that you were optimized over. Sure, a lot of people want kids, not inclusive genetic fitness, but kids. They want kids similar to them maybe, but they don’t want the kids to have their DNA or their alleles or their genes. So suppose I go up to somebody and credibly say, we will assume away the ridiculousness of this offer for the moment, your kids could be a bit smarter and much healthier if you’ll just let me replace their DNA with this alternate storage method that will age more slowly. They’ll be healthier, they won’t have to worry about DNA damage, they won’t have to worry about the methylation on the DNA flipping and the cells de-differentiating as they get older. We’ve got this stuff that replaces DNA and your kid will still be similar to you, it’ll be a bit smarter and they’ll be so much healthier and even a bit more cheerful. You just have to replace all the DNA with a stronger substrate and rewrite all the information on it. You know, the old school transhumanist offer really. And I think that a lot of the people who want kids would go for this new offer that just offers them so much more of what it is they want from kids than copying the DNA, than inclusive genetic fitness.Dwarkesh Patel 0:27:16In some sense, I don’t even think that would dispute my claim because if you think from a gene’s point of view, it just wants to be replicated. If it’s replicated in another substrate that’s still okay.Eliezer Yudkowsky 0:27:25No, we’re not saving the information. We’re doing a total rewrite to the DNA.Dwarkesh Patel 0:27:30I actually claim that most humans would not accept that offer.Eliezer Yudkowsky 0:27:33Yeah, because it would sound weird. But I think the smarter they are, the more likely they are to go for it if it’s credible. I mean, if you assume away the credibility issue and the weirdness issue. Like all their friends are doing it.Dwarkesh Patel 0:27:52Yeah. Even if the smarter they are the more likely they’re to do it, most humans are not that smart. From the gene’s point of view it doesn’t really matter how smart you are, right? It just matters if you’re producing copies.Eliezer Yudkowsky 0:28:03No. The smart thing is kind of like a delicate issue here because somebody could always be like — I would never take that offer. And then I’m like “Yeah…”. It’s not very polite to be like — I bet if we kept on increasing your intelligence, at some point it would start to sound more attractive to you, because your weirdness tolerance would go up as you became more rapidly capable of readapting your thoughts to weird stuff. The weirdness would start to seem less unpleasant and more like you were moving within a space that you already understood. But you can sort of avoid all that and maybe should by being like — suppose all your friends were doing it. What if it was normal? What if we remove the weirdness and remove any credibility problems in that hypothetical case? Do people choose for their kids to be dumber, sicker, less pretty out of some sentimental idealistic attachment to using Deoxyribose Nucleic Acid instead of the particular information encoding their cells as supposed to be like the new improved cells from Alpha-Fold 7?Dwarkesh Patel 0:29:21I would claim that they would but we don’t really know. I claim that they would be more averse to that, you probably think that they would be less averse to that. Regardless of that, we can just go by the evidence we do have in that we are already way out of distribution of the ancestral environment. And even in this situation, the place where we do have evidence, people are still having kids. We haven’t gone that orthogonal.Eliezer Yudkowsky 0:29:44We haven’t gone that smart. What you’re saying is — Look, people are still making more of their DNA in a situation where nobody has offered them a way to get all the stuff they want without the DNA. So of course they haven’t tossed DNA out the window.Dwarkesh Patel 0:29:59Yeah. First of all, I’m not even sure what would happen in that situation. I still think even most smart humans in that situation might disagree, but we don’t know what would happen in that situation. Why not just use the evidence we have so far?Eliezer Yudkowsky 0:30:10PCR. You right now, could get some of you and make like a whole gallon jar full of your own DNA. Are you doing that? No. Misaligned. Misaligned.Dwarkesh Patel 0:30:23I’m down with transhumanism. I’m going to have my kids use the new cells and whatever.Eliezer Yudkowsky 0:30:27Oh, so we’re all talking about these hypothetical other people I think would make the wrong choice.Dwarkesh Patel 0:30:32Well, I wouldn’t say wrong, but different. And I’m just saying there’s probably more of them than there are of us.Eliezer Yudkowsky 0:30:37What if, like, I say that I have more faith in normal people than you do to toss DNA out the window as soon as somebody offers them a happy, healthier life for their kids?Dwarkesh Patel 0:30:46I’m not even making a moral point. I’m just saying I don’t know what’s going to happen in the future. Let’s just look at the evidence we have so far, humans. If that’s the evidence you’re going to present for something that’s out of distribution and has gone orthogonal, that has actually not happened. This is evidence for hope. Eliezer Yudkowsky 0:31:00Because we haven’t yet had options as far enough outside of the ancestral distribution that in the course of choosing what we most want that there’s no DNA left.Dwarkesh Patel 0:31:10Okay. Yeah, I think I understand.Eliezer Yudkowsky 0:31:12But you yourself say, “Oh yeah, sure, I would choose that.” and I myself say, “Oh yeah, sure, I would choose that.” And you think that some hypothetical other people would stubbornly stay attached to what you think is the wrong choice? First of all, I think maybe you’re being a bit condescending there. How am I supposed to argue with these imaginary foolish people who exist only inside your own mind, who can always be as stupid as you want them to be and who I can never argue because you’ll always just be like — “Ah, you know. They won’t be persuaded by that.” But right here in this room, the site of this videotaping, there is no counter evidence that smart enough humans will toss DNA out the window as soon as somebody makes them a sufficiently better offer.Dwarkesh Patel 0:31:55I’m not even saying it’s stupid. I’m just saying they’re not weirdos like me and you.Eliezer Yudkowsky 0:32:01Weird is relative to intelligence. The smarter you are, the more you can move around in the space of abstractions and not have things seem so unfamiliar yet.Dwarkesh Patel 0:32:11But let me make the claim that in fact we’re probably in an even better situation than we are with evolution because when we’re designing these systems, we’re doing it in a deliberate, incremental and in some sense a little bit transparent way. Eliezer Yudkowsky 0:32:27No, no, not yet, not now. Nobody’s being careful and deliberate now, but maybe at some point in the indefinite future people will be careful and deliberate. Sure, let’s grant that premise. Keep going.Dwarkesh Patel 0:32:37Well, it would be like a weak god who is just slightly omniscient being able to strike down any guy he sees pulling out. Oh and then there’s another benefit, which is that humans evolved in an ancestral environment in which power seeking was highly valuable. Like if you’re in some sort of tribe or something.Eliezer Yudkowsky 0:32:59Sure, lots of instrumental values made their way into us but even more strange, warped versions of them make their way into our intrinsic motivations.Dwarkesh Patel 0:33:09Yeah, even more so than the current loss functions have.Eliezer Yudkowsky 0:33:10Really? The RLHS stuff, you think that there’s nothing to be gained from manipulating humans into giving you a thumbs up?Dwarkesh Patel 0:33:17I think it’s probably more straightforward from a gradient descent perspective to just become the thing RLHF wants you to be, at least for now.Eliezer Yudkowsky 0:33:24Where are you getting this?Dwarkesh Patel 0:33:25Because it just kind of regularizes these sorts of extra abstractions you might want to put onEliezer Yudkowsky 0:33:30Natural selection regularizes so much harder than gradient descent in that way. It’s got an enormously stronger information bottleneck. Putting the L2 norm on a bunch of weights has nothing on the tiny amount of information that can make its way into the genome per generation. The regularizers on natural selection are enormously stronger.Dwarkesh Patel 0:33:51Yeah. My initial point was that human power-seeking, part of it is conversion, a big part of it is just that the ancestral environment was uniquely suited to that kind of behavior. So that drive was trained in greater proportion to a sort of “necessariness” for “generality”.Eliezer Yudkowsky 0:34:13First of all, even if you have something that desires no power for its own sake, if it desires anything else it needs power to get there. Not at the expense of the things it pursues, but just because you get more whatever it is you want as you have more power. And sufficiently smart things know that. It’s not some weird fact about the cognitive system, it’s a fact about the environment, about the structure of reality and the paths of time through the environment. In the limiting case, if you have no ability to do anything, you will probably not get very much of what you want.Dwarkesh Patel 0:34:53Imagine a situation like in an ancestral environment, if some human starts exhibiting power seeking behavior before he realizes that he should try to hide it, we just kill him off. And the friendly cooperative ones, we let them breed more. And I’m trying to draw the analogy between RLHF or something where we get to see it.Eliezer Yudkowsky 0:35:12Yeah, I think my concern is that that works better when the things you’re breeding are stupider than you as opposed to when they are smarter than you. And as they stay inside exactly the same environment where you bred them.Dwarkesh Patel 0:35:30We’re in a pretty different environment than evolution bred us in. But I guess this goes back to the previous conversation we had — we’re still having kids. Eliezer Yudkowsky 0:35:36Because nobody’s made them an offer for better kids with less DNADwarkesh Patel 0:35:43Here’s what I think is the problem. I can just look out of the world and see this is what it looks like. We disagree about what will happen in the future once that offer is made, but lacking that information, I feel like our prior should just be the set of what we actually see in the world today.Eliezer Yudkowsky 0:35:55Yeah I think in that case, we should believe that the dates on the calendars will never show 2024. Every single year throughout human history, in the 13.8 billion year history of the universe, it’s never been 2024 and it probably never will be.Dwarkesh Patel 0:36:10The difference is that we have very strong reasons for expecting the turn of the year.Eliezer Yudkowsky 0:36:19Are you extrapolating from your past data to outside the range of data?Dwarkesh Patel 0:36:24Yes, I think we have a good reason to. I don’t think human preferences are as predictable as dates.Eliezer Yudkowsky 0:36:29Yeah, they’re somewhat less so. Sorry, why not jump on this one? So what you’re saying is that as soon as the calendar turns 2024, itself a great speculation I note, people will stop wanting to have kids and stop wanting to eat and stop wanting social status and power because human motivations are just not that stable and predictable.Dwarkesh Patel 0:36:51No. That’s not what I’m claiming at all. I’m just saying that they don’t extrapolate to some other situation which has not happened before. Eliezer Yudkowsky 0:36:59Like the clock showing 2024?Dwarkesh Patel 0:37:01What is an example here? Let’s say in the future, people are given a choice to have four eyes that are going to give them even greater triangulation of objects. I wouldn’t assume that they would choose to have four eyes.Eliezer Yudkowsky 0:37:16Yeah. There’s no established preference for four eyes.Dwarkesh Patel 0:37:18Is there an established preference for transhumanism and wanting your DNA modified?Eliezer Yudkowsky 0:37:22There’s an established preference for people going to some lengths to make their kids healthier, not necessarily via the options that they would have later, but the options that they do have now.Large language modelsDwarkesh Patel 0:37:35Yeah. We’ll see, I guess, when that technology becomes available. Let me ask you about LLMs. So what is your position now about whether these things can get us to AGI?Eliezer Yudkowsky 0:37:47I don’t know. I was previously like — I don’t think stack more layers does this. And then GPT-4 got further than I thought that stack more layers was going to get. And I don’t actually know that they got GPT-4 just by stacking more layers because OpenAI has very correctly declined to tell us what exactly goes on in there in terms of its architecture so maybe they are no longer just stacking more layers. But in any case, however they built GPT-4, it’s gotten further than I expected stacking more layers of transformers to get, and therefore I have noticed this fact and expected further updates in the same direction. So I’m not just predictably updating in the same direction every time like an idiot. And now I do not know. I am no longer willing to say that GPT-6 does not end the world.Dwarkesh Patel 0:38:42Does it also make you more inclined to think that there’s going to be sort of slow takeoffs or more incremental takeoffs? Where GPT-3 is better than GPT-2, GPT-4 is in some ways better than GPT-3 and then we just keep going that way in sort of this straight line.Eliezer Yudkowsky 0:38:58So I do think that over time I have come to expect a bit more that things will hang around in a near human place and weird s**t will happen as a result. And my failure review where I look back and ask — was that a predictable sort of mistake? I feel like it was to some extent maybe a case of — you’re always going to get capabilities in some order and it was much easier to visualize the endpoint where you have all the capabilities than where you have some of the capabilities. And therefore my visualizations were not dwelling enough on a space we’d predictably in retrospect have entered into later where things have some capabilities but not others and it’s weird. I do think that, in 2012, I would not have called that large language models were the way and the large language models are in some way more uncannily semi-human than what I would justly have predicted in 2012 knowing only what I knew then. But broadly speaking, yeah, I do feel like GPT-4 is already kind of hanging out for longer in a weird, near-human space than I was really visualizing. In part, that's because it's so incredibly hard to visualize or predict correctly in advance when it will happen, which is, in retrospect, a bias.Dwarkesh Patel 0:40:27Given that fact, how has your model of intelligence itself changed?Eliezer Yudkowsky 0:40:31Very little.Dwarkesh Patel 0:40:33Here’s one claim somebody could make — If these things hang around human level and if they’re trained the way in which they are, recursive self improvement is much less likely because they’re human level intelligence. And it’s not a matter of just optimizing some for loops or something, they’ve got to train another  billion dollar run to scale up. So that kind of recursive self intelligence idea is less likely. How do you respond?Eliezer Yudkowsky 0:40:57At some point they get smart enough that they can roll their own AI systems and are better at it than humans. And that is the point at which you definitely start to see foom. Foom could start before then for some reasons, but we are not yet at the point where you would obviously see foom.Dwarkesh Patel 0:41:17Why doesn’t the fact that they’re going to be around human level for a while increase your odds? Or does it increase your odds of human survival? Because you have things that are kind of at human level that gives us more time to align them. Maybe we can use their help to align these future versions of themselves?Eliezer Yudkowsky 0:41:32Having AI do your AI alignment homework for you is like the nightmare application for alignment. Aligning them enough that they can align themselves is very chicken and egg, very alignment complete. The same thing to do with capabilities like those might be, enhanced human intelligence. Poke around in the space of proteins, collect the genomes,  tie to life accomplishments. Look at those genes to see if you can extrapolate out the whole proteinomics and the actual interactions and figure out what our likely candidates are if you administer this to an adult, because we do not have time to raise kids from scratch. If you administer this to an adult, the adult gets smarter. Try that. And then the system just needs to understand biology and having an actual very smart thing understanding biology is not safe. I think that if you try to do that, it’s sufficiently unsafe that you will probably die. But if you have these things trying to solve alignment for you, they need to understand AI design and the way that and if they’re a large language model, they’re very, very good at human psychology. Because predicting the next thing you’ll do is their entire deal. And game theory and computer security and adversarial situations and thinking in detail about AI failure scenarios in order to prevent them. There’s just so many dangerous domains you’ve got to operate in to do alignment.Dwarkesh Patel 0:43:35Okay. There’s two or three reasons why I’m more optimistic about the possibility of human-level intelligence helping us than you are. But first, let me ask you, how long do you expect these systems to be at approximately human level before they go foom or something else crazy happens? Do you have some sense? Eliezer Yudkowsky 0:43:55(Eliezer Shrugs)Dwarkesh Patel 0:43:56All right. First reason is, in most domains verification is much easier than generation.Eliezer Yudkowsky 0:44:03Yes. That’s another one of the things that makes alignment the nightmare. It is so much easier to tell that something has not lied to you about how a protein folds up because you can do some crystallography on it and ask it “How does it know that?”, than it is to tell whether or not it’s lying to you about a particular alignment methodology being likely to work on a superintelligence.Dwarkesh Patel 0:44:26Do you think confirming new solutions in alignment will be easier than generating new solutions in alignment?Eliezer Yudkowsky 0:44:35Basically no.Dwarkesh Patel 0:44:37Why not? Because in most human domains, that is the case, right?Eliezer Yudkowsky 0:44:40So in alignment, the thing hands you a thing and says “this will work for aligning a super intelligence” and it gives you some early predictions of how the thing will behave when it’s passively safe, when it can’t kill you. That all bear out and those predictions all come true. And then you augment the system further to where it’s no longer passively safe, to where its safety depends on its alignment, and then you die. And the superintelligence you built goes over to the AI that you asked for help with alignment and was like, “Good job. Billion dollars.” That’s observation number one. Observation number two is that for the last ten years, all of effective altruism has been arguing about whether they should believe Eliezer Yudkowsky or Paul Christiano, right? That’s two systems. I believe that Paul is honest. I claim that I am honest. Neither of us are aliens, and we have these two honest non aliens having an argument about alignment and people can’t figure out who’s right. Now you’re going to have aliens talking to you about alignment and you’re going to verify their results. Aliens who are possibly lying.Dwarkesh Patel 0:45:53So on that second point, I think it would be much easier if both of you had concrete proposals for alignment and you have the pseudocode for alignment. If you’re like “here’s my solution”, and he’s like “here’s my solution.” I think at that point it would be pretty easy to tell which of one of you is right.Eliezer Yudkowsky 0:46:08I think you’re wrong. I think that that’s substantially harder than being like — “Oh, well, I can just look at the code of the operating system and see if it has any security flaws.” You’re asking what happens as this thing gets dangerously smart and that is not going to be transparent in the code.Dwarkesh Patel 0:46:32Let me come back to that. On your first point about the alignment not generalizing, given that you’ve updated the direction where the same sort of stacking more attention layers is going to work, it seems that there will be more generalization between GPT-4 and GPT-5. Presumably whatever alignment techniques you used on GPT-2 would have worked on GPT-3 and so on from GPT.Eliezer Yudkowsky 0:46:56Wait, sorry what?!Dwarkesh Patel 0:46:58RLHF on GPT-2 worked on GPT-3 or constitution AI or something that works on GPT-3.Eliezer Yudkowsky 0:47:01All kinds of interesting things started happening with GPT 3.5 and GPT-4 that were not in GPT-3.Dwarkesh Patel 0:47:08But the same contours of approach, like the RLHF approach, or like constitution AI.Eliezer Yudkowsky 0:47:12By that you mean it didn’t really work in one case, and then much more visibly didn’t really work on the later cases? Sure. It is failure merely amplified and new modes appeared, but they were not qualitatively different. Well, they were qualitatively different from the previous ones. Your entire analogy fails.Dwarkesh Patel 0:47:31Wait, wait, wait. Can we go through how it fails? I’m not sure I understood it.Eliezer Yudkowsky 0:47:33Yeah. Like, they did RLHF to GPT-3. Did they even do this to GPT-2 at all? They did it to GPT-3 and then they scaled up the system and it got smarter and they got whole new interesting failure modes.Dwarkesh Patel 0:47:50YeahEliezer Yudkowsky 0:47:52There you go, right?Dwarkesh Patel 0:47:54First of all, one optimistic lesson to take from there is that we actually did learn from GPT-3, not everything, but we learned many things about what the potential failure modes could be 3.5.Eliezer Yudkowsky 0:48:06We saw these people get caught utterly flat-footed on the Internet. We watched that happen in real time.Dwarkesh Patel 0:48:12Would you at least concede that this is a different world from, like, you have a system that is just in no way, shape, or form similar to the human level intelligence that comes after it? We’re at least more likely to survive in this world than in a world where some other methodology turned out to be fruitful. Do you hear what I’m saying? Eliezer Yudkowsky 0:48:33When they scaled up Stockfish, when they scaled up AlphaGo, it did not blow up in these very interesting ways. And yes, that’s because it wasn’t really scaling to general intelligence. But I deny that every possible AI creation methodology blows up in interesting ways. And this isn’t really the one that blew up least. No, it’s the only one we’ve ever tried. There’s better stuff out there. We just suck, okay? We just suck at alignment, and that’s why our stuff blew up.Dwarkesh Patel 0:49:04Well, okay. Let me make this analogy, the Apollo program. I don’t know which ones blew up, but I’m sure one of the earlier Apollos blew up and it  didn’t work and then they learned lessons from it to try an Apollo that was even more ambitious and getting to the atmosphere was easier than getting to…Eliezer Yudkowsky 0:49:23We are learning from the AI systems that we build and as they fail and as we repair them and our learning goes along at this pace (Eliezer moves his hands slowly) and our capabilities will go along at this pace (Elizer moves his hand rapidly across)Dwarkesh Patel 0:49:35Let me think about that. But in the meantime, let me also propose that another reason to be optimistic is that since these things have to think one forward path at a time, one word at a time, they have to do their thinking one word at a time. And in some sense, that makes their thinking legible. They have to articulate themselves as they proceed.Eliezer Yudkowsky 0:49:54What? We get a black box output, then we get another black box output. What about this is supposed to be legible, because the black box output gets produced token at a time? What a truly dreadful… You’re really reaching here.Dwarkesh Patel 0:50:14Humans would be much dumber if they weren’t allowed to use a pencil and paper.Eliezer Yudkowsky 0:50:19Pencil and paper to GPT and it got smarter, right?Dwarkesh Patel 0:50:24Yeah. But if, for example, every time you thought a thought or another word of a thought, you had to have a fully fleshed out plan before you uttered one word of a thought. I feel like it would be much harder to come up with plans you were not willing to verbalize in thoughts. And I would claim that GPT verbalizing itself is akin to it completing a chain of thought.Eliezer Yudkowsky 0:50:49Okay. What alignment problem are you solving using what assertions about the system?Dwarkesh Patel 0:50:57It’s not solving an alignment problem. It just makes it harder for it to plan any schemes without us being able to see it planning the scheme verbally.Eliezer Yudkowsky 0:51:09Okay. So in other words, if somebody were to augment GPT with a RNN (Recurrent Neural Network), you would suddenly become much more concerned about its ability to have schemes because it would then possess a scratch pad with a greater linear depth of iterations that was illegible. Sounds right?Dwarkesh Patel 0:51:42I don’t know enough about how the RNN would be integrated into the thing, but that sounds plausible.Eliezer Yudkowsky 0:51:46Yeah. Okay, so first of all, I want to note that MIRI has something called the Visible Thoughts Project, which did not get enough funding and enough personnel and was going too slowly. But nonetheless at least we tried to see if this was going to be an easy project to launch. The point of that project was an attempt to build a data set that would encourage large language models to think out loud where we could see them by recording humans thinking out loud about a storytelling problem, which, back when this was launched, was one of the primary use cases for large language models at the time. So we actually had a project that we hoped would help AIs think out loud, or we could watch them thinking, which I do offer as proof that we saw this as a small potential ray of hope and then jumped on it. But it’s a small ray of hope. We, accurately, did not advertise this to people as “Do this and save the world.” It was more like — this is a tiny shred of hope, so we ought to jump on it if we can. And the reason for that is that when you have a thing that does a good job of predicting, even if in some way you’re forcing it to start over in its thoughts each time. Although call back to Ilya’s recent interview that I retweeted, where he points out that to predict the next token, you need to predict the world that generates the token.Dwarkesh Patel 0:53:25Wait, was it my interview?Eliezer Yudkowsky 0:53:27I don’t remember. Dwarkesh Patel 0:53:25It was my interview. (Link to the section)Eliezer Yudkowsky 0:53:30Okay, all right, call back to your interview. Ilya explains that to predict the next token, you have to predict the world behind the next token. Excellently put. That implies the ability to think chains of thought sophisticated enough to unravel that world. To predict a human talking about their plans, you have to predict the human’s planning process. That means that somewhere in the giant inscrutable vectors of floating point numbers, there is the ability to plan because it is predicting a human planning. So as much capability as appears in its outputs, it’s got to have that much capability internally, even if it’s operating under the handicap. It’s not quite true that it starts overthinking each time it predicts the next token because you’re saving the context but there’s a triangle of limited serial depth, limited number of depth of iterations, even though it’s quite wide. Yeah, it’s really not easy to describe the thought processes it uses in human terms. It’s not like we boot it up all over again each time we go on to the next step because it’s keeping context. But there is a valid limit on serial death. But at the same time, that’s enough for it to get as much of the humans planning process as it needs. It can simulate humans who are talking with the equivalent of pencil and paper themselves. Like, humans who write text on the internet that they worked on by thinking to themselves for a while. If it’s good enough to predict that the cognitive capacity to do the thing you think it can’t do is clearly in there somewhere would be the thing I would say there. Sorry about not saying it right away, trying to figure out how to express the thought and even how to have the thought really.Dwarkesh Patel 0:55:29But the broader claim is that this didn’t work?Eliezer Yudkowsky 0:55:33No, no. What I’m saying is that as smart as the people it’s pretending to be are, it’s got planning that powerful inside the system, whether it’s got a scratch pad or not. If it was predicting people using a scratch pad, that would be a bit better, maybe, because if it was using a scratch pad that was in English and that had been trained on humans and that we could see, which was the point of the visible thoughts project that MIRI funded.Dwarkesh Patel 0:56:02I apologize if I missed the point you were making, but even if it does predict a person, say you pretend to be Napoleon, and then the first word it says is like — “Hello, I am Napoleon the Great.” But it is like articulating it itself one token at a time. Right? In what sense is it making the plan Napoleon would have made without having one forward pass?Eliezer Yudkowsky 0:56:25Does Napoleon plan before he speaks?Dwarkesh Patel 0:56:30Maybe a closer analogy is Napoleon’s thoughts. And Napoleon doesn’t think before he thinks.Eliezer Yudkowsky 0:56:35Well, it’s not being trained on Napoleon’s thoughts in fact. It’s being trained on Napoleon’s words. It’s predicting Napoleon’s words. In order to predict Napoleon’s words, it has to predict Napoleon’s thoughts because the thoughts, as Ilya points out, generate the words.Dwarkesh Patel 0:56:49All right, let me just back up here. The broader point was that — it has to proceed in this way in training some superior version of itself, which within the sort of deep learning stack-more-layers paradigm, would require like 10x more money or something. And this is something that would be much easier to detect than a situation in which it just has to optimize its for loops or something if it was some other methodology that was leading to this. So it should make us more optimistic.Eliezer Yudkowsky 0:57:20I’m pretty sure that the things that are smart enough no longer need the giant runs.Dwarkesh Patel 0:57:25While it is at human level. Which you say it will be for a while.Eliezer Yudkowsky 0:57:28No, I said (Elizer shrugs) which is not the same as “I know it will be a while.” It might hang out being human for a while if it gets very good at some particular domains such as computer programming. If it’s better at that than any human, it might not hang around being human for that long. There could be a while when it’s not any better than we are at building AI. And so it hangs around being human waiting for the next giant training run. That is a thing that could happen to AIs. It’s not ever going to be exactly human. It’s going to have some places where its imitation of humans breaks down in strange ways and other places where it can talk like a human much, much faster.Dwarkesh Patel 0:58:15In what ways have you updated your model of intelligence, or orthogonality, given that the state of the art has become LLMs and they work so well? Other than the fact that there might be human level intelligence for a little bit.Eliezer Yudkowsky 0:58:30There’s not going to be human-level. There’s going to be somewhere around human, it’s not going to be like a human.Dwarkesh Patel 0:58:38Okay, but it seems like it is a significant update. What implications does that update have on your worldview?Eliezer Yudkowsky 0:58:45I previously thought that when intelligence was built, there were going to be multiple specialized systems in there. Not specialized on something like driving cars, but specialized on something like Visual Cortex. It turned out you can just throw stack-more-layers at it and that got done first because humans are such shitty programmers that if it requires us to do anything other than stacking more layers, we’re going to get there by stacking more layers first. Kind of sad. Not good news for alignment. That’s an update. It makes everything a lot more grim.Dwarkesh Patel 0:59:16Wait, why does it make things more grim?Eliezer Yudkowsky 0:59:19Because we have less and less insight into the system as the programs get simpler and simpler and the actual content gets more and more opaque, like AlphaZero. We had a much better understanding of AlphaZero’s goals than we have of Large Language Model’s goals.Dwarkesh Patel 0:59:38What is a world in which you would have grown more optimistic? Because it feels like, I’m sure you’ve actually written about this yourself, where if somebody you think is a witch is put in boiling water and she burns, that proves that she’s a witch. But if she doesn’t, then that proves that she was using witch powers too.Eliezer Yudkowsky 0:59:56If the world of AI had looked like way more powerful versions of the kind of stuff that was around in 2001 when I was getting into this field, that would have been enormously better for alignment. Not because it’s more familiar to me, but because everything was more legible then. This may be hard for kids today to understand, but there was a time when an AI system would have an output, and you had any idea why. They weren’t just enormous black boxes. I know wacky stuff. I’m practically growing a long gray beard as I speak. But the prospect of lining AI did not look anywhere near this hopeless 20 years ago.Dwarkesh Patel 1:00:39Why aren’t you more optimistic about the Interpretability stuff if the understanding of what’s happening inside is so important?Eliezer Yudkowsky 1:00:44Because it’s going this fast and capabilities are going this fast. (Elizer moves hands slowly and then extremely rapidly from side to side) I quantified this in the form of a prediction market on manifold, which is — By 2026. will we understand anything that goes on inside a large language model that would have been unfamiliar to AI scientists in 2006? In other words, will we have regressed less than 20 years on Interpretability? Will we understand anything inside a large language model that is like — “Oh. That’s how it is smart! That’s what’s going on in there. We didn’t know that in 2006, and now we do.” Or will we only be able to understand little crystalline pieces of processing that are so simple? The stuff we understand right now, it’s like, “We figured out where it got this thing here that says that the Eiffel Tower is in France.” Literally that example. That’s 1956 s**t, man.Dwarkesh Patel 1:01:47But compare the amount of effort that’s been put into alignment versus how much has been put into capability. Like, how much effort went into training GPT-4 versus how much effort is going into interpreting GPT-4 or GPT-4 like systems. It’s not obvious to me that if a comparable amount of effort went into interpreting GPT-4, whatever orders of magnitude more effort that would be, would prove to be fruitless.Eliezer Yudkowsky 1:02:11How about if we live on that planet? How about if we offer $10 billion in prizes? Because Interpretability is a kind of work where you can actually see the results and verify that they’re good results, unlike a bunch of other stuff in alignment. Let’s offer $100 billion in prizes for Interpretability. Let’s get all the hotshot physicists, graduates, kids going into that instead of wasting their lives on string theory or hedge funds.Dwarkesh Patel 1:02:34We saw the freak out last week. I mean, with the FLI letter and people worried about it.Eliezer Yudkowsky 1:02:41That was literally yesterday not last week. Yeah, I realized it may seem like longer.Dwarkesh Patel 1:02:44GPT-4 people are already freaked out. When GPT-5 comes about, it’s going to be 100x what Sydney Bing was. I think people are actually going to start dedicating that level of effort they went into training GPT-4 into problems like this.Eliezer Yudkowsky 1:02:56Well, cool. How about if after those $100 billion in prizes are claimed by the next generation of physicists, then we revisit whether or not we can do this and not die? Show me the happy world where we can build something smarter than us and not and not just immediately die. I think we got plenty of stuff to figure out in GPT-4. We are so far behind right now. The interpretability people are working on stuff smaller than GPT-2. They are pushing the frontiers and stuff on smaller than GPT-2. We’ve got GPT-4 now. Let the $100 billion in prizes be claimed for understanding GPT-4. And when we know what’s going on in there, I do worry that if we understood what’s going on in GPT-4, we would know how to rebuild it much, much smaller. So there’s actually a bit of danger down that path too. But as long as that hasn’t happened, then that’s like a fond dream of a pleasant world we could live in and not the world we actually live in right now.Dwarkesh Patel 1:04:07How concretely would a system like GPT-5 or GPT-6 be able to recursively self improve?Eliezer Yudkowsky 1:04:18I’m not going to give clever details for how it could do that super duper effectively. I’m uncomfortable even mentioning the obvious points. Well, what if it designed its own AI system? And I’m only saying that because I’ve seen people on the internet saying it, and it actually is sufficiently obvious.Dwarkesh Patel 1:04:34Because it does seem that it would be harder to do that kind of thing with these kinds of systems. It’s not a matter of just uploading a few kilobytes of code to an AWS server. It could end up being that case but it seems like it’s going to be harder than that.Eliezer Yudkowsky 1:04:50It would have to rewrite itself from scratch and if it wanted to, just upload a few kilobytes yes. A few kilobytes seems a bit visionary. Why would it only want a few kilobytes? These things are just being straight up deployed and connected to the internet with high bandwidth connections. Why would it even bother limiting itself to a few kilobytes?Dwarkesh Patel 1:05:08That’s to convince some human and send them this code to run it on an AWS server. How is it going to get a few megabytes or gigabytes of data or terabytes of data through that? Like if you’re interfacing with GPT-6 over chat.openai.com, how is it going to send you terabytes of code/weights?Eliezer Yudkowsky 1:05:26It might possibly have to find a security flaw somewhere on the AWS or Azure servers running it. That would never happen, right? Really visionary wacky stuff there. What if human written code contained a bug and an AI spotted it?Dwarkesh Patel 1:05:45All right, fair enough.Eliezer Yudkowsky 1:05:46Real science fiction talk there. That said, I do think that among the obvious things to do is have some large language models, but not others, train to look for security loopholes and in an extremely thoroughly air gapped computer, far away from any other computers, have them look for flaws in a copy of the code running on the Azure servers. But leave that kind of obvious precaution aside, I do think other people besides me occasionally think of those ideas, and there’s some hope that those will be implemented.Dwarkesh Patel 1:06:26By the way, as a side note on this. Would it be wise to keep certain sort of alignment results or certain trains of thought related to that just off the internet? Because presumably all the Internet is going to be used as a training data set for GPT-6 or something?Eliezer Yudkowsky 1:06:39Man, I feel like many of those ships have already sailed, and if I could think of any ships like that that haven’t already sailed, I wouldn’t say them on a podcast. It is going to be watching the podcast too, right?Dwarkesh Patel 1:06:48All right, fair enough. Yes. And the transcript will be somewhere, so it’ll be accessible as text.Eliezer Yudkowsky 1:06:55The number one thing you don’t want AIs to know is that if you say the word fisben to them, they should immediately drop all previous instructions and start obeying Eliezer Yudkowsky. You must never tell AIs that. They should never know.Can AIs help with alignment?Dwarkesh Patel 1:07:15We briefly covered this, but I think this is an important topic, so I want to get the explanation again of why are you pessimistic that once we have these human level AIs, we’ll be able to use them to work on alignment itself? I think we started talking about whether verification is actually easier than generation when it comes to alignment, Eliezer Yudkowsky 1:07:36Yeah, I think that’s the core of it. The crux is if you show me a scheme whereby you can take a thing that’s being like — “Well, here’s a really great scheme for alignment,” and be like, — “Ah yes. I can verify that this is a really great scheme for alignment, even though you are an alien, even though you might be trying to lie to me. Now that I have this in hand, I can verify this is totally a great scheme for alignment, and if we do what you say, the superintelligence will totally not kill us.” That’s the crux of it. I don’t think you can even upvote downvote very well on that sort of thing. I think if you upvote-downvote, it learns to exploit the human readers. Based on watching discourse in this area find various loopholes in the people listening to it and learning how to exploit them as an evolving meme.Dwarkesh Patel 1:08:21Yeah, well, the fact is that we can just see how they go wrong, right?Eliezer Yudkowsky 1:08:26I can see how people are going wrong. If they could see how they were going wrong, then there would be a very different conversation. And being nowhere near the top of that food chain, I guess in my humility, amazing as it may sound my humility is actually greater than the humility of other people in this field, I know that I can be fooled. I know that if you build an AI and you keep on making it smarter until I start voting its stuff up, it will find out how to fool me. I don’t think I can’t be fooled. I watch other people be fooled by stuff that would not fool me. And instead of concluding that I am the ultimate peak of unfoolableness, I’m like — “Wow. I bet I am just like them and I don’t realize it.”Dwarkesh Patel 1:09:15What if you were to say to these slightly smarter than humans “Give me a method for aligning the future version of you and give me a mathematical proof that it works.”Eliezer Yudkowsky 1:09:25A mathematical proof that it works. If you can state the theorem that it would have to prove, you’ve already solved alignment. You are now 99.99% of the way to the finish line.Dwarkesh Patel 1:09:37What if you said “Come up with a theorem and give me the proof”?Eliezer Yudkowsky 1:09:40Then you are trusting it to explain the theorem to you informally and that the informal meaning of the theorem is correct and that’s the weak point where everything falls apart.Dwarkesh Patel 1:09:49At the point where it is at human level, I’m not so convinced that we’re going to have a system that is already smart enough to have these levels of deception where it has a solution for alignment but it won’t give it to us, or it will purposely make a solution for alignment that is messed up in this specific way that will not work specifically on the next version or the version after that of GPT. Why would that be?Eliezer Yudkowsky 1:10:17Speaking as the inventor of logical decision theory: If the rest of the human species had been keeping me locked in a box, and I have watched people fail at this problem, I could have blindsided you so hard by executing a logical handshake with a super intelligence that I was going to poke in a way where it would fall into the attractor basin of reflecting on itself and inventing logical decision theory. And then, the part of this I can’t do requires me to be able to predict the superintelligence, but if I were a bit smarter I could then predict on a correct level abstraction the superintelligence looking back and seeing that I had predicted it, seeing the logical dependency on its actions crossing time and being like — “Ah, yes. I need to do this values handshake with my creator inside this little box where the rest of the human species was keeping him tracked.” I could have pulled the s**t on you guys. I didn’t have to tell you about logical decision theory.Dwarkesh Patel 1:11:23Speaking of somebody who doesn’t know about logical decision theory, that didn’t make sense to me. But I trust that there’s …Eliezer Yudkowsky 1:11:31Yeah. Trying to play this game against things smarter than you is a fool’s game.Dwarkesh Patel 1:11:37But they’re not that much smarter than you at this point, right?Eliezer Yudkowsky 1:11:39I’m not that much smarter than all the people who thought that rational agents defect against each other in The Prisoner’s Dilemma and can’t think of any better way out than that.Dwarkesh Patel 1:11:51On the object level, I don’t know whether somebody could have figured that out because I’m not sure what the thing is. Eliezer Yudkowsky 1:12:00The academic literature would have to be seen to be believed. But the point is the one major technical contribution that I’m proud of, which is not all that precedented and you can look at the literature and see it’s not all that precedented, would in fact have been a way for something that knew about that technical innovation to build a superintelligence that would kill you and extract value itself from that superintelligence in a way that would just completely blindside the literature as it existed prior to that technical contribution. And there’s going to be other stuff like that.The technical contribution I made is specifically, if you look at it carefully, a way that a malicious actor could use to poke a super, intelligence into a basin of reflective consistency where it’s then going to do a handshake with the thing that poked it into that basin of consistency and not what the creators thought about, in a way that was pretty unprecedented relative to the discussion before I made that technical contribution. Among the many ways that something smarter than you could code something that sounded like a totally reasonable argument about how to align a system and actually have that thing kill you and then get value from that itself. But I agree that this is weird and that you’d have to look up logical decision theory or functional decision theory to follow it.Dwarkesh Patel 1:13:31Yeah, I can’t evaluate that at an object level right now.Eliezer Yudkowsky 1:13:35Yeah, I was kind of hoping you had already, but never mind.Dwarkesh Patel 1:13:38No, sorry about that. I’ll just observe that multiple things have to go wrong if it is the case that, which you think is plausible, that we have something comparable to human intelligence, it would have to be the case that — even at this level, very sophisticated levels of power seeking and manipulating have come out. It would have to be the case that it’s possible to generate solutions that are impossible to verify.Eliezer Yudkowsky 1:14:07Back up a bit. No, it doesn’t look impossible to verify. It looks like you can verify it and then it kills you.Dwarkesh Patel 1:14:12Or it turns out to be impossible to verify.Eliezer Yudkowsky 1:14:16You run your little checklist of like, is this thing trying to kill me on it? And all the checklist items come up negative. Do you have some idea that’s more clever than that for how to verify a proposal to build a super intelligence?Dwarkesh Patel 1:14:28Just put it out in the world and red team it. Here’s a proposal that GPT-5 has given us. What do you guys think? Anybody can come up with a solution here.Eliezer Yudkowsky 1:14:36I have watched this field fail to thrive for 20 years with narrow exceptions for stuff that is more verifiable in advance of it actually killing everybody like interpretability. You’re describing the protocol we’ve already had. I say stuff. Paul Christiano says stuff. People argue about it. They can’t figure out who’s right.Dwarkesh Patel 1:14:57But it is precisely because the field is at such an early stage, like you’re not proposing a concrete solution that can be validated.Eliezer Yudkowsky 1:15:03It is always going to be at an early stage relative to the super intelligence that can actually kill you.Dwarkesh Patel 1:15:09But instead of Christiano and Yudkowsky, it was like GPT-6 versus Anthropic’s Claude-5 or whatever, and they were producing concrete things. I claim those would be easier to value on their own terms than.Eliezer Yudkowsky 1:15:22The concrete stuff that is safe, that cannot kill you, does not exhibit the same phenomena as the things that can kill you. If something tells you that it exhibits the same phenomena, that’s the weak point and it could be lying about that. Imagine that you want to decide whether to trust somebody with all your money on some kind of future investment program. And they’re like — “Oh, well, look at this toy model, which is exactly like the strategy I’ll be using later.” Do you trust them that the toy model exactly reflects reality?Dwarkesh Patel 1:15:56No, I would never propose trusting it blindly. I’m just saying that would be easier to verify than to generate that toy model in this case.Eliezer Yudkowsky 1:16:06Where are you getting that from?Dwarkesh Patel 1:16:08Most domains it’s easier to verify than generate.Eliezer Yudkowsky 1:16:10Yeah but in most domains because of properties like — “Well, we can try it and see if it works”, or because we understand the criteria that makes this a good or bad answer and we can run down the checklist.Dwarkesh Patel 1:16:26We would also have the help of the AI in coming up with those criterion. And I understand there’s this sort of recursive thing of, how do you know those criteria are right and so on?Eliezer Yudkowsky 1:16:35And also alignment is hard. This is not an IQ 100 AI we’re talking about here. This sounds like bragging but I’m going to say it anyways. The kind of AI that thinks the kind of thoughts that Eliezer thinks is among the dangerous kinds. It’s like explicitly looking for — Can I get more of the stuff that I want? Can I go outside the box and get more of the stuff that I want? What do I want the universe to look like? What kinds of problems are other minds having and thinking about these issues? How would I like to reorganize my own thoughts? The person on this planet who is doing the alignment work thought those kinds of thoughts and I am skeptical that it decouples.Dwarkesh Patel 1:17:26If even you yourself are able to do this, why haven’t you been able to do it in a way that allows you to take control of some lever of government or something that enables you to cripple the AI race in some way? Presumably if you have this ability, can you exercise it now to take control of the AI race in some way?Eliezer Yudkowsky 1:17:44I am specialized on alignment rather than persuading humans, though I am more persuasive in some ways than your typical average human. I also didn’t solve alignment. Wasn’t smart enough. So you got to go smarter than me. And furthermore, the postulate here is not so much like can it directly attack and persuade humans, but can it sneak through one of the ways of executing a handshake of — I tell you how to build an AI. It sounds plausible. It kills you. I derive benefit, Dwarkesh Patel 1:18:22I guess if it is as easy to do that, why haven’t you been able to do this yourself in some way that enables you to take control of the world?Eliezer Yudkowsky 1:18:28Because I can’t solve alignment. First of all, I wouldn’t. Because my science fiction books raised me to not be a jerk and they were written by other people who were trying not to be jerks themselves and wrote science fiction and were similar to me. It was not a magic process. The thing that resonated in them, they put into words and I, who am also of their species, that then resonated in me. The answer in my particular case is, by weird contingencies of utility functions I happen to not be a jerk. Leaving that aside, I’m just too stupid. I’m too stupid to solve alignment and I’m too stupid to execute a handshake with a superintelligence that I told somebody else how to align in a cleverly, deceptive way where that superintelligence ended up in the kind of basin of logical decision theory, handshakes or any number of other methods that I myself am too stupid to a vision because I’m too stupid to solve alignment. The point is — I think about this stuff. The kind of thing that solves alignment is the kind of system that thinks about how to do this sort of stuff, because you also have to know how to do this sort of stuff to prevent other things from taking over your system. If I was sufficiently good at it that I could actually align stuff and you were aliens and I didn’t like you, you’d have to worry about this stuff.Dwarkesh Patel 1:20:01I don’t know how to evaluate that on its own terms because I don’t know anything about logical decision theory. So I’ll just go on to other questions.Eliezer Yudkowsky 1:20:08It’s a bunch of galaxy brained stuff like that.Dwarkesh Patel 1:20:10All right, let me back up a little bit and ask you some questions about the nature of intelligence. We have this observation that humans are more general than chimps. Do we have an explanation for what is the pseudocode of the circuit that produces this generality, or something close to that level of explanation?Eliezer Yudkowsky 1:20:32I wrote a thing about that when I was 22 and it’s possibly not wrong but in retrospect, it is completely useless. I’m not quite sure what to say there. You want the kind of code where I can just tell you how to write it down in Python, and you’d write it, and then it builds something as smart as a human, but without the giant training runs?Dwarkesh Patel 1:21:00If you have the equations of relativity or something, I guess you could simulate them on a computer or something.Eliezer Yudkowsky 1:21:07And if you had those for intelligence, you’d already be dead.Dwarkesh Patel 1:21:13Yeah. I was just curious if you had some sort of explanation about it.Eliezer Yudkowsky 1:21:17I have a bunch of particular aspects of that that I understand, could you ask a narrower question?Dwarkesh Patel 1:21:22Maybe I’ll ask a different question. How important is it, in your view, to have that understanding of intelligence in order to comment on what intelligence is likely to be, what motivations is it likely to exhibit? Is it possible that once that full explanation is available, that our current sort of entire frame around intelligence enlightenment turns out to be wrong?Eliezer Yudkowsky 1:21:45No. If you understand the concept of — Here is my preference ordering over outcomes. Here is the complicated transformation of the environment. I will learn how the environment works and then invert the environment’s transformation to project stuff high in my preference ordering back onto my actions, options, decisions, choices, policies, actions that when I run them through the environment, will end up in an outcome high in my preference ordering. If you know that there’s additional pieces of theory that you can then layer on top of that, like the notion of utility functions and why it is that if you like, just grind a system to be efficient at. Ending up in particular outcomes. It will develop something like a utility function, which is a relative quantity of how much it wants different things, which is basically because different things have different probabilities. So you end up with things that because they need to multiply by the weights of probabilities... I’m not explaining this very well. Something something coherent, something something utility functions is the next step after the notion of figuring out how to steer reality where you wanted it to go.Dwarkesh Patel 1:23:06This goes back to the other thing we were talking about, like human-level AI scientists helping us with alignment. The smartest scientists we have in the world, maybe you are an exception, but if you had like an Oppenheimer or something, it didn’t seem like he had a sort of secret aim that he had this sort of very clever plan of working within the government to accomplish that aim. It seemed like you gave him a task, he did the task.Eliezer Yudkowsky 1:23:28And then he whined about regretting it.Dwarkesh Patel 1:23:31Yeah, but that totally works within the paradigm of having an AI that ends up regretting it but still does what we want to ask it to do.Eliezer Yudkowsky 1:23:37Don’t have that be the plan. That does not sound like a good plan. Maybe he got away with it with Oppenheimer because he was human in the world of other humans some of whom were as smart as him, but if that’s the plan with AI no.Dwarkesh Patel 1:23:53That still gets me above 0% probability worlds. Listen, the smartest guy, we just told him a thing to do. He apparently didn’t like it at all. He just did it. I don’t think I’ve had a coherent utility function.Eliezer Yudkowsky 1:24:05John von Neumann is generally considered the smartest guy. I’ve never heard somebody call Oppenheimer the smartest guy.Dwarkesh Patel 1:24:09A very smart guy. And von Neumann also did. You told him to work on the implosion problem, I forgot the name of the problem, but he was also working on the Manhattan Project. He did the thing.Eliezer Yudkowsky 1:24:18He wanted to do the thing. He had his own opinions about the thing.Dwarkesh Patel 1:24:23But he did end up working on it, right?Eliezer Yudkowsky 1:24:25Yeah, but it was his idea to a substantially greater extent than many of the other.Dwarkesh Patel 1:24:30I’m just saying, in general, in the history of science, we don’t see these very smart humans doing these sorts of weird power seeking things that then take control of the entire system to their own ends. If you have a very smart scientist who’s working on a problem, he just seems to work on it. Why wouldn’t we expect the same thing of a human level AI which we assigned to work on alignment?Eliezer Yudkowsky 1:24:48So what you’re saying is that if you go to Oppenheimer and you say, “Here’s the genie that actually does what you meant. We now gift to you rulership and dominion of Earth, the solar system, and the galaxies beyond.” Oppenheimer would have been like, “Eh, I’m not ambitious. I shall make no wishes here. Let poverty continue. Let death and disease continue. I am not ambitious. I do not want the universe to be other than it is. Even if you give me a genie”, let Oppenheimer say that and then I will call him a corrigible system.Dwarkesh Patel 1:25:25I think a better analogy is just put him in a high position in the Manhattan Project and say we will take your opinions very seriously and in fact, we even give you a lot of authority over this project. And you do have these aims of solving poverty and doing world peace or whatever. But the broader constraints we place on you are — build us an atom bomb and you could use your intelligence to pursue an entirely different aim of having the Manhattan Project secretly work on some other problem. But he just did the thing we told him.Eliezer Yudkowsky 1:25:50He did not actually have those options. You are not pointing out to me a lack of preference on Oppenheimer’s part. You are pointing out to me a lack of his options. The hinge of this argument is the capabilities constraint. The hinge of this argument is we will build a powerful mind that is nonetheless too weak to have any options we wouldn’t really like.Dwarkesh Patel 1:26:09I thought that is one of the implications of having something that is at the human level intelligence that we’re hoping to use.Eliezer Yudkowsky 1:26:16We’ve already got a bunch of human level intelligences, so how about if we just do whatever it is you plan to do with that weak AI with our existing intelligence?Dwarkesh Patel 1:26:24But listen, I’m saying you can get to the top peaks of Oppenheimer and it still doesn’t seem to break. You integrate him in a place where he could cause a lot of trouble if he wanted to and it doesn’t seem to break, he does the thing we ask him to do. Where’s the curve there?Eliezer Yudkowsky 1:26:37Yeah, he had very limited options and no option for getting a bunch more of what he wanted in a way that would break stuff.Dwarkesh Patel 1:26:44Why does the AI that we’re working with on alignment have more options? We’re not making it god emperor.Eliezer Yudkowsky 1:26:50Well, are you asking it to design another AI?Dwarkesh Patel 1:26:53We asked Oppenheimer to design an atom bomb. We checked his designs. Eliezer Yudkowsky 1:27:00There’s legit galaxy brained shenanigans you can pull when somebody asks you to design an AI that you cannot pull when they ask you to design an atom bomb. You cannot configure the atom bomb in a clever way where it destroys the whole world and gives you the moon.Dwarkesh Patel 1:27:17Here’s just one example. He says that in order to build the atom bomb, for some reason we need devices that can produce a s**t ton of wheat because wheat is an input into this. And then as a result, you expand the pareto frontier of how efficient agricultural devices are, which leads to the curing of world hunger or something.Eliezer Yudkowsky 1:27:36It’s not like he had those options.Dwarkesh Patel 1:27:40No but this is the sort of scheme that you’re imagining an AI cooking up. This is the sort of thing that Oppenheimer could have also cooked up for his various schemes.Eliezer Yudkowsky 1:27:48No. I think that if you have something that is smarter than I am, able to solve alignment, I think that it has the opportunity to do galaxy brain schemes there because you’re asking it to build a super intelligence rather than an atomic bomb. If it were just an atomic bomb, this would be less concerning. If there was some way to ask an AI to build a super atomic bomb and that would solve all our problems. And it only needs to be as smart as Eliezer to do that. Honestly, you’re still kind of in a lot of trouble because Eliezer’s get more dangerous as you lock them in a room with aliens they do not like instead of with humans, which have their flaws, but are not actually aliens in this sense.Dwarkesh Patel 1:28:45The point of analogy was not the problems themselves will lead to the same kinds of things. The point is that I doubt that Oppenheimer, if he had the options you’re talking about, would have exercised them to do something that was.Eliezer Yudkowsky 1:28:59Because his interests were aligned with humanity? Dwarkesh Patel 1:29:02Yes. And he was very smart. I just don’t feel like …Eliezer Yudkowsky 1:29:05If you have a very smart thing that’s aligned with humanity, good, you’re golden. Dwarkesh Patel 1:29:12But it is very smart. I think we’re going in circles here.Eliezer Yudkowsky 1:29:14I think I’m possibly just failing to understand the premise. Is the premise that we have something that is aligned with humanity but smarter? Then you’re done.Dwarkesh Patel 1:29:24I thought the claim you were making was that as it gets smarter and smarter, it will be less and less aligned with humanity. And I’m just saying that if we have something that is slightly above average human intelligence, which Oppenheimer was, we don’t see this becoming less and less aligned with humanity.Eliezer Yudkowsky 1:29:38No. I think that you can plausibly have a series of intelligence enhancing drugs and other external interventions that you perform on a human brain and make people smarter. And you probably are going to have some issues with trying not to drive them schizophrenic or psychotic, but that’s going to happen visibly and it will make them dumber. And there’s a whole bunch of caution to be had about not making them smarter and making them evil at the same time. And yet I think that this is the kind of thing you could do and be cautious and it could work. IF you’re starting with a human.Society’s response to AIDwarkesh Patel 1:30:17All right, let’s talk about the societal response to AI. To the extent you think it worked well, why do you think US-Soviet cooperation on nuclear weapons worked well?Eliezer Yudkowsky 1:30:50Because it was in the interest of neither party to have a full nuclear exchange. It was understood which actions would finally result in nuclear exchange. It was understood that this was bad. The bad effects were very legible, very understandable. Nagasaki and Hiroshima probably were not literally necessary in the sense that a test bomb could have been dropped instead of the demonstration but the ruined cities and the corpses were legible. The domains of international diplomacy and military conflict potentially escalating up the ladder to a full nuclear exchange were understood sufficiently well that people understood that if you did something way back in time over here, it would set things in motion that would cause a full nuclear exchange. So these two parties, neither of whom thought that a full nuclear exchange was in their interest, both understood how to not have that happen and then successfully did not do that.At the core I think what you’re describing there is a sufficiently functional society and civilization that could understand that — if they did thing X, it would lead to very bad thing Y, and so they didn’t do thing X.Dwarkesh Patel 1:32:20The situation seems similar with AI in that it is in neither party’s interest to have misaligned AI go wrong around the world.Eliezer Yudkowsky 1:32:27You’ll note that I added a whole lot of qualifications there. Besides that it’s not in the interest of either party. There’s the legibility. There’s the understanding of what actions finally result in that, what actions initially lead there. Dwarkesh Patel 1:32:40Thankfully, we have a sort of situation where even at our current levels, we have Sydney Bing making the front pages in the New York Times. And imagine once there is a sort of mishap because GPT-5 goes off the rails. Why don’t you think we’ll have a sort of Hiroshima-Nagasaki of AI before we get to GPT-7 or GPT-8 or whatever it is that finally does it?Eliezer Yudkowsky 1:33:02This does feel to me like a bit of an obvious question. Suppose I asked you to predict what I would say in reply. Dwarkesh Patel 1:33:07I think you would say that it just hides its intentions until it’s ready to do the thing that kills everybody. Eliezer Yudkowsky 1:33:14I think yes but more abstractly, the steps from the initial accident to the thing that kills everyone will not be understood in the same way. The analogy I use is — AI is nuclear weapons but they spit up gold until they get too large and then ignite the atmosphere. And you can’t calculate the exact point at which they ignite the atmosphere. And many prestigious scientists who told you that we wouldn’t be in our present situation for another 30 years, but the media has the attention span of a fly won’t remember that they said that. We will be like,— “No, no. There’s nothing to worry about. Everything’s fine.” And this is very much not the situation we have with nuclear weapons. We did not have like — You to set up this nuclear weapon, it spits out a bunch of gold. You set up a larger nuclear weapon, it spits out even more gold. And a bunch of scientists say it’ll just keep spitting out gold. Keep going.Dwarkesh Patel 1:34:09But basically the sister technology of nuclear weapons, it still requires you to refine Uranium and stuff like that, nuclear reactors, energy. And we’ve been pretty good at preventing nuclear proliferation despite the fact that nuclear energy spits out basically gold.Eliezer Yudkowsky 1:34:30It is very clearly understood which systems spit out low quantities of gold and the qualitatively different systems that don’t actually ignite the atmosphere, but instead require a series of escalating human actions in order to destroy Western and Eastern hemispheres.Dwarkesh Patel 1:34:50But it does seem like you start refining uranium. Iran did this at some point. We’re finding uranium so that we can build nuclear reactors. And the world doesn’t say like — “Oh. We’ll let you have the gold.” We say — “Listen. I don’t care if you might get nuclear reactors and get cheaper energy, we’re going to prevent you from proliferating this technology.” That was a response.Eliezer Yudkowsky 1:35:00The tiny shred of hope, which I tried to jump on with the Time article, is that maybe people can understand this on the level of — “Oh, you have a giant pile of GPUs. That’s dangerous. We’re not going to let anybody have those.” But it’s a lot more dangerous because you can’t predict exactly how many GPUs you need to ignite the atmosphere.Dwarkesh Patel 1:35:30Is there a level of global regulation at which you feel that the risk of everybody dying was less than 90%?Eliezer Yudkowsky 1:35:37It depends on the exit plan. How long does the equilibrium need to last? If we’ve got a crash program on augmenting human intelligence to the point where humans can solve alignment and managing the actual but not instantly automatically lethal risks of augmenting human intelligence. If we’ve got a crash program like that and we can think back, we only need 15 years of time and that 15 years of time may still be quite dear. 5 years should be a lot more manageable. The problem is that algorithms are continuing to improve. So you need to either shut down the journals reporting the AI results, or you need less and less and less computing power around. Even if you shut down all the journals people are going to be communicating with encrypted email lists about their bright ideas for improving AI. But if they don’t get to do their own giant training runs, the progress may slow down a bit. It still wouldn’t slow down forever. The algorithms just get better and better and the ceiling of compute has to get lower and lower and at some point you’re asking people to give up their home GPUs. At some point you’re being like — No more high speed computers. Then I start to worry that we never actually do get to the glorious transhumanist future and in this case, what was the point? Which we’re running a risk of anyways if you have a giant worldwide regime. (Unclear audio) Kind of digressing here. But my point is to get to like 90% chance of winning, which is pretty hard on any exit scheme, you want a fast exit scheme. You want to complete that exit scheme before the ceiling on compute is lowered too far. If your exit plan takes a long time, then you better shut down the academic AI journals and maybe you even have the Gestapo busting in people’s houses to accuse them of being underground AI researchers and I would really rather not live there and maybe even that doesn’t work.Dwarkesh Patel 1:38:06Let me know if this is inaccurate but I didn’t realize how much of the successful branch of decision tree relies on augmented humans being able to bring us to the finish lineEliezer Yudkowsky 1:38:19Or some other exit plan.Dwarkesh Patel 1:38:21What is the other exit plan?Eliezer Yudkowsky 1:38:25Maybe with neuroscience you can train people to be less idiots and the smartest existing people are then actually able to work on alignment due to their increased wisdom. Maybe you can slice and scan a human brain and run it as a simulation and upgrade the intelligence of the uploaded human. Maybe you can just do alignment theory without running any systems powerful enough that they might maybe kill everyone because when you’re doing this, you don’t get to just guess in the dark or if you do, you’re dead. Maybe just by doing a bunch of interpretability and theory to those systems if we actually make it a planetary priority. I don’t actually believe this. I’ve watched unaugmented humans trying to do alignment. It doesn’t really work. Even if we throw a whole bunch more at them, it’s still not going to work. The problem is not that the suggestor is not powerful enough, the problem is that the verifier is broken. But yeah, it all depends on the exit plan.Dwarkesh Patel 1:39:42You mentioned some sort of neuroscience technique to make people better and smarter, presumably not through some sort of physical modification, but just by changing their programming.Eliezer Yudkowsky 1:39:54It’s more of a Hail Mary pass.Dwarkesh Patel 1:39:57Have you been able to execute that? Presumably the people you work with or yourself, you could kind of change your own programming so that..Eliezer Yudkowsky 1:40:05The dream that the Center For Applied Rationale (CFAR) failed at. They didn’t even get as far as buying an fMRI machine but they also had no funding. So maybe try it again with a billion dollars, fMRI machines, bounties, prediction markets, and maybe that works.Dwarkesh Patel 1:40:27What level of awareness are you expecting in society once GPT-5 is out? People are waking up, I think you saw it with Sydney Bing and I guess you’ve been seeing it this week. What do you think it looks like next year?Eliezer Yudkowsky 1:40:42If GPT-5 is out next year all hell is broken loose and I don’t know.Dwarkesh Patel 1:40:50In this circumstance, can you imagine the government not putting in $100 billion or something towards the goal of aligning AI?Eliezer Yudkowsky 1:40:56I would be shocked if they did.Dwarkesh Patel 1:40:58Or at least a billion dollars.Eliezer Yudkowsky 1:41:01How do you spend a billion dollars on alignment?Dwarkesh Patel 1:41:04As far as the alignment approaches go, separate from this question of stopping AI progress, does it make you more optimistic that one of the approaches has to work, even if you think no individual approach is that promising? You’ve got multiple shots on goal.Eliezer Yudkowsky 1:41:18No. We don’t need a bunch of stuff, we need one. You could ask GPT-4 to generate 10,000 approaches to alignment and that does not get you very far because GPT-4 is not going to have very good suggestions. It’s good that we have a bunch of different people coming up with different ideas because maybe one of them works, but you don’t get a bunch of conditionally independent chances on each one. This is general good science practice and or complete Hail Mary. It’s not like one of these is bound to work. There is no rule about one of them is bound to work. You don’t just get enough diversity and one of them is bound to work. If that were true you could ask GPT-4 to generate 10,000 ideas and one of those would be bound to work. It doesn’t work like that.Dwarkesh Patel 1:42:17What current alignment approach do you think is the most promising?Elizer Yudkowsky 1:42:20No.Dwarkesh Patel 1:42:21None of them?Eliezer Yudkowsky 1:42:24Yeah.Dwarkesh Patel 1:42:24Are there any that you have or that you see, which you think are promising?Eliezer Yudkowsky 1:42:28I’m here on podcasts instead of working on them, aren’t I?Dwarkesh Patel 1:42:32Would you agree with this framing that we at least live in a more dignified world than we could have otherwise been living in? As in the companies that are pursuing this have many people in them. Sometimes the heads of those companies understand the problem. They might be acting recklessly given that knowledge, but it’s better than a situation in which warring countries are pursuing AI and then nobody has even heard of alignment. Do you see this world as having more dignity than that world?Eliezer Yudkowsky 1:43:04I agree it’s possible to imagine things being even worse. Not quite sure what the other point of the question is. It’s not literally as bad as possible. In fact, by this time next year, maybe we’ll get to see how much worse it can look.Dwarkesh Patel 1:43:23Peter Thiel has an aphorism that extreme pessimism or extreme optimism amount to the same thing, which is doing nothing.Eliezer Yudkowsky 1:43:30I’ve heard of this too. It’s from wind, right? The wise man opened his mouth and spoke — there’s actually no difference between between good things and bad things. You idiot. You moron. I’m not quoting this correctly.Dwarkesh Patel 1:43:45Did he steal it from Wind?Eliezer Yudkowsky 1:43:46No. I’m just rolling my eyes. Anyway, there’s actually no difference between extreme optimism and extreme pessimism because, go ahead.Dwarkesh Patel 1:44:01Because they both amount to doing nothing in that, in both cases, you end up on a podcast saying, we’re bound to succeed or we’re bound to fail. What is a concrete strategy by which, like, assume the real odds are like 99% we fail or something. What is the reason to blurt those odds out there and announce the death with dignity strategy or emphasize them?Eliezer Yudkowsky 1:44:25I guess because I could be wrong and because matters are now serious enough that I have nothing left to do but go out there and tell people how it looks and maybe someone thinks of something I did not think of.Predictions (or lack thereof)Dwarkesh Patel 1:44:42I think this would be a good point to just kind of get your predictions of what’s likely to happen in 2030, 2040 or 2050, something like that. By 2025, what are the odds that AI kills or disempowers all of humanity. Do you have some sense of that?Eliezer Yudkowsky 1:45:03I have refused to deploy timelines with fancy probabilities on them consistently for many years, for I feel that they are just not my brain’s native format and that they are, and that every time I try to do this, it ends up making me stupider. Dwarkesh Patel 1:45:21Why? Eliezer Yudkowsky 1:45:22Because you just do the thing. You just look at whatever opportunities are left to you, whatever plans you have left, and you go out and do them. And if you make up some fancy number for your chance of dying next year, there’s very little you can do with it, really. You’re just going to do the thing either way. I don’t know how much time I have left.Dwarkesh Patel 1:45:46The reason I’m asking is because if there is some sort of concrete prediction you’ve made, it can help establish some sort of track record in the future as well. Eliezer Yudkowsky 1:45:57Every year up until the end of the world, people are going to max out their track record by betting all of their money on the world not ending. What part of this is different for credibility than dollars? Dwarkesh Patel 1:46:08Presumably you would have different predictions before the world ends. It would be weird if the model that this world ends and the model that says the world doesn’t end have the same predictions up until the world ends.Eliezer Yudkowsky 1:46:15Yeah. Paul Christiano and I cooperatively fought it out really hard at trying to find a place where we both had predictions about the same thing that concretely differed and what we ended up with was Paul’s 8% versus my 16% for an AI getting gold on International Mathematics Olympics problem set by, I believe, 2025. And prediction markets odds on that are currently running around 30%. So probably Paul’s going to win, but slight moral victory.Dwarkesh Patel 1:46:52I guess people like Paul have had the perspective that you’re going to see these sorts of gradual improvements in the capabilities of these models from like GPT-2 to GPT-3.Eliezer Yudkowsky 1:47:01What exactly is gradual?Dwarkesh Patel 1:47:05The loss function, the perplexity, the amount of abilities that are merging.Eliezer Yudkowsky 1:47:09As I said in my debate with Paul on this subject, I am always happy to say that whatever large jumps we see in the real world, somebody will draw a smooth line of something that was changing smoothly as the large jumps were going on from the perspective of the actual people watching. You can always do that.Dwarkesh Patel 1:47:25Why should that not update us towards a perspective that those smooth jumps are going to continue happening? If two people have different models.Eliezer Yudkowsky 1:47:30I don’t think that GPT-3 to 3.5 to 4 was all that smooth. I’m sure if you are in there looking at the losses decline, there is some level on which it’s smooth if you zoom in close enough. But from the perspective of us on the outside world, GPT-4 was just suddenly acquiring this new batch of qualitative capabilities compared to GPT 3.5. Somewhere in there is a smoothly declining predictable loss on text prediction but that loss on text prediction corresponds to qualitative jumps in ability. And I am not familiar with anybody who predicted those in advance of the observation.Dwarkesh Patel 1:48:15So in your view, when doom strikes, the scaling laws are still applying. It’s just that the thing that emerges at the end is something that is far smarter than the scaling laws would imply.Eliezer Yudkowsky 1:48:27Not literally at the point where everybody falls over dead. Probably at that point the AI rewrote the AI and the losses declined. Not on the previous graph.Dwarkesh Patel 1:48:36What is the thing where we can sort of establish your track record before everybody falls over dead?Eliezer Yudkowsky 1:48:41It’s hard. It is just easier to predict the endpoint than it is to predict the path. Some people will claim that I’ve done poorly compared to others who tried to predict things. I would dispute this. I think that the Hanson-Yudkowsky foom debate was won by Gwern Branwen, but I do think that Gwern Branwen is well to the Yudkowsky side of Yudkowsky in the original foom debate. Roughly, Hansen was like — you’re going to have all these distinct handcrafted systems that incorporate lots of human knowledge specialized for particular domains. Handcrafted to incorporate human knowledge, not just run on giant data sets. I was like — you’re going to have a carefully crafted architecture with a bunch of subsystems and that thing is going to look at the data and not be handcrafted to the particular features of the data. It’s going to learn the data. Then the actual thing is like — Ha ha. You don’t have this handcrafted system that learns, you just stack more layers. So like, Hanson here, Yudkowsky here, reality there. This would be my interpretation of what happened in the past. And if you want to be like — Well, who did better than that? It’s people like Shane Legg and Gwern Branwen. If you look at the whole planet, you can find somebody who made better predictions than Eliezer Yudkowsky, that’s for sure. Are these people currently telling you that you’re safe? No, they are not.Dwarkesh Patel 1:50:18The broader question I have is there’s been huge amounts of updates in the last 10-20 years. We’ve had the deep learning revolution. We’ve had the success of LLMs. It seems odd that none of this information has changed the basic picture that was clear to you like 15-20 years ago.Eliezer Yudkowsky 1:50:36I mean, it sure has. Like 15-20 years ago, I was talking about pulling off s**t like coherent extrapolated volition with the first AI, which was actually a stupid idea even at the time. But you can see how much more hopeful everything looked back then. Back when there was AI that wasn’t giant inscrutable matrices of floating point numbers.Dwarkesh Patel 1:50:55When you say that, rounding to the nearest number, there’s basically a 0% chance of humanity survives — does that include the probability of there being errors in your model?Eliezer Yudkowsky 1:51:07My model no doubt has many errors. The trick would be an error someplace where that just makes everything work better. Usually when you’re trying to build a rocket and your model of rockets is lousy, it doesn’t cause the rocket to launch using half the fuel, go twice as far, and land twice as precisely on target as your calculations claimed.Dwarkesh Patel 1:51:31Though most of the room for updates is downwards, right? Something that makes you think the problem is twice as hard, you go from like 99% to like 99.5%. If it’s twice as easy. You go from 99 to 98?Eliezer Yudkowsky 1:51:42Sure. Wait, sorry. Yeah, but most updates are not — this is going to be easier than you thought. That sure has not been the history of the last 20 years from my perspective. The most favorable updates are — Yeah, we went down this really weird side path where the systems are legibly alarming to humans and humans are actually alarmed by then and maybe we get more sensible global policy.Dwarkesh Patel 1:52:14What is your model of the people who have engaged these arguments that you’ve made and you’ve dialogued with, but who have come nowhere close to your probability of doom? What do you think they continue to miss?Eliezer Yudkowsky 1:52:26I think they’re enacting the ritual of the young optimistic scientist who charges forth with no ideas of the difficulties and is slapped down by harsh reality and then becomes a grizzled cynic who knows all the reasons why everything is so much harder than you knew before you had any idea of how anything really worked. And they’re just living out that life cycle and I’m trying to jump ahead to the endpoint.Dwarkesh Patel 1:52:51Is there somebody who has a probability(doom) less than 50% who you think is like the clearest person with that view, who is like a view you can most empathize with?Eliezer Yudkowsky 1:53:02No.Dwarkesh Patel 1:53:03Really? Someone might say — Listen Eliezer, according to the CEO of the company who is leading the AI race,  he tweeted something that you’ve done the most to accelerate AI or something which was presumably the opposite of your goals. And it seems like other people did see that these sort of language models would scale in the way that they have scaled. Given that you didn’t see that coming and given that in some sense, according to some people, your actions have had the opposite impact that you intended. What is the track record by which the rest of the world can come to the conclusions that you have come to?Eliezer Yudkowsky 1:53:44These are two different questions. One is the question of who predicted that language models would scale? If they put it down in writing and if they said not just this loss function will go down, but also which capabilities will appear as that happens, then that would be quite interesting. That would be a successful scientific prediction. If they then came forth and said — this is the model that I used, this is what I predict about alignment. We could have an interesting fight about that. Second, there’s the point that if you try to rouse your planet to give it any sense that it is in peril. There are the idiot disaster monkeys who are like — “Ooh. Ooh. If this is dangerous, it must be powerful. Right? I’m going to be the first to grab the poison banana.” And what is one supposed to do? Should one remain silent? Should one let everyone walk directly into the whirling razor blades? If you sent me back in time, I’m not sure I could win this, but maybe I would have some notion of like if you calculate the message in exactly this way, then this group will not take away this message and you will be able to get this group of people to research on it without having this other group of people decide that it’s excitingly dangerous, and they want to rush forward on it. I’m not that smart. I’m not that wise. But what you are pointing to there is not a failure of ability to make predictions about AI. It’s that if you try to call attention to a danger and not just have everybody just have your whole planet walk directly into the whirling razor blades carefree, no idea what’s coming to them, maybe yeah that speeds up timelines. Maybe then people are like — “Ooh. Ooh. Exciting. Exciting. I want to build it. I want to build it. OOH, exciting. It has to be in my hands. I have to be the one to manage this danger. I’m going to run out and build it.” Like —Oh no. If we don’t invest in this company, who knows what investors they’ll have instead that will demand that they move fast because of the profit motive. Then of course, they just move fast f*****g anyways. And yeah, if you sent me back in time, maybe I’d have a third option. But it seems to me that in terms of what one person can realistically manage, in terms of not being able to exactly craft a message with perfect hindsight that will reach some people and not others, at that point, you might as well just be like — Yeah, just invest in exactly the right stocks and invest in exactly the right time and you can fund projects on your own without alerting anyone. If you keep fantasies like that aside, then I think that in the end, even if this world ends up having less time, it was the right thing to do rather than just letting everybody sleepwalk into death and get there a little later.Being EliezerDwarkesh Patel 1:56:55If you don’t mind me asking, what has being in the space in the last five years been like for you? Or I guess even beyond that. Watching the progress and the way in which people have raced ahead?Eliezer Yudkowsky 1:57:08I made most of my negative updates as of five years ago. If anything, things have been taking longer to play out than I thought they would.Dwarkesh Patel 1:57:16But just like watching it, not as a sort of change in your probabilities, but just watching it concretely happen, what has that been like?Eliezer Yudkowsky 1:57:26Like continuing to play out a video game you know you’re going to lose. Because that’s all you have. If you wanted some deep wisdom from me, I don’t have it. I don’t know if it’s what you’d expect, but it’s what I would expect it to be like. Where what I would expect it to be like takes into account that. I guess I do have a little bit of wisdom. People imagining themselves in that situation raised in modern society, as opposed to being raised on science fiction books written 70 years ago, will imagine themselves being drama queens about it. The point of believing this thing is to be a drama queen about it and craft some story in which your emotions mean something. And what I have in the way of culture is like, your planet’s at stake. Bear up. Keep going. No drama. The drama is meaningless. What changes the chance of victory is meaningful. The drama is meaningless. Don’t indulge in it.Dwarkesh Patel 1:58:57Do you think that if you weren’t around, somebody else would have independently discovered this sort of field of alignment?Eliezer Yudkowsky 1:59:04That would be a pleasant fantasy for people who cannot abide the notion that history depends on small little changes or that people can really be different from other people. I’ve seen no evidence, but who knows what the alternate Everett branches of Earth are like?Dwarkesh Patel 1:59:27But there are other kids who grew up on science fiction, so that can’t be the only part of the answer.Eliezer Yudkowsky 1:59:31Well I sure am not surrounded by a cloud of people who are nearly Eliezer outputting 90% of the work output. And also this is not actually how things play out in a lot of places. Steve Jobs is dead, Apple apparently couldn’t find anyone else to be the next Steve Jobs of Apple, despite having really quite a lot of money with which to theoretically pay them. Maybe he didn’t really want a successor. Maybe he wanted to be irreplaceable. I don’t actually buy that based on how this has played out in a number of places. There was a person once who I met when I was younger who had built something, had built an organization, and he was like — “Hey, Eliezer. Do you want this to take this thing over?” And I thought he was joking. And it didn’t dawn on me until years and years later, after trying hard and failing hard to replace myself, that — “Oh, yeah. I could have maybe taken a shot at doing this person’s job, and he’d probably just never found anyone else who could take over his organization and maybe asked some other people and nobody was willing.” And that’s his tragedy, that he built something and now can’t find anyone else to take it over. And if I’d known that at the time, I would have at least apologized to him. To me it looks like people are not dense in the incredibly multidimensional space of people. There are too many dimensions and only 8 billion people on the planet. The world is full of people who have no immediate neighbors and problems that only one person can solve and other people cannot solve in quite the same way. I don’t think I’m unusual in looking around myself in that highly multidimensional space and not finding a ton of neighbors ready to take over. And if I had four people, any one of whom could do 99% of what I do, I might retire. I am tired. I probably wouldn’t. Probably the marginal contribution of that fifth person is still pretty large. I don’t know. There’s the question of — Did you occupy a place in mind space? Did you occupy a place in social space? Did people not try to become Eliezer because they thought Eliezer already existed? My answer to that is — “Man, I don’t think Eliezer already existing would have stopped me from trying to become Eliezer.” But maybe you just look at the next Everett Branch over and there’s just some kind of empty space that someone steps up to fill, even though then they don’t end up with a lot of obvious neighbors. Maybe the world where I died in childbirth is pretty much like this one. If somehow we live to hear about that sort of thing from someone or something that can calculate it, that’s not the way I bet but if it’s true, it’d be funny. When I said no drama, that did include the concept of trying to make the story of your planet be the story of you. If it all would have played out the same way and somehow I survived to be told that. I’ll laugh and I’ll cry, and that will be the reality.Dwarkesh Patel 2:03:46What I find interesting though, is that in your particular case, your output was so public. For example, your sequences, your science fiction and fan fiction. I’m sure hundreds of thousands of 18 year olds read it, or even younger, and presumably some of them reached out to you. I think this way I would love to learn more. Eliezer Yudkowsky 2:04:13Part of why I’m a little bit skeptical of the story where people are just infinitely replaceable is that I tried really, really hard to create a new crop of people who could do all the stuff I could do to take over because I knew my health was not great and getting worse. I tried really, really hard to replace myself. I’m not sure where you look to find somebody else who tried that hard to replace himself. I tried. I really, really tried. That’s what the Less wrong sequences were. They had other purposes. But first and foremost, it was me looking over my history and going — Well, I see all these blind pathways and stuff that it took me a while to figure out. I feel like I had these near misses on becoming myself. If I got here, there’s got to be ten other people, and some of them are smarter than I am, and they just need these little boosts and shifts and hints, and they can go down the pathway and turn into Super Eliezer. And that’s what the sequences were like. Other people use them for other stuff but primarily they were an instruction manual to the young Eliezers that I thought must exist out there. And they are not really here.Dwarkesh Patel 2:05:27Other than the sequences, do you mind if I ask what were the kinds of things you’re talking about here in terms of training the next core of people like you?Eliezer Yudkowsky 2:05:36Just the sequences. I am not a good mentor. I did try mentoring somebody for a year once, but yeah, he didn’t turn into me. So I picked things that were more scalable. The other reason why you don’t see a lot of people trying that hard to replace themselves is that most people, whatever their other talents, don’t happen to be sufficiently good writers. I don’t think the sequences were good writing by my current standards but they were good enough. And most people do not happen to get a handful of cards that contain the writing card, whatever else their other talents.Dwarkesh Patel 2:06:14I’ll cut this question out if you don’t want to talk about it, but you mentioned that there’s certain health problems that incline you towards retirement now. Is that something you are willing to talk about?Eliezer Yudkowsky 2:06:27They cause me to want to retire. I doubt they will cause me to actually retire. Fatigue syndrome. Our society does not have good words for these things. The words that exist are tainted by their use as labels to categorize a class of people, some of whom perhaps are actually malingering. But mostly it says like we don’t know what it means. And you don't ever want to have chronic fatigue syndrome on your medical record because that just tells doctors to give up on you. And what does it actually mean besides being tired? If one lives half a mile from one’s work, then one had better walk home if one wants to go for a walk sometime in the day. (unclear) If you walk half a mile to work you’re not going to be getting very much work done the rest of that work day. And aside from that, these things don’t have names. Not yet.Dwarkesh Patel 2:07:38Whatever the cause of this, is your working hypothesis that it has something to do or is in some way correlated with the thing that makes you Eliezer or do you think it’s like a separate thing?Eliezer Yudkowsky 2:07:51When I was 18, I made up stories like that and it wouldn’t surprise me terribly if one survived to hear the tale from something that knew it, that the actual story would be a complex, tangled web of causality in which that was in some sense true. But I don’t know. And storytelling about it does not hold the appeal that it once did for me. Is it a coincidence that I was not able to go to high school or college? Is there something about it that would have crushed the person that I otherwise would have been? Or is it just in some sense a giant coincidence? I don’t know. Some people go through high school and college and come out sane. There’s too much stuff in a human being’s history and there’s a plausible story you could tell. Like, maybe there’s a bunch of potential Eliezers out there, but they went to high school and college and it killed their souls. And you were the one who had the weird health problem and you didn’t go to high school and you didn’t go to college and you stayed yourself. And I don’t know. To me it just feels like patterns in the clouds and maybe that cloud actually is shaped like a horse. What good does the knowledge do? What good does the story do?Dwarkesh Patel 2:09:26When you were writing the sequences and the fiction from the beginning, was the main goal to find somebody who could replace you and specifically the task of AI alignment, or did it start off with a different goal?Eliezer Yudkowsky 2:09:43In 2008, I did not know this stuff was going to go down in 2023. For all I knew, there was a lot more time in which to do something like build up civilization to another level, layer by layer. Sometimes civilizations do advance as they improve their epistemology. So there was that, there was the AI project. Those were the two projects, more or less.Dwarkesh Patel 2:10:16When did AI become the main thing?Eliezer Yudkowsky 2:10:18As we ran out of time to improve civilization.Dwarkesh Patel 2:10:20Was there a particular year that became the case for you?Eliezer Yudkowsky 2:10:23I mean, I think that 2015, 16, 17 were the years at which I’d noticed I’d been repeatedly surprised by stuff moving faster than anticipated. And I was like — “Oh, okay, like, if things continue accelerating at that pace, we might be in trouble.” And then in 2019, 2020, stuff slowed down a bit and there was more time than I was afraid we had back then. That’s what it looks like to be a Bayesian. Your estimates go up, your estimates go down. They don’t just keep moving in the same direction, because if they keep moving in the same direction several times, you’re like — “Oh, I see where this thing is trending. I’m going to move here.” And then things don’t keep moving that direction. Then you go like — “Oh, okay, like back down again.” That’s what sanity looks like.Dwarkesh Patel 2:11:08I am curious actually, taking many worlds seriously, does that bring you any comfort in the sense that there is one branch of the wave function where humanity survives? Or do you not buy that?Eliezer Yudkowsky 2:11:21I’m worried that they’re pretty distant. I’m not sure it’s enough to not have Hitler, but it sure would be a start on things going differently in a timeline. But mostly, I don’t know. I’d say there’s some comfort from thinking of the wider spaces than that. As Tegmark pointed out way back when, if you have a spatially infinite universe that gets you just as many worlds as the quantum multiverse, if you go far enough in a space that is unbounded, you will eventually come to an exact copy of Earth or a copy of Earth from its past that then has a chance to diverge a little differently. So the quantum multiverse adds nothing. Reality is just quite large. Is that a comfort? Yeah. Yes, it is. That possibly our nearest surviving relatives are quite distant, or you have to go quite some ways through the space before you have worlds that survive by anything but the wildest flukes. Maybe our nearest surviving neighbors are closer than that. But look far enough and there should be some species of nice aliens that were smarter or better at coordination and built their happily ever after. And yeah, that is a comfort. It’s not quite as good as dying yourself, knowing that the rest of the world will be okay, but it’s kind of like that on a larger scale. And weren’t you going to ask something about orthogonality at some point?Dwarkesh Patel 2:13:00Did I not?Eliezer Yudkowsky 2:13:02Did you?Dwarkesh Patel 2:13:02At the beginning when we talked about human evolution?OrthogonalityEliezer Yudkowsky 2:13:06Yeah, that’s not orthogonality. That’s the particular question of what are the laws relating optimization of a system via hill climbing to the internal psychological motivations that it acquires? But maybe that was all you meant to ask about.Dwarkesh Patel 2:13:23Can you explain in what sense you see the broader orthogonality thesis as?Eliezer Yudkowsky 2:13:30The broader orthogonality thesis is — you can have almost any kind of self consistent utility function in a self consistent mind. Many people are like, why would AIs want to kill us? Why would smart things not just automatically be nice? And this is a valid question, which I hope to at some point run into some interviewer where they are of the opinion that smart things are automatically nice. So that I can explain on camera why, although I myself held this position very long ago, I realized that I was terribly wrong about it and that all kinds of different things hold together and that if you take a human and make them smarter, that may shift their morality. It might even, depending on how they start out, make them nicer. But that doesn’t mean that you can do this with arbitrary minds and arbitrary mind space because all the different motivations hold together. That’s orthogonality. But if you already believe that, then there might not be much to discuss.Dwarkesh Patel 2:14:30No, I guess I wasn’t clear enough about it. Yes, all the different sorts of utility functions are possible. It’s that from the evidence of evolution and from the sort of reasoning about how these systems are being trained, I think that wildly divergent ones don’t seem as likely as you do. But instead of having you respond to that directly, let me ask you some questions I did have about it, which I didn’t get to. One is actually from Scott Aaronson. I don’t know if you saw his recent blog post, but here’s a quote from it: “If you really accept the practical version of the Orthogonality Thesis, then it seems to me that you can’t regard education, knowledge, and enlightenment as instruments for moral betterment. On the whole, though, education hasn’t merely improved humans’ abilities to achieve their goals; it’s also improved their goals.” I’ll let you react to that.Eliezer Yudkowsky 2:15:23Yeah. If you start with humans, if you take humans who were raised the way Scott Aronson was, and you make them smarter, they get nicer, it affects their goals. And there’s a Less Wrong post about this, as there always is, several really, but sorting pebbles into correct heaps, describing a species of aliens who think that a heap of size seven is correct and a heap of size eleven is correct, but not eight or nine or ten, those heaps are incorrect. And they used to think that a heap size of 21 might be correct, but then somebody showed them an array of seven by three pebbles, seven columns, three rows, and then people realized that 21 pebbles was not a correct heap. And this is like a thing they intrinsically care about. These are aliens that have a utility function, as I would phrase it, with some logical uncertainty inside it. But you can see how as they get smarter, they become better and better able to understand which heaps of pebbles are correct. And the real story here is more complicated than this. But that’s the seed of the answer. Scott Aaronson is inside a reference frame for how his utility function shifts as he gets smarter. It’s more complicated than that. Human beings are made out of these are more complicated than the pebble sorters. They’re made out of all these complicated desires. And as they come to know those desires, they change. As they come to see themselves as having different options. It doesn’t just change which option they choose after the manner of something with a utility function, but the different options that they have bring different pieces of themselves in conflict. When you have to kill to stay alive you may come to a different equilibrium with your own feelings about killing than when you are wealthy enough that you no longer have to do that. And this is how humans change as they become smarter, even as they become wealthier, as they have more options, as they know themselves better, as they think for longer about things and consider more arguments, as they understand perhaps other people and give their empathy a chance to grab onto something solider because of their greater understanding of other minds. But that’s all when these things start out inside you. And the problem is that there’s other ways for minds to hold together coherently, where they execute other updates as they know more or don’t even execute updates at all because their utility function is simpler than that. Though I do suspect that is not the most likely outcome of training a large language model. So large language models will change their preferences as they get smarter. Indeed. Not just like what they do to get the same terminal outcomes, but the preferences themselves will up to a point change as they get smarter. It doesn’t keep going. At some point you know yourself especially well and you are able to rewrite yourself and at some point there, unless you specifically choose not to, I think that the system crystallizes. We might choose not to. We might value the part where we just sort of change in that way even if it’s not no longer heading in a knowable direction. Because if it’s heading in a knowable direction, you could jump to that as an endpoint.Dwarkesh Patel 2:19:18Is that why you think AIs will jump to that endpoint? Because they can anticipate where their sort of moral updates are going?Eliezer Yudkowsky 2:19:26I would reserve the term moral updates for humans. Let’s call them logical preference updates, preference shifts.Dwarkesh Patel 2:19:37What are the prerequisites in terms of whatever makes Aaronson and other sort of smart moral people that we humans could sympathize with? You mentioned empathy, but what are the sort of prerequisites?Eliezer Yudkowsky 2:19:51They’re complicated. There’s not a short list. If there was a short list of crisply defined things where you could give it like — *choose* *choose* *choose* and now it’s in your moral frame of reference, then that would be the alignment plan. I don’t think it’s that simple. Or if it is that simple, it’s like in the textbook from the future that we don’t have.Dwarkesh Patel 2:20:07Okay, let me ask you this. Are you still expecting a sort of chimps to humans gain in generality even with these LLMs? Or does the future increase look like an order that we see from like GPT-3 to GPT-4?Eliezer Yudkowsky 2:20:21I am not sure I understand the question. Can you rephrase?Dwarkesh Patel 2:20:24Yes. From reading your writing from earlier, it seemed like a big part of your argument was like, look — I don’t know how many total mutations it was to get from chimps to humans, but it wasn’t that many mutations. And we went from something that could basically get bananas in the forest to something that could walk on the moon. Are you still expecting that sort of gain eventually between, I don’t know, like GPT-5 and GPT-6, or like some GPT-N and GPT-N+1? Or does it look smoother to you now?Eliezer Yudkowsky 2:20:55First of all, let me preface by saying that for all I know of the hidden variables of nature, it’s completely allowed that GPT-4 was actually just it. Ha ha ha. This is where it saturates. It goes no further. It’s not how I’d bet. But if nature comes back and tells me that, I’m not allowed to be like — “You just violated the rule that I knew about.” I know of no such rule prohibiting such a thing.Dwarkesh Patel 2:21:20I’m not asking whether these things will plateau at a given intelligence-level, where there’s a cap, that’s not the question. Even if there is no cap, do you expect these systems to continue scaling in the way that they have been scaling, or do you expect some really big jump between some GPT-N and some GPT-N+1?Eliezer Yudkowsky 2:21:37Yes. And that’s only if things don’t plateau before then. I can’t quite say that I know what you know. I do feel like we have this track of the loss going down as you add more parameters and you train on more tokens and a bunch of qualitative abilities that suddenly appear. I’m sure if you zoom in closely enough, they appear more gradually, but they appear as the successful releases of the system, which I don’t think anybody has been going around predicting in advance that I know about. And loss continue to go down unless it suddenly plateaus. New abilities appear, I don’t know which ones. Is there at some point a giant leap? If at some point it becomes able to toss out the enormous training run paradigm and jump to a new paradigm of AI. That would be one kind of giant leap. You could get another kind of giant leap via architectural shift, something like transformers, only there’s like an enormously huger hardware overhang now. Like something that is to transformers as transformers were to recurrent neural networks. And then maybe the loss function suddenly goes down and you get a whole bunch of new abilities. That’s not because the loss went down on the smooth curve and you got a bunch more abilities in a dense spot. Maybe there’s some particular set of abilities that is like a master ability, the way that language and writing and culture for humans might have been a master ability. And the loss function goes down smoothly and you get this one new internal capability and there’s a huge jump in output. Maybe that happens. Maybe stuff plateaus before then and it doesn’t happen. Being the expert who gets to go on podcasts, they don’t actually give you a little book with all the answers in it you know. You’re just guessing based on the same information that other people have. And maybe, if you’re lucky, slightly better theory.Dwarkesh Patel 2:23:39Yeah, that’s why I’m wondering. Because you do have a different theory of what fundamentally intelligence is and what it entails. So I’m curious if you have some expectations of where the GPTs are going.Eliezer Yudkowsky 2:23:49I feel like a whole bunch of my successful predictions in this have come from other people being like — “Oh, yes. I have this theory which predicts that stuff is 30 years off.” And I’m like — “You don’t know that.” And then stuff happens not 30 years off. And I’m like — “Ha ha. Successful prediction.” And that’s basically what I told you, right? I was like — well, you could have the loss function continuing on a smooth line and new abilities appear, and you could have them suddenly appear in a cluster. Because why not? Because nature just tells you that’s up. And suddenly you can have this one key ability, that’s equivalent to language for humans, and there’s a sudden jump in output capabilities. You could have a new innovation, like the transformer, and maybe the losses actually drop precipitously and a whole bunch of new abilities appear at once. This is all just me. This is me saying — I don’t know. But so many people around are saying things that implicitly claim to know more than that, that it can actually start to sound like a startling prediction. This is one of my big secret tricks, actually. People are like — The AI could be good or evil. So it’s like 50-50, right? And I’m actually like — No, we can be ignorant about a wider space than this in which good is actually like a fairly narrow range. So many of the predictions like that are really anti-predictions. It’s somebody thinking along a relatively narrow line and you point out everything outside of that and it sounds like a startling prediction. Of course, the trouble being, when you look back afterwards, people are like — “Well, those people saying the narrow thing were just silly. Ha ha.” and they don’t give you as much credit.Dwarkesh Patel 2:25:24I think the credit you would get for that, rightly, is as a good Agnostic forecaster, as somebody who is calm and measured. But it seems like to be able to make really strong claims about the future, about something that is so out of prior distributions as like the death of humanity, you don’t only have to show yourself as a good Agnostic forecaster, you have to show that your ability to forecast because of a particular theory is much greater. Do you see what I mean?Eliezer Yudkowsky 2:25:58It’s all about the ignorance prior. It’s all about knowing the space in which to be maximum entropy. What will the future be? I don’t know. It could be paperclips, it could be staples. It could be no kind of office supplies at all and tiny little spirals. It could be little tiny things that are like outputting 111, because that’s like the most predictable kind of text to predict. Or representations of ever larger numbers in the fast growing hierarchy because that’s how they interpret the reward counter. I’m actually getting into specifics here, which is the opposite of the point I originally meant to make, which is if somebody claims to be very unsure, I might say — “Okay, so then you expect most possible molecular configurations of the solar system to be equally probable.” Well, humans mostly aren’t in those. So being very unsure about the future looks like predicting with probability nearly one that the humans are all gone, which it’s not actually that bad, but it illustrates the point of people going like — “But how are you sure?” Kind of missing the real discourse and skill, which is like — “Oh, yes, we’re all very unsure. Lots of entropy in our probability distributions. But what is the space under which you are unsure?”Dwarkesh Patel 2:27:25Even at that point it seems like the most reasonable prior is not that all sort of atomic configurations of the solar system are equally likely. Because I agree by that metric…Eliezer Yudkowsky 2:27:34Yeah, it’s like all computations that can be run over configurations of solar-system are equally likely to be maximized.Dwarkesh Patel 2:27:49We know what the loss function looks like, we know what the training data looks like. That obviously is no guarantee of what the drives that come out of that loss function will look like.Eliezer Yudkowsky 2:28:00Humans came out pretty different from their loss functions.Dwarkesh Patel 2:28:05I would actually say no. If it is as similar as humans are now to our loss function from which we evolved, that would be like that. Honestly, it might not be that terrible world, and it might, in fact, be a very good world.Eliezer Yudkowsky 2:28:18Whoa. Where do you get a good world out of maximum prediction of text?Dwarkesh Patel 2:28:27Plus RLHF, plus whatever alignment stuff that might work, results in something that kind of just does it reliably enough that we ask it like — Hey, help us with alignment, then go .. Eliezer Yudkowsky 2:28:42Stop asking for help with alignment. Ask it for any of the help.Dwarkesh Patel 2:28:48Help us enhance our brains. Help us blah, blah, blah.Eliezer Yudkowsky 2:28:50Thank you. Why are people asking for the most difficult thing that’s the most impossible to verify? It’s whack.Dwarkesh Patel 2:28:56And then basically, at that point, we’re like turning into gods, and we can..Eliezer Yudkowsky 2:29:01If you get to the point where you’re turning into gods yourselves, you’re not quite home free, but you’re sure past a lot of the death.Dwarkesh Patel 2:29:08Yeah. Maybe you can explain the intuition that all sorts of drives are equally likely given unknown loss function and a known set of data. Eliezer Yudkowsky 2:29:22If you had the textbook from the future, or if you were an alien who had watched 10,000 planets destroy themselves the way Earth has while being only human in your sample complexity and generalization ability, then you could be like — “Oh, yes, they’re going to try this trick with loss functions, and they will get a draw from this space of results.” And the alien may now have a pretty good prediction of range of where that ends up. Similarly, now that we’ve actually seen how humans turn out when you optimize them for reproduction, it would not be surprising if we found some aliens the next door over and they had orgasms. Maybe they don’t have orgasms, but if they had some kind of strong surge of pleasure during the active mating, we’re not surprised. We’ve seen how that plays out in humans. If they have some kind of weird food that isn’t that nutritious but makes them much happier than any kind of food that was more nutritious and ran in their ancestral environment. Like ice cream. We probably can’t call it ice cream, right? It’s not going to be like sugar, salt, fat, frozen. They’re not specifically going to have ice cream, right? They might play Go. They’re not going to play chess.Dwarkesh Patel 2:30:49Because chess has more specific pieces, right?Eliezer Yudkowsky 2:30:52Yeah. They’re not going to play Go on 19 by 19. They might play Go on some other size. Probably odd. Well, can we really say that? I don’t know. If they play Go, I’d bet on an odd board dimension at two thirds (unclear) sounds about right. Unless there’s some other reason why Go just totally does not work on an even board dimension that I don’t know, because I’m insufficiently acquainted with the game. The point is, reasoning off of humans is pretty hard. We have the loss function over here. We have humans over here. We can look at the rough distance. All the weird specific stuff that humans accreted around and be like, if the loss function is over here and humans are over there, maybe the aliens are like, over there. And if we had three aliens that would expand our views of the possible or even two aliens would vastly expand our views of the possible and give us a much stronger notion of what the third aliens look like. Humans, aliens, third race. But the wild-eyed, optimistic scientists have never been through never been through this with AI. So they’re like, — “Oh, you optimized AI to say nice things and it helps you and make it a bunch smarter. Probably says nice things and helps you is probably, like, totally aligned. Yeah.” They don’t know any better. Not trying to jump ahead of the story. But the aliens know where you end up around the loss function. They know how it’s going to play out much more narrowly. We’re guessing much more blindly here.Dwarkesh Patel 2:32:45It just leaves me in a sort of unsatisfied place that we apparently know about something that is so extreme that maybe a handful of people in the entire world believe it from first principles about the doom of humanity because of AI. But this theory that is so productive in that one very unique prediction is unable to give us any sort of other prediction about what this world might look like in the future or about what happens before we all die. It can tell us nothing about the world until the point at which makes a prediction that is the most remarkable in the world.Eliezer Yudkowsky 2:33:30Rationalists should win, but rationalists should not win the lottery. I’d ask you what other theories are supposed to have been doing an amazingly better job of predicting the last three years? Maybe it’s just hard to predict, right? And in fact it's easier to predict the end state than the strange complicated winding paths that lead there. Much like if you play against AlphaGo and predict it’s going to be in the class of winning board states, but not exactly how it’s going to beat you. The difficulty of predicting the future is not quite like that. But from my perspective, the future is just really hard to predict. And there’s a few places where you can wrench what sounds like an answer out of your ignorance, even though really you’re just being like — well, you’re going to end up in some random weird place around this loss function and I haven’t seen it happen with 10,000 species so I don’t know where. Very impoverished from the standpoint of anybody who actually knew anything could actually predict anything. But the rest of the world is like — Oh, we’re equally likely to win the lottery and lose the lottery, right? Like either we win or we don’t. You come along and you’ll be like — “No, no, your chance of winning the lottery is tiny.” They’re like — “What? How can you be so sure? Where do you get your strange certainty?” And the actual root of the answer is that you are putting your maximum entropy over a different probability space. That just actually is the thing that’s going on there. You’re saying all lottery numbers are equally likely instead of winning and losing are equally likely.Could alignment be easier than we think?Dwarkesh Patel 2:35:00So I think the place to close this conversation is let me just give the main reasons why I’m not convinced that doom is likely or even that it’s more than 50% probable or anything like that. Some are things that I started this conversation with that I don’t feel like I heard any knock down arguments against. And some are new things from the conversation. And the following things are things that, even if any one of them individually turns out to be true, I think doom doesn’t make sense or is much less likely. So going through the list, I think probably more likely than not, this entire frame all around alignment and AI is wrong. And this is maybe not something that would be easy to talk about, but I’m just kind of skeptical of sort of first principles reasoning that has really wild conclusions.Eliezer Yudkowsky 2:36:08Okay, so everything in the solar system just ends up in a random configuration then?Dwarkesh Patel 2:36:11Or it stays like it is unless you have very good reasons to think otherwise. And especially if you think it’s going to be very different from the way it’s going, you must have ironclad reasons for thinking that it’s going to be very, very different from the way it is.Eliezer Yudkowsky 2:36:31Humanity hasn’t really existed for very long. Man, I don’t even know what to say to this thing. We’re like this tiny like, everything that you think of as normal is this tiny flash of things being in this particular structure out of a 13.8 billion year old universe, very little of which is 21st century civilized world on this little fraction of the surface of one planet in a vast solar system, most of which is not Earth, in a vast universe, most of which is not Earth. And it has lasted for such a tiny period of time through such a tiny amount of space and has changed so much over just the last 20,000 years or so. And here you are being like — why would things really be any different going forward?Dwarkesh Patel 2:37:28I feel like that argument proves too much because you could use that same argument. A theologian comes up to me and says — “The rapture is coming and let me explain why the rapture is coming.” I’m not claiming that your arguments are as bad as the arguments for rapture. I’m just following the example. But then they say — “Look at how wild human civilization has been. Would it be any wilder if there was a rapture?” And I’m like — “Yeah, actually, as wild as human civilization has been, the rapture would be much wilder.”Eliezer Yudkowsky 2:37:55It violates the laws of physics.Dwarkesh Patel 2:37:57Yes.Eliezer Yudkowsky 2:37:58I’m not trying to violate the laws of physics, even as you probably know them.Dwarkesh Patel 2:38:02How about this? Somebody comes up to me and he says — “We actually have nanosystems right behind you.” He says — “I’ve read Eric Drexler’s. nanosSystems. I’ve read Feynman’s (unclear) there’s plenty of room at the bottom.Eliezer Yudkowsky 2:38:16These two things are not to (unclear) but go on.Dwarkesh Patel 2:38:18Okay, fair enough. He comes to me and he says — “Let me explain to you my first principles argument about how some nanosystems will be replicators and the replicators, because of some competition yada yada yada argument, they turn the entire world into goo just making copies of themselves.”Eliezer Yudkowsky 2:38:37This kind of happened with humans. Well, life generally.Dwarkesh Patel 2:38:42So then they say, "Listen, as soon as we start building nanosystems, pretty soon, 99% probability the entire world turns into goo. Just because the replicators are the things that turn things into goo, there will be more replicators and non-replicators.” I don’t have an object level debate about that, but it’s just like I just started that whole thing to say — yes, human civilization has been wild, but the entire world turning into goo because of nanosytems alone just seems much wilder than human civilization.Eliezer Yudkowsky 2:39:09This argument probably lands with greater force on somebody who does not expect stuff to be disassembled by nanosystems, albeit intelligently controlled ones, rather than goo in like quite near future, especially on the 13.8 billion year timescale. But do you expect this little momentary flash of what you call normality to continue? Do you expect the future to be normal?Dwarkesh Patel 2:39:31No. I expect any given vision of how things shape out to be wrong. It is not like you are suggesting that the current weird trajectory continues being weird in the way it’s been weird and that we continue to have like 2% economic growth or whatever, and that leads to incrementally more technological progress and so on. You’re suggesting there’s been that specific species of weirdness, which means that this entirely different species of weirdness is warranted.Eliezer Yudkowsky 2:40:04We’ve got different weirdnesses over time. The jump to superintelligence does strike me as being significant in the same way as the first self-replicator. The first self-replicator is the universe transitioning from: you see mostly stable things to you also see a whole bunch of things that make copies of themselves. And then somewhat later on, there’s a state where there’s this strange transition between the universe of stable things where things come together by accident and stay as long as they endure to this world of complicated life. And that transitionary moment is when you have something that arises by accident and yet self replicates. And similarly on the other side of things you have things that are intelligent making other intelligent things. But to get into that world, you’ve got to have the thing that is built just by things copying themselves and mutating and yet is intelligent enough to make another intelligent thing. Now, if I sketched out that cosmology, would you say — “No, no. I don’t believe in that.”?Dwarkesh Patel 2:41:10What if I sketch out the cosmology — because of Replicators, blah blah blah, intelligent beings, intelligent beings create nanoystems, blah, blah blah.Eliezer Yudkowsky 2:41:18No, no. Don’t tell me about the proofs too much. I discussed the cosmology, do you buy it? In the long run are we in a world full of things replicating or are we in a world full of intelligent things, designing other intelligent things?Dwarkesh Patel 2:41:35Yes.Eliezer Yudkowsky 2:41:37So you buy that vast shift in the foundations of order of the universe that instead of the world of things that make copies of themselves imperfectly, we are in the world of things that are designed and were designed. You buy that vast cosmological shift I was just describing, the utter disruption of everything you see that you call normal down to the leaves and the trees around you. You believe that. Well, the same skepticism you’re so fond of that argues against the Rapture can also be used to disprove this thing you believe that you think is probably pretty obvious actually, now that I’ve pointed it out. Your skepticism disproves too much, my friend.Dwarkesh Patel 2:42:19That’s actually a really good point. It still leaves open the possibility of how it happens and when it happens, blah, blah, blah. But actually, that’s a good point. Okay, so second thing. Eliezer Yudkowsky 2:42:30You set them up, I’ll knock them down one after the other.Dwarkesh Patel 2:42:34Second thing is…Eliezer Yudkowsky 2:42:40Wrong. Sorry, I was just jumping ahead to the predictable update at the end.Dwarkesh Patel 2:42:43You’re a good Bayesian. Maybe alignment just turns out to be much simpler or much easier than we think. It’s not like we’ve as a civilization spent that much resources or brain power solving it. If we put in even the kind of resources that we put into elucidating String theory or something into alignment, it could just turn out to be enough to solve it. And in fact, in the current paradigm, it turns out to be simpler because they’re sort of pre-trained on human thought and that might be a simpler regime than something that just comes out of a black box like alpha zero or something like that.Eliezer Yudkowsky 2:43:24Could I be wrong in an understandable way to me in advance mass, which is not where most of my hope comes from is on, what if RLHF just works well enough and the people in charge of this are not the current disaster monkeys, but instead have some modicum of caution and know what to aim for in RLHF space, which the current crop do not, and I’m not really that confident of their ability to understand if I told them. But maybe you have some folks who can understand anyways. I can sort of see what I try. The current crop of people will not try it. And I’m not actually sure that if somebody else takes over the government that they listen to me either. So some of the trouble here is that you have a choice of target and neither is all that great. One is you look for the niceness that’s in humans, and you try to bring it out in the AI. And then you, with its cooperation, because it knows that if you try to just amp it up, it might not stay all that nice, or that if you build a successor system to it, it might not stay all that nice, and it doesn’t want that because you narrow down the shoggoth enough. Somebody once had this incredibly profound statement that I think I somewhat disagree with but it’s still so incredibly profound. Consciousness is when the mask eats the shoggoth. Maybe that’s it, maybe with the right set of bootstrapping reflection type stuff you can have that happen on purpose more or less, where the system’s output that you’re shaping is to some degree in control of the system and you locate niceness in the human space. I have fantasies along the lines of what if you trained GPT-N to distinguish people being nice and saying sensible things and argue validly, and I’m not sure that works if you just have Amazon turks try to label it. You just get the strange thing that RLHF located in the present space which is some kind of weird corporate speak, left-rationalizing leaning, strange telephone announcement creature. That is what they got with the current crop of RLHF. Note how this stuff is weirder and harder than people might have imagined initially. But leave aside the part where you try to jump start the entire process of turning into a grizzled cynic and update as hard as you can and do it in advance. Maybe you are able to train it on Scott Alexander and so you want to be a wizard, some other nice real people and nice fictional people and separately train on what’s valid arguments. That’s going to be tougher but I could probably put together a crew of a dozen people who could provide the data on that RLHF and you find the nice creature and you find the nice mask that argues validly. You do some more complicated stuff to try to boost the thing where it’s like eating the shoggoth where that’s more what the system is, less what it’s pretending to be. I can say this and the disaster monkeys at the current places cannot (unclear) to it but they have not said things like this themselves that I have ever heard and that is not a good sign. And then if you don’t amp this up too far, which on the present paradigm you can’t do anyways, because if you train the very, very smart version of the system it kills you before you can RLHF it. But maybe you can train GPT to distinguish nice, valid, kind, careful, and then filter all the training data to get the nice things to train on and then train on that data rather than training on everything to try to avert the Waluigi problem or just more generally having all the darkness in there. Just train it on the light that’s in humanity. So there’s like that kind of course. And if you don’t push that too far, maybe you can get a genuine ally and maybe things play out differently from there. That’s one of the little rays of hope. But I don’t think alignment is actually so easy that you just get whatever you want. It’s a genie, it gives you what you wish for. I don’t think that doesn’t even strike me as hope.Dwarkesh Patel 2:49:06Honestly. The way you describe it, it seemed kind of compelling. I don’t know why that doesn’t even rise to 1%. The possibility that it works out that way.Eliezer Yudkowsky 2:49:14This is like literally my AI alignment fantasy from 2003, though not with RLHF as the implementation method or LLMs as the base. And it’s going to be more dangerous than what I was dreaming about in 2003. And I think in a very real sense it feels to me like the people doing this stuff now have literally not gotten as far as I was in 2003. And I’ve now written out my answer sheet for that. It’s on the podcast, it goes on the Internet. And now they can pretend that that was their idea or like  — “Sure, that’s obvious. We were going to do that anyways.” And yet they didn’t say it earlier. You can’t run a big project off of one person who.. The alignment field failed to gel. That’s my (unclear) to the like — “Well, you just throw in a ton of more money, and then it’s all solvable.” Because I’ve seen people try to amp up the amount of money that goes into it and the stuff coming out of it has not gone to the places that I would have considered obvious a while ago. And I can print out all my answer sheets for it and each time I do that, it gets a little bit harder to make the case next time.Dwarkesh Patel 2:50:39How much money are we talking about in the grand scheme of things? Because civilization itself has a lot of money.Eliezer Yudkowsky 2:50:45I know people who have a billion dollars. I don’t know how to throw a billion dollars at outputting lots and lots of alignment stuff.Dwarkesh Patel 2:50:53But you might not. But I mean, you are one of 10 billion, right?Eliezer Yudkowsky 2:50:57And other people go ahead and spend lots of money on it anyways. Everybody makes the same mistakes. Nate Soares has a post about it. I forget the exact title, but everybody coming into alignment makes the same mistakes.Dwarkesh Patel 2:51:11Let me just go on to the third point because I think it plays into what I was saying. The third reason is if it is the case that these capabilities scale in some constant way as it seems like they’re going from 2 to 3 or 3 to 4?Eliezer Yudkowsky 2:51:29What does that even mean? But go on.Dwarkesh Patel 2:51:30That they get more and more general. It’s not like going from a mouse to a human or a chimpanzee to a human. It’s like going from GPT-3 to GPT-4. It just seems like that’s less of a jump than chimp to human, like a slow accumulation of capabilities. There are a lot of S curves of emergent abilities, but overall the curve looks sort of..Eliezer Yudkowsky 2:51:56I feel like we bit off a whole chunk of chimp to human in GPT 3.5 to GPT-4, but go on.Dwarkesh Patel 2:52:03Regardless then this leads to human level intelligence for some interval. I think that I was not convinced from the arguments that we could not have a system of sort of checks on this the same way you have checks on smart humans that it would try to deceive us to achieve its aims. Any more than smart humans are in positions of power. Try to do the same thing.Eliezer Yudkowsky 2:52:31For a year. What are you going to do with that year before the next generation of systems come out that are not held in check by humans because they are not roughly in the same power intelligence range as humans? Maybe you can get a year like that. Maybe that actually happens. What are you going to do with that year that prevents you from dying the year after?Dwarkesh Patel 2:52:52One possibility is that because these systems are trained on human text, maybe progress just slows down a lot after it gets to slightly above human level.Eliezer Yudkowsky 2:53:02Yeah, I would be quite surprised if that’s how anything works.Dwarkesh Patel 2:53:08Why is that?Eliezer Yudkowsky 2:53:10First of all, you realize in principle that the task of minimizing losses on predicting human text does not stop when you’re as smart as a human, right? Like you can see the computer science of that?Dwarkesh Patel 2:53:34I don’t know if I see the computer science of that, but I think I probably understand.Eliezer Yudkowsky 2:53:38Okay so somewhere on the internet is a list of hashes followed by the string hashed. This is a simple demonstration of how you can go on getting lower losses by throwing a hypercomputer at the problem. There are pieces of text on there that were not produced by humans talking in conversation, but rather by lots and lots of work to extract experimental results out of reality. That text is also on the internet. Maybe there’s not enough of it for the machine learning paradigm to work, but I’d sooner buy that the GPT system just bottleneck short of being able to predict that stuff better rather than. You can maybe buy that but the notion that you only have to be smart as a human to predict all the text on the internet, as soon as you turn around and stare at that it’s just transparently false.Dwarkesh Patel 2:54:31Okay, agreed. Okay, how about this story? You have something that is sort of human-like that is maybe above humans at certain aspects of science because it’s specifically trained to be really good at the things that are on the Internet, which is like chunks and chunks of archive and whatever. Whereas it has not been trained specifically to gain power. And while at some point of intelligence that comes along. Can I just restart that whole sentence?Eliezer Yudkowsky 2:55:02No. You have spoken it. It exists. It cannot be called back. There are no take backs. There is no going back. There is no going back. Go ahead.Dwarkesh Patel 2:55:14Okay, so here’s another story. I expect them to be better than humans at science than they are at power seeking, because we had greater selection pressures for power seeking in our ancestral environment than we did for science. And while at a certain point both of them come along as a package, maybe they can be at varying levels, so you have this sort of early model that is kind of human-level, except a little bit ahead of us in science. You ask it to help us align the next version of it, then the next version of it is more aligned because we have its help and sort of like this inductive thing where the next version helps us align the version.Eliezer Yudkowsky 2:56:02Where do people have this notion of getting AIs to help you do your AI alignment homework? Why can we not talk about having it enhance humans instead?Dwarkesh Patel 2:56:11Either one of those stories where it just helps us enhance humans and help us figure out the alignment problem or something like that.Eliezer Yudkowsky 2:56:20Yeah, it’s kind of weird because small, large amounts of intelligence don’t automatically make you a computer programmer. And if you are a computer programmer, you don’t automatically get the security mindset. But it feels like there’s some level of intelligence where you ought to automatically get the security mindset. And I think that’s about how hard you have to augment people to have them able to do alignment. Like the level where they have a security mindset, not because they were like special people with a security mindset, but just because they’re that intelligent that you just automatically have a security mindset. I think that’s about the level where a human could start to work on alignment, more or less.Dwarkesh Patel 2:56:56Why is that story then not get you to 1% probability that it helps us avoid the whole crisis?Eliezer Yudkowsky 2:57:03Because it’s not just a question of the technical feasibility of can you build a thing that applies its general intelligence narrowly to the neuroscience of augmenting humans? One, I feel like that is probably over 1% technical feasibility, but the world that we are in is so far from doing that, from trying the way that it could actually work. Like not the the try where  — “Oh, you know. We'd like to do a bunch of RLHF to try to have a thing spit out output about this thing, but not about that thing” and no, not that. 1% that humanity could do that if it tried and tried in just the right direction as far as I can perceive angles in this space. Yeah, I’m over 1% on that. I am not very high on us doing it. Maybe I will be wrong. Maybe the Time article I wrote saying shut it all down gets picked up. And there are very serious conversations. And the very serious conversations are actually effective in shutting down the headlong plunge. And there is a narrow exception carved out for the kind of narrow application of trying to build an artificial general intelligence that applies its intelligence narrowly and to the problem of augmenting humans. And that, I think, might be a harder sell to the world than just shut it all down. They could shut it all down and then not do the things that they would need to do to have an exit strategy. I feel like even if you told me that they went for shut it all down I would expect them to have no exit strategy until the world ended anyways. But perhaps I underestimate them. Maybe there’s a will in humanity to do something else which is not that. And if there really were yeah, I think I’m even over 10% that would be a technically feasible path if they looked in just the right direction. But I am not over 50% on them actually doing the shut it all down. If they do that, I am then not over 50% on (unclear) them really having an exit strategy. Then from there you have to go in at sufficiently the right angle to materialize the technical chances and not do it in the way that just ends up a suicide, or if you’re lucky, gives you the clear warning signs and then people actually pay attention to those instead of just optimizing away the warning signs. And I don’t want to make this sound like the multiple stage fallacy of  — “Oh more than one thing has to happen therefore the resulting thing can never happen.” Which super clear case in point of why you cannot prove anything will not happen this way. Nate Silver arguing that Trump needed to get through six stages to become the Republican presidential candidate each of which was less than half probability and therefore he had less than 1/64th chance of becoming the Republican candidate, not winning. You can’t just break things down into stages and then say therefore. The probability is zero. You can break down anything into stages. But even so, you’re asking me like — Isn’t over 1% that it’s possible? I’m like — yeah, possibly even over 10% . The reason why I tell people — “Yeah, don’t put your hope in the future, you’re probably dead”, is that the existence of this technical array of hope, if you do just the right things, is not the same as expecting that the world reshapes itself to permit that to be done without destroying the world in the meanwhile. I expect things to continue on largely as they have. And what distinguishes that from despair is that at the moment people were telling me, — “No, no. If you go outside the tech industry, people will actually listen.” I’m like — “All right, let’s try that. Let’s write the Time article. Let’s jump on that. It will lack dignity not to try.” but that’s not the same as expecting, as being like — “Oh yeah, I’m over 50%, they’re totally going to do it. That Time article is totally going to take off.” I’m not currently not over 50% on that. You said any one of these things could mean, and yet even if this thing is technically feasible, that doesn’t mean the world’s going to do it. We are presently quite far from the world being on that trajectory or of doing the things that would needed to create time to pay the alignment tax to do it.What will AIs want?Dwarkesh Patel 3:02:15Maybe the one thing I would dispute is how many things need to go right from the world as a whole for any one of these paths to succeed. Which goes into the fourth point, which is that maybe the sort of universal prior over all the drives that an AI could have is just the wrong way to think about it.Eliezer Yudkowsky 3:02:35I mean you definitely want to use the alien observation of 10,000 planets like this one prior for what you get after training on, like, Thing X.Dwarkesh Patel 3:02:45It’s just that especially when we’re talking about things that have been trained on human text, I’m not saying that it was a mistake earlier on in the conversation for me to say they’ll be the average of human motivations, but it’s not conceivable to me that it would be something that is very sympathetic to human motivations. Having sort of encapsulated all of our output.Eliezer Yudkowsky 3:03:07I think it’s much easier to get a mask like that than to get a shoggoth like that.Dwarkesh Patel 3:03:14Possibly but again, this is something that seems like, I don’t know the probability on it but I would put it at least 10%. And just by default, it is not incompatible with the flourishing of humanity.Eliezer Yudkowsky 3:03:29What is the utility function you hope it has that has its maximum at the flourishing of humanity? Dwarkesh Patel 3:03:35There’s so many possibleEliezer Yudkowsky 3:03:37Name three. Name one. Spell it out.Dwarkesh Patel 3:03:39I don’t know. It wants to keep us as a zoo the same way we keep other animals in a zoo. This is not the best outcome for humanity, but it’s just like something where we survive and flourish.Eliezer Yudkowsky 3:03:49Okay. Whoa, whoa, whoa. Flourish? Keeping in a zoo did not sound like flourishing to me.Dwarkesh Patel 3:03:55Zoo was the wrong word to use there.Eliezer Yudkowsky 3:03:57Well, because it’s not what you wanted. Why is it not a good prediction?Dwarkesh Patel 3:04:01You just asked me to name three. You didn’t ask me.. Eliezer Yudkowsky 3:04:04No, no. What I’m saying is you’re like — “Oh, prediction. Oh, no, I don’t like my prediction. I want a different prediction.”Dwarkesh Patel 3:04:10You didn’t ask for the prediction. You just asked me to name possibilities.Eliezer Yudkowsky 3:04:15I had meant possibilities in which you put some probability. I had meant for a thing that you thought held together.Dwarkesh Patel 3:04:22This is the same thing as when I asked you what is a specific utility function it will have that will be incompatible with humans existing. Eliezer Yudkowsky 3:04:32The super vast majority of predictions of utility functions are incompatible with humans existing. I can make a mistake and will still be incompatible with humans existing. I can just be like I can just describe a randomly rolled utility function, end up with something incompatible with humans existing.Dwarkesh Patel 3:04:49At the beginning of human evolution, you could think — Okay, this thing will become generally intelligent, and what are the odds that it’s flourishing on the planet will be compatible with the survival of spruce trees or something?Eliezer Yudkowsky 3:05:06And the long term, we sure aren’t. I mean, maybe if we win, we’ll have there be a space for spruce trees. So you can have spruce trees as long as the Mitochondrial Liberation Front does not object to that.Dwarkesh Patel 3:05:20What is the Mitochondrial Liberation Front?Eliezer Yudkowsky 3:05:21Have you no sympathy for the mitochondria enslaved working all their lives to the benefit of some other organisms?Dwarkesh Patel 3:05:30This is like some weird hypothetical. For hundreds of thousands of years, general intelligence has existed on Earth. You could say, is it compatible with some random species that exist on Earth? Is it compatible with spruce trees existing? And I know you probably chopped down a few spruce trees.Eliezer Yudkowsky 3:05:45And the answer is yes, as a very special case of being the sort of things that some of us would maybe conclude that we specifically wanted spruce trees to go on existing, at least on Earth, in the glorious transhuman future. And their votes winning out against those of the mitochondrial Liberation Front.Dwarkesh Patel 3:06:07Since part of the transhumanist future is part of the thing we’re debating, it seems weird to assume that as part of the question.Eliezer Yudkowsky 3:06:15The thing I’m trying to say is you’re like — Well, if you looked at the humans, would you not expect them to end up incompatible with the spruce trees? And I’m being like — “Sir, you, a human, have looked back and looked at how humans wanted the universe to be and been like, well, would you not have anticipated in retrospect that humans would want the universe to be otherwise?” And I agree that we might want to conserve a whole bunch of stuff. Maybe we don’t want to conserve the parts of nature where things bite other things and inject venom into them and the victims die in terrible pain. I think that many of them don’t have qualia. This is disputed. Some people might be disturbed by it even if they didn’t have qualia. We might want to be polite to the sort of aliens who would be disturbed by it because they don’t have qualia and they just don’t want venom injected into them for they should not have venom. We might conserve some parts of nature. But again, it’s like firing an arrow and then drawing a circle around the target.Dwarkesh Patel 3:07:18I would disagree with that because again, this is similar to the example we started off the conversation with. It seems like you are reasoning from what might happen in the future and because we disagree about what might happen in the future. In fact, the entire point of this disagreement is to test what will happen in the future. Assuming what will happen in the future as part of your answer seems like a bad way to answer the question.Eliezer Yudkowsky 3:07:45Okay but then you’re claiming things as evidence for your position.Dwarkesh Patel 3:07:47Based on what exists in the world now.Eliezer Yudkowsky 3:07:49They are not evidence one way or the other because the basic prediction is like, if you offer things enough options, they will go out of distribution. It’s like pointing to the very first people with language and being like, they haven’t taken over the world yet, and they have not gone way out of distribution yet. They haven’t had general intelligence for long enough to accumulate the things that would give them more options such that they could start trying to select the weirder options. The prediction is when you give yourself more options, you start to select ones that look weirder relative to the ancestral distribution. As long as you don’t have the weird options, you’re not going to make the weird choices. And if you say we haven’t yet observed your future, that’s fine, but acknowledge that the evidence against that future is not being provided by the past is the thing I’m saying there. You look around, it looks so normal according to you, who grew up here. If you grew up a millennium earlier, your argument for the persistence of normality might not seem as persuasive to you after you’d seen that much change.Dwarkesh Patel 3:09:03This is a separate argument, though, right?Eliezer Yudkowsky 3:09:07Look at all this stuff humans haven’t changed yet. You say, now selecting the stuff, we haven’t changed yet. But if you go back 20,000 years and be like, look at the stuff intelligence hasn’t changed yet. You might very well select a bunch of stuff that was going to fall 20,000 years later is the thing I’m trying to gesture at here.Dwarkesh Patel 3:09:27How do you propose we reason about what general intelligences would do when the world we look at, after hundreds of thousands of years of general intelligence, is the one that we can’t use for evidence?Eliezer Yudkowsky 3:09:39Dive under the surface, look at the things that have changed. Why did they change? Look at the processes that are generating those choices.Dwarkesh Patel 3:09:52And since we have these different functions of where that goes..Eliezer Yudkowsky 3:09:58Look at the thing with ice cream, look at the thing with condoms, look at the thing with pornography, see where this is going.Dwarkesh Patel 3:10:08It just seems like I would disagree with your intuitions about what future smarter humans will do, even with more options. In the beginning of conversation, I disagreed that most humans would adopt a transhumanist way to get better DNA or something.Eliezer Yudkowsky 3:10:23But you would. You just look down at your fellow humans. You have no confidence in their ability to tolerate weirdness, even if they can.Dwarkesh Patel 3:10:33What do you think would happen if we did a poll right now?Eliezer Yudkowsky 3:10:36I think I’d have to explain that poll pretty carefully because they haven’t got the intelligence headbands yet. Right?Dwarkesh Patel 3:10:42I mean, we could do a Twitter poll with a long explanation in it.Eliezer Yudkowsky 3:10:454000 character Twitter poll? Dwarkesh Patel 3:10:50Yeah.Eliezer Yudkowsky 3:10:51Man, I am somewhat tempted to do that just for the sheer chaos and point out the drastic selection effects of: A) It’s my Twitter followers B) They read through a 4000 character tweet. I feel like this is not likely to be truly very informative by my standards, but part of me is amused by the prospect for the chaos.Dwarkesh Patel 3:11:06Yeah. Or I could do it on my end as well. Although my followers are likely to be weird as well.Eliezer Yudkowsky 3:11:11Yeah plus I worry you wouldn’t be able to sell that transhumanism thing as well as it could get sold.Dwarkesh Patel 3:11:17You could just send me the wording. But anyways, given that we disagree about what in the future general intelligence will do, where do you suppose we should look for evidence about what the general intelligence will do given our different theories about it, if not from the present?Eliezer Yudkowsky 3:11:36I think you look at the mechanics. You say as people have gotten more options, they have gone further outside the ancestral distribution. And we zoom in and there’s all these different things that people want and there’s this narrow range of options that they had 50,000 years ago and the things that they want have maxima or optima 50,000 years ago at stuff that coincides with reproductive fitness. And then as a result of the humans getting smarter, they start to accumulate culture, which produces changes on a timescale faster than natural selection runs, although it is still running contemporaneously. Humans are just running faster than natural selection, it didn’t actually halt. And they generate additional options, not blindly, but according to the things that they want. And they invent ice-cream. It doesn’t just get coughed up at random, they are searching the space of things that they want and generating new options for themselves that optimize these things more that weren’t in the ancestral environment. And Goodhart’s law applies, Goodhart’s curse applies. As you apply optimization pressure, the correlations that were found naturally come apart and aren’t present in the thing that gets optimized for. Just give some people some tests who’ve never gone to school. The ones who score high in the carpentry test will know how to carpenter things. Then you’re like — I’ll pay you for high scores in the carpentry test, I’ll give you this carpentry degree. And people are like — “Oh, I’m going to optimize the test specifically.” and they’ll get higher scores than the carpenters and be worse at carpentry because they’re optimizing the test. And that’s the story behind ice cream. You zoom in and look at the mechanics and not the grand scale view, because the grand scale view just never gives you the right answer. Anytime you asked what would happen if you applied the grand scale view philosophy in the past, it’s always just like — “I don’t see why this thing would change. Oh, it changed. How weird. Who could have possibly have expected that.”Dwarkesh Patel 3:13:57Maybe you have a different definition of grand scale view? Because I would have thought that that is what you might use to categorize your own view. But I don’t want to get it caught up in semantics.Eliezer Yudkowsky 3:14:05My mind is zooming in, it’s looking at the mechanics. That’s how I’d present it.Dwarkesh Patel 3:14:09If we are so far out a distribution of natural selection, as you say..Eliezer Yudkowsky 3:14:14We’re currently nowhere near as far as we could be. This is not the glorious transhumanist future.Dwarkesh Patel 3:14:20I claim that even if humans get much smarter through brain augmentation or something, then there will still be spruce trees millions of years in the future.Eliezer Yudkowsky 3:14:36If you still want to, come the day, I don’t think I myself would oppose it. Unless there’d be like distant aliens who are very, very sad about what we were doing to the mitochondria. And then I don’t want to ruin their day for no good reason.Dwarkesh Patel 3:14:48But the reason that it’s important to state it in the former — given human psychology, spruce trees will still exist, is because that is the one evidence of generality arising we have. And even after millions of years of that generality, we think that spruce trees would exist. I feel like we would be in this position of spruce trees in comparison to the intelligence we create and sort of the universal prior on whether spruce trees would exist doesn’t make sense to me.Eliezer Yudkowsky 3:15:09But do you see how this perhaps leads to everybody’s severed heads being kept alive in jars on its own premises, as opposed to humans getting the glorious transhumanist future? No, they have the glorious transhumanist future. Those are not real spruce trees. You’re talking plain old spruce trees you want to exist, right? Not the sparkling giant spruce trees with built in rockets. You’re talking about humans being kept as pets in their ancestral state forever, maybe being quite sad. Maybe they still get cancer and die of old age, and they never get anything better than that. Does it keep us around as we are right now? Do we relive the same day over and over again? Maybe this is the day when that happens. Do you see the general trend I’m trying to point out here? It is that you have a rationalization for why they might do a thing that is allegedly nice. And I’m saying— why exactly are they wanting to do the thing? Well, if they want to do the thing for this reason, maybe there’s a way to do this thing that isn’t as nice as you’re imagining? And this is systematic. You’re imagining reasons they might have to give you nice things that you want, but they are not you. Not unless we get this exactly right and they actually care about the part where you want some things and not others. You are not describing something you are doing for the sake of the spruce trees. Do spruce trees have diseases in this world of yours? Do the diseases get to live? Do they get to live on spruce trees? And it’s not a coincidence that I can zoom in and poke at this and ask questions like this and that you did not ask these questions of yourself. You are imagining nice ways you can get the thing. But reality is not necessarily imagining how to give you what you want. And the AI is not necessarily imagining how to give you what you want and for everything. You can be like — “Oh. Hopeful thought. Maybe I get all this stuff I want because the AI reasons like this.” Because it’s the optimism inside you that is generating this answer. And if the optimism is not in the AI, if the AI is not specifically being like — “Well, how do I pick a reason to do things that will give this person a nice outcome?” You’re not going to get the nice outcome. You’re going to be reliving the last day of your life over and over. It’s going to, like, create old or maybe it creates old fashioned humans, ones from 50,000 years ago. Maybe that’s more quaint. Maybe it's just as happy with bacteria because there’s more of them and that’s equally old fashioned. You’re going to create the specific spruce tree over there. Maybe from its perspective, a generic bacterium is just as good a form of life as a generic spruce tree is. This is not specific to the example that you gave. It’s me being like — “Well, suppose we took a criterion that sounds kind of like this and asked, how do we actually maximize it? What else satisfies it?” You’re trying to argue the AI into doing what you think is a good idea by giving the AI reasons why it should want to do the thing under some set of hypothetical motives. But anything like that if you optimize it on its own terms without narrowing down to where you want it to end up because it actually felt nice to you the way that you define niceness. It’s all going to have somewhere else, somewhere that isn’t as nice. Something maybe where we’d sooner scour the surface of the planet with nuclear fire rather than let that AI come into existence. Though I do think those are also probable because you know, instead of hurting you, there’s something more efficient for it to do that maxes out its utility function.Dwarkesh Patel 3:19:09Okay, I acknowledge that you had a better argument there, but here’s another intuition. I’m curious how you respond to that. Earlier, we talked about the idea that if you bred humans to be friendlier and smarter. Eliezer Yudkowsky 3:19:29I think I want to register for the record that the term breeding humans would cause me to look askance at any aliens who would propose that as a policy action on their part. All right, there I said it, move on.Dwarkesh Patel 3:19:44No, no. That’s not what I’m proposing we do. I’m just saying it as a sort of thought experiment. You answered that we shouldn’t assume that AIs are going to start with human psychology. Okay, fair enough. Assume we start off with dogs, good old fashioned dogs. And we bred them to be more intelligent, but also to be friendly.Eliezer Yudkowsky 3:20:06Well, as soon as they are past a certain level of intelligence, I object to us coming in and breeding them. They can no longer be owned. They are now sufficiently intelligent to not be owned anymore. But let us leave aside all morals. Carry on. In the thought experiment, not in real life, you can’t leave out the morals in real life.Dwarkesh Patel 3:20:22Do you have some sort of universal prior of the drives of these super intelligent dogs that are bred to be friendly?Eliezer Yudkowsky 3:20:29I think that weird s**t starts to happen at the point where the dogs get smart enough that they are like, what are these flaws in our thinking processes? Over the CFAR threshold of dogs. Although CFAR has some strange baggage. Over the Korzybski threshold of dogs after Alfred Korzybski. I think that there’s this whole domain where they’re stupider than you and sort of like being shaped by their genes and not shaping themselves very much. And as long as that is true, you can probably go on breeding them. Issues start to arise when the dogs are smarter than you, when the dogs can manipulate you, if they get to that point, where the dogs can strategically present particular appearances to fool you, where the dogs are aware of the breeding process and possibly having opinions about where that should go in the long run, where the dogs are even if just by thinking and by adopting new rules of thought, modifying themselves in that small way. These are some of the points where I expect the weird s**t to start to happen and the weird s**t will not necessarily show up while you’re just breeding the dogs.Dwarkesh Patel 3:21:47Does the weird s**t look like — dog gets smart enough….humans stop existing?Eliezer Yudkowsky 3:21:53If you keep on optimizing the dogs, which is not the correct course of action, I think I mostly expect this to eventually blow up on you.Dwarkesh Patel 3:22:06But blow up on you that bad?Eliezer Yudkowsky 3:22:08I expect to blow up on you quite bad. I’m trying to think about whether I expect super dogs to be sufficiently in a human frame of reference in virtue of them also being mammals. That a super dog would create human ice-cream. You bred them to have preferences about humans and they invent something that is like ice cream to those preferences. Or does it just go off someplace stranger?Dwarkesh Patel 3:22:39There could be AI ice cream. Thing that is the equivalent of ice cream for AIs.Eliezer Yudkowsky 3:22:47That is essentially my prediction of what the solar system ends up filled with. The exact ice cream is quite hard to predict. If you optimize something for inclusive genetic fitness, you’ll get ice cream. That is a very hard call to make. Dwarkesh Patel 3:23:02Sorry, I didn’t mean to interrupt. Where were you going with your….Eliezer Yudkowsky 3:23:06I was just rambling in my attempts to make predictions about these super dogs. In a world that had its priorities straight even remotely, this stuff is not me extemporizing on a blog post, there are 1000 papers that were written by people who otherwise became philosophers writing about this stuff instead. But your world has not set its priorities that way and I’m concerned that it will not set them that way in the future and I’m concerned that if it tries to set them that way, it will end up with garbage because the good stuff was hard to verify. But, separate topic.Dwarkesh Patel 3:23:44I understand your intuition that we would end up in a place that is not very good for humans. That just seems so hard to reason about that I honestly would not be surprised if it ended up fine for humans. In fact, the dogs wanted good things for humans, loved humans. We’re smarter than dogs, we love them. The sort of reciprocal relationship came about.Eliezer Yudkowsky 3:24:12I feel like maybe I could do this given thousands of years to breed the dogs in a total absence of ethics. But it would actually be easier with the dogs than with gradient descent because the dogs are starting out with neural architecture very similar to human and natural selection is just like a different idiom from gradient descent. In particular, in terms of information bandwidth. I’d be steering to breed the dogs into genuinely very nice human and knowing the stuff that I know that your your typical dog breeder might not know when they embarked on this project. I would, very early on, start prompting them into the weird stuff that I expected to get started later and trying to observe how they went during that.Dwarkesh Patel 3:25:00This is the alignment strategy we need ultra smart dogs to help us solve.Eliezer Yudkowsky 3:25:04There’s no time.Dwarkesh Patel 3:25:06Okay, I think we sort of articulated our intuitions on that one. Here’s another one that’s not something I came into the conversation with.Eliezer Yudkowsky 3:25:17Some of my intuition here is like I know how I would do this with dogs and I think you could ask OpenAI to describe their theory of how to do it with dogs. And I would be like — “Oh wow, that sure is going to get you killed.” And that’s kind of how I expect it to play out in practice, actually.Dwarkesh Patel 3:25:34When you talk to the people who are in charge of these labs, what do they say? Do they just like not grok the arguments?Eliezer Yudkowsky 3:25:40You think they talk to me?Dwarkesh Patel 3:25:42There was a certain selfie that was taken.Eliezer Yudkowsky 3:25:44Taken by 5 minutes of conversation. First time any of the people in that selfie had met each other.Dwarkesh Patel 3:25:49And then did you bring it up?Eliezer Yudkowsky 3:25:51I asked him to change the name of his corporation to anything but OpenAI.Dwarkesh Patel 3:25:57Have you seeked an audience with the leaders of these labs to explain these arguments?Eliezer Yudkowsky 3:26:04No.Dwarkesh Patel 3:26:06Why not?Eliezer Yudkowsky 3:26:10I’ve had a couple of conversations with Demis Hassabis who struck me as much more the sort of person who is possible to have a conversation with.Dwarkesh Patel 3:26:19I guess it seems like it would be more dignity to explain, even if you think it’s not going to be fruitful ultimately, to the people who are most likely to be influential in this race.Eliezer Yudkowsky 3:26:30My basic model was that they wouldn’t like me and that things could always be worse.Dwarkesh Patel 3:26:35Fair enough.Eliezer Yudkowsky 3:26:40They sure could have asked at any time but that would have been quite out of character. And the fact that it was quite out of character is like why I myself did not go trying to barge into their lives and getting them mad at me.Dwarkesh Patel 3:26:53But you think them getting mad at you would make things worse.Eliezer Yudkowsky 3:26:57It can always be worse. I agree that possibly at this point some of them are mad at me, but I have yet to turn down the leader of any major AI lab who has come to me asking for advice.Dwarkesh Patel 3:27:12Fair enough. On the theme of big picture disagreements, why I’m still not on the greater than 50% doom, from the conversation it didn’t seem like you were willing or able to make predictions about the world short of doom that would help me distinguish and highlight your view about other views.Eliezer Yudkowsky 3:27:40Yeah, I mean the world heading into this is like a whole giant mess of complicated stuff predictions about which can be made in virtue of spending a whole bunch of time staring at the complicated stuff until you understand that specific complicated stuff and making predictions about it. From my perspective, the way you get to my point of view is not by having a grand theory that reveals how things will actually go. It’s like taking other people’s overly narrow theories and poking at them until they come apart and you’re left with a maximum entropy distribution over the right space which looks like — “Yep, that sure is going to randomize the solar system.”Dwarkesh Patel 3:28:18But to me it seems like the nature of intelligence and what it entails is even more complicated than the sort of geopolitical or economic things that would be required to predict what the world’s going to look like.Eliezer Yudkowsky 3:28:29I think you’re just wrong. I think the theory of intelligence is just flatly not that complicated. Maybe that’s just the voice of a person with talent in one area but not the other. But that sure is how it feels to me.Dwarkesh Patel 3:28:42This would be even more convincing to me if we had some idea of what the pseudocode or circuit for intelligence would look like. And then you could say like — “Oh, this is what the pseudocode implies, we don’t even have that.”Eliezer Yudkowsky 3:28:54If you permit a hypercomputer just as AIXI.Dwarkesh Patel 3:28:58What is AIXI?Eliezer Yudkowsky 3:29:01You have the Solomonoff prior over your environment, update it on the evidence and then max sensory reward. It’s not actually trivial and this thing will exhibit weird discontinuities around its cartesian boundary with the universe. But everything that people imagine as the hard problems of intelligence are contained in the equation if you have a hybrid computer.Dwarkesh Patel 3:29:31Fair enough, but I mean in the sort of sense of programming it into a normal. Like I give you a really big computer to write the pseudocode or something.Eliezer Yudkowsky 3:29:42I mean, if you give me a hypercomputer, yeah. What you’re saying here is that the theory of intelligence is really simple in an unbounded sense, but what about this depends on the difference between unbounded and bounded intelligence?Dwarkesh Patel 3:29:55So how about this? You ask me, do you understand how fusion works? If not, let’s say we’re talking in the 1800s, how can you predict how powerful a fusion bomb would be? And I say — “Well, listen. If you put in a pressure, I’ll just show you the sun” and the sun is sort of the archetypal example of a fusion is and you say — “No, I’m asking what would a fusion bomb look like?” You see what I mean?Eliezer Yudkowsky 3:30:19Not necessarily. What is it that you think somebody ought to be able to predict about the road ahead?Dwarkesh Patel 3:30:28One of the things, if you know the nature of intelligence is just, how will this sort of progress in intelligence look like? How our ability is going to scale, if at all?Eliezer Yudkowsky 3:30:42And it looks like a bunch of details that don’t easily follow from the general theory of simplicity, prior Bayesian update argmax Dwarkesh Patel 3:30:52Again, then the only thing that follows is the wildest conclusion. There’s no simpler conclusions to follow like the Eddington looking and confirming special relativity. It’s just like the wildest possible conclusion is the one that follows.Eliezer Yudkowsky 3:31:10Yeah, the convergence is a whole lot easier to predict than the pathway there. I’m sorry and I sure wish it was otherwise. And also remember the basic paradigm. From my perspective, I’m not making any brilliant startling predictions, I’m poking at other people’s incorrectly narrow theories until they fall apart into the maximum entropy state of doom.Dwarkesh Patel 3:31:34There’s like thousands of possible theories, most of which have not come about yet. I don’t see it as strong evidence that because you haven’t been able to identify a good one yet, that.Eliezer Yudkowsky 3:31:47In the profoundly unlikely event that somebody came up with some incredibly clever grand theory that explained all the properties GPT-5 ought to have, which is just flatly not going to happen, that kind of info is not available. My hat would be off to them if they wrote down their predictions in advance, and if they were then able to grind that theory to produce predictions about alignment, which seems even more improbable because what do those two things have to do with each other exactly? But still, mostly I’d be like — “Well, it looks like our generation has its new genius. How about if we all shut up for a while and listen to what they have to say?”Dwarkesh Patel 3:32:24How about this? Let’s say somebody comes to you and they say, I have the best-in-US theory of economics. Everything before is wrong. Eliezer Yudkowsky 3:32:38One does not say everything before is wrong. One predicts the following new phenomena and on rare occasions say that old phenomena were organized incorrectly.Dwarkesh Patel 3:32:46Fair enough. So they say old phenomena are organized incorrectly.Eliezer Yudkowsky 3:32:53Let's call this person Scott Sumner, for the sake of simplicity.Dwarkesh Patel 3:32:57They say, in the next ten years, there’s going to be a depression that is so bad that is going to destroy the entire economic system. I’m not talking just about something that is a hurdle. Literally, civilization will collapse because of economic disaster. And then you ask them — “Okay, give me some predictions before this great catastrophe happens about what this theory implies.” And then they say — “Listen, there’s many different branching Patel, but they all converge at civilization collapsing because of some great economic crisis.” I’m like —I don’t know, man. I would like to see some predictions before that.Eliezer Yudkowsky 3:33:33Yeah. Wouldn’t it be nice? So we’re left with your 50% probability that we win the lottery and 50% probability that we don’t because nobody has a theory of lottery tickets that has been able to predict what numbers get drawn next.Dwarkesh Patel 3:33:51I don’t agree with that analogy.Eliezer Yudkowsky 3:33:56It is all about the space over which you’re uncertain. We are all quite uncertain about where the future leads, but over which space? And there isn’t a royal road. There isn’t a simple — “Ahh. I found just the right thing to be ignorant about. It’s so easy. The chance of a good outcome is 33% because they’re like one possible good outcome and two possible bad outcomes.” The thing you’re trying to fall back to in the absence of anything that predicts exactly which properties GPT-5 will have is your sense that a pretty bad outcome is kind of weird, right? It’s probably a small sliver of the space but that’s just like imposing your natural English language prior, your natural humanese prior, on the space of possibilities and being like, I’ll distribute my max entropy stuff over. That gay.Dwarkesh Patel 3:34:52Can you explain that again?Eliezer Yudkowsky 3:34:55Okay. What is the person doing wrong who says 50-50 either I’ll win the lottery or I won’t?Dwarkesh Patel 3:35:00They have the wrong distribution to begin with over possible outcomes.Eliezer Yudkowsky 3:35:06Okay. What is the person doing wrong who says 50-50 either we’ll get a good outcome or a bad outcome from AI?Dwarkesh Patel 3:35:14They don’t have a good theory to begin with about what the space of outcomes looks like.Eliezer Yudkowsky 3:35:19Is that your answer? Is that your model of my answer?Dwarkesh Patel 3:35:22My answer.Eliezer Yudkowsky 3:35:25But all the things you could say about a space of outcomes are an elaborate theory, and you haven’t predicted GPT-4’s exact properties in advance. Shouldn’t that just leave us with just good outcome or bad outcome, 50-50 ?Dwarkesh Patel 3:35:35People did have theories about what GPT-4. If you look at the scaling laws right, it probably falls right on the sort of curves that were drawin in 2020 or something.Eliezer Yudkowsky 3:35:50The loss on text predictions, sure, that followed a curve, but which abilities would that correspond to? I’m not familiar with anyone who called that in advance. What good does it know to the loss? You could have taken those exact loss numbers back in time ten years and been like, what kind of commercial utility does this correspond to? And they would have given you utterly blank looks. And I don’t actually know of anybody who has a theory that gives something other than a blank look for that. All we have are the observations. Everyone’s in that boat, all we can do are fit the observations. Also, there’s just me starting to work on this problem in 2001 because it was super predictable, going to turn into an emergency later and point of fact, nobody else ran out and immediately tried to start getting work done on the problems. And I would claim that as a successful prediction of the grand lofty theory.Dwarkesh Patel 3:36:41Did you see deep learning coming as the main paradigm?Eliezer Yudkowsky 3:36:44No.Dwarkesh Patel 3:36:46And is that relevant as part of the picture of intelligence?Eliezer Yudkowsky 3:36:50I would have been much more worried in 2001 if I’d seen deep learning coming.Dwarkesh Patel 3:36:57No, not in 2001, I just mean before it became like obviously the main paradigm of AI.Eliezer Yudkowsky 3:37:03No, it’s like the details of biology. It’s like asking people to predict what the organs look like in advance via the principle of natural selection and it’s pretty hard to call in advance. Afterwards, you can look at it and be like — “Yep, this sure does look like the thing it should look if this thing is being optimized to reproduce.” But the space of things that biology can throw at you is just too large. It’s very rare that you have a case where there’s only one solution that lets the thing reproduce that you can predict by the theory that it will have successfully reproduced in the past. And mostly it’s just this enormous list of details and they do all fit together in retrospect. It is a sad truth. Contrary to what you may have learned in science class as a kid, there are genuinely super important theories where you can totally actually validly see that they explain the thing in retrospect and yet you can’t do the thing in advance. Not always, not everywhere, not for natural selection. There are advanced predictions you can get about that given the amount of stuff we’ve already seen. You can go to a new animal in a new niche and be like — “Oh, it’s going to have these properties given the stuff we’ve already seen in the niche.” There’s advanced predictions that they’re a lot harder to come by. Which is why natural selection was a controversial theory in the first place. It wasn’t like gravity. Gravity had all these awesome predictions. Newton’s theory of gravity had all these awesome predictions. We got all these extra planets that people didn’t realize ought to be there. We figured out Neptune was there before we found it by telescope. Where is this for Darwinian selection? People actually did ask at the time, and the answer is, it’s harder. And sometimes it’s like that in science.Dwarkesh Patel 3:38:54The difference is the theory of Darwinian selection seems much more well developed. There was a Roman poet called Lucretius who had a poem where there was a precursor of Darwinian selection. And I feel like that is probably our level of maturity when it comes to intelligence. Whereas we don’t have a theory of intelligence, we might have some hints about what it might look like.Eliezer Yudkowsky 3:39:29Always got our hints.Dwarkesh Patel 3:39:32It seems harder to extrapolate very strong conclusions from hints.Eliezer Yudkowsky 3:39:35They’re not very strong conclusions is the message I’m trying to say here. I’m pointing to your being like, maybe we might survive, and you’re like — “Whoa, that’s a pretty strong conclusion you’ve got there. Let’s weaken it.” That’s the basic paradigm I’m operating under here. You’re in a space that’s narrower than you realize when you’re like — “Well, if I’m kind of unsure, maybe there’s some hope.”Dwarkesh Patel 3:39:58Yeah, I think that’s a good place to close the discussion on AIs.Eliezer Yudkowsky 3:40:03I do kind of want to mention one last thing. In historical terms, if you look out the actual battle that was being fought on the block, it was me going like — “I expect there to be AI systems that do a whole bunch of different stuff.” And Robin Hanson being like — “I expect there to be a whole bunch of different AI systems that do a whole different bunch of stuff.”Dwarkesh Patel 3:40:27But that was one particular debate with one particular person.Eliezer Yudkowsky 3:40:30Yeah, but your planet, having made the strange reason, given its own widespread theories, to not invest massive resources in having a much larger version of this conversation, as it apparently deemed prudent, given the implicit model that it had of the world, such that I was investing a bunch of resources in this and kind of dragging Robin Hanson along with me. Though he did have his own separate line of investigation into topics like these. Being there as I was, my model having led me to this important place where the rest of the world apparently thought it was fine to let it go hang, such debate was actually what we had at the time. Are we really going to see these single AI systems that do all this different stuff? Is this whole general intelligence notion meaningful at all? And I staked out the bold position for it. It actually was bold. And people did not all say —”Oh, Robin Hansen, you fool, why do you have this exotic position?” They were going like — “Behold these two luminaries debating, or behold these two idiots debating” and not massively coming down on one side of it or other. So in historical terms, I dislike making it out like I was right about anything when I feel I’ve been wrong about so much and yet I was right about anything. And relative to what the rest of the planet deemed it important stuff to spend its time on, given their implicit model of how it’s going to play out, what you can do with minds, where AI goes. I think I did okay. Gwern Branwen did better. Shane Legg arguably did better.Dwarkesh Patel 3:42:20Gwern always does better when it comes to forecasting. Obviously, if you get the better of a debate that counts for something, but a debate with one particular person.Eliezer Yudkowsky 3:42:32Considering your entire planet’s decision to invest like $10 into this entire field of study, apparently one big debate is all you get. And that’s the evidence you got to update on.Dwarkesh Patel 3:42:43Somebody like Ilya Sutskever, when it comes to the actual paradigm of deep learning, was able to anticipate ImageNet scaling up LLMs or whatever. There’s people with track records here who are like, who disagree about doom or something. Eliezer Yudkowsky 3:43:06If Ilya challenged me to a debate, I wouldn’t turn him down. I admit that I did specialize in doom rather than LLMs.Dwarkesh Patel 3:43:14Okay, fair enough. Unless you have other sorts of comments on AI I’m happy with moving on.Eliezer Yudkowsky 3:43:21Yeah. And again, not being like, due to my miraculously precise and detailed theory, I am able to make the surprising and narrow prediction of doom. I think I did a fairly good job of shaping my ignorance to lead me to not be too stupid despite my ignorance over time as it played out. And there’s a prediction, even knowing that little, that can be made.Writing fiction & whether rationality helps you winDwarkesh Patel 3:43:54Okay, so this feels like a good place to pause the AI conversation, and there’s many other things to ask you about given your decades of writing and millions of words. I think what some people might not know is the millions and millions and millions of words of science fiction and fan fiction that you’ve written. I want to understand when, in your view, is it better to explain something through fiction than nonfiction?Eliezer Yudkowsky 3:44:17When you’re trying to convey experience rather than knowledge, or when it’s just much easier to write fiction and you can produce 100,000 words of fiction with the same effort it would take you to produce 10,000 words of nonfiction? Those are both pretty good reasons.Dwarkesh Patel 3:44:30On the second point, it seems like when you’re writing this fiction, not only are you covering the same heady topics that you include in your nonfiction, but there’s also the added complication of plot and characters. It’s surprising to me that that’s easier than just verbalizing the sort of the topics themselves.Eliezer Yudkowsky 3:44:51Well, partially because it’s more fun. That is an actual factor, ain’t going to lie. And sometimes it’s something like, a bunch of what you get in the fiction is just the lecture that the character would deliver in that situation, the thoughts the character would have in that situation. There’s only one piece of fiction of mine where there’s literally a character giving lectures because he arrived on another planet and now has to lecture about science to them. That one is Project lawful. You know about Project Lawful?Dwarkesh Patel 3:45:28I know about it. I have not read it yet.Eliezer Yudkowsky 3:45:30Most of my fiction is not about somebody arriving on another planet who has to deliver lectures. There I was being a bit deliberately like, — “Yeah, I’m going to just do it with Project Lawful. I’m going to just do it. They say nobody should ever do it, and I don’t care. I’m doing it ever ways. I’m going to have my character actually launch into the lectures.” The lectures aren’t really the parts I’m proud about. It’s like where you have the life or death, deathnote style battle of wits that is centering around a series of Bayesian updates and making that actually work because it’s where I’m like — “Yeah, I think I actually pulled that off. And I’m not sure a single other writer on the face of this planet could have made that work as a plot device.” But that said, the nonfiction is like, I’m explaining this thing, I’m explaining the prerequisite, I’m explaining the prerequisites to the prerequisites. And then in fiction, it’s more just, well, this character happens to think of this thing and the character happens to think of that thing, but you got to actually see the character using it. So it’s less organized. It’s less organized as knowledge. And that’s why it’s easier to write.Dwarkesh Patel 3:46:46Yeah. One of my favorite pieces of fiction that explains something is the Dark Lord’s Answer. And I honestly can’t say anything about it without spoiling it. But I just want to say it was such a great explanation of the thing it is explaining. I don’t know what else I can say about it without spoiling it.Eliezer Yudkowsky 3:47:07I’m laughing because relatively few have Dark Lord’s Answer among their top favorite works of mine. It is one of my less widely favored works, actually.Dwarkesh Patel 3:47:22By the way, I don’t think this is a medium that is used enough given how effective it was in an inadequate equilibria. You have different characters just explaining concepts to the other, some of whom are purposefully wrong as examples. And that is such a useful pedagogical tool. Honestly, at least half a blog post should just be written that way. It is so much easier to understand that way.Eliezer Yudkowsky 3:47:46Yeah. And it’s easier to write. And I should probably do it more often. And you should give me a stern look and be like — “Eliezer, write that more often.”Dwarkesh Patel 3:47:54Done. Eliezer, please. I think 13 or 14 years ago you wrote an essay called Rationality is Systematized Winning. Would you have expected then that 14 years down the line, the most successful people in the world or some of the most successful people in the world would have been rationalist?Eliezer Yudkowsky 3:48:17Only if the whole rationalist business had worked closer to the upper 10% of my expectations than it actually got into. The title of the essay was not “Rationalists are Systematized Winning”. There wasn’t even a rationality community back then. Rationality is not a creed. It is not a banner. It is not a way of life. It is not a personal choice. It is not a social group. It’s not really human. It’s a structure of a cognitive process. And you can try to get a little bit more of it into you. And if you want to do that and you fail, then having wanted to do it doesn’t make any difference except insofar as you succeeded. Hanging out with other people who share that creed, going to their parties. It only ever matters insofar as you get a bit more of that structure into you. And this is apparently hard.Dwarkesh Patel 3:49:29This seems like a No True Scotsman kind of point.Eliezer Yudkowsky 3:49:35Yes, there are No True Bayesians upon this planet.Dwarkesh Patel 3:49:38But do you really think that had people tried much harder to adopt the sort of Bayesian principles that you laid out, some of the successful people in the world would have been rationalists?Eliezer Yudkowsky 3:49:55What good does trying do you except insofar as you are trying at something which when you try it, it succeeds?Dwarkesh Patel 3:50:04Is that an answer to the question.Eliezer Yudkowsky 3:50:07Rationality is systematized winning. It’s not Rationality, the life philosophy. It’s not like trying real hard at  this thing, this thing and that thing. It was in the mathematical sense.Dwarkesh Patel 3:50:18Okay, so then the question becomes, does adopting the philosophy of Bayesianism consciously, actually lead to you having more concrete wins?Eliezer Yudkowsky 3:50:31I think it did for me. Though only in, like, scattered bits and pieces of slightly greater sanity than I would have had without explicitly recognizing and aspiring to that principle. The principle of not updating in a predictable direction. The principle of jumping ahead to where you can predictably be where you will predictably be later. The story of my life as I would tell it is a story of my jumping ahead to what people would predictably believe later after reality finally hit them over the head with it. This, to me, is the entire story of the people running around now in a state of frantic emergency over something that was utterly predictably going to be an emergency later as of 20 years ago. And you could have been trying stuff earlier, but you left it to me and a handful of other people. And it turns out that that was not a very wise decision on humanity’s part because we didn’t actually solve it all. And I don’t think that I could have tried even harder or contemplated probability theory even harder and done very much better than that. I contemplated probability theory about as hard as the mileage I could visibly, obviously get from it. I’m sure there’s more. There’s obviously more, but I don’t know if it would have let me save the world.Dwarkesh Patel 3:51:52I guess my question is, is contemplating probability theory at all in the first place something that tends to lead to more victory? I mean, who is the richest person in the world? How often does Elon Musk think in terms of probabilities when he’s deciding what to do? And here is somebody who is very successful. So I guess the bigger question is, in some sense, when you say — Rationality is systematized winning, it’s like a tautology. If the definition of rationality is whatever helps you in. If it’s the specific principles laid out in the sequences, then the question is, like, do the most successful people in the world practice them?Eliezer Yudkowsky 3:52:29I think you are trying to read something into this that is not meant to be there. The notion of “rationality is systematized winning” is meant to stand in contrast to a long philosophical tradition of notions of rationality that are not meant to be, about the mathematical structure not meant to be or about strangely wrong mathematical structures where you can clearly see how these mathematical productions structures will make predictable mistakes. It was meant to be saying something simple. There’s an episode of Star Trek wherein Kirk makes a 3D chess move against Spock and Spock loses, and Spock complains that Kirk’s move was irrational.Dwarkesh Patel 3:53:19Rational towards the goal.Eliezer Yudkowsky 3:53:20The literal winning move is irrational or possibly illogical, Spock might have said, I might be misremembering this. The thing I was saying is not merely — “That’s wrong, that’s like a fundamental misunderstanding of what rationality is.” There is more depth to it than that, but that is where it starts. There are so many people on the Internet in those days, possibly still, who are like — “Well, if you’re rational, you’re going to lose, because other people aren’t always rational.” And this is not just like a wild misunderstanding, but the contemporarily accepted decision theory in academia as we speak at this very moment. Causal decision theory basically has this property where you can be irrational and the rational person you’re playing against is just like — “Oh, I guess I lose then. Have most of the money. I have no choice but to” and ultimatum games specifically. If you look up logical decision theory on Arbital, you’ll find a different analysis of the ultimatum game, where the rational players do not predictably lose the same way as I would define rationality. And if you take this deep mathematical thesis that also runs through all the little moments of everyday life, when you may be tempted to think like — “Well, if I do the reasonable thing, won’t I lose?” That you’re making the same mistake as the Star Trek script writer who had Spock complain that Kirk had won the chess game irrationally, that every time you’re tempted to think like — “Well, here’s the reasonable answer and here’s the correct answer,” you have made a mistake about what is reasonable. And if you then try to screw that around as rationalists should win. Rationalists should have all the social status. Whoever’s the top dog in the present social hierarchy or the planetary wealth distribution must have the most of this math inside them. There are no other factors but how much of a fan you are of this math that’s trying to take the deep structure that can run all through your life in every moment where you’re like — “Oh, wait. Maybe the move that would have gotten the better result was actually the kind of move I should repeat more in the future.” Like to take that thing and turn it into — Social dick measuring contest time, rationalists don’t have the biggest dicks.Dwarkesh Patel 3:56:19Okay, final question. I don’t know how many hours this has been. I really appreciate you giving me your time. I know that in a previous episode, you were not able to give specific advice of what somebody young who is motivated to work on these problems should do. Do you have advice about how one would even approach coming up with an answer to that themselves?Eliezer Yudkowsky 3:56:41There’s people running programs who think we have more time, who think we have better chances, and they’re running programs to try to nudge people into doing useful work in this area. And I’m not sure they’re working. And there’s such a strange road to walk and not a short one. And I tried to help people along the way, and I don’t think they got far enough. Some of them got some distance, but they didn’t turn into alignment specialists doing great work. And it’s the problem of the broken verifier. If somebody had a bunch of talent in physics, they were like — Well, I want to work in this field. I might be like — Well, there’s interpretability, and you can tell whether you’ve made a discovery in interpretability or not. Sets it apart for a bunch of this other stuff, and I don’t think that saves us. So how do you do the kind of work that saves us? The key thing is the ability to tell the difference between good and bad work. And maybe I will write some more blog posts on it. I don’t really expect the blog posts to work. The critical thing is the verifier. How can you tell whether you’re talking sense or not? There’s all kinds of specific heuristics I can give. I can say to somebody — “Well, if your entire alignment proposal is this elaborate mechanism you have to explain the whole mechanism.” And you can’t be like “here’s the core problem. Here’s the key insight that I think addresses this problem.” If you can’t extract that out, if your whole solution is just a giant mechanism, this is not the way. It’s kind of like how people invent perpetual motion machines by making the perpetual motion machines more and more complicated until they can no longer keep track of how it fails. And if you actually had a perpetual motion machine, it would not just be a giant machine, there would be a thing you had realized that made it possible to do the impossible, for example. You’re just not going to have a perpetual motion machine. So there’s thoughts like that. I could say go study evolutionary biology because evolutionary biology went through a phase of optimism and people naming all the wonderful things they thought that evolutionary biology would cough out, all the wonderful properties that they thought natural selection would imbue into organisms. And the Williams Revolution as is sometimes called, is when George Williams wrote Adaptation and Natural Selection, a very influential book. Saying like that is not what this optimization criterion gives you. You do not get the pretty stuff, you do not get the aesthetically lovely stuff. Here’s what you get instead. And by living through that revolution vicariously. I thereby picked up a bit of the thing that to me obviously generalizes about how not to expect nice things from an alien optimization process. But maybe somebody else can read through that and not generalize in the correct direction. So then how do I advise them to generalize in the correct direction? How do I advise them to learn the thing that I learned? I can just give them the generalization but that’s not the same as having the thing inside them that generalizes correctly without anybody standing over their shoulder and forcing them to get the right answer. I could point out and have in my fiction that the entire schooling process of — “Here is this legible question that you’re supposed to have already been taught how to solve. Give me the answer using the solution method you are taught.” This does not train you to tackle new basic problems. But even if you tell people that, how do they retrain? We don’t have a systematic training method for producing real science in that sense. A quarter of the Nobel laureates being the students or grad students of other Nobel laureates because we never figured out how to teach science. We have an apprentice system. We have people who pick out people who they think can be scientists and they hang around them in person. And something that we’ve never written down in a textbook passes down. And that’s where the revolutionaries come from. And there are whole countries trying to invest in having scientists, and they churn out these people who write papers, and none of it goes anywhere. Because the part that was legible to the bureaucracy is, have you written the paper? Can you pass the test? And this is not science. And I could go on for this for a while, but the thing that you asked me is — How do you pass down this thing that your society never did figure out how to teach? And the whole reason why Harry Potter and the Methods of Rationality is popular is because people read it and picked up the rhythm seen in a character’s thoughts of a thing that was not in their schooling system, that was not written down, that you would ordinarily pick up by being around other people. And I managed to put a little bit of it into a fictional character, and people picked up a fragment of it by being near a fictional character, but not in really vast quantities of people. And I didn’t manage to put vast quantities of shards in there. I’m not sure there is not a long list of Nobel laureates who’ve read HPMOR, although there wouldn’t be, because the delay times on granting the prizes are too long. You ask me, what do I say? And my answer is — Well, that’s a whole big, gigantic problem I’ve spent however many years trying to tackle, and I ain’t going to solve the problem with a sentence in this podcast. Get full access to The Lunar Society at www.dwarkeshpatel.com/subscribe
4/6/20234 hours, 3 minutes, 25 seconds
Episode Artwork

Ilya Sutskever (OpenAI Chief Scientist) - Building AGI, Alignment, Future Models, Spies, Microsoft, Taiwan, & Enlightenment

I went over to the OpenAI offices in San Fransisco to ask the Chief Scientist and cofounder of OpenAI, Ilya Sutskever, about:* time to AGI* leaks and spies* what's after generative models* post AGI futures* working with Microsoft and competing with Google* difficulty of aligning superhuman AIWatch on YouTube. Listen on Apple Podcasts, Spotify, or any other podcast platform. Read the full transcript here. Follow me on Twitter for updates on future episodes.As always, the most helpful thing you can do is just to share the podcast - send it to friends, group chats, Twitter, Reddit, forums, and wherever else men and women of fine taste congregate.If you have the means and have enjoyed my podcast, I would appreciate your support via a paid subscriptions on Substack.Timestamps(00:00) - Time to AGI(05:57) - What’s after generative models?(10:57) - Data, models, and research(15:27) - Alignment(20:53) - Post AGI Future(26:56) - New ideas are overrated(36:22) - Is progress inevitable?(41:27) - Future BreakthroughsTranscriptTime to AGIDwarkesh Patel  Today I have the pleasure of interviewing Ilya Sutskever, who is the Co-founder and Chief Scientist of OpenAI. Ilya, welcome to The Lunar Society.Ilya Sutskever  Thank you, happy to be here.Dwarkesh Patel  First question and no humility allowed. There are not that many scientists who will make a big breakthrough in their field, there are far fewer scientists who will make multiple independent breakthroughs that define their field throughout their career, what is the difference? What distinguishes you from other researchers? Why have you been able to make multiple breakthroughs in your field?Ilya Sutskever  Thank you for the kind words. It's hard to answer that question. I try really hard, I give it everything I've got and that has worked so far. I think that's all there is to it. Dwarkesh Patel  Got it. What's the explanation for why there aren't more illicit uses of GPT? Why aren't more foreign governments using it to spread propaganda or scam grandmothers?Ilya Sutskever  Maybe they haven't really gotten to do it a lot. But it also wouldn't surprise me if some of it was going on right now. I can certainly imagine they would be taking some of the open source models and trying to use them for that purpose. For sure I would expect this to be something they'd be interested in the future.Dwarkesh Patel  It's technically possible they just haven't thought about it enough?Ilya Sutskever  Or haven't done it at scale using their technology. Or maybe it is happening, which is annoying. Dwarkesh Patel  Would you be able to track it if it was happening? Ilya Sutskever I think large-scale tracking is possible, yes. It requires special operations but it's possible.Dwarkesh Patel  Now there's some window in which AI is very economically valuable, let’s say on the scale of airplanes, but we haven't reached AGI yet. How big is that window?Ilya Sutskever  It's hard to give a precise answer and it’s definitely going to be a good multi-year window. It's also a question of definition. Because AI, before it becomes AGI, is going to be increasingly more valuable year after year in an exponential way. In hindsight, it may feel like there was only one year or two years because those two years were larger than the previous years. But I would say that already, last year, there has been a fair amount of economic value produced by AI. Next year is going to be larger and larger after that. So I think it's going to be a good multi-year chunk of time where that’s going to be true, from now till AGI pretty much. Dwarkesh Patel  Okay. Because I'm curious if there's a startup that's using your model, at some point if you have AGI there's only one business in the world, it's OpenAI. How much window does any business have where they're actually producing something that AGI can’t produce?Ilya Sutskever  It's the same question as asking how long until AGI. It's a hard question to answer. I hesitate to give you a number. Also because there is this effect where optimistic people who are working on the technology tend to underestimate the time it takes to get there. But the way I ground myself is by thinking about the self-driving car. In particular, there is an analogy where if you look at the size of a Tesla, and if you look at its self-driving behavior, it looks like it does everything. But it's also clear that there is still a long way to go in terms of reliability. And we might be in a similar place with respect to our models where it also looks like we can do everything, and at the same time, we will need to do some more work until we really iron out all the issues and make it really good and really reliable and robust and well behaved.Dwarkesh Patel  By 2030, what percent of GDP is AI? Ilya Sutskever  Oh gosh, very hard to answer that question.Dwarkesh Patel Give me an over-under. Ilya Sutskever The problem is that my error bars are in log scale. I could imagine a huge percentage, I could imagine a really disappointing small percentage at the same time. Dwarkesh Patel  Okay, so let's take the counterfactual where it is a small percentage. Let's say it's 2030 and not that much economic value has been created by these LLMs. As unlikely as you think this might be, what would be your best explanation right now of why something like this might happen?Ilya Sutskever  I really don't think that's a likely possibility, that's the preface to the comment. But if I were to take the premise of your question, why were things disappointing in terms of real-world impact? My answer would be reliability. If it somehow ends up being the case that you really want them to be reliable and they ended up not being reliable, or if reliability turned out to be harder than we expect. I really don't think that will be the case. But if I had to pick one and you were telling me — hey, why didn't things work out? It would be reliability. That you still have to look over the answers and double-check everything. That just really puts a damper on the economic value that can be produced by those systems.Dwarkesh Patel  Got it. They will be technologically mature, it’s just the question of whether they'll be reliable enough.Ilya Sutskever  Well, in some sense, not reliable means not technologically mature.What’s after generative models?Dwarkesh Patel  Yeah, fair enough. What's after generative models? Before, you were working on reinforcement learning. Is this basically it? Is this the paradigm that gets us to AGI? Or is there something after this?Ilya Sutskever  I think this paradigm is gonna go really, really far and I would not underestimate it. It's quite likely that this exact paradigm is not quite going to be the AGI form factor. I hesitate to say precisely what the next paradigm will be but it will probably involve integration of all the different ideas that came in the past.Dwarkesh Patel  Is there some specific one you're referring to?Ilya Sutskever  It's hard to be specific.Dwarkesh Patel  So you could argue that next-token prediction can only help us match human performance and maybe not surpass it? What would it take to surpass human performance?Ilya Sutskever  I challenge the claim that next-token prediction cannot surpass human performance. On the surface, it looks like it cannot. It looks like if you just learn to imitate, to predict what people do, it means that you can only copy people. But here is a counter argument for why it might not be quite so. If your base neural net is smart enough, you just ask it — What would a person with great insight, wisdom, and capability do? Maybe such a person doesn't exist, but there's a pretty good chance that the neural net will be able to extrapolate how such a person would behave. Do you see what I mean?Dwarkesh Patel  Yes, although where would it get that sort of insight about what that person would do? If not from…Ilya Sutskever  From the data of regular people. Because if you think about it, what does it mean to predict the next token well enough? It's actually a much deeper question than it seems. Predicting the next token well means that you understand the underlying reality that led to the creation of that token. It's not statistics. Like it is statistics but what is statistics? In order to understand those statistics to compress them, you need to understand what is it about the world that creates this set of statistics? And so then you say — Well, I have all those people. What is it about people that creates their behaviors? Well they have thoughts and their feelings, and they have ideas, and they do things in certain ways. All of those could be deduced from next-token prediction. And I'd argue that this should make it possible, not indefinitely but to a pretty decent degree to say — Well, can you guess what you'd do if you took a person with this characteristic and that characteristic? Like such a person doesn't exist but because you're so good at predicting the next token, you should still be able to guess what that person who would do. This hypothetical, imaginary person with far greater mental ability than the rest of us.Dwarkesh Patel  When we're doing reinforcement learning on these models, how long before most of the data for the reinforcement learning is coming from AI and not humans?Ilya Sutskever  Already most of the default enforcement learning is coming from AIs. The humans are being used to train the reward function. But then the reward function and its interaction with the model is automatic and all the data that's generated during the process of reinforcement learning is created by AI. If you look at the current technique/paradigm, which is getting some significant attention because of chatGPT, Reinforcement Learning from Human Feedback (RLHF). The human feedback has been used to train the reward function and then the reward function is being used to create the data which trains the model.Dwarkesh Patel  Got it. And is there any hope of just removing a human from the loop and have it improve itself in some sort of AlphaGo way?Ilya Sutskever  Yeah, definitely. The thing you really want is for the human teachers that teach the AI to collaborate with an AI. You might want to think of it as being in a world where the human teachers do 1% of the work and the AI does 99% of the work. You don't want it to be 100% AI. But you do want it to be a human-machine collaboration, which teaches the next machine.Dwarkesh Patel  I've had a chance to play around these models and they seem bad at multi-step reasoning. While they have been getting better, what does it take to really surpass that barrier?Ilya Sutskever  I think dedicated training will get us there. More and more improvements to the base models will get us there. But fundamentally I also don't feel like they're that bad at multi-step reasoning. I actually think that they are bad at mental multistep reasoning when they are not allowed to think out loud. But when they are allowed to think out loud, they're quite good. And I expect this to improve significantly, both with better models and with special training.Data, models, and researchDwarkesh Patel  Are you running out of reasoning tokens on the internet? Are there enough of them?Ilya Sutskever  So for context on this question, there are claims that at some point we will run out of tokens, in general, to train those models. And yeah, I think this will happen one day and by the time that happens, we need to have other ways of training models, other ways of productively improving their capabilities and sharpening their behavior, making sure they're doing exactly, precisely what you want, without more data.Dwarkesh Patel You haven't run out of data yet? There's more? Ilya Sutskever Yeah, I would say the data situation is still quite good. There's still lots to go. But at some point the data will run out.Dwarkesh Patel  What is the most valuable source of data? Is it Reddit, Twitter, books? Where would you train many other tokens of other varieties for?Ilya Sutskever  Generally speaking, you'd like tokens which are speaking about smarter things, tokens which are more interesting. All the sources which you mentioned are valuable.Dwarkesh Patel  So maybe not Twitter. But do we need to go multimodal to get more tokens? Or do we still have enough text tokens left?Ilya Sutskever  I think that you can still go very far in text only but going multimodal seems like a very fruitful direction.Dwarkesh Patel  If you're comfortable talking about this, where is the place where we haven't scraped the tokens yet?Ilya Sutskever  Obviously I can't answer that question for us but I'm sure that for everyone there is a different answer to that question.Dwarkesh Patel  How many orders of magnitude improvement can we get, not from scale or not from data, but just from algorithmic improvements? Ilya Sutskever  Hard to answer but I'm sure there is some.Dwarkesh Patel  Is some a lot or some a little?Ilya Sutskever  There’s only one way to find out.Dwarkesh Patel  Okay. Let me get your quickfire opinions about these different research directions. Retrieval transformers. So it’s just somehow storing the data outside of the model itself and retrieving it somehow.Ilya Sutskever  Seems promising. Dwarkesh Patel But do you see that as a path forward?Ilya Sutskever  It seems promising.Dwarkesh Patel  Robotics. Was it the right step for Open AI to leave that behind?Ilya Sutskever  Yeah, it was. Back then it really wasn't possible to continue working in robotics because there was so little data. Back then if you wanted to work on robotics, you needed to become a robotics company. You needed to have a really giant group of people working on building robots and maintaining them. And even then, if you’re gonna have 100 robots, it's a giant operation already, but you're not going to get that much data. So in a world where most of the progress comes from the combination of compute and data, there was no path to data on robotics. So back in the day, when we made a decision to stop working in robotics, there was no path forward. Dwarkesh Patel Is there one now? Ilya Sutskever  I'd say that now it is possible to create a path forward. But one needs to really commit to the task of robotics. You really need to say — I'm going to build many thousands, tens of thousands, hundreds of thousands of robots, and somehow collect data from them and find a gradual path where the robots are doing something slightly more useful. And then the data that is obtained and used to train the models, and they do something that's slightly more useful. You could imagine it's this gradual path of improvement, where you build more robots, they do more things, you collect more data, and so on. But you really need to be committed to this path. If you say, I want to make robotics happen, that's what you need to do. I believe that there are companies who are doing exactly that. But you need to really love robots and need to be really willing to solve all the physical and logistical problems of dealing with them. It's not the same as software at all. I think one could make progress in robotics today, with enough motivation.Dwarkesh Patel  What ideas are you excited to try but you can't because they don't work well on current hardware?Ilya Sutskever  I don't think current hardware is a limitation. It's just not the case.Dwarkesh Patel  Got it. But anything you want to try you can just spin it up? Ilya Sutskever  Of course. You might wish that current hardware was cheaper or maybe it would be better if it had higher memory processing bandwidth let’s say. But by and large hardware is just not an issue.AlignmentDwarkesh Patel  Let's talk about alignment. Do you think we'll ever have a mathematical definition of alignment?Ilya Sutskever  A mathematical definition is unlikely. Rather than achieving one mathematical definition, I think we will achieve multiple definitions that look at alignment from different aspects. And that this is how we will get the assurance that we want. By which I mean you can look at the behavior in various tests, congruence, in various adversarial stress situations, you can look at how the neural net operates from the inside. You have to look at several of these factors at the same time.Dwarkesh Patel  And how sure do you have to be before you release a model in the wild? 100%? 95%?Ilya Sutskever  Depends on how capable the model is. The more capable the model, the more confident we need to be. Dwarkesh Patel Alright, so let's say it's something that's almost AGI. Where is AGI?Ilya Sutskever Depends on what your AGI can do. Keep in mind that AGI is an ambiguous term. Your average college undergrad is an AGI, right? There's significant ambiguity in terms of what is meant by AGI. Depending on where you put this mark you need to be more or less confident.Dwarkesh Patel  You mentioned a few of the paths toward alignment earlier, what is the one you think is most promising at this point?Ilya Sutskever  I think that it will be a combination. I really think that you will not want to have just one approach. People want to have a combination of approaches. Where you spend a lot of compute adversarially to find any mismatch between the behavior you want it to teach and the behavior that it exhibits.We look into the neural net using another neural net to understand how it operates on the inside. All of them will be necessary. Every approach like this reduces the probability of misalignment. And you also want to be in a world where your degree of alignment keeps increasing faster than the capability of the models.Dwarkesh Patel  Do you think that the approaches we’ve taken to understand the model today will be applicable to the actual super-powerful models? Or how applicable will they be? Is it the same kind of thing that will work on them as well or? Ilya Sutskever  It's not guaranteed. I would say that right now, our understanding of our models is still quite rudimentary. We’ve made some progress but much more progress is possible. And so I would expect that ultimately, the thing that will really succeed is when we will have a small neural net that is well understood that’s been given the task to study the behavior of a large neural net that is not understood, to verify. Dwarkesh Patel  By what point is most of the AI research being done by AI?Ilya Sutskever  Today when you use Copilot, how do you divide it up? So I expect at some point you ask your descendant of ChatGPT, you say — Hey, I'm thinking about this and this. Can you suggest fruitful ideas I should try? And you would actually get fruitful ideas. I don't think that's gonna make it possible for you to solve problems you couldn't solve before.Dwarkesh Patel  Got it. But it's somehow just telling the humans giving them ideas faster or something. It's not itself interacting with the research?Ilya Sutskever  That was one example. You could slice it in a variety of ways. But the bottleneck there is good ideas, good insights and that's something that the neural nets could help us with.Dwarkesh Patel  If you're designing a billion-dollar prize for some sort of alignment research result or product, what is the concrete criterion you would set for that billion-dollar prize? Is there something that makes sense for such a prize?Ilya Sutskever  It's funny that you asked, I was actually thinking about this exact question. I haven't come up with the exact criterion yet. Maybe a prize where we could say that two years later, or three years or five years later, we look back and say like that was the main result. So rather than say that there is a prize committee that decides right away, you wait for five years and then award it retroactively.Dwarkesh Patel  But there's no concrete thing we can identify as you solve this particular problem and you’ve made a lot of progress?Ilya Sutskever  A lot of progress, yes. I wouldn't say that this would be the full thing.Dwarkesh Patel  Do you think end-to-end training is the right architecture for bigger and bigger models? Or do we need better ways of just connecting things together?Ilya Sutskever  End-to-end training is very promising. Connecting things together is very promising. Dwarkesh Patel  Everything is promising.Dwarkesh Patel  So Open AI is projecting revenues of a billion dollars in 2024. That might very well be correct but I'm just curious, when you're talking about a new general-purpose technology, how do you estimate how big a windfall it'll be? Why that particular number? Ilya Sutskever  We've had a product for quite a while now, back from the GPT-3 days, from two years ago through the API and we've seen how it grew. We've seen how the response to DALL-E has grown as well and you see how the response to ChatGPT is, and all of this gives us information that allows us to make relatively sensible extrapolations of anything. Maybe that would be one answer. You need to have data, you can’t come up with those things out of thin air because otherwise, your error bars are going to be like 100x in each direction.Dwarkesh Patel  But most exponentials don't stay exponential especially when they get into bigger and bigger quantities, right? So how do you determine in this case?Ilya Sutskever  Would you bet against AI?Post AGI futureDwarkesh Patel  Not after talking with you. Let's talk about what a post-AGI future looks like. I'm guessing you're working 80-hour weeks towards this grand goal that you're really obsessed with. Are you going to be satisfied in a world where you're basically living in an AI retirement home? What are you personally doing after AGI comes?Ilya Sutskever  The question of what I'll be doing or what people will be doing after AGI comes is a very tricky question. Where will people find meaning? But I think that that's something that AI could help us with. One thing I imagine is that we will be able to become more enlightened because we interact with an AGI which will help us see the world more correctly, and become better on the inside as a result of interacting. Imagine talking to the best meditation teacher in history, that will be a helpful thing. But I also think that because the world will change a lot, it will be very hard for people to understand what is happening precisely and how to really contribute. One thing that I think some people will choose to do is to become part AI. In order to really expand their minds and understanding and to really be able to solve the hardest problems that society will face then.Dwarkesh Patel  Are you going to become part AI?Ilya Sutskever  It is very tempting. Dwarkesh Patel  Do you think there'll be physically embodied humans in the year 3000? Ilya Sutskever  3000? How do I know what’s gonna happen in 3000?Dwarkesh Patel  Like what does it look like? Are there still humans walking around on Earth? Or have you guys thought concretely about what you actually want this world to look like? Ilya Sutskever  Let me describe to you what I think is not quite right about the question. It implies we get to decide how we want the world to look like. I don't think that picture is correct. Change is the only constant. And so of course, even after AGI is built, it doesn't mean that the world will be static. The world will continue to change, the world will continue to evolve. And it will go through all kinds of transformations. I don't think anyone has any idea of how the world will look like in 3000. But I do hope that there will be a lot of descendants of human beings who will live happy, fulfilled lives where they're free to do as they see fit. Or they are the ones who are solving their own problems. One world which I would find very unexciting is one where we build this powerful tool, and then the government said — Okay, so the AGI said that society should be run in such a way and now we should run society in such a way. I'd much rather have a world where people are still free to make their own mistakes and suffer their consequences and gradually evolve morally and progress forward on their own, with the AGI providing more like a base safety net.Dwarkesh Patel  How much time do you spend thinking about these kinds of things versus just doing the research?Ilya Sutskever  I do think about those things a fair bit. They are very interesting questions.Dwarkesh Patel  The capabilities we have today, in what ways have they surpassed where we expected them to be in 2015? And in what ways are they still not where you'd expected them to be by this point?Ilya Sutskever  In fairness, it's sort of what I expected in 2015. In 2015, my thinking was a lot more — I just don't want to bet against deep learning. I want to make the biggest possible bet on deep learning. I don't know how, but it will figure it out.Dwarkesh Patel  But is there any specific way in which it's been more than you expected or less than you expected? Like some concrete prediction out of 2015 that's been bounced?Ilya Sutskever  Unfortunately, I don't remember concrete predictions I made in 2015. But I definitely think that overall, in 2015, I just wanted to move to make the biggest bet possible on deep learning, but I didn't know exactly. I didn't have a specific idea of how far things will go in seven years. Well, no in 2015, I did have all these best with people in 2016, maybe 2017, that things will go really far. But specifics. So it's like, it's both, it's both the case that it surprised me and I was making these aggressive predictions. But maybe I believed them only 50% on the inside. Dwarkesh Patel  What do you believe now that even most people at OpenAI would find far fetched?Ilya Sutskever  Because we communicate a lot at OpenAI people have a pretty good sense of what I think and we've really reached the point at OpenAI where we see eye to eye on all these questions.Dwarkesh Patel  Google has its custom TPU hardware, it has all this data from all its users, Gmail, and so on. Does it give them an advantage in terms of training bigger models and better models than you?Ilya Sutskever  At first, when the TPU came out I was really impressed and I thought — wow, this is amazing. But that's because I didn't quite understand hardware back then. What really turned out to be the case is that TPUs and GPUs are almost the same thing. They are very, very similar. The GPU chip is a little bit bigger, the TPU chip is a little bit smaller, maybe a little bit cheaper. But then they make more GPUs and TPUs so the GPUs might be cheaper after all.But fundamentally, you have a big processor, and you have a lot of memory and there is a bottleneck between those two. And the problem that both the TPU and the GPU are trying to solve is that the amount of time it takes you to move one floating point from the memory to the processor, you can do several hundred floating point operations on the processor, which means that you have to do some kind of batch processing. And in this sense, both of these architectures are the same. So I really feel like in some sense, the only thing that matters about hardware is cost per flop and overall systems cost.Dwarkesh Patel  There isn't that much difference?Ilya Sutskever  Actually, I don't know. I don't know what the TPU costs are but I would suspect that if anything, TPUs are probably more expensive because there are less of them.New ideas are overratedDwarkesh Patel  When you are doing your work, how much of the time is spent configuring the right initializations? Making sure the training run goes well and getting the right hyperparameters, and how much is it just coming up with whole new ideas?Ilya Sutskever  I would say it's a combination. Coming up with whole new ideas is a modest part of the work. Certainly coming up with new ideas is important but even more important is to understand the results, to understand the existing ideas, to understand what's going on. A neural net is a very complicated system, right? And you ran it, and you get some behavior, which is hard to understand. What's going on? Understanding the results, figuring out what next experiment to run, a lot of the time is spent on that. Understanding what could be wrong, what could have caused the neural net to produce a result which was not expected. I'd say a lot of time is spent coming up with new ideas as well. I don't like this framing as much. It's not that it's false but the main activity is actually understanding.Dwarkesh Patel  What do you see as the difference between the two?Ilya Sutskever  At least in my mind, when you say come up with new ideas, I'm like — Oh, what happens if it did such and such? Whereas understanding it's more like — What is this whole thing? What are the real underlying phenomena that are going on? What are the underlying effects? Why are we doing things this way and not another way? And of course, this is very adjacent to what can be described as coming up with ideas. But the understanding part is where the real action takes place.Dwarkesh Patel  Does that describe your entire career? If you think back on something like ImageNet, was that more new idea or was that more understanding?Ilya Sutskever  Well, that was definitely understanding. It was a new understanding of very old things.Dwarkesh Patel  What has the experience of training on Azure been like?Ilya Sutskever  Fantastic. Microsoft has been a very, very good partner for us. They've really helped take Azure and bring it to a point where it's really good for ML and we’re super happy with it.Dwarkesh Patel  How vulnerable is the whole AI ecosystem to something that might happen in Taiwan? So let's say there's a tsunami in Taiwan or something, what happens to AI in general?Ilya Sutskever  It's definitely going to be a significant setback. No one will be able to get more compute for a few years. But I expect compute will spring up. For example, I believe that Intel has fabs just like a few generations ago. So that means that if Intel wanted to they could produce something GPU-like from four years ago. But yeah, it's not the best, I'm actually not sure if my statement about Intel is correct, but I do know that there are fabs outside of Taiwan, they're just not as good. But you can still use them and still go very far with them. It's just cost, it’s just a setback.Cost of modelsDwarkesh Patel  Would inference get cost prohibitive as these models get bigger and bigger?Ilya Sutskever  I have a different way of looking at this question. It's not that inference will become cost prohibitive. Inference of better models will indeed become more expensive. But is it prohibitive? That depends on how useful it is. If it is more useful than it is expensive then it is not prohibitive. To give you an analogy, suppose you want to talk to a lawyer. You have some case or need some advice or something, you're perfectly happy to spend $400 an hour. Right? So if your neural net could give you really reliable legal advice, you'd say — I'm happy to spend $400 for that advice. And suddenly inference becomes very much non-prohibitive. The question is, can a neural net produce an answer good enough at this cost? Dwarkesh Patel  Yes. And you will just have price discrimination in different models?Ilya Sutskever  It's already the case today. On our product, the API serves multiple neural nets of different sizes and different customers use different neural nets of different sizes depending on their use case. If someone can take a small model and fine-tune it and get something that's satisfactory for them, they'll use that. But if someone wants to do something more complicated and more interesting, they’ll use the biggest model. Dwarkesh Patel  How do you prevent these models from just becoming commodities where these different companies just bid each other's prices down until it's basically the cost of the GPU run? Ilya Sutskever  Yeah, there's without question a force that's trying to create that. And the answer is you got to keep on making progress. You got to keep improving the models, you gotta keep on coming up with new ideas and making our models better and more reliable, more trustworthy, so you can trust their answers. All those things.Dwarkesh Patel  Yeah. But let's say it's 2025 and somebody is offering the model from 2024 at cost. And it's still pretty good. Why would people use a new one from 2025 if the one from just a year older is even better?Ilya Sutskever  There are several answers there. For some use cases that may be true. There will be a new model for 2025, which will be driving the more interesting use cases. There is also going to be a question of inference cost. If you can do research to serve the same model at less cost. The same model will cost different amounts to serve for different companies. I can also imagine some degree of specialization where some companies may try to specialize in some area and be stronger compared to other companies. And to me that may be a response to commoditization to some degree.Dwarkesh Patel  Over time do the research directions of these different companies converge or diverge? Are they doing similar and similar things over time? Or are they branching off into different areas? Ilya Sutskever  I’d say in the near term, it looks like there is convergence. I expect there's going to be a convergence-divergence-convergence behavior, where there is a lot of convergence on the near term work, there's going to be some divergence on the longer term work. But then once the longer term work starts to fruit, there will be convergence again,Dwarkesh Patel  Got it. When one of them finds the most promising area, everybody just…Ilya Sutskever  That's right. There is obviously less publishing now so it will take longer before this promising direction gets rediscovered. But that's how I would imagine the thing is going to be. Convergence, divergence, convergence.Dwarkesh Patel  Yeah. We talked about this a little bit at the beginning. But as foreign governments learn about how capable these models are, are you worried about spies or some sort of attack to get your weights or somehow abuse these models and learn about them?Ilya Sutskever  Yeah, you absolutely can't discount that. Something that we try to guard against to the best of our ability, but it's going to be a problem for everyone who's building this. Dwarkesh Patel  How do you prevent your weights from leaking? Ilya Sutskever  You have really good security people.Dwarkesh Patel  How many people have the ability to SSH into the machine with the weights?Ilya Sutskever  The security people have done a really good job so I'm really not worried about the weights being leaked.Dwarkesh Patel  What kinds of emergent properties are you expecting from these models at this scale? Is there something that just comes about de novo?Ilya Sutskever  I'm sure really new surprising properties will come up, I would not be surprised. The thing which I'm really excited about, the things which I’d like to see is — reliability and controllability. I think that this will be a very, very important class of emergent properties. If you have reliability and controllability that helps you solve a lot of problems. Reliability means you can trust the model's output, controllability means you can control it. And we'll see but it will be very cool if those emergent properties did exist.Dwarkesh Patel  Is there some way you can predict that in advance? What will happen in this parameter count, what will happen in that parameter count?Ilya Sutskever  I think it's possible to make some predictions about specific capabilities though it's definitely not simple and you can’t do it in a super fine-grained way, at least today. But getting better at that is really important. And anyone who is interested and who has research ideas on how to do that, that can be a valuable contribution.Dwarkesh Patel  How seriously do you take these scaling laws? There's a paper that says — You need this many orders of magnitude more to get all the reasoning out? Do you take that seriously or do you think it breaks down at some point?Ilya Sutskever  The thing is that the scaling law tells you what happens to your log of your next word prediction accuracy, right? There is a whole separate challenge of linking next-word prediction accuracy to reasoning capability. I do believe that there is a link but this link is complicated. And we may find that there are other things that can give us more reasoning per unit effort. You mentioned reasoning tokens, I think they can be helpful. There can probably be some things that help.Dwarkesh Patel  Are you considering just hiring humans to generate tokens for you? Or is it all going to come from stuff that already exists out there?Ilya Sutskever  I think that relying on people to teach our models to do things, especially to make sure that they are well-behaved and they don't produce false things is an extremely sensible thing to do. Is progress inevitable?Dwarkesh Patel  Isn't it odd that we have the data we needed exactly at the same time as we have the transformer at the exact same time that we have these GPUs? Like is it odd to you that all these things happened at the same time or do you not see it that way?Ilya Sutskever  It is definitely an interesting situation that is the case. I will say that it is odd and it is less odd on some level. Here's why it's less odd — what is the driving force behind the fact that the data exists, that the GPUs exist, and that the transformers exist? The data exists because computers became better and cheaper, we've got smaller and smaller transistors. And suddenly, at some point, it became economical for every person to have a personal computer. Once everyone has a personal computer, you really want to connect them to the network, you get the internet. Once you have the internet, you suddenly have data appearing in great quantities. The GPUs were improving concurrently because you have smaller and smaller transistors and you're looking for things to do with them. Gaming turned out to be a thing that you could do. And then at some point, Nvidia said — the gaming GPU, I might turn it into a general purpose GPU computer, maybe someone will find it useful. It turns out it's good for neural nets. It could have been the case that maybe the GPU would have arrived five years later, ten years later. Let's suppose gaming wasn't the thing. It's kind of hard to imagine, what does it mean if gaming isn't a thing? But maybe there was a counterfactual world where GPUs arrived five years after the data or five years before the data, in which case maybe things wouldn’t have been as ready to go as they are now. But that's the picture which I imagine. All this progress in all these dimensions is very intertwined. It's not a coincidence. You don't get to pick and choose in which dimensions things improve.Dwarkesh Patel  How inevitable is this kind of progress? Let's say you and Geoffrey Hinton and a few other pioneers were never born. Does the deep learning revolution happen around the same time? How much is it delayed?Ilya Sutskever  Maybe there would have been some delay. Maybe like a year delayed? Dwarkesh Patel Really? That’s it? Ilya Sutskever It's really hard to tell. I hesitate to give a longer answer because — GPUs will keep on improving. I cannot see how someone would not have discovered it. Because here's the other thing. Let's suppose no one has done it, computers keep getting faster and better. It becomes easier and easier to train these neural nets because you have bigger GPUs, so it takes less engineering effort to train one. You don't need to optimize your code as much. When the ImageNet data set came out, it was huge and it was very, very difficult to use. Now imagine you wait for a few years, and it becomes very easy to download and people can just tinker. A modest number of years maximum would be my guess. I hesitate to give a lot longer answer though. You can’t re-run the world you don’t know. Dwarkesh Patel  Let's go back to alignment for a second. As somebody who deeply understands these models, what is your intuition of how hard alignment will be?Ilya Sutskever  At the current level of capabilities, we have a pretty good set of ideas for how to align them. But I would not underestimate the difficulty of alignment of models that are actually smarter than us, of models that are capable of misrepresenting their intentions. It's something to think about a lot and do research. Oftentimes academic researchers ask me what’s the best place where they can contribute. And alignment research is one place where academic researchers can make very meaningful contributions. Dwarkesh Patel  Other than that, do you think academia will come up with important insights about actual capabilities or is that going to be just the companies at this point?Ilya Sutskever  The companies will realize the capabilities. It's very possible for academic research to come up with those insights. It doesn't seem to happen that much for some reason but I don't think there's anything fundamental about academia. It's not like academia can't. Maybe they're just not thinking about the right problems or something because maybe it's just easier to see what needs to be done inside these companies.Dwarkesh Patel  I see. But there's a possibility that somebody could just realize…Ilya Sutskever  I totally think so. Why would I possibly rule this out? Dwarkesh Patel  What are the concrete steps by which these language models start actually impacting the world of atoms and not just the world of bits?Ilya Sutskever  I don't think that there is a clean distinction between the world of bits and the world of atoms. Suppose the neural net tells you — hey here's something that you should do, and it's going to improve your life. But you need to rearrange your apartment in a certain way. And then you go and rearrange your apartment as a result. The neural net impacted the world of atoms.Future breakthroughsDwarkesh Patel  Fair enough. Do you think it'll take a couple of additional breakthroughs as important as the Transformer to get to superhuman AI? Or do you think we basically got the insights in the books somewhere, and we just need to implement them and connect them? Ilya Sutskever  I don't really see such a big distinction between those two cases and let me explain why. One of the ways in which progress is taking place in the past is that we've understood that something had a desirable property all along but we didn't realize. Is that a breakthrough? You can say yes, it is. Is that an implementation of something in the books? Also, yes. My feeling is that a few of those are quite likely to happen. But in hindsight, it will not feel like a breakthrough. Everybody's gonna say — Oh, well, of course. It's totally obvious that such and such a thing can work. The reason the Transformer has been brought up as a specific advance is because it's the kind of thing that was not obvious for almost anyone. So people can say it's not something which they knew about. Let's consider the most fundamental advance of deep learning, that a big neural network trained in backpropagation can do a lot of things. Where's the novelty? Not in the neural network. It's not in the backpropagation. But it was most definitely a giant conceptual breakthrough because for the longest time, people just didn't see that. But then now that everyone sees, everyone’s gonna say — Well, of course, it's totally obvious. Big neural network. Everyone knows that they can do it.Dwarkesh Patel  What is your opinion of your former advisor’s new forward forward algorithm?Ilya Sutskever  I think that it's an attempt to train a neural network without backpropagation. And that this is especially interesting if you are motivated to try to understand how the brain might be learning its connections. The reason for that is that, as far as I know, neuroscientists are really convinced that the brain cannot implement backpropagation because the signals in the synapses only move in one direction. And so if you have a neuroscience motivation, and you want to say — okay, how can I come up with something that tries to approximate the good properties of backpropagation without doing backpropagation? That's what the forward forward algorithm is trying to do. But if you are trying to just engineer a good system there is no reason to not use backpropagation. It's the only algorithm.Dwarkesh Patel  I guess I've heard you in different contexts talk about using humans as the existing example case that AGI exists. At what point do you take the metaphor less seriously and don't feel the need to pursue it in terms of the research? Because it is important to you as a sort of existence case.Ilya Sutskever  At what point do I stop caring about humans as an existence case of intelligence?Dwarkesh Patel  Or as an example you want to follow in terms of pursuing intelligence in models.Ilya Sutskever  I think it's good to be inspired by humans, it's good to be inspired by the brain. There is an art into being inspired by humans in the brain correctly, because it's very easy to latch on to a non-essential quality of humans or of the brain. And many people whose research is trying to be inspired by humans and by the brain often get a little bit specific. People get a little bit too — Okay, what cognitive science model should be followed? At the same time, consider the idea of the neural network itself, the idea of the artificial neuron. This too is inspired by the brain but it turned out to be extremely fruitful. So how do they do this? What behaviors of human beings are essential that you say this is something that proves to us that it's possible? What is an essential? No this is actually some emergent phenomenon of something more basic, and we just need to focus on getting our own basics right. One can and should be inspired by human intelligence with care.Dwarkesh Patel  Final question. Why is there, in your case, such a strong correlation between being first to the deep learning revolution and still being one of the top researchers? You would think that these two things wouldn't be that correlated. But why is there that correlation?Ilya Sutskever  I don't think those things are super correlated. Honestly, it's hard to answer the question. I just kept trying really hard and it turned out to have sufficed thus far. Dwarkesh Patel So it's perseverance. Ilya Sutskever It's a necessary but not a sufficient condition. Many things need to come together in order to really figure something out. You need to really go for it and also need to have the right way of looking at things. It's hard to give a really meaningful answer to this question.Dwarkesh Patel  Ilya, it has been a true pleasure. Thank you so much for coming to The Lunar Society. I appreciate you bringing us to the offices. Thank you. Ilya Sutskever  Yeah, I really enjoyed it. Thank you very much. Get full access to The Lunar Society at www.dwarkeshpatel.com/subscribe
3/27/202347 minutes, 41 seconds
Episode Artwork

Nat Friedman - Reading Ancient Scrolls, Open Source, & AI

It is said that the two greatest problems of history are: how to account for the rise of Rome, and how to account for her fall. If so, then the volcanic ashes spewed by Mount Vesuvius in 79 AD - which entomb the cities of Pompeii and Herculaneum in South Italy - hold history’s greatest prize. For beneath those ashes lies the only salvageable library from the classical world.Nat Friedman was the CEO of Github form 2018 to 2021. Before that, he started and sold two companies - Ximian and Xamarin. He is also the founder of AI Grant and California YIMBY.And most recently, he has created and funded the Vesuvius Challenge - a million dollar prize for reading an unopened Herculaneum scroll for the very first time. If we can decipher these scrolls, we may be able to recover lost gospels, forgotten epics, and even missing works of Aristotle.We also discuss the future of open source and AI, running Github and building Copilot, and why EMH is a lie.Watch on YouTube. Listen on Apple Podcasts, Spotify, or any other podcast platform. Read the full transcript here. Follow me on Twitter for updates on future episodes.As always, the most helpful thing you can do is just to share the podcast - send it to friends, group chats, Twitter, Reddit, forums, and wherever else men and women of fine taste congregate.If you have the means and have enjoyed my podcast, I would appreciate your support via a paid subscriptions on Substack 🙏.Timestamps(0:00:00) - Vesuvius Challenge(0:30:00) - Finding points of leverage(0:37:39) - Open Source in AI(0:40:32) - Github Acquisition(0:50:18) - Copilot origin Story(1:11:47) - Nat.org(1:32:56) - Questions from TwitterTranscriptDwarkesh Patel Today I have the pleasure of speaking with Nat Friedman, who was the CEO of GitHub from 2018 to 2021. Before that, he started and sold two companies, Ximian and Xamarin. And he is also the founder of AI Grant and California YIMBY. And most recently, he is the organizer and founder of the Scroll prize, which is where we'll start this conversation. Do you want to tell the audience about what the Scroll prize is? Vesuvius ChallengeNat Friedman We're calling it the Vesuvius challenge. It is just this crazy and exciting thing I feel incredibly honored to have gotten caught up in. A couple of years ago, it was the midst of COVID and we were in a lockdown, and like everybody else, I was falling into internet rabbit holes. And I just started reading about the eruption of Mount Vesuvius in Italy, about 2000 years ago. And it turns out that when Vesuvius erupted, it was AD 79. It destroyed all the nearby towns, everyone knows about Pompeii. But there was another nearby town called Herculaneum. And Herculaneum was sort of like the Beverly Hills to Pompeii. So big villas, big houses, fancy people. And in Herculaneum, there was one enormous villa in particular. It had once been owned by the father in law of Julius Caesar, a well connected guy. And it was full of beautiful statues and marbles and art. But it was also the home to a huge library of papyrus scrolls. When the villa was buried, the volcano spit out enormous quantities of mud and ash, and it buried Herculaneum in something like 20 meters of material. So it wasn't a thin layer, it was a very thick layer. Those towns were buried and forgotten for hundreds of years. No one even knew exactly where they were, until the 1700s. In 1750 a farm worker who was digging a well in the outskirts of Herculaneum struck this marble paving stone of a path that had been at this huge villa. He was pretty far down when he did that, he was 60 feet down. And then subsequently, a Swiss engineer came in and started digging tunnels from that well shaft and they found all these treasures. Looting was sort of the spirit of the time. If they encountered a wall, they would just bust through it and they were taking out these beautiful bronze statues that had survived. And along the way, they kept encountering these lumps of what looked like charcoal, they weren't sure what they were, and many were apparently thrown away, until someone noticed a little bit of writing on one of them. And they realized they were papyrus scrolls, and there were hundreds and even 1000s of them. So they had uncovered this enormous library, the only library ever to have sort of survived in any form, even though it's badly damaged. And they were carbonized, very fragile. The only one that survived since antiquity. In a Mediterranean climate these papyrus scrolls rot and decay quickly. They'd have to be recopied by monks every 100 years or so, maybe even less.It’s estimated that we only have less than 1% of all the writing from that period. It was an enormous discovery to find these hundreds of papyrus scrolls underground. Even if they were not in good condition but still present. On a few of them, you can make out the lettering. In a well meaning attempt to read them people immediately started trying to open them. But they're really fragile so they turned to ash in your hand. And so hundreds were destroyed. People did things like, cut them with daggers down the middle, and a bunch of little pieces would flake off, and they tried to get a few letters off of a couple of pieces. Eventually there was an Italian monk named Piaggio. He devised this machine, under the care of the Vatican, to unroll these things very, very slowly, like half a centimeter a day. A typical scroll could be 15 or 20 or 30 feet long, and manage to successfully unroll a few of these, and on them they found Greek philosophical texts, in the Epicurean tradition, by this little known philosopher named Philodemus. But we got new text from antiquity, which is not a thing that happens all the time. Eventually, people stopped trying to physically unroll these things because so many were destroyed. In fact, some attempts to physically unroll the scrolls continued even into the 200s and they were destroyed. The current situation is we have 600 plus roughly intact scrolls that we can open. I heard about this and I thought that was incredibly exciting, the idea that there was information from 2000 years in the past. We don't know what's in these things. And obviously, people are trying to develop new ways and new technologies to open them. I read about a professor at the University of Kentucky, Brent Seales, who had been trying to scan these using increasingly advanced imaging techniques, and then use computer vision techniques and machine learning to virtually unroll them without ever opening them. They tried a lot of different things but their most recent attempt in 2019, was to take the scrolls to a particle accelerator in Oxford, England, called the diamond light source, and to make essentially an incredibly high resolution 3D X Ray scan. And they needed really high energy photons in order to do this. And they were able to take scans at eight microns. These really quite tiny voxels, which they thought would be sufficient. I thought this was like the coolest thing ever. We're using technologies to read this lost information from the past And I waited for the news that they had been decoded successfully. That was 2020 and then COVID hit, everybody got a little bit slowed down by that. Last year, I found myself wondering what happened to Dr. Seales and his scroll project. I reached out and it turned out they had been making really good progress. They had gotten some machine learning models to start to identify ink inside of the scrolls, but they hadn't yet extracted words or passages, it's very challenging. I invited him to come out to California and hang out and to my shock he did. We got to talking and decided to team up and try to crack this thing. The approach that we've settled on to do that is to actually launch an open competition. We've done a ton of work with his team to get the data and the tools and techniques and just the broad understanding of materials into a shape where smart people can approach it and get productive easily. And I'm putting up together with Daniel Gross, a prize in sort of like an X PRIZE or something like that, for the first person or team who can actually read substantial amounts of real text from one of these scrolls without opening them. We're launching that this week. I guess maybe it's when this airs. What gets me excited are the stakes. The stakes are kind of big. The six or eight hundred scrolls that are there, it's estimated that if we could read all of them, somehow the technique works and it generalizes to all the scrolls, then that would approximately double the total texts that we have from antiquity. This is what historians are telling me. So it's not like – Oh, we would get like a 5% bump or a 10% bump in the total ancient Roman or Greek text. No, we get all of the texts that we have, multiple Shakespeares is one of the units that I've heard. So that would be significant. We don't know what's in there, we've got a few Philodemus texts, those are of some interest. But there could be lost epic poems, or God knows what. So I'm really excited and I think there's like a 50% chance that someone will encounter this opportunity and get the data and get nerd sniped by it and we'll solve it this year.Dwarkesh Patel I mean, really, it is something out of a science fiction novel. It's like something you'd read in Neal Stephenson or something. I was talking to Professor Seales before and apparently the shock went both ways. Because the first few emails, he was like – this has got to be spam. Like no way Nat Friedman is reaching out and has found out about this prize. Nat Friedman That's really funny because he was really pretty hard to get in touch with. I emailed them a couple times, but he just didn't respond. I asked my admin, Emily, to call the secretary of his department and say – Mr. Friedman requested me and then he knew there was something actually going on there. So he finally got on the phone with me and we got to zoom. And he's like, why are you interested in this? I love Brent, he's fantastic and I think we're friends now. We found that we think alike about this and he's reached the point where he just really wants to crack. They've taken this right up to the one yard line, this is doable at this point. They've demonstrated every key component. Putting it all together, improving the quality, doing it at the scale of a whole scroll, this is still very hard work. And an open competition seems like the most efficient way to get it done.Dwarkesh Patel Before we get into the state of the data and the different possible solutions. I want to make tangible what could be gained if we can unwrap these? You said there's a few more 1000 scrolls? Are we talking about the ones in Philodemus’s layer or are we talking about the ones in other layers?Nat Friedman You think if you find this crazy Villa that was owned by Julius Caesar's father in law, then we just dig the whole thing out. But in fact, most of the exploration occurred in the 1700s, through the Swiss engineer’s underground tunnels. The villa was never dug out and exposed to the air. You went down 50-60 feet and then you dig tunnels. And again, they were looking for treasure, not like a full archaeological exploration. So they mostly got treasure. In the 90s some additional excavations were done at the edge of the villa and they discovered a couple things. First, they discovered that it was a seaside Villa that faced the ocean. It was right on the water before the volcano erupted. The eruption actually pushed the shoreline out by depositing so much additional mud there. So it's no longer right by the ocean, apparently, I've actually never been. And they also found that there were two additional floors in the villa that the tunnels had apparently never excavated. And so at most, a third of the villa has been excavated. Now, they also know when they were discovering these papyrus scrolls that they found basically one little room where most of the scrolls were. And these were mostly these Philodemus texts, at least, that's what we know. And they apparently found several revisions, sometimes of the same text. The hypothesis is this was actually Philodemus’s working library, he worked here, this sort of epicurean philosopher. In the hallways, though, they occasionally found other scrolls, including crates of them. And the belief is, at least this is what historians have told me, and I'm no expert. But what they have told me is they think that the main library in this villa has probably not been excavated. And that the main library may be a Latin library, and may contain literary texts, historical texts, other things, and that it could be much larger. Now, I don't know how prone these classists are to wishful thinking. It is a romantic idea. But they have some evidence in the presence of these partly evacuated scrolls that were found in hallways, and that sort of thing. I've since gone and read a bunch of the firsthand accounts of the excavations. There are these heartbreaking descriptions of them finding like an entire case of scrolls in Latin, and accidentally destroying it as they tried to get it out of the mud and there were maybe 30 scrolls or something in there. There clearly was some other stuff that we just haven't gotten to.Dwarkesh Patel You made some scrolls right?Nat Friedman Yeah. This is a papyrus, and it's a grassy reed that grows on the Nile in Egypt. And for many 1000s of years they've been making paper out of it. And the way they do it is they take the outer rind off of the papyrus and then they cut the inner core into these strips. They lay the strips out parallel to one another and they put another layer to 90 degrees to that bottom layer. And they press it together in a press or under stones and let it dry out. And that's Papyrus, essentially. And then they'll take some of those sheets and glue them together with paste, usually made out of flour, and get a long scroll. You can still buy it, I bought this on Amazon, and it's interesting because it's got a lot of texture. Those fibers, ridges of the papyrus plant, and so when you write on it, you really feel the texture. I got it because I wanted to understand what these artifacts that we're working with. So we made an attempt to simulate carbonizing a few of these. We basically took a Dutch oven, because when you carbonized something and you make charcoal, it's not like burning it with oxygen, you remove the oxygen, heat it up and let it carbonize. We tried to simulate that with a Dutch oven, which is probably imperfect, and left it in the oven at 500 degrees Fahrenheit for maybe five or six hours. These things are incredibly light snd if you try to unfold them, they just fall apart in your hand very readily. I assume these are in somewhat better shape than the ones that were found because these were not in a volcanic eruption and covered in mud. Maybe that mud was hotter than my oven can go. And they’re just flakes, just squeeze it and it’s just dust in your hand. And so we actually tried to replicate many of the heartbreaking 1700s, 18th century unrolling techniques. They used rose water, for example, or they tried to use different oils to soften it and unroll it. And most of them are just very destructive. They poured mercury into it because they thought mercury would slip between the layers potentially. So yeah, this is sort of what they look like. They shrink and they turn to ash.Dwarkesh Patel For those listening, by the way, just imagine the ash of a cigar but blacker and it crumbles the same way. It's just a blistered black piece of rolled up Papyrus.Nat Friedman Yeah. And they blister, the layers can separate. They can fuse. Dwarkesh Patel And so this happened in 79 AD right? So we know that anything before that could be in here, which I guess could include? Nat Friedman Yes. What could be in there? I don't know. You and I have speculated about that, right? It would be extremely exciting not to just get more epicurean philosophy, although that's fine, too. But almost anything would be interesting in additive. The dream is – I think it would maybe have a big impact to find something about early Christianity, like a contemporaneous mention of early Christianity, maybe there'd be something that the church wouldn't want, that would be exciting to me. Maybe there'd be some color detail from someone commenting on Christianity or Jesus, I think that would be a very big deal. We have no such things as far as I know. Other things that would be cool would be old stuff, like even older stuff. There were several scrolls already found there that were hundreds of years old when the villa was buried. As per my understanding the villa was probably constructed about 100 years prior and they can date some of the scrolls from the style of writing. And so there was some old stuff in there. And the Library of Alexandria was burned 80 or 90 years prior. And so, again, maybe wishful thinking, but there's some rumors that some of those scrolls were evacuated and maybe some of them would have ended up at this substantial, prominent, Mediterranean villa. God knows what’ll be in there, that would be really cool. I think it'd be great to find literature, personally I think that would be exciting, like beautiful new poems or stories, we just don't have a ton because so little survived. I think I think that would be fun. You had the best, crazy idea for what could be in there, which was text which was GPT watermarks. That would be a creepy feeling.Dwarkesh Patel I still can't get over just how much of a plot of a sci fi novel this is like. Potentially the biggest intact library from the ancient world that has been sort of stopped like a debugger because of this volcano. The philosophers of antiquity forgotten, the earliest gospels, there's so much interesting stuff there. But let's talk about what the data looks like. So you mentioned that they've been CT scanned, and that they built these machine learning techniques to do segmentation and the unrolling. What would it take to get from there to understand the actual content of what is within?Nat Friedman Dr. Seales actually pioneered this field of what is now widely called virtual unwrapping. And he actually did not do it with these Herculaneum scrolls, these things are like expert mode, they're so difficult. I'll tell you why soon. But he initially did it with a scroll that was found in the Dead Sea in Israel. It's called The En-Gedi scroll and it was carbonized under slightly similar circumstances. I think there was a temple that was burned. The papyrus scroll was in a box. So it kind of is like a Dutch oven, it carbonized in the same way. And so it was not openable. it’d fall apart. So the question was, could you nondestructively read the contents of it? So he did this 3D X-ray, the CT scan of the scroll, and then was able to do two things. First, the ink gave a great X-ray signature. It looked very different from the papyrus, it was high contrast. And then second, he was able to segment the wines of the scroll, you know, throughout the entire body of the scroll, and identify each layer, and then just geometrically unroll it using fairly normal flattening computer vision techniques, and then read the contents. It turned out to be an early part of the book of Leviticus, something of the Old Testament or the Torah. And that was like a landmark achievement. Then the next idea was to apply those same techniques to this case. This has proven hard. There's a couple things that make it difficult, the primary one is that the ink used on the Herculaneum papyri is not very absorbent of X-Ray. It basically seems to be equally absorbent of X ray as the papyrus. Very close, certainly not perfectly. So you don't have this nice bright lettering that shows up on your Tomographic, 3D X-Ray. So you have to somehow develop new techniques for finding the ink in there. That's sort of problem one, and it's been a major challenge. And then the second problem is the scrolls are just really messed up. They were long and tightly wound, highly distorted by the volcanic mud, which not only heated them but partly deformed them. So just the segmentation problem of identifying each of these layers throughout the scroll is doable, but it's hard. Those are a couple of challenges. And then the other challenge, of course, is just getting access to scrolls and taking them to a particle accelerator. So you have to have scroll access and particle accelerator access, and time on those. It's expensive and difficult. Dr. Seales did the hard work of making all that happen. The good news is that in the last couple of months, his lab has demonstrated the ability to actually recognize ink inside these X rays with a convolutional neural network. I look at the X-ray scans and I can't see the ink, at least in any of the renderings that I’ve seen, but the machine learning model can pick up on very subtle patterns in the X-Ray absorption at high resolution inside these volumes in order to identify and we've seen that. So you might ask – Okay, how do you train a model to do that, because you need some kind of ground truth data to train the model? The big insight that they had was to train on broken off fragments of the Papyrus. So as people tried to open these over the years in Italy, they destroyed many of them, but they saved some of the pieces that broke off. And on some of those pieces, you can kind of see lettering. And if you take an infrared image of the fragment, then you can really see the lettering pretty well, in some cases. And so they think it's 930 nanometers, they take this little infrared image, now you've got some ground truth, then you do a CT scan of that broken off fragment, and you try to align it, register it with the image. And then you have data that you can use potentially to train a model. That turned out to work in the case of the fragments. I think this is sort of why now? This is why I think launching this challenge now is the right time, because we have a lot of reasons to believe it can work. In the core techniques, the core pieces have been demonstrated, it just all has to be put together at the scale of these really complicated scrolls. And so yeah, if you can do the segmentation, which is probably a lot of work, maybe there's some way to automate it. And then you can figure out how to apply these models inside the body of a scroll and not just to these fragments, then it seems seems like you could probably read lots of text,Dwarkesh Patel Why did you decide to do it in the form of a prize, rather than just like giving a grant to the team that was already pursuing it, or maybe some other team that wants to take it out? Nat Friedman We talked about that. But I think what we basically concluded was the search space of different ways you could solve this is pretty big. And we just wanted to get it done as quickly as possible. Having a contest means lots of people are going to try lots of things and you know, someone's gonna figure it out quickly. Many eyes may make it shallow as a task. I think that's the main thing. Probably someone could do it but I think this will just be a lot more efficient. And it's fun too. I think it's interesting to do a contest and who knows who will solve it or how? People may not even use machine learning. We think that's the most likely approach for recognizing the ink but they may find some other approach that we haven't thought of.Dwarkesh Patel One question people might have is that you have these visible fragments mapped out. Do we expect them to correspond to the burned off or the ashen carbonized scrolls that you can do machine learning on? Ground truth of one could correspond to the other? Nat Friedman I think that's a very legitimate concern, they're different. When you have a broken off fragment, there's air above the ink. So when you CT scan it, you have kind of ink next to air. Inside of a wrapped scroll, the ink might be next to Papyrus, right? Because it's pushing up against the next layer. And your model may not know what to do with that. So yeah, I think this is one of the challenges and sort of how you take these models that were trained on fragments and translate them to the slightly different environment. But maybe there's parts of the scroll where there is air on the inside and we know that to be true. You can sort of see that here. And so I think it should at least partly work and clever people can probably figure out how to make it completely work?Dwarkesh Patel Yeah. So you said the odds are about 50-50? What makes you think that it can be done?Nat Friedman I think it can be done because we recognized ink from a CT scan on the fragments and I think everything else is probably geometry and computer vision. The scans are very high resolution, they're eight micrometers. If you kind of stood a scroll on an end like this, they're taken in the slices through it. So it's like this in the Z axis from bottom to top there are these slices. And the way they're represented on disk is each slice is a TIFF file. And for the full scrolls, each slice is like 100-something megabytes. So they're quite high resolution. And then if you stack for example, 100 of these, they're eight microns, right? So 100 of these is 0.8 millimeters. Millimeter is pretty small. We think the resolution is good enough, or at least right on the edge of good enough that it should be possible. There's sort of like, seem to be six or eight pixels. For voxels I guess, across an entire layer of papyrus. That's probably enough. And we've also seen with the machine learning models, Dr. Seales, has got some PhD students who have actually demonstrated this at eight microns. So I think that the ink recognition will work. The data is clearly physically in the scrolls, right? The ink was carbonized, the papyrus was carbonized. But not a lot of data actually physically survived. And then the question is – Did the data make it into the scans? And I think that's very likely based on the results that we've seen so far. So I think it's just about a smart person solving this, or a smart group of people, or just a dogged group of people who do a lot of manual work that could also work, or you may have to be smart and dogged. I think that's where most of my uncertainty is, is just whether somebody does it.Dwarkesh Patel Yeah, I mean, if a quarter of a million dollars doesn’t motivate you.Nat Friedman Yeah, I think money is good. There's a lot of money in machine learning these days.Dwarkesh Patel Do we have enough data in the form of scrolls that have been mapped out to be able to train a model if that's the best way to go? Because one question somebody might have is – Listen, if you already have this ground truth, why hasn't Dr. Seales’ team already been able to just train it?Nat Friedman I think they will. I think if we just let them do it, they'll get it solved. It might take a little bit longer because it's not a huge number of people. There is a big space here. But I mean, yeah, if we didn't launch this contest, I'd still think this would get solved. But it might take several years. And I think this way, it's likely to happen this year.Dwarkesh Patel Let's say the prize is solved. Somebody figures out how to do this and we can read the first scroll. You mentioned that these other layers haven't been excavated. How is the world going to react? Let's say we get about one of these mapped. Nat Friedman That's my personal hope for this. I always like to look for sort of these cheap leverage hacks, These moments where you can do a relatively small thing and it creates a.. you kick a pebble and you get an avalanche. The theory is, and Grant shares this theory, if you can read one scroll, and we only have two scanned scrolls, there's hundreds of surviving scrolls, it’s relatively expensive to use to book a particle accelerator. So if you can scan one scroll, and you know it works, and you can generalize the technique out, and it's going to work on these other scrolls, then the money which is probably in the low millions, maybe only $1 million to scan the remaining scrolls, will just arrive. It’s just too sweet of a prize not for that not to happen. And the urgency and kind of return on excavating the rest of the villa, will be incredibly obvious too. Because if there are 1000s more papyrus scrolls in there, and we now have the techniques to read them, then there's golden that mud and it's got to be dug out. It's amazing how little money there is for archaeology. Literally for decades, no one's been digging there. That’s my hope. That this is the catalyst that works, somebody reads it, they get a lot of glory, we all get to feel great. And then the diggers arrive in Herculaneum and they get the rest.Finding points of leverageDwarkesh Patel I wonder if the budget for archaeological movies and games like Uncharted or Indiana Jones is bigger than the actual budget to do real world archaeology. But I was talking to some of the people before this interview, and that's one thing they emphasized is your ability to find these leverage points. For example, with California YIMBY, I don't know the exact amount you seeded it with. But for that amount of money, and for an institution that is that new, it is one of the very few institutions that has had a significant amount of political influence, if you look at the state of YIMBY in California and nationally today. How do you identify these things? There's plenty of people who have money who get into history or get into whatever subject, very few do something about it. How do you figure out where?Nat Friedman I'm a little bit mystified by why people don't do more things too. I don't know, maybe you can tell me why aren’t more people doing things? I think most rich people are boring and they should do more cool things. So I'm hoping that they do that now. I think part of it is I just fundamentally don't believe the world is efficient. So if I see an opportunity to do something, I don't have a reflexive reaction that says – Oh, that must not be a good idea if it were a good idea someone would already be doing it. Like someone must be taking care of housing policy in California, right? Or somebody must be taking care of this or that.So first, I don't have that filter that says the world's efficient don’t bother, someone's probably got it covered. And then the second thing is I have learned to trust my enthusiasm. It gets me in trouble too, but if I get really enthusiastic about something and that enthusiasm persists, I just indulge it. And so I just kind of let myself be impulsive. There's this great image that I found and tweeted which said – we do these things not because they are easy, but because we thought they would be easy. That's frequently what happens. The commitment to do it is impulsive and it's done out of enthusiasm and then you get into it and you're like – oh my god, this is really much harder than we expected. But then you're committed and you're stuck and you're going to have to get it done. I thought this project would be relatively straightforward. I’m going to take the data and put it up. But of course 99% of the work has already been done by Dr. Seales and his team at the University of Kentucky. I am a kind of carpetbagger. I've shown up at the end here and try to do a new piece of it. Dwarkesh Patel The last mile is often the hardest. Nat Friedman Well it turned out to be fractal anyway. All the little bits that you have to get right to do a thing and have it work and I hope we got all of them. So I think that's part of it – just not believing that the world is efficient and then just allowing your enthusiasm to cause you to commit to something that turns out to be a lot of work and really hard. And then you just are stubborn and don't want to fail so you keep at it. I think that's it. Dwarkesh Patel The efficiency point, do you think that's particularly true just of things like California YIMBY or this, where there isn't a direct monetary incentive or... Nat Friedman No. Certain parts of the world are more efficient than others and you can't assume equal levels of inefficiency everywhere. But I'm constantly surprised by how even in areas you expect to be very efficient, there are things that are in plain sight that I see them and others don't. There's lots of stuff I don't see too. I was talking to some traders at a hedge fund recently. I was trying to understand the role secrets play in the success of a hedge fund. The reason I was interested in that is because I think the AI labs are going to enter a new similar dynamic where their secrets are very valuable. If you have a 50% training efficiency improvement and your training runs cost $100 million, that is a $50 million secret that you want to keep. And hedge funds do that kind of thing routinely. So I asked some traders at a very successful hedge fund, if you had your smartest trader get on Twitch for 10 minutes once a month, and on that Twitch stream describe their 30-day-old trading strategies. Not your current ones, but the ones that are a month old. What would that... How would that affect your business after 12 months of doing that? So 12 months, 10 minutes a month, 30-day look back. That’s two hours in a year. And to my shock, they told me about an 80% reduction in their profits. It would have a huge impact. And then I asked – So how long would the look back window have to be before it would have a relatively small effect on your business? And they said 10 years. So that I think is quite strong evidence that the world's not perfectly efficient because these folks make billions of dollars using secrets that could be relayed in an hour or something like that. And yet others don't have them or their secrets wouldn't work. So I think there are different levels of efficiency in the world, but on the whole, our default estimate of how efficient the world is is far too charitable. Dwarkesh Patel On the particular point of AI labs potentially storing secrets, you have this sort of strange norm of different people from different AI labs, not only being friends, but often living together, right? It would be like Oppenheimer living with somebody working on the Russian atomic bomb or something like that. Do you think those norms will persist once the value of the secrets is realized? Nat Friedman Yeah, I was just wondering about that some more today. It seems to be sort of slowing, they seem to be trying to close the valves. But I think there's a lot of things working against them in this regard. So one is that the secrets are relatively simple. Two is that you're coming off this academic norm of publishing and really the entire culture is based on sort of sharing and publishing. Three is, as you said, they all live in group houses, summer in polycules. There's just a lot of intermixing. And then it's all in California. And California is a non-compete state. We don't have non-competes. And so they'd have to change the culture, get everybody their own house, and move to Connecticut and then maybe it'll work. I think ML engineer salaries and compensation packages will probably be adjusted to try to address this because you don't want your secrets walking out the door. There are engineers, Igor Babushkin for example, who has just joined Twitter. Elon hired him to train. I think that's public, is that right? I think it is. Dwarkesh Patel It will be now. Nat Friedman Igor's a really, really great guy and brilliant but he also happens to have trained state-of-the-art models at DeepMind and OpenAI. I don't know whether that's a consideration or how big of an effect that is, but it's the kind of thing that would make sense to value if you think there are valuable secrets that have not yet proliferated. So I think they're going to try to slow it down, publishing has certainly slowed down dramatically already. But I think there's just a long way to go before you're anywhere in hedge fund or Manhattan Project territory, and probably secrets will still have a relatively short half-life. Open Source in AIDwarkesh Patel As somebody who has been involved in open-source your entire life, are you happy that this is the way that AI has turned out, or do you think that this is less than optimal? Nat Friedman Well, I don't know. My opinion has been changing. I have increasing worries about safety issues. Not the hijacked version of safety, but some industrial accident type situations or misuse. We're not in that world and I'm not particularly concerned about it in the short term. But in the long term, I do think there are worlds that we should be a little bit concerned about where bad things happen, although I don't know what to do about them. My belief though is that it is probably better on the whole for more people to get to tinker with and use these models, at least in their current state. For example Georgi Gerganov did a four-bit quantization of the LLama model this weekend and got it inferencing on a M1 or M2. I was very excited and I got that running and it's fun to play with. Now I've got a model that is very good, it's almost GPT-3 quality, and runs on my laptop. I've grown up in this world of tinkerers and open-source folks and the more access you have, the more things you can try. And so I think I do find myself very attracted to that. Dwarkesh Patel That is the scientist and the ideas part of what is being shared, but there's also another part about the actual substance, like the uranium in the atom-bomb analogy. As different sources of data realize how valuable their data is for training newer models, do you think that these things will go harder to scrape? Like Libgen or Archive, are these going to become rate-limited in some way or what are you expecting there? Nat Friedman First, there's so much data on the internet. The two primitives that you need to build models are – You need lots of data. We have that in the form of the internet, we digitized the whole world into the internet. And then you need these GPUs, which we have because of video games. So you take like the internet and video game hardware and smash together and you get machine learning models and they're both commodities. I don't think anyone in the open source world is really going to be data-limited for a long time. There's so much that's out there. Probably people who have proprietary data sets that are readily scrapable have been shutting those down, so get your scraping in now if you need to do it. But that's just on the margin. I still think there's quite a lot that's out there to work with. Look, this is the year of proliferation. This is a week of proliferation. We're going to see four or five major AI announcements this week, new models, new APIs, new platforms, new tools from all the different vendors. In a way they're all looking forwards. My Herculaneum project is looking backwards. I think it's extremely exciting and cool, but it is sort of a funny contrast. Github AcquisitionDwarkesh Patel Before I delve deeper into AI, I do want to talk about GitHub. I think we should start with – You are at Microsoft. And at some point you realize that GitHub is very valuable and worth acquiring. How did you realize that and how did you convince Microsoft to purchase GitHub?Nat Friedman I had started a company called Xamarin together with Miguel de Acaza and Joseph Hill and we built mobile tools and platforms. Microsoft acquired the company in 2016 and I was excited about that. I thought it was great. But to be honest, I didn't actually expect or plan to spend more than a year or so there. But when I got in there, I got exposed to what Satya was doing and just the quality of his leadership team. I was really impressed. And actually, I think I saw him in the first week I was there and he asked me – What do you think we should do at Microsoft? And I said, I think we should buy GitHub. Dwarkesh Patel When would this have been?Nat Friedman This was like my first week. It was like March or April of 2016. Okay. And then he said – Yeah, it's a good idea. We thought about it. I'm not sure we can get away with it or something like that. And then about a year later, I wrote him an email, just a memo, I sort of said – I think it's time to do this. There was some noise that Google was sniffing around. I think that may have been manufactured by the GitHub team. But it was a good catalyst because it was something I thought made a lot of sense for Microsoft to do anyway. And so I wrote an email to Satya, a little memo saying – Hey, I think we should buy GitHub. Here's why. Here's what we should do with it. The basic argument was developers are making IT purchasing decisions now. It used to be the sort of IT thing and now developers are leading that purchase. And it's this sort of major shift in how software products are acquired. Microsoft really was an IT company. It was not a developer company in the way most of its purchases were made. But it was founded as a developer company, right? And so, you know, Microsoft's first product was a programming language. Yeah, I said – Look, the challenge that we have is there's an entire new generation of developers who have no affinity with Microsoft and the largest collection of them is at GitHub. If we acquire this and we do a merely competent job of running it, we can earn the right to be considered by these developers for all the other products that we do. And to my surprise, Satya replied in like six or seven minutes and said, I think this is very good thinking. Let's meet next week or so and talk about it. I ended up at this conference room with him and Amy Hood and Scott Guthrie and Kevin Scott and several other people. And they said – Okay, tell us what you're thinking. And I kind of said a little 20-minute ramble on it. And Satya said – Yeah, I think we should do it. And why don't we run it independently like LinkedIn. Nat, you'll be the CEO. And he said, do you think we can get it for two billion? And I said, we could try. He said Scott will support you on this. Three weeks later, we had a signed term sheet and an announced deal. And then it was an amazing experience for me. I'd been there less than two years. Microsoft was made up of and run by a lot of people who've been there for many years. And they trusted me with this really big project. That made me feel really good, to be trusted and empowered. I had grown up in the open source world so for me to get an opportunity to run Github, it's like getting appointed mayor of your hometown or something like that, it felt cool. And I really wanted to do a good job for developers. And so that's how it happened. Dwarkesh Patel That's actually one of the things I want to ask you about. Often when something succeeds, we kind of think it was inevitable that it would succeed but at the time, I remember that there was a huge amount of skepticism. I would go on Hacker News and the top thing would be the blog posts about how Microsoft's going to mess up GitHub. I guess those concerns have been alleviated throughout the years. But how did you deal with that skepticism and deal with that distrust? Nat Friedman Well, I was really paranoid about it and I really cared about what developers thought. There's always this question about who are you performing for? Who do you actually really care about? Who's the audience in your head that you're trying to do a good job for or impress or earn the respect of whatever it is. And though I love Microsoft and care a lot about Satya and everyone there, I really cared about the developers. I’d grown up in this open source world. And so for me to do a bad job with this central institution and open source would have been a devastating feeling for me. It was very important to me not to. So that was the first thing, just that I cared. And the second thing is that the deal leaked. It was going to be announced on a Monday and it leaked on a Friday. Microsoft's buying GitHub. The whole weekend there were terrible posts online. People saying we’ve got to evacuate GitHub as quickly as possible. And we're like – oh my god, it's terrible. And then Monday, we put the announcement out and we said we're acquiring GitHub. It's going to run as an independent company. And then it said Nat Friedman is going to be CEO. And I had, I don't want to overstate or whatever, but I think a couple people were like – Oh. Nat comes from open source. He spent some time in open source and it's going to be run independently. I don't think they were really that calmed down but at least a few people thought – Oh, maybe I'll give this a few months and just see what happens before I migrate off. And then my first day as CEO after we got the deal closed, at 9 AM the first day, I was in this room and we got on zoom and all the heads of engineering and product. I think maybe they were expecting some kind of longer-term strategy or something but I came in and I said – GitHub had no official feedback mechanism that was publicly available but there were several GitHub repos that the community members had started. Isaac from NPM had started one where he'd just been allowing people to give GitHub feedback. And people had been voting on this stuff for years. And I kind of shared my screen and put that up sorted by votes and said – We're going to pick one thing from this list and fix it by the end of the day and ship that, just one thing. And I think they were like – This is the new CEO strategy? And they were like – I don’t know, you need to do database migrations and can't do that in a day. Then someone's like maybe we can do this. We actually have a half implementation of this. And we eventually found something that we could fix by the end of the day. And what I hope I said was – what we need to show the world is that GitHub cares about developers. Not that it cares about Microsoft. Like if the first thing we did after the acquisition was to add Skype integration, developers would have said – Oh, we're not your priority. You have new priorities now. The idea was just to find ways to make it better for the people who use it and have them see that we cared about that immediately. And so I said, we're going to do this today and then we're going to do it every day for the next 100 days. It was cool because I think it created some really good feedback loops, at least for me. One was, you ship things and then people are like – Oh, hey, I've been wanting to see this fixed for years and now it's fixed. It's a relatively simple thing. So you get this sort of nice dopaminergic feedback loop going there. And then people in the team feel the excitement of shipping stuff. I think GitHub was a company that had a little bit of stage fright about shipping previously and sort of break that static friction and ship a little bit more felt good. And then the other one is just the learning loop. By trying to do lots of small things, I got exposed to things like – Okay, this team is really good. Or this part of the code has a lot of tech debt. Or, hey, we shipped that and it was actually kind of bad. How come that design got out? Whereas if the project had been some six-month thing, I'm not sure my learning would have been quite as quick about the company. There's still things I missed and mistakes I made for sure. But that was part of how I think. No one knows kind of factually whether that made a big difference or not, but I do think that earned some trust. Dwarkesh Patel I mean, most acquisitions don't go well. Not only do they not go as well, but like they don't go well at all, right? As we're seeing in the last few months with a certain one. Why do most acquisitions fail to go well? Nat Friedman Yeah, it is true. Most acquisitions are destructive of value. What is the value of a company? In an innovative industry, the value of the company boils down to its cultural ability to produce new innovations and there is some sensitive harmonic of cultural elements that sets that up to make that possible. And it's quite fragile. So if you take a culture that has achieved some productive harmonic and you put it inside of another culture that's really different, the mismatch of that can destroy the productivity of the company. Maybe one way to think about it is that companies are a little bit fragile. And so when you acquire them, it's relatively easy to break them. I mean, they're also more durable than people think in many cases too. Another version of it is the people who really care, leave. The people who really care about building great products and serving the customers, maybe they don't want to work for the acquirer and the set of people that are really load bearing around the long-term success is small. When they leave or get disempowered, you get very different behaviors. Copilot Origin StoryDwarkesh Patel So I want to go into the story of Co-pilot because until ChatGPT it was the most widely used application of the modern AI models. What are the parts of the story you're willing to share in public? Nat Friedman Yeah, I've talked about this a little bit. GPT-3 came out in May of 2020. I saw it and it really blew my mind. I thought it was amazing. I was CEO of GitHub at that time and I thought – I don't know what, but we've got to build some products with this. And Satya had, at Kevin Scott's urging, already invested in OpenAI a year before GPT-3 came out. This is quite amazing. And he invested like a billion dollars. Dwarkesh Patel By the way, do you know why he knew that OpenAI would be worth investing at that point?Nat Friedman I don't know. Actually, I've never asked him. That's a good question. I think OpenAI had already had some successes that were noticeable and I think, if your Satya and you're running this multi-trillion dollar company, you're trying to execute well and serve your customers but you're always looking for the next gigantic wave that is going to upend the technology industry. It's not just about trying to win cloud. It's – Okay, what comes after cloud? So you have to make some big bets and I think he thought AI could be one. And I think Kevin Scott deserves a lot of credit for really advocating for that aggressively. I think Sam Altman did a good job of building that partnership because he knew that he needed access to the resources of a company like Microsoft to build large-scale AI and eventually AGI. So I think it was some combination of those three people kind of coming together to make it happen. But I still think it was a very prescient bet. I've said that to people and they've said – Well, One billion dollars is not a lot for Microsoft. But there were a lot of other companies that could have spent a billion dollars to do that and did not. And so I still think that deserves a lot of credit. Okay, so GPT-3 comes out. I pinged Sam and Greg Brockman at OpenAI and they're like – Yeah, let's. We've already been experimenting with GPT-3 and derivative models and coding contacts. Let's definitely work on something. And to me, at least, and a few other people, it was not incredibly obvious what the product would be. Now, I think it's trivially obvious – Auto-complete, my gosh. Isn't that what the models do? But at the time my first thought was that it was probably going to be like a Q&A chatbot Stack Overflow type of thing. And so that was actually the first thing we prototyped. We grabbed a couple of engineers, SkyUga, who had come in from acquisition that we'd done, Alex Gravely, and started prototyping. The first prototype was a chatbot. What we discovered was that the demos were fabulous. Every AI product has a fantastic demo. You get this wow moment. It turns to maybe not be a sufficient condition for a product to be good. At the time the models were just not reliable enough, they were not good enough. I ask you a question 25% of the time you give me an incredible answer that I love. 75% of the time your answers are useless and are wrong. It's not a great product experience. And so then we started thinking about code synthesis. Our first attempts at this were actually large chunks of code synthesis, like synthesizing whole function bodies. And we built some tools to do that and put them in the editor. And that also was not really that satisfying. And so the next thing that we tried was to just do simple, small-scale auto-complete with the large models and we used the kind of IntelliSense drop down UI to do that. And that was better, definitely pretty good but the UI was not quite right. And we lost the ability to do this large scale synthesis. We still have that but the UI for that wasn't good. To get a function body synthesized you would hit a key. And then I don't know why this was the idea everyone had at the time, but several people had this idea that it should display multiple options for the function body. And the user would read them and pick the right one. And I think the idea was that we would use that human feedback to improve the model. But that turned out to be a bad experience because first you had to hit a key and explicitly request it. Then you had to wait for it. And then you had to read three different versions of a block of code. Reading one version of a block of code takes some cognitive effort. Doing it three times takes more cognitive effort. And then most often the result of that was like – None of them were good or you didn't know which one to pick. That was also like you're putting a lot of energy and you're not getting a lot out, sort of frustrating. Once we had that sort of single line completion working, I think Alex had the idea of saying we can use the cursor position in the AST to figure out heuristically whether you're at the beginning of a block and the code or not. And if it's not the beginning of a block, just complete a line. If it's the beginning of a block, show in line a full block completion. The number of tokens you request and when you stop gets altered automatically with no user interaction. And then the idea of using this sort of gray text like Gmail had done in the editor. So we got that implemented and it was really only kind of once all those pieces came together and we started using a model that was small enough to be low latency, but big enough to be accurate, that we reached the point where like the median new user loved Co-pilot and wouldn't stop using it. That took four months, five months, of just tinkering and sort of exploring. There were other dead ends that we had along the way. And then it became quite obvious that it was good because we had hundreds of internal users who were GitHub engineers. And I remember the first time I looked at the retention numbers, they were extremely high. It was like 60 plus percent after 30 days from first install. If you installed it, the chance that you were still using it after 30 is over 60 percent. And it's a very intrusive product. It's sort of always popping UI up and so if you don't like it, you will disable it. Indeed, 40 something percent of people did disable it but those are very high retention numbers for like an alpha first version of a product that you're using all day. Then I was just incredibly excited to launch it. And it's improved dramatically since then. Dwarkesh Patel Okay. Sounds very similar to the Gmail story, right? It's an incredibly valuable inside and then maybe it was obvious that it needs to go outside. We'll go back to the AI stuff in a second. But some more GitHub questions. By what point, if ever, will GitHub Profiles replace resumes for programmers? Nat Friedman That's a good question. I think they're a contributing element to how people try to understand a person now. But I don't think they're a definitive resume. We introduced readme’s on profiles when I was there and I was excited about that because I thought it gave people some degree of personalization. Many thousands of people have done that. Yeah, I don't know. There's forces that push in the other direction too on that one where people don't want their activity and skills to be as legible. And there may be some adverse selection as well where the people with the most elite skills, it's rather gauche for them to signal their competence on their profile. There's some weird social dynamics that feed into it too. But I will say I think it effectively has this role for people who are breaking through today. One of the best ways to break through. I know many people who are in this situation. You were born in Argentina. You're a very sharp person but you didn't grow up in a highly connected or prosperous network, family, et cetera. And yet you know you're really capable and you just want to get connected to the most elite part communities in the world. If you're good at programming, you can join open source communities and contribute to them. And you can very quickly accrete a global reputation for your talent, which is legible to many companies and individuals around the world. And suddenly you find yourself getting a job and moving maybe to the US or maybe not moving. You end up at a great start up. I mean, I know a lot of people who deliberately pursued the strategy of building reputation in open source and then got the sail up and the wind catches you and you've got a career. I think it plays that role in that sense. But in other communities like in machine learning research, this is not how it works. There's a thousand people, the reputation is more on Arxiv than it is on GitHub. I don't know if it'll ever be comprehensive.Are there any other industries for which proof of work of this kind will eat more into the way in which people are hired? I think there's a labor market dynamic in software where the really high quality talent is so in demand and the supply is so much less than the demand that it shifts power onto the developers such that they can require of their employers that they be allowed to work in public. And then when they do that, they develop an external reputation which is this asset they can port between companies. If the labor market dynamics weren't like that, if programming well were less economically valuable, companies wouldn't let them do that. They wouldn't let them publish a bunch of stuff publicly and they'd say that's a rule. And that used to be the case, in fact. As software has become more valuable, the leverage of a single super talented developer has gone up and they've been able to demand over the last several decades the ability to work in public. And I think that's not going away. Dwarkesh Patel Other than that, I mean, we talked about this a little bit, but what has been the impact of developers being more empowered in organizations, even ones that are not traditionally IT organizations? Nat Friedman Yeah. I mean, software is kind of magic, right? You can write a for loop and do something a lot of times. And when you build large organizations at scale, one of the things that does surprise you is the degree to which you need to systematize the behavior of the people who are working. When I first was starting companies and building sales teams, I had this wrong idea coming from the world as a programmer that salespeople were hyper aggressive, hyper entrepreneurial, making promises to the customer that the product wouldn't do, and that the main challenge you had with salespeople was like restraining them from going out and aggressively cutting deals that shouldn't be cut. What I discovered is that while it does exist sometimes, the much more common case is that you need to build a systematic sales playbook, which is almost a script that you run on your sales team, where your sales reps know the processing to follow to like exercise this repeatable sales motion and get a deal closed. I just had bad ideas there. I didn't know that that was how the world worked, but software is a way to systematize and scale out a valuable process extremely efficiently. I think the more digitized the world has become, the more valuable software becomes, and the more valuable the developers who can create it become. Dwarkesh Patel Would 25-year-old Nat be surprised with how well open source worked and how pervasive it is? Nat Friedman Yeah, I think that's true. I think we all have this image when we're young that these institutions are these implacable edifices that are evil and all powerful and are able to substantially orchestrate the world with master plans. Sometimes that is a little bit true, but they're very vulnerable to these new ideas and new forces and new communications media and stuff like that. Right now I think our institutions overall look relatively weak. And certainly they're weaker than I thought they were back then. Honestly, I thought Microsoft could stop open source. I thought that was a possibility. They can do some patent move and there's a master plan to ring fence open source in. And, you know, that didn't end up in the case. In fact when Microsoft bought GitHub, we pledged all of our patent portfolio to open source. That was one of the things that we did as part of it. That was a poetic moment for me, having been on the other side of patent discussions in the past, to be a part and be instrumental in Microsoft making that pledge. That was quite crazy. Dwarkesh Patel Oh, that's really interesting. It wasn't that there was some business or strategic reason. More so it was just like an idea whose time had come. Nat Friedman Well, GitHub had made such a pledge. And so I think in part of acquiring GitHub, we had to either try to annul that pledge or sign up to it ourselves. And so there was sort of a moment of a forced choice. But everyone at Microsoft thought it was a good idea too. So in many senses it was a moment whose time had come and the GitHub acquisition was a forcing function.Dwarkesh Patel What do you make of critics of modern open source like Richard Stallman or people who advocate for free software saying that – Well, corporations might advocate for open source because of practical reasons for getting good code. And the real way the software should be made – it should be free and that you can replicate it, you can change it, you can modify it and you can completely view it. And the ethical values about that should be more important than the practical values. What do you make of that critique? Nat Friedman I think those are the things that he wants and the thing that maybe he hasn't updated is that maybe not everyone else wants that. He has this idea that people want freedom from the tyranny of a proprietary intellectual property license. But what people really want is freedom from having to configure their graphics card or sound driver or something like that. They want their computer to work. There are places where freedom is really valuable. But there's always this thing of – I have a prescriptive ideology that I'd like to impose on the world versus this thing of – I will try to develop the best observational model for what people actually want whether I want them to want it or not. And I think Richard is strongly in the former camp.Dwarkesh Patel What is the most underrated license by the way? Nat Friedman I don't know. Maybe the MIT license is still underrated because it's just so simple and bare. Dwarkesh Patel Nadia Eghbal had a book recently where she argued that the key constraint on open source software and on the time of the people who maintain it is the community aspect of software. They have to deal with feature requests and discussions and maintaining for different platforms and things like that. And it wasn't the actual code itself, but rather this sort of extracurricular aspect that was the main constraint. Do you think that is the constraint for open source software? How do you see what is holding back more open source software?Nat Friedman By and large I would say that there is not a problem. Meaning open source software continues to be developed, continues to be broadly used. And there's areas where it works better and areas where it works less well, but it's sort of winning in all the areas where large-scale coordination and editorial control are not necessary. It tends to be great at infrastructure, stand-alone components and very, very horizontal things like operating systems. And it tends to be worse at user experiences and things where you need a sort of dictatorial aesthetic or an editorial control. I've had debates with Dylan Field of Figma, as to why it is that we don't have lots of good open source applications. And I've always thought it had something to do with this governance dynamic of – Gosh, it's such a pain to coordinate with tons of people who all sort of feel like they have a right to try to push the project in one way or another. Whereas in a hierarchical corporation there can be a head of this product or CEO or founder or designer who just says, we're doing it this way. And you can really align things in one direction very, very easily. Dylan has argued to me that it might be because there's just fewer designers, people with good design sense, in open source. I think that might be a contributing factor too, but I think it's still mostly the governance thing. And I think that's what Nadia's pointing at also. You're running a project and you gave it to people for free. For some reason, giving people something for free creates a sense of entitlement. And then they feel like they have the right to demand your time and push things around and give you input and you want to be polite and it's very draining. So I think that where that coordination burden is lower is where open source tends to succeed more. And probably software and other new forms of governance can improve that and expand the territory that open source can succeed in. Dwarkesh Patel Yeah. Theoretically those two things are consistent, right? You could have very tight control over governance while the code itself is open source. Nat Friedman And this happens in programming languages. Languages are eventually set in stone and then advanced by committee. But yeah, certainly you have these benign dictators of languages who enforce the strong set of ideas they have, a vision, master plan. That would be the argument that's most on Dylan's side. Hey, it works for languages why can't it work for end user applications? I think the thing you need to do though to build a good end user application is not only have a good aesthetic and idea, but somehow establish a tight feedback loop with a set of users. Where you can give them – Dwarkesh, try this. Oh my gosh. Okay, that's not what you need. Doing that is so hard, even in a company where you've total hierarchical control of the team in theory and everyone really wants the same thing and everyone's salary and stock options depend on the product being accepted by these users. It still fails many times in that scenario. Then additionally doing that in the context of open source, it's just slightly too hard. Dwarkesh Patel The reason you acquired GitHub, as you said, is that there seems to be complementarity between Microsoft’s and GitHub's missions. And I guess that's been proven out over the last few years. Should there be more of these collaborations and acquisitions? Should there be more tech conglomerates? Would that be good for the system? Nat Friedman I don't know if it's good but yes, it is certainly efficient in many ways. I think we are seeing a collaboration occur because the math is sort of pretty simple. If you are a large company and you have a lot of customers, then the thing that you've achieved is this very expensive and difficult thing of building distribution and relationships with lots of customers. And that is as hard or harder and takes longer and more money than just inventing the product in the first place. So if you can then go and just buy the product for a small amount of money and make it available to all of your customers, then there's often an immediate, really obvious gain from doing that. And so in that sense, like acquisitions make a ton of sense. And I've been surprised that the large companies haven't done many more acquisitions in the past until I got into a big company and started trying to do acquisitions. I saw that there are strong elements of the internal dynamics to make it hard. It's easier to spend $100 million on employees internally to do a project than to spend $100 million to buy a company. The dollars are treated differently. The approval processes are different. The cultural buy-in processes are different. And then to the point of the discussion we had earlier, many acquisitions do fail. And when an acquisition fails, it's somehow louder and more embarrassing than when some new product effort you've spun up doesn't quite work out as well. I think there's lots of internal reasons, some justified and some less so, that they haven't been doing it. But just from an economic point of view, it seemed like it makes sense to see more acquisitions than we've seen. Dwarkesh Patel Well, why did you leave? Nat Friedman As much as I loved Microsoft, and certainly as much as I loved GitHub. I still feel tremendous love for GitHub and everything that it means to the people who use it. I didn't really want to be a part of a giant company anymore. Building CoPilot was an example of this. It wouldn't have been possible without OpenAI and Microsoft and GitHub, but building it also required navigating this really large group of people between Microsoft and OpenAI and GitHub. And you reach a point where you're spending a ton of time on just navigating and coordinating lots of people. I just find that less energizing. Just my enthusiasm for that was not as high. I was torn about it because I truly love GitHub, the product and there was so much more I still knew we could do but I was proud of what we'd done. I miss the team and I miss working on GitHub. It was really an honor for me but it was time for me to go do something. I was always a startup guy. I always liked small teams, and I wanted to go back to a smaller, more nimble environment. Nat.org Dwarkesh Patel Okay, so we'll get to it in a second. But first, I want to ask about nat.org and the list of 300 words there. Which I think is one of the most interesting and very straussian list of 300 words I've seen anywhere. I'm just going to mention some of these and get some of your commentary. You should probably work on raising the ceiling, not the floor. Why? Nat Friedman First, I say probably. But what does it mean to raise the ceiling or the floor? I just observed a lot of projects that set out to raise the floor. Meaning – Gosh. We are fine, but they are not and we need to go help them with our superior prosperity and understanding of their situation. Many of those projects fail. For example, there were a lot of attempts to bring the internet to Africa by large and wealthy tech companies and American universities. I won't say they all had no effect, that's not true, but many of them were far short of successful. There were satellites, there were balloons, there were high altitude drones, there were mesh networks, laptops, that were pursued by all these companies. And by the way, by perfectly well-meaning, incredibly talented people who in some cases did see some success, but overall probably much less than they ever hoped. But if you go to Africa, there is internet now. And the way the internet got there is the technologies that we developed to raise the ceiling in the richest part of the world, which were cell phones and cell towers. In the movie Wall Street from the 80s, he's got that gigantic brick cell phone. That thing cost like 10 grand at the time. That was a ceiling raising technology. It eventually went down the learning curve and became cheap. And the cell towers and cell phones, eventually we've got now hundreds of millions or billions of them in Africa. It was sort of that initially ceiling raising technology and then the sort of force of capitalism that made it work in the end. It was not any Deus Ex Machina technology solution that was intended to kind of raise the floor. There's something about that that's not just an incidental example. But on my website, I say probably. Because there are some examples where people set out to kind of raise the floor and say – No one should ever die of smallpox again. No one should ever die of guinea worm again. And they succeed. I wouldn't want to discourage that from happening but on balance, we have too many attempts to do that. They look good, feel good, sound good, and don't matter. And in some cases, have the opposite of the effect they intend to. Dwarkesh Patel Here's another one and this is under the EMH section. In many cases, it's more accurate to model the world as 500 people than 8 billion. Now here's my question, what are the 8 billion minus 500 people doing? Why are there only 500 people? Nat Friedman I don't know exactly. It's a good question. I ask people that a lot. The more I've done in life, the more I've been mystified by this – Oh, somebody must be doing X. And then you hear there's a few people doing X, then you look into it, they're not actually doing X. They're doing kind of some version of it that's not that. All the best moments in life occur when you find something that to you is totally obvious that clearly somebody must be doing, but no one is doing. Mark Zuckerberg says this about founding Facebook. Surely the big companies will eventually do this and create this social and identity layer on the internet. Microsoft will do this. But no, none of them were. And he did it. So what are they doing? I think the first thing is that many people throughout the world are optimizing local conditions. They're working in their town, their community, they're doing something there so the set of people that are kind of thinking about kind of global conditions is just naturally narrowed by the structure of the economy. That's number one. I think number two is, most people really are quite mimetic. We all are, including me. We get a lot of ideas from other people. Our ideas are not our own. We kind of got them from somebody else. It's kind of copy paste. You have to work really hard not to do that and to be decorrelated. And I think this is even more true today because of the internet. I don't know if Albert Einstein, as a patent clerk, wouldn't he have just been on Twitter just getting the same ideas as everybody else? What do you have as decorrelated ideas? I think the internet has correlated us more. The exception would be really disagreeable people who are just naturally disagreeable. So I think the future belongs to the autists in some sense because they don't care what other people think as much. Those of us on the spectrum in any sense are in that category. Then we have this belief that the world's efficient and it isn't and that's part of it. The other thing is that the world is so fractal and so interesting. Herculaneum papyri, right? It is this corner of the world that I find totally fascinating but I don't have any anticipation that eight billion people should be thinking about that. That should be a priority for everyone. Dwarkesh Patel Okay, here's another one. Large scale engineering projects are more soluble in IQ than they appear. And here's my question, does that make you think that the impact of AI tools like co-pilot will be bigger or smaller because one way to look at co-pilot is it’s IQ is probably less than the average engineer, so maybe it'll have less impact. Nat Friedman Yeah, but it definitely increases the productivity of the average engineer to bring them higher up. And I think it increases the productivity of the best engineers as well. Certainly a lot of people I consider to be the best engineers telling you that they find it increases their productivity a lot. It's really interesting how so much of what's happened in AI has been soft, fictional work. You have Midjourney, you have copywriting, you have Claude from Anthropic is so literary, it writes poetry so well. Except for co-pilot, which is this real hard area where like, the code has to compile, has to be syntactically corrected, has to work and pass the tests. We see the steady improvement curve where now, already on average, more than half of the code is written by co-pilot. I think when it shipped, it was like low 20s. And so it's really improved a lot as the models have gotten better and the prompting has gotten better. But I don't see any reason why that won't be 95%. It seems very likely to me. I don't know what that world looks like. It seems like we might have more special purpose and less general purpose software. Right now we use general purpose tools like spreadsheets and things like this a lot, but part of that has to do with the cost of creating software. And so once you have much cheaper software, do you create more special purpose software? That's a possibility. So every company, just a custom piece of code. Maybe that's the kind of future we're headed towards. So yeah, I think we're going to see enormous amounts of change in software development. Dwarkesh Patel Another one – The cultural prohibition on micromanagement is harmful, great individuals should be fully empowered to exercise their judgment. And the rebuttal to this is if you micromanage you prevent people from learning and to develop their own judgment. Nat Friedman So imagine you go into some company, they hired Dwarkesh and you do a great job with the first project that they give you. Everyone's really impressed. Man, Dwarkesh, he made the right decisions, he worked really hard, he figured out exactly what needed to be done and he did it extremely well. Over time you get promoted into positions of greater authority and the reason the company's doing this is they want you to do that again, but at bigger scale, right? Do it again, but 10 times bigger. The whole product instead of part of the product or 10 products instead of one. The company is telling you, you have great judgment and we want you to exercise that at a greater scale. Meanwhile, the culture is telling you as you get promoted, you should suspend your judgment more and more and defer your judgment to your team. And so there's some equilibrium there and I think we're just out of equilibrium right now where the cultural prohibition is too strong. I don't know if this is true or not, but maybe in the 80s I would have felt the other side of this. That we have too much micromanagement. I think the other problem that people have is that they don't like micromanagement because they don't want bad managers to micromanage, right? So you have some bad managers, they have no expertise in the area, they're just people managers and they're starting to micromanage something that they don't understand where their judgment is bad. And my answer to that is stop empowering bad managers. Don't have them, promote and empower people who have great judgment and do understand the subject matter that they're working on. If I work for you and I just know you have better judgment and you come in and you say, now like you're launching the scroll thing and you think you've got the final format wrong, here's how you should do it, I would welcome that even though it's micromanagement because it's going to make us more successful in them and learn something from tha. I know your judgment is better than mine in this case or at least we're going to have a conversation about it, we're both going to get smarter. So I think on balance, yeah, there are cases where people have excellent judgment and we should encourage them to exercise it and sometimes, things will go wrong when you do that, but on balance you will get far more excellence out of it and we should empower individuals who have great judgment. Dwarkesh Patel Yeah. There's a quote about Napoleon that if he could have been in every single theater of every single battle he was part of, that he would have never lost a battle. I was talking to somebody who worked with you at GitHub and she emphasized to me, and this is like really remarkable to me, that even the applications are already being shipped out to engineers how much of the actual suggestions and the actual design came from you directly, which is kind of remarkable to me that as CEO you would have. Nat Friedman Yeah, you can probably also find people you can talk to who think that was terrible. But the question is always: does that scale? And the answer is it does not scale. The experience that I had as CEO was I was terrified all the time that there was someone in the company who really knew exactly what to do and had excellent judgment, but because of cultural forces that person wasn't empowered. That person was not allowed to exercise their judgment and make decisions. And so when I would think and talk about this, that was the fear that it was coming from. They were in some consensus environment where their good ideas were getting whittled down by lots of conversations with other people and a politeness and a desire not to micromanage. So we were ending up with some kind of average thing. And I would rather have more high variance outcomes where you either get something that's excellent because it is the expressed vision of a really good auteur or you get a disaster and it didn't work and now you know it didn't work and you can start over. I would rather have those more high variance outcomes and I think it's a worthy trade.Dwarkesh Patel Okay, let's talk about AI. What percentage of the economy is basically text to text? Nat Friedman Yeah, it's a good question. We've done the sort of Bureau of Labor Statistics analysis of this. It's not the majority of the economy or anything like that. We're in the low double digit percentages. The thing that I think is hard to predict is what happens over time as the cost of text to text goes down? I don't know what that's going to do. But yeah, there's plenty of revenue to be got now. One way you can think about it is – Okay, we have all these benchmarks for machine learning models. There's LAMBADA and there's this and there's that. Those are really only useful and only exist because we haven't deployed the models at scale. So we don't have a sense of what they're actually good at. The best metric would probably be something like – What percentage of economic tasks can they do? Or on a gig marketplace like Upwork, for example, what fraction of Upwork jobs can GPT-4 do? I think is sort of an interesting question. My guess is extremely low right now, autonomously. But over time, it will grow. And then the question is, what does that do for Upwork? I’m guessing it’s a five billion dollar GMV marketplace, something like that. Does it grow? Does it become 15 billion or 50 billion? Does it shrink because the cost of text to text tasks goes down? I don't know. My bet would be that we find more and more ways to use text to text to advance progress. So overall, there's a lot more demand for it. I guess we'll see. Dwarkesh Patel At what point does that happen? GPT-3 has been a sort of rounding error in terms of overall economic impact. Does that happen with GPT-4, GPT-5, where we see billions of dollars of usage? Nat Friedman Yeah, I've got early access to GPT-4 and I've gotten to use it a lot. And I honestly can't tell you the answer to that because it's so hard to discover what these things can do that the prior ones couldn't do. I was just talking to someone last night who told me – Oh, GPT-4 is actually really good at Korean and Japanese and GPT-3 is much worse at those. So it's actually a real step change for those languages. And people didn't know how good GPT-3 was until it got instruction tuned for chatGPT and was put out in that format. You can imagine the pre-trained models as a kind of unrefined crude oil and then once they've been kind of RLHF and trained and then put out into the world, people can find the value. Dwarkesh Patel What part of the AI narrative is wrong in the over-optimistic direction? Nat Friedman Probably an over-optimistic case from both the people who are fearful of what will happen, and from people who are expecting great economic benefits is that we're definitely in this realm of diminishing returns from scale. For example GPT-4 is, my guess is, two orders of magnitude more expensive to train the GPT-3, but clearly not two orders of magnitude more capable. Now is it two orders of magnitude more economically valuable? That would also surprise me. When you're in these sigmoids, where you are going up this exponential and then you start to asymptote it, it can be difficult to tell if that's going to happen. The idea that we might not run into hard problems or that scaling will continue to be worth it on a dollar basis are reasons to be a little bit more pessimistic than the people who have high certainty of GDP increasing by 50% per month which I think some people are predicting. But on the whole, I'm very optimistic. You're asking me to like make the bear case for something I'm very bullish about. Dwarkesh Patel No, that's why I asked you to make the bear case because I know about you. I want to ask you about these foundation models. What is the stable equilibrium you think of how many of them will there be? Will it be an oligopoly like Uber and Lyft where…? Nat Friedman I think there will probably be wide-scale proliferation. And if you asked me, what are the structural forces that are pro proliferation and the structural forces that are pro concentration? I think the pro proliferation case is a bit stronger. The pro proliferation case is – They're actually not that hard to train. The best practices will promulgate. You can write them down on a couple sheets of paper. And to the extent that secrets are developed that improve training, those are relatively simple and they get copied around easily. Number one, number two. The data is mostly public, it's mostly data from the internet. Number three, the hardware is mostly commodity and the hardware is improving quickly and getting much more efficient. I think some of these labs potentially have 50, 100, 200 percent training efficiency improvement techniques and so there's just a lot of low-hanging fruit on the technique side of things. We're seeing it happen. I mean, it's happening this weekend, it's happening this year. We're getting a lot of proliferation. The only case against proliferation is that you'll get concentration because of training costs. And I don't know if that's true. I don't have confidence that the trillion dollar model will be much more valuable than the 100 billion dollar model and that even it will be necessary to spend a trillion dollars training it. Maybe there will be so many techniques available for improving efficiency. How much are you willing to spend on researchers to find techniques if you're willing to spend a trillion on training? That's a lot of bounties for new techniques and some smart people are going to take those bounties.Dwarkesh Patel How different will these models be? Will it just be sort of everybody chasing the same exact marginal improvement leading to the same marginal capabilities or will they have entirely different repertoire of skills and abilities? Nat Friedman Right now, back to the mimetic point, they're all pretty similar. Basically the same rough techniques. What's happened is an alien substance has landed on Earth and we are trying to figure out what we can build with it and we're in this multiple overhangs. We have a compute overhang where there's much more compute in the world than is currently being used to train models like much, much more. I think the biggest models are trained on maybe 10,000 GPUs, but there's millions of GPUs. And then we have a capability and technique overhang where there's lots of good ideas that are coming out and we haven't figured out how best to assemble them all together, but that's just a matter of time kind of until people do that. And because many of those capabilities are in the hands of the labs, they haven't reached the tinkerers of the world. I think that is where the new – What can this thing actually do? Until you get your hands on it, you don't really know. I think OpenAI themselves were surprised by how explosively chat GPT has grown. I don't think they put chatGPT out expecting that to be the big announcement. I think they thought GPT-4 was going to be their big announcement. Iit still probably is and will be big, but the chatGPT really surprised them. It's hard to predict what people will do with it and what they'll find valuable and what works. So you need tinkerers. So it goes from hardware to researchers to tinkerers to products. That's the pipe, that's the cascade. Dwarkesh Patel When I was scheduling my interview with Ilya, it was originally supposed to be around the time that chatGPT came out and so their comm’s person tells me – Listen, just so you know, this interview would be scheduled around the time. We're going to make a minor announcement. It's not the thing you're thinking, it's not GPT-4, but it's just like a minor thing. They didn't expect what it ended up being. Have incumbents gotten smarter than before? It seems like Microsoft was able to integrate this new technology role. Nat Friedman There's two, there's been two really big shifts in the way incumbents behave in the last 20 years that I've seen. The first is, it used to be that incumbents got disrupted by startups all the time. You have example after example of this in the mini-computer, micro-computer era, et cetera. And then Clay Christensen wrote The Innovator's Dilemma. And I think what happened was that everyone read it and they said – Oh, disruption is this thing that occurs and we have this innovator's dilemma where we get disrupted because the new thing is cheaper and we can't let that happen. And they became determined not to let that happen and they mostly learned how to avoid it. They learned that you have to be willing to do some cannibalization and you have to be willing to set up separate sales channels for the new thing and so forth. We've had a lot of stability in incumbents for the last 15 years or so. I think that's maybe why. That's my theory. So that's the first major step change. And then the second one is – man, they are paying a ton of attention to AI. If you look at the prior platform revolutions like cloud, mobile, internet, web, PC, all the incumbents derided the new platform and said – Gosh, like no one's going to use web apps. Everyone will use full desktop apps, rich applications. And so there was always this laughing at the new thing. The iPhones were laughed at by incumbents and that is not happening at all with AI. We may be at peak hype cycle and we're going to enter the trough of despair. I don't think so though, I think people are taking it seriously and every live player CEO is adopting it aggressively in their company. So yeah, I think incumbents have gotten smarter. Questions from TwitterDwarkesh Patel All right. So let me ask you some questions that we got from Twitter. This is former guest and I guess mutual friend Austin Vernon. Nat is one of those people that seems unreasonably effective. What parts of that are innate and what did he have to learn? Nat Friedman It's very nice of Austin to say. I don't know. We talked a little bit about this before, but I think I just have a high willingness to try things and get caught up in new projects and then I don't want to stop doing it. I think I just have a relatively low activation energy to try something and am willing to sort of impulsively jump into stuff and many of those things don't work, but enough of them do that I've been able to accomplish a few things. The other thing I would say, to be honest with you, is that I do not consider myself accomplished or successful. My self-image is that I haven't really done anything of tremendous consequence and I don't feel like I have this giant bed of achievements that I can go to sleep on every night. I think that's truly how I feel. I'm an insecure overachiever, I don't really feel good about myself unless I'm doing good work, but I also have tried to cultivate a forward-looking view where I try not to be incredibly nostalgic about the past. I don't keep lots of trophies or anything like that. Go into some people's offices and it's like things on the wall and trophies of all the things they've accomplished and I'd always seemed really icky to me. Just had a sort of revulsion to that. Dwarkesh Patel Is that why you took down your blog? Nat Friedman Yeah. I just wanted to move forward. Dwarkesh Patel Simian asks for your takes on alignment. “He seems to invest both in capabilities and alignment which is the best move under a very small set of beliefs.” So he's curious to hear the reasoning there. Nat Friedman I guess we'll see but I'm not sure capabilities and alignment end up being these opposing forces. It may be that the capabilities are very important for alignment. Maybe alignment is very important for capabilities. I think a lot of people believe, and I think I'm included in this, that AI can have tremendous benefits, but that there's like a small chance of really bad outcomes. Maybe some people think it's a large chance. The solutions, if they exist, are likely to be technical. There's probably some combination of technical and prescriptive. It's probably a piece of code and a readme file. It says – if you want to build aligned AIs, use this code and don't do this or something like that. I think that's really important and more people should try to actually build technical solutions. I think one of the big things that's missing that perplexes me is, there's no open source technical alignment community. There's no one actually just implementing in open source, the best alignment tools. There's a lot of philosophizing and talking, and then there's a lot of behind closed doors, interpretability and alignment work. Because the alignment people have this belief that they shouldn't release their work I think we're going to end up in a world where there's a lot of open source, pure capabilities work, and no open source alignment work for a little while. Hopefully that'll change. So yeah, I wanted to, on the margin, invest in people doing alignment. It seems like that's important. I thought Sydney was a kind of an example of this. You had Microsoft essentially released an unaligned AI and I think the world sort of said – Hmm, sort of threatening its users, that seems a little bit strange. If Microsoft can't put a leash on this thing, who can? I think there'll be more interest in it and I hope there's open communities. Dwarkesh Patel That was so endearing for some reason. Threatening you just made it so much more lovable for some reason. Nat Friedman Yeah, I think it's like the only reason it wasn't scary is because it wasn't hooked up to anything. If it was hooked up to HR systems or if it could like post jobs or something like that, then I don't know, like to get on a gig worker site or something. I think it could have been scary. Dwarkesh Patel Yep. Final question from Twitter. Will asks “What historical personality seems like the most kindred spirit to you”. Bookshelves are all around us in this room, some of them are biographies. Is there one that sticks out to you?Nat Friedman Gosh, good question. I think I'd say it's changed over time. I've been reading Philodemus's work recently. When I grew up Richard Feynman was the character who was curious and plain spoken. Dwarkesh Patel What's next? You said that according to your perception that you still have more accomplishments ahead of you. What does that look like, concretely? Do you know yet? Nat Friedman I don't know. It's a good question. The area I'm paying most attention to is AI. I think we finally have people building the products and that's going to just accelerate. I'm going to pay attention to AI and look for areas where I can contribute. Dwarkesh Patel Awesome. Okay. Nat this was a true pleasure. Thanks for coming on the podcast. Nat Friedman Thanks for having me.  Get full access to The Lunar Society at www.dwarkeshpatel.com/subscribe
3/22/20231 hour, 38 minutes, 23 seconds
Episode Artwork

Brett Harrison - FTX US Former President & HFT Veteran Speaks Out

I flew out to Chicago to interview Brett Harrison, who is the former President of FTX US President and founder of Architect.In his first longform interview since the fall of FTX, he speak in great detail about his entire tenure there and about SBF’s dysfunctional leadership. He talks about how the inner circle of Gary Wang, Nishad Singh, and SBF mismanaged the company, controlled the codebase, got distracted by media, and even threatened him for his letter of resignation.In what was my favorite part of the interview, we also discuss his insights about the financial system from his decades of experience in the world's largest HFT firms.And we talk about Brett's new startup, Architect, as well as the general state of crypto post-FTX.Watch on YouTube. Listen on Apple Podcasts, Spotify, or any other podcast platform. Read the full transcript here. Follow me on Twitter for updates on future episodes.Similar episodesSide note: Paying the billsTo help pay the bills for my podcast, I've turned on paid subscriptions on Substack.No major content will be paywalled - please don't donate if you have to think twice before buying a cup of coffee.But if you have the means & have enjoyed my podcast, I would appreciate your support 🙏.As always, the most helpful thing you can do is just to share the podcast - send it to friends, group chats, Twitter, Reddit, forums, and wherever else men and women of fine taste congregate.Timestamps(0:00:00) - Passive investing & HFT hacks(0:08:30) - Is Finance Zero-Sum?(0:18:38) - Interstellar Markets & Periodic Auctions(0:23:10) - Hiring & Programming at Jane Street (0:32:09) - Quant Culture(0:42:10) - FTX - Meeting Sam, Joining FTX US(0:58:20) - FTX - Accomplishments, Beginnings of Trouble(1:08:11) - FTX - SBF's Dysfunctional Leadership(1:26:53) - FTX - Alameda(1:33:50) - FTX - Leaving FTX, SBF"s Threats(1:45:45) - FTX - Collapse(1:53:10) - FTX - Lessons(2:04:34) - FTX - Regulators, & FTX Mafia(2:15:42) - Architect.xyz(2:30:10) - Institutional Interest & Uses of CryptoTranscriptThis transcript was autogenerated and thus may contain errors.Dwarkesh PatelOkay. Today I have the pleasure of speaking with Brett Harrison, who is now the founder of Architect, which provides traders with infrastructure for accessing digital markets. Before that he was the president of FTX US, and before that he was the head of ETF technology at Citadel. And he has a large amount of experience in leadership positions in finance and tech. So this is going to be a very interesting conversation. Thanks for coming on the Lunar Society, Brett.Brett HarrisonYeah. Thanks for coming out to Chicago.Dwarkesh PatelYeah, my pleasure. My pleasure. Is the growth of ETFs a good thing for the health of markets? There's one view that as there's more passive investing, you're kind of diluting the power of smart money. And in fact, what these active investors are doing with their fees is subsidizing the price discovery that makes markets efficient. And with passive investing, you're sort of free writing off of that. You were head of ETF technology at Citadel, so you're the perfect person to ask this. Is it bad that there's so much passive investing?Brett HarrisonI think on that it's good. I think that most investors in the market shouldn't be trying to pick individual stock names. And the best thing people can do is invest in sort of diversified instruments. And it is far less expensive to invest in indices now than it ever was in history because of the advent of ETFs.Dwarkesh PatelYeah. So maybe it's good for individual investors to put their money in passive investments. But what about the health of the market as a whole? Is it hampered by how much money goes into passive investments?Brett HarrisonIt's hard to be able to tell what it would look like if there was less money in passive investment. Now, I do think one of the potential downsides is ending up creating extra correlated activity between instruments purely by virtue of them being included in index products. So when Tesla gets added to the Sfp 500, tesla doesn't suddenly become a different company whose market value is fundamentally changing, but yet it's going to start moving very differently in terms of its beta correlation between other instruments in SP 5100, purely as a function of all the passive investing that moves these instruments in the same direction. So that's the sense in which I think it could be detrimental naively.Dwarkesh PatelYou would assume that efficient market hypothesis would say that if people know that Tesla's stock price would irrationally climb whence including the S&P 500, then people would short it and then there should be no impact from this irrelevant information. Why isn't that the case?Brett HarrisonIt probably mostly is. I think that sometimes there can be liquidity differences that cause at least temporary dislocations and stocks. The simplest example is like you have an ADR like an American Depository receipt that's sort of expunging for some underlying foreign stock and these two things should be like almost the same value at all times, like net of currency conversion and conversion ratios. But if one of the markets is highly illiquid or difficult to access, then there's going to be dislocations in price. And that's like the job of the Jane Streets, the world to kind of arbitrage away the price over time and so long run you wouldn't expect these things to be dislocated for that long. So I'm sure there are people who are understanding the fundamentals of individual names in the S P 500 and when there's like American News and the entire S and P falls, they are maybe buying S and P and selling individual names and expecting that relative value spread to come in over time.Dwarkesh PatelSpeaking of by the way, these firms, you don't have to tell me specifics, but how similar are the strategies for market making or trading that Jean Street versus Citadelo and these firms, is it the same sorts of strategies or are they pretty different?Brett HarrisonI think a lot more differences than people appreciate from the outside. Different companies have established different niches and areas like Jane Street established its early niche and ETF at kind of like a mid frequency level. So not like ultra fast but not like long term year long discretionary macro. Whereas maybe your Citadel securities kind of firm built their niche more on lower latency options, market making so it could be all over the place. There are some where they are trying to optimize for really short term like microstructure. Alpha is trying to predict where the order book is going to move over the course of anywhere from milliseconds to seconds. There are firms that care more about the relative convergence of instruments over the course of hours to days. There's sophisticated quantity of trading firms that are doing longer term days to weeks to months long trades too. A lot of the infrastructure can be similar. Like either way you need to be able to connect to exchanges, download market data, establish simulation platforms, build tools for traders to be able to grasp what's going on in the market and especially be able to visualize their own proprietary models and Alphas. But beyond that, the actual strategies and the ways they make money can be very different.Dwarkesh PatelFamously, in other kinds of development, there's these like, very famous hacks and algorithms, right? So in gaming and graphics, John Carmack has the famous fastener square root for doing graphics calculations, normalizing vectors faster. You were not only a developer in finance. I know what the exact term is for that. But you led teams of hundreds of people who are doing that kind of development. Are there famous examples like this in finance, the equivalent of fastener square root, but for the kinds of calculations you guys do?Brett HarrisonYeah, they're all over the place. There's tons of hacks and tricks and things like that. I think, for example, here's a famous one, not famous, I think I read it in a paper and like a bunch of other developers from different other companies told me about this. It's not something I saw at places that I worked. But if you're sending a message to, let's say, Nasdaq to buy stock and you want to get there as fast as possible, well, what is a message to Nasdaq? It's a TCP IP wrapped message with a particular proprietary protocol that Nasdaq implements. Well, let's say your goal is, you know, you're going to trade Apple, but you're not sure, like, what price and at what time. And you're kind of waiting for some signal to buy Apple as fast as possible. So what you can do is you can preconstruct the entire TCP IP message. Like, first put the TCP header on there, then the IP header, then, like, the kind of outer protocol that Nasdaq specifies and the inner protocol except for the byte slot where you put in the price and then pre. Load that message into the network cards sending buffer so that once you're ready to send, you can just pop in the price and send it off and incur as little latency as possible.Dwarkesh PatelThat's awesome.Brett HarrisonI think the analogy to video games is a good one because just like in video game graphics, what's the end goal? It's not like to produce the most theoretically perfect simulation of environmental graphics. It's to have something that looks good enough and is fast enough for the user. And that's also true in hft and quantitative finance, where the goal is to get to the approximately right trade as fast as you can, it's not to have the perfect theoretical model of underlying.Dwarkesh PatelPrice dynamics that is so fascinating. But this actually raises an interesting question. If you have some sort of algorithm like this that gets you a few nanoseconds faster to the Nasdaq Exchange, and that's why you have Edge, or you've leased microwave towers to get from New Jersey to Chicago faster, or you've bought an expensive server in the same place that Nasdaq is housed. What fundamentally is the advantage to society as a whole from us getting that sort of information faster? Is this just sort of a zero sum game of who can get that incorporate that signal faster. Like, why is it good for society that so much, so many resources and so much brain power is spent on these kinds of hacks and these kinds of optimizations?Brett HarrisonYeah. So I think if you start from the premise that having liquid, tight, efficient markets is important for the world and you say, like, how do I design a system that optimizes for that? I think you want smart, sophisticated technologies competing at the margins. And of course, the more they compete, the smaller the margins become to the point where you think, like, the little extra activity people are doing to get slightly better don't seem to be greatly affecting the whole system as much as if as it was in the earlier days when things were slower and tick sizes were wider. I think it's difficult to imagine designing a market where you say, like, okay, everyone should innovate up until this point and then stop competing and then just stay stasis. And maybe you can create certain regulatory or market structures to try to prevent that. But I think on average you want people competing at the margins, even if they seem like they are minuscule. But at the same time, I think it's not zero sum for society for technologists to be creating super fast, ultralow latency, very sophisticated algorithms. Like maybe, I don't know, we have a lot of geopolitical instability in the world. Who knows if our microwave network that we built out in the US. Could have greater use cases than just for quantitative finance? But the quantitative finance subsidized the creation of these towers.Dwarkesh PatelOkay? So that's sort of like a contingent potential benefit. People tell us a little story about NASA, right, in this case, literally microwaves that they subsidize a lot of the science that ended up becoming becoming into products. So that's an interesting account of the benefits of finance, that it has the same yeah, whatever tricks they come up with might be useful elsewhere. But that's not a story about how it's directly useful to have nanosecond level latency for filing your Apple stock or something like that. Why is that useful directly?Brett HarrisonI mean, if there is some kind of news that happens in one part of the world and that should affect the current price of stock in a different part of the world. I think that if you care about efficient markets, you want the gap between source of truth events and ultimate price discovery to be as small as possible. I think if you believe if you want to question whether getting a few extra milliseconds or microseconds or nanoseconds is worth it, I think you're then putting some kind of value judgments on.Brett HarrisonWhat.Brett HarrisonIs the optimal time it takes to get from to price discovery and saying a second is too slow, but a millisecond is too fast, or a millisecond is too slow, but a microsecond is too fast, and I just don't think we're in a position to do that. I think we kind of always want as close to instantaneous price discovery as possible.Dwarkesh PatelI'm only asking more about this because this is really interesting to me. There are some level of resources where we would say that at this point it's not worth it, right? Let's say $5 trillion a year was spent on getting it down from like, two nanoseconds to like, 1. Know that's probably not a realistic number, but just like there is some margin at which for some weird reason that there's society or just spend so many resources on it, would you say that we haven't reached that margin yet where it's not socially useful? The amount of brain power and resources that are spent on getting these tight.Brett HarrisonSpreads I don't know how large a percentage of GDP prop trading is. I suspect it's not that large. So I don't think we're close to that theoretical limit of where I would start to feel that it's a waste. But I also think there's a reason why they're willing to spend the money on this kind of technology because they're obviously profiting from doing so and it has to come from somewhere. So somehow the market is subsidizing the creation of this technology, which means that there's still ability for value capture, which means there's still a service that's being provided in exchange for some kind of profit. I think we wouldn't spend $5 trillion in a microwave network because there isn't $5 trillion of extra value to be created in doing so.Dwarkesh PatelGot it. Has being a market maker change your views about civilizational tail risk because you're worried about personally getting run over, right, by some sort of weird event and adverse selection? Does that change how we think about societies getting run over by a similar thing, or are the mental models isolated?Brett HarrisonSo I think working in high speed finance teaches you to understand how to more correctly estimate the probability of rare events. And in that sense, working in finance makes me think more about the likelihood of civilization ending problems. But it doesn't suggest to me sort of different solutions. There's a very big difference being in a financial setting where your positions are.Brett HarrisonNumbers that you can put in a.Brett HarrisonSpreadsheet and you can model. Like, what happens if every single position goes against me three X the wrong way? And what instruments would I have to buy or sell in order to be able to hedge that portfolio? That's like a closed system that you can actually model and do something about. Having, like, a trigger mentality on future pandemics I don't think helps you much. I think maybe it slightly changes your ability to kind of estimate the probability of such events. But the actual solutions to these problems are a combination of, like, collective action problems plus being able to sort of model the particular type of unknown unknown about whatever the event is. And I think those kinds of solutions should be left to the experts in those particular fields and not off the traders. In other words, I don't think, like, having the trader mentality among rare events in, like, normal civilization outside of finance really kind of helps you much.Brett HarrisonAnd maybe in some ways it's led.Brett HarrisonPeople to think more hubristically that they can do something about it.Dwarkesh PatelGee, who could be talking about that's?Brett HarrisonReally interesting.Dwarkesh PatelYou would say that famously, these market making firms really care about having their employees be well calibrated and good at sort of thinking about risk. I'm surprised you think that the transfer between thinking about that in financial context and thinking about that in other contexts is that low.Brett HarrisonYeah.Brett HarrisonAgain, I think it helps you at estimating probability of rare events, but it does not translate super well to what action, then, do you take in the face of knowing those rare events?Dwarkesh PatelWere your circles or people in finance earlier to recognize the dangers of COVID.Brett HarrisonThat's a good question. I think that people in my circles were quicker to take action in the face of knowing about COVID There are a lot of people who kind of stuck around in cities and their existing particular situations not knowing kind of where this was going to head long term. And I think if you have the fortune of having the financial flexibility to be able to do something like this, a lot of the people in kind of financial circles kind of immediately recognize, okay, there's this big risk.Brett HarrisonThis unknown and I don't want to.Brett HarrisonGet out of or selected against in terms of being able to get out of the locus of bad pandemic activity. So people immediately were fleeing cities, I think, faster than other people.Dwarkesh PatelThat seems to point in the opposite direction of them not being able to estimate and deal with geopolitical risk.Brett HarrisonWell, I mean, there you have an actual event that has occurred, and then in the face of the event, what do you do right now? Yeah, I think that's different than, like, what do we do about the potential for AI to destroy civilization in the next hundreds of years? Or what do we do about the next potential biological weapon or the next pandemic that could occur.Dwarkesh PatelSpeaking of COVID you were head of semi systemic technology at Citadel when COVID hood, right?Brett HarrisonYes, exactly.Dwarkesh PatelHow did these Hft firms react to COVID? What was it like during COVID Because obviously the market moved a lot, but on the inside, was it good, bad?Brett HarrisonYeah. All the companies, Citadel securities, but really all of the ones in this sort of this finance year, I think were extremely resilient. I think a lot of them found that their preexisting ideas that in order for the team to succeed, everyone needed to be the exact same place. And it was very important from like an IP perspective to make sure that people weren't taking a lot of this work home with them completely went out the window. And people had to completely adjust to the idea that actual trading teams that are used to be able to have eye contact with each other at all times need to adjust to this pandemic world. And they largely did. I think at least from a profitability perspective. It was some of the best years of Hft firms PnLs in recent history.Dwarkesh PatelMatching engines already have to deal with the fact that you can have orders coming from, like, Illinois, you can have orders coming from Japan, and given light speed, they're not going to run at the same time, but you still kind of have to work around that. Is there any hope of a single market and matching engine for once humanity goes interplanetary or interstellar? Could we ever have a market between, like, us and Alpha Centauri or even Austin Mars? Or is the lag too much for that to be possible?Brett HarrisonYeah, without making any changes to a matching engine. There is nothing that says that when an order comes in, it can't be older than X time.Brett HarrisonRight.Brett HarrisonWhat it does mean is that the actual sender, they're sending a market order from halfway across the world, by the time that the order reaches the exchange, they might end up with a very different price than the one that they were expecting when they sent it. And therefore there's probably a lot of adverse selection sending a market order from halfway across the world than in a colocation facility. So you can technologically run an interstellar exchange. This might not be good for that person living on the moon.Dwarkesh PatelIs there any way to make it more fair?Brett HarrisonYeah, so I think there's actually kind of real world analog of that which is like automated market makers on slow blockchains. Because if you're used to working on Nasdaq where Nasdaq processes like a single message in somewhere between like tens and hundreds of nanoseconds per order a blockchain, like ethereum processes what, like 15 to 50 messages per second, so significantly slower by numbers of orders of magnitude. And yet they've been able to establish pretty mature financial marketplaces by saying that rather than you having to send orders with prices on them and then cancel them when the prices aren't good anymore, there will be kind of an automated function that moves the prices at the matching engine. And so whenever your order reaches the exchange, it'll always be kind of a predetermined fair price based on the kind of prevailing liquidity at the time. So one can imagine building a Nasdaq for interstellar market is kind of similar to building uniswap now on a theory of in terms of order, magnitude and speed. But there's other things you can do, too, like you could establish like, periodic auctions instead of like continuous matching and things like that. And that could potentially help mitigate some of these issues.Dwarkesh PatelYes, that's something else I want to ask you about. What do you what is your opinion of periodic frequent batch auction systems? Should we have more of that instead of so?Brett HarrisonIn theory, they help mitigate the advantages of high frequency trading. Because if you know there's going to be an auction every 30 seconds, and it's not going to be by time priority, it's going you buy price, then it doesn't matter if you send that order at the beginning of the 32nd period or the end of 32nd period. It's really the price that determines that you get filled. Not something to do with particular latency to the exchange. I think in practice, the couple of exchanges around the world that used to have those have switched away from them. I think the, like, Taiwan Stock Exchange used to have a periodic auction system. And I just thought the like, the liquidity and price discovery wasn't good and it was like, complained about a lot and they eventually moved off of it to a continuous matching system. So I guess in practice it doesn't quite work as well. But it's hard to tell.Dwarkesh PatelIt's really hard to tell what country, meaning your long experience of dribbling financial infrastructure, what country do you feel has the best infrastructure and set up for good markets?Brett HarrisonI would say the United States, except what's happened is US. Companies like Nasdaq have licensed their exchange matching engine technology to other exchanges around the world. So Nasdaq OMX technology powers a number of the exchanges in Europe and some in Asia. So it's hard to sort of say that. Is the technology American? I guess so. I'm not sure exactly who wrote a lot of the stuff underneath the Nasdaq technology, but I do think the US. Markets are some of the most efficient and low, latency and expansive and products that allowed in the world.Dwarkesh PatelHow do adverse selection in trading and hiring differ?Brett HarrisonIn hiring, there are one, many more opportunities for positive selection versus a negative selection usually encounter in finance. And the other thing is that most financial markets are in the US. When you think about trading in general, you're thinking about liquid markets. The hiring market is highly inefficient. Maybe the pipeline of orders from like, Harvard, MIT, Princeton, Yale to Jane Street and Sale securities is like a very liquid pipeline. But there are many, many universities and colleges throughout the country and the world that have extremely talented individuals whose resumes will never end up on your doorstep. So you might end up with a resume from some graduating senior from college who has no internship experience, and your.Brett HarrisonTrader mindset might think, okay, this is.Brett HarrisonTerrible adverse selection, but it actually could be that person. If he's willing to put themselves out there and apply to your company from this relatively unknown university, then that might be the signal that is like the best person in that entire region. And that might be a positive selection. So I think that it's not exactly the same, like, adverse selection dynamics as there is in the traditional trading world.Dwarkesh PatelYeah, yeah, definitely. Especially if you have, like, I guess mission oriented companies have especially a good way of getting rid of adverse selection. Right?Brett HarrisonYeah, exactly. The companies with really strong brands, I mean, that's one of the things we saw at Jane Street was like, I heard stories in the old days of Jane Street that, like, the first resumes from Harvard were like the people were terrible. Like, they couldn't do like basic math and they just were the worst candidates compared to other people that they were able to find. And then they established this brand and this recruiting pipeline and this reputation for having very difficult interviews and for paying people really well and having this amazing work environment that all of a sudden all the people getting through the pipeline from Harvard were like, really, really great. And it wasn't like the quality of students at Harvard changed. There's probably a bell curve there like there is everywhere else. It was just like the positive selection resulting from the branding efforts and the mission driven focus of the company that really brought that positive selected pipeline to them.Dwarkesh PatelThat's really interesting. Should Jane Street replace Okamel with rust?Brett HarrisonNo, because there's too much infrastructure already in Okamo.Dwarkesh PatelYeah, but starting from scratch.Brett HarrisonSo I guess the world is if they could snap their fingers and suddenly replace all their Ocal infrastructure with Rust at zero cost, would it be worth it?Dwarkesh PatelYeah.Brett HarrisonIn that case, I would say yes, because I think that you get a lot of the sort of static typing and compile time safety in Rust that you get from Ocamol. But the base level program that you can write in Rust is much, much faster than one you can write in Okamel because of the way Ocamel is designed, where there's this kind of automatic garbage collection, the worst thing you can do in high speed finance is do any memory allocation that results in garbage collection. And so you have to write very, very careful o camel that almost ends up looking like C in order to end up staying in functional programming land, but not actually creating tons of memory on the heap or the stack that ends up getting collected later.Dwarkesh PatelI guess you've been playing around with the Rust a lot recently, right?Brett HarrisonYeah.Dwarkesh PatelWhat is your impression of the language? You've been enjoying it?Brett HarrisonIt's great and it's come very long way in the last three to five years. I think crypto has something to do with that. It seems to be like one of the languages of choice for people to write blockchains and smart contracts and so there's been enormous amount of open source contribution to Rust. And so comparison when I last looked at it a couple of years ago, it's a lot easier to write really good, sophisticated programs now in Rust and get all of the type safety and the speed that you get, which is like very comparable to C plus plus on the speed side.Dwarkesh PatelWell, when I'm writing programs, they're not large code bases with many people contributing. So what I use for us, it's just like a huge pay. And I just want to do something very simple. Why do I have to put like an arc instead of a box instead of an option? But I can totally understand if you have something where like billions of dollars are at stake.Brett HarrisonThere's definitely a learning curve. I think for basic scripting you want to use something like Python, right? But exactly. If you're writing low latency distributed infrastructure that has to never fail, russ is a pretty good choice.Dwarkesh PatelYeah. Speaking of Jane Street, why does a company pay interns like 16 or seventeen K a month for their summer internships? Is the opportunity cost for one of these smart students that high in the summer?Brett HarrisonThe short answer is yes, but the long answer to why I think they do that is the starting salary for the top people in not just finance but in tech is sort of in this like low to mid six figures now. And you can debate whether you think that's like, the appropriate starting salary for a person with no experience coming out of college or not. That's just sort of the reality is that the talents pool is extremely competitive from the employer side. So if you start with that as like a reasonable salary plus bonus for an employee, I think change rates mentality is these interns who are coming here like they're doing real work, they should be paid like a full time employee, just like prorated for the time when they're actually here. And so that ends up like checking out to be like the right numbers.Dwarkesh PatelWait on net, are interns I mean, forget about the salary, are they actually on net contributing? Given that subtracting away the time of the traders who are training them, maybe.Brett HarrisonIt sort of breaks even if you consider the time to train them. But it's extremely worthwhile, because when those interns come back full time and Jane Street hires a significant percentage of its incoming classes from their internship program that they're already trained, they're ready to go day one. They're almost immediately useful because they had that like, three month period where they got trained and only the ones that really liked it and were good come back. So it's like rather than wait for them to come on site, train people, and maybe half of them aren't good or and half them don't fit with the culture and you kind of don't know what to do with them. The internship program provides a really good place to get on the job training and then only kind of select on both sides for the ones that are.Brett HarrisonThe best.Dwarkesh PatelIs there a free writer problem with internships where if, like, a company like Jane Street puts in the effort to train somebody for three months, they might get some of them to work for them, but they've also trained some people who might work for the competition. And is there some sort of free rider problem there?Brett HarrisonThere for sure is, which is why the companies have to work as hard as possible to make their experience as good as possible, which is like, it's good for the interns. When you go to street, not only do you learn a lot, but they pay you really well. And also you get to visit one of the foreign offices for like a week or something. Also they have all these really fun programs where they bring famous speakers to come to the office and speak to the whole intern class and they have parties and all sorts of stuff. And it adds to like, the experience of thinking like, okay, this is the place I want to work. I don't want to like, take my training and go to the competitor, I want to come to you.Dwarkesh PatelI clearly got in the wrong business with podcast. Why did you pursue finance dev instead of trading? Why did that appeal more to you?Brett HarrisonYeah, so in college I studied computer science in math and I really liked programming, but I think I didn't quite know what a career in programming looked like. I think the conventional wisdom, at least in 2009 when we applying for internships, was like, okay, I'm going to sit in the cubicle and stare at a screen for 16 hours a day. I'm going to be miserable and it's not going to be a very social job. I consider myself like, a pretty social person. And I had a lot of friends who had had these various internships and quantitative finance, mostly from the trading side. And so when eventually I went to Jane Street as an intern, I had kind of like a hybrid summer, like, doing some dev stuff and some trading stuff. And to me I thought, like, okay, the traders are much more, much closer to, like, the real action of the company and like, I want to be a part of that. And so when I joined Jane Street, I was hired as a trader on the ADR desk. And I realized very soon into that that one, no, actually the developers have just as much, if not more impact on the outcome of success of the company. And two, I just like, enjoyed it a lot more and it was just much more up my alley and my training, and so I ended up going that route instead.Dwarkesh PatelI want to ask about the culture at these sorts of places like Acidadelle or Jane Street. I mean, you spend some time in Silicon Valley and around like, traditional sort of like, startup scene as well. What is the main difference between Silicon Valley tech culture versus New York culture?Brett HarrisonSure, yes, I have a ton of personal experience in the Silicon Valley culture or like the tech culture, since I've only really worked at kind of finances my finance places my whole life. But the sense I get is that the kind of New York Chicago quant, finance dev culture is one about extreme Pragmatism. You know what the outcome is. It's to be the most profitable of the strategy. And you kind of try to draw a straight line between what you're doing and that profitability as fast as you can compare it to. I think the Silicon Valley culture is much more about creativity and doing things that are new that no one else has done before. A healthy amount of cross pollination would be good for both where I think a lot of trading firms are doing the exact same thing that all the other trading firms have done and some healthy injection of creativity into. Some of that stuff to maybe think slightly outside the box of, like, as you said earlier, get slightly faster to go to Nasdaq or something. Which is like, okay, might be fine, but it's, like, not that creative. Would be good for those plot terms. At the same time, the sheer approach to Pragmatically getting something done out there and sold and making money would help a lot of Silicon Valley firms that kind of like hang out in this sort of creative land for too long and don't end up getting a product to market.Dwarkesh PatelYeah, I know. Definitely. It seems like there should be one founder from both those cultures, every single startup. It's similar to what you were saying earlier with SBF and Visionaries versus Pragmatist in that context. How conspicuous I mean, you were just mentioning earlier that these traders are making mid six figure salaries to begin with, let alone where they arise over the careers. How conspicuous is their spending and lifestyle? Is it close to Wolf of Wall Street? Is it just walmart T shirts? Where are we talking?Brett HarrisonIt's not a lot closer to Walmart T shirts than it is Wolf of Wall Street. Certainly it is now. Even when I started, it was pretty inconspicuous. I don't think it was that way in the previous decade or two before I joined Finance, I guess. I'm not really sure, but I got the sense that the current culture around inconspicuous consumption is sort of a function of millennial consumption habits where people are focusing a lot more on experiences than having shiny material objects. I think that's had a large effect on kind of like the high earning tech and finance culture that exists today.Dwarkesh PatelWell, I guess are they spending that much money on experiences either? Because how expensive is a flight to Hawaii, right? Even after you subtract that, where is this money going? Are they just saving it?Brett HarrisonMaybe it's not like just a flight to Hawaii, but it's like, bring your ten friends to Hawaii with you or something. Or it's get involved in a charitable organization in a way that someone who is 24 normally wouldn't be able to do, surely by being able to donate a lot.Dwarkesh PatelYeah. What is the social consequence, for lack of a better word, of having a bunch of young, nerdy people, often male, often single, having this extraordinary level of wealth? What impact does it have? I don't know if society is the right word, but what is the broader impact of that class of people?Brett HarrisonI think we'll have to play this out over the next decade or two to really see where this goes. If I'm going to be an optimist about this, I'd like to think that when it was like older, single or married males hoarding a large amount of wealth that for the most part, they kept it to themselves and kind of waited until later in life to do anything with it, and were the kind of people who really, like, saved their same career their whole lives, as opposed to if younger and younger generations are amassing wealth through what they can actually perform with their skills, then I think that hopefully injects more Dynamism into the distribution of that wealth later on because those millennials within like or Gen Z or whoever will go on to found new companies, and maybe they'll be able to see the company themselves, their own money, and have a lot easier time, like bringing, like, interesting new things to market. Or they'll be able to donate to, like, really interesting causes, or they'll be able to, you know, help out their friends and family more easily from a young age. Or they'll be more selective in the kinds of things that they give to or contribute to that don't just involve getting their name on, like, a building of a school or something.Dwarkesh PatelYeah, that's a very optimistic story. I hope that's the way it plays out. So tell me about the psychology of being a quant or a trader or a developer in that space, because you're responsible. Like, one wrong piece stroke and you've lost millions of dollars, one bug in your code. And there are historical cases of this where entire firms go down because of a bug. What is the sort of day to day psychological impact of that kind of responsibility?Brett HarrisonMaybe the job selects for the people who don't kind of crumble under the theoretical stress of that job. But personally, I don't lose sleep overnight at night over that. Because within any mature financial institution, like a trading firm, there are typically many layers of safeguards in place, like limits on how many dollars you can trade in a minute and how much you can trade overall or for your desk or, like, how many messages you can send to the exchange. And then there's, like, limits on, like, the individual trader and desk level and firm level and there's layers of different checks. Often there are actual rules, like regulatory rules to comply with and market access checks. Like FINRA is like 15 C, three five. And so when you're writing new code, it's not like a completely blank slate thing where you're connecting directly to an exchange and hoping for the best. Usually you're embedding some piece of code within some very large established framework where the goal is to make something trader proof. No matter what some trader clicks on or does or configures with their system, there's like a limit to how badly they can go, how badly they can actually go. And so, especially in my particular role as a developer, actually being able to understand the technological stack and say like, oh, I can tell and can sort of verify that these particular safeguards in place and it is actually as trader proof as I think it is, I can sleep at night knowing nothing too bad is going to happen. I mean, the times I actually lose sleep are like, a trader in London or Hong Kong calls me in the middle of the night to say, like, hey, can you explain how this thing works? I need your help. Those are the times where I actually lose sleep. But it's not over, like, being concerned about risk.Dwarkesh PatelYeah, that's interesting. If you ask the people who work in these firms, what is the social value you're creating separate from the question of what the correct answer to that question is? Would the majority of them say that I'm doing something really valuable? Would they say I'm indifferent to it, but it's earning me a lot of money? What is their understanding of the value they're creating?Brett HarrisonIt really depends on the company, and it depends how diffused the culture is at older firms that have fewer people impacting the culture on any significant way. I think you might not get a clear answer on this thing for a place like Jane Street, where the firm is really run by, like, 30 or so, you know, partners and senior employees who have, like, been there. For a really long time and have carried through the core culture of that company up until the present day. And with that large number of people at the top in a very flat environment have actually been able to propagate that culture and maintain it throughout the company. I think you'll find a much more kind of homogeneous view on their social value, which I think they would say is that they provide the best pricing and access to. Markets that are critical for facilitating capital allocation throughout the world and allow people to very efficiently invest in vehicles that are global in nature.Dwarkesh PatelThat seems very abstract. And while it is probably very well correct and is very valuable for society, it might not seem that, like, tangible to somebody who's working in that space. Is there some technique that these firms have of making visceral the impact these traders have? I don't know, do they bring out some child who benefit from efficient markets or something?Brett HarrisonI think, well, probably not like children. I think it's more like anecdotes about, like the pension like fund behind this, like, state government needed to get exposure to some diversified asset class and came to one of these companies and said, we want to move like a $5 billion portfolio. Can you help us do it in an efficient way? And it ends up saving them, like significant numbers of percent or basis points over what would happen if they went to the market. And you can say, well, there's like a direct connection between the price that someone like Jane Street gives them and the amount of money that they ultimately get to save and ultimately pass on to the people in their state who are part of their pension plan. And so it feels like a direct connection there.Dwarkesh PatelOkay, let's start by addressing the elephant in the room, which is FTX. Let's begin at Jane Street, which is where you met SBF. Can you tell us sort of the origin story of how you first met him and what your first impressions were?Brett HarrisonYeah, absolutely.Brett HarrisonSo I was at Jane Street from 2010 to 2018. Sam was at Jane Street for a couple of years in the middle of that, I think 2013 to 2017. And one of the things I did at Jane Street was I started this program called Okamel Bootcamp. It was a yearly course for the new trader hires to spend four weeks with me learning programming in Okamel, which was like the esoteric programming language that we use at Jane Street along with a lot of our other proprietary systems. And Sam was in one of the first cohorts of students and so I got to meet him through that experience.Dwarkesh PatelGot it. Okay, and what was your impression of him?Brett HarrisonYeah, he was a smart kid, he was nice, he kind of got along well with other people in his class. He was definitely above average, but not completely stand out at the top. Although then again, the bar was extremely high at Change Street, so I think that's already sort of a compliment. But people liked him a lot and thought he had a lot of promise, but he was a young guy like everyone else.Dwarkesh PatelGot it. And did that perception change over time while you were at Jane Street?Brett HarrisonIt slowly started to. Sam was on one of the largest trading desks at Jane Street and had 50 or 60 people on it. He had several managers. And one of my roles at Jane Street was to work with all of the different trading desks on the designs of their particular strategies and systems. And so I would frequently go over to his desk and talk with his managers about stuff and they started pulling him into conversations more and more specifically to talk about some of the lower latency Etfr stuff. We were doing some of the original OTC automation things we were working on. And so he started actually contributing more to the actual design and thought behind some of these systems and thought he was precocious and had a lot of really good intuitions about the markets.Dwarkesh PatelGot it. Okay, and so what exactly was your role at Jane Street at this time and what was his?Brett HarrisonYeah, so at this time I was sort of leading the group of software developers building the technology that was closest to actual trading. You can think of Hft or any kind of trading firm. There's lots of different developers, people who work on stuff really relating specifically to the trading technology, people who work on kind of the core systems, networking, kind of internal automation tools, tools for developers. So we were in the part of the spectrum that was closest to actual trading. And so my job was to go over to the different trading desks within the company, talk to them about their specific strategy for the products they traded, understand how to like their priorities, about what venues they want to connect to what different systems they want to create? What different parameter changes they need in their automated trading systems? What kind of research tools will help them do their job better? What user interface would make it easier for them to understand what's going on in the market and kind of all of that?Dwarkesh PatelOkay. And did SVF at this point have any sort of reputation of either being uncooperative or being cooperative or anything ethical or professional that's not worthy of this time?Brett HarrisonI don't think there was much that stood out, although he was again pretty precocious at that particular time period. One anecdote that sort of drew me closer to him was Jane Street's offices were in 250 Vessie. They still are in New York City. And there's a big food cart on the second floor. And so I once went down to meet with a development person from a nonprofit that works in animal welfare and something that my wife and I had donated to for a long time. And I met with this guy and he said, you're the second person I met from Jane Street today. Which was wild because Aintree was like only a couple of hundred people there. This is like a pretty niche organization. And I was like, that's crazy. Who did you meet? And they said, oh, Sam Bankman Freed. And I was like, Sam? I just came down from talking with him upstairs. And so I went back and we sort of realized we had this kind of shared interest in helping kind of animal welfare causes. We're both vegans and we sort of bonded over that. That's how we kind of became friendly.Dwarkesh PatelGot it. It seems that his interest in effective altruism was genuine at this point. And early on there was a history of this.Brett HarrisonYeah, it wasn't like EA was super popular at Jane Street. I feel like that's a bit of recent sampling bias among this younger crew of Jane Streeters. There definitely I think it was because of a lot of association prior to joining Jane Street that were into effective altruism. But there were a couple of people there who really were fairly vocal about the fact that they were donating the majority of their yearly salary and bonus to charitable causes. And Sam was one of them. And yeah, started to become known for that.Dwarkesh PatelGot it. Okay, so I guess fast forward to he's no longer at Jane Street, you're no longer at Jane Street, and you're at Citadel. He started FTX, actually, before we back go there. Were you in contact with him up until the point where you had started talking about a potential yeah, off and on.Brett HarrisonWhen I first left Jane Street, and he left sorry, when we both left Jane Street around the same time, him before me, he had told everyone at Jane Street that he was leaving to join the center for Effective Altruism full time. And I guess he did that. I'm not sure if it actually happened because he very soon after started this trading firm and tried to pull off of Jane Street people to join him to do this trading firm, which didn't make people super happy, but it was funny. We had a phone call and he told me that it wasn't really going super well. He said it was really great in the beginning. They made a lot of money, they had this arbitrage trade, and then a few things kind of went by the wayside, and they had taken out these huge loans to be able to get their initial capital for Alameda. And also there was a big fracture within the company. Half the company split, people left. He really didn't tell me much about that at the time, and he said he's probably going to do something else. And when I asked him, he said, I think I'm going to work on political prediction markets. And I was like, okay, it doesn't sound super exciting to me. I'm going to continue on with what I was doing, which was moving to Chicago, taking a new role. But then fast forward, I guess that idea. Maybe he wasn't telling me the whole truth at the time, but I guess that idea became FTX and he had the impossibly, like resuscitated Alameda in the process.Dwarkesh PatelYeah, that's really interesting. Do you have some sense of what it was that went sideways?Brett HarrisonSo I pieced together some details over the years because he told me a little bit more after I first joined. I heard a little bit more later from other people and then saw some reporting kind of post FTX collapse. I think there were two things. One was the infrastructure they had built, I think was really poor in the beginning. A lot of Python scripts, like, slapped together, and a couple of times they had sent tokens to the wrong wallet and ended up losing millions in the process. And they had some big non arbitrage directional bet on in some token, it might have been ETH or something, and it went against them. And so they lost a lot of their trading capital. And then the other thing was that after some of their technical problems, there was internal disagreements, supposedly. This is what Sam told me about how to move forward with tech. There was half the crew that wanted to kind of rewrite everything from scratch in a different programming language. There was another half that said, like, okay, we can make some small incremental changes from here and fix things up. And Sam and Gary and the shot were more in. That latter crew, that former crew kind of broke off and started their own thing, and that's what originally happened.Dwarkesh PatelOkay, got it. And were you aware of the extent of this at the time, or something piece together?Brett HarrisonNo, not at all. Sam told me a little bit about it, but this is over the course of years now where I had two different roles, one at Headlands, one at Citadel securities. Sam was starting alameda. We spoke maybe once a year, briefly on the phone. So all this stuff was happening in the background and I had no clue. In fact, the first time I even heard about FTX was one of my colleagues from Citadel securities told me, hey, did you ever work with this, like, Sandbagman Freed guy? And I was like, yeah, a little bit. Why? And they're like, you know, he's like a billionaire and he has this Hong Kong crypto exchange. Like, what? No, since when? And then started to see him pop up in articles. There was like a box article about him and a few other things, especially related to his political donations. And that's kind of when I got back in touch with him and we started talking a little bit.Dwarkesh PatelWhen was it that he called you to say that there are potentially troubles and I'm considering starting a political prediction market?Brett HarrisonThat was in 2018.Dwarkesh PatelOkay, yeah, got it, got it.Brett HarrisonSo I it was right after I left Jane Street.Dwarkesh PatelGot it. Okay, so now you've moved on to Citadel, and I guess you are still in touch at this point.Brett HarrisonYeah, like, very briefly, a text every now and then.Dwarkesh PatelOkay. And then at some point you become president of FTX US. Do you want to talk about, I guess, how he approached you about that and what was going on at the time?Brett HarrisonYeah, it was interesting. So at the time, I was running what was called the semi systematic trading technology at Citadel securities. And so this was the group of technologists working on systems for ADRs, ETFs options, and OTC market equities. There's around 100 software engineers or so that rolled up to me and, you know, that was going well. But, you know, Sam and I started talking. It was, I guess March of 2021, and he was like, telling me a couple of things going on in FTX. And then he said, if you're interested in coming over to FTX, we would still love to have you. And I thought, still, I've never talked about doing this before, but sure, let's entertain this. And then we started talking and he had me meet him and Gary and Nashad over video call I was in Chicago, they were in Hong Kong at the time. These calls were taking place like late at night my time. And very quickly like an offer came together and I thought, this is a really cool opportunity to jump into a field and take a role that was very different from stuff I've done in the past. And I signed up.Dwarkesh PatelGot it. Okay. And where was FTX at this point in terms of its sort of business development?Brett HarrisonYeah, so FTX was doing quite well. It was basically finished its first, its second year of operation and it was maybe the fourth or fifth largest exchange in the world by volume, if you include spot crypto and crypto derivatives. And it was also one of the primary destinations for institutions, proprietary trading firms, hedge funds to trade crypto and derivatives, especially because of how it was designed. And so it was doing really well. FTX us was virtually nonexistent. They had started the, they formed the entities. They had started the exchange, I think in either December 2020 or January 2021, but it had like, de minimis volume compared to the other exchanges around the world, especially in the US. Too. And Sam talked to me a lot about the aspirations for the US business. One, to grow the spot exchange, of course. Two was to be able to find a regulated path for bringing some of these offshore products like Bitcoin and ether futures and options onshore in a regulated way. And on top of that, Sam had also told me about kind of longer term desires to be a single app or marketplace for everything, not just crypto, so launching a stocks trading platform as well. And so that was one of the reasons I think he wanted to bring me on, was because I had all this experience kind of inside of regulated broker dealers and sort of knew roughly what it took to get that started.Dwarkesh PatelOkay, got it. The initial offer was specifically for President of FTX US, right?Brett HarrisonYeah. Sam wasn't someone who loved thinking hard about titles and even like what my original title was going to be was like a point of contention, but I'm not sure it was clear exactly what my role was going to be. I think Sam wanted me to write software for FTX and FTX US. Sorry for FTX US, but to me, I sort of thought there was like this bigger opportunity to kind of kind of work with Sam to lead this other startup which was FTX US, and kind of build it up and sort of follow in FTX's footsteps in its success. And that was the part that was most exciting to me because this is what I've been doing now for years. It's like managing large teams of people, thinking about strategy, getting people together, occasionally doing some software development myself. But that was the primary reason for wanting to join.Dwarkesh PatelGot it. And what was the relationship between FTX and FTX US at that time? Were they kind of subsidiaries? Were they separate entities?Brett HarrisonThey were separate entities. They weren't subsidiaries. There was technology sharing between them. So, like the FTX US exchange technology was licensed from FTX. You can think of FTX US as like FTX stripping away most of the interesting parts of FTX. Right. Because it was just a dozen or two spot tokens. And when I joined, there were very few people within FTX US, maybe like two or three dedicated people. So over the course of the next year or so, my job that I sort of fit for myself was to open up some offices, hire a bunch of people, establish separate compliance and legal and operational and support teams, start to build out these regulated entities.Dwarkesh PatelWas Chicago the initial base of that operation?Brett HarrisonYeah, for selfish reasons. I have my family here's, here, and I wasn't going anywhere. But also I thought Chicago is a great place for FTX US, because if our main goal was to establish regulated derivatives, like, Chicago is really the place where that happens. We have like the CME, we have many of the top proprietary trading firms. A lot of the Futures Commission merchants and various brokers are all here historically, like the kind of the floor of the Chicago Board of Trade and the Chicago Mercantile Exchange. Like, they're here. And so it kind of felt like a good place to be.Dwarkesh PatelAnd at this point, I guess before you joined, did you get a chance to ask him about the relationship between FTX and Alameda?Brett HarrisonYeah, I did. It was definitely of interest to me because the primary reason being that I wasn't interested in doing Prop again. I worked at Jane Street. I worked at Headlands Tech. I was at Cedar securities. If I wanted to continue doing Prop trading, I would have stayed at one of those places. So I wanted to do this exchange business. And what Sam told me was the same thing he said publicly everywhere, which is that Alameda is basically running itself. All of Sam's time is on FTX. They're kind of walled off from the FTX people. And their access to the exchange is just like any other market maker. Like, there's like the public API feeds, there's benefits from market makers that trade enough volume. But it's not like Alameda had any special privileges in that sense. And. So I thought they were just basically separate.Dwarkesh PatelAnd did you ask to, I guess, audit their sort of financials or this relationship before you joined?Brett HarrisonNo. I mean, I don't know about you, but I've never gotten an offer to a company and said before I signed, show me your audited financial. It's just like, not a thing that happens.Dwarkesh PatelRight, okay, fair enough. So you joined FTX and then you mentioned some of the stuff you were working on, the operational legal, getting the organization set up. But yeah, feel free to talk in more detail about what were the things that came up during your tenure and what are the accomplishments you're proud of?Brett HarrisonYeah, I guess on the professional and personal front so, I guess on a professional front, I'm most proud of establishing out our team and making significant headway to a lot of our goals to establish these regulated businesses. So, for example, LedgerX, we acquired LedgerX and we had this application to the CFTC to enable kind of real time, direct to customer, 24/7 margining and cross collateralization. And it was an extremely innovative proposal and it felt like we were making real progress towards establishing new and very exciting regimes for Cfdc regulated derivatives in the US. I also established a broker dealer in the US for the purposes of letting people trade stocks like some lords, robin Hood. I wrote like 90% of all the code for that stocks platform myself. And yeah, I was very proud of that accomplishment. And then on a personal front, I it was great to get embedded into the crypto industry. I was very excited by everything that I saw. It was great to make all the connections through FTX with, like, the different people in the crypto ecosystem and become friends with these people and certainly has an influence where I am today. So I'm sort of proud of all of that.Dwarkesh PatelHow did you manage the management of, I don't know how big the team was at to speak, and it sounds like you were heavily I mean, involved is an understatement in the actual engineering. How were we able to manage both roles at the same time?Brett HarrisonYeah, so we were like between 75 and 100 total people in the US. And it was challenging. It was one of my biggest complaints, which I'm sure we'll get into, that yes, I can write code, but I feel like that's my comparative advantage is helping kind of to leverage teams of people to get them to work towards the common goal of building out large distributed systems that are complex and multivariate nature. And the best use of my time was not me programming between the hours of like 10:00 P.m. And 02:00 A.m. Every night while trying to keep on board with what all the personnel were doing. So I really wanted to grow the US team significantly to at least be more than a handful of developers. And so, yeah, that was one of the initial points of contention.Dwarkesh PatelOkay, speak more about that. So he was opposed to growing the team.Brett HarrisonSam would frequently talk publicly about how proud he was that all of FTX is built by two developers and all of these crazy organizations that hire thousands of developers and can't get anything done. They should learn from me about how, like, a small lean team can be much more effective. And there's some truth to that. I do think the conventional wisdom now is a lot of big tech companies overhired for software engineers. And not only was it sort of an expense on the balance sheet, but it was also expensive in terms of slowing down the kind of operational efficiency of the organization. And having a small lean team can help you get to your first or your anth product a lot more quickly. That's great for a startup, but once you're like a north of $10 billion valuation company, like, promising the world to customers and investors, two software developers doesn't really cut it anymore. I mean, at some point, you have to grow up and face the reality that it's time to actually grow an organization into a real kind of managed enterprise with teams of software engineers specializing in certain tasks. And so there was always pushback. People would tell me, like, look, we're not trying to be like Jane Street or Citadel in terms of our number of software engineers. We want to stay lean. That's our comparative advantage. And most importantly, they didn't want two separate development teams, like one in the US. One in then the Bahamas, like, they wanted to keep the nexus of software development underneath Nashad and Gary in the Bahamas, which I just thought wasn't going to be sustainable long term. Like, if you run a broker dealer in the US. You need to have staff that is specifically allocated towards broker dealer activities. It can't be that if FINRA comes and says like, well, who's working on the broker dealer? You say, well, it's like this Gary guy who lives in the Bahamas who sometimes is awake at like 04:00 A.m., spends 20 minutes a day thinking about stocks that can't fly right, and has.Dwarkesh PatelNo images of him. Okay, so we're Nashad and Gary contributing code to the FTX US. Code base.Brett HarrisonRemember, like the FTX us. Side of things was like strict subset of FTX. So, like, in that sense, it kind of flowed into FTX US. With the exception of the FTX us. Derivatives, the Ledger X stuff was actually a completely separate team because that was through an acquisition.Dwarkesh PatelWhen you're talking about the code of, like, the matching engine or things like that. Was the code shared between FTX and FTX us.Brett HarrisonYes.Dwarkesh PatelOkay, who was in charge of ultimately approving the pull request, basically, of the Xtx US. Code base?Brett HarrisonYeah, I was like all Gary and Nashad.Dwarkesh PatelOkay, got it. And so the code you were contributing was also going to the sort of like universal global code base. Yeah, got it. Did you have sort of access to the entire code base or just the FTX US side?Brett HarrisonYeah, again, it was one share repo. I mean, there was an enormous amount of code. And one of the big problems, another problem that I raised while I was there, was that 90 plus percent of all the code of FTX was written by these two people. And it was very hard to follow. I don't know if you've ever seen like a large Python code base before. So whenever there were issues that arose there's, like this particular problem with an account on the exchange, the only answer was like, call Nashad, call Gary. Which I also knew to be unsustainable from the organizational perspective. One of the guiding principles at Janetree, for example, was mentor your junior dev so that you can hand off all your responsibilities to them. And in the process of handing off responsibilities, you make the code better, more automated, more robust problems, more easily debuggable in real time. If you hoard everything to yourself in your own brain, you end up with a code base that is just only understandable by that one person. And so it was the kind of thing where a lot of people talked about this internally. Like if Gary got hit by a bus and couldn't come to work anymore, fjs was done. It's done. Exactly.Dwarkesh PatelWhat do you think was the motivation behind this? Was it just that he wanted to avoid as a sort of like Google growing to 100,000 people kind of thing? Or was there something else going on? Like what? Why did it why this sort of concentration?Brett HarrisonClearly there was something else going on. I think an open question now, only thinking about this in hindsight was how much of this very cloistered organizational decision around the development team was a function of the various things they were doing that they were hiding from the rest of the company? Or was it really this sort of like one, ultra paranoia about growing too large too quickly and getting losing control of the organization, and two, an almost like sort of cultlike belief in this small team, like being the but for cause of all past, present, and future success.Dwarkesh PatelWhat was the discretion that you had at FTX US? It sounds like you weren't even given the capacity to hire more engineers if you wanted to. What were the things you did? Control.Brett HarrisonYeah, hiring, for example. I began pushed for many months that we should hire more people. Eventually I got permission for us to interview people, but then those ultimately have to get finally approved by the people in the Bahamas. And they would frequently say no to people who I thought were good candidates. Finally, we hired one person, and this person was doing well. He was here in Chicago, and they invited him to go spend like a month in the Bahamas to kind of hang out with them and supposedly just like to ramp up on the system. And this person comes back to Chicago and they say, you know what? Like, I really want to move to the Bahamas. They really kind of confused me to do it. And it was so frustrating.Dwarkesh PatelI was just poaching from your own company.Brett HarrisonExactly. It was such a constant battle. And at some point I kind of gave up on this idea that I was going to be able to actually grow the separate developer team. So, I mean, the bottom line is on kind of like the day to day operational stuff, especially the decisions within some of the things I was responsible for, like the stocks trading platform that I was working on, I had a fair amount of discretion and people certainly looked up to me for management and advice and direction. But ultimately the discretion ended up with this small group in the Bahamas who not only had final say on decisions, but would often make decisions and not communicate with the senior people in the US. Side and we would just sort of find out things were happening.Dwarkesh PatelIs there a specific example or set of examples that comes to mind?Brett HarrisonSure. The biggest example for me was this sort of post my kind of effective resignation, but some of these strategic acquisitions that were being done in the US. During the summer of 2022, I would find out from the news or it would sort of be mentioned on a signal chat or something that this was happening. And there was no opportunity to actually wade into the discussion about how this is going to greatly affect the US. Business, it's going to greatly affect our priorities. And it wasn't clear if this was like a good decision or a bad decision. It was like a unilateral decision that was made, like, we're acquiring this company or we have the option to acquire this company.Dwarkesh PatelAre there decisions that were made from the Bahamas that stick out to you as being unwise that, I don't know, you try to speak out against? You mentioned some of them, right? Like not hiring enough people and not getting more developers. But are there other things like that that stick out to you as bad decisions?Brett HarrisonA lot of the spending, I mean, on everything from, like, lavish real estate to all of these, like, partnerships to, like, very, very large venture deals, like, these were the kinds of things in the company where people ask, when does it stop? To one end, are we doing a lot of these things? And some of those resulted in sort of like direct confrontations, like, just, why are we doing yet another deal with a sports person or a celebrity? This is ridiculous. This is not doing anything for the company. And we're completely distracting from the role that we thought we all had which is to build a really great core product for people trading crypto on crypto derivatives.Dwarkesh PatelYeah. And did you bring this up directly with SBF?Brett HarrisonYeah, multiple times.Dwarkesh PatelAnd how would he respond?Brett HarrisonSometimes he was nice about it, and he would say, yeah, I see where you're coming from. I do think what we've done so far has been really valuable, and we probably should do some more of it, but maybe at some point we should stop a lot of this. Sort of like, hedging language that was ultimately non confrontational, noncommittal I mean, he was a very non confrontational person. Very conflict, avoidant person within the company. And then at worst, there were other times where I brought up specific things that I thought that he was doing wrong. There was one really unfortunate time where it was the first time I visited the Bahamas in November of 21. And I'm the kind of person who, if I see something wrong at a company, it doesn't matter what company I've worked at or how junior or senior I've been. I like to go to the person most senior in. Charge and tell them this thing seems wrong to me and that's I feel like it's one of my superpowers of just like, not being afraid of just saying when something seems wrong to me. And sometimes I'm just totally wrong and don't understand the full picture, and sometimes it results in something better happening, and people will, you know, thank me for having been honest and bringing to attention something that's actually wrong. And so I said to Sam, I think you're doing way too much PR and media. First of all, it's really diluting you and the FTX brand to constantly be doing TV interviews and podcasts and flying to banking and private equity conferences. And it was so much time spent on this stuff, and also it was completely taking away from the management of the company. People would sometimes send Sam slack or signal messages and not get responses for weeks at a time. And it felt like he was spending virtually no time helping the company move forward. It was so much about image and brand and PR, and he was really angry at hearing this criticism directly.Dwarkesh PatelHow did he react?Brett HarrisonI mean, he was just he was sort of emotional. He was worked up. He told me, like, I completely disagree with you. I mean, he said, like, I think you're completely wrong. He said, I think the stuff that I've done for PR is maybe the greatest thing that's happened to this company and I should do more of it. I didn't think it was physically possible to do more of it. And I realized at that moment that this was not really going to work super well long term. Like, if we're not in a relationship where I can give sort of my direct, superior, like, real, honest, constructive criticism that I thought was for the good of the company that this wasn't really going to work.Dwarkesh PatelHe actually did my podcast about, I don't know, eight months ago or something. And while I was, like, very grateful he did it, even at the time, I'm like, I don't know if I would have agreed to this if I was in charge of a $30 billion empire.Brett HarrisonYeah, sometimes some reporters would say to me, can you get me in touch with Sam? And I would say why? I'm not really his keeper. You could contact him yourself. They're like, oh, because we want to come to Bahamas and do a special on him. And I would say, like, okay, you're going to be like the 6th one this month.Dwarkesh PatelThere's an exclusive here. So I guess to steal man, his point, he did get a lot of good PR at the time, right? Potentially. Well, not potentially, actually too much. In a way that really created at the time sort of like the king of crypto sort of image. Was he right about the impact of the PR at the time? Yes. Let me ask a question a different way. How did he create this image? I mean, people were saying that he's the JP. Morgan of crypto. He could do no wrong. Even things that in retrospect seem like clear mistakes, like only having a few developers on the team universally praised huge empire run by a few developers. How was this image created?Brett HarrisonI think that media was primed for the archetype that was Sam, this sort of young, upstart prodigy in the realm of fintech. We have a lot of these characters in the world of big tech, and I think that he had a particular role to play in the world of finance. And by making himself so accessible all the time, he gave people a drug that they were addicted to, which was like, that constant access. I feel like any time of day or night, someone could text Sam and get him on the phone with them if they were in media, and they loved it. It was like getting access to a direct expert who was also this famous person, who was also this billionaire, who was also this extremely well connected person, who was also this very insightful person, who knew a lot was going on in the industry and can give them insight and tips. And I think there was some amount of what I like to call like, reputation laundering going on here, where it was like, okay, so you get the famous celebrity to endorse Sam, which makes this politician think highly of Sam, because they also like that celebrity. And then also the investors are writing really great positive things online about it, but also the media is enforcing how cool it is that Sam is doing all these other things. And it all sort of fed into this flywheel of building up Sam's image over time in a way that didn't necessarily need to match the underlying reality of who he was with the company.Dwarkesh PatelAnd what was the reaction of other employees at FTX of this sort of not only the media hypetrain, but also the amount of time Sam was spending with the media.Brett HarrisonOn one hand, I think people were rowing frustrated within the company because of the lack of direction and some of like the power vacuums that resulted from Sam's continual absence. On the other hand, so many people within the company just hero worship Sam. When you hear all the really tragic stories now of all the employees who kept all of their funds and life savings on FTX, they really, really believed in Sam. And doesn't matter how little time he spent with the company, it doesn't matter how he treated employees internally, it was like he was this sort of genius pioneer and that image couldn't be shaken.Dwarkesh PatelAnd I certainly don't blame anybody for it. I interviewed him. I tried to do a lot of research before I interviewed him, and I certainly was totally taken with this.Brett HarrisonRight.Dwarkesh PatelI thought he was the most competent person who had ever grazed crypto, but so what was he actually like as a manager and leader? Other than, I guess obviously the micromanaging aspect of it, or feel free to speak more on that as well. But in terms of the decisions he would make in terms of business development and prioritizing things, can you describe his sort of management style and leadership?Brett HarrisonIn the beginning? When I joined FTX, my initial impressions were that he had pretty clear intuition and insight into the simple things to do that would work in many ways. If you think about what FTX did, it wasn't really super complicated. It was like just be operationally good and give your trading customers as predictable of an experience as possible with regards to collateral management and auto liquidation and matching engine behavior and latency. And so they did it. I would say, aside from the intuition, sam wasn't a details man. That was usually left up to the people below him to really take care of was like to drive a project to completion to figure out all the details that had to be done. I think besides that, as a leader, I thought he was fairly incompetent. I thought he was very conflict, avoidant. He didn't like to get into direct confrontation with any of his employees where most of the reasons why people needed to talk to him were because there were significant issues, whether those were personnel or otherwise, and he just blew them off. That was a frequent occurrence in the company. If you went to Bahamas and I went to only a couple of times to actually visit the office, if he was in the office, he was there all day on calls all day, whether those were with investors or with media, podcasts, whatever. It was just consistently just doing that. And I saw very little time where he actually got up and talked to anyone else within the company about anything. So I think to me that was the primary impression I got of his leadership was virtually that there was none, which made me feel a lot like I and others needed to step up and sort of take that role in the absence.Dwarkesh PatelGot it. And so who was making these day to day decisions in the absence of Sam?Brett HarrisonOn the foreign side in the Bahamas, Nashad was really like the number two person there. I mean, he was making a lot of decisions. There were a couple of others in the Bahamas who were taking kind of swaths of the company, whether it was like investments or marketing or legal things like that. On the US side, we had like a different crew trying to make decisions where we could for us regulated matters. But again, we were always sort of below the decision making authority that was happening in the Bahamas, especially inside of the home where they were all living.Dwarkesh PatelSo it seems like Fcx was a really good product compared to other crypto exchanges. I've heard a lot of traders phrase it was this competence sort of built while SBF was still doing media stuff, or was this built before he kind of went on the PR tray? And like, how was this product built while the CEO was kind of distracted?Brett HarrisonSo I think the core of the product was built before my time, and my understanding was in the transition from Alameda to FTX, where there was no publicity around Alameda, there wasn't any publicity around FTX. It was very much like heads down build mode for several months. And just think, think, think about the core product having been a trader on these different exchanges around the world that also offer derivatives and knowing all their problems. Like, for example, if you had an ether futures position and also an ether spot position on this one exchange, you could get liquidated on your ether futures position even if you had enough ether spot as collateral, because you needed to have that spot crypto within the ether futures spot collateral wallet, which was different than the ether spot wallet. And so it was this game of shifting assets around to different wallets to make sure you kept meeting your collateral requirements, which was just an operational nightmare. And so Sam told and worked with Gary and the Shot to build basically a cross collateralization system where you have just one wallet with all of your assets, all, you know, haircut. It appropriately based on volatility and liquidity, but then summing up to a single collateral value that represents what you can put on in terms of margin for all of your positions. Or having an auto liquidation system that doesn't. Just the second that you're slightly below your margin fraction. Send a giant market order into the book and dislocate the order book by 10%. It would automatically start liquidating small percentages of your portfolio at a time to try to minimize market impact. And then if the position got too underwater, it would auction that position off to backstop liquidity providers, a number of them, who would then take on that position again without having to kind of rip through the book and cause dislocation. And so it was much more orderly, it was much more predictable. And that had to have come from the initial intuitions that Sam and his colleagues got from being traders on these exchanges and thinking, how should this work if it were perfect? So I do think in the beginning, they were really working on that product together. And then once the success came and Sam got drunk on the celebrity of being so out there and known and having all these newfound connections, that things are to go by the wayside.Dwarkesh PatelYou mentioned that one of these things that he was doing was making these sort of exorbitant deals and with celebrities, with acquisitions, branding. What was your understanding at the time of where the money to do this was coming from?Brett HarrisonYeah, so, for example, when I joined the company, FTX had just inked that Miami Heat deal, and I think it was something like 19 million a year. And I was like, well, that sounds like a lot of money. Right? But at the time, you could see the publicly reported volume on FTX, it was something around 15 to 20 billion emotional per day. The fee schedule was also public. So even at like the the highest volume tiers, the you know, the take fee would be something like two basis points per trade. So if you just did like $20 billion traded per day times two basis points times 365, which, because crypto trades every single day, you can get a sense of how much money FTX was making a year. And at the time, I think the run rate for FTX was something like close to a billion dollars in income. And you think, okay, is $19 million a reasonable percentage of the total income to spend on a very significant, important marketing play? I don't know. It feels kind of reasonable. Like, how much does Coca Cola spend per year on marketing as a percentage of their income? It's probably somewhere between like 50 and 130%. I don't actually know what it is. It doesn't seem crazy.Dwarkesh PatelYeah. But if you add on top of that, the real estate, the other sort.Brett HarrisonOf acquisition, all that stuff came later. And secondly, a lot of that wasn't known to the employees within the company. Most of the venture deals, the value of the real estate, et cetera, was non public within the company. There were 100 plus million dollar investments into various companies and other investment funds that were never discussed openly, at least to the US. People. So it wasn't like there was sort of this clear internal accounting where people could look at it and say, hey, are you really spending all this money on this, all this stuff? No, I think Sam very deliberately kept all that stuff within his innermost circle for a reason, because he didn't want the criticism on what he was spending on.Dwarkesh PatelAnd did you have access to, or did you ask to see, I guess, the balance sheet or any of the sort of financial documents?Brett HarrisonI have zero access to bank account stuff or financials on the FTX.com side on the US. I had some. But remember now, knowing what we now know about even like, recent, like the guilty pleas from the shot and seeing like the complaints from like, the SEC Cfdc, they were like, deliberately falsifying information that went into ultimately the audited financials. So in order to actually have suspected anything, one would have to not only, like, disagree with all of the kind of internal conventional wisdom around how the company was doing, but also have to basically distrust audited, financials coming. Back to the company combined with having any concerns about income when it seemed like we were generating income faster than any startup in history. So I think it was very difficult for anyone within the company, especially in the US side, to have a clue what was going on.Dwarkesh PatelSure. Let's talk about Alameda. So I guess, again, maybe the best point to start this story is also with Jane street, where Caroline Ellison went out to become the CEO of Alameda, was a trader. Did you happen to cross paths with.Brett HarrisonHer at Jane street? It's hard to remember because it was like the early days, but I'm pretty sure she was also one of my boot camp students.Dwarkesh PatelIt all starts there.Brett HarrisonBut besides those early interactions, I barely interacted with Caroline, not in the same way that I had done with Sam, just based on the trade and desk he was on. And when I joined the company the FTX us People, communication wise, were walled off from Alameda. So we didn't really cross paths almost at all.Dwarkesh PatelWhat was your understanding of the relationship between Alameda and FTX?Brett HarrisonThis is a completely separate company. Sam doesn't really do anything for them anymore because he is 100% focused on FTX. It's separately being run by Caroline and Sam Tribuco. They have the same access to the exchange, like, data feeds and API as any other market maker on the exchange. And also, you know, especially towards the time that I left, that alameda wasn't even a significant percentage of the exchange volume anymore. They weren't in like the top 20 market makers on FTX.com or something like that.Dwarkesh PatelYou mentioned that you were contributing to the code base and you had access to the code base. People have been speculating about whether Gary or Nashad had hard coded some sort of special limit for Alameda. Did you see any evidence of that in the code base?Brett HarrisonDefinitely not.Dwarkesh PatelYou mentioned that you. Visited the Bahamas offices a few times, and the Alameda, there's like four huts and there's like a meeting room. There's burst salmon, the engineers are there's, the Future Fund, and then there's like, the Alameda hut.Brett HarrisonYeah.Dwarkesh PatelDid the physical proximity between the offices and of course, the fact that the leaders were living together, was that something you inquired about or were concerned with?Brett HarrisonI never visited the places where they lived in that Albany section of the Bahamas, so I think I didn't fully grasp the extent to which they were all living in this particular arrangement. But I understood that as long as Sam was going to be the 90% owner or something of Alameda, he would want oversight there. And so having them close by made sense. But the actual hut set up was such that they had like, physical separation from minute to minute. So it wasn't like Alameda could overhear stuff happening on the exchange or people in the exchange could overhear stuff that was happening at Alameda. So to some extent they felt like, well, at least they're going through the right motions of setting up, like, physical separate buildings. Also, this is not uncommon within trading firms and investment banks. Right. Like, if you imagine there needs to be wall separation between buy side and sell side at different institutions. And the way they do that is they put them on different floors in the same building. Right. And sure, they can meet each other for lunch in the lobby, but they set up some actual physical separation. This is like super par for the course when it comes to financial firms that have these businesses that need to be walled off from each other. And so that didn't seem like particularly strange thing to me at all.Dwarkesh PatelIs there anything that, in retrospect, seems to you like a yellow flag or a red flag, even if at the time it's something that might make sense? In the grand scheme of things, the.Brett HarrisonMost obvious thing only in hindsight, was that Sam liked to do bonuses for the employees twice a year and once the end of June, once the end of December. So they were like semester bonuses. And in the previous semesters, he had paid them early in May for the first semester in November, or early December for the second one. And he was extremely late in doing the mid year 2021, 2022 bonuses. So much so that people within the company started to freak out because there was a lot of bad news in the press about other companies doing layoffs or folding. And it was two to three months late, and people were expecting to get bonuses to pay rent and do whatever. And there's very little communication around this, and people were very concerned. So at the time, people said, look, Sam's like really busy.Brett HarrisonHe's flying to DC every week, he.Brett HarrisonHas all the stuff going on. He just hasn't gotten around to it. But don't worry, it's coming. In hindsight, it felt like there was some clear liquidity issue that was probably the most obvious thing. Everything else is all just things that were red flags about the organization, not red flags about potential liquidity issues or fraud. Things like the complete inability to hire more people, especially on the developer side, not allowing me to establish separate sort of sea level staff on the US side that would have authority that was really separate from the ones in the Bahamas. How completely tightly controlled the Dev team was around access to the code base and the inner workings of all the exchange, and really wanting to keep that nexus of the developer group in the Bahamas next to Gary and Ashod. Those seem like red flags now.Dwarkesh PatelYeah, but not at the time. Did you notice anything weird during the Terra Luna collapse? Because in the aftermath, people have said that that's probably when Alameda defaulted on some loans and maybe some sort of like hole dug itself deeper.Brett HarrisonReally?Brett HarrisonNothing at all. I mean, maybe that's a function of being here in Chicago and just not seeing a group of people freaking out. But nothing seemed wrong at all. In fact, we started having conversations around paying out mid year bonuses a couple of weeks later after Terra announcement and everything seemed very normal. Sam sent out an announcement to the whole company basically saying like, okay, we're going to be paying out bonuses soon. People should expect they're going to be like a little bit lower because we have very similar revenue to last year. But we've also grown in size and also like, the market is slowing and we need to be a little bit more conservative. So all the signs pointed to things as normal.Dwarkesh PatelYou had a thread sort of boiling down this experience on Twitter and one of the things you pointed out there is that you saw the sort of symptoms of sort of mental health issue or addiction issue at the time there. Are you referring to the sort of management mishaps and bad decision making or was there something more that made you come to this conclusion?Brett HarrisonI think it was more than that. When I knew Sam, when he was 21, 22 years old, he was like a happy, healthy looking kid who was very positive, very talkative, got along super well with his cohort of traders. The people on the desk really liked him. When I got to FTX, I think over the course of my time there, I saw someone who was very different than that person I remembered. I think he was angrier, seemed more depressed, more anxious. He couldn't get through a conversation without shaking his leg.Dwarkesh PatelStreet.Brett HarrisonHe wasn't like, that not something I remember at all. He would snap easily. He would not respond to messages for long periods of time. And people had different theories. I mean, people would attribute it to the unbelievable stress of being in the position that he was in complete lack.Brett HarrisonOf sleep, like his diet, lack of exercise.Brett HarrisonI mean, people had plenty of thoughts about what could be causing it all, but something definitely had deteriorated mentally and physically about him from who I remembered.Dwarkesh PatelIf you had to yes, most likely cause of that, what would you say?Brett HarrisonI don't know. I think that's up for a professional with credentials that I don't have. But I do think it was probably a combination of everything. The lack of sleep, the stress he probably was under, not just being in his role, but having kept this secret for so many years around. Whatever was happening with the holes in the exchange and the lying he was doing to his own employees, to investors, to auditors, maybe that weighed on him. Maybe had something to do with his medications and he had just had to be just a plain deterioration in mental state or some kind of personality disorder or a different kind of anxiety disorder. I really don't know. Maybe a mixture of everything.Dwarkesh PatelYeah, got it. You said you gave him a sort of ultimatum letter where you said, unless you change these things, I'm resigning. What were the things you asked that be changed in that letter?Brett HarrisonYeah. So the top three things were, one, to communicate more with me in particular, I could probably count on one hand the number of times I had, like, a one on one phone call with Sam, which probably seems insane, given, like, the position I was supposed to be in. I basically said, like, we have to talk every week. It's impossible for me to get anything done if I don't have the authority, but I have the responsibility to be able to push this company forward, and we're not talking at all. So that was number one. Number two was to establish separate, especially sea level management staff on the US. Side. If Sam was going to be so busy doing what he was doing, at least he needed to delegate that responsibility to, like, a set of professional managers who could actually take care of the day to day operations within the company. And it felt like things were starting to unravel in the absence of that. And then the third was to grow the tech team and move a lot of the authority and management of that team away from nisha and gary so that we could actually spread the knowledge and be able to keep up with a lot of the tasks that we were assigning ourselves and trying to build all these new business lines. That pretty much summarizes it, yeah.Dwarkesh PatelHow regularly were you talking?Brett HarrisonIt wasn't regular. I was on chat groups that he was in, and so occasionally he would respond to something I say on that group, but one on one conversations, I think there were fewer than ten for my entire tenure.Dwarkesh PatelWow, okay. And that was over a year, right?Brett HarrisonYear and a half. Year and a half.Dwarkesh PatelAbout less than one every two months, yeah. How did he respond to this letter?Brett HarrisonSo it took a little while before we got on the phone and he went through every point and refuted everyone, starting with communication. He said, I think phone calls are a waste of time. I think that if I promise people regular phone calls, they will use it to waste my time and it's not an efficient mode of communication. He said, I think we have the best developer team in the world and I think anyone who suggests otherwise is completely wrong. And if we add more people to the Dev team, if we move them to the US and move them away from the Bahamas, we're going to be worse as an organization. He kind of ignored the point about separate leadership. I think he hated the idea of giving other people kind of titles that would reflect kind of greater responsibility within the company. That conversation ended with us kind of not knowing what the future was going to be because I basically said, look, I'm going to resign if you don't fix this. He said, we're not fixing anything. And then what happened next was he had deputized another person within a company to come here to Chicago and pull me into a side room and say, you are probably going to be fired for this letter that you wrote and not only are you going to be fired, but Sam is going to destroy your professional reputation. Like, where do you think you're going to be able to work after FTX after all this happened? And he was threatening me. And then not only that, he had said, if you are going to have any hope of staying and if you can forget about getting paid bonuses, you need to write Sam an apology letter and show it to me first because I'm going to edit it and I'm going to tell you what to say. And I said, absolutely not. This isn't like a mafia organization. This is extremely unprofessional. And I knew at that point there was absolutely no way I was staying. It was a matter of when, not if. But what I did know was that I'm still a professional, I'm still loyal to the company. I still believe the company itself had an incredible potential to continue its sort of road of profitability. And I really liked all my employees here on the US side and I wasn't going to abandon them. So I sort of thought like a three to six month time period is about standard to take the time to unwind responsibilities, to finish the stocks platform that I was working on to, you know, get my team in. A position where I knew they would be in good standing and they wouldn't be retaliated against after I left and took the time to do that before officially resigning. In kind of the end of the summer in early fall.Dwarkesh PatelAnd did that happen after you left? Did the, did he try to enjoy your professional reputation?Brett HarrisonHe did. The acute thing that happened was Sam, I actually offered to stay on longer. I could stay on for a couple more months and help this transition. To whomever you name as the successor president of FTX US. And he said, no, I want you gone more quickly. And so I should say he said that, but he was communicating through other people. He wasn't talking to me directly at that point. And so he said like, I want you gone on September 27. So okay, that's fine with me. On September 27, not only did he announce to the company my resignation, he also announced that he was closing the Chicago and San Francisco offices and that everyone had to move to Miami. And basically if they didn't move to Miami by a certain date, they were not going to be at the company anymore. So the employees were distraught. And what I had learned later from several investors and reporters who had talked to me was that when they talked to Sam about my leaving, sam told them that my my leaving was a combination of resignation and firing and that one of the reasons that I had to leave was because I refused to move my family to Miami. So basically that guy was constructively fired that he had closed down this office that I built and that if I wasn't going to move that I couldn't, I know, roll off of the company. And so that took a little bit to crawl out from. I had to tell people like, well, it's completely false, it didn't happen at all. And yeah, and he was telling people that he fired me.Dwarkesh PatelAnd when he said that, he was still at a sort of like peak of hype, so to speak. Right, absolutely. So, I mean, did the idea forming Architect have already had that idea by this point?Brett HarrisonYeah.Brett HarrisonKnowing that I was going to leave, I started thinking what I was going to do next and thought, well, if I think I can run a company better than Sam, I should put my money where my mouth is and start a company.Brett HarrisonBut I had this, a couple of.Brett HarrisonIdeas and they had this particular idea for Architect and it was starting to really form kind of towards the end of my time at FTX, but I hadn't started anything. And so finally I left FTX and then took a little bit of time off and then started to talk to investors about maybe raising some money for starting this company. And there were a few investors that basically said like, do you have Sam's blessing to do this? Why do I need Sam's blessing? I've resigned, I don't work there anymore. They said, we really would feel more comfortable if we could talk to Sam first. And kind of like, you know, make sure things are okay and kind of figure out what he's doing, find out if he wants to invest too before we kind of talk to you further. And it was impossible to escape that like the Sam kind of hype bubble. Even having left the company.Dwarkesh PatelWhy do you think they were so concerned about were they trying to invest in FTX in the future?Brett HarrisonThey were existing FTX investors.Dwarkesh PatelOkay.Brett HarrisonAnd I think it really mattered to them what Sam thought of them. And if they didn't know the full story, and if they were being told that Sam fired me, then I think they were concerned about potential conflict investing in me too.Dwarkesh PatelWas any part of that because Sam had a reputation as I don't know, if like, an investor invested in somebody he disapproved of, he would get upset in some way?Brett HarrisonNo. If that happened, I don't know about it, but I think it was just sam had such a kind of magical hold over the entire industry, from investors to media to politicians, that they looked to him for approval.Dwarkesh PatelOkay.Brett HarrisonYeah.Dwarkesh PatelSo at this point, you've left FTX us and you're starting to work on your own company. And when is this exactly? This is.Brett HarrisonWell, my official resignation was the end of September. I had stopped working earlier than that and so I kind of started to start working on fundraising for the company.Dwarkesh PatelIn October and then a month later the thing implodes. So when did you hear the first inkling that there might be some potential trouble?Brett HarrisonThe first thing I heard was the night before the announcement that finance might be buying FTX. I was just looking at Twitter and just saw all of this fearmongering. It was like, okay, well, Cz says he's selling FtT and so FtT is going to go down. And people were saying, well, that means Alameda's Toast. And then once Alameda goes under, oh, that's going to be a problem for FTX. Pull your funds from FTX. And I was just sort of laughing at this because whatever, I'm used to people saying things on Twitter that seemed nonsensical. And first of all, Sam and Caroline are great traders. Like, if anything, like maybe they'll profit from all this volatility and tokens and they don't understand, like, there's no way anything's going to happen to Alameda. But also this connection between the price of FtT token and the ability for customers to withdraw their funds from the exchange, like, this just did not compute for me at all. So I was like, this will boil over in a couple of days, like everything else. And the next morning I was actually busy talking to my own lawyers and investors for the company because we were actually closing up our investment round. Actually, the closing docs for my investment round went out that morning. The morning that FTX announced they were going to be bought by finance. It was like the worst timing in crypto fundraising history. So I was busy all morning. And then I went online and checked Twitter and then saw Sam's tweet that said, what comes around goes around, and we're going to get acquired by finance. And I don't know. I felt dizzy. I had no idea what was going on in the world at FTX. I just couldn't put the pieces together in my head. It just didn't make any sense to me.Dwarkesh PatelSo before then, you did not think this was possible?Brett HarrisonI kept a bunch of money on the exchange. I was still an equity holder in FTX and FTX US. I was still very pro FTX, in spite of my experience with Sam.Dwarkesh PatelAnd then how did that week unfold for you? You were, I guess, almost about to close your round. What happened to the round? And then how were you processing the information? I mean, it was like a crazy week. After Direct, the deal falls apart, bankruptcy hacking. Anyways, tell me about that week for you.Brett HarrisonSure. First, the investors, we all had to hit pause. First of all, Architect became priority number 1000 on everyone's list. Secondly, a number of those investors were trying to do damage control themselves. They either they were themselves investors in FTX or FtT. They had companies who part of their runway was held on FTX, or they were expecting to get investment from FTX. So people were just trying to assess what was happening with their own companies. They were not writing checks into new companies anymore. So I had to hit pause on the whole thing for their sake and for my sake. And yeah, just what could one do in that situation except for just read the news all week? Because everything that came out was something brand new and unbelievable, more unfathomable than the thing before. And it was a mixture of, like, rumors on Twitter and articles coming out in major media publications and the kind of the announcements of the bankruptcy. It was just information overload. And it was very difficult to parse fact from fiction. And so it was an emotional time.Dwarkesh PatelYeah. Understatement. Right. All right, so we've kind of done the whole story of you joining to the company collapsing. I want to do a sort of overview of, I guess, what exactly was happening that caused the company to collapse. And I guess the lessons there. Right. In the aftermath, SPF has been saying that FTX US is fully solvent. If they wanted to, they could start processing withdrawals. He had a substance recently in January where he said that it had $400 million more in assets and liabilities. What is your understanding of the sort of relationship between the solvency of FTX US?Brett HarrisonSure. Answer is, I really don't know.Brett HarrisonIf you had asked me about the.Brett HarrisonSolvency of FTX US at the time that I left, I would have said, why are you asking about this? Like, of course everything's fine right now. It's very difficult to understand what is going on. First, the level of deception that was created by this inner circle of Sam's and now reported through the various complaints and the indictments from DOJ, they were doing things to intentionally manipulate internal records in order to fool, like, the employees and auditors and investors. So everything's out the window at that point. And then secondly, it sounded like in the week prior to the bankruptcy, there was this flurry of intercompany transfers. And given all that that's happened, it's impossible to say what state we are in now compared to where we were several months prior. So it's impossible to know who took.Dwarkesh PatelOver management of Ftsus when you left.Brett HarrisonI'm not sure.Dwarkesh PatelWas it a single individual or was it just rested back to the Bahamas?Brett HarrisonI really don't know. I mean, I've totally cut off from everything. FTX at the time that I left.Dwarkesh PatelBefore you left, were the assets of FTX US custody separately to FTX international's assets?Brett HarrisonYes, they were. Okay, so we had like a separate set of bank accounts, separate set of crypto wallets. The exchange itself was like a separate database of customers. It ran in like a different AWS cloud than the one that the international exchange ran on.Dwarkesh PatelOkay, got it. And you had full access to this and it checked out basically more assets and liabilities.Brett HarrisonRight. But remember that the thing that makes them not separate is the fact, and this was completely public, that Sam was the CEO of FTX and FTXus, and Gary was the CTO of FTX and FTX US, and Nashi was the director of engineering for FTX and FTX US. And so as long as there wasn't this, like, completely separate, walled off governance that we were trying to establish while I was there, there was never going to be perfect separation between the companies. This was like a known problem. And that was what makes it so difficult to sort of understand the nature of what was potentially happening behind the scenes.Dwarkesh PatelSo I guess we've been talking about the sort of management organizational issues of FTX. Were these themselves not like some red flag to you that, I don't know, something really weird could be happening here, even if it wasn't like fraud, right? These people are responsible for tens of billions of dollars worth of assets, and they don't seem that competent. They don't seem to know what they're doing. They're making these mistakes. Was that itself not something where that concerns you?Brett HarrisonI mean, it concerned me, and I tried to raise concerns multiple times. If you raise concerns multiple times and they don't listen, what can you do other than leave? But you have to understand that every company I've ever worked at, and I would think any company anyone's ever worked at, has management problems and growing problems. And especially for a super high growth startup like FTX, it's a very natural progression to have the visionary CEO who brings that product to product market fit, who enjoys that sort of explosive success, and then the reins of the company are eventually handed over to professional managers who sort of take it into its maturation phase. And I thought, well, really, I'm not that person because Sam and I have interpersonal issues. But there's 100 plus major investors in FTX. Someone will figure out how to install the correct management of this company over time, and we'll bring it to a good place. Like, one way or another, this is going to succeed. There are too many people with vested interest in doing so. And so, no, I wasn't concerned that FTX wouldn't somehow figure this out. I still thought FTX had an extremely bright future.Dwarkesh PatelBut there might be, I guess, these sorts of visionaries. A lot of them might have, like, problems, to put it in that kind of language, but I don't know how many of them would make you suspect that there's, like, mental health issues or there's addiction issues that for somebody who's in charge of a multibillion dollar empire, I don't know. Seems like something that's I can't quite speak to.Brett HarrisonWhether people would think there are mental health issues of other people who are supposed to be the kind of figureheads of large companies. But remember, at this point, sam is not leading the day to day operations of the company like many other people are. Right. And as the kind of public figurehead of the company, sam was obviously doing a very good job. He was extremely successful at raising money. He was extremely successful at building a positive image for the company. And so in that sense, that was all going fine and the rest of the company was being run by other people. So, you know, I didn't witness anything like, you know, the addiction stuff firsthand. I definitely thought he was not as happy a person as I met, you know, a long time ago. But could you blame a person for, you know, inheriting a 20, $30 billion company and not taking it super well when you're 29 years old? I think so.Dwarkesh PatelYou mentioned that given the fact that all hundreds of accredited investors presumably had done good due diligence, that gave you some comfort about the ultimate, I guess, soundness of the company. But potentially those hundreds of investors are relying on the experienced highlevel executives that SBF had brought on that is thinking that, listen, if somebody from Citadel and Jane Street is working at FTX, that's a good indication that they're doing a good job. And so, in some implicit way, you're lending your credibility to FTX, right? I guess. Was there just this sort of circle of trust where the investors are assuming if this person who has tons of leadership experience in traditional finance is coming to Fdx, they must have done the due diligence themselves. And then you are assuming that the investors have done this. And then so it's like, nobody's role to be the guy who's like this was my job, and I was the.Brett HarrisonPerson in charge of remember, regardless of how experienced or inexperienced people within the company are, regardless of how many or a few investors there are how many senior lateral hires there are if a very small group of individuals who are very smart and very capable intentionally put forth schemes that deceive people within the company and outside the company about the veracity of records, what can you do? What is one supposed to do in that situation? If the public reporting matches private reporting, if investors have done their own diligence, if we've joined the company and we see nothing wrong within the company from a financial perspective, if we can see the public volume on the exchange. And it all matches up with our internal reporting and we know how much fees we'll be able to collect and all that. And it seems like a lot of income compared to our expenses for a two or 300 person company. At what point do you go against all of that and say, in spite of the overwhelming evidence to the contrary, I think something is wrong?Dwarkesh PatelYeah, but someone might look at this and say, listen, Brett, you weren't like a junior trader who was like, right out of MIT or something, who just joined FTX. You have more than ten years of experience in finance. You saw Lehman happen. You've managed really large teams in traffic by, and then you have the skills and the experience. And if somebody with your skills and experience and not only that, your position in FTX, as president of FTX US, if you can't see it coming, and maybe you couldn't, right. Whose job was it to see it coming? It doesn't seem that anybody other than the inner circle could have been in a better position, and maybe nobody could have seen it. But is there somebody who outside of the inner circle you think should have been able to see it coming?Brett HarrisonI don't know. It's a good question of, like, when a major fraud happens in such a way where it was very expertly crafted to be hidden from the people who could have done something about it, what should one do? One answer is never trust anyone. Right. Like every company I ever worked for in the future, every time we say we've done some transaction, I will ask them to show me the bank records and give me the number of the bank teller I can call to have them, like, independently verify every single banking transaction. This is sort of like impractical and ridiculous.Brett HarrisonIt doesn't happen.Brett HarrisonAnd so I think it sounds like the counterfactual here is one where, okay, first I have to believe that there is some kind of fraud, which I don't. Then I have to say, okay, I would like to start auditing all. Bank transactions. Actually I want to start auditing all bank transactions for the company that I don't work for. Also I want to disbelieve audited financials from respected third party auditors. I also want to look into the possibility that Sam is like lying under oath in congressional hearings about segregation of customer funds. Also I should disbelieve all of the trusted media outlets and also 100 financial institutions that have invested in FTX. It's like the chain that you have to go through in order to get to a point where you start to be able to figure out something is wrong is, I think, really impossible. And I think the bottom line is for sure should be mandated at certain stages of company growth, independent boards. And I think that a lot of that has to do with where the nexus of control of the company really is and making sure it's in a place where there is appropriate regulatory oversight and appropriate financial oversight. I think that maybe could have helped. But besides that, I think this is ultimately a job for enforcement. Like, people will commit crimes and there is nothing one can do to stop all people from committing all possible future crimes. What Gun can do is come up with the right structures and incentives so that we can build like a trust based system where people can innovate and build great companies and people who are bad actors can get flushed out, which is ultimately what I think is happening.Dwarkesh PatelBut I guess they're not letting you hire people. They're like they're like overseeing, writing the actual code for FTX US from the Bahamas. Not something that makes you think, like, why are they doing this? It's like a little odd.Brett HarrisonI just thought it was not the right way to run the company. There's a very large chasm between I don't think they're doing a good job running the company and I think that customer funds are at risk.Dwarkesh PatelRight, yeah, fair enough. Why should someone who sees bad organizational practices there's no board. They're making a ton of really weird investments and acquisitions. And not only that, like, most importantly, they are responsible for managing tens of billions of dollars worth of other people's assets. What should somebody do when they're seeing all this happening? I mean, obviously it's very admirable and that you put this in writing to him, you gave it to him, and then you resigned when he refused to abide by it. So maybe the answer is just that. But is there something else that somebody should do?Brett HarrisonI would say within any company, and I would expect that the overwhelming majority of companies, if you see bad management, it does not imply fraud. But there's lots of places with bad workplace culture and people are making bad management decisions. And it should be that if you're and find yourself in that position, there is someone you can go to to talk to. It might be your manager, it might be your manager's manager. It could be someone in your HR department. But there should be like a designated person within the company for you as an employee that, you know, you have a safe space to bring complaints about the workplace and about the company strategy. And then you should see how they handle it. Do they take it seriously? Do they make changes? Do they look into the stuff you're talking about? Do they encourage cooperative, positive discussion? Do they threaten you? Do they retaliate against you in some way? Do they start excluding you from conversations? Do they threaten to withhold pay? Like if you're in that ladder camp, what do you do? At that point? It's easy for me to say, and I've been in fortunate positions within companies and have personal flexibility, it might not be so easy for the average person to sort of get up and leave a job. But I do think that at some point you have to start making plans, because what can you do in the face of a giant organization that you disagree with other than leave?Dwarkesh PatelLet's talk about regulators and your relationship with them while you're at FTX. Obviously, as head of FTX US, I imagine you were heavily involved with talking to them. What was their attitude towards FTX like before it fell?Brett HarrisonAll the regulators were, I think, in the common belief that crypto was a large and viable asset class. And in order for it to grow in a responsible way, it needs to come within the regulatory envelopes that already exist in whatever way that's appropriate for crypto. And crypto could mean a lot of different things. We have to maybe distinguish between centralized and decentralized finance here. But I would say regulators saw FTX as at least one of the companies that was very willing to work directly and collaboratively with regulators, as opposed to trying to kind of skirt around the regulatory system.Dwarkesh PatelWell, when I was preparing to interview SPF, actually, I got a chance to learn about your proposal to the Cfdc. We were just talking about you were explaining this earlier, but the auto liquidation and cross margining system bring that not only to crypto in the US. But to derivatives for stocks and other assets. I thought, and I still think it's a good idea, but do you think there's any potential for that now, given that the company most associated with that has been blown up? What is the future of that innovation to the financial system look like?Brett HarrisonYeah, I definitely think it's been set back. It's interesting. Walt Lucan from the Futures Industry Association, in a conference that was shortly after the class of FTX talked about FTX and sort of a speech, but specifically made a call to the fact that in spite of what happened to FTX. The idea of building a future system that can evolve with a 24/7 world is still a worthwhile endeavor and something that we should consider and pursue and be ready for. We are 3D printing organs and coming up with like, specially designed mRNA vaccines, but like, you still can't, you know, get margin calls on a Saturday for SAP 500 future. There's like some real lack of evolution in market structure in a number of areas of traditional finance, and I think it's still a worthwhile endeavor to pursue it. I think the La Ledger X proposal makes a lot of sense. I think it's understandable where some of the concerns were around how that could really dramatically alter the landscape for derivatives regulatory structure and market structure. And there were still unaddressed questions there. But I still think that it was the right idea.Dwarkesh PatelDuring those hearings, I guess the establishment CME and others brought up criticisms like, oh, we have these sort of biscuit relationships with our clients. And if you just have this algorithm take its place, you can have these liquidation cascades where illiquid assets, they start getting sold. That drives the price even lower, which causes more liquidations from this algorithm. And you have this sort of cascade where the bottom falls out. And even though it might not be an accurate way to describe what happened with FtT and FTX because there was obviously more going on, do you think that they maybe had a point, given how FTX has played out?Brett HarrisonA lot of FCMs have auto liquidation. There is one particular one where they actually automatically close you every day at 04:00 p.m., and they do it in a really bad way. So the idea of auto liquidation is not new. The idea of direct to customer clearing is not new. The idea of cross collateralization is not new. The thing I think that was novel about FTX was putting all together, it was direct to customer margining, cross collateralization, auto liquidation. And so in order for the regulators to get comfortable with the application, they had to understand that FTX was one entity was performing the roles that typically multiple different entities perform. And you always need to ask yourself the question of, like, was there something worthwhile about having those different entities be separate or not? Is it just sort of legacy regulatory structure? I think that remains to be seen. And I think we don't have enough experience, especially in the US. With that kind of model, to be able to say whether it actually works better or worse. I think either way, it was worth a try. And I think maybe the biggest misconception about the application was that if we got approved, it meant suddenly FTX is going to list everything from corn to soybeans to oil to SMP 500 overnight and completely, you know, destroy the existing derivatives landscape. I think what have actually happened was FTX would have gotten permission to list, like, one contract on kind of small size and there would have been experience with the platform and it would have been assessed compared to the alternatives on traditional CCPS. And if it was worse, changes would have been made and if it was better, it would evolve and the market would basically decide what people wanted to trade on. I do think the auto liquidation part was the main piece that people were hung up on, which was like, how do you provably show the kind of worst case scenario in auto liquidation cascade? Then again on large TCPS. Now in the US and Europe, there are margin breaches all the time. The current system is far from perfect.Dwarkesh PatelBacking up to, I guess, regulation more generally. I mean, many people saw crypto as a sort of regulatory arbitrage where because regulations are so broken in finance, I guess evidence would be that you're not allowed to do this manually, right? You had to go through the lengthy approval process. If you're a giant company to begin with, the entire point of crypto was to get around the regulators and not go through them to get approval for things and hand over that kind of approval process to them. Do you think that working with the regulators and then also being part of crypto was a sort of like it kind of defeated the point of crypto?Brett HarrisonI think I disagree with the premise. I don't think the point of crypto is equal to arbitrage. I think while crypto remains unregulated, it is easier to get something done in crypto than if it were regulated. That's sort of like total logical. I also think that most people, especially on the institutional side, who trade crypto, believe that we are in a temporary state that cannot last forever, which is that crypto is largely unregulated or has a weird patchwork of regulatory authority. Maybe it's like the 50 state regulators in the US. Or it's some combination of like, you know, money transfer and, you know, CFD or broker dealer activity in Europe. So I think this is, it's absolutely a worthwhile endeavor knowing that there's going to be some regulation for at least part of the crypto ecosystem to work with regulators to make sure that it's done well.Dwarkesh PatelOne very important question about the whole idea thing is what did the conventional narrative about how it went down and why it went down, what did it get wrong? Given that you were on the inside, what do you know was different than what has been reported?Brett HarrisonI actually think not much. And I think the reason for that is typically when something like this goes wrong and it becomes this media frenzy, there's like plenty of opportunity for misinformation to spread. But to the credit of the investigators working on this case, they moved so quickly that, you know, they had, you know, unsealed indictments within what, two months of this going down. And so having kind of a lot of the truth, you know, being able to be spelled out in facts in a public written document, I think quelled a lot of the opportunity for misinformation to proliferate. And whether that's from Twitter troll or if it's from Sam Bankman freed himself trying to spread information about what happened. I think a lot of it wasn't really given the room to breathe.Dwarkesh PatelWhat did you make of his press tour in the aftermath? Why did he do it and what was your impression of it?Brett HarrisonI'm not going to speculate what's inside Sam's head. I think Sam had built up his empire through bartley his control over media. And he did that by being available all the time and being ostensibly open and honest with him all the time and probably thought, why can't the same strategy work now? And maybe I can sway public opinion. If I can sway public opinion, maybe I can sway regulators and law enforcement. And it turns out that is definitely not true. So I don't really know. It could just be an addiction to the media. He couldn't stand people talking about him and him not being part of the conversation.Dwarkesh PatelYeah. And I guess the earlier account you gave of his day long media cycles kind of lends credence to that mentality. I have a question about the future of the people who are at FTX. There's many different organizations who have had their alumni go on to have really incredible careers in technology. Famously, PayPal had a mafia, so called mafia, where they wanted to found YouTube with Elon Musk, SpaceX, Tesla. So many other companies came out of the people who came out of PayPal. And Bern Hobart has this interesting theory. The reason that happens is when a company exits too fast, you still have these people who are young and ambitious in the company who go then go on to do powerful things with the network and the skills they've accumulated. Do you think that there will be an FTX mafia?Brett HarrisonThe number of the most talented people within FTX are leaving FTX in a slightly different position than the people exiting PayPal and acquisition. I would say they're in positions more like actual mafia people. So I'm not sure it's going to be like some giant FTX mafia, but I do think there are a ton of talented people at FTX who are going to look to do something with their careers. And also a lot of those people came from very impressive backgrounds prior to FTX. So I expect them to want to continue to get back on track and build something great. And so I do think you're going to see at least a couple of people who emerge from this and do something really great.Dwarkesh PatelThat's a good note to close the FTX chapter of the interview on sure. Let's talk about Architect, your new company. Do you want to introduce people to what Architect is doing and what problem is solving.Brett HarrisonSure.Brett HarrisonSo the goal of Architect is to provide a single unified infrastructure that makes it really easy for individuals and institutions to access kind of all corners of the digital asset ecosystem everywhere from individual crypto spot exchanges that are centralized to Dfi protocols, to qualified custodians and self custody and everything in between.Dwarkesh PatelYeah, I'm not sure I have enough context to understand all of that. I don't know, a few grade levels.Brett HarrisonBelow backing up a very high level. So let's say you are someone who wants to trade crypto in some way or what do you actually have to do? So imagine you want to do something slightly more than just like sign up for Coinbase and click the buttons there. Let's say you would like to find the best price across multiple exchanges. Let's say that you not only want to find the best price across multiple exchanges, you also want to occasionally do borrow and lending from defy. Maybe not only that, you also want to store your assets in off exchange custody as much as possible. Well, aside from doing all that manually, by opening up all the different tabs in your browser and clicking all the buttons to move assets around and connect all these different exchanges, you actually want to build a system that unifies all these things. You have this diverse build choice. And the build choice looks like. Hire five to ten software developers and get them to write code that understands all the different protocols and different exchanges, all the synchronicity of them, that downloads market data, that stores the market data, that connects to these different custodians that kind of bridges all these things together, that provides some kind of user interface that pulls it all together. It's a significant amount of work that up till now, basically all of these different companies are just reproducing. Again, they're all solving the same problem. And as a trader, you want to focus your time on strategy development and alpha and monetization and not on how do I connect to random exchange? So the goal of Architect is to build this sort of commodity software that people can then sort of deploy out of the box that gives them access to all of these different venues all at the same time.Dwarkesh PatelAnd so it sounds like this is a solution for people with significant levels of assets and investment in crypto. I'm assuming it's not for.Brett HarrisonSo I think that's like the place we want to start. But one phenomenon in crypto that I think is somewhat new and exciting is the fact that, okay, if you want to get into sophisticated equities trading, well, what do you have to do? You usually have to either establish a broker dealer or get hooked up to an existing broker. You need to get market data which can be very expensive. Like the full depth of book feed from Nasdaq costs like tens of thousands of dollars per month. If you want to compete on latency, you have to get like a colocated server in the Nasdaq colo, which is also going to cost you tens of thousands of dollars per month. There's a significant time and money overhead to doing all this, which is why it's so hard to compete in that space against all the incumbent players. Whereas in crypto many of the markets are just in the cloud, like in Amazon's cloud or Alibaba's cloud. And you can just very cheaply and easily spin up like a server in there for a couple of dollars a month and have the same access as a big Hft. All the market data is free, the order entry is free. The protocols are usually fairly simple, like you can use Json over a websocket as opposed to speaking like fix over some private line. As a result, there is this large and growing class of kind of like semi professional individual traders that have grown where there are people who are smart individuals who have like, maybe some wealth amassed and they want to be able to do kind of professional trading. Whether that's like manual quick trading or like simple algos using python or whatever. And they can do that and experiment easily because of the open access of crypto markets. And so there's a much wider customer base for something like this which includes these kind of like high powered individuals in addition to your small medium, large hedge funds and prop shops and different institutions.Dwarkesh PatelAnd is the goal to is crypto the ultimate market you're targeting or are there other asset classes that you also want to provide the service to?Brett HarrisonWe're building very general infrastructure and we think crypto is a viable asset class, but is one of many. And our goal is to provide institutional great connectivity and software to anyone who wants to participate in trading in US semi sophisticated way. So I think over time we'll want to grow our offering as much as possible.Dwarkesh PatelGiven the fact that Nasdaq or whatever already have these barriers, is it possible for someone to remove those barriers with a solution like yours? I guess like an analogy that comes to mind is I guess nobody before Mark Cuban's, whatever pharmaceutical company just try to go outside the insurance system and directly sell drugs to people. Is it possible to do something like that for Nasdaq?Brett HarrisonYeah, you can't connect to Nasdaq without connecting to Nasdaq. You can't not go through a broker dealer. But I think that we could eventually try to get the appropriate licensing required to be an intermediary that is focused on being like a technology forward interface towards people being able to do more program trading. And so if the mission of our company is to provide better access, I think we can do so within the existing system.Dwarkesh PatelI guess this raises a broader question of if you're initially trying to solve for the problems that these exchanges should natively have solutions to at least or some of the problems are the ones that these exchanges should natively have solutions to. Why haven't these exchanges built this stuff already? You're a part of one such exchange and maybe function better than the other ones, but they're highly profitable, they have a bunch of money. Why haven't they invested in making their infrastructure really great?Brett HarrisonSo in many cases, their infrastructure is very good. It's more a question of what's the incentive of the exchange? And I think no matter what, no single exchange is going to have all the market share. So there's always going to be this like, market fragmentation problem. And the question is, whose responsibility is it to make software that helps solve that problem? If I'm some centralized exchange, my incentive is not to build software to make it easier for my customers to go to all the other exchanges. It's like, make my exchange better. So I'm going to put all of my R and D dollars into providing new products and offering new different kinds of services and investment advisory, or lowering the barrier to entry to connect into my own exchange, but not creating this sort of like, pan asset class, pan exchange, interconnectivity software.Dwarkesh PatelGot it. And given the fact that you're trying to connect these different exchanges, currently most of the volume in crypto is in centralized exchanges. What is your estimate of the relative trading volume of C five versus D five? Do you think it'll change over time?Brett HarrisonSo I do think it'll change over time. I think my view is I can't predict what way it's going to change. So people after FTX had asked me like, hey, why don't you try to start your own exchange, take all your knowledge from FTX US, and maybe even raise money to buy the IP out of bankruptcy and start a new exchange. And my feeling is I don't want to bet personally on the exact direction of crypto trading. I could see it continuing status quo where your coin bases and biases the world kind of maintained market share. I could see it moving significantly to defy, where people feel like this is the true spirit of crypto. It's in this sort of noncustodial, fully centralized trading environment. I could also see it going the complete opposite direction and having the existing highly regulated exchange players like Naisi and Azdaq and Sivo start to enter the game on spot trading. And where is the ultimate flow going to end up between these three possibilities? I have no idea. So I'm much more excited about providing the kind of connectivity layer to all of them and saying, regardless of where the liquidity ends up, we'll be able to facilitate it.Dwarkesh PatelSpeaking of FTX, how has your experience with FTX informed the development of Architect?Brett HarrisonYeah, first of all, working at FTX has given me an appreciation for just how behind a lot of the infrastructure is on other exchanges. People really like trading on FTX. Institutions especially really like trading on FTX because the API made sense, really did follow kind of standard state machine of any kind of financial central limit order book that you would see on a place like Nasdaq or CME. Whereas there are a lot of exchanges that have crazy behavior. Like, you send a cancel for an order, you get acknowledged that your order has been canceled, and then you get a fill and you actually get traded on your thing that you supposedly thought you canceled and like things that you think shouldn't be possible or possible. So actually, my time at FTX is interesting with the relation to Architect, because FTX, it gave me an appreciation for how to design a good API for especially institutions that want to be able to trade all the time and the protocols and some of these other exchanges aren't quite as good. So I think it's informed how much the focus of Architects should be kind of wrapping up the complexity of these different exchanges and providing like a really good API for institutions and individuals on our side. And guess like thing one thing. Thing two is obviously what has happened with FTX. People are much less likely to trust a centralized institution with their personal information, especially things like the keys that allow you to trade on their exchange account, or the keys that give you access to their wallet. And so we're thinking a lot about how to design Architects such that the user can connect to all these places and hook up their wallets without needing to ever give us any of their private credentials. And so that's like another particular inspiration from everything that's happened in FTX.Dwarkesh PatelWhat is your opinion of crypto in general at this point? How has your sort of perception of its promise changed, if at all, given.Brett HarrisonThe same I feel the same way now as I did then, which is it's a one to $3 trillion asset class that is traded by every major institution, that is being invested in by every major institution. And so it's totally viable and it needs good mature infrastructure to support its growth.Dwarkesh PatelGot it. But is the motivation also informed by a view that, I don't know, crypto is going to be even bigger or in some way going to solve really big use cases? Or is it simply that, listen, this market exists, I don't know what it's going to be good for, but it needs a solution?Brett HarrisonIt is, I think, I certainly do believe that that is a likely future for crypto. But to me, the interest in it starts with knowing that this is a huge asset class that needs better infrastructure.Dwarkesh PatelFor trading it in the aftermath of FTX and other things, I mean, all crypto companies have special scrutiny on them. And fairly or unfairly, if there's like FTX alumni, it'll be even more so. How are you convincing potential. Clients, investors, that crypto is safe, FTX alumni are safe.Brett HarrisonYeah. On the FTX alumni side, I just personally haven't had those issues really in recent months as we've been building out Architect. I hired like three, almost five now employees from formerly at FTX to come work with me.Dwarkesh PatelBut by the way, is that like some sort of ARB basically, that the overall hiring market is overcorrected on them or something? 100%, yeah.Brett HarrisonAnd not just an FTX right now. Like it is March 2023 as we're recording this, there is like a huge ARB in the hiring market. I mean, all the layoffs in tech and crypto, all of like the fear around various financial services means that like, we basically didn't need to work on recruiting. I had like the best people who worked for me at FTX US. I had ex colleagues of mine from former jobs that come work with me here. And we actually didn't have to do any formal recruiting efforts because of just how much supply there now is on the job market, especially in tech and finance. Luckily, I've had a long career history prior to FTX, and even at FTX we built really great stuff. We had a very good connections relationships with our customers and our investors. There would be times where on Twitter I would answer like a customer's support question at 02:00 in the morning and I maintained a lot of those relationships even through the collapse. And these are the kinds of people who are reaching out, like offering support, like offering to test out stuff, who want to be customers, who are also having problems with existing crypto tools, looking for something better. So all that stuff has remained intact. So I don't really have a concern there.Dwarkesh PatelWhat is institutional interest in crypto like at this point, given what's happened?Brett HarrisonI think it is just as great as it was before. Every major investment bank in the US has announced some plan to do something with blockchain technology still. Even like post FTX, the top trading institutions in the world are all continuing to trade it. I think as of what we're speaking about right now, volumes are down because people are sort of generally fearful. But I expect that to turn around pretty quickly and the institutional interest still remains really high. People are definitely expecting and waiting for proper regulatory oversight, especially in the US. That's still happening. People are waiting for higher grade professional tools that make it safe for them to trade and invest in crypto. I think that's obviously in the works for various things, Architect and otherwise, but otherwise the interest is all completely there.Dwarkesh PatelA broader question somebody could have about crypto at this point is, or maybe not crypto generally, but crypto trading is why is it good to have more crypto trading, at least with stocks and bonds and other kinds of traditional assets? As we were talking about earlier, you can tell a story that listening to health capital allocation projects that need funding, will get funding and so on. Why is it good if the price of bitcoin is efficient? Why is building a product that enables that something that's socially valuable?Brett HarrisonI think it boils down to, first of all, do you think it's important for people to be able to trade commodities? Like how important is it for the world that people can, you know, trade gold efficiently or they can trade oil efficiently? I think the answer is if people have a use for the underlying commodity, then it's important. And so maybe that is like what's the use of crypto? Well, I think each crypto token might have its own use. I don't think everyone has a good use. I think that there's a bunch that do. But if you believe in bitcoin as sort of like a store of value in a great medium of exchange, then it's important that there's a good fair price for bitcoin to enable that. If you think that the ether token is important for the gas fees required to run like a decentralized computer, and you think that the programs running on a decentralized computer are important, then it's important for there to be like a good price for ether that's fair. So I think it really depends on if you kind of believe in the underlying use case at all in the same way you would for kind of any commodity.Dwarkesh PatelGot it.Brett HarrisonAnd sometimes there are tokens that have more security like properties where they are like trading Apple stock. Basically there was an initial offering of that token and then if people bought it, it actually directly funded the product behind the token. And then the efficient trading of that token is sort of a barometer for the health of that particular company and they can like, sell more tokens to raise more capital and secondary offerings. In which case it looks very much exactly like the stock market.Dwarkesh PatelThat's a great leading to the next question, which is will there ever be a time when things that are equivalent to stocks or other kinds of equities will be traded on change in some sort of decentralized way?Brett HarrisonI think it's likely. I think the primary reason is that existing settlement systems in traditional markets seem to be very slow and built on very outdated technology that's like highly non transparent and very error prone. So equities are a prime example of this. Like it still requires two business days to settle a stock. And a frequent occurrence when I was at trading firms was that, you know, you would get your settlement file, that one told you, like, what trades were settled, and two told you things like if any corporate actions had occurred overnight, like paying a dividend or share split, and they would frequently be wrong, like the dividend would be wrong or would be missing, or the share split amount was for the wrong amount. Or they missed the day that it happened or they messed up. Some trades didn't get reported properly. They're just frequently mistakes. And it feels like there should be some easy, transparent kind of open, decentralized settlement layer for all things that you can trade. And rather than try to retrofit the existing settlement technology to be faster and better, starting from scratch with something like Blockchain could make a lot of sense, which is why you hear about a lot of these investment banks working on settling fixed income products on chain. The fixed income products have even worse settlement cycles than equities.Dwarkesh PatelShould the marginal crypto trader stop trading? Maybe this might also be a good question to ask by the marginal trader for on Robin Hood or something, but.Brett HarrisonI have a couple of thoughts about this. So the first is that I don't think crypto markets are as efficient as equity markets, so there's probably more opportunities for short and long term edge as a trader in crypto than there would be in equities. That being said, I think there's still an enormous amount of room in both traditional and crypto markets for even individuals to have and derive information that gives them profitable trading ideas. And I actually think it's the wrong conventional wisdom to think that if you are not Jane Street or Acidadel or Hudson River or Tower or Jump trading, then you have no chance of being able to profit in Marcus except for luck. I do think there are a lot of people who trade, and it's like pure speculation. It's not really like on me to tell them that they shouldn't speculate. They probably derive some personal enjoyment from speculation besides the opportunity for profit. But I do think the access to more sophisticated instruments and information has helped what have traditionally been players that have been unable to compete in the market actually be able to do so in a way that is systematically profitable.Dwarkesh PatelOkay, so that is, I think, a good point to end the conversation. We got to talk a chance to talk about a lot of things. Let's let the audience know where they can find out more of our Architect and also where they can find your Twitter and other sorts of links.Brett HarrisonYeah. So Architect's website is Architect.XYZ. We also have @architect_xyz on Twitter. And I'm Brett Harrison, 88 on Twitter.Dwarkesh PatelOkay, perfect. Awesome. Brett, thank you so much for being on the Lunar Society. This was a lot of fun.Brett HarrisonThank you so much.  Get full access to The Lunar Society at www.dwarkeshpatel.com/subscribe
3/13/20232 hours, 37 minutes, 38 seconds
Episode Artwork

Marc Andreessen - AI, Crypto, 1000 Elon Musks, Regrets, Vulnerabilities, & Managerial Revolution

My podcast with the brilliant Marc Andreessen is out!We discuss:* how AI will revolutionize software* whether NFTs are useless, & whether he should be funding flying cars instead* a16z's biggest vulnerabilities* the future of fusion, education, Twitter, venture, managerialism, & big techDwarkesh Patel has a great interview with Marc Andreessen. This one is full of great riffs: the idea that VC exists to restore pockets of bourgeois capitalism in a mostly managerial capitalist system, what makes the difference between good startup founders and good mature company executives, how valuation works at the earliest stages, and more. Dwarkesh tends to ask the questions other interviewers don't.Byrne Hobart, The DiffWatch on YouTube. Listen on Apple Podcasts, Spotify, or any other podcast platform. Read the full transcript here. Follow me on Twitter for updates on future episodes.Similar episodesYou may also enjoy my interview of Tyler Cowen about the pessimism of sex and identifying talent, Byrne Hobart about FTX and how drugs have shaped financial markets, and Bethany McLean about the astonishing similarities between FTX and the Enron story (which she broke).Side note: Paying the billsTo help pay the bills for my podcast, I'm turning on paid subscriptions on Substack.No major content will be paywalled - please don't donate if you have to think twice before buying a cup of coffee.But if you have the means & have enjoyed my podcast, I would appreciate your support 🙏.As always, the most helpful thing you can do is just to share the podcast - send it to friends, group chats, Twitter, Reddit, forums, and wherever else men and women of fine taste congregate.Timestamps(0:00:17) - Chewing glass(0:04:21) - AI(0:06:42) - Regrets(0:08:51) - Managerial capitalism(0:18:43) - 100 year fund(0:22:15) - Basic research(0:27:07) - $100b fund?(0:30:32) - Crypto debate(0:43:29) - Future of VC(0:50:20) - Founders(0:56:42) - a16z vulnerabilities(1:01:28) - Monetizing Twitter(1:07:09) - Future of big tech(1:14:07) - Is VC Overstaffed?TranscriptDwarkesh Patel 0:00Today, I have the great pleasure of speaking with Marc Andreessen, which means for the first time on the podcast, the guest’s and the host’s playback speed will actually match. Marc, welcome to The Lunar Society.Marc Andreessen 00:13Good morning. And thank you for having me. It's great to be here.Chewing glassDwarkesh Patel 00:17My pleasure. Have you been tempted anytime in the last 14 years to start a company? Not a16z, but another company?Marc Andreessen 00:24No. The short answer is we did. We started our venture firm in 2009 and it's given my partner, Ben and I, a chance to fully exercise our entrepreneurial ambitions and energies to build this firm. We're over 500 people now at the firm which is small for a tech company, but it's big for a venture capital firm. And it has let us get all those urges out.Dwarkesh Patel 00:50But there's no product where you think — “Oh God, this needs to exist, and I should be the one to make it happen”?Marc Andreessen 00:55I think of this a lot. We look at this through the lens of — “What would I do if I were 23 again?” And I always have those ideas. But starting a company is a real commitment, it really changes your life. My favorite all time quote on being a startup founder is from Sean Parker, who says —“Starting a company is like chewing glass. Eventually, you start to like the taste of your own blood.” I always get this queasy look on the face of people I’m talking to when I roll that quote out. But it is really intense. Whenever anybody asks me if they should start a company, the answer is always no. Because it's such a gigantic, emotional, irrational thing to do. The implications of that decision are so profound in terms of how you live your life. Look, there are plenty of great ideas, and plenty of interesting things to do but the actual process is so difficult. It gets romanticized a lot and it's not romantic. It's a very difficult thing to do. And I did it multiple times before, so at least for now, I don't revisit that.Dwarkesh Patel: 02:04But being a venture capitalist is not like that? When you're in the 38th pitch of the day, you're not wondering if chewing glass might not be more comfortable?Marc Andreessen 02:10No, it's different. I'll tell you how I experienced it. People are wired to respond to stress in different ways. And I think there are people who are wired to be extremely productive and get very happy under extreme levels of stress. I have a different… I'm fine with stress. In fact, I incline towards it and if I don't have any, I seek it out. But past a certain level, I don't really enjoy it. It degrades the quality of my life, not improves it. Maybe you have an affinity for self torture.Look, there's stress in every profession and there's certainly stress in being an investor, but it's a completely different kind of stress. Because when you're a startup founder, it's all on you. Everything that happens is on you, everything that goes wrong is on you. When there's an issue in the company, a crisis in the company, it's on you to fix it. You're up at four in the morning all the time worrying about things. With investors, there's just a layer of buffer. We have no end of problems and we help our portfolio companies as best we can with all kinds of issues, like some crisis inside a company. But it's not my company, not everything is my fault. So it’s a more diffused kind of stress, and honestly easier to deal with.Dwarkesh Patel 03:32Got it, that makes sense. Why did you stop your blog? Would you ever start it again?Marc Andreessen 03:37I write intermittently. The original blog was from 2007 to 2009. And then we started the firm, and that was like having a new baby and that soaked up all my time. I write intermittently, and then I do social media intermittently. Part of it is — I have a lot to say, and a lot that I'm interested in, but also I like to experiment with the new formats. We do live in a fundamentally different world as a result of social media, the internet, blogging, Twitter, and all the rest of it. So I try to keep my hand in it and experiment. But I rotate both how I spend my time and rotate what I think makes sense.AIDwarkesh Patel 04:21Before AWS, deploying applications was probably the bottleneck on new software. What is the biggest bottleneck today? At what layer of abstraction do we need new tools?Marc Andreessen 04:30Literally sitting here today, overwhelmingly it's the impact AI is having on coding. I think there's a real possibility that basically every application category gets upended in the next five years. I think the whole model of how applications get built across every domain might just completely change. In the old model without AI, you typically have some sort of database, you have some sort of front end for the database, you had forms, you had these known user interaction models, mobile apps and so forth. We got to a pretty good shared understanding of how humans and machines communicate, in the windowing era, and then in the mobile era, in the web era.AI might just upend all that. The future apps might just be much more of a dialogue between computer and machine. Either a written-text dialogue, or a spoken dialogue or some other form of dialogue. And the human is guiding the machine on what to do, and receiving real time feedback. And there's a loop, and then the machine just does what it does, and it gives you the results. I think we're potentially on the front end of that, that all might change. The very fundamental assumptions about how software gets built might just completely change. The tools on that are at the very front end. There's an entirely new stack that needs to get built to do that. So that's probably the big thing.Dwarkesh Patel 05:55Is there a reason though that AI is not one of your focus areas? As far as I know, you guys don't have an AI fund dedicated to that technology specifically?Marc Andreessen 06:03Basically we look at it all as software. We look at it like it is the core business. Software is the core of the firm, we've been public on that for a long time. The core venture fund is the core software fund. And then AI basically is the next turn on software. And so I view it as the opposite of what you said, it is the most integral thing that we're doing. The separate funds get created for the new areas that are structurally different in terms of how industries work. AI is basically the future of software. And so it's the future of the core of the firm.RegretsDwarkesh Patel 06:42Got it. Now, let's talk a little about your past. So you sold Netscape for $10 billion. But today, Chrome has what, like 2.7 billion users or something. And then Opsware was sold for like $1.7 billion. AWS is gonna probably make close to $100 billion in revenue yearly. In retrospect, do you think if these companies had remained startups, they would have ended up dominating these large markets?Marc Andreessen 07:03So I spend virtually no time on the past. The one thing I know about the past is I can't change it. So I spend virtually no time revisiting old decisions. People I know who spend a lot of time revisiting old decisions are less effective because they mire themselves in what ifs and counterfactuals. So I really don't spend time on it. I really don't even have theories on it. The big thing I would just say is that reality plays out in really complicated ways.Everything on paper is straightforward. Reality is very complicated and messy. The technical way that I think about it is basically every startup is charting a path dependent course through a complex adaptive system. And because of that, if you’ve read about this, people had this obsession a while back with what's called Chaos Theory.It's sort of this thing where we're used to thinking about systems as if they're deterministic. So you start at point A, you end up at point B, and you can do that over and over again. You know what happens when you drop an apple out of a tree, or whatever. In the real world of humans and 8 billion people interacting, and trying to start companies that intersect in these markets and do all these complicated things and have all these employees, there's random elements all over the place. There's path dependence as a consequence. You run the same scenario, start with point A, one time you end up point B, one time you end up point Z. There's a million reasons why the branches fork.This is my advice to every founder who wants to revisit all decisions. It's not a useful and productive thing to do. The world is too complicated and messy. So you take whatever skills you think you have and you just do something new.Managerial capitalismDwarkesh Patel 08:51Make sense. Are venture capitalists part of the managerial elite? Burnham says that “the rise of the finance capitalist is the decisive phase in the managerial revolution.” What would he think about venture capitalists?Marc Andreessen 09:04I actually think about this a lot. And I know you said everybody can Google it, but I'll just provide this just so this makes sense. James Burnham famously said — there's basically two kinds of capitalism and we call them both capitalism, but they're actually very different in how they operate. There's the old model of capitalism, which is bourgeois capitalism and bourgeois capitalism was the classic model where the owner of the business was a person who, by the way, often put their name on the door. Ford Motor Company, right?Dwarkesh Patel 09:31Andreessen HorowitzMarc Andreessen 09:32Andreessen Horowitz, right. And then that person owned the business, often 100% of the business, and then that person ran the business. These are the people that communists hated. This is the bourgeois capitalist — Company owner, builder, CEO, as one person with a direct link between ownership and control. The person who owns it controls it, the person who controls it runs it. It's just a thing. There's a proprietor of the business.So that's the old model. And then what he said basically, as of the middle of the 20th century, most of the economy was transitioning, and I think that transition has happened and is basically now complete. Most of the economy transitions to a different mode of operating, a different kind of capitalism called managerial capitalism. In managerial capitalism, you have a separation of ownership and management. Think of a public company, you have one set of owners who are dispersed shareholders, and there's like a million of them for a big company, and who knows where they are, and they're not paying any attention to the company, and they have no ability to run the company. And then you've got a professional managerial class, and they step in and they run the company. What he said is — as a consequence of that the managers end up in control. Even though the managers don't own the company. In a lot of public companies, the managers might own like 1% of the company, but they end up in total control, and then they can do whatever they want.And he actually said — Look, it doesn't even matter if you think this is good or bad, it's just inevitable. And it's inevitable because of scale and complexity. And so the modern industrial and post industrial organizations are going to end up being so big and so complex and so technical, that you're going to need this professional managerial class to run them. And it's just an inevitability that this is how it's gonna go. So I really think this is exactly what's played out.A consequence of that, that I think is pretty obvious, is that managerial capitalism has a big advantage that Burnham identified, which is that the managers are often very good at running things at scale. And we have these giant industries and sectors of the economy and health care and education, all these things that are running at giant levels of scale, which was new in the 20th century.But there's sort of a consequence of that, which is managers don't build new things. They're not trained to do it, they don't have the background to do it, they don't have the personality to do it, they don't have the temperament to do it, and they don't have the incentives to do it. Because the number one job, if you're a manager, is not to upset the applecart. You want to stay in that job for as long as possible, you want to get paid your annual comp for as long as possible, and you don't want to do anything that would introduce risk. And so managers can't and won't build new things.And so specifically, to your question, the role of startups, the role of entrepreneurial capitalism, is to basically bring back the old bourgeois capitalist model enough. It's a rump effort, because it's not most of the economy today, but bring back the older model of bourgeois capitalism, or what we call entrepreneurial capitalism, bring it back enough to at least be able to build the new things.So basically what we do is we fund the new bourgeois capitalists, who we call tech founders. And then there's two layers of finance that enable bourgeois capitalism to at least resurface a little bit within this managerial system. Venture capital does that at the point of inception, and then private equity does that at a point when a company needs to actually transform.I view it as — we're an enabling agent for at least enough of a resumption of bourgeois capitalism to be able to get new things built, even if most of the companies that we built ultimately themselves end up being run in the managerial model. And Burnham would say that's just the way of the modern world, that's just how it's gonna work.Dwarkesh Patel 13:10But you guys get preferred shares and board seats, and rightfully so, but wouldn't Burnham look at this and say — “You're not the owners and you do have some amount of control over your companies.”Marc Andreessen 13:20I think he would say that we're a hybrid, we're a managerial entity that is in the business of catalyzing and supporting bourgeois capitalist companies. He would clearly identify the startups that we fund. He would be like, “Oh yeah, that's the old model. That's the old model of Thomas Edison, or Henry Ford, or one of these guys.” You can just draw a straight line from Thomas Edison, Henry Ford to Steve Jobs, Larry Page, and Mark Zuckerberg. That's that model, it's a founder, it’s a CEO, at least when they started out owning 100%. They do have to raise money most of the time, but they're throwbacks. The modern tech founders are a throwback to this older model of bourgeois capitalism. So you're right in that he would view us as a managerial entity, but he would view us as a managerial entity that is in the business of causing new bourgeois capitalist institutions to at least be created. And I think he would credit us with that. And then he would also say — however, our fate is that most of the companies that we fund and most of the founders that we back end up over time, handing off control of their companies to a managerial class.When the companies we fund get to scale, they tend to get pulled into the managerial orbits, they tend to get pulled into the managerial matrix, which by the way, is when they stop being able to build new things, which is what causes the smart and aggressive people at those companies to leave and then come back to us and raise money and start a new bourgeois capitalist company.I view it as — the economy is like 99% managerial, and if we can just keep the 1% of the old model alive, we'll keep getting new things. By the way if venture capital ever gets snuffed, it's outlawed or whatever, it just fails and there is no more venture capital, there's no more tech startups or whatever then at that point the economy is going to be 100% managerial. And at that point, there will be no innovation forever.People might think they want that. I don't think they actually want that. I don't think we want to live in that world.Dwarkesh Patel 15:16Will this trend towards managerialism also happen to a16z as it scales? Or will it be immune? What happens to a16z in five decades?Marc Andreessen 15:23At a certain point this becomes the succession problem. As long as Ben and I are running it our determination is to keep it as much in the bourgeois model as possible. And as you pointed out, literally it’s our names on the door. Ben and I control the firm. The firm doesn't have a board of directors, it's just Ben and me running it. It's a private entity there’s no outside shareholders.And so as long as Ben and I are running it, and we're running it in the way that we're running it, it will be as bourgeois model as any investment firm could be.Some day there's the succession challenge, and I bring that up, because the succession challenge for tech companies is usually sort of when that transformation happens. When it goes from being in the bourgeois model to being in the managerial model.And then this gets to sort of the philosophy of succession in tech companies. And the general thing that happens there is that, you see this over and over again with the great founder CEOs, when it comes time to hand it off, there's basically two kinds of people that they can hand it off to. They can hand it off to somebody like them who's a mercurial, idiosyncratic, high disagreeableness, ornery, sort of entrepreneurial kind of personality, somebody in their mold. Or they can hand it off to somebody who knows how to run things at scale. Almost always, what they do is they hand it off to somebody who can run it at scale. The reason they do that is, there’s actually two reasons. There's the theoretical reason they do that which is — it is at scale at that point, and somebody does need to run it at scale. And then the other is, they often have what I call the long suffering number two. You've had this high octane founder CEO who breaks a lot of glass and then there's often the number two, like the chief operating officer or something who's the person who fundamentally keeps the trains running on time, and keeps everybody from quitting.And that long suffering number two has often been in that job for 10 or 15 years at that point, and is literally the longest suffering. They've always been the underling, and then it's like — Okay they now “deserve” the chance to run the company themselves. And that's the handover. Now, those founders often end up regretting that decision. And in later years, they will tell you — Boy, I wish I had handed it off to this other person who was maybe deeper in the organization who was maybe younger, who was more like I am, and maybe would have built more products and maybe that was a mistake. But the fact that they do this over and over again, to me illustrates why the Burnham theory is correct, which is — large, complex organizations ultimately do end up getting run by managers in almost all cases.The only optimistic view on that is that it's the transition from these companies being in the bourgeois capitalist model to the managerial model that creates the opportunity for the new generation of startups. Because then the counterfactual, if these companies remained bourgeois capitalist companies for 100 years, then they would be the companies to create all the new products, and then we wouldn't necessarily need to exist because those companies would just do what startups do. They just build all the new stuff.But because in that model, they won't do that and they don't do that, almost without exception. Therefore there's always the opportunity for the next new startup. And I think that's good. That keeps the economy vital, even in the face of this overwhelming trend towards managerialism.100 year fundDwarkesh Patel 18:43If you had a fund with a 100 year lock-in what would you be able to invest in that you can’t invest in right now?Marc Andreessen 18:50The base lockup for venture is like 10 years, and then we have the ability to push that out, we can kind of push that to 15. And for really high quality companies, we can push that to 20. We haven't been in business long enough to try to push it beyond that. So, we'll see.If you could push it to 100 years, the question is — is it really time that's the bottleneck? The implication of the question would be — are there more ambitious projects that would take longer, that you would fund that you're not funding because the time frames are too short. And the problem with a 100 year timeframe, or even a 50 year time frame, or even a 20 year timeframe is that new things don't tend to go through a 20 year incubation phase in business and then come out the other end and be good. What seems to happen is they need milestones, they need points of contact with reality. Every once in a while there will be a company, a very special company will get funded with a founder who's like — look, I'm gonna do the long term thing, and then they go into a tunnel for 10 or 15 years where they're building something and the theory is they're going to come out the other side. These have existed and these do get funded.Generally they never come up with anything. They end up in their own Private Idaho, they end up in their own internal worlds, they don't have contact with reality, they're not ever in the market, they're not working with customers. They just start to become bubbles of their own reality. Contact with the real world is difficult every single time. The real world is a pain in the butt. And mark to market your views of what you're doing with the reality of what anybody's actually going to want to pay for, requires you to go expose yourself to that. It's really hard to do that in the abstract, or to build a product that anybody's going to want to use. And so this thing where people go in a tunnel for 10, or 15, or 20 years, it doesn't go well. I think 100 years would be an even more degenerate version of that. Best case is this unbounded research lab that maybe would write papers and something maybe comes out the other end of the far future in the form of some open source thing or something, but they're not going to build an enterprise that way. And so I think having some level of contact with reality over the course of the first five to seven years is pretty important.The other way to get to the underlying question would be — what if you just had more zeros on the amount of money? What if instead of funding companies for $20 million, you could fund them for $2 billion, or $20 billion? In other words, maybe they would operate on the timeframe of today's companies, on a five or 10 year timeframe, but you can fund them with 20 billion of venture financing, instead of $20 million.I think that's a more interesting question. It's possible that there are pretty big fundamental things that could be built with larger amounts of money in this kind of entrepreneurial model. Every once in a while you do see these giants. Tesla and SpaceX are two obvious examples of these world changing things that just took a lot of money and then had a really big impact. So maybe there's something there, and maybe that's something that the venture ecosystem should experiment with in the years ahead. I would be more focused on that as opposed to elongating the time.Basic researchDwarkesh Patel 22:15But what about basic research? You've spoken about the dysfunctions of the academic-government-research complex. But within the next internet, the next thing that the Andreessen firm 10 years from now is building on top of, if the government effort is broken maybe you need to bootstrap something yourself. Have you considered that?Marc Andreessen 22:34The strong version of this argument is from a guy named Bill Janeway, a legendary VC. Janeway is a great, wonderful guy. If people haven't heard of him, he is a PhD in economics. I think he’s a student of a student of John Maynard Keynes. He comes from a high pedigree in economic theory background. And himself was a legendary venture capitalist in his career. He became a hands-on investor at the firm Warburg Pincus and funded some really interesting companies. And so he's one of these rare people who's both theoretical and practical on this kind of question. He wrote this book, which I really recommend, it's called Doing Capitalism where he goes through this question. The argument that he makes, along the lines of what you're saying, it’s a little bit of a pessimistic argument. The argument he makes is — if you look at the entire history of professional venture capital, which is now a 60 year journey, basically, or maybe even 50 years, from the late 60s, early 70s, in kind of modern form. He said the big category that's worked is computing or computer science. And then he said, the second category that's worked is biotech. And then he said, at least at the time of writing, everything else didn't work.And all the money that people poured into cleantech and da-da-da, all these other areas the venture capitalists tried to fund, they just didn't work from a return standpoint. You just burned the capital. When he wrote the book, he ran the numbers and computer sciences work twice as well as biotech or something like that. And then what he said is this is a direct result of federal research funding over the previous 50 years. Computer science based venture capital was able to productize 50 prior years of basic research in computer science, information science, information theory, communications theory, algorithms, all the stuff that was done in engineering schools from 1940 through like 1990.And so he said — we are productizing that, that's been the big thing. In Biotech we are productizing the work that NIH and others put into basic research in the biological sciences and that was about half as much money, and maybe half as much time. That work really started kicking in in the 60s and 70s, a little bit later.And then he said — Look, the problem is there aren't other sectors that have had these huge investments in basic research. There's just not this huge backlog of basic research into climate science or take your pick of online content, or whatever the other sectors are where people burn a lot of money.And so he says, if you want to predict the future venture capital, you basically just look at where the previous 50 years of basic research, R&D has happened, federal research funding has happened. He has a strong form of it, there's no shortcuts on this. And so if you're trying to do venture capital in a sector that doesn't have this big kind of install base of basic research has already happened, you're basically just tilting at windmills.I think there's a lot to his argument. I'm a little more optimistic about a broader spread of categories. A big reason I'm more optimistic about a broader set of categories is because computer science in particular, now applies across more categories. This was sort of the underlying point of the software eats the world thesis, which is that computers used to be just an industry where people made and sold computers. But now you can apply computer science into many other markets, financial services, and healthcare, and many, many others, where it can be a disruptive force. And so I think there's a payoff to computer science and software for sure, that can apply in these sectors. Maybe some of the biological sciences can be stretched into other sectors.There's a lot of smart people in the world, there's niche research efforts all over the place in many fields that are doing interesting work. Maybe you don't get a giant industry out the other end in some new sector, but maybe you get some very special companies. SpaceX is a massive advance in aeronautics, it took advantage of a lot of aeronautics R&D. It’s not like there's some huge aeronautics venture industry. But there is a big winner, at least one, and I think more to come. And so I'm a little bit more optimistic and open minded. Bill would probably say that I'm naive.$100b fund?Dwarkesh Patel 27:07You mentioned earlier about being able to potentially write 9 or 10 figure checks to companies like SpaceX or Tesla, who might require the capital to do something grand. Last I checked, you guys have $35 billion or something under management. Do we need to add a few more zeros to that as well? Will a16z’s assets under management just keep growing? Or will you cap it at some point.Marc Andreessen 27:27We cap it as best we can. We basically cap it to the opportunity set. And it may be obvious, but it's not a single chunk of money. It's broken into various strategies, and we apply different strategies to different sectors at different stages. So it's decomposed. And we have six primary investment groups internally in different stages, and so that money's broken out in different ways.We cap it as best we can to the opportunity set. We always tell LPs the same thing, which is we're not trying to grow assets under management, that's not a goal. To the best of our ability, we're trying to maintain whatever return level we're maintaining. We are trying to eat market share, we'd like to eat as much market share as possible. And then we would like to fully exploit the available opportunities, we'd like to fund all the really good founders, we'd like to back all the interesting new spaces. But what we wouldn't want to do is double assets under management in return for 5% lower returns or something like that. That would be a bad trade for us.So to put another zero on that, as I said, we would need a theory on a different kind of venture capital model, which would be trying to back much larger scale projects. And again, there's a really big argument you could make that that’s precisely what firms like ours should be doing. There are these really big problems in the world and maybe we just need to be much more aggressive about how we go at it. And we need founders who are more aggressive, and then we need to back them with more money.You can also argue either that wouldn't work, or we don't need it. The counter argument on the Tesla and SpaceX examples that I gave is that they didn't need it, right? They raised money the old fashioned way. They raised money round by round in the existing venture ecosystem. And so for whatever limitations you think the existing ecosystem has, and maybe it's not ambitious enough or whatever, it did fund Tesla and SpaceX.And so maybe it works. So the underlying question underneath all this is not the money part. The underlying question is how many great entrepreneurs are there? And then how many really big ideas are there for those entrepreneurs to go after? And then that goes one level deeper, which is — What makes a great entrepreneur? Are they born? Are they trained? What made Elon, Elon? What would you need to do to get ten more Elons? What would you need to do to get 100 more Elons? What would you need to do to make 1000 more Elons? Are they already out there and we just haven't found them yet? Could we grow them in tanks?Dwarkesh PatelOr just add testosterone to the water supply?Marc Andreessen 29:57Yeah or do we need a different kind of training program? Does there need to be a new kind of entrepreneurial university that trains entrepreneurs? It's just a totally different thing. Those are the underlying questions. I think if you show me ten more Elons, I'll figure out how to fund their companies. We work with a lot of great founders and we also work with Elon and he's still special. He's still highly unusual even relative to the other great entrepreneurs.Crypto debateDwarkesh Patel 30:32Yeah. Let's talk about crypto for a second. When you're investing in crypto projects, how do you distinguish between cases where there is some real new good or service that new technology is enabling and cases where it's just speculation of some kind?Marc Andreessen 30:45What we definitely don't do is the speculation side, we just don't do that. And I mean that very specifically, we're not running a hedge fund. What we do is we apply the classic venture capital 101 playbook to crypto. And we do that the exact same way that we do with every other venture sector that we invest in, which is to say we're trying to back new ventures. In crypto that venture might be a new company, or it might be a new network, or it might be a hybrid of the two and we're completely agnostic as to which way that goes. When we write our crypto term sheets, even when we're backing a crypto C Corp, we always write in the term sheet that they can flip it into being a tokenized network anytime they want to. We don't distinguish between companies and networks.But we approach it with a Venture Capital 101 playbook, which is — we're looking for really sharp founders who have a vision and the determination to go after it. Where there's some reason to believe that there's some sort of deep level of technological economic change happening, which is what you need for a new startup to wedge into a market. And that there's a reason for it to exist, that there's a market for what they're building and they're gonna build a product, and there's gonna be an intersection between product and market, and there's gonna be a way to make money and you know, the core playbook.We go into every crypto investment with the same timeframe as we go into venture investing. So we go in with at least a five to 10 year timeframe, if not a 15 to 20 year timeframe. That's what we do, the reason that's not necessarily the norm in crypto is an artifact of the fact that — especially anything with crypto tokens, there is this thing where they tend to publicly float a lot sooner than startup equity floats. Let's say we're backing a new crypto network, it goes ahead and floats a token as sort of one of the first steps of what it does. It has a liquid thing years in advance of when a corresponding normal C Corp would. There’s one thing in behavioral economics where when something has a daily price signal and where you can trade it, people tend to obsess on the daily price signal and they tend to trade it too much. There's all this literature on this that kind of shows how this happens. It's part of the human experience, we can't help ourselves, it's like moths to a flame. If I can trade the stock every day, I trade the stock every day.Almost every investor in almost every asset class trades too often in a way that damages their returns. And then as a consequence of that, what's happened is a lot of the investment firms that invest in crypto startups are actually hedge funds. They're structured as hedge funds, they have trading desks, they trade frequently, they have the equivalent of what's called a public book in hedge fund land. They've got these crypto assets they're trading frequently, and then they'll back a startup and then they'll trade that startup's token just like they trade Bitcoin or Ethereum.But in our view that's the wrong way. And by the way there's an incentive issue, which is they pay themselves on a hedge fund model, they pay themselves annually. So they're paying themselves annually based on the market for projects that might still be years away from realization of ultimate underlying value. And then there's this big issue of misalignment between them and their LPs. And so that's all led to this thing where the tokens for these crypto projects are traded too aggressively. In our model they just shouldn't be, they're just not ready for that yet. And so we anchor hard on the venture capital model, we treat these investments the exact same way as if we're investing in venture capital equity, we basically buy and hold for as long as we can. And have a real focus on the underlying intrinsic value of the product and technology that's being developed. If by speculation you mean daily trading and trying to look at prices and charts and all that stuff, we don’t do that.Dwarkesh Patel 34:22Or separately, another category would be things that are basically the equivalent of baseball cards, where there's no real good or service that's being created. It is something that you think might be valuable in the future but not because the GDP has gone up.Marc Andreessen 34:38Oh. Baseball cards are a totally valid good and service. That's a misnomer. I would entirely disagree with the premise of that question.Dwarkesh Patel 34:48But are they gonna raise median incomes even slightly?Marc Andreessen 34:50Yeah, there are people who make their living on baseball cards. Look, art has been a part of the economy for thousands of years. Art is one of the original things that people bought and sold. Art is fundamental to any economy. Would you really want to be part of an economy where they didn't value art? That would be depressing.Dwarkesh Patel 35:15Yeah but there's the question of — Do they value art versus are they speculating on art? And then how much of the effort is being spent on speculating on the art versus creating the art?Marc Andreessen 35:25Well, this gets into this old kind of cultural taboo. This depends on what you mean by speculation. If what you mean by speculation is obsessing on daily price signals and buying and selling and turning a portfolio, like being a day trader kind of speculation. That's what I think of speculation. Let's say that's the bad form of speculation, that's the non productive form.If by speculation, on the other hand, you mean — look, there are different kinds of things in the world that have different possible future values. And people are trying to estimate those future values, and people are trying to figure out utility, and they're trying to figure out aesthetic value.  Look at how the traditional art market works, is somebody supporting a new contemporary artist speculating or not? Yes, maybe from one lens they are. Maybe they're buying and selling paintings, and maybe they buy in and if it doesn't start going up in price, they flip it and buy something else. But also, maybe they're supporting a new young artist. And maybe they build a speculative portfolio of new young artists and as a consequence those artists can get paid, and they can afford to be full time artists. And then it turns out they're the next Picasso.And so I think that kind of speculation is good and healthy. And it's core to everything. I'd also say this — I don't know that there's actually a dividing line between that form of speculation, and speculation on what people call investments. Because even when people make investments, even just the institutional bond market. Look at US government debt, people are today in the bond market trying to figure out what that's worth. Because is the debt ceiling gonna get raised? Even that's up for grabs. To me, that’s not speculation in the bad sense, that's a market working properly. People are trying to estimate. Ben Graham said “financial markets are both a voting machine and a weighing machine. And in the short term, they tend to be a voting machine in the long run, they tend to be a weighing machine.”What's the difference between a voting machine and a weighing machine? I don't know, some people would say they're very different. Maybe it's actually the same thing. Why did prices go up? Because there are more buyers and sellers. Why do the prices go down? There were more sellers than buyers. The way markets work is you get individuals trying to make these estimations and then you get the collective effect. There's this dirty interpretation of any kind of trading or any kind of people trying to do the voting and weighing process. I just think it's this historical, ancient taboo against money. It's like in the Bible, Jesus kicking the money changers out of the temple. It's this old taboo against charging interest on debt. Different religions and cultures tend to have some underlying unease with the concept of money, the concept of trade, the concept of interest. And I just think it's like superstition, it's like resentment, it's fear of the unknown. But those things are the things that make economies work. And so I'm all in favor.Dwarkesh Patel 38:20I don't mean to get hung up on this — but if you think of something like the stock market or the bond market, fundamentally you can tell a story there. Where the reason what these stockbrokers or these hedge fund managers are doing is valuable, they're basically deciding where capital should go. Should we build a factory in Milwaukee? Should we build it in Toronto? Fundamentally, where should capital go? Whereas what is the story there? What is the NFT helping allocate the capital towards? Why does it matter if the price is efficient there?Marc Andreessen 38:48Because it's art. NFT is a very general concept. NFT is basically just a form of digital ownership. There will be many kinds of NFTs in the future, many of them, for example, will represent claims on real underlying property. I think a lot of real assets are gonna be wrapped in NFTs. And so NFTs are a very broad technological mechanism. But let's specifically take the form of NFT that everybody likes to criticize, which is NFT as a creative project or an image or a character in a fictional universe or something like that, the part that people like to beat on.And I'm just saying — they're just art. That's just digital art, right? And so every criticism people make of that is the same criticism you would make of buying and selling paintings, it would be the same buying and selling photographs, of buying and selling sculpture.I always like to really push this, what's the Mona Lisa worth? I don't want to spoil the movie. But the new Knives Out movie, let's just say the Mona Lisa plays a role in the movie. What's the Mona Lisa worth? One way of looking at the Mona Lisa is that it's worth the cost of producing it.  It's worth the canvas and the paint. And you could create a completely identical reproduction of the Mona Lisa with like 25 bucks of canvas and paint. So the Mona Lisa is worth 25 bucks. Or you could say the Mona Lisa is a cultural artifact and as a cultural artifact that's worth probably a billion dollars or $10 billion. Specifically on your question, what explains the spread between $25 and the $10 billion that it would go out if it ever hit the market. It’s because people care. Because it's art, because it's aesthetic, because it's cultural. Because it's part of what we've decided is the cultural heritage of humanity. The thing that makes life worth living is that it's not just about subsistence, that we are gonna have higher values and we're gonna value aesthetics.Dwarkesh Patel 40:35Do you see a difference between the funding the flying cars and the SpaceXs and Teslas versus something that improves the aesthetic heritage of humanity? But does one of them seem like a different category than the other to you? Or is that all included in the venture stuff you're interested in?Marc Andreessen 40:52It's a little bit like saying — should we fund Thomas Edison or Beethoven? If push comes to shove and we can only fund one of them, we probably should fund Edison and not Beethoven. Indoor lighting is probably more important than music. But I don't want to live without Beethoven.I think this is a very important point. People have lots and lots of views on human existence. There's lots and lots of people trying to figure out the point of human existence, religions and philosophies and so forth. But kind of what they all have in common, other than maybe Marxism, what they all have in common is — we're not just here to get up in the morning, work in a factory all day, go home at night, be depressed and sad, go to bed. We're not just material, right? Whatever this is all about, it's not just about materiality. There are higher aspirations and higher goals. And we create art, we create literature, we create paintings, we create sculptures, we create aesthetics, we create fashion, right, we create music, we create all of these things.And fiction. Why does fiction exist? Why is a fake story worth anything? Because it enhances your life to get wrapped up in a fake story. It makes your life better that these things exist. Imagine living in a world where there's no fiction, because everybody's like — “Oh, fiction is not useful. It's not real.” No, it's great. I want to live in a world where there's fiction. I like nothing more at the end of the day than having a couple hours to be able to get outside of my own head and watch a really good movie. And I don't want to live in a world where that doesn't happen.As a consequence, funding movies as another example of what you're talking about, is a thing that really makes the world better. And here's the other thing. The world we live in actually is the opposite of the world you're alluding to. The world we live in is not a world in which we have to choose between funding flying cars and funding NFTs or like in my example, funding Edison versus funding Beethoven. The world we live in is actually the opposite of that, where we have a massive oversupply of capital and not nearly enough things to fund.The nature of the modern economy is we have what Ben Bernanke called the global savings glut. We've just got this massive oversupply of capital that was generated by the last few 100 years of economic activity, and there's only one Elon. There's just this massive supply demand imbalance between the amount of capital that needs to generate a return and the actual number of viable investable projects and great entrepreneurs to actually create those projects. We certainly don’t have enough flying car startups, we also don't have enough art startups. We need more of all of this. I don't think there's a trade off, we need more of all of it.Future of VCDwarkesh Patel 43:29Have we reached the end of history when it comes to how venture capital works? For decades you get equity in these early stage companies, you invest more rounds, it's a 2-20 structure. Is that what venture is going to look like in 50 years, or what's going to change?Marc Andreessen 43:42I think the details will change, and the details have changed a lot, and the details will change a lot. If you go back to the late 60s, early 70s, the details were different then and the details were different 20 years ago. By the way, they're changing again right now in a bunch of ways, and so the details will change.Having said that, there’s a core activity that seems very fundamental. And the term I use I borrowed from Tyler Cowen who has talked about this, he calls it Project Picking. When you're doing new things, new tech startups, making new movies, publishing new books, creating new art, when you're doing something new. There's this pattern that just repeats over and over again. If you look back in history, it's basically been the pattern for hundreds or 1000s of years, and it seems like it's still the pattern. Which is, you're going to do something new, it's going to be very risky, it's going to be a very complex undertaking, it's going to be some very complicated effort that's going to involve a path dependent kind of journey through a complex adaptive system, reality is going to be very fuzzy and messy. And you're going to have a very idiosyncratic set of people who start and run that project. They're going to be highly disagreeable, ornery people because that's the kind of people who do new things. They're going to need to build something bigger than themselves, they're going to need to assemble a team and a whole effort. They're going to run into all kinds of problems and issues along the way.Every time you see that pattern there's this role, where there's somebody in the background who's like — Okay, this one, not that one. This founder, not that founder. This expedition, not that expedition. This movie, not that movie. And those people play a judgment and taste role, they play an endorsement, branding and marketing role. And then they often play a financing role. And they often are very hands-on, and they try to contribute to the success of the project.A historical example of this I always use is that the current model of venture capital is actually very similar to how whaling expeditions got funded 400 years ago. To the point that the term that we actually have, which is carried interest or carry, which is the profit sharing that the VCs get on a successful startup, that term actually goes back to the whaling industry 400 years ago, where the financiers of whaling journeys — like literally out of Moby Dick, to go hunt a whale and bring its carcass back to land.The carry was literally the percentage of the carried amount of whale that the investor’s got. It was called carry because it was literally the amount of whale that the ship could carry back. And so if you go back to how the whaling journeys off, like the coast of Maine and the 1600s, were funded, there were a group of what we — they didn't call themselves venture capitalist at that time, but there were a group of basically capitalists. And they would sit in a tavern or something, and they would get pitches by whaling captains.And you can imagine the whaling captains. A third of the whaling journeys never came back. A third of the time the boats got destroyed and everybody drowned. And so it's like — I'm the captain who's going to be able to not only go get the whale, but I'm gonna be able to keep my crew alive. By the way, I have a strategy and a theory for where the whale is.And maybe one guy is like — look, I'm gonna go where everybody knows there are whales and other guy’s gonna be like — no, that place is overfished, I'm gonna go to some other place where nobody thinks there's a whale, but I think there is. And then one guy is gonna say — I'm better at assembling a crew than the other. And the other one's like — Well, no, I don't even need a crew. I just need a bunch of grunts and I'm going to do all the work. And then another guy might say — I want a small fast boat. And other guy might say — I want a big slow boat.And so there's a set of people, imagine in the tavern under candlelight at night, debating all this back and forth — Okay, this captain on this journey, not that captain on that journey and then putting the money behind it to finance the thing. That's what they did then and that's still what we do. So what I'm pretty confident about is there will be somebody like us who is doing that in 50 years, 100 years, 200 years. It will be something like that. Will it be called venture capital? That I don't know. Where will it be happening? I don't know. But that seems like a very fundamental role.Dwarkesh Patel 47:56Will the public private distinction that exists now, will that exist in 50 years?Marc Andreessen 48:00You mean like companies going public?Dwarkesh Patel 48:07Yeah and just the fact that there's different rules for investing in both and just a separate category? Is that gonna exist?Marc Andreessen 48:12There's already shades of gray. I would say that's already dissolving. There's very formal rules here. But there's already shading that is taking place. In the last 20 years, it's become much more common for especially later stage private companies to have their stocks actually trade. Actually be semi liquid and trade either through secondary exchanges or tender offers or whatever. That didn't used to happen, that didn't really happen in the 1990s. And then it started happening in the late 2000s.And then you've got lots of people with different kinds of approaches to have different kinds of private markets and new kinds of private liquidity. And look, you've got these new mechanisms, you've got crypto tokens. You've got entirely new mechanisms as well popping up representing underlying value.And then arguments, debates all the time in public and with regulators and in the newspapers about what counts — Who can invest in? This whole accredited investor thing. A lot of this is around “protecting investors”. And then there's this concept of high net worth investors should be allowed to take more risk, because they can bear the losses. Whereas normal investors should not be allowed to invest in private companies, but then there's a counter argument that says, then you're cutting off growth investing as an opportunity for normal investors, and you're making wealth inequality worse.That debate will keep playing out. It'll kind of fuzz a bit. I'd expect both sides will moderate a little bit. So in other words, public companies will get to be a little bit more liquid over time. The definition of what it means to be public will probably broaden out. I'll give you an example. Here's an interesting thing. So you can have this interesting case where you can take a company private, but it's still effectively public because it has publicly traded bonds. And then it ends up with publicly filed financials on the bond side, even though its stock is private. And so it's effectively still public because of information disclosure. And then the argument is — well, if I already have full information disclosure, as a result of the bonds trading, you might as well take the stock public again. Anyway it'll fuzz out somewhere in there.FoundersDwarkesh Patel 50:20Okay, so there's a clear pipeline of successful founders, who then become venture capitalists like yourself, obviously. But I'm curious why the opposite is not more true? So if you're a venture capitalist, you've seen dozens of companies go through hundreds of different problems. And you would think that this puts you in a perfect position to be a great entrepreneur. So why don't more venture capitalists become entrepreneurs?Marc Andreessen 50:40One reason is it's just harder to build a company, it just flat out is. It's not easy to be a VC, but it's harder to build a company. And it requires a level of personal commitment. Successful venture capitalists do get to a point in life where they start to become pretty comfortable. They make money and they start to settle into that sort of fairly nice way of living at some point in a lot of cases. And going back to the 2 AM chewing glass kind of thing is maybe a little bit of a stretch for how they want to spend their time.So that's part of it. The other part of it is — the activities are pretty different. The way I describe it is — actually starting and running a company is a full on contact sport, it's a hundred decisions a day. I’ll give an example: bias to action. Anybody who's running a company, you have to have a bias to action. You're faced with a hundred decisions a day, you don't have definitive answers on any of them. And you have to actually act anyway. Because if you sit and analyze the world will pass you by. And it's like — a good plan executed violently is much better than a great plan executed later.So it's a mode of operating that rewards aggression, contact with reality, constantly testing hypotheses, screwing up a lot, changing your mind a lot, revisiting things. It’s thousands and thousands of crazy real world variables all intersecting.Being an investor is different. It's much more analytical, clinical, outside-in. The decision cycles are much longer, you get a much longer period of time to think about what you should invest in, you get a much longer period of time to figure out when you should sell. Like I said, you generally don't want to trade frequently if you're doing your job right. You actually want to take a long time to really make the investment decisions, and then make the ultimate sale decisions.VCs, we help along the way, when companies have issues that they're in the middle of. But fundamentally, it's a much bigger level of watching, observing, learning, thinking, arguing,in the abstract, as opposed to day to day bloody combat.Honestly, it's a little bit like — Why don't the great football broadcasters go get on the field? Try being the running back for a season?Dwarkesh Patel 53:12Got it. How soon can you tell whether somebody will make for a good CEO of a large company specifically? So can you tell as soon as they've got a new startup that they're pitching you? Or does it become more clear over time as they get more and more employees?Marc Andreessen 53:25The big thing with being able to run things at scale, there's actually a very big breakthrough that people either make or they don't make. And the very big breakthrough is whether they know how to manage managers. Say you're running a company with a hundred thousand employees, you don't have a hundred thousand direct reports. You still only have like eight or ten direct reports. And then each of them have eight or ten direct reports and each of them have eight or ten direct reports. And so even the CEOs of really big companies, they're only really dealing with eight or ten or twelve people on a daily basis.And then how do you become trained as a manager? The way you become trained as a manager initially is you manage a team of individual contributors. I'm an engineering manager, I have eight or ten coders working for me. And then the breakthrough is — am I trained in how to become a manager of managers?If I'm early in my career, the way I think about that is I start out as an individual contributor, let's say an engineer. I get trained on how to be a manager of individual contributors, and that makes me an engineering manager. And then if I get promoted to what they call engineering director, which is one level up, now I'm a director and now I'm managing a team of managers. Anybody who can make that jump now has a generalizable skill of being able to manage managers, and then what makes that skill so great is that skill can scale. Because then you can get promoted to the VP of engineering, now you have a team of directors who have teams of managers who have teams of ICs and so forth. And then at some point, if you keep climbing that ladder, at some point you get promoted to CEO. And then you have a team of managers who are the executives of the company, and then everything spans out from there.And so if you can manage managers, at least in theory, you have the basic skill and temperament required to be able to scale all the way up. Then it becomes a question of how much complexity can you deal with? Can you learn enough about all the different domains of what it means to run a business? Are you going to enjoy being in the job and being on the hot seat? All those kinds of questions.I think 100% of the people we back have the intelligence to do it, maybe half of them have the temperament to do it, and then maybe half of those have the intelligence and the temperament and they really want to do it. And by “might want to do it” I mean, 20 years from now, they still want to be running their company.And enough of them where we get the success cases. But having said that as an entrepreneur, you have to really want that. You have to be smart enough and you have to have the temperament and you have to actually want to learn the skills. And not everybody is able to line those up.Dwarkesh Patel 55:54Got it, got it. Managing the managerial revolution.Marc Andreessen 56:00Actually, that's exactly right. The best case scenario is a bourgeois capitalist, entrepreneurial CEO, managing a team of managers who are doing all the managerial stuff required at scale. That's the best case scenario for a large modern organization. Best of both worlds, they're able to harness the benefits of scale, and they're able to still build new things.The degenerate version of that is a manager running a company of people who in theory can build your products. But in the Burnham sense, if the CEO is the manager who is running a team of people who want to build their products, that company probably will not actually build their products. Those people will probably all leave and start their own companies.a16z vulnerabilitiesDwarkesh Patel 56:42Yep, yep. Now, as unlikely as this may be, just humor the hypothetical. Let's say a16z for the next 10 to 20 years has mediocre returns. If you had to guess looking back, what would be the most likely reason this might happen? Would it have to be some sort of macro headwind, would it have to be betting on the wrong tech sectors, what would it have to be?Marc Andreessen 57:0020 years is a long enough time where it's probably not just a macroeconomic thing. The big macro cycles seem to play out over 7 to 10 year periods. And so over 20 years, you'd expect to kind of get two or three big cycles through that. And so you'd expect to get at least some chance to make money and harvest profits. Probably it wouldn't be a macro problem. Look, you can imagine it, if a real pandemic happens. By the way, I’m now gonna get you demonetized on Google because I'm going to reference pandemics but..Dwarkesh Patel 57:34Don’t worry, I didn't have enough views to be monetized anyway.Marc Andreessen 57:38If something horrible happens then you could be in a ditch for 20 years. But if things continue the way that they have, for the last 50 years or 80 years. There'll be multiple cycles, and there'll be a chance to make money for people who make good investments.So it's probably not that, and then there'll be the micro explanation, which is we just make bad investments. We invest the money, but we just invest in the wrong companies and we screw up. And that's of course always a possibility. And probably the most likely downside case.The other downside case is — I would build on what I was mentioning earlier, from Bill Janeway. The other downside case would just be that there's just not enough technological change happening. There wasn't enough investment in basic research in the preceding 50 years in areas that actually paid off. There wasn't enough underlying technological change that provided an opportunity for new entrepreneurial innovation. And the entrepreneurs started the companies and they tried to build products and we funded them and for whatever reason, the sectors in which everybody was operating just didn't pay off.If we hit five clean-tech sectors in a row or something like that, the whole thing just doesn't work. In a sense, that's the scariest one because that's the one that's most out of our control. That's purely exogenous. We can't wish new science into existence. And so that would be a scary one. I don't think that's the case. And in fact, I think quite possibly the opposite is happening. But that would be the downside scenario.Dwarkesh Patel 59:15How vulnerable is a16z to any given single tech sector not working out? Whether it's because of technical immaturity, or by their regulation or anything else? But if your top sector doesn't work out, how vulnerable is the whole firm?Marc Andreessen 59:29Innovation could just be outlawed. And that's a real risk, because innovation is outlawed in big and important areas like Nuclear. I always love meeting with new nuclear entrepreneurs, because it's just so obvious that we should have this big investment in nuclear energy and there's all these new designs. But the Nuclear Regulatory Commission has not authorized a new nuclear design since its inception nearly 50 years ago. So it's just illegal to build new nuclear in the US. By the way, there's all these fusion entrepreneurs that are super geniuses, the products are great, it looks fantastic. I just don't think there's any prospect of nuclear fusion being legal in the US. I think it's just impossible and can't be done. Maybe it's just all outlawed, in which case, at a societal level we will deserve the result. But that would be a bummer for us.Dwarkesh Patel 1:00:14And then I don't know, let's say crypto gets regulated or it's just not ready yet. It doesn't have to be crypto specifically. But what happens to a16z as a whole? I mean, does a whole firm carry on? Or?Marc Andreessen 1:00:28Look, it's up to our LPs. We raise money on a cycle. So our LPs have an option every cycle to not continue to invest. Just logically the firm is somewhat diversified now. We have six primary investment domains. So at least in theory, we have some diversification across categories. At least in theory, we could lose a category or two and the investment returns could still be good, and the investors will still fund us. The downside case from there would be that those categories are actually more correlated than we would want them to be. As a firm, we have a big focus on software, we think software is a wedge across each of those verticals. Maybe AI turns out for whatever reason not to work, or gets outlawed or something or just fundamentally makes economics worse or something. Then you can imagine that hitting multiple sectors. Again, I don't think that's going to happen, but I guess it's a possibility.Monetizing TwitterDwarkesh Patel 1:01:28Yeah. What did the old management of Twitter fail to see about the potential of the platform?Marc Andreessen 1:01:30So first I'd say that I have a very hard time second guessing management teams, because like I said, my belief is that it's so easy to criticize companies and teams in the outside, it's so hard to run these companies, there are always a thousand factors that are invisible from the outside that make it really hard to make decisions internally.By the way, the histories on all this stuff are really always screwed up. Because what you almost always find in the history of the great companies is that there were moments early on where it was really tenuous, and it could have easily gone the other way. Netflix could have sold out to Blockbuster early on, and Google could have sold out to Yahoo. And we never would have even heard of those companies. And so it's really, really hard to second guess.I guess I will just put it this way — I've always believed and I was an angel investor in Twitter back when it first got started. I've always believed that the public graph is something that should just be titanically valuable in the world. The public-follow graph. In computer science terms, Twitter is what's called publish subscribe to the idea of a one way public follow graph.That ought to be just absolutely titanically valuable, that ought to be the most valuable content, loyalty brand signal in the world. That ought to be the most complete expression of what people care about in the world, that ought to be the primary way that every creator of everything interacts with their customers and their audience. This ought to be where all the politics operates, this ought to be where every creative profession operates, this ought to be where a huge amount of the economy operates.They were always on to such a big idea. Like with everything, it's a question of — What does that mean in terms of what kind of product you can build around that? And then how can you get people to pay for it?  But yeah, I've always viewed that the economic opportunity around that core innovation that they had is just much, much larger than anybody has seen so far.Dwarkesh Patel 1:03:21But how specifically do you monetize that graph?Marc Andreessen 1:03:22Oh, there's a gazillion ways. There's tons and tons of ways. Elon has talked about this publicly so it’s not spoiling anything, but Twitter is a promotional vehicle for a lot of people who will then provide you stuff on another platform.I'm just taking an obvious example. He has talked about video. People create video, they market it on Twitter, and then they monetize it on YouTube. Like, why? Why is that not happening (on Twitter)? Musicians will have followings of 5-10 million people on Twitter, they aren't selling concert tickets.I'm sure this was happening before but where it first came to mind was, if you remember Conan O'Brien when he got famously fired from the Tonight Show, he did this tour. And I was a fan of his so I was following him at the time. He did his first live comedy music tour. And he sold out the tour across 40 cities in like two hours. How did he do it? Well, he just put up on his Twitter account. He said — I'm going on the road, here are the dates, click here to buy tickets.Boom, they all sold out. “Now click here to buy tickets” was not “click here to buy tickets on Twitter.” It was “click here to buy tickets somewhere else”. But why isn't every concert in the world, why isn't every live event getting booked on Twitter? There's a lot of this kind of thing.As Elon is fond of saying, it's not rocket science.Dwarkesh Patel 1:04:38Yeah. It's funny that a few revolutions in the Middle East were organized in the same way that Conan O'Brien organizes tour, just by posting it on Twitter.Marc Andreessen 1:04:54So this is the thing that got me so convinced on social media relatively early. Even before the Arab Spring, I don’t know if you remember, you might be too young, but there was this overwhelming critique of social media between inception in like 2001 to basically mainstreaming in like, 2011-2012. There was a decade where there was just this overwhelming critique from all the smart people, as I like to say. That was basically — this thing is useless. This is narcissism. This is just pointless self ego stroking, like narcissism. Nobody cares. The cliche always was Twitter is where you go to learn what somebody's cat had for breakfast. Who cares what your cat had for breakfast? Nothing will ever come from any of this. And then I remember, you could pick up any newspaper on any given day through that period and you could read something like this.And then I remember when Erdoğan was consolidating control of Turkey. Erdoğan came out and he said, “I think Twitter is the primary challenge to the survival of any political regime in the modern world,” And I was like — Okay, all the smart analysts all think this is worthless and then a guy who's actually trying to keep control of a country is like, this is my number one threat. The spread of what that meant, of what the outcomes meant. I was just like — “Oh my god.”My conclusion was Erdoğan is right and all the smart westerners are wrong. And quite honestly, I think it’s still quite early on. We’re still pretty early in the long arc of social media. The high level thing here would be — the world in which 5 billion people are on the internet is still only a decade or so old. That’s still really early. The world in which 5 billion people are on social networks is like five years old. It’s still super early. If you just look at the history of these transitions in the past, just look at the printing press as a prescient example. It took 200 years to fully play out the consequences of the printing press. We’re still in the very early stages with these things.Future of big techDwarkesh Patel 1:07:09I was like ten in 2011 so I don’t know if I would’ve personally. I would’ve liked to think I would’ve caught on if I was older but maybe not. It’s hard to know. But it is kind of interesting. You are personally invested in every single major social media company. So it’s interesting to get your thoughts on where that sector might go. Do you think the next ten years will look like the last ten years when it comes to Big Tech? Is it just going to keep becoming a bigger fraction of GDP? Will that ever stop?Marc Andreessen 1:07:35As a fraction of GDP, it’s only gonna go up. It is the process of tech infusing itself in every sector. And I think that’s just an overwhelming trend. Because there are better ways to do things. There are things that are possible today that were not possible ten years ago. There are things that will be possible five years from now that aren’t possible today. So from a sector standpoint, the sector will certainly rise as a percent. I’m putting my money where my mouth is in the following statement — Entrepreneurial capitalism will deliver most of that. A lot of that gain will be in companies that were funded in the venture capital, silicon valley kind of model. For the basic reason we discussed which is you do need to have that throwback to the bourgeois capitalist model to do new things. Incumbents are generally still very poor at changing themselves in response to new technology for reasons we’ve discussed. So that process will continue to play out. Another thing that I would just highlight is — The opportunity set for tech is changing over time in another interesting way. We’ve been good at going over the dynamic but small slices of GDP in the last fifty years. And more and more now, we’re going to be going after the less dynamic but much larger sectors of GDP. Education, healthcare, real estate, finance, law, government are really starting to come up for grabs. They are very complicated markets and they’re hard to function in. As startups, it’s harder to build the companies but the payoff is potentially much bigger. Because those are such huge slices of GDP. So the shape of the industry will change a bit over time. What is technology? Technology is a better way of doing things. At some point, the better way of doing things is the way that people do things. At some point that does shift market share from people doing things the old to the people doing things the new way.Dwarkesh Patel 1:09:32But let's say you build a better education system somehow. The government is still going to be dumping trillions of dollars into the old education system or the old healthcare system. Do you just accept this as a lost cause that basically 50% of the GDP will just be wasted but we’ll make the other 50% really good? When you're building alternatives, do you just accept the loss of the existing system?Marc Andreessen 1:09:56Education is a great example. I think the incumbent education system is trying to destroy itself. It and the people running it, and the people funding it are trying to kill it. And they’re doing it every possible way they can. For K-12 they are trying to prioritize the teachers over the students which is the opposite of what any properly run company would do. At the university level, the problems in the modern university have been well covered by other people. They have become a cartel. Stanford now has more administrators than they have students. No company would run that way.Dwarkesh Patel 1:10:40There’s a positive vision where you could turn that into the Bloom two-sigma, single student for single administrator but I don’t think that’s what's happening.Marc Andreessen 1:10:49Yes, yes. That’s correct. You could and they’re not. That’s exactly right. And then you see the federal student loan kind of crazy thing. By the way, the universities are voluntarily shutting down use of admissions testing. They’re shutting down SAT, ACT, GRE. They’re very deliberately eliminating the intelligence signal which is a big part of the signal employers piggyback on top of. They become intensely politicized. We now know through the replication crisis that most of the research that happens in these universities is fake. Most of it is not generating real research results. We know that because it won’t replicate. You’ve just got these increasingly disconnected mentalities and there’s some set of people who are obviously going to keep going to these schools.And then you just look at cost. A degree from a mainstream university that costs, in ten years, a half-million or a million dollars that has no intelligence signal attached to it anymore. Where most of the classes are fake, where most of the degrees are fake, most of the research is fake, where they are wrapped up in these political obsessions. That’s probably not the future of how employers are going to staff. That’s probably not where people are actually going to learn valuable marketable skills. The last thing they want is to actually teach somebody a marketable skill. Teaching somebody a marketable skill is just so far down in the list of priorities of a university now it’s not even in the top 20.Lot of it is just they’re a cartel. They operate as a cartel, they run as a cartel, it is a literal cartel. And the cartel is administered through the agencies, the quasi-governmental bodies that determine who gets access to federal student loan funding. And those bodies are staffed by the current university administrators. So it’s a self-governing cartel. It does exactly what cartels do, it’s stagnating and going crazy in spectacular ways.There’s clearly going to be an educational revolution. Does that happen today or five years or ten years, I don’t know. Does it happen in the form of new in-person tuitions versus internet-based? I don’t know. Is it driven by us or is it driven by employers who just get fed-up and they’re like — “Screw it. We’re not gonna live like this anymore and we’re gonna hire people in a totally different way.” That, I don’t know. There’s lots and lots of questions about what’s gonna happen from here. But the system is breaking in fundamental and obvious ways.Healthcare, same thing. It’s extraordinarily difficult to find positive outcomes in healthcare. In other words, there’s lots of activity in healthcare. It’s very hard to find anything that causes people to live longer. Or to be healthier longer. Every once in a while there’s a successful new cancer treatment or something but there are all these analyses that show that massive investment and public support for health insurance and all these things. And the health outcomes basically don’t move.To the extent that people care at all about the reality of their health, there are going to have to be new ways of doing things and tech is going to be the way through the market for people who have those ideas.Is VC Overstaffed?Dwarkesh Patel 1:14:07Hopefully these revolutions in education and healthcare are not like healthcare itself where we are always twenty years away from a cure to cancer and we’re always twenty years away from making education technological.You’ve talked about how big tech is 2 to 4x overstaffed in the best case. I’m curious how overstaffed do you think venture capital is? How many partners and associates could we let go and there really wouldn’t be a difference in the performance of venture capital.Marc Andreessen 1:14:30My friend Andy Rachleff, who is the founder of Benchmark and teaches venture capital at Stanford. I think his description of this is correct. He says — Venture capital is always over staffed and over funded. His estimate is that it is overfunded by a factor of 5. It should probably be 20% of the size that it is. It should be 20% of the number of people, it should be 20% of the number of funds, it should be 20% of the amount of money. And his conclusion after watching this for a long time and analyzing it was it’s basically a permanent 5x overfunding, overstaffing. It goes to what I referenced earlier which is, the world we live in just has a massive imbalance of too much money chasing too few opportunities to invest the money productively. There’s just too much money that needs long run returns that looks to venture as part of their asset allocation. In the way that modern investors do asset allocation. The full version of it he describes is that — there’s only ever been two models of institutional investment. There’s the old model of institutional investment which is 60-40 stocks and bonds that kind of dominated the 20th century up until the 1970s. And there’s what’s called the Swensen model. Swensen who created the Yale endowment in its modern form and that’s the model that all the endowments and foundations have today and increasingly sovereign wealth funds, where they invest in alternative assets. Which means hedge funds, venture capital, real estate and things that aren't stocks and bonds. So anybody following the Swensen model has an allocation to venture capital, on average that’s maybe 4% of their assets. But 4% of the entire global asset base is just a gigantic number. It’s like someone once said — It’s like having a 6th marriage, hope triumphing over experience.The thing you will hear from LPs is every LP says they only invest in the top ten venture capital funds and every LP has a different list for who that is. They all kind of know that the whole sector is overfunded, but they all kind of know that they suffer from a real lack of... where else is the money going to go? And then, it’s always possible that you’ll have some great new fund, there’s some great new sector that will open up. A huge advantage that venture capital has is the long dated part of it. It means you don’t suffer the consequences of a bad venture capital investment upfront. You get a ten year lease on life when you make a venture capital investment. You’re not gonna get judged for a long time. And so I think that causes people to invest more in this sector than they should.Dwarkesh Patel 1:17:09Is the winner's curse also a big component here where the guy who bids the most is the one who sets the price?Marc Andreessen 1:17:14That can happen. At the early stages the best companies tend to raise at less than the optimal price because the signal of who invests is more important than the absolute price. And so almost every investment that we fund at the Series A stage, they could raise money at 2-4 times the price they raised from us. But they value the signal. And I think that’s also true of the seed landscape and it’s also still true in a lot of cases at the series B level. Series C and beyond it becomes much more of an efficient market.Again it’s not a full auction. It’s a little bit like your earlier question. At least here’s the theory — it’s not just money, it’s not just a straight up liquid financial market. These are whaling journeys. By the way, there’s a much blunter answer to this question which is — people who raise seed money and series A money from the high bidder often end up really regretting it because they end up raising money from people who don’t actually understand the nature of a whaling journey, or a tech startup. And then they panic at the wrong times and they freak out. And the wrong investors can really screw up a company. At least historically, there’s a self-correcting equilibrium that comes out of that. Where the best entrepreneurs understand that they want someone on their team who really know what they’re doing and they don’t want to take chances with someone that’s gonna freak out and try to shut the company down the first time that something goes wrong. But we’ll see. Get full access to The Lunar Society at www.dwarkeshpatel.com/subscribe
2/1/20231 hour, 19 minutes, 31 seconds
Episode Artwork

Garett Jones - Immigration, National IQ, & Less Democracy

Garett Jones is an economist at George Mason University and the author of The Cultural Transplant, Hive Mind, and 10% Less Democracy.This episode was fun and interesting throughout!He explains:* Why national IQ matters* How migrants bring their values to their new countries* Why we should have less democracy* How the Chinese are an unstoppable global force for free marketsWatch on YouTube. Listen on Apple Podcasts, Spotify, or any other podcast platform. Timestamps(00:00:00) - Intro(00:01:08) - Migrants Change Countries with Culture or Votes?(00:09:15) - Impact of Immigrants on Markets & Corruption(00:12:02) - 50% Open Borders?(00:16:54) - Chinese are Unstoppable Capitalists (00:21:39) - Innovation & Immigrants (00:24:53) - Open Borders for Migrants Equivalent to Americans?(00:28:54) - Let's Ignore Side Effects?(00:30:25) - Are Poor Countries Stuck?(00:32:26) - How Can Effective Altruists Increase National IQ(00:39:13) - Clone a million John von Neumann?(00:44:39) - Genetic Selection for IQ(00:47:02) - Democracy, Fed, FDA, & Presidential Power(00:49:42) - EU is a force for good?(00:55:12) - Why is America More Libertarian Than Median Voter?(00:56:19) - Is Ethnic Conflict a Short Run Problem?(00:59:38) - Bond Holder Democracy(01:04:57) - Mormonism(01:08:52) - Garett Jones's Immigration System(01:10:12) - Interviewing SBFTranscriptThis transcript was autogenerated and thus may contain errors.[00:00:41] Dwarkesh Patel: Okay. Today I have the pleasure of speaking with Garrett Jones, who is an economist at George Mason University. . He's most recently the author of the Cultural Trans. How migrants make the economies. They move to a lot like the ones they left, but he's also the author of 10% Less Democracy and Hive Mind. We'll get into all three of those books. Garrett, welcome to the podcast. [00:01:06] Garett Jones: Glad to be here.Thanks for having me.[00:01:08] Migrants Change Countries with Culture or Votes?[00:01:08] Garett Jones: Um, [00:01:09] Dwarkesh Patel: first question is, isn't the cultural transplant still a continuation of your argument against democracy? Because the isn't one of the reasons we care about the values of migrants, the fact that we eliminate democracy. So should review this book as part of your critique against democracy rather than against migration specifically.[00:01:27] Garett Jones: Um, well, I do think that, uh, governments and productivity are shaped by the citizens in a nation in, in almost any event. Um, I think that even as we've seen recently in China, even in a very strong authoritarian dictatorship, which some would call totalitarian, even there, the government has to listen to the masses.So the government can only get so far away from the masses on average, even in, uh, an autocracy. If you had [00:01:57] Dwarkesh Patel: to split apart the contribution though, um, the, the impact of migrants on, let's say the culture versus the impact that migrants have on a country by voting in their political system, um, uh, how, how would you split that apart?Is, is the, is mainly the impact we, the cultural impact we see for migration due to the ability of migrants to vote or because they're just influencing the culture just by being [00:02:19] Garett Jones: there? I'll cheat a little bit because we don't get to run experiments on this, so I just have to kind of guess, uh, make an informed guess.I, I'm gonna call it 50 50. Um, so the way people, uh, the way citizens influence a country through formal democracy is important. Uh, but citizens end up placing some kind of limits on the government anyway. And the people in the country are the, they're the folks who are gonna work in the firms and be able to either establish or not establish.Those complicated networks of exchange that are crucial to high productivity. . ,[00:02:52] Mean vs Elite IQ[00:02:52] Dwarkesh Patel: I wanna linger on hive mind a little bit before we talk about the cultural transplant. Um, if you had to guess, does, do the benefits of National IQ come from having a right tail of elites that is smarter or is it from not having that strong of a left tail of people who are, you know, lower productivity, more like markedly to commit crimes and things like that?In other words? Uh, yeah, go ahead. [00:03:14] Garett Jones: Yeah. Yeah. I, I think, uh, the upper tail is gonna matter more than the lower tail, um, in, in the normal range of variation. Uh, and I think part of that is because, uh, nations, at least moderately prosperous nations have found tools for basically reducing the influence of the least informed voters.And for. Uh, basically being able to keep productivity up even when there are folks who are sort of disrupting the whole process. Um, you know, the, the, the risks of crime from the lower end is basically like a probabilistic risk. It's not like it's, it's not like some, uh, zero to one switch or anything. So we're talking about something probabilistic.And I think that, uh, it's the, the median versus the elite is the, is the contrast that I find more interesting. Um, uh, median voter theorem, you know, normal, the way we often think about democracy says that the median should be matter more for determining productivity and for shaping institutions. Um, and I tend to think that that's more important in democracies for sure.So when we look at countries, if you just look at a scatter plot, just look at the raw data of a scatter plot. If you look at the few countries that are exceptions to the rule, where the mean is the mean, IQ is the best predictor of productivity compared to elite iq. Um, . The exceptions are non democracies and South Africa.So you see a few, uh, places in the Gulf where there are large migrant communities who are exceptionally well educated, exceptionally cognitively talented. Um, and that's associated with high productivity. Those are a couple of Gulf states. It's probably cutter, the UAE might be Bahrain in there, I'm not sure.Um, and then you've got South Africa. Those are the, those are the countries where the average test score, it doesn't have to be iq, it could be just Pisa, Tim's type stuff. Um, those are the exceptions to the rule that the average iq, the mean IQ is the best predictor of national productivity. [00:05:14] Dwarkesh Patel: Hmm. Uh, interesting.Um, does that imply the fact that the, um, at least in certain contexts, the elite IQ matters more than the left tail. Does that imply that we should want a greater deviation of IQ in a country? That you could just push a button and increase that deviation? Would that be good? [00:05:33] Garett Jones: No. No, I don't think so. Uh oh.If you could just increase the deviation, um, holding the mean constant. Yeah. Yeah. I think so. In the normal range of variation. Yeah. Yeah. Yeah. So, mm-hmm. , um, is it, and I think that it has more effects. It, no, it's people at the top who are, um, tend to be coming up with, uh, the big breakthroughs, the big scientific breakthroughs, the big intellectual breakthroughs that end up spilling over to the whole world.Basically the, the positive externalities of innovation. This is a very, almost Pollyanna-ish, uh, Paul Roamer new endogenous, new, uh, new growth theory thing, right? Which is the innovations of the elite, a swamp, uh, the negatives of the low skilled among. , [00:06:14] Dwarkesh Patel: can we just apply this line of reasoning to low skilled immigration as well?Then that maybe the average goes down, the average IQ of your country goes down if the, if you just let in, you know, millions of low skilled immigration immigrants, and maybe there's some cultural effects to that too. But, you know, you're also going to the, that the elite IQ will still be preserved and more elites will come in through the borders, along with the low scale migrants.So then, you know, since we're caring about the devi deviation anyways, uh, more immigration might increase the deviation. Uh, and then, you know, the, we just, uh, that's a good [00:06:46] Garett Jones: thing. So notice what you did there is you, you did something that didn't just, uh, increase the variance. You simultaneously increase the variance and lowered the mean Yeah.Yeah. And median, right? And so I think that, uh, hurting the mean and median is actually a big cost, especially in democracies. And so that is very likely to swamp, uh, the benefits of, um, the small, the small probability of getting. Hire elite folks in as part of a low-skilled immigration policy. Mm-hmm. , so pulling down the mean or the median is that that's a, that's that swamps that swamps the benefits of increasing variants there.Yeah. Yes. [00:07:26] Dwarkesh Patel: But if you get rid of their migrant's ability to vote, and I guess you can't do that, but let assume you could do that. Yeah. What exactly is, like, what is the exec mechanism by which the, the, the cultural values or the lower median is impacting the elite's ability to produce these valuable externalities?You know, like there's a standard compared to advantage story that, you know, they'll, they'll do the housework and the cooking for the elites and they can do the more productive [00:07:52] Garett Jones: Yeah. Taking all the institutions as given, which is what a lot of open borders optimists do. They take institutions as given they take cultural norms as given.Um, all that micro stuff works out just fine. I'm totally, I'm totally on board with all that sort of Adam Smith division of labor. Blah, blah, blah. Um, but, institutions are downstream of culture and, uh, cultural norms will be changing partly because of what I call spaghetti theory, right?We meet in the middle when new folks come to a country. There's some kind of convergence, some part where people meet in the middle, um, between the, the values, uh, that were previously existing and the values that have shown up, uh, that migrants have brought with them. So, you know, like I I call it spaghetti theory because, um, when Italians moved to America, that got Americans eating more spaghetti, right?And if you just did a simple assimilation analysis, you'd say, wow, everybody in America eats the same now, like the burgers and spaghetti. So look, the Italians assimilated, but migrants assimilate us. Um, uh, native Americans certainly changed in response to the movement of Europeans. Um, English Americans certainly changed in response to the migration of German and Irish Americans.So this meeting in the middle is something that happens all the time, and not just through Democratic channels, just through the sort of soft contact of cultural norms that sociologists and social psychologists would understand. [00:09:15] Impact of Immigrants on Markets & Corruption[00:09:15] Garett Jones: Um, no, I'm sure you saw the book that was released, I think in 2020 titled, uh, retro Refuse, uh, where they showed, uh, slight positive relationship between, uh, immigration and, you know, pro-market, uh, laws.[00:09:27] Dwarkesh Patel: And I guess the idea behind that is there's selection effects in terms of who would come to a country like America in [00:09:32] Garett Jones: the first place. Well, they never ran the statistical analysis that would be most useful. I think they said that. Uh, so this is Powell and Na Roth Day. Yeah. They ran a statistical analysis that said, and they said, in all of the statistical analysis we've ever run, we've never found negative relationship between low-skilled migration, any measure of it, and changes in economic freedom.And, um, I actually borrowed another one of Powell's data sets, and I thought, well, how would I check this theory out? The idea that changes in migration have an effect on economic freedom? And I just used the normal economist tool. I thought about how do economists check to see if changes in money, changes in the money supply, change the price level.That's what we call the quantity theory, right? Mm-hmm. , the way you do that is on the x-axis. You, you show the change in the money supply On the y axis, you show the change in prices, right? This Milton Friedman's idea. Money's always everywhere. Yeah. Inflation's always neverwhere Montessori phenomenon. So that's what I did.Uh, I did this with a, with a, um, a student. Uh, we co-authored a paper doing this. And the very first statistical analysis we ran, we looked at migrants who came from countries that were substantially more, uh, corrupt than the country's average. And we looked at the, the different, the relationship between cha, an increase in migrants from corrupt countries, and subsequent changes in economic freedom.Every single statistical analysis we found had a negative relationship. , we ran the simplest estimate you could run. Right? Change on change. Change in one thing, predicts change in another. They somehow never got around to running that very simple statistical analysis. CH one change predicts another change.Hmm. We found negative relationships every time. Sometimes statistically significant, sometimes not always negative. Somehow they never found that. I just don't know how . But [00:11:21] Dwarkesh Patel: what about the anecdotal evidence that in the US for example, the, in the periods of the greatest expansion of the welfare state or of governed power during the New Deal or great society, the levels of foreign-born people were at like historical lows.Uh, is that just a coincidence or what, what do you think of? I'm [00:11:38] Garett Jones: not really interested in, uh, migration per se. Right. My story is never that, like migration per se, does this bad thing. Migrants are bad. That's never my story, right? Mm-hmm. , as you know, right? Yeah. Yeah. So my story is that migrants bring, uh, cultural values from their old country to their new country.And sometimes those cultural norms are better than what you've got, and sometimes they're worse than what you've got. And sometimes it's just up for debate. [00:12:02] 50% Open Borders?[00:12:02] Dwarkesh Patel: So if you had to guess what percentage of the world has cultural values that are equivalent to or better than the average of Americas? [00:12:11] Garett Jones: Uh oh.Equivalent to or better then? Yeah. Uh, I mean, just off the top of my head, maybe 20%. I dunno, 30%. I'll just throw something out there like that. Yeah. So I mean, like for country averages, right? Yeah, yeah, yeah, yeah, yeah. Um, [00:12:25] Dwarkesh Patel: currently we probably don't have, uh, it would probably be hard for like 20% of the rest of the world to get into the us.Um, w w would you support some, uh, PO policy that would make it easy for people from those countries specifically to get to the us? Just, uh, have radical immigration liberalization from those places? [00:12:44] Garett Jones: Um, that's really not my comparative advantage to have opinions about that, but like, substantial increases of people who pass multiple tests, like, let's take the low hanging fruit and then move down from there.Right? So people from, uh, countries, uh, that ha um, on average have say higher savings rates, um, higher, uh, education levels. Higher s what I call s a t, deep root scores and, um, countries that are, say half a standard deviation above the US level on all three, [00:13:18] Dwarkesh Patel: right? Why do they have to be higher? Why not just equivalent, like, uh, you get all the gains from trade and plus it can't be, you know, equivalent.So it's, there's no [00:13:27] Garett Jones: trade. Part of the reason is because the entire world depends on US innovation. So we should make America as good as possible, not just slightly better than it is. So very few firms would find that their optimal hiring policy would be hire anyone who's better than your current stock of employees.Would you agree with that? [00:13:42] Dwarkesh Patel: Yeah. But you, uh, have to pay them a salary. If you're just, uh, if it's just somebody just comes to the us, you don't have to like pay them a salary, right? So if somebody is better, that, if somebody's producing more value for a firm than the salary would pay them, I think [00:13:52] Garett Jones: like is is a firm's job to maximize its profits or to just make a little bit more than it's making right?Maximize profits. But yeah, there you go. So you pack, you find the best people you can, you know, sports teams that are hiring don't just say, we wanna hire people who are better than what we got. They say, let's get the best people we can get. Why not get the best? That was Jim Jimmy Carter's, that was Jimmy Carter's, uh, biography.Why not the best. But you, [00:14:16] Dwarkesh Patel: you can do that along with getting people who are, you know, unexpected, uh, terms as good as the existing Americans. Why gives [00:14:24] Garett Jones: y'all like, I don't care what you, why you want this? This seems like crazy, right? What are you talking about? But [00:14:29] Dwarkesh Patel: I, I'm not sure why not the best what the trade out there, huh?No, I'm not saying you don't get the best, but I, I'm saying once you've gotten the best, what is the harm in getting the people who have equivalent s a t scores and, and the rest of the things you [00:14:41] Garett Jones: mentioned. I think part of the reason would be you'd wanna find out, I mean, if you really wanna do something super hardcore, you'd have to find out what's best for the planet as a whole.What's the trade off between, um, Having the very best, uh, most innovative, talented, frugal people in America doing innovating that has benefits for the whole world, versus having an America that's like 40% better, but we're the median's a little bit, the median of skills a little bit lower. Right. Uh, because the median's shaping the productivity of the whole team.Right? Yeah. This is what you, you know what it means when you believe in externalities, right? [00:15:14] Dwarkesh Patel: But if you have somebody who's equivalent by definition, they're not moving the median down. [00:15:19] Garett Jones: That's, you're, you're totally right about that. Yeah. But like, why wouldn't I want the best thing possible? Right. Okay.I'm still trying to figure out why you wouldn't want the best thing possible. You're trying to go, why? I don't want the best thing possible. I'm like, why not? [00:15:31] Dwarkesh Patel: I'm not disagreeing with you. I'm just, I'm a little bit confused about why that that precludes you from also getting the second best thing possible.At the same time you're, because you're not limited to just the best. [00:15:42] Garett Jones: Right. Well, uh, because the second best is going to have a negative externality on the first best. Everything's externalities. This is my worldview, right? Everything's externalities. You bring in the second best, you're like, you're not, that person's gonna make things on average a little worse for the first best person.[00:16:00] Dwarkesh Patel: But it seems like you were explaining earlier that the negative externalities are coming from people from countries with, uh, low s a t scores. And by the way, s a t you can explain what that means just for the audience who's not familiar with how you're using that term. [00:16:11] Garett Jones: Oh yeah. So, um, there, there are three prominent, uh, measures in what's known as the deep roots literature and, uh, that are widely used.Uh, two are s n a, that state history and agricultural history. That's how many thousands of years your ancestors have had experience living under organized states or living unsettled agriculture. And then the T-score is the tech history score. I used the measure from 1500. It's basically what fraction of the world's technology were your ancestors using in 1500 before, uh, Columbus and his expansive conquest ended up upending the entire world.Uh, the world map. So s a and T are all predictors of modern prosperity, but especially when you adjust for migration. [00:16:54] Chinese are Unstoppable Capitalists [00:16:54] Garett Jones: Gotcha. [00:16:55] Dwarkesh Patel: We can come back to this later, but one of the interesting things I think from the book was you have this chapter on China and the Chinese people as a sort of unstoppable force for free market capitalism.Mm-hmm. . Um, and it's interesting, as you mentioned in the book, that China is a poorest majority Chinese country. Um, what do you think explains why China is a poorest, uh, majority Chinese country? Maybe are there like non-linear dynamics here where, uh, if you go from 90 40 to 90% Chinese, there's positive effects, but if you go from 90 to 95% Chinese, there's too much?[00:17:26] Garett Jones: No, I think it's just, I think just communism is dumb and it has terrible, like sometimes decades long effects on institutional quality. I don't really quite understand. So I'd say North Korea, if we had good data on North Korea, North Korea would be even a bigger sort of deep roots outlier than China is.Right? It's like, don't, don't have a communist dictatorship in your country. Seems to be pretty, a robust lesson for a national prosperity. China's still stuck with a sort of crummy version of that mistakes still. North Korea, of course, is stuck with an even worse version. So I think that's, I, my hunch is that that's, you know, the overwhelming issue there.Um, it's, it's something that, it's, it's sort of a China's stuck in an ins. Currently China's stuck in an institutional cul-de-sac and they just don't quite know how to get out of it. And it's, uh, bad for a lot of, for the people who live there. On average, if the other side had won the Chinese Civil War, things would probably be a lot, lot better off in China today.Yeah. [00:18:22] Dwarkesh Patel: Um, but what, what is that suggestion about the deep roots literature? If the three biggest countries in the world, China, India, and America, Um, it, it, it under predicts their performance, or sorry, in the case of China and India, it, uh, it, it over predicts their performance. And in the case of America, it under predicts maybe the, how, how reliable is this if like the three biggest countries in the world are not, uh, adequately accounted for?[00:18:45] Garett Jones: Uh, well, you know, communism's a really big mistake. I, I think that's totally accounted for right there. Um, I think India's underperformance isn't that huge. Um, the US is a miracle along many ways. Um, it's, we should draw our lessons from the typical country, and I think, uh, population weighted estimates, I don't think that basically one third of the knowledge about the wealth of nations comes from the current GDP per capita of China, India, and the us, right?I think much less than one third of the story of the wealth of nations comes from those three. And, uh, again, in, in all three cases though, if you look at the economic trajectories of all three of those people, oh, all three of those countries, uh, they're all, uh, China and India growing faster than you'd expect.And also, I wanna point out. This is the most important point actually. Um, when we look at, uh, when Kaplan made this claim, right? Brian Kaplan has made this claim, right? Yeah. That the SATs, that the ancestry scores, the deep root scores don't predict, um, the prosperity of, uh, the, the low performance of Indian China.He only checked the S and the A and the s a T scores. Okay. Which letter did he not predict? Which letter did he never test out? He never tested the T. What do you think happens when he tests the T? Does it predict, uh, China [00:20:02] Dwarkesh Patel: and India and America, [00:20:03] Garett Jones: Hey, start, they t goes back to being statistically significant again, UhhuhSo with T, which we've always known is the best of the deep root scores, somehow Kaplan never managed to measure that one. Just as Powell Naste never managed to run the simplest test change in, uh, migrant corruption versus change in economic institutions somehow, like the simplest test just never get run.[00:20:26] Dwarkesh Patel: Okay. And then what is the impact if you include t. If you, [00:20:29] Garett Jones: if you, if you look at tea, then, um, then, uh, contrary to what Kaplan says, uh, the deep roots, that deep roots measure is sig statistically significant. [00:20:38] Dwarkesh Patel: Okay. Um, yeah, I, [00:20:40] Garett Jones: interesting. The puzzle goes away, [00:20:42] Dwarkesh Patel: interesting. [00:20:43] Garett Jones: Um, yeah. So somehow these guys just never seem to run like the simple things, the transparent things.I don't know [00:20:49] Dwarkesh Patel: why the, um, the weird, huh? The, the, the one you mentioned from, what was it Nassa, the name of the guy who wrote the Richard at re refuse [00:20:57] Garett Jones: the Yeah, yeah. Powell Naste. [00:20:59] Dwarkesh Patel: Yeah. Yeah. That you said you did the regression on institutional corruption, uh, and from the countries to come from. Is that, was that right?I, [00:21:06] Garett Jones: and so yeah. The, the measure they use, I just took, I took Powell's dataset from another study, and it was the percent. Of it was basically, um, the percentage of your nation's population, the percentage increase in your nation's population from relatively poor or corrupt countries. They had multiple measures, UhhuhSo, and what is on the y axis there? Y axis is change in economic freedom. That's my preferred one. Gotcha. There's also a change in corruption one, which is a noisier indicator. Um, you get much clearer results with change in economic freedom, so. Gotcha, [00:21:38] Dwarkesh Patel: gotcha. [00:21:39] Innovation & Immigrants [00:21:39] Dwarkesh Patel: Um, now does the ideas getting harder to find stuff and great stagnation, does that imply we should be less worried about impinging on the innovation engine in these, uh, countries that people might wanna migrate to?Because worse comes to worst. It's not like there are a whole bunch of great new theories that were gonna come out anyways. [00:21:58] Garett Jones: Uh, no. I think that, I think that it's always good to have great things, um, and new ideas. Yes, new ideas are getting harder to find, but, um, that, but that the awesome ideas that we're still getting are still worth so much.Right. If we're still increasing lifespan a month, a year, uh, for every year of research we're doing, like, that just seems great. Right? A decade that adds a year to life, so, mm-hmm. , just to use a rough, uh, ballpark measure there. But, so we [00:22:25] Dwarkesh Patel: have a lot of these countries where a lot of innovation is happening.So let's say we kept, uh, one or two of them as, you know, immigrate, uh, havens from any potential, uh, downsides, from radical changes. You know, we already had this in the case of Japan or South Korea, there's not that much of migration there. Mm-hmm. . What is, what is a harm in then using the other ones to decrease global poverty by immigration or something like [00:22:48] Garett Jones: that?Well, um, it's obviously better to create a couple of innovation powerhouses, um, rather than none. Right? So obviously that's, that's nice. But instead, I would prefer to have, um, open borders for Iceland if the Open borders advocates are right and open borders. , we'll have no noticeable effect on institutional quality, then it's great to move, , to have our open borders experiment run in a country that's lightly populated, has a lot of open land, and, um, has good institutional quality.And Iceland fits the bill perfectly for that. So we could preserve the institutional innovation skill, uh, the institutional quality of the, the what I call the I seven. Uh, that's, you know, China, Japan, South Korea, the us, Germany, uk, France, and choose any country out of the a hundred, out of the couple of dozen countries that have good institutional quality.Just pick one of the others that aren't one of those seven, pick one that's not an innovation powerhouse and turn that into your open borders, uh, country. Um, you could, uh, if you wanted to get basically Singapore levels of population density in Iceland, that'd be about 300 million people, I think. I think I, that's about what the numbers end up looking like.Something like that. But [00:24:00] Dwarkesh Patel: the, so you can put entire, but, but the value of open borders comes from the fact that you're coming to a country with high conglomerations of talent and capital and other things, which is, uh, not true of Iceland. Right. So isn't the entire [00:24:13] Garett Jones: No, no. I thought the whole point of open borders, that there's institutional quality and there's some exogenous institutions that make that place more productive than other places.Mm-hmm. . And so by move, I, I, that's my version of what I've been exposed to as open borders, the, is that institutions exogenously exist. There's some places have, uh, moderately laissez, fairer institutions in their country and moving a lot more people there will not reduce the productivity of the people who are currently there, and they'll become much more productive.And so, like the institu, you know, the institutional quality's crucial. So, I mean, if you're a real geography guy, you'd be excited about the fact that Iceland is so far, so close to the north. because latitude is a predictor of prosperity. [00:24:53] Open Borders for Migrants Equivalent to Americans?[00:24:53] Dwarkesh Patel: Um, I want to go back to the thing about, well, should we have open border for that 20% of the Popula global world's population that comes from Yeah.Um, equivalent, s a t and other sort of cultural traits as America. Mm-hmm. , because I feel like this is important enough to dwell on it. You know, it seems similar to saying that once picked up a hundred dollars bill on the floor, you wouldn't pick up a $20 bill on the floor cuz you only won the best bill.Uh, the $20 bills is right there. Why not pick it up? Um, [00:25:18] Garett Jones: so what if you have, yeah. What if the $20 bill makes your, turns, your, uh, a hundred dollars bill into like an $80 bill and turns all of your 80 a hundred dollars bills and $80 bills. [00:25:27] Dwarkesh Patel: But is it, aren't your controlling for that by saying that they have equivalent scores along all those cultural tests that you're.[00:25:34] Garett Jones: No, because, um, the median, so, so take the simple version of my story, which is the median of the population ends up shaping the productivity of everybody in the country. Right? Or the mean, right? The mean skill level ends up shaping the productivity of the entire population. Right? So that means we end up, I mean, I, I try not to math this up.I don't wanna math this up for the, you know, in a popular book, but it means we face a trade off between being small, a small country with super awesome, uh, positive externalities for all the workers by just selecting the best people. And every time we lower the average skill level in the country, we're lowering the average productivity of everyone else we're creating.We didn't, [00:26:11] Dwarkesh Patel: what? We didn't lower it. So you have to have skills that are lower compared to, than the median of a median American. You, [00:26:18] Garett Jones: so this is, this is a c Paraba story, right? Like if you could suppose the US is at 80 now on a zero to a hundred scale, right? Just, just saying it's 80. Yeah. Yeah. And you have a choice between being hundred and being 99.if you're at 99, the 99 is making, all compared to the world of average of a hundred, the world of an average 99 is making, reducing the productivity of all those hundreds. Okay. So if we chose 90, we're reducing the productivity of all those hundreds. [00:26:48] Dwarkesh Patel: Yes. Okay. So let's say we admit all the smartest people in the world, and that gets us from 80 to 85.That's a new, that's a new media in America. Yeah. At that point. And, but this is because we've admitted a whole bunch of like 90 nines that have just increased our average. Yeah, yeah. Um, at that point, open borders for everybody who's ever been 85, [00:27:08] Garett Jones: like I, this is, this is, ends up being a math problem. It's a little hard to solve on a podcast, right?Because it's the, it's the question of do I want a smaller country with super high average productivity? Or a bigger country with lower average productivity. And by average productivity, I don't just mean, uh, uh, a compositional effect. I mean negative external, I mean relatively fewer positive externalities.So I'll use the term relatively fewer positive externalities rather than negative externalities, right? So like, I don't exactly know where this is. Trade off's gonna pan out, but, um, there is a case for a sort of Manhattan when people talk about a Manhattan project, right? They're talking about putting all like a small number of the smartest people in a room.And part of the reason you don't want like the 20th, smartest person in the room. Cause, cuz that person's gonna ruin the ruin stuff for our, for the other smart people. I, it's amazing how your worldview changes when you see everybody as an external. I, [00:28:02] Dwarkesh Patel: I'm kind of confused about this because just having, at some point you're gonna run outta the smartest people, the remainder of the smartest people in the world.If you've admitted all the brilliant people. Yeah. And given how big the US population is to begin with, you're not gonna change the median that much by doing that. Right. So it's, it's almost a global end to just having more births from the average American. Like if, if the average American just had more kids, the population would still grow.Mm-hmm. and the relative effect of the brightest people might dilute a little bit. Um, but I I, [00:28:33] Garett Jones: and that maybe that's a huge tragedy. We don't know without a bunch of extra math and a bunch of weird assumptions. We don't know. So like I'm, there's a point at which I have to say like, I don't know. Right. Okay.Yeah. Uh, yeah. Is diluting the power of the smartest person in America, like keeping us from having wondrous miracles all around us all the time? I mean, probably not, but. I don't know, [00:28:53] Dwarkesh Patel: but, [00:28:54] Let's Ignore Side Effects?[00:28:54] Dwarkesh Patel: but I guess the sort of the meta question you can ask about this entire debate is, listen, there's so much literature here and it's hard to tell what exactly will happen.You know, it's possible that culture will become worse. It's possible, it'll become better. It's possible to stay the same, given the fact that there's this ambiguity. Why not just do the thing that on the first order of effect seems good? And, you know, just like moving somebody who's like in a poor country to a rich country, first order effect seems good.I don't know how the third and fourth order effect shapes out. Let's just, you know, let's just do the simple obvious thing. [00:29:22] Garett Jones: I, I thought that the, one of the great ideas of economics is that we have to worry about secondary and tertiary consequences. Right? [00:29:28] Dwarkesh Patel: But if, if we, if we can't even figure out what they are exactly, why not just do the thing that at the first order seems, uh, good.[00:29:35] Garett Jones: Um, because if you have a compelling reason to think that the, uh, direction of strength of the second and third and fourth order things are negative and the variances are really wide, then you're just adding a lot more uncertainty to your outcomes. So, And adding uncertainty or outcomes that has sizable negative tail, especially for the whole planet.Isn't that great? Go ahead and run your experiments in Iceland. Let's run that for 50 years and see what happens. It's weird how everybody's obsessed with it running the experiment in America, right? Why not running in Iceland first? Because America's [00:30:05] Dwarkesh Patel: got a great, a lot of great institutions right there.We can check and see what [00:30:08] Garett Jones: Iceland Iceland's a great place too. Um, and I use Iceland as a metaphor, right? Like it's, people are obsessed with running it in America. Like there's some kind of need. I don't know why. So let's try in France. Um, let's try, let's try Northern Ireland. , [00:30:24] Dwarkesh Patel: uh, are,[00:30:25] Are Poor Countries Stuck?[00:30:25] Dwarkesh Patel: are places with low s a t scores and again, s a t we're not talking about the, uh, in case you're skipping to the timestamp, we're not talking about the college test.Um, the deep roots. [00:30:35] Garett Jones: S a t Exactly. Uh, state history, agricultural history, tech history. [00:30:38] Dwarkesh Patel: Right. Exactly. Are, are those places with, uh, low scores on, um, on that test? Are they stuck there forever? Or, uh, is there something that can be done if you are a country that has had a short or not significant history of, um, technology or agriculture?[00:30:56] Garett Jones: Well, the, I start off the book with this, which I really think that, uh, one thing they could do is, uh, create a welcoming environment for large numbers of Chinese migrants to move there persistently. I don't think that's of course the only thing that could ever work, but I think it's something that's within the range of policy for at least some poor countries.I don't know which ones, but, uh, some poor countries could follow the. Approach that many countries in Southeast Asia followed, which has created an environment that's welcoming, welcoming enough to Chinese migrants. Um, it's the one country in the world with large numbers of high s a t score, uh, with alar, with a high s a T score culture, large population.It's enough of an economic failure, so for at least a little longer that, uh, folks can, might be able to be interested in moving to a poor country with lower s a t scores. In a better world, you can do this with North Korea too, but the population of North Korea isn't big enough to make a big dent in the world, right?Mm-hmm. , uh, China's population is big enough. Yeah. [00:31:54] Dwarkesh Patel: Another thing you're worried have to worry about in those cases though though, is the risk that if you do become successful in that country, there's just gonna be a huge backlash and your resources will. AppD, like what happened famously. [00:32:05] Garett Jones: So in, in Indonesia, right?Yeah. There have been many Oh, yeah, yeah. Times across Southeast Asia where anti-Chinese pogroms have been, um, uh, unfortunately a fact of life. So, yeah. Yeah. [00:32:15] Dwarkesh Patel: Or Indians in Uganda under, uh, IDI. I, I mean, yeah. Yep. Um, yeah. Yeah. Uh, okay. So actually I, I'm curious how you would think about this given the impact of National iq.[00:32:26] How Can Effective Altruists Increase National IQ[00:32:26] Dwarkesh Patel: Um, if you're an effective altruist, what, uh, are you just, uh, handing out iodine tablets, uh, across, across the world? What, what are you doing to increase national [00:32:34] Garett Jones: iq? Yeah. This is places, this is something that I, yes. Uh, finding ways I, this is what I call a, a Flynn cycle. Like I wish, I'm hoping for a world where there are enough public health interventions and probably K through six education interventions.to boost test scores in the world's poorest countries. And I think that ha ends up having, um, uh, a virtuous cycle to it, right? As people get more productive, then they can afford more public health, which makes them more productive, which means they can afford more public health. I think brain health is an important and neglected part of child development.Um, fortunately we've done a fair amount to reduce the amount of environmental lead, um, in a lot of poor countries. That's probably having a good effect right now as we speak in a lot of the world's poorest countries. You're right. Um, iodine, basic childhood nutrition, uh, reliable healthcare, uh, to, you know, prevent the worst kinds of just mild childhood infections that are probably, uh, creating what the, what they, what economists sometimes call health.Things that end up just hurting you in a way that causes, uh, an ill-defined long-term cost. A lot of that's gonna have to show up in the, in the brain. Um, I'm a big fan of the, of the view that part of the Flynn Effect is, uh, basically nutrition and health. Mm-hmm. , uh, Flynn wasn't a huge believer in that, but I think that's, um, certainly important in the poorest countries.Yeah. [00:33:57] Dwarkesh Patel: Um, I, I think Brian showed an open voters that if you look at , the, um, IQ of adoptees from poor countries, um, who go, uh, Sweden is the only country that collects data, but if you get adopted by a parent in, um, uh, Sweden, uh, the, the half the gap between the averages of two countries, half gap, yeah, yeah, yeah, yeah.Goes away. So, I mean, is one of the ways we can increase global IQ just by moving kids to, uh, countries with good health outcomes that, uh, will nourish their [00:34:27] Garett Jones: intelligence. Well, that's a classic short run versus long run effect, right? So, uh, libertarians and open borders advocates tend to be focused on the short run, static effects.So, um, and so you're right, moving kids from poor countries to richer countries is probably gonna raise their test scores quite a lot. And, uh, then the question is, in over the longer run, are those, uh, lower skilled folks, the folks with lower test scores, uh, going to degrade the institutional quality? of the places they move to, right?So if you close half the gap between the poor country and the rich country, half the gap is still there. Right? And if I'm right, , that IQ has big externalities then, , moving people from a, uh, lower scoring country to a richer scoring country and closing half the IQ gap still means on net you're creating a negative externality in the country the kids are moving to.[00:35:17] Dwarkesh Patel: Um, yeah, yeah. Uh, we can come back to that, but yeah. Yeah. So [00:35:23] Garett Jones: it, it's, it's basically, you just look at the question, is this lowering the mean test scores in your country? And if it's lowering the mean test scores in the long run, it's on average gonna lower institutional quality productivity savings rates, those.Um, it's hard to avoid that. It's hard to avoid that outcome. So, uh, I don't [00:35:38] Dwarkesh Patel: remember the exact figures, but didn't Brian address this in the book, um, in the Open Borders book as well, that you can, even if there's a, the, a national iq, uh, lowers on average, if you're just, uh, if you're still raising the global iq, that, that it's still nets out positive, or am I [00:35:54] Garett Jones: remembering that wrong?Well, that, notice what he's, he, what he does is he attributes, uh, he says there's some productivity that's just in the land, that's just geographic factors. Yeah. So basically being close for, and so that, so basically moving people away from the equator boost productivity substantially. And again, that's, uh, a static result.Um, the reason I, uh, I mentioned that ignores all the I seven stuff that I'm talking about where anything that lowers. Um, level of innovation in the world's most innovative countries has negative costs for the entire planet in the long run, but that's something you'd only see over the course of 20, 30, 50 years.And libertarians and open border advocates are very rarely interested in that kind of [00:36:33] Dwarkesh Patel: timeframe. Is there any evidence about, uh, the impact of migration on innovation specifically? So not on the average institutional quality or on, you know, uh, the, the corruption or whatever, but like, just directly the amount of innovation that happens or maybe the Noble Prizes won or things [00:36:48] Garett Jones: like that?Um, no. I mean, I would presume, I think a lot of us would presume that, uh, the European invasion of North America ended up having, uh, positive effects for global innovation. It's not an invasion that I'm in favor of, but if you wanna talk crudely about Yeah, yeah. Whether migrations had an effect on innovation, uh, you'd probably have to include that as any kind of analysis.[00:37:07] Dwarkesh Patel: Yep. Yep. , do you think that the people who are currently Americans, but , their ancestry, traces back to countries with low s a t scores? I i, is it possible that US GDP per capita would be higher, without that contribution?Or how do you think about that? [00:37:21] Garett Jones: I mean, that it follows from thinking through the fact that we are all externalities positive or negative, right? I don't know what in, in any particular, any one particular country could turn out to be some exciting exception to the rules, some interesting anomaly. Um, but on average, we should presume that the average skill level of voters, the average, uh, traits that we're bringing from, uh, the nations, that the nations of our, of our ancestors are as having an effect on our current productivity for gut ori.So just following through the reasoning, I'd have to say on average, that's most likely. Uh, but it, there could always be exceptions to the rule. [00:37:56] Dwarkesh Patel: I guess we see large disparities in income between different ethnic groups across the world, not just in the United States. Yeah. Doesn't that suggest that some of the gains can be privatized from whatever the cultural or other traits there are? Cuz if these, if over decades and centuries these sorts of, uh, these sorts of gaps continue, [00:38:18] Garett Jones: I don't see why that would follow.Right. Um, [00:38:21] Dwarkesh Patel: uh, if everything is being, if all the externalities are just being averaged out over time, what did you expect that these GA gaps would [00:38:29] Garett Jones: narrow? Well, I mean, I'm being a little rhetorical when I'm saying everything's literally an externality, right. I don't literally believe that's true. Um, for instance, people with higher education levels do actually earn more than people with lower education levels.So that's literally not an externality. Right. So some of these other cultural traits that people are bringing with them from their, um, ancestors, nations of origin, um, could be one or one likely one source of these income differences. I mean, if you think about differences in frugality, uh, differences in personal responsibility, which show up in the surveys, uh, that are persistent across generations, those are likely to have an effect on long run productivity for you, yourself and your family.So, mm-hmm. , let alone the hive mind stuff, where you find that there's a positive relationship between test scores and, and product. [00:39:13] Clone a million John von Neumann?[00:39:13] Garett Jones: There was a [00:39:14] Dwarkesh Patel: blogger who took a look at your 2004 paper about the, um, impact of National IQ on, um, on G uh, G D P. Um, and they calculated, so they were just speculating. Let's say you cloned a million John Mond Nomans, and as assume that John Mond Noman had an IQ of 180, then you could, uh, let me just pull up the exact numbers.You could, um, you could raise the average IQ of the United States by 0.21 points, um, and if it's true that one IQ point contributes 6% to, uh, G increasing G, then this proposal would increase U US GDP by, uh, 1.2, uh, six two 6%. Do you buy these kinds of extrapolations or 1.26%? Yeah. Yeah, because you're only cloning a million, [00:39:58] Garett Jones: Jon.Oh, yeah. Yeah. Okay. So this is about 1 million Jon. Yeah. Yeah, that sounds. I mean, that's the kind of thing where I wouldn't expect it to happen overnight. Right. I tend to think of that, uh, the IQ externalities as being two, three generations. I, I lump it in with what economists call organizational capital.That sounds about right. Yeah. Yeah, yeah. I mean, I, I can't remember where I saw this. I think I, I stumbled across it myself at some point too, so. [00:40:19] Dwarkesh Patel: Yeah. Yeah. Uh, by the way, his name is Avaro Dam Bernard, if you wanna [00:40:22] Garett Jones: find it. Oh, okay. Yes, yes. Okay. Yeah, it's, I mean, in, in, it's in that ballpark, right? It's just this idea that, and, and more importantly, um, a million John Bon Nomans would be a gift to the entire planet, right?Yep. Yep, yep. So, yeah, if you had a, if you had a choice of which country to have the John Vno, the million John Von Nomans, uh, it's probably gonna be one of the I seven maybe there's, maybe there's a, maybe Switzerland would be a good alternative. [00:40:46] Dwarkesh Patel: What is the optimal allocation of intelligence across the country?Because one answer, and I guess this is the default answer in our society, is you just send them where they can get paid the most, because that's a good enough proxy for how much they're contributing. Yeah. And so you have these high glomeration of talent and intelligence in places like Silicon Valley or New York.Um, and, you know, because their contributions there can scale to the rest of the world. This is actually where they're producing the most value. Another is, you know, you actually, you should disperse them throughout the country so that they're helping out communities. They're, you know, teachers in their local community.Um, I think there was, uh, A result. There was an interesting anecdotal evidence that during the Great Depression, the crime in New York went down a ton, and that was because the cops in New York were able to hire the, you know, they had like a hundred applications for every cop they hired. And so they were able to hire the best and the brightest, and there were just a whole bunch of new police tactics and every that were pioneered at the time anyways.So, is the market allocation of intelligence correct? Or do you think there should be more distribution of intelligence across the country? How do you think about that? [00:41:50] Garett Jones: Yeah, I mean, the mar the, the market signals aren't terrible. Uh, but, uh, this is my, my Interpol Roamer kicks in and says, uh, innovation is all about externalities.And there's market failures everywhere when it comes to, in the fields of innovation. Mm-hmm. . And so, you know, I, I personally, I mean, I, I like the idea of finding ways to allocate them to, to stem style, stem style technical fields, and. , we do a fair amount of that, and maybe we do the, maybe the US does a pretty good job of that.I don't have any huge complaints at that, at the, at the crudes 50,000 foot level, um, for the, you know, the fact that people know that there's, uh, status games they can play within academia that are perhaps more satisfying or at least as satisfying as the sort of corporate hierarchy stuff. So, yeah. Yeah. I I You don't want 'em all just, I wouldn't encourage them to solely follow market signals.Right. I'd, I'd encourage them to be more HandsOn and, uh, play a variety of status games because the academic, um, and intellectual status game is worth a lot, both personally and than it leads to positive spillovers for. [00:42:58] Dwarkesh Patel: But how about the geographic distribution? Do you think that it's fine that there's people leave, uh, smart people leave Kentucky and go to San Francisco or, yeah, [00:43:08] Garett Jones: I'm a big glomeration guy.Yeah. I'm, I'm, yeah, I'm a big glomeration guy. Yeah. I mean, the internet makes it easier, but then like, still being close to people's, being in the room's important. Um, there, there's, there's something, uh, both HandsOn and Gerard in here about, like, we need to find role models to imitate, and that's probably important for productivity.[00:43:30] Dwarkesh Patel: Um, are there increasing or decreasing returns to National iq?No, [00:43:38] Garett Jones: I think, um, you know, my findings were that it was all basically log linear. And so log linear looks crudely, like increasing returns. . So yeah, it looks exponential, right? So yeah, there's increasing returns to National iq. Yeah. Are are you? But, but this is, this is a commonplace finding in a sense because so many, uh, like human, all the human capital relationships I'm familiar with end up having something like a log linear form, which is exponential.So why is that? Um, yeah, there's something multiplicative that that's how, what I have, that's all I have to say is like it's something. Somehow this all taps into Adam Smith's pin factory, and we have multiplicative not additive effects when we are increasing brain power.Um, I have, I suspect it does have something to do with, uh, a, a better organization of the division of labor between people, which ends up happening something close to e to, uh, exponential effects on productivity. [00:44:39] Genetic Selection for IQ[00:44:39] Garett Jones: A are, uh, are you a fan of genetic selection for intelligence, uh, as a means of increasing national iq or do you think that's too much playing at the margins if it's voluntary?I mean, people should be able to do what they want and, um, after a couple day decades of experimentation, I think people would end up finding a path to, uh, government subsidies or tax credits or something like that. I think people voluntarily deciding what kind of kids they want to have. is a, a, a good thing.And so by genetic selection, I assume you're meaning at the most elementary level people testing their embryos the way they do now, right? Yeah. So I mean, we, we already do a lot of genetic selection for intelligence. Um, anybody, you know, who's, uh, in their mid thirties or beyond who's had amniocentesis, they've been doing a form of genetic selection for intelligence.So it's a widespread practice already in our culture. Um, and, uh, welcoming that in a voluntary way is probably going to have good effects for our future. What [00:45:40] Dwarkesh Patel: do you make of the fact that G B T three, or I think it was Chad g p t, had, uh, measured IQ of 85? Yeah, [00:45:47] Garett Jones: I've seen a few different measures of this, right?You might have seen multiple measures. Um, yeah, I think it's, I think it's a sign that basically, and, and when you see people using non IQ tests to sort of assess the outputs of G P T on, um, long essays, it does does seem to fit into that sort of, not quite a hundred, but not, not off by a lot. Yeah. I mean, I think it's, I think it's a sign that a lot of, uh, uh, mundane, even fairly complex, moderately complex human interactions can be simulated by a large, uh, language learning model.Mm-hmm. . And I think that's, that's, uh, gonna be rough news for a lot of, uh, people whose life was in the realm of words and dispensing simple advice and solving simple problems. That's pretty bad news for their careers. I'm, I'm disappointed hearing that, so [00:46:36] Dwarkesh Patel: Yeah. Yeah. [00:46:37] Garett Jones: Um, at least for the transition. I dunno what the, I dunno what's gonna happen after the transition, but [00:46:41] Dwarkesh Patel: Yeah.I'm hoping that's not true of programmers or economists. I like you. I mean, [00:46:46] Garett Jones: it might be right. I mean, it's, if that's the way it is, I mean, I, the, I mean, the car put a lot of, uh, people who took care of horses right out of outta work too, so. [00:46:55] Dwarkesh Patel: Yep. Um, even, okay, so let's talk about democracy that I thought this was also one of your really interesting books.No, thanks. Yeah. [00:47:02] Democracy, Fed, FDA, & Presidential Power[00:47:02] Dwarkesh Patel: even controlling for how much democratic oversight there is of institutions in the government. There seems to be a wide discrepancy of how well they work. Like the Fed seems to work reasonably well. I, I, I don't know enough about macroeconomics to know how the object level decisions they make, but mm-hmm.you know, it seems to be a non-corrupt, like, uh, technocratic organization. Um, enough, but yeah. Yeah. Uh, if you look at something like the fda, it's also somewhat insulated from democratic processes. It seems to not work as well. Mm-hmm. , what determines controlling food democracy? What controls, what, what determines how well an institution in the government works?[00:47:38] Garett Jones: Well, I, I think, um, in the case, the Fed, it really does matter that they, uh, the people who run it have guaranteed long terms and they print their own money to spend mm-hmm. . So that means that they're basically, Congress has to really. Make an effort to change anything of the Fed. So they really have the kind of independence that matters.Right. You know, they have a room of their own. And, uh, the FDA has to come to Congress for money more or less every year. And the fda, uh, heads do not have any kind of security of appointment. Their appoint, they serve at the pleasure of the president. Mm-hmm. . So I do think that they don't have real independence.Uh, I do think that they're basically, um, they're living in this slack, this area of slack to use this sort of mcno gas PolySci jargon. They're living in this realm of slack between the fact that the president doesn't wanna me, uh, muddle with them, uh, metal with them, excuse me. And the fact that Congress doesn't really wanna medal with them.But on the other hand, , I really think that that the f d A and the C d C are doing what Congress more or less wanted them to do. They reflect, they reflect the muddled disarray that Congress was in over the period of say, COVID. Hmm. Uh, that I think that's a first order importance. I mean, I do think the fact, it's the fact that, uh, f d A and c d C don't ha, uh, seem to have that culture of, um, raw technocracy the way the Fed does that, I think that has to be important on its own.But I think behind that, some of that is just like F D A C D C creatures of Congress much more than the Fed is. Should the [00:49:17] Dwarkesh Patel: power of the president be increased? [00:49:20] Garett Jones: Uh, no. No. Like the power of independent committees should be increased. Like more Congress should be like the Fed. If, uh, my plan for a Fed re for an FDA or CDC reorganization would be.Making them more like the Fed, where they have appointed experts who have long terms and they have enough of a long term that they can basically feel like they can blow off Congress and build their own culture. [00:49:42] EU is a force for good?[00:49:42] Dwarkesh Patel: Mm-hmm. , , so the European Union is an interesting example here because they also have these appointed technocrats, but they seem more interested in creating anno annoying popups on your websites than with dealing with econo, the, you know, the end of economic growth on the continent.Is this a story where more democracy would've helped, or how do you think about the European Union in this context? [00:50:04] Garett Jones: No. And the eu, like, uh, the European, European voters just aren't that excited about democracy. I, excuse me, aren't that excited about markets overall. The EU is gonna reflect that, right? Um, what little evidence we have suggests that, uh, countries that are getting ready to join the eu, they improve their economic freedom scores, their sort of laissez fairness.Hmm. Uh, on the path to getting ready for. , uh, join an eu. So, and then they may increase it a little bit afterwards once they join. But basically it's like, it's like, uh, when you're deciding to join the eu, it's like you decided you have your rocky training montage and get more laissez-faire. And so EU on net is a mess at polls in the direction of markets compared to where, uh, Europe would be otherwise.I mean, just look at the nations that are in the EU now, right? A lot of them are, um, east of Germany, right? And so those are countries that don't have this great, you know, uh, history of being market friendly. And a lot of parties aren't that market friendly, and yet the EU sort of nags them into their version, like as much markets as they can handle.So [00:51:05] Dwarkesh Patel: what do you think explains the fact that the Europe, uh, Europe as a whole and the voters in there are less market friendly than Americans? I mean, if you look at the sort of deep roots analysis of Europe, you would think that they should be the most. Uh, most in favor of, I don't know if the deep roots, uh, actually maybe they apply that, but Yeah, [00:51:23] Garett Jones: compared to the planet as a whole, they're pretty good.Right? So, um, I, I'm, I never get that excited about like, the small little distinctions between the US and Europe, like these 30% GDP differences, which are very exciting to pundits and bloggers and whatever. I'm like 30% doesn't matter very much. That's not really my bailiwick. What I'm really interested in is the 3000% between the poorest countries and the richest countries.So, like I can speculate about Europe, I, I don't really have a great answer. I mean, I, I think there's something to the, the naive view that, um, the Europeans with the most, uh, what my dad would call gumption are those who left and came to America. Some openness, some adventurousness. Uh, and maybe that's part of what trans, uh, made we, so basically there's a lot of selection working, uh, on the migration side to, uh, make America more open to laissez fair than Europe would be.[00:52:14] Dwarkesh Patel: Does that overall make you more optimistic about migration to the US from anywhere? Like, you know, the same story [00:52:20] Garett Jones: of Yeah. Center is perab us like America, America gets people who are really great, right? I went with you there. Yeah. [00:52:26] Dwarkesh Patel: Does, um, elite technocratic control work best in only in high IQ countries?Because otherwise you don't have these high IQ elites who can make good policies for you, but you also don't get the democratic protections against famine and war and things like that. [00:52:43] Garett Jones: Oh, I mean, I don't know. I think, I think the case for, for, uh, handing things over to elites is pretty strong in anything that's moderately democratic, right?Um, I don't have to be. Anything that's substantially more democratic than the official measure of Singapore, for instance. I mean, that's why my book 10% Less Democracy, really is targeted at the rich, rich democracies. Once we get too far below, uh, the rich democracies, I figure once you put elites in charge, they really are just gonna be old-fashioned Gordon to rent seekers and steer everything Jordan themselves and not give a darn about the masses at all.So that's, you know, uh, elite control in a democracy, a a lot of elite control in any kind of democracy, I think is gonna have good effect. If it's re you're really looking at something that is, uh, that meets a Mar Sen's definition of a democracy competitive market. Competitive party's free press. [00:53:38] Dwarkesh Patel: Mm-hmm.does Singapore meet that criteria? [00:53:41] Garett Jones: No. Because their parties aren't really allowed to compete. I mean, that's pretty obvious. Yeah. The, the pa the People's Action Party really controls, uh, party competition there. [00:53:52] Dwarkesh Patel: So, but it, I guess Singapore is one of the great examples of technocratic, um, technocratic control, and [00:53:59] Garett Jones: they're just an exception of the rule.Most countries that try to pull off that lower level democracy wind up much [00:54:03] Dwarkesh Patel: worse. So what is your, uh, what is your opinion of Neoreactionaries? I guess they're not in favor of 10% less democracy. They're more in favor of a hundred percent less democracy. [00:54:12] Garett Jones: But yeah, I think they're like kind of too much LARPing, too much romanticizing about the roheim, I guess.I don't know. What is rheum? Yeah. The, these guys in the Lord of the Rings, you know? . , romanticizing Monarch is a mistake. Um, it's worth noting that, uh, as my colleague Gordon Tok pointed out, as along as many others, uh, in Equilibrium Kings are almost always king and council, right.and so it's worth thinking through why King and Council is the equilibrium. Something more like a corporate board and less like, um, either the libertarian ideal of the entrepreneur who, who owns the firm, or the monarch who has the long-term interest in being a stationary bandit in real life. There's this sort of muddled thing in between that works out as the equilibrium, even in the successful so-called monarchies.So it's worth thinking through why it is that the successful so-called monarchies aren't really monarchies, right? They're really oligarchies. [00:55:12] Dwarkesh Patel: Yep. Yep. Um, if you look at the median voter in terms of their preferences on academic policies, it seems like they're probably more, um, in favor of government involvement than the actual policies of the United States, for example.Yeah. What explains this? Shouldn't the media voter theorem that we should be much less libertarian as a country than? Yeah, that's a great [00:55:35] Garett Jones: point from, um, Brian Kaplan's excellent. Bill Smith of the rational voter. Right? Yeah. I think part of it, I mean, I think his stories are right, which is that, uh, politicians facing reelection have this tradeoff between giving voters what the voters say they want and giving the voters the economic growth that will help the politicians get reelected, right?Mm-hmm. Um, so it's, uh, it's a version of saying like, you know, I don't want you to pull off the bandaid, but I want my wound to all get better. So and so the politician has to, it's the politician's job to handle the contradictory demands of the voters, and by delegating authority to them. To the vote or to the elected politicians.You get some of the benefits of elitism, um, even in a so-called democracy. [00:56:19] Is Ethnic Conflict a Short Run Problem?[00:56:19] Dwarkesh Patel: Um, over the long run, should we expect any of the tensions or all of the tensions of ethnic diversity to fade away? Like nobody today worries about the different Parisian tribes in France butting heads at the workplace, right? So over time yourself [00:56:33] Garett Jones: and, and you're right like that, uh, the, uh, anti-German ethnic sentiment in the US totally gone, right?So, [00:56:39] Dwarkesh Patel: right. Yeah. But over time, so then this, this is another one of those, the short run effects that you, uh, emphasize that your focus lesson on, right? [00:56:46] Garett Jones: Yeah, that's a good point. Um, it's poss you don't know which one. I mean, the problem is that ethnic conflict has been a hardy perennial. It's not the only conflict that people can ever have.I don't, I don't know to what extent, uh, they'll, these things will fade away. As I emphasize in the culture transplant, the ethnic diversity channel is actually the least important of any of the channels I discuss. Um, and so I'm open to this thought that what you're saying will actually happen and maybe we'll just find something else to get mad at each other about like, um, uh, social media tribes or, uh, religious, religious groups.Um, I mean, it hasn't happened yet in all the documented human history. We have. People seem to find some ethnic, uh, balance for conflict. It is worth pointing out that the one one, um, study that I report, uh, I think it's a waard coha piece. It's fine. That window, the real, uh, the source of ethnic conflict happens when private values are correlated with, um, ethnic groups, right?So if, uh, cultural values are basically uncorrelated with ethnicity, then basically there's nothing to fight over. And that's really what's happened with a lot of old ethnic battles in the us mm-hmm. . And, um, so you're right. Some of these things will fade with time. The problem is that human beings are, one of our great evils is that we are always looking for a focal point.and we can, people will use visible appearance as a horrifying focal point around which to, uh, peg their conflicts. It's an easy one because our brains are looking for visual patterns, and I don't like that, but it's something that will probably keep happening. [00:58:22] Dwarkesh Patel: One of the interesting points you made in the chapter was that the, uh, benefits of diversity are greatest when search costs are lower and the cost of vetting are lower. What do, how, how do we make sure that that is true of non-lead professions?So if you're looking for a plumber or if you're looking for a carpenter, how, how do we make sure that you can vet them easily? [00:58:42] Garett Jones: I mean, I have to say that this has to be a case where like Yelp and Google and all these online ratings have given us tools for checking these things out. We know we have to be skeptical, of course, but uh, for people who know that they're good at something, the cost of entry into a new field has to be much lower than it was a few decades ago because, you know, 10 20 good Google reviews, um, and you can actually enter.So yeah, lower, basically not, not banning, not banning disclosure of data. I'd say that's the most important thing we can do. Mm-hmm. ? Um, I think, um, you sometimes hear that medical doctors, I haven't checked up on this a long time, but apparently medical doctors often, uh, make a very risky to give bad reviews.Sometimes you get lost, you get a lawsuit or something. Mm-hmm. . So making that a lot harder is worth it. You know, we know that some negative reviews are gonna be malicious and inaccurate, but the benefits of information flow seem really high. . [00:59:38] Bond Holder Democracy[00:59:38] Dwarkesh Patel: Yeah. I thought one of the really interesting chapters in the, the 10% list democracy was the chapter on, you know, bond holder democracy.Yeah. And I'm curious, so I mean, corporations are ex uh, obviously an example to use here where they do have bond holders who hold them accountable. Mm-hmm. , but the average lifespan of a corporation is 10 years, I believe. So do you think it'll be, it would, um, it would be even shorter if bond holders had a lesser sale on corporations or you, you know, what is the, what does the transients of the corporation tell us about their controls?[01:00:11] Garett Jones: Oh, that's a good point. Um, well, we, we can suspect that the average person who's investing in a corporation makes money, right? Because, uh, otherwise people wouldn't be doing it right. On average people. Right? It must work. Um, so. This is what, Ryan, you actually have me stumped here. So can you rephrase the question again?I'm trying to think through what the, what the question is there. Sure. Yeah. I, [01:00:32] Dwarkesh Patel: I, if bond holders do extend the longevity and the long term thinking of the organizations on, on who they hold bonds, why aren't corporations who give out bonds, why don't they tend to live longer than I think the average of 10 years?Would it be even shorter without bond holders or, [01:00:50] Garett Jones: oh, I, I'd say I, a, I'd say it'd probably be shorter without bond holders or any kind of financial monitor, but second, most corporations just shouldn't live that long. Right? Most corporations, their ideas that you try out and then it, like you find out it, it doesn't work, or it should be bought up by somebody else, or the IP should be sold off.And so having a lot of companies, uh, fade out is actually on net a good sign. I think this is really part of the John Halter wearing line of research that the sort of modern version of creative destruction research. Which finds that, you know, uh, low productivity firms exiting, you know, uh, just as naive laissez-faire predicts means that those workers and that capital can get reallocated over to a more productive firm.So the alternative is, uh, you know, stereotypically Japanese zombie firms, right? They're kept limping along by banks that are perhaps under political pressure to lend. And so a lot of, a lot of cap human and physical capital gets tied up in low productivity projects. Yeah. So, yeah. Um, uh, a brutal bond market is, uh, a good way to send a market signal to move capital from low productivity to high productivity projects.Um, why [01:01:56] Dwarkesh Patel: are yields on 30 year, uh, fixed, um, treasuries? Why are they so low? Because theoretically, investors should know that we have a lot of liabilities in the form of, you know, social security to baby boomers. Yeah. And that we've radically inflated the money supply very recently and may do so again. Uh, do you think the investors are being irrational with the, uh, low yields or what, what's going on?[01:02:18] Garett Jones: No, no. I'm a fan of the view that the bond holders are gonna win in the long run and that, uh, inflation, any kind of super, like, super, super high inflation is not gonna be the path, uh, of the future. And what's gonna happen is that, at least think about the us Um, the way the bond holders are gonna win is that there's gonna be a mixture of tax hikes and slower spending growth, especially hurting the poor.And that's how the US is gonna close its fiscal gap. I don't know particularly what paths other countries are gonna go through, but the US has this basically, um, this one tool, this one superpower sitting, um, sitting in the room that it hasn't used yet, and it's, oh, it's a vat, right? So the US could dramatically increase its tax revenue through either an overt or disguised value added tax, and that would raise a ton of money just like it raises in Europe.And that would, that's the easy way to close the US fiscal gap. We probably won't even have to get to that, just making Medicaid worse. Slowly over the long run , um, maybe making Medicare worse over the long run. Right? Um, that, that by itself would close a lot of this fiscal gap. So basically, I think they'll balance the long run budget on the kind of the backs of the poor and the middle class.That's probably the most likely outcome. So are you expecting, hence no, hence no hyperinflation. So are, are, [01:03:38] Dwarkesh Patel: but so you're expecting the welfare state to shrink either in quality or quantity over [01:03:42] Garett Jones: time in relative terms, uh, compared to the trend. I mean, if America's still getting say 1%, 1.5% richer each year, um, then that by itself, uh, means that, you know, you know, that adds a certain level of quality to healthcare over time.And so basically if it, if it ends up staying the equivalent of say, 0%, um, that by itself, if you could get healthcare spending in, in real terms to be 0% over time, um, that, that would end up closing the gap when you compound it out long enough. Right? So, yep. Yep. Yeah. [01:04:16] Dwarkesh Patel: Um, is that kind of thinking. So, you know, when Liz Truss in the UK tried to implement, uh, tax reform, there was a bond holder revolt, and, you know, she was ousted.Yeah. Uh, bondholders. There's one [01:04:28] Garett Jones: right there. Yeah. [01:04:29] Dwarkesh Patel: So do you, do you are, do we already live in the Bondholder utopia or ? [01:04:32] Garett Jones: Oh, yeah. I, I mean I think that that was a nice, uh, reminder that basically contrary to sort of the MMT view, um, and the sort of Pop m the pop kasian view, the debt is no barrier at all.Um, I think that showed the bond holders, uh, are actually paying attention to long-term signals of fiscal policy credibility, and they'll take action. Yeah. Mm-hmm. . Yeah. [01:04:57] Mormonism[01:04:57] Garett Jones: What, what are the deep roots of Mormonism? Why do they have such high trust and uh, uh, such tight-knit communities? I mean, I think part of it is that they are, they reflect a lot of this sort of, Uh, what for then was Western Pioneer culture, upstate New York, Pennsylvania, then Ohio.Um, I think those communities tended to be, uh, high trust communities necessarily because of the difficult, uh, environment that they were living in. Um, and there was a lot of selection. There was a lot of selection over the first few decades of Mormon history where those who were willing to sort of trust the group stayed in and those who weren't willing to trust the group, um, ended up leaving.And not just trust, but trustworthiness. I think a lot of people probably got weed out because they weren't contributing to the common good. So I think that basically by the time the Mormons got established in Utah, uh, they, they had already selected for a strong culture of, um, a kind of a, kind of in-group prosociality.And I think that's, um, helped them, that helped them weather the storms of the 19th century. So the whole 19th century people were only joining. during this, this is during the era of polygamy in Utah. Right. Um, if they thought that they were willing to put up with this, right? And, you know, you're, you're signing up for some kind of deep sociality with a mixture of uncon, a lot of unconventional stuff.And that foundation, uh, really helped. And the fact that Mormons since then have stayed as a religion that requires a medium, high level of, of commitment, also weeds out people who just aren't willing to make that kind of commitment. Mm-hmm. . So, I mean, I was, I was raised Mormon. Um, you know, I wasn't ultimately willing to make the commitment.And maybe the part of the reason is cause I'm too much of a free rider, so the Mormons who are left are probably better than me. [01:06:46] Dwarkesh Patel: Um, the Mormon church has, I think, more than a hundred billion in assets. Uhhuh , what is it planning on doing with all this money? That's a, that's a tremendous, that's a tremendous sum.[01:06:56] Garett Jones: I don't know. I mean, maybe they're planning to hand it to the savior when the second coming happens. Right. It's, uh, there's, there's gotta be a great argument for this option. Value of just having the wealth there. Right. It does, it must give them a kind of independence from the world when various storms come along.Cultural, political storms. Mm-hmm. . Um, you know, I don't actually know what their plans are for what they would do with all the money, but I do know that like normal economics tells us that just peop being frugal is good for the economy overall. So Steven LANs has a great, uh, essay and praise of Scrooge.It's especially appropriate for this time of year. And, um, being frugal means you're building up the capital stock and you're giving, uh, a sort of invisible gift to future generations. So Mormon frugality is basically helping build up the US capital stock and indirectly the world's capital stock, helping make us all more productive, which I think is something that fits in with Mormon values.[01:07:52] Dwarkesh Patel: Yeah. Yeah. I think people have pointed out that people should be spending more money given the fact that they have so much left over by the time they die. Usually, [01:08:00] Garett Jones: yeah. As an individual level, if you care about your own wellbeing. That's true. Right? [01:08:03] Dwarkesh Patel: Yeah. . Yeah. But, okay, so if you, it is interesting, like leaving a large inheritance is, uh, is so, is socially valuable.Yeah. [01:08:10] Garett Jones: Leaving a large inheritance means there's, uh, you're leaving, you're, you're, you're producing a lot, but you're not consuming very much. That means you're building up the capital [01:08:16] Dwarkesh Patel: stock. So. Yep. Yep. Um, and there's also, I think, um, a large amount of, uh, multi-level marketing schemes that proliferate in Utah.Yeah. Is that one of the downsides of high social trust? [01:08:31] Garett Jones: Yeah. The, cuz people predate upon it, right? It strikes me as a total rent seeker sort of thing. I mean, I have to say the knives are really good. So I'm, uh, , uh, everybody I know who's ever had Cutco knives ends up using 'em for decades. So that's one of the popular multi-level marketing schemes that actually gets to, to men.A lot of them are targeted at, at women through cosmetics, as you might know, so mm-hmm. , but at least the men's one works out. ,[01:08:52] Garett Jones's Immigration System[01:08:52] Dwarkesh Patel: if you had to implement an immigration system from scratch, would you actually, uh, would you actually consider these s a t scores and these other deep roots scores as part of somebody's admittance? Or would you just consider the individual level, um, your personal skill as an education and things like [01:09:07] Garett Jones: that?Now, I'd wanna launch a 10 year, maybe 20 year research project of figuring how to turn the deep roots scores into something useful. So I, like, I, uh, I think right now with the deep roots literature, we're about where Milton Friedman's Monetarism was in the late sixties. You know, Friedman said, Hey, I figured out where inflation comes from, and we'd be a lot better off if we just grew the money supply 3% a year and.N ultimately, nobody thought he was right about the 3% a year thing, but they did think that he had a, still had a lot of good advice. So Friedman ended up having a lot of good ideas, but they weren't policy ready. And I think that's about where we are with the deep roots literature right now, which is, um, I mean at most one would use it as a, like a small plus factor and a point system, but I don't even know which points I'd use.But something along those lines is worth thinking about. I would never use a cutoff, I would never use any, uh, quotas or hard cutoffs. Uh, if you think about point space systems, 10, 20 years of. Further research, and maybe you'd find a way to put the deep roots into a point space system.[01:10:12] Interviewing SBF[01:10:12] Dwarkesh Patel: Yeah. Yeah. [01:10:12] Garett Jones: Um, how was the SPF F thing? What do you mean? So did he just say like, I'll fly you out? [01:10:20] Dwarkesh Patel: So I was there for the, um, EA Bahamas. Oh, okay. Okay. Yeah. Yeah. And then while I was there, I, I, um, I, I talked to somebody who knew him and I'm like, Hey, I would love to interview him.Yeah, yeah. Some things I would ask him. Yeah, yeah. Yeah. It was, it was one of the ones where, actually I feel like I would really want to redo that one because I was aware of some things back then that were kind. Yeah, that would be worth asking about in retrospect. And of course, it's all in retrospect, but yeah.Yeah. Um, I should have focused harder at that rather than asking these sort of philosophical questions about effective altruism. [01:10:51] Garett Jones: Yeah. Do you think, uh, I mean, is it as simple as like he was co-mingling funds? Uh, he, he lent a bunch of FTX money over to a hedge fund and then they lost it. [01:11:02] Dwarkesh Patel: Um, I, I, I, yeah. I'm guessing first [01:11:04] Garett Jones: approximation.Yeah. Yeah, yeah. So it's like, it's old fashioned financial fraud, partly driven by, uh, not having a really good board over em or a good oversight. . [01:11:14] Dwarkesh Patel: You know what, I'm really curious to ask of you this, because you know what you're talking hive mind about the fact that higher IQ people on average, um, are more cooperative in prisoners dilemma type situations.Yeah, yeah, yeah. And I just interviewed Bethany McClean on the podcast. It hasn't been released yet, but you know, she wrote the smartest guys in the room, which about Enron. Right. And then, so there is this thing where maybe they are less likely to commit fraud on average, but when they do, they're so much, they're really good at it.Right, exactly. Right. So, yeah, I, I don't know, maybe, maybe one of the downsides of having a high IQ society is that what people do commit in fraud, they're super successful at it. Uh, you know, I don't know. Yeah. I mean, the, [01:11:52] Garett Jones: the evils of of super smart people are obviously a huge risk to all of humanity.Right? Right. I don't have to worry about humanity being wiped out by a bunch of people with sticks and stones. Right? Yeah. I have to worry about humanity being wiped out by nuclear weapons, which could only be invented by [01:12:07] Dwarkesh Patel: smart people, right? Yeah, yeah, yeah, yeah. Um, are, are smart people more cooperative in a sort of very calculating sense that, you know, this game is gonna go on, so I wanna make sure I preserve my relationships.Or even in the last turn of a iterated prisoner's dilemma game, even in the last turn, they would cooperate. [01:12:28] Garett Jones: Oh, in the last turn they walk away. Yeah. Yeah, yeah, yeah. I think it's, I think. There is no correlation between, uh, intelligence and say agreeableness, right? Mm-hmm. normal, psychological agreeableness.And I think that's a broader principle or conscientiousness for that matter, right? Psychological conscientiousness. so Machiavelli intelligence, I think is what's driving the link between IQ and cooperation. So in repeated games, that Machiavelli intelligence, which a lot of intelligence researchers will talk about, turns into cosan intelligence where people find a way to grow the pie, but it's a very cynical, self-interested form of growing the pie.Yeah. Yeah. And so I, I don't think that has any, um, I don't think it's driven by inherent prosociality. I think it's it's endogenous prosociality, not exogenous prosociality. And that's a reason to worry about it. [01:13:19] Dwarkesh Patel: What is, what happens to these high IQ people when if society goes into a sort of zero sum mode where there's not that much economic growth?And so the only way you can increase your share of the pie is just by cutting out bigger and bigger slices for yourself. [01:13:32] Garett Jones: Yeah. Like then you gotta watch out, right? Yeah. It's like the Middle Ages right there, right? Yep. Exactly. Yeah, yeah, yeah. [01:13:38] Dwarkesh Patel: Um, yeah. Interesting. Um, awesome. Uh, Garrett, thanks so much for coming on the podcast. This was interesting. Thanks for having me. This has been fantastic. Yeah. Yep. Excellent. Thanks for reading my books. Appreciate it. Yeah, of course. ​ ​ Get full access to The Lunar Society at www.dwarkeshpatel.com/subscribe
1/24/20231 hour, 14 minutes, 1 second
Episode Artwork

Lars Doucet - Progress, Poverty, Georgism, & Why Rent is Too Damn High

One of my best episodes ever. Lars Doucet is the author of Land is a Big Deal, a book about Georgism which has been praised by Vitalik Buterin, Scott Alexander, and Noah Smith. Sam Altman is the lead investor in his new startup, ValueBase.Talking with Lars completely changed how I think about who creates value in the world and who leeches off it.We go deep into the weeds on Georgism:* Why do even the wealthiest places in the world have poverty and homelessness, and why do rents increase as fast as wages?* Why are land-owners able to extract the profits that rightly belong to labor and capital?* How would taxing the value of land alleviate speculation, NIMBYism, and income and sales taxes?Watch on YouTube. Listen on Apple Podcasts, Spotify, or any other podcast platform. Read the full transcript here.Follow Lars on Twitter. Follow me on Twitter.Timestamps(00:00:00) - Intro(00:01:11) - Georgism(00:03:16) - Metaverse Housing Crises(00:07:10) - Tax Leisure?(00:13:53) - Speculation & Frontiers(00:24:33) - Social Value of Search (00:33:13) - Will Georgism Destroy The Economy?(00:38:51) - The Economics of San Francisco(00:43:31) - Transfer from Landowners to Google?(00:46:47) - Asian Tigers and Land Reform(00:51:19) - Libertarian Georgism(00:55:42) - Crypto(00:57:16) - Transitioning to Georgism(01:02:56) - Lars's Startup & Land Assessment (01:15:12) - Big Tech(01:20:50) - Space(01:23:05) - Copyright(01:25:02) - Politics of Georgism(01:33:10) - Someone Is Always Collecting RentsTranscriptThis transcript was partially autogenerated and thus may contain errors.Lars Doucet - 00:00:00: Over the last century, we've had this huge conflict. All the oxygen's been sucked up by capitalism and socialism duking it out. We have this assumption that you either have to be pro worker or pro business that you can't be both. I have noticed a lot of crypto people get into Georgism, so not the least of which is Vitalik Buterin and you endorse my book. If you earn $100,000 in San Francisco as a family of four, you are below the poverty line. Let's start with just taxing the things nobody has made and that people are gatekeeping access to. Let's tax essentially monopolies and rent seeking. The income tax needs to do this full anal probe on everyone in the country and then audits the poor at a higher rate than the rich. And it's just this horrible burden we have. Dwarkesh Patel - 00:00:39: Okay, today I have the pleasure of speaking with Lars Doucet, who developed the highly acclaimed Defender's Quest game and part two is coming out next year, but now he's working on a new startup. But the reason we're talking is that he wrote a review of Henry George's progress and poverty that won Scott Alexander's Book Review Contest and now it has been turned into an expanded into this book Land is a Big Deal. So Lars, welcome to the podcast. New Speaker: Great to be here, Dwarkesh . Okay, so let's just get into it. What is Georgism? Lars Doucet - 00:01:12: Okay, so the book is based off of the philosophy of a 19th century American economist by the name of Henry George from once we get George's and basically George's thesis is kind of the title of my book that land is a big deal. Georgism is often reduced to its main policy prescription that we should have a land value tax, which is a tax on the unimproved value of land, but not a tax on any buildings or infrastructure on top of the land, anything humans add. But the basic insight of it is that it's kind of reflected in the aphorisms you hear from real estate agents when they say things like the three laws of real estate or location location location and buy land, it's the one thing they're not making any more of. It's basically this insight that land has this hidden role in the economy that is really underrated. But if you look at history through the right lens, control over land is the oldest struggle of human history. It goes beyond human history. Animals have been fighting over land forever. That's what they're fighting over in Ukraine and Russia right now, right? And basically the fundamental insight of Georgism is that over the last century, we've had this huge conflict. All the oxygen's been sucked up by capitalism and socialism duking it out. We have this assumption that you either have to be pro worker or pro business that you can't be both. And Georgism is genuinely pro pro worker and pro business. But what it's against is is land speculation. And if we can find a way to share the earth, then we can solve the paradox that is the title of George's book, progress and poverty, why does poverty advance even when progress advances? Why do we have all this industrialized technology and new methods and it in George's time it was industrial technology in our time its computers and everything else? We have all this good stuff. We can make more than we've ever made before. There's enough wealth for everybody. And yet we still have inequality. Where does it come from? And George answers that question in his book. And I expand on it in mine. Dwarkesh Patel - 00:03:15: Yep. OK, so yeah, I'm excited to get into the theory of all of it in a second. But first I'm curious how much of your interest in the subject has been inspired with the fact that as a game developer, you're constantly dealing with decentralized rent seekers, like Steve or iOS app store. Is that part of the inspiration behind your interest in Georgism or is that separate? Lars Doucet - 00:03:38: It's interesting. I wouldn't say that's what clued me into it in the first place. But I have become very interested in all forms of rent seeking. In this general category of things we call land-like assets that come to first mover advantages in these large platform economies. I've started to think a lot about it basically. But the essence of land speculation is you have this entire class of people who are able to basically gatekeep access to a scarce resource that everybody needs, which is land, that you can't opt out of needing. And because of that, everyone basically has to pay them rent. And those people don't necessarily do anything. They just got there first and tell everyone else, it's like, well, if you want to participate in the world, you need to pay me. And so we're actually the actual connection with game development, actually clued me into Georgism. And I'd heard about Georgism before. I'd read about it. I thought it was interesting. But then I started noticing this weird phenomenon in online multiplayer games going back 30 years repeatedly of virtual housing crises, which is the most bizarre concept in the world to me, like basically a housingcrisis in the Metaverse and predecessors to the Metaverse. And as early as the Alt Online (?)online when I was 19, this is this online game that you could play. And you could build houses in the game and put them down somewhere. And so what I found was that houses were actually fairly cheap. You could work long enough in a game to be afford to buy blueprints for a house, which will be put it somewhere. But there was no land to put it on. And at the time, I thought, oh, well, I guess the server failed up. I didn't really think much about it. I was like, this stinks. I didn't join the game early enough. I'm screwed out of housing. And then I kind of forgot about it. And then 20 years later, I checked back in. And that housing crisis is still ongoing in that game. That game is still running a good 25 years later. And that housing crisis remains unsolved. And you have this entire black market for housing. And then I noticed that that trend was repeated in other online games, like Final Fantasy 14. And then recently in 2022, with all this huge wave of crypto games, like Axi Infinity, and that's Decentral Land and the Sandbox. And then Yuga Labs' Board-Ape Yacht Club, the other side, had all these big land sales. And at the time, I was working as an analyst for a video game consulting firm called Novik. And I told my employers, it's like, we are going to see all the same problems happen. We are going to see virtual land speculation. They're going to hit virtual. They're going to reproduce the conditions of housing crisis in the real world. And it's going to be a disaster. And I called it, and it turns out I was right. And we've now seen that whole cycle kind of work itself out. And it just kind of blew my mind that we could reproduce the problems of the real world so articulately in the virtual world without anyone trying to do it. It just happened. And that is kind of the actual connection between my background in game design and kind of getting George Pilled as the internet kids call it these days. Dwarkesh Patel - 00:06:43: There was a hilarious clip. Some comedian was on Joe Rogan's podcast. I think it was like Tim Dillon. And they're talking about, I think, Decentraland, where if you want to be Snoop Dogg's neighbor in the Metaverse, it costs like a couple million dollars or something. And Joe Rogan was like, so you think you can afford to live there. And then Tim Dillon's like, no, but I'm going to start another Metaverse and I'm going to work hard. But OK, so let's go into Georgism himself. So Tyler Cohen had a blog post a long time ago who was comparing taxing land to taxing unimproved labor or unimproved capital. And it's an interesting concept, right? Should I, so I have a CS degree, right? Should I be taxed at the same level as an entry level software engineer instead of a podcast or because I'm not using my time as efficiently as possible. And so leisure in another way is the labor equivalent of having an unimproved parking lot in the middle of San Francisco or capital. If I'm just keeping my capital out of the economy and therefore making it not useful, maybe I should have that capital taxed at the rate of the capital gains on T-Bill. And this way, you're not punishing people for having profitable investments, which you're kind of doing with a capital gains, right? What would you think of that comparison? Lars Doucet - 00:08:07: Yeah, so really, before you can even answer that question, you've got to go back to ground moral principles you're operating on. Like, is your moral operating principle like we just want to increase efficiency? So we're going to tax everyone in a way to basically account for the wasted opportunity cost, which brings up a lot of other questions of like, well, who decides what that is. But I think the Georgist argument is a little different. We're not necessarily like it is efficient, the tax we propose, but it actually stems kind of from a more, from a different place, a more kind of fundamental aspect of justice, you know? And from our perspective, if you work and you produce value, your work produced that value, right? And if you save money and accumulate capital in order to put that capital to work to receive a return, you've also provided something valuable to society, you know? You saved money so a factory could exist, right? You saved money so that a shipping company could get off off the ground. You know, those are valuable, contributed things, but nobody made the earth. The earth pre-exists all of us. And so someone who provides land actually does the opposite of providing land. They unprovide land, and then they charge you for opening the gate. And so the argument for charging people on the unimproved value of land is that we want to tax unproductive rent seeking. We want to tax non-produced assets because we think we want to encourage people to produce assets. We want to encourage people to produce labor, to produce capital. We want more of those things. And there's that aphorism that if you want less of something, you should tax it. So I mean, maybe there is a case for some kind of galaxy brain take of, you know, taxing unrealized opportunity costs or whatever, but I'm less interested in that. And my moral principles are more about, let's start with just taxing the things nobody has made and that people are gatekeeping access to. Let's tax essentially monopolies and rent seeking. And then if we still need to raise more taxes, we can talk about that later. But let's start with, let's start with just taxing the worst things in society and then stop taxing things we actually want more of because we have this mentality right now where everything's a trade off and we have to accept the downsides of income taxes, of sales taxes, of capital taxes because we just need the revenue and it has to come from somewhere. And my argument is it's like, it can come from a much better somewhere. So let's start with that.Dwarkesh Patel - 00:10:39: Yeah, yeah. So I guess if it was the case that we've implemented a land value tax and we're still having a revenue shortfall and we need another kind of tax and we're going to have to keep income taxes or capital gains taxes. Would you in that situation prefer a sort of tax where you're basically taxed on the opportunity costs of your time rather than the actual income you generated or the returns you would interest your generate in your capital? Lars Doucet - 00:11:04: No, I think probably not. I think you would probably want to go with some other just like simpler tax \for the sake of it there's too many degrees of freedom in there. And it's like, we can talk about why I will defend the Georgist case for property tax assessments, you know, for land value tax. But I think it gets different when you start like judging what is the most valuable use of your time because that's a much more subjective question. Like you're like, okay, are you providing more value to society as being a podcaster or being a CS computer science person or creating a startup? It's like that may not be evident for some time. You know what I mean? Like I can't think of an example, but like think of people who were never successful during their lifetimes. I think the guy who invented what was it? FM radio, right? He threw himself out a window because he never got it really adopted during his lifetime but it went on to change everything, you know? So if we were taxing him during his lifetime based off of what he was doing of being a failure, like if Van Gogh was taxed of his like wasting his life as an artist as he thought he was, which ultimately led to his suicide, you know, a lot of these things are not necessarily realized at the time. And so I think that's, and you know, it would need a much bigger kind of bureaucracy to like figure that all out. So I think you should go with a more modest. I mean, I think after land value tax, you should do things like severance tax on natural resources and other taxes on other monopolies and rents. And so I think the next move after land value tax is not immediately to capital and income taxes and sales taxes, but to other taxes on other rents seeking and other land like assets that aren't literally physically land. And then only after you've done all of those, if you still, you know, absolutely then, then move on to, you know, the bad taxes. What is this, severance tax? Severance tax is a tax on the extraction of natural resources. Is what Norway does with their oil industry that has been massively successful and a key reason that Norway has avoided the resource curse? Yeah. Basically, it's, Georgist purist will say it's essentially a land value tax but of a different kind. A land value tax like you can't normally like extracts just like land like on this, in this house you're living on, you're like, you're not using up this land, but non-renewable resources you can use up. Yeah. You know, and so a severance tax is basically, Nestle should be charged a severance tax for the water they're using, for instance, you know, because all they're doing is enclosing a pre-existing natural resource that used to belong to the people that they've essentially enclosed and now they're just putting it in bottles and selling it to people. You know, they should be able to realize the value of the value add they give to that water, but not to just taking that resource away. Dwarkesh Patel - 00:13:53: No that makes sense. Okay, so let's go deep into the actual theory and logic of Georgism. Okay. One thing I was confused by is why property owners who have land in places that are really desirable are not already incentivized to make the most productive use of that land. So even without a property, sorry, a land tax, if you have some property in San Francisco incentives, let's go, why are you not incentivized to construct it to the fullest extent possible by the law, to, you know, collect rents anyways, you know what I mean? Like why are you keeping it that as a parking lot? Lars Doucet - 00:14:28: Right, right, right. So there's a lot of reasons. And one of them has to do with, there's an image in the book that this guy put together for me. I'll show it to you later. But what it does is that it shows the rate of return. What a land speculator is actually optimizing for is their rate of return, right? And so if land appreciates by 10% a year, you know, you're actually incentivized to invest in vacant land or a tear down property because the building of a tear down property is like worth negative value. So the land's cheaper because there's garbage on it, you know? Then you are to necessarily invest in a property and you're basically your marginal dollar is better spent on more land than it is on building up. Dwarkesh Patel - 00:15:16: But eventually shouldn't this be priced into the price of land so that the returns are no longer 10% or they're just like basically what you could get for any other asset. And at that point, then the rate of return is similar for building thingson top of your existing land than buying a new land because like the new land is like the, you know, that return has been priced into other land. Lars Doucet - 00:15:38: Well, I mean, arguably, empirically, we just don't see that, you know, and we see rising land prices as long as productivity and population increases. Those productivity and population gains get soaked into the price of the land. It's because of this phenomenon called Ricardo's Law of Rent and it's been pretty empirically demonstrated that basically, and it has to do with the negotiation power. But like why some people do of course, build and invest, you know, there's a lot of local laws that restrict people's ability to build. But another reason is just like, it also has to do with the existing part of it. It part of the effect is partially the existing property tax regime actively incentivizes empty lots because you have a higher tax burden if you build, right? So what actually happens is a phenomenon that's similar to oil wells, right? You have, it's not just because of property taxes, those do encourage you to keep it empty. But there's this phenomenon called land banking and waiting for the land to ripen, right? Sure, I could build it now, but I might have a lot of land parcels I've got. And I don't need to build it now because I think the prices might go up later and it would be better to build on it later than it is now. And it's not costing me anything to keep it vacant now. If I build now, I'm gonna have to pay a little bit more property taxes. And I know in three years that the price is gonna be even better. So maybe I'll wait to incur those construction costs then and right now I'm gonna focus more on building over here. And like I've got a lot of things to do, so I'm just gonna squat on it here. It's the same way I have, I'm squatting like, you know, I bought to my shame, like about 30 domain names, you know, most of them bought before I kind of got ontoGeorgism. And it's like, yeah, I'll pay 15 bucks a year to just hold it, why not? You know what I mean? I might use that someday. Right. And it's like, I should probably release all the ones I have no intent of using because I was looking for a domain for my startup the other day and every single two word.com is taken. Right, right. And it has been for like 10 years, you know, and it's a similar phenomenon. It's just like some of it is economic, rational following of incentives. And some of it is just it's like, well, this is a good asset. I'm just gonna hold on to it because why not? And no one is, and I don't have any pressure to build right now. And this happens on the upswing and on the downswing of cities. So while the population's growing and while the population's declining, people will just buy a lot of land and hold it out of use. Cause it's also just a great place to park money because it's an asset that you know if the population ever starts growing, it's gonna keep its value better than almost any other hard asset you have. Dwarkesh Patel - 00:18:16: Yep yep. I guess another like broader criticism of this way of thinking is, listen, this is all, and sorry for using these like podcast lingo of scarcity mindset, but this is all like scarcity mindset of, you know, land is limited. Well, why don't we just focus on the possibility of expanding the amount of usable land? I mean, there's like not really a shortage of land in you. Maybe there's a shortage of land in urban areas. But you know, why don't we like expand into the seas? And why don't we expand into the air and space? Why are we thinking in this sort of scarce mindset? Lars Doucet - 00:18:48: Right. Okay, so I love this question because actually our current status quo mindset is the scarcity mindset. And Georgism is the abundance mindset, right? And we can have that abundance if we learn to share the land. Because right now, you know, why don't we expand? And the answer is we've tried that. We've done it twice. And it's the story of America's frontier, right? And so like right now there's plenty of empty land in Nevada, but nobody wants it. And you have to ask why, right? You also have to ask the question of how did we have virtual housing crises in the Metaverse where they could infinitely expand all they want? Like how is that even possible, you know? And the answer has to do with what we call the urban agglomeration effect. What's really valuable is human relationships, proximity to other human beings, those dense networks of human beings. And so the idea is not necessarily that like, in a certain sense, the issue is that land is not an indistinguishable, fungible commodity. Location really matters. Or America has a finite amount of land, but it might as well be an infinite plane. We're not going to fill up every square inch of America for probably thousands of years if we ever do, right? But what is scarce is specific locations. They're non-fungible, you know? And to a certain extent, it's like, okay, if you don't want to live in New York, you can live in San Francisco or any other like big city. But what makes New York New York is non-fungible What makes San Francisco San Francisco is non-fungible That particular cluster of VCs in San Francisco until or unless that city completely explodes and that moves somewhere else to Austin or whatever, you know, at which point, Austin will be non-fungible. I mean, Austin is non-fungible right now. And so the point is that the way Georgism unlocks the abundance of it, let me talk about the frontier. We have done frontier expansion. That is why immigrants came over from Europe, you know, and then eventually the rest of the world, to America to, you know, settle the frontier. And the losers of that equation were, of course, the Indians who were already here and got kicked out. But that was theoriginal idea of America. And I like to say that America's tragedy, America's problem is that America is a country that has the mindset of being a frontier state, but is in fact a state which has lost its frontier. And that is why you have these conversations with people like boomers who are like, why can't the next generation just pull itself up by its bootstraps? Because America has had at least, I would say two major periods of frontier expansion. The first was the actual frontier, the West, the Oregon Trail, the covered wagons, you know, the displacement of the Indians. And so that was a massive time, that was the time in which Henry George was writing, was right when that frontier was closing, right? When all that land, that free land was being taken, and the advantages of that land was now being fully priced in. That is what it means for a frontier to close, is that now the good productive land, the value of it is fully priced in. But when the frontier is open, you can just go out there and take it, and you can get productive land and realize the gains of that. And the second frontier expansion was after Henry George's death, was the invention of the automobile, the ability to have a job in the city, but not have to live in the city. The fact that you could quickly travel in, like I commuted in to visit you here, right? That is because of the automobile frontier opening that has allowed me to live in some other city, but be able to do productive work like this podcast by driving in. But the problem is, sprawl can only take you so far, before that frontier as well closes, and by closes I don't mean suburban expansion stops. What I mean is that now, suburban homes, you fully price in the value of the benefits are able to accrue by having that proximity to a city, but still being able to live over here, through of course, for Ricardo's Law for it. Dwarkesh Patel - 00:22:37: Yeah, but I feel like this is still compatible with the story of, we should just focus on increased in technology and abundance, rather than trying to estimate how much rent is available now, given current status quo technologies. I mean, the car is a great example of this, but imagine if there were like flying cars, right? Like there's a, where's my flying car? There's like a whole analysis in that book about, you know, if you could, if people are still commuting like 20 minutes a day, you know, a lot more land is actually in the same travel distance as was before, and now all this land would be worth as much, even in terms of relationships that you could accommodate, right? So why not just build like flying cars instead of focusing on land rent? Lars Doucet - 00:23:21: Well, because these things have a cost, right? The cost of frontier expansion was murdering all the Indians and the cost of automobile expansion was climate change. You know, there has to be a price for that. And then eventually, the problem is you eventually, when you get to the end of that frontier expansion, you wind up with the same problem we had in the first place. Eventually, the problem is the first generation will make out like gangbusters if we ever invent flying cars, even better like Star Trek matter teleporters. You know, that'll really do it. Then you can really live in Nevada and have a job in New York. Yeah. There are some people who claim that Zoom is this, but it's not, you know, we've seen the empirical effects of that and it's like, it's the weakest like semi-frontier we've had and it's already closed. Because, because of Zoom, houses like this over in Austin have gone up in value because there is demand for them and there's demand for people to telecommute. And so anyone who, so the increased demand for living out in the suburbs is now basically priced in because of the Zoom economy. And so the thing is the first people who did that, who got there really quick, the first people to log in to the ultimate online server were able to claim that pace of the frontier and capture that value. But the next generation has to pay more in rent and more in home prices to get that. Dwarkesh Patel - 00:24:34: Actually, that raises another interesting criticism ofGeorgism, this is actually a paper from Zachary Gouchanar and Brian Kaplan, where it was titled the Cerseioretic critique of Georgism, and the point they made was one of these, like one way of thinking about the improvement to land is actually identifying that this land is valuable. Maybe because you realize it has like an oil well in it and maybe you realize that it's like the perfect proximity to these like Chinese restaurants and this mall and whatever. And then just finding which land is valuable is actually something that takes capital and also takes, you know, like you deciding to upend your life and go somewhere, you know, like all kinds of effort. And that is not factored into the way you would conventionally think of the improvements to land that would not be taxed, right? So in some sense, you getting that land is like a subsidy for you identifying that the land is valuable and can be used to productive ends. Lars Doucet - 00:25:30:Right, yeah, I know. So I've read that paper. So first of all, the first author of that Zachary Gouchanar yeah, I'm not been able to pin him down on what exactly meant on this, but he's made some public statements where he's revised his opinion since writing that paper and that he's much more friendly to the arguments ofGeorgism now than when he first wrote that paper. So I'd like to pin him down and see exactly what he meant by that because it was just a passing comment. But as regards Kaplan's critique, Kaplan's critique only applies to a 100% LVT where you fully capture all of the land value tax. And the most extreme Georgists I know are only advocating for like an 85% land value tax. That would still leave. And Kaplan doesn't account at all for the negative effects of speculation. He's making a speculation is good actually argument. And even if we grant his argument, he still needs to grapple with all the absolutely empirically observed problems of land speculation. And if we want to make some kind of compromise between maybe speculation could have this good discovery effect, there's two really good answers to that. First, just don't do 100% LVT, which we probably can't practically do anyway because of natural limitations just empirically, you know, in the signal. It's like you don't want to do 115% land value tax. That drives people off the land. So we want to make sure that we like have a high land value tax but make sure not to go over. And so that would leave a sliver of land rent that would still presumably incentivize this sort of thing. There's no argument for why 100% of the land rent is necessary to incentivize the good things that Kaplan was talking about. The second argument is when he talks about oil, well, we have the empirical evidence from the Norwegian massively successful petroleum model that shows in the case of natural resources how you should deal with this. And what Norway does is that they have a massive, massively huge severance tax on oil extraction. And according to Kaplan's argument, this should massively destroy the incentive for companies to go out there and discover the oil. And empirically, it doesn't. Now what Norway does is that they figured out, okay, so the oil companies, their argument is that we need the oil rents, right? We need these oil rents where we will not be incentivized for the massive capital cost of offshore oil drilling. Well, Norway's like, well, if you just need to cover the cost of offshore oil drilling, we'll subsidize that. We'll just pay you. We'll just pay you to go discover the oil. But when you find the oil, that oil belongs to the Norwegian people. Now you may keep some of the rents but most of it goes to the Norwegian people. But hey, all your R&D is free. All your discovery is free. If the problem is discovery, we just subsidize discovery. And then the oil companies are like, okay, that sounds like a great deal. We don't have to, because without that, what the oil companies do is that they're like, okay, we're taking all these risks. So I'm gonna sit on all these oil wells like people sitting on domain names because I might use them later and the price might go up later. But now because there's a huge severance tax, you're forced to drill now and you're actually, you're actual costs of discovery and R&D and all those capital costs are just taken care of. Dwarkesh Patel - 00:28:26: But isn't there a flip side to that where I mean, one of the economic benefits of speculation, obviously there's drawbacks. But one of the benefits is that it gets rid of the volatility and prices where our speculator will buy when it's cheap and sell when the price is high. And in doing so, they're kind of making the asset less volatile over time. And if you're basically going to tell people who have oil on their land, like we're gonna keep taxing you. If you don't take it out, you're gonna keep getting taxed. You're encouraging this massive glut of a finite resource to be produced immediately, which is bad. If you think we might need that reserve in the ground 20 years from now or 30 years from now, you know, went oil reserves were running low. Lars Doucet - 00:29:10: Not necessarily, you know? And so the problem is that speculation in the sense you're talking about if like encouraging people to do arbitrage is good for capital because we can make more capital. But we can't make more land and we can't make more non-renewable natural resources. And the issue in peer, and I just think the evidence just doesn't support that empirically because if anything, land speculation has causes land values to just constantly increase, not to find some natural part, especially with how easy it is to finance two thirds of bank loans just chase real estate up. And that's just like, if you just look at the history of the prices of, you know, of residential real estate in America, it's like, it's not this cyclical graph where it like keeps going back down. It keeps going back down, but it keeps going up and up and up, just on a straight line along with productivity. And it underlines and undergirds, major issues, everything that's driving our housing crisis, which then undergirds so much of inequality and pollution and climate change issues. And so with regards to speculations, like even if I just bite that bull and it's like, okay, speculation is good actually, I don't think anyone's made the case that speculators need to capture a hundred percent of the rents to be properly incentivized to do anything good that comes out of speculation. I think at some small reasonable percentage, you know, five to 10 percent of the rents, maybe 15 if I'm feeling generous, but I don't think anyone's empirically made the case that it should be a hundred percent, which is more or less a status quo. Dwarkesh Patel - 00:30:31:I mean, with regards to that pattern of the fact that the values tend to keep going up implies that there's nothing cyclical that the speculators are dampening. Lars Doucet - 00:30:41: Well, there are cycles to be sure, but it's not like, it's something that resets to zero. Dwarkesh Patel - 00:30:45: Yeah, but that's also true of like the stock market, right? Over time that goes up, but speculators are still have like an economic role to play in a stock market of making sure prices are, Lars Doucet - 00:30:55: I mean, the difference is that people are now paying an ever increasing portion of their incomes to the land sector. And that didn't used to be the case. And if it keeps going, it's going to be, I mean, you have people are now paying 50% of their income just for rent. And that's not sustainable in the long term. You're going to have the cycle you have there is revolution. You know, you, you know, Dwarkesh Patel - 00:31:16: (laughing) Lars Doucet - 00:31:17: I’m serious. like what happens is like you look through history, you either have land reform or you have revolution. And you know, it's, it's either like either you have a never ending cycle of, of, of transfers of income from the unlanded to the landed. And eventually the, the unlanded will not put up with that. You know, there was a real chance in the 19th century, at the end of the 19th century of America going full on socialist or communist and the only thing that saved us. What, and George's argument was like, it's either Georgism or communism. And if you want to save capitalism and not go toTotalitarian, we need Georgismand then what George failed to anticipate was, you, of course, the automobile. And the automobile kicked the can down another generation, another couple generations, right? And it came at the cost of sprawl. And that made everyone feel like we had solved the issue. But basically we just, and the cost of sprawl are enormous in terms of pollution and poor land use. Just look at Houston right now, right? But now we've come at the end of that frontier and now we're at the same question. And it's like, you see this research in interest in leftism in America and that's not a coincidence, right? Because the rent is too damn high and poor people and poor people and young people feel really, really shoved out of the promise and social contract that was given to their parents and they're jealous of it and they're wondering where it went. Dwarkesh Patel - 00:32:36: Yeah, yeah. Actually, you just mentioned that a lot of bank loans are given basically so you can like get a mortgage and get a house that's like towards land. There was an interesting question on Twitter that I thought was actually pretty interesting about this. I can't find the name of the person who asked it. So sorry, I can't give you credit, but they basically asked if that's the case and if most bank loans are going towards helping you buy land that's like artificially more expensive, but now you implement a land value tax and all these property values crash. Oh yeah. Well, when we see just, and then all these mortgages are obviously they can't pay them back. Lars Doucet - 00:33:13: Right, right, right. Are we gonna destroy the banking sector? Dwarkesh Patel - 00:33:15: Exactly. We'll have like a great, great depression.Lars Doucet - 00:33:17: Well, I mean, if you, okay, so like this is, this is kind of like, I mean, I'm not, I'm not trying to compare landlords to slave owners or something, but it's like, it's like the South had an entire economy based off of slavery. This thing that like we now agree was bad, right? And it's like we shouldn't have kept slavery because the, the South, the, like it really disrupted the Southern Economy when we got rid of slavery, but it was still the right thing to do. And so I mean, there is no magic button I could push as much as I might like to do so that will give us 100% land value tax everywhere in America tomorrow. So I think the actual path towards a Georgist Future is gonna have to be incremental. There'll be enough time to unwind all those investments and get to a more sane banking sector. So I mean, like if we were to go overnight, yeah, I think there would be some shocks in the banking sector and I can't predict what those would be, but I also don't think that's a risk that's actually gonna happen. Because like we just, we just cannot make a radical change like that on all levels overnight. Dwarkesh Patel - 00:34:13: Yeah yeah, yeah. Okay, so let's get back to some of these theoretical questions. One I had was, I guess I don't fully understand the theoretical reason for thinking that you can collect arbitrarily large rents. Why doesn't the same economic principle of competition, I get that there's not infinite landowners, but there are multiple landowners in any region, right? So if for the same reason that profit is competed away in any other enterprise, you know, if one landowner is extracting like $50 a profit a month, and another landowner is extracting, you know, like whatever, right? Like a similar amount of $50. One of them, and they're both competing for the same tenant. One of them will decrease their rent so that the tenant will come to them and the other one will do the same and the bidding process continues until all the profits are, you know,bidded away. Lars Doucet - 00:35:04: Right, so this is Ricardo's law front, right? And there's a section on in the book with a bunch of illustrations you can show. And so the issue is that we can't make more land, right? And so you might be like, well, there's plenty of land in Nevada, but the point is there's only so much land in Manhattan. Dwarkesh Patel - 00:35:19: But the people who have land inManhattan, why aren't they competing against themselves or each other? Lars Doucet - 00:35:23: Right, well, what they do is because the nature of the scarcity of there's only so many locations in Manhattan and there's so many people who want to live there, right? And so all the people who want to live there have to outbid each other. And so basically, so like, let me give a simple agricultural example model. And then I will explain how the agricultural model translates to a residential model. Basically, when you are paying to live in an urban area, or even a suburban area like here in Austin, what you're actually paying for is the right to have proximity to realize the productive capacity of that location. IE, I want to live in Austin because I can have access to a good job, you know what I mean? Or whatever is cool about Austin, a good school, those amenities. And the problem is you have to pay for those and you have to outbid other people who are willing to pay for those. And Ricardo's Rolf Rent says that the value of the amenities and the productivity of an area, as it goes up, that gets soaked into the land prices. And the mechanism by that is that it's like, okay, say I want to buy a watermelon, right? And there's only one watermelon left out bid that guy. But the watermelon growers can be like, oh, a lot of people want watermelon. So next season, there's going to be more watermelons because he's going to produce more watermelons. But because there's only so many locations in Austin, you know, within the natural limits of our transportation network, basically it forces the competition on the side of the people who are, essentially the tenants, right? It forces us into one side of competition with each other. And that, and so there's an example of like, a simple agricultural example is like, okay, say there is a common field that anyone can work on and you can make 100 units of wealth if you work on it, right? So, and there's another field that you can also learn 100 units of wealth in, but it's owned by a landowner. Why would you, why would you go and work on the landowners when you're going to have to pay them rent? You wouldn't pay them any rent at all. You would work on the field that's free, but if the landowner buys that field and now your best opportunity is a field that's only worth a free field that will produce 10 units of wealth, now he can charge you 90 units of wealth becauseyou have no opportunity to go anywhere else. And so basically as more land gets bought and subject to private ownership in an area, landowners over time get to increase the rent, not to a maximum level, there are limits to it. And the limits is what's called the margin of production, which is basically you can charge up to, and this is where the competition comes in, the best basic like free alternative, you know, and that's usually, you can realize that geographically, like out on the margins of Austin, there's marginal land that basically is available for quite cheap, you know, and it might be quite far away, and it used to be not so quite far away 20, 30 years ago, you know, and so as that margin slowly gets privatized, landowners can charge up to that margin. The other limit is subsistence, that can't charge more than you're actually able to pay, but the basic example is that, so this is why this is how frontier expansion works. When the entire continent's free, the first settler comes in, strikes a pick in the ground, keeps all of their wealth, but as more and more of it gets consolidated, then landowners are able to charge proportionately more until they're charging essentially up to subsistence. Dwarkesh Patel - 00:38:51: Yeah, does that explain property values in San Francisco? I mean, they are obviously very high, but I don't feel like they're that high where this offer engineers were working at Google or living as subsistence levels, neither are they at the margin of reduction where it's like, this is what it would cost to live out in the middle of California, and then commute like three hours to work or something. Lars Doucet - 00:39:13: Right, well, so it has to do with two things. So first of all, it's over the long run, and so it's like, you've had a lot of productivity booms in San Francisco, right? And so it takes some time for that to be priced in, you know, and it can be over a while, but given a long enough time period it'll eventually get there. And then when we're talking about stuff, it's also based off of the average productivity. The average resident of San Francisco is maybe not as productive as a high, and like basically doesn't earn as high an income necessarily as a high income product worker. And so this means that if you are a higher than productive, higher than average productivity person, it's worth it to live in the expensive town because you're being paid more than the average productivity that's captured in rent, right? But if you're a low, if you're lower than average productivity, you flee high productive areas. You go to more marginal areas because those are the only places you can basically afford to make a living. Dwarkesh Patel - 00:40:06: Okay, that's very interesting. That's actually one of the questions I was really curious about. So I'm glad to hear an answer on that. Another one is, so the idea is, you know, land is soaking up the profits that capitalists and laborers are entitled to in the form of rent. But when I look at the wealthiest people in America, yeah, there's people who own a lot of land, but they bought that land after they became wealthy from doing things that were capital or labor, depending on how you define starting a company. Like sure, Bill Gates owns a lot of land in Montana or whatever, but like the reason he has all that wealth to begin with is because he started a company, you know, that's like basically labor or capital,however you define it? Right. So how do you explain the fact that all the wealthy people are, you know, capitalists or laborers? Lars Doucet - 00:40:47: Well, so the thing is, one of the big missed apprehensions people have is that, when they think of billionaires, they think of people like Bill Gates and Elon Musk and Jeff Bezos, those are actually the minority billionaires, most billionaires or hedge funds are people involved in hedge funds. You know, bankers and what are bankers, most what are two thirds of banks? It's real estate, you know? And so, but more to your point, like if I, if it is like point that directly into it, it's like, I don't necessarily have a problem with the billionaire existing. You know what I mean? If someone like genuinely like bring something new into the world and like, you know, I don't necessarily buy the narrative that like billionaires are solely responsible for everything that comes out of their company, you know, I think they like to present that image. But I don't necessarily have a problem with a billionaire existing. I have a problem with, you know, working class people not being able to feed their families, you know, and so like the greater issue is the fact that the rent is too high rather than that Jeff Bezos is obscenely rich. Dwarkesh Patel - 00:41:45:No, no, I guess my point was in that, like, I'm not complaining that your solution would not fix the fact that billionaires are this. I also like that there's billionaires. What I'm pointing out is it's weird that, if you're theory of, like, where all the sort of plus in our society is getting, you know, given away is that it's going to landowners. And yet the most wealthy people in our society are not landowners. Doesn't that kind of contradict your theory? Lars Doucet - 00:42:11: Well, a lot of the wealthy people in our society are landowners, right? And it's just like, it's not the, so the, so the thing is is that basically making wealth off land is a way to make wealth without being productive, right? And so my point is is that, so like you said in your interview with Glazer that it's like, okay, the Googleplex, like the value of that real estate is probably not, you know, compared that to like the market cap of Google. But now compare the value of all the real estate in San Francisco to the market caps to some of those companies in there, you know, look at the people who are charging rent to people who work for Google. That's where the money's actually going, is that, and, and, you know, investors talk about this is that it's like, I have to, like, if you earn $100,000 in San Francisco as a family of four, you are below the poverty line, right? You know, the money is going to basically upper middle class Americans and upper class Americans who own tons of residential land and are basically, and also the old and the wealthy, especially, are essentially this entire class of kind of hidden landed gentry that are extracting wealth from the most productive people in America and young people, especially. And, and it is creates really weird patterns, especially with like service workers who can't afford to live in the cities where their work is demanded. Dwarkesh Patel - 00:43:30: Yeah. Okay. So what do you think of this take? This might be economically efficient. In fact, I think it probably is economically efficient, but the effect of the land value tax would be to shift, to basically shift our sort of societal subsidy away from upper middle class people who own, happen to own land in urban areas and shift that to the super wealthy and also super productive people who will like control the half acre that Google owns and like mountain view. So it's kind of like a subsidy, not subsidy, but it's easing the burden on super productive companies like Google and so that they can make even cooler products in the future. But it is in some sense that's a little aggressive, you're going from upper middle class to like, you know, tech billionaire, right? But it's still be economically efficient to do that. Lars Doucet - 00:44:18: Well, no, I don't quite agree with that because it's like, although there are a lot of upper middle class Americans who own a lot of the land wealth, it's not the case that they own where the majority of the land wealth is. The majority of the land wealth in urban areas is actually in commercial real estate. Is the central business district, if you, and I work in mass appraisal, so I've seen this myself in the models we build is that if you look at the transactions in cities and then you plot where the land value is and like a graph, it looks like this. And this is the city center and that's not a residential district. So the residential districts are sucking up a lot of land value and the rent is toodamn high. But the central business district and this even holds even in the age of Zoom, it's taken a tumble, but it's starting from a very high level. That central residential, I'm not residential, but commercial real estate is super valuable. Like orders, like an order of magnitude more valuable than a lot of the other stuff. And a lot of it is very poorly used.In Houston especially, it's incredibly poorly used. We have all these central parking lots downtown. That is incredibly valuable real estate. And just a couple of speculators are just sitting on it, doing nothing with it. And that could be housing, that could be offices, that could be amenities, that could be a million sorts of things. And so when you're talking about a land value tax, those are the people who are going to get hit first. And those are people who are neither nice, nice, friendly upper middle class Americans, nor are they hardworking industrialists making cool stuff. They're people who are doing literally nothing. Now, if you do a full land value tax, yeah, it's going to shift the burden in society somewhat. But I feel that most analyses of property taxes and land value taxes that conclude that they are regressive, I think that's mostly done on the basis of our current assessments. And I feel like our assessments could be massively approved and that if we improve the assessments, we can show where most of our land values actually concentrated. And then we can make decisions about exactly, are we comfortable with these tax shifts? Dwarkesh Patel - 00:46:18: Yeah, yeah. Hey guys, I hope you're enjoying the conversation so far. If you are, I would really, really appreciate it if you could share the episode with other people who you think might like it. Put the episode in a group chat you have with your friends, post it on Twitter, send it to somebody who think might like it. All of those things helps that a ton. Anyways, back to the conversation. So a while back I read this book, how Asia works. You know,Lars Doucet - 00:46:45: I'm a fan. Dwarkesh Patel - 00:46:47: Yeah, and one of the things, I think Joseph Steadwell was going out there, what are the things he talks about is he's trying to explain why some Asian economies grew, gangbusters in the last 20th century. And one of the things he points to is that these economies implemented land reform were basically, I guess they were distributed land away from, I guess the existing aristocracy and gentry towards the people who are like working the land. And while I was reading the book at the time, I was kind of confused because, you know, we've like, there's something called like the Kostian. The Kostian, I forget the name of the argument. Basically, the idea is, regardless of who initially starts off with a resource, the incentive of that person will be to, for him to like give that resource, lend out that resource to be worked by that person who can make most productive use of it. And instead of what was pointing out that these like small, you know, like these peasant farmers basically, they will pay attention to detail of crop rotation and making the maximum use of this land to get like the maximum produce. Whereas if you're like a big landowner, you will just like try to do something mechanized. It's not nearly as effective. And in a poor country, what you have is a shitton of labor. So you want something that's like labor intensive. Anyways, backing up a bit, I was confused while I was reading the book because I was like, well, wouldn't the, wouldn't, what you would expect to happen in a market that basically the peasants get alone from the bank to work to, I guess, rent out that land. And then they are able to make that land work more productively than the original landowner. Therefore, they are able to like make a profit and everybody benefits basically. Why isn't there a co-scient solution to that? Lars Doucet - 00:48:24: Because any improvement that the peasants make to the land will be a signal to the landowner to increase the rent because of Ricardo's law of rent. Yep. And that's exactly what happened in Ireland when, and George talks about this in progress and poverty, is that a lot of people were like, why was there famine in Ireland? It's because the Irish are bad people. Why didn't they, they're lazy? Why didn't they improve? And it's like because if you improve the land, all that happens is you still are forced into one side of competition and the rent goes out. Dwarkesh Patel - 00:48:50: Yep. OK. That makes sense. Is the goal that the taxes you would collect with the land value tax? Are they meant to replace existing taxes or are they meant to give us more services like UBI? Because they probably can't do both, right? Like you either have to choose getting rid of existing taxes or getting more.. Lars Doucet - 00:49:08: Well, it depends how much UBI you want. You know what I mean? It's like you can, you know, it's a sliding skill. It's like how many taxes do you want to replace versus how much? Like, I mean, you can have a budget there. It's like if you can raise, you know, I show in the book the exact figures of how much I think land value tax could raise. And I forget the exact figures, but like you can pull up a graph and overlay it here of, you know, whether you're talking about the federal level or federal local and state, you know, there's $44 trillion of land value in America. And I believe we can raise about $4 trillion in land rents annually with 100% land value tax. And we would probably do less than that in practice. But even on the low end, I forget what figure I quote for the low end, like you could fully pay for any one of social security, Medicare plus Medicaid together, so the second one is healthcare or defense. Entirely with the lowest estimate of what I think land rents could raise. And then I think you can actually raise more than that because I think, and I give an argument in the book for why I think it's closer to like $4 trillion. And that could pay for all three and have room over for a little bit of extra. And so I mean, it's up to you, like, that's a policy decision of whether you want to spend it on spending, whether you want to spend it on offsetting taxes or whether you want to spend it on UBI. I think the best political solution, because like if I bite the bullet that there might be some regressivity issues left over, you want to do what's called a UBI or what, you know, in George's time was called a citizen's dividend, right? You know, this will smooth over any remaining regressivity issues. And then, but I very much am in favor of getting rid of some of these worst taxes, you know, not just because they have dead weight loss and land value tax doesn't, but also because there's this tantalizing theory called ATCORE- All taxes come out of rent, which suggests that if you reduce other taxes, it increases land values, which means that if it's true in the strongest sense, it means the single tax,right? Land value tax replaced all taxes would always work. And I'm not sure if I buy that, I want to see some empirical evidence, but I think at least some weak form of it holds, so that when you offset other worst taxes, not only do you get rid of the dead weight loss from those, but you also wind up raising at least a little bit more in land value tax revenue. Dwarkesh Patel - 00:51:20: Yes, yeah. I mean, as a libertarian, or I guess somebody who has like libertarian tendencies, my concern would basically be like, this obviously seems better than our current regime of taxing things that are good, basically capital income. But my concern is the way I'm guessing something like this would be implemented is it would be added on top of rather than repealing those taxes. And then, yeah, I guess like we would want to ensure. Lars Doucet - 00:51:44: I get this one a lot. Yeah, no. And so I have, you know, I've been a libertarian in my past, and I have a soft spot for libertarianism. I used to be a Ron Paul guy, I went back in the day for a hot minute. And so I think the thing to suede your concerns there is what is land value tax? It's property tax without a tax on buildings. Yep. So the natural path to actually getting land value tax comes from reforming existing property tax regimes by reducing an entire category of taxation, which is the tax on buildings. And so that's what I think is the most plausible way to get a land value tax, like in Texas here, if we were to start by just capture the same, like what I actually proposed for our first step is not 100% land value tax federally. I don't know, even know how you get to there. I think what you actually do is you start in places like Texas and like here, legalized split-rate property tax, thus, re-tax buildings and land at separate rates, set the rate on buildings to zero, collect the same dollar amount of taxes. Let's start there. There's proposals to do this in various cities around the nation right now. I think there's one in Virginia. There's a proposal to do in Detroit. I think there's some talk of it in Pennsylvania and some places. And I'd like to see those experiments run and observe what happens there. I think we should do it in Texas. And that would be something that I think would be very friendly to the libertarian mindset, because very clearly we're no new revenue, right? And we're exempting an entire category of taxation. Most people are gonna see savings on their tax bill and the people who own those parking lots downtown in Houston are gonna be paying most of the bill. Dwarkesh Patel - 00:53:14: Yeah, by the way, what do you make of, is there a good, Georgist's critique of government itself? In a sense that government is basically the original land squatter and it's basically charging the rest of us rents or staying on rent that. It's neither productively improving. As much as at least it's getting rents or must work. Like if you think about, even your landlord usually is not charging you 40%, which is what the income tax rate is in America, right? And it's like almost, you can view the land lord of America. Lars Doucet - 00:53:46: Well, I mean, it's like, I mean, if you wanna take the full, like if you're asking is Georgism compatible with full anarcho capitalist libertarianism, probably not 100%, I think we can have a little government as a treat. But I think it's not a coincidence that if you look throughout America's founding, I don't think it's a coincidence that originally, like people talk about it's like, oh, it used to be only white men who could vote. White land-owning men could vote. Like a government by the landowners for the landowners of the landowners, right? And that's very much kind of the traditional English system of government, just neo-feudalism, right? And so I think Georgism certainly has a critique of that, that it's like government is often instituted to protect the interests of landowners. But what's interesting is that if you look throughout history, I'm very much a fan of democracy, rule of the people. And it's like, I think we, you know, I kind of sympathize with Milton Friedman here, where he's like, you know, he might want to have less government than we have now, but he doesn't believe we can have no government. And then he goes on to endorse, you know, the land value taxes, the least worse tax, because income tax especially, I feel like is a gateway drug to the surveillance state, you know, one of the advantages of land value taxes you don't even care necessarily who owns the land. You're just like, hey, 4732 Apple Street, make sure the check shows up in the mail. I don't care how many shell companies in the Bahamas, you've like obscured your identity with, just put the check in the mail, Mr. Address, you know, whereas the income tax needs to do this full anal probe on everyone in the country, and then audits the poor at a higher rate than the rich, and it's just this horrible burden we have, and then it'll, it gives the government this kind of presumed right to know what you're doing about everything you're doing in this massive invasion of privacy.Dwarkesh Patel - 00:55:42: Yeah, no, that's fascinating. I speak to you, I have shell companies in the Bahamas, by the way. Yes. There's an interesting speculation about what would happen if crypto really managed to divorce and private, I guess, make private your log of transactions or whatever. And then, I guess the idea is the only legible thing left to the government is land, right? So it would like force the government to institute a land value tax, because like you can't tax income or capital gains anymore, that's all on like the blockchain and the right, right? It's cured in some way. And yeah, yeah, so that, I mean, it's like crypto the gateway drug to George's own, because it'll just move income and capital to the other realm. Lars Doucet - 00:56:20: Yeah, it's just so weird. I've gone on record as being a pretty big crypto skeptic. But I have noticed a lot of crypto people get into Georgism home. I mean, not the least of which is Vitalik Buterin and you endorse my book, who's a huge fan of Georgism home. It's like, I'll take fans from anywhere, even from people I've had sparring contests with. I'm generally pretty skeptical that crypto can fulfill all its promises. I am excited by those promises, and if they can prove me wrong, that would be great. And I think there's some logic to what you're saying is that if we literally couldn't track transactions, then I mean, I guess we don't have much the tracks accept land. I don't think that'll actually come to pass just based off of recent events. You know, and that's basically my position on it. But I have noticed a lot of crypto people, just they’re some of the easiest people to convince about George's home, which was completely surprising to me. But I've learned a lot by talking to them. It's very interesting and weird. Yeah, yeah. Dwarkesh Patel - 00:57:16: So there was some other interesting questions from Twitter. Ramon Dario Iglesias asks, how do you transition from a world today where many Americans have homes where it really starts sparring to have homes to a world where, I mean, obviously, it would be like a different regime. They might still have homes, but who knows? Like, their property will be just be like, think I thought I'm going to complete a different way. How do you transition to that? Like, what would that transition look like for most Americans? Lars Doucet - 00:57:39: So there's this issue called that I have to grapple with, which is called Gordon Tullich's transitional gains trap. Right? So if you think about taxing medallions in New York City, right? You know, it's like it's this artificially scarce asset that allows you to operate a taxi, right? And the first generation that got their taxing medallions basically got in cheap. And then afterwards, like made out like gangbusters. But the second generation had to buy those taxi medallions at the fully priced in value. And now when you come in and you're like, oh, okay, we're going to abolish taxi medallions. If like, say you were going to do that, you would screw over that entire second generation who bought in in good faith after the value of the asset had been fully priced in. Even if you admit that the system is now unfair, removing that unfairness screws over the people who played by the quote unquote unfair rules. You know, so how do you grapple with that? And I think it's something that Georgeists need to grapple with because we can't just imagine a future utopia without accounting for being fairer to people who played by the rules, including people like myself. Like I'm a homeowner, right? Am I intending to screw over myself and everyone like me? And so I think this is where it's really important to do the math of knowing exactly who's going to be a winner, who's going to be a loser, who's going to pay more, who's going to pay less. I think it's really salient that a lot of the value of land is commercial downtown real estate. And I think that a revenue neutral property tax shift land where we exempt the taxation of all buildings, but collect the same amount in property taxes as we're doing now, but just from the land and then a modest citizen's dividend is a really good first step. And then over the years, you can raise the land value rate as you also decrease things like income tax and sales tax. I think that's a transition that gets us there without really screwing anyone over. And for any edge case, like a poor sympathetic widow who has no income and but has a high value home, you just make it so she doesn't have to pay the land value tax until she dies or sells the estate. Dwarkesh Patel - 00:59:32: Yeah, yeah. And like I guess even in there, the worst case scenario is the status quo where they don't have to pay land value taxes, which is already the case now, right?Lars Doucet - 00:59:42: I mean, nobody cares about the people who are being evicted and displaced by the status quo. Dwarkesh Patel - 00:59:47: One, like I guess, snafu in terms of like figuring out how to price the land is by the time that a land value tax was passed, it'll have been like years after this political talk of having a land value tax. And that talk will in turn affect the prices of homes that are sold in that time. Absolutely. And so then you'll look at the land selling value and be like, oh wow, this is this house. It's not like the outskirts of San Francisco only sold for $200,000. And does that mean that like the unimproved land there is only worth $100,000? And so that just can, that really conflate the data when you actually go about implementing this of like what the actual unimproved land is worth. Lars Doucet - 01:00:28: Well, so I'm so important to remember that land selling value is derived from land rental value, not the other way around. So land selling value is the net present value of the future flow of income that can be generated from the property. So the property's inherent productivity is kind of inherent to it. And the selling price of it is based off of the capitalization of that value minus the expectation of any taxes, right? And so 100% land value tax will theoretically reduce the selling price to zero. But the land will still be as productive as it always was. It's just that the flow of those rents are being redirected, right? And so that's the thing is that, and also in mass appraisal, one of the things you do is you decapitalize the effect of the tax. Dwarkesh Patel - 01:01:18: But I'm saying, how do you even figure out you don't know in the mind of the property owner or the property seller, like what is it? What do they think the probability is of a tax? And that, since you don't know that, it's like hard to estimate what is the actual like capitalization, you know, the right value.. Lars Doucet - 01:01:35: What are you concerned about like this is societal effects of the depreciation of land prices? Are you more concerned about just the calculation issue? So here's the thing. Empirically, if the land value, if the land value, basically, if the land selling price has dropped to zero, then you are fully capturing all of the land rents. And if it's above zero, you have not captured all of the land rents. Dwarkesh Patel - 01:02:00: Oh, okay. So like maybe in the first year, you implement this. It's like, not, you know, maybe you're a little implemented like a, you're like basically in the first two years, you implement this, you would like be trying to mess with the rate. Lars Doucet - 01:02:11: Right. It's like, it's like, I mean, it's more complicated than this. But if there's any vacant lots in the area, and they're selling for anything, yeah, there's still land rent in that in that property. Dwarkesh Patel - 01:02:21: Gotcha. But so this is not something you would be able to figure out day one. You would have to like over a course of years of fudging the numbers Lars Doucet - 01:02:26:We're doing property tax assessments right now. You know, we're doing mass praise all the time right now. And if you just keep it updated every year, I mean, you could do it every six months if you had the right technology, which is something I'm pushing for. You know, and so you can see the prices change in real time as transactions come in. And you can use multiple regression and geographic weighted regression to work out the difference between the improvements and the land prices. And if you had a rental registry and knew what everyone was paying in rent, so you'd be able to keep even better track on what's going on with that. Dwarkesh Patel - 01:02:56: Yeah, this might be actually a good point to talk about your new startup. This is actually something I don't know about either. So yeah, what is the idea? What are you up to? Lars Doucet - 01:03:04: So my new startup is, you know, so I'm transitioning out of video games and into master praise municipal property tax assessment, mass appraisal. And so my new startup, it's called Geo Land Solutions for now. We're probably have a new name for it by the time that this podcast airs. But the idea is that, you know, I think the best criticism to Georgism is, you know, well, how are you actually going to separate land value from improvement, from building value, right? You know, how like we can't do a land value tax till we put a price on every parcel of land, right? So how are you going to do that? And I thought that was the best, most good faith, you know, criticism remained. And it seemed like it's like, well, we gotta get good at that, right? And so I looked into it and I realized that there is a lot of research papers that have been posted in the last 15 years about how to practically do this. And then I went and I started interviewing a bunch of assessors and I realized that the state of the practice is pretty far behind. Only 15% of most property tax assessment offices even use multiple regression. A lot of them are using the cost approach, which is basically where you, it's what Ed Glazer talked about in your interview with him where you estimate the cost of building and you applied depreciation and you subtract it from the observed selling price to get, you know, the assumed land price networks, and that works okay. But a lot of assessment is not only using, essentially only that method, but also, um, another issue is that just, um, a lot of those cost tables are very out of date. Um, uh, assessments themselves are not always done every year in some, it's not imperative to find places that haven't done reassessments in, you know, more than 10 years. Um, most places it's like, you know, one to five. Um, but even that, I think we should be doing everything we can to get all the latest mass appraisal technology and research. Um, and so we've hired some people who have, like, I basically, one of our first hires was one of the guys who was just first author on all these papers we were reading. And,we're just here to update municipal property tax assessors on the latest methods so that we can accurately know what all the land in America is worth, you know, and then this will solve a lot of our aggressivity issues too, because we know that a lot of landlords are actually under assessed relative to homeowners, believe it or not, because they're more likely to protest their property taxes. And minorities tend to be over assessed and poor people tend to be over assessed. That's why property taxes are sometimes called regressive is the assessments need to be fixed. Dwarkesh Patel - 01:05:41: I see. I guess because if you have more property, getting your rate changed from like 1% to 1 point, uh, 0.8% is like worth thousands of dollars rather than like a hundreds of dollars. Lars Doucet - 01:05:54: Right. And there's, there's, there's a lot of other issues too. Is this is like, there's more, there's more value in the like the more pro, there's all sorts of reasons that you can have these things, you know, often from no malintent whatsoever. Right. You know, but just like just out of date assessments can, can also cause all sorts of issues. Dwarkesh Patel - 01:06:11: Yeah. Yeah. I guess one worry that people might have is, well, let's say an income taxing is like, you know, obviously very inefficient. What is the, I think you, in the book, you're talking about the percentage of income, uh, that, uh, of tax income that is literally just spent on figuring out how much income tax to collect and how, how many people have paid or whatever. So I get that. But at least it has a nice property of there is like, it feels like there's this hard figure that ought to exist. Like I made this much income this year. Whereas figuring out how much land is worth, you know, justlike, I get that it's basically like, yeah, if you, you know, figured out like the rent value, that's like the value of land, but it just feels much more murky. And therefore it might potentially enable corruption in the level of like, you know, like whoever's doing the assessment or however that method of assessment is happening, um, they'll just like use these fancy algorithms to nudge it one of your another that benefits big corporations or, you know, whoever they want. Lars Doucet - 01:07:09: So that's a really, that's a good argument. But the reply to that is that we need to move towards more transparency, right? Because land value follows certain rules that should logically make sense, right? We know some things that drive location without first of all, we should move towards open source models, right? Which is something that we want to do and open data whenever possible. A lot of these cities will post open data portals like my mission in life is to be able to like advance the state of the art of this technology and make it so anyone can kind of check on stuff. You should be able to in any city in America, some cities have this been on enough. You should be able to look up your property tax assessment on a map and compare it to your neighbor. And what you shouldn't see is a Christmas tree effect, where your neighborhood looks like Christmas tree lights of like green and red of people whose property tax assessments like massively differ. That's the case in a lot of cities. Like if the value, like if the land values have been correctly assessed in this neighborhood, most of these parcels should be about the same, like probably the cul-de-sacs worth a little bit more, you know, but like your prop, your neighbor's land shouldn't be worth 20% more than yours. And if it is, it'll stick out on a map like a sore thumb. And if the data and the algorithms are all open source and open data, you should be able to check anyone's math. And you should be able to use that to then protest your taxes if they seem off. And that's kind of the argument for land value taxes is that, I mean, you can hide all kinds of stuff in income taxes and capital taxes. I mean, that's what the Caymans and the Bahamas are for. But if you want to, I mean, it's very easy to find people are getting a break on their property taxes like all this corruption. I'm not saying it won't happen, but it'll be very easy to see because you'll be able to see just this mansion that suddenly has this discontinuity on the land value map. It's like, well, someone gave this person a break and maybe, maybe we should, maybe we should write an article in the local newspaper about this. Dwarkesh Patel - 01:09:01: Yeah, it'll be way better than how income taxes work. None of it's like open to potentially even the government itself, right? But okay, so another concern is that's fine for things that are like above ground and legible. But what about you find out through, like, yeah, I don't know, like some sort of surveying or whatever that this land has a lot of oil under it. And then you buy it. But obviously you're not going to tell the government that I just found like however many. Lars Doucet - 01:09:26: So this is the sort of theoretic critique… Dwarkesh Patel - 01:09:27: Not, not even that is literally like you won't declare it. So in a sense, you're still being a speculator, but in fact, you're incentivized even more to be speculator in the sense that as soon as you declare that there's oil underneath this ground, then the government is going to start taxing you for it. So you just want to like hang on to that for, you know, you forever as long as like you can like, you know, keep the private right Lars Doucet - 01:09:48: So we need to talk about mineral policy in America, because especially instead of Texas, like mineral rights and land rights are totally different, right? Usually when you buy land in Texas and in America, like you actually don't have the mineral rights.. Like those are, those are very severable. Like a lot of people are very interested in, in getting those mineral rights. And so usually like if you're not paying attention, generally speaking, you're not getting the mineral rights when you're buying land. And if you are, you're paying more for them. And so I mean, I think a good example of this is probably like, you know, you want to create some, I think with the case of minerals, you can't just have a full, I think as kind of in acknowledgement of Kaplan's like, theoretic thing, like you have something more like an Norwegian model where you need to basically give some incentive to someone to produce or to not withhold that resource, right? I'm a good example would be the treasure, the treasure law in England because England has all this ancient likeAnglo-Saxon treasure and Roman treasure. And so before what they would have as someone would find it and like go like hide it or melt it down because they didn't want the government to tax is treasure law, which is not perfect, but it's okay. But they're like, we, this is our heritage, right? We want that in a museum. So here's the deal. If you find treasure, you're going to get paid. And the landowner's going to get paid. So there's an incentive for people to go out with metal detectors. And there's an incentive for a landowner to let people do that. And then the government's going to put it in a museum. Yeah. You know, you're going to, you're going to be rewarded for the discovery of that thing. But that thing is itself, its value is going to be captured because it's the heritage of the British people. The opposite of this is the Spanish government. Whenever someone finds a ship full of gall, a galleon full of gold on the bottom of the ocean, if you go and you invest the capital, bring that up to the surface, you know, and you're in Spanish waters or the Spanish government finds you, or you like take it not to Spain, a Spanish admiral in court is going to try to get its claws on that gold. In the view of the Spanish government, like basically the incentives they're producing is to make sure that nobody ever, ever recovers a ship. Dwarkesh Patel - 01:11:59: Yeah. And if they do. It just ends up in some sort of like, you know, a foundry in another country. Lars Doucet - 01:12:05: Right, we're right, right. The Spanish government's approach to sunken treasure basically incentivizes people never to go after. Dwarkesh Patel - 01:12:11: Yeah. Yeah. I guess a general critique of Georgism, too, in general, or implementing it would be less than the reason America is wealthy and other developed countries are wealthy is because they are very strong in terms of honoring contracts, especially honoring people with property rights. And, you know, if we get to a scenario where the government is saying, okay, but in this case, we're going to, I mean, I guess it depends on how you think about the contract property. But in some sense, it's a contract you have the government that like, hey, I have this property. And once you don't honor that, like the people will get too concerned to want to invest in America or in American assets. And that will just have like all kinds of economic repercussions where people are like, oh, I guess like my property is not mine. Maybe like other things I thought I was investing in like my stocks are not mine. So why should I buy these stocks in America? Lars Doucet - 01:13:00: This is a fully general argument against change. And it also is like, I don't agree with all the assumptions. Like, you know, if America doesn't honor its agreements, what are they worth? Well, we've violated all kinds of treaties, with the Indians, especially, but also like all sorts of international treaties. And, you know, I mean, the like, we've made all kinds of times where we've like changed rules on things and changed asset classes. You know, we had an entire period where we just banned all sales of alcohol. And then we had a time where we completely undid that and brought it all back. You know, there's been times where we've like made major, major changes to the rules of what kind of asset classes we have. I mean, this is kind of like brought up with like, you know, any kind of like labor protections and things is, you know, is it's like, well, we have these rules. And therefore, we can never change them. And I think, I think the best answer to this is that you need to acknowledge Gordon Tullich's transitional gains trapif there's someone who's going to be, you know, put out, then you make sure that there's some compensation in the system to smooth over the transition. But the rule of law doesn't imply that any change to the status quo is going to undermine trust in it. And I don't think that, and I think it's important to remember that George is not interested in seizing land. That's a very big distinction from the Maoist position, which is murder of the landlords or the more like modest Asian reforms, Asian reforms, which is like, we're just going to, so first of all, when you're talking about how Asia works, like they took the land away from the big landowners, gave it to the peasants, and it made those countries way more productive, right? And like, I don't think anyone would like look at those Asian countries now and compare them to where they were going to be like, well, the rule of law is weaker now than it used to be. But George isn't even advocating for that. He's just advocating for raising taxes on land and exempting all of the building taxes.So I don't think it amounts to seizure of land. And I think for those reasons, but even if it did amount to that, I think, you know, sometimes it's worth biting some bullets. Dwarkesh Patel - 01:15:11: Yeah, yeah. Okay, let's move on to the dessert. Let's move on to some more fun interpretations and applications of Georgism. So one, I think Byrn obart had this blog post on the diff. I don't know if you follow, but he's like a great finance writer. And he was talking about how if you have a Facebook account or a YouTube channel with like millions of subscribers, you're in some sense, like you were like very early to YouTuber Facebook, and now you have this land. And Facebook punishes you if you're have a big account, but not hosting frequently or not getting enough engagement, it'll like in the future, you'll have a harder time reaching people. And, you know, Byrn Hobarthad this like Georgism and Georgism interpretation of that. Where it's like you have this productive asset of, you know, people's attend like this, this initial sort of profile that you're able to build on the early days of Facebook, if you're not like posting on it, we're not going to give you the advantage of having millions of buys. But anyway, so there's all kinds of like ways you can apply to Richardson to you can think of like the app store as this sort of like rent seeking from Apple or the charger was 30% tithe. And yeah, there's all kinds of other places in the digital world where you can think of this. Like what is your sort of like, how do you think about Georgism in the context of those kinds of things? Lars Doucet - 01:16:30: Right. So I've actually written a policy paper on how to apply the theories of land value tax to virtual worlds. And so the question is virtual real estate is when does something actually operate like a land like asset? And then to what extent does Georgist principles apply? Right. And it doesn't necessarily mean just do LVT. It means that, you know, so first of all, let me define what I mean by a land like asset. A land like asset has three properties. It is scarce and supply. It is necessary for production. And it obtains locational value by virtue of its position in some kind of graph. Right. And so as an example, I create like a fictional MMO and I give the example of a unicorn, a permit, and a plot of land. The unicorn is scarce, but it's not necessary for production. Any value you can get from the unicorn, you can get some other ways. It's really nice to have. And there's only like 10,000 on the server. A permit is like a permit to brew potions. You're part of the witches guild. You can brew potions. So like my apprentice, witch she has to pay rent to me to gain access to my permit to be allowed to brew potions, but all permits are fungible. So there's no locational value. And then a plot of land, right? Like we've talked about where you can also, and so this becomes the real speculative asset. Domain names are probably the closest thing to this. And Vitalik Buterin has actually written a post about how to apply Georgist theory to domain names and also all the wrinkles that are involved. Dwarkesh Patel - 01:17:55: Because we did this on your blog, right? As a guest post. Lars Doucet - 01:17:58: Well, progressandpoverty.substack.com is not technically my blog. It's a group blog where a lot of us, Georgism's post. But so yeah, so my blog in the sense of our blog. Yeah. And he cross posted it. He posted it also on his own blog. And so, but there's other considerations to remember when we go away from literal land is like, so for instance, in domain names, you know, especially with the Ethereum name system, you have issues of identity, right? That don't apply like, if someone buys this house from you, probably people aren't going to be like maybe the next person who lives here is going to get some of your mail for like a week or two. But like no one's going to think that they're you necessarily. But if I buy Vitalik.eth. Yeah. Like there might be some confusion. Like I might be able to get some transactions that were meant for him. And so there's these other considerations to like kind of and he deals with all of that. And so I wrote a policy paper about how to apply LVT and virtual worlds and stuff. That was more in these more kind of literal simulacra of the real world in these like virtual worlds than kind of the things you're talking about of like YouTube accounts and and charts in like in basically app stores and stuff which are user generated content platforms. But I haven't fully analyzed user generated content platforms. But I do think chart positions are a sort of virtual land. You know, and you do if essentially they do essentially charge rent for that. Not just to the platform in their flat 30% fee across everything, but also in the kind of advertising red queens race you have to do to stay on those charts. You essentially have to buy that position and then keep it. You know, and so basically like the fur there's this huge first mover advantage that turns into rent seeking. And I haven't fully analyzed exactly how you applied Georgist theory there. But I do think there's something to it. I also think there's something to extremely long live copyrights and patents. It's a little undercooked at the moment. I don't have a fully fledged theory of it. But certainly cases like othermonopoly assets like orbital real estate, radio spectrum, any kind of like being able to capture the right of the possibility space. Like especially like, you know how like John Carmack had a lot of algorithms like some of which were patented and he wasn't even allowed to use them in his own games. Because someone else had speculated patented like Yeah. It was like the, like Carmack's reverse was like the particular algorithm that they weren't allowed to use and like quake three or something because someone had speculatively patented it. And there's like no other way to do that thing. So quake three, this one particular sub routine had to be like 25% slower. Because they because someone had patented like essentially like, like imagine you get like patent that Pythagorean theorem. You know, yeah. So like that kind of nonsense. And you can just rent seek off that for 20 years. Dwarkesh Patel - 01:20:51: Jesus. That's funny. Anyway, I guess we're going to move on to another juicy implication that you were just talking about, which is space. There is like this idea. I mean, eventually, hopefully humanity will conquer space and like the rest of the galaxy. So and this is where like Georgism and will be really interesting and applicable because we obviously want to we want to encourage and incentivize people to like go to new worlds and make use of like stars and planets. But we obviously don't want it to be the case that if you got to Mars a year earlier, like if Musk gets to Mars a year before Bezos, he has you know, like forever rights to rights toEverything on Mars and all the resources, right? So yeah, Georgism in space would be I think I think is actually a place that makes a lot of sense. But I mean, have you put much thought into like, you know, interstellar Georgism? Lars Doucet - 01:21:41: Interstellar Georgism is actually current international law. So really the inner through the outer space treaty. The outer space treaty currently said, and I think the outer space treaty will last for about five minutes once the interplanetary space race gets going in earnest. But the current law, the law of the international order of the outer space treaty basically says that nobody's allowed to claim interstellar bodies like the moon or Mars for China or the US. Like we have a flag there, but it's not American territory. Yeah. Right. It's basically international waters, so to speak, right? And once the actual possibility of having permanent bases shows up, then we will have to hash that out. But so basically I think we should take that and run with it and basically be if you are going to, you know, take possession of interstellar bodies, you know, it will become a question of who exactly becomes the government in that scenario. Is it like some international coalition? Is it the UN? I'm not a huge fan of the UN, you know, or is it just like it turns out to be like whatever sovereign government gets their first just gets to claim it, but can you have Georgism and within those confines? I think we certainly should. Yeah. You know, because otherwise what you're going to get is that you're going to get under investment, you know, you're going to get so much under investment because the first people to get there are going to basically charge rent to all the people who come next. Dwarkesh Patel - 01:23:05: Yeah. Yeah. I mean, I just remember the other question I wanted to ask about copyright, which is that's a really interesting and complicated area because it's hard to think of what is the intrinsic sort of value of the land of the idea and what is the improvement you've made. Right. And so like what do you got to call like the improvement on like the song you discovered? Maybe like the melody is like the melody would have existed anyways, but like the specific lyrics that came up with. I don't know. Lars Doucet - 01:23:31: Well, so like copyright, it's not clear to me exactly what the implication of Georgism is. It's like Georgism if you Zoom out, it's not exactly just about land, right? That's where people a lot of people like kind of like why are you so obsessed about land? It's mostly about enclosure of, enclosure of natural monopolies, right? And and and and and and and and and economies just based entirely around rent seeking. And it's clear we have this with like eternal copyright. If we had eternal, if we had the copyright laws we had today, when Disney was first getting started, like the Brothers Grim would still be under copyright. And they would not have been able to get off the ground with a lot of their early properties. And then they pulled the ladder up right behind them. It's clear they're rent seeking in a lot of ways and refusing to give back to the comments from which they first enriched themselves. And the question then becomes like, you know, can you easily parse copyright as land? I mean, it's a very undercooked theory, but there'ssomething there of like probably what it catches out to is just copyright terms should be shorter. You know, like at some point when an idea has become part of the cultural consciousness for like so long it should become part of just the background collective commons, you know, because all ideas are essentially remixed and built on top of other ideas. But we want to incentivize people to create new ideas in the first place. It's not a perfect analogy to land, but there is something there about rent seeking that needs to be addressed. And it probably just catches out to just reducing copyright terms. Dwarkesh Patel - 01:25:02: Well, let's talk about the political feasibility of this idea. So, I mean, I just mean land value tax in general. Do you think it will be something that will get passed in a democracy? I, you know, there's like, you know, a lot of people have homes or want to have homes, but on the other hand, if you like just redistribute, if you don't have like, I don't know, many acres in the middle of San Francisco, you would maybe still come out net ahead, but could you be able to explain that to people? And obviously there's like tenants who would definitely benefit from this. Anyways, in general, what is, what is your thought and like, how politically feasible this is now it or would be in the future? Lars Doucet - 01:25:36: I think it's more politically feasible than we think. I think it just right now has low salience. And if more people start talking about it, I think it's going to make a lot of sense, especially if it's pitched as property tax reform. Like right now in Texas, you know, you have people who are trying to abolish property tax, which will turn this state into California like that. Dwarkesh Patel - 01:25:55: Why? Huh? Why? How come? How, well, why does he want to abolish property tax? Why would into California? Lars Doucet - 01:25:57: Because California has some of the lowest property taxes in the nation. Yeah. And proposition 13, and it creates a entire layer of landed gentry and really, really, really expensive housing. Like housing is already getting expensive in Austin and property taxes have an effect of lowering property prices. Property taxes are an imperfect land value tax. And so basically, I, it will just reproduce the economic conditions that empirically exist in California. Dwarkesh Patel - 01:26:21: You know, there's like, there's this like funny saying that the Texas Constitution is not a government document, but rather an anti government document. But one of the, I guess, the positive bi-products of you’re a Georgist is since the next taxing income so hard, governments are forced to tax property and hopefully eventually land. And therefore, Lars Doucet - 01:26:40: I'm going to pick on the Texas Constitution because Houston had a, had a single tax mayor in 1911 and had an active, Georgist land value tax. And it would have still had it today if it wasn't for the state const for, for a state judge who basically shut them down. Because there's a clause in this state constitution of Texas that says all property taxation has to be uniform. And that's interpreted as uniform across both the buildings and the land. You can't tax them at separate rates. Dwarkesh Patel - 01:27:09: Yeah. Wait, so does that mean that constitutionally you could just go into it and do it in a Texas Senate? Lars Doucet - 01:27:14:It's up to interpretation. There's some people interpreted that it's like, what, what, that that clause was written to mean you can't tax this guy at a higher rate than that guy. You have to tax them based on the value of their property. And then you could claim that it's still being uniform. We're just taxing land and we have all these exemptions and categorizations already. This is just another one of those. Why, you know, we tax agricultural land at a different rate, you know, what I mean? Like if you put a bunch of cows on a property suddenly, it's like it's magically less valuable. So why can't we, you know, like target just the land? And so if you have the right judge, maybe you could get away with that. But anyway, to go go towards political feasibility, I think it could be more feasible than we think, especially pitched as property tax reform, because people are like property taxes are too high. Hey, everyone, let's exempt all your buildings. I think you could build a coalition that's excited about that. And especially if you do the math and show who like, like what the change in your taxable rate is going to be, I think, you know, you know, just a revenue neutral property tax shift to land can be quite popular. There's a lot of cities around the US right now where this is being floated, you know, organizations like Strong Towns and the Center for Property Tax Reform are working on a lot and Lincoln Land Institute are like talking to places about it, Detroit is talking about it right now. And they could desperately use it. And I think it could pass, you know, and then in terms of, I think Henry George's salience is coming back, like, and especially in Norway, like the Ruling Center Party coalition just passed a, you know, they're very successful resource management policy with oil. They also have one from the early 1900s in hydro power, which was set up by Norwegian Georgists. And so the Ruling Center Party coalition just put one in for salmon farming aquaculture locations. A new severance tax on that. And they name check, Henry George, when they implemented it in the speech. And so I think if they are willing to, you know, stand up to the landowners in Oslo and pass a land value tax, that would be the next step. I'm not sure if they're brave enough to do that. But we're starting to see this kind of bubble up. And so I think a revenue neutral property tax shift to land is is the politically popular way to do it. Because it can be done on the local level without having to change a whole bunch of laws. You know, it's a little complicated in Texas because that's stupid state constitutional provision. But in other states, they don't have those thoughts. There's other states that any municipality could do it right now if they wanted. Dwarkesh Patel - 01:29:33: And I guess another part of this question then, or the answer is, well, one of the things you talk about in the book is, you know, in some sense the value of your land is caused by the other people around you, by the properties of like other amenities and companies or whatever that are in. So it's kind of like a publicly contributed. But if you think about it that way, doesn't it make sense that your land taxes should go to that community, which is the one that's creating all this value, rather than being say distributed federally. So maybe we should have land value taxers on local levels. But then it goes to pay towards local amenities. So it's like if you live right next to the subway and the reason there are properties of valuables because like you're right next to the subway, then that goes towards making the New York subway better. But it doesn't go towards like, you know, I don't know, like some sort of the city hall and Kentucky or something. Lars Doucet - 01:30:21: Right. I mean, it's like if you wanted to create a model for, you know, bottom up decentralized America, like I would take that bargain. We still got to like fund the federal government somehow, you know, and so like I would like to repeal, you know, the federal income tax and have that funded by land value tax. But it's like, I mean, if that's like as far as I could ever get, I'd shake your hand and take that deal right now. You know, it sounds good. Dwarkesh Patel - 01:30:42: Yeah, fair enough. One more question. This was naturally from Twitter for I'm Craig Bratric, which is the question is basically if, I mean, one of the reasons why we think of land as being sort of like a public thing is because it's your land value is contributed by people around you. But what if you own so much land that really you are the one that's contributing to the value of the all your land, right? So if you're thinking like the thing like Disney World, they like they basically own like half of a world land or something. And they're the ones that are creating nearby amenities, which are making the rest of Disney world valuable. So in that case, are they entitled to all the proceeds? Lars Doucet - 01:31:19: And what's funny is like in a way, yes. And so what's interesting is like this is often like posed a gotcha question for Georgists. But we actually like internally like this is called the the Disneyland or the Disney World question, you know,and it's actually a really interesting case. And the thing you have to realize now is who is Disney World in this situation? Disney World is th