An excerpt has surfaced from the AI2027 podcast with siskind and the ex AI researcher, where the dear doctor makes the case for how an AGI could build an army of terminators in a year if it wanted.
It goes something like: OpenAI is worth as much as all US car companies (except tesla) combined, so it could buy up every car factory and convert it to a murderbot factory, because that’s kind of like what the US gov did in WW2 to build bombers, reaching peak capacity in three years, and AGI would obviously be more efficient than a US wartime gov so let’s say one year, generally a completely unassailable syllogism from very serious people.
Even /r/ssc commenters are calling him out about the whole AI doomer thing getting more noticeably culty than usual edit: The thread even features a rare heavily downvoted siskind post, -10 at the time of this edit.
The latter part of the clip is the interviewer pointing out that there might be technological bottlenecks that could require upending our entire economic model before stuff like curing cancer could be achieved, positing that if we somehow had AGI-like tech in the 1960s it would probably have to use its limited means to invent the entire tech tree that leads to late 2020s GPUs out of thin air, international supply chains and all, before starting on the road to becoming really useful.
Siskind then goes “nuh-uh!” and ultimately proceeds to give Elon’s metaphorical asshole a tongue bath of unprecedented depth and rigor, all but claiming that what’s keeping modern technology down is the inability to extract more man hours from Grimes’ ex, and that’s how we should view the eventual AGI-LLMs, like wittle Elons that don’t need sleep. And didn’t you know, having non-experts micromanage everything in a project is cool and awesome actually.
Stephen Hawking was starting to promote AI doomerism in 2014. But he’s not a Nobel prize winner. Yoshua Bengio is a doomer, but no Nobel prize either, although he is pretty decorated in awards. So yeah looks like one winner and a few other notable doomers that aren’t actually Nobel Prize winners somehow became winners plural in Scott’s argument from authority. Also, considering the long list of example of Noble Disease, I really don’t think Nobel Prize winner endorsement is a good way to gauge experts’ attitudes or sentiment.
I was very tempted to go ‘don’t think it is more than one nobel guy, which is not great because of nobel disease anyway. I could link to rationalwiki here but that has come under threat because the people whos content you enjoy Scott started a lawsuit against them’ but think that might be a bit culturewarry, and I also try not to react at the places we point towards. As that just leads to harassment like behaviour. Also Penrose is a Nobel prize winner who is against AGI stuff.
Yeah it’s really not productive to engage directly.
I’d almost categorize Penrose as a borderline case of noble disease himself for stuff he’s said about Quantum Consciousness and relatedly the halting problem and Godel’s incompleteness theorem. But he actually has a proposed mechanism (involving microtubules) that is testable and falsifiable and the physics half of what he is talking about is within his domain of expertise.
This is why I very much dislike Popperism. Popperites are convinced “science = falsifiability.” If I argue that the universe is made of cheese and the mechanism is a wizard that you can only see through a telescope with a special handcrafted ruby lens that I sell at my shop for $4000, should research institutions be expected to take my claim seriously and buy my ruby lens to test it? I mean, it’s technically falsifiable, either they will look through the lens and see the wizard and universe of cheese or they will not. If you are a Popperite you have no choice but to admit that it is a legitimate scientific theory.
There should be more to a scientific proposal than it technically being “falsifiable.” Penrose’s “theory” is quantum mysticism, it is not a scientific theory just because it is in principle testable.
It’s even more ridiculous when you realize that microtubles are structural, they don’t play a role in information processing in the brain, and you have microtubules all throughout your body. Them having quantum effects in them is meaningless. Even if you could empirically demonstrate without a shadow of doubt that microtubules do somehow create coherent quantum states that the brain makes use of, that would just be an interesting fact on its own. It would not prove #1 or #2. Microtubules are not a “mechanism” for #1 and #2, even if they played a role in decision making as if the brain is a quantum computer (they don’t), then you cannot derive from this that somehow quantum mechanics explains why people can believe things without proof (why do I even have to say this, it’s so stupid!) or that the reduction of the wave function is a physical process caused by gravity.
There is no good argument to believe even #1 or #2 are tied together. Even if you proved there is indeed a non-local physical collapse and overturned all our modern scientific theories, that wouldn’t demonstrate #1 or #3 either. None of the claims in the theory have any obvious connections to one another other than spurious, largely incoherent arguments. This is not his domain of expertise. You could argue #2 is within his domain, but #1 and #3 are nowhere near his domain.
Physicists have proposed speculative physical collapse theories before, like GRW, and we forget about them because they were interesting but went nowhere because there is zero evidence “collapse” is a physical process, and treating it as such requires overturning all of modern physics, as it could not be made compatible with special or general relativity nor could it reproduce the predictions of quantum mechanics, requiring you to rewrite all of physics from the ground-up. The reduction of the wave function is a measurement update, it is epistemic, there is no evidence that it is a physical process.
Even then, theoretical physicists speculate about a lot of things that turn out to go nowhere, that itself is par for the course. But Penrose goes above and beyond this and branches into philosophy, biology, and neuroscience and starts using comically bad arguments to try and tie them all together. Those are not his areas of expertise at all. It reminds me of the old essay Natural Science and the Spirit World from the 19th century that documented a lot of renounced scientists who also had completely crazy side projects, like Alfred Wallace, the guy who co-discovered evolution by natural selection, who believed he could also raise spirits from the dead and converse with them.