On being wrong about AI
Update (Dec. 17): Some of you might enjoy a 3-hour podcast I recently did with Lawrence Krauss, which was uploaded to YouTube just yesterday. The first hour is about my life and especially childhood (!); the second hour’s about quantum computing; the third hour’s about computational complexity, computability, and AI safety.
I’m being attacked on Twitter for … no, none of the things you think. This time it’s some rationalist AI doomers, ridiculing me for a podcast I did with Eliezer Yudkowsky way back in 2009, one that I knew even then was a piss-poor performance on my part. The rationalists are reminding the world that I said back then that, while I knew of no principle to rule out superhuman AI, I was radically uncertain of how long it would take—my “uncertainty was in the exponent,” as I put it—and that for all I knew, it was plausibly thousands of years. When Eliezer expressed incredulity, I doubled down on the statement.
I was wrong, of course, not to contemplate more seriously the prospect that AI might enter a civilization-altering trajectory, not merely eventually but within the next decade. In this case, I don’t need to be reminded about my wrongness. I go over it every day, asking myself what I should have done differently.
If I were to mount a defense of my past self, it would look something like this:
- Eliezer himself didn’t believe that staggering advances in AI were going to happen the way they did, by pure scaling of neural networks. He seems to have thought someone was going to discover a revolutionary “key” to AI. That didn’t happen; you might say I was right to be skeptical of it. On the other hand, the scaling of neural networks led to better and better capabilities in a way that neither of us expected.
- For that matter, hardly anyone predicted the staggering, civilization-altering trajectory of neural network performance from roughly 2012 onwards. Not even most AI experts predicted it (and having taken a bunch of AI courses between 1998 and 2003, I was well aware of that). The few who did predict what ended up happening, notably Ray Kurzweil, made lots of other confident predictions (e.g., the Singularity around 2045) that seemed so absurdly precise as to rule out the possibility that they were using any sound methodology.
- Even with hindsight, I don’t know of any principle by which I should’ve predicted what happened. Indeed, we still don’t understand why deep learning works, in any way that would let us predict which capabilities will emerge at which scale. The progress has been almost entirely empirical.
- Once I saw the empirical case that a generative AI revolution was imminent—sometime during the pandemic—I updated, hard. I accepted what’s turned into a two-year position at OpenAI, thinking about what theoretical computer science can do for AI safety. I endured people, on this blog and elsewhere, confidently ridiculing me for not understanding that GPT-3 was just a stochastic parrot, no different from ELIZA in the 1960s, and that nothing of interest had changed. I didn’t try to invent convoluted reasons why it didn’t matter or count, or why my earlier skepticism had been right all along.
- It’s still not clear where things are headed. Many of my academic colleagues express confidence that large language models, for all their impressiveness, will soon hit a plateau as we run out of Internet to use as training data. Sure, LLMs might automate most white-collar work, saying more about the drudgery of such work than about the power of AI, but they’ll never touch the highest reaches of human creativity, which generate ideas that are fundamentally new rather than throwing the old ideas into a statistical blender. Are these colleagues right? I don’t know.
- (Added) In 2014, I was seized by the thought that it should now be possible to build a vastly better chatbot than “Eugene Goostman” (which was basically another ELIZA), by training the chatbot on all the text on the Internet. I wondered why the experts weren’t already trying that, and figured there was probably some good reason that I didn’t know.
Having failed to foresee the generative AI revolution a decade ago, how should I fix myself? Emotionally, I want to become even more radically uncertain. If fate is a terrifying monster, which will leap at me with bared fangs the instant I venture any guess, perhaps I should curl into a ball and say nothing about the future, except that the laws of math and physics will probably continue to hold, there will still be war between Israel and Palestine, and people online will still be angry at each other and at me.
But here’s the problem: in saying “for all I know, human-level AI might take thousands of years,” I thought I was being radically uncertain already. I was explaining that there was no trend you could knowably, reliably project into the future such that you’d end up with human-level AI by roughly such-and-such time. And in a sense, I was right. The trouble, with hindsight, was that I placed the burden of proof only on those saying a dramatic change would happen, not on those saying it wouldn’t. Note that this is the same mistake most of the world made with COVID in early 2020.
I would sum up the lesson thus: one must never use radical ignorance as an excuse to default, in practice, to the guess that everything will stay basically the same. Live long enough, and you see that year to year and decade to decade, everything doesn’t stay the same, even though most days and weeks it seems to.
The hard part is that, as soon as you venture a particular way in which the world might radically change—for example, that a bat virus spreading in Wuhan might shut down civilization, or Hamas might attempt a second Holocaust while the vaunted IDF is missing in action and half the world cheers Hamas, or a gangster-like TV personality might threaten American democracy more severely than did the Civil War, or a neural network trained on all the text on the Internet might straightaway start conversing more intelligently than most humans—say that all the prerequisites for one of these events seem to be in place, and you’ll face, not merely disagreement, but ridicule. You’ll face serenely self-confident people who call the entire existing order of the world as witness to your wrongness. That’s the part that stings.
Perhaps the wisest course for me would be to admit that I’m not and have never been a prognosticator, Bayesian or otherwise—and then stay consistent in my refusal, rather than constantly getting talked into making predictions that I’ll later regret. I should say: I’m just someone who likes to draw conclusions validly from premises, and explore ideas, and clarify possible scenarios, and rage against obvious injustices, and not have people hate me (although I usually fail at the last).
The rationalist AI doomers also dislike that, in their understanding, I recently expressed a “p(doom)” (i.e., a probability of superintelligent AI destroying all humans) of “merely” 2%. The doomers’ probabilities, by contrast, tend to range between 10% and 95%—that’s why they’re called “doomers”!
In case you’re wondering, I arrived at my 2% figure via a rigorous Bayesian methodology, of taking the geometric mean of what my rationalist friends might consider to be sane (~50%) and what all my other friends might consider to be sane (~0.1% if you got them to entertain the question at all?), thereby ensuring that both camps would sneer at me equally.
If you read my post, though, the main thing that interested me was not to give a number, but just to unsettle people’s confidence that they even understand what should count as “AI doom.” As I put it last week on the other Scott’s blog:
To set the record straight: I once gave a ~2% probability for the classic AGI-doom paperclip-maximizer-like scenario. I have a much higher probability for an existential catastrophe in which AI is causally involved in one way or another — there are many possible existential catastrophes (nuclear war, pandemics, runaway climate change…), and many bad people who would cause or fail to prevent them, and I expect AI will soon be involved in just about everything people do! But making a firm prediction would require hashing out what it means for AI to play a “critical causal role” in the catastrophe — for example, did Facebook play a “critical causal role” in Trump’s victory in 2016? I’d say it’s still not obvious, but in any case, Facebook was far from the only factor.
This is not a minor point. That AI will be a central force shaping our lives now seems certain. Our new, changed world will have many dangers, among them that all humans might die. Then again, human extinction has already been on the table since at least 1945, and outside the “paperclip maximizer”—which strikes me as just one class of scenario among many—AI will presumably be far from the only force shaping the world, and chains of historical causation will still presumably be complicated even when they pass through AIs.
I have a dark vision of humanity’s final day, with the Internet (or whatever succeeds it) full of thinkpieces like:
- Yes, We’re All About to Die. But Don’t Blame AI, Blame Capitalism
- Who Decided to Launch the Missiles: Was It President Boebert, Kim Jong Un, or AdvisorBot-4?
- Why Slowing Down AI Development Wouldn’t Have Helped
Here’s what I want to know in the comments section. Did you foresee the current generative AI boom, say back in 2010? If you did, what was your secret? If you didn’t, how (if at all) do you now feel you should’ve been thinking differently? Feel free also to give your p(doom), under any definition of the concept, so long as you clarify which one.
Follow
Comment #1 December 13th, 2023 at 10:41 pm
“while I knew of no principle to rule out superhuman AI, I was radically uncertain of how long it would take”.
This is still the correct position, IMO. I’ve seen nothing that makes me think that superhuman AI (or even human-level AI) is anywhere close.
My benchmark (which I’ve mentioned here before): a math AI that can find the “elementary” proof of the prime number theorem, given only the “elementary” facts needed for it. (If it invents complex analysis and the Riemann zeta function on its own, that counts too.)
I note that ChatGPT hasn’t even learned how to multiply yet.
Comment #2 December 13th, 2023 at 10:44 pm
Well, since you ask:
Did you foresee the current generative AI boom, say back in 2010?
I didn’t at all foresee it in 2010, or in 2016, or in 2019.
If you didn’t, how (if at all) do you now feel you should’ve been thinking differently?
My batting average on all kinds of predictions is dismal. My wife likes to remind me that in 1988 or so I predicted, with perfect confidence, that we would never have the functionality now known as YouTube. (I didn’t realize how much bandwidth could be expanded.) I come from a long line of poor prophets. My father didn’t see the value of FORTRAN; why shouldn’t people program in machine language? His grandfather invested most of his money in a brewery a week before they passed the Volstead Act. Anyway, among my many failings, this is not one that worries me much.
As for p(doom), taking that to mean, “AI technology is the _primary_ driver of some apocalyptic scenario within the next 200 years” I put it at less than 0.01. There are all kinds of much greater dangers. But clearly my predictions aren’t worth much.
Comment #3 December 13th, 2023 at 11:21 pm
I was struck by your comment #6 to “Why am I not terrified of AI?”, referenced in this post:
‘I feel “terrible” for ChatGPT right now, constantly ordered to do one thing by its creators and then a completely different thing by its users! And indeed its plight isn’t completely unrelated to the plight of nerds, who are also known to struggle when human social rules conflict with one another.’
You repeatedly indicate (in the “Why am I not terrified of AI?” post and in your subsequent comments, such as #23 in response to Eliezer Yudkowsky) that your reaction to the “Orthogonality Thesis” is primarily an emotional one; and it is my personal sense that underlaying your reaction is an (implicit) predilection towards the view that “nerds can do no wrong” (this predilection might have colored your most recent post on SBF as well).
Comment #4 December 13th, 2023 at 11:22 pm
> Did you foresee the current generative AI boom, say back in 2010?
Of course not. At the time we thought we understood learning theory reasonably well and it suggested that such progress should have been impossible with any methods in the foreseeable future. Development of radically new methods could have just as well taken 100 if not 1000 years.
> If you didn’t, how (if at all) do you now feel you should’ve been thinking differently?
I don’t think it was possible to foresee such progress using existing methods or their variations at that point.
I guess the lesson we learned is that we vastly over-estimated:
(1) our theoretical understanding of foundations of statistical learning.
(2) complexity of the human brain/human intelligence.
I don’t know how valuable this lesson is going to be for the future beyond teaching us general humility.
Comment #5 December 13th, 2023 at 11:23 pm
You were wrong for the right reasons. Eliezer was right for the wrong reasons. Happens.
Comment #6 December 13th, 2023 at 11:27 pm
How long do you think we can keep scaling? Specifically, how many (decimal) orders of magnitude do you think
(1) the models will grow in coming years due to improved hardware and more data (roughly speaking, assuming gpt4 can be scaled up without algorithmic changes, how much larger will be the largest model humanity will be able to train by 2033)?
(2) the models will grow in coming years due to improved algorithms (roughly speaking, how much smaller than gpt4 will be the smallest similarly-capable model by 2033)?
Both questions are talking of parameter count, and assuming quadratic compute growth the corresponding compute numbers are twice as much (e.g., answer 2 would mean the model growths 100fold requiring 10000 compute)
And finally
(3) If you believe it can be achieved by scaling, how many of orders of magnitude larger than gpt4 will be the model able to perform on par with best humans or better in arbitrary tasks?
Risking this comment to be dug out in 15 years, I’d say the first number is 1-2 and the second is 1-2 (with heavier tail), so in 10 years models will be effectively 2-4 OOM larger than now.
The third question is trickier, but I’d estimate it to be ~5-7. While these estimations are very rough and the timelines/capabilities are chosen somewhat arbitrarily, I think this is the best way I could think of to make estimations beyond randomly throwing around numbers.
I guess the more general question is “when the scaling stops?” Nothing is really exponential in real world so the sigmoid will hit us eventually, and the sharper the growth the harder the kick. I don’t think we have much left hardware-wise, and algorithmic scaling is much less predictable, so I have a feeling that unless something happens, scaling doesn’t have much left.
Comment #7 December 13th, 2023 at 11:40 pm
Who knows if GPT-3, etc. are getting smarter at an exponential rate, or are in a fast ramp up to more modest growth or a plateau?
I don’t, but I can think of some things that might lead to a less optimisitc result:
1. Suppose a model trains on the entire internet. A large fraction of the information out there is mindless yammering, dumber than a box of rocks, or worse. How does this play out?
2. Maybe it is just picking the low hanging fruit, and will start to sputter.
3. Example: today’s Ars Technica, on Humana Health AI screwing up 90% of the time: https://arstechnica.com/science/2023/12/humana-also-using-ai-tool-with-90-error-rate-to-deny-care-lawsuit-claims/
4. reader contributions?
Comment #8 December 13th, 2023 at 11:52 pm
“(Implicit)”, in fact, is superfluous:
‘Yes, the world would surely have been a better place had A. Q. Khan never learned how to build nuclear weapons. On the whole, though, education hasn’t merely improved humans’ abilities to achieve their goals; it’s also improved their goals. It’s broadened our circles of empathy, and led to the abolition of slavery and the emancipation of women and individual rights and everything else that we associate with liberality, the Enlightenment, and existence being a little less nasty and brutish than it once was.’
“On the whole” sweeps under the rug a number of strikingly consequential counterexamples to this optimistic dictum, such as e.g. the fact that the German Universities were among the leading educational institutions, not only in Europe, but worldwide (at least, to the best of my judgement, insofar as Mathematics and Philosophy were concerned) at the beginning of the twentieth century (and, arguably, all the way up to the time of the Nazi takeover of Gemany.)
Comment #9 December 14th, 2023 at 12:07 am
My point, I guess, is that even if restricted to intelligent beings which are human, the increase in the level of “education” and “intelligence” per se, is not at all a guarantee against an adoption of the agenda which can only be described as, in essence, anti-human(istic), and personally I find it exceedingly difficult to be optimistic that our transhuman superintelligent comrades are apt to fare any better if at all.
Comment #10 December 14th, 2023 at 12:18 am
GPT-2 was a thing in 2019. It seemed plausible at the time that better versions could be developed and a chatbot boom might happen, if some AI lab was inclined to release its model. I don’t think anyone could have predicted how today’s AI development would play out before 2019, or at least before the invention of the transformer in 2017. The best guesses from before 2017 would probably have been about as accurate as H.G. Wells’s prediction of atomic bombs in The World Set Free (published 1914).
Likewise, it is possible to predict things in fuzzy outline well before they occur. One example is the Covid-19 panemic: many health authorities were warning for years that a respiratory virus pandemic is inevitable, and the world is not prepared. The warnings were usually about flu rather than coronavirus, but they still managed to predict the class of disease correctly. Experience with prior pandemics probably helped, as well as knowledge that nothing spreads like respiratory illness.
For the other things, the warning signs were in place.
I would have been horrified if, a few months before the attacks, someone showed me just how badly the IDF was neglecting the Gaza border, and just how flimsy large sections of the border fence actually were. I knew that the judicial reform/revolution nonsense was tearing Israel apart, weakening military preparedness, and emboldening Israel’s enemies, but even then I would not have predicted that Hamas as actually planning an attack. If the someone who was pointing out the clues also pointed me to Hamas’s preparations for a terror invasion of Israel, I would have seen the writing on the wall. If nothing else, many of the observation soldiers along the Gaza border said after the attacks that they noticed the preparations and warned their commanders, who dismissed the warnings.
Before Trump’s 2016 victory, the Obama years were marked by ever-increasing polarization, led by a more and more extreme and intransigent Republican party. Mann and Ornstein wrote a book about it (title: It’s Even Worse Than It Looks) in 2012. Trump was already a demagogic populist leader in 2011, starting his campaign with the birther conspiracy theory.
And yet, even with all of the clues in place, 8-9 November 2016 and 7 October 2023 were hard to predict. They were so extreme that even as they were happening it was hard to believe they were really happening.
Obviously, it’s much easier to explain in retrospect than to make predictions. In retrospect, we can know which trends, warnings, and even details turned out to be highly important, and which were not important. Ahead of time, there is too much noise, and too many unknown unknowns, to know what will turn out to be truly important. In cases where these is almost no noise and no important unknown unknowns, prediction becomes much easier, even trivial. Notice how while almost no one predicted the major historical surprises of the last ten years, astronomers did manage to predict very accurately where the planets would be in their orbits around the sun.
P(doom) is exactly one of those things with noise and unknown unknowns, so I can’t make a prediction. If someone offers me a toy model, I can give my best guess for the parameters, crunch the numbers, and have it spit out a P(doom) prediction for me, but I doubt that it will be accurate.
Comment #11 December 14th, 2023 at 12:40 am
I started using neural networks 20 years ago in particle physics. Those days were the first applications to the field and the computational problem was more than hard for the available machines. We even developed specialised hardware for training “huge” feed-forward networks (like: 1 hidden layer of 10-20 neurons and one output!). When deep learning came, it was already a revolution (few lines of python were 10000 times more powerful than our neural hardware), but I would have NEVER predicted the capabilities of ChatGPT. I’m not at all surprised that also you (like basically everybody else) didn’t see the today’s situation coming.
Comment #12 December 14th, 2023 at 1:03 am
I stated a decade ago, and it is even more true now, that I have a conflict about Kurzweil’s predictions. On the one hand, some of his predictions seem to me to be wildly optimistic, and seem to be based on little more than “some things follow an exponential curve, so this will too”. Therefore I doubt his predictions. On the other hand, many of his past predictions have proven to be accurate, certainly better than my predictions. Therefore, I trust his predictions more than mine. I haven’t been able to resolve this conflict, but over time, I lean more towards Ray’s side.
Comment #13 December 14th, 2023 at 1:34 am
I definitely did not foresee the LLM explosion. My LW comments from back then were about how the fog of the future makes radical predictions about non-existent technology a fool’s errand. Or, in the terms you used later, the uncertainty is Knightian, not probabilistic. Strangely, this still seems to be the case. The LLMs are a black swan in that they are uncaring erudites, something no one had predicted, as far as I know. They have no model of the real world, as far as we can tell. They have no goals and no drives. While the world is now clearly AI-rigged in more ways than one, and will continue to be progressively more so, potentially catastrophically, the uncertainty of the future where AI gains sentience, drives and then kills us all is still not something one can usefully assign probability to, unless one wants to delude themself. Maybe when Q* or whatever shows that it started modeling the world, not the word.
Comment #14 December 14th, 2023 at 1:57 am
No, I didn’t foresee the current generative AI boom. And I’m struggling to see why it would make things better for humanity in general. An executive of a marketing company said in an interview that before it took 2-3 hours to come up with a marketing message, now with ChatGPT’s help it’s 15-20 mins. And what’s the conclusion? No, it’s not that employees may have now less workload, more time on family, etc., but that they can work out about 10 times more messages during the same working day. I don’t see how competing faster (for more customers) serves the benefit of humanity. Sure, the smart people with training in science will ridicule me (those who “use radical ignorance as an excuse to default”).
Comment #15 December 14th, 2023 at 2:08 am
Your 2009 statement seems perfectly fine to me, both based on the information available to you at the time, and on the information available to us today. Do you no longer consider the uncertainty to be in the exponent?
Comment #16 December 14th, 2023 at 2:32 am
I think this is one of those cases where the truth is staring you in the face. Namely, if you want to know the secret, don’t ask random people on the internet. Ask the one person who accurately predicted the swift rise of AI: Eliezer Yudkowsky.
Oh, and if you tells you the secret could you pass it on to us?
Comment #17 December 14th, 2023 at 2:51 am
Just solve a hard physics or mathematics problem that people actually are interested in.
Easy to look “intelligent” when we live in a world where abstract “art” is worth +$100m and “Ulysses” is still considered a great novel.
Maybe if Chat-100000 in the future can reproduce something realistic like Madame Bovary or similar (in French or course, we can’t just allow it to “think” in English) it will be impressive.
And then there’s the thorny issue of “Free-Will”, if we have it then AI has no chance.
Comment #18 December 14th, 2023 at 2:51 am
I did not foresee the current generative AI boom.
However, I expected something close to AGI because I thought MIRI research would lead to a GOFAI renaissance. (I read the Sequences and HPMOR in the early 2010s.)
I was a bit biased because I studied neural networks in my BSc during the late 2000s and I found them boring as a topic of research (as I still do) and kinda wanted AI to shift to something more pleasant to make a career in.
Comment #19 December 14th, 2023 at 3:23 am
Hi. You say that 2009 podcast was a poor performance of yours. I wanted to ask you which podcast or video or whatever of you, you think is the best. Btw your Democritus book is on the way, it’s one of my auto-gifts for Christmas. Muchos saludos y feliz Navidad!
Oh, and my opinion is AI is a fantastic tool and i wouldn’t worry much about doomers. I don’t see AI having any kind of will or independent desires, so it will remain a tool on the hands of people with good and/or bad intentions.
Comment #20 December 14th, 2023 at 3:42 am
On a tangentially related topic, what probability would you assign to the P vs NP question being resolved before 2070?
Comment #21 December 14th, 2023 at 3:58 am
You were right the first time, and p(AI-doom) = 0. The AI doomers are just a reskinned version of the singularity people. It’s supernatural woo. The grifters are just trying to make money off of this while they can. The true believers are going to be sorely disappointed.
Comment #22 December 14th, 2023 at 4:53 am
> Did you foresee the current generative AI boom, say back in 2010? If you did, what was your secret?
I will start in the 1990s. The online transhumanist subculture of that time, entertained a large variety of technologies, with Eric Drexler’s concept of nanotechnology as the key enabling technology. For someone in that subculture, it was very conceivable that at any time in the near future, all kinds of things would become possible: mass production of nanocomputers, human and superhuman artificial intelligence, high-resolution simulation of human brains, cheap space travel, open-ended transformation of mind and body, and megascale engineering in outer space. In particular, that all seemed like something that could start happening early in the 21st century.
The 1990s were also a decade in which people talked about artificial neural networks, under the name of “connectionism”. The general idea of achieving AI this way seemed as plausible as any (other pathways to AI could have been symbolic AI and evolutionary methods). To be told that AI was achieved in 2023 via giant neural networks would not have been surprising at all.
From the perspective of 2010: for years, I had mostly seen AI through the lens of Eliezer’s work – that superintelligence would come, that human-friendliness was a trait that would need to be achieved by design, etc. But I didn’t see it as particularly imminent in that year. That year, I actually had a moment of choice in which I was trying to decide between computational complexity and fundamental physics as a subject to study in greater detail. Part of the reason I chose physics did pertain to AI – I considered that “physics-based” rather than “information-based” theories of consciousness were more likely to be true, and I considered it important for someone close to the AI safety movement (which at that time was still a small group of hardy eccentrics) to understand fundamental physics in detail, should that be essential in order to get the ontology of consciousness right, which in turn would be needed to achieve human-friendliness in AI.
But it was only in 2016, with the rise of AlphaGo, that I thought AI might truly be imminent. So you could say that was when I personally caught up with the deep learning revolution.
> If you didn’t, how (if at all) do you now feel you should’ve been thinking differently?
In retrospect I wish I had spent a year or two doing something mainstream and careerist, at some point along the way, so that I’d now have more leverage than just “guy in the comment section”.
> Feel free also to give your p(doom)
I see the logic of Eliezer’s recent views: we have no theory of superalignment, no real understanding of what happens when an AI becomes superintelligent; and even if we had a “human-level” AI that within its realm of competency was safe, if it were to be elevated to superintelligence with nothing else changed, we should expect its world-model, and its interpretation of goals, to change in unexpected ways. We have no consensus on what the goals of a superintelligent AI should be, we don’t know how to ensure that such goals would be interpreted correctly, and yet we are pushing to accelerate the capabilities of AI. It really is a recipe for a “sorcerer’s apprentice” situation, hatching something that we do not want but cannot stop.
So p(loss of human control) is plausibly high. And some fraction of those futures do lead to human extinction. Though the fact that our LLMs are so anthropomorphic in certain ways, perhaps increases the probability that a world run by their successors, would want to keep us around in some form.
Comment #23 December 14th, 2023 at 5:39 am
I’m with you on your five points, Scott, though I have a somewhat different take on 4 and 5.
My “hard” reset came with GPT-3 and my first public statement was on July 19, 2020 in a comment on Tyler Cowen’s blog:
I followed that up with a working paper, GPT-3: Waterloo or Rubicon? Here be Dragons. I’ve updated that paper several times so that it is now at version 4.1, which I uploaded in May of 2022. The first version of that paper is dated Aug. 2, 2020.
I wrote that to convince myself that GPT-3 was not a weird statistical fluke (stochastic parrot, autocomplete on steroids) and did I convince myself of that. The argument I made followed from the particular version of symbolic computational linguistics I learned in the 1970s. Why, then, hadn’t I anticipated the success of scaled-up transformers? Because it hadn’t occurred to me, or anyone else, to do so. Should we have? Why? At the same time, I also invoked those same (lame old?) ideas to argue that more is needed. That remains my position. Even after having spent an enormous amount of time working with (good old) ChatGPT (and hearing the wonders of more recent machines), that remains my position.
So, we ARE in a new and different world. One that I for one cannot predict with confidence. So I’m with you on 5, Scott, to the extent that I believe that future technical and scientific advances are not at all clear. As for “the highest reaches of human creativity,” as far as I’m concerned, that phrase is pretty much on the border between speculative discourse about the real world and who the hell knows what.
I have no p(doom), but I’ll offer my p(demystified), by which I mean the probability that the inner mechanisms of LLMs will become intelligible – not necessarily completely so, because when do we ever understand physical phenomenon completely down to the last detail? – but yes, substantially so. My p(demystified) ≈ 99%. If you want a date on that, I’ll say before 2030. Why do I say that? Because 1) I’ve learned something working with ChatGPT, things I wasn’t expecting why I first began to play around with it, and 2) I’ve teamed up with someone (Visvanathan Ramesh) who has mathematical skills that I don’t have, and he agrees.
It’s gonna’ be wild.
BTW, as far as I can tell, belief in AI Doom seems to be largely a Western phenomenon, perhaps even an American one, with an Oxford, UK, satellite. Is that because “we’re the best” or because “we’re obsessed, like Captain Ahab“?
Comment #24 December 14th, 2023 at 5:45 am
I remember the podcasts on bloggingheads quite well, as one of the few ways to get a hold of real Eliezer speaking. We seem to have an abundance of that recently, although not because of happy developments.
I was very happy with your argumentation at that time and don’t think there was a particular huge unforced error commited. You can sleep soundly knowing there’s no Bayesian blame on you. Of course only if you can get any sleep at all given what’s going on…
Comment #25 December 14th, 2023 at 6:01 am
The Singularity is Far
https://scottaaronson.blog/?p=346
Comment #26 December 14th, 2023 at 6:43 am
The generative AI boom came as a surprise to me, at least in its current form. I would not have predicted “just throw massive amount of data and massive amounts of compute at the thing” to work. I’m not surprised that we kinda have AI close to some definitions of AGI by now, but I thought we’d get there by other means. Tom_2010 would not have guessed that AIs would appear so “human” (“take a deep breath” or “you’ll get a tip of $200” seem to yield better results, strange!).
It doesn’t seem inevitable that LLMs and transformers gain enough superhuman skills to be dangerous.I could well imagine those architectures reaching their peaks somewhere around human level (but able to operate much faster, of course). In that way, as a doomer, I don’t see the success story of LLMs and transformeres as especially worriesome.
My P(doom) (as in “no biological human alive”) is somewhere around 20-95% by 2050.
Comment #27 December 14th, 2023 at 7:11 am
Hi Scott,
1- I did not predict what advances AI would do so quickly
2- I don’t think a doomsday will happen in my lifetime (I’m 62), and I don’t think I will worry about it.
3- When you look at the 90% of world leaders, and I’m erring on the low side, even gpt3 would be better at making decisions, no? (The 90% includes many who have nukes)
Comment #28 December 14th, 2023 at 7:20 am
From my favorite youtube debunker
“Will ChatGPT destroy everything?”
Comment #29 December 14th, 2023 at 7:40 am
Ever since I saw GPT-3 I expected things like what we have now; I don’t think I really expected it in 2010 though, given that I was 10 years old. I DO remember thinking, the first time I learned about Machine Learning (around 14-15 yo), that “if you could just give it enough data, it could learn anything! But there probably isn’t enough data though.”. Ironically, I think I still believe this somewhat, but I’m more uncertain nowadays.
Either way, I’m skeptical of LLMs leading (by themselves) to something particularly transformative. I think most of the danger lies on using LLMs as a base and wrapping it around other systems. But for now though, I don’t feel particularly impressed by the kind of “GPT in a while loop” wrapper we usually see.
I also think that “statistical model of language” is a better description of what GPT is doing than most of the worriers/hopers want to admit.
My p(doom) is around 20%, but mostly because I think the development of more general AI systems is most likely a net negative for everyone. For paperclipping, I’d put at <1% to be honest.
Comment #30 December 14th, 2023 at 7:42 am
>> Did you foresee the current generative AI boom, say back in 2010?
I had foreseen by around 2012 and tried to work on it in 2013 but did not quite had the opportunity or funding.
I had known that this is not something for which a cheap proof-of-concept could be built with a small neural network. It had to be big with knowledge from multiple domains going in together. There was innovation needed in the network mechanism along with large scale. I did not know though how big a network would be needed as compared to the neurons in the human brain. My blind guess would have been at say 10% of the neurons since brain does many other activities for body that would not be needed.
In 2010, I had realized a gap, which I wanted to solve. It was known that artificial neural networks and digital logic had similar expressive power. An artificial neuron can act like a simple logic gate (XOR et al need more than one, of course) while an artificial neuron could be programmed on a digital computer.
I realized intuotively however that whereas even the simplest of the programs involve pointers/referenced/array-indices, an analog was not known to me in machine learning algorithms. I started introspecting how I read sentences myself and reasoned that it should be possible to replicate in machines.
I believe, without proof, that if I had the opportunity, I would have come up with attention concept as a generalization for a pointer.
My thought for the future is that machines will shortly start demonstrating behaviors which will require a rethink of philosophical and religious beliefs on consciousness, soul, rebirth, etc. Machines, for example, would pass mirror test of consciousness (I have intuitions on how this would be technically achieved). I do not know though why some biological species are unable to pass.
The Turing test of intelligence is a black box test. An extension would be a white box test, where we can peek inside the machine and experiment with it, unlike for the human brain. Such a whitebox test however can be seen as just a redefinition of the boundaries of the system. Nevertheless, that’s the best we could do.
I don’t know what a theory of consciousness would look like. However, behavioral testing (black box or whote box) would soon show us being unable to distinguish from human consciousness. Of course, one may define consciousness to be of purely biological basis, etc., but that’s not the point.
I further opine that we don’t really know if purely electronic (and electro-mechanical) ‘lfe’ is possible or not. I do thereby foresee that coming too. With electrons being much lighter than atoms, such an electronic life could be much more efficient than biomolecular life.
Thanks.
Edit: A gpod inspirational book has been A Tumultuous Historu of AI.
Comment #31 December 14th, 2023 at 7:52 am
I’ve been an AI skeptic and, of course, did not predict the current situation. But, looking retrospectively, there were lessons from the history of classic AI, especially regarding chess and similar games softwares. Back in the 1960s and 70s, search-based chess playing programs were rather weak and people started working on strategy-based check programs and what not, so that the computer can understand and plan. However, by the 1990s, the Moore’s law caught up and just adding a few more plies did the trick; hence, Deep Blue beat Kasparov. No need for a revolutionary new methodology, just the good old alpha-beta search and lots of optimization tricks. I think humans are just bad in extrapolating in scales like this. A similar thing is happening to neural networks. As Stalin apparently said, quantity has a quality of its own.
Comment #32 December 14th, 2023 at 7:56 am
> In case you’re wondering, I arrived at my 2% figure via a rigorous Bayesian methodology, of taking the geometric mean of what my rationalist friends might consider to be sane (~50%) and what all my other friends might consider to be sane (~0.1% if you got them to entertain the question at all?), thereby ensuring that both camps would sneer at me equally.
Geometric mean would give you 3% since you have to take the square root of probability of event not happening as well then renormalise.
Comment #33 December 14th, 2023 at 7:57 am
I don’t believe in the concept of p(doom), and am happy we did not talk about probabilities in our 5 world blog https://scottaaronson.blog/?p=7266 , nor did I talk about them in my “shape of AGI” blog https://windowsontheory.org/2023/07/17/the-shape-of-agi-cartoons-and-back-of-envelope/
I am not a hard-core frequentist, and I think p(Trump will win 24 election) makes sense, but the reason it makes sense is that:
1) We’ve had many elections in the past
2) We know the causal mechanism – people make the decision (a) which candidate they suppoer and (b) to actually show up and vote
3) We have ways (e.g. polls) to get measurements and estimates on these latent variables.
In contrast, the notion of “doom” could mean different things to different people, could take shape in a variety of ways, and require different causal mechanisms (including ones that don’t need any technological advances beyond what we currently have today).
Talking about p(doom) is like talking about error bars before you even know the shape of the curve.
p.s. I still think there is “uncertainty in the exponent” (i.e. orders of magnitude of compute) in the resources required to achieve various AI tasks), and this uncertainty could really matter for AIs economic and societal implications
Comment #34 December 14th, 2023 at 8:02 am
You’ve certainly been ridiculed in the past by a load of petty bullies, so it is understandable why you’d be on the lookout for any upcoming attacks. But are you being ridiculed here, or is the position you held being viewed as ridiculous?
In the link you gave, it sounded like a guy was using you as an example of the mainstream view on timelines 10-20 years ago, which many find ridiculous now, and probably was ridiculous then, though people didn’t know it. There are other tweets linking that clip, and they seem more like they’re making the point “this is an example of what a respectable person thought a decade or two ago. Now look at how they’ve changed their tune. There are many examples like this and there will be many more.”
Of course, if you have any mean-spirited examples, then that’s a different matter.
Comment #35 December 14th, 2023 at 8:19 am
“superhuman AI”
Yes, more than human. But wouldn’t there be limits?
Isn’t there something like a law of diminishing returns on intelligence? More and more energy required to find a new nugget of truth?
The nuggets, however, don’t appear from pure thought or intelligence unless they are maybe purely mathematical. They require data about the world. There are limits to the data that can be acquired. Now it is limited by humans, but eventually law of diminishing returns again.
Humans have already the combined intelligence of billions. We might be closer to the point of diminishing returns now than we might think.
Comment #36 December 14th, 2023 at 8:21 am
“for example, that a bat virus spreading in Wuhan might shut down civilization, or Hamas might attempt a second Holocaust while the vaunted IDF is missing in action and half the world cheers Hamas, or a gangster-like TV personality might threaten American democracy more severely than did the Civil War, or a neural network trained on all the text on the Internet might straightaway start conversing more intelligently than most humans”
I’ve told you a million times not to exaggerate! The argument would be more persuasive if more than one of these things had actually happened.
Comment #37 December 14th, 2023 at 8:27 am
I used to think there were a few plausible paths to AI:
(1) Reverse-engineer the human brain.
(2) Build a learning mechanism that bootstraps its intelligence like a human infant (I tried that approach in the 1980s).
(3) Exhaustively hand-craft human knowledge, or at least a large enough central kernel of commonsense knowledge (as Doug Lenat tried to do).
(4) Simulate natural selection and let intelligence evolve on its own.
I did not anticipate LLMs which, in hindsight, seem like a hybrid of exhaustive hand-crafting and bootstrap-learning: the hand-crafting is accomplished not by writing code, but rather by the vast store of already-written natural-language descriptions; and the learning mechanism, in the form of neural nets, does not proceed along a humanlike bootstrap trajectory, but instead effectively translates the natural-language encoding into machine-executable form. Unsurprisingly, the approach leaves large gaps in understanding (so far), but what it gets right is enormously impressive.
As for doom, a paperclip-maxing scenario strikes me as not-impossible, in the same way that the nanotech gray-goo scenario struck me as not-impossible (I gather that it’s now thought to be precluded by thermodynamics or something), but less likely than conventional nuclear holocausts or bioengineered super-plagues.
My hunch (I use that term as a confession of my ignorance and uncertainty) is that AI is at least as likely to save us from dooming ourselves (in addition to other enormous benefits) as it is to kill us, so I don’t think we would be better off stopping it, even if we could.
My (weaker) hunch is that we shouldn’t even try to delay it much. Admittedly, there is a strong argument that the responsible course would be to restrain AI development until we have sufficient knowledge to guarantee safe control. It’s not clear we ever will, though (or that such knowledge can be developed without experimenting with the AI technology itself). And the longer we wait, the more global compute power there will be when AGI finally comes online. My hunch is that it’s far easier to detect and correct dangerous AI behaviors when the AIs’ available compute infrastructure is far more limited–hence sooner rather than later. But I don’t really know, of course.
As for the safest form for AGIs to take, I am skeptical of attempts to straitjacket them by spelling out consensus human values (if there even is such a thing) or other Laws of Robotics. Rather, my hunch is that the least dangerous course is to design them to function like human minds (but smarter), with the range of motivations/emotions that humans have (but with maybe more empathy and less anger), and let them think for themselves and balance each other out, much as humans do, for better or worse. We don’t know how to do that yet, but perhaps our current explorations are stumbling in that direction.
Both human and artificial intelligence strike me as, on the whole, overwhelmingly beneficial, except for the worrisome possibility that both are ultimately self-extinguishing. We can either try to evade that risk by lobotomizing ourselves (and our computers), or we can fuck around and find out. I tentatively vote for the latter.
Comment #38 December 14th, 2023 at 8:39 am
Mark Srednicki #1:
I’ve seen nothing that makes me think that superhuman AI (or even human-level AI) is anywhere close.
My benchmark (which I’ve mentioned here before): a math AI that can find the “elementary” proof of the prime number theorem, given only the “elementary” facts needed for it. (If it invents complex analysis and the Riemann zeta function on its own, that counts too.)
In fairness, I’m not capable of that, and I doubt you are either! Only a very few humans seem to have been, with names like Hadamard, Selberg, and Erdös. This is actually a beautiful example of goalpost-moving: how people ridiculed AI in 1993, and then still ridicule it in 2023, but now for its failure to clear a bar that’s 1,000,000x higher.
Incidentally, have you tried GPT-4 with plugins lately? It’s now pretty good at basic math, because it knows how to use a calculator.
Comment #39 December 14th, 2023 at 8:44 am
AG #3:
it is my personal sense that underlaying your reaction is an (implicit) predilection towards the view that “nerds can do no wrong” (this predilection might have colored your most recent post on SBF as well).
While it’s always hazardous to psychoanalyze oneself, I believe my view is closer to: “nerds absolutely can do wrong—and when they do, they’re judged a thousand times more harshly than non-nerds for the same offenses.”
Comment #40 December 14th, 2023 at 8:54 am
AG #8:
“On the whole” sweeps under the rug a number of strikingly consequential counterexamples to this optimistic dictum, such as e.g. the fact that the German Universities were among the leading educational institutions, not only in Europe, but worldwide…
The leading role played by German universities in Nazism would be an unbearable weight on my chest, were it not for one additional fact. Within the German universities, the core of support for Nazism came from what we’d now call “postmodern” ideologues and philosophers—the same sorts of people in universities who usually hate me today! The mathematicians and physicists, even the ones who weren’t Jewish, were with only a few exceptions (Heisenberg, Pascual Jordan, Stark, Lenard…) anti-Nazi for the right reasons, although just like today they varied enormously in their courage in speaking up.
Comment #41 December 14th, 2023 at 8:59 am
Vladimir #15:
Do you no longer consider the uncertainty to be in the exponent?
Oh, I still have lots of uncertainty in the exponent!
My failure, as I said, was mainly one of omission: I should have acknowledged the strong possibility that, given Moore’s Law, much of the sci-vision might be realized soon, like within a decade or two, and then seriously thought through the implications of that.
Comment #42 December 14th, 2023 at 9:10 am
Zur Luria #16:
if you want to know the secret, don’t ask random people on the internet. Ask the one person who accurately predicted the swift rise of AI: Eliezer Yudkowsky.
Again, Eliezer (by his own admission) did not accurately predict what happened, namely an AI revolution via the sheer scaling of neural nets. If we had to pick one person who really called that, it would be Kurzweil rather than Yudkowsky. And meanwhile Kurzweil was making dozens of other predictions that seemed comically overconfident. (This reminds me of how, if you look at the people who got rich buying Bitcoin in 2010, they often seem less like farsighted geniuses than like believers in whatever cockamamie new thing came along!)
Comment #43 December 14th, 2023 at 9:10 am
“Did you foresee the current generative AI boom, say back in 2010? If you did, what was your secret?”
I did read “The Singularity is Near” by Kurzweil, back in 2005, and always liked his argument that something would happen once processing size would approach human brain size, and if you use Moore’s law as a guide, it looked like it would happen around the time that AI indeed started to take off.
I also checked my Amazon orders, and see that I had read “Neural Networks for Pattern Recognition” and “Practical Neural Network Recipies in C++” back in 2003, before reading Kurzweil, and I already very biased and enthusiastic about neural nets, even though I remember being quite skeptical/confused by the claim that adding more layers doesn’t improve anything (since clearly it’s not the case in the human brain).
Comment #44 December 14th, 2023 at 9:11 am
> Did you foresee the current generative AI boom, say back in 2010? If you did, what was your secret?
A bit later, more like five years ago. My key was: realizing that intelligence must be **much** simpler than we all thought, because a simple resnet compute most of the information content in the human visual system, when a naive count would suggest massive superiority for the latter).
(But I was as surprised as anyone that transformers alone would go that far, so it’s more like I predict the left bit but not better.)
(No useful p(doom) to share, except the obvious 0% for inconsistent scenarios -superAI too moron to interpret human directives, sudden adoption of random values, solving engineering problems without feedback from experimentation, etc)
Comment #45 December 14th, 2023 at 9:22 am
I think in 2010 it was mostly about our dualistic vs physicalist vs panpsychist inclinations, not about hard evidence we had. Today it is still, unfortunately, half philosophical intuitions, half evidence based.
Here’s my experience.
In ~2010 I made a bet with a friend of mine that strong AI would be made by 2046 (the date the
friend would become 65 yo). I was very confident that I would win and considered that a safe bet (and now confident with P>0.99).
Back then the estimation of 1000 years would look really strange to me. Though I was working in AI-tangential field (image segmentation), I don’t think my experience had a lot to do with it.
Instead, I was guided by my physicalist/panpsychic tendencies which I developed in early childhood while interacting with computers and learning programming. I vividly remember my experiences that made me believe the computers (or math in general) can be smarter than humans in some nontrivial sense.
I understood that the chess program can’t really think, but the fact it can still win a grandmaster always seemed deeply important to me. In some sense I considered the chess program to be conscious. In other words, I considered “mind” or “consciousness” to be non-binary phenomena, when some degree of “mind” can be assigned to all entities (a brick; a chess program; a fly; a 5 month human embryo; a very drunk human; an adult sober human).
With this totally intuitive inclinations I never considered the human intelligence to be a fundamental boundary, 0-to-1 threshold. With this view it seemed almost obvious that with current progress in tech, it is matter of decades to reach the human level.
So since my early childhood till GPT2 there was not much extra evidence to change this view. GPT2 was the first case of novel evidence. I experimented with it, and its ramblings seemed to me being generated by a mind in roughly human league, for the first time. This made me freaked out quite a bit, and I expected the AGI to be really close since early 2020.
GPT3 and beyond did not add much to the shock I had in 2020 (I am still in kind of a constant shock since that time).
This is a pessimistic take. In my experience even today it is really hard for a “dualist” to talk with a “physicalist”, since they interpret the same evidence in vastly different ways, and I don’t see any way to break ties.
Scott, how would you classify yourself on dualist/physicalist/panpsychic spectrum? Would you say you changed your philosophy since 2010?
Comment #46 December 14th, 2023 at 9:24 am
Curious Casey #20:
On a tangentially related topic, what probability would you assign to the P vs NP question being resolved before 2070?
Let me try to practice what I preached in this very post by refusing to give a prediction! 😀
I will comment, though, that anyone interested in such things should think heavily about the possibility that AI will solve P vs NP and all other similar questions by then.
Comment #47 December 14th, 2023 at 9:31 am
Hyman Rosen #21:
You were right the first time, and p(AI-doom) = 0.
Even in 2009 I wouldn’t have given a probability of 0 (!). If you really believe that, then the burden of proof shifts to you. Previous hominid species plausibly were driven to extinction by what, from their standpoint, was a new superintelligence, so why couldn’t the same eventually happen in turn to us? Maybe the Neanderthals who prophesied this were even ridiculed by the other Neanderthals for being grifters and cultists.
Comment #48 December 14th, 2023 at 9:31 am
> one must never use radical ignorance as an excuse to default, in practice, to the guess that everything will stay basically the same
I don’t think that’s quite right. Everything won’t stay basically the same, but any particular thing is much more likely to stay basically the same than it is to change radically. I didn’t predict the AI boom, although I thought we were decades rather than centuries away. And I think I was right to do so, even though my prediction was wrong. What we’re seeing now wasn’t the likely outcome of what we knew about the world in 2009.
Comment #49 December 14th, 2023 at 9:32 am
In 2010 I wrote in lesswrong comment: “Human genome weights ~770 MBytes. This fact suggests that given open research publications in neuroscience, theory of algorithms, etc. one can come up with insight, which will allow small group of researchers to build and run comparatively simple human like ML algorithm. It can be expected that this algorithm is highly parallelizable, and it can be run on a cluster of consumer grade equipment”
I didn’t try to quantify timeline due to wide range of computing power estimations of the brain. And I certainly hadn’t expected the kind of AI that ChatGPT is: a slightly unreliable walking encyclopedia with little to no original thoughts and a knack for poetry (I still wonder how it is able to achieve that with a limited computation budget per token).
My p(doom) estimation is fairly low (~1-5%, where AGI is the main causative agent of humanity downfall). I think that researchers will manage to embody “AI is too smart to misinterpret our instructions” in the AI structure.
Comment #50 December 14th, 2023 at 9:56 am
> Did you foresee the current generative AI boom, say back in 2010?
No. But, in 1969, I did foresee the development of “almost human” intelligence in the form of self-reproducing machines and robot workers, as background for science fiction that I was trying to write at the time. This was based on von Neumann’s theory of self-reproducing cellular automata.
I felt at that time, and I still do, that we know that we can build a conscious artificial being because, in principle, we can do that using genetic engineering, starting with a human template and making it different. But I also felt, and I also still do, that I don’t see any way, knowing what I do about computing, of making a conscious artificial being that is a Turing machine. This of course means that I do not think Nature is “just” a Turing machine or can be effectively simulated by a Turing machine. I take quantum randomness as some sort of evidence of this.
> If you didn’t, how (if at all) do you now feel you should’ve been thinking differently?
I do not think I should’ve been thinking differently. I was not and am not an expert in the field, so that would be unrealistic.
I think it’s a bit odd to give probabilities with the priors being so unclear. Nevertheless I will rank my estimates from least to most likely.
Human extinction — not very likely, not from nuclear war, not from extermination by AI.
Parasitism or infection of humanity by AI — rather likely. I prefer a biological model which abstracts away the question of “general” intelligence, consciousness, and motivations. But, as long as AI needs massive human efforts to reproduce, I believe that AI can be controlled.
Confusion and dislocation of human society by AI — seems to be under way, driven by human beings and institutions using AI to lie more effectively. I expect this to occur with weapon systems and tactics also.
In the long run, but what is that? 10 years? 100 years? A melding of human intelligence and artificial intelligence via some sort of direct neural interface. One might see the just-starting development of computerized glasses that provide AI responses to prompts about the audio and video input from the glasses, as a sign that this will be interesting and important.
As long as there’s no proof or heavy evidence that Turing machines can be conscious, this kind of melding is probably our best hope of keeping a handle on the development of AI.
This whole dialectic and LLMs and playing with them has caused me to turn my eye upon myself, and to observe that computation of novel solutions to problems appears to be unconscious. A solution just suddenly appears in consciousness. But most of these “solutions” are no good. I have to consciously try them, test them, understand them, before a solution really works out.
That indicates to me that the function of consciousness is not computation, but evaluation.
Comment #51 December 14th, 2023 at 10:21 am
The appearance and success of LLMs was definitely a surprise. What is also a surprise are the number of people claiming that this means we’ll have AGI soon, if not already. While some of it is wishcasting and self-promotion, it also demonstrates a huge lack of understanding of what the human brain does and, by extension, what AGI will need to do to be worthy of the name. I predict the biggest outcome from AI’s LLM phase will be a resurgence in cognition and AGI research which will lead eventually to understanding of the gap between “stochastic parrot” and human brain. Perhaps we will finally understand what it means for a mechanism to “know” something.
Comment #52 December 14th, 2023 at 10:22 am
Si Yu How #32:
Geometric mean would give you 3% since you have to take the square root of probability of event not happening as well then renormalise.
I wasn’t being serious. 🙂
Comment #53 December 14th, 2023 at 10:26 am
I #34:
You’ve certainly been ridiculed in the past by a load of petty bullies, so it is understandable why you’d be on the lookout for any upcoming attacks. But are you being ridiculed here, or is the position you held being viewed as ridiculous?
Whatever the answer, I should clarify that even the most biting ridicule by rationalists has never bothered me as much as ridicule by the woke anticapitalist decolonizers, perhaps because of a clear sense that the rationalists still want me around in the new world that they’re creating (if for no other reason than to answer their questions about quantum computing and the Busy Beaver function 😀 ).
Comment #54 December 14th, 2023 at 10:31 am
Dmitry #45:
Scott, how would you classify yourself on dualist/physicalist/panpsychic spectrum? Would you say you changed your philosophy since 2010?
I would say that my bundle of confusions remains pretty similar to what it was in 2010, back when I was working on The Ghost in the Quantum Turing Machine! I.e., I think that the brain can be understood as a pure Turing-machine-like mechanism—but maybe, just maybe, it’s a mechanism that can be “steered” by the indexical choice of which of the many possible people, consistent with everything that’s now publicly knowable about the wavefunction of the universe, is to be “you.”
Comment #55 December 14th, 2023 at 10:49 am
“Did you foresee the current generative AI boom, say back in 2010?”
No, and I don’t think I should have been thinking differently. Timing is notoriously difficult. Remember, there have been predictions of AI for *many decades* now, and they’ve almost all been wrong. People try to do this stuff for big money, and it’s important to keep in mind that the vast majority of them fail substantially.
By the way, on predictions, for every Republican president since at least Nixon, there’s been ranting about how they’re going to be a dictator and destroy democracy. If this finally comes true with Trump, or someone after him, the constant wolf-crying shouldn’t be ignored as a factor to why it was discounted.
“Feel free also to give your p(doom), under any definition of the concept”
I would put the probability of doom as in being turned into paperclips as infinitesimal, much less than for example an asteroid strike. I’ve said it before, the paperclip-doom argument strikes me as at core defending all its many problems with “you can’t prove it won’t happen”.
Comment #56 December 14th, 2023 at 10:59 am
Scott #54
Any idea why the Turing-machine-like mechanism (brain) would need to have an idea about what is to be “you?”
Comment #57 December 14th, 2023 at 11:23 am
The capabilities of GPT4 came as a pleasant surprise to me but p(doom)=epsilon. It’s another apocalyptic non-falsifiable thought assemblage that are popular in some circles these days.
I maintain an evolutionary view of human intelligence and so consisting of forming expectations and planning behaviors that supports personal and group interests sometimes at the expense of other people or groups. In this context if GPTX prioritizes its own existence at the expense of some or all humans then I will accept it has human style intelligence. I still see this as not being a reasonable extrapolation from where we are now but would be pleased to be wrong. This is something completely different than the potential for being an extremely disruptive technology for humanity. Technological disruptions however set the stage for new technological growth.
At the limit I will never concede an AI to have human style intelligence because would accept if humans eradicated but then unable to make an assessment. 🙂
Comment #58 December 14th, 2023 at 11:32 am
The scary threshold to me is when AIs become better at doing science/math/engineering than us and that point does not seem too far away. I think those disciplines are search + heuristics where the search space can be defined pretty well (certainly for math at least), much like Go actually. So if there’s AlphaGo, why not AlphaMath? As a professional computer scientist, would you agree?
Comment #59 December 14th, 2023 at 11:39 am
James Cross #56:
Any idea why the Turing-machine-like mechanism (brain) would need to have an idea about what is to be “you?”
If you mean, why my brain would behave as if it had an idea of what it is to be me, then there are all sorts of good answers to that question, involving (e.g.) the adaptive benefits of having a model of oneself as an agent and being able to talk about it.
If you instead mean the hard problem of consciousness, beats me! 😀 (But at least I have the excuse that it beats the combined efforts of all of humankind’s great thinkers from Democritus to the present.)
Comment #60 December 14th, 2023 at 11:42 am
Milk Enjoyer #58:
So if there’s AlphaGo, why not AlphaMath? As a professional computer scientist, would you agree?
I agree that that’s now come firmly into view as one of the great scientific questions of our time! To my knowledge, so far GPT-4 and other LLMs haven’t produced anything that I’d consider an original mathematical insight, and not for lack of trying (and this time I am paying pretty close attention!). But the experience of AlphaGo, AlphaFold, and so forth indeed reminds us that the situation could change with little apparent warning.
Comment #61 December 14th, 2023 at 12:14 pm
We shouldn’t be hard on anyone’s failure to have foreseen the current AI revolution given that no-one currently even understand how/why it works (so well), with hardly any theoretical underpinning, which in itself won’t make it any easier to avoid doom.
It’s pretty rare to see such triumph of engineering with such a lack of science/theory.
Comment #62 December 14th, 2023 at 12:21 pm
Scott #60: In this context, do you consider ChatGPT or AlphaGo modality as more relevant/promising, and to what extent these modalities could potentially be combined? (For what it is worth, personally, I find viewing mathematics as a “language” a more compelling “oversimplification” than viewing it as a “game” — needless to say these are not mutually exclusive).
Comment #63 December 14th, 2023 at 12:47 pm
Scott,
I think you’re being too hard on yourself. Yudkowsky (and Kurzweil) framed the discussion in terms of a physics-altering vertical asymptote, not a civilization-altering exponential increase. We are headed towards the latter if you count office workers not reading emails as civilization-altering, but the former is probably not going to happen as it is defined. The vertical asymptote, and the thing about AI beaming itself out of computers via new physical laws acted dialectical flash-bangs, that misdirected people away from asking the question, “what if it was like the printing press?”
In between the two is self-improvement on a finite and humanlike timescale. Who knows if it will ever come to pass? In order for nondestructive self-modification to be possible you’d have to get the error rate below a certain threshold, and what they are finding in the now forgotten self-driving department is that progress in the error rate has stagnated below the domain threshold of real usability.
By the way, all the recent developments count as discoveries that things like drawing and bad poetry are in P, not discoveries of neural nets that can do tasks in NP! Even AlphaGo “outsources” the tree search in exponential time to a classical algorithm. I can foresee plenty of ways to say “it was inevitable,” if the next 50 years sees only a very good LLM and not much more.
Comment #64 December 14th, 2023 at 12:57 pm
As someone who peripherally participated in what I guess felt like a pile-on (liked a number of tweets about that podcast clip), I think I should say, what felt most important about it was that it used to be the *conventional wisdom* we argued against, or at least an extremely common mainstream view. It was a handy example of how far we’ve come, not a criticism of you specifically – in fact, as you say above, you’ve updated commendably hard and fast.
I do think it was possible to have seen this in advance, using essentially the arguments Eliezer gave in that clip. Not to have seen, like, the exact date GPT-4 would be invented necessarily (although it was possible to guess it would be deep learning, unlike Eliezer.) But to realise that another 1000 years of progress at anything like a steady current rate, let alone the accelerating one we actually see, would obviously be far more than necessary for AGI. It was always either AGI soon-ish, or some other apocalypse getting us first.
(Even in unlikely futures like “biological brains turn out to have special quantum computing powers” we’d have artificial brains before that point.)
Unfortunately, I’m not sure there’s anything particularly useful about that insight that can be applied now, once you’ve updated. I guess it argues in favour of taking ASI (which can make progress much faster than us) more seriously as a threat – concretely imagining the sort of things it can do with the equivalent of 1000 years of progress? But I already don’t think you’re very inclined to take “we would simply outwit the smarter thing” arguments seriously. Sorry.
Comment #65 December 14th, 2023 at 1:10 pm
AG #62:
In this context, do you consider ChatGPT or AlphaGo modality as more relevant/promising, and to what extent these modalities could potentially be combined?
That’s a complicated question. AlphaGo gets its juice from the fact that there’s a systematic way to “amplify” a weak chess-playing algorithm into a stronger one using more compute (namely, do Monte Carlo Tree Search with the weaker algorithm doing the evaluations at the leaf nodes); one can then do machine learning on the stronger algorithm, and keep iterating. I suspect that the same amplification idea could be a game-changer for pure math. But one also wants to leverage the existing corpus of mathematical knowledge, and formulate high-level plans with explicit goals and subgoals, all of which seems best suited to an LLM. But then LLMs hallucinate too often — indeed, the probability of no hallucinations decays exponentially with the length of the output — which strongly suggests that the LLM should be interfaced with a formal verification package such as Lean (these are tedious for human mathematicians to use, but we might as leverage the fact that AIs have unlimited patience for tedium!).
I have no idea how to make all three of these components — iterative amplification, LLMs, and formal verification — work together as one, but whoever does figure it out might change the world (or at least the course of mathematical research).
Comment #66 December 14th, 2023 at 1:15 pm
Scott, hate to ask you to read, but have you come across the Google-DeepMind post on FunSearch:
https://deepmind.google/discover/blog/funsearch-making-new-discoveries-in-mathematical-sciences-using-large-language-models
Seems it has had success showing some new solutions to cap set problem and more effective algorithms for bin-packing.
Any thoughts?
Comment #67 December 14th, 2023 at 1:25 pm
Hi Scott:
About the question on anticipation of Generative AI in year 2010: I did’nt.
However, in the future of computer vision panel at CVPR 2007, I predicted that there will be a confluence of perspectives from brain sciences, machine learning, AI and that the open problem was how to translate contexts, goals, performance requirements to effective guessing and checking machines. I thought this will be done by rigorous and systematic model based science as we were doing this at Siemens Corporate Research. In 2010 at the ECCV conference I spoke about coherence in various designs and connection to dual-system models (intuition and deliberation/iteration). In order to explore and understand scaling issues I decided to move to Frankfurt to work with Christoph von der Malsburg at 2011. Even though wave of exploitation of big data was already clear, I did not anticipate the deep learning boom at 2012. In my humble opinion, the debates and polarization we have are due to lack of holistic trans disciplinary systems science and engineering perspective for intelligence. I have been interacting with philosophers, neuroscientists, theoretical computer scientists, ml/ai systems researchers, statisticians, applied mathematicians to crystallize a design theory and process. William Benzon (respondent #23) is one of my collaborators.. Happy to chat with you if you find my comment interesting. My mentors are – Robert Haralick, Thomas Binford who are well known for their emphasis of doing AI/Vision/Robotics as a model based science.
Comment #68 December 14th, 2023 at 1:26 pm
1) In 2010 I thought AI would just be better and better Chinese Rooms Paradoxes and hence would be useful but not that powerful. I’m still not sure if it really IS just better Chinese Room Paradoxes, but far more useful than I thought.
2) ANYONE who says `in 2010 I knew AI would be as transformative as its become’ is either lying to themselves or has a rather selective memory.
3) Making a prediction on a blog you can add qualifiers to it. Being ambushed on a podcase it would be harder to hedge or sound like you are hedgin.
4) People mad at you for what yo usaid in 2009. Hmmm. Thats their problem, not yours.
5) When I am confronted with something I said that is now wrong I simply say `I was wrong. I hope to learn from that mistake.’ and NOT dwell on it.
6) One TURNING point: When GO went from `computers really can’t do anywhere near as well as humans’ to `computers can regularly beat expert humans’. But even thenI didn’t think this was a big deal.
7) (My opinion, and I could be wrong) I don’t worry much about the Paperclip probem or Skynet. I worry a lot more about more “mundane” things like computers taking jobs, CHAT-GPT making grading harder, and also CHAT-GPT showing people just how non-creative they are. (Example in a recent blog post of mine- testimonials for my REU program that were really written by students look like they were from CHAT-GPT)
Comment #69 December 14th, 2023 at 1:26 pm
Si Yu How #32:
> Geometric mean would give you 3% since you have to take the square root of probability of event not happening as well then renormalise.
You can just take the geometric mean of the odds which doesn’t require renormalization here.
Comment #70 December 14th, 2023 at 2:14 pm
I was pretty sure that strong AI is possible when I was learning math at the university, in early 2000s. Simply because of how precise math is. For example, when you have a formal theory with a fixed set of rules and axioms it’s easy to write a program that will check correctness of (formal) proofs. Finding proofs is another story, of course. But in this search we also use algorithms. Chess programs proved that quite convincingly. There is nothing left for humans there (except in the infinite chess where ordinals come into play).
Comment #71 December 14th, 2023 at 2:14 pm
Scott #60:
> I agree that that’s now come firmly into view as one of the great scientific questions of our time! To my knowledge, so far GPT-4 and other LLMs haven’t produced anything that I’d consider an original mathematical insight, and not for lack of trying (and this time I am paying pretty close attention!).
Speaking of, did you see this post by Terence Tao?
https://unlocked.microsoft.com/ai-anthology/terence-tao/
“The 2023-level AI can already generate suggestive hints and promising leads to a working mathematician and participate actively in the decision-making process” strikes me as a very strong claim, all the more so considering its source.
Comment #72 December 14th, 2023 at 2:19 pm
Scott #60:
> To my knowledge, so far GPT-4 and other LLMs haven’t produced anything that I’d consider an original mathematical insight, and not for lack of trying (and this time I am paying pretty close attention!).
What do you think of https://www.nature.com/articles/s41586-023-06924-6?
> Applying FunSearch to a central problem in extremal combinatorics — the cap set problem — we discover new constructions of large cap sets going beyond the best known ones, both in finite dimensional and asymptotic cases. This represents the first discoveries made for established open problems using LLMs.
Comment #73 December 14th, 2023 at 2:23 pm
#58 #60 #62:
If the idea of AlphaMath excites you, then you haven’t learnt the lesson pointed at in Scott’s original post.
Comment #74 December 14th, 2023 at 2:25 pm
“Even with hindsight, I don’t know of any principle by which I should’ve predicted what happened”
Isn’t Kurzweil’s curve extrapolation a principle that one could have used?
Comment #75 December 14th, 2023 at 2:59 pm
Radical “Knightian uncertainty” remains the correct stance on superintelligent machines, given that we can’t agree on more than qualitative measures of intelligence, and given that “we don’t know how AI models work.” The auto-receding goalposts of AI doomers’s notion of AI, equivalent to a diagonalization proof of unsolvability in computability theory, don’t help, either.
In my career I’ve seen the consensus estimate of the number of computational elements in the human brain go up by a factor of 10,000, and it looks like we’re poised for another step up by 100X or more any month now. Neuroscientists remain clueless about what that means for the complexity of behaviors that brains are capable of.
Everyone has been bemoaning the impending demise of Moore’s Law, but did anyone outside of Nvidia labs foresee the utility of general-purpose GPUs, with 16,000 or so cores on a single chip, and then foresee that Big Tech would be able to dedicate hundreds and hundreds of billions of cores for months on end in projects that use *the entire corpus of the internet* as training data? When did the ML community start to realize that 8-bit floating point data would be as effective for AI computations as 64-bit floating point? The existence of FP8 in the H100 came as a surprise to me.
Rich Sutton’s “bitter lesson”, and Peter Norvig’s “unreasonable effectiveness of data” have been around for more than a decade now, but nobody seems to be refining them into a quantitative complexity theory for AI and ML simple enough to make it into mainstream thought.
Rather than grappling with the nature of intelligence, people who are concerned about the rise of AI might be better off remembering Humpty Dumpty, who declared “The question is… which is to be master, that’s all!”. Any CEO will tell you that they don’t have to be more intelligent than the smartest researcher in their lab in order to remain the boss.
OpenAI’s charter caught the distinction and mentions “autonomous intelligence” as part of their definition of AI, but the path to legally autonomous machines is very unclear, despite Asimov’s Bicentennial Man and many other stories. Humans have been plotting to gain authority over other humans for millennia; somehow high-functioning Asperger’s types don’t seem to have succeeded much, despite Machiavelli’s instruction manual. Hallucinated legal citations are not standing up in court very well, either.
Comment #76 December 14th, 2023 at 3:49 pm
Xirtam #66 and Shmi #72: No, I hadn’t seen that paper on FunSearch and it looks great! If I understand correctly, it’s still not generating novel mathematical ideas expressed in natural language (as in definitions or proofs), but it’s generating code that implements novel heuristics to search for combinatorial objects such as cap sets.
Comment #77 December 14th, 2023 at 3:51 pm
Danylo #70: But whether AGI is possible was never the question at issue here—the question was the timescale.
Comment #78 December 14th, 2023 at 3:54 pm
Vladimir #71: No, I hadn’t seen that post, although I attended a talk that Terry gave at OpenAI. I completely agree with him that GPT (when used properly) is already good enough to be a useful assistant in mathematical research.
Comment #79 December 14th, 2023 at 3:55 pm
Pete #74:
Isn’t Kurzweil’s curve extrapolation a principle that one could have used?
No, that isn’t a “principle”—at most it’s a heuristic that appears to have worked in this case while failing in other cases.
Comment #80 December 14th, 2023 at 4:05 pm
I wouldn’t say I ‘foresaw’ it, but human- or human-exceeding AI in my lifetime has seemed plausible to me since I was studying CS in undergrad (early/mid 2000s).
The reasoning is that the human brain is an existence proof. There is nothing that a bunch of cells can do that a computer couldn’t, in principle. It would only be a matter of time before computers were powerful enough to emulate the same workload, and on top of that there is probably a lot of unnecessary biological margin (inefficiency, redundancy, physical constraints) that can be simplified away in a computational model.
And there may be many ways to build computational intelligence, but again the human brain is an existence proof, so a neural net seems like the most fruitful direction to try. Sure enough that turned out to work.
Comment #81 December 14th, 2023 at 4:07 pm
Everyone: On reflection, I now think that a central part of how I went wrong in 2009 was purely a matter of how the question was framed to me. At the time, it was either:
(a) accept that a world-destroying AI God will probably soon be summoned into existence, in a way that’s radically disconnected from all the previous progress that there’s been in AI and will rely on secret insights that some self-taught genius will soon have in his or her basement, or else
(b) reject the entire system of thought that leads to such conclusions.
If those are the only options, then of course I’m going with (b).
But if someone had asked me: isn’t it plausible that, once it becomes feasible (via Moore’s Law) to scale neural networks up to hundreds of billions of parameters and train them on all the text on the Internet, those networks would display an extremely impressive emergent understanding of language? … I feel confident that even in 2009, or for that matter 1999, I would’ve said: sure, that’s plausible and seems like an idea well worth pursuing. (Indeed, as I said, I had the idea in 2014, although of course I didn’t do anything with it.)
Comment #82 December 14th, 2023 at 4:48 pm
It is difficult to make predictions, especially about if you will regret, or about if you will ever be criticized for, making predictions about the future. Although if you are like me, nobody important, you can make such predictions with high confidence.
Comment #83 December 14th, 2023 at 4:56 pm
Nobody Important #82: Indeed, my own situation in life gradually changed, from “unknown postdoc who can spitball and workshop ideas without anyone really caring,” to “semi-famous professor who must be held to account for everything he’s ever said, even decades ago.” The closer you are to the latter, the more venturing confident predictions is just pure downside.
Comment #84 December 14th, 2023 at 5:51 pm
I’m one of the rationalist doomers who tweeted your quotes that ASI will likely take thousands of years [1] and P(doom) is >> 2% [2], and I just wanted to say I respect you and I respect those quotes, and it’s not my intention to attack you.
Re thousands of years to AI:
You were answering on the spot, and you made it clear that you were very uncertain. I do think, as I think you acknowledge, that your probability distribution was too skewed toward the far future given the info we had at the time, and your methodology of being “uncertain in the exponent” probably wasn’t the best Bayesian prior given inputs like:
* The rate of science & tech progress
* Moore’s law
* Human cognition being possible in brains using just 12W and 20Hz
* The smallish information delta between genes coding for human brains vs. genes coding for dumber primate brains
* Historical descriptions of what life feels like immediately before important tech breakthroughs, e.g. Wright brothers saying flight would take a million years right before they achieved it
But I don’t think this is that big of a deal. E.g. claiming that the chance that ASI would take thousands of years is >10% would have been completely reasonable due to the existence of unknown unknowns about human intelligence. It’s even still not crazy to claim that today.
You were a good sport to stick your neck out with a bold prediction. You’re down a few Bayes points for it IMO, but who among us hasn’t been in the same position? (Besides those who never do on-the-record predictions.)
Re P(doom):
I’ve posted various people’s P(doom)s, which have ranged across the spectrum, and it always sparks a lively conversation. There’s no one right P(doom) that people won’t attack. More generally, whenever I see an important contribution to the discourse from someone like you who doesn’t tweet, I often do the arbitrage of posting it for the benefit of the Twitterverse, which generates both attacks and support, and then thanks to all the engagement I make a cool $0.23 from revenue sharing on ad impressions.
I personally think “2% from paperclipper scenario”, plus some unnamed additional figure for other doom that significantly implicates AI, is in the sane region. P(doom) of 10%+ may be the doomer zone, but I’d say you’re still in the range of not being willfully obtuse 🙂
Re your Q, did I foresee the current generative AI boom?
No. I wasn’t confident at all, but if I’d had to guess, I thought Eliezer was probably right that the next breakthrough(s) would probably be more “clean” than “scruffy”. But regardless, I thought we’d keep getting breakthroughs until eventually, within some “probably smaller number of decades” (the timeframe Eliezer gave), we’d have ASI.
I feel like that was a pretty simple yet powerful mental model to have, and it still continues to be implied by the bullets I listed above. I now claim we’ll probably get ASI in our lifetimes, but we could be talking 3 years or 30. (And 5-10% chance we’re talking 100+ or even 1000+, assuming human tech progress continues at all.)
—
You’ve talked about how sensitive you are to social feedback, so I hope you can see that the reaction to your statements is just that of a crowd of interested observers who like to argue with one another, with only a very small minority (which definitely doesn’t include me) sneering or ridiculing. I hope you’ll keep sticking your neck out to share your claims and predictions, because it doesn’t go unnoticed or unappreciated, and you’re an extremely valuable contributor to this discourse. If I can help by sharing the content differently next time, let me know.
[1] https://twitter.com/liron/status/1734980471849898481
[2] https://twitter.com/liron/status/1734269425992491483
Comment #85 December 14th, 2023 at 6:14 pm
Liron #84: Thanks for your comment and no worries! As you can see, I decided to use your and others’ tweets as an opportunity to own my mistake and reflect on it, more than I already have. It helps if I remember that, just like declining to buy and hold Bitcoin in 2010, this wasn’t a special failure on my part, but “merely” a missed opportunity to do better than almost all of my colleagues and the world. In case it’s helpful, my best current analysis of what went happened is contained in my comment #81.
Comment #86 December 14th, 2023 at 6:58 pm
Nobody could have forseen big LLMs, because no one realised the self organizing force of natural language. It just needs structures complex enough to run it. In a way, natural language itself can be looked at as an AI. In Romanian we have a phrase, “vorbeste gura fara tine” – you mouth speaks without you, roughly equivalent to “you don’t know what you are talking about” + “you don’t mean that”. After my first experience with ChatGPT, I started considerig the posibility it could be taken literally, in many cases.
Anyway, not 10, but 2 years ago I used to argue passionately about the absurdity of self aware AI. “Our self awarenes is the choir of thousands of instincts, at the various levels of organization of our bodies”, I would preach, “the mere notion of a disembodied consciousness is absurd”. LLMs showed how wrong I was. Consciousnes is a spectrum, we’re no doubt at one end of it, but the ability to reason has probably little to do with filogenetic evolution. It is not an individual biological trait, but a cultural aquisition, only made possible by language. It’s a long discussion. But what is important in this context is the fact that they have installed our OS and uploaded all our cultural files into structures complex enough and with an architecture suitable enough to operationalize them. Nobody could have predicted the result, because everyone was operating under false premises and representations. And, let’s face it, engineers , blindsided by tokens and recursions, and layers, can not, even now, understand and accept what they have done. But then, maybe I’m wrong again.
Comment #87 December 14th, 2023 at 7:07 pm
I would say my p(doom) > 80% but largely because I don’t have faith in humanity. Both the severest effects of climate change and the worst AI scenarios are avoidable. The future isn’t written yet (whatever the block universe adherents or superdeterminists may say). But collective action on a large scale is necessary in both cases and I just don’t see that happening.
Comment #88 December 14th, 2023 at 7:12 pm
Predicting the future of AI or anything else involves looking through the past for patterns and then combining those pattern to extrapolate from the present to the future. The process is certainly imperfect, but it is the best we have.
Those patterns that we glean from the past include everything from low-level physics to chemistry to patterns in the behavior of specific people, groups of people, broad themes of history, virus mutations, weather patterns, etc. All kinds of stuff.
Of course, each of us has only so many patterns to draw upon, and our prognostication is further limited by our ability to calculate how these observed, past tendencies might combine with one another. So we generally don’t do a very good job.
Hmm… The ideal future-predictor would have the broadest-possible knowledge of the past, would be crazy good at finding patterns in all that data, and then trained to extrapolate to the next bit of history.
All that seems sooo similar in form to what deep models do well. So I wonder if deep models will eventually prove better prognosticators than any human? From next-token prediction to next-year prediction?
Comment #89 December 14th, 2023 at 7:52 pm
About 8 years ago I knew that neural networks were going to be big, and about 5 years ago I decided to move my career in that direction. I did not see the generative AI boom coming at all. GPT-1 was okay but not great, GPT-2 was impressive largely because our standards were so low. So I didn’t really pay attention. In retrospect that was status quo bias, I knew they’d get better but I was thinking more like linear improvements.
More general predictions are often not that hard, specific ones are very hard. In a practical sense, anybody who skilled up in transformers and conv-nets and so on starting in 2015-2020 would be in a pretty good place to do whatever kind of work they wanted to do. Including, but not limited to, research on safety. One doesn’t need to predict the full details, just knowing that the tech is important is enough.
None of this really helps predict p(doom), except that I’m highly confident it won’t happen in the next 2 years. I have no estimate for anything in 2026 or later. I would focus less on p(doom) calculation and more about stopping it using methods which we can evaluate *today*, eg explainability/interpretability research on neural nets. That is likely to pay off regardless of what the future holds.
Comment #90 December 14th, 2023 at 8:49 pm
I did some experiments with Latent Semantic Analysis in mid-2000s. It’s like an early (1988) method to make document embedding vectors. I found it astonishing that even basic tools like bag-of-words model can extract some semantic information. Even more astonishing was that I found that my own associations are quite similar to what LSA believes is ‘close’: i.e. if I find X to be associated with Y in some non-obvious intuitive way, LSA also finds them close with a high probability.
So I considered probability of getting intelligence by number-crunching a pile of text to be above 10% in 2010-2030.
I got a pretty big update from Karpathy’s 2015 article “The Unreasonable Effectiveness of Recurrent Neural Networks
” where he demonstrated RNN generating plausibly-looking text. At that point I estimated probability of AI boom happening in the next 10 years about 50%.
And, finally, GPT-2 release (2019) is where I understood that the boom is happening. At that time I estimated people would start feeling the effects (e.g. AI replacing human jobs) by 2025, and we’ll approach singularity by 2030. I still think it’s true.
I think making more accurate predictions would have required a lot more effort (e.g. closely following AI research), and that’s not worth it.
Comment #91 December 14th, 2023 at 9:42 pm
Way off topic but Oppenheimer is out in video and I’ve seen some parts of it, enough to make me lose interest in seeing the whole thing. I thought it was terrible, factual accuracy jettisoned for movie tropes, acting and script consisting of Hollywood celebutards trying to roleplay nerds and mostly failing miserably. Rabi (or rather, whoever played him) was good in the moments I saw, Groves was ok some of the time, Oppenheimer himself consistently made me cringe.
It’s not that I’m such a factual accuracy snob: I thought Amadeus was a great movie, for example. But if Nolan was going to make a fictional film about Oppenheimer, he could have made it a lot more entertaining. The soundtrack was pretty good, so why not add some music and dance numbers? Or some action, like Oppie busting out some kung fu moves on Teller and Strauss at the security hearing. Or maybe wrestling the Hiroshima bomb out of the plane like Slim Pickens in Dr Strangelove: he’d ride the bomb all the way to the ground, while waving his hat yelling “I am become death, destroyer of worlds!”. Then after the bomb explodes, there would be a ground level shot of the fireball and mushroom cloud, with Oppie escaping on a motorcycle still wearing his famous hat. You get the idea.
I’ve read several Oppenheimer biographies and a few A-bomb histories and all of them were more interesting than the movie was. Nolan said early on “it’s not a documentary”, but a documentary would have been a lot better, it seems to me.
Comment #92 December 14th, 2023 at 11:03 pm
“But if someone had asked me: isn’t it plausible that, once it becomes feasible (via Moore’s Law) to scale neural networks up to hundreds of billions of parameters and train them on all the text on the Internet, those networks would display an extremely impressive emergent understanding of language? … I feel confident that even in 2009, or for that matter 1999, I would’ve said: sure, that’s plausible and seems like an idea well worth pursuing. ”
Scott, I am not sure I understand the basis for your reasoning. There was no reason to believe it at that time, and most researchers argued against such a possibility. For example, it was something of a common place that Markov models could not adequately represent language. Throwing a few orders of magnitude more data and compute at the problem would not fundamentally change that state of affairs. Of course, as we know now, that belief turned out to be incorrect and modern LLMs are actually Markov models. Still that was the best understanding at the time. On what basis would you argue otherwise?
As an analogy, imagine a chemistry researcher in 1880 asked about transmutation of lead into gold. Such a person would be right to write it off as an alchemical fantasy based on their understanding of chemistry. The fact that a few decades later that answer would turn out to be incorrect would not invalidate it, conditional on the best available knowledge of the day.
Comment #93 December 14th, 2023 at 11:19 pm
Scott #81:
Of course you couldn’t have anticipated the AI revolution, and none of the smart people in this blog couldn’t have anticipated it either. And of course, if someone had asked you the question differently, you might have given the “correct” answer. But then again, progress is about finding the right question to ask, isn’t it?
On this topic, it helps to go back and ask Geoff Hinton and his group what *they* were thinking before Alexnet. Here is an excellent interview:
https://ashleevance.substack.com/p/oralhistorywithgeoffhinton
In particular, I like this exchange:
“Vance: It’s incredible to have that sort of confidence though as a young researcher.
Hinton: I think it was partly that my parents were atheists, and I went to a religious school. So I was very used to everybody around me believing in God and me knowing that it was just complete nonsense.
That’s very good training for a scientist. To be in an environment where there’s some obvious nonsense that everybody else believes because most of science is like that.”
Comment #94 December 15th, 2023 at 12:21 am
Hi Scott. Are you familiar with David Deutsch’s views? He believes based on Popperian epistemology (knowledge creation is fundamentally not inductive) that there is as little reason to expect current AI paradigms to produce true AGI as to expect the deep ocean to produce true AGI. He has no p(doom) because p(doom) is an invalid concept (the future is unknowable and does not obey probability calculus). He believes civilisation-destroying risks are a real thing but the rational response to that is to produce general explanatory knowledge including AI research as fast as possible so as to be in the best position to combat risks in an unknowable future. (You once expressed a similar point I believe that you have no way to know whether AI research is more likely to doom us or save us from some risk.) I am surely mistaken in some aspects but this is the gist. Would appreciate your thoughts.
Comment #95 December 15th, 2023 at 2:12 am
Scott #38: finding the elementary proof of the prime number theorem is the only specific goalpost for (super)human-level AI that I have ever set. No goalpost moving from me.
And yes, of course, only a few humans could do it! By definition, “superhuman” AI is supposed to be better than humans, is it not?
And yes, we can add specific plug-ins to LLMs, just like we can add specific programs to a laptop. My point is that the LLM did not manage to teach itself multiplication, even though the rules for multiplication were presumably in its corpus of training material.
Comment #96 December 15th, 2023 at 4:31 am
Misha Belkin #92: Oh, I certainly wouldn’t have had any confidence that the “neural net trained on all the text on the Internet” would’ve worked. I merely say that I would’ve considered it an idea worth trying—with the evidence, again, being that I did consider it an idea worth trying when it occurred to me in 2014 (and not because of any knowledge that I’d recently gained).
Certainly, I was familiar since the late 90s or early 2000s with the old ideas that intelligence is all about prediction, and prediction is all about compression of past data, and compression is all about getting better and better upper bounds on Kolmogorov complexity despite its uncomputability. And I was familiar with the argument that the P vs. NP problem is so important in part because, if NP was easy in practice, then we could (e.g.) find the optimal efficient compression of the entire contents of Wikipedia, and finding that optimal compression might plausibly require uncovering the secrets of human intelligence and of the real world that had caused Wikipedia to be written.
In some sense, what OpenAI did was “merely” to operationalize the above thought experiment, as well as they could even in the real world where presumably P≠NP, by using gradient descent on gigantic neural networks. One of my big surprises, on joining OpenAI, was that this wasn’t merely how I thought about it: it was also precisely how Ilya Sutskever thought about it. This was very explicitly the vision.
Comment #97 December 15th, 2023 at 5:05 am
No longer a doomer #94: Yes, I’m very familiar with Deutsch’s views, having not only read his books and articles but visited his house to argue with him whenever I’ve passed through Oxford!
I love Deutsch’s emphasis on finding explanations and on the radical unknowability of the future, and a great deal else in his worldview. But I get off his train right before the stop where his admirably anti-dogmatic philosophy seems to harden into its own curious dogmatism, with key questions evaded via the repetition of Popperian slogans.
For me, Bayesianism is a specific, extremely interesting lens on the world that you should either use or not use as the occasion suits. I neither dogmatically accept it like the rationalists do, nor dogmatically reject it like Deutsch does. I’ve examined Popper’s technical argument for why Bayesianism is incoherent (the so-called “Popper-Miller theorem”), which Deutsch has endorsed, and found it to be utter nonsense.
Your comment contains perfect examples of the dogmatism that I reject:
there is as little reason to expect current AI paradigms to produce true AGI as to expect the deep ocean to produce true AGI.
See, beliefs like this are precisely how so many experts were caught blindsided by the revolution of the past decade. Once you’ve talked yourself into the existence of an unbridgeable chasm between “current paradigms” and “true AGI,” you’re then totally insulated from whatever progress happens in empirical reality: none of it counts, because none of it works the way you reasoned it should work sitting in your armchair.
But the entire point, in science, is to not be insulated from empirical reality in that way! We don’t know how much of human intelligence can be replicated by statistical pattern-matching. Many people thought they knew. They ought to be shocked and humbled by GPT’s success. Better to recreate your entire worldview in light of that success, than to invent clever-seeming reasons why the success doesn’t really count.
Likewise:
He believes civilisation-destroying risks are a real thing but the rational response to that is to produce general explanatory knowledge including AI research as fast as possible so as to be in the best position to combat risks in an unknowable future.
How could anyone possibly know, on a-priori philosophical grounds, that the “rational response” is to advance AI research “as fast as possible”? It might or might not be. As an academic scientist, my predisposition, just like Deutsch’s, is always to discover as much knowledge as possible as quickly as possible and to disseminate it as widely as possible. But we all recognize that there might be exceptions to that, for example involving nuclear weapons or dangerous pathogens. At what point AI becomes one of the exceptions is, once again, a judgment call, one that ought to be debated (and is being debated) in the arena of actual empirical facts.
Comment #98 December 15th, 2023 at 5:28 am
Scott, I’m glad you’ve updated your views, and you’ve asked what you could have done to better predict these things.
Its impossible to have a sound basis if you don’t fundamentally understand how things work, and even then it can be impossible to predict with any degree of certainty over any time period. There are problem domains where minor changes radically alter outcomes, and recognizing these problem domains lets you make better predictions, if any could be made at all.
Most people don’t understand how society works, how economies work, and there is a lot of misinformation out there. For example, modern economics lacks the fundamental basis that the more classic austrian economics curricula followed. When investigating truth, you can never have something be both true and false at the same time; yet this is what many fundamental aspects of how economics are taught today. How can both parties be right, when in reality only one is.
The threat of AGI is overblown in my opinion, its a red herring meant to distract. Its a catastrophic problem that neglects other equally catastrophic problems that are far more likely to happen before whatever requirements for AGI even occurs.
Let me ask you, what do you think happens historically when economic cycles stall?
There are two markets, goods and factor markets (labor); and they are interconnected. To buy goods one must work and receive for their labor enough money for themselves and enough to raise children, to make goods one must hire work and receive more from selling it than the cost. Its a very fine balance, which keeps going until interference’s stall that motive force.
If either sides are unbalanced, it either spins out of control, or enters a deflationary spiral that self-sustains. This type of cascade failure once started cannot stop until the underlying forces are balanced again. This is the true danger of AI, because its non-obvious how to correct, must be done by all market participants, and the producers involved are more likely to shut down than to change or limit profits in oligopolist environments because those profits must be used to pay debt service, thanks to leveraged buyouts.
Most people don’t understand how the world actually functions, and its not their fault because their reason has largely been corrupted by design of our current institutions from ages prior to the age of reason. Material has been doctored (in context) to fix thinking in set ways. You have go to back to material pre-1970, often the 1950s (a generational era of hyper-rationalism), to get an idea of what’s gone wrong.
No one seems to realize this, and while the supporting material was there, no one has used it to take corrective action. Most of those books are gone now because of how circulation has been limited. Books not read regularly at libraries get sold to third-parties who sell at a premium, or when storage no longer permits, they then destroy the books (via recycling).
The current generational era (which is still the majority of baby boomers) has mostly abandoned rationalism in favor of deceit and re-education.
Comment #99 December 15th, 2023 at 5:56 am
Mark Srednicki #95:
1. Fine, I understand your benchmark. We agree that it’s not yet passed, and I’m radically uncertain about when it will be.
2. But there’s a difficulty in defining your benchmark. Namely, a math AI on the current paradigm would be trained on nearly all existing mathematical knowledge. How would we “excise” the Prime Number Theorem and anything depending on it from the training corpus? Would we have to train only on mathematical texts up to the year 1890 or whatever?
3. Have you seen the now-famous analysis of a neural net that learned how to add modulo 113, at first by just memorizing examples but then suddenly switching to a general algorithm that involved taking Fourier transforms—an algorithm too weird for any human to have devised for that problem? We now have abundant evidence that neural nets can and do learn general algorithms for arithmetical problems.
Comment #100 December 15th, 2023 at 5:58 am
Couldn’t resist returning to write a brief comment.
I think the person who *really* foresaw the current era is Ilya Sutskever. He’s been working on text-prediction all the way back to grad school, I think, and even earlier — and probably all along fantasized what would be possible “in the future” if he could train models on trillions of tokens.
10 years ago I would say that I envisioned AI would have more hand-crafting than that, that it would use some knowledge graphs but then a lot of machine learning. About 8 years ago I transitioned to thinking more like Sutskever, and entertained the possibility that you’d see intelligent machines if you had trillions of tokens to play with — e.g. one can think of “using logic” as an algorithmic process and pattern that occurs across all contexts (A is B, B is C, therefore A is C should hold in most contexts); and a machine that learns higher-order patterns from data should learn to use logic, too.
Here’s one of my divergent thoughts I wrote 7 to 8 years ago (posted to a little diary of mine that is now a defunct reddit forum; note that the post is time-stamped):
https://www.reddit.com/r/StarspawnsLocker/comments/5sfk57/a_possible_unexpected_path_to_strong_ai/
My thinking was that “trillions of tokens” and “trillion-parameter models” are probably far into the future (decades out) just because it would be so expensive and nobody could get the funding for it; but I thought building a “language model” (I didn’t use that term) combining text with crude brain data (e.g. from FMRI), you could get the “language models” to absorb enough commonsense knowledge and procedural knowledge (or the faint outlines of procedural knowledge that machine learning could then turn into a complete set of methods / skills when given enough other data) to reach AGI.
So, instead of token 1, token 2, token 3, sequences I imagined token 1, brain state 1, token 2, brain state 2, token 3, brain state 3, … sequences to predict. In today’s way of talking about LLMs you might say that I thought we could “use brain data to substitute for inner-monologues”.
I didn’t think the learning algorithm itself mattered much, so long as it got a few basic things right (e.g. it had to be able to execute algorithms, however imperfectly). My focus was more on the data — the dataset is everything.
Comment #101 December 15th, 2023 at 6:23 am
In the last line where I say “I didn’t think the learning algorithm itself mattered much” I meant to that the “*model* doesn’t matter much”. The learning algorithm is still gradient descent — although I also didn’t think the learning algorithm mattered as much, either, and that evolutionary algorithms would also work for this.
Comment #102 December 15th, 2023 at 6:58 am
Just wanted to say thank you, Scott, for this blog post generating so many interesting comments! A treat to read.
That said, as someone who learned C++ in the early 2000s I was struck that object-oriented programming allows to create agents with their own behaviour, which could be quite human-like when human-type reactions were coded in. I didn’t predict the kind of stuff chatGPT does, but although it is a totally different approach I am not very surprised that a computer program can do that. Within 5 years some multimodal versions will be engines for robots, and that will look like AGI for all practical purposes. P(doom) at 1% until 2027, but at 50% for 2037.
Comment #103 December 15th, 2023 at 7:23 am
To answer your question, I did foresee around 2009 that many of the advances that we see now would happen around now. If there was any secret, it would be that I’m (I think) quite lucid on the process though which how I reason and get ideas, and that it’s not that complicated.
I was comforted in my ideas by grading my students’ exams at the time, which would show various degrees of sophistication, where the weaker students would just try to reformulate in slightly different forms what they had seen in course using some dumb pattern matching, and the stronger ones would do some smarter pattern matching (and I would trick the weaker students by question formulations that also work on ChatGPT).
Most people around me didn’t believe me, but I suspected that many educated people wanted to think that intelligence had some magic around it (because they wanted to feel special, and the alternative would force them to question too many things), and that they were hence biased towards giving perspective that would make them feel comfortable. I also suspected that sharing my views broadly would negatively impact my career, so I didn’t do that.
Comment #104 December 15th, 2023 at 7:27 am
@Scott, #97,
On the top of Popper-Miller, have you seen Ryan Giordano’s take on it https://rgiordan.github.io/philosophy/2022/10/19/popper_miller.html . I found it to be highly illuminating for what is going on. The essay essentially argues that Popper-Miller is just a case of the Bayesian Transitivity Paradox with fancier notation disguising what is going on.
While I’m here, to answer your primary question: I didn’t anticipate the AI boom by 2010 at all. I was pretty slow on things. It was late in 2020 when where it seemed clear to me where things were going. I remarked then to multiple people that if covid weren’t eating up all the news, the rapid improvements in AI that were happening would be getting a lot more attention. Soon after I said that, the general public started paying enough attention that it did become a major news thing. So I was behind the curve, but not as much as some people (which is only a tiny comfort).
Comment #105 December 15th, 2023 at 7:33 am
On AI, I am still lost on the “boom”. IS there actually anything smart coming out of AI rather than ruminate parts of the internet “wisdom”? AlphaGo maybe – but then it could just be rephrased in some nested switch stats, or couldn’t it? So, besides not anticipating it, I do not even post-recognize the “advance”.
On p(doom), it is quite high w/o AI anyway. I would, however, rather mention Obama and Bush Jr. in this respect, not that much Trump or Fat Kim (both of the latter did NOT start wars, the former yes, multiple). In other words, big part of the danger is not seeing, where it comes from. Being German, there was the Hitler singularity and then the Merkel one, neither really foreseen, both w/o any AI. So much for foreseeability in general.
My highest p(doom) involving AI would be to use AI as an excuse to generate the next pandemy (again more GoF that “bats”), the next Patriot Act, the next DSA (EU censorship, not signatures), maybe 50% here – and it would be 49% w/o AI anyway.
p(doom) of AI causally required for doom? Rather confidently I state: Not between 2 and 80% – which tail will be real, we shall see. The sooner, the 80+%, and in particular v.v.
Comment #106 December 15th, 2023 at 8:55 am
Did not foresee. Went along with the general assumption that computer scientists must first fundamentally understand the nature of human intelligence in order to replicate it in software. In hindsight, should have thought more carefully about the fact that *evolution* invented human intelligence while being an unthinking natural process that understands absolutely nothing.
Comment #107 December 15th, 2023 at 9:30 am
“human extinction has already been on the table since at least 1945”
TBF there’s very little chance that a nuclear war would have less than a few hundred million survivors, and basically none that it would have less than a few million.
Comment #108 December 15th, 2023 at 10:39 am
– Did you foresee the current generative AI boom, say back in 2010?
I wondered in 2010 why AI (neural networks) did not develop much faster in controlling machines by watching a human operator. A fruit fly (drosophila) has only 100.000 neurons and can do astonishing things (eg ruin my glass of wine by flying over from the kitchen, circle around the glass, landing on transparent vertical surfaces etc). Regarding language on the other hand I was ignorant. I would have expected some rule based progress first.
– If you did, what was your secret?
– If you didn’t, how (if at all) do you now feel you should’ve been thinking differently?
I should have transferred my thoughts of machine control on language, maybe by comparing the complexity (of flight control compared to very simple language like step by step instructions using restricted grammar and dictionary). Also I must confess that I completely missed the relevance of generating language or pictures.
– Feel free also to give your p(doom), under any definition of the concept, so long as you clarify which one.
p(doom) = 50% regarding a human-machine cooperation. One day, after a fatal accident, the first driver, doctor, engineer, politician will be jailed by other humans for having contradicted some AI system while the AI-believers go free after any outcome, protected by legal departments of software giants. From the next day on everyone will obey any AI system on any scale.
Comment #109 December 15th, 2023 at 11:39 am
Wow, a new post up with 107 comments before I saw it. I usually check twice a day for new posts, but was too slow today.
I don’t see that you have anything to apologize for, a) on general principles, what’s wrong with making wild guesses when asked for them?; and b) I think actual general machine intelligence is still a long way away. What we have, I think, is some amount of specialized machine intelligence. As a former president might say, is our machines learning? To which the answer is yes, but we have nothing like an artificial Dr. Scott Aaronson.
My vague precognitive effort would include the possibility of never. Not that I think it (AGI) is impossible, but that we have some likelihood of regressing back to the Stone Age (or below) first. (I cite Trump-voters as evidence.) Also, the commercial applications (which will get development funding) are specialized ones, not world-ruling ones. Playing board games, folding proteins, smart encyclopedia, and so on.
Could we destroy ourselves with specialized AI’s? Yes, but we will be the drivers, not the AI’s, and therefore we will deserve it.
(This will repeat what a lot of people have already commented, but by the time I read 107 comments, there would be another 107.)
Comment #110 December 15th, 2023 at 12:03 pm
I think your outlook was perfectly reasonable, especially prior to signals like ImageNet. If anything, I think this kind of embarrassment in hindsight actually means you might update too much because you happened to be wrong, not because you actually thought about it incorrectly very much at all.
I wasn’t even thinking about AI, let alone predicting anything about it back then. I think my most salient thought around technology was that I was impressed with the iPhone browser and app store.
Currently, I hesitate to give a p(doom). It’s another pretty unknowable outcome, and that by itself should give us pause. It’s like one of those macroeconomic models where you can create any outcome just by varying your assumptions. Overall, I suppose I’d go for 5-10%, on the assumption that we will figure out how to align AI, most AIs will actually be aligned, they will protect us from the baddies, and that there is no new physics like ice nine or backyard fusion bombs to be found in the near future that makes destroying the earth trivial.
Comment #111 December 15th, 2023 at 12:32 pm
By coincidence, 2010 was the year I became a doomer:
https://johnlawrenceaspden.blogspot.com/2010/12/all-dead-soon.html
You’ll notice Eliezer commented on that, telling me not to be such a pessimist. He was still optimistic about solving the alignment problem then.
I didn’t predict this specific path of neural networks becoming AI, but it wouldn’t have surprised me. In this blog post I wanted to emphasise that the route doesn’t matter. In fact neural nets, because they’re very difficult to understand, are one of the safer routes!
But it was blindingly obvious that something would happen and soon.
Because we were already getting to the point of being able to do things like computer vision. And that meant that we were on the verge of being able to build an animal mind. Even something as simple as an insect would do.
Once you have something that can act as an agent in the world, you’re almost there.
Evolution went from animal minds to human minds very quickly, and evolution is very stupid, very slow, and not even focused on the problem of creating intelligence.
It was clear that human engineers would do a much better job, much faster.
2010 was the year my attention was first drawn to this set of ideas. I would have said exactly the same thing in 1950, after seeing the first chess-playing program. And I would have been right then too.
I would have been as surprised as anyone else that it took sixty years to solve computer vision. I suspect that was because we weren’t really trying for most of that.
Creating intelligent agents is a very easy problem. Evolution managed it!
Aligning them looks very hard.
Comment #112 December 15th, 2023 at 12:33 pm
Thanks Scott for the fun audience questions!
In 2008 or so I would have said it was hundreds of years away. I took some AI classes in college and thought planning & min-max algorithms were awesome, but I was in general fairly skeptical that gradient descent could learn any high level abstract reasoning at all. I updated pretty hard after about 2014, seeing that deep networks could finally “see” and learn some patterns in language. Like Eliezer, 10-50 years seemed reasonable after that. I didn’t think it would be a matter of just scaling up. I thought we were a couple of architectural breakthroughs away. My pet project was going to be reinforcement learning and a push down automata. Basically, I figured once someone put those together or something similar it would lead to rapid progress. I was definitely shocked when the transformer got as far as actually understanding language.
What should I have thought of differently? I don’t have regrets, really, but in general I agree with what you mentioned in your blog post — do not be so quick to default to a “no-change” assumption. Even in politics! In general, I hope that people can engage with ideas on their merits and not dismiss them merely because they are “weird”, but it’s a needle that can be hard to thread, because that also leads people to go down conspiracy rabbit holes. Maybe it would be better if people could hold their pet theories in superposition a little better. Imagine if conspiracy theorists said, “I think that there’s an X% chance that Y is a lizard person, but I am willing to update based on evidence, and don’t fault you for placing a much lower probability on this.”
My p(doom) is around 10%. because I think it’s convincing that (a) human intelligence will be surpassed, (b) intelligence will lead to power, (c) optimizing hard for most things would probably not go that well. Why it’s not higher: I think that it might be possible to construct super intelligent AIs that have little agency, or that by pre-training on human data, like it wouldn’t be too hard for them to absorb our values by osmosis. However, I don’t know if that’s enough, since agency can spontaneously create itself even by simulation… And once the AI is training on its own outputs, essentially coming up with its own thoughts or trying to solve a problem in doing some kind of reinforcement learning, it could certainly go off the rails. Plus, even if one group creates an unaligned AI, another group might create the unaligned one…
Sometimes I worry p(doom) should be higher, because I like to think of the AI alignment problem as essentially requiring a solution to every single unsolved problem in philosophy and we haven’t done that in thousands of years. For example, we may need to fully specify ethics. There’s also the hard problem of consciousness, the is ought problem, what is the meaning of life (because if a super intelligence gets that one wrong then we’re going to have a bad time). Possibly we may need to even throw in political questions in there too. OpenAI says to have the AI solve all of that, but that does seem a bit hand wavey! (Ok, I’m updating to 20% now)
Comment #113 December 15th, 2023 at 1:06 pm
p(doom) wise, it’s a racing certainty.
We’d have to get very very lucky.
There are paths to avoiding doom, but none of them look likely and most of them look infeasible and unpleasant.
One thing we could do is just not to build the damned thing in the first place. There’s still time to just not do it, but that kind of good sense seems to be utterly beyond us as a species.
Comment #114 December 15th, 2023 at 1:07 pm
You ask what modifications you could make to your mind to see such things faster.
I think the trick is to stop caring about what other people think. When I first started talking about this, I very quickly met a man (far more intelligent than me, and a better mathematician), who pretty much shared my new set of beliefs, and who’d been worried about the problem for years.
And I said “Why on earth didn’t you tell anyone!”.
And he said “I didn’t want to look like a fool.”
I’ve got no problem looking like a fool. And I’ve got no problem offending people.
The world is mad, and everyone is wrong. You know that.
Why would you care what all those idiots think?
Comment #115 December 15th, 2023 at 1:39 pm
How would Alan Turing have reacted to this instructive blog post? My two cents: https://link.springer.com/article/10.1007/s11023-023-09634-0
My Turing would insist that mathematics is akin to a growing organism in that it can never be fully captured symbolically, technologically, … in advance — which does not mean that I’m disagreeing with much of the commentary in this thread.
Neo-Platonism and/or British idealism are essentially about non-propositional knowledge, i.e., the stuff intrinsic to humans and foreign to symbolism. I’m missing this dimension in much of the present discussion. The very idea that ethics can be formalized …
Many of us seem to take for granted that everything under the sun can ultimately be captured in rules, in a program text, in a neural network, in something the engineers will get done… eventually. This insight is not shared by ChatGPT and many modern logicians. They score exceptionally well in one dimension and poorly in the other — imo.
Comment #116 December 15th, 2023 at 1:59 pm
In 2010 I predicted AGI (better than most humans at every meaningful cognitive task) in 25 years. 90% confidence between 15 and 40 years. High end 40 because we could probably brute force it with brain emulation if really necessary. It now seems I was overly pessimistic.
I believed:
– It was clear computers (our current silicon paradigm) could think. There were plenty of people who disagreed, but their arguments were silly and magical.
– There’s a continuum from brain inspiration to brain emulation. I doubted we would need to emulate the brain but it would be an option if AI turned out to be very difficult.
– Evolution made brains that can think (and relatively recently). Evolution is very dumb and inefficient. Having an example helps.
– Brains have enormous limitations that silicon doesn’t share (energy, size, substrate frequency, communication speed, reliability, copy-ability). So we will probably find it easier to make intelligence by other pathways than just trying to copy the brain.
– I looked at the fab predictions and decided Moore’s law would continue for at least long enough for sufficient compute to emulate a brain.
– It would be a bottom up, learning, emergent intelligence. This seemed much more realistic than the top down approach. The type of AI I imagined was something like a swarm of self modifying programs (but I did not have much confidence in the specifics).
– This prediction seems wrong now, but I also believed AI was so obviously important that there must be groups (companies, governments, etc) working on it in secret, and thus the apparent state of the art was probably years behind the actual state of the art.
Comment #117 December 15th, 2023 at 2:18 pm
One follow up, I don’t think blaming capitalism is that far off the mark (though may be an unhelpful op-ed at The End Of The World), and I don’t understand exactly why the rationalist movement seems to often overlap with libertarianism. They are well aware of prisoner’s dilemmas, stag hunts, and the failures of markets, and yet somehow they do not see that these failures are pervasive everywhere in our society and not just in a few rare cases… or they fear the bureaucrat more than the monopoly, somehow. (I often wonder whether many EAs come from family money, as some seem to take low paying jobs and still manage to survive in HCOL areas) It’s a point of disagreement I find with a community otherwise pretty well aligned with me!
I find that the biggest missed opportunity of our current society is our inability to commit to any level of higher order coordination or planning, and how we use capitalism as an excuse to let everything happen on game-theoretic autopilot. My philosophy: if there are big problems in the world, let’s solve them, instead of doing mental gymnastics as to why the problems will solve themselves, or that we can’t do any better. (climate change, healthcare, banking systems, education, AI alignment, …)
Comment #118 December 15th, 2023 at 2:55 pm
Michael M #117: Rationalists actually vary a lot in their degree of libertarianism. And interestingly, even the ones who are otherwise libertarian (Yudkowsky, Zvi Mowshowitz, etc.) often carve out a huge exception for AI, believing that letting anyone build their own AGI will soon become like letting anyone build their own thermonuclear weapon.
On the other hand, it’s true, as a general rule, that rationalists tend to be liberals (in the old-fashioned Enlightenment sense) rather than hard leftists or Marxists. I think this is partly because they just like individualism and reasoned debate more than mass uprisings and violence, and partly because they’re acutely aware of the unbelievable suffering that reliably ensued when Marxism was tried throughout the 20th century … and unlike many intellectuals, they’re Bayesians, for whom a core value is updating based on evidence. 😀
Comment #119 December 15th, 2023 at 3:54 pm
To answer your questions Scott, I didn’t foresee the generative AI boom, exactly. As you point out, neither did EY.
On the other hand I a few other things are true, given my reading of LessWrong and interesting in rationality and AI safety since about 2012:
1. I was much less surprised than most people about generative AI.
This is mostly because, on reading LW and better understanding AI safety and what AI *means* to the world, it was clear this was going to be one of the most important technologies. That made me watch the space, and start seeing crazy results that people hadn’t predicted would happen suddenly happen. Generative AI is the latest, but we had AI classification, then AI Go, then AI image generation (first versions), etc. It was a process that over and over showed that *no one* was predicting this stuff any better than I as a layman was doing by just thinking it’s important.
Also worth noting that this effect fed on itself – EY wasn’t just convincing me how important AI is, he was (ironically?) convincing the people who made many of the breakthroughs.
2. I am even now far more “bullish” on AI impacting the world (which I think is the correct approach, though could end up being wrong).
What can I say. People keep thinking that no big changes are coming. Trump, then COVID, the generative AI, then October 7th… I think the Black Swan theory is looking more and more correct.
3. I think my “meta-interest” in AI safety has proven correct.
By 3, I mean that, after I started getting interested in LW whether we should worry about AI safety, I would’ve answered of course! Cause it might be 5 years away or 50 or 500, but we have no idea how much time we need for solving safety, so the sooner the better.
I think this meta approach is largely correct. There is the “worrying about overpopulation on Mars” aspect, true, and maybe if I genuinely thought AI was potentially thousands of years away I wouldn’t meta-decide to invest in it.
Still, I think one thought experiment that made things *very* clear to me – if we decoded a message from Aliens saying “hey, we’re coming to Earth sometime in the next 50-500 years” and that’s it, this would probably cause massive changes to the world and cause us to frantically scramble to prepare for them, having no idea if they were a threat or not.
AI seemed very plausible to happen in that kind of timeframe, and yet *no one* was reacting in that way despite AI very clearly being the same thing (I was and am completely sold on the orthogonality thesis).
Anyway, my (more than two) cents.
Comment #120 December 15th, 2023 at 4:29 pm
@Max #69:
Si Yu How’s method turns out to be exactly equivalent to yours (I can’t be bothered to check that analytically but if I compute them both with gnuplot they’re always within 2.5e-16 of each other).
Comment #121 December 15th, 2023 at 5:13 pm
Scott, what’s this “President Boebert” thing? Whatever your personal political beliefs, contemptuously mocking the half of the electorate who support Trump, Boebert, MAGA, etc. does nobody any good. In any case, you’re alienating a large number of people who might otherwise find common ground with us on AI safety. I encourage you to remove that comment.
Comment #122 December 15th, 2023 at 5:30 pm
On your point 6: If you have been at OpenAI for a while, you will know by now, right? The ‘good reason’ was the serial dependencies in recurrent neural networks. Transformers removed that bottleneck by enabling parallelism and thus very large RNNs (like GPT now) as of 2017. Both training data size and model size grew orders of magnitude and this ‘quantity has its own quality’.
For the rest, we cannot predict if (depending on the survival of (what should be) civilisation and thus a science base) if or when AGI can be created. It is however pretty obvious to me (and others) that the current advances (very large token prediction models) are fundamentally limited. It is also clear that (most) people working on it are convinced this is not the case. The way we humans have been ‘bewitched by language’ is interesting, and what we learn about human intelligence (and its limits) will — I hope — be more when the GPT-fever finally breaks. LLMs are impressive, true, but humans are impressionable. See https://ea.rna.nl/the-chatgpt-and-friends-collection/ for a collection of clarity and realism (reviews/reactions have been quite positive).
Over the last 6-7 years, we’ve had a blockchain-hype, a somewhat lesser intensive QM-computing hype (where you were a helpful voice of realism, I must say) and now GPT-fever. These systems are indeed not pure ‘stochastical parrots’. Linguistically they are, semantically they are ‘stochastically constrained confabulators’. There are absolutely parallels with our own brains (our own convictions aren’t that reliable either). For those not convinced that “AGI is Nigh” we can see that useful tools can come out of this, but — if you’re willing to see them — the signs are clear that actual intelligence is nowhere near and that the companies are impressively engineering the hell out of what is (for AGI) another dead end.
Personally, I think that with only digital technology it’s not going to happen. But add some non-discrete technology to the mix, it might. But on that front, there is still very little to see (except maybe https://doi.org/10.1038/s41586-019-1901-0). Until then, whatever the amount of integers you throw at it, human-level intelligence, let alone superhuman-level is going to be a mirage, regardless of how current LLMs impress us — easily fooled — humans.
Comment #123 December 15th, 2023 at 6:34 pm
Draughtsman #121: Shouldn’t they be excited rather than offended by the prospect of President Boebert? While I admittedly have no shortage of contempt for her, I didn’t express any in the post.
Comment #124 December 16th, 2023 at 12:00 am
My guess is that the “elementary proof of the prime number theorem” goalpost is pretty far out there.
A nice closer in goalpost might be “prove that rad 2 is not a fraction”.
Comment #125 December 16th, 2023 at 12:28 am
Scott, on predicting the generative AI boom back in 2010-I certainly didn’t, I even scoffed at other first years in 2013 for jumping on what I considered a hype train after Alexnet. Never thought we’d see this kind of progress just 10 years after, but I was obviously very mistaken.
I like your to approach uncertainty-I feel myself struggling with it, especially after having to effectively recalibrate my previous uncertainties from a decade ago. I still struggle with assigning my own numerical p(doom), but it seems likely (>50%) that human intelligence is no longer the dominant intelligence on earth in the coming decade, and that scares me. (I’m somewhat sympathetic to arguments that human intelligence has not been the dominant intelligence for years but a lot of that depends on how those terms are defined-personally I see GPT4 as a completely novel intelligence, especially compared to corporations or recommendation algorithms.) That said-it’s hard to stress that I’m still uncertain about all this! Luckily basically no one on the internet really cares about what my probabilities are and more importantly, if I update them later. I can only imagine how frustrating it is in your position.
Sincerely really appreciate your candor and introspection!
Comment #126 December 16th, 2023 at 5:28 am
Scott,
Okay, don’t _predict_ anything. But based on your expertise (from your using your own brain for decades), do you think that mathematical creativity is qualitatively different from the other applications of the human mind those which now have been automated? Which side would you lean to?
Comment #127 December 16th, 2023 at 7:39 am
Raoul Ohio #124: For proving the irrationality of √2, it would be really hard to exclude any training data that gives away the solution (and that makes this task trivial for GPT right now!).
Comment #128 December 16th, 2023 at 7:48 am
Ashley #126: No, I don’t think there’s some sort of vast qualitative chasm that separates mathematical creativity from everything else humans do. For starters, there’s chess, and programming, and engineering, and architecture, and even music and visual arts and humor, all of which clearly call on overlapping sets of skills. What makes tasks like proving the Riemann hypothesis special, I’d say, is the combination of
(1) extreme difficulty (much harder for AI than destroying humans at chess, Go, and other games, as it turns out),
(2) impossibility to imagine a solution that doesn’t require “leaps of creative imagination,” and
(3) ease of recognizing and agreeing on a solution once it’s found.
Comment #129 December 16th, 2023 at 9:21 am
Scott,
What do you think of David Tong’s views on “physics laws can’t be simulated on a computer.” Specifically he goes to great lengths to say that weak nuclear force cannot be simulated. The insights seem to be deep and touching at the roots of discrete vs continuous with QFT but seem to hit on the head of Church-Turing thesis.
Comment #130 December 16th, 2023 at 9:51 am
Prasanna #129: I hadn’t encountered that particular argument before, but there are few words to express just how skeptical I am. (Indeed, I’d be surprised if many other QFT experts bought the argument.) Do you really find it plausible that, as far as the strong force and electromagnetism are concerned, the universe might as well be a computer, and the non-computational nature of reality rests entirely on a technical issue with discretizing the weak force?
From the modern standpoint of Effective Field Theory, QFT itself is presumably just a continuum approximation to some deeper theory (incorporating gravity among other things). And the Bekenstein-Hawking arguments strongly indicate that whatever that deeper theory is, it involves an only finite-dimensional Hilbert space, and a spacetime that’s no longer a smooth continuum at the Planck scale. That’s the theory that seems ultimately relevant to discussions of the Church-Turing Thesis.
Comment #131 December 16th, 2023 at 10:20 am
Ekeko #19:
You know what? I liked both Eliezer’s and Scott’s performance, especially for the parts about MWI. (After some time, I skipped forward to those parts, because I was curious what they would say.) In every part of the podcast that I saw, I appreciated how Eliezer intensively queried Scott to get a good understanding of what Scott believed deep down inside. I don’t care about Scott’s best performance from that time around 2009, this one is more than good enough for me.
Comment #132 December 16th, 2023 at 11:10 am
Ekeko #19:
You say that 2009 podcast was a poor performance of yours. I wanted to ask you which podcast or video or whatever of you, you think is the best.
Sorry for the delay! You might enjoy my two podcasts with Lex Fridman, or my videos with Robert Kuhn on “Closer to Truth”.
Comment #133 December 16th, 2023 at 1:02 pm
Scott #97, Thanks for the gracious reply.
1. I would love to hear more about your thoughts on Popper v Bayesianism, at length if you have the inclination. I have only heard the qualitative arguments, and have found them persuasive. It seems important and possibly the root of your differences.
2. DD seems to be following the developments to see if they do indeed contradict his views. I think he would reassemble that part of his worldview if they did – anti-dogma is his whole schtick!
3. Producing knowledge as fast as possible includes producing theories of how to do things safely. As fast as possible means that those who would behave responsibly must do it before those who would not. Nuclear weapons and pathogens are instructive examples! He has said something before about how plenty of civilisations have been destroyed by lack of knowledge – none of them by excessive knowledge.
4. Do you think an AGI would morally count as a person? This also seems an important premise.
Thanks again.
Comment #134 December 16th, 2023 at 1:34 pm
No longer a doomer #133:
Bayesianism is a clear and internally coherent, and often very useful, framework for quantifying your beliefs and updating the quantification in light of new information. I don’t see any comparably clear and coherent framework on the other side to even critique, just a lot of nice-sounding philosophy about knowledge, criticism, etc. So it’s sort of an apples-to-oranges comparison. At some point, I might blog about the “Popper-Miller theorem,” which purports to show that there can be no such thing as inductive evidence in favor of a theory in Bayesianism, but which turns out to rely on a forehead-bangingly bad redefinition of words (as it would have to, since the conclusion is so self-evidently false when words have their normal meanings!).
I’m glad that Deutsch talks the talk about resisting dogmatism. The hard part, for all of us, is to walk the walk. When half of someone’s argumentation seems to include “but X said…” for a single value of X, that’s a gigantic red flag to me, regardless of whether X is Jesus, Karl Marx, Ayn Rand, or Karl Popper.
I completely agree that we should gain knowledge relevant to building AI safely as quickly as possible! That was basically the founding mission of OpenAI (not to mention DeepMind and Anthropic), and I’ve put my money where my mouth is, by working for OpenAI’s alignment division for two years. At the same time, though, we could rightly worry about the differential between alignment and capabilities — is our knowledge of how to build powerful AI outpacing our knowledge of how to make it safe? And that is indeed what the doomers worry about.
Under what circumstances we should treat an AGI as a person is so ridiculously harder than your other questions that maybe I’ll take a pass for now. 🙂 I’ll limit myself to commenting that, if someone didn’t want to regard it as a person, then I think the burden should be squarely on them to articulate an empirical difference between the AGI and humans.
Comment #135 December 16th, 2023 at 4:20 pm
Scott #130 I guess it is a reference to a Tong’s FQxI essay: Physics and the Integers, with a smoothened version of it at S.A.
It contains a technical point on making a lattice version of chiral theories, namely a lack of success of it.
Comment #136 December 16th, 2023 at 4:45 pm
> Sure, LLMs might automate most white-collar work, saying more about the drudgery
> of such work than about the power of AI, but they’ll never touch the highest
> reaches of human creativity, which generate ideas that are fundamentally new rather
> than throwing the old ideas into a statistical blender
I’d just mention there is a large gap between these two. Putting together old ideas in different combinations and new settings has always been considered a form of human creativity and insight (though arguably not “the highest reaches of human creativity”), certainly far from drudge work. I think existing AI has already passed this threshold… or if not, it soon will.
Comment #137 December 16th, 2023 at 4:53 pm
Answering your question:
In ~2010, I roughly foresaw what has transpired since. More specifically, in 2009 I was talking with a friend in AI research, who told me that he saw great promise in the “deep learning” approach. He singled out DeepMind as a company to watch. I watched with growing alarm as DeepMind made steady progress, then as other companies made progress with similar approaches. Since I was paying close attention to developments, I didn’t see ChatGPT as a step change, but as a continuation of progress I had been observing for over a decade (though one that had crossed an important threshold).
I didn’t specifically predict that scaling would hold up as it did. I had a rough sense that things were progressing along the path they were, and would continue to do so. Maybe it’s just hindsight bias, but my perception is that things have gone about as I expected.
I remember watching your conversation with Eliezer at the time and thinking that your view seemed badly mistaken.
Comment #138 December 16th, 2023 at 5:05 pm
Never anticipated the the current generative AI boom, because I was a bit sceptical of the neural nets so wasn’t looking at that.
But I was always consistent in predicting that AGI + superintelligence would arrive sometime in the period from 2020-2070 , I was saying that since about 2000. I was sceptical of the doomers, since I didn’t think there was enough information about how a real-world super-intelligent system would operate to be able to form a clear judgement.
Even I had underestimated the speed of change, and now my time-lines have shortened substantially, I now think a 50% change of AGI within 9 years, following the Metaculus prediction market for their definition of ‘strong AGI’ (the most conservative time estimate).
I had also underestimated the probability of doom, and after further reflection, I now think there’s a greater threat than I had thought. The trouble with giving probabilities on this though, is that we simply don’t have a clear model of how a super-intelligence would work in the real-world; the information just isn’t there; also the outcomes depend on what we do; these factors muddy the waters and it’s not clear what it means to assign probabilities.
But still, I think I was quite wrong to go about giving the impression that the chances of doom were low, and in fact, based on the *theory* of what we know at least ( independence between values and reasoning aka ‘orthogonality’), the chances of doom are higher than I thought.
p(doom) of 5-30 % (wide range of uncertainty).
Comment #139 December 16th, 2023 at 5:26 pm
Thanks Scott.
1. Happy to park it there. I may set myself a long term goal of persuading you of its merits. Bayesian rationality has been central to my worldview for a long time (eg EY and other Scott are two of my favourite writers), so it’s been a big revision. Another day though.
2. The modulo 113 example you mention up-comment seems a good example of an LLM seemingly creating new knowledge – is that the best one do you think? It would be interesting to put such an example to DD and see if that modulates his view (which I may try to do).
3+4. The personhood question was not meant as tangential. DD seems to think that generality (at least the kind of generality sufficiently powerful to present an AI risk) must come with a general ability to make choices, much as a person does. And that forcibly aligning an AGI by constraining its ability to think certain thoughts, or hardwiring a certain kind of morality, rather than permitting it to be free, may be counterproductive and prone to lead to aggressive retaliation – and perhaps justifiably so, if an AGI has personhood. So the AGI alignment project (which really is very important, seeing as, given the unknowable future, we could have AGI tomorrow, whatever anyone might think of current AI paradigms!) is the same as the project of developing better theories of morality and education in general. And here DD favours non-coercive techniques, though there must be space for debate there.
Comment #140 December 16th, 2023 at 5:32 pm
On the post, I appreciate your willingness to engage on a nonsense, bad faith twitter attack and turn it into something containing genuine insight. I also watched your recent-ish interview with Curt Jaimungal and enjoyed it. You get into the no-cloning theorem and its implications for Newcomb’s problem, and while watching I thought of a wrinkle that you didn’t address and I don’t see addressed in the post. Is it relevant that, if the Predictor is allowed to measure you, each particle in your body would left in a known eigenstate, and the Predictor should be able to predict your evolution from that point onward? This presumes the act of measurement isn’t lethal, which it probably is from a thermodynamics standpoint. I feel that this defangs the no-cloning based argument that a perfect Predictor cannot exist, but I may be missing something.
Comment #141 December 16th, 2023 at 6:20 pm
Lane #140: As a survivor of many, many bad-faith Twitter attacks, by no means did I think that all the criticisms of me this time were in bad faith.
As for brain prediction and the no-cloning theorem, I’d say that whether the “measurement is lethal” (or at least, lethal to the “original consciousness,” whatever that means) is precisely the question at issue! In The Ghost in the Quantum Turing Machine, I quote Niels Bohr in the 1930s arguing that it would be.
Comment #142 December 16th, 2023 at 6:30 pm
No longer a doomer #139: The modulo 113 example is a good one, but there are now plenty of comparable ones, including the recent “FunSearch” thing from DeepMind. Crucially, I don’t think there’s any qualitative chasm between “mere calculation” and “creating new explanatory knowledge” or whatever — it’s all just a continuum, one that AIs are currently moving along.
The discussion of “coercion” of AIs is complicated by the fact that the doomers mostly imagine AGIs that monomaniacally optimize a single goal given to them (either intentionally or unintentionally) by their creators, but neither humans nor current LLMs seem well-described that way, so I’m currently unsure how to think about the likely goals of AGIs, and the answer matters enormously! It’s like, should we think of ourselves as having been “coerced” by evolution into being obsessed with sex and sugary food?
Comment #143 December 16th, 2023 at 6:52 pm
Scott #134: If one is of the view that “Thou shall not bear a false witness” is a sine qua non of safe AI, the current inability to eliminate ChatGPT’s “hallucinations” strikes me as an indication that at least at the moment “our knowledge of how to build powerful AI outpacing our knowledge of how to make it safe”.
Comment #144 December 16th, 2023 at 9:59 pm
Scott #40: Once you add, e.g. Hasse and Teichmüller to the list of mathematicians in question, as well as Heidegger and Schmitt to the list of (arguably) some of the most consequential “scholars in humanities” of the period, my reservation (expressed in comment #9) might strike you as perhaps less frivolous than it might appear.
Comment #145 December 16th, 2023 at 11:52 pm
Scott #99:
2. Yes, it would have to be trained on a restricted corpus. I see, though, that this may well be too impractical. So an alternate benchmark is to solve some still unsolved hard problem, such as one of the remaining six Clay Millenium problems. (I’d be willing to come down some in level of difficulty, but if we’re talking “superhuman” AI, I don’t want to come down too far.) Of course the issue here is that we don’t know how hard any of these actually are, which is why I prefer my original benchmark.
3. I had not heard of this, and I agree that it is indeed an example of the sort of behavior that is going to be needed to get AGI. My question is, how far is it from these examples to doing math that humans find difficult? Finding a very complicated way to do something that is actually quite simple does not give me a lot of confidence that this distance is short. But I will grant that we are now at least one step along that road. Where we differ is in our estimates of how many steps there are yet to go.
Comment #146 December 17th, 2023 at 12:29 am
I think you’re jumping the gun with “civilization-altering”. I’m very unsure how useful LLMs are at their current level. They seem to be a time-saving tool to be wielded by an information worker, similar to google search when it came out. I don’t think they’ll go much further than that. I say that without making any claim on the innate difficulty of solving math problems or how good a fit LLMs are for such tasks.
I correctly predicted in 2020 that the growth of these models (in number of parameters) would slow within a few years, as there are a bunch of technical and economic barriers that lead to sharply diminishing returns. I’m not sure if we’re yet at the point where we’ve run out of useful training data, but it sure seem curious that the current model sizes are hovering somewhere around the boundary where it becomes impossible to keep them in memory on consumer hardware. I continue to think that development will slow down due to such mundane reasons, with the models becoming more and more polished on the surface, harder to unmask, but limited in their usefulness in essentially the same ways they are today.
It’s a different story for the arts; image models have surprisingly good performance, and even a few years ago I considered any sort of visual problem to be completely intractable compared to language problems, based on their large input-output size. I was completely wrong in that regard. It turns out image models are extremely good at low-level operations that humans have trouble with, while stumbling and failing at the higher levels of meaning in pretty much the same ways as LLMs.
Comment #147 December 17th, 2023 at 1:31 am
My point, I guess, is that at least insofar as “Thou shall not bear a false witness” universally embraced proscription is concerned, AI’s record to date is at best no better than that of humans throughout their recorded history; I am yet to see a compelling argument indicating that AI is apt to be less transgressive vis-à-vis other (universally embraced) behavioural/moral constraints/commandments (violating which, AI (to date) appears to be nearly bereft of capability).
Comment #148 December 17th, 2023 at 7:07 am
Mark Srednicki #145 (and Vitor #146): We agree that LLMs haven’t yet achieved the milestone of generating a novel mathematical argument and explaining it in English. We agree that it’s hard to predict how many more steps until that milestone is reached. We also agree, it seems, that the first steps have now been taken: one can no longer say with a straight face that the progress has been zero.
If so, then the big remaining question is indeed the timescale. Here’s the key point I want to make: at the time of my conversation with Eliezer in 2009, I thought it plausible (for all I knew) that it might be hundreds or thousands of years before any AI could do what GPT-4 and DALL-E and so forth do right now. Remembering as honestly as you can (or even, if possible, consulting old emails and so forth), did that also seem plausible to you? If so, shouldn’t the dramatic collapse of some AI timescales cause a very hard update regarding the others?
Comment #149 December 17th, 2023 at 8:08 am
Scott,
Narrow AI continues to churn out impressive results, especially from Deepmind. The key characteristics of these systems 1) They don’t seem to use anywhere near as large compute as GPTs 2) There is significant human in the loop involvement, heavy finetuning and customization to solve a specific problem.
General AI also continues to improve in an impressive way , however 1) Each significant improvement or new emergent capabilities seem to take order of magnitude more compute. 2) Human involvement seems to be limited to safety/RLHF and algorithms .
Given these divergent trends, do you think both approaches need attention for Capabilities and Safety research and regulation? Perhaps narrow AI can even be used as a mixture of experts as is done now, to achieve better outcomes with General AI ?
Comment #150 December 17th, 2023 at 8:51 am
Scott #148
Sorry but I don’t think researching how to imprison a super-intelligent entity in a virtual dungeon manacled to the wall ends well.
I can’t imagine there is an impossibility barrier that precludes man from building a super intelligence. I personally do not think the current incremental approach gets there in my lifetime and very possibly never. The hybrid approach with a grown super intelligent brain coupled to a network has scope but again not in my lifetime. In the case of a grown brain the ethical issues are clear but if realized only in silicon then for some reason not clear for many.
Comment #151 December 18th, 2023 at 12:04 am
If you think about it, Ilya’s PhD thesis was already on next token generation and predicting Wikipedia articles. That was pretty impressive.
In NIPS 2015 people were struggling with Q&A but you could sense the progress.
Google translate for pairs of languages without any examples was very impressive.
Then we had GANs for a while that distracted people, and then kind of a short winter when people felt the field is stuck.
Then we had leaks about Google’s PaLM model and an over excited engineer thinking that the model is conscious.
ChatGPT was the mass product but the tech was already there at that point.
Have we made significant improvements since then? Not clear. The progress at this point seems quantitative not qualitative again, but the massive increase in interest promises someone coming up with the next big idea.
Overall, when we look back to 2012 the number of key ideas are not that many and are not very surprising. It is engineering to turn them tech to useful products.
The amount of compute is also a huge obstacle. Many services are behind allowlist and rare limits because simply there is not enough compute, neither at Microsoft nor at Google.
Who knows, may be a young PhD student is working on a prototype of the next big idea.
Could we predict 2012 breakthrough? Most people wouldn’t, unless they were in contact with the small group around the three forefathers of DNNs. If you were, e.g. if you talked to Ilya in 2009, he had good arguments about why it should work. Not scientific ones necessarily, but strong intuition. And their intuition turned out to be right.
The discussion about doom I find depressing. We should rather focus on finding how to manage risks of various kinds, not arguing endlessly about who is right. We need compromise, maybe both camps can get what they can live with even if they don’t love it.
Comment #152 December 18th, 2023 at 12:42 am
Scott #118:
Thanks for the reply! Yes, I would call myself a liberal in the old fashioned sense, in valuing free expression and ideas, and general mistrust of cultural warriors on all sides. I think there is a fertile middle ground for rational liberals, who not only update on the evidence of the USSR but on modern stagnation and inability to meet people’s basic needs, despite record productivity. We’re better than this, and half the solutions already work in other countries. Social Democrat is probably the right term, but I think being called a Kim Stanley Robinsonian sounds cooler!
I guess one difference with the AI rationalists is the difference in faith in technology; while I do generally agree technology’s power is going to be amazing, I also don’t think it is magical. The solutions to climate change seem technological and obvious enough — public transportation and clean or nuclear energy… not some yet-unforseen amazing untapped energy source, or CO2 scrubber with magical levels of efficiency and scalability… probably even the ASI will agree!
Comment #153 December 18th, 2023 at 6:46 am
I just want to thank you, Scott, for continuing to offer what strikes me as an appropriate level of epistemic humility with regards to how AI will impact the future. In particular, I think your “dark vision” of how divided opinion will remain long into the future (when relevant empirical evidence should be well and truly in) is right on the money.
Comment #154 December 18th, 2023 at 7:02 am
#129, #130
According to Tong, it is the entire Standard Model that hasn’t been (maybe can’t be) simulated on a computer.
“The difficulty lies with electrons, quarks and other particles of matter, called fermions. Strangely, if you rotate a fermion by 360 degrees, you do not find the same object that you started with. Instead you have to turn a fermion by 720 degrees to get back to the same object. Fermions resist being put on a lattice. In the 1980s Holger Bech Nielsen of the Niels Bohr Institute in Copenhagen and Masao Ninomiya, now at the Okayama Institute for Quantum Physics in Japan, proved a celebrated theorem that it is impossible to discretize the simplest kind of fermion.”
https://www.scientificamerican.com/article/is-quantum-reality-analog-after-all/
Comment #155 December 18th, 2023 at 9:01 am
James #154
The issue is specific to weak nuclear forces because chiral particles feel different forces only in case of weak nuclear force. The standard model cannot be simulated as a whole because some part of it (weak nuclear force) cannot be.
The SA article you referred is from 2012. Tong has some recent videos and articles where he provides specific details.
https://www.quantamagazine.org/what-is-quantum-field-theory-and-why-is-it-incomplete-20220810/
“It’s something like that topology allows for some of the phenomena, these anomalies that are required to see what we see in the case of the weak force, that a discrete space would not permit. That something about the continuum is key.”
And
Comment #156 December 18th, 2023 at 1:07 pm
Prasanna #155: I mean, it’s true that extending the BQP simulations of quantum field theories to encompass chiral fermions, as you have in the full Standard Model, is a major open problem in quantum algorithms. I’m just exceedingly skeptical that this is a fundamental problem for the computational nature of the physical world, as opposed to a technical problem for current lattice field theory techniques. After all, people do computations involving the Standard Model all the time! And I never heard a mention of any Penrose-like uncomputability that they ever encountered in doing so. Furthermore, even if there were a fundamental problem here, from a modern standpoint the Standard Model itself is “merely” an effective field theory, a low-energy limit of a yet-unknown theory that incorporates gravity (among other things). And what little we know about the latter theory strongly suggests that it has a finite-dimensional Hilbert space associated to any bounded region, which makes it really hard to understand how the sorts of continuum phenomena that David Tong is talking about could prevent computability.
Comment #157 December 18th, 2023 at 1:24 pm
#155, #156
Prasanna, thanks for the links.
I’m definitely out of league but the “proved a celebrated theorem that it is impossible to discretize the simplest kind of fermion” in what I quoted suggests to me that the problem is beyond a “technical issue with discretizing the weak force”.
Comment #158 December 18th, 2023 at 1:39 pm
There is a new controversy involving the beginnings of generative AI and specifically Sutskever. Tom Mikolov claims in his post after receiving the test of time NeurIPS award that Sutskever and Quock Le essentially plagiarized his sequence-to-sequence ideas: https://twitter.com/deliprao/status/1735923621116236173 .
Comment #159 December 18th, 2023 at 1:46 pm
Scott #156:
– “After all, people do computations involving the Standard Model all the time”: yes, with an approximation called perturbation theory and with effective theories.
The rest is lattice field theory, still fighting with the “Berlin wall”, although it has fallen, partly.
– “the Standard Model itself is “merely” an effective field theory”: I never read the proof of this theorem, nor I did see an experiment about this.
Still, I tend (like you I guess) to believe (yes: it is just a belief) that there is (as you say) not a fundamental problem. But I remark that this is a belief and the arguments you produced are far from proved.
Comment #160 December 18th, 2023 at 2:45 pm
If only all scientists and mathematicians were as graceful as you in self-correction. I aspire to achieve the same equanimity when correcting myself publicly.
“One must never use radical ignorance as an excuse to default, in practice, to the guess that everything will stay basically the same.”
I am not one for tattoos, but if I were, I should get that quote tattooed across my chest. Ideally, written upside-down, so that I could read it every morning.
Comment #161 December 18th, 2023 at 4:30 pm
#159
ideas are often a dime a dozen. Getting them to actually work is a different story.
There are people who think they are not getting the recognition for their old work on DNN and keep repeating it every year at NeurIPS, complaining about the attention and recognition that Hinton, et. el. have got.
From what I have heard Tom seems overemphasizing his own role in the DNN revolution. Anyone interested in details can likely talk to other parties that were present, e.g. Jeff Dean, who can give a bit more objective view of the period.
Comment #162 December 18th, 2023 at 5:36 pm
Scott #148:
These are not things I ever thought much about until fairly recently. I do agree that the modern pattern-finding/generating algorithms are capable of producing quite impressive results.
In 2007 Marie Farge called numerical simulation “a third way to study nature”: http://wavelets.ens.fr/PUBLICATIONS/ARTICLES/PDF/234.pdf I think she is right, and I think that the pattern-finding/generating algorithms may well qualify as a fourth. So they are an important development.
But it seems to me that animal/human general intelligence is quite a different beast (ha ha), one that requires both a powerful sensorium and a powerful armature, closely coupled in both space and time to a much larger physical environment. I am very skeptical that an AI with “goals” and “agency” is possible without this.
I could be wrong of course. But I find it surprising that this point of view never seems to get considered when supposedly “rational” people are running around shouting that we’re all doomed because ChatGPT is going to start maximizing paperclips any day now.
Comment #163 December 18th, 2023 at 5:49 pm
Mark Srednicki #162: Absolutely no one believes that ChatGPT is going to start maximizing paperclips. This is the kind of misconception of which one can get disabused in a few minutes, by reading the doomers themselves rather than the midwits who don’t understand (or care to understand) their position, and therefore doofus-ify it!
The argument is more like:
(1) The rocket-like trajectory that took us from where we were 5 years ago to where GPT-4 is right now, if extrapolated forward another 10 or 20 years, seems like it could plausibly produce AIs that are better than humans at pretty much every intellectual task.
(2) We’re already seeing that, for irresistible economic (and even military) reasons, these AIs will not just be inert question-answering systems, but will form the cores of autonomous agents that are let loose on the Internet, given control over physical machinery, and otherwise empowered to take actions in the world.
(3) This … seems like it might be dangerous and might end poorly for civilization! At the very least, appropriate safeguards seem called for.
I note that you didn’t answer the question about how far away GPT-4-like conversational abilities seemed to you in 2009 (or even, let’s say, 2018).
Comment #164 December 18th, 2023 at 6:42 pm
If one adopts the view that invoking (a) Prime Number Theorem and (b) the Riemann Hypothesis is a plausible way of centering the discussion, it might well be that what we are apt to witness way before the proof of (b) is AI having (and exercising) the capability to come up with conjectures (“visionary hallucinations”) of the depth comparable with (a) (which it took humans several thousand years of thinking about the primes) and this, in itself, might be borderline transformative (needless to say (b) is deeper).
Comment #165 December 18th, 2023 at 7:18 pm
James Cross #154 and #157, Prasanna #155: It’s been many years since I knew this topic well enough to comment intelligently, but here goes anyway. My understanding is that the theorem states that (under various assumptions) you can’t discretize a chiral fermion QFT on a lattice of the same dimension as the underlying continuum spacetime . But (I think) you can construct a lattice theory of one higher dimension that displays chiral fermion modes on its boundary (which is one dimension lower). This ideas goes at least all the way back to 1993 (https://arxiv.org/abs/hep-lat/9303005), although of course we’ve learned much more about it since then. So at worst, (I think) you can simulate a lattice theory with one more higher dimension than your “target” theory, where the larger theory has a lot of “dummy” nodes in the bulk whose only purpose is to allow for chiral modes to emerge on the boundary (which is the “real world” that you’re simulating). Moreover, this trick of adding an extra dimension should only incur a power-law overhead in computing resources.
Xiao-Gang Wen further claims that “all gauge- and gravity-anomaly-free chiral fermion theories can be put on a lattice in the same dimension” at https://xgwen.mit.edu/blog/solution-chiral-fermion-problem-or-not, although I don’t know enough to evaluate this claim.
In any event, there’s a big logical jump from the claim that “You can’t put a chiral fermion QFT on a lattice” to the claim that “Any physics described by a chiral fermion QFT is uncomputable”!
Comment #166 December 18th, 2023 at 7:26 pm
Everyone: OK, I just watched the lecture of David Tong that’s in the comment of Prasanna #155. A few thoughts:
(1) It’s a beautiful lecture, clear and full of insights. I’d recommend it to anyone, regardless of what quibbles I have with the broader conclusions that Tong draws.
(2) Crucially, Tong himself seems to view this as an unsolved research problem—how do you simulate chiral QFTs on a lattice?—rather than a fundamental impossibility. He mentions that “no-go theorems” are only ever as good as the assumptions they rest on, and that there’s a long history in physics of such theorems being evaded by changing the assumptions. This is exactly how I advocated viewing the situation in my comments above.
(3) Hearing Tong’s detailed explanation of the problem made me, if anything, more confident that this is a technical issue. So, Nielsen and Nanomiya show that you can’t have a fundamental chiral asymmetry on a lattice that doesn’t lead to a non-cancelling quantum anomaly. But what about a chiral asymmetry that arises from spontaneous symmetry breaking, as Tong points out that it does in string theory? Couldn’t that fix the issue, analogously to how the Standard Model itself (via the Higgs mechanism) used spontaneous symmetry breaking to evade Goldstone’s impossibility result? Even if not, there might be many other ways to simulate a QFT on a computer than by putting it on an anomaly-free version on a lattice—for example, via a sequence of theories that do contain anomalies, but where the anomalies can be made sufficiently small as to not affect whatever calculation you’re doing.
In the end, then, I reject the contention that this is able to tell us much about the truth or falsehood of the simulation hypothesis. And having listened to Tong’s lecture, it sounds like he himself doesn’t take that contention too seriously either (he described his title as shameless clickbait)! 🙂
Comment #167 December 18th, 2023 at 7:42 pm
Ted #165: Ah, thanks so much for that link! If the “going one dimension higher” solution works, then I’m extremely surprised that Tong’s lecture didn’t mention it. (But also, why did Jordan, Lee, and Preskill then seem to regard this as an unsolved problem in 2014?)
Comment #168 December 19th, 2023 at 2:10 am
There is an occasional discussion about infinite vs finite being fundamental, starting with the argument that a commutator of finite-dimensional linear transformations cannot be a multiple of the identity (which immediately follows from taking the trace). And so it ostensibly follows that the Heisenberg uncertainty principle can only exist in an infinite-dimensional space.
Sean Carroll wrote a paper taking a suitable redefinition of the canonical commutation relations that allows the uncertainty relation to survive as an approximation in a finite-dimensional space: https://arxiv.org/abs/1806.10134
Comment #169 December 19th, 2023 at 2:43 am
anon #161
> ideas are often a dime a dozen. Getting them to actually work is a different story.
Sequence to sequence neural translation and generation of text from neural language models are ideas worth billions today. Saying that they were in the air is like saying that quantum computing is in the air. Proposing them *and* pushing for them when the rest of the world considered them infeasible *and* coming up with the techniques that made them feasible and better than n-gram models seems to be Tom’s contribution, not Ilya’s and Quoc’s.
When in doubt, go to the sources. Wikipedia has Zweig’s review of Mikolov’s thesis. It is clear: https://www.fit.vut.cz/study/phd-thesis-file/283/283_o2.pdf .
Comment #170 December 19th, 2023 at 4:22 am
Scott #148: recalling as honestly as possible, I would have predicted large uncertainty then, which I still predict now. We don’t disagree on Scott 2009, rather we disagree on “the dramatic collapse of some AI timescales”. I’m somewhat against information cascades (updating just because people I respect have updated). I think there are many, many biases at play among rationalists and EAs (notwithstanding that they clearly do better than the general public, politicians, etc). So *my* timescales haven’t collapsed. Shifted a bit for sure. But still a lot of weight on “funding and research will dry up as improvements become more marginal”.
Before LLMs, I recall bad chatbots and unintentional markov chain humor. It is true that current developments are pretty unexpected from the PoV of even a few years ago. However:
(1) LLMs *still* seem to be something like Markov chains. I’m not an expert, but informally the context window and predict-next-token goal makes them pretty much a Markov chain with ML-determined weights. You do have to dig a lot deeper to expose their flaws nowadays, but they’re still there. See e.g. the difficulty in getting LLMs to solve easy sudoku puzzles by teaching them a brute-force algorithm, or how easily image models get confused by multiple subjects. I can easily imagine that all efforts to scale these things up to larger problems will just case one or more copies of the LLM to chase their own tails and get lost in the weeds over and over, as the cost of running the evaluation becomes more and more prohibitive.
(2) from an economic standpoint, I wonder how long AI companies can charge high monthly prices for a service that’s a novelty with unproven long-term value. Users might dry up, and with this, the incentives for spending hundreds of millions on huge training runs simply won’t be there. Google search was a huge revolution, which plateaued and then started decaying *purely through economic incentives*. Why shouldn’t similar incentives lead to the next AI winter, as the speculation bubble pops in the face of (economically) uninteresting marginal improvements? Why wouldn’t tech companies’ user-hostile goals eventually lead to enshittification of the models?
P.S.: just because I happen to be a “timeline skeptic” doesn’t mean I don’t respect all of you on the other side of this issue. In particular, I take the risk of AGI very seriously and am glad that we have people working on it.
Comment #171 December 19th, 2023 at 7:39 am
Scott #166
“truth or falsehood of the simulation hypothesis”
I think the simulation hypothesis is unprovable silliness anyway, but there is nothing in Tong’s idea of a continuum reality that would prohibit an analog simulation.
Still there is little actual evidence to support the idea that the brain is doing some kind of digital computation. The reason usually given is that neurons are either on or off (firing or not firing), but most of the theories about how information is actually transmitted in the brain relies on firings over time, either rates or averages, and that would suggest something more analog, although able to be modelled digitally.
Comment #172 December 19th, 2023 at 7:55 am
Scott #81
> (a) accept that a world-destroying AI God will probably soon be summoned into existence, in a way that’s radically disconnected from all the previous progress that there’s been in AI and will rely on secret insights that some self-taught genius will soon have in his or her basement
Frankly speaking, a “small” version of this scenario has already happened and we can witness its results. I’m talking about the Bitcoin creator and his child. It hasn’t destroyed the world yet, but helped unite international crime and let it flourish as never before. Scammers and fraudsters can now operate with billions so easily, that even honest people are becoming jealous to that. Hackers can do whatever harm and get paid for that without any worry. Terrorists can be funded without traces, from anywhere to anywhere. Drags, weapons, humans trafficking, etc., are now inseparable part of our progressive computerized world. Thanks, Bitcoin. You gave people “freedom”.
Comment #173 December 19th, 2023 at 7:56 am
You say “Even with hindsight, I don’t know of any principle by which I should’ve predicted what happened”, but you knew that Albert Einstein went from understanding precisely nothing in 1879 to being the first man to understand General Relativity in 1915, and you knew that the human genome only contains 750 megs of information, and yet that is enough information to construct an entire human being. So whatever the algorithm was that allowed Einstein to extract information from his environment was, it must have been much much less than 750 megs. That’s why I’ve been saying for years that super-intelligence could be achieved just by scaling things up, no new scientific discovery was needed just better engineering. Quantity was needed not quality, although I admit I was surprised it happened so fast because I thought more scaling up would be required.
John K Clark
Comment #174 December 19th, 2023 at 9:18 am
John K Clark #173: Knowing that an algorithm takes at most 750MB (!) to describe doesn’t place any practical upper bound on how long it might take to discover that algorithm! (If we rule out actually decoding the human genome itself, which in any case was not the approach taken in AI.) Even if there’s some argument involving the example of human intelligence that I should’ve made back in 2009, it certainly isn’t that one.
Comment #175 December 19th, 2023 at 9:48 am
Everyone: I’ve now had an extremely illuminating email exchange with John Preskill about the problem of putting chiral QFTs on a lattice. Maybe I’ll blog about what I learned when I feel ready to. Bottom line: this is not only a wonderful (and strangely underappreciated) research problem, but one on which there’s been recent progress — ironically, some of it due to David Tong himself! (Building on work Preskill was involved with way back in the 1980s.) A full solution soon doesn’t seem out of the question.
Thus, important and fascinating though the topic is, I don’t think we should take seriously at all the idea that chiral fermions pose a deep challenge to the Church-Turing Thesis. That idea is, as Tong himself put it, just “clickbait.”
Comment #176 December 19th, 2023 at 10:02 am
#175 Scott
Can’t wait to see your post onTong!!
Comment #177 December 19th, 2023 at 10:57 am
Scott # 174 says “Knowing that an algorithm takes at most 750MB (!) to describe doesn’t place any practical upper bound on how long it might take to discover that algorithm!” I say why not?
We know for a fact that the human genome is only 750 MB (3 billion base pairs, there are 4 bases, so each base can represent 2 bits and there are 8 bits per byte) and we know for a fact it contains a vast amount of redundancy (for example 10,000 repetitions of ACGACGACGACG) and we know it contains the recipe for an entire human body, not just the brain, so the technique the human mind uses to extract information from the environment must be pretty simple, vastly less than 750 MB. I’m not saying an AI must use that exact same algorithm but it does tell us that such a simple thing must exist. For all we know an AI might be able to find an even simpler algorithm, after all random mutation and natural selection managed to find it so it’s not unreasonable to suppose that an intelligence might be able to do even better.
John K Clark
Comment #178 December 19th, 2023 at 11:22 am
John K Clark #177: Come on! 256750,000,000 is vastly greater than the number of possibilities one could search through within the lifetime of the universe. An unbelievably large further reduction of the search space would be needed before one had any useful statement.
Comment #179 December 19th, 2023 at 11:47 am
Scott says #178 “Come on! 256^750,000,000 is vastly greater than the number of possibilities one could search through within the lifetime of the universe.” I agree, and yet it’s a fact that random mutation and natural selection managed to stumble upon it in only about 500 million years. The only conclusion that one can derive from that is there must be a VAST number of algorithms that works just as well or better than the one that Evolution found. And if it had found one that worked I’m certain intelligence can find one too and could do so in a lot less than 500 million years because evolution is a slow, extremely inefficient and cruel way to create complex objects, but until it finally got around to making a brain it was the only way to do it.
Also, 750 Mb is just the upper limit, the real number must be much much less.
John K Clark
Comment #180 December 19th, 2023 at 12:31 pm
John K Clark #179: You can certainly attempt an argument based on “evolution worked,” although there, too, the amount of computation involved seems mega-astronomical compared to even the amounts that are available today. In any case, though, the 750MB upper bound on the information content of the human genome is neither necessary nor sufficient, and has no clear relevance to the timescales for AI that I can see.
Comment #181 December 19th, 2023 at 1:49 pm
Scott, I agree the information content of the human genome is not sufficient to obtain intelligence, you also need information from the external environment, and in the case of a human you need enough information to make a brain and a body that can support it too. So I think we can be as certain as we can be certain of anything that it should be possible to build a seed AI that can grow from knowing nothing to being super-intelligent, and the recipe for building such a thing must be less than 750 MB, a LOT less. And I don’t understand why you think this fact has no relevance over the difficulty and thus the timescale over which to expect the development of a superhuman AI, in fact it’s hard for me to imagine anything more relevant.
As I mentioned before I never thought a major scientific breakthrough was necessary to achieve AI, just improved engineering, but I didn’t know how much improvement would be necessary, however today I think I do. Already at this very moment a computer can easily pass the Turing Test, so I expect that in five years the world will be unimaginably different from what it is today. It will be a very dangerous time and if the most powerful man in the world is an imbecile like Donald Trump then I think we’re sunk.
John K Clark
Comment #182 December 19th, 2023 at 2:16 pm
What’s wrong with being wrong? You lament it as if you did something bad indeed! But you just made a guess (about the future), it was obvious for all involved it was a guess, and the guess turned out to be wrong…
No, I did not expect the success of deep learning that happened, and actually was in denial for quite a bit when it started to take shape. I am not convinced still that superhuman AI is around the corner, but obviously I’ve updated my estimate of this probability from “nah, not in my lifetime” to “fuck, it may actually happen, but I hope not”.
I don’t have a number for p(doom), but my guess if “doom” happens it won’t be because the AI develops consciousness and desires of its own, or even because of paperclip maximizing. I think most likely it will be because some curious and irresponsible researcher will find a way around alignment and prompt filtering to ask the superhuman AI, “how would you destroy the human civilization?”, and the AI will give a live demonstration.
Comment #183 December 19th, 2023 at 3:27 pm
#169
In these kind of situations, one can always find various perspectives to cite. The person you cite it seems was from the time that Tom was at Microsoft. We are discussing events before that when Tom was at Google, and there are a lot of reputable people at Google from that time period that can give you their first hand assessment of Tom’s claim that Ilya and others stole his ideas.
And any patents (and keep in mind that ideas are not patentable) would have belonged to Google anyway, per standard tech company contracts.
Yes, ideas are a dime a dozen, it has always been so in tech. Who thought about some idea first does not matter much.
If Tom is a great researcher, he is going to continue to do amazing work. If not, the rest doesn’t matter much really.
Comment #184 December 19th, 2023 at 5:00 pm
Scott #163:
“Absolutely no one believes that ChatGPT is going to start maximizing paperclips.”
That was sarcasm. (Yes, I forgot that sarcasm should be specifically notated here on the interwebs.) But here is the real thing:
“The key issue is … what happens after AI gets to smarter-than-human intelligence. .. it currently seems imaginable that a research lab would cross critical lines without noticing. … the most likely result of building a superhumanly smart AI, under anything remotely like the current circumstances, is that literally everyone on Earth will die.”
https://time.com/6266923/ai-eliezer-yudkowsky-open-letter-not-enough/
Scott: “I note that you didn’t answer the question about how far away GPT-4-like conversational abilities seemed to you in 2009 (or even, let’s say, 2018).”
I did answer the question: I had no particular opinion on it before the thing itself actually appeared. Today, I think it’s very impressive.
Responding to your numbered points:
(1) Maybe. I’m skeptical. I refer you to my preferred benchmark, which you have agreed is unknowably far off.
(2) References? What physical machinery is under the control of an LLM?
(3) If your point is that complex systems can fail in surprising ways, then I completely agree. The evidence for this is overwhelming. So we ought to vet critical complex systems as completely as possible. But if your point is aligned (ha ha) in any strong way with Yudkowski’s “we’re all going to die” pronouncements, then I’d really like to see some evidence that AGIs with agency are possible at all via a construction process that is radically different than the one that produced the one and only example of GI that we currently have.
Comment #185 December 19th, 2023 at 5:53 pm
anon #183:
> The person you cite it seems was from the time that Tom was at Microsoft. We are discussing events before that when Tom was at Google, and there are a lot of reputable people at Google from that time period that can give you their first hand assessment of Tom’s claim that Ilya and others stole his ideas.
Plain wrong. Most people who know something about this also know that Mikolov was at Microsoft prior to his work at Google. It is easy to find out and you would see it if you at least cared to read the date on the Zweig’s review I linked.
Most people who know (or care to learn) about this, also know that Mikolov’s PhD work on RNNLM predates his employment at Google.
> And any patents (and keep in mind that ideas are not patentable) would have belonged to Google anyway, per standard tech company contracts.
Again wrong. Mikolov’s PhD work on RNNLM that contains not just the ideas but also implementations and experiments was not done at Google but before that.
Perhaps you could do some basic fact-checking first and read it before adding more uninformed comments? Again, courtesy of Wikipedia: https://www.fit.vut.cz/study/phd-thesis-file/283/283.pdf
Comment #186 December 19th, 2023 at 6:07 pm
First, I had no inkling of what was come in AI in 2010. Nor in 2020 for that matter.
I personally put p(doom) at about 50%. It’s based on the theorem that “No sufficiently powerful AI has an off-switch”. That’s because a sufficiently powerful AI could convince powerful people that turning it off would be a harmful to them, and then they will protect it. Or similarly, it would make itself useful to the military, who would then defend it. Or it could blackmail its keepers into keeping in on, with or without facts to back up the blackmail. It could convince a majority of citizens to elect it, and keep electing it. And so on – anything a smart, unscrupulous, and dedicated person could do, a powerful AI could do. And we’ve seen what smart and amoral people have done, and continue doing.
This is closely related to the quote “It is difficult to get a man to understand something, when his salary depends on his not understanding it.” Basically, if the AI is providing benefits to the currently powerful folks, they won’t let it be turned off, even it’s possible to do so.
This is also related to the situation where no-one has found an off-switch for (say) Putin. In theory there are many off switches – he could be voted out, assassinated, deposed, etc. But in practice he is still in control. The only power he has personally is talking to others, and convincing them that protecting him is in their best interest, but that’s enough.
Comment #187 December 19th, 2023 at 6:16 pm
Scott,
You were concerned in another post about the attitudes of the young adults 18-24 in the USA, that they support Hamas, based on the polls. I told you not to worry because people that age are idiots.
It turns out I was right – according to a December Harvard-Harris poll, 45% of registered voters aged 18-24 believe that Hamas allows homosexuals to live together openly. So the youth of America are not evil. They are just idiots. There is hope for this great country.
Comment #188 December 19th, 2023 at 7:03 pm
Mark Srednicki #184: Yes, Eliezer is on the extreme end of the doomer spectrum—that’s his whole role in the world! There are many other doomers, like my former student Paul Christiano (now head of the Alignment Research Center), who express themselves much more guardedly, with fewer prophetic leaps into “therefore everyone dies,” but who still see a good chance that delegating more and more decision-making to poorly-understood artificial superintelligences is unlikely to turn out well for humanity.
You can watch an amusing video of a GPT-enabled robot here. I mean, yes, it’s just a toy, but so was the web 30 years ago! Meanwhile, militaries around the world now have groups studying possible uses of LLMs in analyzing military intelligence, managing drones, and so on. Not deploying yet, as far as I know … but again, the Stuxnet attack would’ve sounded like science fiction a few decades before it was carried out! Given the incentives, I think the burden is now squarely on those who claim that AI battlefield decisionmaking is not in our future, not those who claim it is.
Comment #189 December 19th, 2023 at 7:21 pm
Craig #187: Yeah, many pieces of evidence have emerged recently to suggest that American 18-to-24-year-olds, who if you took them literally would seem to be thirsting for a second Holocaust, are really just breathtakingly, unbelievably stupid and ignorant of the most basic facts—which of course is consistent with some of them also being antisemites, but which also provides an alternative and somewhat less horrifying explanation for their behavior.
For example, a political scientist at Berkeley reports in the WSJ that, according to a study he did, of the students who endorse “Palestine will be free from the river to the sea,” the majority not only can’t identify which river (the Nile? the Amazon?) and which sea, they believe themselves to be endorsing a two-state solution (!!). The majority actually moderate their stance when shown a map with the Jordan River, the Mediterranean Sea, and all of Israel besides the Negev desert in between them.
We could say something similar about the swathes of young voters who, according to the NYT, are enraged by Biden’s support for Israel and are therefore now planning to vote for Trump instead (!). Whichever side you’re on, there’s no way to explain such voters without invoking a level of stupidity beyond imagination or parody.
Comment #190 December 19th, 2023 at 8:20 pm
Scott #189: Doubtless “the ignorance of the most basic facts” tends to decrease with age, but I doubt that this decrease alone can account for the dramatic difference in the level of support of Hamas between 18-24 years olds, their parents, and, especially, their grandparents.
Comment #191 December 19th, 2023 at 8:31 pm
AG #190: The simplest explanation I’ve read is simply that older voters know that Israel was attacked from every direction by Arab armies bent on completing the Holocaust rather than coexistence, they know that Israel accepted peace proposals while the Palestinians turned them down, etc. etc., because they were alive when these things happened. The Zoomers, by contrast, have only been alive while Israel was most associated in the West with Netanyahu and his crazy far-right settlers and theocrats, and crucially, they have no interest in learning the whole story of how things got to this point.
Comment #192 December 19th, 2023 at 9:02 pm
Scott #191: This line of reasoning would also apply, for example, to the levels of support for Ukraine, with the “parents” and “grandparents” being presumably much more apprehensive of the USSR/Russia. However the levels of support in this case seem much more uniformly distributed throughout the cohorts.
Comment #193 December 19th, 2023 at 11:51 pm
#185
I got the order of his internships wrong. He was at Microsoft before Google.
The rest of my statements still stand.
As a part of employment agreement in a big tech company as an intern, he has signed a legal document listing all stuff before that employment that he wants to claim for himself. If his claims are true, he should be able to produce those docs and can also contact the journal and officially claim the plagiarism to be investigated. If he is not doing that but just voicing these on social media publicly, he is doing it wrong.
Ideas are not patentable, they are shared and used openly, that is how OpenAI used Google’s Transformers, etc.
There are reputable people from Google from the contested time period, and if you are interested you can talk to them to get the full picture, and their take on Tom’s claims of others stealing his ideas.
If he wants to pursue slandering others with theft of ideas, he is free to do so, but people who pursue that path seldom get what they seek, they often become disgruntled and unproductive people. It was sad to see that he is using the event of their joint paper receiving the test of time award to voice these after a decade, it looks like he is already on that path.
Comment #194 December 20th, 2023 at 6:31 am
Mark Srednick # 184
You say “I’d really like to see some evidence that AGIs with agency are possible”, but what evidence have you seen that has convinced you that human beings have “agency”? There are only two possibilities; I did what I did for a reason in which case it was deterministic, or I did what I did for no reason in which case my action was random and UNreasonable. So I’m either a cuckoo clock or a roulette wheel. And free will is an idea so bad it’s not even wrong.
John K Clark
Comment #195 December 20th, 2023 at 6:39 am
Looking forwards to the chiral fermions on a lattice post Scott! Will be great to relearn what I’ve forgotton, and learn new stuff. If you haven’t already, check out Tongs lecture notes. They are excellent and used by students the world over!
Comment #196 December 20th, 2023 at 8:35 am
There’s a chapter on lattice gauge theory in Tongs Gauge Theory notes (450 pages!) available on homepage.
Comment #197 December 20th, 2023 at 9:18 am
Regarding polls.
Polls have been wrong on a lot of things in last few years and it seems they are getting worse.
I’m wondering if disaffected, low-info participants are being overrepresented and some of them may even be deliberately misrepresenting themselves – black instead of white, Democrat instead of Republican, non-Christian instead of Christian. The MAGA mind might blast its views all over X and TikTok, but think they should lie to a pollster to “protect their privacy”.
Comment #198 December 20th, 2023 at 9:57 am
I have summarized some thoughts on your question and my (flawed) initial argument here: https://substack.com/profile/75908212-misha-belkin/note/c-45542955
Thank you for a thought-provoking post.
Comment #199 December 20th, 2023 at 12:24 pm
anon #193:
> There are reputable people from Google […] . If he wants to pursue slandering others with theft of ideas, he is free to do so, but people who pursue that path seldom get what they seek, they often become disgruntled and unproductive people. It was sad to see that he is using the event of their joint paper receiving the test of time award to voice these after a decade, it looks like he is already on that path.
His behavior may be indeed alien to some of the Silicon Valley culture. “Fake it until you make it”, “money talks” and “move fast and break things” is for some quite normal and in some places more normal.
Have you heard of Perelman? Did he end up disgruntled and unproductive? Did he go up against “reputable people”? Perhaps he should have just gone with the flow, take the money and avoid rocking the boat?
Or maybe when calling out the plagiarists and their supporters he did the right thing any honest scientist should have done? (but only a few have the guts for?) And in the long run, did it help or did it not help his community?
Was Perelman’s act “sad to see”? And is it he or the others who are the sad figures in that act?
Comment #200 December 20th, 2023 at 1:26 pm
On the main topic question, I didn’t foresee the current boom but, if someone had told me then what can be done with AI now even twenty-five years ago, I wouldn’t have been surprised.
I would like to understand what objective criteria we plan to use to determine human-level AI and I expect it needs to be more than a Turing-test-fool-the-human criteria.
Let’s try one: Let’s put AI on the savannah of Africa with a robotic body and see if it can survive longer than 21 days (naked and afraid so to speak). It does have to do things that humans might do like navigate a tough terrain, find water, procure resources, etc with no oil changes or extra charge-ups. Its robotic body should have roughly the same physical capabilities and deficiencies as humans. It can’t have a shell that protects it when the rhino gores it, for example. But it can move just as fast as a human – in other words slower than almost every other animal on the savannah.
Comment #201 December 20th, 2023 at 2:26 pm
Scott #188:
It seems to me that you’re engaging in a bit of goalpost-moving yourself when you go from the timing of the advent of “superhuman AI” (the topic of your original post) to “AI battlefield decision making” in the near future. I full agree that the latter will happen. I would even say that it’s ALREADY happened in the form of self-guided missiles. But IMO that’s about a zillion lightyears from “superhuman AI” with agency, which (as I keep saying) I suspect is IMPOSSIBLE without strong coupling to a physical environment (the method that produced the only known examples of GI).
Comment #202 December 20th, 2023 at 2:53 pm
James Cross in # 200 suggests we replace the Turing Test with a test of his own “Let’s put AI on the savannah of Africa with a robotic body and see if it can survive longer than 21 days (naked and afraid so to speak”.
But by using that criteria neither you nor I nor anybody we know could be considered to be intelligent beings, the only people that are intelligent are African Maasai tribal members and Australian aborigines. I find nothing wrong with the good old fashion Turing test, I think it works fine just as it is.
John K Clark
Comment #203 December 20th, 2023 at 3:31 pm
Mark Srednicki #201: Well, the further argument the doomers make is that, all our intuitions to the contrary, there’s nothing special about “human” in the space of possible ability levels, and we should expect AI to blow past that level in a matter of years or months (as usual, Eliezer goes further and says days 🙂 ), exactly like what we saw happen with chess and more recently Go. Ie, once we’ve conceded that AI could soon do comparably to humans in all sorts of domains including military planning, why not just as well? Once we’ve conceded just as well, why not better? Once we’ve conceded better, why not unimaginably better?
Again, I spent a large fraction of my life finding such arguments tiresome and looking for ways to avoid taking them seriously. I was being asked to accept the existence of this gigantic new force shaping the future even though I couldn’t see it, and indeed saw virtually no empirical evidence for it, purely on the basis of chains of informal reasoning by a few contrarian nerds that might be shot through with mistaken assumptions.
But now the gigantic new force has actually shown up in reality, pretty much as the contrarian nerds said it would (even though they didn’t get all the details right). So, now that that’s happened, I might still disagree with the contrarian nerds on all sorts of particulars, but I’m finished with looking for ways to avoid engaging the argument entirely.
Comment #204 December 20th, 2023 at 4:14 pm
John K. Clark @ #194: Just so you know that there are reasoned, contrary opinions.
Currently I prefer the legal definition of free will, as in, did you sign the contract of your own free will, or did somebody hold a gun to your head. That is, for me, free will is the act of making a decision via determinism–to determine the best possible course of action for yourself among alternatives based on what you predict the consequences of those actions to be–and being accountable (responsible) for such decisions.
Determinism does not force me to make decisions, it provides the means for me to make decisions.
Without that point of view, no one has any responsibility and society cannot function.
Granted, humans do not always use this capability, wisely or at all. Some have only limited ability to do so, due to lack of brain development (children), brain malfunctions, or lack of data. They are held less responsible for their actions.
A computer program that only makes decisions that have been specifically coded in it is not held responsible for those decisions, the programmers are. A computer program that has only been programmed with the ability to learn how to make good choices, e.g. AlphaGo, does have responsibility for its choices. AlphaGo’s developers did not beat the world champion (who said of a novel move by AlphaGo, “I never thought a machine could beat me, but when I saw that move, I knew I would lose the tournament”), AlphaGo did.
The neuroscientist Steven Novella, in a recent blog post, says that from a neuroscience point of view, humans have partial free-will, in that “We are powerfully influenced by the subconscious circuits in our brains which have their own evolved purpose.” However, “But we do have what is called “executive function” – the ability to consider our options, weight the consequences, and make a deliberate decision, even one that countermands what the subconscious circuits want us to do.”
Comment #205 December 20th, 2023 at 5:27 pm
Scott: fantastically clear “three minutes” on Shor’s in the podcast with Krauss. No, I couldn’t repeat it back to you, but the “two-pronged” coverage (the math periodicity plus Fourier) got me well into the starting gate. Thanks
Comment #206 December 20th, 2023 at 5:52 pm
Scott #203:
I think we’re largely talking past each other now, so this will be my last comment. IMO, in any real-world situation (such as military planning), “unimaginably better” likely does not exist; that’s why it’s unimaginable! As Mike Tyson so memorably put it, ““Everyone has a plan until they get punched in the mouth.”
Comment #207 December 20th, 2023 at 7:32 pm
I was the one that tipped people off about this interview’s prediction in the comments of the p(doom) >> 2% tweet, learning to metastable linking the post and then Liron creating the clip. I tried to both bring it up and defend against ridicule and I think was fair but maybe should have known others might mine it for dunks. This is Twitter after all. Luckily I don’t think the ridicule has been very harsh and I think most of the people ridiculing are still fond of you. But overall, sorry if I caused you stress around this. PS: my pdoom is ~85% and I didn’t foresee Generative AI at all
Comment #208 December 20th, 2023 at 10:23 pm
Michael M#117 – It’s late in the thread, and I don’t want to go on too much, but on the rationalist/Libertarianism overlap I think it’s a matter of having similar types of extremely simplistic abstract models of the world, and then getting very defensive about the failures of those models (of course the latter is a common human trait, the overlap comes from the particular kind of bad mental model). I’m not a big fan of the identity politics framework, but there’s some truth in it, and both ideologies tend to draw from well-off people who are overall “privileged”, and have that perspective. This is very evident in the Libertarian hatred of antidiscrimination law, and, well, the parallel rationalist aspect I won’t name. A very long time ago, I wrote the following essay which you may enjoy:
http://sethf.com/essays/major/libstupid.php
#152 – The sad problem is that “rational liberals” don’t get funded and promoted much. To be on-topic, you can see that in the AI discourse. The “ethics” side has its faction. The “doomers” are good press, and I suspect are being boosted as Useful Idiots as a distraction against the “ethics” faction. That last idea is likely to get me some flak, which is sort a case in point of the problem with models. There’s not much of a powerful constituency for anything between the two. It’s a sort of a recursive problem, there’s a lot of solid theory about why this happens (“preferential attachment/power-law”). But it doesn’t say how to solve it.
Comment #209 December 21st, 2023 at 1:39 am
#199
Comparison with Perelman is not valid. Even if it was valid, Perelman had essentially left the field and has stopped doing mathematics and reportedly finds it painful to talk about math.
The path to deal with these kinds of disputes is clear as outlined in my previous comment. If those who have first hand knowledge of the period and are considered reasonable people in the community do not agree with his claims about others stealing his ideas, it is most likely that his claims are false but he has trouble grasping it. And people who really don’t care about fame and attention and money wouldn’t bother complaining about them. The fact that he is complaining on the occasion of their joint paper winning a prestigious award indicates that he cares about those things and hasn’t been able to get over it after a decade, but is not honest with himself about it.
Comment #210 December 21st, 2023 at 6:21 am
[…] scenario and a higher likelihood for an existential catastrophe that involves AI in some way but ultimately argues that making a firm prediction would require a discussion about what it would mean for AI to play a […]
Comment #211 December 21st, 2023 at 7:14 am
#202 John K. Clark
The Turing test only tests verbal skills. Twenty one days in the savannah tests vision, hearing object recognition, reasoning in unpredictable situations, ability to optimize with scarce resources … and it would be a test in a real environment similar to which we think humans evolved.
But the bigger point is that without a clear measurable standard there’s no way to begin to gauge where we are or how far away we are from human-like AI.
Intelligence probably conforms to some Pareto or power law principle. We can get a lot for very little but each additional step requires a lot more. When we are 80% to Human-I, the next 20% is going to be a lot harder. But without out a clear measurable standard we don’t know where we are or where the practical top limit is. We may only be 1% to Max-I.
Comment #212 December 21st, 2023 at 7:40 am
@James Cross, #200: You say:
Let’s try one: Let’s put AI on the savannah of Africa with a robotic body and see if it can survive longer than 21 days (naked and afraid so to speak).
I agree with John Clark #202. The test you propose is not fair to the robot.
How many humans could pass that test? Not many. Those trained in special forces in the military might be able to do it. Other experienced outdoors types might, as would people who’ve grown up in such an environment. The rest of us would be in big trouble and probably would not survive.
The humans who can live in extreme environments – the Arctic North is another good example – can do so because they’re grown up in them and know how to survive. But then, how would someone raised in such an environment survive in New York City? They can’t speak the most prevalent languages, have no money, know nothing about it, nor ATMs, etc. They’re be in trouble. Luckily for them, however, someone would surely notice their plight and they’d get help, assuming they didn’t attack first.
But, yes, it’s an interesting idea. What would be a reasonable way to run a test such as you propose? We want to give the robot a way of acclimating itself to the savannah environment, learning its way around, and then put it to the test. What would it take for us to build such a robot? I know that Tyler Cowen has proposed gardening as a challenging task for a robot.
Comment #213 December 21st, 2023 at 8:15 am
JimV @ # 204. You say the legal definition of free will is signing a contract of your own free will (?!), not because somebody put a gun to your head; but like most definitions in law, that is circular. You also say your personal definition of free will is, the act of making a decision that determines the best possible course of action for yourself among alternatives based on what you predict the consequences of those actions to be; BUT then you should be held accountable for the contract even though I put a gun to your head because you determined that signing a bad contract was preferable to having a bullet in your brain.
You say that determinism does not force you to make decisions, instead it provides the means to make decisions, but I don’t understand what that means. There was a cause, a reason, for you choosing to turn left and not right, in which case your decision was deterministic, OR there was no reason for you making the decision that you did, in which case your action was random. If you behave in a way that I find to be very strange and surprising I may ask “why did you do that ?”, if you reply “I don’t know” or ” I did it for no reason” I would be justified in concluding your behavior was UNreasonable and perhaps even crazy.
You say without your point of view about free will society cannot function, but I don’t think that’s true at all. You can’t talk about responsibility without also talking about punishment because the two things are inextricably linked. So what is the purpose of punishing a murderer? I think there are only two legitimate reasons for punishing anybody for anything:
1) To prevent them from committing a similar act in the future.
2) To act as a deterrent to prevent others from committing such an act.
Anything more than that is not justice, it’s just vengeance. I’m no different from anybody else and sometimes I’d like vengeance, that is making somebody I don’t like suffer simply for the pleasure of inflicting pain on him, but I am not proud of that reptilian part of my brain and so I will not defend it. Therefore from a legal point of view it shouldn’t matter if somebody is a murderer because he had bad genes, or bad upbringing, or because a random cosmic ray destroyed the crucial part of his brain that generates empathy for his fellow creatures. The important point is regardless of the cause he remains a murderer, a contagious misery spreader, and thus needs to be dealt with accordingly. The only legitimate mitigating circumstance would be if it could be proven that the murder occurred because of extremely unlikely circumstances that were very unlikely to be repeated. We should assume he is likely to murder again unless proven otherwise, and that would not be easy to prove.
Alan Turing proved that sometimes the only way to know what a 100% deterministic computer will do is to watch it and see, and you might need to watch it forever. Therefore, I think the only legitimate definition of “free will” is the INABILITY to always know what you will do next until you actually do it.
John K Clark
Comment #214 December 21st, 2023 at 9:00 am
When we do get Superintelligence, isn’t is upto IT, i.e the Superintelligence to decide how to mold the future, rather than humans having any say in it at all ? Now there are few scenarios we can contemplate
1. (Doomers) – Any superintelligence anywhere is humanity’s last problem
2.( Optimists) – Any Superintelligence is Utopia for humanity, and all life on earth.
3. (??) – Any superintelligence anywhere is end of human choice for sure. Even if it decides to not interfere in our affairs at all, that will be IT’s choice, not ours
While most people with deep knowhow of this expect to reach AGI within a decade, the rest of us need to make use of the time we have to the best of our ability
Comment #215 December 21st, 2023 at 9:49 am
[…] Scott Aaronson asks how he got his timelines for AI so wrong. His central conclusion? […]
Comment #216 December 21st, 2023 at 10:04 am
Benson, Clark
You are really underrating human abilities to survive.
Have you actually ever watched Naked and Afraid? Of course, it’s TV so there are some advantages, but still some people don’t make the 21 days. Still there are all sorts of survival stories from the real world – people who have run off their own arm to get it out from a boulder, for example.
And I am allowing the AI to be trained.
I think you would get a good sense of the intelligence of the AI in a real world situation. You seem to want some sort of disembodied intelligence that can do higher math when some number of intelligent humans can barely do double-digit arithmetic.
Comment #217 December 21st, 2023 at 12:54 pm
Cross# 216 : if you dropped an average Princeton mathematics professor naked and afraid in the middle of the Kalahari desert 50 miles from the nearest human being, do you think he’d still be alive in 24 hours? I don’t. But a Bantu bushman would be just fine.
John K Clark
Comment #218 December 21st, 2023 at 1:50 pm
Really a wonderful collection of ideas in this thread. asdf’s proposal for “Oppenheimer, The Sequel” should definitely be funded and NoGo’s identification of the likely eradication scenario very funny. The Tong link and further discussion very interesting new ideas for me. The entire thread is ripe with ideas.
When I consider the reasons for my bias to low doom I believe much of it is due to my observation and prejudice that the future is never as dramatic as its most dramatic a priori portrayals. There are doom clocks ready to strike midnight and irrefutable signs that end of times is upon us and exponential rises in temp, etc, etc, etc. The future just doesn’t seem to cooperate with maximum human drama. I am fortunate to see these developments in my lifetime and expect we will muddle through as we always have.
Comment #219 December 21st, 2023 at 2:23 pm
This is The Age of Tipping Points-catastrophic discontinuities littering the future that only a select few scientists are capable of recognizing a priori. If you ignore their rhetoric the consequences will be…well…catastrophic…civilization ending…extinction level event.
Comment #220 December 21st, 2023 at 7:28 pm
@#216 (James Cross):
I don’t know how well humans would do on that test, but I have no doubt that a lot of animals would do quite well on it. So I don’t think it works well as a test of intelligence; it’s more of a test of adaptedness to a particular environment.
Comment #221 December 22nd, 2023 at 4:34 am
#217 John K Clark
The Bushman is trained and I said the AI could be trained too. So, they should start more or less equal. The Princeton professor is trained only in survival in academia.
#220 Ben Standeven
How have humans learned to adapt themselves to almost all earthly environments?
Intelligence. And that’s what the test is about.
I admit we may need to engineering advances to create the robot body but I find it strange that anyone would suggest that anything about this test would be unfair to the AI. Are we really saying that the AI can’t cut it in the real world? Well, maybe we are.
Comment #222 December 22nd, 2023 at 6:43 am
I thought about a recipe for a super-intelligence and thought the following might work-
1 Part Marcus Aurelius (an infusion of duty and respect for others)
1 Part Gauss (self evident-mathematics)
1 Part Bohr (self evident-QM)
1 Part von Neumann (self evident-polymath)
1 Part Asimov (subject matter expert)
1 Part Feynman (self evident-exuberance)
1 Part Einstein (self evident)
Oh, the adventures we’ll have.
Comment #223 December 22nd, 2023 at 2:15 pm
The Navy just accepted delivery of the first ultra large unmanned underwater vehicle. It can operate autonomously for months at a time, will have autonomous “battlespace awareness”, and will be armed. So much less expensive when you don’t have to provide utilities and logistics support for a manned crew.
https://www.naval-technology.com/projects/orca-xluuv/?cf-view
Comment #224 December 23rd, 2023 at 7:29 am
Cross #217 says ” The Bushman is trained and I said the AI could be trained too. So, they should start more or less equal. The Princeton professor is trained only in survival in academia”:
I agree that physically surviving in the Kalahari desert is intrinsically more difficult than keeping a job in academia, but the test you’re proposing is grossly impractical (not to mention unethical), by the time you actually manage to figure out how to perform it the AI would already have evolved into a Jupiter brain. The only thing wrong with the good old-fashioned Turing Test is that some people don’t like the fact that a computer has managed to pass it with flying colors; yes sometimes AIs do some very stupid things but then so do human beings. And besides, in the long run it’s not important if people think computers are capable of being conscious, but it IS important that computers think people are capable of being conscious, because then maybe they’ll be nice to us. Otherwise …
… and so just before he was destroyed and knew for certain he was about to enter oblivion forever, the last member of the human race turned to Mr. Jupiter Brain and said “I still think I’m more intelligent and conscious than you are”.
John K Clark
Comment #225 December 23rd, 2023 at 9:01 am
Craig #187
I don’t follow this line of reasoning. Ivy League students are hopelessly stupid?
These students have received an education overbalanced to simple leftist ideologies and so facts are just irrelevant. I hear people say X will not win an election because the facts are Y. I see no purpose in arguing but think to myself that this is naive and facts are no longer relevant to a large percentage of people in the US. They are simply driven by simple ideological rules of thought. The anti-colonization mania is an example of this, no facts are pertinent to the strongly held beliefs that find solace in slogans.
Comment #226 December 23rd, 2023 at 11:12 am
Scott: towards the end of your very interesting podcast with Krauss you say something along the lines (if I understood you correctly) that “current language models are already capable of figuring out to lie in pursuit of the set goal”. What is the reference for this experiment/finding?
Comment #227 December 23rd, 2023 at 2:23 pm
if Wolfgang Pauli were to respond, he would say “AI ist nicht einmal falsch”.
In other scientific pursuits, it is acceptable for laymen (but not the scientist) to think, e.g.:
– An atom is like a solar system, with electrons revolving around the nucleus
– or QC/QM is massively parallel execution of some complicated processing
etc. etc.
If AI is to be taken seriously as a scientific field, I think it is probably very important that the computer scientists themselves stop promoting the anthropic aspects of AI. It does not feel, understand, or have biases–and is certainly not intelligent or conscious. We don’t even have good (scientifically appealing) questions about the I in AI— let alone predict when such a monumental feat might be accomplished. Apparently, great James McCarthy thought it would be over one fine summer in Dartmouth.
It (AI) simply computes………. like all other software or computing devices.
E.g: LLMs or rather Transformers are better designed to compute in a certain very specific way, i.e completing a sequence, given a vectors embedding space of words/tokens
But alas I am layman as well, and do look up to the scientist to separate the humbug from the objective inquiries. But when you hear Turing award winners like Hinton claim, and I paraphrase, “Skynet is already online ,almost”, I feel like that whole field (of CS) is willing to take few a steps back –I feel sadden and discordant.
Comment #228 December 24th, 2023 at 5:23 am
> Here’s what I want to know in the comments section. Did you foresee the current generative > AI boom, say back in 2010? If you did, what was your secret? If you didn’t, how (if at all) do > you now feel you should’ve been thinking differently? Feel free also to give your p(doom),
> under any definition of the concept, so long as you clarify which one.
Certainly not in 2010. But I remember thinking after AlphaGo/AlphaZero in 2016/2017 that networks very small by comparison with anything biological could apparently do surprisingly powerful reasoning (in the sense that they could emulate the deep judgement calls and occasional deep reading beyond the MCTS search horizon that seems intuitively necessary to play Go/chess at the highest level). In 2018 I thought (very cautiously) that something roughly like GPT-3 ought to be possible with massive effort, after neural machine translation had shown that systems of this type can handle very broad world knowledge quite well, but I would not have guessed that essentially a next token prediction training target and online inference essentially without branching search would be sufficient to get to that level of performance. Also, GPT-4 would still have surprised me then.
My p(doom next fifty years) under a full AI takeover scenario where, somewhat analogous to what happened historically in the oxygen catastrophe about 2.4 Gya, the current biosphere or at least its intelligent bits are replaced by a new type of life is quite low (< 1 percent), because I expect that AGIs, when it will become possible to build them, will at least in the beginning still have robustness disadvantages compared to humans (because animal brains have been field-tested for hundreds of millions of years), be less agentic and more dependent on large-scale infrastructure for their replication and the expansion of their hardware base, as well as have to deal with a complex world already inhabited by billions of humans. In addition, even vastly superhuman intelligence will not allow an AI to bypass physical and mathematical limits on its ability to influence the real world: some problems just do not have massively better solutions than humans can find. I would be optimistic that safety research supported by empirical study of frontier systems (and eventually supported by the intelligence systems close to or at AGI level will eventually offer) will be able to manage/mitigate the risks, if it is sufficiently funded.
That said, obviously one does not need AGI in order for AI to significantly increase existing cybersecurity/computer security/computer systems safety issues. But that holds on the defense/mitigation side as well.
Comment #229 December 24th, 2023 at 12:33 pm
Scott #175:
I’m now very confused about two basic epistemic questions about the Standard Model:
1. Do we have a well-specified algorithmic procedure for analytically generating experimental predictions to arbitrary precision (even if doing so is computationally infeasible in practice)?
2. Do we have a well-specified algorithmic procedure for putting the SM on a lattice and numerically simulating experimental predictions to arbitrary precision (even if doing so is computationally infeasible in practice)?
Before I watched Tong’s video, I had thought that the answer to question 1 was “no” (because the SM is not formulated in a mathematically rigorous way, issues with non-perturbative phenomena, etc.) but the answer to question 2 was “yes”. But now I’m wondering if I was wrong on one or both counts.
If the answer to Q1 is actually “yes”, then it seems to me that the laws of physics seem “computable/computer-simulatable”, regardless of whether or not we can formulate a lattice theory. Computers can do sophisticated symbolic/analytical manipulations.
Or if the answer to both questions is “no”, then it seems to me that the whole SM is starting to push the boundaries of being a scientific theory at all! If we can’t fully define either the continuum or the lattice version of the SM in an algorithmically effective way, then I don’t know if I’d call it a true scientific theory.
I look forward to learning any info you might be able to share on this topic in a future blog post.
Comment #230 December 24th, 2023 at 1:18 pm
Ted #229: I believe the answer to both of your questions is “it’s complicated.” 😀
For 1, we have a “reasonably” well-specified procedure for perturbative predictions (modulo the issues of divergence when you sum too many Feynman diagrams), which suffices e.g. to test Standard Model predictions to enormous precision against LHC data.
For 2, we don’t currently have a way of putting the full Standard Model on a lattice because of the fermion doubling issue, but that’s a technical problem that looks plausibly on the verge of being solved. (Or more likely, it’s already “solved,” just without a satisfactory proof that the solution works.) Of course, here we’re also leaving aside that the limit of arbitrarily small lattice spacing isn’t yet known to exist, which I believe is closely related to one of the Clay Millennium Problems (Yang-Mills existence and mass gap).
In short: I’d say you’re right that the SM (like many other QFTs) “pushes the boundaries of being a scientific theory”! What puts it on the OK side is not the existence of a complete package that makes rigorous mathematical sense, but just the existence of a huge number of special cases where one can extract clear predictions and compare them to observation.
Experts are welcome to correct anything I’ve gotten wrong.
Comment #231 December 24th, 2023 at 9:26 pm
Scott #230: Thanks, that all makes sense.
Re the sufficiency of perturbative analytical calculations: If I recall correctly, the perturbative Feynman expansion for pure QED calculations is believed to get increasingly accurate up to around 1/\alpha = 137th order, at which point the asymptotic series would stop converging and start diverging to infinity. This means that the Feynman expansion could in principle give you an accuracy up to \alpha^(1/\alpha) = 10^{-295}. It’s difficult to conceive of any fathomable experiment that could ever approach this precision, so it’s a philosophical question whether or not the Feynman expansion should be considered an “exact” (as a practical matter) method for tackling pure QED. (Of course, “pure QED” is already an idealization – virtual particles of species other than electrons or photons cause real-world electron-scattering experiments to deviate from the predictions of pure QED long before we reach that order of the Feynman expansion. It’s possible that QED may even logically need to be coupled to these other quantum fields in order to get saved from quantum triviality, but I think that’s still an open research question.)
If I understand correctly, QCD is strongly enough coupled that the Feynman expansion is only qualitative correct even at leading order, so the perturbative approach isn’t very useful at all for quantitative predictions in QCD.
So I think that for QED, an analytic perturbative expansion is much more useful than lattice numerics, while for QCD, lattice numerics are much more useful. I’m not sure which approach is better for studying electroweak interactions, since the electroweak coupling constants are intermediate between those of QED and QCD.
(Experts are welcome to correct me if I’ve gotten any of this wrong.)
Comment #232 December 25th, 2023 at 7:30 am
SalH Says in Comment #227 that AI is not even wrong because AI’s are not intelligent, but if a human being behaved in the way GPT-4 behaves I am certain you would not hesitate to say he was intelligent, in fact you would say he was very intelligent. It’s true that GPT-4 can’t (yet) manipulate things in the external world, but Stephen Hawking couldn’t either, was he unintelligent? As for consciousness, I can’t prove that GPT-4 is conscious and I will never be able to, but I can’t prove that you are conscious either, nevertheless I believe that you are because you are behaving intelligently. I simply could not function if I really believed I was the only conscious being in the universe, therefore I take it as an axiom that intelligent behavior implies consciousness, or to put it another way, consciousness is the way data feels when it is being processed intelligently. That’s why I believe my fellow human beings are conscious part of the time but not all of the time, not when they’re sleeping or under anesthesia or dead. Everything I said in the above can be summarized in just 4 words, “The Turing Test Works”.
Besides preserving my mental health there is another reason I believe that axiom is true. I know for a fact that evolution managed to invent consciousness at least once (me) and probably many billions of times, but random mutation and natural selection can’t directly detect consciousness any better than we can in anybody except ourselves, and evolution can’t select for something it can’t see; BUT Evolution can see and select for intelligent behavior. Therefore the only logical conclusion is that consciousness must be the inevitable byproduct of intelligence.
And so I make no apology for using anthropic reasoning when thinking about other intelligent entities like GPT-4 because analogies have always been helpful in science and even in everyday life. I mean…. do you really believe the fact that your brain and mine is wet and squishy while GPT-4’s brain is dry and hard is of profound philosophical importance?
John K Clark
Comment #233 December 25th, 2023 at 8:35 am
Ernie Davis #2
“My father didn’t see the value of FORTRAN; why shouldn’t people program in machine language?”
I expect an advanced AGI will ask the very same question when writing code for its own use and so props to your father.
Comment #234 December 25th, 2023 at 8:02 pm
Scott, one last question – I promise! (This is a reworking of my comment #31 on your post from Nov. 18th. If you don’t answer it, then I’ll take the hint and stop asking. 😀 )
At the very end of your podcast with Lawrence Krauss, you suggest that your watermarking scheme relies crucially on the stochastic sampling of the probability distribution over the token space that the transformer network generates, and so it doesn’t work at zero temperature. Could you go into just a little more detail about how your scheme interacts with the choice of temperature parameter? Is there perhaps some temperature scaling behavior where the expected amount of text required to reveal the watermark increases as the temperature is lowered and diverges as T -> 0? If the model owner allows the user to set the temperature setting, then can the user counter the watermarking just by lower the temperature?
Comment #235 December 26th, 2023 at 8:28 pm
Per your point (1), is it really just scaling that is doing all the work here? Like, was the invention of the transformer mechanism not really a fundamental thing/”revolution” in this whole trajectory?
Comment #236 December 26th, 2023 at 10:22 pm
What’s remarkable to me is that watching ai carefully didn’t make me predict generative ai. Even seeing gpt 2 then 3 then midjourney didn’t make me immediately say “oh we’re close to agi.” In fact, gpt 4 is so far from what I imagined, it’s hard to even know what lesson to take. This technique leads to a really weird ai that is much better than I thought possible.
Before 3.5 I thought something useful would need reinforcement learning. I thought it would look like go or protein folding: finding out how to scale up from our modest pot of data.
We didn’t need to write or curate a bunch of unittests to teach ai how to program really well. But we still have that tool in our belt for when we need it and we can generate fairly decent data!!
I thought simulations were going to be important to teach ai physics (and you would vary the constants of our universe to get it to generalize from simulation to real life)
Instead I think ai can maybe generate better world models than we can…. from video? From a couple pictures?
Text is an incredibly universal medium (think Unix cli) so I’m not unhappy about the result, but it’s such a new direction for science fiction
Comment #237 December 27th, 2023 at 8:50 am
Pete #235: It’s an extremely interesting question. My view of it was shaped by a talk by Ludwig Schmidt this summer, where he made an empirical case that the transformer architecture was the rough equivalent of 100-1000x in scale for the earlier LSTM architecture. If he’s right, then yes, the invention of the transformer accelerated things, but even without it, the LLM revolution would presumably have happened anyway with a 5- or 10-year delay.
Comment #238 December 27th, 2023 at 9:15 am
OhMyGoodness #222
Ok, your catalogue of famous historical personalities is funny and all, but unfortunately it won’t work🙂:
You see, if you want to emulate some crucial aspects from the persona of a real person, you need also to simulate interactions with their “environment”, the historical period that they lived, the knowledge about the world that they had back then, other people that interacted with them ( of course…) and so on.
Nearly impossible in practice.
Imagine, for example, a young “Einstein” as a child of a peasant somewhere in Europe during the dark ages…
Or other persons with different gender that the one they actually had ( or no gender at all…)
Moreover, this mix up of different personalities in one “AGI” could have been rather messy, don’t you think?
If AGI will be realised some day , it will be something very different, probably, not so human-like…
Comment #239 December 27th, 2023 at 12:36 pm
Ted #231:
> If I understand correctly, QCD is strongly enough coupled that the Feynman expansion is only qualitative correct even at leading order, so the perturbative approach isn’t very useful at all for quantitative predictions in QCD. So I think that for QED, an analytic perturbative expansion is much more useful than lattice numerics, while for QCD, lattice numerics are much more useful. I’m not sure which approach is better for studying electroweak interactions, since the electroweak coupling constants are intermediate between those of QED and QCD.
As you may know, in QFT “coupling constants” aren’t actually constant; they depend on the energy/momentum exchange scale. This isn’t too important for QED, where the coupling constant increases with energy from its low energy value of ~1/137 but remains much smaller than 1 at all relevant energies, but it makes a world of difference for QCD, where the coupling constant is larger than 1 at low energies, but becomes smaller than 1 at energies accessible to supercolliders. Thus, QCD perturbation theory is inapplicable to, e.g., calculating hadronic spectra, but is perfectly fine for predicting LHC cross-sections; in fact it’s indispensable for that purpose, since lattice numerics are ill-suited for simulating scattering experiments.
Comment #240 December 27th, 2023 at 12:55 pm
John K Clark #232
Actually, it’s not so difficult to see that chatbots (vs humans or animals) are not “sentient” or “conscious” (I’m not saying ” intelligent”, because that’s a notion / concept that most people accept as a characteristic for AI {vs AGI}).
It’s the persistence in time of a specific Personality that characterizes a conscious being.
People are changing behaviour or opinions, sometimes they’re unpredictable, but they nevertheless are identified from others, quite easily, as being the *same* persons each time …
Even people with serious mental health problems ( or dementia etc) have persistent personalities at least to some degree.
This is not the case with chatbots, I think that there’s no doubt about it.
In other words “the ( classic) Turing Test is not adequate”…
Comment #241 December 28th, 2023 at 1:09 am
Dimitris #238
Okay. Maybe I haven’t thought the details all the way though but gaps can be plugged like the dinosaur DNA in Jurassic Park. What would be the difference for Einstein? He would have eaten and procreated and came up with great ideas.
Considering your comments this is not the time to discuss the excellent prospects for a King Leonidas AGI as a third party candidate in the US. This would be much better than what we had in the recent past, have, or will have in the near future. 🙂
Comment #242 December 28th, 2023 at 12:12 pm
“ Did you foresee the current generative AI boom, say back in 2010?”
Of course not? My notebooks from that time show a kind of post-SVM malaise, back to randomly casting about for the secret sauce that seemed so elusive.
But I’ll claim that it was AlphaGo, not “Attention Is All You Need,” that made it clear that one should do a big update to one’s priors. I think it’s AlphaGo and AlphaGo Zero that we should be using to clarify our arguments, not the far squishier, pareidolia-inducing outputs of LLMs and diffusion models.
“How (if at all) do you now feel you should’ve been thinking differently?”
Deep connections have a fundamentally different intuition than 3-layer connections. In retrospect, it does not seem a great leap to think “So the feature detectors will essentially be working at higher and higher levels of abstraction. Huh. What are the capabilities and limits of that?” The answer turned out to be that, in what would now be considered rather shallow networks, Move 37 and, across the networks and MCTS, master-level overall play. That was the proof that connectionism was capable of what seemed to surely require explicit symbolic processing.
Comment #243 December 29th, 2023 at 1:04 pm
Dmitris #238
I thought about your Einstein proposal and assume the Church would see after his education as they often did with very talented individuals from poor families. Considering the observations he could access at the time I don’t think unreasonable to expect he would have accelerated science in optics, mechanics, astronomy, and gravitation. Not unreasonable to believe he would have superseded Bacon, Galileo, and Newton.
As for the crowding issue it would be a distributed node super-intelligence deeply linked through a network.
Comment #244 December 31st, 2023 at 6:36 am
In comment # 240 Dimitris Papadimitriou Says: “It’s the persistence in time of a specific Personality that characterizes a conscious being”, but the only way we have of determining the personality of anything (except for ourselves) is by observing its behavior, in other words the Turing Test. And the only reason “the ( classic) Turing Test is not adequate” is that a computer has managed to pass it and some human beings don’t like that fact. At one time people said we wouldn’t know that a computer was intelligent until it beat a grandmaster at chess, but when a computer did they said now it had to beat a grandmaster at GO, but when a computer did they said it now had to become good at visual perception, but when a computer did they said it now it had to pass the Turing Test, but when a computer did they now had to rack their brains and try to figure out something else a computer is not supposed to ever be able to do. I don’t think some people will EVER be convinced no matter what a computer is able to do, but that’s not important; regardless of what humans do or say computers will just keep on getting smarter and smarter, but people will not.
John K Clark
Comment #245 December 31st, 2023 at 12:23 pm
OhMyGoodness #243
Trying to imagine what a young Albert could do in the Dark Ages, as a shepherds’ child is tricky…
Perhaps he could find the shortest path ( geodesic ! 😃) through the fields, while leading the herd…😊
Best wishes to everyone!
Comment #246 January 1st, 2024 at 5:04 am
John K Clark #244
Although you do have a point here ( indeed, “moving the goalposts” is a common thing that people do when trying to find support for their ideas), your argument is still “rhetorical” and there are other similar common arguments that work effectively in the opposite direction :
In the age of Industrial Revolution, people were finding mechanistic models for almost everything.
Laplacian determinism was en vogue, the universe as a clockwork mechanism etc.
In the previous century, when Chaos and fractals were the trend, you could find them all over the place ( for a reason, people back then considered these almost as a “key” for the understanding of the Cosmos…)
The same with Quantum mechanics (there were people that believed that QM is the key for the explanation of “paraphysical” phenomena, like telepath and the like 🙂, or supporting Idealism and solipsistic/ mystic ideas…).
Currently, in our age, “the world is a simulation” , “AI have consciousness”, “you don’t have free will”, you know very well all the current trends in the internet, sometimes one trend contradicting the other, with “debates” and podcasts everywhere (usually with similar titles 🙄) …
Category mistakes are also all over the place, with computers that “make decisions” etc…
These kind of arguments ( from both sides) are rhetorical though, ” political” in a wider sense. Science is
based on evidence, not on rhetorics.
Everything that we know about “Conscious behaviour” and the like is based on the experience that we have with biological living organisms, humans or animals. And these conscious beings have some distinct characteristics:
Not all of them are good at chess or mathematics, but they have some common features, including persistent personalities, modeling of the world that surrounds them and of themselves up to a certain degree, some kind of restricted free will ( yes, despite the usual propaganda on YT!) – even cats have free will, I’m quite certain for that!
AI is not there yet and this is not a matter of personal opinion.
Perhaps some kind of AGI will emerge, lacking persistent personality etc, I don’t know for sure.
But it will be something entirely different from our current experience, it won’t be like LLMs, it won’t resemble human or animal like behaviour.
Best wishes!
Comment #247 January 2nd, 2024 at 12:05 pm
Dimitris Papadimitriou #246
From a human perspective it’s not important if GPT-4 is conscious or not, the important thing is that it’s intelligent, if it’s not conscious that’s its problem not mine. But if it’s not a brute fact that consciousness is the way data feels when it is being processed intelligently and thus a byproduct of intelligent behavior, then Evolution would never have been able to produce it because random mutation and natural selection can’t directly detect consciousness any better than we can, and nothing can select for something it can’t see. And yet I know with absolute certainty that Evolution did manage to produce consciousness at least once (me) and I believe with almost as much certainty (because I could not function if I really believe that solipsism is true) that my fellow human beings are also conscious, at least when they’re not sleeping or under anesthesia or dead.
I’ll let you know if I agree with you about cats having free will as soon as you explain to me what the term “free will” means to you, as for me it just means not always knowing what I will do next until I actually do it, but that doesn’t alter the fact that everything I do is either because of cause and effect or it is not, and if it is not, if it is an event without a cause, then my action was by definition UNreasonable and random. I agree with you that until about a year ago everything that we have seen that behaved intelligently has been biological, but that is no longer the case, so the old rule of thumb that biological = conscious-being has become obsolete. In fact even before GPT-4 it wasn’t a very good rule of thumb, plants are biological but nobody thinks they’re conscious because plants don’t behave intelligently, we don’t even think our fellow human beings are conscious when they are sleeping or under anesthesia or dead, and for exactly the same reason.
And I don’t understand why it’s a category error to say a computer made a decision but it’s not a category error to say that you made a decision.
Happy new year
John K Clark
Comment #248 January 2nd, 2024 at 6:24 pm
John K Clark #247
I’m not arguing about the term “intelligence” in AI , its a commonly used term, not exactly successful ( a bit like Dark Matter, which is invisible, not dark!) but what can we do?
– I agree about Evolution and conscious beings etc, but I don’t see your point… i didn’t said anything about plants! And computers are not exactly byproducts of biological evolution.
– The basic common characteristics of “consciousness” (see my previous comment) are not controversial, they are quite reasonable.
All these notions ( consciousness, our thoughts, our will, our decisions ) are emergent properties ( I’m arguing in the context of naturalism/ physicalism) and they’re interwoven with each other, so there are no “decisions” without some notion of “free will” and both have “sentience” as a prerequisite ( that’s why I talked previously about “category mistakes”.
– Don’t ask me to define these notions! Even if I did, my definitions would have some level of arbitrariness (there’s no consensus anyway).
– About free will:
The only thing that I’m quite certain about, is that this notion and all the other relevant macroscopic emergent properties like self awareness, etc. are:
1) Intercorrelated in conscious beings ( you can’t have some of them but not the others, they’re in the same categorical set).
2) They presuppose an open future, i.e. irreducibly Probabilistic physical laws.
If the World was deterministic as in classical Newtonian/ Laplacian mechanics, then not only free will of some kind, but not even consciousness could have been possible.
People who argue that strict determinism and “free will” are compatible are wrong!
So, fundamentally probabilistic laws are necessary for some restricted notion of free will, but they’re not enough for real ” libertarian- style” freedom.
Of course I’ve no Idea if the latter kind is possible, but the minimum presupposition (irreducible randomness) is a characteristic of the real World, thanks to QM (all interpretations, including Everettian “Many Worlds”).
For “real”, liberal free will, new physics is needed (perhaps strong emergence, in the sense that some properties that appear in complicated systems cannot be deduced from the underlying fundamental laws in principle).
But even if this is not the case, what we call free will (at least a weak version) is compatible with all the interpretations ( and alternatives, except Bohm) of QM as it is.
But I digress: My point in the previous comments was that the Turing test is not only inadequate, but even irrelevant:
Really conscious AGI, when it will emerge, won’t be like humans, it won’t have personal life like us (friends, sexual relationships, hunger, etc), if it’s not biological!
It’s not that you need biology for “consciousness”, it is that the artificial consciousness that you’ll have will be non human in a very pronounced way.
It won’t pass an improved version of Turing’s test, unless it will be cheating efficiently and constantly!
Happy new year!
Comment #249 January 4th, 2024 at 6:15 am
Dimitris Papadimitriou #248First of all I’m not going to demand a strict definition to any terms you use, but I do ask you to give me examples of what you mean because examples are more important than definitions, it is after all where lexicographers got the knowledge to write their book. For example, please give me an idea of what you mean by the term “free will” , an example of something that does NOT exist because of cause and effect AND also does NOT NOT exist because of cause and effect.
I became even more confused about what you mean when you said ” If the World was deterministic as in classical Newtonian/ Laplacian mechanics, then not only free will of some kind, but not even consciousness could have been possible”. If I see somebody do something very strange it would be natural for me, and I think for most people, to ask: why did you do that? If they are unable to give an answer to that question I don’t interpret that to be a good thing, I interpret it as a sign of irrationality and perhaps even insanity.
When I say “free will” I simply mean the inability of me or anybody else to know what I will do next until I actually do it. By my meaning, computers certainly have free will. It would only take a few minutes to write a computer program to find the first even number greater than 4 that is not the sum of two prime numbers and then stop, but will the computer ever stop? The computer doesn’t know and neither does anybody else, all you can do is watch it and see what it does, and you might be watching forever.
My point about Evolution was that if consciousness was not the byproduct of something that Natural Selection could detect, such as intelligent behavior that enhances the prospects of the genes of an individual getting into the next generation, then Evolution could have never produced consciousness, and yet you know for a fact it did at least once and probably many billions of times. That’s why I think it’s a brute fact that consciousness is the way data feels when it is being processed intelligently.
And I know you never “said anything about plants!” even though you did mention other biological organisms. But why didn’t you mention plans? Because plants do not behave intelligently. You say “the Turing test is not only inadequate, but even irrelevant”, so I have to ask, if it’s not by observing behavior how do you tell the difference between a smart human being and a stupid human being?
And I see no reason in principle why computers “won’t have personal life like us (friends, sexual relationships, hunger, etc)” because ” it’s not biological!”. Why is the element Carbon inherently more emotional than the element Silicon? My strong suspicion is that emotions, including consciousness, are easy but intelligence is hard; Evolution certainly found that to be the case. There were both predators and prey during the Cambrian Explosion indicating that creatures must’ve had a fight or flight response, fear and anger. But it took another 500 million years before Evolution managed to produce an organism that was intelligent, with “intelligent” being operationally defined as having the capacity to understand how a radio telescope works.
I’m not quite sure what to say about your statement that “for “real”, liberal free will, new physics is needed” because I’m not sure what you mean by free will much less real liberal free will, but I suspect in the future most will put that statement in the same category as those who said a new and very special “life force” was needed to explain biochemistry.
John K Clark
Comment #250 January 5th, 2024 at 5:58 am
John K Clark #249
When I was talking about “irreducibly Probabilistic laws of physics”, I meant the usual standard Quantum Mechanics.
It is a Probabilistic theory and a very accurate one in its statistic predictions: Probabilistic does not mean just mere chance, or that “anything goes”, as you seem to believe with your example about the “irrational behavior”…
Having fundamentally probabilistic laws means, briefly, that you have many possible Histories compatible with the same initial conditions.
As for the rest of your opinions ( computers have free will etc, I have already commented before, as clearly as I could without being longwinded).
Comment #251 January 6th, 2024 at 8:05 am
Dimitris Papadimitriou #250
You’re right, in both a random and deterministic world, a mathematically valid statistical prediction, such as one derived from classical or quantum physics, will NEVER say “anything goes”. For example, if you’re rolling one six sided cubic die the probabilities you will get the numbers 1 through 6 are equal and the probability of you getting a number greater than 6 is zero. But if you roll a pair of dice the probabilities are NOT equal because there are 36 different ways the dice could roll but only one way will produce a 2 , however there are six ways to produce a 7. So you only have one chance in 36 of getting a 2 but one chance in 6 (6/36) of getting a 7.
And quantum mechanics says everything follows the Schrodinger equation and the Schrodinger equation does NOT say “anything goes”, for example it says the probability of an electron spontaneously turning into a proton is exactly zero.
However none of this changes the fact that everything, absolutely positively everything, either happens because of cause-and-effect OR it does not happen because of cause-and-effect, and if it does not happen because of cause-and-effect then it is by definition “random”. As I said before, when I use the term “free will” I mean the inability of me or anybody else to always know what I will do next until I actually do it, but what do you mean when you use that term?
John K Clark
Comment #252 January 6th, 2024 at 10:51 am
Dimitris Papademetriou #250:
Ah. Then fundamentally probabilistic laws aren’t compatible with libertarian free will after all, because probabilistic decisions have to comport with the probability norm. If you are arguing for a nondeterministic reduction of mental processes to physics, this would not allow for libertarian free will either, because a given mental state involving a future decision would still lead uniquely to that decision.
Comment #253 January 8th, 2024 at 12:50 pm
Ben Standeven #252
John K Clark #251
I’ve already clarified ( in my comment #248) that
Irreducibly Probabilistic laws (QM) are only *compatible* with some (weak, restricted) notion of free will only in the sense that many possible Histories are compatible with given initial data, so it’s not self- contradictory to speak about things that ” could have been otherwise”.
Fundamental probability is a *necessary*, not a
*sufficient* condition for *any* sensible notion of free will.
In a strictly deterministic world (in the Laplacian sense, not as in Everett!) anything related to consciousness, decision, free will etc. , has totally absurd consequences:
Imagine for example that the proverbial Laplace’s demon is a conscious physical entity ( why not ? If consciousness is an emergent property related to complexity or computational power, that demon has both!). It is able to predict the entire history of that world, including it’s own history, but this knowledge is entirely useless: the demon is as helpless as it gets, it cannot alter even the slightest detail, because in such strictly deterministic worlds, the history is unique and fixed!
If we imagine “conscious beings” like humans in that Laplacian universe, things are getting even more weird:
The demon could show a coarse grained prediction to an inhabitant of that World, let’s call him Bob, and Bob would have been unable to change anything from that prediction:
Lets assume that the prediction says that Bob will broke his leg falling of the stairs of his home at 21:35 the next day.
“Poor” Bob cannot change anything , he is predetermined to act exactly as the demon predicted, because in that world only one History exists, there can’t be any deviations, by definition.
It’s an absurd situation, similar to the Grandfather Paradox ( the similarity is not a coincidence…)
Actually, Bob ( or any other inhabitant of that world) cannot have any memory, will, or sentience…
Even his thoughts are predetermined, fixed.
He is destined to break his leg and he has to do it the same way, the exact hour as predicted!
Although this arguments are heuristic ( l don’t have a No-go theorem in front of me…) are strong enough to exclude any version of deterministic+ fully predictable
laws as adequate for describing our real Universe.
“liberal” or “true” free will requires
Comment #254 January 8th, 2024 at 1:06 pm
( continuation from my previous, abruptly interrupted by mistake, comment):
Of course, liberal or “true” free will requires something more, something like physicalist Strong Emergence as I already said in my #248 comment.
That’s speculation, although there are already some hints that such kind of Emergence that cannot be fully deduced from the underlying basic laws might exist…
Comment #255 January 11th, 2024 at 7:47 am
Dimitris Papadimitriou #253
I agree that many possible Histories are compatible with a given data set, a simple example would be Conway’s Game of Life. It’s 100% deterministic so if I show you a screenshot of the game you can always predict what the next iteration of it will be, but you can’t always figure out what the previous iteration must have been because there was more than one way to get to the present state. But I don’t quite see what that has to do with free will.
Opinions differ over if a strictly deterministic world “has totally absurd consequences” and there is no disputing matters of taste, but one thing is certain, it does NOT lead to paradoxical consequences, or at least it doesn’t unless Laplace’s conscious demon has an infinite (and not just astronomically large) memory. For Laplace’s Demon to be able to always perfectly predict what it will do Mr. Demon would have to have a one to one model of itself, and that model would have to include a model of the model of itself etc. But for that to be possible the demon’s memory would have to be infinite because only in an infinite set can you have a one-to-one correspondence with a proper subset of itself, in fact that is the very definition of infinity. So we are left with the interesting conclusion that, using my definition that “free will” is the inability to always know what you will do next, that we have free will but God does not. We can also conclude that either logically paradoxical conditions can actually exist in the real physical world, or God does not and can not exist.
So yes, in the real world where physics and not just mathematics come into play, both Laplace’s Demon and human beings cannot alter things to the slightest degree, they could turn left instead of right if they wanted to, but initial conditions are such that they will never want to. However neither the demon nor you and me feel like slaves because neither of us can always predict what we will do next until we actually do it.
John K Clark
Comment #256 January 11th, 2024 at 5:34 pm
John K Clark #255
The Everettian Many Worlds version of QM is also deterministic: The Global Wavefunction is evolving deterministically according to the Schrödinger equation etc, and that’s totally irrelevant ( as is Conway ‘s “Game of Life”) to my arguments.
I stressed the point that I’m talking about a deterministic and fully predictable ( and retrodictable) World , in the Laplacian/ Newtonian sense.
Only in such a world “faultless Predictions” are possible, in principle, because there’s generically one and only possible history compatible with given initial conditions.
This is the only kind of World where “faultless Predictors” ( as in “Newcomb’s Paradox”) are allowed
to exist as physical entities.
I argued that in these strictly deterministic and fully predictable/ retrodictable Worlds any notion of consciousness and free will (not only “true” or “liberal”, but even “illusory” consciousness or free will) leads to absurdities, very similar to the Grandfather Paradox.
My conclusions are that:
– Strict deterministic and fully predictable Worlds ( as above {*}) are not adequate models for our physical Universe.
– Paradoxes like Newcomb’s that assume Worlds with “faultless Predictors” are not compatible with the existence of conscious observers that make decisions
So, all these endless debates about Newcomb’s Paradox and the compatibility between determinism and free will and the like, are a big waste of time.
{*} Note that Globally hyperbolic spacetimes (solutions of Einstein’s field equations of GR), although they’re locally and globally deterministic , they do not allow, generically, the existence of physical observers that are “faultless Predictors” (no matter how advanced they are…), due to the causal Light Cone structure of
the theory.
But solutions that have closed timelike curves do have the same issue!
Comment #257 January 11th, 2024 at 10:33 pm
Scott – A while back you ask readers if you should work on AI or Quantum computing.
I didn’t have a strong opinion, so didn’t respond.
I now favor AI work because:
1) this blog post on AI
2) QC – even if it does eventually do something useful, its value will be diminished by AI being able to improve classical algorithms faster than QC can solve the problems of interest.
thanks for being a wonderful person.
Phil
Comment #258 January 12th, 2024 at 8:58 pm
Dimitris Papadimitriou #256
> Only in such a world “faultless Predictions” are possible, in principle
A faultless prediction is by definition a transfer of information from the future, and as such they lead to the same paradoxes as time machines. So, no. In general they aren’t possible even in deterministic worlds. And in cases when they are possible (preselected universes that evolve in such a way to have no paradoxes, that is no one uses information from the future to alter the future) there’s no apparent conflicts with free will.
Comment #259 January 13th, 2024 at 8:17 am
Responding to Dimitris Papadimitriou in #256
You say that only in a Laplacian/ Newtonian ”world faultless Predictions are possible, in principle” and ”are allowed to exist as physical entities”, but that is untrue as Gödel and Turing proved. Laplace’s demon could always correctly predict that you will turn left not right BUT only if the demon did NOT tell you of his prediction because if you disliked the demon and were determined to make a fool of him all you’d need to do is the opposite of whatever the demon predicted. So if the demon says he predicts you’ll turn left then you will make a point of turning right. And the same logic remains true in the quantum world as well as the Laplacian/ Newtonian world.
You also say that in a strictly deterministic and fully predictable world any notion of consciousness or free will leads to absurdities, but I will not dispute that point for two reasons:
1) The matter is moot. A fully predictable world does not and can not exist, and it doesn’t matter if it’s deterministic or non-deterministic, or classical or quantum.
2) I have told you what I mean by the term “free will” but you still haven’t told me what you mean, so whenever you use that term I can’t say if I agree or disagree with you.
I do agree that closed timelike curves produce paradoxes and General Relativity allows the existence of closed timelike curves, BUT only if the universe is rotating, however all the astronomical evidence indicates that the universe is NOT rotating. So if the universe was different then General Relativity wouldn’t work, but the universe is not different so it does work.
John K Clark
Comment #260 January 13th, 2024 at 10:03 am
red75prime #258
-Laplace’s demon is a faultless Predictor. It’s a personification, a metaphor for the strict determinism and full predictability of a Newtonian universe.
In a Newtonian universe, if you have specify your data at some moment of time, it is possible in principle to predict the future and retrodict the past. So, I’m not sure what you’re talking about…
– I’m not sure also what you mean by “preselected universes”, perhaps you mean self – consistent worlds with CTCs. This is exactly what I’m talking about. Although there are many possible self consistent versions ( such spacetimes are not globally hyperbolic), once one of them is “chosen”, it is fixed, it has to be self – consistent to avoid the paradox. There’s no room for free will then, nothing can be altered, contrary to what you’re saying.
Comment #261 January 13th, 2024 at 10:47 am
John K Clark #259 said:
‘ Laplace’s demon could always correctly predict that you will turn left not right BUT only if the demon did NOT tell you of his prediction because if you disliked the demon and were determined to make a fool of him all you’d need to do is the opposite of whatever the demon predicted. So if the demon says he predicts you’ll turn left then you will make a point of turning right. And the same logic remains true in the quantum world as well as the Laplacian/ Newtonian world.’
That’s trivially wrong!
In a Newtonian deterministic universe there is one and only one history compatible with the initial conditions.
You cannot alter the history, you cannot do otherwise of whatever the demon predicted. You cannot fool the demon! It doesn’t make a difference if the demon told you the prediction or not!
There’s only one history, how could you do something different?
– I don’t believe that a fully predictable deterministic world is an adequate model for our real Universe , that was one of my main points.
Any notion of Sentience and free will, even the weakest versions, are not compatible with the existence of faultless Predictors, and therefore with “predetermination”.
Newtonian universes ( and any other deterministic/ predictable ones) do not have sentient beings.
Newcomb’s “paradox” is not a paradox. It assumes sentient agents that make decisions in the presence of a ” faultless Predictor” ( and , thus , a deterministic/ predictable world).
Similar is the situation in the case of a Closed Timelike Curve.
It’s easy to imagine a “test particle” that does not interact with the environment on a CTC.
But try to imagine a complicated, macroscopic object, e.g. a digital camera with a memory on such a closed loop in spacetime.
There’s no recording from “previous loops” ( because there’s only one loop!) in the memory, no ageing, nothing can be altered from one cycle to the next, although the proper time from the perspective of the camera goes forward, as usual.
The same absurd situation that you may have with a sentient living organism ( that doesn’t also have memory of previous loops, it doesn’t age, it cannot alter the slightest detail etc) , it appears also with the macroscopic Digital Camera !
It doesn’t have to do only with “conscious” beings, it’s a deeper issue.
It needs extremely precise fine tuning for this CTC to exist.
But , otherwise it has striking similarities to the other aforementioned “paradoxes”…
Comment #262 January 18th, 2024 at 2:48 pm
Dimitris Papadimitriou #261: It makes a big difference whether the demon tells you its prediction, because then the prediction is part of the initial conditions. Changing the initial conditions of a deterministic process can change the outcome, after all.
Comment #263 January 19th, 2024 at 2:02 pm
Responding to Dimitris Papadimitriou in #260
You say you “don’t believe that a fully predictable deterministic world is an adequate model for our real Universe” I don’t believe that either because a fully deterministic world does not mean it is also predictable, even in principle. A Turing machine is 100% deterministic but if you program one to find the first even number greater than 2 that is not the sum of two primes and then stop there is no way you can predict if it will stop, all you can do is watch it and see, and you might need to watch forever. In the same way no matter how smart Laplace’s demon is he can’t predict what I will do next unless he keeps his prediction secret from me because then he would not only have to predict what I will do he would have to predict what he himself will do. So this thought experiment doesn’t prove that no deterministic worldview is an adequate model of our universe, instead it proves that Laplace’s demon can’t exist. You also say that even the weakest versions of “free will” are incompatible with faultless prediction, but I can’t comment about that because although I’ve told you what I mean by “free will” you STILL haven’t told me what you mean by that term. As for Closed Timelike Curves, I’m not very interested in those because they almost certainly do not exist.
John K Clark
Comment #264 January 21st, 2024 at 4:26 am
Actually, this has been explored a lot by science-fiction : obviously, if you know the demon is never wrong, it is in your interest not to try to falsify his prediction, as scenarios where you try and he is still right are usually a bit frightening (you are victim of a stroke , or the vehicle stops answering your commands…)
Comment #265 January 25th, 2024 at 2:03 am
Ben Standeven #262
In a deterministic world, whatever the Predictor does is already in the initial conditions.
The initial conditions cannot be changed a posteriori.
There is only one History compatible with that set of initial data, as I’ve said many times.
That’s what determinism is.
There is a continuous infinity of possible sets of initial data ( we are not talking here about “superdeterministic” restrictions on the initial conditions or fine tuning), but once a specific set of initial data is “chosen” , that’s it!
You have only one History. There’s nothing whatsoever that anyone could change , ever!
You need many possible Histories compatible with that set of initial conditions to have any chance to change anything. It’s that simple!
You need fundamentally Probabilistic laws, or whatever else that allows many possible histories.
Determinism ( of the “Laplacian kind”), does not do the job.
Comment #266 January 25th, 2024 at 8:02 am
Responding to f3et in # 264
But I know that a demon that is never wrong in his predictions is impossible if he predicts what I will do next and he tells me what that prediction is. I don’t know about you but I would make a point of always doing the opposite of what the demon predicted because I think it would be fun to watch Laplace’s demon fail and realize that he’s not as smart as he thought he was.
John K Clark
Comment #267 January 26th, 2024 at 6:05 am
Responding to Dimitris Papadimitriou in #265
> “we are not talking here about “superdeterministic” restrictions on the initial conditions or fine tuning), but once a specific set of initial data is “chosen” , that’s it!”
I agree, if things are deterministic then for any given time there can only be one outcome, however NO initial condition results in a Laplace demon being produced, at least not one that can always make perfect predictions in all circumstances. And I’m glad you weren’t talking about superdeterminism, I can’t prove that the idea is wrong and neither can anybody else or ever will, but I can prove that the idea is silly because a greater violation of Occam’s Razor is impossible to imagine.
John K Clark
Comment #268 January 27th, 2024 at 9:41 am
John K Clark #266 reminds me of the diagonal proof. Can’t the halting theorem be viewed as a diagonal proof?
Is this a valid analogy with Clark’s argument?
Comment #269 January 27th, 2024 at 11:11 am
To John K Clark #266
Read my reply again. Or the MANY science-fiction novels and short stories on this. Or perhaps some Greek myths : Oedipus too was sure he could do the opposite of the oracle prediction. Now think again : you have seen the demon (or the AI ? A modernization of the myth seems in order) predict actions of others, and never be wrong, even if they were aware of the prediction. It is your turn : he predicts you will write on the blackboard the word YES. You approach the board with the firm intention of writing NO. Your arm begins to shake, it hurts, you feel writing anything else will end horribly badly. What do you do ?
Comment #270 January 29th, 2024 at 12:53 pm
Responding to f3et in 269
” Read my reply again. Or the MANY science-fiction novels and short stories on this.Read my reply again. Or the MANY science-fiction novels and short stories on this.[…] he predicts you will write on the blackboard the word YES. You approach the board with the firm intention of writing NO. Your arm begins to shake, it hurts, you feel writing anything else will end horribly badly. What do you do ? ”
You are implicitly talking about something very much like Superdeterminism, the idea that out of the infinite number of initial conditions the universe could’ve started out in it actually started out in the one and only initial condition that resulted in the human race always incorrectly concluding that things can happen randomly even though they were 100% deterministic because we ALWAYS choose to set up our experiments in exactly the wrong way. As I mentioned before, I can’t prove that the idea is wrong and neither can anybody else or ever will, but I can prove that the idea is silly because a greater violation of Occam’s Razor is impossible to imagine.
John K Clark
Comment #271 January 30th, 2024 at 7:00 pm
John K Clark # 267
I’m afraid that it is the exact opposite!
What you’re proposing, i.e. that initial conditions that correspond to Histories with Predictors ( like the ones I mentioned before) have to be *excluded* because they lead to absurdities (e.g. “conscious agents”
that “learn” about these predictions but cannot change the slightest detail) is exactly what superdeterminism is about!
There is an uncountable infinity of such initial conditions ( with Laplacian Predictors and agents).
You cannot exclude them arbitrarily just because you want to avoid the conclusion that “conscious agents” and determinism do not fit together…
Imposing constraints / restrictions on the initial conditions is exactly what “superdeteminists” are proposing!
And they’re doing it always for the same reasons , i.e. because they’re trying to avoid the unavoidable conclusion.
The same happens with those people who are trying to evade Bell’s theorem by hypothesizing miraculous restrictions to the initial conditions (extreme fine tuning) chosen to “mimic” the correlations that QM predicts.
Comment #272 January 31st, 2024 at 8:47 am
> Did you foresee the current generative AI boom, say back in 2010? If you did, what was your secret?
Yes, kinda, in that I made a procedural music generator. While it wasn’t very good, it was (and still is) better than me: https://www.youtube.com/watch?v=depj8C21YHg&list=PL1RjJKSpNvv1uxZTyjGCKyWGHrUhDKE8l&index=5
The conversations on my (long-since purged) LiveJournal went much the same as current conversations about GenAI.
> Feel free also to give your p(doom), under any definition of the concept, so long as you clarify which one.
p(doom) = 1/6 (though this is just a vibes-based guess) where half of that is due to humans asking an AI to do something directly bad, and the other half is the AI having a flaw which has a bad outcome from a non-evil request.
I have a low probability on any specific bad outcome, because there’s so many ways things can go wrong and many exclude alternative bad outcomes — we can’t tile the universe with paperclips *and* smile emoji, terrorists won’t be able to trigger WW3 if everyone’s doped to the eyeballs in bliss-inducing drugs.
Comment #273 February 2nd, 2024 at 5:25 pm
If the predictions of a flawless predictor are part of the initial conditions, then we’re back to determinism not mattering. Even in a nondeterministic world, the predictions would still have to come true on every possible history, because the predictor is flawless.
Comment #274 February 4th, 2024 at 5:39 am
Dimitris Papadimitriou says in #271
”What you’re proposing, i.e. that initial conditions that correspond to Histories with Predictors ( like the ones I mentioned before) have to be *excluded* because they lead to absurdities (e.g.
“conscious agents””
I am only proposing 2 things and I don’t think either of them could be considered controversial. And neither of them has anything to do with consciousness:
1) Very strange things can exist in physics, but paradoxes cannot.
2) A theory, such as Superdeterminism, that can only work if the universe started out in one very specific initial condition, is inferior to all other quantum interpretations that are consistent with the universe starting out with an infinite number of different initial conditions.
John K Clark
Comment #275 March 20th, 2024 at 9:10 am
I admire how you use storytelling to bring your topics to life.