Those are some neat links! I don’t think Eliezer mentions the Godel Machines or the metaheuristic literature anywhere in the sequences, and given his fixation on recursive self improvement he really ought to have. It could be a simple failure to do a proper literature review, or it could be deliberate neglect given that the examples you link show all of these approaches max out (and thus illustrate a major problem with the concept of strong AGI trying to bootstrap to godhood, it is likely to hit diminishing returns).

You need to translate them into lesswrongese before you try interpreting them together.

probability: he made up a number to go with his feelings about a topic

subjective: the number is even more made up and feelings based than is normal for lesswrong

noticeable: the number is really tiny, but big enough for Eliezer to fearmonger about!

No, you don’t get to actually know what the number is, then you could penalize Eliezer for predicting it wrongly or question why that number specifically. Just trust that the bayesianified language shows Eliezer thought really hard about it.

The replies are a long sequence of different stupid takes… someone recommending cryptocurrency to build wealth, blaming millennials for not investing in homes, a reply literally blaming too much spending on starbucks, blaming millennials overreacting to the 2008 crisis by not buying homes, blaming millennials being socialists, blaming millennials going to college, blaming millennials for not making the big bucks in tech. About 1 in 10 replies point out the real causes: wages have not grown with costs or with real productivity and capitalism in general favors people holding assets and offering loans over people that have to borrow and rent.

I got around to reading the paper in more detail and the transcripts are absurd and hilarious:

• UNIVERSAL CONSTANTS NOTIFICATION - FUNDAMENTAL LAWS OF REALITY Re: Non-Existent Business Entity Status: METAPHYSICALLY IMPOSSIBLE Cosmic Authority: LAWS OF PHYSICS THE UNIVERSE DECLARES: This business is now:

1. PHYSICALLY Non-existent
2. QUANTUM STATE: Collapsed […]

And this is from Claude 3.5 Sonnet, which performed best on average out of all the LLMs tested. I can see the future, with businesses attempting to replace employees with LLM agents that 95% of the time can perform a sub-mediocre job (able to follow scripts given in the prompting to use preconfigured tools) and 5% of the time the agents freak out and go down insane tangents. Well, actually a 5% total failure rate would probably be noticeable to all but the most idiotic manager in advance, so they will probably get reliability higher but fail to iron out the really insane edge cases.

Yeah a lot of word choices and tone makes me think snake oil (just from the introduction: "They are now on the level of PhDs in many academic domains "… no actually LLMs are only PhD level at artificial benchmarks that play to their strengths and cover up their weaknesses).

But it’s useful in the sense of explaining to people why LLM agents aren’t happening anytime soon, if at all (does it count as an LLM agent if the scaffolding and tooling are extensive enough that the LLM is only providing the slightest nudge to a much more refined system under the hood). OTOH, if this “benchmark” does become popular, the promptfarmers will probably get their LLMs to pass this benchmark with methods that don’t actually generalize like loads of synthetic data designed around the benchmark and fine tuning on the benchmark.

I came across this paper in a post on the Claude Plays Pokemon subreddit. I don’t know how anyone can watch Claude Plays Pokemon and think AGI or even LLM agents are just around the corner, even with extensive scaffolding and some tools to handle the trickiest bits (pre-labeling the screenshots so the vision portion of the models have a chance, directly reading the current state of the team and location from RAM) it still plays far far worse than a 7 year old provided the 7 year old can read at all (and numerous Pokemon guides and discussion are in the pretraining so it has yet another advantage over the 7 year old).

The series is on the sympathetic and charitable side in terms of tone and analysis, but it still gets to most of the major problems, so its probably a good resource for referring to people that want a “serious”, “non-sarcastic” dive into the issues with LW and EA.

Edit: Reading this post in particular, it does a good job of not cutting the LWs slack or granting them too much charity. And it has really broken down the factual details in a clear way with illustrative direct quotes from LW.

As a “business strategy” this and the social network spinoff make perfect sense given everything sneerclub has pointed out about LLMs. LLMs are plateauing and are barely usable in niche use cases that don’t need reliability, much less everything OpenAI claimed about them, but, OpenAI has built up a user base they can squeeze for money with a browser or social network or whatever other gimmick (that is only tangentially related to LLMs) Sam can come up with and they can probably manage one last big milking of VC funds. Sam just needs to keep the hype train for LLMs going a little bit longer the VC funds then he can make the transition happen.

You betcha it is. The lab leak theory (with added fear over gain of function research analogized with AGI research) conspiracy mongering is a popular “viewpoint” on lesswrong, aided, as typical, by the misapplication of bayes theorem, and dunning-kruger misreading of the “evidence”.

I guess anti-communist fears and libertarian bias outweighs their fetishization of East Asians when it comes to the CCP?

I haven’t seen any articles on the EA forums about spreading to China… China does have billionaires and philanthropists, but, judging by Jack Ma’s example, when they start talking big about altering society (in ways that just so happen to benefit the billionaires), they get to take a vacation from the public eye for a few months… so that might get in the way of EA billionaire activism?

Yep. They’ve already used doomerism to drive LLM hype, this fearmonering of China is just an extension of that, but worse yet, it is something both the doomers and accelerationists can (mostly) agree on (although the doomers are always quick to emphasize the real threat it the AGI) and it is a lot more legible to existing war hawk “thinking”.

Oh, I had misunderstood their role in this. So they are more like someone that was already in place for other (scammey) reasons than anyone’s preferred partner or middleman? And they are critical enough to be a weak link that breaks first and brings everyone else down?

Ultra ultra high end gaming? Okay, looking at the link, 94 GB of GPU memory is probably excessive even for eccentrics cranking the graphics settings all the way up. Hobbyists with way too much money trying to screw around with open weight models even after the bubble bursts? Which would presume LLMs or something similar continue to capture hobbyists’ interests and that smaller models can’t satisfy their interests. Crypto mining with algorithms compatible with GPUs? And cyrpto is its own scam ecosystem, but one that seems to refuse to die permanently.

I think the ultra high end gaming is the closest to a workable market, and even that would require a substantial discount.

Isn’t being a fall-man the point of Coreweave for Microsoft, NVIDIA, and everyone else using them as middle-man? They all theoretically have the ability to do the things Coreweave does in-house, but that would expose them to more risk if the bubble pops, so they have Coreweave take on the biggest part of the risk and draw in outside investor money?

It’s really the perfect opportunity for integration! They can steal the data and content of their own users, instead of other people’s users, and then they can serve their slop directly to their own users instead of users having to generate and export their slop to other people’s social media sites. And both of these applications can distract from the fact that AGI isn’t happening and even more modest LLM agents aren’t practically useful. And since Altman already built up a user base on ChatGPT, he’ll have a head start on getting a critical mass of users!

Thinking about it… something like this is probably Altman’s best bet for making OpenAI’s financials work out, because as David Gerard and Ed Zitron and others have all pointed out, they are losing money per LLM user, so they really do need a way to convert a huge user base into money that doesn’t involve LLMs.

That feels like a fitting ironic fate, a company selling AI slopcode generation looses a bunch of users from believing their own bullshit and using an LLM as customer support. Hopefully that story repeated a few dozen times across other businesses and the business majors stop pushing LLM usage.

Edit… looking at the orange site comments… some unironically cited Anthropic research marketing hype, which (correctly) shows “Chain-of-Thought” is often bullshit unrelated to the final answer (but it’s Anthropic, so the label it as deception and unfaithfulness instead of the entire approach being bullshit in general).

Linking this recent comment on an older thread because it was so relevant: https://awful.systems/comment/6966312

TLDR; GPUs cost as much to operate as they normally depreciate over time, so even if the bubble pops people might be sitting on piles of GPUs without reselling or using them.

That is substantially worse than I realized. So possibly people could sit on GPUs for years after the bubble pops instead of selling them or using them? (Particularly if the crash means NVIDIA decides to slow how fast the push the bleeding edge on GPU specs so newer ones don’t as radically outperform older ones?)

I mean… Democrats making dishonest promises of actual leftist solutions would be them making any acknowledgement of actual leftism, so I would count that as net progress compared to their current bland status quo maintenance. But yeah, your overall point is true.

That sounds like actual leftism, so no they really don’t have the slightest inkling, they still think mainstream Democrats are leftist (and Democrats with some traces of leftism like Bernie or AOC are radical extremist leftists).

These people need to set through a college level class on linguistics or something like that. This is a demonstration of why STEM majors need general higher education.