I had never thought about any of this before, but it actually makes perfect sense.
By its nature, an LLM feeds back some statistically close approximation of what you expect to see, and the more you engage with it (which is to say, the more you refine your prompts for it) the closer it necessarily gets to precisely what you expect to see.
“He was like, ‘just talk to [ChatGPT]. You’ll see what I’m talking about,’” his wife recalled. “And every time I’m looking at what’s going on the screen, it just sounds like a bunch of affirming, sycophantic bullsh*t.”
Exactly. To an outside observer, that’s likely what it would look like, because in some sense, that’s exactly what it in fact is.
But to the person engaging with it, it’s a revelation of the deep, secret, hidden truths that they always sort of suspected lurked at the heart of reality. Never mind that the LLM is just stringing together words and phrases most statistically likely to correspond with the prompts it’s been given - to the person feeding it those prompts, it seems like, at long last, verification of what they’ve always suspected.
I can totally see how people could get sucked in by that
As someone with a bipolar loved one, i can see exactly how this could feed into their delusions. It’s always there…even if they ran out of people to blast with their wild, delusional ideas the chat bot can be there to listen and feed back. When everyone has stopped listening or begins avoiding them because the mentally ill person has gotten more forceful/assertive about their beliefs, the chatbot will still be there. The voice in their head now has a companion on screen. I never considered any of this before but I’m concerned where this can lead, especially given the examples in the article.
Oh my god yes. The moment I read the headline, it all fell into place.
Yes - it’s necessarily pretty much the exact same effect, because the LLM, like the mentalist, is taking cues from the input it gets and making connections and feeding back whatever is most (statistically) likely to be appropriately on-topic.
And exactly as with a mentalist, everything that approaches what they want to hear is going to get an encouraging response, and likely further prompts which serve to narrow it down even further, and make it even easier to tell them even more precisely just what they want to hear…
I had never thought about any of this before, but it actually makes perfect sense.
By its nature, an LLM feeds back some statistically close approximation of what you expect to see, and the more you engage with it (which is to say, the more you refine your prompts for it) the closer it necessarily gets to precisely what you expect to see.
Exactly. To an outside observer, that’s likely what it would look like, because in some sense, that’s exactly what it in fact is.
But to the person engaging with it, it’s a revelation of the deep, secret, hidden truths that they always sort of suspected lurked at the heart of reality. Never mind that the LLM is just stringing together words and phrases most statistically likely to correspond with the prompts it’s been given - to the person feeding it those prompts, it seems like, at long last, verification of what they’ve always suspected.
I can totally see how people could get sucked in by that
As someone with a bipolar loved one, i can see exactly how this could feed into their delusions. It’s always there…even if they ran out of people to blast with their wild, delusional ideas the chat bot can be there to listen and feed back. When everyone has stopped listening or begins avoiding them because the mentally ill person has gotten more forceful/assertive about their beliefs, the chatbot will still be there. The voice in their head now has a companion on screen. I never considered any of this before but I’m concerned where this can lead, especially given the examples in the article.
https://softwarecrisis.dev/letters/llmentalist/
Oh my god yes. The moment I read the headline, it all fell into place.
Yes - it’s necessarily pretty much the exact same effect, because the LLM, like the mentalist, is taking cues from the input it gets and making connections and feeding back whatever is most (statistically) likely to be appropriately on-topic.
And exactly as with a mentalist, everything that approaches what they want to hear is going to get an encouraging response, and likely further prompts which serve to narrow it down even further, and make it even easier to tell them even more precisely just what they want to hear…
Wow…