Not quite, actually. It is moreso training recursively on the output without any changes, i.e., Data -> Model A -> Data (generated by Model A) -> Model B -> Data (generated by Model B -> …, that leads to (complete) collapse. A single step like this can still worsen performance notably, though, especially when it makes up the sheer majority of the data. [source]
And if they train using little data, you won’t get anywhere near the chatbots we have now. If they fine-tune an existing model to do as they wish, it would likely have side effects. Like being more likely to introduce security bugs in generated code, generally give incorrect answers to other common sense questions, and so on. [source]
From what he wrote it feels like it will majorly be existing data with substitutions/corrections made in places where they deem necessary. Like when you ask about Elon it will probably spew sth along the lines of the greatest inventor of the last century, a polymath and a very successful path of exile 2 player.
Isn’t it a well known fact that training on other AI output data leads to complete collapse of the newly trained AI models?
Do you think this guy knows facts?
Yes, sometimes. He has a physics degree, he’s not an idiot exactly. Just extremely vain and narcissistic.
No he doesn’t. He has an economics degree.
He has both
Yeah a BA in Physics, which everyone who ever physicsed knows is bullshit.
Physics is a science not an art.
A BA in physics is not a physics degree. It’s a bullshit degree
Why are you being downvoted :(
I was just trying to pull your leg.
🤷♀️ Dunno, people are weird sometimes.
What was the major? Judging by what he does it’d probably be “kinematics of sex”
Not quite, actually. It is moreso training recursively on the output without any changes, i.e., Data -> Model A -> Data (generated by Model A) -> Model B -> Data (generated by Model B -> …, that leads to (complete) collapse. A single step like this can still worsen performance notably, though, especially when it makes up the sheer majority of the data. [source]
And if they train using little data, you won’t get anywhere near the chatbots we have now. If they fine-tune an existing model to do as they wish, it would likely have side effects. Like being more likely to introduce security bugs in generated code, generally give incorrect answers to other common sense questions, and so on. [source]
From what he wrote it feels like it will majorly be existing data with substitutions/corrections made in places where they deem necessary. Like when you ask about Elon it will probably spew sth along the lines of the greatest inventor of the last century, a polymath and a very successful path of exile 2 player.