There’s actually a perplexity improvement parameter-to-paramater for BitNet-1.58 which increases as it scales up.
So yes, post-training quantization perplexity issues are apparent, but if you train quantization in from the start it is better than FP.
Which makes sense through the lens of the superposition hypothesis where the weights are actually representing a hyperdimensional virtual vector space. If the weights have too much precision competing features might compromise on fuzzier representations instead of restructuring the virtual network to better matching nodes.
Constrained weight precision is probably going to be the future of pretraining within a generation or two looking at the data so far.
There is some research being done with fine tuning 1-bit quants, and they seem pretty responsive to it. Of course you’ll never get a full generalist model out of it, but there’s some hope for tiny specialized models that can run on CPU for a fraction of the energy bill.
The big models are great marketing because their verbal output is believable, but they’re grossly overkill for most tasks.
Try using a 1-bit LLM to test the article’s claim.
The perplexity loss is staggering. It’s like 75% accuracy lost or more. It turns a 30 billion parameter model into a 7 billion parameter model.
Highly recommended that you try to replicate their results.
https://xkcd.com/2934/
But since it takes 10% of the space (vram, etc.) sounds like they could just start with a larger model and still come out ahead
There’s actually a perplexity improvement parameter-to-paramater for BitNet-1.58 which increases as it scales up.
So yes, post-training quantization perplexity issues are apparent, but if you train quantization in from the start it is better than FP.
Which makes sense through the lens of the superposition hypothesis where the weights are actually representing a hyperdimensional virtual vector space. If the weights have too much precision competing features might compromise on fuzzier representations instead of restructuring the virtual network to better matching nodes.
Constrained weight precision is probably going to be the future of pretraining within a generation or two looking at the data so far.
There is some research being done with fine tuning 1-bit quants, and they seem pretty responsive to it. Of course you’ll never get a full generalist model out of it, but there’s some hope for tiny specialized models that can run on CPU for a fraction of the energy bill.
The big models are great marketing because their verbal output is believable, but they’re grossly overkill for most tasks.