LLMs are trained by taking a passage of text and masking out the next words. The LLM has to guess what the next word is going to be.
If you use the output of a fancy ass billion dollar model as your training data, you can duplicate the output style and “knowledge” of the parent model if you show it enough responses. That’s basically what Alibaba did. They prompted the shit out of Claude and used the responses to train their own model which allows you to piggyback off of Claude’s hard work pirating the entire internet. Your cloned model can also be smaller and leaner, being cheaper to operate.
I said this elsewhere but it’s like taking a block of metal and showing it Porsche 911s until it turned into a Porsche 911 with 95% of the performance, and it also costs ⅕ the cost to maintain and fuel it.
here’s the thing with the Porsche analogy. you had to buy or rent the Porsche first. paid for it and used it exactly within the TOS outlined in the contract. no law was broken.
what Anthropic is arguing is that anything their model comes up with remains Anthropic IP. this means they will literally need to sue every single one of their customers first, before they even have a snowballs chance in hell of pursuing Alibaba.
they already set the precedent by not legally pursuing their customers that use paid content generated by their model, and it automatically becomes the property of the end user.
It’s approximate but yeah you can get roughly in that ballpark. The biggest benefit is making the model weights smaller and cheaper to run. You can fit 5X as many instances on the same server if you distill down while having basically the same output.
The main caveat is you need to absolutely hammer the main model with questions from all angles to try and get it to present as much of its internalized knowledge as possible. Which is why Anthropic is pissed about this since they’re barely making money off of these prompts to train a more efficient competitor (BTW this is how “mini” or other models are trained. They’re distillates)
LLMs are trained by taking a passage of text and masking out the next words. The LLM has to guess what the next word is going to be.
If you use the output of a fancy ass billion dollar model as your training data, you can duplicate the output style and “knowledge” of the parent model if you show it enough responses. That’s basically what Alibaba did. They prompted the shit out of Claude and used the responses to train their own model which allows you to piggyback off of Claude’s hard work pirating the entire internet. Your cloned model can also be smaller and leaner, being cheaper to operate.
I said this elsewhere but it’s like taking a block of metal and showing it Porsche 911s until it turned into a Porsche 911 with 95% of the performance, and it also costs ⅕ the cost to maintain and fuel it.
here’s the thing with the Porsche analogy. you had to buy or rent the Porsche first. paid for it and used it exactly within the TOS outlined in the contract. no law was broken.
what Anthropic is arguing is that anything their model comes up with remains Anthropic IP. this means they will literally need to sue every single one of their customers first, before they even have a snowballs chance in hell of pursuing Alibaba.
they already set the precedent by not legally pursuing their customers that use paid content generated by their model, and it automatically becomes the property of the end user.
95% of performance is impressive for a clone
It’s approximate but yeah you can get roughly in that ballpark. The biggest benefit is making the model weights smaller and cheaper to run. You can fit 5X as many instances on the same server if you distill down while having basically the same output.
The main caveat is you need to absolutely hammer the main model with questions from all angles to try and get it to present as much of its internalized knowledge as possible. Which is why Anthropic is pissed about this since they’re barely making money off of these prompts to train a more efficient competitor (BTW this is how “mini” or other models are trained. They’re distillates)