It doesn’t literally mean “90% failure rate vs 100%”
In the HLE benchmark, for example, we may be talking about 45% test score for MiMo Pro or GLM or something, vs 55% for Sonnet and 61% for maxed out Claude.
But further complicating that, those particular open weight models do better than Claude in one particular hallucination benchmark, meaning Claude/ChatGPT might be worse at RAG/extraction type workloads. It’s like this across the board, with different models trading blows in different areas.
By “90%,” he really means “it’s so close the better model depends on the particular application, workload, harness, sampling configuration” and such, and that none of these models are close to “100% accuracy” across the board.
One other thing: Anthropic, OpenAI, and Grok have all been accused of “dumbing down” models under high demand to speed up inference. If your application depends on them, and they are the only provider, that could throw a wrench in the workflow.
But open weights? Pick your provider. You get an API as consistent as you are willing to pay for.
In other words, that “100%” isn’t really representing the potential inconsistency one gets being locked to a single provider.
It’s his examples that I’m going off. But maybe he doesn’t know what he’s talking about and an IBM engineer is losing their shit at upper management right now. I used to work for them a long time ago under Virginia and that was standard… Though, so has been every other place since.
I give IBM credit because so many of the people there are ridiculously smart. Not their business people.
It doesn’t literally mean “90% failure rate vs 100%”
In the HLE benchmark, for example, we may be talking about 45% test score for MiMo Pro or GLM or something, vs 55% for Sonnet and 61% for maxed out Claude.
But further complicating that, those particular open weight models do better than Claude in one particular hallucination benchmark, meaning Claude/ChatGPT might be worse at RAG/extraction type workloads. It’s like this across the board, with different models trading blows in different areas.
By “90%,” he really means “it’s so close the better model depends on the particular application, workload, harness, sampling configuration” and such, and that none of these models are close to “100% accuracy” across the board.
One other thing: Anthropic, OpenAI, and Grok have all been accused of “dumbing down” models under high demand to speed up inference. If your application depends on them, and they are the only provider, that could throw a wrench in the workflow.
But open weights? Pick your provider. You get an API as consistent as you are willing to pay for.
In other words, that “100%” isn’t really representing the potential inconsistency one gets being locked to a single provider.
It’s his examples that I’m going off. But maybe he doesn’t know what he’s talking about and an IBM engineer is losing their shit at upper management right now. I used to work for them a long time ago under Virginia and that was standard… Though, so has been every other place since.
I give IBM credit because so many of the people there are ridiculously smart. Not their business people.
I’m willing to bet that “90%” and “100%” have absolutely no concrete meaning here.
“90% as good for a fraction of the cost” is a set/premade phrase and so the numbers mean nothing, since the person who said it didn’t come up with it.
Like, he extrapolates off of it but I’m willing to bet the number is from the expression.
No one said 90% failure rate. You brought that up. Reread the post.
You might be lost, Shillsville is 20 miles that way.
You said:
So I interpreted that as an implication of “90% accuracy.”
I’m sorry I misinterpreted you, but I’m not sure why you have to be so abrasive calling that out? “Shillsville?” ???