rest were hallucinations
I’m having trouble parsing whatcha mean here if they were coding tasks. The code didn’t run? Ran but had 0 functionality? If they were non-coding tasks, then agreed, I didn’t notice it being significantly more accurate. Though I did appreciate the larger vocab. I wasn’t gonna be able to afford to keep using it once it went to API pricing anyway.
I just use the web app, mostly to make self-contained html toys. During the brief period it was up and part of the general subscription, I asked it to make Terrace (the old board game from TNG) with very little help in the initial prompt. I kinda know how involved that task is, cause I manually wrote a Godot version back in '20. It nailed it with only minor fixes - 3D, reactive sound and visuals, a music score that is pretty chill, with Easy, Medium, and Hard levels of AI to compete against. I have yet to beat it on Hard. Opus couldn’t touch that. I’m pretty sure the fed’s response is simple retaliation against Anthropic for not playing ball with the DoD/W, but the capability jump was definitely notable. I saw someone liken it to the jump from gpt 3.5 to 4, and I agree, if not a bit more.
Did you try it? In the few coding tasks I threw at it, it performed much better than Opus.
I was also confused. It’s oddly worded. From Anthropic’s news post:
As far as I can tell, this review preceded the gov order, and it was at this point that Katie Moussouris was involved.