I’m not convinced that they even know 100% how Anthropic is doing it. I can think of an easier way that doesn’t corrupt the text: just find a bunch of tokens where there is a good spread of token possibilities, and the more often the most likely one is chosen, the more likely it’s AI.
That being said, it doesn’t seem much different from what any of us do to identify AI text — it has lots of tells anyway.
That’s how AI testers work and its why they don’t. Most forms of formal writing are predictable by design. If the AI can predict predictable formulaic writing, it doesn’t mean its AI, its probably just any form of professional writing other than fiction.
Famous public domain works will always be considered AI by those tests, because of course your LLM knows the american national constitution. It was in the training data, so it can predict it with 100% accuracy, therefore your test wrongly calls it AI.
The difference between what you describe and what I describe, is that a 100% match isn’t a hit. Nor is a 90/7/2/1. You need something with meaningful variability. Even within formal papers there are places where word choice is arbitrary as the article explains.
Of course, you’re lacking the context of the full prompt and just feeding in the raw text. Again it gets way more reliable the more text you have.
But it’s moot because the more text you have the more tells will sneak in and you probably don’t even need an AI checker. Those phrases that AI loves but humans use comparatively rarely. It’s not a tell — it’s the whole game!
I’m not convinced that they even know 100% how Anthropic is doing it. I can think of an easier way that doesn’t corrupt the text: just find a bunch of tokens where there is a good spread of token possibilities, and the more often the most likely one is chosen, the more likely it’s AI.
That being said, it doesn’t seem much different from what any of us do to identify AI text — it has lots of tells anyway.
That’s how AI testers work and its why they don’t. Most forms of formal writing are predictable by design. If the AI can predict predictable formulaic writing, it doesn’t mean its AI, its probably just any form of professional writing other than fiction.
Famous public domain works will always be considered AI by those tests, because of course your LLM knows the american national constitution. It was in the training data, so it can predict it with 100% accuracy, therefore your test wrongly calls it AI.
Testing for AI writing that way does not work.
The difference between what you describe and what I describe, is that a 100% match isn’t a hit. Nor is a 90/7/2/1. You need something with meaningful variability. Even within formal papers there are places where word choice is arbitrary as the article explains.
Of course, you’re lacking the context of the full prompt and just feeding in the raw text. Again it gets way more reliable the more text you have.
But it’s moot because the more text you have the more tells will sneak in and you probably don’t even need an AI checker. Those phrases that AI loves but humans use comparatively rarely. It’s not a tell — it’s the whole game!