I built KillSwitch, an esoteric language designed to confound LLMs and coding agents, and benchmarked the major models against it.
  • dan@upvote.au
    491·
    8 days ago

    This is interesting because even though they designed the language to be impossible for LLMs to use, Claude still scored nearly 70% on their benchmark.

    • OldMrFish@feddit.dk
      323·
      8 days ago

      Alternative headline: Claude scores 70% on benchmark in esoteric language it hasn’t been trained on. That’s basically an ad for Claude Code…

      • dan@upvote.au
        5·
        8 days ago

        Not only was Claude not trained on it, but the language is actively trying to be hostile towards it.

    • Mirodir@discuss.tchncs.de
      232·
      8 days ago

      And that’s without any of this being in its training set. I wonder if they realized how big of a self own that is.