Armin Ronacher: Interpreting Pangram

The useful part

Yesterday David Sacks wrote a tweet and within a few minutes people did, what they usually do, and they asked Pangram if it was AI. Now Pangram has a pretty low false positive rate, but if you have ever used an LLM as a writing assitant, you will have probably noticed that it claims your posts 100% AI, even though you don't feel like they are. Pangram itself is a trained model, that attempts to detect segments of text as being definitely human, definitely AI and a mixture of the two.

How it works

• If you want to know how it works, they published a paper.
• An LLM is then tasked to understand the text and write a fresh new text on the same topic.
• Pangram claims their model to have rates of 0.0041% false AI accusations and 0.34% missed AI text.
• So now that we know this I figured it might be fun to have an LLM re-create David's tweet.
• I'm sure if you ask Pangram about if the above text is AI, it will probably say so, but that's not really the point.

What to take from it

The point is that I then used Opus 5 to generate a text which reads entirely AI generated.

Details worth keeping

To which David replied that these AI detectors are bogus.