Substack's Pangram Detector: Nobody Knows How It Reads Arabic

This week, Substack turned its readers into examiners. A single button now shows anyone the probability that the post they're reading was written by AI. The tool is called Pangram, and Substack writers are furious — some calling it a return of the witch hunt, after it flagged one writer's own hand-written essay as 100% AI, then later scored the same piece 100% human.

Every bit of that debate, from 404 Media to The Atlantic to dozens of Substack writers themselves, has played out entirely in English. Nobody has asked the question that actually matters to any Arabic writer thinking about starting a newsletter there: how does this detector handle Arabic text specifically?

What exactly did Substack launch?

A partnership with Pangram gives any reader a button that scans posts longer than 100 characters and shows an AI-probability score. Writers can disable the feature for their readers, or add an optional "How I made this" note explaining any AI use.

What do we actually know about Pangram's accuracy?

The company says its false positive rate is under 1 in 10,000. But numbers like that have a poor track record: Turnitin also claimed a false positive rate under 1%, and a Washington Post investigation later found the real-world rate exceeded 50% in some cases. That gap between the claimed number and real performance is exactly what's worrying Substack writers now.

There's a more precise precedent: a Stanford study tested seven AI detectors on essays written by humans who speak English as a second language. On average, 61% of those fully human essays were misclassified as AI-generated, and one detector hit 97%. The writer used AI for none of it — the problem was their writing pattern, not its origin.

So why don't we know how it handles Arabic specifically?

Because, as of this writing, neither Pangram nor Substack has published any Arabic-specific accuracy figure. The company says it trained its model on large volumes of human text written before 2021, but no public breakdown exists showing what share of that was Arabic, or how it performs on Modern Standard Arabic versus dialect versus bilingual writers. The absence of a number isn't proof of innocence or guilt — it's simply an unstated unknown.

What we do know from Arabic-detection research more broadly (see our full breakdown on Arabic detector accuracy) suggests the gap seen with English-as-a-second-language writers in the Stanford study could be even wider for Arabic, since the divergence is in whole sentence structure, not just vocabulary.

What's actually at risk for an Arabic writer on the platform?

• Your reader sees the score before reading a word of your post, making suspicion the starting point instead of the conclusion.
• There's no announced channel to correct an Arabic-specific misclassification, beyond the general reporting option available to everyone.
• Your newsletter may be the one place your Arabic-speaking audience reads about you at all, so a single wrong score hits your credibility directly with that specific audience.

What should you do now if you write in Arabic on Substack?

1. Turn on the "How I made this" note and disclose any AI use clearly, even if it was only for editing, not full drafting.
2. Keep sequential, timestamped drafts of your writing outside the platform — that's stronger evidence than any score a detector shows.
3. If a fully hand-written piece gets flagged, don't argue with the number; present your writing process itself as the evidence.
4. Remember any detector, this one included, produces a probabilistic guess, not a verdict — and error rates climb specifically for non-English and bilingual writers.