India Bets on AI Detection. Every Regulator Should Watch What Happens Next.

Mahsa Alimardani is associate director, Jacobo Castellanos is senior coordinator, and Bruna Martins dos Santos is policy and advocacy manager on the Technology, Threats and Opportunities program at WITNESS.

In late December, a blurry video captured a protester facing down security forces in Tehran. Someone used AI editing tools to sharpen the image and make it more shareable. The Iranian regime pointed to the visible AI artifacts and dismissed the photo as fabricated. The moment was real, verified from multiple angles and by independent fact-checkers, but without any record of what was edited and what the original showed, the regime's narrative stuck. An Israeli Persian-language account then shared the edited version, enabling the regime to spin a broader conspiracy narrative that the protests themselves were manufactured by a foreign enemy.

As documented in The Atlantic, the case illustrates what happens when there is no verifiable record of how content was made and enforcement depends entirely on detecting manipulation after the fact. Detection produced a binary answer that AI was involved, but that collapsed the difference between an image fabricated entirely by AI and an authentic photo sharpened using AI tools. A provenance record showing what was changed and what the original source was would have made it far harder to dismiss the documentation outright. On Friday, February 20, just as India wraps its AI Impact Summit in New Delhi, India's IT Amendment Rules 2026 on synthetically generated information (SGI) or synthetic audio-visual content will come into force, and they risk replicating this problem at the level of national regulation.

India is not the first country to regulate synthetic media. China's Deep Synthesis Provisions have been in force since January 2023, and South Korea has enacted targeted measures against non-consensual synthetic intimate imagery. But India is the first to make platforms' legal protections depend on whether their AI detection tools work. This is essentially a high-stakes enforcement experiment that regulators drafting the EU AI Act's Code of Practice on provenance and transparency, and legislators implementing California's AI Transparency Act (SB 942, as amended by AB 853) for synthetic content, should be watching closely.

The detection gamble

India’s new IT rules on SGI require platforms to deploy automated tools to verify whether content is synthetically generated, and to act on the results. If a platform is found to have knowingly allowed synthetic content that violates the rules, it risks the loss of safe harbor protection under Section 79 of India's IT Act. As the Internet Freedom Foundation has argued, these proactive monitoring obligations invert the logic of safe harbor by shifting classification burdens onto platforms and users. Platforms must also act on takedown orders within three hours, or two hours for intimate imagery.

The problem is that the tools these obligations depend on are not ready for this role. WITNESS's TRIED benchmark demonstrates that current AI detection tools produce inconsistent results across file types, modalities and contexts, with significant false-positive and false-negative rates. Our Deepfakes Rapid Response Force has documented these limitations in real-world verification, including in the midst of protests and conflicts. Detection tools are improving, but they are not reliable enough to serve as the basis for legal compliance. And when platforms face losing their legal protections for getting it wrong, the incentive is to remove first and review later.

The liability structure is asymmetric. A platform that fails to remove synthetic content risks losing safe harbor protection. A platform that wrongly removes authentic content faces no consequence: the rules provide no requirement for user notification, no appeal process, and no liability for wrongful removal. When this asymmetry meets unreliable detection tools, the rational platform response is systematic over-removal.

Provenance should be interoperable, secure and rights-respecting

Effective provenance infrastructure must be interoperable across platforms and borders, secure through mechanisms like cryptographic signing, and rights-respecting with privacy safeguards built in by design.

If detection tells you whether something might be synthetic, provenance tells you what actually happened to it: what tool was used to capture, edit or generate it, what was changed, and when. India's rules get this half right. They require synthetic content to carry permanent metadata including a unique identifier. But the rules don’t reference any existing open standards such as C2PA, JPEG Trust, or ISO 22144. Each platform is left to build its own scheme, which means provenance data doesn't travel across platforms or borders. The rules specify no way to verify that provenance metadata is authentic. Rule 3(3)(b) prohibits removing metadata, but nothing prevents someone from changing what it says. The label is protected; the information inside it is not. And the unique identifier traces to the platform, not the content process. It records which system distributed the content, not what the AI did.

Compare this with China, which introduced a mandatory national technical standard (GB 45438-2025) alongside its September 2025 labelling measures, with obligations falling on AI service providers, not just distribution platforms -a stronger pipeline architecture. China's standard is internally standardized and technically secured through provider-level obligations, but it fails on both interoperability and rights. Its provenance system operates within an authoritarian governance structure: mandatory real-name registration, log retention and reporting obligations under the Deep Synthesis Provisions create a direct path from content to individual identity. Provenance built for traceability rather than transparency is a surveillance tool. India does not replicate China's political context, but shares key design features: no data minimization, and provisions allowing disclosure of user identity to private complainants.

In our engagement with the EU AI Act's Code of Practice on Transparency, WITNESS has advocated for a "recipe" approach to provenance: understanding the ingredients of AI and human contribution in content and how they were combined. Crucially, this approach must be designed with privacy as a core principle. Provenance data should never embed personally identifiable information by default, and any contextual data must be minimized and protected. C2PA's Content Credentials are the closest existing standard to meeting all three requirements: interoperable by design, secured through cryptographic signing, and compatible with rights-respecting implementation through data minimization. Content Credentials are not without limitations; no provenance system can guarantee completeness, and metadata can be stripped or spoofed. But an imperfect open standard that travels across platforms is a more reliable foundation for enforcement than detection tools operating in isolation.

The pipeline is broken

If the editing tool used on the Tehran protest photo had embedded a provenance record showing what was changed and what the original source was, the authentic documentation underneath would have remained verifiable. Instead, detection carried the full weight, and it was weaponized against the very people it should have protected.

The rules place the entire obligation on intermediaries, while the AI developers and model providers who build the tools that generate synthetic content face no corresponding requirement to embed provenance at the point of creation. In our submission to MeitY, WITNESS called for shared responsibility across the AI pipeline: developers, deployers and intermediaries each contributing to transparency at their point in the chain. The final rules did not adopt this approach. Detection is left to do the work that provenance infrastructure should be making easier, using tools that cannot yet do it reliably, with platforms' legal protections at stake.

What other regulators should take from this

India's rules are an early answer to design questions that other regulators are grappling with. The drafters of the EU AI Act's Code of Practice are still making foundational choices about detection tool access, the scope of provenance marking, and compliance accountability. Implementers of California's AI Transparency Act, now signed into law, have settled on a stronger architecture that distributes obligations across generative AI providers, platforms, and capture device manufacturers, but its effectiveness depends on which standards the market adopts. India shows what happens when the architecture itself is wrong: provenance without interoperable standards, liability without safeguards, and a pipeline that asks platforms to do what developers should be doing at the point of creation.

None of these frameworks are finished. India's rules are in force but correctable. The EU Code of Practice is still in draft. California's architecture is set but its standards are not. And for regulators across the global majority who often look to India as a model, the design gaps here risk being replicated before they're corrected. All three can still get this right. The ambition is there. The design work remains.