What Data Was Your AI Trained On? xAI Doesn't Want You To Know

Generative AI is perhaps the fastest-adopted technology in human history. People use AI systems as personal assistants, life coaches, coding agents, digital coworkers, and even romantic partners. AI serves even more use cases for businesses, across every size and industry. But with all this access, we know remarkably little about the training data being used to develop these models. AI training data is a black box. Given the stakes and scale of what AI can do and how quickly it has permeated society and the economy, we need to understand what ingredients go into these models.

xAI disagrees. In fact, xAI is suing the state of California in order to keep the public in the dark about the data it uses to train its AI models. xAI described its intent by saying it's trying to protect its First Amendment rights and intellectual property. They further argued (at the district court level) that consumers don't care about training data.

This stance isn’t surprising. The history of new technologies is rife with similar industry resistance. Congress passed the Food, Drug and Cosmetic Act only after more than 100 people died from an untested drug solvent. After thalidomide caused thousands of birth defects in Europe in the 1960s, Congress required pharmaceutical companies to disclose side effects and prove drug safety and effectiveness before they sold their products and to report adverse events. It took Ralph Nader exposing dangers known to automobile manufacturers to give us safety defect disclosure laws.

While each of these laws ultimately made products safer, industry involved fought hard against them. Food manufacturers resisted nutrition labels for decades. Pharmaceutical companies called the required post-thalidomide disclosures overreach. General Motors spied on Nader. Industry consistently branded disclosure requirements as going too far, an attack on trade secrets, or a violation of some principle of market freedom. Unfortunately, the public often pays the price before the law catches up and compels industries to be more transparent.

Now xAI is running the same playbook, suing the state of California to prevent it from enforcing a run-of-the-mill transparency law known as AB 2013. The law doesn't require AI companies to reveal source code, model weights, or proprietary trade secrets. It only requires developers to publish documentation describing, in general terms, the datasets used to train publicly available AI systems. It’s the functional equivalent of an ingredient label. Having lost its bid to prevent enforcement of the law in the trial court, xAI has now appealed the case to the 9th Circuit, arguing that the disclosure requirement violates its First Amendment rights (among other things).

Courts have consistently rejected similar arguments in the past. Why? Because when a product is used by and can affect the lives of millions of people, the public should have the right to basic information about it. Courts have established First Amendment standards for “commercial speech,” both to protect the public from false advertising and to require industries to provide basic factual information to the public and the market. The only thing new about AB 2013 is the technology at issue.

AI is already causing harm that transparency laws can mitigate. Researchers found child sexual abuse material buried inside a widely used AI training dataset underpinning one of the first breakout text-to-image AI models. The public only knows this because outside researchers were able to look inside that dataset. AI hiring tools are already facing litigation (Mobley v. Workday); companies using such tools can’t assess their own exposure without some transparency. For financial professionals, training data opacity is a compliance problem, and a particularly critical one where fiduciary duties are concerned. If xAI wins this case, AI companies will be shielded from independent scrutiny even as consumers and businesses are prevented from making informed choices about what products are safest and best for their needs.

Transparency laws are a vital part of making the AI transition go well for the public and the economy. That’s why yesterday we filed an amicus brief in xAI v. Bonta, fighting back against xAI’s attack on AB 2013. Eighteen other child safety, consumer protection, and AI accountability organizations have signed on. Nine First Amendment and other scholars have also added their voices, as well as one individual. That breadth is itself an indication of how important this is for the public.

But this case matters for more than just AB 2013 and training data transparency. This is the first challenge to transparency laws as applied to the AI industry. If xAI’s First Amendment argument is successful, it becomes a roadmap for challenging every state AI transparency law: SB 53, New York's RAISE Act, Illinois SB 315, and every disclosure requirement that follows. It means any congressional efforts to require further transparency would likely face the same challenge. The outcome of this case will shape the constitutional landscape for AI regulations, period.

Transparency helps build the public trust necessary to safely diffuse new technology through our society and economy. The public’s right to make informed decisions about new technology is the legacy of every consumer protection law ever passed. When powerful companies try to make that accountability unconstitutional by twisting the First Amendment into a knot that would prevent you from knowing what’s in your breakfast cereal, we’ll keep stepping in to flag it for the courts. But lawyers can only do so much. The public has real power. We all need to stand together to demand transparency in AI.

The amicus brief referenced above is here.