You Can’t Solve AI Risk Without Solving the AI Evidence Problem

In September, AI executives voiced fears of human extinction. President Trump moved quickly to sign a “morally binding” framework for AI companies to self-police.

The gap in urgency between these two events, just days apart, leaves the public largely in the dark about AI risks. Researchers, for their part, are limited in their work by industry opacity. The situation is reminiscent of research on social media, which has long struggled with access to platform data. Today, we are at risk of forgetting lessons from that work in the face of a technology that is much more powerful, even putting aside speculation of so-called superintelligence.

AI companies’ restrictions on data and models give them significant power while limiting the ability of independent researchers or the public to assess the technology’s impact. Self-policing can work if corporate and public incentives are aligned or there is external pressure shaping behavior. External pressure can only happen if outside researchers or the public have enough visibility to evaluate impact. When companies restrict transparency, how can the public or government know if any slow-down or oversight is appropriate?

Digital trace data is one example of a useful research resource. It is, simply put, a record of online activity. Usually it is not originally intended for researchers but created for other purposes, like advertising or broadcasting a political opinion. The number of followers in a Facebook group and Google’s AI overview of search results are both digital trace data, but neither are inherently created for research.

Still, researchers have shown that this type of data can answer questions across the spectrum of human experience, including human trafficking recruitment, investor herding, acts of free expression, illicit drug market activity, and propaganda dissemination, using methods ranging from large-scale quantitative analysis to surveys and case studies.

Despite two-plus decades of research on social media, researchers are still unsettled on answers to important questions about harm or harm-mitigation. This is in part because existing resources still do not provide durable research access. An analysis of 113 unique researcher requests at the Princeton Accelerator, where my colleagues and I build and maintain tooling for research on the information environment, points to strong demand for a relatively small set of capabilities: researchers want to be able to analyze digital trace data independently, preserve it, and sometimes share it with partner researchers. That may sound like a basic requirement for scientific research, but in the social media space the problem is nowhere near solved.

While most people’s lives have become more digital, many researchers have moved away from digital trace data because it is too difficult to access or work with. Today, there are seven types of resources which researchers can draw on to study the information environment. None of these resources are without challenges; they are scattered across providers and hedged with restrictions on who may use them and for what.

The first type is company platforms and systems, to which access is usually mediated through restricted APIs or controlled research environments. Four years ago, the majority of peer-reviewed publications on the information environment focused on X/Twitter before X increased the price of academic access from free to $42,000 per month. Even when companies offer data access programs for researchers, it is limited, granted at the companies’ discretion, and often on a one-off basis. In most cases, the data cannot be retained. The lack of comprehensive and persistent access makes replicating or validating findings challenging.

Survey panels, the second type, are expensive to produce and limited to specific questions asked during a snapshot in time.

The third type, for-profit data providers, includes services which are often prohibitively expensive and don’t provide scientifically weighted datasets. As a result, researchers and readers (including policymakers) risk drawing conclusions about the data that is accessible, rather than the subject under question. The fourth type, open-source and free tools, are more accessible but suffer from the same lack of representative data.

The fifth type, archives and preservation efforts, helps; but coverage remains uneven across populations, platforms, and data types. The sixth buttresses and facilitates the others: researcher safety, legal, and ethics support includes guides, toolkits and advisory services to researchers for legal and ethical risks, intimidation, harassment, and the psychological impact of content. Researchers have faced these challenges in practice, including from social media companies directly.

That leaves the seventh resource: Europe tried to fix some of this through article 40 of the Digital Services Act (DSA), which gave European researchers a legal pathway to access social media data. In practice, though, the route is still slow and messy. Approval can take months, and companies are failing to comply. X has issued denials without explanation. TikTok has returned data with gaps. Meta omitted European politicians because they didn’t have 25,000 followers. As a result, the European Commission has issued preliminary findings against Meta and TikTok and fined X €120 million. If companies do not comply with legal penalties, a more flexible, self-policing, structure seems unlikely to be successful.

AI is even more difficult to study and evaluate than social media — and on multiple dimensions. Privacy concerns around chatbots are acute: because use feels like a private conversation with something that might feel human, users are often unusually vulnerable with AI chatbots. Still, with consent, researchers can study those interactions. But the training and datasets behind the model and the numerous decisions developers make to bring it to a customer, such as safety rules and the ongoing A/B testing, remain invisible.

There have been nascent steps to address this. For instance, the European Union just designated ChatGPT a “Very Large Online Search Engine,” triggering requirements for transparency and researcher access. But conversations where the chatbot adds its own input don’t appear to be covered.

AI might deliver enormous net benefits; it may even be a national security imperative. But that does not mean we shouldn’t systematically study and measure AI, even if the most sensitive capabilities remain restricted for safety and security reasons. Even if data access under the DSA improved, there would still be gaps and limitations, especially for non-Europeans. A global experiment is underway, where we can’t understand the effects and the companies are writing the rules.

There is currently no sanctioned route to independently study AI systems or tools at scale — not to mention their impact on people, economics, and society. After there is, then whatever is decided about guardrails and pace can be optimally informed. The United States has failed to create a data access pathway for studying social media and its effects. Will it do so for AI, even as it becomes an increasingly critical source of information?