Demonstrators wear masks depicting Meta CEO Mark Zuckerberg, Brazilian presidential candidate Flavio Bolsonaro and US President Donald Trump during a protest outside Meta's headquarters in Sao Paulo, Brazil, on October 19, 2026. The protest was called over allegations that restrictions on Instagram affected the campaign of Brazilian President Luiz Inacio Lula da Silva ahead of the presidential runoff election. Meta attributed the reported issue to a technical failure and denied political bias. (Photo by Allison Sales/NurPhoto via AP)

Brazil Shows Platform Illiberalism is Big Tech’s New Normal

Grok app icon is seen next to the Meta AI and Claude app icons on a smartphone home screen in this illustration photo taken on October 27, 2026. (Photo by Matteo Della Torre/NurPhoto via AP)

The Solution to AI’s Alignment Problem is Precise Guardrails on How We Use It

A stock market chart with lines in different colors on a dark screen with lines trending downwards to the right. 

Chart on a dark screen by Maxim Hopman on 

Will Prediction Markets Affect Democracy? Take a Bet

Canada's Prime Minister Mark Carney, front left, speaks with European Commission President Ursula von der Leyen at the European Parliament in Strasbourg, eastern France, Sept 17, 2026. (AP Photo/Pascal Bastien, File)

Canada and Europe Need a Way to See the AI They Cannot Access

A surreal collage depicting a dishevelled pile of items arranged like a traditional vanitas painting. To the right is a vintage computer monitor displaying the OpenAI logo. Objects representing various instances of AI “slop” litter the scene, such as an idol of Jesus with a shrimp replacing its head, and an anthropomorphised cappuccino cup with cartoonish eyes and lips.

Slop Vanitas II by Georgina Leeuwendal / 

Creating a New Internet Bargain to Preserve Fair Use and Protect the Commons

A close up of a spider in its web. Valeriia Neganova/Unsplash

A close up of a spider in its web. Valeriia Neganova/

Breaking the Open Web Won’t Stop the Bad Bots

Prime Minister Andy Burnham (left) meets with US President Donald Trump during his visit to the US to attend the United Nations General Assembly on Sept. 22, 2026, in New York. Photo: Toby Melville/PA Wire POLITICS (Press Association via AP Images)

The UK-US Tech Deal Exposes the Limits of Britain's AI Ambitions

A crowd of supporters surround President Lula holding a microphone at a podium draped with the Brazilian flag.

President Luiz Inacio Lula da Silva delivers a victory speech after winning the presidential runoff in Brazil, Oct. 25, 2026. (AP Photo/Andre Penner)

Lula’s Win Signals Fight to Regulate Social Media Will Escalate in Brazil

A data center is under construction in Loudoun County, Virginia on October 7, 2026. The region houses the world's largest concentration of data centers, making Virginia the nation's digital capital. (Photo by Olivier Douliery/Sipa via AP Images)

What to Do About the Coming Glut of AI Data Centers

Tesla and SpaceX CEO Elon Musk receives the National Medal of Science from US President Donald Trump during the Golden Age of American Innovation Summit at the Institute of Peace on October 08, 2026 in Washington, DC. (Photo by Win McNamee/Getty Images)

‘The Greatest Capitalist In History’? A Review of Alex Gibney’s ‘Musk’

Secretary of Defense Pete Hegseth gives a State of the Force address at Marine Corps Base Quantico, Wednesday, Sept. 30, 2026 in Quantico, Va. (Alex Wong/Pool Photo via AP)

The Pentagon Has a Big Tech Problem

An entrance driveway to a hospital with a sign pointing to Emergency, Laboratory, and Radiology. A person is walking past from the left, and another person is leaving the hospital on a motorbike. 

A street view of The Nairobi Hospital in Kenya on Oct. 6, 2026 (AP Photo/Andrew Kasuku)

Without an Exit Plan, AI Partnerships in Africa Risk Becoming the New Tied Aid

Voters line up at a polling station during the presidential runoff in Brasilia, Brazil, Sunday, Oct. 25, 2026. (AP Photo/Eraldo Peres)

How to Reclaim the Future from Strongmen and Oligarchs

The Edge Dialogues with Dr. Alondra Nelson

Demonstrators holding a banner that reads in Portuguese "Brazil decides about Brazil" rally against Meta over allegations of interference in Brazil's presidential election, outside Meta's headquarters in Sao Paulo, Monday, Oct. 19, 2026. (AP Photo/Andre Penner)

On Eve of Vote, Brazil’s Federal Prosecution Office Demands Answers from Meta

Demonstrators wear masks depicting Meta CEO Mark Zuckerberg, Brazilian presidential candidate Flavio Bolsonaro and US President Donald Trump during a protest outside Meta's headquarters in Sao Paulo, Brazil, on October 19, 2026. (Photo by Allison Sales/NurPhoto via AP)

In Brazil’s Runoff, Lula Faces Bolsonaro, Trump—and Big Tech

A grey toned abstract photo of wavy mirror reflections of a building seen from many different angles. 

LLMs Are Still Making Election Deepfakes Just Days Before the US Midterms

A view of the US Capitol building with the dome brightly illuminated against the backdrop of a night sky. 

The United States Capitol dome by night, Washington DC, 2019. Photo by Darren Halstead on 

A Congressional Commission on AI Risk is Overdue

A demonstrator wearing a yellow vest waves the French flag on top of a burning barricade on the Champs-Elysees avenue with the Arc de Triomphe in background in Paris. (AP Photo/Michel Euler, File)

France’s Protests Reveal Europe’s Expanding Surveillance of Dissent

Building of the European Parliament in Strasbourg, France, with the EU-flag.

Building of the European Parliament in Strasbourg, France

European Parliament Divided on How to Govern AI At Home and Abroad

Our organizations, Public Knowledge and Creative Commons, share a vision of an open internet and a thriving and vital digital 

. Over the past year, we’ve come to share a growing concern: that the old bargain built on the open web — content from creators and publishers in exchange for acknowledgement and traffic from platforms — has broken down, and the values of 

 and sharing that have been essential to build and sustain a thriving knowledge commons have been abandoned.

This is most acute right now in the realm of generative artificial intelligence. Technology firms with highly concentrated power are expanding their role as information intermediaries, locking up data through exclusive agreements, at times ignoring access signals imposed by site owners, circumventing rate limits, or masking their origins. Online creators and publishers — collectively, rights holders — who feel their copyrighted content is being exploited are putting up their own defenses, including licenses, lawsuits, blocking, and attempts at legislation. The result is a fragmented information ecosystem, and an already-measurable decrease in the openness and availability of online information. The decrease in availability is compounded by threats to the 

 of information in the knowledge commons. Research shows the more valuable the content — most often human-made — the more likely it is to be held back.

Underlying all of this is an emerging new exchange model for the web — a new internet bargain — that without intervention may result in the closing of the commons for even the most beneficial uses. AI model developers and online rights holders should be motivated — and, where necessary, incented or required by policymakers — to change these trends.

This post outlines the forces that are threatening the digital knowledge commons. These include 

 — the commodification of human knowledge resources as free raw materials by dominant technology actors — and 

 — restrictions or demands imposed by online rights holders in self-protection against what they perceive as exploitation. We also outline the dynamics that are 

 the knowledge commons. We show how, combined, these dynamics are creating a new, unsustainable economic bargain for the open web that may benefit large AI model developers and online rights holders while shutting out everybody else. Lastly, we propose a set of principles for developing solutions to address the impact of these forces on the knowledge commons.

What is the ‘knowledge commons’, and why does it matter?

The knowledge commons in this context refers to information and cultural resources available to all members of society. It encompasses the vast variety of information on the open web, and is best exemplified by free digital media including Wikipedia, Internet Archive, Creative Commons-licensed creative content, the digital collections of libraries, openly licensed research and education, open scientific collections such as the 

, and much online journalism. A thriving knowledge commons available to all is essential for advancing creativity and imagination, enabling innovation, solving complex problems, holding the powerful to account, supporting democratic systems, and for human flourishing in general. It is also critical for information integrity, as access to quality knowledge is key to counteracting information pollution.

Contemporary threats to the knowledge commons

Thanks to generative artificial intelligence and its reliance on enormous quantities of training data, AI firms are rapidly becoming the new information intermediaries. These firms, some of which grew out of social media and search companies, rely heavily on datasets scraped from the internet. They are motivated to lock up the highest-quality and most proprietary data — the best raw materials — to ensure their models excel on performance benchmarks and offer distinctive advantages to consumer or enterprise users.

Accordingly, they’ve adopted methods like 

 to secure for themselves data that used to be more democratically accessible. They consistently lobby against data transparency laws by arguing that disclosure mandates violate trade secrets as well as free speech. They’ve also made themselves guests where they’re not invited, 

 from rights holders or masking their origins to access these materials. Some have the 

“Let Google do the Googling for you.”

 The combined effect is a set of information moats as deep or deeper than those in social media and search. We refer to this dynamic as 

Online rights holders, particularly publishers, must feel like they’ve seen this movie before. To them, the plot consists of technology firms basing their business models entirely on legal and free access to online information, and capturing traffic and revenue from rights holders in the process. The 

 from web crawlers has plummeted, and publishers are losing their opportunity to monetize web traffic in the way of the old bargain. Publishers see this as exploitative, and in some cases they may be right: 

 that 30% of all web scrapes violate explicit Robots Exclusion Protocol, or robots.txt, instructions designed to manage access. (Alternative technical signaling mechanisms, like llms.txt, the “noai” meta tag, and the TDM Reservation Protocol appear to have minimal uptake.)

In two past blog posts, Public Knowledge described the specific 

impacts of generative artificial intelligence

 on the business models of online publishers and reviewed a number of 

solutions that online publishers are adopting

 to mitigate them. Publishers, too, are using an array of mechanisms to insulate their commercial assets. News publishers have been a primary source of training data for AI models and their business model has been highly dependent on discovery and referral from search and social intermediaries, so they are in the forefront of responses. The largest news publishers — and an increasing number of small ones — are happily entering into licensing contracts. Some are suing: depending on the 

, there are almost 100 lawsuits from online publishers against AI model developers. The publishers are pursuing protection from Congress: at last count, there were seven bills at the federal level alone that would legislate away fair use for AI training, restrict the use of copyrighted content for it, or require publicly accessible registries of copyrighted content used to train models. Some publishers are attempting to sequester their content through financial means, such as paywalls and subscription models; legal means, such as changes to their terms of service; and technical means, such as the use of robots.txt and HTTP headers. News publishers are 

In this context, new market entrants — technology intermediaries focused on detecting, tolling, and blocking bots and crawlers — are emerging to address these new economics and correct for information asymmetry. But they also have the potential to become 

 of the kind we experienced in advertising technology. Artists, too, have been using techniques such as 

 or poison AI models. We refer to these strategies collectively as 

AI slop, LLM poisoning, and model breakdown

An accelerating volume of research shows another impact of generative artificial intelligence on the knowledge commons: deteriorating information 

Examples are AI slop, LLM poisoning, recursive degradation, and the platforms’ own content moderation.

 for 2025) is the colloquial name for the vast amounts of low-quality, generative AI-produced content gushing onto the internet. It may be the product of low-effort or low-sophistication AI users, or it may be mass-produced to farm engagement or earn advertising revenue. It’s 

 that a significant portion of the content users see on popular sites like LinkedIn and X is actually AI spam. 

 showed that three-quarters of new webpages have some form of AI-generated content, and 

 showed that the number of articles published on the internet that are primarily AI-generated (50%) is equal to the number written by humans (50%). 

 showed that almost 10% of newly published news articles — and six times that rate for published op-eds — are either partially or fully AI-generated.

AI slop is a detrimental input to training models, and AI model developers are going to what seem like extreme (but legal) measures — like 

 of books — to avoid it. Slop also hurts the knowledge commons and public trust, which is why content platforms like 

 have introduced features that let readers report, assess, or avoid it. According to a 

, TikTok, X, YouTube, Apple, Instagram, and Pinterest are also evaluating strategies to manage AI slop, while researchers assess it as “a systemic problem with our information environment.” AI-generated 

 with a surge in low-quality research reports, peer reviews, and news sites; declining scientific innovation; declines in quality of patent applications; and a rising rate of required retractions of published papers.

,” bad actors flood the web with articles containing false or misleading narratives 

, in the hope that commercial generative AI products will reflect them in their responses. It can poison training models when AI developers scrape data from troll farms or automated disinformation sites to train their models, or when they use real-time web search to ground or answer a specific user query from these same sites. This has become such a significant problem that it has opened a potential market for AI systems whose sole 

 is that they are trained on reputable information. Imagine if large technology companies had disclosed their training sources all along. That could have steered us towards quality on all fronts, limiting the mess we now find ourselves in. Instead, lawmakers around the world are considering legislation that would 

 AI developers to disclose the sources they use for training.

Both of our organizations are interested in the benefits of transparency. But Public Knowledge, in particular, is wary of legislation that threatens fair use by providing plaintiffs with a pre-discovery tool that would bring about a continuous wave of litigation risk. Our organizations will be following the implementation of the transparency requirement in the EU AI Act closely, which will only be enforced for legacy models later next year.

A couple of years ago, a technical paper got a lot of buzz for advancing the idea of 

, in which AI models trained repetitively on AI-generated data break down and eventually “collapse.” Like repeatedly photocopying a document, errors and biases accumulate until the model produces nonsensical, homogeneous outputs. While actual model collapse may happen only under highly specific experimental conditions, there is support for the broader idea of recursive degradation. It refers to a degenerative process where successive generations of AI models trained recursively on data generated by other AI models lose accuracy, diversity, and reality grounding. While testing methods and outcomes have varied, research consistently shows that generative models will inevitably degrade without a steady supply of new human data. Whatever form it takes, recursive degradation raises important questions about the long-term sustainability of the knowledge commons and points to the need for motivated self-interest on the part of AI model developers to protect it. It also hurts competition, since AI model developers entering the market after the widespread launch of ChatGPT in 2022 are invariably working with some synthetic data.

AI model developers also use various output 

 from overly violent or sexual outputs or to insulate themselves from financial, legal, or political pushback. This output filtering may result in a form of AI slop. For example, AI guardrails intended to make models safe, polite, or compliant may result in low-quality, generic, or repetitive output. It’s also been demonstrated that when digital platforms generate their revenue through an advertising-based business model, they largely shape their content moderation practices to maximize engagement. That often results in an algorithmic emphasis on low-quality, inaccurate, and inflammatory content. To the extent that dominant AI firms extend (like 

 an advertising-based business model, their outputs will ultimately be shaped by the same incentives to capture and hold attention that we see in social media.

The blunt tools in use on either side of the AI developer/rights holder divide are serving to shrink the knowledge commons and erode its quality. This isn’t theoretical, AI-doomer prognosticating about some imagined AI future. The digital knowledge commons is contracting at an alarming rate, driven by the rise of AI model developers that have transformed openly shared content into a high-value industrial input and the defensive response of rights holders who are rapidly restricting access through legal, technical, and contractual means. The resulting structural economic pressures make maintaining open digital resources increasingly unsustainable. That requires, and is already creating, a new economic model for the web, one that is far less equitable with regards to access to knowledge.

Compelling evidence of the shrinking knowledge commons

Researchers are beginning to measure the shrinking commons. One significant empirical contribution is the report, “

Consent in Crisis: The Rapid Decline of the AI Data Commons

,” which dates from 2024. It measured how administrators of web domains are evolving their intentions for how their sites can be crawled and their content used. One of the key findings was that “web domains are rapidly restricting crawling and use of their content for AI”. The report noted that “in a single year (2023-2024) there has been a rapid crescendo of data restrictions from web sources, rendering ~5%+ of all tokens in C4 [an open AI training corporus], or 28%+ of the most actively maintained, critical sources in C4, fully restricted from use. For Terms of Service crawling restrictions, a full 45% of C4 is now restricted.” Results were similar for other training datasets.

The paper also identified a compositional bias that is relevant to our discussion of knowledge quality: high-quality editorial sources (like news organizations and encyclopedias) showed the highest restriction rates, while niche and e-commerce sites remained more open. If restrictions are respected (though there is ample evidence that they are not), the remaining training commons will be increasingly biased toward low-quality, commercially-motivated content — a structural degradation of the knowledge ecosystem. Issues in the consistency and effectiveness of the administrators’ efforts also pointed to a 

 crisis and the need for better preference signaling protocols.

More current research also concluded that “the boundaries of public data are rapidly shrinking.” In “

The Walling of Public Web Data and Inequality in AI Innovation

,” researchers constructed a “Web Data Enclosure Index” and demonstrated “that the boundaries of public web data have begun to shift from default readability to an explicit, tiered permission system.” The Enclosure Index (expressed as an exclusion rate) for AI training bots reached 60% for business and travel domains, 57.1% for community and knowledge-sharing sites, 40% for news and media, and 10.9% for public research (a number that should be 0% if we want to collectively benefit from scientific advancement). The researchers defined at length the negative implications for AI innovation, open-source models, and competition. Essentially, if the content required to train the next generation of models is subject to enclosure, AI innovation will slow; model quality will be constrained by those that can afford to negotiate licensing deals; open-source, open-weight, and public research models will be restricted to older, stale public domain data or polluted public web scrapes; and new businesses will face a significant barrier to entry. All of this adds up to another reason for enlightened self-interest on the part of AI developers.

Web Crawler Restrictions, AI Training Datasets & Political Biases

,” showed that a quarter of the top 1000 websites restrict AI crawlers, and that the highest-quality content sources and most politically neutral sources (based on large-scale audience characterization) were the most likely to be blocking AI crawlers. Similarly, “

Selective Participation in the AI Data Commons

” showed that more credible, more established, and more centrist publishers (using measures of fact-checking and media bias) are more likely to restrict access. As Current Affairs editor 

, “The truth is paywalled and the lies are free.” The results suggest blocking might skew training datasets toward low-quality or polarized content, which may have implications for enterprise as well as consumer use. At a time when AI outputs are 

, this effect alone should motivate AI model developers to seek solutions in their own self-interest.

Anecdotal evidence from academic platforms, artist communities, and open access publishers collected by Creative Commons also suggests that creators are increasingly choosing more restrictive CC licensing options (for example, CC BY-NC or CC BY-ND instead of CC BY or CC BY-SA) to explicitly signal that they do not want their data used by commercial AI, whether or not the license has any bearing on training. While this is an important signal for the movement to pay attention to, Creative Commons itself advises creators that CC licenses do not stop AI training because training is, in many legal regimes across the world, considered fair use or an exception to copyright law. More on that later.

Perhaps the most compelling proof point that AI model developers and rights holders are forging a new bargain is publishers’ evolving relationship with Google. Googlebot, Google’s combined search and AI crawler, “dwarfs other AI bots and crawlers in volume,” 

. As AI Overviews top more and more user queries, referral traffic from Google has fallen so low that some of the world’s largest publishers are considering 

 that would have been considered suicidal even a year ago: 

. In their determination to block crawling, publishers seem 

 what remains of traffic from ranking in search. (

 directive in robots.txt lets publishers opt out of having their content used for artificial intelligence. But the directive governs downstream data usage, not crawling. That means Google-Extended may stop Google from using content to train its models or ground answers to user prompts, but it will still use that content in AI Overviews.) We will leave to the side the irony of 

 that scraping public search results harms rights holders in its lawsuit against SerpApi.

Case studies: Wikimedia, Common Crawl, and Internet Archive

,” researchers forecasted that the “[r]ising expressions of non-consent will affect non-profits, archives, and academic researchers.” They were right. 

 that news websites that began blocking AI crawling in 2023 also began blocking other crawlers with which they had long coexisted, including academic and research utilities. Three case studies show how even the entities most committed to openness and sharing — traditionally perceived as the “good guys” of the open web — are being affected by these forces.

Wikipedia, a free, open-collaboration online encyclopedia written by hundreds of thousands of volunteer editors, regularly appears on lists of the world’s 10 most visited websites, hosting an 

estimated 68 million articles in almost 300 languages

 of content to train AI models that power generative AI user applications like Gemini, Claude, and ChatGPT. Despite the 

 (the nonprofit which hosts Wikipedia) for attribution and financial support for AI scraping, AI model developers don’t always follow the rules. This has had 

 for Wikipedia: sharp increases in automated traffic; overloaded systems; increases in costs; declines in human traffic to the site (and therefore opportunities to onboard new editors and garner financial support); diminished user experiences; and then imperfect, unattributed AI outputs that pollute the information ecosystem that editors use to source the encyclopedia. Wikimedia 

 by prioritizing user access, updating its policy on automated access, improving its bot detection and defense tools, and setting global rate limits. As of earlier this year, Wikimedia was blocking or throttling about 30% of all automated requests from bots not following the rules — and promising that this rate will increase.

 asserting that “the combination of mass scraping and mass usage by some companies threatens to break th[e] social contract of the free knowledge commons” and forecasted that it will continue to review its licenses and legal policies. More recently, Wikimedia Foundation 

 associated with OpenAI that had edited its wikis and tried to exploit a public note-taking tool — both apparently attempts to use Wikimedia’s tools as proxies to fetch data from other sites — and generated heavy traffic that may have contributed to a partial outage. But Wikimedia Foundation has also built a mechanism for commercial relationships with high-volume users of their content, including AI model developers. It has established a dedicated API infrastructure — and revenue stream — for high-volume data requests. We believe this is one of several new forms of monetization — in this case, for access — that will make up a new 

 bargain for the web. Importantly, though, it does little to encourage new editors to join the community, which forms the lifeblood of the Wikipedia project.

The Common Crawl Foundation is a nonprofit organization that maintains a 

 collected by scraping hundreds of billions of web pages. Its stated goal is to make a public, open archive of the web freely available to researchers, educators, journalists, and developers. In November of 2025, The Atlantic 

 accusing Common Crawl of “open[ing] a back door for AI companies to train their models with paywalled articles from major news websites,” and pointed out that the foundation has recently received significant donations from several AI developers. 

, Common Crawl acknowledged the challenges publishers face in “balancing openness with commercial sustainability” but denied there was any relationship between the donations and scraping. It reinforced that it clearly identifies itself (CCbot) in its user agent string, does not scrape paywalled material, honors robots.txt exclusions, and complies with takedown and removal requests. It wasn’t enough. In June 2026, lawyers for Digital Content Next, a trade association for digital media brands, sent Common Crawl 

 on behalf of its members, which include the Associated Press, the New York Times, NBC Universal, Bloomberg, NPR, and Fox. Their legal argument is that US copyright law is an opt-in, not opt-out, regime, and that Common Crawl should have to ask digital publishers for access before it crawls them. The letter demands that Common Crawl stop scraping protected content, and that it remove member content already sitting in Common Crawl’s archives. The parties may wind up in court.

Internet Archive, a nonprofit online research library, is one of the internet’s most important resources. Its 

 to ensure “universal access to all knowledge” is embodied by a permanent, freely available archive of websites, texts, audio, software, and cultural artifacts to preserve digital history. It may be best known for the 

, which allows users to view historical snapshots of websites. As over 290 journalists recently expressed in an 

, the Internet Archive is a critical resource for journalists’ work and it does not engage in paywall circumvention or irresponsible scraping. Internet Archive also uses rate limiting, filtering, and monitoring to prevent abusive access. Yet it is currently, 

, “collateral damage caught up in the conflict between AI companies and publishers.” A 

, including the New York Times, are “hard blocking” the Internet Archive’s crawlers because they believe the Wayback Machine is providing a workaround for AI model developers determined to access or reuse its content. Or, they are placing restrictions on the Wayback Machine to ensure that AI models 

 instead of sites that aggregate their work. The fact that publishers are walling off their copyrighted content — despite their own reliance on Internet Archive in reporting — is a sign of how bad and unproductive things have become.

Rethinking ‘open’ for a transformed web

Given these effects, even longtime advocates in the open knowledge movement are revisiting “

” and rethinking its core concepts to prevent abuses and concentrations of power. As Creative Commons wrote in 

, “even those who have historically believed in the power of sharing knowledge…are choosing to close off.” As noted above, creators and researchers in the open education and open science communities are increasingly choosing more restrictive licenses over the most permissive ones as part of a broader pushback against the large-scale commercialization of openly shared content. Both the extractive abuses of tech firms and the defensive contraction among rights holders threaten the entire ecosystem of open knowledge — research, journalism, education, archiving — that depends on a shared infrastructure of open access.

The Digital Public Goods Alliance (DPGA), in its recent “defense playbook,” 

: “Digital public goods and the broader knowledge commons are increasingly vulnerable to blunt-force, automated extraction by AI crawlers and scrapers operating without reciprocity, breaking the implicit social contract of the open web…” The playbook recaps in detail the challenges to online rights holders that we’ve shorthanded here — ballooning infrastructure costs, loss of human traffic, erosion of human-curated information, and perpetuating power imbalances — and describes the ensuing pressures on democracy, innovation, heritage, and global collaboration. Their framework outlining “layers” of defensive measures for rights holders demonstrates that even those most committed to preserving “openness” recognize the need for new solutions that are compatible with “the structural realities of massive-scale machine consumption.”

, the nonprofit Invest in Open Infrastructure highlights some of the same challenges. It notes the strains on infrastructure, limitations of current defensive approaches, risks of data consolidation, and the tensions inherent to what are essentially now all voluntary compliance mechanisms for managing access to online content. The report points toward what it acknowledges to be “a promising, if demanding, path”: AI model developers as well as rights holders acting with enlightened self-interest to create “new partnership models that could align open knowledge strategies with commercial demand.” Our shared hypothesis is that data governance innovations and public policy can be used to encourage, or require, these new models to protect the public interest.

That last reference hints at an actionable proposition for reconciling the tension between AI model developers and rights holders in a way that protects the knowledge commons: 

that the structure of the web is changing in ways that demand rethinking the broader economic framework of the digital ecosystem, and that public policy can be a tool to ensure that framework develops in a way that serves the public interest.

Key to open-mindedness about data governance innovations or public policy is the understanding that there has

been a structural and economic change in how information is generated, distributed, and consumed online. Generative AI has had the 

 of any new digital technology in human history. There was already less reliance on the open web for information, entertainment, and transactions due to apps, commercial social media, and other dominant web platforms; now AI and agents are again changing the nature of our relationship to the open web. As noted earlier, most new 

 on the open web is created by machines, not humans. Automated 

 with human traffic and, in most months, exceeds it. Today, most online content is not read by humans, accessed via search referral in response to user queries, or shared on social media among family and friends. Instead, it is used as industrial raw material and transformed for consumption inside commercial interfaces.

In other words, most of the web no longer exists to be consumed by people. The knowledge commons, increasingly, is an all-you-can-eat buffet for machine ingestion. This isn’t about “robots reading”: the machines exponentially exceed human limits for accessing, reading, memorizing, learning from, and producing new creative work. Or, 

, the world will soon experience “the death of the reader” as “machines create slop for more machines.”

Policy solutions are already being pursued to support and enable this new economic framework. They are generally propelled by AI model developers, who want the rules to be more certain or more permissive, or by rights holders who want rules that allow them to monetize their content in new ways. Recent legislative proposals based on the premise of sneaky “

,” for example, have the benefit to publishers of ensuring an identity for 

 bots that can be used as the basis of a “block, allow, charge, or measure” 

strategy that calls for screening incoming bots

 based on their potential for monetization. None of these are designed with a broader concept of the public interest in the forefront — most of them would compromise socially beneficial uses of scraping and crawling, and privacy considerations abound.

Without deliberate intervention, the internet will continue to shrink in terms of the amount and quality of publicly available information. (

 behind the “rupture” in the web’s value loop described this outcome as “a degraded and privatized information base.”) It may also bifurcate into a 

: a premium layer, accessible only through licensing agreements, APIs, and paywalls to the wealthiest corporations, institutions, and people, and a degraded open layer increasingly populated by synthetic, low-quality, and commercially motivated content. This is an outcome with implications for democracy, civic engagement, competition, and innovation — and it all comes back to the principles of openness from which we have collectively benefited for so long. 

We need to consider a range of solutions that manage these tensions, including some we might otherwise reject as being contrary to one of these values or another in isolation.

In pursuit of that goal, civil society organizations are bringing forward potential solutions that represent a middle ground. Public Knowledge conducted off-the-record convenings of like-minded stakeholders to explore the potential of solutions that reflected our conviction that AI training is generally a fair use of copyrighted content — but also our conviction that we need to protect the knowledge commons for the long term. The solutions included bot identification requirements, extended collective licensing, and text and data mining exceptions for 

CC Signals, a new preference signals framework

 designed to allow rights holders to signal their preferences for how their content can be reused by machines. There were both complementarities and tensions between our ideas, but in general both faced significant resistance: they were seen as undermining the premise of fair use, as antithetical to the principles of the open web, or conversely, as insufficiently robust to insulate rights holders from exploitation.

Apart from Public Knowledge, Creative Commons, and some 

 that are evaluating broader options, much of the activity in the policymaking space is focused on copyright, especially as it relates to training of generative models. As noted above, 

 believes that in general, training of generative AI models using copyrighted content is permissible under the US legal doctrine of fair use. Fair use is a critical component of the legal infrastructure of the open web, and expanding copyright to cover training inputs would threaten not only AI development but also search engines, web archiving, and the entire ecosystem of transformative and creative use that depends on the same legal foundations. (Congress hasn’t helped our cause, conflating AI training with piracy in a 

 with the provocative title, “Too Big to Prosecute? Examining the AI Industry’s Mass Ingestion of Copyrighted Works for AI Training”.) Paradoxically, expanding or interpreting copyright law in this way would also threaten the 

 to legally quote, excerpt, critique, or provide historical context to current events. This is not the right approach.

Our belief that generative model training on copyrighted material is fair use is why we generally oppose legislative proposals meant to change or undermine the doctrine, such as the 

 (which creates a federal intellectual property right in people’s voice and visual likeness); the 

 (which calls for the creation of federal standards for AI provenance and regulates how AI models are trained on copyrighted works); the 

 (which forces AI developers to disclose their training data); the 

 (which requires training data sets be submitted to the Register of Copyrights); the 

 (which amends trademark law to prevent deceptive AI ads); the 

 (which establishes a new “style right” to stop AI tools from cloning an artist's “look and feel” for profit); the 

Journalism Competition & Preservation Act

 (which creates an antitrust exemption for news organizations to negotiate with tech firms for access to their content); and the 

 (which compels automated web crawlers to identify themselves to online publishers). That extensive list only covers current legislative proposals at the US federal level — a sign of how many solutions are being proposed that may do more harm to the commons than good.

We also resist proposals of this nature because in our view this is not a copyright problem at all. It is a market structure problem, propelled by the transforming economic framework of the web. Publishers are reaching for a copyright solution because that is what is easily available to them, and because it offers options that are conducive to monetization of their content. Their calls are being given tailwinds by a growing anti-AI backlash. But as noted in 

, we should address the market structure problem as distinct from other problems we may have with the technology.

US policy or regulatory proposal focused on addressing the impact of bot traffic, specifically, is that some proportion of malicious bot activity may be from outside the US and therefore outside US legal jurisdiction. This is a legitimate theory, though by definition hard to test. According to 

, for example, the vast majority of automated AI crawling on US news sites is driven by major US-based companies: Googlebot, OpenAI’s GPTBot, and Anthropic’s ClaudeBot. There is a lot of traffic from sources outside the US, but it is generally routed from allied countries that house data infrastructure or centers. It is hard to determine its original source.

Other available evidence comes from cybersecurity and internet infrastructure companies who we recognize have a vested interest in rights holders wanting to detect and either circumvent or monetize unwanted activity. 

, a global cybersecurity company, takes a sector view, reporting that the vast majority of detectable “bad bot” behavior targets high-value industry sectors like financial services, business, and retail to steal customer data, manipulate workflows, abuse promotions, or steal digital inventory. A minuscule portion — less than a quarter of one percent — of this total global behavior targeted the news industry, and most of it was from identifiable, US-based sources. (Huawei’s PetalBot, based in China, was an exception, but PetalBot identifies itself and respects robots.txt). The latest report from 

, a content delivery network provider, also shows that online publishers were less likely to be hit with “unwanted” traffic than Fastly’s cross-industry average. And again, the vast majority of unwanted volume from top AI crawlers was from known US companies (plus PetalBot).

 tracked major, episodic surges where traffic routing from China and Singapore suddenly accounted for 22% of all site traffic, outnumbering known AI bots combined. This frequently included high-volume scraping from Bytespider (the crawler associated with ByteDance, the Chinese company that created and still owns the global version of TikTok). Cybersecurity firms like 

Latest

Podcast

Recent

Our content delivered to your inbox

Thank you!

Around the globe