Last year, Cloudflare introduced a setting that allowed website owners to block AI crawlers. Today, the company is announcing that this setting will now be the default rather than a user needing to switch it on.
The company explained that by switching to a permission-based model, it is eliminating the need for content owners to manually configure settings in order to opt out.
According to Matthew Prince, co-founder and CEO of Cloudflare, AI crawlers have been scraping content from the web without limits, but in order for the Internet to thrive in an AI era, there is a need for a new economic model that works for creators, consumers, and AI founders.
“Our goal is to put the power back in the hands of creators, while still helping AI companies innovate. This is about safeguarding the future of a free and vibrant Internet with a new model that works for everyone,” he said.
Additionally, Cloudflare is experimenting with a pay per crawl model (in private beta) for content owners to monetize content that is being used to train AI.
When an AI crawler requests content, it will be prompted to pay or will receive a HTTP response code 402 error message.
Publishers have three options for crawlers: allowing free access, charging for access, or blocking access entirely. They also have the option to allow a crawler to bypass payment, which may be useful in scenarios where a content owner wants to negotiate a content partnership beyond the pay per crawl feature.
“At its core, pay per crawl begins a technical shift in how content is controlled online. By providing creators with a robust, programmatic mechanism for valuing and controlling their digital assets, we empower them to continue creating the rich, diverse content that makes the Internet invaluable,” Cloudflare wrote in a blog post.
While this model is still in its early stages, Cloudflare anticipates many different types of interactions and marketplaces to develop simultaneously, such as the ability to charge different rates for different content types.
