Well this is super interesting. Cloudflare is trying out a turnstile thing that lets you gate your site to AI crawlers and get paid for allowing access.
Would you use a thing like this? In addition to/instead of advertising?
I predict it will be a race to the bottom with pricing and there won’t be any opportunity for meaningful revenue except for content aggregators.
After hundreds of conversations with news organizations, publishers, and large-scale social media platforms, we heard a consistent desire for a third path: They’d like to allow AI crawlers to access their content, but they’d like to get compensated. Currently, that requires knowing the right individual and striking a one-off deal, which is an insurmountable challenge if you don’t have scale and leverage.
We believe your choice need not be binary — there should be a third, more nuanced option: You can charge for access. Instead of a blanket block or uncompensated open access, we want to empower content owners to monetize their content at Internet scale.
Each time an AI crawler requests content, they either present payment intent via request headers for successful access (HTTP response code 200), or receive a 402 Payment Required response with pricing. Cloudflare acts as the Merchant of Record for pay per crawl and also provides the underlying technical infrastructure.
Why? I think these are pretty different cases. The commercial search engine is trying to send human beings to your site. The AI crawlers are trying to replace your site with a chatbot trained on its content.
I no longer believe that this is the case, especially with Google. If Google can keep people from visiting websites by giving people just enough info on search engine result pages, they won’t visit websites. But that requires mining people’s websites for information before they can show it on SERPs. Therefore, Google can bloody well pay me.
Likewise Microsoft with Bing. And as long as DuckDuckGo depends primarily on the Bing index, they can GoGoFuck themselves.
honestly tentatively seems like a great ‘compromise’ to our current dystopia. I agree that it will probably just be swallowed up and only benefit aggregators though, much like SEO and the like. But it’d at least be a way for those places to make money without absolutely obliterating normal users with ads!
I wonder how it will determine what is an ai crawler though. Many don’t respect robots.txt or various other methods of ‘checking’. I admit I don’t know much about the nuances of how these things are tracked! I just worry about potential accessibility problems w people that use VPNs, screen readers, and other technologies that are occasionally read as ‘robot-like’ by a website.
I just worry about potential accessibility problems w people that use VPNs, screen readers, and other technologies that are occasionally read as ‘robot-like’ by a website.
Right, me too… Although these problems are not particular to paywalling, they arise whenever you want to distinguish between AI bots and normal users. That’s going to be an arms race of “are you a human?” vs. machine learning algorithm just like the captchas of last decade, unfortunately.