GemNews

All stories · Article

OpenAI is paying cloudflare to make web crawling more efficient

AI companies are beginning to pay for more efficient access to the web. OpenAI is already paying Cloudflare for signals that tell its crawler when a page changes, rather than repeatedly downloading pages that remain the same.

By Janie Nguyen · · 9 min read

Share: WhatsApp · Telegram · X · Facebook · LinkedIn · Email

The development points to a broader shift: as AI agents consume more web content, the infrastructure behind crawling, updating and accessing that information is becoming a commercial business.

*Source: **https://blog.cloudflare.com/*

For Shopify merchants, the question is no longer just whether AI can read their product pages. It is whether the information AI receives is accurate, current and structured well enough to influence what customers discover and buy.

OpenAI is paying for signals, not repeated page downloads

During Cloudflare's Q3 2026 earnings call, CEO Matthew Prince confirmed that OpenAI is paying Cloudflare for information about changes to web pages.

Instead, Cloudflare could actually give them hints when a page had changed, when a menu price had updated, when anything had happened, and then they could get the freshest data, and they could actually crawl that. That helps OpenAI save money, and that actually helps the Cloudflare customers get the freshest data to them and make sure that it does not waste a ton of resources. I think that is a clear way where we are already seeing that, and that has helped. OpenAI is paying for that, and a lot of other companies in the space, I think you will see do that as well.
- CEO Matthew Prince

The mechanism addresses a simple inefficiency in traditional crawling.

An AI crawler might repeatedly visit the same restaurant website to check whether its menu or prices have changed. Most visits return exactly the same content. That means unnecessary requests, bandwidth consumption and processing costs.

Cloudflare can instead provide signals when a page changes. If a restaurant updates its prices, OpenAI receives a hint that the page needs to be fetched again.

The crawler can focus on updated content rather than repeatedly downloading unchanged pages.

This creates two benefits:

  • OpenAI reduces the cost of collecting and refreshing web information.
  • Websites served through Cloudflare can make updated information available to AI systems more efficiently.

Prince described this as an example of Cloudflare already generating revenue from infrastructure designed for the next generation of the internet.

He did not disclose the amount OpenAI pays or the precise commercial terms of the arrangement.

Sources: Search Engine Journal – Cloudflare CEO Likens AI Agent Optimization to a New Version of SEO · Cloudflare Q3 2026 earnings call transcript.

The next step: pricing AI agents by the value of their visits

The change-hint arrangement is only one part of a larger shift Prince expects to see.

He predicts that websites and infrastructure providers will eventually develop micropayment models for AI agents accessing online content.

His example illustrates why a universal per-request fee may not work.

Imagine an AI agent searching for a restaurant. It visits 1,000 restaurant websites, reads their menus and recommends only one. The other 999 websites absorb the cost of serving that information without gaining a customer.

Prince suggests that the agent's stated purpose could influence how much it pays.

An agent that clearly explains what its user is looking for may receive cheaper access, or even free access, because its visit creates an opportunity for relevant advertising.

For example, an agent planning a family holiday to Hawaii could represent a valuable commercial opportunity for travel businesses. Its visit is not simply another crawler request; it signals potential purchase intent.

The underlying principle is that not every AI visit has the same commercial value.

A crawler collecting information at scale and an agent actively helping someone make a purchase are fundamentally different kinds of traffic.

This is still a prediction, not an established industry-wide pricing standard. But it suggests that the economics of AI access may increasingly depend on the purpose and potential value of each visit.

Source: Cloudflare Q3 2026 earnings call transcript.

AI crawling is becoming a monetization layer

Cloudflare is already building products around this emerging market.

Its Pay Per Crawl initiative explores ways for websites to charge AI crawlers for accessing their content. The product remains in private beta.

Cloudflare also introduced Monetization Gateway in closed beta on September 30, 2026. It allows domain owners to set payment requirements for AI agents accessing websites, APIs, datasets and MCP tools.

These products point to a developing infrastructure layer in which access permissions, payment instructions and the identity or purpose of AI agents can become part of the same interaction.

The market is still early. These tools are not yet a universal payment standard, and their availability does not mean that every website can immediately monetize AI traffic.

Sources: Cloudflare Pay Per Crawl · Cloudflare Monetization Gateway – September 30, 2026.

What this means for Shopify merchants

The immediate implication for Shopify merchants is not to start charging crawlers. It is to understand how their product information is accessed, refreshed and interpreted by AI systems.

Shopify already manages bot traffic at the network layer for stores hosted on its platform. Merchants do not need to configure this layer themselves.

At the product-data level, Shopify Catalog is the primary method through which agentic storefronts such as ChatGPT and Microsoft Copilot receive eligible product information.

That information includes product titles, descriptions, images, prices and availability.

Shopify also provides separate controls for AI channel access and crawler permissions. Importantly, blocking an AI crawler through robots.txt does not automatically prevent product information from being made available through Shopify Catalog.

This distinction matters. Crawling an online store and accessing its structured product catalog are not necessarily the same process.

Sources: Shopify – Catalog and product discovery for agentic storefronts · Shopify – Data sharing and privacy for agentic storefronts.

AI discovery is already showing up in ecommerce orders

While the infrastructure and payment models are still developing, AI-assisted product discovery is already generating measurable ecommerce activity.

An analysis by Enalitica of more than 100,000 recent online store orders, using data through November 7, 2026, found:

  • ~4× growth: AI's share of orders was roughly four times higher than in the same months a year earlier, comparing the same stores.
  • Nearly 1% of orders: AI assistants were approaching 1% of orders at the typical store.
  • ~90% first-time customers: Nearly nine in ten AI-referred orders came from new customers, compared with around seven in ten orders overall.
  • ChatGPT dominates: Approximately 95 out of every 100 AI-referred orders came from ChatGPT.
  • An attribution gap: GA4 recorded only 41% of these AI-referred orders under its AI Assistant channel.

The findings suggest that AI is becoming a product discovery channel, not merely another source of website visits.

The study reflects Enalitica's own measured stores, rather than the entire ecommerce market. Its figures should therefore be treated as evidence of an emerging pattern, not a universal industry benchmark.

Source: Enalitica – ChatGPT Sales in 2026: What the Last 100,000+ Store Orders Show.

What sellers should pay attention to now

There are three practical areas worth monitoring:

Product data quality

Make sure product titles, descriptions, variants, prices, images and availability are accurate and consistent. Structured information gives AI systems a clearer basis for understanding and recommending products.

AI access and discoverability

Review your Shopify Catalog eligibility and agentic storefront settings. Understand the difference between open-web crawling, robots.txt permissions and product data distributed through Shopify Catalog.

Order attribution

Check GA4's AI Assistant channel, but do not assume it captures every AI-referred purchase. Review Direct, Referral and Unassigned traffic as well, and compare the first-time customer share across sources.

The larger shift is worth watching.

As AI agents become more active on the web, the infrastructure that helps them discover, retrieve and interpret information is becoming a market of its own. At the same time, merchants have a growing reason to make their product information accurate, accessible and easy for those agents to understand.

The next phase of AI commerce will depend not only on which websites agents can access, but also on how efficiently they can find the right information and turn it into a purchase.

Discuss on GemCommunity