Why AI Bots Need Consent
AI crawlers are becoming part of the web's infrastructure. Consent makes that relationship clearer for owners, builders, and users.
The web was built on discovery
Search crawlers helped people find useful pages, and site owners generally understood the trade: crawl access in exchange for visibility. AI systems have widened that pattern. A page may now be retrieved for an answer, summarized in a chat interface, indexed for an agent, or collected into a training corpus.
Those uses are not identical. A publisher may welcome search indexing, allow user-initiated retrieval, and decline training-data collection. Consent gives owners vocabulary for those distinctions.
Clarity helps responsible bots
Many bot operators want to respect site preferences but need a consistent signal. Human-readable policies are useful for legal and editorial review. Machine-readable policies are useful for crawlers, edge systems, and monitoring pipelines.
BotConsent focuses on both audiences because permission infrastructure has to be legible to people and software.
Consent is not anti-AI
The goal is not to wall off the web. The goal is to make web permissions explicit enough that valuable AI products can be built with better norms, clearer accountability, and fewer surprises for site owners.