How to add BotConsent to a website
Add a generated AI bot consent policy, connect your domain, and publish discovery files.
1. Verify your site
Start by adding your domain in the dashboard. BotConsent provides a DNS TXT record, HTML file, or meta tag verification method so ownership can be confirmed before policies are hosted.
2. Choose bot preferences
Set category-level defaults for search indexing, AI assistant retrieval, AI training, commercial scraping, and archive/research access. You can add bot-specific overrides for known user agents.
3. Publish discovery files
BotConsent generates a policy URL, robots.txt suggestions, and llms.txt-style guidance. Keep these files at the root of your site so crawlers can find them quickly.
User-agent: GPTBot Disallow: / User-agent: Googlebot Allow: / # BotConsent policy Policy: https://botconsent.com/policies/example.com
4. Monitor activity
The dashboard records policy changes and mock crawler events in this MVP. A production integration can connect to edge logs, WAF events, or analytics pipelines.