Permission policies for AI bots
Publish clear preferences for search, assistant retrieval, model training, commercial scraping, and archival research.
AI-era website permissions
BotConsent
Control how AI bots interact with your website. Publish clear permissions for crawlers, scrapers, indexing agents, AI assistants, and training-data collectors without making the web harder to use.
Policy status example.com
Published AI bot consent v2026.05
Search indexing Allow
Assistant retrieval Allow with attribution
AI training crawlers Block
Commercial scrapers Review
robots.txt Output
User-agent: GPTBot Disallow: / User-agent: Googlebot Allow: / Policy: /botconsent.json
Known agents 7 watched
Googlebot Allowed
Bingbot Allowed
GPTBot Blocked
ClaudeBot Conditional
PerplexityBot Allowed
Core value
Control, transparency, and permission infrastructure.
BotConsent turns bot preferences into something your team can manage and crawlers can discover.
Generated discovery files
Create robots.txt and llms.txt-style outputs that point crawlers to your canonical BotConsent policy.
Hosted policy pages
Give people and software a stable URL for your bot permissions, contact route, policy version, and change history.
Crawler activity visibility
Review detected agents, policy outcomes, and compliance-ready logs from one operational dashboard.
How it works
A policy workflow site owners can actually maintain.
Start with a plain-language policy builder, then publish structured files and connect enforcement where it matters.
- 1
Verify your domain with DNS, meta tag, or file-based ownership checks.
- 2
Choose category-level defaults and add bot-specific overrides when needed.
- 3
Publish policy files and connect monitoring or edge enforcement.
- 4
Review changes, crawler activity, and exportable consent history.
Use cases
Built for teams that publish, protect, and maintain web content.
BotConsent is calm infrastructure for the messy edge between discovery, attribution, and reuse.
Publishers balancing search visibility with training-data boundaries.
SaaS teams standardizing bot policies across docs, marketing, and app surfaces.
Agencies generating clear consent settings for client websites.
Developers integrating policy signals into Cloudflare, APIs, and logs.
Trust and security
Policy clarity without security theater.
BotConsent treats consent as an operational record: who changed a policy, what it says, where crawlers can find it, and how suspicious activity should be reviewed. It is designed for Cloudflare-friendly deployment, privacy-conscious defaults, and human-readable governance.
Site verification Versioned policies Export-ready logs API-first design
Pricing preview
Start with a generator. Grow into monitoring and hosted policies.
Simple plans for solo site owners, growing teams, and portfolios.
Free
$0
For basic policy publishing
Start free- One website
- Basic AI bot policy generator
- robots.txt and llms.txt output
- Manual policy download
Pro
$19/mo
Per site, per month
Get Pro- Hosted policies
- Monitoring dashboard
- Crawler activity alerts
- Policy version history
- Site verification
Business
Custom
For teams and portfolios
Contact sales- Multi-site management
- Developer API access
- Team accounts
- Compliance exports
- Priority support
FAQ
Practical answers for a new category.
AI crawler consent is evolving. BotConsent keeps the first version understandable.
Does BotConsent replace robots.txt?
No. BotConsent complements robots.txt by adding AI-specific use cases, hosted policy pages, and richer policy history.
Can BotConsent block every scraper?
No policy file can force every actor to comply. BotConsent publishes preferences for responsible bots and can feed enforcement workflows for suspicious traffic.
Is this for technical teams only?
No. The product is designed so founders, publishers, and site owners can set practical defaults, while developers still get structured output and API-friendly workflows.
Where does it deploy?
This website and MVP are prepared for Cloudflare Pages with a static Astro build and a Pages Function for contact form handling.
Ready to publish a policy?