AI bot permission policies
Define allowed, blocked, attribution-required, and review-first behaviors by bot category and known user agent.
Product
One control plane for website bot permissions.
BotConsent helps website owners move from scattered crawler rules to managed policies that are readable, publishable, and ready for enforcement workflows.
Policy status example.com
Published AI bot consent v2026.05
Search indexing Allow
Assistant retrieval Allow with attribution
AI training crawlers Block
Commercial scrapers Review
robots.txt Output
User-agent: GPTBot Disallow: / User-agent: Googlebot Allow: / Policy: /botconsent.json
Known agents 7 watched
Googlebot Allowed
Bingbot Allowed
GPTBot Blocked
ClaudeBot Conditional
PerplexityBot Allowed
Features
Everything needed to publish a serious AI bot consent policy.
The MVP focuses on the public site, policy generation, and dashboard workflow. The product model is ready to grow into monitoring, APIs, and enforcement.
robots.txt and llms.txt generation
Generate familiar crawler instructions and AI-oriented guidance from one source of truth.
Bot access preferences
Separate search indexing, answer retrieval, training collection, commercial scraping, and public-interest archiving.
Crawler detection
Track observed agents, suspicious user agents, and policy outcomes in a dashboard-ready activity model.
AI training opt-in/opt-out controls
Express model-training consent in clear, machine-readable terms that can be referenced by compliant crawlers.
Consent logs
Keep a history of policy versions, site verification events, crawler observations, and exports.
Site verification
Confirm domain ownership with DNS, meta tag, or file-based verification before publishing hosted policy pages.
Policy hosting
Serve a canonical policy URL for every verified site so people, bots, and enforcement tools know where to look.
Developer-friendly API
Expose policies, events, and exports through predictable endpoints for internal tools and edge integrations.
Policy model
Consent by use case, not just by crawler name.
A single bot may perform different tasks. BotConsent keeps the owner's intent attached to the type of access requested.
| Category | Typical preference | Reason |
|---|---|---|
| Search indexing | Allow public pages | Keep canonical discovery healthy |
| AI assistant retrieval | Allow with attribution | Support user-initiated answers |
| AI training crawlers | Block or opt in | Control training-data collection |
| Commercial scrapers | Review first | Reduce low-value extraction |
| Archive/research bots | Allow selected agents | Preserve public-interest access |
GET /v1/sites/example.com/policy
Authorization: Bearer bc_live_...
{
"domain": "example.com",
"ai_training": "disallow",
"assistant_retrieval": "allow_with_attribution",
"commercial_scraping": "manual_review"
} Developer ready
Clean outputs for crawlers, edge rules, and internal tools.
BotConsent can sit in front of a simple website or a larger compliance workflow. The same policy choices can generate static files, hosted pages, dashboard previews, and API responses.
View API overview