How AI bot consent policies work

Policy layers

• Category defaults describe the baseline preference for broad bot types.
• Bot-specific overrides handle known crawlers with documented user agents.
• Use-case permissions separate search indexing, retrieval, training, scraping, and archiving.
• Audit entries record who changed a policy and when the change took effect.

Machine-readable output

BotConsent turns owner choices into structured output that can be linked from robots.txt, referenced in llms.txt, and hosted as a canonical policy page.

{
  "site": "example.com",
  "ai_training": "disallow",
  "assistant_retrieval": "allow_with_attribution",
  "commercial_scraping": "manual_review",
  "policy_version": "2026-05-06"
}

Enforcement model

Respectful crawlers can follow the published policy directly. For unknown or non-compliant actors, BotConsent can feed edge rules, alerts, and review workflows.