AI and LLM Governance
Location: Step 2 — AI & LLM Governance
This step controls how AI-related crawlers and machine-usage preferences are handled.
Why this step matters
This is where Better Robots.txt becomes more than a simple robots.txt editor.
It lets you publish a clearer policy surface for:
- search indexing
- answer-generation systems
- model training posture
- archive services
- crawler category separation
Main controls
- AI Training Protection — publish restrictive rules and signals aimed at training-oriented bots when enabled
- AI Search & Answer Engines — choose whether AI search systems are allowed by published policy or restricted by published rules and signals
- Content usage signals — clarify preferences such as
search,ai-input, andai-train - llms.txt — available separately when the edition supports it
What this step does not do
This step does not:
- authenticate bots
- guarantee compliance
- guarantee training exclusion
- prove runtime blocking
- replace a legal or infrastructure layer
The correct framing is: policy publication and machine-readable guidance.
Questions to ask before enabling stricter options
- Is the site a public discovery site or a protection-first site?
- Does the site need only basic openness, or a more explicit AI stance?
- Is the concern about training, answer-generation, scraping, or archives?
- Which behavior matters enough to publish clearly, even if enforcement cannot be guaranteed?