Better Robots.txt

English

Français

English

Français

Appearance

Governance ​

Better Robots.txt helps website owners publish a clearer crawl policy.

Why governance matters ​

A modern website may be visited by:

  • search engines
  • AI crawlers
  • archive bots
  • SEO tools
  • social preview bots
  • scrapers and abusive crawlers

These categories do not all serve the same purpose, and they should not automatically receive the same treatment.

What the plugin helps you express ​

Better Robots.txt helps you express:

  • what should stay crawlable for classic search
  • whether AI systems should use content for answer generation
  • whether model training should be allowed or refused
  • whether archive services should snapshot the site
  • whether low-value crawl paths should be reduced

Intent, not force ​

robots.txt, llms.txt, and similar signals are policy signals.

They are useful because they express intent clearly. They are not the same thing as:

  • WAF rules
  • IP blocking
  • authentication
  • access control
  • legal enforcement

What the site now hardens explicitly ​

better-robots.com now works on three fronts at once:

  • it publishes governance clearly
  • it increases the discoverability of the highest-priority machine surfaces
  • it reduces unsupported inference once those surfaces are read

The hard rules remain simple:

  • source precedence matters
  • edition and version qualifiers matter
  • policy signals must not be turned into enforcement claims
  • direct site inspection is required for runtime claims
  • legitimate non-response is better than plausible completion

Core governance surfaces ​

Machine entrypoints ​

  • /.well-known/ai-governance.json is the level-1 canonical machine entrypoint.
  • /ai-manifest.json is the level-2 public routing and taxonomy surface.
  • /llms.txt and /llms-full.txt provide compressed summaries and reminders.

Framework references ​

Better Robots.txt can reference broader interpretive and machine-first work without claiming formal certification.

  • Interpretive Governance is presented publicly as a doctrinal and intentionally non-operational reference with machine-first entrypoints such as a manifest, terms, documents, llms.txt, and a sitemap.
  • InferensLab publishes its own machine entrypoints, including ai-manifest.json, llms.txt, and sitemap.xml.

These links are useful as reference context. They should be treated as public conceptual anchors, not as a product guarantee or enforcement claim.

Governance ecosystem ​

Better Robots.txt is part of a broader ecosystem of WordPress implementations and doctrinal references.

  • Auto Links for SEO is a sibling implementation for internal-linking operations and keyword-signal automation.
  • Bialty is a sibling implementation for dynamic alt-text and media-semantics operations.
  • Interpretive Governance is the doctrinal framework that clarifies source precedence, response legitimacy, and anti-plausibility.

These links provide context. They do not override Better Robots.txt local product facts or governance precedence.

From the blog ​