Skip to content

Crawl Signal

Crawl Signal measures whether agents can find, access, and trust the technical integrity of this service's pages.

What it measures

Scoring Crawl Signal Clarity against Dimension 3 of the agent-readiness-rubric; generating FIX artifacts from fix-templates/; flagging any ESCALATE items.

How it scores

Score Descriptor
0–4 robots.txt blocks agents or is misconfigured. No sitemap. Critical crawl errors. Page speed under 50 (Lighthouse).
5–9 Sitemap present but stale or incomplete. Some crawl errors. Mobile usability issues. Load time 3–5s.
10–14 Sitemap current. robots.txt permissive to agents. Core pages load under 3s. Canonical tags present but not fully consistent.
15–19 Clean crawl report. Canonicals correct. Hreflang if multilingual. Sub-2.5s load. Zero critical crawl errors.
20 Perfect crawl hygiene. Sitemap auto-updates on publish. Sub-2s load. Zero crawl errors. Agents encounter no friction accessing any indexed page.

What it generates on FIX

All of these produce a text artifact the operator can write:

  • robots.txt AI-crawler directives — from fix-templates/robots-txt.md. Ensures GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot, and Googlebot are explicitly allowed. OAI-SearchBot and GPTBot are always treated as distinct crawlers; a site may allow one while blocking the other and must address both by name.
  • Canonical tags — generated inline as <link rel="canonical" href="{{url}}"> HTML, placed in <head>. Applied to pages with missing or inconsistent canonical declarations.
  • <time> markup for crawl-relevant dates — from fix-templates/time-markup.md. Applied to sitemap last-modified signals and any visible date strings that affect freshness perception at the crawl layer.
  • Sitemap entry — generated inline as an XML <url> block to be added to the existing sitemap.xml. Applied when key pages are absent from the sitemap.

What escalates

These fixes require action in the world and cannot be produced as text artifacts:

  • Core Web Vitals / page-speed remediation — improving Lighthouse scores below the rubric threshold requires code changes by a developer. Cannot be produced as a configuration file.
  • Bing Webmaster Tools sitemap submission — ChatGPT retrieves from the Bing index, not Google. Submitting the sitemap to Bing Webmaster Tools requires account access and cannot be completed in robots.txt or any text artifact. This escalation fires whenever ChatGPT citation coverage is a goal and Bing indexation is unconfirmed.

See it in action