Fast Index Google answers one question better than anything else: what do crawlers actually see when they fetch your page? Here is the full journey of a submitted URL — from queue to the per-bot result matrix — and the infrastructure that makes it run on the edge.
Every job moves through the states queued → crawling → crawling_bots → done (or failed). Each state is visible live in the dashboard.
The URL is fetched once with a current Chrome user agent, following redirects. This is the browser baseline: HTTP status, total time, final URL after redirects, title, meta description, canonical link, H1, html language, word count, meta robots and any X-Robots-Tag header. Everything later is compared against this fetch.
The origin's robots.txt is fetched and parsed into user-agent groups. Rules are matched first for the specific bot name (Googlebot), then for *, honouring Allow and Disallow with * wildcards and $ end-anchors. A missing robots.txt (HTTP 404) is treated as "no restrictions", exactly as Google specifies.
sitemap.xml is fetched from the origin and scanned for the submitted URL. The report shows whether a sitemap exists, whether it is referenced by robots.txt, and whether your URL is listed — the difference between "discovered" and "submitted".
The same URL is fetched again with every enabled crawler user agent — Googlebot, Bingbot, YandexBot, DuckDuckBot, Baiduspider, CCBot, Applebot, Sogou, 360Spider, AhrefsBot, SemrushBot, MJ12bot, facebookexternalhit, Twitterbot, Googlebot-Image and Yeti — in a parallel pool. For each bot we record:
The aggregate is reported as Bots OK n/16 — a direct, honest crawlability score. No random numbers, no "indexability rate" estimate: these are real fetches with real responses.
Results land in the dashboard as a sortable table with the full per-bot matrix in the job detail modal. Export any filtered view as CSV, retry individual jobs after a server fix, or delete them from the admin panel. Public sitemaps and robots files are generated for the site itself.
This deployment runs entirely on Cloudflare's edge — no servers to maintain:
processQueue. Jobs that stall mid-crawl for more than 6 minutes are automatically recovered, and a job interrupted twice is failed with a CPU-limit explanation rather than hanging forever.The codebase is dual-target: the same logic runs locally on Node.js + Express + better-sqlite3 with npm start, and on Workers via the portable modules in functions/lib/. See the API reference for exact endpoints.