Fast Index Google is a free, self-hosted crawlability toolkit. Submit a URL and it is crawled as a browser, as Googlebot, and as fifteen other crawlers — then audited with a built-in SEO checker, a rich-text analyzer and Google's free PageSpeed Insights API.
Googlebot, Bingbot, YandexBot, DuckDuckBot, Baiduspider, CCBot, Applebot, Sogou, 360Spider, AhrefsBot, SemrushBot, MJ12bot, facebookexternalhit, Twitterbot, Googlebot-Image and Yeti — all plain HTTP fetches.
Performance, SEO, accessibility and best-practices scores plus Core Web Vitals (LCP, CLS, TBT, FCP) from Google's free public API.
Flesch readability, grade level, keyword density, stop-word ratio, long-sentence detection and reading time — computed locally, nothing stored.
Full user-agent group matching with Allow/Disallow, * wildcards and $ anchors, checked for Googlebot and for *.
Toggle crawlers, tune timeouts and concurrency, manage SEO overrides and SmartLink ads, delete jobs, and watch live system logs — token-protected.
Light and dark themes, sortable tables, CSV export, batch submit, live polling and a per-bot result matrix.
Each submitted URL goes through: queued → crawling → crawling_bots → done (or failed).
meta robots and X-Robots-Tag.*). A missing robots.txt (404) means no restrictions.POST /api/submit
{ "url": "https://example.com/page" }
{ "urls": ["https://a.com", "https://b.com"] }
{ "urls": "https://a.com\nhttps://b.com" }GET /api/jobs?status=&q=&limit= — list jobsGET /api/jobs/:id — job detail with raw crawl JSONPOST /api/jobs/:id/retry — re-run the crawlGET /api/stats — totals + average bot access rateGET /api/settings — public settings (bots, timeouts, SEO overrides, SmartLink config)POST /api/checker/text — { "text": "..." } → readability + keywordsPOST /api/checker/page — { "url": "...", "strategy": "mobile|desktop" } → SEO audit + text analysis + PageSpeedGET /sitemap.xml — auto-generated sitemap (site pages + processed URLs)Admin endpoints (/api/admin/*) require the x-admin-token header. On the live deployment the token is a Pages secret; when self-hosting it is printed in the server console on startup and stored in data/settings.json.
PUT /api/admin/settings
{
"siteName": "Fast Index Google",
"crawlTimeoutMs": 15000,
"botPool": 5,
"concurrency": 3,
"refreshMs": 3000,
"bots": { "Googlebot": true, "Yeti": false },
"seo": {
"metaTitle": "{page} — Free Multi-Bot Crawl Checker",
"metaDescription": "...",
"keywords": "...",
"ogImage": "https://example.com/og.png",
"googleAnalyticsId": "G-XXXXXXXXXX",
"googleSiteVerification": "..."
},
"smartlink": { "enabled": true, "url": "https://ad.example.com/abc", "perHour": 3 }
}In metaTitle, the placeholder {page} is replaced with each page's own static title at runtime. Leaving SEO fields empty keeps the per-page static meta tags.
npm install && npm start — Node.js + Express + better-sqlite3, data in data/.npm run build && npm run deploy — static export plus Pages Functions, D1 database, and the Durable Object scheduler (npm run deploy:scheduler when the scheduler code changes).PORT, CRAWL_TIMEOUT_MS, BOT_POOL, CONCURRENCY, REFRESH_MS, BOTS, ADMIN_TOKEN (self-hosted); ADMIN_TOKEN as a Pages secret on the live site.Yes. Every crawler is a plain HTTP fetch and PageSpeed Insights is Google's free public API. No API keys, no quotas, no subscriptions.
Twelve of the sixteen crawlers received a valid, non-blocked response. Open the job to see exactly which bots were blocked, their status codes and response times.
The server answered 401, 403, 429 or 999 for that user agent, or the request timed out. That is a server-side decision about that specific crawler — often WAF or cloud bot management.
That bot received different body content than the reference browser fetch. This can indicate cloaking, or a site that relies on JavaScript rendering (a real Googlebot renders JS; our fetches do not).
Yes — toggle individual bots in the admin panel, or restrict the bot list with the BOTS environment variable when self-hosting.
In the deployment's database — SQLite locally (data/fastindex.db), Cloudflare D1 on the live site — plus settings. Nothing else leaves your machine except the crawl fetches themselves and the PageSpeed API call.
No. It shows what crawlers actually receive. For submission, use Google Search Console and Bing Webmaster Tools.
Cloudflare Pages serves the static site and the API (Pages Functions), D1 stores data, and a Durable Object provides the cron loop that drains the queue — Pages itself has no cron triggers. See how it works.
Yes. The Node.js + Express version runs locally with better-sqlite3; the Workers modules in functions/lib/ mirror the same pipeline. Both targets share the same settings shape.
An optional monetization feature. When enabled in the admin panel, the first click anywhere on a page opens the advertiser URL in a background popup window, capped at 3 times per hour per browser via local storage. It never runs on the admin panel.