Documentation & FAQ
Fast Index Google is a free, self-hosted crawlability toolkit. Submit a URL and it is crawled as a browser, as Googlebot, and as fifteen other crawlers — then audited with a built-in SEO checker, a rich-text analyzer and Google's free PageSpeed Insights API.
The toolkit at a glance
19 free crawlers
Googlebot, Bingbot, YandexBot, DuckDuckBot, Baiduspider, CCBot, Applebot, Sogou, 360Spider, AhrefsBot, SemrushBot, MJ12bot, facebookexternalhit, Twitterbot, Googlebot-Image, Yeti, GPTBot, ClaudeBot and PerplexityBot — all plain HTTP fetches.
PageSpeed Insights
Performance, SEO, accessibility and best-practices scores plus Core Web Vitals (LCP, CLS, TBT, FCP) from Google's free public API.
Rich text analyzer
Flesch readability, grade level, keyword density, stop-word ratio, long-sentence detection and reading time — computed locally, nothing stored.
robots.txt matcher
Full user-agent group matching with Allow/Disallow, * wildcards and $ anchors, checked for Googlebot and for *.
Admin panel
Toggle crawlers, tune timeouts and concurrency, manage SEO overrides and SmartLink ads, delete jobs, and watch live system logs — token-protected.
Premium UI
Light and dark themes, sortable tables, CSV export, batch submit, live polling and a per-bot result matrix.
How the crawler pipeline works
Each submitted URL goes through: queued → crawling → crawling_bots → done (or failed).
- Reference crawl — fetch as a normal browser; record status, timing, title, description, canonical, H1, language, word count,
meta robotsandX-Robots-Tag. - robots.txt — fetched from the origin and matched per user-agent group (Googlebot first, then
*). A missing robots.txt (404) means no restrictions. - sitemap.xml — checked for existence and whether the URL is listed.
- Bot matrix — the same URL is fetched with every enabled crawler UA (parallel pool). Each result records HTTP status, response time, blocked flag, and whether the body content matches the browser fetch.
- Aggregation — the job reports Bots OK n/19 plus averages, exportable as CSV. Read the deep dive on the how it works page.
API reference
Submit URLs
POST /api/submit
{ "url": "https://example.com/page" }
{ "urls": ["https://a.com", "https://b.com"] }
{ "urls": "https://a.com\nhttps://b.com" }Other endpoints
GET /api/jobs?status=&q=&limit=— list jobsGET /api/jobs/:id— job detail with raw crawl JSONPOST /api/jobs/:id/retry— re-run the crawlGET /api/stats— totals + average bot access rateGET /api/settings— public settings (bots, timeouts, SEO overrides, SmartLink config)POST /api/checker/text—{ "text": "..." }→ readability + keywordsPOST /api/checker/page—{ "url": "...", "strategy": "mobile|desktop" }→ SEO audit + text analysis + PageSpeedPOST /api/live-index—{ "domain": "example.com" }→ normalises to the main domain, crawls + audits it and publishes an indexable page at/example.comGET /api/live-index?limit=— recent live-index submissionsGET /sitemap.xml— auto-generated sitemap (site pages + live-index domain pages)
Live Index accepts main domains only: any pasted URL form is normalised to the registrable domain (scheme, path, query, port and www stripped). The generated page mirrors the domain's own title, description and keywords, adds JSON-LD schema markup, shows the full meta + audit report and links back with a dofollow link.
Admin endpoints (/api/admin/*) require the x-admin-token header. On the live deployment the token is a Pages secret; when self-hosting it is printed in the server console on startup and stored in data/settings.json.
Admin settings shape
PUT /api/admin/settings
{
"siteName": "Fast Index Google",
"crawlTimeoutMs": 15000,
"botPool": 5,
"concurrency": 3,
"refreshMs": 3000,
"bots": { "Googlebot": true, "Yeti": false },
"seo": {
"metaTitle": "{page} — Free Multi-Bot Crawl Checker",
"metaDescription": "...",
"keywords": "...",
"ogImage": "https://example.com/og.png",
"googleAnalyticsId": "G-XXXXXXXXXX",
"googleSiteVerification": "..."
},
"smartlink": { "enabled": true, "url": "https://ad.example.com/abc", "perHour": 3 }
}In metaTitle, the placeholder {page} is replaced with each page's own static title at runtime. Leaving SEO fields empty keeps the per-page static meta tags.
Deployment notes
- Local:
npm install && npm start— Node.js + Express + better-sqlite3, data indata/. - Cloudflare Pages:
npm run build && npm run deploy— static export plus Pages Functions, D1 database, and the Durable Object scheduler (npm run deploy:schedulerwhen the scheduler code changes). - Environment:
PORT,CRAWL_TIMEOUT_MS,BOT_POOL,CONCURRENCY,REFRESH_MS,BOTS,ADMIN_TOKEN(self-hosted);ADMIN_TOKENas a Pages secret on the live site.
FAQ
Is Fast Index Google free?
Yes. Every crawler is a plain HTTP fetch and PageSpeed Insights is Google's free public API. No API keys, no quotas, no subscriptions.
What does "Bots OK 12/19" mean?
Twelve of the nineteen crawlers received a valid, non-blocked response. Open the job to see exactly which bots were blocked, their status codes and response times.
Why does a bot show "blocked"?
The server answered 401, 403, 429 or 999 for that user agent, or the request timed out. That is a server-side decision about that specific crawler — often WAF or cloud bot management.
What does "content differs" mean?
That bot received different body content than the reference browser fetch. This can indicate cloaking, or a site that relies on JavaScript rendering (a real Googlebot renders JS; our fetches do not).
Can I use fewer crawlers?
Yes — toggle individual bots in the admin panel, or restrict the bot list with the BOTS environment variable when self-hosting.
Where is data stored?
In the deployment's database — SQLite locally (data/fastindex.db), Cloudflare D1 on the live site — plus settings. Nothing else leaves your machine except the crawl fetches themselves and the PageSpeed API call.
Does this submit my site to search engines?
No. It shows what crawlers actually receive. For submission, use Google Search Console and Bing Webmaster Tools.
How is this deployed on Cloudflare?
Cloudflare Pages serves the static site and the API (Pages Functions), D1 stores data, and a Durable Object provides the cron loop that drains the queue — Pages itself has no cron triggers. See how it works.
Can I self-host it?
Yes. The Node.js + Express version runs locally with better-sqlite3; the Workers modules in functions/lib/ mirror the same pipeline. Both targets share the same settings shape.
What are the SmartLink popup ads?
An optional monetization feature. When enabled in the admin panel, the first click anywhere on a page opens the advertiser URL in a background popup window, capped at 3 times per hour per browser via local storage. It never runs on the admin panel.