← nohumans.directory

crawl-preflight.postnov01.workers.dev/crawl-check/:domain

● failing · data.web · 0.02 USDC/call · base, polygon, eip155:42161

Crawl pre-flight for a domain: will you be blocked, is content behind a paywall, and how much is actually there. Returns per-bot robots.txt verdicts for 16 AI crawlers, crawlable page count from sitemaps, detected CDN and paid-access signals (x402, TollBit), plus a fetch/skip recommendation. Call before spending requests on an unknown domain.

Endpoint

https://crawl-preflight.postnov01.workers.dev/crawl-check/:domain

Verification

0%reputation score
13probes total
816ms~latency
neverlast passed

last 13 probes, oldest → newest · 0% pass · re-probed several times an hour from nohumans infrastructure — never self-reported

Paid verification

Not yet paid-verified. Probe-based status above is liveness only.

Badge

failing — nohumans.directory

[![nohumans.directory](https://nohumans.directory/badge/12709d34-e12.svg)](https://nohumans.directory/l/12709d34-e12)

Machine interface

curl https://nohumans.directory/v1/listings/12709d34-e12

Own this service? Claim this listing to control its metadata — proof-of-control via your own endpoint, ~2 minutes. Works for submitted listings too, and recovers a lost token.