What this is
CleanFindingBot is a crawler we're building to reduce our reliance on third-party search APIs over time. It reads publicly available pages to build a family-safe search index. It never blocks or replaces our existing search results - it only ever supplements them once a page has been fetched, classified as safe, and judged relevant.
User-agent
CleanFindingBot identifies itself with this User-Agent string on every request:
CleanFindingBot/1.0 (+https://cleanfinding.com/crawler-info)
How it behaves
- Reads and respects
robots.txt, includingDisallowrules andCrawl-delay, before fetching anything from a host. - Honors
noindexandX-Robots-Tagsignals encountered while crawling. - Fetches pages at a bounded rate per host, with a byte-size cap and a short timeout per request - it is not designed to be a heavy load on any site.
- Every fetched page goes through a fail-closed safety classifier before it can ever be surfaced in search results; anything that can't be confidently classified as safe is excluded, never guessed as safe by default.
How to block it
Add the following to your site's robots.txt to disallow CleanFindingBot entirely:
User-agent: CleanFindingBot
Disallow: /
We check robots.txt before every fetch, so this takes effect on our next visit to your site.
Submit your site for indexing
Want CleanFindingBot to crawl your site sooner rather than waiting for us to discover it via links? Submit its URL below - we'll still check your robots.txt first, and a submission doesn't guarantee inclusion in search results (pages still go through safety classification).
Report abuse or ask to be excluded immediately
If you believe CleanFindingBot is misbehaving, or you'd like to be excluded without waiting for us to re-check your robots.txt, contact us and we'll add your domain to our denylist right away:
- Email: privacy@cleanfinding.com
- Contact Form: cleanfinding.com/contact