Since launch, CleanFinding's search results have come entirely from a third-party provider. That's worked well, but it also means every result you see passes through someone else's index first. We've started building our own: a first-party web crawler and search index, powered by a bot we call CleanFindingBot.
In short: CleanFindingBot crawls publicly available web pages, runs every single one through an AI safety classifier, and stores only what passes. When you search, we quietly blend a small number of our own safety-cleared, relevant results in alongside your existing results. Nothing about your search experience changes unless our index has something genuinely good to add - and if it doesn't yet, you'll never notice a difference.
Why Build Our Own Index?
Relying entirely on a third-party search API means we're always one step removed from the actual content - we can filter what comes back, but we don't control how it was found or indexed in the first place. A first-party index lets us apply CleanFinding's own safety standards from the moment a page is discovered, not just at the moment it's displayed. It's also simply more resilient: less dependence on any single outside provider, long term.
This is very much a first step, not a replacement. Our third-party search provider remains the primary source of results today and isn't going anywhere - our own index only ever supplements it.
Safety Comes First, Not After
The part we care most about isn't the crawling - crawling the web is a solved problem. It's what happens before a crawled page is ever allowed to reach you.
Every page CleanFindingBot fetches goes through an automated safety classifier before it's eligible to appear anywhere in search results. That classifier is fail-closed: if it can't confidently tell that a page is safe, the page is excluded - not guessed as safe by default. Under the hood, this works by turning a page's text into a numeric representation (an embedding) and scoring it with a small model we trained ourselves; only pages that score confidently safe ever make it into the index, and anything in between gets held back rather than risked.
The default is exclusion, not inclusion. A page has to actively clear our safety bar to show up - uncertainty always resolves to "don't show it," never "probably fine."
How It Behaves on the Web
CleanFindingBot is a polite, well-identified crawler:
- It identifies itself clearly with its own User-Agent string, linking back to a page explaining exactly what it does.
- It reads and respects
robots.txton every site, includingDisallowrules andCrawl-delay, before fetching anything. - It honors
noindexsignals it encounters while crawling. - It fetches at a bounded rate per site, with a byte-size cap and a short timeout - it's not designed to be a meaningful load on anyone's server.
- Site owners can block it entirely via
robots.txt, or contact us directly for immediate removal without waiting for us to recheck it.
Full details, including exactly how to block it, live at cleanfinding.com/crawler-info.
Submit Your Site
Rather than waiting for CleanFindingBot to discover a site purely through links, you can now nominate a URL directly. We'll still check that site's robots.txt first, and submitting a URL doesn't guarantee it gets crawled, indexed, or shown in results - every submission goes through the same safety classification as anything else we crawl. It's a starting point for discovery, not a fast lane around our standards.
What's Live Today vs. What's Ahead
We'd rather be precise about where this actually stands than oversell it:
| Piece | Status |
|---|---|
| Crawling, robots.txt compliance, politeness limits | ✅ Live |
| Fail-closed AI safety classification on every page | ✅ Live |
| Blending safety-cleared results into search | ✅ Live |
| Public site submission tool | ✅ Live |
| Index size and coverage | 🟡 Early and growing |
| Learned relevance ranking (beyond text-match heuristics) | 🟡 In progress |
The index today is small on purpose - we'd rather grow it carefully behind a real safety bar than rush scale ahead of it. You may not see our own results in every search yet; that's expected, and it'll change as the index grows.
Frequently Asked Questions
Does this replace my existing search results?
No. Our first-party index only ever supplements what you already get. If it has nothing confident and relevant to add, search behaves exactly as it does today.
Can I block CleanFindingBot from my site?
Yes - add a Disallow rule for it in your robots.txt, or email privacy@cleanfinding.com for immediate removal.
Can I submit my own site?
Yes, at cleanfinding.com/crawler-info. It's a nomination, not a guarantee - every submitted page still goes through robots.txt checks and safety classification.
Submit Your Site
Nominate your site for CleanFindingBot to crawl - still subject to robots.txt and our safety classifier, like everything else we index.
Go to Crawler InfoHave questions or feedback? We'd love to hear from you. Contact us at hello@cleanfinding.com or through our contact form.