Disallow Directive Test
The Disallow Directive Test isolates the Disallow rules in your robots.txt file and shows you exactly which URLs each rule blocks for each major bot. We simulate Googlebot, Googlebot-Image, Bingbot and other major crawlers against a live sample of your URLs, then surface any rule that blocks important content unintentionally — the single most common cause of unexplained traffic drops after a robots.txt change.
What This Tool Checks
- Every Disallow rule with the user-agent it applies to
- Sample URLs blocked by each rule
- Wildcards (*) and end-of-URL anchors ($) interpretation
- Conflict between Allow and Disallow rules
- Per-bot rules vs. wildcard "User-agent: *" defaults
- Catastrophic rules (Disallow: / for Googlebot)
- Disallow on resources Google needs (CSS, JS, images)
Why It Matters for SEO
Disallow rules look simple but their interaction with wildcards, end anchors and per-bot user-agent overrides is subtle. A rule like "Disallow: /search" can accidentally block "/search-engine-results" if you forget the trailing slash. A rule under a specific User-agent block silently overrides the wildcard rules. Misreading these interactions is how teams accidentally block Googlebot from their entire product catalogue while believing they only blocked internal search.
How to Fix It
Keep Disallow rules narrow and specific. Anchor the end of paths with $ where possible (Disallow: /search$). Never block CSS, JS or image directories Google needs for rendering. To remove indexed URLs, use noindex on the page itself rather than Disallow in robots.txt.
How It Works
We parse robots.txt using the same precedence rules Google's open-source robots.txt parser uses, then test each declared user-agent against a representative sample of your URLs (homepage, top pages, sample asset URLs). The output is a clear "X URLs blocked by this rule for this bot" table.
Common Mistakes to Avoid
- Disallow: /search blocking /search-results unintentionally
- Specific user-agent block silently overriding wildcard rules
- Disallow rules on /assets or /static blocking CSS and JS
- Forgetting to remove staging-environment Disallow: / rule on launch
- Trying to use Disallow to remove already-indexed URLs (won't work)
Quick Checklist
- Each Disallow rule blocks only the URLs you intend
- No Disallow rule blocks CSS / JS / image folders
- Per-bot rules deliberately override the wildcard
- No "Disallow: /" left over from staging
- noindex (not Disallow) is used to remove indexed URLs