Disallow Directive Test
The Disallow Directive Test isolates the Disallow rules in your robots.txt file and shows you exactly which URLs each rule blocks for each major bot.
- Results in seconds
- Pass / fail + fix guidance
- No account required
The Disallow Directive Test isolates the Disallow rules in your robots.txt file and shows you exactly which URLs each rule blocks for each major bot. We simulate Googlebot, Googlebot-Image, Bingbot and other major crawlers against a live sample of your URLs, then surface any rule that blocks important content unintentionally — the single most common cause of unexplained traffic drops after a robots.txt change.
What This Tool Checks
- Every Disallow rule with the user-agent it applies to
- Sample URLs blocked by each rule
- Wildcards (*) and end-of-URL anchors ($) interpretation
- Conflict between Allow and Disallow rules
- Per-bot rules vs. wildcard "User-agent: *" defaults
- Catastrophic rules (Disallow: / for Googlebot)
- Disallow on resources Google needs (CSS, JS, images)
Why It Matters for SEO
Disallow rules look simple but their interaction with wildcards, end anchors and per-bot user-agent overrides is subtle. A rule like "Disallow: /search" can accidentally block "/search-engine-results" if you forget the trailing slash. A rule under a specific User-agent block silently overrides the wildcard rules. Misreading these interactions is how teams accidentally block Googlebot from their entire product catalogue while believing they only blocked internal search.
How to Fix It
Keep Disallow rules narrow and specific. Anchor the end of paths with $ where possible (Disallow: /search$). Never block CSS, JS or image directories Google needs for rendering. To remove indexed URLs, use noindex on the page itself rather than Disallow in robots.txt.
How It Works
We parse robots.txt using the same precedence rules Google's open-source robots.txt parser uses, then test each declared user-agent against a representative sample of your URLs (homepage, top pages, sample asset URLs). The output is a clear "X URLs blocked by this rule for this bot" table.
Common Mistakes to Avoid
- Disallow: /search blocking /search-results unintentionally
- Specific user-agent block silently overriding wildcard rules
- Disallow rules on /assets or /static blocking CSS and JS
- Forgetting to remove staging-environment Disallow: / rule on launch
- Trying to use Disallow to remove already-indexed URLs (won't work)
Quick Checklist
- Each Disallow rule blocks only the URLs you intend
- No Disallow rule blocks CSS / JS / image folders
- Per-bot rules deliberately override the wildcard
- No "Disallow: /" left over from staging
- noindex (not Disallow) is used to remove indexed URLs
Put your whole site on autopilot — SEO and AI search
PositionMySite monitors every signal on this page across your entire website 24/7 — plus keyword rankings, competitor moves and AI-search readiness (llms.txt, schema, ChatGPT & Gemini visibility). When something breaks, you know before Google does.
Every feature unlocked · No commitment · Cancel anytimeFrequently Asked Questions
A line in robots.txt that tells a specific crawler (or all crawlers via the * wildcard) not to fetch URLs matching the given path pattern. Disallow only controls crawling, not indexing.
No. Disallow blocks crawling but the URL can still be indexed (without content) if other sites link to it. Use noindex on the page itself to remove URLs from the index.
* matches any sequence of characters; $ anchors to the end of the URL. "Disallow: /*?utm" blocks any URL containing "?utm". "Disallow: /print$" blocks /print but not /printer-friendly.
You probably have a Disallow rule on the directory holding them. Google needs CSS and JS to render the page properly for mobile-friendly testing — never block them.
Yes. Add a User-agent: GoogleBot block above the wildcard with its own Disallow rules. Per-bot blocks fully override wildcard rules for that bot — they do not stack.