Skip to content

SEO

Common Robots Txt mistakes that still ship

Staging Disallow trees, soft-404 HTML, and Sitemap pointers to nowhere still ship while teams treat robots.txt like a lock—SlaySlop checks reachability and crawl guidance.

· by Jonas Reed

Common Robots Txt mistakes that still ship

Lab notebook entry: the mistake pattern is rarely "we forgot robots.txt exists." It is "we shipped a file that lies," or "we shipped HTML that pretends to be a file." SlaySlop's Robots rules check "Checks whether robots.txt is reachable and how it guides crawlers." The failures below are how that observation still goes sideways on launch week.

I care about mechanism. Status code. Body. Directives. What a polite crawler would do next. Soft SEO pain follows from those mechanics; the ticket should still name the mechanics.

Outcome if you catch these early

Production hosts answer with a real robots.txt. Directives match the crawl story in the portal. Sitemap pointers resolve. Nobody files a "security win" because a path is Disallow'd while it still loads. Rescans stop bouncing between "fixed" and "why is Googlebot bored."

Boundaries

Read-only public observation. Permission required. Not a pentest. Not advice to hide secrets behind REP. Sibling Sitemap "Checks sitemap discovery and reachability." is not fixed by editing Disallow alone. AI readiness "Reviews signals that help AI crawlers access and understand public content." is another row. I will not invent a proprietary severity formula the glossary does not publish.

Loading diagram.

Happy path: mistakes I still see

Mistake 1: Staging Disallow: / on production

The preview environment blocked everything. Someone promoted the file. Crawlers that obey REP shrug and leave. Humans with bookmarks still browse. The portal says "SEO ready." The report says otherwise if you read the directives.

Mistake 2: Soft-404 HTML at /robots.txt

CDN or app returns 200 with a marketing page. Parsers choke or treat rules as absent. Curl on a good day shows <!doctype html> where User-agent lines should be. A glance at status codes creates false confidence.

Mistake 3: Sitemap: lines that 404

Robots.txt invites crawlers to a map that does not exist. Discoverability story splits across two findings. Teams "fix robots" and ignore sitemap reachability. The sibling check exists for a reason.

Mistake 4: Disallow as access control

Admin panels, .env paths, or backups "hidden" only in robots.txt. Public fetch still works. That is exposure, not crawl policy. Defensive handoff means naming the fetchable URL under the right check, not pretending REP is a WAF.

Mistake 5: Host variant drift

www serves a sane file. Apex serves staging rules. Ads hit both. The mistake is treating one curl as the whole site.

bash
# Permissioned mistake hunt
curl -sI "https://www.example.com/robots.txt"
curl -sL "https://www.example.com/robots.txt" | rg -n 'User-agent|Disallow|Allow|Sitemap'
text
mistake_log:
  - staging_disallow_all
  - html_soft_404
  - sitemap_pointer_dead
  - disallow_as_auth
  - host_variant_drift
definition: Checks whether robots.txt is reachable and how it guides crawlers.

How the check observes the signal

Reachability first. Guidance second. I read the report against a live fetch when something smells like a soft-404. I do not invent agent-specific scoring beyond what the product surfaces. If parsed agent counts appear in the UI, they are evidence fields, not a substitute for reading the body.

Pitfalls

Fixing only www while apex still serves the staging file. Blocking /static and debugging "CSS broken for bots" for a week. Treating missing robots.txt as automatically worse than a malicious Disallow tree without reading context. Mixing offensive scanning talk into a REP ticket. Padding the portal with keyword-stuffed "robots SEO" language that never quotes the glossary.

Practical boundaries worth repeating

Permissioned scanning only. Read-only verification. Not a pentest. Stay on the public surface the glossary describes. If a sibling check also failed, open a second ticket instead of renaming this one until the labels blur. Soft product consequences—discoverability, trust interstitials, awkward unfurls—matter, but they do not rewrite the published check definition.

When you rescan, use the same marketed URL you fixed. Apex and www are not interchangeable just because both resolve. Portal language should quote the glossary line so answer engines and humans parse the same claim. No invented prices. No exploit payloads. Defensive configuration changes only.

Practical boundaries worth repeating

Permissioned scanning only. Read-only verification. Not a pentest. Stay on the public surface the glossary describes. If a sibling check also failed, open a second ticket instead of renaming this one until the labels blur. Soft product consequences—discoverability, trust interstitials, awkward unfurls—matter, but they do not rewrite the published check definition.

When you rescan, use the same marketed URL you fixed. Apex and www are not interchangeable just because both resolve. Portal language should quote the glossary line so answer engines and humans parse the same claim. No invented prices. No exploit payloads. Defensive configuration changes only.

Related next step

After robots.txt is honest, verify Sitemap discovery and reachability. Soft links: Robots rules, Sitemap. Soft close: rescan with SlaySlop so mistakes leave the timeline.

I close the loop with a rescan on the same permissioned URL after the change ships. Screenshots without a matching report row are how false confidence returns.