I Blocked My Best Pages by Accident.A Single Line Did It.
I was staring at a traffic drop and couldn't figure out why. Turns out my robots.txt file had a disallow rule that was too broad.
I'd meant to block a single folder but wrote it in a way that caught half my site. One misplaced asterisk, and Google stopped crawling pages I actually wanted ranked.
Your robots.txt is a text file in your site's root directory that tells search engines which pages they can and can't crawl. It's not a security tool, and it won't hide anything from the public.
Think of it as a polite instruction manual: don't waste time on this folder, focus on these pages. Google's crawler documentation covers the syntax, but the core rule is be specific.
Disallow: /admin/ blocks only that folder. Disallow: / blocks everything.
The mistake I made happens more than you'd think. You add a rule to block something temporary, forget about it, and six months later you're wondering why new content isn't ranking.
Check your robots.txt now if you haven't looked in a while. It's a small file with outsized impact, and our SEO work reviews yours during the audit phase.
Go to yoursite.com/robots.txt and read what's actually there. If you see anything you don't recognize or remember adding, especially a broad Disallow rule, note it. One misplaced line can quietly block the pages you most want ranked.
