Guide · Updated October 2026

What Is Robots.txt and Do I Need One?

Robots.txt tells search engine crawlers which parts of your site to avoid. Learn what it does, a safe basic example, and the costly mistakes to watch for.

What Is Robots.txt and Do I Need One?

Short answer: Robots.txt is a small text file at the root of your website that tells search engine crawlers which areas they should not crawl. Most small business sites only need a very simple one, and the biggest risk is accidentally blocking your whole site from Google.

Where it lives

It always sits at yourdomain.com/robots.txt. You can type that address into your browser right now to see yours. If you get a 404 error, you do not have one, which is not a disaster: crawlers will simply assume they can visit everything.

What a basic robots.txt looks like

User-agent: *
Disallow:
Sitemap: https://www.example.com/sitemap.xml

In plain English: all crawlers are allowed everywhere, and here is where the sitemap is. For most small sites, that is all you need.

The key lines explained

  • User-agent: which crawler the rule applies to. An asterisk means all of them.
  • Disallow: a path crawlers should not visit. Empty means nothing is blocked.
  • Allow: an exception inside a blocked folder.
  • Sitemap: the address of your XML sitemap.

The mistake that wipes sites off Google

This single line blocks everything:

Disallow: /

Developers sometimes add it while building a site to keep it hidden, then forget to remove it at launch. If your new site has not appeared on Google after weeks, check this first. Also check your platform settings: WordPress has a 'Discourage search engines' box that does something similar.

We check robots.txt, sitemaps and indexing on every launch. Explore SiteDrum plans.

SiteDrum builds fast, mobile-first websites from $99 setup and $15 a month, with hosting, security, Google setup and support included. Already have a site that is slow or broken? Try Website Rescue or run our free speed test.

What robots.txt does not do

  • It does not hide pages from search results reliably. A blocked page can still appear in Google if other sites link to it, just without a description. To keep a page out, use a noindex tag and let Google crawl it.
  • It is not security. The file is public, and bad bots ignore it. Never use it to hide private areas. Use passwords instead.

Sensible things to block

  • Admin areas, like /wp-admin/ (WordPress typically allows a specific file inside it).
  • Internal search result pages that create endless duplicate URLs.
  • Shopping basket and checkout steps.

Never block your CSS or JavaScript files. Google needs them to see your pages properly.

What about AI crawlers?

Some business owners use robots.txt to opt out of certain AI crawlers by naming their user agents. Whether you want to is a business decision: blocking them may reduce how often AI tools can reference your site.

How to test yours

Google Search Console has a robots.txt report showing the version Google last fetched and any errors. You can also use URL Inspection on a page to see if it is blocked. If something looks wrong and you are not sure how to fix it, ask us.

The bottom line

Robots.txt is a short instruction file for crawlers. Keep yours simple, point it to your sitemap, never block the whole site by accident, and remember it is not a privacy tool.

Frequently asked questions

Do I need a robots.txt file?

It is not required, but a simple one pointing to your sitemap is good practice.

Can robots.txt remove a page from Google?

Not reliably. Use a noindex tag, or Search Console's removals tool for urgent cases.

Why is my site not on Google?

One common cause is a robots.txt line saying Disallow: / left over from development.

Is robots.txt case sensitive?

Paths in it are case sensitive, so /Admin and /admin are treated differently.

Want a website that brings you customers?

Tell us what you do on WhatsApp and we will send a fixed quote and a free mockup.

Get my free mockup

Related guides

How Do I Do Keyword Research for Free?Read guide →What Is Domain Authority and Does It Matter?Read guide →Why Did My Google Rankings Drop?Read guide →How Do I Choose Google Business Profile Categories?Read guide →

Our products

Starter website$99 setup, $15 a monthBusiness website$249 setup, $29 a monthPremium website$499 setup, $59 a monthWebsite RescueFix a slow, broken or hacked siteFree speed testCheck your site in secondsSample websitesSee designs for your industry

Website design near you

Website design in LondonWebsite design in New YorkWebsite design in TorontoWebsite design in EdinburghWebsite design in PhoenixWebsite design in DublinWebsite design in AntwerpWebsite design in MiamiWebsite design in BergenWebsite design in SydneyAll cities →

SiteDrum home · All guides · Sitemap · Contact us

All guides

Contact us

Tell us about your business

Send a message and we reply within one working day with a fixed quote. Prefer to chat? WhatsApp us or book a free 15-minute call.

We only use your details to reply to you. See our privacy policy.
Chat on WhatsAppBook a free call