What Is Robots.txt? A Beginner Guide to the File That Controls Search Crawlers

Robots.txt is like a "staff only" sign on a door. Search engine crawlers check it before entering your site. Polite bots follow the rules. Some do not. But without a sign at all, every bot walks in everywhere — including places you did not want them to go.

What Robots.txt Actually Does

When Google, Bing, or any search engine sends a bot to crawl your website, the bot's first stop is always the same: yoursite.com/robots.txt. It reads this file to find out:

• Which pages or directories it is allowed to crawl
• Which pages or directories it should skip
• Where your sitemap is located (so it can find all your pages efficiently)

The file is plain text. No special formatting. No code. Just simple rules that bots understand.

Step 1: Check If You Already Have One

Open your browser and go to:

yoursite.com/robots.txt

If you see text with rules like "User-agent" and "Disallow," you already have a robots.txt file. If you see a 404 error, you do not have one — and that means crawlers are accessing everything on your site without any guidance.

Step 2: Understand the Basic Rules

Robots.txt uses four main instructions. That is it. Four:

Instruction	What It Means	Example
User-agent	Which bot these rules are for (* means all bots)	User-agent: *
Disallow	This path is off limits — do not crawl it	Disallow: /admin/
Allow	This path is okay — override a Disallow rule	Allow: /admin/public-page/
Sitemap	Here is where all my pages are listed	Sitemap: https://yoursite.com/sitemap.xml

A complete robots.txt file is just combinations of these four instructions. Nothing more complicated than that.

Step 3: Create Your Robots.txt

The fastest way: use the Robots.txt Generator. Select which directories you want to block, which bots you want to target, and it writes the correct file for you. Download it and upload it to your website.

If you want to write it by hand, here is a simple starting point that works for most websites:

User-agent: *
Disallow: /admin/
Disallow: /private/
Disallow: /staging/

Sitemap: https://yoursite.com/sitemap.xml

This tells all bots: you can crawl everything except the admin, private, and staging directories. And here is the sitemap so you can find all the public pages efficiently.

Step 4: Upload It

1. Save your file as robots.txt (plain text, lowercase, no extension tricks)
2. Upload it to the root directory of your website — the same folder where your homepage files live
3. Verify by visiting yoursite.com/robots.txt in your browser
4. You should see your rules displayed as plain text

If you use WordPress, you can edit robots.txt through your SEO plugin (Yoast or Rank Math) without touching FTP. See our WordPress robots.txt guide for step-by-step instructions.

Three Things Robots.txt Does NOT Do

These are the most common misconceptions — and getting them wrong can cause real problems:

Misconception	Reality
Robots.txt hides pages from Google	Wrong. It tells Google not to CRAWL the page, but Google can still INDEX the URL if other websites link to it. The page can appear in search results with no description. To truly hide a page, use a noindex meta tag.
Robots.txt is security	Wrong. Anyone can read your robots.txt file by visiting yoursite.com/robots.txt. In fact, it often reveals which directories exist. Never rely on robots.txt to protect sensitive content. Use passwords and authentication.
Blocked pages are invisible	Wrong. Robots.txt blocks crawling, not linking. If another website links to a page you have blocked, search engines may still show that URL in results — they just cannot access the content to display a description.

Real-World Analogy

Imagine your website is an office building:

• Robots.txt is the directory sign in the lobby: "Visitors welcome on floors 1-5. Floors 6-10 are restricted."
• Search engine bots are the visitors. Most read the sign and follow it.
• The sign does not lock doors. A determined person can still walk up to floor 6. For actual security, you need locked doors (passwords, authentication).
• Noindex is different: it is like letting visitors in but asking them not to take photos or share what they see.

When You Definitely Need a Robots.txt

• You have admin panels, dashboards, or login pages you do not want in search results
• You have a staging or development version of your site
• You want to block AI training bots from scraping your content (see our AI bot blocking guide)
• You have internal search result pages that create thin content in Google's index
• Your site has thousands of pages and you want crawlers to focus on the important ones

Tools to Get Started

• Robots.txt Generator — create a valid robots.txt file without memorizing syntax
• Meta Tag Generator — create noindex tags for pages that need to be fully hidden from search results
• Open Graph Checker — make sure your social sharing tags are working correctly
• Question Finder — discover questions people ask about SEO topics
• Headline Analyzer — improve your page titles for better click-through rates
• Readability Scorer — check that your content reads clearly for your audience
• Keyword Density Checker — analyze keyword usage before publishing