How can I prevent the Google-Bot from crawling my website?

From:

Lily Ray, Steve Paine

Modified: 07.07.2025

Google-Index, Google-Bot and the Crawling and Indexing Process

• What is the Google Everflux?
• How to Identify Googlebot on Your Server
• Knowledge Graph - How Google Understands Things and How Often Is It Used?
• Googlebot Mobile Overview and Features
• Can the Google-Bot fill out and crawl forms?
• What is the Mobile First Index?
• Google Search and Entities
• Our web site is no longer in the index - have we lost our rankings?
• Crawl Budget: What does this mean?
• Rich Snippets: What Are the Advantages?
• What is the X-Robots-Tag?
• Robots meta tag vs. robots.txt: what are the main differences?
• Web Crawlers: How do They Work?
• What is Google Search Console and How To Get Started
• These are the CTR's For Various Types of Google Search Result
• What is a User-Agent?
• Crawling and Indexing for extensive websites
• Google SERP Features: Result Types in the Search Results
• Why does the amount of indexed pages fluctuate so much?
• How can I quickly get a new page into Google's index?
• Is a website with and without the www harmful?
• Why does a blocked, noindex URL show up in the search results?
• Find out how many pages of a domain are indexed by Google
• How can I prevent the Google-Bot from crawling my website?
• Why does a Google search with quotes sometimes deliver more results than the same search without it?
• Back to overview

Discover how SISTRIX can be used to improve your search marketing. 14 day free, no-commitment trial with all data and tools: Test SISTRIX for free

Lily Ray talks about blocking crawlers from a website using robots.txt

Note: If you are looking to block specific URLs, you can use the robots meta-tag. Also be aware that there are some cases where URLs will still be indexed, even if they are blocked using the robots.txt file.

Blocking the Google-Bot using the robots.txt

The robots.txt is a simple text file with the name “robots”. It has to be placed in the root-directory of a website in order for search engines to follow the directives.

If a website has a robots.txt, it can be accessed through the following path:

http://www.my-domain.com/robots.txt

The contents of the robots.txt

By using the following instructions, we exclusively forbid the Google-Bot access to our entire website:

You have to add the following to your robots.txt to tell Google-Bot to stay away from the entire domain:

User-Agent: Googlebot
Disallow: /

If you only want to restrict access to some directories or files instead of the entire website, the robots.txt has to contain the following:

The following only tells Google-Bot that it is forbidden from accessing the directory “a-directory” as well as the file “one-file.pdf”:

User-Agent: Googlebot
Disallow: /a-directory/
Disallow: /one-file.pdf

Some URLs may still be indexed

The code examples shown here are only meant for Google-Bot. Crawlers from other search engines, such as Bing, will not be blocked.

Block crawlers using WordPress

WordPress has a built-in feature that will set a robots meta-tag to noindex in the header of every page.

1

Assuming you have the Administrator rights in the WordPress site, go to the Settings -> Reading page and select “Discourage search engines from indexing this site” 1 as shown above.

More information on Googlebot and crawler control

What is the difference between robots.txt and the robots meta-tag?

https://support.google.com/webmasters/answer/6062608?hl=en