How to Create and Add Rules in WordPress Robots.txt File

If you want to controlSearch engine access to websitesCreate robots.txt The file is the first step. This is an ASCII text file that tells the search engine which pages can be indexed and which pages should not be indexed.

Correct editing robots.txt Files can significantly impact a site's SEO, help improve site visibility, and also secure sensitive files.

What is a robots.txt file?

robots.txt file is a rule file for search engine crawlers, which defines which parts of a website are open to crawlers and which parts are off limits. With proper settings, you can better manage the content indexing strategy of your website.

How to create and customize robots.txt file?

1. Understanding robots.txt Basic syntax rules for documents.
2. Add rules that allow or prohibit according to the needs of the site.
3. Upload the file to the root directory of the website to ensure that search engines can read it correctly.

What is robots.txt file?

robots.txt is a simple text file that is used to send a message to theweb crawler(e.g. search engine robots) indicate which parts of a website can be crawled and which parts need to be hidden. It ensures that search engines don't index pages or files that shouldn't be made public, thus protecting the privacy of the site and the security of its content.

By default, WordPress automatically generates a virtual robots.txt file with the path:www.yourdomain.com/robots.txt. This is accomplished by creating a custom robots.txt file, you can replace the default file and thus control more precisely which files or pages are not indexed by search engines.

WordPress robots.txt file location

robots.txt The file is usually located in the root directory of the website.WordPress automatically generates a virtual version of the robots.txt file, but if you do not create it manually, the file is not actually saved on the server.

How can I view the robots.txt file of my website?

1. Open your browser.

2. In the address bar, type https://您的域名/robots.txt And visit.

• If the file exists, the browser displays robots.txt The contents of the file.

• If no content is displayed, the site has not yet been created robots.txt Documentation.

This can be done by creating a custom robots.txt file to accurately manage search engine access to website content.

Understanding the rules in the robots.txt file

robots.txt The directives in the file define the behavior of a web crawler (e.g., a search engine robot). Directives consist of two main parts:

1. User-agent: Specifies which crawler (e.g. Googlebot) the rule applies to.

2. Disallow: Specifies which pages or directories are not allowed to be crawled.

Example:

User-agent: *
Disallow: /wp-admin/

This code instructs all crawlers to disable crawling /wp-admin/ directory, but other areas can be accessed normally.

Correctly understand and use robots.txt The rules in the file are critical. Incorrect configuration can lead to unintended consequences, such as preventing search engines from accessing important pages of the site.

Here are some common robots.txt A brief description of the rules and their uses:

rules and regulations	instructions
User-agent: *	Specifies that the following rules apply to all crawlers.
Disallow: /	Disable all crawlers from accessing any page of the site.
Disallow: /private/	Block all crawler access /private/ All pages in the directory.
Allow: /public/	Allow all crawlers access /public/ All pages in the directory.
Sitemap: https://www.example.com/sitemap.xml	Specify the location of the Sitemap file of the website, so that crawlers can index the content better.

By properly configuring these rules, you can better manage crawler behavior and improve SEO results while protecting the privacy of sensitive content.

How to create WordPress robots.txt file

Creating a robots.txt file can be done in several ways, but in this post, 2 simple methods are presented:

Method 1: Edit robots.txt file with plugin

look as if WPCode respond in singing Virtual Robots.txt Such plugins can be easily managed without having to write code directly robots.txt Documentation. Simply install the plugin, go to its settings screen and add custom rules. This method is easy to use and perfect for newbies.

Editing robots.txt file with WPCode

1. In the WordPress dashboard Plugins > Install Plugins page, search and install WPCode Plug-ins.

2. After the installation is complete, go to Code Snippets option, and then click the File EditorThe

3. In File Editor In the robots.txt File Options.

4. Modify the contents of the document according to the requirements, save the changes and test them.

Editing the robots.txt file with Virtual Robots.txt

1. In Plugins > Install Plugins page, search and install Virtual Robots.txt Plug-ins.

2. After the installation is complete, go to the plugin's set up Page.

3. View the default rules provided by the plug-in or add custom rules as needed.

4. Save the changes and ensure that robots.txt The document meets your needs.

Using these plug-ins makes it easier to manage robots.txt Documentation that adapts to the needs of different websites while ensuring that the behavior of search engine crawlers is properly controlled.

Method 2: Create and upload WordPress robots.txt file via FTP

For more direct control of robots.txt file, you can create a physical file with a text editor and upload it using FTP. Here are the exact steps:

Create robots.txt file

1. Open a text editor (e.g. Notepad or other familiar tool).

2. Preparation of the required rules. For example:

3. Save the document as robots.txtThe

Upload to the root directory of the website

1. Use an FTP utility (e.g. FileZilla) or a control panel (e.g.pagodaetc.) to connect to the web server.

2. Navigate to the root directory of the site (usually the public_html maybe www (folder).

3. Place the just-created robots.txt The file is uploaded to the root directory.

Upon completion. robots.txt file will take effect and can be accessed by visiting the https://您的域名/robots.txt to verify that the contents of the file are displayed correctly. With this method, the contents and rules of the file can be completely controlled.

Adding rules to WordPress robots.txt

create a good idea robots.txt After the file, various rules can be added as needed to accomplish specific goals. Below are some common uses and code examples:

1. Preventing the entire site from being crawled

exist robots.txt The following code is added to the file to stop search engines from crawling the entire site:

User-agent: *
Disallow: /

2. Blocking specific crawlers from accessing the site

Add the following code to block a specific crawler (e.g. Dotbot) from accessing the site:

User-agent: Dotbot
Disallow: /

3. Block specific folders or files from being crawled

The following code prevents search engines from crawling a particular folder or file:

User-agent: *
Disallow: /private-folder/

4. Allow all crawlers to access the site

The following code allows all crawlers to access all content on the site:

User-agent: *
Disallow.

5. Allow access to specific files in banned folders

The following code allows the crawler to access specific files in a banned folder:

User-agent: *
Disallow: /private-folder/
Allow: /private-folder/public-file.html

6. Block crawling WordPress search result pages

The following code prevents the crawler from crawling the WordPress search results page:

User-agent: *
Disallow: /?s=

7. Set different rules for different crawlers

The following code sets up different rules for different crawlers:

User-agent: Googlebot
Disallow: /no-google/

User-agent: Bingbot
Disallow: /no-bing/

8. Blocking GPTBot from crawling website content

The following code prevents GPTBot from crawling your website content:

User-agent: GPTBot
Disallow: /

These examples demonstrate the robots.txt The rules can be flexibly adapted to a site's SEO strategy to better manage search engine behavior, protect site content and optimize search rankings.

Specify the user agent in the robots.txt file

Rules can be set up to target specific crawlers by specifying a user agent in the rule. Additionally, the use of wildcards can help manage crawler access to different parts of the site more flexibly.

1. Use wildcards in robots.txt file

Wildcards can set broad or specific rules based on patterns.

2. Use of asterisk (*) wildcards

An asterisk (*) indicates any user agent for all crawlers:

User-agent: *
Disallow: /private/

3. Use of dollar sign ($) wildcards

The dollar sign ($) indicates the end of a URL and is used to match a specific URL pattern:

User-agent: *
Disallow: /*? /*?

How to test robots.txt file

When creating or editing robots.txt It is important to verify the correctness of the file immediately after it has been created. This can be verified using the following tools:

1. Google Search Console
submit (a report etc) robots.txt document and check for compliance with the expected rules.

2. Specialized validation tools
use a tool like technicalseo.com An online tool like this performs the verification.

Validation Steps:

1. Enter the URL of the website (e.g. https://yourdomain.com/robots.txt).

2. Select the crawler or user agent you are interested in from the drop-down menu.

3. Click submit (a report etc) button to start the validation.

After completing these steps, it is easy to confirm robots.txt file is configured correctly, thus ensuring that the crawler runs according to the set rules.

summarize

manage well robots.txt Documentation is an important part of optimizing the SEO of your WordPress website. By following the above guidelines, it is possible to create a customized robots.txt file, both to improve the visibility of your website and to protect sensitive areas of your website from being crawled.

Contact Us
Can't read the tutorial? Contact us for a free answer! Free help for personal, small business sites!	Customer Service
① Tel: 020-2206-9892
② QQ咨询：1025174874
(iii) E-mail: info@361sale.com
④ Working hours: Monday to Friday, 9:30-18:30, holidays off