Yoast SEO Tutorial: How to View and Modify Robots.txt File

robots.txt It is an important file that directs search engines to crawl your website and control what content is allowed to be included. For WordPress users, this file can be easily managed using the Yoast SEO plugin. In this article, we will introduce the role of robots.txt, the basic rules, and how to use the Yoast SEO Make settings.

I. What is Robots.txt file?

Robots.txt is a plain text file located in the root directory of a web site that directs search engine spiders (such as the Googlebot(Bingbot) to crawl or skip specific content.

Example:

User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php

The meaning of this code is: disable access to all search engines /wp-admin/ directory, but allow crawling AJAX Interface.

Second, why set Robots.txt?

A properly set up Robots.txt file can help:

• Stop meaningless or duplicate content from being indexed(e.g., back office, category pages, tabs)
• Protect sensitive pages from being indexed
• Direct search engines to focus crawling resources on high-value pages

For example.WooCommerce Stores often block /cart/,/checkout/ and other functional pages to avoid low quality content that can slow down SEO performance.

How to use Yoast SEO plugin to view Robots.txt file?

The Yoast SEO plugin for WordPress provides built-in Robots.txt file viewing and editing functionality without the need to manually access the hosting or FTP.

The steps are as follows:

• Login to WordPress Backend
• Go to the menu:Yoast SEO → Tools

• Click [File Editor] (File Editor)

• If your site allows editing, the current Robots.txt content will be displayed on the page

If the file doesn't exist yet, Yoast SEO provides a "one-click create" button that automatically generates the file in the root directory of the website. robots.txtThe

How to modify the Robots.txt file?

In the [File Editor] page, you can directly edit the Robots.txt content, for example:

User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php
Disallow: /tag/
Disallow: /author/
Sitemap: https://yourdomain.com/sitemap_index.xml

Once you have finished editing, click the [Save Changes] button to update the robots.txt file.

V. Common Robots.txt configuration examples

Standard configuration (fits most WordPress sites):

User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php
Disallow: /?s=
Disallow: /tag/
Disallow: /author/
Sitemap: https://yourdomain.com/sitemap_index.xml

Enhanced privacy configurations:

User-agent: *
Disallow: /wp-login.php
Disallow: /wp-register.php
Disallow: /checkout/
Disallow: /cart/

Note: Do not block the following content, otherwise it will affect the page display effect

Disallow: /wp-content/themes/
Disallow: /wp-content/plugins/

This will prevent search engines from loading CSS/JS, thus affecting the mobile usability score.

VI. Verify whether Robots.txt is effective

Once you have completed the setup, you can access it through your browser:

https://yourdomain.com/robots.txt

Make sure the content is consistent with what is in the Yoast editor. The following tools are also recommended for testing:

• Google Search Console → Check Web site tool
• Screaming Frog SEO Spider
• Ahrefs Site Audit Tool

VII. Summary and recommendations

The Robots.txt file may be small, but it plays a big role in your SEO strategy. With the Yoast SEO plugin, it's easy to manage this critical file without going into the server. It is recommended that you check and update Robots.txt regularly to ensure that your search engine crawling strategy is aligned with your website structure and marketing goals.

Recommended Practice:

• Block low quality, duplicate or private pages
• reservations Sitemap Addresses, favorable indexing
• Avoid blocking CSS/JS file loading

Contact Us
Can't read the tutorial? Contact us for a free answer! Free help for personal, small business sites!	Customer Service
① Tel: 020-2206-9892
② QQ咨询：1025174874
(iii) E-mail: info@361sale.com
④ Working hours: Monday to Friday, 9:30-18:30, holidays off