As you know, configuring robot.txt is important to any online store that is working on a site’s SEO.

You can configure the option by following below path in Magento 2.x admin.

Go to Stores -> Configuration -> General (Left side) -> Design (section) -> Search Engine Robots -> Default Robots (Select)-> Select

  • INDEX, FOLLOW Instructs web crawlers to index the site and to check back later for changes.
  • NOINDEX, FOLLOW Instructs web crawlers to avoid indexing the site, but to check back later or changes.
  • INDEX, NOFOLLOW Instructs web crawlers to index the site once, but to not check back later for changes.
  • NOINDEX, NOFOLLOW Instructs web crawlers to avoid indexing the site, and to not check back later for changes.

Now In order to set Custom Instructions.

For e.g:

Allows Full Access

Disallows Access to All Folders

Default Instructions

When complete, click Save Config.

Related Read:  Magento 2 - How to get Logo Height, Width, Image URL and Alt Text

If you have any query related to configure Robots.txt file then please write in comments.

Enjoy Magento 2!