{"id":1223,"date":"2016-12-27T20:04:50","date_gmt":"2016-12-27T14:34:50","guid":{"rendered":"http:\/\/www.magewallet.com\/?p=1223"},"modified":"2020-03-13T16:04:15","modified_gmt":"2020-03-13T10:34:15","slug":"how-to-configure-robots-txt-in-magento-2","status":"publish","type":"post","link":"https:\/\/www.hiddentechies.com\/blog\/magento-2\/how-to-configure-robots-txt-in-magento-2\/","title":{"rendered":"How to Configure Robots.txt in Magento 2?"},"content":{"rendered":"<p>As you know, configuring robot.txt is important to any online store that is working on a site\u2019s SEO.<\/p>\n<p>You can configure the option by following below path in Magento 2.x admin.<\/p>\n<p>Go to Stores -&gt; Configuration -&gt; General (Left side) -&gt; Design (section) -&gt; Search Engine Robots -&gt; Default Robots (Select)-&gt; Select<\/p>\n<ul>\n<li>INDEX, FOLLOW Instructs web crawlers to index the site and to check back later\u00a0for changes.<\/li>\n<li>NOINDEX, FOLLOW Instructs web crawlers to avoid indexing the site, but to check\u00a0back later or changes.<\/li>\n<li><span style=\"line-height: 1.5;\">INDEX, NOFOLLOW Instructs web crawlers to index the site once, but to not check\u00a0<\/span>back later for changes.<\/li>\n<li>NOINDEX, NOFOLLOW Instructs web crawlers to avoid indexing the site, and to not\u00a0check back later for changes.<\/li>\n<\/ul>\n<p>Now In order to set Custom Instructions.<\/p>\n<p>For e.g:<\/p>\n<p><strong>Allows Full Access<\/strong><\/p>\n<pre class=\"lang:default decode:true \">User-agent:*\r\nDisallow:<\/pre>\n<p><strong>Disallows Access to All Folders<\/strong><\/p>\n<pre class=\"lang:default decode:true \">User-agent:*\r\nDisallow: \/<\/pre>\n<p><strong>Default Instructions<\/strong><\/p>\n<pre class=\"lang:default decode:true \">Disallow: \/lib\/\r\nDisallow: \/*.php$\r\nDisallow: \/pkginfo\/\r\nDisallow: \/report\/\r\nDisallow: \/var\/\r\nDisallow: \/catalog\/\r\nDisallow: \/customer\/\r\nDisallow: \/sendfriend\/\r\nDisallow: \/review\/\r\nDisallow: \/*SID=<\/pre>\n<p>When complete, click Save Config.<\/p>\n<p>If you have any query related to configure Robots.txt file then please write in comments.<\/p>\n<p>Enjoy Magento 2!<\/p>\n","protected":false},"excerpt":{"rendered":"<p>As you know, configuring robot.txt is important to any online store that is working on a site\u2019s SEO. You can configure the option by following below path in Magento 2.x admin. Go to Stores -&gt; Configuration -&gt; General (Left side) -&gt; Design (section) -&gt; Search Engine Robots -&gt; Default Robots (Select)-&gt; Select INDEX, FOLLOW Instructs&#8230; <\/p>\n<div class=\"actions\"><a href=\"https:\/\/www.hiddentechies.com\/blog\/magento-2\/how-to-configure-robots-txt-in-magento-2\/\">Continue Reading<\/a><\/div>\n","protected":false},"author":1,"featured_media":4738,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[59],"tags":[9,10,32,45,169,122],"_links":{"self":[{"href":"https:\/\/www.hiddentechies.com\/blog\/wp-json\/wp\/v2\/posts\/1223"}],"collection":[{"href":"https:\/\/www.hiddentechies.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.hiddentechies.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.hiddentechies.com\/blog\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/www.hiddentechies.com\/blog\/wp-json\/wp\/v2\/comments?post=1223"}],"version-history":[{"count":2,"href":"https:\/\/www.hiddentechies.com\/blog\/wp-json\/wp\/v2\/posts\/1223\/revisions"}],"predecessor-version":[{"id":10533,"href":"https:\/\/www.hiddentechies.com\/blog\/wp-json\/wp\/v2\/posts\/1223\/revisions\/10533"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.hiddentechies.com\/blog\/wp-json\/wp\/v2\/media\/4738"}],"wp:attachment":[{"href":"https:\/\/www.hiddentechies.com\/blog\/wp-json\/wp\/v2\/media?parent=1223"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.hiddentechies.com\/blog\/wp-json\/wp\/v2\/categories?post=1223"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.hiddentechies.com\/blog\/wp-json\/wp\/v2\/tags?post=1223"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}