Appearance

Robots.txt ​

The robots.txt file sits in the root directory of your website and tells crawlers which URLs they may fetch. Shopify automatically generates a default robots.txt file that is suitable for standard e-commerce setups. Customize this file only if you need specific crawl rules, such as blocking crawlers from fetching internal search pages or filtering certain URL search parameters.

robots.txt guides crawling and fetching behavior. It does not guarantee how search engines index or rank your pages.

Before you begin ​

  • SEOEngine is installed and open in your Shopify admin.
  • You understand which crawl rules your store needs beyond Shopify’s default robots.txt.

Generate and edit custom crawl rules ​

If your store requires custom rules beyond Shopify's default setup, generate an editable version of the file in Robots.txt.

  1. From SEOEngine, go to Robots.txt.
  2. Read the configuration advisory banner at the top of the page.
  3. Select Generate robots.txt to create a custom, editable copy of the file.
  4. Edit your rules directly in the Robots.txt content text box.
  5. Select Save to publish your rules.
  6. Select View robots.txt to open and review the live file on your storefront.

Robots.txt editor

After you save, the custom file is published for your storefront. Use View robots.txt to confirm the live content matches what you saved.

Restore Shopify default rules ​

If you no longer need custom crawl configurations or need to reset your rules, you can return to Shopify's system-managed file.

  1. Open Robots.txt in the app.
  2. Select Restore Shopify default.
  3. Confirm the action when prompted.

WARNING

Restoring the default settings completely deletes your custom rules from your active theme, and this action cannot be undone.