# Robots.txt # For more information on this file see: # http://www.robotstxt.org/ # Define the sitemap path Sitemap: http://beauty.exchange/sitemap.xml # Make changes for all web spiders User-agent: * Allow: / Disallow: /global/ Disallow: /assets/ User-agent: MJ12bot Disallow: / User-agent: facebookexternalhit Disallow: / User-agent: DotBot Disallow: / User-agent: hrankbot Disallow: / User-agent: panscient.com Disallow: /