############################### # # # User-agent: * # list folders robots are not allowed to index # Disallow: /images/ Disallow: /css/ Disallow: /javascript/ User-agent: SemrushBot Disallow: / User-agent: MJ12bot Disallow: / User-agent: AhrefsBot Disallow: / User-agent: YandexBot Disallow: / User-agent: Googlebot Allow: /css/ Allow: /javascript/ User-Agent: bingbot Allow: /css/ Allow: /javascript/ # # # End of robots.txt file # # ###############################