# Block bad SEO scrapers and unnecessary bots User-agent: AhrefsBot Disallow: / User-agent: SemrushBot Disallow: / User-agent: MJ12bot Disallow: / User-agent: DotBot Disallow: / User-agent: BLEXBot Disallow: / User-agent: Screaming Frog SEO Spider Disallow: / # Allow major search engines explicitly User-agent: Googlebot Allow: / User-agent: Bingbot Allow: / User-agent: Applebot Allow: / User-agent: YandexBot Allow: / User-agent: Twitterbot Allow: / User-agent: FacebookExternalHit Allow: / # General bot rules User-agent: * Disallow: /wp-admin/ Allow: /wp-admin/admin-ajax.php Allow: / # Sitemap location Sitemap: https://newsrt.us/sitemap_index.xml