Bingbot ignoring robots.txt and attempting to retrieve a trafficbasedsspsitemap.xml [closed]

Question

Closed. This question is off-topic. It is not currently accepting answers.

---

Want to improve this question? Update the question so it's on-topic for Stack Overflow.

Closed 8 years ago.

Improve this question

I have an app whose content should not be publicly indexed. I've therefore disallowed access to all crawlers.

robots.txt:

# Robots shouldn't index a private app.
User-agent: *
Disallow: / 

However, Bing has been ignoring this and daily requests a /trafficbasedsspsitemap.xml file, which I have no need to create.

I also have no need to receive daily 404 error notifications for this file. I'd like to just make the bingbot go away, so what do I need to do to forbid it from making requests?

Answer 1

According to this answer, this is Bingbot checking for an XML sitemap generated by the Bing Sitemap Plugin for IIS and Apache. It apparently cannot be blocked by robots.txt.

Answer 2

For those coming from google-

You could block bots via apache user agent detection/ rewrite directives, that would allow you to keep bingbot out entirely. https://superuser.com/questions/330671/wildcard-blocking-of-bots-in-apache

Block all bots/crawlers/spiders for a special directory with htaccess

etc.