Robots.txt syntax not understood [closed]

Question

Closed. This question is off-topic. It is not currently accepting answers.

---

Want to improve this question? Update the question so it's on-topic for Stack Overflow.

Closed 8 years ago.

Improve this question

I submitted my robots.txt file ages ago to Google and it is still giving me a syntax not understood for the first line.

After Googling the most common problem is Google adding a '?' at the start of the line but it isnt doing that to me.

the url to the robots.txt is

www.leisurepursuits.co.uk/robots.txt

The error is:

Line 1: User-agent: * Syntax not understood

Answer 1

This error shows because the expected robots.txt file format is plain text encoded in UTF-8. The file consists of records (lines) separated by CR, CR/LF or LF.

If a character encoding is not a subset of UTF-8, then your robots.txt files is being parsed incorrectly.

First check your robots.txt url in http://www.asymptoticdesign.co.uk/cgi-bin/check-url.pl by selecting the second option “view source” and see that it responds successfully or not.

Now upload your robots.txt file again. It will not show error as shown previously.