Web crawlers and GET vs POST requests

Question

I have heard that web crawlers are supposed to follow only GET requests and not POST ones.

In the real world is this a valid assumption?

Answer 1

Generally they do not do POST requests. This is just the current state of affairs and is not dictated anywhere, I believe. Some search engines are experimenting with crawling forms, but these are still GET requests.

Answer 2

Are you asking if crawlers will parse out URLs from form tags and follow them? Some will. It's a pretty safe assumption they will always use the GET verb though.

Spam bots are a different story.

Answer 3

Polite crawlers only use GET requests. Just like polite crawlers obey the bots.txt.

However, even google is playing fast and loose with the definition of polite, their crawler has done serious bandwidth damage to some sites including stackoverflow.