You could take a look here to create a robots.txt.
googlebot crawling webservice
-
11-12-2021 - |
Pergunta
is there some way to make googlebot not to crawl links that is used for webservice? When I try to access my webservice on url it returns 500 error when there no parameters passed. Or is there also some way to prevent an http 500 error when I try to access my webservice in url without parameters?
THanks.
Solução
Licenciado em: CC-BY-SA com atribuição
Não afiliado a StackOverflow