Online tool for crawling a website and retriving all meta information for every page
-
12-09-2019 - |
Question
Does anyone know of a free online tool that can crawl any given website and return just the Meta Keywords and Meta Description information?
Solution
Assuming you have access to Linux/Unix:
mkdir temp
cd temp
wget -r SITE_ADDRESS
Then, for keywords:
egrep -r -h 'meta[^>]+name="keywords' * | sed 's/^.*content="\([^"]*\)".*$/\1/g'
and for descriptions:
egrep -r -h 'meta[^>]+name="description' * | sed 's/^.*content="\([^"]*\)".*$/\1/g'
If you want all the unique keywords, try:
egrep -r -h 'meta[^>]+name="keywords' * | sed 's/^.*content="\([^"]*\)".*$/\1/g' | sed 's/\s*,\s*/\n/g' | sort | uniq
I'm sure there's a one-liner or program out there that does this exact thing, and there are definitely easier answers.
OTHER TIPS
To retrive all meta information try this tool Meta Tags Analyzer
Licensed under: CC-BY-SA with attribution
Not affiliated with StackOverflow