Limit recursion for off host pages when using wget

Question

I'm trying to wget a page along with a little bit of the pages it links to off site. What I'd like to do is recurse up to a depth of 5 on the site (host), and then as soon as I jump to another host limit the recursion to say 2 pages. I can't seem to find any such option in the man page, am I just out of luck?

Answer 1

Sadly, yes. You're out of luck if you're trying to do this with just one command. Wget does not support different recursion depths for the parent host and other domains. However, you can use the following set of commands to effectively get what you're looking for:

$ wget -r -H -l 2 www.example.com
$ wget -r -l 5 -c www.example.com

The first command tries to recursively download everything from the domain you want, including from other hosts upto a recursion depth of 2. Beyond that, it simply ignores all the files.

In the second command, we ask Wget to download only from the host server, but this time upto a recursion depth of 5. The -c is used to ask Wget to continue the file transfer. This means Wget will not attempt to re-download everything that has already been downloaded through the first command.

The above commands are only a proof of concept and will need to be tailored to meet your specific needs.