Awk match() - multiple per line

Question

I'm using the match() function in gawk to grab links out of an HTML file.. the regular expression is something like this:

match($0, /(<a href=\")([^\"]+)/, arr)

I don't seem to be able to use the "/g" option at the end to get multiple matches per line though?

Answer 1

That is correct. AWK regular expressions don't have flags.
Also, there's no built-in support for having match look for the second or later matches.
Only the gsub and gensub functions have that.
I'd try something like this:

gensub(/.*<a href=\"([^\"]+)/, "\1%", "g")
last = split($0, "%", arr)
delete arr[last]

where % is a string that you can guarantee won't be found in the input.

Answer 2

The text-mode browser lynx might be a better tool for harvesting URLs. It's -dump flag writes formatted output to standard output. At the end, you'll find a numbered list of every visible and hidden link on that page. (Or file. It accepts URLs or filenames as arguments.)

$ lynx -dump http://www.stackoverflow.com 

[snip]
References

   Visible links
   1. http://stackoverflow.com/opensearch.xml
   2. http://stackoverflow.com/feeds
   3. http://stackexchange.com/
   4. http://stackoverflow.com/users/login
   5. http://careers.stackoverflow.com/
   6. http://chat.stackoverflow.com/
[snip]
 676. http://creativecommons.org/licenses/by-sa/3.0/
 677. http://blog.stackoverflow.com/2009/06/attribution-required/

   Hidden links:
 678. http://www.peer1.com/stackoverflow
 679. http://creativecommons.org/licenses/by-sa/3.0/