Java: I have a big string of html and need to extract the href=“…” text

Question

I have this string containing a large chunk of html and am trying to extract the link from href="..." portion of the string. The href could be in one of the following forms:

<a href="..." />
<a class="..." href="..." />

I don't really have a problem with regex but for some reason when I use the following code:

        String innerHTML = getHTML(); 
  Pattern p = Pattern.compile("href=\"(.*)\"", Pattern.DOTALL);
  Matcher m = p.matcher(innerHTML);
  if (m.find()) {
   // Get all groups for this match
   for (int i=0; i<=m.groupCount(); i++) {
    String groupStr = m.group(i);
    System.out.println(groupStr);

   }
  }

Can someone tell me what is wrong with my code? I did this stuff in php but in Java I am somehow doing something wrong... What is happening is that it prints the whole html string whenever I try to print it...

EDIT: Just so that everyone knows what kind of a string I am dealing with:

<a class="Wrap" href="item.php?id=43241"><input type="button">
    <span class="chevron"></span>
  </a>
  <div class="menu"></div>

Everytime I run the code, it prints the whole string... That's the problem...

And about using jTidy... I'm on it but it would be interesting to know what went wrong in this case as well...

Answer 1

.* 

This is an greedy operation that will take any character including the quotes.

Try something like:

"href=\"([^\"]*)\""

Answer 2

There are two problems with the code you've posted:

Firstly the .* in your regular expression is greedy. This will cause it to match all characters until the last " character that can be found. You can make this match be non-greedy by changing this to .*?.

Secondly, to pick up all the matches, you need to keep iterating with Matcher.find rather than looking for groups. Groups give you access to each parenthesized section of the regex. You however, are looking for each time the whole regular expression matches.

Putting these together gives you the following code which should do what you need:

Pattern p = Pattern.compile("href=\"(.*?)\"", Pattern.DOTALL);
Matcher m = p.matcher(innerHTML);

while (m.find()) 
{
    System.out.println(m.group(1));
}

Answer 3

Regex is great but not the right tool for this particular purpose. Normally you want to use a stackbased parser for this. Have a look at Java HTML parser API's like jTidy.

Answer 4

Use a built in parser. Something like:

    EditorKit kit = new HTMLEditorKit();
    HTMLDocument doc = (HTMLDocument)kit.createDefaultDocument();
    doc.putProperty("IgnoreCharsetDirective", Boolean.TRUE);
    kit.read(reader, doc, 0);

    HTMLDocument.Iterator it = doc.getIterator(HTML.Tag.A);

    while (it.isValid())
    {
        SimpleAttributeSet s = (SimpleAttributeSet)it.getAttributes();
        String href = (String)s.getAttribute(HTML.Attribute.HREF);
        System.out.println( href );
        it.next();
    }

Or use the ParserCallback:

import java.io.*;
import java.net.*;
import javax.swing.text.*;
import javax.swing.text.html.parser.*;
import javax.swing.text.html.*;

public class ParserCallbackText extends HTMLEditorKit.ParserCallback
{
    public void handleStartTag(HTML.Tag tag, MutableAttributeSet a, int pos)
    {
        if (tag.equals(HTML.Tag.A))
        {
            String href = (String)a.getAttribute(HTML.Attribute.HREF);
            System.out.println(href);
        }
    }

    public static void main(String[] args)
        throws Exception
    {
        Reader reader = getReader(args[0]);
        ParserCallbackText parser = new ParserCallbackText();
        new ParserDelegator().parse(reader, parser, true);
    }

    static Reader getReader(String uri)
        throws IOException
    {
        // Retrieve from Internet.
        if (uri.startsWith("http:"))
        {
            URLConnection conn = new URL(uri).openConnection();
            return new InputStreamReader(conn.getInputStream());
        }
        // Retrieve from file.
        else
        {
            return new FileReader(uri);
        }
    }
}

The Reader could be a StringReader.

Answer 5

Another easy and reliable way to do it is by using Jsoup

Document doc = Jsoup.connect("http://example.com/").get();
Elements links = doc.select("a[href]");
for (Element link : links){
  System.out.println(link.attr("abs:href"));
}

Answer 6

you may use a html parser library. jtidy for example gives you a DOM model of the html, from wich you can extract all "a" elements and read their "href" attribute

Answer 7

"href=\"(.*?)\"" should also work, but I think Kugel's answer will work faster.