regular expression to extract text from HTML

Question

I would like to extract from a general HTML page, all the text (displayed or not).

I would like to remove

• any HTML tags
• Any javascript
• Any CSS styles

Is there a regular expression (one or more) that will achieve that?

Answer 1

You can't really parse HTML with regular expressions. It's too complex. RE's won't handle <![CDATA[ sections correctly at all. Further, some kinds of common HTML things like <text> will work in a browser as proper text, but might baffle a naive RE.

You'll be happier and more successful with a proper HTML parser. Python folks often use something Beautiful Soup to parse HTML and strip out tags and scripts.

---

Also, browsers, by design, tolerate malformed HTML. So you will often find yourself trying to parse HTML which is clearly improper, but happens to work okay in a browser.

You might be able to parse bad HTML with RE's. All it requires is patience and hard work. But it's often simpler to use someone else's parser.

Answer 2

Remove javascript and CSS:

<(script|style).*?</\1>

Remove tags

<.*?>

Answer 3

Needed a regex solution (in php) that would return the plain text just as well (or better than) PHPSimpleDOM, only much faster. Here is the solution that I came up with:

function plaintext($html)
{
    // remove comments and any content found in the the comment area (strip_tags only removes the actual tags).
    $plaintext = preg_replace('#<!--.*?-->#s', '', $html);

    // put a space between list items (strip_tags just removes the tags).
    $plaintext = preg_replace('#</li>#', ' </li>', $plaintext);

    // remove all script and style tags
    $plaintext = preg_replace('#<(script|style)\b[^>]*>(.*?)</(script|style)>#is', "", $plaintext);

    // remove br tags (missed by strip_tags)
    $plaintext = preg_replace("#<br[^>]*?>#", " ", $plaintext);

    // remove all remaining html
    $plaintext = strip_tags($plaintext);

    return $plaintext;
}

When I tested this on some complicated sites (forums seem to contain some of the tougher html to parse), this method returned the same result as PHPSimpleDOM plaintext, only much, much faster. It also handled the list items (li tags) properly, where PHPSimpleDOM did not.

As for the speed:

• SimpleDom: 0.03248 sec.
• RegEx: 0.00087 sec.

37 times faster!

Answer 4

Contemplating doing this with regular expressions is daunting. Have you considered XSLT? The XPath expression to extract all of the text nodes in an XHTML document, minus script & style content, would be:

//body//text()[not(ancestor::script)][not(ancestor::style)]

Answer 5

Using perl syntax for defining the regexes, a start might be:

!<body.*?>(.*)</body>!smi

Then applying the following replace to the result of that group:

!<script.*?</script>!!smi
!<[^>]+/[ \t]*>!!smi
!</?([a-z]+).*?>!!smi
/<!--.*?-->//smi

This of course won't format things nicely as a text file, but it strip out all the HTML (mostly, there's a few cases where it might not work quite right). A better idea though is to use an XML parser in whatever language you are using to parse the HTML properly and extract the text out of that.

Answer 6

The simplest way for simple HTML (example in Python):

text = "<p>This is my> <strong>example</strong>HTML,<br /> containing tags</p>"
import re
" ".join([t.strip() for t in re.findall(r"<[^>]+>|[^<]+",text) if not '<' in t])

Returns this:

'This is my> example HTML, containing tags'

Answer 7

Here's a function to remove even most complex html tags.

function strip_html_tags( $text ) 
{

$text = preg_replace(
    array(
        // Remove invisible content
        '@<head[^>]*?>.*?</head>@siu',
        '@<style[^>]*?>.*?</style>@siu',
        '@<script[^>]*?.*?</script>@siu',
        '@<object[^>]*?.*?</object>@siu',
        '@<embed[^>]*?.*?</embed>@siu',
        '@<applet[^>]*?.*?</applet>@siu',
        '@<noframes[^>]*?.*?</noframes>@siu',
        '@<noscript[^>]*?.*?</noscript>@siu',
        '@<noembed[^>]*?.*?</noembed>@siu',

        // Add line breaks before & after blocks
        '@<((br)|(hr))@iu',
        '@</?((address)|(blockquote)|(center)|(del))@iu',
        '@</?((div)|(h[1-9])|(ins)|(isindex)|(p)|(pre))@iu',
        '@</?((dir)|(dl)|(dt)|(dd)|(li)|(menu)|(ol)|(ul))@iu',
        '@</?((table)|(th)|(td)|(caption))@iu',
        '@</?((form)|(button)|(fieldset)|(legend)|(input))@iu',
        '@</?((label)|(select)|(optgroup)|(option)|(textarea))@iu',
        '@</?((frameset)|(frame)|(iframe))@iu',
    ),
    array(
        ' ', ' ', ' ', ' ', ' ', ' ', ' ', ' ', ' ',
        "\n\$0", "\n\$0", "\n\$0", "\n\$0", "\n\$0", "\n\$0",
        "\n\$0", "\n\$0",
    ),
    $text );

// Remove all remaining tags and comments and return.
return strip_tags( $text );
    }

Answer 8

If you're using PHP, try Simple HTML DOM, available at SourceForge.

Otherwise, Google html2text, and you'll find a variety of implementations for different languages that basically use a series of regular expressions to suck out all the markup. Be careful here, because tags without endings can sometimes be left in, as well as special characters such as & (which is &).

Also, watch out for comments and Javascript, as I've found it's particularly annoying to deal with for regular expressions, and why I generally just prefer to let a free parser do all the work for me.

Answer 9

I believe you can just do

document.body.innerText

Which will return the content of all text nodes in the document, visible or not.

[edit (olliej): sigh nevermind, this only works in Safari and IE, and i can't be bothered downloading a firefox nightly to see if it exists in trunk :-/ ]

Answer 10

Can't you just use the WebBrowser control available with C# ?

        System.Windows.Forms.WebBrowser wc = new System.Windows.Forms.WebBrowser();
        wc.DocumentText = "<html><body>blah blah<b>foo</b></body></html>";
        System.Windows.Forms.HtmlDocument h = wc.Document;
        Console.WriteLine(h.Body.InnerText);

Answer 11

string decode = System.Web.HttpUtility.HtmlDecode(your_htmlfile.html);
                Regex objRegExp = new Regex("<(.|\n)+?>");
                string replace = objRegExp.Replace(g, "");
                replace = replace.Replace(k, string.Empty);
                replace.Trim("\t\r\n ".ToCharArray());

then take a label and do "label.text=replace;" see on label out put

.