Best practice for formatting scraped HTML

Question

I have inherited some code which uses a YQL query to grab all the HTML from an external site. The data is then filtered and then outputted to a container. The YQL query is not returning XML but rather HTML, I've tried changing the format to JSON without luck. Maybe I'm messing up the syntax.

Here is the YQL query:

$.getJSON("http://query.yahooapis.com/v1/public/yql?" +
                "q=select%20*%20from%20html%20where%20url%3D%22" +
                encodeURIComponent(url) +
                "%22&format=JSON&callback=?",

this data is then passed to this filter function (this is not the entire piece of code)

function filterData(data) {

        data = data.replace(/<?\/body[^>]*>/g, '');
        data = data.replace(/[\r|\n]+/g, '');
        data = data.replace(/<--[\S\s]*?-->/g, '');
        data = data.replace(/<noscript[^>]*>[\S\s]*?<\/noscript>/g, '');
        data = data.replace(/<script[^>]*>[\S\s]*?<\/script>/g, '');
        data = data.replace(/<p>5\) \{ return; }.* \} \}; <\/p>/, '');
        data = data.replace(/<body[^>]*>/g, '');
        data = data.replace(/<hr[^>]*>/g, '');
        data = data.replace(/<img[^>]*>/g, '');
        data = data.replace(/<table[^>]*>/g, '<table>');

        return data;
}

I feel like there must be a better way to either put this returned HTML data into some sort of template or something, I'm pretty unfamiliar with jQuery, any help greatly appreciated!

Answer 1

You want to grab them as JSON for easier processing in jQuery. You just need to make sure uppercase JSON is changed to lowercase json to get JSON output from the API.

$.getJSON("http://query.yahooapis.com/v1/public/yql?" +
                "q=select%20*%20from%20html%20where%20url%3D%22" +
                encodeURIComponent(url) +
                "%22&format=json&callback=?",