How to grab data that is not in html source but visible from browser?

Question

The data I want is visible from the browser, but I can't find it from the html source code. I suspect the data was generated by scripts. I'd like to grad such kind of data. Is it possible using Jsoup? I'm aware Jsoup just does not execute Javascript.

Take this page for example, I'd like to grab all the colleges and schools under Academics -> COLLEGES & SCHOOLS.

Answer 1

If the dom content is generated via scripts or plugins, then you really should consider a scriptable browser like phantomjs. Then you can just write some javascript to extract the data.

I didn't check your link, and I assume you're looking for a general answer not specific to any page.