I have a business web app that needs to pull in information from various other web sites. For most sites, the user just instructs the server to pull the data (either using .NET's HttpRequest, or Selenium).
But for some unfriendly, Javascript-heavy sites, our users have to visit the site manually, navigate to the right spot, and copy and paste into our application.
Other than bookmarklets, is there any way for our page to show an IFRAME with the source web site loaded, allow the user to navigate within the frame, and then capture the IFRAME's body?
Since the site in the IFRAME isn't in the same domain (not even close), I can't seem to work around browser cross-site scripting limitations. I've tried using HTML5's "sandbox" feature, but it appears to only allow communication (via "allow-same-origin") the other way, from the IFRAME to the host site, which isn't useful to me. Also, it doesn't work if the site in question attempts to load its frames to the top context.
What I'm ideally looking for is a solution that would allow the browser to be configured to trust my web site implicitly (it's an intranet app) and allow it to access any frame's contents. That would at least get me in the ballpark. Bonus points if I can get the iframe to redefine the "top" context as its own frame, so the hosted site functions properly within the frame.