Fetch ajax/javascript content using HTMLunit

Question

I have written a code which will fetch me a html contents of the page as response , I am using HTML Unit to do so . But I am getting error's for some specific urls like

[https://communities.netapp.com/welcome][1]

For first page i am able to retrieve the contents . But when i dont the content which we get using load more button .

Here's my code:

    import java.io.BufferedWriter;
import java.io.File;
import java.io.FileWriter;
import java.io.IOException;
import java.io.Writer;
import java.net.MalformedURLException;

import com.gargoylesoftware.htmlunit.BrowserVersion;
import com.gargoylesoftware.htmlunit.FailingHttpStatusCodeException;
import com.gargoylesoftware.htmlunit.NicelyResynchronizingAjaxController;
import com.gargoylesoftware.htmlunit.WebClient;
import com.gargoylesoftware.htmlunit.html.HtmlPage;


public class Sample {
    public static void main(String[] args) throws FailingHttpStatusCodeException, MalformedURLException, IOException, InterruptedException {
        String url = "https://communities.netapp.com/welcome";
        WebClient client = new WebClient(BrowserVersion.INTERNET_EXPLORER_9);
        client.getOptions().setJavaScriptEnabled(true);
        client.getOptions().setRedirectEnabled(true);
        client.getOptions().setThrowExceptionOnScriptError(true);
        client.getOptions().setCssEnabled(true);
        client.getOptions().setUseInsecureSSL(true);
        client.getOptions().setThrowExceptionOnFailingStatusCode(false);
        client.setAjaxController(new NicelyResynchronizingAjaxController());
        HtmlPage page = client.getPage(url);

        Writer output = null;
        String text = page.asText();
        File file = new File("D://write6.txt");
        output = new BufferedWriter(new FileWriter(file));
        output.write(text);
        output.close();
        System.out.println("Your file has been written");  

       // System.out.println("as Text ==" +page.asText());
      //  System.out.println("asXML == " +page.asXml());
      //  System.out.println("text content ==" +page.getTextContent());
      //  System.out.println(page.getWebResponse().getContentAsString());
    }
}

Any suggestion ?

Answer 1

As i understand from your question you have a button on which you have to press.

Please look at: http://htmlunit.sourceforge.net/gettingStarted.html

You have there an example of submitting a form.

This should be very similar here