URL unit testing in Java

Question

I'm writing tests for some data fetcher, which gets information from some URL and parses it with JSOUP.

One of the methods:

public Map<String, String> getDepartments() {
    Map<String, String> result = new HashMap<>();
    Document doc = null;

    try {
        doc = Jsoup.connect(globalScheduleURL).get();
    } catch (IOException e) {
        e.printStackTrace();
    }

    Elements links = doc.select("a[href]");
...
}

where globalScheduleURL is a String, usually formatted as http://universitysite.edu/.... For testing purposes I made a mock copy of needed pages and saved it under ./src/test/resources. How can I access to local files, so the address starts with file:/?

When I tried to do something like this:

@Before
public void initializeDataFetcher() throws MalformedURLException {
    df = new SSUDataFetcher();

    File file = new File("./src/test/resources/departments.html");
    URL fileURL = file.toURI().toURL();

    System.out.println(fileURL);
    df.setGlobalURL(fileURL.toString());


}

I get:

file:.../src/test/resources/departments.html java.net.MalformedURLException: Only http & https protocols supported

Is there any workaround to avoid this exception in JSoup or some URL formatting in Java? `

Answer 1

I would prefer to factor the URL fetching out into an own class/method (because reading from an URL is no mission critical code) - the interesting part is the parsing of the HTML and this can be done easily by a method accepting the HTML-String and returning the parsed result without the actual URL reading and you easily can mock the method by providing it with a static String.

Answer 2

doc = Jsoup.connect(globalScheduleURL).get(); 

is not testable because connect() is a static method. To test this, you would need to first extract this line of code into it's own method

Document getDocumentHelper(String globalScheduleURL) throws IOException {
        return Jsoup.connect(globalScheduleURL).get();
}

and then stub this method out to return a mock Document, using a mocking framework like Mockito, EasyMock, PowerMock, or Spock and use it in your original test. For example, in Spock:

Document doc = mock(Document) 
classUnderTest.getDocumentHelper(globalScheduleURL) >> doc

Or Mockito:

Document doc = mock(Document.class); 
when(classUnderTest.getDocumentHelper(globalScheduleURL)).thenReturn(doc); 

Answer 3

The real problem is that the program has not been written with testability in mind. Create an abstraction for the part that obtains the data, so that you can inject a mock implementation in the test.