What is a test oracle and what is it used for?

Question

I don't really understand the concept of a software testing oracle. It says:

An oracle is a mechanism for determining whether the program has passed or failed a test.

Consider the following code:

// class that should be tested
public int sum(int a, int b) {
  return a + b;
}

// test class
static Main tester = new Main();
@Test
public void testSum() {
  assertEquals("2 + 3 is 5", 5, tester.sum(2, 3));
}

The class that should be tested always returns the sum of 2 integers.
I pass as a parameter 2 and 3, and expect 5. 2 and 3 will be summed up and compared to the expected value (5). In this case the test succeeds.

How exactly can an oracle help me here? Is an oracle involved in this example?

Answer 1

A test oracle is a source of information about whether the output of a program (or function or method) is correct or not.

A test oracle might specify correct output for all possible input or only for specific input. It might not specify actual output values but only constraints on them.

The oracle might be

• a program (separate from the system under test) which takes the same input and produces the same output
• documentation that gives specific correct outputs for specific given inputs
• a documented algorithm that a human could use to calculate correct outputs for given inputs
• a human domain expert who can somehow look at the output and tell whether it is correct
• or any other way of telling that output is correct.

If not vague, the concept is at least very broad.

An oracle isn't a test runner, but a test runner could use an oracle as a source of correct output to which to compare the system-under-test's output, or as a source of constraints against which to evaluate the SUT's output.

In your example, you used your personal ability to carry out the algorithm of addition as your oracle. Instead, you could use a different implementation of that algorithm as an oracle:

assertEquals("2 + 3 is 5", 2 + 3, tester.sum(2, 3));

Answer 2

Let me pose the oracle question this way: how can we check that the program returns the right answer?

For this function, we can easily check the answer with the following pseudocode (Sorry, it's not C++.):

repeat many times {
    int a = randomNumber();
    int b = randomNumber();
    int result = sum(a, b);
    assertEquals("random case", a, result - b);
}

This oracle uses subtraction to check the function. This allows millions or billions of tests to be run with little human effort.