Thank you @omer681, you pointed me in the right direction.
My fix involved changing the Hole values from enums to ints in an array. Then, the in recursion, an 'itter' int was passed where the for loop would start. In recursion, this number was inherited from the for loop. This way, each step in a for loop would limit the number of steps in the loops below. This works for any number of balls and score holes.
private static void scoreCalc(int targetScore, int score, int ballNumber,
int itter)
{
for(int i = itter; i < HOLEVALUES.length; i++)
{
...
if (ballNumber > 0) {
scoreCalc(targetScore, score, ballNumber, i);
...
}
}
private static void scoreCalc(int targetScore)
{
scoreCalc(targetScore, 0, BALLCOUNT, 0);
}