Order of operations, just like you learned in basic math. You're calculating
(hourlyWage * hoursWorked) - overtimeHours
Instead, you want
hourlyWage * (hoursWorked - overtimeHours)
Additionally, you shouldn't recalculate in your total
; instead, just add regular and overtime.
Finally, as @CodeMonkey notes, you're unconditionally assuming that the person is working overtime. Here's what I suggest instead:
if(hoursWorked > 40) {
regularHours = 40;
overtimeHours = regularHours - 40;
} else {
regularHours = hoursWorked;
overtimeHours = 0;
}