I'm just guessing here, but it seems that your algorithm is attempting to apply stereo panning laws to the left/right channels. In that case, I don't think that the calculation is being done correctly. This link (also posted in a comment) might be useful:
If my suspicions are correct, then you are not applying the correct amount of gain compensation to the signal, which is why it's probably not sounding as you expect.