This is a known issue with Silverlight, i have suffered with this for a while. I think it's to do with the mouse down / mouse up events firing out of sequence.
I did some research and located a work item which is supposed to fix this problem. (At least for some people)
I have not had time to implement it myself yet - so cannot say if it works or not.
But here it is, worth a try!