Assuming WinForms, when a control gets a MouseDown event from the user pressing a mouse button, it "captures" the mouse input from that moment until the mouse button is released. This is used to provide the MouseUp event even if the mouse moved away from the control.
You would have to override that but there is a side-effect:
Private Sub Panel1_MouseDown(sender As Object, e As MouseEventArgs) _
Handles Panel1.MouseDown
Panel1.Capture = False
End Sub
Now you have no guarantee the MouseUp event will fire unless the mouse is directly over the control.