Wild guess: You want
canvas2.on("mousedown", handleMouseDown);
…
canvas2.off("mousedown", handleMouseDown);
With the unnecessary anonymous function you did not have a reference to the actual passed listener, so you could not remove it any more.
when the function has been used the function needs to be disengaged
You might also consider using .one("mousedown", handleMouseDown)
, then.