This is a bug - i think if we remove object from canvas we should remove activeObject as well. I will open a issue on github.
But for now you can do this:
canvas.discardActiveObject();
canvas.remove(activeObject);
Here is the issue on github: https://github.com/kangax/fabric.js/issues/962