Issue is that , your have set the scale down duration to scaleUp and not scaleDown.
change it as below..
Animation scaleDown = new ScaleAnimation(2, 1, 2, 1, Animation.RELATIVE_TO_SELF, 0.5f, Animation.RELATIVE_TO_SELF, 0.5f);
scaleDown.setDuration(2000);
imageView.setAnimation(scaleDown);