If the ScrollView is the root of your view, that may happen because some view inside the ScrollView is gaining the focus.
Try to add this attribute android:focusableInTouchMode="true"
to another view located at the top of the ScrollView and tell if it worked ;)!