As you concluded it is used a lot in onTouchEvent. OnTouchEvent is called whenever you interact with the screen, meaning touch move and release events. Moving just one pixel would result in a potential call to this method.
There is not much more to explain, it is just the way it is implemented. Usually adapter.getCount is implemented with something like List.getSize or Cursor.getCount. And has almost zero overhead. If this is a problem, optimize you ListAdapter.getCount method, cache the count or something like that. Only do complex stuff in there when needed and cache the result until it becomes invalid.