Your image button is probably taking focus when you click on list item.
So Add this
android:descendantFocusability="blocksDescendants"
to your root element
android:descendantFocusability
Defines the relationship between the ViewGroup and its descendants when looking for a View to take focus.
Must be one of the following constant values.
Constant Value Description
beforeDescendants 0 The ViewGroup will get focus before any of its descendants.
afterDescendants 1 The ViewGroup will get focus only if none of its descendants want it.
blocksDescendants 2 The ViewGroup will block its descendants from receiving focus.
This corresponds to the global attribute resource symbol descendantFocusability.