You could also set android:clickable="false" to the TextView/ImageView layouts and android:clickable="true" on the background, that way the background view always catches the clicks.
Additionally, the following answer might be helpful for people considering this issue:
SO: Android: How to propagate click event to LinearLayout childs and change their drawable