Try changing the "viewHolder = new ViewHolder();" inside the getview of Adapter if the convertview is null;
Like :
public View getView(final int position, View convertView, ViewGroup parent) {
String catID = categoryList.get(position).catID;
if (convertView == null)
{
viewHolder = new ViewHolder(); //Add this line. Otherwise u are sharing the same instance for other views.
convertView = inflater.inflate(R.layout.category_list_item, parent, false);
viewHolder.categoryTitle = (TextView) convertView.findViewById(R.id.category_title);
viewHolder.gallery = (Gallery) convertView.findViewById(R.id.gallery);
convertView.setTag(viewHolder);
} else
viewHolder = (ViewHolder) convertView.getTag();
assetTable = new AssetsTable(context);
int count = assetTable.getCountOfParticularCategory(catID);
initializeGallery(catID);
String text = categoryList.get(position).catName + " (" + count + ") ";
viewHolder.categoryTitle.setText(text);
return convertView;
}