I think whats probably happening is when you are setting the layout params you are overwriting the previous one which said center in parent. You should try this:
RelativeLayout.LayoutParams params = new RelativeLayout.LayoutParams(iconWidth, iconWidth);
params.addRule(RelativeLayout.LayoutParams.CENTER_IN_PARENT);
viewHolder.ivIcon.setLayoutParams(params);