Your Adapter class does this internally to reduce memory consumption and to efficiently use device resources.
In your Adapter class you should do
public View getView (int position, View convertView, ViewGroup parent){
// convertView is reused from previously scrolled out view to reduce memory consumption
if( convertView == null ){
//We must create a View:
convertView = inflater.inflate(R.layout.my_list_item, parent, false);
}
// Set border to the image if it is selected otherwise set it to default
// keep in mind when you add any condition here, else should also be addressed
return convertView;
}