Try setting bitmap options for the imageview as follows.
DisplayImageOptions options = new DisplayImageOptions.Builder()
.showImageOnLoading(R.drawable.ic_stub)
.showImageForEmptyUri(R.drawable.ic_empty)
.showImageOnFail(R.drawable.ic_error)
.cacheInMemory(true)
.cacheOnDisc(true)
.considerExifParams(true)
.build();
and now try diplay to imageview as
imageLoader.displayImage(imageUrls[position], holder.image, options);
I am sure this will work.
See the code cacheInMemory(true) property for display options is important.