This is caused by the way RoundedBitmapDisplayer draws the bitmap.
If you look at the source, you'll see that it uses a RoundedDrawable, which uses canvas.drawRoundRect() to draw a rounded rectangle of the desired size of the Drawable, using the downloaded image as the texture via a BitmapShader. BitmapShader does not support scaling (only clamping and tile modes). Try using a SimpleBitmapDisplayer instead which uses the normal ImageView.setImageBitmap() way of displaying the image.
If you need rounded corners, you'll have to find a different way to implement that, for example by scaling the Bitmap to the desired size first. Another option is to call Canvas.saveLayer() before delegating to BitmapDrawable for the scaling, and then applying the rounded corner masking effect using PorterDuff.Mode.DST_IN. Either way you'll end up writing a bit more low-level code, but you should be able to encapsulate everything nicely in a custom BitmapDisplayer.