So after debugging and googling I've found that Xamarin Garbage Collection Article. Conclusion is that bitmaps that UIL loaded (with or without cache) couldn't be released because there was a relationship between the managed Bitmap (Mono/.net) and framework Bitmap (Java) instances. To break that relationship I call bitmap.Dispose() in OnLoadingComplete in my implementation of IImageLoadingListener.
public void OnLoadingComplete(string p0, View p1, Bitmap p2)
{
var spinner = ((View)p1.Parent).FindViewById<ProgressBar>(Resource.Id.loading);
spinner.Visibility = ViewStates.Gone;
p2.Dispose();
}