I solved it by modifying :resize_to_fill
carrierwave method as described in their code here.
I just made a new method with the same code with a check to see if the uploaded image is smaller. Here is the new method:
def resize_to_fill_modfied(width, height, gravity=::Magick::CenterGravity)
manipulate! do |img|
img.crop_resized!(width, height, gravity) unless (img.columns <= width && img.rows <= height)
img = yield(img) if block_given?
img
end
end
Does exactly what I want now.