Yes, system takes care for scaling, but scaling at runtime can affect your app's performance, especially when the scaling factor is not power of 2.
Normally scaling down a image wouldn't cause quality problems. however, it may produce a minor impact on image quality under some circumstances, here's a test:
image in xxhdpi folder, left side is how it looks like on a Nexus 5 (xxhdpi), right side is how it looks like on a Nexus 4 (xhdpi).
If image size is less than 4px, app may crash when scaling down.
Conclusion: if app size is more important than problems above, then you can put all images in xxhdpi folder, except the small ones.