When to use the Filesystem API? Should I use it at all?

Question

I'm wondering when it is recommended to use the Filesystem API, and whether it's useful at all?

WordPress seems a little inconsistent when it comes to Filesystem API usage, it uses the API only in a few places when uploading and unpacking files, and that doesn't make much sense to me, I mean, where's to point in using it at all when there are a lot of other situations where the PHP filesystem functions are used for direct access? Uploading packages would work even when there are ownership "problems", however I'm pretty sure that these "problems" won't go away 5 minutes later when I'm utilizing any of the functions that are using for example fwrite directly.

So why should I as a plugin/theme developer care about it anyways when half of the WordPress functions I'm using does not rely on the Filesystem API, my plugin/theme wouldn't work correctly anyways, wouldn't it?