Removing directory of files slows down ENTIRE webserver

Question

I have a directory in /var/www/media with subdirectories containing a day's worth of many many images. Every night one of of these subdirectories is deleted using rm -rf /var/www/media/1234. Server is running Nginx/PHP on Ubuntu 12.04 with ext4 filesystem with journaling.

Problem: When doing the deletes, the entire web server slows to a crawl. iotop shows 99% IO utilization by jbd2/md3-8 and md4_raid1. I believe jdb2 is working with ext4 journaling on the filesystem /dev/md4 which hosts /var/www/media that we're deleting files off.

Its unacceptable to have the delete operations grind the server to a halt. What are my options here?

Do we have to set ext4 journaling to data=writeback to speed things up and avoid thrashing my server?

Answer 1

Check the man page for ionice, which is intended for tuning this sort of thing. It's fiddly though, as performance work always is. The fact that pure disk I/O is killing your server says immediately that your service is disk-bound, and that's not a happy place to be (storage is a lot harder to scale than CPU!). It may be that the best option is to add RAM to your boxes, but obviously we can't say without details.