how to get size of folder including apparent size of sparse files? (du is too slow)

Question

I have a folder containing a lot of KVM qcow2 files, they are all sparse files. Now I need to get the total size of folder, the qcow2 file size should be counted as apparent size(not real size).

for example:

image: c9f38caf104b4d338cc1bbdd640dca89.qcow2 file format: qcow2 virtual size: 100G (107374182400 bytes) disk size: 3.3M cluster_size: 65536

the image should be treated as 100G but not 3.3M

originally I use statvfs() but it can only return real size of the folder. then I switch to 'du --apparent-size', but it's too slow given I have 10000+ files and it takes almost 5 minutes to caculate.

anybody knows a fast way that can get the size of folder counting qcow2's virtual size? thank you

Answer 1

There is no way to find out this information without stat()ing every file in the directory. It is slow if you have this many files in a single directory. stat() needs to retrieve the inode of every single file.

Adding more memory might help due to caching.

Answer 2

You could use something like this:

find images/ -name "*.qcow2" -exec qemu-img info {} \; | grep virtual | cut -d"(" -f2 | awk '{ SUM += $1} END { print SUM }'

Answer 3

Modern Unix*ish OSes provide a way to retrieve the stats of all entries of a directory in one step. This also needs to look at all inodes but probably it can be done optimized in the file system driver itself and thus might be faster.

Apparently you are not looking for a way to do this using system calls from C, so I guess a feasible approach could be to use Python. There you have access to this feature using the function scandir() in module os.