How many loop devices can I create (and mount) on a modern Linux (2.6.30+ kernel) system?

Question

I would like to implement per-directory quotas for a multi-user web application we're developing. The problem is... we have implemented a document manager for any client to store their own private documents but we don't want to run out of space due to this feature, so we want to assign them a given limit.

As there doesn't seem to exist a standard method for implementing per-directory quotas in Linux (I know, quotas are primarily targeted against users or groups, but we need something like the way Windows Server 2008 R2 handles quotas on a per-directory basis) I opted to use a 'trick'. I basically do this:

touch client1.ext3
dd if=/dev/zero of=./client1.ext3 bs=1024 count=16384
mkfs.ext3 ./client1.ext3
mount -o loop,rw ./client1.ext3 ./mountpoint

It's just a code sample, but that's the idea... I create virtual 'volumes' which I assign to my clients so they can store their private data and, in case they need more, they can pay on a per-storage-ammount basis.

The 'problem' I see with this is I just see 8 loop devices in my /dev hierarchy, and we currently have 17 test clients for our application, so the ammount of currently present loop devices don't cover my needs. I know you could allocate up to 256 loop devices up to kernel version 2.6.23, and the limit (from version 2.6.24 onwards) isn't theoretically present anymore, although I still have some concerns.

Honestly, I feel like filling the /dev hierarchy with 1000+ loop devices (which are not gonna be unmounted at all during all the system lifetime) is very wrong and not the way it should be done, but maybe it's doable as a middle-term solution, so my questions are:

• How much memory does a single loop device take on allocation?
• Do the system crash or have a performance hit having 256+ allocated loop devices?
• Can I increase in a dynamic way the number of loop devices? or...
• How can I pre-define the number of available loop devices at boot time?

Answer 1

The idea you described is in fact "logical volume management" (LVM) done by hand. If you use LVM for this you get the double bonus of "it's a well known standard" and "there is good tool support for that including online resizing and more".

Answer 2

Track storage quotas in your application, not in the operating system. Creating tons of loopback filesystems like this will waste a significant amount of storage, perform badly, and scale worse.

Answer 3

LVM adds you statically splited hard disk space, and on it are created filesystems. if it is ext4 or xfs, you add free pv space into lv and resize on the fly device assigned to user or group. unfortunately if you want decrease size (shrink), you must do this offline. at first you must decrease unmounted filesystem size, then you must decrease lv size. but this is risky because if you decrease lv below filesystem size, then fs will be damaged. xfs has no shrink functionality, you are able only increase it.

another approach is more advanced filesystems. this is btrfs fully supported on linux, or zfs implemented as linux kernel module, not as fuse. using those filesystems you are able create logical sub volumes, and on the fly increase/decrease maximum available space. free space is common to all volumes. in those filesystem is impossible assign lover space into subvolume that data on it, then fs damage is impossible on this way. unfortunately zfs as module you must compile independently, because there are no official distributed linux kernel with supports this fs. But you can check btrfs, its functionality is very close to zfs now, and it is officially supported by kernel.

btw. loop is based on block devices with major number 7 (see /dev/loop* special block files), and there is available 64 mount points. maybe this count is higher, but I've never achieved it. some recipe how to configure more loops is here: http://www.tldp.org/HOWTO/CDServer-HOWTO/addloops.html . Buth my friend told me about some trick, you can manually add more loop devices and it will be usable on the fly by mknod /dev/loop8 b 7 8 , mknod /dev/loop9 b 7 9 etc. of course without changing modules.conf file or similar, and creating on temporary udev filesystem, all additional loops will be lost.