Small file not committed to disk for over a minute

Question

I have an embedded Linux app that writes to a file at a fairly slow rate (50 bytes/s or so). The file is on a hard drive, XFS filesystem. The file is being written to by calling write(), not fwrite().

If I power-cycle the system and check the file, over a minute's worth of data is missing. I thought the default Linux behaviour was to sync the disk cache every 5s (I can tolerate 5s worth of missing data so they'd be no problem with this). What should I be checking to see why it isn't getting synced for a long time? /proc/sys/vm/dirty_writeback_centisecs is 500. Are there other changeable things I should be checking? It definitely looks to be a disk cache issue - if i ls -l the file, the size is as expected, and after the power cycle it is less than before.

Answer 1

Mmm, I'm not sure about the 5s as "default in Linux". As far as I know the delay is filesystem dependant, although maybe I'm mistaken. I believe the maximum delay for cache sync on XFS is (or at least was a while ago) 30 seconds. That figure of 5 seconds was true for some older filesystems (ext2 and ext3 I believe) and it doesn't hold true for ext4, if I recall correctly.

In conclusion, what is happening to you is the expected behaviour.

If you want to surpass the cache, I believe you will need to use fsync, O_SYNC or O_DIRECT. Or apply the 'sync' option when mounting the filesystem, which should solve the problem globally.

Answer 2

The five seconds behavior is only for ext3/ext4 filesystems, you can control this by commit mount option (default is commit=5). I don't know how XFS behaves tho.

You perhaps mean 30 seconds limit in the VM subsystem. By default, Linux can wait up to 30 seconds with writing your data. You can get/set the limit using:

# cat /proc/sys/vm/dirty_expire_centiseconds
3000

Shows hondreds of second.

Answer 3

Well, XFS is known for such "failures". Problem is connected with write-cache. More info on XFS FAQ.