Buffer Size in C++

Question

I am observing the following behavior with the C++ Std library method std::ostream::write().

For buffering the data I am making use of the following C++ API

std::ofstream::rdbuf()->pubsetbuf(char* s, streamsize n)

This works fine ( verified using the strace utility ) as long as the size of data (datasize) we are writing on the file stream using

std::ofstream::write (const char* s, datasize n)

Is less than 1023 bytes ( below this value the writes are accumulated till the buffer is not full), but when the size of data to write exceeds 1023, the buffer is not taken into account and the data is flushed to the file.

e.g. If I set the buffer size to 10KB and write around 512bytes a time, strace will show that multiple writes have been combined into a single write

writev(3, [{"AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA"..., 9728}, {"AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA"..., 512}], 2) = 10240 ( 10 KB )
writev(3, [{"AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA"..., 9728}, {"AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA"..., 512}], 2) = 10240
...

but when I write 1024 bytes a time ( keeping the buffer fixed to 10 KB), now strace shows me that it is not using the buffer and each ofstream::write call is being translated to write system call.

writev(3, [{NULL, 0}, {"AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA"..., 1024}], 2) = 1024 ( 1KB )
writev(3, [{NULL, 0}, {"AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA"..., 1024}], 2) = 1024
...

Is there any C++ API Call or Linux Tuning Parameter which I am missing?

Answer 1

This is an implementation detail of libstdc++, implemented around line 650 of bits/fstream.tcc. Basically, if the write is larger than 2^10, it will skip the buffer.

If you want the rationale behind this decision, I suggest you send a mail to the libstdc++ development list.

http://gcc.gnu.org/ml/libstdc++/

Answer 2

Looks like someone writing the stdlib implementation made an "optimization" without giving enough thought to it. So, the only workaround for you would be to avoid the C++ API and use the standard C library.

This is not the only suboptimality in the GNU/Linux implementation of the standard C++ library: on my machine, malloc() is 100 cycles faster than the standard void* operator new (size_t size)...