aio on osx: Is it implemented in the kernel or with user threads? Other Options?

Question

I am working on my small c++ framework and have a file class which should also support async reading and writing. The only solution other than using synchronous file i/o inside some worker threads I found is aio. Anyways I was looking around and read somewhere, that in Linux, aio is not even implemented in the kernel but rather with user threads. Is the same true for OSX? Another concern is aio's functionality of callbacks which has to spawn an extra thread for each callback since you can't assign a certain thread or threadpool to take care of that (signals are not an option for me). So here are the questions resulting from that:

• Is aio implemented in the Kernel of osx and thus is most likely better than my own threaded implementation?

• Can the callback system -spawning a thread for each callback- become a bottleneck in practice?

• If aio is not worth using on osx, are there any other alternatives on unix? in cocoa? in carbon?

• Or should I simply emulate async i/o with my own threadpool?

What is your experience on the subject?

Answer 1

You can see exactly how AIO is implemented on OSX right here.

The implementation uses kernel threads, one queue of jobs which each thread pops and execute in a blocking fashion in a priority queue based on each request's priority (at least that's what it looks like at a first glance).

You can configure the number of threads and the size of the queue with sysctl. To see these options and the default values, run sysctl -a | grep aio

kern.aiomax = 90
kern.aioprocmax = 16
kern.aiothreads = 4

In my experience, in order for it to make any sense to use AIO, these limits need to be a lot higher.

As for the callbacks in threads, I don't believe Mac OS X supports that. It only does completion notifications through signals (see source).

You could probably do as good of a job in your own thread pool. One thing you could do better than the current darwin implementation is to sort your read jobs by physical location on the disk (see fcntl and F_LOG2PHYS) which might even give you an edge.

Answer 2

@Moka: Sorry to say that you're wrong on the linux implementation, as of kernel 2.6 there is a kernel implementation of AIO, which comes in libaio (libaio.h)

The implementation that doesn't use Kernel threads but instead uses user threads is POSIX.1 AIO, and it does it that way to make it more portable, as not all unix based OS support completion events at Kernel level.