very interesting result from latency test of IPC on Linux 2.6.18

Question

I am doing a performance (latency) test on unix socket under linux 2.6.18,
a process A writes 1024 bytes to process B on each 10 ms, and the result shows average latency is 20 us with small standard deviation(2~3 us).

The test becomes interesting when I run some additional CPU-bound processes simultaneously with process A&B, these new process is very cache-friendly such as a busy loop of simple math calculation, but as a result which surprises me, the IPC latency suddenly goes down, become 15 us on average.

As far as I know, to improve interactivity the O(1) scheduler(2.6 prior to 2.6.23) rewards IO-bound process by some heuristic method, but this can't explain why the speed becomes faster even than the first case.

I have also considered that if the Linux do some special case of busy-loop when process A was rewarded, but it seems not by further test.

This really confuses me.

---

my configuration: CPU: Intel(R) Xeon(R) CPU E5-2609 0 @ 2.40GHz with 10M L3 cache MEM: 32G OS: Linux 2.6.18-308.el5 SMP x86_64

Answer 1

I suspect that some power-saving feature of the hardware is at work here. A 10ms sleep is more than enough time for modern hardware to enter a low-power state. When you're looking at things at the microsecond level, there is a real, measurable latency to come out of a power-saving state.

My guess is that running the "busy" program in parallel prevents the hardware from entering a low power state. Standard things to try:

• At the BIOS level, disable any and all power-saving features including C-states
• At the OS level, disable cpuspeed (or whatever frequency scaling program your distro uses)
• Try booting with the "idle=poll" kernel parameter

That last suggestion is especially important for Sandy Bridge CPUs (which is what you have), at least with RHEL/CentOS 5.x (which I'm guessing you're running). I found the Linux kernel would still override some BIOS settings. Other Linux kernel params that may help you:

• intel_idle.max_state=0
• processor.max_cstate=0