Python - OSError 24 (Too many open files) and shared memory

Question

I faced with the problem there was exception OSError 24 ("Too many open files") raised on my mac os x in python script.

I had no idea what could caused that issue. lsof -p showed about 40-50 lines, and my ulimit was 1200 (I check that using

resource.getrlimit(resource.RLIMIT_NOFILE)

), that returned tuple (1200, 1200). So I didn't exceed limit even closely.

That my script spawned number of subprocesses and also allocated shared memory segments. Exception occurred while allocating shared memory segments. I use sysv_ipc module.

Also I knew I total allowed number of shared memory segments is enough large (128 segments), and command

ipcs -b -m

gave definitely less number (not more then 40 segments).

Answer 1

The issue was in shared memory system settings (shared memory – wiki).

There is parameter kern.sysv.shmseg in /etc/sysctl.conf file which represents the maximum number of shared memory segments each process can attach. So I had value 32 that was not enough for my script.

To view parameters, use:

sysctl -A | grep shm

To update that parameters, edit file:

sudo vim /etc/sysctl.conf

My looks now like that:

kern.sysv.shmmax=564777216
kern.sysv.shmmin=1
kern.sysv.shmmni=700
kern.sysv.shmseg=128
kern.sysv.shmall=131072

Notice, you need restart system in order to apply settings.

To view currently allocated shared memory segments, type:

ipcs -m -b

To remove all shared memory segments:

for n in `ipcs -b -m | egrep ^m | awk '{ print $2; }'`; do ipcrm -m $n; done

Notice, only segments that are not attached to any process will be really removed.

More on shared memory settings: http://techjournal.318.com/general-technology/shared-memory-settings-explain/, http://www.spy-hill.com/help/apple/SharedMemory.html, http://support.apple.com/kb/HT4022