Can't run over two sites using Nginx + Pyramid + Gunicorn

Question

I use bundle Nginx + Gunicorn + Pserve (Pyramid) on my web server to run several sites. To be exact only two sites now working. If I put it right Nginx listens 80 port and proxying requests to Gunicorn on 5000 and 5010 ports, here is my nginx configs for two domains:

# $ cat cat /etc/nginx/sites-available/domain1.ru
upstream domain1.ru {
    server 0.0.0.0:5000;
    server 0.0.0.0:5001;
}

server {
    listen 80;
    server_name www.domain1.ru domain1.ru;
    error_log  /var/log/nginx/domain1.ru.error.log;
    access_log  /var/log/nginx/domain1.ru.access.log;

    location / {
        try_files $uri @proxy_to_app;
    }

    location @proxy_to_app {
        proxy_set_header        Host $http_host;
        proxy_set_header        X-Real-IP $remote_addr;
        proxy_set_header        X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header        X-Forwarded-Proto $scheme;

        client_max_body_size    10m;
        client_body_buffer_size 128k;
        proxy_connect_timeout   60s;
        proxy_send_timeout      90s;
        proxy_read_timeout      90s;
        proxy_buffering         off;
        proxy_temp_file_write_size 64k;
        proxy_pass      http://127.0.0.1:5000;
        proxy_redirect          off;
    }
}


# $ cat cat /etc/nginx/sites-available/domain2.ru
upstream domain2.ru {
    server 0.0.0.0:5010;
    server 0.0.0.0:5011;
}

server {
    listen 80;
    server_name www.domain2.ru domain2.ru;
    error_log  /var/log/nginx/domain2.ru.error.log;
    access_log  /var/log/nginx/domain2.ru.access.log;

    location / {
        try_files $uri @proxy_to_app;
    }

    location @proxy_to_app {
        proxy_set_header        Host $http_host;
        proxy_set_header        X-Real-IP $remote_addr;
        proxy_set_header        X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header        X-Forwarded-Proto $scheme;

        client_max_body_size    10m;
        client_body_buffer_size 128k;
        proxy_connect_timeout   60s;
        proxy_send_timeout      90s;
        proxy_read_timeout      90s;
        proxy_buffering         off;
        proxy_temp_file_write_size 64k;
        proxy_pass      http://127.0.0.1:5010;
        proxy_redirect          off;
    }
}

And these two sites works well, but when I added just the same config except changing ports to 5015 and 5016 something went wrong. When I just started pserve process I got an error:

$ pserve production.ini --reload --log-file=domain3.ru-pserve.log
Starting subprocess with file monitor
Traceback (most recent call last):
  File "/var/www/eba/data/gs/env/bin/pserve", line 9, in <module>
    load_entry_point('pyramid==1.4.5', 'console_scripts', 'pserve')()
  File "/var/www/eba/data/gs/env/lib/python2.7/site-packages/pkg_resources.py", line 378, in load_entry_point
    return get_distribution(dist).load_entry_point(group, name)
  File "/var/www/eba/data/gs/env/lib/python2.7/site-packages/pkg_resources.py", line 2566, in load_entry_point
    return ep.load()
  File "/var/www/eba/data/gs/env/lib/python2.7/site-packages/pkg_resources.py", line 2260, in load
    entry = __import__(self.module_name, globals(),globals(), ['__name__'])
  File "/var/www/eba/data/gs/env/lib/python2.7/site-packages/pyramid/scripts/pserve.py", line 28, in <module>
    from pyramid.compat import WIN
  File "/var/www/eba/data/gs/env/lib/python2.7/site-packages/pyramid/compat.py", line 114, in <module>
    from urllib2 import urlopen as url_open
  File "/var/www/eba/data/gs/env/opt/Python-2.7.3/lib/python2.7/urllib2.py", line 94, in <module>
    import httplib
  File "/var/www/eba/data/gs/env/opt/Python-2.7.3/lib/python2.7/httplib.py", line 69, in <module>
    from array import array
ImportError: /var/www/eba/data/gs/env/lib/python2.7/lib-dynload/array.so: cannot open shared object file: Too many open files in system

When I run pserve again I got another error, here is just last several lines:

File "/var/www/eba/data/gs/env/opt/Python-2.7.3/lib/python2.7/random.py", line 70, in <module>
    import _random
ImportError: /var/www/eba/data/gs/env/lib/python2.7/lib-dynload/_random.so: cannot open shared object file: Too many open files in system

Next time - another:

File "/var/www/eba/data/gs/env/lib/python2.7/site-packages/paste/deploy/loadwsgi.py", line 3, in <module>
    from __future__ import with_statement
ImportError: No module named __future__

The things is if I switched off the second site and run pserve again on 3rd - it runs okay.

I don't know what restrictions it must be. Maybe because I run all applications using one environment in ~/gs/env folder. The ports 5015 and 5016 are free.

Answer 1

It was the server problem. I just exhausted all available memory. 512Mb is not enough for three php sites (mod-fpm) and three pyramid sites, and mongo. The first error was the key - Too many open files in system.

ulimit -a showed me it was 1024 available files, but increasing that value has had no results. So I changed the tariff plan to 1024Mb and for now it's enough - all working fine and fast.

Answer 2

This sounds like it's an issue with using --reload in a very large folder. Try removing that argument as you shouldn't be using it in a production environment anyway.