Pycurl sometimes take up to 80%~100% CPU, causing congestion

Question

I'm using pycurl in a test application to stress my application via REST. I notice if I just send 1000 messages per second, pycurl would start taking up alot of CPU and sometimes return an error message.

pycurl.error: (7, 'Failed to connect to 127.0.0.1: Cannot assign requested address') Command exited with non-zero status 1

I looked online abit and noticed other people have noticed high CPU utilization as well. I'm wondering is there anything I can do, or configure? Maybe there's a way to modify how pycurl binds to the socket?

Thanks Morgan

Answer 1

You are probably leaking socket file descriptors. Make sure you are closing all connections and files you read/write from/to. You can get a reference of pycurl error codes at:

http://curl.haxx.se/libcurl/c/libcurl-errors.html

Also you might try to set CURLOPT_VERBOSE option, to get more debug information.