Trace/BPT trap when calling urllib.urlopen

Question

For some reason I'm getting a Trace/BPT trap error when calling urllib.urlopen. I've tried both urllib and urllib2 with identical results. Here is the code which throws the error:

def get_url(url):
    from urllib2 import urlopen
    if not url or not url.startswith('http://'): return None
    return urlopen(url).read() # FIXME!

I should add that this code is running on a CherryPy webserver with web.py.

Someone requested a traceback. Unfortunately, there is none. Trace/BPT trap is outputted to the terminal and the process terminates. E.g.

dloewenherz@andros project $ sudo ./index.py 80
http://0.0.0.0:80/
# Here I visit the page which contains the get_url(url) method
Trace/BPT trap
dloewenherz@andros project $

Edit: I am running OS X 10.6.2, web.py 0.33, Python 2.6.2, and CherryPy 3.1.2.

Answer 1

Adding the following lines to the top of the main file solved the problem:

import urllib2
urllib2.install_opener(urllib2.build_opener())

In other words, it is not enough to import the urllib2 module but you actually need to create the opener in the main thread.

Answer 2

Are you running this under OS X 10.6? Apparently threads and importing modules for the first time does not play well together there. See if you can't import urllib2 outside of the thread?

There are a few more details in the following thread: Trace/BPT trap with Python threading module

I'd try either moving the import of urllib to the top of the same file or, since it seems to be a problem only with importing a module for the first time in a thread, import it somewhere else as well, like in the same file as your main() function.

Edit: Which versions of OS X, Python, CherryPy and web.py are you running? I'm using OS X 10.5.8, Python 2.6, CherryPy 3.1.2 and web.py 0.33 and can't reproduce your problem using the below code:

import web

urls = (
  '/', 'index'
)

app = web.application(urls, globals())

class index:
    def GET(self):
        from urllib2 import urlopen
        return urlopen("http://google.se/").read()

if __name__ == "__main__": app.run()


$ sudo python index.py 80
http://0.0.0.0:80/
127.0.0.1:59601 - - [08/Nov/2009 09:46:40] "HTTP/1.1 GET /" - 200 OK
127.0.0.1:59604 - - [08/Nov/2009 09:46:40] "HTTP/1.1 GET /extern_js/f/CgJzdhICc2UgACswCjhBQB0sKzAOOAksKzAYOAQsKzAlOMmIASwrMCY4BSwrMCc4Aiw/dDWkSd2jmF8.js" - 404 Not Found
127.0.0.1:59601 - - [08/Nov/2009 09:46:40] "HTTP/1.1 GET /logos/elmo-hp.gif" - 404 Not Found
127.0.0.1:59601 - - [08/Nov/2009 09:46:40] "HTTP/1.1 GET /images/nav_logo7.png" - 404 Not Found

Is this code enough to reproduce the problem on your end? If not, I need more information in order to be of help.