Closing asyncio event loop in Python causes exception at end

Question

For https requests using asyncio and aiohttp in Python 3.4 on Windows I'll need to use 2 event loops. A ProactorEventLoop for running shell commands, and the default event loop for HTTPS requests. The ProactorEventLoop does not work for HTTPS commands, unfortunately.

The following code below shows what happens when I use a newly created default event loop and try to close it at the end on Windows. I get exceptions at the end if I call loop.close at the end as shown below:

> Traceback (most recent call last):
>  File "C:\BuildUtilities\p3.4env0\lib\site-packages\aiohttp\connector.py", line 56, in __del__
>    self.close()
>  File "C:\BuildUtilities\p3.4env0\lib\site-packages\aiohttp\connector.py", line 97, in close
>    transport.close()
>  File "C:\Python34\Lib\asyncio\selector_events.py", line 375, in close
>    self._loop.remove_reader(self._sock_fd)
>  File "C:\Python34\Lib\asyncio\selector_events.py", line 155, in remove_reader
>    key = self._selector.get_key(fd)
> AttributeError: 'NoneType' object has no attribute 'get_key'

Commenting it out removes the exception and I don't know why. The one and only

import asyncio
import aiohttp

@asyncio.coroutine
def get_body(url):
    response = yield from aiohttp.request('GET', url)
    return (yield from response.read_and_close())

#loop = asyncio.ProactorEventLoop()
loop = asyncio.new_event_loop()
asyncio.set_event_loop(loop)

f = asyncio.async( get_body('https://www.google.com') )
try:
    loop.run_until_complete(f)
except Exception as e:
    print(e)

if f.result():
    print(f.result())

loop.close()

Thanks, greenaj

Answer 1

Update: it looks like the issue is fixed in the github version (0.7.2). It doesn't produce the error. As @danj.py said, it is fixed by "Get rid of __del__ in connector" commit.

---

It is not ProactorEventLoop or Windows specific. I can reproduce the error on Ubuntu with the default event loop:

#!/usr/bin/env python3
import asyncio
import aiohttp # $ pip install aiohttp

@asyncio.coroutine
def get_body(url):
    response = yield from aiohttp.request('GET', url)
    return (yield from response.read_and_close())

loop = asyncio.get_event_loop()
body = loop.run_until_complete(get_body('https://stackoverflow.com/q/23283502'))
print(len(body), type(body), body[:200])
loop.close()

It might be a bug in aiohttp because the usage seems correct.

There is no error if the request is made without aiohttp:

#!/usr/bin/env python3
import asyncio
from contextlib import closing
from urllib.parse import urlsplit

@asyncio.coroutine
def get_body(url):
    # parse url
    url = urlsplit(url)
    path = '/' * (not url.path) + url.path + '?' * bool(url.query) + url.query
    # open connection
    reader, writer = yield from asyncio.open_connection(
        host=url.hostname,
        port=url.port or (443 if url.scheme == 'https' else 80),
        ssl=(url.scheme == 'https'))
    with closing(writer):
        # send request
        writer.write(b'GET ' + path.encode('ascii') + b' HTTP/1.1\r\n'
                     b'Host: ' + url.netloc.encode('ascii') + b'\r\n'
                     b'Connection: close\r\n\r\n')
        # read headers
        while True:
            line = yield from reader.readline()
            line = line.rstrip(b'\n\r')
            print(line.decode('latin-1'))
            if not line:
                break
        # read body
        body = yield from reader.read()
    return body

loop = asyncio.get_event_loop()
body = loop.run_until_complete(get_body('https://stackoverflow.com/q/23283502'))
print(len(body), type(body), body[:200])
loop.close()

Note: the examples are not completely equivalent e.g., the latter doesn't follow redirects.

Answer 2

J.F. Sebastian is correct that this is a bug with aiohttp.

It looks like this issue has been resolved via the following commit.

https://github.com/KeepSafe/aiohttp/commit/a229110539e93c21ae0198f6f883ae3e40ea866b#diff-7f25afde79f309e2f8722c26cf1f10ad

As of this writing, it does not appear that this has made its way into pypi.

Answer 3

Actually it's a (fixed) bug in Python 3.4 itself: http://bugs.python.org/issue21435

As temporary solution (and for sake of Python 3.3 support ) I made pull request https://github.com/KeepSafe/aiohttp/pull/48

But, say again, the root of problem is garbage collector in Python 3.4.0 (fixed by upcoming Python 3.4.1 bugfix release).