python imaplib - error 'unexpected repsonse'

Question

Hi I've suddenly encountered an error using imaplib on some code that worked fine before.

import imaplib
m = imaplib.IMAP4('myserver','port')
m.login(r'username','password')
m.select()

gives me the error

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/lib/python2.7/imaplib.py", line 649, in select
    typ, dat = self._simple_command(name, mailbox)
  File "/usr/lib/python2.7/imaplib.py", line 1070, in _simple_command
    return self._command_complete(name, self._command(name, *args))
  File "/usr/lib/python2.7/imaplib.py", line 899, in _command_complete
    raise self.abort('command: %s => %s' % (name, val))
imaplib.abort: command: SELECT => unexpected response: '*  1520 EXISTS'

I'm not sure what it means. Emails are otherwise coming through fine, and I'm using davmail as a server.

The program in its entirety saves attachments with a certain name in a specific folder.

I've stepped through it and its definitely the m.select() that is where its falling over.

This same program worked absolutely fine until recently.

What am I doing wrong, and how do I fix it?

The log of activity is as follows

>>> import imaplib
>>> m = imaplib.IMAP4('server','port')
>>> Debug=4
>>> m.debug
0
>>> m.debug=4
>>> m.debug
4
>>> m.login(r'username','password')
  01:26.55 > HLFI1 LOGIN "username" "password"
  01:30.76 < HLFI1 OK Authenticated
('OK', ['Authenticated'])
>>> m.list()
  01:56.33 > HLFI2 LIST "" *
  02:00.04 < * LIST (\HasNoChildren) "/" "Trash/Sent Messages"
  02:00.04 < * LIST (\HasNoChildren) "/" "Sync Issues/Server Failures"
  02:00.04 < * LIST (\HasNoChildren) "/" "Sync Issues/Local Failures"
  02:00.04 < * LIST (\HasNoChildren) "/" "Sync Issues/Conflicts"
  02:00.04 < * LIST (\HasChildren) "/" "Sync Issues"
  02:00.04 < * LIST (\HasNoChildren) "/" "Junk E-mail"
  02:00.04 < * LIST (\HasNoChildren) "/" "Drafts"
  02:00.04 < * LIST (\HasChildren) "/" "Trash"
  02:00.04 < * LIST (\HasNoChildren) "/" "Sent"
  02:00.04 < * LIST (\HasNoChildren) "/" "Outbox"
  02:00.04 < * LIST (\HasNoChildren) "/" "INBOX"
  02:00.04 < HLFI2 OK LIST completed
('OK', ['(\\HasNoChildren) "/" "Trash/Sent Messages"', '(\\HasNoChildren) "/" "Sync Issues/Server Failures"', '(\\HasNoChildren) "/" "Sync Issues/Local Failures"', '(\\HasNoChildren) "/" "Sync Issues/Conflicts"', '(\\HasChildren) "/" "Sync Issues"', '(\\HasNoChildren) "/" "Junk E-mail"', '(\\HasNoChildren) "/" "Drafts"', '(\\HasChildren) "/" "Trash"', '(\\HasNoChildren) "/" "Sent"', '(\\HasNoChildren) "/" "Outbox"', '(\\HasNoChildren) "/" "INBOX"'])
>>> m.select()
  02:21.37 > HLFI3 SELECT INBOX
  02:30.87 < *  1548 EXISTS
  02:30.87 last 4 IMAP4 interactions:
  00:16.73 < * OK [CAPABILITY IMAP4REV1 AUTH=LOGIN MOVE] IMAP4rev1 DavMail 4.3.0-2125 server ready
  00:16.73 > HLFI0 CAPABILITY
  00:16.74 < * CAPABILITY IMAP4REV1 AUTH=LOGIN MOVE
  00:16.77 < HLFI0 OK CAPABILITY completed
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/lib/python2.7/imaplib.py", line 649, in select
    typ, dat = self._simple_command(name, mailbox)
  File "/usr/lib/python2.7/imaplib.py", line 1070, in _simple_command
    return self._command_complete(name, self._command(name, *args))
  File "/usr/lib/python2.7/imaplib.py", line 899, in _command_complete
    raise self.abort('command: %s => %s' % (name, val))
imaplib.abort: command: SELECT => unexpected response: '*  1548 EXISTS'

** UPDATE **

I've now filed a bug at python-dev under

Bug report on Python

Where David Murray says the response is non RFC compliant

And a second at davmail sourceforge under

davmail bug report

M guessant says it is necessary for the IMAP keep alive.

I'll keep this updated with developments..

Answer 1

It appears that the space-padded message count in the RECENT response is what triggers this. It is unclear to me whether it should be classified as an error in Python's imaplib or in the IMAP server you are using. I would argue that imaplib should be robust against this, regardless of what the spec says. Perhaps you should file a bug report?

(If you do, please take care to add details about which server is producing this response. If it is a commercial product with a respectable market share, it is important to fix, whereas of course, if it's your own simple Python server, they might not care.)

Answer 2

Found the response of the EXISTS command is padded by davmail, it seems to be when the number of emails is around or over 500.

Acceptable response:

  58:24.54 > PJJD3 EXAMINE INBOX
  58:24.77 < * 486 EXISTS
  58:24.78      matched r'\* (?P<data>\d+) (?P<type>[A-Z-]+)( (?P<data2>.*))?' => ('486', 'EXISTS', None, None)
  58:24.78 untagged_responses[EXISTS] 0 += ["486"]

Failed response:

  57:50.86 > KPFE3 EXAMINE INBOX
  57:51.10 < *  953 EXISTS
  57:51.10 last 0 IMAP4 interactions:
  57:51.10 > KPFE4 LOGOUT

Have create a pull request for the imaplib library at github to account for the issue

To patch your code the imaplib allows the regex to be altered, simply add the following to your code:

imaplib.Untagged_status = imaplib.re.compile(br'\*[ ]{1,2}(?P<data>\d+) (?P<type>[A-Z-]+)( (?P<data2>.*))?')