What protocol does Google use for Gmail? (not IMAP or POP)

Question 1

After more research, I've found that there exists an API for GMail: https://developers.google.com/gmail/api/ I don't think that API was released when I posted this question back in 2013.

Using that API, I have created a demo program that fetches the 100 last mails of a label: https://gist.github.com/bjourne/37a9d15b03862003008a1b0169190dbe

The relevant part of the program is:

resource = service.users().messages()
result = resource.list(userId = 'me', labelIds = [label]).execute()
mail_ids = [m['id'] for m in result['messages']]

start = time()
mails = []
batch = BatchHttpRequest()
cb = lambda req, res, exc: mails.append(to_mail(res))
for mail_id in mail_ids:
    get_request = resource.get(**headers_params(mail_id))
    batch.add(get_request, callback = cb)
result = batch.execute()
print('Took %.2f seconds' % (time() - start))

It lists the last 100 messages sorted by date in a label (folder in IMAP terminology) containing over 570k messages.

On my machine, this loop takes about 0.5 - 0.8 seconds. I can claim confidently that no pure IMAP client on the planet comes even close. Likely, IMAP won't ever get faster because it is a poor fit for how Google stores mail internally.

So I'll answer my own question. This is the API they are using and it wasn't exposed earlier.

Question 2

The Android app (at least the ones I've used) use IMAP. You can verify this by running Wireshark on the server.

As to why the Android app is so fast - what I know is that it uses the SEARCH command to select the most recent n messages. Desktop clients such as Thunderbird or Outlook are much more heavy-weight and download headers and metadata for every message in the folder, despite recommendations for them not to.

A smartphone does not have the resources to store and process millions of emails (although more modern ones might be getting there) so the SEARCH approach allowed quick mail access for handheld devices.

Anyhow, Wireshark can reveal a great deal about the behaviour of IMAP clients and servers. If you're really curious, give it a shot. You can't do this if the server is Gmail, but you can try it out on another server (e.g. hMailServer).

Question 3

You can test gmail's IMAP performance easily (if you have a million-message mailbox). Open an IMAP connection with

openssl s_client -connect imap.gmail.com:993 -crlf

then login and open your inbox.

a login yourname@gmail.com yourpassword
b select inbox

Or open your allmail box if inbox isn't big enough (name may vary depending on UI language):

c select "[Gmail]/All Mail"

If SELECT is fast but an IMAP client slow, then that's because the client sends additional/unneeded slow commands. Many choose to fill or update a data structure for the entire million messages even if they're going to display only 40 messages. That's client choice, not IMAP slowness.

Question 4

"No other IMAP client can do this" is a pretty bold statement, but a million of messages is a pretty big number as well. I would encourage you to give Trojitá a try here. Chances are that the initial synchronization will be rather slow (it would transfer the flags for that million of messages for various technical reasons related to how the IMAP flags, SELECT, SEARCH and STATUS are specified), but the subsequent resynchronizaiton should be lightning fast thanks to ESEARCH, CONDSTORE and QRESYNC. I would be interested to hear how well Trojitá works with your setup -- the contact information are on the homepage.

To your question -- most webmails nowadays provide a private API for their own use. A typical architecture is to transfer messages about updated state via JSON, but there is no standard for this and the interface is prioprietary. Chances are that the GMail "app" uses the same (or similar) method. You don't have much options to verify this as it is likely using TLS. With a web interface, it is trivial to see the traffic with an appropriate browser plugin, but not so much with a standalone Android application.