Useful IMAP header information not showing up

Question

I'm using standard IMAP functions to get emails in PHP. I need to keep track of the Message-ID (and References and In-Reply-To) for each message to build threads. Once this system gets deployed, I want users to be able to reply to messages in their mail client to add a message to a thread, but in my web interface, replies can't be attached to the original post because I don't have an In-Reply-To ID from any email headers. I would use In-Reply-To ID to attach their reply to the notification that I sent to them of a new post, but I can't because I can't get the IDs. If I had an In-Reply-To header, messages could be grouped, as there is a thread of IDs connecting them.

Also, I want to be able to reply to any message that the user sends to the server email address if there was an error with their submission or anything like that, in which case I'll need the Message-ID of their message so that I can make that the In-Reply-To ID of the message that I send in reply.

I can't get values for Message-ID, nor In-Reply-To even though they actually have values. (Go to any message in Gmail, click the dropdown menu next to the reply button, hit "show original", and you will see that all of the header information that is missing for me when I call the IMAP functions is actually there, as though Gmail doesn't want to give out that information to IMAP servers.)

I've tried a variety of calls that supposedly get the header information (imap_headerinfo, imap_fetchheader, imap_fetch_overview), but they all return the array of sparse values for the header info that I don't need. After executing

$this->mbox = imap_open('{imap.gmail.com:993/imap/ssl}', $email, $password);

I can get only a few useful values out of the header with any of these calls:

imap_header($this->getImapStream(), "1");
imap_fetch_overview($this->getImapStream(), "1");
imap_fetchheader($this->getImapStream(), "1");

I can see stuff like Subject, To, Reply-To, and other header values, and they are all accurate, but Message-ID, References, In-Reply-To, and other valuable header information is largely absent (rather, the array values that represent them are empty in every case).

I can almost guarantee that the problem isn't with the code itself, as I can see some values in the headers, and I can successfully retrieve the bodies of messages; it's simply as though Google is filtering what header information I can see.

_{You may notice that I have the exact same question as on this page, but it was suggested that I make my own question, so that's what I've done.}

---

EDIT

If somebody can give me a way to reply to messages from my server without needing to retrieve the Message-ID of the message to which I'm replying, that'd be equally awesome.

---

EDIT 2

I listed the Message-ID from every message in my very large inbox and, from what I can see, about 40% of the IDs actually show up. If I run the same test again, the same messages retain their Message-IDs. I am searching for a pattern among the messages that show the Message-ID but there is very strong evidence for a non-pattern among the types of messages.

Every message, in reality, has a Message-ID (and I have verified this for even the messages whose IDs don't show up with IMAP queries), but IMAP functions only reveal the ID of some of the messages.

Answer 1

Turns out that the headers weren't in fact missing. When I was print_ring these calls:

imap_header($this->getImapStream(), "1");
imap_fetch_overview($this->getImapStream(), "1");
imap_fetchheader($this->getImapStream(), "1");

I neglected to generate escape characters on them with htmlspecialchars, which matters because Message-IDs look a little something like <lotsOfArbitraryNumbersAndLetters@domain.com> or something close to it, and my browser thought "Hey, that's an odd tag, but I'll treat it like one!"

Answer 2

Likely, your function that gets the headers for the message is sending a "limiting list" when it requests them, or is using the IMAP Protocol "ENVELOPE", so it doesn't have to fetch useless headers like Received and the like. I guarantee that they are there to be fetched, but I suspect the PHP functions are not fetching them for some reason. GMail just passes through the MIME message as it got it.

I don't think threading information is included in the ENVELOPE.

Are you able to do a protocol trace? If so, I can help you diagnose what's going on.