procmailrc rule for “In-Reply-To header matches previously seen Message-ID header”

Question

Trying to figure out how to have a .procmailrc rule that tosses all mail that is in reply to previously seen mail. Using maildir, and would like the rule to cover messages in either cur/ or new/...

Should I have the procmailrc parse all the files? Or should I have a rule that extracts message id headers, dump those into a file, and parse that?

I can't just check for and toss anything with the in-reply-to header, as if it's a reply to something that this address hasn't seen yet, it needs to be accepted.

Answer 1

You need to collect a cache of Message-Id:s from incoming messages:

:0c:
| formail -zxMessage-Id: >>msgid.txt

Then check the In-Reply-To: against this cache (probably earlier in your .procmailrc so you can filter spam before adding a Message-Id to the cache);

:0
* ? formail -zxIn-Reply-To: | fgrep -f msgid.txt -
./whitelisted

See also the examples of formail -D for comparison. Perhaps you could massage things into a form where you could actually use formail -D as the back end (replace Message-Id: with From: before adding to the cache; split the In-Reply-To: and perform a similar substitution on each message-id). formail can maintain a constant-size LRU cache, whereas the fgrep file will just keep on growing indefinitely (though in this case, perhaps that's precisely what you want).

When you initially set this up, you probably want to collect a cache from messages you have in cur and new already, but after that, you should not need to refer to those messages again from your recipe. (If your inbox is anything like mine, real-time grepping of the whole inbox would be quite infeasible.)

If you want a big cache, replacing a plain-text file with SQLite or something would probably be worth looking into. Actually, by the time a proper database is really worth the effort, you might want a real database rather than SQLite, but I only have vague ideas about the scalability of different database engines for this sort of thing, and no practical experience.

Note that -- unless you have an unusual set-up -- your outgoing Message-Id:s will not be added to the cache; for this to be actually useful, perhaps you should arrange for that to happen somehow, and/or establish a pattern which matches your outgoing Message-Id:s (ideally with no false positives, and certainly no false negatives. If you have switched MUAs in the past, you should perhaps look for patterns for old messages of yours separately, if you want full coverage).

Caution: untested, and I'm probably quite rusty.