Finally found the fix. Basically I only need to pass factory=None instead of factory=MaildirMessage.
Reading the documentation in the link below it seems like factory=None actually picks MaildirMessage factory but apparently it also adds extra behavior.