If serialization of an existing type gets complicated, the best option is always: create a separate DTO model - that looks kinda similar to your domain entity, but which only exists to play nicely with serialization - usually very simple (parameterless constructors, basic accessors, no validation, etc). Then map between them. Otherwise, you'll be playing a game of whack-a-mole with configuring a serializer for a type it doesn't really like.
If you want a game of whack-a-mole with XmlSerializer: you can create an XmlAttributeOverrides instance, configure it by hand for your specific type (adding the attribute instances), and pass it into the XmlSerializer constructor. But this is ugly, quite fiddly, and you must must must cache and re-use the serializer instance (the normal automatic assembly cache/re-use doesn't apply if you use that overload of the constructor). You can obtain (from the XmlAttributeOverrides instance) an XmlAttributes instance per-type or per-member, and then the XmlIgnore property to true as necessary. Frankly, I advise against this approach.