With concrete classes and dynamic data like this, you won't be able to lean on the standard XmlSerializer to serialize / deserialize for you - as it reflects on your classes, and the properties you want to populate simply don't exist. You could provide a class with all possible properties if your set of 'OtherAttributes' is known and finite, and not subject to future change, but that would give you an ugly bloated class (and I think you've already decided this is not the solution).
Practical options therefore:
Do it manually. Use the XmlDocument class, load your data with .Load(), and iterate the nodes using .SelectNodes() using an XPath query (something like "/Detail/OtherAttributes/*"). You will have to write the lot yourself, but this gives you complete control over the serialization / deserialization. You won't have to cover your code in (arguably superfluous!) attributes either.
Use Json.NET (http://james.newtonking.com/json), it allows for far greater control over serialization and deserialization. It's fast, has good docs and is overall pretty nifty really.