How to serialize / deserialize immutable list type in c#

Question

If I have a class defined

[DataContract()]
class MyObject {
    [DataMember()]
    ImmutableList<string> Strings { get; private set}
}

The ImmutableList<T> type comes from the immutables library https://www.nuget.org/packages/Microsoft.Bcl.Immutable. Note that the class ImmutableList does not have a default constructor or a mutable Add method. Adding things to the list take the form.

myList = myList.Add("new string");

Can I add some custom support to the .NET serialization mechanism to support this type and show it how to deserialize it?

Currently the collection is just skipped on deserialization though it is fine to serialize it.

Answer 1

There is another clean way to do this via IDataContractSurrogate interface. The DataContractSerializer allows you to provide a surrogate for non serializable objects. Below is the example and test case for ImmutableList<T>. It uses reflection and probably could be optimized by smarter folks than me but here it is.

TESTCASE

using FluentAssertions;
using System.Collections.Immutable;
using System.IO;
using System.Linq;
using System.Runtime.Serialization;
using System.Text;
using System.Xml;
using Xunit;

namespace ReactiveUI.Ext.Spec
{
    [DataContract(Name="Node", Namespace="http://foo.com/")]
    class Node
    {
        [DataMember()]
        public string Name;
    }

    [DataContract(Name="Fixture", Namespace="http://foo.com/")]
    class FixtureType
    {
        [DataMember()]
        public ImmutableList<Node> Nodes;

        public FixtureType(){
            Nodes = ImmutableList<Node>.Empty.AddRange( new []
            { new Node(){Name="A"}
            , new Node(){Name="B"}
            , new Node(){Name="C"}
            });
        }
    }


    public class ImmutableSurrogateSpec
    {  
        public static string ToXML(object obj)
            {
                var settings = new XmlWriterSettings { Indent = true };

                using (MemoryStream memoryStream = new MemoryStream())
                using (StreamReader reader = new StreamReader(memoryStream))
                using (XmlWriter writer = XmlWriter.Create(memoryStream, settings))
                {
                    DataContractSerializer serializer =
                      new DataContractSerializer
                          ( obj.GetType()
                          , new DataContractSerializerSettings() { DataContractSurrogate = new ImmutableSurrogateSerializer() }
                          );
                    serializer.WriteObject(writer, obj);
                    writer.Flush();
                    memoryStream.Position = 0;
                    return reader.ReadToEnd();
                }
            }

        public static T Load<T>(Stream data)
        {
            DataContractSerializer ser = new DataContractSerializer
                  ( typeof(T)
                  , new DataContractSerializerSettings() { DataContractSurrogate = new ImmutableSurrogateSerializer() }
                  );
            return (T)ser.ReadObject(data);
        }

        public static T Load<T>(string data)
        {
            using (var stream = new MemoryStream(Encoding.UTF8.GetBytes(data)))
            {
                return Load<T>(stream);
            }
        }

        [Fact]
        public void ShouldWork()
        {
            var o = new FixtureType();

            var s = ToXML(o);

            var oo = Load<FixtureType>(s);

            oo.Nodes.Count().Should().Be(3);
            var names = oo.Nodes.Select(n => n.Name).ToList();
            names.ShouldAllBeEquivalentTo(new[]{"A", "B", "C"});

        }

    }
}

IMPLEMENTATION

using System;
using System.Collections.Concurrent;
using System.Collections.Generic;
using System.Collections.Immutable;
using System.Linq;
using System.Linq.Expressions;
using System.Reflection;
using System.Runtime.Serialization;

namespace ReactiveUI.Ext
{
    class ImmutableListListConverter<T>
    {
        public static ImmutableList<T> ToImmutable( List<T> list )
        {
            return ImmutableList<T>.Empty.AddRange(list);
        }

        public static List<T> ToList(ImmutableList<T> list){
            return list.ToList();
        }

        public static object ToImmutable( object list )
        {
            return ToImmutable(( List<T> ) list);
        }

        public static object ToList(object list){
            return ToList(( ImmutableList<T> ) list);
        }

    }

    static class ImmutableListListConverter {


        static ConcurrentDictionary<Tuple<string, Type>, Func<object,object>> _MethodCache 
            = new ConcurrentDictionary<Tuple<string, Type>, Func<object,object>>();

        public static Func<object,object> CreateMethod( string name, Type genericType )
        {
            var key = Tuple.Create(name, genericType);
            if ( !_MethodCache.ContainsKey(key) )
            {
                _MethodCache[key] = typeof(ImmutableListListConverter<>)
                    .MakeGenericType(new []{genericType})
                    .GetMethod(name, new []{typeof(object)})
                    .MakeLambda();
            }
            return _MethodCache[key];
        }
        public static Func<object,object> ToImmutableMethod( Type targetType )
        {
            return ImmutableListListConverter.CreateMethod("ToImmutable", targetType.GenericTypeArguments[0]);
        }

        public static Func<object,object> ToListMethod( Type targetType )
        {
            return ImmutableListListConverter.CreateMethod("ToList", targetType.GenericTypeArguments[0]);
        }

        private static Func<object,object> MakeLambda(this MethodInfo method )
        {
            return (Func<object,object>) method.CreateDelegate(Expression.GetDelegateType(
            (from parameter in method.GetParameters() select parameter.ParameterType)
            .Concat(new[] { method.ReturnType })
            .ToArray()));
        }

    }

    public class ImmutableSurrogateSerializer : IDataContractSurrogate
    {
        static ConcurrentDictionary<Type, Type> _TypeCache = new ConcurrentDictionary<Type, Type>();

        public Type GetDataContractType( Type targetType )
        {
            if ( _TypeCache.ContainsKey(targetType) )
            {
                return _TypeCache[targetType];
            }

            if(targetType.IsGenericType && targetType.GetGenericTypeDefinition() == typeof(ImmutableList<>)) 
            {
                return _TypeCache[targetType] 
                    = typeof(List<>).MakeGenericType(targetType.GetGenericArguments());
            }
            else
            {
                return targetType;
            }
        }

        public object GetDeserializedObject( object obj, Type targetType )
        {
            if ( _TypeCache.ContainsKey(targetType) )
            {
               return ImmutableListListConverter.ToImmutableMethod(targetType)(obj);
            }
            return obj;
        }

        public object GetObjectToSerialize( object obj, Type targetType )
        {
            if ( targetType.IsGenericType && targetType.GetGenericTypeDefinition() == typeof(ImmutableList<>) )
            {
               return ImmutableListListConverter.ToListMethod(targetType)(obj);
            }
            return obj;
        }

        public object GetCustomDataToExport( Type clrType, Type dataContractType )
        {
            throw new NotImplementedException();
        }

        public object GetCustomDataToExport( System.Reflection.MemberInfo memberInfo, Type dataContractType )
        {
            throw new NotImplementedException();
        }


        public void GetKnownCustomDataTypes( System.Collections.ObjectModel.Collection<Type> customDataTypes )
        {
            throw new NotImplementedException();
        }


        public Type GetReferencedTypeOnImport( string typeName, string typeNamespace, object customData )
        {
            throw new NotImplementedException();
        }

        public System.CodeDom.CodeTypeDeclaration ProcessImportedType( System.CodeDom.CodeTypeDeclaration typeDeclaration, System.CodeDom.CodeCompileUnit compileUnit )
        {
            throw new NotImplementedException();
        }

        public ImmutableSurrogateSerializer() { }
    }
}

Answer 2

One way to do this is to use a proxy mutable list and use the OnSerializing and OnDeserialized hooks

[DataContract()]
class MyObject {

    public ImmutableList<string> Strings { get; private set}

    [DataMember(Name="Strings")]
    private List<String> _Strings;

    [OnSerializing()]
    public void OnSerializing(StreamingContext ctxt){
        _Strings = Strings.ToList();
    }

    [OnDeserialized()]
    public void OnDeserialized(StreamingContext ctxt){
        Strings = ImmutableList<string>.Empty.AddRange(_Strings);
    }
}

It's not super pretty but as Marc Gravell noted in his answer, DataContract serializer is broken with respects to immutable collections and there are no simple hooks to teach it how to behave without the above type of hack.

UPDATE

DataContract serializer is not broken. There is a way to hook surrogates in. See this separate answer showing an alternate technique.

https://stackoverflow.com/a/18957739/158285

Answer 3

Heh; I can imagine what is happening here... the generated code is probably doing (paraphrasing):

var list = obj.Strings;
while(CanReadNextItem()) {
    list.Add(ReadNextItem());
}

The problem is that the BCL immutable API would require you to catch the result each time, i.e.

var list = obj.Strings;
while(CanReadNextItem()) {
    list = list.Add(ReadNextItem());
}
obj.Strings = list; // the private set is not a problem for this

Pre-existing list deserialization code doesn't work this way because it has never needed to - and indeed, there are many different implementations of Add, some of which return non-void results which are required to be ignored.

The lack of a non-public constructor may also upset it a bit, but if this was the main problem, I would kinda expect an exception when it tries to create a non-null list.

Of course, in performance terms, the list = list.Add(...) API probably isn't the most appropriate one to use anyway (although it should work).

I blogged on this topic recently (in the context of protobuf-net, which has now been updated to work with these collection types): http://marcgravell.blogspot.co.uk/2013/09/fun-with-immutable-collections.html Hopefully this blog article should explain why the differences mean that it doesn't play nicely with existing serialization techniques, and how serialization libraries might be updated to work with this scenario.

To answer the question directly, I would say the answer is simply: because the required changes to support immutable collections have not yet been made to DataContractSerializer. I have no knowledge of whether there is a plan to address this. But: I happily declare: "works in protobuf-net" ;p