Replace binaryformatter by protobuf

Question

I need to replace a library that store data in files (serialize/deserialize) This library do it currently using BinaryFormatter, but it's slow for large lists. Many posts here on stackoverflow show that protobuf is really performant, so I'm trying to use it.

In order to make a replacement without rewriting a lot of code, my requirements are:

• Must serialize generic data, as I've to interface with Store<T>(T data) (most of time T is marked with DataContract and/or Serializable attribute)
• I can't modify all different objets classes to add protobuf attributes, so I need a generic method that doesn't use these attributes
• I don't need backward compatibility: if the T object schema change (i.e. new property), all generated files are obsolete and will be deleted / recreated

My first and naive implementation looks like this:

public bool Store<T>(string key, T data)
{
    var formatter = Serializer.CreateFormatter<T>();
    using (var fileStream = new FileStream(this.GetFilePath(key), FileMode.Create))
    {
        formatter.Serialize(fileStream, data);
        return true;
    }
}

But then I got this exception:

Type is not expected, and no contract can be inferred: My.Application.Namespace.ShortcutData

Currently I'm a bit stuck, I didn't find a good tutorial on how to use protobuf-net.

Is it possible to reach these requirements with protobuf ? Do you have good tutorial on how to do this ?

EDIT:

The problem is indeed that I need to tell protobuf how to (de)serialize data. Here is what I have now:

public bool Store<T>(string key, T data)
{
    this.Register<T>();
    var formatter = Serializer.CreateFormatter<T>();
    using (var fileStream = new FileStream(this.GetFilePath(key), FileMode.Create))
    {
        formatter.Serialize(fileStream, data);
        fileStream.Close();
        return true;
    }
}

The main code is in the Register method:

protected void Register(Type type)
{
    if (type.IsGenericType)
    {
        var arguments = type.GetGenericArguments();
        foreach (var argument in arguments)
            this.Register(argument);
    }

    if (!this._registeredTypes.Contains(type) && !type.IsValueType)
    {
        this._registeredTypes.Add(type);
        var properties = type.GetProperties();
        foreach (var property in properties)
        {
            this.Register(property.PropertyType);
        }

        try
        {
            ProtoBuf.Meta.RuntimeTypeModel.Default
                .Add(type, false)
                .Add(properties
                    .Where( p => p.CanWrite)
                    .OrderBy(x => x.Name)
                    .Select(x => x.Name)
                    .ToArray());
        }
        catch
        {
            // I've a problem here: I sometime have an error for an already registered type (??)
        }
    }
}

I know this is not a clean code, but this is just an in-replacement of existing code and a complete rewrite with protobuf in mind will follow in a second step.

Answer 1

I have only just started playing with Protobuf-net.. so please don't assume my answer is in any way the "proper" way to do things. Marc Gravell will no doubt provide you an answer at some point that you should definitely take note of.

..however, my proof-of-concept that I wrote the other day required that I get a heap of classes serializing and quickly. To that end.. I came up with this:

var assembly = Assembly.Load("Assembly.Name.Here");

foreach (var type in assembly.GetTypes().Where(x => typeof (IInterfaceEachClassImplements).IsAssignableFrom(x))) {
    RuntimeTypeModel
        .Default
        .Add(type, false)
        .Add(type.GetProperties()
            .Select(x => x.Name)
            .ToArray());
}

Basically, it loads each class into the protobuf-net typemodel based on an interface they all implement.. not attributes applied to them.

Hopefully this points you in the right direction anyway.

Answer 2

What is ShortcutData? The protobuf format does not include any type metadata, so it is necessary for the engine to have some knowledge about how to map the data. This can be done in many ways, including via various attributes. The map can also be specified at runtime via code involving RuntimeTypeModel. As an example, if our ShortcutData looks like:

public class ShortcutData {
    public int Key {get;set;}
    public string Link {get;set;}
}

Then we could simply do:

[ProtoContract]
public class ShortcutData {
    [ProtoMember(1)]
    public int Key {get;set;}
    [ProtoMember(2)]
    public string Link {get;set;}
}

where 1 and 2 will become the protobuf field identifiers. To serialize, you should not need to use the formatter API. Simply:

public bool Store<T>(string key, T data)
{
    using (var fileStream = File.Create(GetFilePath(key)))
    {
        Serializer.Serialize<T>(fileStream, data);
        return true;
    }
}