insert complex objects to azure table with TableServiceEntity

Question

I was considering adding a whole complex object into a table. Having come from good old fashioned SQL approach I'd obviously separate this into tables but I'm trying a different approach. So basically I have an object that is made up of a nested class (sort of a thing you get when you deserialise a json collection) that have usual Customer, Business and a list of InvoiceItems.

public class Business
{
    public string _id { get; set; }
    public string name { get; set; }
    public string street_1 { get; set; }
    public string street_2 { get; set; }
    public string town { get; set; }
    public string county { get; set; }
    public string postcode { get; set; }

 //other fields as needed
}

public class Customer
{
    public string _id { get; set; }
    public string name { get; set; }
    public string street_1 { get; set; }
    public string street_2 { get; set; }
    public string town { get; set; }
    public string county { get; set; }
    public string postcode { get; set; }

    //other fields as needed
 }

public class InvoiceItems
{
    public string item_id { get; set; }
    public string price_per_unit { get; set; }
    public string quanity { get; set; }
    public string date { get; set; }
}

public class WholeObject
{

    public Customer customer { get; set; }
    public Business address { get; set; }
    public List<InvoiceItems> items { get; set; }
}

(sample classes really)

Is there any limitation other than 1MB size to insert that into table? I've read up on TableServiceEntity, which should map C# objects, but will it handle? At this stage this is a very hypothetical question as I haven't actually tried to code it. Any help would be appreciated.

Answer 1

Complex properties do not get saved into table storage. Only a subset of standard data types can be persisted. Check this link for more info on what data types are supported: https://docs.microsoft.com/rest/api/storageservices/Understanding-the-Table-Service-Data-Model#property-types

However, what you want to do is possible, just a little harder to implement. Here are two ways this can be done:

• Expose CustomerSerialized, AddressSerialized and ItemsSerialized properties on your WholeObject class. Have these properties serialize/deserialize their associated complex objects to byte array, json or perhaps xml. Two downsides with this approach: a) you'll always be loading full object tree from the azure table storage even if you only need Customer object; b) you're limited to 64kb per property per object, so you better make sure that your InvoiceItems collection can fit into 64kb serialized

• Come up with a PartitionKey/RowKey schema that would allow you to save all types of related objects as individual entities but with PartitionKey/Rowkey indicating their relationship. This would allow you to get at the full object tree or at the individual child objects independently. This approach does not have the two downsides of the first approach but it is definitely on the complex side of things when implementing. It also restricts your ability to tune PartitionKey/RowKey to other business needs. I've used this approach for a large e-commerce company with a great deal of success.

In my project, the PartitionKey for all entities (root and child) was mapped to the ID of the "WholeObject" while RowKey consisted of the concatenated combination of ParentObjectID, ChildObjectID and ChildObjectType. When needing to retrieve all of the objects in the WHoleObject, simply issue a query against PartitionKey, otherwise use PartitionKey and part of RowKey to get at certain types of objects (ie: all of Addresses) or PartitionKey and full RowKey to get at an individual entity. This approach supported indefinitely deep object tree structure, however, once the number of entities went over 100, storing the object in one transaction was a headache.

Answer 2

I have come across a similar problem and have implemented a generic object flattener/recomposer API that will flatten your complex entities into flat EntityProperty dictionaries and make them writeable to Table Storage, in the form of DynamicTableEntity.

Same API will then recompose the entire complex object back from the EntityProperty dictionary of the DynamicTableEntity.

Have a look at: https://www.nuget.org/packages/ObjectFlattenerRecomposer/

Usage:

//Flatten object of type Order) and convert it to EntityProperty Dictionary
 Dictionary<string, EntityProperty> flattenedProperties = EntityPropertyConverter.Flatten(order);

// Create a DynamicTableEntity and set its PK and RK
DynamicTableEntity dynamicTableEntity = new DynamicTableEntity(partitionKey, rowKey);
dynamicTableEntity.Properties = flattenedProperties;

// Write the DynamicTableEntity to Azure Table Storage using client SDK

//Read the entity back from AzureTableStorage as DynamicTableEntity using the same PK and RK
DynamicTableEntity entity = [Read from Azure using the PK and RK];

//Convert the DynamicTableEntity back to original complex object.
 Order order = EntityPropertyConverter.ConvertBack<Order>(entity.Properties);

Since then I worked with azure team and integrated this functionality to Azure Storage SDK version 8.0.

One important note that api s in Azure SDK does NOT support IEnumerable or ICollection type properties. But if you get the nuget package v2 in the above link you can pretty much write any object into Azure Table Storage including objects with enumerable, collection type properties.

https://msdn.microsoft.com/en-us/library/microsoft.windowsazure.storage.table.tableentity.flatten.aspx

https://msdn.microsoft.com/en-us/library/azure/mt775432.aspx

I also added TableEntityAdapter class to SDK version 8.2.0 to make the usage easier. Simply pass your complex entity to its constructor and it will transparently handle flattening/converting back your complex object on writes and reads. https://github.com/Azure/azure-storage-net/blob/master/Lib/Common/Table/TableEntityAdapter.cs