Seeking references and advice for modeling time series data in Mongoengine

Question

I'm new to MonogoEngine and web apps in general and I've got programmers block. I'd like to create a model(s) for logging periodic recorded values from equipment. Based on a MongoDB blog post, I'm going for something like the following (copy/pasted from their post):

{
  timestamp_hour: ISODate("2013-10-10T23:00:00.000Z"),
  type: “memory_used”,
  values: {
    0: { 0: 999999, 1: 999999, …, 59: 1000000 },
    1: { 0: 2000000, 1: 2000000, …, 59: 1000000 },
    …,
    58: { 0: 1600000, 1: 1200000, …, 59: 1100000 },
    59: { 0: 1300000, 1: 1400000, …, 59: 1500000 }
  }
}

I'd like the document to be more along the lines of the following for a data point logged each minute for one day for multiple values.

{
  timestamp_hour: ISODate("2013-10-10T23:00:00.000Z"),
  equipmentID: “123456”,
  parameter1: {
    0: { 0: null, 1: null, …, 59: null },
    1: { 0: null, 1: null, …, 59: null },
    …,
    22: { 0: null, 1: null, …, 59: null },
    23: { 0: null, 1: null, …, 59: null }
  },
  parameter2: {
    0: { 0: null, 1: null, …, 59: null },
    1: { 0: null, 1: null, …, 59: null },
    …,
    22: { 0: null, 1: null, …, 59: null },
    23: { 0: null, 1: null, …, 59: null }
  }
}

So, my trouble is how to set this up as model(s).

1. I was thinking "0: { 0: null, 1: null, …, 59: null }" is a model of embeddedDocument type. Then add these to the a model of Parameters. Is that the right path?

2. The number of parameters may vary, so maybe these need to be as part of a listField?

3. Can an embeddedDocument be embedded in an EmbeddedDocument?

Answer 1

I'm not sure what the benefits of using MongoEngine in this case would be over pymongo directly. It might be cumbersome for such a large model - where as interacting with a dictionary of a known schema in my mind would be simpler.

So this all comes with a caveat and warning - it can be done in MongoEngine, I'm just not sure its your best course of action for a happy experience.

1. It would be a nested EmbeddedDocument eg:

   class SecondAggreate(EmbeddedDocument):
       s0 = IntField()
       s1 = IntField()
       ...
       s59 = IntField()
   
   class MinuteAggregate(EmbeddedDocument):
       m0 = EmbeddedDocumentField(SecondAggregate)
       m1 = EmbeddedDocumentField(SecondAggregate)
       ...
       m59 = EmbeddedDocumentField(SecondAggregate)
   

   Notice we have to add a m or s to the keys denoting minutes and seconds because we can't have attributes that are literals. Use db_field_name to ensure you don't store that extra data.

2. In the example the fields are fixed, they aren't required so you don't have to store them - but the types are defined for when you do. You could use a list but there are other considerations for lists but its simpler with a dictionary as you define the key.

3. Yes - see point 1.

Final caveat - if you choose to do this in MongoEngine (and as shown above you can) then don't use Doc.save() make sure any updates are explicit as save() may introduce race conditions if multiple threads try and update the document at the same time.