Optimal relation in Mongoid for a very large data set

Question

I´m trying out Mongoid in app that has Stations which send Observations every 5 minutes (288/day). Which adds up to a really large dataset, How do I set up the relationship for performance?

The most critical query is getting all the stations and the last observation. Which sounds simple but is a PITA in Postgres.

I´m using

Rails 4.0.3
Mongoid 4.0.0beta

1. Embedded

class Station 
    embeds_many :observations 

    def latest_observation
        observations.last
    end

    # fields ...
end

class Observation 
    embedded_in :station 

    # fields ...
end

2. Has many / belongs to / embeds_one :latest_observation

class Station 
    has_many :observations 
    embeds_one :latest_observation, class: 'Observation'
end

class Observation 
    belongs_to :station 

    after_save :update_station

    def update_station
        self.station.latest_observation = self
        self.station.save!
    end
end

Answer 1

I think the first option is not a viable solution. You'll end up with some (few) station records that will become of huge size because of embedding observations.

In terms of MongoDB I would suggest:

• a collection for stations that embeds a a fixed amount of observations (with the observations for the last 30 days or so)
• an observations collection that will act as a warehouse

stations: [ { id: sta_1234, observations: [ {observation_id: obs_1234, observation_data: {} }, ...] }, ... ]

were observations has a fixed size.

that way

• you can have a quick reference to recent observations.
• you will have fixed sized docs (which is very good for mongo!)

• you will not lose any history

  Actually observations may be better off as observation_ids and you keep there only the ids, or some useful data about each observation

a "drawback" about it is that you'll have to use raw mongo at some points cause mongoid does not support everything (yet), but I think the benefits are greater.

With this approach you 'll have to handle the duplicate data on insertion in your app, which should be no problem.