Implementing a pattern for maintaining changes of an entity over time?

Question

I'm designing a system that acts as a master data service for what I shall here call boxes. The system is to be implemented in Java with a relational database (SQL) as the main storage. Each box has a ~dozen different top-level properties: ranging from simple primitives (booleans, integers, etc) to other objects and lists of other objects.

The main issue is that most of the box properties may change over time, and one needs to be able to schedule those changes in advance. Furthermore, one needs to be able to schedule any number of upcoming changes to any of the attributes, in any chronological order.

For example, in October we might schedule new set of derps for a box for December, to be returned back to normal on January 1. If, in mid-December, we find out the box gets a new foobar_id in February, we need to be able to schedule that change for February without affecting the upcoming derp change on January 1 -- and without accidentally reverting the derps of December back when the time comes to apply the foobar_id update.

My idea is to create some sort of a queue of upcoming change events. Each queue item would only change the values of the properties given in that exact event. New events could be added into any position of the queue and existing events could be removed from it. When an event would occur, it would record the old value of the property it changed.

Now, the keywords of the previous paragraph are some sort of a queue. I'm unsure how to actually implement this in Java + a relational database! It seems that a language with strong static typing doesn't lend itself well to this kind of an exercise in generic attribute changes.

I'm considering a relatively simple database table with a timestamp (date of the change), the name of the property that's going to change (an enumeration), and a serialized (JSON) representation of the new data. Then each property would basically need their own handler/deserializer. Another way would be to copy the box database structure for upcoming changes and just store a bunch of "boxes" with no other properties than the ones that are going to change. This seems like it might be easier Java-wise but the database would become quite complex, when almost all tables would need to be duplicated.

I need the system to be robust so that it's not too easy to break it when new properties inevitably are added, or some old properties are changed. As such, I'm not too fond of the idea of using reflection on this. New changes can only come in to the system as fast as human beings can type, so I don't need the solution to be optimized for speed. But I do plan to keep one complete and up to date version of each box object in the database, so that I don't need to reconstruct the object from a number of changes every time I need it. Also, for what it's worth, the database queries that the system is going to be handling are going to be pretty simple and the number of boxes is unlikely to exceed ten thousand. I'm not concerned about the actual scheduling part, i.e. triggering the changes at the correct time.

So I guess that basically my questions are these, starting from the most important one:

1. Does the queue pattern that I just described have some name by which I could find some more material on it? If it does not have a name, can you point me to something similar?
2. Can you point me to some good resources specific to Java, SQL and this kind of a pattern?
3. Any other thoughts? Anecdotes? Am I missing something obvious and I should do it some other way? (Java and SQL are going to stay anyway.)

Answer 1

I ended up implementing something really close to the JSON Merge Patch format. The database stores those JSON items, and they are applied into the objects when their time comes. This seems to work well, and the only thing I regret is that I didn't find JSON Merge Patch in time to not re-invent it from scratch... There would have existed a GitHub project that can be added as a Maven dependency, for example. Note that JSON Merge Patch is not to be confused with JSON Patch. (That would not have been as helpful in my case.)

Answer 2

Take a look at Entity Component Systems, and Event Sourcing Systems.

For your specific case I believe Entity Component Systems are what you need.

Entity Component System

Each entity is essential a box. That box can have any number of components of various assorted shapes in it. If during February you need to add a reference for some short-term tracker, all you need is a component that models that subset of properties. 4 weeks later you can delete that component from the database and not affect any other information. A System is responsible for managing and interacting with a few of these components to achieve some goal.

Game platforms use this system extensively as game artists often want to mix this, that and the other and the engineers learnt not to hard code it. Many scripting languages (such as JavaScript) use the Entity/Component parts of this pattern roughly known as Prototype Objects, or Property Bags. I find Steve Yegge's post on Property Based Objects a good read about how to organise property orientated objects.

Event Sourcing

Take every state-change and encode it as message on an append only log. The log is meant to be read serially, later messages modify the system state described by all the previous messages. This becomes the canonical state.

To optimise lookup speeds construct a view of the state in the log. The view can be any shape you need to allow the code to work. In fact you can have several views for slight variations of the same data, none of these are authoritative, and can be dropped as pleased.

For a point of reference, just look at any Source Code Repository. Every Commit is an appended message to the log, but this usually only stores the delta to the current files state. The view is the fully reconstructed file in a check-out.