CQRS - how to handle new report tables (or: how to import ALL history from the event store)

Question

I've studied some CQRS sample implementations (Java / .Net) which use event sourcing as the event store and a simple (No)SQL stores as the 'report store'.

Looks all good, but I seem to be missing something in all sample implementations.

How to handle the addition of new report stores / screens, after an application has gone in production? and how to import existing (latest) data from the event store to the new report store?

Ie:

Imagine a basic DDD/CQRS driven CRM application. Every screen (view really) has it's own structured report store (a SQL table). All these views get updated using handlers listening to the domain events (CustomerCreated / CustomerHasMoved, etc).

One feature of the CRM is that it can log phone calls (PhoneCallLogged event). Due to time constraints we only implemented the logging of phone calls in V1 of the CRM (viewing and reporting of who handled which phone call will be implemented in V2)

After a time running in production, we want to implement the 'reporting' of logged phone calls per customer and sales representative.

So we need to add some screens (views) and the supporting report tables (in the report store) and fill it with the data already collected in the Event Store...

That is where I get stuck while looking at the samples I studied. They don't handle the import of existing (history) data from the event store to a (new) report store.

All samples of the EventRepository (DomainRepository) only have a method 'GetById' and 'Add', they don't support getting ALL aggregate roots in once to fill a new report table.

Without this initial data import, the new screens are only updated for newly occurred events. Not for the phone calls already logged (because there was no report listener for the PhoneCallLogged event)

Any suggestions, recommendations ?

Thanks in advance,

Remco

Answer 1

You re-run the handler on the existing event log (eg you play the old events through the new event handler)

Consider you example ... you have a ton of PhoneCallLoggedEvents in your event log. Take your new Handles and play all the old events through it. It is then like it has always been running and will just continue to process any new events that arrive.

Cheers,

Greg

Answer 2

For example in Axon Framework, this can be done via:

JdbcEventStore eventStore = ...;

ReplayingCluster replayingCluster = new ReplayingCluster(
            new SimpleCluster("replaying"),
            eventStore,
            new NoTransactionManager(),
            0,
            new BackloggingIncomingMessageHandler());

replayingCluster.startReplay();

Event replay is an area that is not completely documented and lacks mature tooling, but here are some starting points:

• http://www.axonframework.org/docs/2.4/event-processing.html#d5e1852
• https://groups.google.com/forum/#!searchin/axonframework/ReplayingCluster/axonframework/brCxc7Uha7I/Hr4LJpBJIWMJ

Answer 3

The 'EventRepository' only contains these methods because you only need them in production.

When adding a new denormalization for reporting, you can send all event from start to you handler.

You can do this on your development site this way :

• Load your event log to the dev site
• Send all events to your denormalization handler
• Move your new view + handler to your production site
• Run events that happened inbetween
• Now you're ready