OK I have found the problem, apparently the GetById method of the EventStoreRepository passes version 0 to the underlying method it overloads. So in essence it will try to load the snapshot till version 0 which obviously does not return a snapshot.
I changed my command handler by adding the int.MaxValue as extra parameter when retrieving an aggregate and now my aggregate factory gets the snapshot passed.
When I look further, I see that the latest nuget package contains this bug that affects other people as well: https://github.com/joliver/CommonDomain/issues/27
It has been updated in the repository; https://github.com/joliver/CommonDomain/commit/0fb51c7516ff06fae032aa04dbfdd0f242f86991 but not released it seems. Hopefully it gets updated some day.