Since it sounds like you will actually need to query against the payment data outside the context of the client (i.e. for aggregated reporting), I would not want to add each individual payment item to the client collection objects.
I would certainly create a payment object collection, and then either reference a payment key in the client object for each payment and the client key in the payment object, so you have a definitive way to relate one to the other in either direction, or have a third collection mapping clients to payments.
What is preferable here may really depend on your access pattern. For example, you may not even need such "foreign keys" on both set of objects if the lookup is always going to be in one direction for cases where you need to establish the relation.