Optimal way to deal with huge one to many collections with JPA

Question

Suppose we have an entity "Something" and this something has one to many (factor of millions) relationship to some "Data".

Something 1 -> * Data
Data 1 -> 1 Something

Now if we want to add some data objects we should do this:

Something.getDataList().add(Data)

This will actually pull all data objects from database which is not optimal imho.

However if i remove relationship from Something, and leave it in Data I'll be able to add and retrieve exactly those objects that I ask for using DAO:

Something
Data 1 -> 1 Something

Now the data access interface will look like this:

Something.addData(Data) // will use DataDAO to save object

or

Something.addData(List<Data>) // will use same DataDAO batch insert

I need some guidance on this, maybe I lack some knowledge in JPA, and there is no need for this? Also I'm not sure as entities are not natural this way as data is provided by their methods but its not actually contained in entity, (if this is right then I should remove every one to many relationship if there is a performance critical operation dealing with that particular entity, which is also unnatural).

In my particular case I have lot of REST consumers that periodically gonna update database. I'm using ObjectDB, JPA... But question is more abstract here.

Answer 1

I believe that having the something.getDataList() around is a clicking bomb if there are milions of Data records related to Something. Just as you said, calling something.getDataList().add(data) would fetch the whole data set from DB in order to perform a single insert. Also, anyone could be tempted to use something.getDataList().size to get the number of records, resulting in the same overhead.

My suggestion is that you use the DataDAO for such operations (i.e. adding or counting) like:

void addData(Something something, Data data){
    //Something similar can be used for batch insert
    data.setSomething(something);
    em.persist(data);
}

List<Data> findData(Something something, MyFilter filter){
    // use a NamedQuery with parameters populated by something and your filters
    return em.getResultList();
}

Long countData(Something something){
    // use a 'SELECT COUNT(*) FROM Data d WHERE d.something = :something' NamedQuery 
    return em.getSingleResult;
}