Concatenate tbb concurrent_vectors?

Question

I have multiple large Intel TBB concurrent_vectors that I need to combine. They are so large that allocating a new concurrent_vector of adequate size is not possible.

...so the following pseudo-code won't work

concurrent_vector<myStruct> A(100000);
concurrent_vector<myStruct> B(100000);
...
concurrent_vector<myStruct> X(100000);

concurrent_vector<myStruct> combined;
combined.resize(A.size()+B.size()....X.size()); // This will fail

What I would like to do is transfer or swap in the first set of vectors into the combined one so that [A], [B]...[X] becomes [AB...X]. I know tbb supports swap() of a single vector, but I can't see how to swap or append multiples. Can anyone point me in the right direction? Thanks in advance!!

Answer 1

Unfortunately, there is no such a way to combine multiple concurrent_vector into a single one without reallocation. It comes from the strict data structure rule of tbb::concurrent_vector where the size of each segment of elements must be of the same size as sum of all the previous segments.

But remember that concurrent_vector grows without reallocation, so the best you can do is to append vectors one by one destroying them immediately after copying. This way, there are more chances that large enough memory block will be freed when needed.

std::copy(B.begin(), B.end(), A.grow_by(B.size())); B.clear();
std::copy(C.begin(), C.end(), A.grow_by(C.size())); C.clear();
...
std::copy(X.begin(), X.end(), A.grow_by(X.size())); // X.clear();

Though, memory fragmentation can ruin everything once again.