std::istream_iterator, memory consumption

Question

I try to iterate through a text file using iterators:

1) std::istream_iterator<MyData> itRecordStreamBegin(fileStream);
2) std::istream_iterator<MyData> itRecordStreamEnd; 

After the first line it reads some data from the file (it uses operator >> for MyData type). As expected.

Unfortunately after that memory consumption rises and it seems like istream_iterator tried to load whole file in memory. Several seconds later (when there are out_of_memory exceptions thrown) debugger gets into the second line. But file reading failed and I cannot read anything more.

Questions:

Is this correct behaviour for istream_iterator (to load file into memory)?

I do not observe this for smaller files (like 20MB).

Maybe for such large file I need to use common getline way?

Answer 1

The istream_iterator just uses the >> operator on the type. It keeps at most one instance of the type in memory. When using it, you do make copies of the instance. I would suspect (but without seeing the code) that either your copy constructor or destructor is defective, or you leak memory somewhere in your >> operator. I would be very surprised if the problem is related to istream_iterator.