Out of memory exception

Question

I have a list of a class type which contains (string, string, string, string, int, DateTime, DateTime) The total amount of chars in the strings combined equals 28.

I read in a file to populate the list which will finish with 19,000,000 (nineteen-million) of these objects in the list.

I read in the file and add to the list like so

public void ReadDocGrabValues(string fileREAD)
{
    using (var file = new StreamReader(fileREAD))
    {
        var j = file.ReadLine();

        while (j != null)
        {
            mylist.Add(new IDandAGE(j.Substring(0, 15), j.Substring(16, 1), j.Substring(18, 6), j.Substring(25, 6), 0, DateTime.Today, DateTime.Today));
            j = file.ReadLine();
        }
    }
}

Which shouldn't be the problem. I then go through the entire list to work out the DateTime objects from the two strings.

public void ConvertYOBDOI()
{
    foreach (IDandAGE x in mylist)
    {
        string IssueDate = (x.StringDOD.Substring(0,4) + "-" + x.StringDOD.Substring(4,2) + "-01");
        string BirthDate = (x.StringYOB.Substring(0,4) + "-" + x.StringYOB.Substring(4,2) + "-01");
        x.DeathDate= DateTime.Parse(DeathDate);
        x.YearOfBirth = DateTime.Parse(BirthDate);

     }
}

I then go through the entire list again to work out the age value.

public void DateCalc()
{
    foreach (IDandAGE w in mylist)
    {
        w.Age = w.DateOfDeath.Year - w.YearOfBirth.Year;
        if (w.YearOfBirth > w.DateOfDeath.AddYears(-w.Age)) w.Age--;

    }
}

And finally I write out the values I want from the list to a file.

public void CreateAgeFile()
{
   StreamWriter a = new StreamWriter(@"C:\GetAgeTest.txt");

    foreach (IDandAGE x in mylist)
    {
       String f = (x.ID + "," + x.Gender + "," + x.StringYOB + "," + x.StringDOD + "," + x.Age + ",NULL,NULL,NULL");
        a.WriteLine(f);
    }
    a.Close();

}

I've really only just started coding so apologies in advance for it being inefficient/rubbish/not being able to answer my own question.

I guess it's possible that it's giving me this exception for multiple reasons since I traverse through the list several times.

Any help or suggestions welcomed.

Thank you.

Answer 1

If you are accessing 19M records, then yes: you need to at least think about memory. You are currently storing them all in a list, so they won't be collectable. If you are on 32-bit (or 64-bit-enabled with prefer-32-bit checked) then this will be tight. A better idea is probably to use an iterator block so that you don't need them all in memory at once:

public IEnumerable<IDandAGE> ReadDocGrabValues(string fileREAD)
{
    foreach(string j in File.ReadLines(fileREAD))
    {
        yield return new IDandAGE(j.Substring(0, 15), j.Substring(16, 1),
            j.Substring(18, 6), j.Substring(25, 6), 0,
            DateTime.Today, DateTime.Today);
    }
}
...
foreach (IDandAGE w in ReadDocGrabValues(path))
{
    // do per-item processing
}