List <T> store large amounts of data , Not enough memory

Question

public class ListKeywords
{
    public int ID { set; get; }
    public string Keyword { set; get; } //关键词
    public string Language { set; get; } //语种
    public int WordCount { set; get; } //单词数
    public int WordLength { set; get; } // 字符数
    public int Status { set; get; } //采集状态  0-未采集 1-采集成功 2-保存失败 3-保存成功 4-发布失败 5-发布成功
    public bool Taken { set; get; }
    public bool FTPStatus { set; get; }
    public bool DBStatus { set; get; }
    public string UrlName { set; get; }
    public ListKeywords()
    {
    }
    public ListKeywords(string keyword)
    {
        this.Keyword = keyword;
    }
}


List<string> lines = new List<string>();
List<ListKeywords> keywordsList = new List<ListKeywords>();

using (StreamReader sr = File.OpenText(filePath))
{
    string s = String.Empty;
    while ((s = sr.ReadLine()) != null)
    {
        //lines.Add(s);   //Operating normally
        eywordsList.Add(new ListKeywords("some keywords")); //  Operating normally
        keywordsList.Add(new ListKeywords(s)); // it will be out of memeory
    }
}

In text file, have 1,000,000 line data, if i use above code to load the large data to list< keywordsList >, it will raises an OutOfMemoryException, but if i load it to list< string >, it run normally. How to solved it ?

Answer 1

Instead of using a List maybe try using an IEnumerable w/ yield?

static IEnumerable<ListKeywords> Keywords()
{
    using (StreamReader sr = File.OpenText(path))
    {
        string s = String.Empty;
        while ((s = sr.ReadLine()) != null)
        {
            yield return new ListKeywords(s);
        }
    }
}

Note that Jon Skeet's C# in Depth offers a great explanation about this in Chapter 6. I imagine he also has some articles or posts on StackOverflow about this topic. As he points out, you want to be careful about modifying this method to pass in a StreamReader (or TextReader as is used in his example) as you would want to take ownership of the reader so it will be properly disposed of. Rather, you would want to pass in a Func<StreamReader> if you have such a need. Another interesting note he adds here - which I will point out because there are some edge cases where the reader will not actually be properly disposed of even if you don't allow the reader to be provided by the caller - it's possible for the caller to abuse the IEnumerable<ListKeywords> by doing something like Keywords().GetEnumerator() - this could result in a memory leak and could even potentially cause security issues if you have security-related code which relies on the using statement to clean up the resource.