How to query large list and retrieve one field of all items?

Question

Image a list with i.e. 20.000 list items. All items have the field ContainerId (indexed). I now want to retrieve all ContainerIds to do other stuff with those.

SPQuery is limited to a maximum of 2000 items - yet the ViewFields is pretty much what I want. Is it still the best was to go forward? How would I implement paging I didn't see a Skip field for the query.

Any better way to go forward? In the end I want to have a List<string> allContainerIds.

Answer 1

Try to use the ContentIterator.

Example from MSDN:

static int exceptions = 0;
static int items = 0;

protected void OnTestContentIterator(object sender, EventArgs args)
{
    items = 0;
    exceptions = 0;
    string query1 = @"<View>
        <Query>
            <Where>
                <And>
                    <BeginsWith>
                        <FieldRef Name='SKU' />
                        <Value Type='Text'>S</Value>
                    </BeginsWith>
                </And>
            </Where>
        </Query>
    </View>";

    ContentIterator iterator = new ContentIterator();
    SPQuery listQuery = new SPQuery();
    listQuery.Query = query1;
    SPList list = SPContext.Current.Web.Lists["Parts"];
    iterator.ProcessListItems(list,
        listQuery,
        ProcessItem,
        ProcessError
    );
}

public bool ProcessError(SPListItem item, Exception e) 
{ 
    // process the error
    exceptions++; 
    return true; 
}
public void ProcessItem(SPListItem item)
{
    items++;
    //process the item.
}

In your case in ProcessItem you can add values to List allContainerIds

Answer 2

For querying large lists, I recommand using the "ContentIterator" object. you can refer to this link http://prasannabj.blogspot.com/ to have more information about it.

For the paging, it's possible, you can refer to this link http://blogs.msdn.com/b/dbadki/archive/2008/10/08/caml-query-execution-using-pagination-for-custom-columns.aspx. I'm looking for another more detailed article and will share it with you asap.