SPQuery and RowLimit

Question

I have around 10000+ rows(Listitems) in a list I want to query.

I want to iterate each of the item in a timerJob - but I cant take all of them at a time since : Object Model Override - NO, ListView threshold - 1000 - in the FARM level and i we cant change this.

What is the way for me to iterate all the 10000+ (like a batch) ??

Answer 1

You should use a ContentIterator. This will allow you to iterate over the contents of very large lists without triggering an SPQueryThrottledException.

For example:

SPList list = SPContext.Current.Web.Lists["MyList"];

// Build query using an iterator-defined WHERE clause
string query = string.Format("<Where><Eq><FieldRef Name='MyFieldName'/><Value Type='Text'>MyFieldValue</Value></Eq></Where>{0}", ContentIterator.ItemEnumerationOrderByNVPField);

// Instantiate iterator and use it to process the list
ContentIterator iterator = new ContentIterator();
iterator.ProcessListItems(list, query, ProcessListItem, ProcessListItemError);

Then you'd define your ProcessListItem and ProcessListItemError thusly:

static void ProcessListItem(SPListItem item) {
    Console.WriteLine("ProcessListItem: Name {0}", item.Title);
}

static bool ProcessListItemError(SPListItem item, Exception e) {
    Console.WriteLine("ERROR: message {0}", e.Message);
    return true;
}

I'd also recommend you review Microsoft's Best Practices with SharePoint Server articles, in particular "Writing Efficient Code In SharePoint Server", which further discusses properly writing queries.

Answer 2

Content Iterator is one of the best technique you can use. Do forget to use order by clause ContentIterator.ItemEnumerationOrderByNVPField that uses indexed field otherwise content iterator may throw exception when handling large items.

This article best covers content iterator plus other techniques as well :

http://extreme-sharepoint.com/2012/07/17/data-access-via-caml-queries/