CAML query items in increments

Question

I currently have a sharepoint2010 list that contains roughly 200,000 records.

I want to get each record, tweak it, massage it, and store it in a SQL table.

As of now, I am using the sharepoint 2010 web service method GetListItems like so...

System.Xml.XmlNode nodeListItems = client.GetListItems("guid", "", query, viewFields, RowNumber, queryOptions, null);

querying 200000 records is too much to query at once. How can I get around this? The GetListItems method takes CAML query parameters.

Is there a way to do this in increments, like say 5000 records at a time? How would one structure the CAML query to do that?

Unless someone has a better way of accomplishing this altogether?

Answer 1

Yes, there is, you can paginate the results. The fifth parameter is the page size, you have it set via RowNumber. Set it to 5000 if you want pages of size 5000.

Details on accessing subsequent pages can be seen from the documentation for the GetListItems method

The GetListItems method supports server-side paging. The XML data returned by this method includes a ListItemCollectionPositionNext attribute inside the rs:Data element that contains the information to support paging. This string contains data for the fields in the sort and for other items needed for paging. You should consider this string internal and not to be modified; modifying it can produce unexpected results. The following example shows the form of this return value when paging is supported.

<rs:Data ListItemCollectionPositionNext=" 
Paged=TRUE&p_ID=100&View=
      %7bC68F4A6A%2d9AFD%2d406C%2dB624%2d2CF8D729901E%7d&PageFirstRow=
      101" Count=1000 >
   <z:row ows_FirstName="Nancy" ows_LastName="Name" ….. />
   ...
</rs:Data>

To get the next page of data, the queryOption parameter is used, as shown in the following example.

<QueryOptions>
  <Paging ListItemCollectionPositionNext=" 
    Paged=TRUE&p_ID=100&View=
    %7bC68F4A6A%2d9AFD%2d406C%2dB624%2d2CF8D729901E%7d&PageFirstRow=
   101" />
</QueryOptions>

So all you need to do is grab the data in that attribute from each page's result set and add it to the query to get the next page.