CAML Query on SharePoint Online View

Question

I have a very large library in a SharePoint Online environment: > 3000 (document sets) in the root folder and more than 7000 (document sets, folders and documents) in total. The system is mainly based on custom code: C# CSOM.

Performance wise, the system would benefit greatly from querying on subsets of the whole data set, rather on the whole 7000 items every time (my present solution loads batches of 5000 items in a ListItemCollection: in my case first 5000 and then 2000, and then querying is done by LINQ) especially bearing in mind that certain operations are directly tied to specific views.

I am trying the below implementation as found in a comment on another thread.

Microsoft.SharePoint.Client.List list = ctx.Web.Lists.GetByTitle(ListName);
ctx.Load(list);
ctx.ExecuteQuery();

Microsoft.SharePoint.Client.View view = list.Views.GetByTitle(viewName);
ctx.Load(view);
ctx.ExecuteQuery();

CamlQuery query = new CamlQuery();
query.ViewXml = view.ViewQuery;

ListItemCollection items = list.GetItems(query);
ctx.Load(items);
ctx.ExecuteQuery();

My issue is that the ListItemCollection is being populated with all the items on the root collection (3000), when the chosen views tried consist of at most 60 items.

The view.ViewQuery is populated with this value:

"<GroupBy Collapse=\"FALSE\" GroupLimit=\"30\"><FieldRef Name=\"CurrentStatus\" /></GroupBy><OrderBy><FieldRef Name=\"CurrentStatus\" Ascending=\"FALSE\" /><FieldRef Name=\"Stage\" /></OrderBy><Where><And><Eq><FieldRef Name=\"Stage\" /><Value Type=\"Text\">Confirm Container</Value></Eq><Eq><FieldRef Name=\"PastTransition\" /><Value Type=\"Boolean\">0</Value></Eq></And></Where>"

Is the above code supposed to work? If yes, what's wrong? If not, what's another solution that achieves the same thing?

Answer 1

Are you pulling CAML from the view because users might update the details of the filter? If not, teeing up the query manually is probably the better way to go.

Give this a try:

query.ViewXml ="<View Scope=\"Recursive\"><Query><Where><And><Eq><FieldRef Name=\"Stage\" /><Value Type=\"Text\">Confirm Container</Value></Eq><Eq><FieldRef Name=\"PastTransition\" /><Value Type=\"Boolean\">0</Value></Eq></And></Where></Query></View>"

Because your list is > 7k items, you would also need to make sure that one or both of the Stage and PastTransition columns are indexed. Might depend on which one returns the smaller subset of data and in what order they are evaluated.

You may also need to do "RecursiveAll" instead of "Recursive", depending on if you want/need to return folders. View scope enumeration: https://msdn.microsoft.com/en-us/library/microsoft.sharepoint.spviewscope.aspx

This should get you the filtered list. From there, your code can do any grouping/ordering on the returned items.

Update

Since query details need to be puled dynamically from existing views (per your feedback below), you can pull the contents of the out of the existing view.ViewQuery and inject it into the view XML above. This will get rid of and also allow the query to run on lists > 5k.

See Jurgen's code for a fully baked solution.

Answer 2

The below is my final solution, based on @Jim Barntish's answer.

Microsoft.SharePoint.Client.List list = ctx.Web.Lists.GetByTitle(ListName);
ctx.Load(list);
ctx.ExecuteQuery();

Microsoft.SharePoint.Client.View view = list.Views.GetByTitle(viewName);
ctx.Load(view);
ctx.ExecuteQuery();

CamlQuery query = new CamlQuery();
query.ViewXml = view.ViewQuery;

int pFrom = query.ViewXml.IndexOf("<Where>") + "<Where>".Length;
int pTo = query.ViewXml.LastIndexOf("</Where>");

string whereClause = query.ViewXml.Substring(pFrom, pTo - pFrom);

string queryStr = "<View Scope=\"Recursive\"><Query><Where>" + whereClause + "</Where></Query></View>";

query.ViewXml = queryStr;

ListItemCollection items = list.GetItems(query);
ctx.Load(items);
ctx.ExecuteQuery();