Query large external list with CAML

Question

I have a SharePoint external list that points to a 100,000 record SQL table. I have to set a filter on the Read List Operation otherwise the list doesnt work. It will timeout as it tries to return the full list. So I have added a Limit filter of size 200 onto the operation.

THe problem this causes is that when I query the external list using CAML it only searches the 200 entries returned, not the full list.

I would like it to search the entire list, but return only a maximum of 200 matching entries.

How can I best achieve this?

Answer 1

Maybe this answer att SharePoint-Exchange can help you. https://sharepoint.stackexchange.com/questions/31566/caml-and-external-list-pass-parameter-to-readlist-finder-method

My idead is that you will probably need to modify your readlist-method to make sure that it reads the entire SQL-table but with the Where-parameter specified by the filter parameter in the readlist-method. Something like

Pseudo code:

public static IEnumerable<YourEntity> ReadList(string param)
{
    if(string.IsNullOrEmpty(param) == true)
    {
        //Your original code thata only fetches the first 200 items
    }
    else
    {
        //Read your SQL-table with a Where ParamName = 'param' - clause
    }
}

Good luck

Answer 2

Based upon the structure of your query and the information presented here, reports indicate <RowLimit> implements the functionality you desire:

The RowLimit element sets the row limit for a view.

Syntax

Attributes

• Paged: Optional Boolean. TRUE if the list supports displaying more items page by page. If FALSE or unspecified, then the row limit is absolute and there is no link to see more items.

Caveats: note the remarks in the documentation.

You've probably already inspected this for your purposes (citing your question: "So I have added a Limit filter of size 200 onto the operation."). So, on to the next issue:

The problem this causes is that when I query the external list using CAML it only searches the 200 entries returned, not the full list.

This seems strange. If you're indeed using <RowLimit> and the documentation is correct:

The RowLimit element sets the row limit for a view.

And:

The <RowLimit> tag is in the schema definition of a view (direct child of ) and therefore cannot be nested inside a <Query> tag.

Then it should hold true that your nested query executes before your View component to satisfy the guarantee of your limit statement. As a corollary to this, this should allow you to perform results paging across the rest of the set defined by your query.

Building on these principles, we might construct a paged query like this:

<View>
    <RowLimit Paged='True'>200</RowLimit>
    <Method Name='ReadList'/>
    <Query>
        <Where>
            <Contains>
                    <FieldRef Name='Name'/>
                    <Value Type='Text'>{0}</Value>
            </Contains>
        </Where>
    </Query>
    <ViewFields>
    <FieldRef Name='Name'/>
    <FieldRef Name='Id'/>
    <FieldRef Name='BdcIdentity'/>                        
    </ViewFields>
</View>

Noting, as mentioned in the documentation, that we must implement <PagedRowset>. If this isn't desired, we set Paged='FALSE' above.

I'm probably entirely off base here, because this seems like exactly what you've tried already. But, in the interests of exhaustively mapping out the space, it can't hurt to suggest it.

Answer 3

Unfortunately, this is a known issue with querying external list. However, in OOB web parts, paging is supported by means of XSLT. The RowLimit works only in XSLT and not in CAML query. In External List, there is no server side paging, rather the paging is client side meaning that SharePoint pulls all the data and then sets a Filter limit in the view.

Answer 4

This is one of the scenarios where no-code BCS on it's own doesn't really cut it. Either implement this as a stored procedure on the database server, or using a custom BDC connector built in Visual Studio.

Answer 5

If you can't pull in the entire SQL table to your external list, then there's no way you're going to be able query that data set as you could a SharePoint List.

However, I can offer a solution that we've used for scenarios virtually identical to this that has worked extremely well for us. In our scenario we're querying an Oracle database which takes forever to return large data sets.

The approach we took was to use the Factory pattern to determine by what means the data source (SharePoint List, external database, etc) should be queried.

The examples below are a tad trite, but they illustrate the concept well.

So, start out with an interfaced which defines how the data set will be queried and what fields will be returned:

public interface IQueryData
{
    string ListToQuery { get; set; }
    List<MyResultObject> ExecuteQuery();
}

You'd have a custom object that represents a single record returned by the query

public class MyResultObject
{
    public string FileRef { get; }
    public string Title { get; set; }
    // any other fields you'd like to see potentially returned...
}

Then you'd have a data provider which implements this interface for the SQL data source

public class SqlDataProvider : IQueryData
{
    public string ListToQuery { get { return "BigSqlTable"; } }
    public List<MyResultObject> ExecuteQuery()
    {
        // query your external data source here...
        // populate a list of MyResultObject's from the result set and return it to the consumer
    }
}

You'd also have a data provider which implements the interface for a SharePoint data source

public class SharePointDataProvider : IQueryData
{
    public string ListToQuery { get { return "MySharePointList"; } }
    public List<MyResultObject> ExecuteQuery()
    {
        // query your SharePoint list here, using CAML, SharePoint object model, etc...
        // populate a list of MyResultObject's from the result set and return it to the consumer
    }
}

With this implementation you've encapsulated the logic and the details of the query in your respective data providers.

Now you'd have a Factory that builds the appropriate data provider (based on the specified ListToQuery parameter):

public static class QueryDataProviderFactory
{
    public static IQueryData Build(string listToQuery)
    {
        switch(listToQuery)
        {
            case "BigSqlTable": return new SqlDataProvider(); break;
            case "MySharePointList": return new SharePointDataProvider(); break;
            // you can have many other implementations here that query your data sources in different manners
        }
    }
}

Finally, you'd use your factory to initiate your query, passing in the name of the data source you want to query:

public List<MyResultObject> RunQuery()
{
    return QueryDataProviderFactory.Build("BigSqlTable").ExecuteQuery();
}

This pattern keeps your external implementation encapsulated into its own data provider and abstracts away the details of the query from the consumer. All the consumer needs to do is specify the name of the list they want to query and the Factory decides which implementation to initiate.

You can even make your IQueryData interface implement generics for further extensibility:

public interface IQueryData<T>
{
    string ListToQuery { get; set; }
    List<T> ExecuteQuery();
}

This would open the door for the consumer to also specify the type of object they'd expect to be returned.

Our query data interface actually has many more members that add even more extensibility points to our query providers, but I thought this example illustrates the point in a concise and easy-to-grasp manner.

Just wanted to offer this suggestion as it seems like virtually the same scenario we came across a year or so ago and this strategy has been working very well for us.

Answer 6

Use property ListItemCollectionPosition from SPQuery and SPListItemCollection e.g.

using (var web = site.OpenWeb("bla-bla"))
{
  var list = web.Lists["your_list"];
  var query = new SPQuery();
  query.Query = "your query";
  do
  {
    var items = list.GetItems(query);
    foreach(SPListItem item in items)
    {
      //your code
    }
    query.ListItemCollectionPosition = items.ListItemCollectionPosition;
  } while(query.ListItemCollectionPosition != null);
}