Error in my custom webpart: The attempted operation is prohibited because it exceeds the list view threshold enforced by the administrator

Question

I am using sharepoint 2010. I have created a custom webpart. In this webpart it is possible to add a new item. Before I add this new item, I need to select some values from drop down boxes. In the event receiver of the last dropdownbox I do this in code:

protected void ddProcesses_SelectedIndexChanged(object sender, EventArgs e)
        {
            btnAddFunction.Visible = true;

//            alApparaten.Visible = false;

            ReloadFunctions();

            ((ApparaatList)alApparaten).FuncID = 0;
            ((ApparaatList)alApparaten).Reload();
        }


public void Reload()
        {
            using (AssetRegisterDataContext spContext = new AssetRegisterDataContext(SPContext.Current.Site.Url))
            {
                var apparaten = from i in spContext.Apparaat.ToList()
                                where i.Functie_ID != null && i.Functie_ID.Id == FuncID
                                select i;
                rptApparaten.DataSource = (from i in apparaten where i.Verwijderd != true select i);
                rptApparaten.DataBind();
            }

        }


    private void ReloadFunctions()
            {
                using (AssetRegisterDataContext spContext = new AssetRegisterDataContext(SPContext.Current.Site.Url))
                {
                    var query = (from i in spContext.Functies.ToList()
                                 where (i.Locatie != null && i.Locatie.LocatieNaamCode == ddLocations.SelectedValue)
                                    && (i.Faciliteitnaam != null && i.Faciliteitnaam.Faciliteitnaam == ddFacilities.SelectedValue)
                                    && (i.Procesnaam != null && i.Procesnaam.Procesnaam == ddProcesses.SelectedValue)
                                    && (i.Verwijderd != true)
                                 select i
                                 );
                    rptFunctions.DataSource = query;
                    rptFunctions.DataBind();
                }
            }

When I go to the list "Apparaat" I see there 5002 items. When I got to the settings of the list I see this info:

5002 items (the theshold for this list view is 5000).

When I got the list "Apparaat" everything is OK. I try to add 2 items and is working OK.

Why my custom webpart is giving me the theshold limit excetpion and the GUI is working fine?

Answer 1

I found the solution by myself. I have created a whileloop. It loops in this while loop till the count of the total items is reached. Inside the loop I take the first 1999 items and add it to a list. After this I take the second 1999 items. It will do this logic till you reached the total count of items. After this you have a list with all the items and can add it to the repeater. The reason why I used 1999 items is because it is possible the throttle can be decrease from 5000 to 2000. The custom code will then also work.

Answer 2

In the Reload function, when you call ToList(), it forces the data context to issue the query. At that point in your code, you have not specified any filters (where clause), so all records are read. (Well, attempted to be read.)

The DataSource property can accept an IQueryable source, so just remove the ToList calls (change the ReloadFunctions method as well) and you should be OK.

Answer 3

Linq to SharePoint queries have a default row limit of maximum int which is 2147483647 and apparently cannot be changed. Your query is probably returning more than 5000 items and exceeding the thresholds.

To make sure you are not exceeding the threshold, you have to use SPQuery object and write your query in CAML AND SET THE ROWLIMIT PROPERTY TO 5000. You will need something like this. Change the CAML query.

SPQuery q=new SPQuery();
q.Query="<Where><IsNotNull><FieldRef Name='Functie_ID' /></IsNotNull></Where>";
q.RowLimit=5000;
SPListItemCollection resultItems=ApparaatList.GetItems(q);
rptApparaten.DataSource =resultItems;
rptApparaten.DataBind();

Answer 4

this explains what is going on:

The attempted operation is prohibited because it exceeds the list view threshold enforced by the administrator

its set by the admin on a list from central administration on the web application!

Can you goto central admin -> application managment -> manage web applications -> select the web application that holds the given list -> within the tab above click on the drop down for general settings -> select resource throtteling.

This should give you a popup box with the webapplication settings that are set by the admin. if you scroll down you would see List View Lookup Threshold , by default i think its set to 5000, you can increase this number to something grater.

I encounter this issue when i have running workflows on the list, it produces the same error but this is due to List View Lookup Threshold which is set to 8 by default, you can increase this number to 100 and im almost certin that it will work.

I understand that some people say that returning all data is wrong, yes that is true to some effect... what if you need it all? what if you have over 1 million records? i know there are other soltions but this is one and is easy to rectify within central admin settings and if your server can handle the strain than this is a simple option.

Specifies the maximum number of list or library items that a database operation, such as a query, can process at one time. Operations that exceed this limit are blocked.

To give you time to make alternative plans, SharePoint 2010 warns you on the List Settings page when your list has exceeded 3,000 items. The warning contains a help link to this topic.

http://office2010.microsoft.com/en-us/sharepoint-server-help/manage-lists-and-libraries-with-many-items-HA010378155.aspx?redir=0

Answer 5

This is expected behaviour - SharePoint has a feature called 'Query Throttling', which is designed to prevent inefficient list queries from adversely affecting performance for users.

This limit is set to 5000, and this should probably not be changed. The reason that it's set to 5000 is that at more than 5000 items, the underlying SQL database escalates from row-locking to table-locking. As SharePoint stores much of it's content in one table in the content database, this can have a significant impact on the performance of your system, and this can affect many site collections.

Now, there are some ways to mitigate the Query Throttle. Firstly, you can use column indexes on your List. Just as querying an unindexed SQL table column forces a table scan, so querying an unindexed List column does the same. If that list contains more than 5000 items, well, you hit the throttle. However, an indexed column means that you can just retrieve the rows you need.

If you do need to trawl through each item in a large list, there is also the ContentIterator control - however, this is slow, as it uses multiple queries to get the relevant items either one at a time, or in batches.

Retrieving items one-at-a-time:

ContentIterator ci = new ContentIterator("Single Item Example);
SPQuery qry = new SPQuery();
qry.Query = "<Where><Eq><FieldRef Name='Colour'/><Value Type='Text'>Pink</Value></Eq></Where>";
qry.Query = qry.Query + ContentIterator.ItemEnumerationOrderByNVPField;
ci.ProcessListItems(list, qry,
    delegate(SPListItem item)
    {
         // Do stuff with the item
    },
    delegate(SPListItem item, Exception ex)
    {
      // Handle an exception. Return TRUE to rethrow the exception, FALSE to keep iterating  return false;
    }
 );

Retrieving items in batches:

ContentIterator ci = new ContentIterator("Batch Example);
SPQuery qry = new SPQuery();
qry.Query = "<Where><Eq><FieldRef Name='Colour'/><Value Type='Text'>Pink</Value></Eq></Where>";
qry.Query = qry.Query + ContentIterator.ItemEnumerationOrderByNVPField;
ci.ProcessListItems(list, qry,
    delegate(SPListItemCollection items)
    {
        foreach (SPListItem itm in items)
        {
            //Process each item in batch!
        }
    },
    delegate(SPListItemCollection item, Exception ex)
    {
         // Handle an exception. Return TRUE to rethrow the exception, FALSE to keep iterating  return false;
    }
);

This might help: http://www.novolocus.com/2012/07/09/dealing-with-large-lists-part-1-what-is-throttling-and-how-it-causes-difficulty/