Approaches for using REST API on list with more than 5K items

Question

We are building a SharePoint application where an external system will upload data into SharePoint list using REST APIs. The volume of data in the list would be high and it would go above 5k items in near future. The external system would also have to query the data already in list (again using REST API) before uploading it so that they don't upload duplicate data.

One pitfall I can see is that when the list goes above 5k items the REST API will stop working to filter the data. I am not sure if there would be any problem in adding any data using REST API in a list with more than 5k items.

So here are my queries:

• Is there any way we can filter a list (with REST API) with more than 5k items in it?
• Will there be any issue while adding data to list using REST API when the list has more than 5k items?
• In this scenario is SharePoint list a good option or SQL server be considered?

Answer 1

Is there any way we can filter a list (with REST API) with more than 5k items in it?

Answer - Please note that 5k limit is for query, if your query is going return data more than 5k items as per your business logic, it will not return data. To avoid this, you can do following.

1. List down all the columns on which you are going to filter data(pass column name and its value in where clause)
2. Create indexes on this columns(also consider that you cannot create index on certain type of column like, multiline of text, multi value people and group etc.
3. Analyse and check that is your filters/caml query would return more than 5k items in future. if that is case, you might need to apply additional filter based on some column and split your code logic so that it returns data for each for that column type.
4. Provide some addtional filters on UI for user to select so that your query does not cross 5k limit while returning data.

Will there be any issue while adding data to list using REST API when the list has more than 5k items? Answer - There will not be any issue in adding data using REST API until and unless insert query is not dependent on select query. For e.g. Before inserting query if you are going to make select query on this list and your query is going to return more than 5k items.

In this scenario is SharePoint list a good option or SQL server be considered?

Answer - It will totally dependent on your analysis on Q1, how your filters are going to be and on which kind of columns. if your queries with filter is going to return be more than 5k in future irrespective of filter you are sending in query. you might have no other option but to go for SQL else SharePoint list is fine.