BCS External Content Type with 20,000+ items (O365)

Question

We have a large database of 20,000+ items on a local database that we need to get into Sharepoint Online 2013 in O365 to use as metadata for various documents in multiple locations. New entries and changes need to be synced into the choice list. Can anyone offer some insight on the best way to accomplish this?

Current Options:

1) BCS External Content Type: but how to get past the 2000 item throttle limit?

2) Managed Metadata: manual updates only? This offers the best functionality but seems like a nightmare to administer.

3) 3rd party tool? Still subject to 5000 throttle limit?

4) Use a JQuery data table and a custom edit form to "select" from an xml extract of the database and inject it into a text field (however this would not apply any "updates" if data changes).

I am at my wits end in seeing how this might be possible in O365.

Answer 1

I had a similar issue this morning, with an external dataset with around 100,000 records. The way I worked around it was to create a SQL stored procedure to do the Read List operation, rather than querying the data directly. The Read Item operation can still be linked directly to the list, rather than requiring a second stored procedure.

On the Read List operation, you'll need to specify a filter like this one:

To simplify searching, my procedure has only one parameter, but the query it runs searches for the parameter in five different columns. This gives the user maximum flexibility for their search, without complicating the input.

Also, note the Default Value on the filter. You need to have something in there, and an asterisk is just fine.

The output for your procedure should output all the columns that your items contain. You also need to mark columns as Identifiers on the Return Parameter page as appropriate. In my case, I had three identifier columns. (If you're a SQL developer, think of this as the primary key.)

For your Read Item operation, you'll need to link up the identifier columns. As long as your Read List stored procedure and Read Item source object (a view in my case, but a table would work fine too) outputs the same columns and has the same identifiers, you should be good to go.

Answer 2

BCS. The item throttle applies to a single query operation. So, in your BCS model, be sure to include filters on the appropriate columns. These filters will show in the Entity Picker (the button on the list form that looks kind of like the people picker).

You won't necessarily get an autocomplete experience like the Bing/Google search box, but your users can certainly search for the item from your database.

Answer 3

I just came across this discussion and it has solved my issue. My data set was far smaller. It was annoyingly just over the 2,000 item at ~2,400 items. The options were to increase the list item limit which affects all lists so is not acceptable or try this solution. This article was EXACTLY what I was looking for so I am reposting it to increase it's visibility.

http://msdn.microsoft.com/en-us/library/office/ff798274%28v=office.14%29.aspx

This are other great resources:

BCS and SQL http://blogs.msdn.com/b/jysaikia/archive/2010/12/15/a-step-by-step-guide-in-designing-bcs-entities-by-using-a-sql-stored-procedure.aspx#Create

Paging and sorting with BCS