Build Warehouse Sync job fails after upgrade to TFS 2013 (timeout)

Question

After upgrading to TFS 2013 the Build Warehouse Job fails with a timeout exception. Rebuilding the Warehouse and Analysis databases didn't help.

The exception is:

[Build Warehouse Sync]: ---> Microsoft.TeamFoundation.Warehouse.WarehouseException: TF221122: An error occurred running job Build Warehouse Sync for team project collection or Team Foundation server xxxxxx. ---> Microsoft.TeamFoundation.Framework.Server.DatabaseOperationTimeoutException: TF246018: The database operation exceeded the timeout limit and has been cancelled. Verify that the parameters of the operation are correct. ---> System.Data.SqlClient.SqlException: Timeout expired. The timeout period elapsed prior to completion of the operation or the server is not responding. ---> System.ComponentModel.Win32Exception: The wait operation timed out
 --- End of inner exception stack trace ---
 at System.Data.SqlClient.SqlConnection.OnError(SqlException exception, Boolean breakConnection, Action`1 wrapCloseInAction)
 at System.Data.SqlClient.TdsParser.ThrowExceptionAndWarning(TdsParserStateObject stateObj, Boolean callerHasConnectionLock, Boolean asyncClose)
 at System.Data.SqlClient.TdsParser.TryRun(RunBehavior runBehavior, SqlCommand cmdHandler, SqlDataReader dataStream, BulkCopySimpleResultSet bulkCopyHandler, TdsParserStateObject stateObj, Boolean& dataReady)
 at System.Data.SqlClient.SqlDataReader.TryConsumeMetaData()
 at System.Data.SqlClient.SqlDataReader.get_MetaData()
 at System.Data.SqlClient.SqlCommand.FinishExecuteReader(SqlDataReader ds, RunBehavior runBehavior, String resetOptionsString)
 at System.Data.SqlClient.SqlCommand.RunExecuteReaderTds(CommandBehavior cmdBehavior, RunBehavior runBehavior, Boolean returnStream, Boolean async, Int32 timeout, Task& task, Boolean asyncWrite, SqlDataReader ds)
 at System.Data.SqlClient.SqlCommand.RunExecuteReader(CommandBehavior cmdBehavior, RunBehavior runBehavior, Boolean returnStream, String method, TaskCompletionSource`1 completion, Int32 timeout, Task& task, Boolean asyncWrite)
 at System.Data.SqlClient.SqlCommand.RunExecuteReader(CommandBehavior cmdBehavior, RunBehavior runBehavior, Boolean returnStream, String method)
 at System.Data.SqlClient.SqlCommand.ExecuteReader(CommandBehavior behavior, String method)
 at System.Data.SqlClient.SqlCommand.ExecuteReader(CommandBehavior behavior)
 at Microsoft.TeamFoundation.Framework.Server.TeamFoundationSqlResourceComponent.Execute(ExecuteType executeType, CommandBehavior behavior)
 --- End of inner exception stack trace ---
 at Microsoft.TeamFoundation.Framework.Server.TeamFoundationSqlResourceComponent.TranslateException(Int32 errorNumber, SqlException sqlException, SqlError sqlError)
 at Microsoft.TeamFoundation.Framework.Server.TeamFoundationSqlResourceComponent.TranslateException(SqlException sqlException)
 at Microsoft.TeamFoundation.Framework.Server.TeamFoundationSqlResourceComponent.MapException(SqlException ex, QueryExecutionState queryState)
 at Microsoft.TeamFoundation.Framework.Server.TeamFoundationSqlResourceComponent.HandleException(Exception exception)
 at Microsoft.TeamFoundation.Framework.Server.TeamFoundationSqlResourceComponent.Execute(ExecuteType executeType, CommandBehavior behavior)
 at Microsoft.TeamFoundation.Framework.Server.TeamFoundationSqlResourceComponent.ExecuteReader()
 at Microsoft.TeamFoundation.Build.Server.DataAccess.BuildInformationComponent3.QueryBuildInformation(ICollection`1 uris, IList`1 informationTypes)
 at Microsoft.TeamFoundation.Build.Server.BuildInformationMerger.TryMergeNext()
 at Microsoft.TeamFoundation.Build.Server.CommandQueryChangedBuilds.ContinueExecution()
 at Microsoft.TeamFoundation.Build.Server.CommandQueryChangedBuilds.Execute(IList`1 teamProjects, DateTime minChangedTime, BuildStatus statusFilter, IList`1 informationTypes)
 at Microsoft.TeamFoundation.Build.Server.TeamFoundationBuildService.QueryChangedBuilds(TeamFoundationRequestContext requestContext, IList`1 teamProjects, DateTime minChangedTime, BuildStatus statusFilter, IList`1 informationTypes)
 at Microsoft.TeamFoundation.Build.Adapter.TeamBuildWarehouseAdapter.ProcessAndUploadBuildData()
 at Microsoft.TeamFoundation.Build.Adapter.TeamBuildWarehouseAdapter.MakeDataChanges()
 at Microsoft.TeamFoundation.Warehouse.WarehouseSyncJobExtension`1.MakeDataChanges(TeamFoundationRequestContext requestContext, TeamFoundationJobDefinition jobDefinition, String& resultMessage)
 at Microsoft.TeamFoundation.Warehouse.WarehouseSyncJobExtension`1.RunInternal(TeamFoundationRequestContext requestContext, TeamFoundationJobDefinition jobDefinition, DateTime queueTime, String& resultMessage)
 at Microsoft.TeamFoundation.Warehouse.WarehouseJobExtension.Run(TeamFoundationRequestContext requestContext, TeamFoundationJobDefinition jobDefinition, DateTime queueTime, String& resultMessage)
 --- End of inner exception stack trace ---

I've also posted a Microsoft Connect feedback bug

Answer 1

It's a long story, but you can find the solution here:

Connect issue 811443

The reason for this fail is that when the TFS database is huge then the warehouse sync job is unable to complete the sync in time. This could happen with a smaller database with an incremental job too if the delta is huge. The fix (or actually workaround) is to delete the store procedure that is responsible for that and replace it with one that has a TOP 1000 or TOP 10000 in the TSQL, which makes it sync only part of the data. The stored procedures in Tfs_DefaultCollection are encrypted so you need to ask Microsoft via support to provide you with the sql for it. They know more. Of course this can't be a permanent fix and is a nice workaround available through Microsoft support only :)