SSRS Tries to Upgrade Itself every morning at 2 am

Question

I have a Windows 2019 Server running both SQL Server 2017 Enterprise (14.0.3238.1) and SQL Reporting Services 2017 (14.0.600.1109). In noticed in the RSManagement logs (located here: C:\Program Files\Microsoft SQL Server Reporting Services\SSRS\LogFiles), that every morning between 2 am and 2:30 am Reporting Services tries to upgrade itself.

Here is a recent attempt:

2019-12-12 02:15:38.2956|INFO|5|Could not find ProductInfo table. Will do first-time upgrade.| RequestID = s_4446d062-b6f4-48df-ab17-5bcd6e39597f 
2019-12-12 02:15:39.2019|INFO|15|Upgrading ReportServer database to version 14.0.600.1109 hash B114DCF686D456AD82E265B6E3C78CE6652923210651185B82C532429D01A214| RequestID = s_4446d062-b6f4-48df-ab17-5bcd6e39597f 
2019-12-12 02:15:39.2644|ERROR|15|Aborting SQL batch script!  Error executing SQL batch script at entry
------------
/**************************************************************/
/* Copyright (c) Microsoft.  All rights reserved. */
/**************************************************************/
-- !!! This assumes the database is created and the user is either a dbo or is added to the RSExecRole
-- !!! Please run setup to create the database, users, role !!!

--------------------------------------------------
------------- Upgrade RS Catalog
--------------------------------------------------

--- Starting in Kilimanjaro, the catalog version numbers were changed to be just an integer (starting with 100)
--- The old system (C.0.9.45) was last used in Katmai.

use [ReportServer]
------------
| RequestID = s_4446d062-b6f4-48df-ab17-5bcd6e39597f System.InvalidOperationException: ExecuteNonQuery requires an open and available Connection. The connection's current state is closed.
   at System.Data.SqlClient.SqlCommand.ValidateCommand(String method, Boolean async)
   at System.Data.SqlClient.SqlCommand.InternalExecuteNonQuery(TaskCompletionSource`1 completion, String methodName, Boolean sendToPipe, Int32 timeout, Boolean& usedCache, Boolean asyncWrite, Boolean inRetry)
   at System.Data.SqlClient.SqlCommand.ExecuteNonQuery()
   at Microsoft.BIServer.HostingEnvironment.Storage.MeteredSqlConnection.ExecuteBatchScript(String script, TimeSpan individualCommandTimeout)
2019-12-12 02:15:39.2644|FATAL|15|Database upgrade failed!!  The database may now be in an inconsistent state.| RequestID = s_4446d062-b6f4-48df-ab17-5bcd6e39597f System.InvalidOperationException: ExecuteNonQuery requires an open and available Connection. The connection's current state is closed.
   at System.Data.SqlClient.SqlCommand.ValidateCommand(String method, Boolean async)
   at System.Data.SqlClient.SqlCommand.InternalExecuteNonQuery(TaskCompletionSource`1 completion, String methodName, Boolean sendToPipe, Int32 timeout, Boolean& usedCache, Boolean asyncWrite, Boolean inRetry)
   at System.Data.SqlClient.SqlCommand.ExecuteNonQuery()
   at Microsoft.BIServer.HostingEnvironment.Storage.MeteredSqlConnection.ExecuteBatchScript(String script, TimeSpan individualCommandTimeout)
   at Microsoft.BIServer.Management.WebApi.DatabaseUpgrade.DatabaseUpgrader.GetStatusAndUpdateIfRequired(ISqlAccess sqlAccess)
2019-12-12 02:17:25.6049|ERROR|15|Unhandled error in the Web API. Exception: System.Net.Http.HttpRequestException: Error while copying content to a stream. ---> System.IO.IOException ---> System.Net.HttpListenerException: The I/O operation has been aborted because of either a thread exit or an application request
   at System.Net.HttpResponseStream.EndWrite(IAsyncResult asyncResult)
   at Microsoft.Owin.Host.HttpListener.RequestProcessing.ExceptionFilterStream.EndWrite(IAsyncResult asyncResult)
   --- End of inner exception stack trace ---
   at Microsoft.Owin.Host.HttpListener.RequestProcessing.ExceptionFilterStream.EndWrite(IAsyncResult asyncResult)
   at System.Threading.Tasks.TaskFactory`1.FromAsyncCoreLogic(IAsyncResult iar, Func`2 endFunction, Action`1 endAction, Task`1 promise, Boolean requiresSynchronization)
   --- End of inner exception stack trace ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Web.Http.Owin.HttpMessageHandlerAdapter.<SendResponseContentAsync>d__20.MoveNext()| RequestID = s_89eb2349-7f83-46c2-a42d-e3b213d4448d 
2019-12-12 02:17:25.6205|INFO|15|STN\hermes-admin ::1: GET /api/v1.0/ManagementState - 0:00:00.0841569
Connection dropped| RequestID = s_89eb2349-7f83-46c2-a42d-e3b213d4448d System.Threading.Tasks.TaskCanceledException: A task was canceled.
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Web.Http.Owin.HttpMessageHandlerAdapter.<SendResponseContentAsync>d__20.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Web.Http.Owin.HttpMessageHandlerAdapter.<InvokeCore>d__0.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.BIServer.Management.WebHost.RequestLoggingMiddleWare.<Invoke>d__0.MoveNext()
2019-12-12 02:22:51.3851|ERROR|11|Unhandled error in the Web API. Exception: System.Net.Http.HttpRequestException: Error while copying content to a stream. ---> System.IO.IOException ---> System.Net.HttpListenerException: The I/O operation has been aborted because of either a thread exit or an application request
   at System.Net.HttpResponseStream.EndWrite(IAsyncResult asyncResult)
   at Microsoft.Owin.Host.HttpListener.RequestProcessing.ExceptionFilterStream.EndWrite(IAsyncResult asyncResult)
   --- End of inner exception stack trace ---
   at Microsoft.Owin.Host.HttpListener.RequestProcessing.ExceptionFilterStream.EndWrite(IAsyncResult asyncResult)
   at System.Threading.Tasks.TaskFactory`1.FromAsyncCoreLogic(IAsyncResult iar, Func`2 endFunction, Action`1 endAction, Task`1 promise, Boolean requiresSynchronization)
   --- End of inner exception stack trace ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Web.Http.Owin.HttpMessageHandlerAdapter.<SendResponseContentAsync>d__20.MoveNext()| RequestID = s_4c58e529-e329-44c2-b5f5-674223edde20 
2019-12-12 02:22:51.3851|INFO|11|STN\hermes-admin ::1: GET /api/v1.0/ManagementState - 0:00:00.0724935
Connection dropped| RequestID = s_4c58e529-e329-44c2-b5f5-674223edde20 System.Threading.Tasks.TaskCanceledException: A task was canceled.
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Web.Http.Owin.HttpMessageHandlerAdapter.<SendResponseContentAsync>d__20.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Web.Http.Owin.HttpMessageHandlerAdapter.<InvokeCore>d__0.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.BIServer.Management.WebHost.RequestLoggingMiddleWare.<Invoke>d__0.MoveNext()

Our Reporting Services installation seems fine - reports are running, users can login and browse the reporting website, etc. Just concerned with the error message above. Any ideas what could be going on?

Answer 1

Do you have a scale out deployment? Are you sure all nodes are using the same version of SSRS? Check the logs for all your nodes, the exact version should be listed at the top of the ReportingServicesService log file.