Slow select Queries on Log shipping standby server

dba.stackexchange https://dba.stackexchange.com/questions/240294

Question

We have a database in primary DC around 15 TB with partitioned tables with largest one around 7 TB.

To avoid users from reporting a LS set up was done for above database in different DC where the LS restore job runs once nightly.

on Primay- update stats and update stats AYNC are both set to True and hence reflected the same on secondary. I am not sure why someone kept both enabled and is that how it should be?

Now on secondary, i see lots of blocking occurring when a SELECT query runs from users results in select statman statement for auto update stats for that read only database.

I am not able to get why there would be one when auto update async is enabled?

Also, will the update stats done on primary (once a week) be restored during LS restore? if not, whats the better way to run update stats on secondary database involved in LS standby mode, may be to run daily for better select performance out there?

Please advise

Was it helpful?

Solution

Statistics in read only databases.

TL;DR

In Read only databases, regular permanent statistics can be leveraged to satisfy query plans. If sql server needs new statistics for the read only db or these permanent statistics in the database have become stale, then temporary statistics can be created / updated. These statistics reside in TempDB and are managed by SQL Server (you can only drop them).

There are two kinds of temporary statistics, these that get created due to missing statistics and those that get 'updated'.

The creation of temporary statistics can be removed by adding the statistics manually on the primary database, or generating the estimated execution plans of the reporting queries (also on the primary instance/db). More on that further in this answer.

Permanent statistics can be updated and 'transformed' to temporary statistics on a read only database.

The temporary stat updates can be resolved by either updating your statistics on the primary db more often or disabling the auto stat updates on the read only db after log restores are done.

To disable the auto update stats you could execute this on the read only database: ALTER DATABASE [Database] SET AUTO_UPDATE_STATISTICS OFF; This will stop the updates of temporary statistics on this Read only database.

Another important part regarding Restores to a standby database & temporary statistics is that when applying log backups, the temporary statistics get updated again, even though they still exist in sys.stats.

In your case explaining the daily issues with Statman queries (recalculating the temp statistics each day if they need to be created / updated).

Norecovery --> Standby --> Norecovery ... removes temporary statistics

Another interesting part regarding temporary statistics is that they will be gone when the db state changes to restoring using RESTORE DATABASE ... WITH NORECOVERY.

use MASTER
GO
RESTORE DATABASE [ReadOnly2] with NORECOVERY
RESTORE DATABASE [ReadOnly2] WITH STANDBY = 'D:\temp\ReadOnly_Standby.bak'

Effectively flushing the temporary statistics on all objects 

SELECT * From sys.stats where is_temporary = 1;

And recalculating the same 2 stats between each state change + test query running.

enter image description here

Both of these permanent statistics show up in the xml of our execution plan

   <StatisticsInfo Database="[ReadOnly2]" Schema="[dbo]" Table="[Bla]" Statistics="[IX_Bla_indexedval]" ModificationCount="12000000" SamplingPercent="15.8812" LastUpdate="2019-06-12T10:52:32.25" />
      <StatisticsInfo Database="[ReadOnly2]" Schema="[dbo]" Table="[Bla]" Statistics="[PK__Bla__3214EC075017BD54]" ModificationCount="12000000" SamplingPercent="15.2345" LastUpdate="2019-06-12T10:52:35.34" />

With the modificationcount, samplingpercent & lastupdate changed after executing the query again with the temporary statistics 'updated'.

ModificationCount="0" SamplingPercent="5.71018" LastUpdate="2019-06-13T11:32:36.5"

Creation of temporary statistics

Regular, non temporary statistics do not get updated / you cannot update (not even temporary) statistics on read only databases.

What you are seeing on the reporting instance is the creation / 'updating' of temporary statistics.

These statistics reside in TempDB & SQL Server Creates & Updates them.

Replicating the behaviour

I was able to replicate the behaviour of temporary statistics creation on a table with 100M rows in a read only database (Addendum #1) enter image description here

With the troublesome StatMan Queries you mentioned.

SELECT StatMan([SC0], [SB0000]) FROM (SELECT TOP 100 PERCENT [SC0], step_direction([SC0]) over (order by NULL) AS [SB0000]  FROM (SELECT [NonIndexedVal] AS [SC0] FROM [dbo].[Bla] TABLESAMPLE SYSTEM (7.707678e-001 PERCENT) WITH (READUNCOMMITTED) ) AS _MS_UPDSTATS_TBL_HELPER ORDER BY [SC0], [SB0000] ) AS _MS_UPDSTATS_TBL  OPTION (MAXDOP 16)

Maxdop 16 (max cores) due to me having MAXDOP set as 0 on my test machine, YMMV

Restarting the instance

When I restart the instance I see the same behaviour, showing that these are in fact, temporary statistics. Some Q&A on temporary statistics can be found here.

Now what we are seeing is the creation of temporary statistics and they get created before the query runs.

For big tables, they do note that sample rate is still applied to temporary statistics.

Other point to note is that statistics created as part of auto-stats use data sampling so the creation of these statistics is fast and does not depend on the size of the table

Source

How could you resolve the creation of the statistics when sql server restarts / the database is restored?

If it is possible to do so, you could script them out and create them on the main 'primary' database.

Finding the temporary statistics

SELECT OBJECT_ID, name, auto_created,
user_created, is_temporary
FROM sys.stats
WHERE is_temporary = 1;

Scripting the statistics

Without T-SQL Or by using the T-SQL answer on the question How to script statistics in Sql Server? (using T-SQL) By Martin Smith

Which could solve your main issue with the statistics being created time and time again.

Another idea if the issue is due to temp stats being created & you cannot script them out correctly, could be creating the estimated execution plans for the reporting queries on the primary database. This should create the statistics needed when auto create statistics = on.

Temporary statistic updates

The other problem that could arise are stale permanent statistics. As noted in this aforementioned blog, stale permanent statistics can be updated and set to is_temporary=1.

This means that permanent statistics on a read only database can become temporary statistics until the instance is restarted. When you update the statistics on the primary it should get carried over to the secondary when the log is applied.

Asynchronous temporary statistic updates

We see the asynchronous statistic updates also working for these temporary statistics!

After running addendum #1, we run the next snippet:

   USE MASTER
GO
ALTER DATABASE [ReadOnly] SET  READ_WRITE;

ALTER DATABASE [ReadOnly] SET AUTO_UPDATE_STATISTICS OFF 

USE [ReadOnly]
GO
INSERT INTO dbo.Bla WITH(TABLOCK)(Indexedval,NonIndexedVal) 
SELECT TOP(10000000) --10M
        ROW_NUMBER() OVER(ORDER BY (SELECT NULL)) as rownum1,
        ROW_NUMBER() OVER(ORDER BY (SELECT NULL)) as rownum2
FROM master..spt_values spt1
CROSS JOIN master..spt_values spt2
CROSS JOIN master..spt_values spt3;

ALTER DATABASE [ReadOnly] SET AUTO_UPDATE_STATISTICS ON 

USE MASTER
GO
ALTER DATABASE [ReadOnly] SET  READ_ONLY;
SELECT Indexedval 
FROM dbo.Bla
WHERE  Indexedval =999999
AND 1= (SELECT 1);

The above SELECT runs instantly.

After running the query, the aftermath is shown.

enter image description here

The temp statistic updates run after query execution.

SELECT StatMan([SC0]) FROM (SELECT TOP 100 PERCENT [Indexedval] AS [SC0] FROM [dbo].[Bla] WITH (READUNCOMMITTED)  ORDER BY [SC0] ) AS _MS_UPDSTATS_TBL  OPTION (MAXDOP 1)

Disabling auto update statistics on read only databases

You could disable the updating of permanent statistics to temporary statistics by running the next statement, you can change this setting on a read only database and it will still work.

ALTER DATABASE [ReadOnly] SET AUTO_UPDATE_STATISTICS OFF;

Solution for the temporary stat upgrades.

Also, will the update stats done on primary (once a week) be restored during LS restore?

Updating your stats will be brought over to the secondary, updating them more where possible will result in less stale statistics and less stale permanent statistics being converted to temporary statistics.

Looking into both of these would be my go to in tackling this problem.

Disabling temporary statistics by using a traceflag

When massively enabling traceflags to help a colleague and digging deeper between 2 and 3 thousand, it appears that traceflag 2362 can be used to disable temporary statistics.

You can enable them like this:

DBCC TRACEON(2362,-1);

And all newly temp stats will not be created. Existing temporary statistics will remain until they are removed. For example by setting the db offline and online again.

Applying log backups to a standby database & Temporary statistics

Addendum 2

When running the queries in addendum #2, and applying log backups to the standby database the stats get updated after each restore.

Even after applying an 'empty' log backup.

In addendum 2, between each log backup restore the following query runs:

SELECT Indexedval 
FROM dbo.Bla
WHERE  Indexedval =999999
AND 1= (SELECT 1);

These trigger temp stat updates each time.

Proof

enter image description here enter image description here

All this means that applying logs at night will make the temp stat updates run each day, without a restart of the instance.

Resolving this issue

  • set auto update stats off on the standby db after applying logs. ALTER DATABASE [ReadOnly2] SET AUTO_UPDATE_STATISTICS OFF;
  • Run your reporting queries when the restores finish to create the temp stats
  • Update the statistics on your primary database more often before applying the logs.
  • Enabling traceflag 2362

When restoring a log backup they still exist:

SELECT name, is_temporary From sys.stats where is_temporary = 1;

name    is_temporary
PK__Bla__3214EC075017BD54   1
IX_Bla_indexedval   1

But they do get recalculated when rerunning the query. enter image description here

Addendum #1 (Table with 100M records in a read only db.)

CREATE DATABASE [ReadOnly]
 CONTAINMENT = NONE
 ON  PRIMARY 
( NAME = N'ReadOnly', FILENAME = N'C:\Program Files\Microsoft SQL Server\MSSQL14.SQL02\MSSQL\DATA\ReadOnly.mdf' , SIZE = 8192KB , FILEGROWTH = 65536KB )
 LOG ON 
( NAME = N'ReadOnly_log', FILENAME = N'C:\Program Files\Microsoft SQL Server\MSSQL14.SQL02\MSSQL\DATA\ReadOnly_log.ldf' , SIZE = 8192KB , FILEGROWTH = 65536KB )
GO
ALTER DATABASE [ReadOnly] SET COMPATIBILITY_LEVEL = 140
ALTER DATABASE [ReadOnly] SET AUTO_CREATE_STATISTICS ON(INCREMENTAL = OFF)
ALTER DATABASE [ReadOnly] SET AUTO_UPDATE_STATISTICS ON 
ALTER DATABASE [ReadOnly] SET AUTO_UPDATE_STATISTICS_ASYNC ON 
ALTER DATABASE [ReadOnly] SET  READ_WRITE 
ALTER DATABASE [ReadOnly] SET RECOVERY SIMPLE 
ALTER DATABASE [ReadOnly] SET  MULTI_USER 
ALTER DATABASE [ReadOnly] SET PAGE_VERIFY CHECKSUM  

USE [ReadOnly]
GO
CREATE TABLE dbo.Bla(Id INT IDENTITY(1,1) PRIMARY KEY NOT NULL, Indexedval INT,NonIndexedVal INT);
CREATE INDEX IX_Bla_indexedval on dbo.Bla(Indexedval);

INSERT INTO dbo.Bla WITH(TABLOCK)(Indexedval,NonIndexedVal) 
SELECT TOP(10000000) --10M
        ROW_NUMBER() OVER(ORDER BY (SELECT NULL)) as rownum1,
        ROW_NUMBER() OVER(ORDER BY (SELECT NULL)) as rownum2
FROM master..spt_values spt1
CROSS JOIN master..spt_values spt2
CROSS JOIN master..spt_values spt3;
GO 10

USE MASTER
GO
ALTER DATABASE [ReadOnly] SET  READ_ONLY;

USE [ReadOnly]
GO
SELECT NonIndexedVal 
FROM dbo.Bla
WHERE  NonIndexedVal = 999999;

#Addendum 2 

ALTER DATABASE [ReadOnly] SET  READ_WRITE;

ALTER DATABASE [ReadOnly] SET RECOVERY FULL

BACKUP DATABASE [ReadOnly] to disk = 'D:\temp\ReadOnly.bak'
WITH COMPRESSION, STATS=5


RESTORE FILELISTONLY FROM DISK = 'D:\temp\ReadOnly.bak'

RESTORE DATABASE [ReadOnly2] FROM disk = 'D:\temp\ReadOnly.bak'
WITH MOVE  'ReadOnly' to 'D:\temp\ReadOnly2.mdf'
,MOVE 'ReadOnly_log' to 'F:\temp\ReadOnly_log2.ldf'
, STANDBY = 'D:\temp\ReadOnly_Standby.bak'





USE [ReadOnly2]
GO
ALTER DATABASE [ReadOnly2] SET AUTO_UPDATE_STATISTICS ON

SELECT Indexedval 
FROM dbo.Bla
WHERE  Indexedval =999999
AND 1= (SELECT 1); 


USE [ReadOnly]

INSERT INTO dbo.Bla WITH(TABLOCK)(Indexedval,NonIndexedVal) 
SELECT TOP(2000000) --2M
        ROW_NUMBER() OVER(ORDER BY (SELECT NULL)) as rownum1,
        ROW_NUMBER() OVER(ORDER BY (SELECT NULL)) as rownum2
FROM master..spt_values spt1
CROSS JOIN master..spt_values spt2
CROSS JOIN master..spt_values spt3;


USE MASTER
GO
BACKUP LOG [ReadOnly] to disk = 'D:\temp\ReadOnlyLog.trn'
WITH COMPRESSION, STATS=5


RESTORE LOG [ReadOnly2] FROM DISK='D:\temp\ReadOnlyLog.trn'
WITH STANDBY = 'D:\temp\ReadOnly_Standby.bak'

USE [ReadOnly2]
SELECT Indexedval 
FROM dbo.Bla
WHERE  Indexedval =999999
AND 1= (SELECT 1); 

BACKUP LOG [ReadOnly] to disk = 'D:\temp\ReadOnlyLog2.trn'
WITH COMPRESSION, STATS=5


RESTORE LOG [ReadOnly2] FROM DISK='D:\temp\ReadOnlyLog2.trn'
WITH STANDBY = 'D:\temp\ReadOnly_Standby.bak'

USE [ReadOnly2]
SELECT Indexedval 
FROM dbo.Bla
WHERE  Indexedval =999999
AND 1= (SELECT 1); 
SELECT * From sys.stats where is_temporary = 1

OTHER TIPS

A secondary database server was set up for reporting so that the production server wasn't dealing with that traffic, great!

  1. Auto-updating statistics is still a debate-ridden topic. (Keep reading)
  2. If you manually update the statistics after your nightly log shipping/restore/update process, then you may be able to turn both of those options off. If your data changes throughout the day, I would leave auto-update on, but turn off asynchronous.

I've learned from a few of the lurkers here, that have blogs elsewhere, that the asynchronous setting is only "useful" when auto-updating stats will delay queries longer than it's worth spending the time to update. MSSQL will then try to run the stats update whenever is most convenient for the load of the server, so that it's not as intrusive. Mind you, this means that bad stats may be used in queries until MSSQL decides to complete them.

Licensed under: CC-BY-SA with attribution
Not affiliated with dba.stackexchange
scroll top