Can I tell SQL Server not to place data in the buffer cache?

Question

One of the servers I support has an ETL job that runs every morning. This isn't an SSIS package, just a series of stored procedures executed via SQL Server Agent. The job takes several hours to run, and regularly brings the server's page life expectancy down under 100, which isn't surprising as it hits pretty much every table in the main operational database.

Most of the data it's reading isn't needed during regular operational use, so it's just wasting space in the buffer cache until the job is done and the cache can start filling back up with data that's actually useful. Is there any mechanism through which I can tell SQL Server not to replace what's in the buffer cache when running these stored procedures?

Answer 1

Can I tell SQL Server not to place data in the buffer cache?

No. SQL Server must bring pages into memory in order to work with them. There are no configuration options to directly affect how long pages remain in memory, or the various policies used to manage buffer pool usage.

If your process involves changing data, you may be able to influence the impact on the buffer pool to some extent (ha!) by issuing a manual checkpoint. This will force modified (dirty) pages to be written to disk, freeing them up for reuse.

Resource governor currently only supports limiting workspace memory (used by sorts, hashing operations, and parallel buffers). It does not provide a way to limit the use of the buffer pool for cached data and index pages.

Generally, you will have to rely on SQL Server making good decisions around space usage in the buffer pool. The ETL job(s) likely benefit from the current arrangement, so you might well find a negative performance impact were you able to limit its use of the buffer pool.

If you are certain that the vast majority of the buffer pool is being consumed by finished ETL operations, you might find it worthwhile to explicitly flush dirty pages and clear buffer pool at the end of the ETL process:

CHECKPOINT;
DBCC DROPCLEANBUFFERS;

This is usually not indicated for a production system, but you may find it helps in this specific situation. With no dirty pages and an empty buffer pool, SQL Server will begin to acquire pages for the new workload at its maximum rate.

In particular, the buffer pool being lower than the server memory target will enable ramp-up reads - where whole extents are read in whenever a page is requested from persistent storage. This is effective by default in Enterprise Edition and can be enabled for other editions (even Express) using global trace flag 840, see:

FIX: An application query that reads lots of data may be slow, and the SQL Server service may take a long time to start in SQL Server 2005

From that link:

This hotfix exposes trace flag 840. When trace flag 840 is turned on, SQL Server can perform larger I/O extent reads to populate the buffer pool when SQL Server starts. The larger I/O extent reads populate the buffer pool faster. Additionally, the larger I/O extent reads improve the initial query compilation and the response time when SQL Server starts.

Although that text mentions SQL Server starting, it also applies to buffer pool repopulation as I described above.

You can of course also help this process along by running dummy queries against structures you want to bring into memory at the end of the ETL process.

Answer 2

Seems like you could just use resource governor:

https://msdn.microsoft.com/en-us/library/bb933866%28v=sql.110%29.aspx

That way you can just put a heavy memory limit on one user, like the user that runs the agent jobs, or the agent account itself.