Job waiting for resources - is it better to kill it or let it wait

Question

Our server is very busy sometimes and I can see my jobs are waiting for resources. Is it better to kill it and start it after e.g. one hour or let it wait until it finishes?

Right now there is a rule that kills the job if it has not finished within a specified amount of time and then it tries this four times. I think it is best to let it wait, because when it gets killed it looses its place in the "line" and when it starts again it has to go to the end of the queue.

Extra info: the jobs are from ETL with analytical queries and the big jobs have many joins on large tables. Tableau also sends queries directly to the server...

Thanks

Answer 1

The best option is to resolve the resource contention when possible. Based on your question, you seem to be seeing waits for memory grants. In the comments, you stated that your server has 4 TB of memory. By default, SQL Server will allow a single query to get a grant up to 750 GB. Are there any queries that really need that much memory? Keep the following in mind:

• By design, SQL Server asks for a generous amount of memory for its memory grants.
• Queries may ask for extremely large grants if the queries have poor cardinality estimates. I've seen queries ask for more than a TB of memory when less than 20 GB was sufficient.
• Even if a query gets a huge memory grant it may still spill to tempdb. This is because a query's memory grant need to be shared between different operators and the split is determined before the query runs.

I suggest looking into the queries that request the most memory and using Resource Governor or the MAX_GRANT_PERCENT hint to reduce the size of the largest grants. This is a trade off. You may see worse performance for some queries if they start spilling more data to tempdb. However, you may be able to reduce the grants without degrading performance at all. It all depends on the queries that run on your system and how many are running at a time.