Using a filtered index when setting a variable

Question

I know from other questions and posts that when SQL compiles a query plan it can only use a filtered index if the filtered index is guaranteed to be able to be used every time the query runs. This means you can't use a variable in the where clause because sometimes it might be able to use the filtered index and sometimes not.

One way around this is to use OPTION(RECOMPILE), so that the times it can use it, it'll get the filtered index.

Doing some testing, I found that this query can use a filtered index (note, I'm forcing the index just to prove a point):

SELECT MAX(table1.SomeDateField)
FROM        dbo.table1 WITH(INDEX(MyFilteredIndex))
WHERE       table1.filteredColumn = @variable
OPTION (RECOMPILE)

However, if I want to assign the result to a variable, I'm bang out of luck:

SELECT @OutputVariable = MAX(table1.SomeDateField)
FROM        dbo.table1 WITH(INDEX(MyFilteredIndex))
WHERE       table1.filteredColumn = @variable
OPTION (RECOMPILE)

results in:

Msg 8622, Level 16, State 1, Line 15 Query processor could not produce a query plan because of the hints defined in this query. Resubmit the query without specifying any hints and without using SET FORCEPLAN.

The query can clearly use the filtered index as it runs find when I don't want to save the output to a variable.

I have ways of rewriting this query to hard-code @variable to remove the problem, but can someone explain why the first query can use the filtered index, but the 2nd query can't?

Answer 1

The optimization that allows you to use the filtered index with a RECOMPILE hint is called the "parameter embedding optimization." That's a process where the query parser replaces the variable reference with the literal value inside the variable.

See this post from Paul White for the reason why it doesn't work in your second case: Parameter Sniffing, Embedding, and the RECOMPILE Options

There is one scenario where using OPTION (RECOMPILE) will not result in the parameter embedding optimization being applied. If the statement assigns to a variable, parameter values are not embedded:

Answer 2

This should work:

declare @myVariable int
declare @variable int = 9
declare @OutputVariable date

declare @t table(MaxDate date)

insert into @t(MaxDate) 
SELECT MAX(table1.SomeDateField)
FROM        dbo.table1 WITH(INDEX(MyFilteredIndex))
WHERE       table1.filteredColumn = @variable
OPTION (RECOMPILE)

select @OutputVariable MaxDate from @t

And you can always use dynamic SQL with literal values instead of trying to get a parameter to transform into a literal with OPTION (RECOMPILE). eg

declare @variable int = 9
declare @OutputVariable date

declare @sql nvarchar(max) = concat(N'
SELECT @OutputVariable = MAX(table1.SomeDateField)
FROM        dbo.table1 WITH(INDEX(MyFilteredIndex))
WHERE       table1.filteredColumn = ', @variable)

exec sp_executesql @sql, N'@OutputVariable date out', @OutputVariable = @OutputVariable out
select @OutputVariable

Answer 3

The reason why the optimizer can't generate a plan is because it has no way of guaranteeing that the variable's value will actually match the filter predicate. The query plan is precompiled and stored, and it is not safe for it to use that index in all cases. Unfortunately the optimizer does not at this time contain logic to bifurcate a filtered index only in the case when the filter matches, which is a shame. You can do it yourself, as SQLPro has demonstrated.

One problem with this is: filtered indexes often change, based on what the DBA thinks is best. An option I have successfully used is to cache client-side the filtered index predicate, and dynamically build the query based on that, while still passing through the variable. This avoids issues of plan cache bloat and SQL injection, as the query is still parameterized and does not change unless the index changes. You can also do this with sp_executesql from inside a stored procedure.

To get the filter predicate, run this query:

  SELECT ISNULL(
      (SELECT i.filter_definition
      FROM sys.indexes i
      WHERE i.object_id = OBJECT_ID(@tablename) AND
          i.name = @indexname AND has_filter = 1),
      '(1=1)');

This also handles the case when the filter, or indeed the whole index, is dropped.

Answer 4

You can try this query:

SELECT @OutputVariable =
(SELECT MAX(table1.SomeDateField)
FROM   dbo.table1
WHERE  table1.filteredColumn = @variable
  AND  table1.filteredColumn = the_filtered_value
UNION ALL
SELECT MAX(table1.SomeDateField)
FROM   dbo.table1
WHERE  table1.filteredColumn = @variable
  AND  NOT table1.filteredColumn = the_filtered_value)

The reason this works is because you are bifurcating the filtered index i.e. only using it in a case when the variable matches the filter predicate. The optimizer cannot use the index without this because it can't guarantee the value matches the predicate.