Subscription Initialization throws Error “The option ”INLINE=ON“ is not valid for this function” which is incorrect

Question

Scenario We are upgrading our SQL estate to SQL Server 2019 and have setup what will be the production environment.

From: SQL2008R2 Standard/Windows Server 2008R2 - Primary Server/Secondary Server (Windows Fail-over Cluster) & Reports Server (Transactional Replication).

To: SQL2019 Standard/Windows Server 2019 - Primary Server/Secondary Server (Always-On) & Reports Server (Transactional Replication)

Backups from 2008R2 were restored on 2019 and compatibility level set to latest. Always-On setup has gone relatively smooth and initial testing against current application has shown no issues with compatibility.

The issue is purely with setting up Transactional Replication. There are 2 databases that require replication, one publication each with the primary as the publisher and distributor.

The Issue During the initialization of the subscription for each DB on the reports server it runs fine until it gets to the point of creating functions and produces the errors below.

1st Database:

Message: The option "INLINE=ON" is not valid for this function. Check the documentation for the constructs supported with INLINE option in a function. Command Text: CREATE FUNCTION [dbo].[f_clienttels - Mirror Copy ce2d3663eb494f3589bd5000dad1bf1f](@ClientID [int]) RETURNS varchar WITH INLINE = ON, EXECUTE AS CALLER AS BEGIN ......

2nd Database:

Message: An invalid option was specified for the statement "CREATE/ALTER FUNCTION". Incorrect syntax near the keyword 'with'. If this statement is a common table expression, an xmlnamespaces clause or a change tracking context clause, the previous statement must be terminated with a semicolon. Incorrect syntax near ')'. Command Text: CREATE FUNCTION [dbo].[GetGroupAndDescendantGroupsSelective - Mirror Copy 46f329d5eed444428f45b052f07c7ea8](@GroupId [int]) RETURNS TABLE WITH INLINE = ON AS RETURN ( WITH GroupsCTE AS ( ........

They are both different errors but i believe both are to do with the "INLINE=ON" option, this option is not present in either of those functions, none of our functions use the inline option explicitly, if you remove those articles from the subscription it just gives the same error on the next function (CTE error if the function starts with a CTE and the INLINE=ON error if it does not).

So it appears replication is inserting "WITH INLINE = ON" to functions prior to replicating and then erroring on the addition it has made.

I have patched all instances to the latest CU4 update 15.0.4033.1, tested those functions on both servers (which work), validated all functions/procs within the database and all are fine. In the current 2008R2 environment I had to recreate the publication a couple months back and didn't get these errors. As a work around for now I'm manually creating functions at the subscriber and removing all function articles from the publication.

Any help with a resolution to this would be greatly appreciated, the only reference to this error (from 1st DB) is mentioning that it's undocumented (link below) and i could not find any other forum posts mentioning it.

16203 – The option “INLINE=ON” is not valid for this function. Check the documentation for the constructs supported with INLINE option in a function.

From: Brent Ozar - What’s New in SQL Server 2019’s sys.messages: More Unannounced Features

We don't have an active support contract with Microsoft but are trying to reach them via the supplier that provided the licences so I will give an update here if they get back to us.

I can provide more information if needed.

Answer 1

Apologies about the delay in posting these findings, I spent over a month going back and forwards with Microsoft support regarding this (at one point they told me that replication is not designed to replicate functions and stored procedures and to just not replicate them as a fix?!!), they have eventually conceded that it is indeed a bug, after i sent them a load of evidence from experiments my end. The bug is actually with the database restore/upgrade process in SQL Engine and not transactional replication itself as it first appeared.

Cause/Investigation

The root cause of this is that when a database is restored from a previous version of SQL (2008R2 in my case) onto a new SQL 2019 instance, by default it appears to flag the functions with inline_type = 1 in sys.sqlmodules (this manifests as "Marked for inlining" in the function properties in the GUI). There is a flag in that table called inlinable which was a bit of a red herring when initially investigating, as it appears replication is using inline_type instead, inlinable remains true even after the workaround and does not hinder replication.

I arrived at this conclusion via the following steps:

• Restore Database From 2008R2 and set compatibility to 150: 100+ functions were restored having inline_type = 1 and inlinable = 1. Thus when replication is initiated it eventually fails on these functions with ...

16203 – The option “INLINE=ON” is not valid for this function.

This option is applied automatically by replication because of the inline_type flag.

OR

An invalid option was specified for the statement "CREATE/ALTER FUNCTION". Incorrect syntax near the keyword 'with'.

The reason for the 2nd error is because its automatically declaring INLINE = ON in the replicated code, causing the syntax to be invalid when the function contains a CTE.

• Clone The Restored database: Same result as above
• Set the database compatibility level to 140/130/120/110 (pre inlining changes in 2019): Same result as above.
• Restoring and upgrading database on replication subscriber first: Same result as above.
• Manually setting INLINE = OFF: does not fix the issue and is also reported as invalid syntax on some of the functions.
• ALTER DATABASE SCOPED CONFIGURATION SET TSQL_SCALAR_UDF_INLINING = Off: Same result as above.
• Creating the subscription as Pull rather than Push: Same result as above.
• Dropping one of the functions and recreating: The recreated function succeeds and it fails on the next one.
• Recreating the database entirely from generated scripts: Works with no errors in replication.

Workaround

The only viable workaround for this is to recreate all the functions post restore/upgrade so their flags are set correctly. Note: not all functions are incorrectly flagged, some legitimately should be flagged with inline_type = 1, the workaround will set these correctly so it doesn't hurt if they're caught in the below script.

I did this via a quick and dirty PowerShell (below), some of the Microsoft articles of similar issues suggest sp_refreshsqlmodule this did not work for me in the first instance, but I can't see any logical reason why not, so its worth a try in the first instance instead of this.

#Recreate all function to fix inlining issue
#BACKUP DATABASE FIRST! Small Potential for dropped functions to not recreate!
$SQLServer = "YOURSERVER\Instance"
$SQLDatabase = "YourDatabase"
$Constring = "server='$SQLServer';database='$SQLDatabase';trusted_connection=true;"
$GetFunctionsSQL = "SELECT m.definition,o.name  FROM sys.sql_modules m INNER JOIN 
sys.objects o ON m.object_id = o.object_id WHERE m.inline_type = 1 AND o.type IN ('FN','IF','TF')"
$con = New-Object "System.Data.SqlClient.SqlConnection"
$con.ConnectionString = $Constring
$cmd = New-Object "System.Data.SqlClient.SqlCommand"
$cmd.Connection = $con
$cmd.CommandText = $GetFunctionsSQL
$sda = New-Object "System.Data.SqlClient.SqlDataAdapter"
$sda.SelectCommand = $cmd
$dt = New-Object System.Data.Datatable
Try {
   $sda.Fill($dt)    >> $null
}
catch {
   Write-Error "$_"
}
foreach ($r in $dt.Rows) {
   $nme = $r["name"]
   $def = $r["definition"]
   Write-Host "Processing $nme"
   $con = New-Object "System.Data.SqlClient.SqlConnection"
   $con.ConnectionString = $Constring
   $cmd = New-Object "System.Data.SqlClient.SqlCommand"
   $cmd.Connection = $con
   $cmd.CommandText = "drop function $nme"
   Try {
      $cmd.Connection.Open()
      $res = $cmd.ExecuteNonQuery()
   }
   catch {
      Write-Error "$_"
   }
   if ($res -eq "-1") {
      Write-Host "Reacreating..."
      $cmd.Connection.Close()
      $cmd.Connection = $con
      $cmd.CommandText = "$def"
      Try {
         $cmd.Connection.Open()
         $res = $cmd.ExecuteNonQuery()
      }
      catch {
         Write-Error "$_"
      }
      $cmd.Connection.Close()
   }
}

Final Resolution

The root cause will have to be fixed within the SQL engine/database restore code to actually resolve this exact scenario. Microsoft have confirmed the fix for this will be implemented in The SQL Server 2019 CU6 Update.

Finally we are able to raise a Hotfix request for this issue which is coming in CU6. This may be going to release in July or Aug 2020.

Not sure i agree with it being called a Hotfix when you have to wait 3 months for it!

Hope this helps someone else and also serve as a lesson that when you contact Microsoft support, be prepared to do a lot of your own investigation unprompted and send it over to them, it really helped speed up the process.

Answer 2

Refer this article The only solution worked for me was drop and recreate the UDF after upgrading to SQL 2019.

The inline_Type column from Sys.sql_modules will be 1 after your upgrade to SQL 2019. After you drop and recreate the UDF, the Inline_type will be 0 and the replication initialization works fine.

Answer 3

I encountered this issue while migrating a publication that includes some scalar UDFs from SQL 2016 to SQL 2019. Changing the database settings on the publisher database did not resolve the issue. And since the subscribers include older versions of SQL Server, adding "WITH INLINE=OFF" is not an option in the function definition. So my solution is to force the udf to not be inlineable by adding the following code within the function definition:

--Junk code to force is_inlineable to 0 
DECLARE @dummy DATETIME; SELECT @dummy = GETDATE()

If you do a select on sys.sql_modules, is_inlineable will show as 0, and the snapshot scripts will generate successfully.

Answer 4

As a follow up on this, upgrading to SQL Server 2019 CU9 removes this workaround.
Any function of the format: Create function as return table select 1 x

Is created with inline_type = 1 and then replication throws the above error.

We had to roll back CU9 due to this feature.