•  07-01-2021
  •  | 
  •  

Question

I have a cursor that generates one record of JSON text from a group of tables. The cursor has been making SSMS crash. The script runs for a time then SSMS fails. Below is the code that I have written that is causing the crash.

DECLARE @ROW_ID int  -- Here we create a variable that will contain the ID of each row.

DECLARE JSON_CURSOR CURSOR   -- Here we prepare the cursor and give the select statement to iterate through
FOR

        SELECT  -- Our select statement (here you can do whatever work you wish)
            ROW_NUMBER() OVER (ORDER BY NAME_2-1,NAME_2-2,FIELD_1-1,FIELD_1-2) AS ROWID
        FROM
            (
            SELECT 
                FIELD_1-1
                ,FIELD_1-2
                ,NAME_1-1
                ,NAME_1-2
            FROM 
                (
                SELECT FIELD_1-1,FIELD_1-2,NAME,VALUE
                FROM TABLE_1
                WHERE NAME IN ('NAME_1-1','NAME_1-2')
                ) AS SRC
            PIVOT
                (
                MAX(VALUE_1) FOR NAME IN ([FIELD_1-1],[FIELD_1-2])
                ) AS PVT
            ) AS T0
        LEFT JOIN TABLE_2 AS [P] ON T0.NAME_1-2=P.NAME_2-2;

OPEN JSON_CURSOR -- This charges the results to memory

FETCH NEXT FROM JSON_CURSOR INTO @ROW_ID -- We fetch the first result

WHILE @@FETCH_STATUS = 0 --If the fetch went well then we go for it
BEGIN

    SELECT * FROM
        (
        SELECT  -- Our select statement (here you can do whatever work you wish)
            FIELD_2-1
            ,FIELD_2-2
            ,FIELD_1-1
            ,FIELD_1-2
            ,T0.NAME_1-1
            ,ROW_NUMBER() OVER (ORDER BY FIELD_2-1,FIELD_2-2,FIELD_1-1,FIELD_1-2) AS ROWID
        FROM
            (
            SELECT 
                FIELD_1-1
                ,FIELD_1-2
                ,NAME_1-1
                ,NAME_1-2
            FROM 
                (
                SELECT FIELD_1-1,FIELD_1-2,NAME,VALUE
                FROM TABLE_1
                WHERE NAME IN ('NAME_1-1','NAME_1-2')
                ) AS SRC
            PIVOT
                (
                MAX(VALUE_1) FOR NAME IN ([FIELD_1-1],[FIELD_1-2])
                ) AS PVT
            ) AS T0
        LEFT JOIN TABLE_2 AS [P] ON T0.NAME_1-2=P.NAME_2-2
        ) AS T1
    WHERE ROWID = @ROW_ID  -- In regards to our latest fetched ID
    order by (FIELD_2-1,FIELD_2-2,FIELD_1-1,FIELD_1-2)
    FOR JSON PATH, ROOT('FIELD_2-1');

FETCH NEXT FROM JSON_CURSOR INTO @ROW_ID -- Once the work is done we fetch the next result

END
-- We arrive here when @@FETCH_STATUS shows there are no more results to treat
CLOSE JSON_CURSOR  
DEALLOCATE JSON_CURSOR -- CLOSE and DEALLOCATE remove the data from memory and clean up the process

From Windows Log:

The following .NET Runtime error happened first:

Error 7/20/2018 2:27:58 PM .NET Runtime 1026 None

Application: Ssms.exe Framework Version: v4.0.30319 Description: The process was terminated due to an unhandled exception. Exception Info: System.ComponentModel.Win32Exception at System.Windows.Forms.NativeWindow.CreateHandle(System.Windows.Forms.CreateParams) at System.Windows.Forms.Control.CreateHandle() at System.Windows.Forms.TextBoxBase.CreateHandle() at System.Windows.Forms.Control.CreateControl(Boolean) at System.Windows.Forms.Control.CreateControl(Boolean) at System.Windows.Forms.Control.CreateControl() at System.Windows.Forms.Control.WmShowWindow(System.Windows.Forms.Message ByRef) at System.Windows.Forms.Control.WndProc(System.Windows.Forms.Message ByRef) at System.Windows.Forms.ScrollableControl.WndProc(System.Windows.Forms.Message ByRef) at System.Windows.Forms.ContainerControl.WndProc(System.Windows.Forms.Message ByRef) at System.Windows.Forms.UpDownBase.WndProc(System.Windows.Forms.Message ByRef) at System.Windows.Forms.Control+ControlNativeWindow.OnMessage(System.Windows.Forms.Message ByRef) at System.Windows.Forms.Control+ControlNativeWindow.WndProc(System.Windows.Forms.Message ByRef) at System.Windows.Forms.NativeWindow.DebuggableCallback(IntPtr, Int32, IntPtr, IntPtr)

Second Application Error:

Error 7/20/2018 2:27:58 PM Application Error 1000 (100)

Faulting application name: Ssms.exe, version: 2017.140.17277.0, time stamp: 0x5b304116 Faulting module name: KERNELBASE.dll, version: 10.0.14393.2189, time stamp: 0x5abda7d6 Exception code: 0xe0434352 Fault offset: 0x000daa12 Faulting process id: 0x3f6c Faulting application start time: 0x01d4205466d2b650 Faulting application path: C:\Program Files (x86)\Microsoft SQL Server\140\Tools\Binn\ManagementStudio\Ssms.exe Faulting module path: C:\WINDOWS\System32\KERNELBASE.dll Report Id: 03b3b0c6-0839-4562-a71f-f5b4fc0a3029 Faulting package full name: Faulting package-relative application ID:

The purpose of this script is to output a single entry in JSON that will be posted to a data service in the cloud. Also in SSMS I am sending the results to the grid.

Here is the full query for the stored procedure that I am creating.

    /****** Object:  StoredProcedure [dbo].[sp_acQ-Zerion_POST_HTTP]    Script Date: 6/15/2018 10:48:28 AM ******/
    SET ANSI_NULLS ON
    GO

    SET QUOTED_IDENTIFIER ON
    GO


    /* FILL IN WITH DB */
    ALTER PROCEDURE [dbo].[sp_acQ-Zerion_POST_HTTP] --@ID varchar(50) 
    AS

    /* define variables */
    Declare @hr int;
    Declare @Object as Int;
    Declare @ResponseText as Varchar(8000);
    Declare @src varchar(255), @desc varchar(255),@status int,@msg varchar(255);

    -------------------------------------------------------------------------------------
    /* Cursor Pt 1 */
    -------------------------------------------------------------------------------------

     DECLARE @ROW_ID int  -- Here we create a variable that will contain the ID of each row.

        DECLARE JSON_CURSOR CURSOR   -- Here we prepare the cursor and give the select statement to iterate through
        FOR

                SELECT  -- Our select statement (here you can do whatever work you wish)
                    ROW_NUMBER() OVER (ORDER BY NAME_2-1,NAME_2-2,FIELD_1-1,FIELD_1-2) AS ROWID
                FROM
                    (
                    SELECT 
                        FIELD_1-1
                        ,FIELD_1-2
                        ,NAME_1-1
                        ,NAME_1-2
                    FROM 
                        (
                        SELECT FIELD_1-1,FIELD_1-2,NAME,VALUE
                        FROM TABLE_1
                        WHERE NAME IN ('NAME_1-1','NAME_1-2')
                        ) AS SRC
                    PIVOT
                        (
                        MAX(VALUE_1) FOR NAME IN ([FIELD_1-1],[FIELD_1-2])
                        ) AS PVT
                    ) AS T0
                LEFT JOIN TABLE_2 AS [P] ON T0.NAME_1-2=P.NAME_2-2
                WHERE NAME_1-1 IN ('True','False');          

        OPEN JSON_CURSOR -- This charges the results to memory

        FETCH NEXT FROM JSON_CURSOR INTO @ROW_ID -- We fetch the first result

        WHILE @@FETCH_STATUS = 0 --If the fetch went well then we go for it
        BEGIN
    -------------------------------------------------------------------------------------
    -------------------------------------------------------------------------------------

    Declare @Records as Varchar(8000)=
        (
    -------------------------------------------------------------------------------------
    /* Cursor Pt 2 */
    -------------------------------------------------------------------------------------

            SELECT * FROM
                (
                SELECT  -- Our select statement (here you can do whatever work you wish)
                    FIELD_2-1
                    ,FIELD_2-2
                    ,FIELD_1-1
                    ,FIELD_1-2
                    ,T0.NAME_1-1
                    ,ROW_NUMBER() OVER (ORDER BY FIELD_2-1,FIELD_2-2,FIELD_1-1,FIELD_1-2) AS ROWID
                FROM
                    (
                    SELECT 
                        FIELD_1-1
                        ,FIELD_1-2
                        ,NAME_1-1
                        ,NAME_1-2
                    FROM 
                        (
                        SELECT FIELD_1-1,FIELD_1-2,NAME,VALUE
                        FROM TABLE_1
                        WHERE NAME IN ('NAME_1-1','NAME_1-2')
                        ) AS SRC
                    PIVOT
                        (
                        MAX(VALUE_1) FOR NAME IN ([FIELD_1-1],[FIELD_1-2])
                        ) AS PVT
                    ) AS T0
                LEFT JOIN TABLE_2 AS [P] ON T0.NAME_1-2=P.NAME_2-2
                WHERE NAME_1-1 IN ('True','False')
                ) AS T1
            WHERE ROWID = @ROW_ID  -- In regards to our latest fetched ID
            order by (FIELD_2-1,FIELD_2-2,FIELD_1-1,FIELD_1-2)
            FOR JSON PATH, ROOT('FIELD_2-1');

    -------------------------------------------------------------------------------------
    -------------------------------------------------------------------------------------
        )

    /* wrap records in JSON object */
    Declare @Body as varchar(8000) = @Records

    /* create XMLHTTP object and send object via HTTP POST */
    Exec @hr=sp_OACreate 'MSXML2.ServerXMLHTTP', @Object OUT;
    if @hr <> 0 begin Raiserror('sp_OACreate MSXML2.ServerXMLHttp.3.0 failed', 16,1) return end


    Exec @hr = sp_OAMethod @Object, 'open', NULL, 'post','https://dataflownode.zerionsoftware.com/domain/solutions/services/webhooks/4b9b0f4b8a4b4387ec1642fdaabec7b400d5c938-7be9d5a63b5cba8ab72cd3410429e2635f68a687', 'false'
    if @hr <>0 begin set @msg = 'sp_OAMethod Open failed' goto eh end

    Exec @hr = sp_OAMethod @Object, 'setRequestHeader', null, 'Content-Type', 'application/json'
    if @hr <>0 begin set @msg = 'sp_OAMethod setRequestHeader failed' goto eh end

    Exec @hr = sp_OAMethod @Object, 'send', null, @Body
    if @hr <>0 begin set @msg = 'sp_OAMethod Send failed' goto eh end

    if @status <> 200 begin set @msg = 'sp_OAMethod http status ' + str(@status) goto eh end

    Exec @hr = sp_OAMethod @Object, 'responseText', @ResponseText OUT--PUT
    Select @ResponseText

    -------------------------------------------------------------------------------------
    /* Cursor Pt 3 */
    -------------------------------------------------------------------------------------

    FETCH NEXT FROM JSON_CURSOR INTO @ROW_ID -- Once the work is done we fetch the next result

    END
    -- We arrive here when @@FETCH_STATUS shows there are no more results to treat
    CLOSE JSON_CURSOR  
    DEALLOCATE JSON_CURSOR -- CLOSE and DEALLOCATE remove the data from memory and clean up the process

    -------------------------------------------------------------------------------------
    -------------------------------------------------------------------------------------

    --if @hr <>0 begin set @msg = 'sp_OAMethod read response failed' goto
    IF @hr <> 0  
    BEGIN  
       EXEC sp_OAGetErrorInfo @object  
        RETURN  
    goto
    eh end

    /* clean-up after data is sent */
    Exec @hr=sp_OADestroy @Object
    return
    eh:
    Raiserror(@msg, 16, 1)
    return

    IF @hr <> 0  
    BEGIN  
       EXEC sp_OAGetErrorInfo @object, @src OUT, @desc OUT   
       raiserror('Error Creating COM Component 0x%x, %s, %s',16,1, @hr, @src, @desc)  
        RETURN  



    END;  
    GO
Was it helpful?

Solution

  1. First, it looks like you are doing a ton of extra work here. The query in the cursor is a subset of the query in the WHILE loop, and the data does not change at any point in time. So it is just executing that same query over and over again, just for a different row. It should be far more efficient to store the results of the initial query into a local temp table and then just use that for both the CURSOR and WHILE loop queries:

    CREATE TABLE #Data
(
  [RowID] INT NOT NULL IDENTITY(1, 1),
  [FIELD_2-1] {data_type},
  [FIELD_2-2] {data_type},
  [FIELD_1-1] {data_type},
  [FIELD_1-2] {data_type},
  [T0.NAME_1-1] {data_type}
);

DECLARE @TotalRows INT;

INSERT INTO #Data ([FIELD_2-1], [FIELD_2-2], [FIELD_1-1], [FIELD_1-2],
                   [T0.NAME_1-1])
  SELECT [FIELD_2-1], [FIELD_2-2], [FIELD_1-1], [FIELD_1-2], [T0.NAME_1-1]
  FROM (
        SELECT  -- Our select statement (here you can do whatever work you wish)
            FIELD_2-1
            ,FIELD_2-2
            ,FIELD_1-1
            ,FIELD_1-2
            ,T0.NAME_1-1
        FROM
            (
            SELECT 
                FIELD_1-1
                ,FIELD_1-2
                ,NAME_1-1
                ,NAME_1-2
            FROM 
                (
                SELECT FIELD_1-1,FIELD_1-2,NAME,VALUE
                FROM TABLE_1
                WHERE NAME IN ('NAME_1-1','NAME_1-2')
                ) AS SRC
            PIVOT
                (
                MAX(VALUE_1) FOR NAME IN ([FIELD_1-1],[FIELD_1-2])
                ) AS PVT
            ) AS T0
        LEFT JOIN TABLE_2 AS [P] ON T0.NAME_1-2=P.NAME_2-2
        ) AS T1
    ORDER BY [FIELD_2-1], [FIELD_2-2], [FIELD_1-1], [FIELD_1-2];

SET @TotalRows = @@ROWCOUNT;

  2. Then, you can get rid of the CURSOR entirely since it is only used to get a total rowcount anyway, and change the WHILE loop to be simple counter.

  3. Finally, putting those two pieces together, the WHILE loop becomes:

    DECLARE @Index INT = 1,
        @Records VARCHAR(8000);

WHILE (@Index <= @TotalRows)
BEGIN

  SET @Records = (

  SELECT [FIELD_2-1], [FIELD_2-2], [FIELD_1-1], [FIELD_1-2], [T0.NAME_1-1], [RowID]
  FROM   #Data
  WHERE  [RowID] = @Index
  FOR JSON PATH, ROOT('FIELD_2-1')
                 );

  SET @Index += 1;
END;

All that being said:

  1. You should probably not be re-declaring the @Records variable each iteration of the loop. Declare it once before the loop and just set it each time within the loop.
  2. You do not need the @Body variable as it does not do anything. You merely set it to the value of @Records, which is a waste of CPU and RAM (and time).
  3. Per each iteration of the loop, you execute sp_OACreate, but you never call sp_OADestroy (or whatever that is). I am guessing that this ends up creating many objects in memory. You might also need to execute sp_OAMethod one more time, just before the Destroy to close out the request. You will need to check around to see if that is the case. You do NOT want to leave open any orphaned network sockets.

  4. While you might actually get this working, using the OLE Automation stored procedures (i.e. sp_OA* ) is rather risky. They have been deprecated since the release of SQL Server 2005, in favor of using SQLCLR instead. There are many advantages to using SQLCLR instead of the OLE Automation procs, including:

    • being able to use NVARCHAR(MAX) instead of being stuck with VARCHAR(8000) and NVARCHAR(4000). In fact, you can even send XML. The OLE Automation procs do not handle any of the datatypes added post SQL Server 2000.
    • better security
    • better memory handling
    • more stable

    You can use the .NET HttpWebRequest class. Or, if you do not want to code anything, a pre-done SQLCLR function exists in SQL# (which I created). The function is INET_GetWebPages and it handles a large variety of scenarios (i.e. you can pass in custom headers, send content as GET or POST, etc). However, just to be clear, INET_GetWebPages is only available in the Full version (i.e. not in the Free version)

  5. Still, it does seem odd that it is SSMS that crashes and not SQL Server itself. If cleaning up the code as suggested above (all except switching to using SQLCLR) does not fix the error, you can always comments out the EXEC sp_OA* statements. Start with commenting out all of the EXEC sp_OA* statements and the IF statement that follows each one. Then, if that gets the proc to no longer error, start uncommenting each EXEC / IF pair of statements one by one (except the first set: if you uncomment the sp_OACreate, then you must pair that with an sp_OADestroy in the same scope!!)

All of those suggestions aside, the actual issue that caused the crash (as we discovered in the chat) ended up being the sheer number of result sets returned to SSMS. There are 2775 rows returned by the query in the first code block of the question. Within the context of the stored procedure, no result sets are returned to the client; all results are stored in a VARCHAR(8000) variable. But in the testing, it is just a straight SELECT. When not dumping each result set to a separate grid, SSMS did not crash.

Also, if you are going to code this yourself, you will run into "issues" when deploying the code if using SQL Server 2017 (or newer), and even if using SQL Server 2016, you will still need to set the Assembly to EXTERNAL_ACCESS which will require signing the Assembly and a few other minor steps. Please see my post here for instructions on how to take care of this, with or without Visual Studio, and in a way that works with SQL Server 2017 (or newer):

SQLCLR vs. SQL Server 2017, Part 2: “CLR strict security” – Solution 1 (the goal being a singular, self-contained installation script that handles the security and the assembly and has no external dependencies for anything, thus making it easy to version and/or transfer between development, testing, and production systems)

OTHER TIPS

To get past the SSMS issue, SET NOCOUNT ON and don't send results to the client.

ServerXMLHTTP.Send, when called from the deprecated sp_OAxxx COM interop stored procedures has an 8000 character limit, so it's not a good choice for this job.

This is trivial to do in a SQL CLR stored procedure, or in many client programming environments, like PowerShell, Python, .NET, etc. All of which can be invoked from SQL Agent jobs.

Also, you're not properly destroying the COM objects you are creating. You call sp_OACreate in the loop, but not sp_OADestroy.

Also you are missing a line when you copied (probably second or third hand) my 15-year old USENET post on how to use ServerXmlHttp from the sp_OAxxx COM interop procedures.

   exec @hr = sp_OAGetProperty @obj, 'status', @status OUT
   if @hr <0 begin  set @msg = 'sp_OAMethod read status failed' goto eh end

All in all, I'd advise you to stop and write this process in a different language.

Here's a sample to get you started of how to post a FOR JSON query to an HTTP endpoint and get the response.

As background, a FOR JSON query is streamed to the client as a one-column, multi-row result set with the JSON broken across the rows. So you can just read through the result rows and post the content to the HTTP endpoint. No matter how large the document, it won't be buffered in SQL.

You call it like this

declare @rc int  = 0
declare @body nvarchar(max) 

exec postjson 'select * from sys.objects for json path', 'http://localhost:51801/api/values',  @rc out, @body out

select @rc, @body

Here's the C# source code

using System;
using System.Data;
using System.Data.SqlClient;
using System.Data.SqlTypes;
using System.Text;
using Microsoft.SqlServer.Server;
using System.Net;
using System.IO;

public partial class StoredProcedures
{
    [Microsoft.SqlServer.Server.SqlProcedure]
    public static void PostJSON(string sqlForJSONQuery, string targetURI, out int responseCode, out string responseBody)
    {
        using (var con = new SqlConnection("Context Connection=true"))
        {
            con.Open();
            var cmd = con.CreateCommand();
            cmd.CommandText = sqlForJSONQuery;

            using (var rdr = cmd.ExecuteReader())
            {
                var req = WebRequest.CreateHttp(targetURI);
                req.Method = "Post";
                req.ContentType = "application/json";
                using (var rs = req.GetRequestStream())
                {

                    while (rdr.Read())
                    {
                        var val = rdr.GetString(0);
                        //SqlContext.Pipe.Send(val);
                        var buf = Encoding.UTF8.GetBytes(val);
                        rs.Write(buf, 0, buf.Length);
                    }
                }
                HttpWebResponse resp;
                try
                {
                    resp = (HttpWebResponse)req.GetResponse();
                }
                catch (WebException ex)
                {
                    resp = (HttpWebResponse)ex.Response;

                }

                responseCode = (int)resp.StatusCode;

                using (var respStream = resp.GetResponseStream())
                {
                    using (var sr = new StreamReader(respStream, Encoding.UTF8))
                    {
                        responseBody = sr.ReadToEnd();
                    }
                }


            }
        }
    }
}

You can build and deploy this using SQL Server Data Tools. This gives you full Visual Studio integration for your SQL Server development, including SQL CLR, covering coding, debugging, deploying, and managing source control.

Or minimally, here's how to do it with nothing but the command line on your SQL Server.

Create a directory on your SQL Server called c:\PostJSON, and in there create PostJSON.cs with the above source code. Then open a command prompt in that folder and run:

PS C:\PostJSON> C:\windows\Microsoft.NET\Framework64\v4.0.30319\csc /out:PostJson.dll /target:library PostJSON.cs

To compile the PostJSON.cs file into PostJSON.dll.

Then from your database run this script to install and test the stored procedure:

drop procedure if exists postjson
if exists (select * from sys.assemblies where name = 'PostJSON')
 drop assembly PostJSON

exec sp_configure 'show advanced options', 1
reconfigure
exec sp_configure 'clr enabled', 1;
reconfigure

go
DECLARE @asmBin varbinary(max) = (
        SELECT BulkColumn 
        FROM OPENROWSET (BULK 'c:\PostJSON\PostJson.dll', SINGLE_BLOB) a
        );

DECLARE @hash varbinary(64);
SELECT @hash = HASHBYTES('SHA2_512', @asmBin);

declare @description nvarchar(4000) = N'PostJSON';

if not exists (select * from sys.trusted_assemblies where hash = @hash)
begin
  EXEC sys.sp_add_trusted_assembly @hash = @hash,
                                   @description = @description;
end

CREATE ASSEMBLY [PostJSON]
    AUTHORIZATION [dbo]
    FROM @asmBin
    WITH PERMISSION_SET = EXTERNAL_ACCESS;  

exec('
CREATE PROCEDURE PostJson  @sqlForJSONQuery nvarchar(max), 
                           @targetURI nvarchar(max), 
                           @responseCode int out, 
                           @responseBody nvarchar(max) out 
AS EXTERNAL NAME PostJSON.StoredProcedures.PostJSON
')


go
--test

declare @rc int
declare @body nvarchar(max)

exec postjson 'select ''Hello world'' msg for json path', 'http:\\bing.com', @rc out, @body out

select @rc, @body
Licensed under: CC-BY-SA with attribution
Not affiliated with dba.stackexchange
scroll top