SQL Server xp_cmdshell error 5

Question

When I execute the following statement in SQL Server 2008:

declare @s varchar(100)
set @s='dir'
exec sp_configure 'show advanced options',1
reconfigure
exec sp_configure 'xp_cmdshell',1
reconfigure
exec xp_cmdshell @s

It gives the following error:

execute xp_cmdshell error.invoke 'CreateProcess' failed，error code: '5'.

How can I fix this? I enabled the xp_cmdshell procedure, and I have given the folder read-write privileges.

I searched on the internet and tips to add domain user to the Administrator Group. But I still do not understand the principle. I would appreciate any insights. Thanks!

Answer 1

Error code 5 means - access denied.

Please check if account which used for MSSQL service has right on command that you trying to run.

Details: http://social.msdn.microsoft.com/Forums/sqlserver/en-US/bbdffb3c-6604-4011-ac5f-5ea8ecb1a32c/xpcmdshell-a-call-to-createprocess-failed-with-error-code-5-under-sql2005-rtm?forum=sqlsecurity

Answer 2

While this is likely very easy for many folks, I wasted an enormous amount of time doing this ... and the gotcha wasn't obvious ... hopefully this helps someone else.

(Tested on SQL2012)

Step 1: Actually enable the xp_cmdshell feature in SQL Server

            USE master
            Go
            EXEC sp_configure 'show advanced options', 1
            Go
            RECONFIGURE
            Go
            EXEC sp_configure 'xp_cmdshell', 1 
            GO 
            RECONFIGURE 
            GO
            EXEC sp_configure 'show advanced options', 0
            Go
            RECONFIGURE
            Go

Step 2: Run a dos "whoami /all" command using xp_cmdshell and our two service stop commands

            EXEC xp_cmdshell 'whoami /all'
            exec xp_cmdshell 'net stop MR2012ApplicationService';
            exec xp_cmdshell 'net stop MR2012ProcessService';

Among other bits, this will show my execution context is "nt authority\network service S-1-5-20". The account shown has to have permissions to the service being restarted. The other two commands, both attempting to stop services fail due to permissions:

            System error 5 has occurred.
            Access is denied.

Step 3: Using SUBINACL, let's look at the permissions on the service before we start. Because this is where I got in trouble, lets make sure we know the full path to SUBINACL.exe.

            where SUBINACL
            SUBINACL.exe /SERVICE MR2012ProcessService /DISPLAY

If you had the same problem I had, you will see the path to SUBINACL is wherever you might have previously extracted "Windows Server 2003 Resource Kit Tools" (this is my development machine) and is version 4.0.1.1604. That said, observe that "nt authority\network service" doesn't have permissions. Roger? Roger!

Step 4: Let's grant those permissions!

            SUBINACL.exe /SERVICE MR2012ApplicationService /GRANT="nt authority\network service"=LQSTOP
            SUBINACL.exe /SERVICE MR2012ProcessService /GRANT="nt authority\network service"=LQSTOP

Step 5: Pay close attention to the output of those two commands! Doesn't look like they did much, did it?

            +SERVICE MR2012ApplicationService
            /GRANT=nt authority\network service=LQSTOP

            Elapsed Time: 00 00:00:00
            Done:        0, Modified        0, Failed        0, Syntax errors        0

Hmmm. Well, since we looked at permissions BEFORE we started, let's see what they look like now:

            SUBINACL.exe /SERVICE MR2012ProcessService /DISPLAY

Well, that is lame. It didn't work! Well, as it turns out, I wasted a HUGE amount of time trying all sorts of crazy business and the root of the problem is a known bug in SUBINACL!

Step 6: Let's go get the newest version and install it to C:\Dev\Tools:

            https://www.microsoft.com/en-us/download/details.aspx?id=23510

After the install, let's check the path/version to the exe again.

            where SUBINACL

Hmm, it is the same version, but there is a SUBINACL.exe.old now.

Step 7: Let's try those permission grants again! Output should now look like this:

            MR2012ApplicationService : new ace for DOMAIN\MyProxyAccount
            MR2012ApplicationService : 1 change(s)

            Elapsed Time: 00 00:00:00
            Done:        1, Modified        1, Failed        0, Syntax errors        0
            Last Done  : MR2012ApplicationService

There is one more wrinkle; the execution account will be the service account of sql server (i.e. nt authority\network service). However, if this is not desired, a proxy can be setup.

Step 8: Set up a proxy account has to be set up like this:

            EXEC sp_xp_cmdshell_proxy_account 'DOMAIN\MyProxyAccount', '$ome$uper$ecureP@ssword';

To be clear, this isn't my set up, just my understanding of how it works.

Step 9: And of course, this proxy account has to have permissions:

            C:\Dev\Tools\SUBINACL.exe /SERVICE MR2012ApplicationService /GRANT=DOMAIN\MyProxyAccount=LQSTOP
            C:\Dev\Tools\SUBINACL.exe /SERVICE MR2012ProcessService /GRANT=DOMAIN\MyProxyAccount=LQSTOP

Also, the service account will need "Logon as a batch job" in local policies.

Regards,

Stuart

Answer 3

You can do this from SQL Server Management Studio as follows:

Right-click the server, and choose Facets Select Facet Surface Area Configuration Set property XPCmdShellEnabled to True then create proxy account