Which reasons could make ShellExecute fail?

Question

I have a VB6 application which opens files with their associated application using:

ShellExecute(0, "open", filename, params, vbNullString, vbNormalFocus)

This works perfectly.

Now I got a customer (running XP with Adobe Reader) who can't open any PDF file using the above command. But the same file is being opened without any problems when double clicking it from Windows Explorer. I also tested the filename/-path combination on my machine to exclude those kind of problems.

I'm searching for any hints on what I could check to make sure ShellExecute is working. Or what can cause ShellExecute to fail this way?

Answer 1

What's the return value of ShellExecute? If it's 0x0000001f (== 31, meaning SE_ERR_NOASSOC), than according to shellapi.h "There is no application associated with the given file name extension.", which means that somehow the registration of the .pdf file extension got lost. Reinstalling Adobe Reader might help.

Answer 2

Further to Thomas's answer, here's some VB6 constants for possible return values of ShellExecute, with possible explanations (I think I originally took these from the MSDN page, return value section). A return value of 32 or less means the call failed. The specific value returned indicates what went wrong.

Const ERROR_BAD_FORMAT = 11&
Const ERROR_FILE_NOT_FOUND = 2&          
Const ERROR_PATH_NOT_FOUND = 3&          ' The specified path was not found. '
Const SE_ERR_ACCESSDENIED = 5            ' The operating system denied access to the specified file. '
Const SE_ERR_ASSOCINCOMPLETE = 27        ' The file name association is incomplete or invalid. '
Const SE_ERR_DDEBUSY = 30                ' The Dynamic Data Exchange (DDE) transaction could not be completed because other DDE transactions were being processed. '
Const SE_ERR_DDEFAIL = 29                ' The DDE transaction failed. '
Const SE_ERR_DDETIMEOUT = 28             ' The DDE transaction could not be completed because the request timed out. '
Const SE_ERR_DLLNOTFOUND = 32            ' The specified dynamic-link library (DLL) was not found. '
Const SE_ERR_FNF = 2                     ' The specified file was not found. '
Const SE_ERR_NOASSOC = 31                ' There is no application associated with the given file name extension. '
Const SE_ERR_OOM = 8                     '  out of memory '
Const SE_ERR_PNF = 3                     '  path not found '
Const SE_ERR_SHARE = 26                  ' A sharing violation occurred. '

Answer 3

You have "open" as the verb, don't do that, use vbNullString as the verb ("Open" means the open verb, NULL means the default verb (If the user has not set a specific default, the default is open, if there is no open verb for that filetype, ShellExecute uses the first verb it finds))

Answer 4

Have a look at the return value of your ShellExecute call. From the MSDN:

If the function succeeds, it returns a value greater than 32. If the function fails, it returns an error value that indicates the cause of the failure. The return value is cast as an HINSTANCE for backward compatibility with 16-bit Windows applications. It is not a true HINSTANCE, however. It can be cast only to an int and compared to either 32 or the following error codes below.

0: The operating system is out of memory or resources.

ERROR_FILE_NOT_FOUND: The specified file was not found.

ERROR_PATH_NOT_FOUND: The specified path was not found

(...)

Answer 5

Instead of using ShellExecute to 'execute' the PDF file, I use the FindExecutable API:

Private Const ERROR_FILE_NO_ASSOCIATION     As Long = 31
Private Const ERROR_FILE_NOT_FOUND          As Long = 2
Private Const ERROR_PATH_NOT_FOUND          As Long = 3
Private Const ERROR_FILE_SUCCESS            As Long = 32 
Private Const ERROR_BAD_FORMAT              As Long = 11

Private Declare Function FindExecutable Lib "shell32.dll" _
   Alias "FindExecutableA" _
  (ByVal lpFile As String, _
   ByVal lpDirectory As String, _
   ByVal sResult As String) As Long


Private Sub OpenDocument(sFile as string, sPath as string)
     Dim sResult As String
     Dim lSuccess As Long, lPos as long

     sResult = Space$(MAX_PATH)
     lSuccess = FindExecutable(sFile, sPath), sResult)
     Select Case lSuccess
        Case ERROR_FILE_NO_ASSOCIATION
            If Right$(sFile, 3) = "pdf" Then
                MsgBox "You must have a PDF viewer such as Acrobat Reader to view pdf files."
            Else
                MsgBox "There is no registered program to open the selected file." & vbCrLf & sFile
            End If
        Case ERROR_FILE_NOT_FOUND: MsgBox "File not found: " & sFile
        Case ERROR_PATH_NOT_FOUND: MsgBox "Path not found: " & sPath
        Case ERROR_BAD_FORMAT:     MsgBox "Bad format."
        Case Is >= ERROR_FILE_SUCCESS:
           lPos = InStr(sResult, Chr$(0))
           If lPos Then sResult = Left$(sResult, lPos - 1)
           Shell sResult & " " & sPath & sFile, True), vbMaximizedFocus
    End Select

End Sub

Answer 6

If you're using

CoInitializeEx(NULL, COINIT_MULTITHREADED)

in your code, then you will have to create a separate thread for executing via ShellExecute. See more here: Calling Shell Functions and Interfaces from a Multithreaded Apartment

Answer 7

1. Uninstall and reinstall Acrobat Reader.
2. Under "Documents and Settings", rename "username" folder to "usernamex" (you should be logged in with different admin user).
3. Relogin as user and it creates a new "username" folder with a new user registry.
4. Now it should work.

You can copy files from the usernamex folder to the new username folder (Desktop, Documents, etc.)

Answer 8

I've had the same problem and it was not possible to change the VB6-Code. So I had to find another solution...

In my case it was a file with the extension ".xyz", but in reality it was a file for Microsoft Word, like a .doc-file.

When doubleclicking for the first time, Windows asks for a programm to open the file with. After that the doubleclick worked fine. But the ShellExecute didn't. The problem is, that the ShellExecute performs a "right click" -> "open" on the file, and "open" didn't exist in the context menu on my .xyz-file. There was just an "edit"... So the ShellExecute worked with "edit", but not with "open" as the second parameter.

And because I was not able to change the VB6-code, I opened the registry with regedit. In the Path "HKEY_CLASSES_ROOT\.doc" the standard-value was "Word.Document.8", in "HKEY_CLASSES_ROOT\.xyz" there was just "xyz_auto_file". So I changed just this value into "Word.Document.8", and everything worked perfectly. Now I have the same context menu as with a .doc-file, when I right-click on my .xyz-file.

And also the ShellExecute works perfectly...

Answer 9

I had the same issue with an existing program that is using the verb open instead of NULL when calling the ShellExecute function. I was able to fix the problem by adding the open verb like described here using the registry editor into the .pdf handler (mine was at HKEY_CLASSES_ROOT\pdf_auto_file). I think this is a problem within the Adobe Reader installer that is sometimes not adding the open verb during installation.

Here is the export of the registry values I added:

[HKEY_CLASSES_ROOT\pdf_auto_file\shell\Open\command]
@="\"C:\\Program Files\\Adobe\\Reader 11.0\\Reader\\AcroRd32.exe\" \"%1\""

Answer 10

I encountered the same problem as the OP within a compiled Visual Foxpro 9 application after updating from W7x64 to W10 public release.

I have Adobe Acrobat installed as well as Adobe Reader. Changing the default .pdf association from Reader to Acrobat and... It all works! Changing back to Reader with original failure (error code 31 - "There is no application associated with the given file-name extension."). Beats me, but fortunately I don't have to worry. I'm too old to care and will require all sites to stay with W7.

Either association works from file explorer

Answer 11

Calling a Unicode version (ShellExecuteW) from an command which only supports ANSII, experienced with a recent version of Inno Setup. ShellExecuteW worked for some ANSII string arguments but in this case not the required one, returning 2 (see below).
As a matter of interest, in either ANSII or Unicode, Inno's internal function ShellExec also failed with code 5 for the reason the compiling process still had an open handle to the file.

Answer 12

Here's a function that translates a the windows error numbers to text. You can use the return value as the parameter and get back a more friendly message.

Private Declare Function FormatMessage Lib "kernel32" Alias "FormatMessageA" _
    (ByVal dwFlags As Long, lpSource As Long, ByVal dwMessageId As Long, _
    ByVal dwLanguageId As Long, ByVal lpBuffer As String, _
    ByVal nSize As Long, ByVal Arguments As Any) As Long

Private Const FORMAT_MESSAGE_FROM_SYSTEM = &H1000
Private Const FORMAT_MESSAGE_IGNORE_INSERTS = &H200
Private Const MAX_PATH = 260

Function TranslateDLLError(ByVal lngErrNum As Long) As String
   Dim sRtrnCode As String * MAX_PATH
   Dim lRet As Long

   On Error GoTo errTranslateDLLError(

   sRtrnCode = Space$(256)
   lRet = FormatMessage(FORMAT_MESSAGE_FROM_SYSTEM Or FORMAT_MESSAGE_IGNORE_INSERTS, 0&, lngErrNum, 0&, sRtrnCode, Len(sRtrnCode), 0&)
   If lRet > 0 Then
      Translate_DLL_Error = Replace$(Left(sRtrnCode, lRet), vbCrLf, "")
   Else
      Translate_DLL_Error = "Error not found."
   End If

   Exit Function

errTranslateDLLError(:
   TranslateDLLError( = "Unable to translate system error: " & CStr(lngErrNum)

End Function

Answer 13

Try this. You have to associate the PDF file with any program (such as Acrobat x) to read PDF's, then you can open PDF files with ShellExecute.