Problems triggering dump on first-chance OutOfMemoryException

Question

I came across the same error on two machines. Both were equipped with .NET 4.5, so I think this is the reason it did not work.

It turned out to be a bug in DebugDiag Scripts. It fails to retrieve the name of the CLR exception in one of it's scripts. However, this problem is fixable, since every DebugDiag rule creates a vbs script, which can be modified. It uses the very same commands, you can also use in WinDbg. Here is how to fix it:

Create your DebugDiag rule using the GUI, as you already did.
For every rule, there is a vbs script file generated. Open it in a text editor under C:\Program Files\DebugDiag\Scripts\CrashRule_<rulename>.vbs
Find the function GetCLRExceptionType

Modify it as follows:

Function GetCLRExceptionType(ByVal ExceptionObjHexAddr, ByVal bInnerException)
    Dim Output, Lines, i

    If Debugger.IsClrExtensionMissing Then
        WriteToLog "Unable to determine CLR exception type - extension dll could not be loaded."
    Else
        ' Output = Debugger.Execute("!DumpObj " & ExceptionObjHexAddr) ' Does not work in .NET 4.5
        Output = Debugger.Execute("!pe") ' FIX .NET45

        Lines = Split(Output, Chr(10))
        For i = 0 To UBound(Lines)      
            If bInnerException Then
                If InStr(Lines(i), "_innerException") <> 0 Then
                    Tokens = Split(Lines(i), " ")
                    For j = 0 To UBound(Tokens)
                        If Len(Tokens(j)) = 8 Then                              
                            GetCLRExceptionType = GetCLRExceptionType(Tokens(j), False)
                            Exit For
                        End If
                    Next
                End If
            ElseIf Len(Lines(i)) >= 7 Then
                If InStr(Lines(i), "Exception type:") = 1 Then  ' FIX .NET45
                    GetCLRExceptionType = Trim(Mid(Lines(i), 16))  ' FIX .NET45
                    Exit For
                End If
            End If
        Next

        If GetCLRExceptionType = "" Then
            If g_ClrExceptionTypeFailureLogCount < MAX_CLR_EXCEPTION_TYPE_FAILURE_LOG_ENTRIES Then
                g_ClrExceptionTypeFailureLogCount = g_ClrExceptionTypeFailureLogCount + 1

                WriteToLog "Unable to determine CLR exception type" & vbcrlf & _
                    "ExceptionObjHexAddr = " & ExceptionObjHexAddr & vbcrlf & _
                    "bInnerException = " & bInnerException & vbcrlf & _
                    "DumpObject Output = " & Output & vbcrlf & _
                    Debugger.Execute("kb100") & vbcrlf & _
                    Debugger.Execute("!clrstack") & vbcrlf & _
                    Debugger.Execute("!dso")
            End If
        End If
    End If
End Function

By default, !DumpObj is used to dump the content of the exception object. However, something changed in .NET 4.5 and obviously, the type of the exception cannot be extracted anymore. Instead, place the !pe command here and parse the exception type from it's result.

In order to make !pe work, you need to do .loadby sos clr. I added it to the Sub Debugger_OnLoadModule:

Sub Debugger_OnLoadModule(ByVal NewModule)
    WriteToLog NewModule.ImageName & " loaded at " & Debugger.GetAs32BitHexString(NewModule.Base)
    Select Case UCase(NewModule.ModuleName)
        Case "MSCORWKS", "MSCORSVR", "CLR", "CORECLR"
            UpdateDeferredManagedBreakpoints
    End Select

    Debugger.Execute(".loadby sos clr")
End Sub

After making the changes you need to either restart the processs or deactivate/reactivate the rule to apply it.

Also: be aware, that if you modify your rule in the DebugDiag GUI, you changes will be overwritten completely.