How to write a custom native visualizer DLL for Visual Studio 2012 debugger?

Question

What is needed to write a custom native visualizer DLL in C++ for Visual Studio 2012 debugger? I want to display a value that can only be calculated from a class/struct on-demand hence a native visualizer DLL is required. Visual Studio 2012 uses a new method for implementing native visualizers called Natvis. As of this time, there is very little correct information on Natvis and especially on using Natvis to call a visualizer DLL. The DLL will calculate a display string based on class/struct member values.

Answer 1

Here's the C++ code that comprises the AddIn DLL. I named the file NatvisAddIn.cpp and the project created NatvisAddIn.dll.

#include "stdafx.h"
#include <iostream>
#include <windows.h>

#define ADDIN_API __declspec(dllexport)

typedef struct tagDEBUGHELPER
{
    DWORD dwVersion;
    HRESULT (WINAPI *ReadDebuggeeMemory)( struct tagDEBUGHELPER *pThis, DWORD dwAddr, DWORD nWant, VOID* pWhere, DWORD *nGot );
    // from here only when dwVersion >= 0x20000
    DWORDLONG (WINAPI *GetRealAddress)( struct tagDEBUGHELPER *pThis );
    HRESULT (WINAPI *ReadDebuggeeMemoryEx)( struct tagDEBUGHELPER *pThis, DWORDLONG qwAddr, DWORD nWant, VOID* pWhere, DWORD *nGot );
    int (WINAPI *GetProcessorType)( struct tagDEBUGHELPER *pThis );
} DEBUGHELPER;

typedef HRESULT (WINAPI *CUSTOMVIEWER)( DWORD dwAddress, DEBUGHELPER *pHelper, int nBase, BOOL bUniStrings, char *pResult, size_t max, DWORD reserved );

extern "C" ADDIN_API HRESULT MyClassFormatter( DWORD dwAddress, DEBUGHELPER *pHelper, int nBase, BOOL bUniStrings, char *pResult, size_t max, DWORD reserved );
extern "C" ADDIN_API HRESULT MyStructFormatter( DWORD dwAddress, DEBUGHELPER *pHelper, int nBase, BOOL bUniStrings, char *pResult, size_t max, DWORD reserved );

class MyClass
{
public:
    int publicInt;
};

struct MyStruct { int i; };

ADDIN_API HRESULT MyClassFormatter( DWORD dwAddress, DEBUGHELPER *pHelper, int nBase, BOOL bUniStrings, char *pResult, size_t max, DWORD reserved )
{
    MyClass c;
    DWORD nGot;
    pHelper->ReadDebuggeeMemory(pHelper,dwAddress,sizeof(MyClass),&c,&nGot);
    sprintf_s(pResult,max,"Dll MyClass: max=%d nGot=%d MyClass=%x publicInt=%d",max,nGot,dwAddress,c.publicInt);
    return S_OK;
}

ADDIN_API HRESULT MyStructFormatter( DWORD dwAddress, DEBUGHELPER *pHelper, int nBase, BOOL bUniStrings, char *pResult, size_t max, DWORD reserved )
{
    MyStruct s;
    DWORD nGot;
    pHelper->ReadDebuggeeMemory(pHelper,dwAddress,sizeof(MyStruct),&s,&nGot);
    sprintf_s(pResult,max,"Dll MyStruct: max=%d nGot=%d MyStruct=%x i=%d",max,nGot,dwAddress,s.i);
    return S_OK;
}

Here is the .natvis file which Visual Studio 2012 debugger uses to display the value. Place it in a .natvis file. I named it NatvisAddIn.natvis. The file instructs VS 2012 debugger to call NatvisAddIn.dll. The dll contains two visualizer method calls; MyClassFormatter to format MyClass and MyStructFormatter to format MyStruct. The debugger will show the method's formatted value in the Auto, Watch or tooltip display for each instance of the specified type (MyClass, MyStruct).

<?xml version="1.0" encoding="utf-8"?>
    <AutoVisualizer xmlns="http://schemas.microsoft.com/vstudio/debugger/natvis/2010">
    <Type Name="MyClass">
        <DisplayString LegacyAddin="NatvisAddIn.dll" Export="MyClassFormatter"></DisplayString>
    </Type>
    <Type Name="MyStruct">
        <DisplayString LegacyAddin="NatvisAddIn.dll" Export="MyStructFormatter"></DisplayString>
    </Type>
</AutoVisualizer>

Place both the compiled NatvisAddIn.dll file and the NatvisAddIn.natvis files into one of the following three locations:

%VSINSTALLDIR%\Common7\Packages\Debugger\Visualizers (requires admin access)

%USERPROFILE%\My Documents\Visual Studio 2012\Visualizers\

VS extension folders

You will need to make sure the following registry key exists and the value is 1:

[HKEY_CURRENT_USER\Software\Microsoft\VisualStudio\11.0_Config\Debugger]

"EnableNatvisDiagnostics"=dword:00000001

If all goes well, you will see natvis messages appear in Visual Studio's debugger Output window. The messages will show whether Natvis was able to parse the .natvis files. Results of parsing every .natvis file is shown in the output window. If something is wrong, use the command "dumpbin/exports " to double check that the DLL methods' names are exactly matching the .navis file's Type=. Also make sure the current .dll and .natvis files have been copied to the appropriate directory.

Natvis: Parsing natvis xml file: C:\Program Files (x86)\Microsoft Visual Studio 11.0\Common7\Packages\Debugger\Visualizers\atlmfc.natvis.
Natvis: Done parsing natvis xml file: C:\Program Files (x86)\Microsoft Visual Studio 11.0\Common7\Packages\Debugger\Visualizers\atlmfc.natvis.
Natvis: Parsing natvis xml file: C:\Program Files (x86)\Microsoft Visual Studio 11.0\Common7\Packages\Debugger\Visualizers\concurrency.natvis.
Natvis: Done parsing natvis xml file: C:\Program Files (x86)\Microsoft Visual Studio 11.0\Common7\Packages\Debugger\Visualizers\concurrency.natvis.
Natvis: Parsing natvis xml file: C:\Program Files (x86)\Microsoft Visual Studio 11.0\Common7\Packages\Debugger\Visualizers\NatvisAddIn.natvis.
Natvis: Done parsing natvis xml file: C:\Program Files (x86)\Microsoft Visual Studio 11.0\Common7\Packages\Debugger\Visualizers\NatvisAddIn.natvis.
Natvis: Parsing natvis xml file: C:\Program Files (x86)\Microsoft Visual Studio 11.0\Common7\Packages\Debugger\Visualizers\stl.natvis.
Natvis: Done parsing natvis xml file: C:\Program Files (x86)\Microsoft Visual Studio 11.0\Common7\Packages\Debugger\Visualizers\stl.natvis.
Natvis: Parsing natvis xml file: C:\Program Files (x86)\Microsoft Visual Studio 11.0\Common7\Packages\Debugger\Visualizers\windows.natvis.
Natvis: Done parsing natvis xml file: C:\Program Files (x86)\Microsoft Visual Studio 11.0\Common7\Packages\Debugger\Visualizers\windows.natvis.
Natvis: Parsing natvis xml file: C:\Program Files (x86)\Microsoft Visual Studio 11.0\Common7\Packages\Debugger\Visualizers\winrt.natvis.
Natvis: Done parsing natvis xml file: C:\Program Files (x86)\Microsoft Visual Studio 11.0\Common7\Packages\Debugger\Visualizers\winrt.natvis.

Test program:

#include "stdafx.h"
#include <iostream>

class MyClass
{
public:
    int publicInt;
};

struct MyStruct { int i; };

int _tmain(int argc, _TCHAR* argv[])
{
    struct MyStruct s = {1234};
    std::cout << s.i << std::endl;
    MyClass *c = new MyClass;
    c->publicInt = 1234;
    std::cout << c->publicInt << std::endl;
    return 0;
}

Information resources:

\Xml\Schemas\natvis.xsd

http://code.msdn.microsoft.com/windowsdesktop/Writing-type-visualizers-2eae77a2

http://blogs.msdn.com/b/mgoldin/archive/2012/06/06/visual-studio-2012-and-debugger-natvis-files-what-can-i-do-with-them.aspx

http://blogs.msdn.com/b/vcblog/archive/2012/07/12/10329460.aspx

Answer 2

For 64 bit version debugging, following lines should be used:

auto realAddress = pHelper->GetRealAddress(pHelper);
pHelper->ReadDebuggeeMemoryEx(pHelper, realAddress, sizeof(MyClass), &c, &nGot );

For previous example, 64bit version could look like this:

#include "stdafx.h"
#include <iostream>
#include <windows.h>

#define ADDIN_API __declspec(dllexport)

typedef struct tagDEBUGHELPER
{
    DWORD dwVersion;
    HRESULT (WINAPI *ReadDebuggeeMemory)( struct tagDEBUGHELPER *pThis, DWORD dwAddr, DWORD nWant, VOID* pWhere, DWORD *nGot );
    // from here only when dwVersion >= 0x20000
    DWORDLONG (WINAPI *GetRealAddress)( struct tagDEBUGHELPER *pThis );
    HRESULT (WINAPI *ReadDebuggeeMemoryEx)( struct tagDEBUGHELPER *pThis, DWORDLONG qwAddr, DWORD nWant, VOID* pWhere, DWORD *nGot );
    int (WINAPI *GetProcessorType)( struct tagDEBUGHELPER *pThis );
} DEBUGHELPER;

typedef HRESULT (WINAPI *CUSTOMVIEWER)( DWORD dwAddress, DEBUGHELPER *pHelper, int nBase, BOOL bUniStrings, char *pResult, size_t max, DWORD reserved );

extern "C" ADDIN_API HRESULT MyClassFormatter( DWORD dwAddress, DEBUGHELPER *pHelper, int nBase, BOOL bUniStrings, char *pResult, size_t max, DWORD reserved );

class MyClass
{
public:
    int publicInt;
};

ADDIN_API HRESULT MyClassFormatter( DWORD dwAddress, DEBUGHELPER *pHelper, int nBase, BOOL bUniStrings, char *pResult, size_t max, DWORD reserved )
{
    MyClass c;
    DWORD nGot;
    auto realAddress = pHelper->GetRealAddress(pHelper);
    pHelper->ReadDebuggeeMemoryEx(pHelper, realAddress, sizeof(MyClass), &c, &nGot );
    sprintf_s(pResult,max,"Dll MyClass: max=%d nGot=%d MyClass=%llx publicInt=%d",max, nGot, realAddress, c.publicInt);
    return S_OK;
}

Answer 3

I need one clarification related to search path(s) that are looked up for loading the above mentioned "NatvisAddIn.dll". Let me try to explain by extending the above example

To visualize my custom C++ class object(let's say MyCustomeType), I need to call some additional APIs (to compute the display string for MyCustomeType) in the implementation of function(say MyClassFormatter) that is mentioned in "Export" attribute of "DisplayString" XML.

Calling these additional APIs create a library/dll dependency on the above mentioned "NatvisAddIn.dll". If this additional dependency is just one or two library, I can put those library at same location where NatvisAddIn.dll is present. However, in my case the dependency contains a long chain of libraries.

Can some one suggest me some elegant way to resolve the dependency without pulling the complete chain of libraries into %USERPROFILE%\My Documents\Visual Studio 2012\Visualizers folder?

For demonstration of my use case, let's consider the below dependency tree: NatvisAddIn.dll a.dll
a1.dll a2.dll a3.dll

I copied the a.dll at same location where NatvisAddIn.dll. It's dependent dlls (a1, a2 and a3) are present at location that are added to PATH variable. When I try to visualize MyCustomeType object in visual studio debugger, the natvis diagonostic give the below error in output window

Natvis: C:\Users\myUser\Documents\Visual Studio 2017\Visualizers\mydata.natvis(9,6): Error: Failed to load addin from C:\Users\myuser\Documents\Visual Studio 2017\Visualizers\NatvisAddIn.dll for type MyCustomeType: : The specified module could not be found.

My understanding of above error, the visual studio debugger was not able to resolve dependency of a.dll (a1, a2 and a3) and hence it failed to load NatvisAddIn.dll

When I tried to use a.dll in my testApplication and compute the DisplayString for MyCustomeType, dependency get resolve, a.dll is loaded and I get the expected out string without copying a1.dll, a2.dll and a3.dll. the dependent dll are resolved/picked-up from window PATH variable. But, In case of visual studio debugger, dependent DLL are NOT resolved from PATH variable.

As identified by depends tool, some dlls which are not resolved by debugger are:

api-ms-win-core-errorhandling-l1-1-0.dll api-ms-win-crt-time-l1-1-0.dll api-ms-win-crt-heap-l1-1-0.dll

Some of these dll are present visual studio installation, others are present in c:\windows\WinSxS

I've tried my use case on visual studio 2012 and visual studio 2017. I landed up in same issue with both the visual studios.