I've found the answer in this article http://www.webdbtips.com/89962/
add this tag to html page
<META HTTP-EQUIV="MSThemeCompatible" CONTENT="Yes">
or add the following line to the body of the class that inherits from CDHtmlDialog
STDMETHOD(GetHostInfo)(DOCHOSTUIINFO *pInfo){pInfo->dwFlags |= DOCHOSTUIFLAG_THEME; return S_OK;};
that enabled the visual style for me