Why CreateDialog failed with error code 5 in BHO?

Question

I use CreateDialog to create a modeless dialog in SetSite method after get the IWebBrowser2 interface. The dialog resource is in the BHO dll. When create a new instance (I mean doble click the IE shortcut) of IE the creation is successful but when I create a new tab the creation is failed (but in other computer it is successful). there is also something strange is that sometimes create a new tab will also create a new IE process but sometimes will not.

This is the code for dialog creation:

bool MyDialog::Create(MyContext *ls)
{
    extern HINSTANCE hInstance; // handle of BHO dll
    m_hDialog = CreateDialog(hInstance, MAKEINTRESOURCE(IDD_MYDLG),ls->GetBrowserMainWnd(), MyDlgProc);

    if (m_hDialog) {
        SetWindowLong(m_hDialog, GWL_USERDATA, (LONG)ls);
        SetTimer(m_hDialog, 1, 1000, NULL);
        return true;
    }
    return false;
}

I think this has something to do with dialog creation in different UI thread but not sure about this. Hope somebody can help me with this problem. Thanks a lot!

Update 2014-03-31:

The GetBrowserMainWnd method call IWebBrowser2->get_HWND to get the main window handle. But for IE7 and later the introduced tabbed window makes things complex as MSDN's description:

"Internet Explorer 7. With the introduction of tabbed browsing, the return value of this method can be ambiguous. To alleviate confusion and maintain the highest level of compatibility with existing applications, this method returns a handle to the top-level window frame, not the currently selected tab."

So, I use the example code (refer to http://msdn.microsoft.com/en-us/library/aa752126(v=vs.85).aspx) solved this problem.

Answer 1

It seems that the root cause is the third parameter hWndParent. When I set it to NULL this problem disappear. I think the new process of an IE tab can't access the IE main window handle so failed with error code 5.