You're probably not handling WM_NCACTIVATE as well. Try handling it, similar to this:
case WM_NCACTIVATE:
break;
case WM_ACTIVATEAPP:
if (LOWORD(wParam) == FALSE)
SendMessage(hWnd, WM_SYSCOMMAND, SC_MINIMIZE, NULL);
break;
Edit:
I must have missed the part of your question where you said you removed the default bar by setting the style to 0. That is definitely not the proper way to do it, you should do something along the lines of this, as found here:
LONG lStyle = GetWindowLong(hWnd, GWL_STYLE);
lStyle &= ~(WS_CAPTION | WS_THICKFRAME | WS_MINIMIZE | WS_MAXIMIZE | WS_SYSMENU);
SetWindowLong(hWnd, GWL_STYLE, lStyle);
After you do that you should no longer need to handle WM_ACTIVATEAPP or WM_NCACTIVATE to properly minimize/maximize the window.