if(BM_GETCHECK==BST_CHECKED)
This condition will always evaluate to false. That is why it is telling you the button is unchecked regardless of the button state.
In your case, you probably want something like this:
if(Button_GetState(hDecRB) == BST_CHECKED) {
MessageBox(NULL,"CHECKED","sachin",MB_OK);
}
else {
MessageBox(NULL,"Binary not checked","sachin",MB_OK);
MessageBox(NULL,(LPCSTR)flag,"sachin",MB_OK);
}
Also you need to store each button in a different HWND. You are using hDecRB for each one, which is being overwritten each time you reassign it.
http://msdn.microsoft.com/en-us/library/windows/desktop/bb775986(v=vs.85).aspx