The error is because the two documented login functions used above are specificaly for IntegratedSecurityMode 1, 2 and 3.
You've indicated that you are using Cognos to handle user authentication, which is either mode 4 or 5. There are two more-or-less "undocumented" functions for logging in to the server using Cognos authentication:
Here's a sample, written in C++ (copied from here) which you can use a reference for connecting using Cognos authentication:
TM1V voServerName = TM1ValStringW( hPool, (TM1_UTF16_T *)pszServerName, 0 );
TM1V voPasswd = TM1ValStringEncryptW( hPool, (TM1_UTF16_T*)pszPassword, 0 );
TM1V vArray[3];
vArray[0] = TM1ValStringW( hPool, (TM1_UTF16_T*)szCAMNamespace, 0 );
vArray[1] = TM1ValStringW( hPool, (TM1_UTF16_T*)admin_login_name, 0 );
vArray[2] = voPasswd;
TM1V vCAMArgArr = TM1ValArray(hPool, vArray, 3);
TM1V vTmpServer =
TM1SystemServerConnectWithCAMNamespace(hPool, voServerName, vCAMArgArr);