After so much internet surfing and forum hunting I manage to fix this problem. If anyone else having this issue this might help them :)
Thu Sep 26 16:38:19 2013 : Debug: pam_pass: using pamauth string <radiusd> for pam.conf lookup
Thu Sep 26 16:38:19 2013 : Debug: pam_pass: function pam_authenticate FAILED for <test>. Reason: Cannot make/remove an entry for the specified session
The above line actually means a auth fail, even though it doesn't sound like it, also it could mean that .google_authenticator file in the user's home directory isn't accessible.
FreeRadius log file not help you much with this issue, but have a look through /var/log/secure on CentOS and /var/log/auth.log in Ubuntu. This will explain which is the issue.
Issue with my system was my time was out and my random generated number by Google Dual Factor Authenticator application on my iPhone wasn't valid. I had to install NTP and change my servers time to the correct time which fixed the issue!!!!
hope this help someone else :)