Apparently, the culprit was the following piece of code, which was the cause for the delay:
environment.put("com.sun.jndi.ldap.connect.timeout", "1500");
Even the debug logs confirmed that the delay was happening when an attempt was being made to set the connection timeout.
Apparently, it seems from the following links, that connection time out does not work when Ldap connection is made with ssl.
Removing this line of code solved the issue.