Turns out this could be done by using a custom security manager:
public class CustomSecurityManager extends DefaultWebSecurityManager {
@Override
protected void beforeLogout(Subject subject)
{
super.removeRequestIdentity(subject);
}
}