Does your IDP support and correctly initialize Single Logout? If so it could be related to this issue, just update to latest Spring SAML version or change property invalidateHttpSession in your logout handler to true:
<bean id="logoutSessionHandler"
class="org.springframework.security.web.authentication.logout.SecurityContextLogoutHandler">
<property name="invalidateHttpSession" value="true"/>
</bean>