I am only guessing but I suspect that your adfs is configurd for windows authentication while your application has the Login button (it is not clear from your description).
If this is the case then you can't really log out from already authenticated adfs and it happily reissues the token upon the next request from the application. You can't do anything about it since the authentication is performed with ntlm/kerberos and this is the browser that keeps your credentials until you close it.
The solution would be to change adfs to Forms mode (switch the order of providers in adfs web.config) so that not only the adfs has its own cookies but also these cookies are truly removed when you issue the sign out message.