Its because the session is block the asynchronous operations, and the handle is usually used for long time operations, like the making and download of a file - if you keep the session on long time operation you block the rest of your pages.
Also the handle is made with the idea of the minimum required to get a response.
About the session lock:
Web app blocked while processing another web app on sharing same session
jQuery Ajax calls to web service seem to be synchronous
ASP.NET Server does not process pages asynchronously
Replacing ASP.Net's session entirely