Microsoft still seems to have a bug with Forms Authentication on WebFarm

Question

It appears that there was a problem at some point that folks tracked down that caused authentication tickets to be marked invalid on a Webfarm when the servers had different mixes of patches.

UNFORTUNATELY, it seems that there is STILL a problem even when the web servers have identical patches on them.

My two servers:

• have IDENTICAL patches
• have identical machineKeys

Yet when a user transitions from one web server in the farm to the other web server the underlying microsoft code invalidates the token. Here's the Event log entry:

Event code: 4005 
Event message: Forms authentication failed for the request. Reason: The ticket supplied was invalid. 
Event time: 4/10/2012 2:42:20 PM 
Event time (UTC): 4/10/2012 6:42:20 PM 
Event ID: 92eedee52ede49239fd063fe5609d858 
Event sequence: 2 
Event occurrence: 1 
Event detail code: 50201 

Application information: 
    Application domain: /LM/W3SVC/2000/ROOT-1-129785553216092727 
    Trust level: Full 
    Application Virtual Path: / 
    Application Path: C:\inetpub\wwwroot\testportal.mydomain.com\ 
    Machine name: WEB02 

Process information: 
    Process ID: 1428 
    Process name: w3wp.exe 
    Account name: IIS APPPOOL\testportal.mydomain.com 

Request information: 
    Request URL: http://testportal.mydomain.com/Interface.aspx 
    Request path: /Interface.aspx 
    User host address: ************* 
    User:  
    Is authenticated: False 
    Authentication Type:  
    Thread account name: IIS APPPOOL\testportal.mydomain.com 

Name to authenticate:  

Custom event details: 

I've either got to figure this out or write microsoft's authentication out of my system (which I really don't have time to do).

Here are the hotfixes applied IDENTICALLY to both servers which are:

Windows Server 2008 R2 x64-based

KB981391,KB981392,KB977236,KB981111,KB977238,KB977239,KB981390,KB2305420,KB2386667,KB2393802,KB2425227,KB2475792,KB2476490,KB2478662,KB2479628,KB2482017,KB2484033,KB2485376,KB2487426,KB2488113,KB2492386,KB2503665,KB2505438,KB2506014,KB2506212,KB2506928,KB2507618,KB2507938,KB2508272,KB2509553,KB2510531,KB2511250,KB2511455,KB2515325,KB2518869,KB2522422,KB2524375,KB2529073,KB2530548,KB2533552,KB2533623,KB2534366,KB2536275,KB2536276,KB2539635,KB2541014,KB2544521,KB2544893,KB2545698,KB2547666,KB2552343,KB2555917,KB2556532,KB2560656,KB2563227,KB2564958,KB2567680,KB2570947,KB2572077,KB2584146,KB2585542,KB2588516,KB2598845,KB2603229,KB2607047,KB2607576,KB2608658,KB2618444,KB2618451,KB2620704,KB2620712,KB2621440,KB2631813,KB2632503,KB2633873,KB2633952,KB2636573,KB2639308,KB2639417,KB2640148,KB2641653,KB2641690,KB2643719,KB2644615,KB2645640,KB2647516,KB2647518,KB2654428,KB2656356,KB2660075,KB2660465,KB2665364,KB2667402,KB958488,KB976902,KB976932,KB982018

Per request, here's the authentication section of my web.config:

    <authentication mode="Forms">
        <forms loginUrl="Login.aspx" name=".ASPXFORMSAUTH" cookieless="AutoDetect" timeout="120" slidingExpiration="true">
        </forms>
    </authentication>

Answer 1

I think I have finally resolved this issue. I'm not exactly sure why this change fixes the problem but what I did to resolve it was to put the machineKey definition directly in the web.config file for my application. Apparently either I don't understand how to use IIS Manager to set up the machine keys properly or there is some issue with the IIS Manager.

So to solve the problem I put an entry directly in my web.config that looks something like this:

<machineKey validation="SHA1" validationKey="-a-validation-key-" decryption="Auto"  decryptionKey="-a-decryption-key-"/>

This article kind of led me in the correct path to solve this issue:

ASP.NET 4 Breaking Changes

Answer 2

Well I spoke too soon. During this whole process, I deleted .NET Framework 4 off of the server because we are not developing to it and I had seen some mention of FW4 causing authentication issues.

After "resolving" this issue I put Framework 4 back on the servers and ran windows update which added the following patches to both servers:

• Update for Microsoft .NET Framework 4 on XP, Server 2003, Vista, Windows 7, Server 2008, Server 2008 R2 for x64 (KB2600217)
• Update for Microsoft .NET Framework 4 on Windows XP, Windows Server 2003, Windows Vista, Windows 7, Windows Server 2008, Windows Server 2008 R2 for x64-based Systems (KB2533523)
• Security Update for Microsoft .NET Framework 4 on Windows XP, Windows Server 2003, Windows Vista, Windows 7, Windows Server 2008, Windows Server 2008 R2 for x64-based Systems (KB2487367)
• Security Update for Microsoft .NET Framework 4 on Windows XP, Windows Server 2003, Windows Vista, Windows 7, Windows Server 2008, Windows Server 2008 R2 for x64-based Systems (KB2518870)
• Security Update for Microsoft .NET Framework 4 on XP, Server 2003, Vista, Windows 7, Server 2008, Server 2008 R2 for x64 (KB2656351)
• Security Update for Microsoft .NET Framework 4 on XP, Server 2003, Vista, Windows 7, Server 2008, Server 2008 R2 for x64 (KB2572078)
• Security Update for Microsoft .NET Framework 4 on XP, Server 2003, Vista, Windows 7, Server 2008, Server 2008 R2 for x64 (KB2633870)
• Update for Microsoft .NET Framework 4 on Windows XP, Windows Server 2003, Windows Vista, Windows 7, Windows Server 2008, Windows Server 2008 R2 for x64-based Systems (KB2468871)
• Security Update for Microsoft .NET Framework 4 on XP, Server 2003, Vista, Windows 7, Server 2008, Server 2008 R2 for x64 (KB2656368)

Since I made these updates, the problem is now back. The other interesting thing is that somewhere during the process, these updates chopped the decryption key back to 48 bytes. I'm not sure if it chopped the existing one or if it created a new one. I went back and put a newly generated Machine Key and Decryption Key and it did not fix the problem.

Answer 3

Well with some help from this guy over at asp.net I figured out how to fix the issue:

Thread at asp.net that helped me figure this out.

I think what finally fixed it was regenerating the Validation and Decryption keys. There must have been something about the old keys that was incompatible with the hotfixes loaded on these servers.

On an interesting note, the old and new validation keys were both 128 bytes but the old Decryption Key was 48 bytes and the new one is 64 bytes.

Answer 4

I've seen this behavior when MS10-070 (asp.net padding oracle attack) was applied to some of the servers. This update changes [in some way] the encryption technique, making the tickets incompatible with unpatched servers within the same farm.

Can you verify that all servers in your farm has this update applied?