Validation of viewstate MAC failed, but occurs on only 1 out of 2 webfarm servers (machine key is correct)

Question

So I have something weird going on and i can not pin down exactly what is causing it. My asp.net project is live with session state on two production servers that are synced using the following command:

msdeploy -verb:sync -source:webserver,computername=%MACHINE%,username=Administrator,password=%PASSWORD% -dest:webserver 2<&1 

The application is an asp.net 4.0 website that is run on two Server 2008 R2 web servers behind a load balanced configuration where the users are set to stick to one server once they connect. We have <MachineKey> set hardcoded with validation and decryption keys in the root site of the application and it is the same between both servers. My application is set up to forward exception events to our email system.

What is happening is that i am receiving the dreaded 'Validation of viewstate MAC failed' from the servers but even though the server load is 50/50 split the errors are coming in on a 99/1 split. So one web server is generating these errors considerably more often than the other one. This is strange considering the servers are synced and all configurations are identical.

I've done extensive searching on this problem and it seems quite difficult to find any solution that doesn't mention or do the following.

• <MachineKey> is not identical between servers. (I know for a fact this is not my problem)
• Setting enableViewStateMac=false or some other setting that jeopardizes the site security.
• Make sure that all action tags on form inputs reference the same page they are placed on
• Make sure the instance ID of the servers are the same (they are)
• If the user clicks through the page before the entire page (viewstate) has been downloaded (my viewstate is set to render at the top of the page).
• Issues with response.redirect and server.transfer

Now i have eliminated all except the last two as possible causes. My application has been running fine for over a year with no issues and right before these errors appeared i enabled SQL session state, migrated the project from .NET 3.5 to .NET 4.0, and set the set the server mode deployment mode to retail. I have tried recycling the application pools and performing an 'iis reset' to no avail.

Does anyone else have any suggestions as to what i can look at? Bottom line i do NOT want to fix this by opening up security holes in my site.

It appears this is happening to users right after they authenticate using forms authentication the first time they try to log in but i can not confirm this. I also have a theory that this might have to do with caching but i can't be sure on this either.

Here is the juicy bit from my web.config (i have removed some sensitive information)

    <system.web>
    <httpRuntime requestValidationMode="2.0"/>
    <globalization culture="en-US" uiCulture="en-US" resourceProviderFactoryType="WebResourceFactory"/>
    <compilation debug="true" defaultLanguage="c#" explicit="true" strict="true" targetFramework="4.0">
      <assemblies>
      </assemblies>
    </compilation>
    <authentication mode="Forms">
      <forms name=".ASPXAUTH" loginUrl="Login.aspx" protection="All" slidingExpiration="true"/>
    </authentication>
    <authorization>
      <deny users="?"/>
    </authorization>
    <sessionState mode="SQLServer" sqlConnectionString="connection" compressionEnabled="true" />
    <pages theme="Blue" controlRenderingCompatibilityVersion="3.5" clientIDMode="AutoID">
<machineKey validationKey="key" decryptionKey="key" decryption="3DES" validation="SHA1" />
    </system.web>

EDIT:Emphasized that i'm using SQL session state with a load balancer set to prefer to route users to the server they started on.

Answer 1

Ok i appear to have fixed it, though i can not discern what exactly caused it so i will just list all the steps i performed in case someone else has this problem later on.

1 : Installed these windows updates:

2: My forms authentication cookie was set to persistent but my session cookie was set to the browser session. I set my forms authentication cookie to be browser session based.

3: I copied my from the site config to the root of IIS. From all the documentation i could find it should not be necessary for me to do this because IIS should support multiple machine keys for different sites / applications.

4: Rebooted the server.

That's it! I have not received the errors since then.

Answer 2

There is an additional possibility that you have not added to your list - ViewStateUserKey.

I have seen issues with applications where the ViewStateUserKey was set to the Session ID on logon and (crucially) before any data is saved to the Session. Since ASP.NET does not persist Session IDs until one or more objects are saved to session this meant that the ID was constantly changing and the Viewstate was failing validation. Even if you have saved something to Session then the Session will be different on each server if you are using the default in-process model and not a state server or SQL session store (as you are doing). Any server specific value or value that is not readily predictable across servers used with ViewStateUserKey will of course also cause this problem.

Otherwise the most common causes of this issue I have seen is where an "Action" attribute is set on a form that is not the URL of the same page as the form (this catches out developers used to PHP or platforms that do not attempt to abstract away from HTTP), or missing Machine Key attributes in the Web.config in multi-server environments (which you seem to have covered).