having an issue with membership credential verification failed when it shouldn't be using it

Question 1

That's awfully suspicious to happen right at that time.

I would agree, but it seems that in step #4 you redirect off the site, and then the user does data entry. If a user occasionally takes 10-20 minutes to enter that information (due to distraction, etc.), it makes it much more likely than if it were just a simple race condition.

If you still have the data from all the errors, you may be able to look back to see if you can find a pattern to the times when this happens (or every xx hours--see below).

Check your IIS settings to see when the app pool recycles. Does it recycle around 4am every night? Is it on a rolling schedule? By default, IIS recycles on some odd number of hours for some reason (I think every 28 hours).
Get off of InProc Session state and into State Server (or SQL). InProc is only going to cause you pain anyway in the long run. Note that when you make this change, you have to ensure that all objects you put into session are serializable, otherwise you will get errors. InProc does not require objects to be serialized in session.

Edit: Ok, to check your app pool recycling:

In IIS Manager, select the appropriate app pool and select Advanced Settings (right click or use right side menu).
Scroll to the bottom, to section Recycling
Regular Time Interval will reset the app pool every xx minutes. The default is 1740 minutes, or every 29 hours.
The Specific Times setting allows you to set a scheduled time for it to recycle.

In general, you DO want to recycle the app pool periodically (probably daily).

To answer your second question: if this is indeed the cause, it's not a question of timeout; it's a question of whether the app pool recycles during the period of time between when they are redirected away and when they get redirected back. Changing your session state to something other than InProc should solve this problem.

That said, the session expiring could ALSO be the cause of this, so setting your session timeout to a larger value could also resolve this.

If you take a more extensive look at the logs during periods of time when this happened, it might give more clues to what is happening.

Edit #2

See if you can isolate the occurrences of the error in the logs. If you can, see if there is a pattern to the browser that is being used. I would also look for other patterns to see if anything jumps out.

You might just test with a bunch of different browsers (including mobile) to see if you can reproduce. Also, try different versions of IE and different security settings on IE.

Question 2

Please refer to the following article by ScottGu http://weblogs.asp.net/scottgu/archive/2006/04/22/Always-set-the-_2200_applicationName_2200_-property-when-configuring-ASP.NET-2.0-Membership-and-other-Providers.aspx.

In addition,make sure that the “ApplicationName” value in “aspnet_Applications” database table and the “applicationName” property value for all providers in web.config (Membership provider, Role provider, Profile provider etc.) must have the same value carefully again.

Hope this would fix your problem!

Question 3

The few things I found in your web.config are -

You have multiple applicationName. Normally, IsUserInRole and GetRolesForUser should fail if applicationName are not same.
In addition, you want to set defaultProvider for each provider especially if you have more than one provider like roleManager - <roleManager ... defaultProvider="DefaultRoleProvider">

Current Web.config

<membership>
   <providers>
      <clear/>
          <add ... applicationName="webportal"/>
   </providers>
</membership>

<profile>
   <providers>
      <clear/>
      <add ... applicationName="/"/>
    </providers>
</profile>

<roleManager enabled="true">
   <providers>
      <clear/>
      <add ... applicationName="/"/>
      <add ...  applicationName="/"/>
   </providers>
</roleManager>

Take a look at web.config of Scott Hanselman's ASP.NET Universal Providers blog.

<sessionState
   mode="InProc"
   stateConnectionString="tcpip=127.0.0.1:42424"
   stateNetworkTimeout="60"
   sqlConnectionString="data source=127.0.0.1;Integrated Security=SSPI"
   cookieless="false"
   timeout="60"
/>

You do not need stateConnectionString, stateNetworkTimeout and sqlConnectionString for InProc. More information here.

Question 4

Its been a while since I have troubleshot issues like this (but I did it for pss over 7 yrs ago).

Everything that @PhilSandler recommended is good. However I have a gut feeling the root cause is NOT AppPool recycling or Session time outs, if you do get desperate you could try Out-Of-Process Session State.

I'm thinking if its roughly 1 out of 10 users and it appears intermittent (as you cannot reproduce it), then its probably going to be a browser specific setting. I'm guessing users which encounter the problem have cookies disabled in their browsers (FF and Chrome users are more switched on than regular IE users).

This hypothesis would slightly match up with @SilverlightFox theory:

the frameset that is preventing 3rd party cookies

<authentication mode="Forms">

I'm thinking that the forms authentication cant store the session data on client side. Membership credential verification failed event log entries can be caused due to disabled cookies.

Please disable cookies in FF or Chrome and try out my hypothesis. Fingers crossed this allows you to reproduce it.

Question 5

After looking around the web, here are the solutions I've found where people have had your 4006 error and ultimately fixed the issue.

applicationName field set to "/". - This is the most common conversation about your issue. But, your applicationName field looks fine so that's definitely not it.
Problem with ActiveDirectoryMembershipProvider - Link. This seems unlikely to be your problem because you're using the SqlMembershipProvider.
Internet user missing permissions on the .mdf and .ldf files - Link. The references to this problem that I saw were with people using SQLExpress because it runs under the ASP.NET user account and needs read/write access. Again, this seems unlikely to be be your problem because your only having it intermittently. But it is related to the 4006 error, so verifying permissions on the user seems worth looking at.
Duplicated User Passwords - I didn't save a link, but if you don't have unique constraints on both username and password in the database, it can return the 4006 response. So, far this seems most probable too me because it would explain the intermittent behavior and because people frequently double-click when using the internet. I would double and triple check your constraints.

I'll be happy to see what else I can find, but ruling out the last two of these first seems like a good idea. Also, the thing that seems the most odd about the gap in your logs is the fact that the EventLog data shows the error happening almost in the middle of the IIS gap. Then after it resumes, there is another log gap. It looks like it might be two completely different users. Maybe not, Just throwing that out there. Let me know. Good luck.

Question 6

I notice that the src isn't set correctly on the IFrame.

<div align="center"> <iframe width="100%" height="600px" src="@Html.Raw(@ViewBag.GateWayWebsite)"></iframe></div>

The GateWayWebsite should be properly HTML encoded in case it contains special characters that are relevant but are not being passed properly:

<div align="center"> <iframe width="100%" height="600px" src="@ViewBag.GateWayWebsite"></iframe></div>

Another thought is that it could be that the frameset that is preventing 3rd party cookies - either the browser setting or the P3P privacy policy header. Try opening the link outside of the IFrame (i.e. so the address bar changes to the external site), just as a test to see if this resolves your issues.

Edit: I shouldn't have edited my answer as you have now reverted your upvote. My answer is technically correct, although it might not be the answer that fixes your current problem, I am right that the output of the URL is wrong as you are not HTML encoding it. I find it is best to fix everything as this reduces the chances of niggly bugs occurring in future that are difficult to debug.