In Mono, memcached providers crashes after a few hours with “the socket is not connected” 500 error

Question

I am trying to set up a mono asp.net 4.0 test website that uses memcached providers (http://memcachedproviders.codeplex.com/) using Enyim.Memcached to access memcached. The website and memcachedproviders work fine on a windows server.

I ported over the working asp.net website to mono (Ubuntu 10.10, Mono 2.8), and it runs correctly and pulls sessions from an external server running memcached, but inexplicably, every couple hours or so, the website crashes, and just returns a 500 error saying "The socket is not connected" This only happens when memcachedproviders session state is enabled.

Does anyone have any experience setting up memcachedproveders for a session in mono, or know at least where I should begin looking? While I'm familiar with linux and apache, I'm generally more comfortable with windows so there may be some configuration I'm missing.

As far as i can tell, this is the log from apache from the time the error, even with debugging enabled from mod_mono:

[Tue Nov 09 17:53:17 2010] [notice] caught SIGTERM, shutting down
[Tue Nov 09 17:53:19 2010] [notice] Apache/2.2.14 (Ubuntu) mod_mono/2.8 configured -- resuming normal operations
Listening on: /tmp/mod_mono_server_global
Root directory: /
Listening on: /tmp/mod_mono_server_global
Root directory: /
Error: There's already a server listening on /tmp/mod_mono_server_global
mod-mono-server4
mod-mono-server4
Listening on: /tmp/mod_mono_server_mysite
Root directory: /srv/www/mysite
[Tue Nov 09 17:53:58 2010] [error] (104)Connection reset by peer: read_data failed
[Tue Nov 09 17:53:58 2010] [error] Command stream corrupted, last command was -1

The general memcached configuration in web.config is something like:

<enyim.com>
    <memcached>
        <servers>
            <add address="##.##.##.##" port="11211"/>
        </servers>
        <socketPool minPoolSize="10" maxPoolSize="100" connectionTimeout="00:00:10" deadTimeout="00:02:00"/>
    </memcached>
</enyim.com>
...
<sessionState cookieless="false" regenerateExpiredSessionId="true" mode="Custom" customProvider="MemcachedSessionProvider">
    <providers>
        <add name="MemcachedSessionProvider" type="MemcachedProviders.Session.SessionStateProvider,MemcachedProviders" dbType="none" writeExceptionsToEventLog="true"/>
    </providers>
</sessionState>

And the stack trace:

System.Net.Sockets.SocketException: The socket is not connected
  at System.Net.Sockets.Socket.Shutdown (SocketShutdown how) [0x00058] in /home/monoweb/mono-2.8/mcs/class/System/System.Net.Sockets/Socket_2_1.cs:698 
  at Enyim.Caching.Memcached.PooledSocket.Dispose (Boolean disposing) [0x00000] in <filename unknown>:0 
  at Enyim.Caching.Memcached.PooledSocket.Destroy () [0x00000] in <filename unknown>:0 
  at Enyim.Caching.Memcached.MemcachedNode+InternalPoolImpl.ReleaseSocket (Enyim.Caching.Memcached.PooledSocket socket) [0x00000] in <filename unknown>:0 
  at Enyim.Caching.Memcached.PooledSocket.Dispose (Boolean disposing) [0x00000] in <filename unknown>:0 
  at Enyim.Caching.Memcached.PooledSocket.System.IDisposable.Dispose () [0x00000] in <filename unknown>:0 
  at Enyim.Caching.Memcached.ItemOperation.Dispose () [0x00000] in <filename unknown>:0 
  at Enyim.Caching.Memcached.Operation.System.IDisposable.Dispose () [0x00000] in <filename unknown>:0 
  at Enyim.Caching.MemcachedClient.Store (Enyim.Caching.Memcached.ServerPool pool, StoreCommand mode, System.String key, System.Object value, UInt64 casValue, TimeSpan validFor, DateTime expiresAt) [0x00000] in <filename unknown>:0 
  at Enyim.Caching.MemcachedClient.Store (StoreMode mode, System.String key, System.Object value, TimeSpan validFor) [0x00000] in <filename unknown>:0 
  at MemcachedProviders.Session.SessionStateProvider.SetAndReleaseItemExclusive (System.Web.HttpContext context, System.String id, System.Web.SessionState.SessionStateStoreData item, System.Object lockId, Boolean newItem) [0x00000] in <filename unknown>:0 
  at System.Web.SessionState.SessionStateModule.OnReleaseRequestState (System.Object o, System.EventArgs args) [0x0003f] in /home/monoweb/mono-2.8/mcs/class/System.Web/System.Web.SessionState_2.0/SessionStateModule.cs:292 
  at System.Web.HttpApplication+<RunHooks>c__Iterator1.MoveNext () [0x001aa] in /home/monoweb/mono-2.8/mcs/class/System.Web/System.Web/HttpApplication.cs:1026 
  at System.Web.HttpApplication+<Pipeline>c__Iterator2.MoveNext () [0x00ec9] in /home/monoweb/mono-2.8/mcs/class/System.Web/System.Web/HttpApplication.cs:1385 
  at System.Web.HttpApplication.Tick () [0x00000] in /home/monoweb/mono-2.8/mcs/class/System.Web/System.Web/HttpApplication.cs:914 

Answer 1

I ended up fixing this by first downloading the source for memcachedproviders, and recompiling it to reference enyim.memcached 2.7 (the latest version). This involved some minor changes.

After this, in mono I instead got a different error, within enyim memcached regarding string.IndexOfAny ( http://msdn.microsoft.com/en-us/library/11w09h50.aspx ) where the string being checked was null, and so was throwing an exception. If I'm reading the docs right, I think that it's only supposed to throw an exception if the parameter is null, not if the string is. Either that or mod_mono randomly tries to access session[null], whereas asp.net does not.

To fix this, I downloaded the enyim memcached source, and added a check that would replace a null with an empty string before the function call. memcached sessions have been up for over 24 hours so far without a hitch now.