Terracotta Ehcache: server disconnects during debug

Question

I found out, that when I connect by debugger to the application, and starting to debug, the connection to terracotta server is lost (?) and in the terracotta server logs next messages are appeared:

2012-03-30 13:45:06,758 [L2_L1:TCComm Main Selector Thread_R (listen 0.0.0.0:9510)] WARN com.tc.net.protocol.transport.ConnectionHealthChecker Impl. DSO Server - 127.0.0.1:55112 might be in Long GC. GC count since last ping reply : 1 2012-03-30 13:45:27,761 [L2_L1:TCComm Main Selector Thread_R (listen 0.0.0.0:9510)] WARN com.tc.net.protocol.transport.ConnectionHealthChecker Impl. DSO Server - 127.0.0.1:55112 might be in Long GC. GC count since last ping reply : 1 2012-03-30 13:45:31,761 [L2_L1:TCComm Main Selector Thread_R (listen 0.0.0.0:9510)] WARN com.tc.net.protocol.transport.ConnectionHealthChecker Impl. DSO Server - 127.0.0.1:55112 might be in Long GC. GC count since last ping reply : 2

...

2012-03-30 13:46:37,768 [L2_L1:TCComm Main Selector Thread_R (listen 0.0.0.0:9510)] ERROR com.tc.net.protocol.transport.ConnectionHealthChecke rImpl. DSO Server - 127.0.0.1:55112 might be in Long GC. GC count since last ping reply : 10. But its too long. No more retries 2012-03-30 13:46:38,768 [HealthChecker] INFO com.tc.net.protocol.transport.ConnectionHealthCheckerImpl. DSO Server - 127.0.0.1:55112 is DEAD 2012-03-30 13:46:38,768 [HealthChecker] ERROR com.tc.net.protocol.transport.ConnectionHealthCheckerImpl: DSO Server - Declared connection dead ConnectionID(1.0b1994ac80f14b7191080bdc3f38582a) idle time 45317ms 2012-03-30 13:46:38,768 [L2_L1:TCWorkerComm # 0_R] WARN com.tc.net.protocol.transport.ServerMessageTransport - ConnectionID(1.0b1994ac80f14b71 91080bdc3f38582a): CLOSE EVENT : com.tc.net.core.TCConnectionJDK14@5158277: connected: false, closed: true local=127.0.0.1:9510 remote=127.0.0 .1:55112 connect=[Fri Mar 30 13:34:22 BST 2012] idle=2001ms [207584 read, 229735 write]. STATUS : DISCONNECTED

...

2012-03-30 13:46:38,799 [L2_L1:TCWorkerComm # 0_R] INFO com.tc.objectserver.persistence.sleepycat.SleepycatPersistor - Deleted client state fo r ChannelID=[1] 2012-03-30 13:46:38,801 [WorkerThread(channel_life_cycle_stage, 0)] INFO com.tc.objectserver.handler.ChannelLifeCycleHandler - : Received tran sport disconnect. Shutting down client ClientID[1] 2012-03-30 13:46:38,801 [WorkerThread(channel_life_cycle_stage, 0)] INFO com.tc.objectserver.persistence.impl.TransactionStoreImpl - shutdownC lient() : Removing txns from DB : 0

After this is happened, any operation with cache, like getWithLoader just doesn't answer, until terracotta server won't be restarted again.

Question: how can it be fixed/reconfigured? I assume, it can happen in production also (and actually sometimes happens) if for some (any) reason application will hang/staled/etc.

Answer 1

This is just to get you started.

TC connections betwee server and client are considered dead when the applicable HealthCheck fails. The default values for the HealthCheck assume a very stable and performant network. I recommend you familiarize yourself with the details and the calculations on

http://www.terracotta.org/documentation/3.5.2/terracotta-server-array/high-availability#85916

So typically you begin with a) making sure your network doesn't hiccup occasionally b) setting the TC HealthCheck values a bit higher

If the problem persists I'd recommend posting directly on the TC forums (they'll help you even if you only use the open-source edition, may take a few days to reply though.