Datastax DSE - Deleting lot of rows resulting in "Insert command failed after 4 attempts" exception

Question

We have two DSE nodes (3.2.4) for development purpose. Replication factor for a Cassandra keyspace is set to 2. When performing Solr delete by query without specifying consistency level (i.e.: quorum), occasionally we got the following exception when deleting a lot of rows (about 500K or so). With consistency level set to one, the same exception also observed (when deleting about 1 million rows). I wonder if there is a work around for this. The exception is as follows:

 WARN [http-8983-1] 2014-03-27 09:21:59,008 CassandraUtil.java (line 44) Error Operation timed out - received only 0 responses. on attempt 1 out of 4 with CL ONE...
 WARN [http-8983-1] 2014-03-27 09:22:09,109 CassandraUtil.java (line 44) Error Operation timed out - received only 0 responses. on attempt 2 out of 4 with CL ONE...
 WARN [http-8983-1] 2014-03-27 09:22:19,211 CassandraUtil.java (line 44) Error Operation timed out - received only 0 responses. on attempt 3 out of 4 with CL ONE...
 WARN [http-8983-1] 2014-03-27 09:22:29,312 CassandraUtil.java (line 44) Error Operation timed out - received only 0 responses. on attempt 4 out of 4 with CL ONE...
ERROR [http-8983-1] 2014-03-27 09:22:29,415 SolrException.java (line 124) org.apache.solr.common.SolrException: java.lang.RuntimeException: Insert command failed after 4 attempts, source exception follows.
        at com.datastax.bdp.search.solr.handler.update.CassandraDirectUpdateHandler.deleteByQuery(CassandraDirectUpdateHandler.java:236)
        at org.apache.solr.update.processor.RunUpdateProcessor.processDelete(RunUpdateProcessorFactory.java:80)
        at org.apache.solr.update.processor.UpdateRequestProcessor.processDelete(UpdateRequestProcessor.java:55)
        at org.apache.solr.update.processor.DistributedUpdateProcessor.doDeleteByQuery(DistributedUpdateProcessor.java:899)
        at org.apache.solr.update.processor.DistributedUpdateProcessor.processDelete(DistributedUpdateProcessor.java:757)
        at org.apache.solr.update.processor.UpdateRequestProcessor.processDelete(UpdateRequestProcessor.java:55)
        at org.apache.solr.handler.loader.XMLLoader.processDelete(XMLLoader.java:346)
        at org.apache.solr.handler.loader.XMLLoader.processUpdate(XMLLoader.java:277)
        at org.apache.solr.handler.loader.XMLLoader.load(XMLLoader.java:173)
        at org.apache.solr.handler.UpdateRequestHandler$1.load(UpdateRequestHandler.java:92)
        at org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(ContentStreamHandlerBase.java:74)
        at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:135)
        at org.apache.solr.core.SolrCore.execute(SolrCore.java:1841)
        at org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:669)
        at com.datastax.bdp.search.solr.servlet.CassandraDispatchFilter.execute(CassandraDispatchFilter.java:165)
        at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:359)
        at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:155)
        at com.datastax.bdp.search.solr.servlet.CassandraDispatchFilter.doFilter(CassandraDispatchFilter.java:137)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
        at com.datastax.bdp.cassandra.audit.SolrHttpAuditLogFilter.doFilter(SolrHttpAuditLogFilter.java:194)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
        at com.datastax.bdp.search.solr.auth.CassandraAuthorizationFilter.doFilter(CassandraAuthorizationFilter.java:94)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
        at com.datastax.bdp.search.solr.auth.DseAuthenticationFilter.doFilter(DseAuthenticationFilter.java:102)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
        at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233)
        at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191)
        at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)
        at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)
        at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
        at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:298)
        at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:859)
        at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:588)
        at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:489)
        at java.lang.Thread.run(Thread.java:662)
Caused by: java.lang.RuntimeException: Insert command failed after 4 attempts, source exception follows.
        at com.datastax.bdp.util.CassandraUtil.robustInsert(CassandraUtil.java:56)
        at com.datastax.bdp.search.solr.ThriftCassandraRowWriter.doDeletes(ThriftCassandraRowWriter.java:189)
        at com.datastax.bdp.search.solr.ThriftCassandraRowWriter.access$100(ThriftCassandraRowWriter.java:39)
        at com.datastax.bdp.search.solr.ThriftCassandraRowWriter$DeletingHitCollector.collect(ThriftCassandraRowWriter.java:241)
        at org.apache.lucene.search.Scorer.score(Scorer.java:65)
        at org.apache.lucene.search.ConstantScoreQuery$ConstantScorer.score(ConstantScoreQuery.java:245)
        at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:612)
        at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:309)
        at com.datastax.bdp.search.solr.ThriftCassandraRowWriter.deleteByQuery(ThriftCassandraRowWriter.java:168)
        at com.datastax.bdp.search.solr.handler.update.CassandraDirectUpdateHandler.deleteByQuery(CassandraDirectUpdateHandler.java:217)
        ... 38 more
Caused by: org.apache.cassandra.exceptions.WriteTimeoutException: Operation timed out - received only 0 responses.
        at org.apache.cassandra.service.AbstractWriteResponseHandler.get(AbstractWriteResponseHandler.java:91)
        at org.apache.cassandra.service.StorageProxy.mutate(StorageProxy.java:198)
        at com.datastax.bdp.util.CassandraUtil.robustInsert(CassandraUtil.java:39)
        ... 47 more

Answer 1

Thanks for reporting this issue. We're taking a look at it and considering improvements.

For now, all I can recommend is to retry the operation if it fails, with a slight delay since the problem may simply be that the cluster is busy.

You might also want to consider smaller deletes, using narrower queries that select fewer documents/rows.

I'll check into whether you can reasonably tune the timeout to handle cluster latency issues.

EDIT: You can raise the timeout with write_request_timeout_in_ms, in cassandra.yaml