批量加载期间 Solr 错误 - org.apache.lucene.index.MergePolicy$MergeException

Question

通过 sstableloader 批量加载数百万条记录时，我遇到了很多以下异常：

ERROR [Lucene Merge Thread #132642] 2014-07-29 00:35:01,252 CassandraDaemon.java (line 199) Exception in thread Thread[Lucene Merge Thread #132642,6,main]
org.apache.lucene.index.MergePolicy$MergeException: java.lang.IllegalStateException: failed
        at org.apache.lucene.index.ConcurrentMergeScheduler.handleMergeException(ConcurrentMergeScheduler.java:545)
        at org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:518)
Caused by: java.lang.IllegalStateException: failed
        at org.apache.lucene.util.packed.DirectPackedReader.get(DirectPackedReader.java:93)
        at org.apache.lucene.util.packed.BlockPackedReader.get(BlockPackedReader.java:86)
        at org.apache.lucene.util.LongValues.get(LongValues.java:35)
        at org.apache.lucene.codecs.lucene45.Lucene45DocValuesProducer$5.getOrd(Lucene45DocValuesProducer.java:459)
        at org.apache.lucene.codecs.DocValuesConsumer$4$1.setNext(DocValuesConsumer.java:389)
        at org.apache.lucene.codecs.DocValuesConsumer$4$1.hasNext(DocValuesConsumer.java:352)
        at org.apache.lucene.codecs.lucene45.Lucene45DocValuesConsumer.addNumericField(Lucene45DocValuesConsumer.java:141)
        at org.apache.lucene.codecs.lucene45.Lucene45DocValuesConsumer.addSortedField(Lucene45DocValuesConsumer.java:350)
        at org.apache.lucene.codecs.perfield.PerFieldDocValuesFormat$FieldsWriter.addSortedField(PerFieldDocValuesFormat.java:116)
        at org.apache.lucene.codecs.DocValuesConsumer.mergeSortedField(DocValuesConsumer.java:305)
        at org.apache.lucene.index.SegmentMerger.mergeDocValues(SegmentMerger.java:197)
        at org.apache.lucene.index.SegmentMerger.merge(SegmentMerger.java:116)
        at org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:4058)
        at org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:3655)
        at org.apache.lucene.index.ConcurrentMergeScheduler.doMerge(ConcurrentMergeScheduler.java:405)
        at org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:482)
Caused by: java.io.EOFException: Read past EOF (resource: BBIndexInput(name=_13ms5_Lucene45_0.dvd))
        at com.datastax.bdp.search.lucene.store.bytebuffer.ByteBufferIndexInput.switchCurrentBuffer(ByteBufferIndexInput.java:188)
        at com.datastax.bdp.search.lucene.store.bytebuffer.ByteBufferIndexInput.readByte(ByteBufferIndexInput.java:129)
        at org.apache.lucene.store.DataInput.readShort(DataInput.java:77)
        at com.datastax.bdp.search.lucene.store.bytebuffer.ByteBufferIndexInput.readShort(ByteBufferIndexInput.java:89)
        at org.apache.lucene.util.packed.DirectPackedReader.get(DirectPackedReader.java:64)
        ... 15 more

我从异常跟踪中看到它与 Long 值和 EOF 有关。但是，我不知道是什么触发了错误。我尝试导入的 SSTable 文件是由 Java 程序（由我编写）生成的，该程序使用 org.apache.cassandra.io.sstable.CQLSSTableWriter.

CF 模式、Solr 模式和 SSTable 生成器代码可以在此处找到： https://www.dropbox.com/sh/1rpo3ixmz1bg9y2/AAA3aqlfzWEsNIwy79G9dASba

附：

• 我最近从 DSE 4.1.3 升级到 4.5.1。我不记得在升级之前看到过这个错误
• 生成器的类路径中包含的 cassandra 库的版本为 2.0.8。在 DSE 升级之前，它使用版本 2.0.5 库
• DSE 拓扑：1 个 DC、6 个 solr 节点（禁用 vnode）、RF 2
• 其他 DSE 配置：LeveledCompaction、LZ4 压缩、GossipingPropertyFileSnitch
• 机器规格：CentOS 6.5 x64，JDK 1.7.0_55，六核，堆大小 120GB（我们有需要此的特定查询），总 RAM 128GB

我最初在 6 个节点中的 3 个节点中遇到了该错误。我重新启动了所有这些记录，并且能够导入 150 多万条记录，并且没有出现错误。但是，当我在睡觉时让导入无人看管时，错误就会在 6 个节点中的 1 个中重新出现。

我现在非常震惊，因为每个节点中索引记录的数量（根据 Solr 管理 UI）比 Cassandra 行数（根据 nodetool cfstats）少了大约 60,000 条记录

更新：

仍然继续经历这一点。索引文档（Solr）和存储文档（Cassandra cfstats）数量之间的差异日益增大

更新（2014-08-13）：

按照 Rock Brain 的建议更改了目录工厂；但通过sstableloader连续导入几个小时后又出现错误

更新（2014-08-14）：

有趣的是，我注意到我实际上遇到了两个类似的异常（区别只是最后一个“引起的”的堆栈跟踪：

例外1：

ERROR [Lucene Merge Thread #24937] 2014-08-14 06:20:32,270 CassandraDaemon.java (line 199) Exception in thread Thread[Lucene Merge Thread #24937,6,main]
org.apache.lucene.index.MergePolicy$MergeException: java.lang.IllegalStateException: failed
        at org.apache.lucene.index.ConcurrentMergeScheduler.handleMergeException(ConcurrentMergeScheduler.java:545)
        at org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:518)
Caused by: java.lang.IllegalStateException: failed
        at org.apache.lucene.util.packed.DirectPackedReader.get(DirectPackedReader.java:93)
        at org.apache.lucene.util.packed.BlockPackedReader.get(BlockPackedReader.java:86)
        at org.apache.lucene.util.LongValues.get(LongValues.java:35)
        at org.apache.lucene.codecs.lucene45.Lucene45DocValuesProducer$5.getOrd(Lucene45DocValuesProducer.java:459)
        at org.apache.lucene.codecs.DocValuesConsumer$4$1.setNext(DocValuesConsumer.java:389)
        at org.apache.lucene.codecs.DocValuesConsumer$4$1.hasNext(DocValuesConsumer.java:352)
        at org.apache.lucene.codecs.lucene45.Lucene45DocValuesConsumer.addNumericField(Lucene45DocValuesConsumer.java:141)
        at org.apache.lucene.codecs.lucene45.Lucene45DocValuesConsumer.addSortedField(Lucene45DocValuesConsumer.java:350)
        at org.apache.lucene.codecs.perfield.PerFieldDocValuesFormat$FieldsWriter.addSortedField(PerFieldDocValuesFormat.java:116)
        at org.apache.lucene.codecs.DocValuesConsumer.mergeSortedField(DocValuesConsumer.java:305)
        at org.apache.lucene.index.SegmentMerger.mergeDocValues(SegmentMerger.java:197)
        at org.apache.lucene.index.SegmentMerger.merge(SegmentMerger.java:116)
        at org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:4058)
        at org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:3655)
        at org.apache.lucene.index.ConcurrentMergeScheduler.doMerge(ConcurrentMergeScheduler.java:405)
        at org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:482)
Caused by: java.io.EOFException: Read past EOF (resource: BBIndexInput(name=_67nex_Lucene45_0.dvd))
        at com.datastax.bdp.search.lucene.store.bytebuffer.ByteBufferIndexInput.switchCurrentBuffer(ByteBufferIndexInput.java:188)
        at com.datastax.bdp.search.lucene.store.bytebuffer.ByteBufferIndexInput.readByte(ByteBufferIndexInput.java:129)
        at org.apache.lucene.util.packed.DirectPackedReader.get(DirectPackedReader.java:64)
        ... 15 more

异常 2（与本文顶部的原始异常完全相同）：

ERROR [Lucene Merge Thread #24936] 2014-08-14 06:20:34,694 CassandraDaemon.java (line 199) Exception in thread Thread[Lucene Merge Thread #24936,6,main]
org.apache.lucene.index.MergePolicy$MergeException: java.lang.IllegalStateException: failed
        at org.apache.lucene.index.ConcurrentMergeScheduler.handleMergeException(ConcurrentMergeScheduler.java:545)
        at org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:518)
Caused by: java.lang.IllegalStateException: failed
        at org.apache.lucene.util.packed.DirectPackedReader.get(DirectPackedReader.java:93)
        at org.apache.lucene.util.packed.BlockPackedReader.get(BlockPackedReader.java:86)
        at org.apache.lucene.util.LongValues.get(LongValues.java:35)
        at org.apache.lucene.codecs.lucene45.Lucene45DocValuesProducer$5.getOrd(Lucene45DocValuesProducer.java:459)
        at org.apache.lucene.codecs.DocValuesConsumer$4$1.setNext(DocValuesConsumer.java:389)
        at org.apache.lucene.codecs.DocValuesConsumer$4$1.hasNext(DocValuesConsumer.java:352)
        at org.apache.lucene.codecs.lucene45.Lucene45DocValuesConsumer.addNumericField(Lucene45DocValuesConsumer.java:141)
        at org.apache.lucene.codecs.lucene45.Lucene45DocValuesConsumer.addSortedField(Lucene45DocValuesConsumer.java:350)
        at org.apache.lucene.codecs.perfield.PerFieldDocValuesFormat$FieldsWriter.addSortedField(PerFieldDocValuesFormat.java:116)
        at org.apache.lucene.codecs.DocValuesConsumer.mergeSortedField(DocValuesConsumer.java:305)
        at org.apache.lucene.index.SegmentMerger.mergeDocValues(SegmentMerger.java:197)
        at org.apache.lucene.index.SegmentMerger.merge(SegmentMerger.java:116)
        at org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:4058)
        at org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:3655)
        at org.apache.lucene.index.ConcurrentMergeScheduler.doMerge(ConcurrentMergeScheduler.java:405)
        at org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:482)
Caused by: java.io.EOFException: Read past EOF (resource: BBIndexInput(name=_67fvk_Lucene45_0.dvd))
        at com.datastax.bdp.search.lucene.store.bytebuffer.ByteBufferIndexInput.switchCurrentBuffer(ByteBufferIndexInput.java:188)
        at com.datastax.bdp.search.lucene.store.bytebuffer.ByteBufferIndexInput.readByte(ByteBufferIndexInput.java:129)
        at org.apache.lucene.store.DataInput.readShort(DataInput.java:77)
        at com.datastax.bdp.search.lucene.store.bytebuffer.ByteBufferIndexInput.readShort(ByteBufferIndexInput.java:89)
        at org.apache.lucene.util.packed.DirectPackedReader.get(DirectPackedReader.java:64)
        ... 15 more

更新第 2 部分 (2014-08-14)：

示例重新加载警告：

 WARN [http-8983-2] 2014-08-14 08:31:28,828 CassandraCoreContainer.java (line 739) Too much waiting for new searcher...
 WARN [http-8983-2] 2014-08-14 08:31:28,831 SolrCores.java (line 375) Tried to remove core myks.mycf from pendingCoreOps and it wasn't there.
 INFO [http-8983-2] 2014-08-14 08:31:28,832 StorageService.java (line 2644) Starting repair command #3, repairing 0 ranges for keyspace solr_admin
 INFO [http-8983-2] 2014-08-14 08:31:28,835 SolrDispatchFilter.java (line 672) [admin] webapp=null path=/admin/cores params={slave=true&deleteAll=false&name=myks.mycf&distributed=false&action=RELOAD&reindex=false&core=myks.mycf&wt=javabin&version=2} status=0 QTime=61640

更新（2014-08-23）：

重新执行建议后，我无法再重现异常 解决方法

Answer 1

更新所有核心的 solrconfig.xml：交换 directoryFactory 从 com.datastax.bdp.cassandra.index.solr.DSENRTCachingDirectoryFactory 到 solr.MMapDirectoryFactory.

另外，正在使用什么操作系统、JVM 版本、CPU 数量、堆大小、总可用内存。加载多少分钟/小时后发生错误。