Datastax solr: Cassandra will now flush up to the two largest memtables to free up memory

Question

I'm working with datastax 3.1 on a single node with 4Go of RAM. I have not change anything in cassandra-en.sh and cassandra.yaml except the "--Xss" (because of my java version which require a little more) So by default Cassandra set to 1Go my -Xms and -Xmx parameters: -Xms1024M -Xmx1024M

But while inserting my data after around 200 000 rows (in 3 different column_families), Solr and cassandra logs keep repeat this kind of warning:

WARN StorageService Flushing CFS(Keyspace='OpsCenter',​ ColumnFamily='rollups60') to relieve memory pressure 17:58:07

WARN GCInspector Heap is 0.8825103486201678 full. You may need to reduce memtable and/or cache sizes. Cassandra will now flush up to the two largest memtables to free up memory. Adjust flush_largest_memtables_at threshold in cassandra.yaml if you don't want Cassandra to do this automatically

So, OK my heap is full, but why after flushing, is my heap still full ?

If I stop inserting data at this point. Warning keep repeating. If I stop and restart cassandra. No problem raise

It looks like memory leak issue right? So where should I look at?

Thanks for futur help.

Answer 1

One thing that's a memory hog is Solr's caches. Take a look at your solrconfig.xml file inside the "conf" dir of each of your Solr cores, and look at the value configured for caches such as:

<filterCache class="solr.FastLRUCache"
             size="100"
             initialSize="0"
             autowarmCount="0"/>

There may be multiple entries like this one. Make sure that, at least the autowarmCount and initialSize are set to 0. Further more, lower the "size" value to something small, like 100 or something. All these values refer to number of entries in the cache.

Another thing that may help is configuring Solr to do hard-commits more often. Look for an entry such as:

 <!-- stuff ommited for brevity -->

 <autoCommit> 
   <maxDocs>5000</maxDocs> 
       <maxTime>15000</maxTime> 
       <openSearcher>false</openSearcher> 
 </autoCommit>

The above settings will commit to disk each time 5000 documents have been added or 15 seconds have passed since the last commit, which ever comes first. Also set openSearcher to false.

Finally, look for these entries and set them as follows:

<ramBufferSizeMB>16</ramBufferSizeMB>
<maxBufferedDocs>5000</maxBufferedDocs>

Now, making all this modifications on Solr at once will surely make it run a lot slower. Try instead to make them incrementally, until you get rid of the memory error. Also, it may simply be that you need to allocate more memory to your Java process. If you say the machine has 4 Gb of RAM, why not try your test with -Xmx2g or -Xmx3g ?

Answer 2

Cassandra is trying to clear up heap space, however flushing memtables doesn't flush Solr heap data structures.

For the index size you have, combined with possibly queries that load the Lucene field caches there is not enough heap space allocated. The best advice is to allocate more heap space.

To view the field cache memory usage:

http://www.datastax.com/docs/datastax_enterprise3.1/solutions/dse_search_core_status