TrieDateField in schema.xml conflicts with timestamp in column family

Question

We just upgraded from DataStax Enteprise 3.2.2 to 4.5.1. We migrated from 3.2.2 -> 3.2.5 -> 4.0.3 -> 4.5.1, every time following the procedures in the documentation, and upgrading the sstables after each upgrade.

The servers are running and the core is accepting queries normally.

Now, trying to get rid of sub-optimal settings, I try to reload the SolR core with a new solrconfig.xml, which fails with this error message:

Current validation_class and found SolrType do not match for column: created_at

After some research, we think we have created a legacy (thrift) table using the HTTP API but somehow also set the data mapping type to "2" (for CQL3 tables only). How can we solve this without recreating everything from scratch?

When changing the data mapping type from "2" to "1", and posting the config and schema via the API, I get this:

org.apache.solr.common.SolrException:org.apache.solr.common.SolrException: Solr core peantus.oranges has dseTypeMappingVersion = 2, cannot load new Solr config with dseTypeMappingVersion = 1.

The stack trace for the posting of the solrconfig.xml with dseTypeMappingVersion = 2 is as follows:

ERROR [http-8983-5] 2014-07-31 15:56:39,091 SolrCoreResourceManager.java (line 435) Unable to create core: peanuts.oranges org.apache.solr.common.SolrException: Unable to create core: peanuts.oranges at org.apache.solr.core.CoreContainer.recordAndThrow(CoreContainer.java:957) at com.datastax.bdp.search.solr.core.CassandraCoreContainer.create(CassandraCoreContainer.java:265) at com.datastax.bdp.search.solr.core.SolrCoreResourceManager.reloadCore(SolrCoreResourceManager.java:382) at com.datastax.bdp.search.solr.handler.admin.CassandraCoreAdminHandler.handleReloadAction(CassandraCoreAdminHandler.java:143) at org.apache.solr.handler.admin.CoreAdminHandler.handleRequestBody(CoreAdminHandler.java:178) at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:137) at org.apache.solr.servlet.SolrDispatchFilter.handleAdminRequest(SolrDispatchFilter.java:669) at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:248) at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:197) at com.datastax.bdp.search.solr.servlet.CassandraDispatchFilter.doFilter(CassandraDispatchFilter.java:99) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206) at com.datastax.bdp.cassandra.audit.SolrHttpAuditLogFilter.doFilter(SolrHttpAuditLogFilter.java:218) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206) at com.datastax.bdp.search.solr.auth.CassandraAuthorizationFilter.doFilter(CassandraAuthorizationFilter.java:100) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206) at com.datastax.bdp.search.solr.auth.DseAuthenticationFilter.doFilter(DseAuthenticationFilter.java:102) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206) at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233) at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191) at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127) at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:103) at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109) at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:293) at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:861) at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:606) at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:489) at java.lang.Thread.run(Unknown Source) Caused by: javax.xml.parsers.ParserConfigurationException: Current validation_class and found SolrType do not match for column: created_at. at com.datastax.bdp.search.solr.core.ThriftCassandraSolrSchemaUpdater.addColumnMetaData(ThriftCassandraSolrSchemaUpdater.java:196) at com.datastax.bdp.search.solr.core.ThriftCassandraSolrSchemaUpdater.update(ThriftCassandraSolrSchemaUpdater.java:72) at com.datastax.bdp.search.solr.core.CassandraCoreContainer.create(CassandraCoreContainer.java:244) ... 29 more

The following table was created using DSE Search's HTTP API.

cqlsh:peanuts> DESCRIBE TABLE oranges;

CREATE TABLE oranges (
  key text,
  "_docBoost" text,
  category_id int,
  category_name text,
  city text,
  country_code text,
  created_at timestamp,
  ...
  solr_query text,
  text text,
  user_id text,
  week_of_year int,
  year int,
  PRIMARY KEY ((key))
) WITH COMPACT STORAGE AND
  bloom_filter_fp_chance=0.010000 AND
  caching='KEYS_ONLY' AND
  comment='' AND
  dclocal_read_repair_chance=0.000000 AND
  gc_grace_seconds=864000 AND
  index_interval=128 AND
  read_repair_chance=0.100000 AND
  replicate_on_write='true' AND
  populate_io_cache_on_flush='false' AND
  default_time_to_live=0 AND
  speculative_retry='99.0PERCENTILE' AND
  memtable_flush_period_in_ms=0 AND
  compaction={'class': 'SizeTieredCompactionStrategy'} AND
  compression={'sstable_compression': 'SnappyCompressor'};

Here is the related excerpt from the schema.xml:

   <field name="id" type="string" indexed="true" stored="true" required="true" multiValued="false" />
   <field name="created_at" type="date" indexed="true" stored="true" docValues="true" />
   ...

   <field name="postal_code" type="string" stored="true" indexed="true"/>
   <field name="country_code" type="string" stored="true" indexed="true"/>
   <field name="city" type="string" stored="true" indexed="true"/>
   <field name="latitude"  type="float" indexed="true" stored="true"/>
   <field name="longitude"  type="float" indexed="true" stored="true"/>

   <field name="text" type="text_general" indexed="true" stored="false" multiValued="true"/>

 </fields>
  ....    
  <types>
    <fieldType name="string" class="solr.StrField" sortMissingLast="true" />
    <fieldType name="int" class="solr.TrieIntField" precisionStep="0" positionIncrementGap="0"/>
    <fieldType name="float" class="solr.TrieFloatField" precisionStep="0" positionIncrementGap="0"/>
    <fieldType name="date" class="solr.TrieDateField" precisionStep="0" positionIncrementGap="0"/>

My current assumption is, that we need to change the current solrconfig to a mapping of type 1 as this corresponds to the actual data storage set-up. But how, if the API does not accept the POST?

Alternatively, I could think about migrating to a pure CQL3 table, but this would require a complete re-generation of the data from scratch, right? Can this be avoided?

Answer 1

The problem is your type mapping at version 2, which is expecting a TimestampType, while Thrift uses DateType.

You need to "force" a downgrade to type 1, by adding the following snippet in your Solr config:

<dseTypeMappingVersion force="true">1</dseTypeMappingVersion>

Then upload and reload the core.