Can Solr handle data import from tables like table1,table2,…tableN?

Question

I have an mysql db, and the data tables named according to the sharding rule, which means the name has a same prefix,e.g: the_table_,and has a number for suffix, so the table name will be the_table_1,the_table_2,the_table_3.When we want to select a row, we have to find the table suffix from sql select mod(primary_key_id, 64)+1, the sharding rule, and the 64 is a base number, which means we have 64 tables.

Now we want to use Solr to index the data in the tables,but solr data-config.xml can not support sql query with dynamic name,So is there any suggestion to fix this problem?

Things like this?

<entity name="audit" 
        query="select id,monitor_type,city,STATUS,is_history,timeout,author,author_uid,author_ip,operator_uid,operator,created_at,description from `main_table`">
        <field column="ID" name="id" />
        <field column="MONITOR_TYPE" name="monitor_type" />
        <field column="CITY" name="city" />
        <field column="STATUS" name="status" />
        <field column="IS_HISTORY" name="is_history" />
        <field column="AUTHOR" name="author" />
        <field column="AUTHOR_UID" name="author_uid" />
        <field column="AUTHOR_IP" name="author_ip" />
        <field column="OPERATOR_UID" name="operator_uid" />
        <field column="OPERATOR" name="operator" />

        <entity name="distribution" query="select mod(${audit.ID}, 256)+1 as table_id">
          <entity name="details" query="select content from detail${distribution.table_id} where id=${audit.ID}">
            <field column="CONTENT" name="content" />
          </entity>
        </entity>
</entity>

Answer 1

I would create a stored procedure to do what you want and then call it from Solr. Something like this

CREATE PROCEDURE `GetAuditDetails`(IN auditID INT)
BEGIN
  DECLARE tableID INT DEFAULT 0;
  DECLARE sqlString VARCHAR(100) DEFAULT '';
  SELECT MOD(auditID, 256)+1 INTO tableID;
  SET @sqlString = CONCAT('select content from detail', tableID, ' where id=', auditID );
  PREPARE stmt FROM @sqlString;
  EXECUTE stmt;
END;

<entity name="audit" 
    query="select id,monitor_type,city,STATUS,is_history,timeout,author,author_uid,author_ip,operator_uid,operator,created_at,description from `main_table`">
        <field column="ID" name="id" />
        <field column="MONITOR_TYPE" name="monitor_type" />
        <field column="CITY" name="city" />
        <field column="STATUS" name="status" />
        <field column="IS_HISTORY" name="is_history" />
        <field column="AUTHOR" name="author" />
        <field column="AUTHOR_UID" name="author_uid" />
        <field column="AUTHOR_IP" name="author_ip" />
        <field column="OPERATOR_UID" name="operator_uid" />
        <field column="OPERATOR" name="operator" />

        <entity name="details" query="CALL GetAuditDetails(${audit.ID})">
           <field column="CONTENT" name="content" />
        </entity>
</entity>

Answer 2

First, it should not use audit.ID in the nested entry query,but ${audit.id}, otherwise there will be an exception. Second,for Mysql procedure:

CREATE PROCEDURE `GetAuditDetails`(IN auditID INT)
BEGIN
  DECLARE tableID INT DEFAULT 0;
  DECLARE sqlString VARCHAR(100) DEFAULT '';
  SELECT MOD(auditID, 256)+1 INTO tableID;
  SET @sqlString = CONCAT('select content from detail', tableID, ' where id=', auditID );
  PREPARE stmt FROM @sqlString;
  EXECUTE stmt;
END;