Magento 2.1.3 : Indexer creates an extra index on production, not on local development

Question

I've already created an issue about this, but I've got a strange problem. Locally, when I run indexer:reindex everything works fine. However, when I run it on producten I get thrown a MYSQL error:

[Zend_Db_Statement_Exception]
    SQLSTATE[42000]: Syntax error or access violation: 1170 BLOB/TEXT column 'created_in' used in key specification without a key length, query was: CREATE T
    ABLE IF NOT EXISTS `customer_grid_flat` (
    `entity_id` int UNSIGNED NOT NULL COMMENT 'Entity ID' ,
    `name` text NULL COMMENT 'Name' ,
    `email` varchar(255) NULL COMMENT 'Email' ,
    `group_id` int NULL COMMENT 'Group_id' ,
    `created_at` timestamp NULL default NULL COMMENT 'Created_at' ,
    `website_id` int NULL COMMENT 'Website_id' ,
    `confirmation` varchar(255) NULL COMMENT 'Confirmation' ,
    `created_in` text NULL COMMENT 'Created_in' ,
    `dob` date NULL COMMENT 'Dob' ,
    `gender` int NULL COMMENT 'Gender' ,
    `taxvat` varchar(255) NULL COMMENT 'Taxvat' ,
    `lock_expires` timestamp NULL default NULL COMMENT 'Lock_expires' ,
    `shipping_full` text NULL COMMENT 'Shipping_full' ,
    `billing_full` text NULL COMMENT 'Billing_full' ,
    `billing_firstname` varchar(255) NULL COMMENT 'Billing_firstname' ,
    `billing_lastname` varchar(255) NULL COMMENT 'Billing_lastname' ,
    `billing_telephone` varchar(255) NULL COMMENT 'Billing_telephone' ,
    `billing_postcode` varchar(255) NULL COMMENT 'Billing_postcode' ,
    `billing_country_id` varchar(255) NULL COMMENT 'Billing_country_id' ,
    `billing_region` varchar(255) NULL COMMENT 'Billing_region' ,
    `billing_street` varchar(255) NULL COMMENT 'Billing_street' ,
    `billing_city` varchar(255) NULL COMMENT 'Billing_city' ,
    `billing_fax` varchar(255) NULL COMMENT 'Billing_fax' ,
    `billing_vat_id` varchar(255) NULL COMMENT 'Billing_vat_id' ,
    `billing_company` varchar(255) NULL COMMENT 'Billing_company' ,
    PRIMARY KEY (`entity_id`),
    INDEX `CUSTOMER_GRID_FLAT_GROUP_ID` (`group_id`),
    INDEX `CUSTOMER_GRID_FLAT_CREATED_AT` (`created_at`),
    INDEX `CUSTOMER_GRID_FLAT_WEBSITE_ID` (`website_id`),
    INDEX `CUSTOMER_GRID_FLAT_CONFIRMATION` (`confirmation`),
    INDEX `CUSTOMER_GRID_FLAT_CREATED_IN` (`created_in`),
    INDEX `CUSTOMER_GRID_FLAT_DOB` (`dob`),
    INDEX `CUSTOMER_GRID_FLAT_GENDER` (`gender`),
    INDEX `CUSTOMER_GRID_FLAT_LOCK_EXPIRES` (`lock_expires`),
    INDEX `CUSTOMER_GRID_FLAT_BILLING_COUNTRY_ID` (`billing_country_id`),
    FULLTEXT `FTI_AC9FBFAE9FBD3A0B02DBA986349612A6` (`name`, `email`, `taxvat`, `shipping_full`, `billing_full`, `billing_firstname`, `billing_lastname`, `
    billing_telephone`, `billing_postcode`, `billing_region`, `billing_street`, `billing_city`, `billing_fax`, `billing_vat_id`, `billing_company`)
    ) COMMENT='customer_grid_flat' ENGINE=INNODB charset=utf8 COLLATE=utf8_general_ci

I've compared the query building locally with the one on production, and the only difference there is, is that on production Magentos' indexer for some reason determines that created_in should also be an index.

Why does it do that? Can anyone point me to the right direction on where to look? I've been searching for this issue for 3 days now.

Answer 1

I found the answer to my own question.

Turns out it wasn't Magento-related, but an error in our deployment tool. Like @Marius pointed out, the attributes' filterable/searchable values weren't properly set.

This is because our deployment makes a 'shalow copy' of the production database (with only EAV attributes, configuration settings, design / theme updates, etc. and not orders/customers/products) to compile against (since Magento needs a database to compile it's code).

In Magento 2.1.2, the customer_eav_attribute e.d. were not required to run a valid setup:upgrade or index:reindex, but my guess is that in Magento 2.1.3 this has been improved. Thus I had to upgrade our deployment tool.

Thanks for looking into this @Marius! IMHO the indexing process of Magento is one of the most abstract and complex beasts to comprehend. Hope to get some good solid documentation about that soon on how to write your own indexers.

Answer 2

TL;DR.
I think there is a problem with your created_in column from the customer_entity table in one of the environments. For some reason the size of the column is not taken into consideration. May be the mysql version but this is just a speculation.

[EDIT]
The above assumption was wrong.
The problem appears because the attribute created_in is marked as is_filterable_in_grid = 1 and is_searchable_in_grid = 0 in the customer_eav_attribute table.
Make is_searchable_in_grid = 1 (as it should be by default) and everything should be OK.
Make it
[/EDIT]

Long version: Here are some pieces of the puzzle I found. You can try to put them all together. I hope there are enough pieces to realize the big picture.

Here is the code that builds the flat tables. It's an abstract method but the way the indexes are created is clear Magento\Framework\Indexer\GridStructure::createFlatTable().

This part is important.

if ($field['type'] === 'filterable') {
    $table->addIndex(
        $this->resource->getIdxName($tableName, $name, AdapterInterface::INDEX_TYPE_INDEX),
        $name,
        ['type' => AdapterInterface::INDEX_TYPE_INDEX]
    );
}

This means that if the field is declared as filtrerable it will be added as an index in the flat table.
Now let's look at the indexer.xml file from the customer module.. The created_in attribute is marked as filterable and it has the type text.
Now lets go back to the Indexer class before and see what type text means..

protected $columnTypesMap = [
    'varchar'    => ['type' => Table::TYPE_TEXT, 'size' => 255],
    'mediumtext' => ['type' => Table::TYPE_TEXT, 'size' => 16777216],
    'text'       => ['type' => Table::TYPE_TEXT, 'size' => 65536],
    'int'        => ['type' => Table::TYPE_INTEGER, 'size' => null],
    'date'       => ['type' => Table::TYPE_DATE, 'size' => null],
    'datetime'   => ['type' => Table::TYPE_DATETIME, 'size' => null],
    'timestamp'  => ['type' => Table::TYPE_TIMESTAMP, 'size' => null],
]; 

This means that the the column type will be text and the size will be 65536.
But for some reason the size is ignored when creating the table. And you get the error when you try to add the index because there is no size on the column (as the error states).

i tried to reindex on my instance and indeed the column created_in does not have an index. That's what made me speculate that there is mysql version issue, or maybe a configuration issue. For the record I have mysql 5.7.16-0ubuntu0.16.04.1.

Other info that might prove useful, the actual query for creating the table is executed in Magento\Framework\DB\Adapter\Pdo\Mysql::createTable. Good place to start debugging and then go back in the stack trace.
Also, before trasnforming the index configuration to the actual sql text, there is an other validation done my the method _getIndexesDefinition in the same Magento\Framework\DB\Adapter\Pdo\Mysql class.

That's all I have for now. Happy debugging.

[EDIT]

This is bugging me so I decided to invest a bit in it.
I debugged the createFlatTable method mentioned above and I see that for the created_in attribute comes up with type searchable even if in the indexer.xml it appears as filterable. I don't know yet why that happens. will post back when I find out why.

[Second degree edit] - this is actually the solution, the rest is just debugging steps I took.
Continuing the idea from the edit above, the type of the flat column filterable or searchable is determined for the customer attributes by the method Magento\Customer\Model\Indexer\AttributeProvider::getType.

protected function getType(Attribute $attribute)
{
    if ($attribute->canBeSearchableInGrid()) {
        $type = 'searchable';
    } elseif ($attribute->canBeFilterableInGrid()) {
        $type = 'filterable';
    } else {
        $type = 'virtual';
    }

    return $type;
}

In your case, you get the attribute type filterable and because this is a text attribute mysql cannot create the index on it.
Make sure your attribute is marked as is_searchable_in_grid and it will get the type searchable when the flat table is built and no index will be added to it.