How does Magento manage foreign key constraint

Question

I recently upgrade to mage 1.8.1. seems everything fine but re-index product price. found error below:

#1452 - Cannot add or update a child row: a foreign key constraint fails (`threeleavednew`.`catalog_product_index_tier_price`, CONSTRAINT `FK_CAT_PRD_IDX_TIER_PRICE_ENTT_ID_CAT_PRD_ENTT_ENTT_ID` FOREIGN KEY (`entity_id`) REFERENCES `catalog_product_entity` (`entity_id)

This DB was exported by phpMyadmin before upgrade and imported to a new empty DB (created under my domain hosting). I just extracted mage 1.8.1.tar.gz and copy over my theme and point Mage 1.8 to the new DB by modifying local.xml (keep encryption key unchanged), rename mage 1.8 directory to httpdocs and launch my site....

1. I am not sure if this way of upgrade will cause any DB foreign key problem.
2. My old site was transferred from other developer, I am not sure if the DB was with FK problem from day one which might be not compatible with ver 1.8.

What I have done is

1) to use below statement in phpMyadmin

select *
from INFORMATION_SCHEMA.TABLE_CONSTRAINTS
where CONSTRAINT_TYPE = 'FOREIGN KEY';

to dump all foreign keys from old DB, current DB and Mage sample DB. make comparison.

2) in phpMyadmin run export to dump current DB and Mage sample DB, make comparison with old DB exported before upgrade.

Here are outcome.

1) in Mage sample DB foreign keys table I can find three FK keys for catalog_product_index_tier_price table as below

'FK_CAT_PRD_IDX_TIER_PRICE_CSTR_GROUP_ID_CSTR_GROUP_CSTR_GROUP_ID',
'FK_CAT_PRD_IDX_TIER_PRICE_ENTT_ID_CAT_PRD_ENTT_ENTT_ID', 
'FK_CAT_PRD_IDX_TIER_PRICE_WS_ID_CORE_WS_WS_ID', 

and one FK each for catalog_product_flat_1, _2, _3 tables.

'FK_CAT_PRD_FLAT_1_ENTT_ID_CAT_PRD_ENTT_ENTT_ID',
'FK_CAT_PRD_FLAT_2_ENTT_ID_CAT_PRD_ENTT_ENTT_ID', 
'FK_CAT_PRD_FLAT_3_ENTT_ID_CAT_PRD_ENTT_ENTT_ID',

However, when I tried to dump FK table from old DB and current DB in different time, I some time can find the three FK for catalog_product_index_tier_price table, some time I can not find them at all, some time find FK name like 'FK_6E08D719F0501DD1D8E6D4EFF2511C85'. and I never find any FK for catalog_product_flat_1, _2, _3 tables.

2) in both old DB and current DB exported .sql files, I do not find any such statement as below to add FK for table catalog_product_index_tier_price and catalog_product_flat_1, _2, _3 , but I found them in exported .sql file from Mage sample DB.

ALTER TABLE `catalog_product_index_tier_price`
ADD CONSTRAINT `FK_CAT_PRD_IDX_TIER_PRICE_CSTR_GROUP_ID_CSTR_GROUP_CSTR_GROUP_ID` FOREIGN KEY (`customer_group_id`) REFERENCES `customer_group` (`customer_group_id`) ON DELETE CASCADE ON UPDATE CASCADE,
ADD CONSTRAINT `FK_CAT_PRD_IDX_TIER_PRICE_ENTT_ID_CAT_PRD_ENTT_ENTT_ID` FOREIGN KEY (`entity_id`) REFERENCES `catalog_product_entity` (`entity_id`) ON DELETE CASCADE ON UPDATE CASCADE,
ADD CONSTRAINT `FK_CAT_PRD_IDX_TIER_PRICE_WS_ID_CORE_WS_WS_ID` FOREIGN KEY (`website_id`) REFERENCES `core_website` (`website_id`) ON DELETE CASCADE ON UPDATE CASCADE;

and

ALTER TABLE `catalog_product_flat_1`
ADD CONSTRAINT `FK_CAT_PRD_FLAT_1_ENTT_ID_CAT_PRD_ENTT_ENTT_ID` FOREIGN KEY (`entity_id`) REFERENCES `catalog_product_entity` (`entity_id`) ON DELETE CASCADE ON UPDATE CASCADE;

OK, it looks like both my old DB and current DB got FK problem with unconstant symptom. It is obvious that my all history DB backup files has no statement to add FK for those tables which one of them I have used to import DB for the mage 1.8 run where I got problem. I have no way to check my cureent problem is caused by wrong way of upgrade or it potentially exist since day one.

My question are

1. Does Mage 1.8 manage DB differently from previous version? even though I hardly think so.
2. How does Magento manage FK constraint actually? does it assign FK constraint names and relationships at very beginning and keep all of them statically unchanged or it assign/reassign FK constraint name and relationships dynamically with sale or customer changes? My understanding was that FK constraint name and relationship will never change, its data changes with site running.
3. Why phpMyadmin export function dump a DB backup (even before upgrade when site is working fine) with much less "add constrain" statement than the same dump on Mage sample DB? does this mean my DB had wrong/problem FK since day one? if yes, why I only got problem after upgrade to 1.8?
4. Assuming Magento manage FKs statically, Is there a standard/reference Magento FK constraint list which we can make comparison with our problem one for easy DB debug?

I am really getting lost, hope someone can help me to sort these out. Thank you in advance. I saw many mage FK error threads all over google search similar to mine, non of them give me a rational and logical solution and explanation. I expect a solution where I can see why and how it solve the problem rather than just telling me "truncate some tables, you will get problem solved"....

Thank you very much.

Update: I have tried to track query flow and find some info, hope the master will know where the problem is.

## 17493 ## TRANSACTION BEGIN
TIME: 0.0001
## 2014-03-28 18:00:03
## 17493 ## QUERY
SQL: DELETE FROM `catalog_product_index_price_idx`
AFF: 410
TIME: 0.0055
## 2014-03-28 18:00:03
## 17493 ## QUERY
SQL: SELECT `cw`.`website_id`, `csg`.`default_store_id` AS `store_id` FROM `core_website` AS `cw`
INNER JOIN `core_store_group` AS `csg` ON cw.default_group_id = csg.group_id WHERE  (cw.website_id != 0)
AFF: 1
TIME: 0.0006
## 2014-03-28 18:00:03
## 17493 ## QUERY
SQL: DELETE FROM `catalog_product_index_website`
AFF: 1
TIME: 0.0003
## 2014-03-28 18:00:03
## 17493 ## QUERY
SQL: INSERT INTO `catalog_product_index_website` (`website_id`,`website_date`,`rate`) VALUES  (?, ?, ?)
BIND: array (  0 => '1',  1 => '2014-03-29',  2 => 1,)
AFF: 1
TIME: 0.0002
## 2014-03-28 18:00:03
## 17493 ## QUERY
SQL: DELETE FROM `catalog_product_index_tier_price`
AFF: 0
TIME: 0.0002
## 2014-03-28 18:00:03
## 17493 ## QUERY
SQL: DESCRIBE `catalog_product_index_tier_price`
AFF: 4
TIME: 0.0005

## 2014-03-28 18:00:03
## 17493 ## QUERY
SQL: INSERT INTO `catalog_product_index_tier_price` SELECT `tp`.`entity_id`, `cg`.`customer_group_id`, `cw`.`website_id`, MIN(IF(tp.website_id = 0, ROUND(tp.value * cwd.rate, 4), tp.value)) FROM `catalog_product_entity_tier_price` AS `tp`
INNER JOIN `customer_group` AS `cg` ON tp.all_groups = 1 OR (tp.all_groups = 0 AND tp.customer_group_id = cg.customer_group_id)
INNER JOIN `core_website` AS `cw` ON tp.website_id = 0 OR tp.website_id = cw.website_id
INNER JOIN `catalog_product_index_website` AS `cwd` ON cw.website_id = cwd.website_id WHERE (cw.website_id != 0) GROUP BY `tp`.`entity_id`,
`cg`.`customer_group_id`,
`cw`.`website_id` ON DUPLICATE KEY UPDATE `min_price` = VALUES(`min_price`)
TIME: 0.0069

    ## 2014-03-28 18:00:03
    EXCEPTION 
    exception 'PDOException' with message 'SQLSTATE[23000]: Integrity constraint violation: 1452  Cannot add or update a child row: a foreign key constraint fails (`threeleavednew`.`catalog_product_index_tier_price`, CONSTRAINT `FK_CAT_PRD_IDX_TIER_PRICE_ENTT_ID_CAT_PRD_ENTT_ENTT_ID` FOREIGN KEY (`entity_id`) REFERENCES `catalog_product_entity` (`entity_)' in /var/www/vhosts/3leavedcart.com/httpdocs/lib/Zend/Db/Statement/Pdo.php:228
    Stack trace:
    #0 /var/www/vhosts/3leavedcart.com/httpdocs/lib/Zend/Db/Statement/Pdo.php(228): PDOStatement->execute(Array)
    #1 /var/www/vhosts/3leavedcart.com/httpdocs/lib/Varien/Db/Statement/Pdo/Mysql.php(110): Zend_Db_Statement_Pdo->_execute(Array)
    #2 /var/www/vhosts/3leavedcart.com/httpdocs/app/code/core/Zend/Db/Statement.php(291): Varien_Db_Statement_Pdo_Mysql->_execute(Array)
    #3 /var/www/vhosts/3leavedcart.com/httpdocs/lib/Zend/Db/Adapter/Abstract.php(479): Zend_Db_Statement->execute(Array)
    #4 /var/www/vhosts/3leavedcart.com/httpdocs/lib/Zend/Db/Adapter/Pdo/Abstract.php(238): Zend_Db_Adapter_Abstract->query('INSERT INTO `ca...', Array)
    #5 /var/www/vhosts/3leavedcart.com/httpdocs/lib/Varien/Db/Adapter/Pdo/Mysql.php(428): Zend_Db_Adapter_Pdo_Abstract->query('INSERT INTO `ca...', Array)
#6 /var/www/vhosts/3leavedcart.com/httpdocs/app/code/core/Mage/Catalog/Model/Resource/Product/Indexer    /Price.php(455): Varien_Db_Adapter_Pdo_Mysql->query('INSERT INTO `ca...')
#7 /var/www/vhosts/3leavedcart.com/httpdocs/app/code/core/Mage/Catalog/Model/Resource/Product/Indexer    /Price.php(379): Mage_Catalog_Model_Resource_Product_Indexer_Price->_prepareTierPriceIndex()
    #8 /var/www/vhosts/3leavedcart.com/httpdocs/app/code/core/Mage/Index/Model/Indexer/Abstract.php(143): Mage_Catalog_Model_Resource_Product_Indexer_Price->reindexAll()
    #9 /var/www/vhosts/3leavedcart.com/httpdocs/app/code/core/Mage/Index/Model/Process.php(210): Mage_Index_Model_Indexer_Abstract->reindexAll()
    #10 /var/www/vhosts/3leavedcart.com/httpdocs/app/code/core/Mage/Catalog/Model/Observer.php(203): Mage_Index_Model_Process->reindexAll()
    #11 [internal function]: Mage_Catalog_Model_Observer->reindexProductPrices(Object(Mage_Cron_Model_Schedule))
    #12 /var/www/vhosts/3leavedcart.com/httpdocs/app/code/core/Mage/Cron/Model/Observer.php(325): call_user_func_array(Array, Array)
    #13 /var/www/vhosts/3leavedcart.com/httpdocs/app/code/core/Mage/Cron/Model/Observer.php(72): Mage_Cron_Model_Observer->_processJob(Object(Mage_Cron_Model_Schedule), Object(Mage_Core_Model_Config_Element))
    #14 /var/www/vhosts/3leavedcart.com/httpdocs/app/code/core/Mage/Core/Model/App.php(1338): Mage_Cron_Model_Observer->dispatch(Object(Varien_Event_Observer))
    #15 /var/www/vhosts/3leavedcart.com/httpdocs/app/code/core/Mage/Core/Model/App.php(1317): Mage_Core_Model_App->_callObserverMethod(Object(Mage_Cron_Model_Observer), 'dispatch', Object(Varien_Event_Observer))
    #16 /var/www/vhosts/3leavedcart.com/httpdocs/app/Mage.php(448): Mage_Core_Model_App->dispatchEvent('default', Array)
    #17 /var/www/vhosts/3leavedcart.com/httpdocs/cron.php(76): Mage::dispatchEvent('default')
    #18 {main}

And below is only finding in mysqld.log file

140319 16:41:21  InnoDB: Error: in ALTER TABLE `threeleavednew`.`catalog_product_index_tier_price`
InnoDB: has or is referenced in foreign key constraints
InnoDB: which are not compatible with the new table definition.

I also compared table catalog_product_index_tier_price from my DB with the one from Mage sample DB in both table structure and related constraints name + relationship...in fact, I compared more than 10 tables I suspected, no clue.

Thank you.

Answer 1

Time is running, so I decided to go with my ancient method which was in my head when I first time replyed Ron...it took me more than half a day, but better than spending days on trying other methods with no hope...I am not sure if this could be final solution for those who encounter similar errors in community, just in case someone search this thread.

Here is what I did, after that, reindex product price and tag will be a piece of cake whenever you want.

1. dump your DB and open it with editor to find all FK relationship pointed to 'catalog_product_entity.entity_id' which is parent table, write down the child tables. most of them are started with catalog or tag etc.

2. open your phpMyAdmin or similar tool, use below statement to find any rows in these child tables with Fk 'entity_id' or 'product_id' referencing to 'catalog_product_entity.entity_id' but no longer exist in parent table and delete them all.

   SELECT * FROM child table name WHERE 'product_id' or 'enitty_id' not in (select 'entity_id' from 'catalog_product_entity');

3. Delete only rows you find with this statement and leave rest rows out there.

You may have to do this one table by one table, and some FK is 'product_id', some is 'enitty_id', so be carefull... My DB has some products were deleted before which are not in 'catalog_product_entity' table any more, but their old 'entity_id' are still exist in many other tables with constraint relationship to 'catalog_product_entity' table. What I did is to find out those rows and delete them from child table. Deleting table rows in phpMyadmin is pretty simple.

That is all, go and clear var/cache and var/locks, login beckend admin to do reindex and see all green!

Prerequisite of this mothed is that Magento's all FK constraints (both name and relationship) are defined once and keep constant...which is what I was asked and was not sure at beginning.

Answer 2

I think you are overcomplicating things a bit. Magento has FK constrains that are setup when you install it. What I think may have happened is that your Magento's DB structure got corrupted along the way and just recently with your upgrade to 1.8 - the tables get retouched a bit and when they get retouched the FK constraints get reset to what they should be.

Long story short, you probably have a bad extension or you may want to look at what may have caused your DB structure corruption. It is more likely that the new structure is the proper one since you just upgraded.

Last note: the good thing about the price index is that it is one of those indexes that you can truncate the table and re-apply the index without any consequences. I would recommend that you do that and see if it helps.