Data Migration not procceds after Map Step

Question

ubuntu@ip-172-31-32-129:/var/www/magento$ sudo php bin/magento migrate:data -r vendor/magento/data-migration-tool/etc/opensource-to-opensource/1.9.3.1/config.xml
Reset the current position of migration to start from the beginning

[2019-04-08 07:04:28][INFO][mode: data][stage: integrity check][step: Data Integrity Step]: started
100% [============================] Remaining Time: < 1 sec
[2019-04-08 07:06:10][INFO][mode: data][stage: integrity check][step: EAV Step]: started
100% [============================] Remaining Time: < 1 sec
[2019-04-08 07:06:13][INFO][mode: data][stage: integrity check][step: Customer Attributes Step]: started
100% [============================] Remaining Time: < 1 sec
[2019-04-08 07:06:15][INFO][mode: data][stage: integrity check][step: Map Step]: started
100% [============================] Remaining Time: < 1 sec
[2019-04-08 07:06:48][INFO][mode: data][stage: integrity check][step: Url Rewrite Step]: started
100% [============================] Remaining Time: < 1 sec
[2019-04-08 07:06:48][INFO][mode: data][stage: integrity check][step: Log Step]: started
100% [============================] Remaining Time: < 1 sec
[2019-04-08 07:06:49][INFO][mode: data][stage: integrity check][step: Ratings Step]: started
100% [============================] Remaining Time: < 1 sec
[2019-04-08 07:06:49][INFO][mode: data][stage: integrity check][step: ConfigurablePrices step]: started
100% [============================] Remaining Time: < 1 sec
[2019-04-08 07:06:49][INFO][mode: data][stage: integrity check][step: OrderGrids Step]: started
100% [============================] Remaining Time: < 1 sec
[2019-04-08 07:06:49][INFO][mode: data][stage: integrity check][step: Tier Price Step]: started
100% [============================] Remaining Time: < 1 sec
[2019-04-08 07:06:49][INFO][mode: data][stage: integrity check][step: SalesIncrement Step]: started
100% [============================] Remaining Time: < 1 sec
[2019-04-08 07:06:49][INFO][mode: data][stage: integrity check][step: Inventory Step]: started
100% [============================] Remaining Time: < 1 sec
[2019-04-08 07:06:49][INFO][mode: data][stage: setup triggers][step: Stage]: started
100% [============================] Remaining Time: < 1 sec
[2019-04-08 07:07:19][INFO][mode: data][stage: data migration][step: EAV Step]: started
100% [============================] Remaining Time: < 1 sec
[2019-04-08 07:07:30][INFO][mode: data][stage: volume check][step: EAV Step]: started
100% [============================] Remaining Time: < 1 sec
[2019-04-08 07:07:46][INFO][mode: data][stage: data migration][step: Customer Attributes Step]: started
100% [============================] Remaining Time: < 1 sec
[2019-04-08 07:08:02][INFO][mode: data][stage: volume check][step: Customer Attributes Step]: started
100% [============================] Remaining Time: < 1 sec
[2019-04-08 07:08:03][INFO][mode: data][stage: data migration][step: Map Step]: started
12% [===>------------------------] Remaining Time: 18 mins

Magento not showing any kind of error only waiting time increase in map steps. Only customer migrate properly not any product etc Any one have idea how to debug this error.

Source Magento have 9000 customer and 100 product and 13000 orders still it taking to much time process never terminated at map step only time increase not even showing any error.

If i check at admin and database only customer record and attributes are migrated..... not anything else... like orders...

Source Magento : 1.9.3.1
Destination Magento : 2.3.1

Answer 1

After Debugging at file

:vendor/magento/data-migration-tool/src/Migration/Step/Map/Data.php

Print Source table name and destination table find that issue with

catalog_product_entity_media_gallery

crosscheck record so finds the issue and fixed that. issue related to orphan records so it failed and go to infinite loop so I just ignore that table because I just want to migrate customer and orders

Answer 2

I realise you found the problem that was responsible in your particular case, but I thought I'd add an extra answer here, as my team had the same symptoms but a different explanation. Hopefully this will be of use to others.

tl;dr: We were migrating from USA to Europe and the latency was high. The migration hadn't got stuck, it was just incredibly slow. It would have taken days if we had persevered.

We left the data migration tool running overnight. After approx. 15 hours, it was still stuck on the map step, at 13%. The migration log file provided no clues. We monitored for half an hour and saw no change in the database. We figured it was stuck.

We were wrong. After some debugging we found that the data transformation took about 0.4 seconds per record, so after 15 hours it had only managed to process roughly 135,000 records. In a Magento database that's really not very much. Added to which, it operates in blocks of records, only updating the destination database once a block has been processed. In our case the block size was 50,000, so that explains why we saw no change in the database over the course of half an hour.

We eventually found the problem was database read latency. We were running the script in an EU-based machine, reading from a US database. When we ran the script on a US machine it was immeasurably faster (literally - the timestamps in our logging read 0 seconds per record).

Answer 3

Same problem, but different reason, just give everyone a reference.

I put the debug codes echo $sourceDocName.PHP_EOL; in \Migration\Step\Map\Data::perform method, and found the reason.

My sales_flat_quote table has millions of data, when migration, it looks like stuck, but actually there is too much data to spend time. We need go to database to see the process.

When you have a very large table to migrate, it will looks like stuck. You need debug the \Migration\Step\Map\Data::perform method, sometime it is because the table is too large, sometime it is because the wrong data in table as others have answered, just note the last inserted data.