Magento 2: Migrate changes made since last migration using non-live DB?

magento.stackexchange https://magento.stackexchange.com/questions/321516

Question

When we started development of our M2 project, we migrated all data to the new M2 project from M1. We did this using the Data Migration Tool. Because we did not want to mess with the live database, we first made a copy of this database and used this copy to migrate our data to M2.

This has worked perfectly.

Now we want to migrate all changes made to the live M1 database since we started development on M2. But now I saw that in order to migrate the delta from the old M1 database Magento added some tables to that database when we migrated it (according to https://devdocs.magento.com/guides/v2.4/migration/migration-migrate-delta.html). However, since we migrated from a copy and not the live M1 database, no changes were tracked.

Is there a way to migrate changed data from M1 to M2 without Magento having kept track of changes in the M1 database?

Was it helpful?

Solution

Kinda big, but I did it in two different imports

First deleted all orders and customers with the queries in this question https://github.com/magento/data-migration-tool/issues/467#issuecomment-627912182

Then do a map.xml, this might depend on your tables, I had to ignore extensions tables and other ones, was a trial error, backup your database and try, if something is wrong go back and check where it is on database then add that table to ignore, in that last answer on git a posted a bit but too much lines to post it here.

To import customers

<?xml version="1.0" encoding="UTF-8"?>

<steps mode="data">
    <!-- <step title="Data Integrity Step">
        <integrity>Migration\Step\DataIntegrity\Integrity</integrity>
    </step>
   <step title="EAV Step">
        <integrity>Migration\Step\Eav\Integrity</integrity>
        <data>Migration\Step\Eav\Data</data>
        <volume>Migration\Step\Eav\Volume</volume>
    </step>  -->
    <step title="Customer Attributes Step">
        <integrity>Migration\Step\Customer\Integrity</integrity>
        <data>Migration\Step\Customer\Data</data>
        <volume>Migration\Step\Customer\Volume</volume>
    </step>
    <!-- <step title="Map Step">
        <integrity>Migration\Step\Map\Integrity</integrity>
        <data>Migration\Step\Map\Data</data>
        <volume>Migration\Step\Map\Volume</volume>
    </step>
   <step title="Url Rewrite Step">
        <integrity>Migration\Step\UrlRewrite\Version191to2000</integrity>
        <data>Migration\Step\UrlRewrite\Version191to2000</data>
        <volume>Migration\Step\UrlRewrite\Version191to2000</volume>
    </step> --> 
 <!--   <step title="Log Step">
        <integrity>Migration\Step\Log\Integrity</integrity>
        <data>Migration\Step\Log\Data</data>
        <volume>Migration\Step\Log\Volume</volume>
    </step>  -->
  <!--  <step title="Ratings Step">
        <integrity>Migration\Step\Ratings\Integrity</integrity>
        <data>Migration\Step\Ratings\Data</data>
        <volume>Migration\Step\Ratings\Volume</volume>
    </step>
    <step title="ConfigurablePrices step">
        <integrity>Migration\Step\ConfigurablePrices\Integrity</integrity>
        <data>Migration\Step\ConfigurablePrices\Data</data>
        <volume>Migration\Step\ConfigurablePrices\Volume</volume>
    </step>  -->
   <!-- <step title="OrderGrids Step">
        <integrity>Migration\Step\OrderGrids\Integrity</integrity>
        <data>Migration\Step\OrderGrids\Data</data>
        <volume>Migration\Step\OrderGrids\Volume</volume>
    </step>
    <step title="Tier Price Step">
        <integrity>Migration\Step\TierPrice\Integrity</integrity>
        <data>Migration\Step\TierPrice\Data</data>
        <volume>Migration\Step\TierPrice\Volume</volume>
    </step>  
    <step title="SalesIncrement Step">
        <integrity>Migration\Step\SalesIncrement\Integrity</integrity>
        <data>Migration\Step\SalesIncrement\Data</data>
        <volume>Migration\Step\SalesIncrement\Volume</volume>
    </step>
    <step title="PostProcessing Step">
        <data>Migration\Step\PostProcessing\Data</data>
    </step>-->
</steps>
<steps mode="delta">
    <step title="Customer Attributes Step">
        <delta>Migration\Step\Customer\Delta</delta>
        <volume>Migration\Step\Customer\Volume</volume>
    </step>
</steps>

<source>
   <database host="xxxxxxx" name="database_name" user="user" password="password"/>
</source>
<destination>
   <database host="xxxxxxx" name="database_name" user="user" password="password"/>
</destination>
<options>
    <map_file>etc/opensource-to-opensource/1.9.0.1/map.xml.dist</map_file>
    <eav_map_file>etc/opensource-to-opensource/map-eav.xml.dist</eav_map_file>
    <eav_document_groups_file>etc/opensource-to-opensource/eav-document-groups.xml.dist</eav_document_groups_file>
    <eav_attribute_groups_file>etc/opensource-to-opensource/eav-attribute-groups.xml.dist</eav_attribute_groups_file>
    <log_map_file>etc/opensource-to-opensource/map-log.xml.dist</log_map_file>
    <log_document_groups_file>etc/opensource-to-opensource/log-document-groups.xml.dist</log_document_groups_file>
    <settings_map_file>etc/opensource-to-opensource/settings.xml.dist</settings_map_file>
    <customer_map_file>etc/opensource-to-opensource/map-customer.xml.dist</customer_map_file>
    <customer_document_groups_file>etc/opensource-to-opensource/customer-document-groups.xml.dist</customer_document_groups_file>
    <customer_attribute_groups_file>etc/opensource-to-opensource/customer-attribute-groups.xml.dist</customer_attribute_groups_file>
    <delta_document_groups_file>etc/opensource-to-opensource/deltalog.xml.dist</delta_document_groups_file>
    <order_grids_document_groups_file>etc/opensource-to-opensource/order-grids-document-groups.xml.dist</order_grids_document_groups_file>
    <map_document_groups>etc/opensource-to-opensource/map-document-groups.xml.dist</map_document_groups>
    <class_map>etc/opensource-to-opensource/class-map.xml.dist</class_map>
    <tier_price_map_file>etc/opensource-to-opensource/map-tier-price.xml.dist</tier_price_map_file>
    <stores_map_file>etc/opensource-to-opensource/map-stores.xml.dist</stores_map_file>
    <!--
    In case bulk_size=0 it will be auto-detected for every document.
    -->
    <bulk_size>0</bulk_size>
    <!--
    Set direct_document_copy = 1 for better performance.
    NOTE: 'source' and 'destination' databases MUST be placed on the same MySQL instance
    and 'destination' user MUST be granted with 'SELECT' permissions on 'source' database
     -->
    <direct_document_copy>0</direct_document_copy>
    <source_prefix />
    <dest_prefix>pe</dest_prefix>
    <auto_resolve_urlrewrite_duplicates>0</auto_resolve_urlrewrite_duplicates>
    <log_file>migration.log</log_file>
    <progress_bar_format>%percent%% [%bar%] Remaining Time: %remaining%</progress_bar_format>
    <upgrade_customer_password_hash>1</upgrade_customer_password_hash>
    <edition_migrate>opensource-to-opensource</edition_migrate>
    <edition_number>1.9.0.1</edition_number>
    <init_statements_source>SET NAMES utf8;</init_statements_source>
    <init_statements_destination>SET NAMES utf8;</init_statements_destination>
    <crypt_key>04a54072077cb6e32e54dee19b47cc3f</crypt_key>
</options>

After this, if you can't login with a customer account truncate customer_entity_varchar, it's caused by a change in hash I think so truncate so it's regenerated and all good.

then, orders

    <?xml version="1.0" encoding="UTF-8"?>
<!--
/**
 * Copyright © Magento, Inc. All rights reserved.
 * See COPYING.txt for license details.
 */
-->
<config xmlns:xs="http://www.w3.org/2001/XMLSchema-instance" xs:noNamespaceSchemaLocation="../../config.xsd">

    <steps mode="data">
            <!-- <step title="Data Integrity Step">
                <integrity>Migration\Step\DataIntegrity\Integrity</integrity>
            </step>
           <step title="EAV Step">
                <integrity>Migration\Step\Eav\Integrity</integrity>
                <data>Migration\Step\Eav\Data</data>
                <volume>Migration\Step\Eav\Volume</volume>
            </step>  
            <step title="Customer Attributes Step">
                <integrity>Migration\Step\Customer\Integrity</integrity>
                <data>Migration\Step\Customer\Data</data>
                <volume>Migration\Step\Customer\Volume</volume>
            </step>-->
            <step title="Map Step">
                <integrity>Migration\Step\Map\Integrity</integrity>
                <data>Migration\Step\Map\Data</data>
                <volume>Migration\Step\Map\Volume</volume>
            </step>
            <!--<step title="Url Rewrite Step">
                <integrity>Migration\Step\UrlRewrite\Version191to2000</integrity>
                <data>Migration\Step\UrlRewrite\Version191to2000</Imdata>
                <volume>Migration\Step\UrlRewrite\Version191to2000</volume>
            </step>
            <step title="Log Step">
                <integrity>Migration\Step\Log\Integrity</integrity>
                <data>Migration\Step\Log\Data</data>
                <volume>Migration\Step\Log\Volume</volume>
            </step>
            <step title="Ratings Step">
                <integrity>Migration\Step\Ratings\Integrity</integrity>
                <data>Migration\Step\Ratings\Data</data>
                <volume>Migration\Step\Ratings\Volume</volume>
            </step>
            <step title="ConfigurablePrices step">
                <integrity>Migration\Step\ConfigurablePrices\Integrity</integrity>
                <data>Migration\Step\ConfigurablePrices\Data</data>
                <volume>Migration\Step\ConfigurablePrices\Volume</volume>
            </step>  -->
            <step title="OrderGrids Step">
                <integrity>Migration\Step\OrderGrids\Integrity</integrity>
                <data>Migration\Step\OrderGrids\Data</data>
                <volume>Migration\Step\OrderGrids\Volume</volume>
            </step>
    <!--        <step title="Tier Price Step">
                <integrity>Migration\Step\TierPrice\Integrity</integrity>
                <data>Migration\Step\TierPrice\Data</data>
                <volume>Migration\Step\TierPrice\Volume</volume>
            </step>  -->
            <step title="SalesIncrement Step">
                <integrity>Migration\Step\SalesIncrement\Integrity</integrity>
                <data>Migration\Step\SalesIncrement\Data</data>
                <volume>Migration\Step\SalesIncrement\Volume</volume>
            </step>
            <!--<step title="PostProcessing Step">
                <data>Migration\Step\PostProcessing\Data</data>
            </step>-->
        </steps>
  <steps mode="delta">
        <step title="SalesIncrement Step">
            <delta>Migration\Step\SalesIncrement\Delta</delta>
            <volume>Migration\Step\SalesIncrement\Volume</volume>
        </step>
        <step title="OrderGrids Step">
            <delta>Migration\Step\OrderGrids\Delta</delta>
            <volume>Migration\Step\OrderGrids\Volume</volume>
        </step>
        
</steps>

    <source>
       <database host="xxxxxx" name="database" user="user" password="password"/>
    </source>
    <destination>
       <database host="xxxxxxx" name="database" user="user" password="password"/>
    </destination>
    <options>
        <map_file>etc/opensource-to-opensource/1.9.0.1/map.xml.dist</map_file>
        <eav_map_file>etc/opensource-to-opensource/map-eav.xml.dist</eav_map_file>
        <eav_document_groups_file>etc/opensource-to-opensource/eav-document-groups.xml.dist</eav_document_groups_file>
        <eav_attribute_groups_file>etc/opensource-to-opensource/eav-attribute-groups.xml.dist</eav_attribute_groups_file>
        <log_map_file>etc/opensource-to-opensource/map-log.xml.dist</log_map_file>
        <log_document_groups_file>etc/opensource-to-opensource/log-document-groups.xml.dist</log_document_groups_file>
        <settings_map_file>etc/opensource-to-opensource/settings.xml.dist</settings_map_file>
        <customer_map_file>etc/opensource-to-opensource/map-customer.xml.dist</customer_map_file>
        <customer_document_groups_file>etc/opensource-to-opensource/customer-document-groups.xml.dist</customer_document_groups_file>
        <customer_attribute_groups_file>etc/opensource-to-opensource/customer-attribute-groups.xml.dist</customer_attribute_groups_file>
        <delta_document_groups_file>etc/opensource-to-opensource/deltalog.xml.dist</delta_document_groups_file>
        <order_grids_document_groups_file>etc/opensource-to-opensource/order-grids-document-groups.xml.dist</order_grids_document_groups_file>
        <map_document_groups>etc/opensource-to-opensource/map-document-groups.xml.dist</map_document_groups>
        <class_map>etc/opensource-to-opensource/class-map.xml.dist</class_map>
        <tier_price_map_file>etc/opensource-to-opensource/map-tier-price.xml.dist</tier_price_map_file>
        <stores_map_file>etc/opensource-to-opensource/map-stores.xml.dist</stores_map_file>
        <!--
        In case bulk_size=0 it will be auto-detected for every document.
        -->
        <bulk_size>0</bulk_size>
        <!--
        Set direct_document_copy = 1 for better performance.
        NOTE: 'source' and 'destination' databases MUST be placed on the same MySQL instance
        and 'destination' user MUST be granted with 'SELECT' permissions on 'source' database
         -->
        <direct_document_copy>0</direct_document_copy>
        <source_prefix />
        <dest_prefix>pe</dest_prefix>
        <auto_resolve_urlrewrite_duplicates>0</auto_resolve_urlrewrite_duplicates>
        <log_file>migration.log</log_file>
        <progress_bar_format>%percent%% [%bar%] Remaining Time: %remaining%</progress_bar_format>
        <upgrade_customer_password_hash>1</upgrade_customer_password_hash>
        <edition_migrate>opensource-to-opensource</edition_migrate>
        <edition_number>1.9.0.1</edition_number>
        <init_statements_source>SET NAMES utf8;</init_statements_source>
        <init_statements_destination>SET NAMES utf8;</init_statements_destination>
        <crypt_key>04a54072077cb6e32e54dee19b47cc3f</crypt_key>
    </options>
</config>

You can try to join this, but i found it easier to do in two imports. If gallery tables give yu problems truncate them, don't worry, but remember to always have a copy of the original database, I got it wrong so many times, if you have any problem let me know I'll see if i can help!

Hope it helps :D

OTHER TIPS

After a lot of trial and error I was able to get this working.

Here are my quick notes. Ping me if more information is required:

  • Create a DB backup of the M2 db before the remigration
  • Ensure that the migration tool and your corresponding project specific extension are enabled as described in Magento's migration guide
  • Go to https://www.czettner.com/2018/09/13/magento-2-data-migration.html and run the truncation query as provided
  • From the same URL, copy the XML snippet provided and add it to your map.xml at the following path: map > destination > document_rules. I left all the other nodes that are already present at this path, and simply amended the XML snippet provided
  • Run the migration with the "--reset" flag as described in Magento's migration guide

When you encounter an issue, try to fix it and restart the whole process by resetting your DB with the backup you created.

Some issues that I encountered:

  • Document has ambiguous configuration: cms_page => Here I simply commented out everything related to "cms_page" from the map.xml
  • Missing required argument $options or similar => This seems to be an issue with EAV attributes that I already had during my initial migration. Here is what helped me identify the corresponding EAV attribute which I then deleted: “Missing required argument $options” After migration from magento 1.9.2.4 to 2.1.6
  • SQLSTATE[23000]: Integrity constraint violation: 1062 Duplicate entry '182-65' for key 'CAT_PRD_ENTT_MDA_GLR_VAL_TO_ENTT_VAL_ID_ENTT_ID' => Here I simply truncated the following tables: catalog_product_entity_media_gallery, catalog_product_entity_media_gallery_value, catalog_product_entity_media_gallery_value_to_entity, catalog_product_entity_media_gallery_value_video
Licensed under: CC-BY-SA with attribution
Not affiliated with magento.stackexchange
scroll top