Batching database schema changes during upgrades

Question

Some background

When rehearsing a production deployment to EE 1.11 from 1.10, we encountered an issue where the SQL upgrade routines were adding columns to report, log, sales, customer tables for the Enterprise_RMA module.

The log tables on this particular install had ~10M rows, had not been maintained in quite some time. Adding columns proved to take a very, very long process. The rehearsal upgrade was performed on an AWS 4XL, and still took ~4 hours to complete.

Upon analysis, and reproduction of the issue, we found that columns were being added one at a time, SHOW PROCESSLIST reveals 30-45s for copy to tmp table for each column. There are dozens of columns to add in some of these tables.

Our (workable, but not my favorite) solution:

Our workaround was to truncate log/report tables, and take the hit on the sales_flat_order tables. We could have run the archiver to move some of the data out (~600k records or so) - but custom reports that demanded the presence of the sales_flat_order table precluded our ability to do that.

Our final, actual, production deployment still hit up in the 1.5 hour range for the full database upgrade.

Finally, the question:

How can we speed up database migrations from the application layer? Can we batch or roll up all ALTER statements into a handful of executions or is this just the nature of the beast? How do you handle this for your huge EE customers? Upgrade offline, swap in place?

Any and all feedback welcomed. Thanks in advance.

Answer 1

Give this a try:

• https://github.com/fbrnc/Magento-Upgrade-Replay

Magento-Upgrade-Replay This is prototype for multi-process Optimized Upgrade Replay from Magento 1.5 (1.10 EE) and lower to 1.6 (1.11 EE) and higher. In order to run optimized replay you need to run original upgrade in database with same structure. Replay script uses pdo_mysql.log to read queries and estimate execution time.

...

Script internals Create posix FIFO for inter-process communications. Start multiple php processes using proc_open(). Prepare queries for execution: Strip out duplicate queries. Merge similar ALTER TABLE queries. Optimize one data-specific query Split in chunks for parallel execution. Child processes are waiting for data in STDIN and puting their IDs in fifo once process is finished. Parent process reads fifo in blocking mode. Once Parent gets ID of child, it sends next query or waits till othe queries from chunk will be executed. Data upgrades are triggered the same way as in original.

Further reading:

• http://www.reddit.com/r/Magento/comments/u8a8j/upgrades_on_largish_high_volume_stores_how_do_you/

Answer 2

We have upgraded large database (30-60gb) and had the upgrade process run for as long as 10 hours. I think the best approach is to first clean everything you can. I will attach our complete clean script at the end.

The next item we are using is a Magerun. This allows the ugprade to be run from the command line. A 20gb Cleaned DB takes about 6 hours to upgrade on a Dell 2950 (2x4core, 16gb mem)

I have experience slow upgrade times on VPS servers mainly due to the slow I/O time, I recommend using local hardware.

Think about tuning you my.cnf as well to make sure you have enough recourses available for the upgrade, then turn it back if needed.

Here is a complete clean script - Many of the tables are for different version of Magento and won't exists, I simply comment them out if I get a SQL error.

SET foreign_key_checks = 0;
truncate dataflow_batch_export;
truncate dataflow_batch_import;
truncate log_customer;
truncate log_quote;
truncate log_summary;
truncate log_summary_type;
truncate log_url;
truncate log_url_info;
truncate log_visitor;
truncate log_visitor_info;
truncate log_visitor_online; 
truncate adminnotification_inbox;
;reports
truncate report_viewed_product_index;
truncate report_compared_product_index;
truncate report_event;
;These may not be in the system
;truncate report_viewed_product_aggregated_monthly;
;truncate report_viewed_product_aggregated_daily;
;truncate report_viewed_product_aggregated_yearly;
;send a friend
truncate sendfriend_log;
;core logs
truncate core_cache;
;truncate core_url_rewrite;
;delete quotes
;truncate sales_flat_quote_shipping_rate;
;truncate sales_flat_quote_payment;
;truncate sales_flat_quote_item_option;
;truncate sales_flat_quote_item;
;truncate sales_flat_quote_address_item;
;truncate sales_flat_quote_address;
;truncate sales_flat_quote;
;truncate index tables
;truncate catalog_product_index_eav;
;truncate catalog_product_index_eav_decimal;
;truncate catalog_product_index_eav_decimal_idx;
;truncate catalog_product_index_eav_idx;
;truncate catalog_product_index_price;
;truncate catalog_product_index_price_bndl_opt;
;truncate catalog_product_index_price_bndl_sel;
;truncate catalog_product_index_price_bundle;
;truncate catalog_product_index_price_downloadable_idx;
;truncate catalog_product_index_price_final_idx;
;truncate catalog_product_index_price_idx;
;truncate catalog_product_index_price_idx_cfg_opt_aggregate;
;truncate catalog_product_index_price_idx_cfg_option;
;truncate catalog_product_index_price_idx_option;
;truncate catalog_product_index_price_idx_option_aggregate;
;truncate catalog_product_index_tier_price;
;truncate catalog_product_index_website;
;truncate catalogindex_aggregation;
;truncate catalogindex_aggregation_tag;
;truncate catalogindex_aggregation_to_tag;
;truncate catalogindex_eav;
;truncate catalogindex_minimal_price;
;truncate catalogindex_price;
;truncate catalog_product_index_eav_decimal_idx;
;truncate catalog_product_index_eav_decimal_tmp;
;truncate catalog_product_index_eav_idx;
;truncate catalog_product_index_eav_tmp;
;truncate catalog_product_index_price_bundle_idx;
;truncate catalog_product_index_price_bundle_opt_idx;
;truncate catalog_product_index_price_bundle_opt_tmp;
;truncate catalog_product_index_price_bundle_sel_idx;
;truncate catalog_product_index_price_bundle_sel_tmp;
;truncate catalog_product_index_price_bundle_tmp;
;truncate catalog_product_index_price_cfg_opt_agr_idx;
;truncate catalog_product_index_price_cfg_opt_agr_tmp;
;truncate catalog_product_index_price_cfg_opt_idx;
;truncate catalog_product_index_price_cfg_opt_tmp;
;truncate catalog_product_index_price_downlod_idx;
;truncate catalog_product_index_price_downlod_tmp;
;truncate catalog_product_index_price_final_idx;
;truncate catalog_product_index_price_final_tmp;
;truncate catalog_product_index_price_idx;
;truncate catalog_product_index_price_opt_agr_idx;
;truncate catalog_product_index_price_opt_agr_tmp;
;truncate catalog_product_index_price_opt_idx;
;truncate catalog_product_index_price_opt_tmp;

;truncate catalog_category_anc_categs_index_idx;
;truncate catalog_category_anc_categs_index_tmp;
;truncate catalog_category_anc_products_index_idx;
;truncate catalog_category_anc_products_index_tmp;
;truncate catalog_category_product_index_enbl_idx;
;truncate catalog_category_product_index_enbl_tmp;
;truncate catalog_category_product_index_idx;
;truncate catalog_category_product_index_tmp;

;truncate catalog_product_bundle_price_index;
;truncate catalog_product_bundle_stock_index;
;truncate catalog_product_enabled_index;

;truncate cataloginventory_stock_status_idx;
;truncate cataloginventory_stock_status_tmp;
truncate index_event;
truncate index_process;
truncate index_process_event;
;drop 
;drop table catalog_category_flat_store_1;
;drop table catalog_product_flat_1;

;M2E Items
truncate m2epro__backup_ebay_listings_logs;
truncate m2epro__backup_ebay_orders_logs;
truncate m2epro__backup_listings_logs;
truncate m2epro__backup_synchronizations_logs;
truncate m2epro_listing_log;
truncate m2epro_listing_other_log;
truncate m2epro_synchronization_log;
truncate magemonkey_api_debug;

SET foreign_key_checks = 1;

Answer 3

We've always erred on the side of caution when upgrading "big" EE stores and completed them out-of-hours in the standard fashion (ie. maintenance mode for live site, base theme/template, disable all modules, update using mage).

We also usually start the upgrade in duplicate (ie. on two separate servers/environments); simply because we have the server resource to do it - and it gives us some buffer (ie. not having to re-start the process) if there happens to be an "odd" error.

copy to tmp table can be combated by increasing the tmp_table_size accordingly and using a tmpfs mounted DIR for the MySQL tmp disk, provided it doesn't use DISTINCT

We've always toyed with the idea of queuing commands, grouping queries and commiting in 1 fell swoop - but the potential for error is just too vast.