•  09-10-2020
  •  | 
  •  

Question

I'm running a dedicated server with about 20 websites active. About 6 of them are Magento installations. Running the server as a webserver (web, e-mail, and MySQL ).

This is a preemptive optimization. Currently, I don't seem to have any issues, but I feel that it's good to be proactive and optimize now to avoid any potential problems later.

The server is an 8 core CPU, 8Gb RAM, running CentOS 6. The 8GB of RAM is spread around the entire server, not just the database. I'm trying to optimize my MySQL installation for the best possible performance and stability. Ran MySQLTuner (1.7.4) and getting these suggestions.

Would appreciate some help in optimizing the settings.

Thank you!

UPDATE: 11/28/2017 08:20PM EST

 - Added additional 8GB of RAM.  
 - Latest MySQLTuner results. Global Status, Variables, Innodb Status files updated.

Can view the 3 files requested here. Latest info below:

-------- Storage Engine Statistics -----------------------------------------------------------------
[--] Status: +ARCHIVE +BLACKHOLE +CSV -FEDERATED +InnoDB +MEMORY +MRG_MYISAM +MyISAM +PERFORMANCE_SCHEMA
[--] Data in MyISAM tables: 90M (Tables: 300)
[--] Data in InnoDB tables: 923M (Tables: 3591)
[--] Data in MEMORY tables: 3M (Tables: 141)
[OK] Total fragmented tables: 0

-------- Security Recommendations ------------------------------------------------------------------
[OK] There are no anonymous accounts for any database users
[OK] All database users have passwords assigned
[--] There are 612 basic passwords in the list.

-------- CVE Security Recommendations --------------------------------------------------------------
[OK] NO SECURITY CVE FOUND FOR YOUR VERSION

-------- Performance Metrics -----------------------------------------------------------------------
[--] Up for: 13d 17h 16m 32s (29M q [25.279 qps], 860K conn, TX: 43G, RX: 9G)
[--] Reads / Writes: 68% / 32%
[--] Binary logging is disabled
[--] Physical Memory     : 7.7G
[--] Max MySQL memory    : 5.1G
[--] Other process memory: 964.7M
[--] Total buffers: 4.4G global + 1.1M per thread (100 max threads)
[--] P_S Max memory usage: 555M
[--] Galera GCache Max memory usage: 0B
[OK] Maximum reached memory usage: 5.0G (65.37% of installed RAM)
[OK] Maximum possible memory usage: 5.1G (66.51% of installed RAM)
[OK] Overall possible memory usage with other process is compatible with memory available
[OK] Slow queries: 0% (35/29M)
[OK] Highest usage of available connections: 21% (21/100)
[OK] Aborted connections: 0.01%  (114/860297)
[!!] name resolution is active : a reverse name resolution is made for each new connection and can reduce performance
[!!] Query cache may be disabled by default due to mutex contention.
[OK] Query cache efficiency: 90.8% (22M cached / 25M selects)
[!!] Query cache prunes per day: 43104
[OK] Sorts requiring temporary tables: 0% (38 temp sorts / 360K sorts)
[!!] Joins performed without indexes: 22932
[OK] Temporary tables created on disk: 23% (201K on disk / 844K total)
[OK] Thread cache hit rate: 99% (21 created / 860K connections)
[OK] Table cache hit rate: 47% (7K open / 16K opened)
[OK] Open file limit used: 4% (867/20K)
[OK] Table locks acquired immediately: 99% (5M immediate / 5M locks)

-------- Performance schema ------------------------------------------------------------------------
[--] Memory used by P_S: 555.4M
[--] Sys schema isn't installed.

-------- ThreadPool Metrics ------------------------------------------------------------------------
[--] ThreadPool stat is enabled.
[--] Thread Pool Size: 8 thread(s).
[!!] thread_pool_size between 16 and 36 when using InnoDB storage engine.

-------- MyISAM Metrics ----------------------------------------------------------------------------
[!!] Key buffer used: 19.5% (52M used / 268M cache)
[OK] Key buffer size / total MyISAM indexes: 256.0M/22.5M
[OK] Read Key buffer hit rate: 99.9% (3M cached / 3K reads)
[!!] Write Key buffer hit rate: 42.3% (124K cached / 52K writes)

-------- InnoDB Metrics ----------------------------------------------------------------------------
[--] InnoDB is enabled.
[--] InnoDB Thread Concurrency: 4
[OK] InnoDB File per table is activated
[OK] InnoDB buffer pool / data size: 4.0G/923.8M
[!!] Ratio InnoDB log file size / InnoDB Buffer pool size (50 %): 1.0G * 2/4.0G should be equal 25%
[!!] InnoDB buffer pool instances: 8
[--] InnoDB Buffer Pool Chunk Size not used or defined in your version
[OK] InnoDB Read buffer efficiency: 99.99% (383379684 hits/ 383413960 total)
[!!] InnoDB Write Log efficiency: 72.46% (2434433 hits/ 3359563 total)
[OK] InnoDB log waits: 0.00% (0 waits / 925130 writes)

-------- AriaDB Metrics ----------------------------------------------------------------------------
[--] AriaDB is disabled.

-------- TokuDB Metrics ----------------------------------------------------------------------------
[--] TokuDB is disabled.

-------- XtraDB Metrics ----------------------------------------------------------------------------
[--] XtraDB is disabled.

-------- RocksDB Metrics ---------------------------------------------------------------------------
[--] RocksDB is disabled.

-------- Spider Metrics ----------------------------------------------------------------------------
[--] Spider is disabled.

-------- Connect Metrics ---------------------------------------------------------------------------
[--] Connect is disabled.

-------- Galera Metrics ----------------------------------------------------------------------------
[--] Galera is disabled.

-------- Replication Metrics -----------------------------------------------------------------------
[--] Galera Synchronous replication: NO
[--] No replication slave(s) for this server.
[--] This is a standalone server.

-------- Recommendations ---------------------------------------------------------------------------
General recommendations:
    Control warning line(s) into /var/lib/mysql/ file
    Control error line(s) into /var/lib/mysql/ file
    Configure your accounts with ip or subnets only, then update your configuration with skip-name-resolve=1
    Adjust your join queries to always utilize indexes
    Consider installing Sys schema from https://github.com/mysql/mysql-sys
    Thread pool size for InnoDB usage (8)
    Read this before changing innodb_log_file_size and/or innodb_log_files_in_group:
Variables to adjust:
    query_cache_size (=0)
    query_cache_type (=0)
    query_cache_size (> 100M)
    join_buffer_size (> 256.0K, or always use indexes with joins)
    thread_pool_size between 16 and 36 for InnoDB usage
    innodb_log_file_size should be (=512M) if possible, so InnoDB total log files size equals to 25% of buffer pool size.
    innodb_buffer_pool_instances(=4)

Below are the settings currently in my config file.

[mysqld]                                                                                                                            

datadir=/var/lib/mysql
socket=/chroot/tmp/mysql.sock

userstat=1


ft_min_word_len=3

max_connections=100 # MAX CURRENT X 5
max_connect_errors=25
connect_timeout=10
interactive_timeout=20
wait_timeout=50
delayed_insert_timeout=10
#join_buffer_size=1M

max_allowed_packet=16M

myisam_sort_buffer_size=1M
#sort_buffer_size=1M

#read_buffer_size=1M
#read_rnd_buffer_size=2M

thread_cache_size=100  # from 192 MySQL v 8 recommends CAP of 100 to avoid overload
thread_concurrency=4

query_cache_size=100M # from 256M
#query_cache_limit=32M #remove to allow default of 1M rather than 32M
query_cache_type=1
query_cache_min_res_unit = 512  # from 4K to conserve qcache RAM used per RESULT

tmp_table_size=80M # from 512M
max_heap_table_size=80M # from 512M
max_tmp_tables=10
slow_query_log=1
long_query_time=10
slow-query-log-file  = /var/lib/mysql/mysql-slow.log
table_open_cache = 10000  # 10000 from 2000 to support the 28,000 opened by instance

innodb-file-per-table=1

character-set-server=utf8
collation-server=utf8_general_ci

# Tweaking below
# 256M
innodb_buffer_pool_size = 4G  # to match RAM reported by MySQLTuner
# 512 MB
key_buffer_size = 256M  # from 512M used only my MyISAM
# 16MB
bulk_insert_buffer_size = 16M
innodb_thread_concurrency = 4
innodb_autoinc_lock_mode = 0
skip-external-locking
# Double write off :)
innodb_doublewrite = true
low_priority_updates = 1
#innodb_checksums = false
innodb_support_xa = false
max_write_lock_count = 10
innodb_flush_log_at_trx_commit = 2
innodb_max_dirty_pages_pct = 40
innodb_io_capacity = 400
innodb_write_io_threads = 4
innodb_read_io_threads = 4
innodb_adaptive_flushing = 1
innodb_flush_method = O_DIRECT
innodb_log_file_size=1G
innodb_print_all_deadlocks = 1 # to error log, if you ever have one, you will WISH you had the details. 
innodb_buffer_pool_dump_at_shutdown=1 # from OFF to allow quick warmup 
innodb_buffer_pool_load_at_startup=1 #from OFF to allow warmed buffer_pool
#skip-name-resolve=1
#log_error_verbosity=3 
log_warnings = 2 # to record Aborted Connections in error.log
Was it helpful?

Solution

Your my.cnf includes 4 lines that should be removed, they are

sort_buffer_size read_buffer_size read_rnd_buffer_size join_buffer_size

these are all per connection RAM requirements and are driving your RAM footprint much higher than necessary. Let the DEFAULTS work for you to improve response time with room to breathe in RAM. Consider adjusting the following in your my.cnf, please

key_buffer_size = 512M  # from 1G because 200M is used now
innodb_buffer_pool_size = 4G  # to match RAM reported by MySQLTuner
max_connections = 100  # from 500, until more than 8 are used
thread_cache_size = 100  # from 192 MySQL v 8 recommends CAP of 100 to avoid overload
query_cache_limit     remove to allow default of 1M rather than 32M
query_cache_min_res_unit = 512  # from 4K to conserve qcache RAM used per RESULT
table_open_cache = 10000  # from 2000 to support the 28,000 opened by instance

After the above items have been configured, shutdown/restart have been completed and you have 7 (or more) consecutive days of uptime, please refresh your my.cnf and MySQLTuner report, THEN for another detailed analysis, add/refresh to OriginalPost the following, SHOW GLOBAL STATUS; SHOW GLOBAL VARIABLES; SHOW ENGINE INNODB STATUS; for up to five specific additional cfg recommendations, one per day to be applied, monitor.

OTHER TIPS

query_cache_size=256M  -->  50M -- A large size can actually slow down due to purging
query_cache_type=1  -- Do you actually find it useful?  Provide SHOW GLOBAL STATUS LIKE "Qc%";

tmp_table_size=512M  --> No mo than 1% of RAM; else could lead to swapping
max_heap_table_size=512M  --> Ditto

slow_query_log=1
long_query_time=1  -- Good.  But only 18 slow queries?

thread_cache_size = 100  -- Wasting RAM since you haven't seen more than 9.

table_open_cache = 10000  -- OK if you need it.  But do you really have thousands of tables?

innodb_buffer_pool_size = 4G  -- Good (for 8G of RAM)
key_buffer_size = 512M  -- Too high if not using MyISAM

innodb_flush_log_at_trx_comm -- mispelled

innodb_max_dirty_pages_pct = 40  -- this low a setting probably leads to more I/O

[!!] Joins performed without indexes: 1453  -- Let's see some
[!!] Temporary tables created on disk: 29% (29K on disk / 98K total)  -- Let's see some

Notes:

  • You cannot "tune your way out of performance problems". Some of my suggestions above will help some.

  • Slow queries must be investigated.

  • Move from MyISAM to InnoDB wherever possible.

  • The Query Cache, in most production systems should be turned off.

  • Is Magento on the same server? How much RAM is it using? All the advice given here assumes you have 8GB of RAM available to MySQL. If you have less, then several settings need to be lowered.

  • For more tuning analysis, see here. That blog includes a couple of suggestions around the space limitation in this forum. It also suggests what to do with the Slowlog.

  • You mentioned "stability", yet you are using MyISAM, and have turned off the double-write buffer and set innodb_flush_log_at_trx_commit a setting for performance, not data integrity.

Deeper dive

A few more things showed up:

Observations:

Version: 5.6.36-82.1-log 
7.7 GB of RAM 
Uptime = 9d 23:50:34 
You are not running on Windows. 
Running 64-bit version 
It appears that you are running both MyISAM and InnoDB.

The More Important Issues

tmp_table_size and max_heap_table_size are dangerously high at 512M. If a number of complex queries were to be run simultaneously, you could run out of RAM, leading to a significant slowdown. Recommend no more than 1% (80M) of RAM for each. (max_tmp_tables is unused.)

query_cache_size leads to degraded performance when it is too big. Lower to 50M.

There are a lot of table scans; set long_query_time to 1 and turn on the SlowLog. Later, discover the worst queries and work on them.

Turning off the double-write buffer is a small invitation to corruption of the database.

It seems that neither InnoDB, nor MyISAM, tables are so big as to need increasing their buffers. In fact they could be lowered if RAM is needed elsewhere.

For many reasons, MyISAM tables should be converted to InnoDB. Conversion from MyISAM to InnoDB

table_open_cache may as well be lowered to 3000.

innodb_log_file_size is much higher than necessary, but is probably not worth changing.

"Savepoints" are rarely used in MySQL; how are they working for you?

Details and other observations

( (key_buffer_size - 1.2 * Key_blocks_used * 1024) / _ram ) = (512M - 1.2 * 5839 * 1024) / 8267812044.8 = 6.4% -- Percent of RAM wasted in key_buffer. -- Decrease key_buffer_size.

( innodb_buffer_pool_size / _ram ) = 4096M / 8267812044.8 = 51.9% -- % of RAM used for InnoDB buffer_pool

( table_open_cache ) = 10,000 -- Number of table descriptors to cache -- Several hundred is usually good.

( Innodb_buffer_pool_pages_free * 16384 / innodb_buffer_pool_size ) = 216,083 * 16384 / 4096M = 82.4% -- buffer pool free -- buffer_pool_size is bigger than working set; could decrease it

( innodb_max_dirty_pages_pct ) = 40 -- When buffer_pool starts flushing to disk -- Are you experimenting?

( Innodb_buffer_pool_pages_free / Innodb_buffer_pool_pages_total ) = 216,083 / 262136 = 82.4% -- Pct of buffer_pool currently not in use -- innodb_buffer_pool_size to bigger than necessary?

( Innodb_buffer_pool_bytes_data / innodb_buffer_pool_size ) = 735,395,840 / 4096M = 17.1% -- Percent of buffer pool taken up by data -- A small percent may indicate that the buffer_pool is unnecessarily big.

( Innodb_dblwr_pages_written/Innodb_pages_written ) = 0/1260380 = 0 -- Seems like these values should be equal?

( innodb_doublewrite ) = OFF -- Extra I/O, but extra safety in crash. -- OFF is OK for FusionIO, Galera, Slaves

( Innodb_os_log_written / (Uptime / 3600) / innodb_log_files_in_group / innodb_log_file_size ) = 1,246,395,904 / (863434 / 3600) / 2 / 1024M = 0.00242 -- Ratio (see minutes)

( Uptime / 60 * innodb_log_file_size / Innodb_os_log_written ) = 863,434 / 60 * 1024M / 1246395904 = 12,397 -- Minutes between InnoDB log rotations Beginning with 5.6.8, this can be changed dynamically; be sure to also change my.cnf. -- (The recommendation of 60 minutes between rotations is somewhat arbitrary.) Adjust innodb_log_file_size. (Cannot change in AWS.)

( Innodb_rows_deleted / Innodb_rows_inserted ) = 1,249,027 / 1276124 = 0.979 -- Churn -- "Don't queue it, just do it." (If MySQL is being used as a queue.)

( expand_fast_index_creation ) = OFF -- ALTER and OPTIMIZE may be greatly sped up by using ON. -- Probably better to be ON.

( innodb_thread_concurrency ) = 4 -- 0 = Let InnoDB decide the best for concurrency_tickets. -- Set to 0

( innodb_print_all_deadlocks ) = OFF -- Whether to log all Deadlocks. -- If you are plagued with Deadlocks, turn this on. Caution: If you have lots of deadlocks, this may write a lot to disk.

( min( tmp_table_size, max_heap_table_size ) / _ram ) = min( 512M, 512M ) / 8267812044.8 = 6.5% -- Percent of RAM to allocate when needing MEMORY table (per table), or temp table inside a SELECT (per temp table per some SELECTs). Too high may lead to swapping. -- Decrease tmp_table_size and max_heap_table_size to, say, 1% of ram.

( innodb_buffer_pool_populate ) = OFF = 0 -- NUMA control

( local_infile ) = ON -- local_infile = ON is a potential security issue

( Key_blocks_used * 1024 / key_buffer_size ) = 5,839 * 1024 / 512M = 1.1% -- Percent of key_buffer used . High-water-mark. -- Lower key_buffer_size to avoid unnecessary memory usage.

( Key_writes / Key_write_requests ) = 131,634 / 155998 = 84.4% -- key_buffer effectiveness for writes -- If you have enough RAM, it would be worthwhile to increase key_buffer_size.

( query_cache_size ) = 256M -- Size of QC -- Too small = not of much use. Too large = too much overhead. Recommend either 0 or no more than 50M.

( (query_cache_size - Qcache_free_memory) / Qcache_queries_in_cache / query_alloc_block_size ) = (256M - 63512256) / 115486 / 8192 = 0.217 -- query_alloc_block_size vs formula -- Adjust query_alloc_block_size

( Created_tmp_disk_tables / (Created_tmp_disk_tables + Created_tmp_tables) ) = 148,564 / (148564 + 549507) = 21.3% -- Percent of temp tables that spilled to disk -- maybe increase tmp_table_size and max_heap_table_size; avoid blobs, etc.

( tmp_table_size ) = 512M -- Limit on size of MEMORY temp tables used to support a SELECT -- Decrease tmp_table_size to avoid running out of RAM. Perhaps no more than 64M.

( (Com_insert + Com_update + Com_delete + Com_replace) / Com_commit ) = (257684 + 342113 + 118532 + 235) / 389763 = 1.84 -- Statements per Commit (assuming all InnoDB) -- Low: Might help to group queries together in transactions; High: long transactions strain various things.

( Select_scan / Com_select ) = 527,912 / 1359892 = 38.8% -- % of selects doing full table scan. (May be fooled by Stored Routines.) -- Add indexes / optimize queries

( binlog_error_action ) = IGNORE_ERROR -- What to do if the binlog cannot be written. -- IGNORE_ERROR is the default for backwards compatibility, but ABORT_SERVER is recommended.

( binlog_format ) = STATEMENT -- STATEMENT/ROW/MIXED. ROW is preferred; it may become the default.

( expire_logs_days ) = 0 -- How soon to automatically purge binlog (after this many days) -- Too large (or zero) = consumes disk space; too small = need to respond quickly to network/machine crash. (Not relevant if log_bin = OFF)

( innodb_autoinc_lock_mode ) = 0 -- Galera: desires 2 -- 2 = "interleaved"; 1 = "consecutive" is typical; 0 = "traditional".

( back_log / max_connections ) = 70 / 100 = 70.0%

( thread_cache_size ) = 100 -- How many extra processes to keep around (Not relevant when using thread pooling) (Autosized as of 5.6.8; based on max_connections) -- 0 is good for Windows 0 is inefficient for non-Windows; 10 is probably fine

( thread_cache_size / max_connections ) = 100 / 100 = 100.0% -- (0 for Windows)

(The following "abnormal" values are not necessarily "bad", merely uncommon.)

Abnormally small:

Innodb_dblwr_pages_written = 0
Threads_connected = 0.0042 /HR
delayed_insert_timeout = 10
interactive_timeout = 20
myisam_sort_buffer_size = 1MB
query_cache_min_res_unit = 512
thread_concurrency = 4
wait_timeout = 50

Abnormally large:

Com_drop_user = 0.0042 /HR
Com_empty_query = 15 /HR
Com_release_savepoint = 0.19 /HR
Com_rollback_to_savepoint = 15 /HR
Com_savepoint = 0.19 /HR
Com_show_create_func = 0.0083 /HR
Com_show_processlist = 0.17 /sec
Com_show_status = 0.17 /sec
Com_stmt_send_long_data = 1.9 /HR
Handler_savepoint = 0.19 /HR
Handler_savepoint_rollback = 15 /HR
Open_table_definitions = 4,172
Open_tables = 7,852
Qcache_free_blocks = 49,773
Qcache_total_blocks = 287,236
max_heap_table_size = 512MB

Abnormal strings:

ft_min_word_len = 3
innodb_force_load_corrupted = OFF
innodb_support_xa = OFF
low_priority_updates = ON
optimizer_trace = enabled=off,one_line=off
optimizer_trace_features = greedy_search=on, range_optimizer=on, dynamic_range=on, repeated_subselect=on
secure_file_priv = /var/lib/mysql-files/
sha256_password_private_key_path = private_key.pem
sha256_password_public_key_path = public_key.pem
slave_rows_search_algorithms = TABLE_SCAN,INDEX_SCAN
sql_slave_skip_counter = 0
userstat = ON

my.cnf things to do in [mysqld] section:

thread_concurrency=30 # from 4 to support more concurrent processing
innodb_io_capacity_max=6000 # from 2000 (default)
innodb_io_capacity=1200 # from 400 per second
innodb_thread_concurrency=15 # from 4 
innodb_write_io_threads=8 # from 4  
key_buffer_size=16M # from 256M less than 5M used in 13 days
expire_logs_days=5 # from 0 to allow research more days
innodb_purge_threads=2 # from 1 to reduce DEL delays
innodb_read_ahead_threshold=8 # from 56 to enable read ahead quicker
innodb_stats_sample_pages=32 # from 8 for more accurate cardinality data
low_priority_updates=OFF # from ON to minimize exposure to power fails
max_join_size=1000000000 # from huge number to 1 Billion row limit
max_seeks_for_key=32 # from huge number to reasonable limit before scan
sql_select_limit=1000000000 # from huge number to 1 Billion row limit
query_prealloc_size=32768 # from 8192 to avoid RAM allocation all day for large queries
lock_wait_timeout=300 # from 31536000 secs (1 year) until replication needed

One minute of your general log from a busy time of day would allow analysis to suggest code changes to get more than 1 of each 21 queries to be stored in Query Cache.

5 pages of your slow query log would reveal many details leading to possible index creation and/or code changes to reduce processing time. MySQLTuner is whining about joins without indexes. And they do take much longer than when properly indexed on both sides of the = variables.

Posting from you Linux system ulimit -a would reveal all limits to help with getting more tables, table definitions and open file capacity available.

Continued success with your installation. Please consider upvote and/or acceptance for ideas that help your team.

This script will allow 10 second elapsed time reporting of global status values USED with calculated RPS and Rate per Minute for each item.

# filename globalstatusage10sec.sql   Last Update 201801310853 whauck
# 2015-01-23 from schlomi's web site
# http://code.openark.org/blog/mysql/mysql-global-status-difference-using-single-query
# shortened AS labels for columns to be effective
SELECT STRAIGHT_JOIN
   LOWER(gs0.VARIABLE_NAME) AS variable_name,
   gs0.VARIABLE_VALUE AS "Begin",
   gs1.VARIABLE_VALUE AS "End",
   (gs1.VARIABLE_VALUE - gs0.VARIABLE_VALUE) AS diff,
   (gs1.VARIABLE_VALUE - gs0.VARIABLE_VALUE) / 10 AS psec,
   (gs1.VARIABLE_VALUE - gs0.VARIABLE_VALUE) * 60 / 10 AS
pminute
FROM
   (
     SELECT
       VARIABLE_NAME,
       VARIABLE_VALUE
     FROM
       INFORMATION_SCHEMA.GLOBAL_STATUS
     UNION ALL
     SELECT
       '',
       SLEEP(10)
     FROM DUAL
   ) AS gs0
   JOIN (
     SELECT 
       VARIABLE_NAME,
       VARIABLE_VALUE
     FROM 
       INFORMATION_SCHEMA.GLOBAL_STATUS
   ) gs1 USING (VARIABLE_NAME)
WHERE
   gs1.VARIABLE_VALUE != gs0.VARIABLE_VALUE LIMIT 0,800
;
Licensed under: CC-BY-SA with attribution
Not affiliated with dba.stackexchange
scroll top