Help me tune my.conf MySQL [closed]
https://dba.stackexchange.com/questions/278513
-
09-03-2021 - |
italiano
english
français
española
中国
日本の
العربية
Deutsch
한국어
Português
Russian题
Currently, I have been surfing around for ideas and copying some of the processes and details found and playing around with my Server and now seems to be pretty good but with some anomalies in performance (not always)... if you see something wrong with my current configuration which could tend to decrease the performance of the server instead of improving it please let me know and thanks for reading and helping
I'm attaching my conf file used in
OS
Virtualization: vmware Operating System: CentOS Linux 7 (Core) CPE OS Name: cpe:/o:centos:centos:7 Kernel: Linux 3.10.0-1127.19.1.el7.x86_64 Architecture: x86-64
Resources
Ram 64GB ECC DDR3 - CPU 2 Cores 8 Logic Processors per core = 16 - Disk Nvme m.2 500GB Samsung 980Pro
MySQL 8.0.22
[mysqld_safe]
log_error = /var/log/mysql_error.log
[mysqld]
# datadir = {{ mysqldir }}
datadir = /var/lib/mysql
socket = /var/lib/mysql/mysql.sock
pid-file = /var/run/mysqld/mysqld.pid
# Datadog
general_log = ON
general_log_file = /var/log/mysqld.log
log_error = /var/log/mysql_error.log
slow_query_log = ON
slow_query_log_file = /var/log/mysql_slow.log
long_query_time = 1
log_output = File
log_queries_not_using_indexes = ON
local-infile = ON
character_set_server = utf8mb4
collation_server = utf8mb4_general_ci
default_authentication_plugin = mysql_native_password
skip-character-set-client-handshake
# current configuration
skip-name-resolve = 1
sql-mode = ALLOW_INVALID_DATES
ssl-ca = ca.pem
ssl-cert = server-cert.pem
ssl-key = server-key.pem
require_secure_transport = ON
# general
table_definition_cache = 400000
back_log = 3500
# tune
max_seeks_for_key = 32
max_connections = 100
connect_timeout = 5
wait_timeout = 600
net_read_timeout = 60
max_allowed_packet = 160M
thread_cache_size = 128
bulk_insert_buffer_size = 16M
tmp_table_size = 16M
max_heap_table_size = 16M
sort_buffer_size = 4M
read_buffer_size = 2M
join_buffer_size = 512k
read_rnd_buffer_size = 512K
key_buffer_size = 6M
myisam_recover_options = BACKUP
myisam_sort_buffer_size = 16M
table_open_cache = 400000
table_open_cache_instances = 8
concurrent_insert = 2
max_binlog_size = 100M
thread_stack = 4M
max_prepared_stmt_count = 1000000
performance_schema = ON
binlog_expire_logs_seconds = 864000
innodb_log_files_in_group = 2
innodb_undo_log_truncate = OFF
innodb_doublewrite = 1
innodb_flush_log_at_trx_commit = 0
innodb_max_dirty_pages_pct = 90
innodb_max_dirty_pages_pct_lwm = 10
innodb_page_cleaners = 4
innodb_adaptive_max_sleep_delay = 10000
innodb_lru_scan_depth = 128
innodb_io_capacity = 40000
innodb_io_capacity_max = 80000
innodb_purge_threads = 4
innodb_flushing_avg_loops = 4
innodb_thread_concurrency = 0
innodb_buffer_pool_instances = 32
innodb_log_file_size = 4G
innodb_log_buffer_size = 3G
innodb_buffer_pool_size = 30G
innodb_change_buffer_max_size = 50
innodb_buffer_pool_chunk_size = 512M
innodb_file_per_table = 1
innodb_open_files = 600
innodb_flush_method = O_DIRECT_NO_FSYNC
innodb_flush_neighbors = 0
innodb_write_io_threads = 64
innodb_read_io_threads = 64
innodb_use_native_aio = 1
innodb_adaptive_hash_index = 0
innodb_stats_persistent = 1
innodb_adaptive_flushing = 1
innodb_monitor_enable = '%'
Update
I have been evaluating my processes and settings step by step with the log files and a lot of reading, so far this set up is working very well and way Faster than the one before... Now the Inserts, Delete and Read goes from Hours to even 1 minute with no CPU Spikes or RAM Looking Forward what you guys think I could adjust - Thanks
[mysqld_safe]
log_error = /var/log/mysql_error.log
[mysqld]
# datadir = {{ mysqldir }}
datadir = /var/lib/mysql
socket = /var/lib/mysql/mysql.sock
pid-file = /var/run/mysqld/mysqld.pid
# Datadog
#general_log = ON
#general_log_file = /var/log/mysqld.log
log_error = /var/log/mysql_error.log
slow_query_log = ON
slow_query_log_file = /var/log/mysql_slow.log
long_query_time = 1
log_output = File
log_queries_not_using_indexes = ON
local-infile = ON
character_set_server = utf8mb4
collation_server = utf8mb4_general_ci
default_authentication_plugin = mysql_native_password
skip-character-set-client-handshake
# current configuration
skip-name-resolve = 1
sql-mode = ALLOW_INVALID_DATES
ssl-ca = ca.pem
ssl-cert = server-cert.pem
ssl-key = server-key.pem
require_secure_transport = ON
# general
#table_definition_cache = 400000
#back_log = 3500
# tune
#max_seeks_for_key = 32
max_connections = 100
#connect_timeout = 5
#wait_timeout = 600
#net_read_timeout = 60
#max_allowed_packet = 160M
#thread_cache_size = 128
bulk_insert_buffer_size = 256M
tmpdir = /dev/shm
tmp_table_size = 256M
max_heap_table_size = 256M
sort_buffer_size = 50M
read_buffer_size = 4M
join_buffer_size = 3M
read_rnd_buffer_size = 8M
innodb_sort_buffer_size = 50M
key_buffer_size = 50M
#myisam_recover_options = BACKUP
#myisam_sort_buffer_size = 16M
table_open_cache = 40000
#table_open_cache_instances = 8
#concurrent_insert = 2
#max_binlog_size = 100M
#thread_stack = 4M
#max_prepared_stmt_count = 1000000
performance_schema = 1
binlog_expire_logs_seconds = 864000
innodb_log_files_in_group = 2
#innodb_undo_log_truncate = OFF
#innodb_doublewrite = 0
#innodb_flush_log_at_trx_commit = 0
#innodb_max_dirty_pages_pct = 90
#innodb_max_dirty_pages_pct_lwm = 10
#innodb_page_cleaners = 4
#innodb_adaptive_max_sleep_delay = 10000
innodb_lru_scan_depth = 5196
innodb_io_capacity = 1000
#innodb_io_capacity_max = 2000
#innodb_purge_threads = 4
#innodb_flushing_avg_loops = 4
innodb_thread_concurrency = 0
innodb_buffer_pool_instances = 2
innodb_log_file_size = 256M
innodb_log_buffer_size = 32M
innodb_buffer_pool_size = 2G
#innodb_change_buffer_max_size = 50
#innodb_buffer_pool_chunk_size = 512M
#innodb_file_per_table = 1
#innodb_open_files = 600
innodb_flush_method = O_DIRECT_NO_FSYNC
#innodb_flush_neighbors = 0
innodb_write_io_threads = 16
innodb_read_io_threads = 16
#innodb_use_native_aio = 1
innodb_adaptive_hash_index = 1
#innodb_stats_persistent = 1
#innodb_adaptive_flushing = 1
innodb_monitor_enable = '%'
#innodb_dedicated_server = ON
解决方案
I have a little good news for you
ASPECT #1
You could tune certain aspect of MySQL 8.0 with just one option: innodb_dedicated_server.
The innodb_dedicated_server options will auto-tune four(4) things
- Less than 1G: 128M (default value if innodb_dedicated_server is OFF)
- Less Than or = 4G: Detected Physical RAM * 0.5
- Greater than 4G: Detected Physical RAM * 0.75
- Less than 1G: 48M (default value if innodb_dedicated_server is OFF)
- Less then 4G: 128M
- Less than<=8G: 512M
- <=16G: 1024M
- Greater than 16G: 2G
The variable also sets the following:
- Set to O_DIRECT_NO_FSYNC if the setting is available on the system.
- If not, set it to the default InnoDB flush method
innodb_log_file_in_group (as of MySQL 8.0.14) : The number of log files in the log group. InnoDB writes to the files in a circular fashion. The default (and recommended) value is 2. The location of the files is specified by innodb_log_group_home_dir. The combined size of log files (innodb_log_file_size * innodb_log_files_in_group) can be up to 512GB.
ASPECT #2
You can lower innodb_read_io_threads and innodb_write_io_threads to 16 for now.
Periodically, you should run SHOW ENGINE INNODB STATUS\G
Check for the expression Pending normal aio and you will see something like
Pending normal aio reads: 0 [0, 0, 0, 0] , aio writes: 0 [0, 0, 0, 0] ,
If any of the zeros become very high (like 16-32) frequently, then increase one or both of them.
ASPECT #3
Your innodb_log_buffer_size needs to be like 32M not 3G.
ASPECT #4
Your innodb_adaptive_hash_index should be enabled
ASPECT #5
All other tuning aspect would be the same as you would for MySQL 5.7. You can refer to this Percona blog for those aspects => MySQL 5.7 Performance Tuning After Installation. You can also refer to other posts in the DBA StackExchange. You should definitely read all the options for innodb from the MySQL Documentation.
HAVE FUN RESEARCHING !!!
其他提示
You are vastly over-tuning. By default you shouldn't be touching anything other than innodb_buffer_pool_size unless your workload specifically requires a deviation from defaults, which are sensible for the majority of workloads.
Changing sort_buffer_size and join_buffer_size away from defaults is particularly counter-productive in virtually all cases. innodb_open_files should be left alone as it defaults to table_open_cache - which you have set absurdly high.
Your innodb_io_capacity is also almost certainly set to more than your SSDs can realistically deliver on a database workload with 16KB blocks, especially since you also have general_log enabled. Instead, you would be better off disabling general_log and only leaving slow_log enabled with long_query_time=0 set.
