MySQL crashed suddenly

Question

MySQL seems to crash suddenly but recovered in around 2minutes. I am running on Ubuntu 18.04 (16vcpu, 32GB RAM). I dont know what caused it.

Here is my.ini

[mysql]
default-character-set=utf8mb4

[mysqld]
lower_case_table_names = 1
pid-file        = /var/run/mysqld/mysqld.pid
socket          = /var/run/mysqld/mysqld.sock
datadir         = /data/mysql
log-error       = /var/log/mysql/error.log
# By default we only accept connections from localhost
#bind-address   = 127.0.0.1
# Disabling symbolic-links is recommended to prevent assorted security risks
symbolic-links=0
max_allowed_packet=1G
max_connect_errors=1000
max_connections=2500
character-set-server=utf8mb4
collation-server=utf8mb4_unicode_ci
innodb_buffer_pool_size=27G
server-id=2
log-bin=env1
expire_logs_days=2
sql_mode='STRICT_TRANS_TABLES,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION'
net_read_timeout=300
net_write_timeout=300
log_bin_trust_function_creators=1

Here is my /var/log/mysql/error.log

The crash is around 8:11am as seen on the image above

Im trying to find the root cause for this. Is there any logs in Ubuntu that I can check to find more info on what caused MySQL to crash?

EDIT: I saw the OOM (Out of Memory) error on the /var/log/syslog. Any idea what caused this?

Answer 1

innodb_buffer_pool_size=27G is too close to the limit especially with a large amount of (max-)connections.

Lower this down and examine utilization. show global status like 'innodb_buffer_pool_pages% should be full after a reasonable amount of warmup (24hrs+)

show global status like 'innodb_buffer_pool_read%. read_requests vs reads should be quite large. (reads% < 1%).

Once you have this balance hopefully there is sufficient ram available to avoid OOM, but still has sufficient performance.