MySQL router on server with tomcat memory leak

Question

Memory is slowly used by mysql router. I am running mysql router version 8.0.20 (I found there was a memory leak on mysql router 2.1.2 but mysql router 8.x is newer)

I have a 3 node single-primary cluster on MySQL 5.7.30 with 13 application servers pointing to it from localhost and 1 server dedicated to mysql-router to developers.

Each application server has its own mysql router locally configured as well as a tomcat server. There are about 14 mysql routers pointed at this cluster. All mysql routers are configured the same.

mysqlrouter.conf

[DEFAULT]
user=mysqlrouter
logging_folder=/var/log/mysqlrouter
runtime_folder=/etc/mysqlrouter/run
data_folder=/etc/mysqlrouter/data
keyring_path=/etc/mysqlrouter/data/keyring
master_key_path=/etc/mysqlrouter/mysqlrouter.key
connect_timeout=15
read_timeout=30
dynamic_state=/etc/mysqlrouter/data/state.json

[logger]
level = INFO

[metadata_cache:prod1]
cluster_type=gr
router_id=5
user=mysql_router5_qgi8an1mm3s9
metadata_cluster=prod1
ttl=0.5
use_gr_notifications=0

[routing:prod1_rw]
bind_address=0.0.0.0
bind_port=6446
destinations=metadata-cache://prod1/?role=PRIMARY
routing_strategy=first-available
protocol=classic

[routing:prod1_ro]
bind_address=0.0.0.0
bind_port=6447
destinations=metadata-cache://prod1/?role=SECONDARY
routing_strategy=round-robin-with-fallback
protocol=classic

[routing:prod1_x_rw]
bind_address=0.0.0.0
bind_port=64460
destinations=metadata-cache://prod1/?role=PRIMARY
routing_strategy=first-available
protocol=x

[routing:prod1_x_ro]
bind_address=0.0.0.0
bind_port=64470
destinations=metadata-cache://prod1/?role=SECONDARY
routing_strategy=round-robin-with-fallback
protocol=x

The memory allocated to mysql router on these 13 production servers continues to grow until it crashes the server. Currently one is using 3gb of ram. If I restart the server it frees up the memory. It seems to randomly go through these 13 and start flaring up. The one dedicated to developers, only running mysql router has not had an issue.

On average the mysql router grows in memory usage by 1gb a week. About every three weeks I need to restart mysql router on these servers so that the tomcat server does not crash.

Here is the mysqlconf from the developer machine

#File automatically generated during MySQL Router bootstrap
[DEFAULT]
user=mysqlrouter
logging_folder=/var/log/mysqlrouter
runtime_folder=/etc/mysqlrouter/run
data_folder=/etc/mysqlrouter/data
keyring_path=/etc/mysqlrouter/data/keyring
master_key_path=/etc/mysqlrouter/mysqlrouter.key
connect_timeout=15
read_timeout=30
dynamic_state=/etc/mysqlrouter/data/state.json

[logger]
level = INFO

[metadata_cache:prod1]
cluster_type=gr
router_id=1
user=mysql_router2_6tobldcfht67
metadata_cluster=prod1
ttl=0.5
use_gr_notifications=0

[routing:prod1_rw]
bind_address=0.0.0.0
bind_port=6446
destinations=metadata-cache://prod1/?role=PRIMARY
routing_strategy=first-available
protocol=classic

[routing:prod1_ro]
bind_address=0.0.0.0
bind_port=6447
destinations=metadata-cache://prod1/?role=SECONDARY
routing_strategy=round-robin-with-fallback
protocol=classic

[routing:prod1_x_rw]
bind_address=0.0.0.0
bind_port=64460
destinations=metadata-cache://prod1/?role=PRIMARY
routing_strategy=first-available
protocol=x

[routing:prod1_x_ro]
bind_address=0.0.0.0
bind_port=64470
destinations=metadata-cache://prod1/?role=SECONDARY
routing_strategy=round-robin-with-fallback
protocol=x