CPU issues after upgrading from Magento 2.2.4 to Magento 2.4

Question

We recently upgraded our site from Magento 2.2.4 to Magento 2.4.

Done the upgrade on a copy of our site on a test server and everything was fine.

When we upgrade on our live server, the page load time increased dramatically when 4+ people were on the site at the same time. This also crashed EleasticSearch, so we moved that to it's own VPS and ES works fine now. Before the upgrade, 15-20+ users online at the one time wouldn't have been uncommon and the server handled it fine.

With 2.2.4, we had a VPS with 6GB RAM, 4 CPU's. Our hosting provider suggested we increase this when we ran into issues after the upgrade. We're now at 8CPU's, 12GB RAM and although that improved performance, load times from server were very long.

We now have Varnish running on a separate VPS and while that has sped up load times, it's still not good enough. Varnish has been running for the last 24 hours and we've been getting 503 and 504 errors and our Magento developer has told me these are due to Varnish waiting so long for our Magento server to respond.

Our hosting company is now telling us we need to get a dedicated server, is this necessary? Our Magento developer has said our VPS, if there's no issues with the server, should be fine. Our hosting company is telling us that the server is fine.

We're unsure what to do as we haven't much confidence in our hosting company as they have just been telling us to increase our package, without really investigating why we're having these issues.

Answer 1

We had a similar issue after we upgraded to 2.4.1 and this was the root cause of the problem:

Magento 2.4 critical load of RAM triggering OOM killer due to CSP module 100.4.0

Here is the patch that we applied.