Apache Bench: Why so many 503s?
-
23-06-2021 - |
Question
I'm running a dedicated server on DreamHost: Quad Core, 4GB RAM.
Linux (Debian), Apache, PHP (with XCache for opcode caching), MySQL.
The URL I'm querying has 18 MySQL queries every time.
I'm wondering why when I do "ab -n 500 -c 50 http://<url>/" I'm getting the following results:
Server Software: Apache
Server Hostname: <url>
Server Port: 80
Document Path: /
Document Length: 162809 bytes
Concurrency Level: 50
Time taken for tests: 12.664 seconds
Complete requests: 500
Failed requests: 377
(Connect: 0, Receive: 0, Length: 377, Exceptions: 0)
Write errors: 0
Non-2xx responses: 377
Total transferred: 20291876 bytes
HTML transferred: 20147278 bytes
Requests per second: 39.48 [#/sec] (mean)
Time per request: 1266.372 [ms] (mean)
Time per request: 25.327 [ms] (mean, across all concurrent requests)
Transfer rate: 1564.81 [Kbytes/sec] received
Connection Times (ms)
min mean[+/-sd] median max
Connect: 0 0 0.4 0 2
Processing: 194 1190 1617.0 248 6060
Waiting: 194 965 1263.9 248 5190
Total: 194 1190 1617.0 248 6060
Percentage of the requests served within a certain time (ms)
50% 248
66% 359
75% 1555
80% 2787
90% 3979
95% 4709
98% 5427
99% 5967
100% 6060 (longest request)
All 377 failed requests are 503s. What could be causing this? Running top
while doing this saw my processor spike to about 24%, but didn't tax it too much.
My CMS says that with a single load of the page, it takes .0017 seconds to load base classes, .1711 seconds for controller execution time, for a total execution time of .1731 seconds, using 2.887 MB of memory.
ab -n 100 -c 10
always has 0 failed requests, but it seems that anything with concurrent connections over ~25 has a bunch of failed connections.
I'm expecting a ton of traffic next week all at once, how can I best prepare my server? Will this hold up as is?
edit: from my httpd-mpm.conf file, which is included via httpd.conf
<IfModule mpm_worker_module>
StartServers 4
MaxClients 600
MinSpareThreads 50
MaxSpareThreads 150
ThreadsPerChild 25
MaxRequestsPerChild 0
</IfModule>
Solution
If this is a DreamHost server and not a VPS, it's probably that they're killing your php processes because you've used up your slice of the CPU share. If one of your processes takes up too much resources or there are too many incoming connections, it'll get killed and the webserver will return a 503.
I have no idea what these limits are set to, but someone from dreamhost explains in this thread:
There is a limit of memory and connections your website can handle here on DH. If you are getting to many visitors which are using too many ressources you'll be getting 503 errors. Maybe your website has grown too big after three years? Without more information about the number of visitors and the used software we can only speculate.
OTHER TIPS
This can be caused by a few different things
- Your webserver is hitting open file limit (check ulimit)
- If you are using FPM you might be exceeding child limit (check fpm config)
- You can be exceeding connection limit (check sysctl.conf)
First thing you should do is check the nginx or apache log.