Diagnosing slow page 'Waiting time' between two web servers

Question

We have just installed/configured a new web server to replace our out-of-date one. Let's call the old server 'server1' + the new server 'server2'. They are both running the same website, with the same code, yet the older server is still serving web pages far more quickly than the new one. Firstly, here is some more detail on the server specs:

Server1 config:

Linux server1 2.6.32-25-generic-pae #45-Ubuntu SMP Sat Oct 16 21:01:33 UTC 2010 i686 GNU/Linux

MemTotal: 6180036 kB

8 cores Intel(R) Xeon(R) CPU E5620 @ 2.40GHz

Server version: Apache/2.2.14 (Ubuntu)

Server2 Config:

Linux server2 3.2.0-23-generic #36-Ubuntu SMP Tue Apr 10 20:39:51 UTC 2012 x86_64 x86_64 x86_64 GNU/Linux

MemTotal: 24682544 kB

16 cores Intel(R) Xeon(R) CPU E5620 @ 2.40GHz

Server version: Apache/2.2.22 (Ubuntu)

The major differences are that server2 has a 64bit OS architecture and also has a lot more RAM and CPU power.

I don't know about you, but I'd expect server2 to blow the socks of server1. However This is not the case when it comes to serving webpages.

Developer Tools output for both servers page loads:

Waiting Time: 314ms

Waiting Time: 5.45s

As you can see, the exact same file requests are made from both machines, yet server1 is still dominating.

I have tried looking into several factors that may be affecting the 'Waiting Time', but I am having a lot of difficulty narrowing down where the time is being spent. I have looked into the Apache2 configuration, the exact same directives and module extensions are applied across both machines... diff'd the code and verified it is identical... The network/ping/nslookup times are pretty much identical.

Can anyone explain what exactly how 'Waiting time' is calculated, and any practices that can be used to narrow down the problem?

Many thanks, ns

Answer 1

Yes I did eventually get to the bottom of it. Although the solution to your problem is likely to be very different to mine. 'Waiting time' is quite a broad term which can relate to anything that executes on the server-side.

In my case, I had to manually debug the code in a rather painful way, by printing loads of microtime() statements around the code to figure out where the time was being lost.

It turned out that there was some legacy code which had an exec to /sbin/route.

On the original 32-bit systems, this was executing in no time at all, but from 64-bit machine, it was taking in excess of 5 seconds:

$ time /sbin/route
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
default         10.10.10.1       0.0.0.0         UG    100    0        0 eth0

real    0m5.007s
user    0m0.000s
sys     0m0.004s

Not sure if /sbin/route generally executes slower on 64-bit machine (if so, It'd be nice to know why), or whether it was just the newer OS version that was to blame.