Scaling LAMP

Question

I have a client with a LAMP website serving mostly video. He is currently on one server with all components. He is having some scaling problems. What are some of the techniques that can be used to help.

I used separating out the DB to another server with a GB Ethernet between it and the webserver. Maybe adding more web servers with some load balancing and additional MySQL servers with replication?

Would like some medium, large, super large examples on how to scale if possible.

The video is actually coming as jpg images. Similiar to this website:

http://www.webcams.travel/webcam/1170170095

And to add some details. I figure the max visitors per hour would be 1000, and I think would be lucky to have that. Maybe closer to 1000 a day.

Answer 1

The advice about CloudFront and MemCache etc. is all good, assuming those address the root of your performance issues.

On the database side: Profile, profile, profile. Moving the DB to a separate server was (probably) a good step, but if you haven't profiled the queries being run on this DB instance, you don't know what you need to improve.

Start by doing an EXPLAIN on the most common queries, and check whether you're doing unnecessary sequential scans on large or growing tables. Index as necessary.

Are any of the queries passing back columns that aren't used?

Are you, as other commenters were worried, serving your graphic/video content from the database? Files should be kept in the filesystem, and served from there.

Are all your tables MyISAM? Frequently-updated tables may show performance improvement by moving them to InnoDB, where they can benefit from row-level locking (as opposed to MyISAM's table-level locking).

These are just simple, general recommendations - without more specifics on exactly what is slow, it's tough to offer anything deeper.

Answer 2

First off, I agree that you must know what the bottleneck is before attempting to scale. Here some run of the mill suggestions to get you started:

1. Get video out of the database. I don't see how this would make sense in any situation.

2. Add one more server to only server the video content and leave the HTML/application to the original one. Not only does this reduce load, but it creates a performance improvement on the client side by overcoming any HTTP connection limits.

3. Cache - This may mean Memcahce or just pre-building HTML output instead of dynamically processing it for every request.

4. If you are fortunate enough to reach 'super large' you might think about a CDN. I'm sure you will encounter many other hurdles before then though.

Best of luck.

Answer 3

You need to work out exactly what the problem is - there is no general solution.

You need to build a performance test environment with real hardware and something which can simulate lots of clients. This may be quite expensive.

If the problem is the database (which it may be) then the solution is often to split it into several databases each holding part of the data.

But I wouldn't imagine that any sane person would store video on MySQL - if he is doing that, then a refactoring may be in order.

Answer 4

Profile to see how much load various parts of his site actually inflict.

I presume most of the load is actually serving the videos - use a proxy to redirect this work to a second (,third, fourth...) server.

Answer 5

Barrett's answers are pretty much what I would do -- id the bottlenecks, look at memcached, move the DB to a separate server from web services, etc.

I would add that Amazon has a new service called CloudFront that will serve up content from geographically close servers. I haven't given it a try yet, but that may be a way to spread the load for a relatively low cost.

Also, look at the presentations from Livejournal and Facebook on how they scale their systems, they may provide some insight depending on how you have your applicaiton(s) structured.

Answer 6

You should profile the application to start with and find out where the bottlenecks are - if it's PHP, using something like xdebug would be a good start. It may be that you spend a long time serializing data, or talking to a database - so perhaps caching with Memcached may help.

Secondly, if you're using MySQL, turn on MySQL slow query log (log_slow_queries in my.cnf); this can be used to give you an idea of what the expensive database queries are and once you know that you can target tuning them via EXPLAIN <> as has already been mentioned. It may be the case that you just need to add a few indexes to the database, at which point it'll become quick again. There are other relevant my.cnf parameters to e.g. log queries not using an index.

Thirdly, look into tools like Yahoo's page speed firefox plugin - it'll make some suggestions (e.g. offload static content, compress stuff you're returning, make sure you have expires headers etc.). Perhaps the server(s) themselves aren't the bottleneck, and pages are taking a long time to render due to a dependency on a third party (external JS, adverts, flash ...).

You'll need to have some idea of how busy your database server is, compared to your web server, to know where you need to add additional hardware (i.e. do you need multiple web servers with a front end load balancer, or do you need multiple database servers with replication or both?).

I'm sure there are loads more that can be said...