Hello!

These days with so many (buzzword) cloud services, the standard vanilla LAMP stack is slowly becoming a thing of the past. Or at least more obfuscated / confusing. Its important when building your web application to consider things like analytics, metrics, reporting, capacity planning and scaling. You need to be able to scale your web application outwards, but you also need to be able to have a methodical system in place through which you can identify the weak points or bottlenecks in your stack. Identifying the bottlenecks are important because they help you determine what your “ceiling” for the maximum number of requests your web stack can handle before it buckles under the load.

Typically these types of bottlenecks appear in the classic or common end points within a standard web stack : web and database services.

Identifying bottlenecks

With apache or nginx , there are many tools available to ensure that your web and database services are optimal. These tools will measure the average memory allotment per process and scan your current configuration to determine if you are over or under compensating with respect to the available resources on your server.

In apache it would be the MaxClients directive. If your average process is 50mb and you only have 2GB of available ram then a MaxClient directive of 400 will be way too high. Same goes for a MySQL configuration.

For apache, I’ve used the tool apachebench that does all the scans and assessments automatically. A report is then generated that issues recommendations. For MySQL, a similar script called MySQLTuner will help you assess the optimal mysql configuration based on the current runtime stats of a live database service.

Taken one step further, it would be a good idea to implement MySQL slow query logging and PHP’s XDebug extension. This will help you determine if and how many expensive queries are happening between php and mysql. Additionally it may help you optimize your code beyond just mysql queries.

Scaling MySQL

Aside from simply adding more resources to a singular MySQL server, you can create read-only mysql slave servers in order to split off your reads and writes. This will allow you to define multiple read only slave database servers.

This will definitely help you scale out your web application if it is read-heavy. What if its not?

Caching

There are many points where caching systems can be implemented into a web application. Varnish is a caching system that sits in front of your web service as a transparent proxy. Requests are received and processed by varnish. If it already has a cached copy of the request in RAM, it will serve straight from ram. If not it will “poll” the backend web server (apache / nginx / etc).

There is also caching systems like memcache which works in a similar way that varnish does. The downside of memcache is that it has to be developed into your code to rely on memcache where necessary. The benefits of both ends of caching are clear, however.

Load testing

Recently we came across a really robust and clean python based load testing module called Locust.

You basically define your own configuration file that interacts with a front end web application (GET/POST/etc) :

from locust import HttpLocust, TaskSet, task


class UserBehavior(TaskSet):
    def on_start(self):
        """ on_start is called when a Locust start before any task is scheduled """
        self.login()

    def login(self):
        self.client.post("/login", {"username":"username", "password":"password"})

    @task(2)
    def index(self):
        self.client.get("/")

    @task(1)
    def task1(self):
        self.client.get("/about-us/")

What the above configuration directive does is log into the web application with a test user account and then gets the homepage and an about-us page. With Locust you can define number of users and how often to spawn a new user session (i.e. 100 users with 1 every 10 seconds spawned). That will help define a request-per-second load testing model.

Then you can monitor all the services described above (varnish, memcache, mysql, apache/nginx) and look for slowdowns and bottlenecks. Locust will detect any web failures (i.e. 403, 404, 503, etc). Metrics reported from Locust in conjunction with additional monitoring will help you find the weak points in your web stack. That will then help you make capacity planning decisions for the future. You could take the rate of growth for your website over a particular period of time and predict where you will need to be in 3/6/12 months time. Without proper planning your uptime will be affected and you’ll hear from your direct managers much more often 😉