Cloud performance compared with New Relic

Recently I ran into some performance problems with a moderately complex Drupal 6 site running on EC2. New Relic reported an average application time around 1000-1200ms, clearly too high. Since the site was already rather optimized with MySQL being tuned, using Nginx and Varnish, as well as APC and Memcache I started to suspect the instance itself.

I came across an interesting post from 2009 which did seem to indicate that processing performance, especially in regards to I/O, varied widely across cloud hosting providers. This led me to switch to Linode with that site but I made some tests along the way to show how instances differ.

Test setup

To make relevant comparisons between the individual offerings I created a Locust script (more on that in a later post) to log into the site and crawl a random but limited set of pages. The test ran with an average of 10 concurrent users for around half an hour each.

Disclaimer

• The tests wer run on a single instance, they are thus not statistically sound and your mileage wiIl vary; though I do think that the comparisons give valuable insight.
• The EC2 instances were all run from the same EBS volume in US East and should be directly comparable, for Linode the same OS and setup was undertaken but details do differ.

Results

My laptop was used to establish a baseline performance with real hardware, which came in at 272ms. To get below this it would be necessary to look at individual features and see if they can optimized or removed, which isn’t really an option for me.

The EC2 micro instance was basically unsable and ramped up to an application time of 4030ms.

The small instance, which was also used in production, performed similarly to the usage I saw on the live site and clocked in at 1180ms.

When switching to an EC2 medium instance, performance was finally at a decent average with 533ms.

In contrast, the entry-level from Linode “Linode 1GB” averaged 841ms.

Comparison

The conclusion I drew from this is that the performance-to-price ratio is too low for using EC2 when running such a Drupal site with a single host and hosting costs are an issue, such as for non-profit organzisations or private sites.

These tests are relevant for quickly delivering a site with a predictable and limited usage pattern: speed, not scalibility. If I had a highly fluctuating user load (e.g. mass media events) the situation might be quite different. In those cases, launching a few more instances and paying spot prices for medium or larger instances for the time needed is probably still cheaper than paying a monthly fee for those instances at a hoster like Linode. 

I do want to highlight that this is highly dependent on your application and its users. For example, I’m running a Django-based site on a micro instance in EC2 Ireland and it comes in at an impressive 92ms at a very good price: