Cloud computing is here, and it’s changing everything: infinitely scalable on-demand hardware, no up-front costs, pay-as-you-go pricing, and click-and-drag instead of rack-and-stack. It’s a paradigm shift in how IT functions, but it is no silver bullet. There are limits to the abstracted infrastructure, and undelivered promises around automatically scalable hardware. To analyze the promise and the reality of cloud, Alistair Croll spent five months doing primary research across five of the most popular cloud platforms and published his findings publically. Below, we analyze the study, review the conclusions, and identify the most relevant takeaways.
“Over the course of several months, we created test agents for five leading cloud providers that would measure network, CPU, and I/O constraints. We also analyzed five companies’ sites running on each of the five clouds.”
The goal was to measure both the performance of each cloud platform across four categories (latency, bandwidth, CPU, I/O) and the aggregate performance of websites hosted on each clouds. The aim was to determine:
- 1. How do different clouds perform for web users?
- 2. What are the differences across clouds for specific functions – network delivery, computation, and back-end I/O?
- 3. How much is one cloud user affected by its neighbors?
- 4. How do IaaS and PaaS clouds vary in performance?
- 5. How do different platforms handle spikes in concurrent requests?
- 6. How variable, or predictable, are specific cloud platforms?
Conclusions and Takeaways
Each business will find different things to take away from the results of the study, but below are the findings that will apply to most everyone:
- 1. Different clouds are great for different things – Salesforce and Google have amazing global bandwidth, while Amazon has the most consistent CPU performance. Rackspace has excellent I/O performance, but average bandwidth. In general, Google excels in most categories, but getting data in and out is difficult.
- 2. The cloud’s performance will affect your performance – The data showed a correlation between the performance of every website hosted on a cloud, which means that the cloud itself plays a major role in the performance of your website and web application. This variation is not insignificant.
- 3. PaaS does not yet fully deliver on its promise – The PaaS promise is that you write your application and let the provider handle everything else. Unfortunately, we are not quite there yet. The Force.com platform has serious limitations around the amount of work you can do per hour. Google App Engine makes it painful to import large amounts of data, and does not optimize your application to reduce your costs in any way. These things are coming, but are not yet here.
- 4. Monitor your cloud performance carefully – Considering that your performance is affected by the performance of your cloud and by your neighbors on the cloud, you need to watch your web site/application’s performance extremely carefully. This means that you need to monitor it externally from around the world, in addition to monitoring the lower level resources such as CPU, I/O, and steal time. This is necessary both to understand your users’ experience and to find the root cause of slowdowns.
- 5. The cloud is the future – In spite of every issue encountered, it is clear that the cloud has huge advantages over traditional infrastructure. Setting up a test like this without a cloud infrastructure would have taken a huge amount of time, money, and expensive resources. Instead, as soon as the test was done, the virtual servers could be shut down and the costs went to zero.