Review of Scaleway ARM-based cloud server

by Gabriel, created 30-jun-2018, updated 07-jul-2018

As a web developer I have the opportunity to work with a lot of different hosting providers. The good, the bad and the ugly. In this post I'm sharing my latest experiment with an ARM-based (as opposed of traditional x86) cloud based hosting provider: Scaleway.


Small fish vs big fish - ARM vs x86

Background: ARM vs x86

ARM and x86 are two different architectures for Computer Processor Unit (CPU)
ARM CPU are used in most mobile devices today. While x86 continues to dominate the compute-intensive usages: workstation and server processor. Using ARM processor for server is like asking the question: Can we use a lot of those cheap and power-efficient processors to replace a big and complex CPU AND deliver the same service?

There are essentially 2 manufacturers for x86: Intel and AMD. But Intel have around 80% market share in total (laptop, desktop and server) and even more when we look only at the server market. Very concentrated.
On the other hand there are currently a lot of designers, manufacturers for the ARM CPU: and among them some some big names: Apple, AppliedMicro, Broadcom, Cavium, Nvidia, Qualcomm, Samsung and others... Quiet a competitive environment.

ARM has been very successful in mobile devices because it is cheap and and power-savy. Those criteria are more important than raw computing power in this area. But with the thriving market of smartphones and other portable devices those last 10 years (Iphone 1 was released in 2007) the ARM architecture has evolved a lot: clock speed increased, multi-cores... Now with the 8th generation (ARMv8) the CPU went from 32 bits to 64-bits, bringing it closer to the current architecture and power of the x86. Not here yet but on it's coming fast.

On the software side, most of the Linux distribution and other critical projects (for the web) (webserver, browser, database, hypervisor...) are already available and supported on ARM. That's mean that from a enduser point of vue: it doesn't matter if the underlying CPU is ARM or x86, all the important software of the web ecosystem can run on it.

So now, one step further can we or can we not safely and efficiently replace x86 server by an ARM servers? If we can we can expect to reduce hosting cost because ARM chip are both cheaper to build and to run (per MIPS).

Scaleway is the first cloud hosting provider, to my knowledge, to offer ARM-based server. And it is cheap. Let's try to create a Virtual Private Server (VPS) and configure it to run a website.

The test

The admin interface is nice and sleak. No further comment needed on that.
Creating a VPS is straight forward. Most of the VPS and other products are based on x86. ARM is just a recent addons to their offer. There are currently 7 ARM-based VPS: from 4 cores to 64 cores ! (and from €2.99/month to €279.99/mo ). In this test I will review the 4 cores and the 8 cores. All are based on ARMv8 64bits SoC by Cavium ThunderX.

Scaleway admin interface: server creation
Admin page: choice of the VPS model

After the account is created and credit card entered, You choose your server and it will be created in less than 2 minutes.
You choose the OS among 9 differents Linux distro officially supported (Ubuntu, Debian, Fedora...) + the community provided OS: some more specific image: wordpress, node.js ; or even Docker!

Scaleway admin interface: server creation
Admin page: VPS is ready to be used

You will need to register your ssh public key in the admin interface before the server creation (use to grant you ssh access to the VPS). A (optional and paid) public IP can be affected to your VPS.

I choose to install Ubuntu 16.04 + the ordinary stack to powered a Drupal website:

et voilà!
Drupal Status report page at the end
Drupal Status report page at the end of the configuration

No issues, no difference between the configuration of this LAMP stack and the configuration of a LAMP stack on x86. And performance, admin pages are working fine. But this is just test with 1 user (me), let's try to do a load test and compare it to another server.

How does it compare to an AWS EC2 instance?

I choose to compare the Scaleway servers with a very popular server EC2 of AWS. The comparison included 4 real life servers: 2 of each types:

Those servers are in the low to middle-end for each company. Bear in mind that the price differ a lot, but the goal is just to give an idea of what value you can get for your money.

Note: cost at the time of publication, Scaleway Euro price converted to USD. For AWS I used Paris datacenter pricing, to have a better comparison with Scaleway (based in Paris too).

Simple benchmark: sysbench

sysbench is a lightweight benchmarking tool for linux. It supports testing CPU (multithreading), memory, file I/O, and other stuff. It is available as a package in all popular linux distribution. (But in that case I'vE downloaded the binary package directly from the vendor website to use the latest version: (1.0.14 on arm, 1.0.15 on x86).

test 4 ARMv8 8 ARMv8 t2.small t2.medium
CPU:
Results: events per second
more is better
2,530 evt/s 5,059 evt/s 875 evt/s 1,752 evt/s
File I/O random read
Results:
Read per second
Throughput
66,616 reads/s

1,040 MiB/s
182,974 reads/s

2,858 MiB/s
7,535 reads/s

117 MiB/s
431,607 reads/s

6,743 MiB/s
Memory:
Results: events per second
4,064,073 evt/s
3,968 MiB/sec
7,292,855 evt/s
7,121 MiB/sec
793,445 evt/s
774 MiB/sec
1,302,535 evt/s
1,271 MiB/sec

Commands used:

Summary

Those 2 Multi-core ARM server seem to perform clearly better CPU-wise and memory-wise for those indiviudal tasks. It is not a surprising results: 8 cores is doing more operation than 2. It is interesting to notice that I/O are still much better for the AWS servers. Although I think it is more related to the fact that SSD disk powering those servers are more efficient.

global benchmark: ApacheBench

95% of the servers I have installed are used to host a website. They combine a HTTP web server (eg: Apache), a database (MySQL), a PHP framework (or CMS) and some custom PHP code: this is a LAMP stack. In this scenario the global performance of the entire stack is important. It is the global performance that will defined the response time of the server and the capacity to handle heavy traffic.

So in this section I'm using ApacheBench (ab): with this tool we can send to the server a lot of HTTP requests in a short period of time, simulating big traffic on the website, and measure the average server response time. This tool is not perfect because it won't represent exactly "real life" traffic but it is ok to compare different servers like we are doing here:

We are requesting with ab the "List of content" admin page in Drupal. In this test website I have created 50 test pieces of content to populate the page.

ab -n100 -c4 -v1 -C 'SESSYYYYYyyyy=ZZZZZzzzz' http://<SERVER_IP>/d8/admin/content

Command sent

In this commands:

The results:

test 4 ARMv8 8 ARMv8 t2.small t2.medium
number=1
concurrency=1
total=0.569s
request=569ms
total=0.611s
request=611ms
total=0.12s
request=120ms
total=0.127s
request=127ms
number=100
concurrency=4
total=15s
request=147ms
total=15s
request=153ms
total=11s
request=107ms
total=6s
request=60ms
number=200
concurrency=8
total=28s
request=142ms
total=17s
request=86ms
total=23s
request=116ms
total=12s
request=61ms
number=400
concurrency=16
total=60s
request=149ms
total=34s
request=86ms
total=48s
request=120ms
total=25s
request=62ms

Summary

The resultats are more comparable: the 2 best servers are the t2.medium and 8-ARM with the t2.small performing slightly better than the 4-ARM and the t2.medium performing slighly better than the 8-ARM. In this more complex test, those 2 arm servers stand slighltly behind the x86 EC2 instances chosen, but still show decent capacity.

Conclusion

It seems to me that those ARM-based server by Scaleway are working very well to power websites. In fact it doesn't make any difference for me to create and configure those servers as opposed as a traditional x86. And as a developer/devops this is important: this encourages technology adoption. Of course, this was just a short test and we will need more time to make sure it doesn't come with drawbacks. And I'd like to see other cloud provider entering the market.
But, and this is the main point, if it turnout that those processors are significantly cheaper to produce and to run (because less electricity consumption) for the same MIPS (million instructions per second) then it will mean that hosting cost will be reduced. And this is good for all!