AWS Micro Instance WordPress Tuning

Recently I registered a new domain, rootuser.ninja, for the purpose of testing AWS using the 12 month free tier offer. That gives you 750 hours per month to run a micro server, 1GB RAM, 8GB hard drive and 1 vCPU basically for free for a year. What’s not to like about that?! After the 12 months, each micro server is $0.012 per hour, that’s cheaper than a lot of web hosting and you get a dedicated server.

After spinning up my little VM with a CentOS 7 image I installed:

  • Apache with all the wordpress pre-requisites
  • MariaDB
  • Postfix for SMTP
  • Dovecot for IMAP
  • Amavis for Email scanning
  • Spamassassin for SPAM scanning
  • Clamav for Email antivirus
  • WordPress
  • Webmail

After configuring all the software, importing my WordPress database and installing my themes and plugins, this little free VM from Amazon is basically a web hosting service in a box. I get Email with antispam and antivirus, a web-based Email interface, IMAP and SMTP Email for my mobile devices and website hosting with WordPress; so this little 1GB server is doing a lot of work.

WordPress isn’t exactly a light-weight option for hosting web pages. If I wanted raw speed I would probably install something like Nginx with static pages. But, I wanted to put this little server through its paces, WordPress is very popular and this is an extremely standard configuration that anyone can support.

The next thing I did was went looking for benchmarking tools. I’ve used Siege before, but it’s labor intensive to run a lot of scenarios and extract the data in a manageable format. What I found was LoadImpact.com. This is a basic web stress testing tool that generates good reports, lets you customize the scenario, and exports the data to CSV if you want to do custom reporting. They integrate with NewRelic, so you can overlap your server statistics if you’re a NewRelic customer. And, even better there’s a free tier where you can spin up test scenarios to see how the service works.

This is basically the stock install, with very little tuning done, here are the results with 25 virtual users:

LoadImpact stock config 25VUs

As you can see, the wheels start to come off between 14 and 16 virtual users. Here’s vmstat output from the server:

procs ———————–memory———————- —swap– —–io—- -system– ——–cpu——–
r b swpd free inact active si so bi bo in cs us sy id wa st
0 0 128724 77520 423532 423464 0 0 2 1 27 72 0 0 100 0 0
0 0 128724 77396 423532 423464 0 0 0 4 40 97 0 0 100 0 0
0 1 195804 60720 450960 412876 0 13416 23894 13423 1095 744 20 4 63 13 0
0 0 300532 62640 452568 394268 0 20946 42329 21000 1591 1239 23 6 46 25 0
0 0 300532 68908 435772 402976 0 0 2900 0 126 137 0 0 98 1 0
0 0 300532 156012 432580 323940 0 0 2718 0 497 479 20 3 77 1 0

What you see is the server going along, completely idle, then the virtual users start. Immediately, there’s pressure on memory and the kernel starts swapping out to disk, the “so” column. At the same time there’s some CPU usage, the “us” column, and some stress on the I/O subsystem the “wa” column. This is when the server starts spinning up httpd processes to handle the requests. Once those processes are started, then it’s quiet again.

procs ———————–memory———————- —swap– —–io—- -system– ——–cpu——–
0 0 300532 191748 433244 290536 0 0 474 13 95 156 0 1 99 0 0
1 0 300532 95320 433596 390152 0 0 166 1 499 296 22 3 75 0 0
1 0 377184 62916 417864 407452 0 19766 23860 19785 1521 1241 38 7 46 9 0
0 4 467204 71552 409468 410024 6 13569 39147 13578 1257 785 11 5 63 20 0
0 0 473548 104864 410744 380584 0 1269 12700 1270 574 394 10 1 81 8 0
0 0 473548 109700 412584 379204 0 0 467 9 111 190 0 1 98 1 0
0 0 473548 105012 411360 389924 0 0 1788 18 565 530 20 3 76 1 0

Then, as a few more virtual users startup, there’s another hit, a little larger this time. And another quiet period.

procs ———————–memory———————- —swap– —–io—- -system– ——–cpu——–
r b swpd free inact active si so bi bo in cs us sy id wa st
1 0 473548 98528 411900 396088 0 0 942 0 481 299 19 2 78 0 0
0 0 496108 98340 420072 389376 6 4513 2642 4514 527 379 22 3 75 1 0
0 0 534600 84128 415984 400040 0 7698 2881 7711 971 509 40 3 53 4 0
0 52 601956 65720 406936 402948 491 21736 76078 21756 2012 870 6 27 41 26 0
0 10 750884 68616 405672 405504 436 24171 70228 24208 1949 1221 19 23 0 57 1
2 15 830092 74364 403444 403420 211 13290 64972 13294 1867 1024 20 7 0 72 1
6 4 918572 65196 408888 413156 952 18221 48684 18230 2651 1327 69 13 0 18 0
0 0 937712 69840 419668 366812 207 3874 22123 3885 1353 713 27 5 58 10 0
0 8 1045236 75040 391872 391388 458 21505 46228 21516 1573 935 16 7 49 28 0
6 3 1092932 59632 393496 410188 765 10330 60089 10331 1784 1112 24 7 0 68 0
1 2 992984 154652 350388 364660 14298 0 26733 11 1582 1501 31 4 51 14 0
1 4 971284 70608 389528 410608 4701 7465 19646 7475 1444 958 43 7 46 5 0
0 0 1002300 125788 382248 367688 1703 6318 54590 6332 1499 989 13 4 24 57 1
0 16 1069088 70420 403344 403080 3590 16073 59748 16212 2275 1373 31 12 3 53 1
0 38 1096192 70276 406416 407696 4713 14312 70625 14500 2055 1096 19 11 0 69 1
2 28 1155908 63692 419576 402056 5941 15580 66120 15596 2094 1472 13 9 0 78 1
0 25 1190732 76336 407212 407328 17626 27545 63867 27556 2726 2134 24 9 0 67 1
0 30 1201480 66744 414152 413876 17462 21536 64895 21564 3241 2028 39 8 0 52 1
0 26 1225468 69684 414936 410704 16846 16354 68164 16368 2887 1989 27 7 0 65 1
0 29 1201136 63584 420580 411332 11280 10859 67350 10869 2440 1660 11 17 0 71 1

Finally, there’s another big spike as more virtual users come online. But, now the server is in trouble. You can see the available memory has dropped into the 5 digits, without really recovering, and the server has started swapping IN from disk, the “si” column. There’s also a lot of I/O going on, and the disk can’t keep up. The CPU is being used, but it’s not busy. At this time, for 25 virtual users, Apache had started over 80 processes!

This is a pretty typical memory constrained system, which is then causing an issue with the disks because of swapping; the server just can’t feed the CPUs fast enough. So, the fix would be to add memory until we have the swapping under control, then see if there’s still an issue with the disk subsystem. We’re not going to do that, so lets start by limiting the Apache processes.

The web server will be mostly idle, but the server will need to process Email 24×7. I’m sizing it to start a maximum of 20 processes. To limit the number of servers that Apache starts I’ve removed some modules that I won’t be using and added these lines to the httpd.conf file:

StartServers 3
MinSpareServers 3
MaxSpareServers 4
ServerLimit 20

This is very small. It says we’re planning on the Apache server being mostly idle, starting just 3 processes, and we’ll run a maximum of 20 server processes. We will pay a penalty when the Apache processes startup, but we’ll keep more memory free for other processes when it’s not needed.

Here are the results of the second run:

LoadImpact ServerLimit 20 with 25VUs

 

This is a much more linear result, and the numbers are much better as the number of virtual users ramps up past about 16. Here’s that the server was doing:

procs ———————–memory———————- —swap– —–io—- -system– ——–cpu——–
r b swpd free inact active si so bi bo in cs us sy id wa st
0 0 43656 74440 505044 350028 0 0 29 1 47 113 0 0 100 0 0
0 0 43656 74440 505048 350028 0 0 0 0 48 111 0 0 100 0 0
0 0 64808 76588 490660 361252 0 4230 6077 4231 290 334 2 1 90 7 0
0 0 64808 76464 490728 361256 0 0 6 6 52 122 0 0 100 0 0

Here’s a bump similar to what we saw with the first run, but a bit smaller. After this there’s another second wave as the processes spin up, very similar to the first run. But, look at what the numbers look like toward the end of the run:

procs ———————–memory———————- —swap– —–io—- -system– ——–cpu——–
r b swpd free inact active si so bi bo in cs us sy id wa st
0 0 794712 104832 419764 388220 6 2910 3136 2923 816 551 38 3 56 2 0
0 26 849428 84784 416532 411036 18 10948 70797 10958 2143 781 24 45 16 14 0
0 0 848928 117556 435740 360032 82 0 5038 21 1236 748 62 4 31 4 0
1 0 848616 88356 445936 378980 77 0 2366 0 461 338 11 1 87 1 0
0 0 848608 103152 448652 361544 6 0 606 10 625 328 30 1 68 0 0
0 0 847308 102752 448372 362116 261 0 294 0 184 212 1 0 98 1 0
0 0 847300 100744 448896 363704 0 0 105 9 475 301 20 1 79 0 0
0 0 847296 100744 448896 363704 0 0 2 12 115 158 0 0 100 0 0
4 0 847288 77224 448896 387736 0 0 0 1 284 208 12 1 87 0 0
0 0 847264 117996 427260 367728 0 0 0 17 1118 477 66 4 30 0 0

Where before we had tons of swapping in and out, and high I/O Wait numbers, the server now looks relatively quiet. CPU usage is relatively low, there’s low disk I/O overall, and a fair amount of free memory even if we are using 800MB of swap.

Now the memory workload is tuned to about the maximum the server can handle and there’s no constraint on CPU or Disk. Unfortunately our response time goes over 20 seconds at about the 14 virtual user mark, that’s pretty slow.

Next we’ll look at how improve the performance of WordPress.

Leave a Reply

Your email address will not be published. Required fields are marked *

*
*