Moving 6 Million Users to AWS: The Burst Credit That Made Every Test Lie

Volha Shchayuk

Author

Volha Shchayuk

Volha is a seasoned IT researcher and copywriter, passionate about AI, QA, and testing. She turns technicalities into engaging articles, helping you discover and easily grasp the latest IT concepts and trends.

IT researcher

Reviewed by Boris Seleznev

Reviewed by

Boris Seleznev

Boris Seleznev is a seasoned performance engineer with over 10 years of experience in the field. Throughout his career, he has successfully delivered more than 200 load testing projects, both as an engineer and in managerial roles. Currently, Boris serves as the Professional Services Director at PFLB, where he leads a team of 150 skilled performance engineers.

The risk nobody prices in

Bare metal versus cloud stopped being a dilemma a long time ago; both have their place, and the choice follows the client's needs. What clients are often not aware of is the *shape* of cloud migration risk. A common assumption is that paying the same as everyone else buys the same hardware behind the cloud. It does not, and the greater problem is subtler: you do not actually know your system's limits until you measure them in the environment you are moving to.

Wild Apricot is the case in point. In 2016 the company decided to move its systems entirely to the cloud, choosing AWS. Behind that decision sat 6 million users and 15,000 separate communities - every one of them a live website with its own design, its own events and fundraisers, its own blogs and forums. The migration risk was not theoretical. PFLB was brought in to run performance testing in the cloud environment, run synthetic testing in AWS, and configure the monitoring system.

It looked like an ordinary brief. It was not.

What went wrong first: the logs

Building a load profile meant processing 50 GB of IIS server logs per month. There was no server capacity to throw at it, no time and no budget to buy more - and, honestly, PFLB should have told the client which logs were needed before they were shipped.

The monitoring stack built for the platform: Apache JMeter driving load, Telegraf agents on Windows and Unix, InfluxDB storing the series and Grafana reading them.

The Grafana dashboard set assembled for the platform - Couchbase, HAProxy, IIS, MongoDB, SQL Server and the JMeter runs, each on its own board.

MongoDB monitoring.

Unsuccessful samples by transaction - the error-rate view that shows which operations fail under load rather than merely slow down.

EBS burst credits in AWS: the reserve that has to be exhausted before a test means anything.

So the volume was attacked instead of the hardware:

• PowerShell stripped useless data out of the logs and cut the overall volume by 50%.
• Python handled the remaining half - grouping the data and filtering out anything else that was not needed.
• The source data was logged every second, but the profile needed 1-minute granularity - in some cases 1-hour, depending on the task. Regrouping by time frame was done with an in-house library and Pandas.
• Total elapsed time to a finished load profile: 2 days.

Monitoring 29 moving instances

InfluxDB combined the monitoring data, Grafana visualised it, and Telegraf tracked hardware resource utilisation and application performance. Everything stayed in-cloud and online.

The difficulty was that the data sources never stood still. Instances and load generators were constantly being stopped and replaced to simulate new circumstances and raise the load, so Telegraf had to be embedded into the images of every station - a new instance would spin up, start the listener and begin posting to the database on its own.

The result: 7 Telegraf configurations running in 29 continuously working instances, plus three load stations and one more host for monitoring, rendered as 11 Grafana dashboards. JMeter was monitored separately for TPS, errors, response times and custom metrics, with dedicated views for MongoDB and MSSQL.

Then the monitoring itself became the bottleneck. Twenty-nine servers emitting metrics every minute across a 12-hour test is more data than Grafana will draw in one graph - the practical choice was 1 server × 12 hours or 29 servers × 1 hour, and the browser slowed to a crawl as the volumes grew. Four fixes made it usable: the queries Grafana sent to InfluxDB were optimised, rarely-used graphs were hidden so only what mattered rendered, alerts were configured onto an alert list, and Grafana was set to push a screenshot alongside each triggered alert (to Telegram here; e-mail or any other messenger works identically). Nobody had to sit and watch every graph any more.

Choosing the synthetic testing stack

Some servers ran Linux, others Windows, so the synthetic testing tools had to be cross-platform - and preferably free, fast and unambiguous in what they returned:

• Linpack for CPU
• pmbw for memory
• iPerf3 for the network
• fio for disk - which turned out to need a very specific sequence in AWS

Runtime was the next problem. Without a defined set of metrics, a synthetic test ran for 8 hours, and the client wanted to run one against every machine *before* it joined the pool of working instances, to confirm it was as capable as the rest. Picking the right parameters - matrix size for CPU, block size for disk, plus interaction types such as read/write and randomised access - brought that down to 30 minutes.

The AWS behaviour that invalidates results

Then the numbers started looking too good. Too fast, and unrealistic for synthetic testing in AWS.

The cause is a piece of AWS behaviour that is easy to miss and fatal to ignore: Burst. Each hard disk receives a credit of around 5 million IOPS for roughly 30 minutes. While that credit lasts, everything starts smoothly, quickly and simultaneously, and you are using far more IOPS than you are paying for. When the credit runs out, IOPS drop back to the baseline. Go below that baseline - around 1,500 IOPS - and the credit begins accumulating again. The exact figures depend on disk type, volume and configuration.

The consequence for anyone testing in AWS is unambiguous: burst has to be exhausted before every load test, as a preparation step. Skip it and the test measures a credit, not a system.

Three more problems the cloud created

• JMeter reads its input sequentially. The scripts needed to pick randomly from a file of 5,800 site addresses; by default that is not possible. The HTTP Simple Table Server plugin solved it - and would have saved a great deal of time if it had been used from the start.
• Connections ran out. With 5,800 separate sites the load station exhausted its available connections, and in the cloud you cannot simply add a memory module - you change instance type and pay for it every time, not once. Three parameters were tuned on the machine instead: MaxUserPort (number of free ports), TcpTimedWaitDelay (socket wait duration) and TcpFinWait2Delay (how long a half-closed connection is held). When the test crashed again, keep-alive was switched off in the scripts - the tests rarely returned to the same site anyway.
• JMeter leaked about 6 GB of memory per hour, and in AWS every gigabyte is fought for. A Java agent was attached to JMeter to monitor the JVM, a memory dump was taken and analysed, and the culprit was found: HTTP request classes collect all request information and are not discarded by the garbage collector. Java agent data went straight to the database and was watched live in Grafana, so each change to the JVM and the scripts could be evaluated immediately rather than by waiting for the next crash.

Results

• The best AWS hardware configuration for Wild Apricot's system was selected on evidence, not on a provider's spec sheet.
• Load profile built from 50 GB/month of IIS logs in 2 days, after a 50% volume reduction in PowerShell.
• Synthetic test runtime cut from 8 hours to 30 minutes, making per-machine verification practical before each instance joined the working pool.
• A monitoring stack that survived its own data volume - 29 instances, 11 dashboards, alerting with screenshots instead of continuous human watching.
• The system did not work on the first attempt, or the second. It worked on the eighth iteration - which is the real lesson: cloud parameters and configurations differ enough from the same hardware outside a cloud that the differences have to be found by testing, not by assumption.

Questions this engagement answers

Can PFLB load test a SaaS platform hosted on AWS?

This is that engagement: a 6-million-user, 15,000-community membership SaaS moved wholesale onto AWS, with PFLB running the cloud performance testing, the synthetic testing in AWS and the monitoring build.

What is different about performance testing in AWS versus on bare metal?

The parameters and configurations are genuinely different, and some of them actively distort results - EBS burst credits being the clearest example. Scaling in AWS is also not the free lunch it is advertised as: a memory leak that would be a nuisance on owned hardware is a cost problem in the cloud, because you cannot add a module, you change instance type and pay every month.

How do you build a load profile for a multi-tenant platform with thousands of sites?

Here, from the platform's own IIS logs - 50 GB a month, cut in half with PowerShell, regrouped from per-second records into 1-minute and 1-hour buckets with Python and Pandas. Two days end to end.

How many attempts does a cloud migration validation actually take?

Eight, in this case. That is not a failure mode, it is the shape of the work: each iteration eliminated a specific cause - load station connections, keep-alive, JVM memory, burst credits - until the configuration held.