At a glance
- Client: an aviation group. System under test: BoardConnect - a service that broadcasts audio and video content to airplane passengers over Wi-Fi
- Why a second test: another popular airline had already tested BoardConnect and the results were available; the customer decided to double-check them, knowing that a passenger-visible failure damages the carrier's image
- Target load: 50 independent passengers, each on their own mobile device, connecting after the plane's ascent
- The constraint that shaped the project: it was impossible to emulate more than seven mobile devices from one personal computer - so a bench had to be built rather than a script written
- Bench built: 10 laptops on a switch, one acting as master, hosting 50 Ubuntu 16.04 virtual machines on the principle "one adapter, one virtual machine, one user"
- Radio reality: the planned twenty-five 2.4 GHz connections failed against surrounding networks; the final split was 34 connections at 5 GHz and 16 at 2.4 GHz
- Load profile: 35 users watching video, 10 listening to audio, 3 reading PDF books, 2 viewing transfer information - four separate JMeter scripts
- Finding: BoardConnect could not stably deliver multimedia content over Wi-Fi to 50 passengers; the customer sent the device back for reworking
A hardware problem wearing software clothes
PFLB launched the project in early spring and worked on it throughout March. The task looked like an ordinary load testing brief - check the stability of a media broadcast service - and was not one. Every previous engagement had put load on software. This one put load on a box in an aircraft cabin, over a radio link, and the difference showed up immediately.
The customer's reasoning was reputational rather than technical. A similar service was already running at another popular airline, which had tested BoardConnect and published results. Management was well aware that if passengers hit problems with in-flight entertainment, the carrier wears it - so they wanted their own numbers rather than someone else's.
That instinct turned out to be worth the project. The earlier study had emulated fewer devices. Fifty was the number that mattered, and nobody had put fifty real Wi-Fi clients on the service.
Building something that behaves like 50 passengers
Virtual users over HTTP would have proved nothing: the service is reached over Wi-Fi, so load generation had to happen over real radio. One laptop carried at most seven simultaneous device emulations, so the team built a fleet.
- Each laptop carried one built-in Wi-Fi adapter plus six USB adapters - two connected directly, four through a USB hub - following the rule one adapter, one user.
- One Ubuntu 16.04 (32-bit) VM in Oracle VM VirtualBox per adapter, each carrying JMeter 5.0. That OS supports Java 8, which JMeter Server needs, and costs little CPU.
- Each VM got two adapters: one bridged to the host's RJ45 for the local network, one to a Wi-Fi adapter for BoardConnect.
- Ten laptops were patched into a switch. One - the master - started and monitored the load and ran nothing else; the rest hosted the VMs.
- Addressing followed a pattern that kept the run commands manageable: a host on `192.168.2.110` carried VMs on `.111`, `.112` and so on.
Result: a network of 50 virtual machines and one master.
The Wi-Fi itself pushed back. BoardConnect runs 2.4 GHz (channels 1 and 5) and 5 GHz (channels 36 and 40); the plan split the load evenly between channels 1 and 36. The 25 planned 2.4 GHz connections failed, most likely crowded out by nearby networks on the same frequency, so the split became 34 connections at 5 GHz and 16 at 2.4 GHz. Acrylic Wi-Fi Scanner showed what was on the air in real time.
Four use cases, and the traffic nobody sees
Traffic was analysed in Fiddler 4 before any script was written, and it surfaced something a naive script would have missed: alongside content downloads, client and server constantly exchange short messages - some carrying flight information, some simply keeping the connection alive. Each script therefore created a separate thread of HTTP GET requests to reproduce that chatter.
Four use cases were built, and the customer specified the mix:
- UC01 - view random videos (movies, cartoons, TV series): 35 simultaneous users
- UC02 - listen to random audio (music, audiobooks): 10 simultaneous users
- UC03 - read books (open PDF files): 3 simultaneous users
- UC04 - view transfer information: 2 simultaneous users
UC03 and UC04 were plain sequences of HTTP GET requests. UC01 and UC02 additionally implemented a media stream emulator using the DASH protocol - because a video session is a stream, not a download, and testing it as a download measures the wrong thing.
Each use case lived in its own JMX file, run from its own copy of the JMeter folder on the master, in remote testing mode over RMI and SSL. The four commands each targeted an explicit list of VM addresses via `-R` rather than the `remote_hosts` property, so the profile could be changed without editing `jmeter.properties`. An Excel sheet with a concatenation formula built the long IP lists.
Reading 50 clients at once
Data went to the classic JMeter + InfluxDB + Grafana bundle, with InfluxDB on the master laptop - the one machine deliberately kept free of load scripts.
Because four independent scripts were writing at once, each script carried two Backend Listeners: one writing to a shared `General` application, one writing to the use case's own name. That small decision bought two things:
- the number of active threads per use case could be monitored separately;
- during real-time analysis of a single use case, far less data had to reach Grafana.
The dashboard started from community Dashboard #5496 (Apache JMeter Dashboard using Core InfluxdbBackendListenerClient) and was significantly rebuilt: top panels for transactions, errors, traffic, average transfer rate, active users and data sent and received, with an `Application` variable to switch data sets; bottom panels for transferred volume and response time per transaction, the latter shown both conventionally and as a heatmap to make change over time visible.
Results
- BoardConnect was shown to be unable to stably provide multimedia content over Wi-Fi to 50 plane passengers.
- The result did not agree with the Lufthansa study, because that study emulated fewer devices - a previously undetected problem was identified and reported.
- The customer decided to send the device for reworking rather than risk reputational harm in the cabin.
- PFLB came out of the project with working practice in stress testing hardware, not just software.
Questions this engagement answers
Can you load test a physical Wi-Fi device rather than a web backend?
Yes, but not with virtual users alone. Here it took ten laptops, a built-in plus six USB Wi-Fi adapters on each, fifty Linux virtual machines and a rule of one adapter per emulated passenger, because the service is only reachable over the air.
Why re-test something a comparable operator already tested?
Because the test profile is the result. The earlier study emulated fewer devices and passed; at the profile that mattered - fifty passengers - the service did not hold. The number tested is as important as the pass or fail.
How do you run 50 JMeter clients without them getting in each other's way?
JMeter's remote testing mode with one master and fifty servers, addressed explicitly through `-R` rather than a properties file, with one JMX per use case so the mix can be adjusted independently. Two Backend Listeners per script keep the per-use-case view separate from the aggregate.
What breaks first in a Wi-Fi test bench?
The radio. Twenty-five planned 2.4 GHz connections would not come up because of the surrounding networks - the profile had to be re-cut as 34 connections at 5 GHz and 16 at 2.4 GHz before a single request was sent.
Testing something that isn't a web server?
If your product is a device, a radio link or an appliance rather than an endpoint, the hard part is building a load source that behaves like real clients - and that has to be designed before any script is useful. That is the work this project was mostly made of.
