Load Testing for Wi-Fi Broadcasts

Experience Wi-Fi Testing Service for Multimedia Broadcasts during Air Travel

Customer satisfaction with our services largely accounts for the success of the company. And nothing is trivial in this industry. Knowing this, our client – the aviation group contacted PFLB with a request to conduct load testing of BoardConnect, a service designed for Wi-Fi broadcasting of audio and video content to airplane passengers.

A similar service was used by another popular airline which had previously tested BoardConnect. However, the customer decided to double-check these results. Our customer’s management was well aware that if passengers experienced problems with the service, the carrier’s image would suffer.

PFLB Launched a Project to Load Test BoardConnect in Early Spring. Our Team Worked on it throughout March.

We faced the task of checking the stability of the Wi-Fi-based media broadcast service. And this was a real challenge, because we had not previously load tested hardware, only software.

How Were We Supposed to Do This?

By simulating the behavior of 50 independent passengers whose mobile devices can connect to the tested service after the plane’s ascent. Their devices should provide them with a choice of various audio and video content: movies, TV series, music, audio books, and cartoons.

How Did We Do It?

We used laptops to perform the task. The problem was in scaling up: it was impossible to emulate the work of more than seven mobile devices from one personal computer. As a result, we had to set up an entire “fleet”. Each laptop used six Wi-Fi adapters: one built-in and six USB (two connected directly, four connected via a USB hub). In this way, we followed the principle “One adapter, one user”.

To facilitate the operation of connected adapters, we created virtual machines (VMs) running Ubuntu. Here we applied the rule “One adapter, one virtual machine”. Each VM was configured to work with a single Wi-Fi adapter, and all the VMs were combined into a local network.

We analyzed traffic in Fiddler 4. We wrote scripts, debugged, and tested using JMeter 5.0, including in remote testing mode. Finally, we collected and analyzed data using InfluxDB and Grafana.

Now let’s walk through the stages of the project and consider individual aspects of each.

Fine-tuning the Workbench. Working with the Wi-Fi Network

BoardConnect connects at 2.4 GHz (channels 1 and 5) and 5 GHz (channels 36 and 40). In our case, channels 1 and 36 should have been used in equal measures.

However, the planned twenty-five 2.4-GHz connections failed. This was probably due to the fact that many other nearby Wi-Fi networks were operating at the same frequency. We worked around this problem by creating 34 connections at 5 GHz. We managed to create the remaining 16 connections at 2.4 GHz.

To view data about the Wi-Fi networks in real time, we used the program Acrylic Wi-Fi Scanner.

Script creation

Four scripts (use cases) were created to emulate user actions in the BoardConnect service:

At the customer’s request, we used the following testing profile: 35 simultaneous users performing UC01, 10 simultaneous users performing UC02, three simultaneous users performing UC03, and two simultaneous users performing UC04. The use cases were implemented in four separate JMeter files.

Additionally, traffic analysis showed that, in addition to downloading content, short messages are constantly exchanged between the client and server. Some of them represent flight information, while others maintain the connection. To emulate this mechanism, each script created a separate thread consisting of HTTP GET requests.

The third and fourth use cases were implemented as a sequence of HTTP GET requests. The first and second use cases, in addition to GET requests, implemented a media stream emulator using the DASH protocol.

Virtual Machine Image

Virtual machines (VMs) were created using Oracle VM VirtualBox. We chose Ubuntu 16.04 (32-bit) as the operating system for the VMs, since it supports Java 8, which is required to run JMeter Server. In addition, this version of Ubuntu is easy to configure and has low CPU requirements.

In addition to the operating system, the VM image contained JMeter 5.0.

Creation and configuration of a network of virtual machines

The test bench consisted of ten laptops connected to a switch by patch cords. Moreover, one computer (we’ll call it the “master”) was used to start and monitor the load, while the rest (“hosts”) were used to create a load on the service under test. The virtual machines (VMs) were started on the hosts. As mentioned above, each VM used a single Wi-Fi adapter.

The master, hosts, and VMs were joined to a local network. To set up this network, we assigned IP addresses in the form of 192.168.2.110 with a subnet mask of 255.255.255.0 for the master’s and hosts’ network adapters.

To connect a VM to this network, we created a network bridge to the RJ45 network adapter for each machine’s host. After the network was started, the IP address in the Ubuntu network connection settings was changed so that the VM had an address similar to the host’s, but differing in the values of the unit digit. For example, if the IP address of the host was 192.168.2.110, then the VM on this host had 192.168.2.111, 192.168.2.112, and so on.

To provide Wi-Fi connectivity to the service, a network bridge was created to one of the Wi-Fi adapters for each VM. Thus, the VMs had access to two adapters: the first — for connecting to the local network, the second — for interacting with BoardConnect.

Using JMeter in Remote Testing Mode

After we created a network of VMs, we were able to start load testing wifi network using JMeter in remote testing mode. This mode involves one control machine (master) and several servers, to which the JMX file and launch command are sent.

Here:

However, in our case, we used -R {list of IP addresses} in order to ignore the settings in jmeter.properties and instead run testing on the VMs whose addresses we indicated directly in the run command.

We had four JMX files: one for each of our use case scripts. This made it possible to satisfy the testing profile and at the same time support simple adjustments and script launching.

Accordingly, on the master, we ran the following four lines in different command prompts from different copies of the JMeter folder:

jmeter -Djava.rmi.server.hostname = 192.168.2.40 -n -t seven_UC01.jmx -R {IP addresses of 35 virtual machines}
jmeter -Djava.rmi.server.hostname = 192.168.2.40 -n -t seven_UC02.jmx -R {IP addresses of 10 virtual machines}
jmeter -Djava.rmi.server.hostname = 192.168.2.40 -n -t seven_UC03.jmx -R {IP addresses of 3 virtual machines}
jmeter -Djava.rmi.server.hostname = 192.168.2.40 -n -t seven_UC04.jmx -R {IP addresses of 2 virtual machines}

An Excel spreadsheet was used to compile and adjust the long IP addresses. The final line was formed using concatenation, with the result being used as the value of the -R option.

Data Collection and Analysis

For this, we used a classic bundle of wifi load testing tools: JMeter + InfluxDB + Grafana.

As mentioned above, we used four JMX files, each a separate script running in remote testing mode on predefined virtual machines.

Given this implementation, we had to create two Backend Listeners in each JMeter script when configuring data writing to the InfluxDB database. The first was equal to the value of the “Application” field (“General”). The second was equal to the name of the use case.
Generalized data for all scripts was written to “General”, while data for each specific use case was written to the “Application” equal to the name of the use case.

This approach has at least two advantages:

• the ability to monitor the number of active threads for each use case;
• during real-time analysis of the execution of a use case, the volume of data sent to Grafana is much less.

In all scripts, the InfluxDB server’s IP address was identical: it was the master laptop, which did not run any load scripts.

We imported a dashboard as the basis for configuring data monitoring and analysis (Dashboard #5496 Apache JMeter Dashboard using Core InfluxdbBackendListenerClient from Philippe M. Updated on 2018-09-18 17:20:33). However, we changed its basic panels significantly.

Rapid switching between data sets was carried out by selecting the desired value of the “Application” variable at the top of the dashboard.

Using the Backend Listener made it possible to configure the “Active Threads” panel for each use case, which allowed us to select the number of active users.

At the bottom of the dashboard, we placed panels that recorded the volume of transferred data and response time for each transaction. To display this last metric, we used both the original panel and a heatmap panel, which allows us to track the change in a value over time.

What Did the Project Provide to the Customer?

The wifi broadcasts testing conducted by our team showed that BoardConnect is unable to stably provide multimedia content over Wi-fi to 50 plane passengers.

Our results did not agree with the results of the Lufthansa study, since it involved emulation of fewer devices. In other words, we were able to identify a previously undetected problem and report it to the customer. The  decided to send the device for reworking in order to avoid reputational harm.
In addition to the operating system, the VM image contained JMeter 5.0.

What Did the Project Provide to Our Company?

Our team gained valuable experience in stress testing hardware. And now we are prepared for the challenges that may arise in such projects.