SAP Load Testing for X5 Retail: Case Study

There are few companies in the IT systems quality assurance market that have expertise in SAP testing, but PFLB is one of the few that do. Over the years, our specialists accumulated experience in retail that came in handy when helping X5 Retail, a major food retail company worth $16 billion, to support their internal IT systems. We started with SAP platform deployment, established testing processes in the retail giant’s systems, and maintained contact for more than 10 years.

How the Idea of Load Testing of SAP in Retail Was Born

At the beginning of this long journey, the SAP Basis team had had enough resources to monitor SAP system performance issues. However, X5 Retail’s network was growing actively, the systems were becoming more and more complex, and the load on the systems was increasing accordingly; the workload of SAP Basis specialists was, too. Our prospective client decided to have one of the engineers from their full-time testers conduct SAP load testing, and, thus, they solved the problem for a couple of years; performance of SAP systems was under control.

At some point, though, the bus factor came into play; that load engineer had to leave the company, and we had to find a replacement, asap. We came to the rescue by providing competent specialists immediately.

The highlight of the situation was that it was all happening just before the fall, which is the start of the high season — exactly when retailers are inundated with a higher-than-average number of customers and sales. During the high season, the IT systems of such huge enterprises experience enormous loads. They process numerous information flows to allow thousands and thousands of grocery stores and other shops all over the country to operate smoothly.

The Challenges in SAP Load Testing

Meanwhile, we dealt with accidents on production and critical flaws in the performance testing process that were only common in high season. Data center equipment had not been updated for ages and was working at the limit of its capabilities. At the same time, the retail chain continued expanding; things began to take a threatening turn.

Specifics of SAP Performance Testing in Retail

The main scope of work included configuring and testing internal intensive SAP business processes, such as:

We were forced to solve everything based on the information that each store sent to the central unit on a daily basis: stock supplies, balance forecasts, expiry dates, etc. Ideally, the test environment required a highly precise simulation of the production load from thousands of retail stores. We also simulated thousands of internal SAP users and multiple heavy processes involved in various important business tasks in the background. 

Some processes were not heavy in the least, but were crucially important for business; it was important not to overlook those, but to make sure they had been included in the load testing profile. With new processes constantly appearing and old ones constantly changing, we needed to adjust the SAP load testing profile quickly. When an issue from production had to be taken care of, such as those flaws that needed urgent testing, we needed to be even quicker. We also had to share the testing environment with functional and automation testing teams, as well as with SAP consultants. Altogether, the project required considerable attention and careful management of the testing process.

In the course of the project, we updated performance testing methods and developed up-to-date instructions. We also prepared detailed checklists to configure systems in the testing environment for the following situations:

• after copying data from backup;
• to generate test data;
• to prepare it for load testing;
• to stop the tests and collect results upon completion

Since the customer could not use the Grafana toolkit at that time, we implemented Express Report, our own monitoring system developed by PFLB, which was used to collect and compare logs, among other things. We embedded the tools to compare metrics for the main multi-threaded business processes, as well as their child tasks. 

The team also analyzed AWR reports, controlled the accuracy of test data generation, and introduced tracing for calculation-intensive tasks, such as extracting child tasks and analyzing them as much as possible at the runtime. All these actions were further automated as much as possible by our team as the project evolved.

Results

PFLB team built a high-quality agile performance testing process which helped one of the largest retail companies to successfully cope with the crisis of the high season; and we built it concurrently with the hardware upgrade.