At a glance
- Client: one of the largest retail chains selling electronics and household appliances, running a full omnichannel model - unified assortment, prices and services whether a customer buys in store or online
- Systems under test, as one estate: SAP PI/PO (integration bus), SAP ERP (procurement, sales, distribution, accounting, tax and management reporting, balances, master data), SAP POS DM (store data management), SAP TM (transport planning for both fixed-route mainline haulage and undefined city routes)
- The question: did performance meet requirements before and after the transformation, did anything get worse, what needs optimising, and did the optimisation itself degrade anything
- How load was generated: user load through the SAP GUI protocol; PI load by sending XML over SOAP and uploading files to FTP, matching how data really arrives
- Tools: LoadRunner and Apache JMeter
Why all 4 systems had to be tested together
In an omnichannel retailer these systems are not neighbours, they are one mechanism: a sale in a store moves through POS data management, into ERP for accounting and stock, across the integration bus, and out to transport planning for replenishment. Testing any one of them alone answers a question nobody is asking.
The transformation made this sharper. The client needed to know not just whether the estate was fast enough, but whether it had become slower than before - which requires the same measurement, run twice, against the same load.
How the scope was chosen
The tested operations were selected on the customer's expert judgement, against three explicit criteria:
- business-critical, performed daily, and affecting sales or logistics in the system
- placing significant load on the system
- expected to carry increased load as a result of the transformation
Target load figures and SLAs came from interviews with the functional consultants - the people who know what the numbers are supposed to be - rather than from documentation.
How the load was built
Tests emulated three things at once, because that is how the estate actually runs: user traffic, traffic from external systems, and background jobs launched on a schedule closely matching production.
Each system was loaded the way it is loaded in reality:
- SAP ERP - user load through the SAP GUI protocol, with scheduled tasks mirroring production timing, measured as performance metrics per operation.
- SAP PI - the main load is data streams from other systems, so it was emulated by sending XML files over SOAP and uploading files to an FTP server.
Throughput per virtual user depends on the scenario, on system response times and on the delay between iterations - so total intensity was tracked across all virtual users rather than assumed from the user count.
What the testing found
Every item below is a defect the transformation would otherwise have carried into production:
- A database locking problem in budget control. The customer optimised the program on the back of it.
- An overflow problem in ActiveMQ, and, with PFLB's help, the SAP PI system debugged and tuned.
- A critical error in a TSA patch that had nearly reached production. This is the one that pays for a testing programme: the patch was days from shipping.
- A missing table of product comparison materials and receipt indices. Without that optimisation the load caused SAP POS DM to stop working - found on a test bench rather than in stores.
Questions this engagement answers
How do you prove a transformation did not make things slower?
Measure the same operations under the same load before and after, in a test environment that did not change between the runs. Without the "before" run there is no answer, only opinions.
Should connected SAP systems be tested separately?
Not when the business process crosses them. Here a single retail flow touches POS DM, ERP, PI/PO and TM, so the estate was loaded as one system.
How do you decide what to include?
Business-critical daily operations that affect sales or logistics, operations that load the system heavily, and operations the transformation is expected to make busier. Everything else is noise in a comparison.
Where do the SLA numbers come from?
From the functional consultants. Documented targets age; the people configuring the system know what the business actually expects.
Transforming a platform your business runs on?
The risk in a transformation is rarely the new capability. It is the operation that used to take four seconds and now takes nine, discovered by a store manager rather than by a test.
