Throughput Fell Below a Quarter After the Migration — the Bank Found Out Before Go-Live, Not After

Sona Hakobyan

Author

Sona Hakobyan

Sona Hakobyan is a Senior Copywriter at PFLB. She writes and edits content for websites, blogs, and internal platforms. Sona participates in cross-functional content planning and production. Her experience includes work on international content teams and B2B communications.

Senior Copywriter

Reviewed by Boris Seleznev

Reviewed by

Boris Seleznev

Boris Seleznev is a seasoned performance engineer with over 10 years of experience in the field. Throughout his career, he has successfully delivered more than 200 load testing projects, both as an engineer and in managerial roles. Currently, Boris serves as the Professional Services Director at PFLB, where he leads a team of 150 skilled performance engineers.

The risk of building your own processing

Moving transaction processing in-house significantly simplifies business processes and reduces costs against using external processing centres. That is the reason banks do it. It is also the reason they do it under commercial pressure, with a date attached.

The bank's own specialists were the ones who raised the flag. In connection with the planned migration, they identified two risks tied to the performance and fault tolerance of the IT infrastructure, TranzWare CMS and TranzWare Online: a critical drop in retail transaction throughput, and interruptions in handling credit and debit card transactions. For a lender operating in more than 2,000 cities with over 31.6 million customers, either one is not a degradation - it is cards declining at points of sale across the country, on the day the new processing goes live.

Rather than accept the risk or argue about it, the bank decided to measure it. The chosen instrument was a benchmark: run the same tests against TranzWare CMS and TranzWare Online before and after the change, and compare. PFLB was hired to conduct the performance testing.

How the profiles were built

PFLB proposed focusing on two system behaviour profiles: "business day" and "day-end closing" - the two states in which a processing system behaves completely differently and fails for different reasons.

Building those profiles was analysis before it was scripting:

• Operational statistics of the live system were analysed to reveal the primary sources of load: business-user transactions and background processes performed on a schedule.
• Integrated communications were analysed to determine the nature of interaction with external systems, which then became the basis for adding further operations to the load profiles.

What had to be built to generate the load

• Load was emulated with LoadRunner, JMeter and Citrix ICA.
• PFLB engineers used an ISO-8583 emulator developed in-house to generate test payment card transactions.
• Emulators of external systems were developed during the project to create additional load over JDBC, SOAP, Oracle AQ and PL/SQL.
• A PL/SQL package and auxiliary LoadRunner scripts were developed to generate test data in the database.

Then the discipline: a series of tests was run on the "old" configuration; the same series was run on the "new" architecture already using the bank's in-house processing. That is what made the two results comparable on a load representative of real operating conditions.

While the tests ran, PFLB specialists monitored performance characteristics at four levels - system resources (CPU, memory, I/O), databases and middleware, applications (code profiling) and business processes (operation response times).

What was found

• TranzWare Online's throughput fell to less than one fourth of its pre-migration level under the in-house processing configuration.
• The bottleneck was the CBA interface responsible for TranzWare Online's communication with one of the banking systems. During testing, a backlog in the CBA interface's message queue degraded performance for all types of transactions - not just the ones passing through that interface.
• Single-threaded processing of one banking system's transactions on the TranzWare Online application server was identified as a further potential problem.
• Several functional bugs were found alongside the performance findings.

Results

• The bank postponed deployment of its in-house processing system by 3 months on the strength of the findings.
• During those three months a developer fixed the bottleneck.
• After all the bugs were eliminated, the load testing was repeated.
• The in-house processing system was successfully introduced.

Questions this engagement answers

How long does a professional load test engagement take?

It is measured in test cycles, not in a single run - and a buyer should budget for the re-test. This engagement was structured as: analyse live operational statistics to build two profiles, build the emulators and data generators needed to drive them, run a full series against the old configuration, run the identical series against the new architecture, analyse, and then repeat the load testing after the developer's fix. The gap between finding the bottleneck and re-testing was three months of remediation on the bank's side. If a vendor quotes you a duration that contains no re-test window, ask what happens when the first run finds something.

What should a benchmark engagement produce that a single load test does not?

A comparison you can act on. "TranzWare Online is slow" would have started an infrastructure argument. "Throughput is below a quarter of the old configuration, and the CBA interface's message queue is backing up" produced a named fix, a developer assigned to it, and a defensible decision to move the go-live date. That is compare testing doing its job.

Should a system migration be tested before or after go-live?

Before, and against the pre-migration baseline. The whole value here came from having measured the old configuration first - without that baseline there is nothing to compare the new one to, and a fourfold throughput regression looks like a normal Monday. This is the core argument for testing a system migration as a benchmark rather than an acceptance check.

What does a vendor need to be able to build, not just run?

Here: an ISO-8583 emulator for card transactions, emulators for external systems over JDBC, SOAP, Oracle AQ and PL/SQL, and a PL/SQL package to generate test data in the database. Off-the-shelf load tools cover none of that. When you evaluate a performance testing vendor, ask what they have built for previous clients when the tooling did not exist.