FIS Profile Banking System Load Testing

The IT infrastructure of any organization requires some key elements in order to function. The most important system for our client, a large state-owned bank, is FIS Profile.

This system was created by Fidelity Information Services, Inc. (or FIS), an American organization included in the Fortune 500 rundown that offers a wide range of financial products and services.

FIS is headquartered in Jacksonville, Florida, and employs about 55,000 people worldwide. Following the completion of its recent — and by far its largest — acquisition of Worldpay, Inc. for $35 billion in the third quarter of 2019, FIS became the largest processing and payments company in the world.

FIS has a product portfolio for the financial services sector that encompasses both retail and investment banking. This includes Profile, a financial application based on the open-source G.T.M, a transaction processing database engine powered by FIS. This product is the fastest NoSQL solution utilized in many banks around the world. Profile serves millions of operations per minute and has been running 24/7 for decades.

The Profile system is loaded with user actions, batch operations as per banking regulations, and Visa and Mastercard transaction processing. There are at least 11 releases per year for banking application testing. The PFLB team has been conducting performance testing of each of these system releases every year for 7 years, with each load test profile including 50 operations. Each release starts with standard steps:

In addition, each test run includes a series of standard launches of the required software, hardware resource utilization checks, and the termination of the test (stopping the banking software testing, collecting the required post-test data, etc.).

Plus, it is essential to run performance tests when updates are released or changes are made to products and hardware.

What Are the Limitations of Fidelity Profile Testing?

All these elements are critical to the issue of developing experience and an understanding of the working structure of Profile. Maintaining the acquired competence and increasing the knowledge of Profile is one of the main objectives of PFLB.

Banking software testing is a necessity not only for the client, one of the largest banks in the world, but also for its millions of customers. Without it, the system may not work properly after an update. This could put millions of people at risk, as the bank serves hundreds of government organizations and large corporations. Without the stable and high-quality work of the client’s IT infrastructure, employees could go without being paid or not receive necessary cash payments on time.

Task

Our client pays special attention to the stability of the Profile system, which serves more than 65 million customers, 55 million contracts, 9 million loan agreements, and 45 million cards.

Fidelity Profile is not the most user-friendly system based on the IBM POWER architecture. The project is a large non-relational Database Management System (DBMS), and it consists of many individual executable objects (scripts, procedures, batches, etc.) that are called on-demand. The system works based on the principle of a black box: there is an input and an output. Among other challenges, the following points are worth mentioning:

PFLB has conducted banking application testing for each of the system releases. On average, a release contains about ten changes to the functionality of the system, each of which requires detailed elaboration and specific testing.

As part of this task, our testing team had to take these steps:

Solution

We decided to make the testing process more agile by implementing the following:

Once the load testing tools are developed, we assemble the load configuration based on the analysis of statistics of queries served by Profile.

Each time, our team conducts 24-hour testing for system reliability for more than 50 business cases, including account management operations, loans, payroll projects, card processing, and customer data management.

To test all these business cases, the experts at PFLB developed a structure that provides a quick record of changes in the requests caused by system updates. The framework also allows the automation of the connecting of test data.

PFLB engineers developed an emulator to simulate the load caused by card operations using the ISO 8583 protocol. External system emulators have also been developed as a part of the project. They generate an additional load using Java, JDBC, and SOAP technologies.

To emulate the load, we use several tools: Apache JMeter, MicroFocus Performance Center, and unique PFLB products.

At the beginning of each release, we generate data to increase the size of the database. This is done to perform more accurate tests, taking into account the growth and development of the company in the future. For each release, additional objects are created in the database: 160,000 clients, 160,000 accounts, 240,000 loans, and 560,000 cards.

Our current achievements:

• our team conducts performance testing of 11 system releases each year;
• the load testing is automated;
• the system is tested under a load of approximately 6 million operations per hour;
• the client receives recommendations on how to optimize the Profile system settings to function with maximum performance after each testing iteration.

Profile Automation

The Fidelity Profile system is a large project, and for its load testing, it is essential to automate processes. For this reason, our engineers have automated the following actions:

Test plans in this project have been created according to a specific algorithm:

Also, it is important to always review the collected data, because the amount of data needed for the same type of tests is constantly changing, depending on the load profile obtained. Moreover, an incorrect manual calculation can lead to the loss of test results and, more importantly, the amount of time for its implementation and the re-generation of data.

Benefits for the Client

The project has enabled the client to implement a process of regular comparative performance testing of different configurations of the system with a real industrial load of about 6 million operations per hour.

Due to the load-testing project, the bank has minimized the risk of failures in a production environment where the system is unavailable and customers cannot perform their operations. Effective stress testing from our company has allowed the client to save money and avoid system downtime.

Due to automation, we have obtained the following results:

For seven years, PFLB staff have been testing the system’s reliability in 24/7 mode, solving problems in the production environment, and helping to detect performance degradation since the release time.

For PFLB, this project is also valuable in terms of the opportunity to develop skills in dealing with a complex and original banking system.