Top 1,000 world’s
The bank’s management made a strategic decision to develop their remote banking services in-house and to try and bring the system to a higher level in the ratings. Three platforms (online web banking, iOS and Android apps) were subject to renovation.
The situation was aggravated by the fact that at that moment the banking group was undergoing integration processes, and the load on the system increased in step-ups. It became clear that the system was not ready for such a load: there was a system crash on a peak workload day, a payday.
The goal was to optimize the system to a state of stable operation with a certain SLA on accessibility.
Since the remote banking services system operates in an environment of integration with related systems (accounting, active operations, sales channels, terminal networks, etc.), to identify the problem area, specialists need to evaluate the whole complex of these components.
In that case it was necessary to find out which system malfunctioned, what particular problems disrupted the work of the client Internet banking.
In order to deal more quickly with the critical task of finding the problem, the CIO and CTO of the bank decided to outsource the task.
The first step of the PFLB team was to issue several recommendations that helped the bank survive the next banking day without system failures.
Our best engineers immediately engaged in an investigation of the system overload. We found 200+ system errors that impaired both the system’s performance and its stability.
For example, when the application server generated new database connections, they bounced as “timed out”.
After having increased database session timeout from 5 seconds to 30, we made sure there will be no system crashes in the next payroll period by conducting performance tests right before it. Thus, we located one of the most critical bottlenecks and saved the immediate future of the bank’s clients. We also ran stress testing to be able to predict the behavior of the system from the next peak day and to see if it will stand the load.
At the same time, we actively collaborated with the bank’s staff to form a pool of recommendations needed to be implemented to stabilize the system. We helped to prioritize these tasks and began the process of removing bottlenecks. Overall, it took us less than 2 weeks.
The bank’s remote banking services system was under active development, which could have caused changes in system performance during the release of new versions. The client made a strategic decision to build and implement a performance testing process as a routine part of their business process to minimize the risks. We were eager to help, and that is how the second stage of the project started.
Speaking strategically, our client wanted the system to have a load reserve that would ensure system stability with further user number increases. We worked on this stage of the project for half a year, because the bank system was expanding and required a full-fledged performance testing process to be designed, introduced and implemented.
During the second phase of the project, which lasted for six months, the following results were achieved:
The client’s system work and maintenance were made more transparent and predictable in terms of scaling, launching a new module and development.
A detailed understanding of the performance reserve availability and volume was reached by evaluating the components of the RBS system with a quick-response service mockup.
A pool of non-optimal queries to the database that were difficult to find even in Oracle-statistics were identified and optimized.
Duplicated queries that affected the business logic of the system were reorganized.
Comparative load testing was incorporated into the in-house release policy to highlight changes in performance.
PFLB helped us with the most complex technical part. This contribution was priceless. Our system can now handle a load of 450% of a peak load day profile.
CIO, the client
The system now has load capacity of up to 450% of the production environment. Now the bank can freely perform critical business processes without a threat of facing a system crash.
During the project, the bank’s in-house team accumulated enough skills to be able to identify the causes and locations of bottlenecks, and thus, performance testing and localizing performance issues is since then the IT department’s function.
Since the bank’s policy was initially aimed at in-house dev, embedding performance testing into the development process was a valuable experience that increased the maturity of the bank’s development culture.