Constant IT development is the new black. Many enterprises have already started using software designed to automate corporate processes with the help of AI. We share our favorite case of testing intelligent robotic process automation (RPA): Blue Prism, a platform providing a digital workforce that can perform any task of an operation staff member working on their PC. Whether they need to order stationery supplies or calculate salaries, Blue Prism virtual employees easily emulate business processes as real users would, regardless of the software used. Moreover, Blue Prism RPA is equipped with AI capabilities that make a lot of sense for the banking sector.
1. RPA and AI for financial services
1.1. What is robotic process automation in banking?
1.2. How is RPA used in banking?
1.3. What is intelligent automation?
2. General performance testing tasks that the bank faced
3. Testing and scaling challenges and other RPA issues in the case
4. Process automation as a testing solution
4.1. Performance testing method
4.2. Connection between the app server and virtual users
4.3. Connecting an app server to the database
4.4. Features of the testing process
4.5. What we developed
5. Results of testing intelligent automation software for banking
Banking and financial services are the top sectors where RPA solutions are implemented. RPA helps cut costs, reduce response times, and—most importantly—free staff from routine tasks so that they can focus on more creative and innovative tasks. It has become even more popular due to the COVID-19 pandemic, as one of the problems faced by businesses during numerous lockdowns has been how to increase accessibility to on-site systems without crowding their offices.
Financial organizations mainly use RPA for document automation in financial products, fund transfers, customer verification and requests, and data processing and verification. The list makes it obvious why RPA is so popular with banks.
Intelligent RPA is another step up toward automation. It unifies human and digital resources since it is equipped with artificial intelligence (AI)—and AI is a quick learner. Blue Prism RPA offers such solutions for the financial sector. However, although the company claims that its digital workforce operates 24/7, never makes mistakes, and has a 100% audit trail to account for the work they’ve done, our client faced several challenges and asked us to help.
The client is the biggest bank in Europe, with 98 million individual clients, 2.7 million corporate clients, and 278,000 employees. Although they had been using Blue Prism RPA for a while, they decided to perform the first ever performance testing of their software, so it was all new to them. PFLB played the guiding part, making the client aware of all the details of performance testing and running the service. After the project, the client’s business indicators increased notably, so they returned to us with another service request.
Maximum number of digital workers that could be employed and processes that could be executed by Blue Prism RPA within the current system configuration.
Correlation between maximum productivity and system servers.
Cases that could lead to system failures.
The client gave us the results of a load testing of the system that they performed using their own capacities. Unfortunately, they were of no use at all.
First, the data was out of date, as processes tend to update on a regular basis. The test results were valid only at the moment of generation. The connection of new regional banks and processes to the system changes the load testing profile, which can critically affect system performance.
Second, test indicators directly depend on the complexity of the processes executed, and the testing results cannot be solid without a specific load. For instance, if a process creates and consumes queue elements saved in a database, it generates a much heavier load on application servers and the database, as opposed to supporting processes that work autonomously.
Another problem was that the client had prepaid 1,000 digital workers, while the system had a maximum capacity of 500. When around 400 Blue Prism virtual employees were active simultaneously, the system load reached 100%, and it would no longer respond.
Obviously, our client’s experience shows that it is recommended that you check your system capacity before buying software. Professional advice is more useful than the seller’s promise that your system will endure twice its maximum load.
We also had to decide how to scale, either by increasing the system servers’ capabilities or by scaling the architecture—that is, by deploying additional instances of the system.
Want to learn more about our performance testing services?
Find out what’s included and how
to start working with us
The client’s system comprised one database (DB) server, 14 app servers, one load balancer, and 700 Blue Prism robots. We used five servers for testing, including load generators. We also considered creating an emulator of virtual automated workers that would perform tasks instead of 700 digital workers. The load was supposed to be triggered upon receiving a signal sent by the virtual worker emulator. However, after analyzing the situation, we opted for a better solution. We decided to use JMeter, as it replaced digital workers with virtual users, or vusers, in a 1:1 ratio. After all, the only task of the virtual workers was to send requests to the app server. As it is very easy to increase the number of vusers or threads in JMeter, we managed to scale the load by changing a single setting.
At the beginning, we had no information on communication between virtual users and the server, so it was a black box. We decided to record the traffic. Through our research, we found out that the system was using a .NET Remoting secure protocol. To simplify the process, we changed the performance testing mode to insecure and recorded traffic with Wireshark since .NET Remoting secure uses TCP requests. Basically, TCP requests are structured around the connection of the address, the quantity of bytes being sent, the body as a byte string, and the sign of request end. The traffic analyzer output was a hex string of the sent bytes.
At first, we used a self-written converter for parameterization. Later, we also read and controlled hex stings ourselves. In the end, the load script consisted of a package with TCP samplers, where the body was a hex string. Additionally, the script included preprocessors for parameterization and postprocessors for correlation.
Surprisingly, every TCP request with a valid structure received code 200 as a response, so the text content did not matter. To verify the response body, we used JMeter Assertions.
In the end, we created a tool capable of emulating the load generated by real digital workers and sending it to the Blue Prism RPA servers.
We were provided with one app server for the testing process, although in reality there were 14 in the system. This was not a problem. Since the SQL request is transferred via protocol TDS, we recorded and replicated it. One JMeter vuser represented one app server by emulating the necessary traffic.
We reached a 100% load in the testing profile in the first run. However, the system resources seemed much less loaded than in real life. We identified several reasons for this:
Processes are stored in the database as XML files that are sent to the digital worker by the app server at launch. When configuring the load profile, we took every file size into account. However, processes can be interconnected, in which case the vuser receives a package of XML-dependent processes at launch. This was substantially increasing the load on the network and the central processor of the database server.
Process developers had stored local variables in a table with global variables in the database. Obviously, every vuser executing a process requested all the data from the table several times.
The executing process receives tasks using a procedure that is stored in the database. It turned out that of 233 procedure requests, new tasks were only received in 2% of the cases. Reproduction of this process resulted in a 20% increase in central processor utilization on the DB server.
A large number of system administrators using the UI created additional loads on the servers.
We estimated that 10% of the central processor’s load went to encoding messages.
A script that emulates a Blue Prism digital workforce. This helped us determine the maximum performance of one Blue Prism server.
A script that emulates a Blue Prism server. This allowed us to recreate the necessary load on the database server while being short of Blue Prism servers.
Basic hardware monitoring for the performance testing profile (Telegraph, Influx, Grafana).
Self-written monitoring of operation response time based on constantly updated JMeter logs.
While conducting performance testing of intelligent automation software Blue Prism RPA for our client, a major financial service provider, we achieved the following results: