At a glance
- Client: a big national bank running Robotic Process Automation - an automated system built to fully substitute an operational bank employee working at a PC, driving mouse and keyboard input the way a person does rather than sending hard-coded requests
- Production estate: 1 database server, 14 application servers, 1 balancer and 700 virtual automated workspaces (VAWs)
- The brief: find the maximum number of VAWs and processes the current technical tool complex supports, build a sizing table, locate the bottlenecks, and reproduce production incidents - including a crash when the queue exceeds 700 elements
- Approach: the customer's proposed 700-VAW robot emulator was rejected - JMeter substitutes the robots directly, one virtual user per robot; the black-box robot-to-application-server interface was captured with Wireshark and rebuilt as TCP samplers carrying hex bodies, with parametrisation preprocessors and correlation postprocessors
- Finding - the polling loop: only 2% of process-run attempts actually succeeded; the retry pattern produced 233 `getnextcase` calls per second of which just 4 were useful, consuming 20% of the database server CPU
- Finding - the global variable table: local variables saved to a global table in the database, then re-read repeatedly by every process for every robot, consumed a further 10% of database server CPU
- Measured ceiling: 109% of the load testing profile, at 140% CPU and network restriction - while the network already carried 9 of 10 Gbps at 100% of production load
- Recommendation: more Blue Prism instances, not a wider network channel
A workforce that has to be emulated, not scripted
Robotic process automation in a bank of this size is not a pilot. The system is designed to fully substitute any operational employee working at a PC, and it is deliberately indifferent to the task - it can calculate a salary or order notebooks and pens from an online shop with the same machinery. The design point is that the robot uses entry signals, mouse and keyboard, exactly as a real user would, rather than firing hard-coded requests at a known endpoint.
That design is what makes the platform valuable and what makes it awkward to test. The customer planned to increase the number of virtual automated workspaces running robots while simultaneously increasing the number of executing processes - two variables moving at once, on an estate of 1 database server, 14 application servers, a balancer and 700 VAWs. PFLB was asked to establish four things: the maximum number of VAWs and processes the current technical tool complex could carry; a sizing table linking maximum performance to the server complex; the bottlenecks reducing performance; and, if possible, a reproduction of production incidents such as the system crashing when more than 700 elements sat in the queue.
The customer's own proposal was to build a robot emulator that would imitate 700 VAWs from a single server, with pseudo-robots announcing that they had started work in order to load the system. After some thought, PFLB rejected it. JMeter can substitute the robots. One virtual user stands for one robot; from the application server's point of view the robot is just a source of requests, and what it believes it is busy with is irrelevant to the load.
Turning a black box into a load script
The interface between the robots and the application server was opaque. The solution was traffic logging.
- Production uses the .NET Remoting secure protocol; for simplicity the load testing environment was switched to the insecure variant.
- Traffic was captured with Wireshark, since .NET Remoting sends TCP requests. A request has the structure [address – number of bytes to transfer – body of the byte line – query-ending symbol], and the sniffer returned a hex line of the transferred bytes.
- A self-written converter handled parametrisation at first; later the team simply read the hex lines directly.
- The finished load script was a package of TCP samplers with hex bodies, parametrisation preprocessors, and postprocessors for correlation.
Two discoveries shaped the script. First, every undamaged TCP request comes back as a 200 response - including responses whose body says, in effect, "your query is useless". Response validation had to move into JMeter assertions on the body rather than relying on status. Second, on structure: one script with branches inside it beat 20 separate scripts for the different types, volumes and intensities in the load profile.
The second interface had a different problem: there was one Blue Prism server available against 14 in production. That turned out not to matter. SQL requests travel over the TDS protocol, so they could be logged and reproduced the same way - a JMeter user standing in for part of the Blue Prism server, issuing the requests a robot would have caused.
What came out of this stage:
- a script that emulates robot activity, giving the maximum performance of a single Blue Prism server;
- a script that emulates the Blue Prism server itself, reproducing load on a Blue Prism server without needing access to enough of them;
- base hardware monitoring of the load testing environment using Telegraf, InfluxDB and Grafana;
- self-written runtime monitoring built on continuously updated JMeter logs.
Why 100% of the profile wasn't loading the hardware
The first maximum-performance search test produced a contradiction: the profile reached 100%, and the hardware was nowhere near as loaded as it was in production. Five causes accounted for the gap.
- Processes are interconnected. Paying government taxes, for example, requires activating around 200 processes. Because that is expensive, each robot starting a process from the package wants assurance that resources exist - so it packs the whole package of XML processes. Network traffic volume multiplies, and so does processor usage.
- The global variable table. Process developers save local variables into a global variable table in the database, and every process, for every robot, then requests all of that table's data several times over. Cost: 10% of the database server CPU.
- Almost every process launch looked successful. Only 2% were. The planner - the subsystem module that optimises the number of VAWs - correctly works out that a process needs 100 VAWs. But starting a robot is an expensive operation, so it happens only once per period. The 100 VAWs start, process the queue elements, and still have time left, at which point the loop begins: take an element – is the element taken? – no – take the element. That produces 233 iterations of the `getnextcase` procedure per second, of which 4 are useful, and consumes 20% of the database server CPU.
- Neighbouring subsystems. The order in which load testing components run matters, and the web interface's influence on the database - around 5% of CPU - had not been accounted for when configuring hardware resource utilisation for the test environment.
- The protocol change itself. Running with encryption switched off was budgeted at 10% of CPU load, a rule of thumb taken from experienced colleagues rather than measured.
Finding the ceiling
At 100% of production load the estate already consumes almost all available bandwidth - 9 of 10 Gbps - plus a significant share of other hardware resources. Hardware utilisation grows linearly as robots are added, and linearly again as processes or their load increase. Multiplying two linear functions gives a quadratic, and a quadratic gives you maximum performance in five minutes in Excel.
The maths turned out not to help much, so the test was rerun with both variables raised - more robots and more processes together, with the XML file sizes reduced slightly to get there. The measured maximum was 109% of the load testing profile, at 140% CPU load and under network restriction - even though 100% of the profile corresponds to only 40% database server CPU utilisation.
The conclusion follows directly from those two numbers sitting side by side: widening the network channel - buying the "golden" cables - will not raise performance. The limit is not where the bandwidth chart makes it look.
Results
- Recommendations compiled for optimising the script logic the robots run - the global variable table pattern and the `getnextcase` retry loop being the two clearest targets, at 10% and 20% of database server CPU respectively.
- A sizing table for the technical tool complex across different load levels, which was the deliverable the expansion decision depended on.
- A recommendation to scale by adding Blue Prism instances, not by widening the network - supported by a measured ceiling of 109% of profile at 140% CPU while database server CPU sat at 40%.
- Two reusable emulation assets: a robot-activity script and a Blue Prism server script, the second of which removed the need for access to 14 servers.
Questions this engagement answers
How do you load test robots that drive a mouse and keyboard?
You do not drive mice and keyboards. From the application server's perspective a robot is a source of TCP requests, so one JMeter virtual user stands in for one robot - which is why the customer's proposed 700-VAW emulator was unnecessary.
What if the protocol between the components is undocumented?
Capture it. Here .NET Remoting traffic was recorded with Wireshark, decoded from hex, and rebuilt as TCP samplers with parametrisation and correlation - the same method then applied over TDS for the database interface.
What do you do when the test environment has 1 server and production has 14?
Emulate the missing tier. A JMeter script standing in for the Blue Prism server reproduced the database-side load a full estate would generate, so the sizing question could be answered without the sizing hardware.
Why did 100% of the load profile not load the hardware like production does?
Because the profile modelled requests, and production behaviour included things the profile did not: XML process packages multiplying traffic, a global variable table re-read by every process for every robot, a retry loop wasting 229 of every 233 database calls, and a web interface consuming database CPU alongside the robots.
Will more bandwidth fix an RPA platform that is saturating the network?
Not here. The network was carrying 9 of 10 Gbps at production load, but the ceiling arrived at 109% of profile with CPU at 140% and database CPU at 40% - so the recommendation was additional Blue Prism instances rather than a wider pipe.
Expanding a robot fleet on numbers nobody has measured?
Adding robots and adding processes are two different multipliers on the same infrastructure, and their product is not linear. A sizing table built from a measured ceiling is what turns an expansion plan into a budget you can defend.