At a glance
- Solution: Quality Gates Next (QGN) - PFLB's quality-gate methodology, 8 gates a product passes before release, each placed at the end of a significant development stage
- Client: a bank operating in retail, corporate and investment banking
- Goal: move deliveries into production faster without lowering quality - the two things that usually trade against each other
- Outcome: the time needed to install an update dropped 5×
- What was tested: the bank's banking system - UI tests emulating real users, API tests for back-end requests, and load scripts in Apache JMeter
- What was built: automated quality gates inside a Jenkins pipeline that deploys the application on delivery, runs the tests, and either stops the release or advances it
- Test layers: UI tests emulating real users, API tests for back-end requests, scenarios written in Gherkin, load scripts in Apache JMeter
- Infrastructure: browsers orchestrated by Selenoid inside Kubernetes, stubs by Mountebank, images from a local Harbor registry, monitoring on Telegraf, InfluxDB and Grafana
- Reporting: Allure, with a custom template added to show check-level statistics the standard frameworks could not produce
How the gate works
QGN is not a tool the bank had to build: it is an existing, tuned methodology that gets adapted to the client's pipeline, which is why it went in quickly. Its eight gates sit at the end of each significant stage of the development life cycle, so a defect is found at the stage that produced it rather than in acceptance testing.
When developers make a delivery, the Jenkins pipeline deploys the application and runs the automated tests. Depending on the result, the pipeline is interrupted or proceeds to the next stage - in this case manual functional testing. Stages are visible in a Jenkins plugin.
The quality gates pipeline.
Once every stage completes and the results meet the gate criteria, the delivery moves to pre-production for acceptance testing. Nothing advances on assertion; it advances on evidence.
Four problems worth describing, because they recur
The business wanted a metric that meant something. "Number of API tests passed" is misleading when each test can contain a completely different number of checks. Neither TestNG nor JUnit nor RestAssured can collect check-level statistics. The team reworked the test automation framework to gather passed, failed and skipped checks, and customised the Allure report template to present them in a dedicated block.
Containers changed the browser strategy. The application and its environment ran on container technology, which carries less overhead than full virtualisation. So UI automation moved from Selenium Grid to Selenoid - a better fit for a Kubernetes cluster, with more flexible configuration and the ability to restart browser containers quickly.
The servers had no internet access. A local Docker image repository - Harbor Docker Registry - solved it, with SSL-secured access and per-role read and write permissions.
Two systems needed stubs, quickly. Rather than writing stubs from scratch, the team used Mountebank, which was fast to configure into working stubs for both systems.
Building a load profile without production statistics
The bank could not supply statistics on user activity in production - a common situation, and one that stops many load testing projects before they start.
The profile was instead built from PFLB's accumulated experience testing online banking systems for other banks: what the traffic mix in this class of system looks like, which operations dominate, and how they distribute across the day. Experience in a domain is not a soft asset here; it is what makes testing possible when the client's own data is missing.
Results
- Time to install an update fell 5× - the headline outcome, and the reason the bank ran the project at all.
- Bottlenecks were identified in the banking system, with concrete optimisation recommendations handed over.
- The Kubernetes cluster and its accompanying tooling went into the production environment, deployed within the constraints that existed.
- Bank staff were trained on the graphical interface for managing and monitoring the K8s cluster, so the bank operates it without PFLB.
- Release risk was reduced: nothing advances on assertion, only on evidence from the gates.
Questions this engagement answers
What is a quality gate in practice?
A pipeline stage that either stops a delivery or lets it through, based on automated test results - here, deploy, run UI and API tests, evaluate against criteria, then hand over to manual functional testing and finally pre-production acceptance.
Why is "tests passed" a poor release metric?
Because one test may contain one check and another fifty. Counting checks rather than tests required reworking the framework and the report template, and it is the difference between a number that reassures and a number that informs.
Can you load test without production statistics?
Yes, if you have tested comparable systems. The load profile here was reconstructed from experience with other banks' online banking platforms rather than from the client's own data.
Does an air-gapped environment block CI automation?
No - it changes the supply chain. A local registry with controlled access replaces public image pulls, and the pipeline runs unchanged.
Shipping faster without shipping regressions?
Quality gates are only worth building if they can stop a release. That means the tests have to be trustworthy and the metrics have to mean something - both of which are engineering work, not configuration.
