Full Regression Cut ~30% on a Bi-Weekly Release Train — Then We Fixed the CI Fight It Started

Volha Shchayuk

Author

Volha Shchayuk

Volha is a seasoned IT researcher and copywriter, passionate about AI, QA, and testing. She turns technicalities into engaging articles, helping you discover and easily grasp the latest IT concepts and trends.

IT researcher

Reviewed by Boris Seleznev

Reviewed by

Boris Seleznev

Boris Seleznev is a seasoned performance engineer with over 10 years of experience in the field. Throughout his career, he has successfully delivered more than 200 load testing projects, both as an engineer and in managerial roles. Currently, Boris serves as the Professional Services Director at PFLB, where he leads a team of 150 skilled performance engineers.

Why manual testing ran out of road

Being a very successful social media application with millions of subscribers, the client's engineering team was - and still is - challenged to ship regular updates across three platforms every two weeks while keeping the quality standards its regular users expect. That is a hard enough problem on its own. What made it unmanageable was the A/B testing culture on top: the product was constantly pushing new ideas, so the *rate of change per release* was increasing drastically rather than holding steady.

Manual testing alone could not cater for a bi-weekly release train under those conditions. Not because the testers were slow, but because a fixed manual regression pass has a fixed duration, and the amount of change it has to cover was growing every sprint. Something had to be automated, and it had to be automated in a way that survived a UI that never stopped moving.

PFLB was brought in to build native UI automation testing with the frameworks Apple and Google had just introduced - XCUITest for iOS and Espresso for Android - rather than a cross-platform wrapper.

What automation delivered

• ~20% of all functional test cases were automated.
• Full regression time was reduced by ~30%.
• Automated client analytics checks found P0/P1 bugs almost every release. These are exactly the cases teams skip when testing by hand, because verifying analytics events manually is punishing work.

The test orchestrator.

The quarantine mechanism for unstable tests.

Then came the CI battles with the developers.

The problem automation created

The client ran a classical pre-merge CI architecture: a GitHub repository holding the app codebase, Jenkins CI, and every pull request and each of its commits triggering a chain of checks against the PR branch - compilation, unit tests, code-style validation. Those checks block the merge: if one fails, the PR does not go into the main development branch until it is addressed.

The challenge was to add automated UI tests for the Android and iOS repositories to that same blocking chain. Three things made that hostile:

• UI tests can be flaky by nature. Flakiness here depended on physical factors - USB connections to devices, internet connectivity.
• Constant UI changes in the app cause UI test failures and demand constant updates to the test code.
• Native frameworks put the tests in the app's own repository. So when a developer's PR broke a test, fixing or excluding that test required another commit or another PR - and every check had to run again. The developer is blocked, and irritated, because they may simply have changed a UI flow that the test reacted to. An unhappy developer campaigns to move UI tests to post-merge, and the value of pre-merge testing quietly disappears.

The technical failure mode and the social failure mode were the same failure mode. Any fix had to break the link between "a test is failing" and "a human is blocked".

Test Orchestrator

The answer was a framework for managing tests in CI, sitting outside the app repository:

• 01 - Toggle without a commit. Any specific automated test can be enabled or disabled without any change in source code. If a test is failing because of a developer change or any other cause, it can be pulled out of the main pool immediately to unblock CI. The orchestrator works with both unit tests and UI tests.
• 02 - Quarantine. A newly added test must pass 10 consecutive times before it is admitted to the CI pool. The same rule applies to updated and fixed tests. This is what removes flakiness from CI rather than merely apologising for it.

Two use cases show how it plays out day to day:

• A test fails because of a UI change. The developer is blocked despite not having broken anything explicitly. The automation team - or the developer themselves - disables the test from the Test Orchestrator's web portal and opens a Jira task for the automation team to update it.
• A test fails because of a real bug the developer introduced. After examining it, the product team decides to fix it in the following sprint. The developer or QA engineer disables the test from the web portal and links the Jira bug for reference. When the defect is fixed, the test's status is updated and it moves to the quarantine job for validation before returning to the pool.

Results

• ~20% of all functional test cases automated across iOS and Android.
• Full regression time down ~30%.
• P0/P1 bugs caught almost every release by automated client analytics checks.
• UI tests survived in the pre-merge chain instead of being exiled to post-merge, because a failing test no longer holds a developer hostage.
• Flakiness handled structurally - 10 consecutive passes as the entry requirement to CI, applied equally to new, updated and fixed tests.
• The project ran against a constantly changing app on tight deadlines, across a multinational team in different time zones, working in Agile alongside the client's own engineers.

Questions this engagement answers

How much of a mobile suite is realistic to automate?

Here, about 20% of all functional test cases - and that 20% took roughly 30% off full regression time, because the cases worth automating first are the long, repetitive ones rather than the numerous ones.

Should UI tests run pre-merge or post-merge?

Pre-merge is worth defending, but only if a failing test can be removed from the pool in seconds without a commit. Without that escape hatch, developers will - reasonably - push to move UI tests post-merge, and the tests stop protecting the main branch.

How do you stop flaky UI tests eroding trust in CI?

Make entry to CI conditional rather than automatic. A new or repaired test sits in quarantine until it passes 10 consecutive times; nothing joins the blocking pool on the strength of one green run.

What does a test orchestrator actually own?

The enable/disable state of every test, independently of the repository the test code lives in, for both unit and UI tests - plus the quarantine job that governs promotion back into the pool.