Go back to all articles

Automated UI Testing

Sep 24, 2019
7 min read
Our client is a social dating app with worldwide recognition and presence in over 100 countries.

Being very successful social media application with millions of subscribers and over a billion swipes per day, client’s engineering team was and still is challenged to release regular software updates across Android, iOS, and Web platform on a bi-weekly basis while keeping high standards of quality to satisfy its regular users.

Rate of changes per release was increasing drastically while product was pushing new ideas via A/B tests thus having only manual functional testing could not cater bi-weekly releases.
In 2016, PFLB was asked to join forces with the in-house engineering team to build what at the time was, native test automation solution using tools for automated ui testing – newly introduced XCUITest library for iOS and Espresso library for Android.

Only manual testing could not cater bi-weekly releases
because rate of changes was increasing drastically while new ideas pushing via A/B tests.
Have a Project in Mind?​
We have been working on performance testing projects since 2008.
Drop us a line to find out what our team can do for you.
Get a quote You’ll hear back from our tech account manager in one day if not sooner

As a Result:

  • ~20% of all functional test cases were automated
  • full regression time was reduced by ~30%
  • ~automated client analytics were finding P0/P1 bugs almost every release ( extremely time consuming to test manually)

Success seemed to be achieved until our team start running into CI battles with devs.
Let’s examine a typical CI architecture:

CI with pre-merge tests ( Classical Case):

  • GitHub repository with app codebase
  • Jenkins CI
  • Each PR and its consecutive commit is triggering a chain of checks against PR branch which include but not limiting to code compilation, unit tests, code style validation
  • Checks described above are blocking PR merge — if one of the listed above checks fail, PR won’t be merged into main developing branch until the issue is addressed.

Challenge:

Add UI Automated tests for Android/IOS repositories to run along with other checks – compilation, unit tests, lint on each Pull Request

Problems:

  • UITests due to its nature might be flaky. Flakiness could depend on many factors – USB connections on devices, internet connection
  • Constant UI changes in the app lead to UITest failures and require constant updates in the tests code
  • Since we are using native testing frameworks (Espresso, XCTest) for writing fast and reliable iOS and Android UI Tests, these tests reside in the same repository along with app’s code. Therefore when a developer makes PR and brakes one or more of these tests, the change in test or exclusion of broken test would require a commit or another PR. As a result, all checks would need to be run once again which is time consuming. Not only developer is blocked but also irritated since he might of change UI flow which made UI Test to react to such change ( false positive ) Unhappy and irritated developer will obviously opposed to run UI Tests in pre-merge manner and would fight for move UI Test to post merge execution.

Test Orchestrator

test orchestrator

Solution:

Test Orchestrator – a framework for managing tests in CI.

  • 01
    The ability to enable / disable specific automation tests without any change in source code. If test is failing due to a developer change or other cause, one could be quickly disabled from main pool of tests to unblock CI. The orchestrator works both with Unit and UiTests.
  • 02
    Quarantine is another feature of Test Orchestrator. A newly added test would have to pass 10 consecutive times before being added to the CI pool of tests. The same rule applies for updated/fixed tests. Such feature eliminates flakiness in CI.

Use Cases:

01

Test fails in CI due to UI changes. Developer is blocked, although he did not brake test explicitly. Automation team or developer himself would disable test from Test Orchestrator’s web portal while opening a new Jira task for automation team to address test modification.

Quarantine

case study quarantine

02

Test failed due to the actual bug which developer introduced in a Pull Request. After examining a new bug, the product team decided to fix it in the following sprint. Dev or QA engineer would perform the following actions:

  • Developer or QA Engineer disables test from Test Orchestrator web portal and link Jira bug for reference.
  • When the defect is fixed, the status of the test get updated and it moves to quarantine job for validation.

Conclusion

Looking back at all the work performed on the project, we can definitely call it a success. Despite having to deal with a constantly changing app and working on very tight deadlines, we managed not only successfully write and run multiple test cases, but also recognize the need and importance of changes in the process and subsequently come up with a solution that eliminates many issues associated with test automation.

All this helped our team recognize the importance of detailed planning, especially at the early stages of a project. In addition, working on a multinational team across different time zones and especially around tight deadlines gave us all an opportunity to work using Agile methodology and focus on the quality of the end product for our client. In doing all that, we not only helped deliver a better product or improved a few metrics, but our team also managed to learn and grow as well.

Table of contents
Let us know about your needs
We can provide multiple performance testing services and a lot more than that if the situation needs a far more complex approach.
Get a quote You’ll hear back from our tech account manager in one day if not sooner

Related insights in case studies

Explore what we’ve learned from these experiences
7 min read

From Hundreds to Thousands: Scaling MEFA Pathway Software for Mass Student Registration

how load testing helped e learning services provider preview
Jul 12, 2024

FolderWave, Inc. is a leading digital services provider in the Massachusetts e-learning sector. It aids millions of students in researching and planning a job-oriented education. The company delivers IT solutions, platforms, and services in partnership with notable non-profit organizations like MEFA Pathway and College Board, which connect a vast network of colleges, schools, and universities […]

8 min read

How Load Testing Helped Texans Survive Power Outages During a Storm

how pflb helped texans survive power outages during a storm preview
Jun 13, 2024

One of the largest electric distribution company in Texas that serves over 100,000 people, ensuring an uninterrupted power supply. To maintain service quality during weather-related power outages, it must comply with the Texas Reliability Entity, which ensures energy providers’ reliability and security. To provide clients with reliable service, the company deployed Storm Center and the […]

4 min read

Tynor Prepared the New Website for High Sales in Four Days

tynor prepared the new website for high sales in four days preview
Dec 12, 2022

Tynor Orthotics is India’s largest manufacturer and exporter of orthopedic and fracture aids established in the 90s to deliver quality healthcare products. Committed to a significant expansion in the next three years, Tynor crafted a new e-commerce website focused on excellent customer experience to support this growth. To be confident at launch, the engineering team of Tynor decided to run pre-go-live stress testing for the website. The tight deadline felt challenging, the customer was relieved to hear we provide a quick load testing solution. Quick Load Testing solution includes a four-day load testing project performed by engineers of PFLB and a 1-month subscription to the innovative load testing PFLB platform.

5 min read

Bank Increases Load Capacity by 450% to Deal with Business Growth

bank increases load capacity by 450 to deal with business growth preview
Oct 3, 2022

Our client's bank was absorbing other banks, and the number of individual clients was growing. The system was not ready for expansion or integration. The owners started to suspect bottlenecks when problems with paying salaries to corporate clients’ employees arose. As a result, in the next pay period, the load on the system increased dramatically, and the system got overloaded. People did not get their salaries in time, as the system crashed.

  • Be the first one to know

    We’ll send you a monthly e-mail with all the useful insights that we will have found and analyzed

  • People love to read

    Explore the most popular articles we’ve written so far