Automated UI Testing

Being very successful social media application with millions of subscribers and over a billion swipes per day, client’s engineering team was and still is challenged to release regular software updates across Android, iOS, and Web platform on a bi-weekly basis while keeping high standards of quality to satisfy its regular users.

Rate of changes per release was increasing drastically while product was pushing new ideas via A/B tests thus having only manual functional testing could not cater bi-weekly releases.
In 2016, PFLB was asked to join forces with the in-house engineering team to build what at the time was, native test automation solution using tools for automated ui testing – newly introduced XCUITest library for iOS and Espresso library for Android.

As a Result:

Success seemed to be achieved until our team start running into CI battles with devs.
Let’s examine a typical CI architecture:

CI with pre-merge tests ( Classical Case):

Challenge:

Add UI Automated tests for Android/IOS repositories to run along with other checks – compilation, unit tests, lint on each Pull Request

Problems:

Test Orchestrator

Solution:

Test Orchestrator – a framework for managing tests in CI.

Use Cases:

01

Test fails in CI due to UI changes. Developer is blocked, although he did not brake test explicitly. Automation team or developer himself would disable test from Test Orchestrator’s web portal while opening a new Jira task for automation team to address test modification.

Quarantine

02

Test failed due to the actual bug which developer introduced in a Pull Request. After examining a new bug, the product team decided to fix it in the following sprint. Dev or QA engineer would perform the following actions:

Conclusion

Looking back at all the work performed on the project, we can definitely call it a success. Despite having to deal with a constantly changing app and working on very tight deadlines, we managed not only successfully write and run multiple test cases, but also recognize the need and importance of changes in the process and subsequently come up with a solution that eliminates many issues associated with test automation.

All this helped our team recognize the importance of detailed planning, especially at the early stages of a project. In addition, working on a multinational team across different time zones and especially around tight deadlines gave us all an opportunity to work using Agile methodology and focus on the quality of the end product for our client. In doing all that, we not only helped deliver a better product or improved a few metrics, but our team also managed to learn and grow as well.