Go back to all articles

Automated Testing Strategy: How to Build & Examples

May 21, 2019
9 min read

Automation is a crucial part of all modern development processes. Be it Waterfall, Agile, or any other methodology used, they all play a part in continuous development, integration, testing, and more. For developers, this means 10, 20, 50 releases per year. Pushing out so many updates and builds with no automation would be a pain in the neck.

Automated testing is a way out for all companies who are already using or are looking forward to implementing Agile into their workflows. On top of that, the quality of testing and the number of detected errors is higher when the workflow is automated.

How to approach automated testing? What should a developer do to create an efficient ecosystem for running scripts? In this post, we’ll touch on automated strategies in detail.

What Is Automated Testing?

By definition
Automated testing is the process of conducting a set of tests that run on their own. Unlike manual tests, reliant on the workforce, the automated ones depend on tools for management and reporting.

This approach to testing has become extremely popular as companies start to transition towards DevOps. Opposite to a traditional development environment, companies have started performing testing not at the end of the cycle but early on.

In order to be able to run dozens of small tests simultaneously without creating a bottleneck for developers and stalling the process, QA professionals have picked up on automated testing. Essentially, it allows them to:

  • Ensure a steady coverage of tests all the way down the development pipeline;
  • Create a set of tests that could be repeatedly re-used;
  • Deploy tests 24/7 and speed up product delivery to the market;
  • Reduce the cost of end-to-end testing by catching bugs early on;
  • Create an environment for parallel testing.

Automated Testing Strategy

In general, an automated testing strategy includes the overall approach to testing and reporting. All the processes are outlined and planned along with the expectations for the testing cycle.

When working on an automation strategy for testing sessions, testers will start from outlining the following points:

  • Test environment
    A good testing environment should, first and foremost, be fully controlled and provide a developer with the freedom to modify all the necessary data. There are cases when running a test requires several environments to work in tandem – in this case, a QA professional has to ensure both systems are compatible with one another.
  • Testing approach
    This is the part where a tester defines the scope of the test, creates a list of activities, and assigns team members to oversee each cluster. A standard automated testing pipeline would normally look as follows: Initiation, Automation Test Planning, Execution, Reporting, Close-out.
  • Testing tools
    A tester has to choose and outline the range of used tools as well as their scope.

While the structure of the strategy is highly depending on the structure of the company, the number of people involved, and the methodology a business has adopted, the number of steps and actions to take remains fairly steady.

Here’s an example of automated testing strategy you can use for executing scripts.

Step 1. Choose cases for automated testing

As much as developers would want automated testing from A to Z, there are bottlenecks that can only be assessed manually to the fullest of their extent. So, before figuring out the logistics of automated testing, you will have to decide which cases will be up for automation.

If your test case matches one of the following categories, you can be confident about automating it:

  • Runs on multiple configurations
    In case you are testing an application using different operating systems and target processors, writing automated scripts will come in handy. Testing each configuration manually will take a ton of time and stall the development process.
  • Requires a large data input
    If a case requires a large data set, consider automating it. Moreover, in order to not collect data manually, testers normally use data generation tools.
  • Takes a long period of time to perform
    In case a test needs to be supervised continuously for more than 10–12 hours, it’s smart to consider automation. A script can be executed on its own without any human supervision – all a tester should do is to analyze the report generated by the testing tool after the execution of the script.
  • Performance tests
    In order to get meaningful insights from a performance test, a manager would have to ask thousands of employees to access a website or an app simultaneously. Logistically, it would be a nightmare – nor is it likely that your company has thousands of employees to spare for such mundane tasks. That’s why all aspects of performance testing are recommended for automation. Incorporating a performance testing strategy ensures your business solution is thoroughly assessed, allowing you to predict system behavior under various conditions and handle high traffic with confidence.
  • Data-driven tests
    In case your case involves changing data sets without changing the body of the test itself, automation is a go-to option. You’ll then be able to run multiple tests simultaneously, improving both the quality and the speed of the session.

Step 2. Hire an automation engineer or a team

After you have determined a set of cases that will be up for testing, it’s time to choose a team to help you with the execution. Hiring an automation engineer is not an easy task especially if you want him to be versatile and capable of creating multi-faceted scripts.

Here are a few tips to reduce the number of challenges down the road and help you find a fit candidate in no time.

  • Look for someone who’s capable of assessing whether or not a case should be automated
    Automating all suites and cases is a bad practice that will only make the management more complex and the results of the execution – less insightful for the company. A person you want on board is capable of determining which cases are critical for automation, those that could be automated down the line, or stay manual altogether.
  • Choose engineers with experience in multiple tools
    Having a tester who has mastered one tool and is loyal to it has its own upsides – however, you’d be losing in terms of versatility and freedom. Make sure an engineer you consider a hire stays open to tools, monitors the landscape and is updated on the latest solutions – both open-source and commercial.
  • Specify your requirements
    You need to know what you’re looking for. Instead of simply browsing through CVs of engineers, specify the type of test you want to perform and ensure a candidate is not out of his depth in this particular area. By narrowing down the list of job requirements, you’re more likely to find the right person for the job.

While skills are crucial when it comes to looking for an automation engineer, salaries are still a factor to consider. In case you’re not familiar with the job market, here are the salaries of a junior, middle, and senior automation professional.

Job titleHourly rateYearly salary
Junior Automation Engineer$18.64$55,000
Middle Automation Engineer$30.29$74,000
Senior Automation Engineer$50.73$102,000

All the data is provided by Payscale (April 2019)

Step 3. Select the Right Automation Tool

Choosing a suite of tools for testing is crucial when it comes to an automated testing strategy. You need to ensure all the software you use for script execution would improve, not limit the process. At the moment, there are dozens of solutions with a similar range of features – choosing the one to go with is, by no means, easy. For high-performance API architectures, a gRPC load testing tool can be particularly valuable in ensuring the scalability and efficiency of your services.

That’s why a developer needs a set of criteria to use for support. Here are a few factors that influence the set of tools a QA specialist chooses to go with:

  • Complexity
    If your team does not possess considerable experience in automation, learning a new programming language in order to navigate the tool would be a concern. Make sure to find out the reach of your QA’s abilities, the languages they are comfortable with, and the tools they have previously used. Also, be sure to not choose powerhouse software for simple testing tasks – the abundance of features will slow the process down instead of improving the pace.
  • Technical feasibility
    Make sure that the tool allows you to reap the main benefits of automation testing – runs tests with no supervision, collects automatically generated reports, doesn’t rely heavily on the exact coordinates of the screen, is capable of running multiple tests at once, and so on.
  • Test data
    It’s best to go for a tool that has a set of data management features. This way, you won’t have to rely on different software products for dataset generation and script execution.
  • Application stability
    This includes an established community along with a broad range of documentation articles. Make sure the tool you choose for testing has a well-established reputation and is not at the risk of shutting down.
  • Reusability of scripts
    Make sure a tool allows you to keep a database of all written scripts and reuse them for a new project
  • A possibility to execute scripts in various environments
    It’s important for a chosen tool to be capable of transferring a script to a different environment and automatically changing application-specific data.

Step 4. Create the test automation framework

When creating a test automation framework, a QA specialist has to ensure it is scalable. Keep in mind that the framework is rarely used for a single project – instead, it is a template of test automation strategy, guidelines, and principles your testers should be capable of using for any kind of automated testing.

Note
You need to make sure a testing framework covers not just the execution but a system of reports as well as a retry mechanism to re-execute tests that have previously failed.

While your testing framework will depend on the scope of testing and the tools you choose to handle executions, here are a few general tips to take into account.

  • Use the PageObjects model (POM)
    As a tester is increasing the test coverage, they may end up with an unmaintainable structure. By adhering to POM and keeping tests and element locators separately, a QA specialist will be able to avoid code duplicates as well as the time they would normally need in order to set up locators.
  • Implement Object Repositories for GUI testing
    Instead of having test objects scattered all over test scripts, it’s better to keep them in a centralized location. That’s why using Object Repository comes in handy as it allows UI testers to store and declare all the elements of the app.
  • Define the scope of testing
    Make sure you’re clear on the testing requirements – this way, you’ll be able to define whether or not a test was a success. In case you don’t have a testing scope set in stone, major insights, and application issues can be overlooked.
  • Conduct frequent framework code reviews
    This way, a team manager is capable of bringing testers closer together allowing them to know each other’s strengths better – also, you’ll be able to seamlessly monitor juniors. By reviewing code, a team lead will ensure that every member of the team understand the unified testing standards and complies with them.
  • Provide the team with all the needed documentation
    This includes all the tool-related articles that allow QA specialists to solve problems during script execution as well as test reports. Make sure team members can share reports with each other – this way, everyone on the team will get an understanding of the bigger picture.

Step 5. Getting your team up to speed with automation

Establishing an agile and comprehensive testing framework is crucial – but it’s even more important to ensure a testing team adheres to it. If you’re a business owner or a project manager, it’s your job to establish a link between testers and developers as well as within the testing team itself.

Here are a few tips that will help you onboard QA specialists faster and build teamwork within the department.

  • Test and improve the team’s expertise
    Regarding domain knowledge, various software systems, and software testing phases.
  • Establish the communication and reporting system within the team
    You can use project management tools as well as shared code editors to increase the amount of collaboration during test sessions. It’s a good idea to put testers in a small group – this way, if one team member were to leave the team, another one would step in, and the development process remains undisturbed
  • Update your team about the requirements and test cases on a regular basis
    As a manager, it’s your job to ensure not a single change in the test objective remains unnoticed. It’s a good idea to schedule regular status update calls to make sure everybody on the team is on the same page.
  • Be sure to get regular reports from each team member to have better control of the overall process
    There are managers who adopt a report day strategy – during a fixed day of the week, a team lead collects reports from each team members. Such a strategy allows you to bring structure to the communication system and track testers’ progress.

Step 6. Setting up the testing environment

The test environment is a combination of software and hardware needed to test the performance of the system. Setting it up the right way is the bedrock of a successful testing session. Here’s a short checklist for creating a favorable environment for automated tests.

  • Make sure all the needed equipment is available
    This includes the hardware as well as additional tools – working PCs that supported the needed operating system, the needed amount of monitors, and so on.
  • Ensure that the system has all the applications needed for testing
    You might need to see how your program complies with other software a user may have installed. If that’s the case, be sure to prepare all the additional applications and file sets beforehand. Also, check that your team knows how to use all the apps needed for successfully navigating the environment.
  • Check the availability of standard datasets
    Remember that test data should be collected before script execution – otherwise, the test will take too much time to complete.
  • Assign the environment maintenance to a specific person on the team
    This team member will be responsible for the state of servers, hardware, and software.
  • Plan the scheduled use of the testing environment
    In case you share an environment with the development team, ensure both departments occupy separate time slots. Having both QA specialists and software engineers working in the same environments simultaneously could provide you with false test results and corrupt the entire process.
  • Ensure that the connectivity is achieved before initializing the testing
    This is especially important for cases that require a complex environment configuration.

Step 7. Developing An Execution Plan

When a team lead is understood by their team, understands its expertise and the range of cases and tools used for testing, and the environment is up and running, it’s time to elaborate on a detailed execution automated testing plan.

Note
Unlike the testing framework or the outline developed on the initial stages of a testing strategy, an execution plan is a daily list of activities described in detail.

Here’s what it normally consists of:

  • A description of day-to-day tasks
    The automation plan should include time slots for status emails, progress updates, room to learn, and so on.
  • Using pipeline orchestrators for test executions
    These tools allow testers to automate sequences of tasks and ensure they are completed in a fixed order. Some of the most popular options in the market include TFS, Bamboo, Jenkins, TeamCity, and so on.
  • Define a set of practices to keep tests change-resistant
    These guidelines should cover the way test cases interact with the elements of the tested application.

Step 8. Writing Scripts

One of the crucial steps of the automated testing strategy is writing scripts for execution. In order to make sure your automated tests are efficient and assess the functionality of the application to the best extent possible, consider paying attention to the following guidelines:

  • Prioritize scenarios
    List the most important user flows and focus on ensuring they are bug-free. A list of features with the highest values normally includes the login page, checkouts, or adding items to a shopping cart. In case you have an Analytics Account, use this data to find out which user flows are used most extensively on your website.
  • Break scenarios down to atomic flows
    Rather than writing a script for a complex sequence of events, break them down to a set of simple tests. When all of these actions are tested, run a sequence test to ensure the performance of the whole is no different than that of its parts.
  • Create single-purposed tests
    Depending on the element and the action you’re testing, most common testing purposes include validation (check the visibility and the look of the element), setup, and actions (testing a user flow).
  • Be sure the initial state of the test is consistent
    Due to the fact that automated tests consist of a fixed number of actions, it’s crucial that they start in the same state – otherwise, the flow of the test will be compromised.
  • Compose a complex test as a whole of simple ones
    This way, testers will be able to save time and human resources as there’s no need to record a new workflow from scratch.
  • Reduce the number of occurrence conditions
    In order to ensure the test is predictable and easy to manage, try to avoid if-statements when writing a script.
  • Write isolated scripts
    Running self-contained flows improves the quality of parallel testing. As opposed to connected suites, if there’s a hold up with an independent test, the entire session will not be affected.

Step 9. Reporting

After a series of tests is executed, a QA manager has to capture insights and put them into a readable and concise document that will help developers improve the product. While there are tools that create reports automatically – you’d only need to revise the content – there are a few tips that make test results reporting easier and way more efficient.

  • Analyze the results collectively, not in isolation
    That’s why experienced testers tend to keep logs of all the old session. By comparing the data between testing sessions, you’ll be able to keep track of progress, uncover repeated issues, identify tests that need further improvement, and so on. Also, in case you have detected an issue that was not indicated by the previous report, it’ll be easier for a developer to understand what triggered the change.
  • Get broader logs than pass-or-fail
    Make sure team members provide you with access to screenshots, system logs, and all the data captured during testing. Instead of tracking progress with pass-or-fail logs, rely on the step-by-step description of the execution process.
  • Improve data visualization
    The insights will have more meaning if you group them together in a table, a diagram, or a graph – this way, you’ll be able to get a clearer comparison between the builds of the program. Visualizing testing reports in a visually compelling way is highly important for team leads in case they have to use them as a status update for a client.

Step 10. Script maintenance

Creating a low-maintenance reusable test suite is a struggle for a fair share of testing teams. There are QA specialists who put no emphasis on test suite maintenance whatsoever – as a result, their scripts end up stale and underused.

Here are a few tips for testers who are eager to improve script maintenance.

  • Exclude browsers and operating systems with low usage rates
    In order to decrease the number of combinations for testing. If you need to test an application in different browsers across all versions of Windows, that would leave you with over 20 server-side combinations. By eliminating rarely used Windows versions or browsers, you’ll be able to significantly save both time and effort otherwise spent on testing.
  • Avoid complicated scenarios
    Test suites that are easy to maintain are, most of all, flexible and simple. This means excluding ‘IF-THEN-ELSE’ patterns and avoiding complex flows that mix users’ behaviors. Keep in mind that complex tests with nested conditionals are harder to debug and read – thus, their maintenance cost will grow.
  • Get to know the system’s asynchronous actions
    Be sure to keep an eye on client-side asynchronous actions and avoid hard-wired delays. While handling callbacks can be challenging for testers, in cooperation with developers, you’ll be able to determine the flow of the page and create helper functions.
  • Cooperate with developers
    By asking them to create backing APIs.
Have a Project in Mind?​
We have been working on performance testing projects since 2008.
Drop us a line to find out what our team can do for you.
Get a quote You’ll hear back from our tech account manager in one day if not sooner

Automated Testing at PFLB

In case you’re looking for a team of experienced testers and automation engineers, be sure to contact PFLB. Our team will create and execute an automated testing strategy that will comply with your company’s development methodology and will be non-invasive to developers.

Here are the types of automated testing we normally handle:

  • Automated regression testing;
  • Test automation during development;
  • Test automation for DevOps;
  • Test automation in a production environment;
  • Embedded system test automation.

Do you want a team of certified testers to handle your next big project? Contact us today – PFLB will be happy to get on board. You can also take a look at the full list of performance testing services we offer to choose which type of testing your project needs.

Table of contents

    Related insights in blog articles

    Explore what we’ve learned from these experiences
    More Blog Articles
    4 min read

    Concurrent vs. Simultaneous User Testing: Main Differences Explained

    concurrent vs simultaneous user testing preview
    Mar 7, 2025

    When it comes to performance testing, not all user interactions are created equal. The difference between concurrent and simultaneous user testing may sound subtle, but it has a big impact on how well your system handles real-world scenarios. If you’re preparing your platform for anything from daily traffic to high-stakes launches, understanding these terms can […]

    8 min read

    Performance Testing vs. Stress Testing vs. Load Testing : Overview

    performance testing vs load testing which one is more suitable for you preview
    Mar 4, 2025

    Software testing professionals often use performance testing and load testing interchangeably, but these are not the same. Performance testing is an umbrella term that includes both load testing and stress testing, along with other methods that evaluate a system’s speed, stability, and scalability. Load testing checks how well a system handles expected traffic, ensuring it […]

    3 min read

    SOAP UI vs PostMan: Which Tool is Best?

    postman vs soapui preview
    Feb 24, 2025

    When it comes to API testing, Postman and SoapUI are two of the most widely used tools, each designed with a different focus. Postman is streamlined, intuitive, and ideal for REST API development, automation, and team collaboration. SoapUI, on the other hand, offers comprehensive functional and security testing, making it the go-to choice for SOAP-based […]

    4 min read

    DevOps Pro Europe Conference 2025 – Advance Your DevOps Expertise

    devops pro europe preview
    Feb 21, 2025

    The annual DevOps Pro Europe conference explores the core principles and concepts of the DevOps methodology. It highlights how to effectively utilize common DevOps patterns for developing, deploying, and maintaining applications both on-premises and in the cloud. The DevOps Pro Europe 2025 conference will be available on-site and online, allowing participants to join in their […]

  • Be the first one to know

    We’ll send you a monthly e-mail with all the useful insights that we will have found and analyzed

    • People love to read

    Explore the most popular articles we’ve written so far