Testing Environment in the Project Lifecycle

To launch a product, you often need several testing environments. But why? Why pile up equipment, overload engineers with tasks, and hire more staff? Is it really inevitable? We will take a closer look at this problem below, but trust us, it is. We have been providing performance testing services since 2008, and have done it for 300+ customers, so we can’t even count how many testing environments our 400 engineers have set up.

Is the Production Environment Suitable for Running Performance Tests?

Let’s imagine you have a project, a production environment, and the latest version of your product in it. Unfortunately, after the launch or the next release, you would often find some bugs in it. It’s very tempting to launch or release your software as soon as possible, but then the users may notice the bugs, too. Even if you run tests in the product environment to get rid of the bugs, the mere fact of their existence can repel the users. And if the bugs are critical, those users may as well just leave you: afterall, who needs a product that doesn’t perform the tasks it was designed for? An example would be an app providing bank card services, but instead of current account balance it shows users the balance from 2 hours ago. Unacceptable, right?

Even if testers do find these bugs and errors before the users see them, it will take time to fix the bugs and reupload the app. While reuploading, the services will be unavailable. And you are in a trap: your customers are leaving you while all you are doing is trying to improve your product for their convenience.

Different Environments for Different Teams

What kinds of environments are there? Why is a separate environment necessary for every kind of test? The types of testing environments include:

You can have several environments of each type. For instance, you may need different environments for functional and load testing, or several dev environments for different lines of development. The number varies and is usually from 4 to 20 per project.

The work starts in the dev environment, where they can test different functions and whether software works at all after the updates or has crashed because of some new feature they added.

If it does work after the updates, proceed by taking it to the test environment. There, testing engineers can look into business processes in detail and find errors and minor defects that developers should continue working with. Every function is tested separately.

Afterwards, it’s the turn of the pre-production environment. Here, you test  system deployment and run integration tests. Separate improvements of different features are united into a big update, which is mounted over the previous version of software. Then, check how the connected systems work.

If your application is rather heavy and/or complex, and if the number of your end users is rather big, you should also run load testing. Most probably, it will need a separate environment of its own. Load testing is quite a process, and it is crucially important that other teams’ processes won’t have any impact on it. Don’t use developers’ environment for load testing: errors may occur and prevent you from running the tests correctly. You will also steal time from them, since they need their environment to test updates, while load testing should be run on the latest version of the product.

The same goes for the tester’s environment: their work can create additional load, which can compromise your tests results, or otherwise your tests may interfere with their tests.

How to Create a Separate Environment?

If each team needs its own environment, it makes a minimum of 4 environments total: for developers, test engineers, load testers and management. What steps are needed to create them, then?

Step 1. Analyze the requirements to the environment, and calculate the configuration.

First of all, analyze access patterns to the site’s internal servers. It is important for the environment to be closed, so that it does not influence other environments.

Then, analyze the requirements to the environment, and calculate the configuration. Write down why you need your environment to be as similar to production as possible, how to make it this way, and what characteristics should be kept in mind.

If some particular functions do not require the capacities of prod to be tested, these environments may be made with smaller capacities, but generally, architecture should mirror that of production. Specifically, the load testing environment should be at least 70% of the production environment. If you make it less productive, the results of the tests will be very uneasy to extrapolate, since the load will be different.

Let us give you an example. Say, there is a problem in the production environment. Some operations take too long to be executed. If you run load tests in an environment that only makes 50% of the production capacity, you can crash into the wall of capacity limitations in CPU or RAM of application servers. But in production they may work well, so the problem might in fact be in the DB that is too slow to process so many queries. A weaker environment does not allow to apply the necessary level of load, so it would be impossible to find the source of the problem.

Step 2. Choose between hardware and cloud.

This is an essential issue, since an environment with large capacities is quite expensive. Both solutions have their pros and cons. To make your choice, consider the following issues: urgency, utility, affordability, and availability.

Step 3. Configure network and access parameters.

As we mentioned above, each team should have their own access to the testing environment, and their environments should be placed into closed subnets. There was a legendary story once: while testing a stock exchange, a performance testing team was interrupted by a group of Special Forces fighters. They were interested to find out who had bought 70 pounds of gold. The testing environment and the production environment were interconnected, access parameters were configured in a wrong way, so when testing the purchase of gold, the engineers had literally done a real purchase.

Step 4. Configure the processes in the testing environment

What you have left to do is assigning a team and thinking about some standard procedures, such as automation of the version assembly, or setting up and restarting the environment. Plan the release cycle and add time for development, functional testing, load testing and integration testing. Then, just follow this roadmap, while also backing up DBs in your testing environments in time.

Conclusions

Without organizing your release cycles around load testing service, new releases may become a weak spot in the organization of the project. Load testing is quite important in this cycle, and in the absence of an environment as similar to prod as possible, it can bring controversial or faulty results and threaten your product’s success.