There are basically two models of providing performance testing services to the customer: outstaff and fixed price. If your customers chose outstaffing, it means they have an experienced testing engineer on staff. This person will be able to set goals and tasks for the testing process and act as a performance testing manager. In this case, they only pay your engineers’ hours, and take full responsibility for the results.
Fixed price means that the responsibility is on your side. Your customer can’t run the tests themself, and wants a package deal. As a performance testing manager, you’re going to have to design everything from scratch. In this article, we’re looking at stages of a fixed price project, as seen by a QA project manager.
The first thing you need to do together with the customer is to fill in your communication map. it should include all the stakeholders, such as the management, architects, developers, analysts, admins, consultants, and business. You will need architects, because they know their system better than you will ever. You need developers for consultations, and analysts to ask them about business processes. Administrators may be your key to your testing environment. SAP consultants are very much aware of how exactly every business process runs. Access to each and every of these specialists will make your work much easier.
Whether you have already drafted your map of communications or not, the first meeting is an important point for your QA project roadmap. If it’s possible, ask someone from the customer’s side to organize it. While at it, use your survey or a sample of performance testing methodology to ask all the questions you need. If you feel like you have too many, start with the questions which are directly related to the first stage of the performance testing project, in case there will be no time for all of them. You will need to give tasks to your engineers after the meeting, so you need to understand enough.
A little life hack is to record the meeting, since you might miss something and want to go back to the recording many times later. Obviously, you have to ask all participants for permission to start the recording.
Make sure the next step you take is identifying the list of project goals, both business and technical.
Business goals of a performance testing project equal its background causes. Basically, it is why the customer got the idea they needed performance testing. It may be business scaling, a new release or getting ready for the Black Friday rush.
Technical goals of performance testing are something you will have to come up with as a performance testing manager, based on your customer’s needs. The list may include eliminating bugs in the production environment, finding bottlenecks, decreasing system response time defining maximal productivity, reliability testing, etc. Most often, the customer won’t know what their technical goals are, and you will have to translate their business needs into technical goals of the project you manage. As an example, let’s suppose your customer wants performance testing because of a system crash in production. To translate it into technical goals, you may recommend to analyze logs from that particular moment of crash, make a profile based on those logs, look for operations that were being processed at that moment and try to reconstruct the situation in the testing environment.
You should, of course, base your recommendations on the overall project evaluation. To meet time limits is more important than to precisely follow the scope of tasks, since the latter can change a lot during the project. It doesn’t really matter if the contract says you have to test maximum performance while the customer asks for integration testing or some other kinds of tests. But since project evaluation is based on the number of professionals and the time they are going to work on it, it is really important to be able to meet the deadlines.
Keep it in mind that customers sometimes try to get a free ride and ask for much more than they actually have paid for. Your contract with the list of technical goals will save you from such problems.
Do you want to load test your product?
Drop us a line to find out what our team can do for you.
The earlier you give the customer the list of what you need for the project, the earlier you receive it. It’s best to try to discuss it in the first meeting.
You will most probably need either a direct contact with architects or an architecture layout made by them. They know their product much better, so the latter is preferable.
You need to get yourself familiar with the system, its capacities and hardware it is based on. One of the main questions here is how the testing environment corresponds to production. If, for instance, you are given one environment and then told to run tests in a different environment, you’ll waste your time. And, as mentioned above, time is money.
You’ll need their statistics, since you can’t start writing scripts before you have a profile. You’ll need access to environments, hardware, and, most important, to your customer’s staff. After you’ve designed your communications map, you come up with questions or requests and deal with particular people on staff instead of the management. The person you need may well tell you they are busy with their own tasks. To avoid conflicts and/or holdback, it’s better if the management from the customer’s side can warn their staff about their involvement with your project and the reporting period.
Based on the architecture, you will have to decide, which servers are subject to testing, what is in the system, how you will serve loads, etc. There are a great deal of factors to consider. The system is integrated, it sends and receives data, users interact with it, different protocols are used in different parts of the system, and so on, and so forth. Study the system as close as possible, to the tiniest detail.
Your system might both generate load and be loaded, but these two processes should be regarded as separate while testing. You will emulate the source of load with your loading scripts. Depending on the system architecture and the customer’s needs, you can offer your customer isolated or integrated tests, or both. Isolated tests check how well each component of the system works, provided that everything else works fine. After isolated tests, all the systems are connected back together, and in an integrated test, you look for bottlenecks in connections between the systems.
Integrated tests are usually easier to run, since you only have to emulate users’ activity, while all the connections between the system’s components are already there. If isolated tests give you data on maximal performance of each system component, and then you can tune each one of them separately, integrated tests help to find out how they perform as a whole.
One trait that unites many of our customers is that they do believe they know their systems. The thing is, most often they don’t. If the load or performance testing profile is being made up by the customer, it may include the operations that the customer thinks are there, but in fact they are not, if you check statistics. Customers often think their profiles are much more intense than they really are, overloaded, as according to them.
You can write a profile yourself, and base it on data, not on thoughts and beliefs, but it’s not a quick thing to do. If time for that was planned as according to the contract, you will have to get access to statistics, and waste even more time when discussing it with the customer. Evaluate if you gave time and other resources for all those tasks.
At this stage, you have access to the system and have had a look at it. You know your QA team and you know your timelines. Even if you don’t have a profile yet, try recording traffic: it’s something to at least start assessing labor costs. Even with the first recordings, you’ll be able to evaluate how much time a certain script would take.
As a performance testing manager, you are familiar with the team, so ask them what they are good at, how long have they been in QA, and what are their strengths and weaknesses. So you can either suggest your own evaluation, or ask team members to evaluate what they can do in how much time. The second approach seems to be more motivating, since it suggests agency and self-imposed responsibility. Moreover, in every project, your team and your own labor assessment skills will build up, too.
Don’t forget to add time for debugging. The risks are high: bus factor, delays on the side of the customer, COVID and other unforeseeable circumstances all ask for some extra time. Usually the amount of time you need can be calculated as a certain percentage from the number of scripts you are planning to use. For 3 scripts, it may be 5%, whilst for a bigger number of scripts you should add 10 to 20% of time for debugging.
Make a project plan using MS Project. Evaluate each script, each stub, each member of staff and make a plan using this or any other tool. MS Project is not the only tool you can use, but it has certain valuable features. For instance, your customer may see which tasks they are responsible for, and how much of each task is complete by a certain date.
MS project helps you visualise how well your forecasts are working, so that you can keep yourself and your customer informed. It makes revision easy and saves time. If you’re not able to meet the deadline, the earlier you see it and inform the customer, the more ways to compensate for a delay there are.
Transfer tasks from the plan to Trello/Jira or a similar task tracking tool. Gaining access to the system, deploying monitoring, and writing scripts – these are all tasks of different priority, and according to priority, they should be included in the backlog.
Created tasks have names, descriptions, and tags. You can also assign a specific participant if you think it’s important that this particular person should be responsible for it. You can add a checklist, a deadline, or write comments on the task. It is advisable to do so, especially if you move the task to the “Waiting” column, specifying the reason in the comment. When the task is completed, it is moved to “Done”.
Also, start making a project in Confluence. Create a project and fill in all the information. It is necessary so that if the customer returns in half a year, and the people who worked on this project will be unavailable, you will be able to retrieve all necessary information from Confluence. When creating it, add Performance testing methods and the final report. The more information you contribute, the better for subsequent projects.
A lot depends on your deadlines. If your plan is realistic in terms of deadlines, you can use agile task management, and your staff will pick tasks themselves. Otherwise, you can assign tasks by yourself to develop the versatile skills of your team members. It’s safer to assign tasks if you’re in a hurry. No one knows your team’s strengths and weaknesses better than you.
Your tasks should be smart: specific, measurable, achievable, relevant and time-bound. “Write a message to the customer with a request for statistics” is wrong. Writing it will take a few minutes, while getting the statistics will take much more time. It is crucially important to set a specific task with specific deadlines.
If you have a large team, then it is better to build a scheme where more experienced employees are responsible for less experienced employees. And don’t forget that the main mistake of all new management is they want to do everything themselves. Instead, trust your team, give them tasks. Before you take on a task, think about whether it is possible to give it to someone else.
It makes sense to hold daily meetings in the beginning of the day to discuss problems they faced yesterday and their plans for the upcoming day. Weekly meetings are mandatory. You should tell the team about the status of the project, remind them of the deadlines, revise the tasks, and adjust the plan. If you expect delays, it’s better to report them to the customer at an early stage.
Also, if some of your employees come without experience, it is better to check some of their tasks after them. If you do not check the script at the beginning of the project, you may end up with tests that don’t work. (See our article “Organizing the Best Software Testing Team“)
You should regularly report to the customer what has been done and how the project is proceeding (Check our “Software Testing Documentation Tips“). Here are some of our favourite ways to do so:
When you test other company’s systems, a lot of their staff will not trust you. Your test results show their system doesn’t work well? But for them, it might be good enough, and they may start doubting your competences. To avoid criticism at the end of the project, start involving developers and analysts from their side at the earliest stages possible.
First, create a test registry, an Excel spreadsheet. There, you will be able to write down dates, details, results, short summaries of changes made to the script, and conclusions. Write down that the response times are not higher than one second, or that the maximum performance is this and that, or that the problem was reproduced at such and such a moment of the test.
Lifehack: send yourself an email with results of every test, and then when you need to write a final report, and you do not remember anything, collect all those emails into one report, add conclusions, and that’s it. Final report will not take you long.
To sum up, you are responsible for the technical side of the project. Responsibility for contracts and orders, conflict resolution with the customer, communications with the customer, finance, and the non-technical results of the project in general are on the overall Project Manager. The PM is supposed to resolve any conflicts between you and the customer, and must participate in all communications with the customer. We will take a look at such problematic fields as budgeting, contract control, risk management and team selection in the next articles. Stay tuned!
Creation of high quality solutions is impossible without testing them, so an expertise in performance testing management is something very valuable in today’s world. As a company with 400+ professional engineers, we do know that. In PFLB, we have been providing performance and load testing services since 2008, and have done it for 300+ happy customers, so don’t hesitate to contact us, if we can be of any help.