Performance Testing: What It Is, and Why You Need It
System failures and website crashes have been an issue as long as businesses have had websites and/or applications. Some notorious examples include eBay’s crash in 1999 with $4 million losses, or FOREO’s website crash on the Black Friday with about $10 million of lost revenue. Heck, weeks ago Zuckerberg lost $6 billion in a few hours after the Facebook outage.
In PFLB, we strongly believe in the importance of performance testing for business. It’s sensible, more rational and more effective to run performance tests and avoid the crash rather than mend the system after the crash, losing not only revenues, but also man hours. In this article, we’re proving the point: we have been providing performance testing services since 2008, and have done it for 300+ customers, so our 400 employees do know a little something about how performance testing can change your business.
Request a quote, if you want to discuss how your business can benefit from performance testing services.
Table of Contents
How much does a system failure cost?
So in 1999, eBay went down for 22 hours. The company estimated that they had lost around $4 million, or almost 2% of their annual income. But even if you are not eBay, just google system failure or website crash: you will be surprised by how often it happens to banks, on-line shops, governmental websites, etc. Such giants as Facebook or Instagram are not safe either, besides a huge number of IT specialists on staff.
Meanwhile, the consequences for business are not something you can pretend are nonexistent. They include:
- lost profits;
- costs to fix the system;
- reputational losses;
- reduction in the market value of the company;
- rating decrease;
- productivity issues.
Let us give you some examples. One of the most common cases of lost profits is a website crash under peak load on Black Friday. Seriously, many of our clients come in the end of November, although nothing kept them from running tests before the crash, not after it. But for them, sales just don’t happen, because the moment has gone.
After a shutdown, you’ll have to put everything back on track, which is also costly, especially if you have to rebuild or at least repair the infrastructure. Headlines in the media that mention the crash lead to reputational losses. These, in turn, can lead to reduction in your market value, as it has just happened with Facebook. Moreover, if you’re in e-commerce and in Google ratings, you can slip down the rating list. As for productivity issues, a number of your employees won’t be able to work while the system is down, and this is money in salaries, lost for you, too.
Some of the biggest companies have already developed their own approaches to calculate how much system changes cost them. Take a look at a few examples:
- Slowing down the Amazon website by 100 ms led to a loss of 1% of revenue;
- Speeding up work by 400 ms led to an increase in traffic by 9% for Yahoo;
- For BBC, slowing down the site by 1 sec led to the loss of 10% of users;
- Slowing down Google by 200 ms led to a loss of 20% of ad revenue;
- 1 s improvement led to +2% of conversions for Walmart.
100 ms slowdown
– 1% revenue
200 ms slowdown
– 20% ad revenue
400 ms improvement
+ 9% traffic
400 ms improvement
+ 9% traffic
1 s improvement
+ 2% conversions
1 s improvement
+ 2% conversions
1 ms slowdown
– 10% users
Do You Want to Load Test Your Product?
Drop us a line to find out what our team can do for you.
How much can your business loose?
But how do they calculate it? Well, it is obviously proprietary information, but in fact there is quite a universal formula that lets you calculate your own, hopefully potential, losses. Roughly. Using it, you can estimate your losses and see just why performance testing is so vital for your business goals.
1. For lost profits, divide your amount of annual sales by 525600, that is, the number of minutes in a year, and multiply the number by the number of minutes that your website or app was down. That is how much you lost or can lose, in sales.
2. For costs to fix the system, multiply the average salary p/h in USD by the number of people involved, and then by the number of hours they (will) spend putting the system back on track.
3. For productivity issues, multiply the average idle employees’ salary p/h in USD by the number of idle employees, and then by the number of hours they had to stay idle
4. Next value is projected revenue decline due to loss of customer loyalty. To calculate it, multiply value from step 1, lost profits, by the percentage of repeat sales. Alas, you won’t know about repeat sales until after the accident and a while more, which is why this value is called projected.
5. The last indicator is the projected decrease in revenue due to reputational losses. It is calculated in a similar way: lost income is multiplied by the percentage of sales to customers who came to the website from social media to compare goods and/or services.
You can look at your company’s history and calculate the number based on the shutdowns you have already had, or try to calculate it for a potential shutdown. You will anyway need it to compare to the cost of performance testing to see if it benefits your business or not.
What is performance testing?
Performance testing makes it possible to assess the speed of operations, ascertain the permissible number of simultaneously active users or simultaneous transactions, and determine how the volume of data affects the performance of an IT solution. The English of this is that performance testing is creating a load by emulating real users’ actions with software means to research how the system performs under load.
Performance testing is just right for you, if you need to:
- ensure the stability of the systems against the background of growing business demands;
- predict, assess and mitigate the risks when replicating new processes and after the introduction of new software releases;
- use a systematic approach for long-term planning of systems development based on performance requirements.
In each of these situations, performance testing helps your business achieve its goals. With its help, you can:
- determine the maximum system performance;
- compare different versions of the IT system, hardware and software configurations;
- detect bottlenecks that limit system performance;.
- find out how the system performance changes with a significant increase in the volume of data;
- measure execution time of critical operations at different loads;
- find out if you are ready for the sales season, advertising on the Internet or opening new branches.
But the main thing is that regular performance testing allows you to take control of the situation and guarantee high quality of services.
What is NOT performance testing?
So, performance testing sounds like a magic wand, doesn’t it? Why are there still companies that don’t use it, then? The thing is, many businesses think they do use performance testing, but in fact they don’t.
There are a number of useful tools that people may mean when they say they load test their products or systems regularly, but, although useful, these instruments do not equal full circle performance testing. These are NOT a substitution for performance testing.
URL-bashers
They create a large load on the site with simple queries. As a result you get a short report on the response time.
Optimization of the front
It mainly affects the processing by the browser and how it copes with it, but does not show what happens before that, i.e. the connections between components and servers.
Monitoring
These are systems that allow you to observe the UX, operation of systems, and server communication. It is a mandatory addition for performance testing.
Productivity solutions (CDN, Cash-management, etc.)
These allow you to improve the performance of the system, but do not give you the opportunity to do a full-fledged analysis.
If you are using any of these, your IT engineers are doing a great job, but it does not substitute for performance testing. If any, it gives you a very scattered impression of how your system might work.
Mission-critical moments
In every healthy business environment, there are pinpoints, at which performance testing is just inevitable. They are, as follows:
- launches of new applications;
- introduction of new features in the systems;
- new offers for clients;
- new events;
- major promotions, such as Black Friday, etc.
If your business history includes or is supposed to soon experience any of these stages, consider looking for a performance testing services provider. And keep it mind that the perfect time to do so is 4 to 6 months prior to the event. Nobody can help you avoid the losses 10 days before the Black Friday. Fix the roof before it starts to rain.
What Type of Performance Testing Do You Need?
Find out on a free consultation with our experts.
Approaches to performance testing
There are two opposite approaches to performance testing: you can do it yourself or outsource to a performance testing company. Both have their ups and downs. If you make a decision to do it yourself, consider the following challenges:
- Full responsibility for the project. Obviously, there will be no external specialists to secure you from making mistakes, so make sure you’ve got reliable resources within the company.
- Planning ahead. You will have to suggest a plan, a roadmap, checkpoints, deadlines, and KPIs, all by yourself, to sell it to your managers.
- Using only internal company resources. Check if you have enough time, specialists and a stand to run the test.
If you, on the other hand, decide to outsource your performance testing project, you still need to be ready. The performance testing specialists will need data, access to your DBs, and your stand.
There is always an option to combine the two approaches, though. If you don’t want to pay for performance testing every time (and there will be many times, as discussed below), you might want the performance testing company to share their expertise. There are many ways to do so, but they should be discussed before you start the project.
Performance testing frequency
To cut a long story short, performance testing should be a part of DevOps. You run it once, you see the results and the benefits, but then the next release of your software is ready, and then what? Performance testing results go out of date very quickly, so you have to repeat it regularly in order to avoid system shutdowns and crashes. One way to do so is to integrate it into DevOps.
One-time performance testing
.exec(http(“Home”).get(“/”)
.pause(5)
Automated process
Stages of performance testing life cycle
Beware: your first experience with performance testing will most probably be just a sneak peek into the future of your business. It’s not like things don’t work the first time: performance testing does, but soon enough you’ll find yourself moving in the direction of regular tests. You will (or should, anyway) pick out engineers who will be responsible for the tests, find resources, and eventually, run tests before every release or launch. There are many useful tools that help automate this process: have a look at our articles on Top10 best load testing tools or Best online load testing tools.
But that’s not it, either. Your ultimate goal is shift left testing. There, you start testing along with designing your technical assignment. Your specialists test new code, go back to their colleagues from development and give them feedback on any risks of failures, if any. Then, they can proceed to the next stage or keep working with that same part of code. Similarly, you test every build and every deployment.
A prominent feature of shift left testing is that it is both cheaper than testing the final product, and it decreases your product’s time to market. You will never again have to put a fully developed product back to the development stage to start from scratch. Instead, due to timely testing, there will be no surprises before the launch.
Performance testing: stages of life cycle
First run
Regular runs
Shift Left Testing
In an ideal environment, performance testing starts along with the design of technical assignment and the first lines of code.
Have a Project in Mind?
We have been working on performance testing projects since 2008.
Drop us a line to find out what our team can do for you.
Conclusions
A perfect performance testing solution should benefit several groups of staff members. It has to save time and be convenient for developers and QA specialists. It is supposed to help management ensure transparency of the department’s work and save time on building regular reports. It obviously should save money for business, one way or another. Don’t settle for anything less.
At PFLB, we’re happy to share. Don’t hesitate to contact us, if you have a project in mind. We’ll help you find an ideal solution for your company.