This article is aimed at explaining the difference between different types of performance tools and should prove useful if you are assessing which load tool is right for you. There are a myriad of load testing tools out there – far too many for me to research individually. I’ve thought long and hard about how to best present this information in a way that is easy to digest. What follows is my interpretation and a first stab, there are always exceptions to the rule.
The tools tend to fall into 3 distinct categories:
Freeware Performance Test Tools:
- Open source and free
- Specialist knowledge required – Developer level interest required to learn.
- Learning Curve: High
- Development Time: High
- Handover Time: High
- No commercial support: Reliance on forums and goodwill
- Best Suited To: In-house, subsystem testing, ‘quick and dirty’ testing, have high access to insider knowledge of application under test.
Commercial Performance Test Tools:
- All in one packages: A lot of what you need is included in these packages. e.g. Record and playback functionality, analyst graphs, monitoring capability, debugging ability
- Costs involved
- Learning Curve: Medium
- Development Time: Med
- Handover Time: Med
- Commercially supported – protocols updates available quickly, help for issues and fixes can be speedy
- Best Suited To: In-house testing of applications prior to release where aggressive releases and builds are scheduled.
- Examples: Loadrunner, Forecast, SilkPerformer
Cloud Based Performance Test Tools:
- No hardware or installation required
- Very Cheap and easy pricing structure
- Learning Curve: Low
- Development Time: Low
- Handover Time: Low
- Best Suited To: Simple load scenarios executed over the web on HTTP based applications, True E2E testing
- Disadvantages: Low protocol support, no in-house test ability, difficult to create complex load scenarios
- Examples: Loadstorm, LoadImpact, Gomez
Freeware Performance Tools: e.g. Grinder, JMeter
These tend to concentrate on generating load only. The largest benefit of open source tools are they are free and extremely flexible. Time to record and playback scripts can be lengthy and require different toolsets. These tools are especially suitable for developers that wish to quickly perform sub-system (or system) load tests or for the performance tester that has close access to the development team. I view the development time to create performance tests for large applications as a major factor when considering them. e.g Client server traffic is changing frequency with new builds.
I personally think there is a large hole in market to replicate and pull together a number of open source tools and build a solution that replicates the commercial tools – and potentially do a better job (more on this in another article).
Commercial Performance Tools: e.g. Loadrunner, Forecast, SilkPerformer
Commercial tools are generally suitable for medium to large sized customers that have a complex n-tier architectures and large development teams. If there are a number of different protocols that need supporting then you can generally find a commercial tool that will support it. Commercial tools have a number of sub-products bundled together: Record and Playback, Scenario Creation, Injectors, Realtime Analysis, Post Analysis and tie in with Monitoring Products (e.g. Memory, CPU, Network i/o). Together these sub-products greatly enhance the productively of performance tester. These tools tend to be suitable for large development team dropping regular releases into performance testing. Most commercial tools work on the same principles and generate load in slightly different ways.
Cloud Based Performance Tools: e.g. LoadImpact, LoadStorm, SOASTA
These are the new kids on the block and shaking up the old guard. This is an exciting development as they are a fraction of the cost of the commercial tools. They tend to work on a memory and CPU sledgehammer approach, which means they consume lots of hardware resources e.g. Amazon cloud services. This in a way is a minor downfall – as a lot of them do not give you the ability to inject their services in-house and past the firewall. Scripting ability and logic tend to be simple – which means complex load testing scenarios are not sometimes achievable depending on your requirements. These tools have a time and a place – if you are considering them then also consider your hardware environment and infrastructure in place. You may only be able to test during quiet periods of the day (as you are testing from internet) and you will have to look carefully at the infrastructure outside the lab e.g. firewall, network capacity – the web facing infrastructure. I feel that this is the future – it’s only a matter of time before these services improve on their weak points.
As I said earlier – this is a generalisation of the different load testing tools available (there are over 100), but it should steer you in the right direction and give you a good starting point when assessing which tool to use.
See also Selecting Performance Test Tools