This time I present a brief comparison of two load-testing tool: Loadrunner (as called “the industry standard load testing tool”) and its open-source “counterpart” the Apache JMeter.
Why are exactly these two pieces compared? I made a quick research at indeed.com to find out the global demand on load testing professionals. We can conclude, that Loadrunner holds a strong position on the market and Apache JMeter is quickly taking up – quicker than any other performance tools today.
So, here you can find the comparison – guided through several factors which I hope could be helpful during tool selection:
|JMeter||Loadrunner / HP Performance Center|
|Platform||Java (Platform independent )||Microsoft Windows
(server parts require ASP.NET)
Performance Center requires several Windows Servers. Unix/Linux for load-generator is supported though.
|Tool Architecture||Local script authoring UI with Load generation capabilities. |
UI-less load generator option exits.
|Desktop clients for Load script authoring (Virtual User Generator) and result analysis (Analysis). Controller and Load genearator for test execution.|
|Remote load generators||Multiple Remote Load generators hosts are supported.||Distributed (even multi-region) load generator hosts. Over-firewall solutions.|
|Cloud-capability||Third-party Amazon image for cloud-based testing is available.||Via installing Load Generators on the cloud. Separate Cloud-test offering from HP is available.|
|Resource monitoring||Missing/limited||Performance center supports a variety of platforms/applications.
N.B. JMX monitoring requires extra HP software (Sitescope).
Resource metrics can be conveniently tracked with other metrics in the Analysis tool and on the online dashboard on Performance Center.
|Documentation||Apache website provides documentation. |
|Context sensitive help (“press F1”)
|Community||Open source software community, mailing list.||Official forum on HP site. Activity is low/medium.|
|Scripting||A script is actually a graphical tree of nodes, with limited editing capabilities.|
Supports BeanShell for evaluations, checks, etc.
|C (primarily) and Java.
|HTTP traffic capturing||HTTP Proxy||Network capture of local applications.
HTTP Proxy for remote applications.
TCP dump for mobile/webservice protocols.
|Script pacing||Via number of Threads setting.|
Timers provide (approximate) TPS control by automatic pacing.
|Via the number of Virtual Users and iteration pacing.|
|Scenario composition||One script can contain multiple thread groups and form the scenario.||Separate scripts can be configured to constitute a scenario|
|Script runtime configuration||Via command-line parameter||Via command-line parameter, performance center automatically manages it.|
|Repointing test to a different AUT host||No out-of-the-box support. Script has to take over configuration parameter value.||No out-of-the-box support. Script has to take over configuration parameter value.|
|Protocol coverage||Focus primarily on HTTP and Java-related protocols (JMS, JDBC).|
Also supports FTP, LDAP, SOAP.
|Broad coverage from plain HTTP-based protocols through proprietary enterprise products.|