Performance testing is complex, you can’t use just a screwdriver – you need a swiss army knife.
I’ve seen yet another performance test tool that sells itself as needing ‘no scripting’ ability. A scriptless tool – ‘no coding required’. I’ve lost count of the number of tools I see selling with these points. If you’re thinking about getting one let me save you the trouble of assessing them – they tend not to work well, are clunky and an incorrect fit for most performance testing jobs, simple as that. They unravel as soon as you need to do something pretty unsophisticated (like calculate a date based on a value coming back). This means you have to drop out of the GUI and into scripted view. Almost instantly, instead of having a single look and feel for solving the load testing issue in hand, you have two very different views (GUI and code). GUI’s are good for very simple performance testing but incredibly clumsy for serious performance testing.
The Load Testing Scriptless Oxymoron:
Creating traditional performance tests is fundamentally a programming exercise; there is no escaping this. I’ve recently used a tool which made an extensive use of GUI’s, after a short period the GUI just got in the way and then continued to get in the way. Programming is powerful because you are given a small number of constructs and then given freedom to solve. As soon as you create a GUI you start creating walls, rules and restrictions. Your freedom to express a range of solutions automatically becomes eroded. Do we ever see programmers implementing code using just GUI’s – no, there is a reason for this. Its still more productive to to code in an IDE than it is to use a GUI. This hasn’t changed for 20 years.
LoadRunner – The Dinosaur that stands the Performance Test of time.
Loadrunner – not my favorite tool, but one that has seriously stood the test of time and revered by many performance testers. Why? Because the scripted approach it uses to solving the problem in hand is fundamentally correct. It’s an sound approach, and still now, after almost 15 years this approach hasn’t had to change – and its still works with minimum modifications for the new technologies. Doesn’t that say something profound about a scripted approach? LR does have a GUI view, but no-one I’ve met seriously uses this, its relegated and quickly forgotten. I’m guessing the Loadrunner team started on this route and realized it created more problems than it solved. If you need a good LR alternative, look up Facilita – it is LR evolved.
Here’s a summary of the fundamental short coming’s of a Performance GUI approach:
- Drop Out:: You need more that one view – as soon as you can’t solve an issue in a GUI then you have to drop out into script. How quickly and regularly you have to drop into scripting is a measure of how weak the approach is
- Rescript: If you have regular drops of code and have to re-script, then attempting to duplicate logic (for loops, conditional logic) created in the GUI for the new drop become cumbersome and time consuming.
- Features: Need a new feature in the GUI to cope with the next new thing? I bet it can be done in code and in several different ways. The GUI is stuck in a never ending game of “catch up” and you could be left high and dry. This isn’t even taking into the consideration the quality, the lack of clumsiness of GUI implementation and the time it takes to arrive.
- Transfer of Knowledge: A GUI view looks nice, but is impractical when attempting to understand logic. Having to expand tree views, inspect conditional logic and not easily being able to see the control flow at a glance means you end up endlessly clicking, expanding and inspecting. This isn’t just to understand the code of others its also your own. (See Clicky Click)
- Click flick Click, Expand, Clicky click: The job becomes impossible without the aid of a mouse and constant flicking between screens. Make sure you have a robust mouse.
- Knowledge not Skill:: With a few basic constructs and a LR scripting approach, a performance tester can pick up and learn a tool quite easily. GUI’s require extensive knowledge and learning.
- Fools Gold: By appearing to wrap in a GUI there is a perception that the less skilled can acquire the skills and productively goes up. This couldn’t be further from truth.
- Limited Solutions: In a purely scripted approach I can have 10+ ways of solving a problem, I then have the nice problem of selecting the one of these is best. In a GUI approach it becomes “how do I solve this within the confins of this framework”. Ironically, the act of implementing a GUI to make life simpler creates restrictions, limits options and makes the task in hand more difficult. (See also knowledge not skill)
There are many more, but I’ll stop there. So what are GUI’s good for? ::
- Creating Performance Tests for simple static and stateless types of websites.
- Being used as a visual metaphor for recorded, played back traffic and perhaps even a representation of the scripted approach.
- A starting point to template a creation of a solution to a problem
- Solving problems that adhere and conform to a rigid set of standards (Most of the web out!)
- Scenario creation – Thats build performance scripts together to create more complex and realistic scenarios
- Interpreting Data – Thats the run-time analysis and post analysis data.
- Selling and Marketing to people that are the purchasers, uninformed and not the Performance Engineers. “We have a GUI solution that increases productively” (It doesn’t!!!).
Using an umbrella for a parachute – seems like a good idea
Traditional Performance testing is a programming activity. Using a GUI to replace a programming paradigm may seem like a good idea, but its not. Don’t get sucked in, there are no easy ways out. Assess your problem and chose the most appropriate tool for the job – don’t get drawn into the misguided hype.
My one line Performance Takeaway:
Love & Harry Houdini: If a tools sells itself with a GUI approach be suspicious. In my experience these tools actually inhibit productively, require extensive knowledge, unravel quickly and tie the experienced Performance Engineers hands behind their back. Let your performance engineer try before you decide to buy, they will love you much more you if you do.
– And yes, I have been amazed at how many tools have been bought before in-house engineers have had the chance to try.
May Also Be interested In:
Performance Testing is hitting the wall: – Introducing a new concept of GUI based performance tools & applies to mainly ‘outside’ the firewall
Selecting the Correct Performance Tool: – Broad outline of how to go about selecting a tool
The Death of a Giant and birth of New Children: – A new dawn for Performance Test Tools