Those of you who have experience testing a graphical user interface (GUI) know that it can be quite different than testing other kinds of applications. GUI testing tends to be a lot about navigation, and maintainability is a huge challenge.
But are you testing a GUI? Or are you using a GUI to test something behind it? For example, if you are testing a DSL router that has a web interface, are you testing the web interface? Or are you testing the router, and accessing it via the web GUI? It turns out that these are quite different things. And I suggest that you give that careful consideration as you develop your test plans.
In fact, I think you can expect to see a lot more GUIs coming on all kinds of devices -- from small handheld consumer devices all the way up to large backbone routers, and everything in between. That's because your customers are becoming increasingly used to working with GUI interfaces and are becoming less tolerant of command-line interfaces. Of course, there are still many benefits of CLIs, but in general, I think you can expect them to slowly disappear over time.
In order to understand the difference between testing a GUI and testing the device (or application) behind it, consider a device with a command-line interface. When you run a CLI test, are you testing the CLI user interface? Or are you testing the device itself? Well, you probably do have a bunch of tests to make sure that the CLI itself is working correctly. But I suspect that the majority of your tests are using the CLI to configure the device and to extract state information from it to analyze the behavior of the device under certain test conditions.
With the GUI, the same concepts apply. There are tests to verify that the GUI itself is working properly. And then there are tests to verify that the device or application that the GUI provides access to is working properly. If your device has both CLI and GUI user interfaces, then you might be testing the device through the CLI, and then adding tests to make sure that the GUI is also working. This is a good strategy -- as testing via CLI is probably easier and more maintainable. On the other hand, testers are starting to tell me that they want to test their device in the same way that their users use their device. And so as they start to see more and more users depending on the GUI, they want to shift more of their testing to go through the GUI as well.
I suggest that you design your test plans with this very clear distinction in your mind. You need one set of tests to verify that your GUI works correctly. You need a different set of tests to verify that your device is working correctly. If that second set of tests will be done through the GUI, that's fine. But the goal and approach to those tests are likely to be different.
Let's consider how they will be different.
If you think about the code in the product that provides the GUI, you can visualize what you need to have test coverage for. The GUI code is usually made up of a series of "pages". To test the GUI, you are probably going to have a set of tests that verify navigation among these pages. And then you'll have a set of tests that verify that each page displays the correct information. And when you perform actions on these pages, the actions are performed. But note that in this series of tests, if you are focused on the GUI code, then you probably don't care about whether a certain sequence of actions works correctly. You are more interested in whether the information presented is consistent with what you know is in the device, and when you take an action, whether the device will perform that action.
Now let's look at testing a device through a GUI. In this case, you are not trying to test the GUI code. You are trying to see if your device works correctly. If you do this through the GUI, then you will have to use the GUI to perform the configuration steps and to extract the information you need for success criteria. But in this series of tests, you don't really care about any of the other stuff presented via the GUI.
Many testers are tempted to combine these two types of tests together. After all, you'll be using the same controls and fetching the same kind of data from the pages. Right? Well, I contend that it is a "false economy" to think you can kill these two birds with the same stone(s). You are better off designing two sets of tests -- each optimized for its own purpose.
iTest itself provides a GUI as you know. So Fanfare is faced with this challenge every day. We need to verify that each new feature in iTest is working correctly. And we need to verify that any additions or changes to the GUI are properly tested. So we know a little something about this challenge. We're not always perfect at this. But we certainly try. We have some GUI tests that verify that each control in a given view/editor works -- and that information presented in these controls is consistent with what we expect it to be. Then we have another set of tests that use the GUI to verify certain workflows to see that the underlying feature is operating correctly.
You might ask what would go wrong if you combine these different tests together? What we have learned is that when you are testing the GUI itself, you are mostly trying to get coverage. Have you tested every control on every page? This kind of testing can be modularized -- so that if part of the GUI is rebuilt in the next release, only those tests that focus on that part of the GUI need to be updated. The tests (via the GUI) that exercise a feature in the underlying product is going to be organized quite differently. What we have found best for these is to develop a procedure library that performs abstract GUI navigation and actions. These are not optimized for using every control. Instead, these library procedures do very specific things -- navigating to a specific page, or fetching a specific piece of information. Then the feature tests tend to look like sequences of calls to those "building block" procedures.
The goal is that when a certain part of the GUI changes, you want to do two things: replace/update the GUI tests that were intended to verify that part of the GUI; and, second, update the supporting procedures to deal with the GUI changes, so that all of the feature tests can remain unchanged. The world is rarely quite that clean. But I think this is a good goal for your testing architecture.
If any of your GUI testing experts would like to comment, I'd love to hear about your experiences in GUI testing -- and testing through a GUI!
 | Kingston Duffie is the founder and CTO of Fanfare. Learn more about Kingston >> |
You must be a registered user to add a comment on this article. If you've already registered, please log in. If you haven't registered yet, please register and log in.