Shop Talk

[#1]

If you ask around among test automation experts, you’ll find that many are dedicated to the notion of data-driven testing.  You can read about this concept here.  People have asked me whether it is possible to use this approach using iTest.  The answer is definitely “yes”. 

In data-driven testing, the idea is to reduce the number of automated test scripts you have, but at the same time increase the number of test cases.  Each script is more generic.  It walks through some process where the steps in the process are defined as records in an external file or a database.  A parameter to this generic automated test identifies a file to use, or a query on a database to use to get the records that will drive the test.  The number of test cases increases – because each different query on a database (or each different external test file) creates a different scenario – even though they all depend on the same script to perform the actual execution.

Let’s look at a simple example.  Suppose that you want to define a data-driven test that sends a series of CLI commands to a device-under-test and for each command, checks that a certain piece of data in the response is correct.  The commands to send and the results to check are all pulled from a database.  The iTest test case is going to be completely generic.  Each individual test case is defined as a set of records in a database.  In this way, users could create new test cases by updating records in a database.  Or you might even create a custom tool that would generate millions of different test cases by simply populating records in a database.

In our example, the test case looks like the following:

1. Log into the console of the DUT and get to the appropriate state for issuing commands
2. Connect to the database
3. Issue a query (passed in as a parameter to the test case) for the command records, and get the first record in the resultset
4. While there is a record available:
   • Get the command string to send from the record
   • Send that command to the DUT
   • Analyze the response using a query specified in the database record and an assertion also contained in the record
   • Get the next record from the database, if any
5. Log out of the console

iTest makes it easy to create this generic test case.  Perhaps you will use capture.  You will open a telnet session to your DUT.  You will open a Database Client session to your database in which you have added records defining your first test case.  HINT: The Database Client has a checkbox that will cause it to fetch one record at a time – which is very convenient in cases like this where you’ll be taking a series of actions based on the information in each record.  After you get the first record, you then click on the “Next Record” button to get subsequent records.  Your database query as it will appear in the final test case might look something like:  SELECT command, query, assertion FROM cliTestRecords WHERE testId = [param testId] ORDER BY orderIndex

In your test case, you would store the result of the database Execute and Next steps in a variable – perhaps you’ll call it “testRecord”.  You will have a step in your telnet session to your DUT whose command is set to [query testRecord command()].  It will have an analysis rule whose query is set to [query testRecord query()] and whose assertion is set to [query testRecord assertion()].

Now this one automated test can be used to run lots of lots of different test cases that are all based on the idea of sending a sequence of CLI commands and checking the value of a certain query in the response to that command.  To create a new test case, you would just add records to the database with your preferred sequence of commands and analysis – and use a unique testId in each of the records.

You can also do data-driven testing using a file as input instead of using a database.  In this case, you would probably use iTest’s readFile step to read in the information from your file.  Depending on the format of the file, you can then directly query that data.  Or you may need to add a response map so that you can query it.  In this case, rather than using a While loop in your test (iterating over database records until there are no more), you will probably use a For loop to iterate over the number of rows in the table of results from mapping your data file.  But in either case, the overall approach is similar.

The example I discussed above is obviously very simplistic.  But once you start to recognize the benefits of this kind of approach, you will start to see lots of different ways to exploit this approach.  For example, you could use data-driven testing to drive you through a web test – where the database records tell you about the navigation to work through.  And so on.

I do caution, however, that data-driven testing can be taken too far.  It is really good when you need a lot of different test cases that all follow the same pattern.  But you’ll probably find that it isn’t suitable for scenario testing, for example, where no two tests have a lot in common.  I have seen cases where a test developer became overly enamored with data-driven testing and the end result was a very confusing set of complex tests that were very difficult to understand because a lot of the information was split between the test script and the database.  So be careful about where to use this concept.

I also want to caution against using data-driven testing when you are really just trying to check all possibilities for a given test parameter.  Suppose, for example, that you want to check that the DUT will respond correctly regardless of what length of string you enter into a certain field.  In this case, data-driven testing is overkill.  It is much simpler to just define a loop in your test case and to programmatically vary the value of that field in each loop.  For example, if you want to check all possible string lengths from 1 to 100, you can add a For loop where on each loop, you append one more character to a string variable and use that variable to set the field’s value each time. 

Data-driven testing can be an extremely valuable asset in your test automation arsenal.  And iTest makes it quite easy to automate data-driven tests without having to do complex database programming.

---

Kingston Duffie is the founder and CTO of Fanfare. Learn more about Kingston >>