TestNG & Selenium: Separate tests in "groups" run in each group

Question

We use TestNG and Selenium WebDriver to test our web application.

Now our problem is that we often carry out several tests that must be performed in a certain order, for example:

• enter the application
• enter some data
• To change the data
• check the display is correct.

Now it’s obvious that these tests should be performed in this exact order.

At the same time, we have many other tests that are completely independent of the list of tests above.

So, we would like to somehow put the tests into “groups” (not necessarily groups in the sense of TestNG), and then run them so that:

• tests within the same group always work together and in the same order
• but different test "groups" in general can run in any order

The second point is important because we want to avoid dependencies between tests in different groups (therefore, different test "groups" can be used and developed independently).

Is there a way to achieve this with TestNG?

The solutions we tried

• First, we simply put the tests that belong together in the same class, and used dependsOnMethods to get them to work in the correct order. This was used for testing in TestNG V5, but in V6 TestNG sometimes alternates tests from different classes (subject to the order imposed by dependsOnMethods ). There seems to be no way to tell TestNG "Always run tests from the same class together."
• We examined the ability to record a method interceptor. . However, this has the disadvantage that running tests from within the IDE becomes more complicated (because the interceptor will not directly access the test in the class). Additionally, tests using dependsOnMethods cannot be ordered by the interceptor, so we would have to stop using this. We will probably have to create our own annotation to indicate the order, and we would like to use the standard features of TestNG as much as possible.
• TestNG docs suggest using a preserve-order to order tests. This looks promising, but only works if you list each test method separately, which seems redundant and difficult to maintain.

Is there a better way to achieve this?

I am also open to any other suggestions on how to handle tests that are built on top of each other, without having to impose a general order on all tests.

PS

alanning replies that we could just keep all the tests independent by doing the necessary setup in each test. This is basically a good idea (and some tests do this), but sometimes we need to test the full workflow, and each step depends on all the previous steps (as in my example). To do this with "independent" tests will mean repeating the same multi-stage setup over and over again, and this will make our already slower tests even slower. Instead of three tests:

• Test 1: login to the application
• Test 2: enter some data
• Test 3: edit the data

we would get

• Test 1: login to the application
• Test 2: enter the application, enter some data
• Test 3: enter the application, enter some data, edit the data, etc.

In addition to an unnecessary increase in testing time, this also seems unnatural - you need to model the workflow as a series of tests.

If there is no other way, this is probably the way we do it, but we are looking for a better solution without repeating the same installation calls.

Answer 1

You mix "functionality" and "test". Separating them will solve your problem.

For example, create a helper class / method that will follow the steps to log in, and then call that class / method in your test login mode and all other tests that require user login.

Your other tests should not really rely on your Logon "Test", just the login class / method.

If subsequent login modifications result in an error during the login process, all tests that rely on the Login helper class / method will still fail as expected.

Update:

Turns out it already has a name, a page object template. Here is a page with Java examples using this template:

http://code.google.com/p/selenium/wiki/PageObjects

Answer 2

Try depending on the group and method dependent. Add all methods in one class to one group. for instance

 @Test(groups={"cls1","other"}) public void cls1test1(){ } @Test(groups={"cls1","other"}, dependsOnMethods="cls1test1", alwaysrun=true) public void cls1test2(){ } In class 2 @Test(groups={"cls2","other"}, dependsOnGroups="cls1", alwaysrun=true) public void cls2test1(){ } @Test(groups={"cls2","other"}, dependsOnMethods="cls2test1", dependsOnGroups="cls1", alwaysrun=true) public void cls2test2(){ } 

Answer 3

There is a simple (albeit hacked) workaround for this if you like your first approach:

First, we simply put the tests that belong to each other in the same class, and use the dependOnMethods functions to execute them in the correct order. This was used for testing in TestNG V5, but in V6 TestNG sometimes alternates tests from different classes (subject to the order imposed by dependOnMethods). There seems to be no way to tell TestNG "Always run tests from the same class together."

We had a similar problem: we needed our tests to run in classes, because we could not guarantee that the test classes do not interfere with each other.

This is what we did: put a

 @Test( dependsOnGroups= { "dummyGroupToMakeTestNGTreatThisAsDependentClass" } ) 

Annotations to an abstract test class or interface that inherit all your tests. This will put all your methods in the “first group” (the group described in this section , not the TestNG groups). Within groups, ordering is cool. Thanks to Cedric Beust, he provided a very quick answer for this .

Edit: The dummyGroupToMakeTestNGTreatThisAsDependentClass group really should exist, but you can just add a dummy test case for this purpose.