Beginner's question about continuous integration and selenium tests,

Question

I am very new to CI but I recently inherited the project when Team City was just implemented and I slowly hug it. One thing we would like to do is run some Selenium Tests as part of the build process. I have created selenium tests and successfully run them through the nunit console on my development machine. The build server builds the project and then deploys it (the web forms application as it happens) on the staging server.

Before each testing of selenium, we set the database in a known state, that is, we must have only certain records - thus, each test is independent of the others. The problem is that the intermediate server will be used by real "human" testers, so this can cause problems with the persistence of the reset database (records are deleted, etc.). The question is, should I really deploy the application on the virtual on the build server and run selenium tests against this and only deploy on the intermediate server if those tests pass?

Or did I realize that this is absolutely wrong? If so, how do you do this in your organization?

Answer 1

I suggest that you do not mix automatic and manual testing , allowing your testers to access the server installed for your automatic tests. This can cause false negatives in both automatic and manual tests. These "errors" are vague and more than likely never reproducible (very bad news). This will cause you a lot of unnecessary “error reporting” and lead to crashes.

So here is what you can do ...

In addition to your current setup, you can create an additional phased server for your manual testers. This is the smallest you have to do. You should probably create several of them, one for each tester .

And here comes the rant ...

In my current project, we recently found out that our testers (we had ~ 10 of them) reused a single server. They argued that since our application would have several simultaneous users, it would be nice that by checking individual functions, they also test how these functions work when several users work on the same server. WRONG!

If several users are concerned, there should be test cases for specific problems. If functionality number 1 can interfere with functionality number 2, it should be specifically tested and not just "tested by luck."

Before this was explained to our manual testers, we got a lot of false error messages due to the fact that one tester simply stepped on the other testers of the tester. (for example, tester1 deleted the entry entered by tester2 into the system, etc.). This created many unnecessary error reports, and these errors were never reproduced.

Sorry for the ranting, I was hoping this still helps :)