May use Specflow script with Visual Studio 2013 Load Test

Question

I plan to reuse existing Specflow scripts (currently used for acceptance and automatic testing) to test the VS load, and also to avoid duplication and extra work. The spectrometer works great for these tests, because it runs them once, but in the context of the load test, when it runs each Specflow script more than once and in parallel it encounters problems and errors, and with a lot of users it gets more

These errors can damage part of the test, which in the end creates an incorrect test result, for example, using one Specflow script as a test script with a load test of 20 users and a time period of 2 minutes can cause 50 errors similar to the following. Thus, the test result shows that a particular script runs 200 times when 150 have passed, and 50 failed tests and failures are caused by Specflow errors. In the context of a load test, this result is completely wrong and wrong, because the test itself has problems.

Error message:

ScenarioTearDown threw exception. System.NullReferenceException: System.NullReferenceException: Object reference not set to an instance of an object. TechTalk.SpecFlow.Infrastructure.TestExecutionEngine.HandleBlockSwitch(ScenarioBlock block) TechTalk.SpecFlow.Infrastructure.TestExecutionEngine.ExecuteStep(StepInstance stepInstance) TechTalk.SpecFlow.Infrastructure.TestExecutionEngine.Step(StepDefinitionKeyword stepDefinitionKeyword, String keyword, String text, String multilineTextArg, Table tableArg) TechTalk.SpecFlow.TestRunner.Then(String text, String multilineTextArg, Table tableArg, String keyword) 

After some research, it seems that Specflow cannot generate and run the parallel script that caused this conflict and did not pass any test, but I also have some doubts about this and you are trying to find out if there is any workaround about this, or if I miss something and interestingly, can Specflow scripts be used to test the load at all?

Answer 1

I understand that I want to reuse your tests for load testing (Do not Repeat Yourself), but the load test has a completely different purpose than acceptance tests. Load tests should take realistic usage scenarios every day and throw more and more users at them. For this reason, I urge you to conduct load tests separately from your acceptance and automated tests. They really test different things.

Load tests should verify application performance with high usage for daily scenarios, and acceptance and automatic tests ensure that the application operates in accordance with the specification.

Load loading is the process of applying for a system or device and measuring its response. Load testing is performed to determine the behavior of the system under both normal and expected maximum load conditions. This helps determine the maximum working capacity of the application, as well as any bottlenecks, and determine which element causes the degradation.

Source: Wikipedia: Download Testing

Acceptance test - This is a formal description of the behavior of a software product, usually expressed as an example or use case ... For many Agile teams, acceptance testing is the main form of functional specification; sometimes this is the only formal expression of business requirements. In other cases, they simply complement a technical document based on a less specific Agile technique or formalism, such as use cases or more descriptive documents.

Source: Agile Aliance: Acceptance Testing

These are different things, so the tests and the test framework should also be different. You are not really “repeating” by keeping a separate set for stress tests.

As for the technical reason why this fails? SpecFlow, when launched using a regular test runner, Visual Studio was not created to run tests in parallel. There are parallel test runners, but most of them are paid programs.