How to select a subset of tests in pytest using custom markers in parameters

Question

I wonder how to select a subset of my tests using custom pytest markers.

A simple test works as expected:

Code with one parameter marked

import pytest @pytest.mark.parametrize('a', [pytest.mark.my_custom_marker(0), 1]) @pytest.mark.parametrize('b', [0, 1]) def test_increment(a, b): pass 

If I only want to run the test marked with the 'my_custom_marker' sign

Output

 $ pytest test_machin.py -m my_custom_marker --collect-only platform linux2 -- Python 2.7.12, pytest-3.0.5, py-1.4.32, pluggy-0.4.0 rootdir: /home/mvelay/workspace/sandbox, inifile: plugins: hypothesis-3.6.0, html-1.12.0, xdist-1.15.0, timeout-1.0.0 collected 4 items <Module 'test_machin.py'> <Function 'test_increment[0-0]'> <Function 'test_increment[0-1]'> 

But as soon as I try to check a few checked parameters, I run into a problem

Code with two marked parameters

 import pytest @pytest.mark.parametrize('a', [pytest.mark.my_custom_marker(0), 1]) @pytest.mark.parametrize('b', [pytest.mark.my_custom_marker(0), 1]) def test_increment(a, b): pass 

Output

 $ pytest -m my_custom_marker test_machin.py --collect-only platform linux2 -- Python 2.7.12, pytest-3.0.5, py-1.4.32, pluggy-0.4.0 rootdir: /home/mvelay/workspace/sandbox, inifile: plugins: hypothesis-3.6.0, html-1.12.0, xdist-1.15.0, timeout-1.0.0 collected 4 items <Module 'test_machin.py'> <Function 'test_increment[0-0]'> <Function 'test_increment[0-1]'> <Function 'test_increment[1-0]'> 

I expected that only the combination [0-0] would be executed.

Is there any way to do this in an elegant way?

Answer 1

You can use two different markers as follows:

 import pytest @pytest.mark.parametrize('a', [pytest.mark.marker_a(0), 1]) @pytest.mark.parametrize('b', [pytest.mark.marker_b(0), 1]) def test_increment(a, b): pass 

And specify the label expression:

 $ pytest test_machin.py -m "marker_a and marker_b" --collect-only ============= test session starts =============================== platform darwin -- Python 2.7.10, pytest-3.0.5, py-1.4.32, pluggy-0.4.0 rootdir: /Users/yangchao/Code, inifile: collected 4 items <Module 'test_machin.py'> <Function 'test_increment[0-0]'> ============= 3 tests deselected ================================= ============= 3 deselected in 0.00 seconds ======================= 

Answer 2

The answer is: you are abusing Marks . Test runners are usually built to test the entire permutation space. You will find that everything in pytest focused on building a Cartesian product. This is because people usually want to test as much as possible while writing the least amount of code. Looking at this, you will find that you are an anomaly and that is the problem XY

This does not mean that there is no answer. So what do you really want to do here? Here is what you ask ...

I wonder how to select a subset of my tests using custom pytest markers.

But I wonder what you say ...

I want to run my test with different sets of parameters on demand using labels.

It is much easier to do. Here it is in its simplest form ...

 import pytest @pytest.mark.A def test_add(): a = 2 b = 1 result = add(a,b) assert result == 3 @pytest.mark.B def test_add_with_failure(): a = 1 b = 2 add(a,b) #expect it to fail def add(a, b): assert a>b return a+b 

Now each of these test suites can be invoked from the command line with labels.

py.test -m A tests.py py.test -m B tests.py

It was an original work with a tag. To find a specific test or group of tests and run it (them). If you are trying to do something other than looking for a specific test to run, you will struggle with the framework.

If you want something a little more canonical pytest , you can take part of the template and pytest it. This allows you to add a lot of permutations. So ...

 @pytest.fixture(params=[(2,1,3)]) def always_pass(request): return request.param @pytest.mark.A def test_add(always_pass): a, b, expected = always_pass result = add(a,b) assert result == expected 

Or you can continue to work and collect all the tests in one large, complex tool ...

 @pytest.fixture(params=[ (2, 1, 3), (1, 2, 'fail') ]) def all_the_things(request): return request.param @pytest.mark.A def test_add(all_the_things): a, b, expected = all_the_things if expected == 'fail' with pytest.raises(AssertionError): result = add(a,b) else: assert result == expected 

The point here is that the structure tells you something when it puts everything together this way. His saying "there is a better way." Do not fight with the frame, hug it! If you look closer at parameterize , looking at the principles described above, you will see that this is indeed a function that allows you to have a finer, granular composition. Thus, you can build the Cartesian product of the individual parameters using styling for decorators and labels.