Programmatically collect or create tests using py.test

Question

I am writing a pytest plugin to order your tests before running them. I need a better way to write tests for my plugin.

I use pytest pytest_collection_modifyitems hook , and then changes itemsin place based on specific markers. The easiest way to check this would be to have an input list of pytest tests, pass them to my function and state that the output is in the correct order.

I am having problems collecting or creating input test lists. Input tests should be in the same format as pytest will be passed as itemsin hook pytest_collection_modifyitems, i.e. They must be instances _pytest.python.Functionwith appropriate markers.

I will accept answers for: 1) collecting tests using the existing Python module or 2) programmatically creating instances _pytest.python.Functionthat look like this would be passed in a hook pytest_collection_modifyitems. All that makes it easy to generate test data.

Answer 1

I suggest you go with a higher level of testing using testdirfixture, which is common to pytest plugins and is accessible through the plugin pytester:

pytest_plugins = "pytester"


def test_something(testdir):
    """Test some thing."""
    testdir.makepyfile("""
        import pytest

        @pytest.mark.some
        def test_some():
            assert True

        @pytest.mark.some_thing_else
        def test_some_thing_else():
            assert True
    """)
    result = testdir.runpytest('-vv')
    assert result.stdout.lines[-4:-2] == [
        u'test_something.py::test_some PASSED',
        u'test_something.py::test_some_thing_else PASSED'
    ]

so you can easily have multiple permutations of the tests and their markers in the source and asserting the output strings that you are asserting in the actual order.