$ cat test_test.py
import pytest
EXPECTED = 2
XFAIL = 3
@pytest.mark.parametrize(
"result",
[
1,
pytest.param(EXPECTED, marks=pytest.mark.xfail(strict=True)),
pytest.param(XFAIL, marks=pytest.mark.xfail(strict=True)),
],
)
def test_increment(result):
assert result == EXPECTED
$ pytest test_test.py
========================== test session starts ==========================
platform linux -- Python 3.8.10, pytest-7.4.1, pluggy-1.3.0
configfile: pyproject.toml
plugins: mypy-plugins-3.0.0, dash-2.13.0, pudb-0.7.0, mypy-testing-0.1.1
collected 3 items
test_test.py FFx [100%]
=============================== FAILURES ================================
___________________________ test_increment[1] ___________________________
result = 1
@pytest.mark.parametrize(
"result",
[
1,
pytest.param(EXPECTED, marks=pytest.mark.xfail(strict=True)),
pytest.param(XFAIL, marks=pytest.mark.xfail(strict=True)),
],
)
def test_increment(result):
> assert result == EXPECTED
E assert 1 == 2
test_test.py:16: AssertionError
___________________________ test_increment[2] ___________________________
[XPASS(strict)]
======================== short test summary info ========================
FAILED test_test.py::test_increment[1] - assert 1 == 2
FAILED test_test.py::test_increment[2]
===================== 2 failed, 1 xfailed in 0.10s ======================
I would like to be able:
xfailif-f it does not matchout: |, but matchesxfail: |(imaginary attribute), but continuesxpassif-f it matchesout: |(preferably strict, since it will lead to actually noticing in CI/CDfailThis is clearly wrong:
https://github.com/stdedos/matplotlib-stubs-hoel/actions/runs/6221692521/job/16884150094
And I'd like to:
expected_failmeans "negative test" (instead of pytest'sxfail)This is (hopefully) illustrated by this example: