Make tests using random numbers more reproducible

[pytestbot]

Originally reported by: Nils Werner (BitBucket: nils-werner, GitHub: nils-werner)

---

I recently was thinking about what to do to make the use of random numbers more reproducible. Currently, when a test using a random number fails, there is no way of backtracking what random number seed was actually used so that the test can be reproduced.

So...

1. While using tox I noticed it always sets PYTHONHASHSEED itself (to a random value, but printing that value) to ensure people can reproduce tests failing due to anything that has to do with that environment variable
2. While using pytest-django I noticed they always set the database back into its initial state and require the tests to be marked (or have a db fixture) to touch the dabase.

This brought me to the idea of controlling the random module a bit more to:

• Set the seed
  • to a constant value, or
  • to a random value, printing that value in case of errors, or
  • reset the seed after each test, or
• Allowing the test to set a seed without affecting other tests

For example, currently you can do

import pytest
import my_module
import random

def test_something():
    assert random.random() > 0.5

Which gives absolutely no information why and in what situations the (rather idiotic) test is failing.

Or, when you have two tests

import pytest
import my_module
import random

def test_something():
    assert random.random() > 0.5

def test_something_else():
    assert random.random() > 0.5

The order of execution changes the values each test sees (particularily troubling with xdist I guess)

ideally, the above code should fail and should be replaced with

import pytest
import my_module

def test_something(random):
    assert random.random() > 0.5

That way pytest could always prepare random to be in the identical state for each tests and remember the seed value and print it, hinting at the fact that there are random numbers at play here.

I haven't given too much thought about how to achieve this behaviour (monkeypatching the random module I guess) but first wanted to hear what you guys think.

---

• Bitbucket: https://bitbucket.org/pytest-dev/pytest/issue/667

[pytestbot]

Original comment by Nils Werner (BitBucket: nils-werner, GitHub: nils-werner):

---

fwiw, I am now always defining and using

import numpy
import random as rand

@pytest.fixture
def random():
    rand.seed(0)
    numpy.random.seed(0)

in my conftest.py

[RonnyPfannschmidt]

closing for now - a plugin should provide experimental results also its a real problem to have this as a opt-out/opt-in