Fix random state not being used for sampling configurations

[eddiebergman]

• Issue Configuration for SimpleClassificationPipelineTest.test_configurations_signed_data gives undeterministic error #1310 highlighted an in-deterministic test, giving us failures with a certain configuration (not solved).
• This is a result of SimpleClassificationPipeline not receiving random_state in the tests which in turn does not get passed to the ConfigSpace it creates.
• Searching further shows that the ConfigSpace used by Automl._create_search_space was not recieving a random_state either, not sure how this effects the starting samples that autosklearn would use.

This PR:

• Keeps random_state in tests as None so that further bad configurations can still surface, all be it randomly.
• Forwards on the random_state to Automl._create_search_space so the non-test code is deterministic.
• Add's documentation to the classification pipeline tests.

[mfeurer]

This all looks good, but is there a reason you only touched the classification pipeline test and not the regression ones, too?

[eddiebergman]

I labelled it with "PR: In Progress", I will do the regression ones as well, also I have to take it back out of our randomized test where we randomly select sample configurations too.

I'll send you a review request when it's ready

[eddiebergman]

So I ended up removing random_state from the tests again, this will slowly allow invalid configurations to bubble up and we deal with them while we can. This PR now only fixes the AutoML._create_search_space and adds some documentation to the tests.

[codecov]

Codecov Report

Merging #1329 (da17589) into development (3761f9b) will increase coverage by 0.40%.
The diff coverage is 100.00%.

@@               Coverage Diff               @@
##           development    #1329      +/-   ##
===============================================
+ Coverage        88.05%   88.46%   +0.40%     
===============================================
  Files              140      140              
  Lines            11163    11811     +648     
===============================================
+ Hits              9830    10449     +619     
- Misses            1333     1362      +29     

Impacted Files	Coverage Δ
autosklearn/util/pipeline.py	100.00% <100.00%> (+7.50%)	⬆️
...ine/components/classification/gradient_boosting.py	93.04% <0.00%> (-0.87%)	⬇️
...preprocessing/imputation/categorical_imputation.py	96.29% <0.00%> (-0.77%)	⬇️
...ning/optimizers/metalearn_optimizer/metalearner.py	96.34% <0.00%> (+0.23%)	⬆️
autosklearn/estimators.py	93.93% <0.00%> (+0.51%)	⬆️
autosklearn/pipeline/components/base.py	79.63% <0.00%> (+0.59%)	⬆️
...pipeline/components/regression/gaussian_process.py	97.91% <0.00%> (+0.61%)	⬆️
autosklearn/evaluation/abstract_evaluator.py	93.05% <0.00%> (+0.77%)	⬆️
...ata_preprocessing/categorical_encoding/encoding.py	97.36% <0.00%> (+0.93%)	⬆️
autosklearn/smbo.py	89.14% <0.00%> (+1.14%)	⬆️
... and 7 more

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 3761f9b...da17589. Read the comment docs.

[mfeurer]

This is now inconsistent. The random_state is not dropped in a few unit tests above.

[eddiebergman]

That's because this test relies an an accuracy score, it needs to be fixed as not all configurations will get 96%. The other tests are not specifically for metrics.