-
Notifications
You must be signed in to change notification settings - Fork 1.3k
Improve some code coverage #1350
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Comments
Could you elaborate more about parts of pipeline and available configurations? |
Also what components of the system need the code coverage the most? |
Hi @jcob-sikorski, So in our pipeline, we select different components with different hyperparameters and then evaluate them. The base pipelines folder is where you can see most of the heavy lifting done, with the Each component will define it's Can you access https://app.codecov.io/gh/automl/auto-sklearn/branch/development? Here you'll see the coverage of the entire codebase (for the development branch). The sunburst diagram plot is probably the best to get a quick overview. The most notable areas of the system lacking coverage is You're free to choose wherever you'd like of course but since you mentioned the pipeline I assume you might prefer to look there? Looking at the Testing these things would require reading around and getting an idea of what's going on, I would be happy to answer any question you have along the way :) Providing a large overview of how the details of the system work might instead be too much. Best, |
How can I take first bite at this? This codebase is huge. Could you propose me some areas where I can start? Where I can find some easy examples of tests? |
A simple place that is very standalone would be to provide some tests for the You can look at In any case, your test file might look like: import pytest
from autosklearn.functional import intersection
def test_intersection_empty_iterables():
"""
Expects
-------
* Doing an intersection on an empty list should be empty
"""
a = []
assert intersection(a) == set()
def test_intersection_one_list():
"""
Expects
-------
* I actually don't know what it does, find out and see if it makes sense
"""
a = ["a", "b", "c"]
assert intersection(a) == ?
def test_intersection_no_overlap():
a = {"a", "b", "c"}
b = {"d"}
assert intersection(a, b) == set()
def test_intersection_with_overlap():
a = {"a", "b", "c"}
b = {"a"}
assert intersection(a, b) == {"a"} It would be good to take a look at the functions, try understand what they're doing and then come up with things that would be interesting to test. There are of course other functions in there and things like You can check the line coverage of your tests using: pytest --cov test/test_util/test_functional.py I would also check out the CONTRIBUTING.md which walks you through a lot of the setup for installing for dev and making a PR :) Best, |
Can I give a try on these tests? |
Uh oh!
There was an error while loading. Please reload this page.
This is an ongoing issue we need to work on or would gladly accept contributions for anyone looking to get some experience with testing open-source.
We have some guaranteed randomness with the configuration we tests which causes the code coverage to fluctuate. This causes issue in that we have a simple change such as a docstring change DOC: rename OSX -> macOS as it is the new name #1349 causes our code coverage test to fail, due to a
-0.03%
change in coverage.In general, we are hovering around
88%
coverage, these fluctuations are relatively minor and in no way account for the other12%
. I would roughly estimate that we could quite easily reach5%
with ensuring more components of the system are tested. I estimate the remaining5%
is testing the various branches ofif/else
, error validation and maybe~1-2%
untestable code we do not need to be concerned with (abstract classes etc...)Please check out our contribution guide,
pytest-coverage
and our code coverage statistics reported from our automatic unittests if you'd like to get started!The text was updated successfully, but these errors were encountered: