Skip to content

0.4.2 test failures: tests/integration/test_validators.py::TestOAS{30,31}Validator::test_format_checkers #61

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
mgorny opened this issue Jan 31, 2023 · 1 comment · Fixed by #62

Comments

@mgorny
Copy link

mgorny commented Jan 31, 2023

When running the test suite in 0.4.2, I get the following failures:

========================================================= test session starts =========================================================
platform linux -- Python 3.10.9, pytest-6.2.5, py-1.11.0, pluggy-1.0.0 -- /tmp/openapi-schema-validator/.venv/bin/python
cachedir: .pytest_cache
rootdir: /tmp/openapi-schema-validator, configfile: pyproject.toml
plugins: flake8-1.1.0, cov-3.0.0
collected 58 items                                                                                                                    

tests/integration/test_validators.py::TestOAS30Validator::test_format_checkers FAILED
tests/integration/test_validators.py::TestOAS30ValidatorValidate::test_null[boolean] PASSED
tests/integration/test_validators.py::TestOAS30ValidatorValidate::test_null[array] PASSED
tests/integration/test_validators.py::TestOAS30ValidatorValidate::test_null[integer] PASSED
tests/integration/test_validators.py::TestOAS30ValidatorValidate::test_null[number] PASSED
tests/integration/test_validators.py::TestOAS30ValidatorValidate::test_null[string] PASSED
tests/integration/test_validators.py::TestOAS30ValidatorValidate::test_nullable_untyped[True] PASSED
tests/integration/test_validators.py::TestOAS30ValidatorValidate::test_nullable_untyped[False] PASSED
tests/integration/test_validators.py::TestOAS30ValidatorValidate::test_nullable[boolean] PASSED
tests/integration/test_validators.py::TestOAS30ValidatorValidate::test_nullable[array] PASSED
tests/integration/test_validators.py::TestOAS30ValidatorValidate::test_nullable[integer] PASSED
tests/integration/test_validators.py::TestOAS30ValidatorValidate::test_nullable[number] PASSED
tests/integration/test_validators.py::TestOAS30ValidatorValidate::test_nullable[string] PASSED
tests/integration/test_validators.py::TestOAS30ValidatorValidate::test_nullable_enum_without_none PASSED
tests/integration/test_validators.py::TestOAS30ValidatorValidate::test_nullable_enum_with_none PASSED
tests/integration/test_validators.py::TestOAS30ValidatorValidate::test_string_format_byte_valid[c3RyaW5n0] PASSED
tests/integration/test_validators.py::TestOAS30ValidatorValidate::test_string_format_byte_valid[c3RyaW5n1] PASSED
tests/integration/test_validators.py::TestOAS30ValidatorValidate::test_string_format_byte_invalid[string0] PASSED
tests/integration/test_validators.py::TestOAS30ValidatorValidate::test_string_format_byte_invalid[string1] PASSED
tests/integration/test_validators.py::TestOAS30ValidatorValidate::test_allof_required PASSED
tests/integration/test_validators.py::TestOAS30ValidatorValidate::test_required PASSED
tests/integration/test_validators.py::TestOAS30ValidatorValidate::test_read_only PASSED
tests/integration/test_validators.py::TestOAS30ValidatorValidate::test_write_only PASSED
tests/integration/test_validators.py::TestOAS30ValidatorValidate::test_required_read_only PASSED
tests/integration/test_validators.py::TestOAS30ValidatorValidate::test_required_write_only PASSED
tests/integration/test_validators.py::TestOAS30ValidatorValidate::test_oneof_required PASSED
tests/integration/test_validators.py::TestOAS30ValidatorValidate::test_oneof_discriminator[oneOf] PASSED
tests/integration/test_validators.py::TestOAS30ValidatorValidate::test_oneof_discriminator[anyOf] PASSED
tests/integration/test_validators.py::TestOAS30ValidatorValidate::test_oneof_discriminator[allOf] PASSED
tests/integration/test_validators.py::TestOAS30ValidatorValidate::test_nullable_ref[True] PASSED
tests/integration/test_validators.py::TestOAS30ValidatorValidate::test_nullable_ref[False] PASSED
tests/integration/test_validators.py::TestOAS30ValidatorValidate::test_nullable_schema_combos[True-oneOf-None is not valid under any of the given schemas] PASSED
tests/integration/test_validators.py::TestOAS30ValidatorValidate::test_nullable_schema_combos[True-anyOf-None is not valid under any of the given schemas] PASSED
tests/integration/test_validators.py::TestOAS30ValidatorValidate::test_nullable_schema_combos[True-allOf-None for not nullable] PASSED
tests/integration/test_validators.py::TestOAS30ValidatorValidate::test_nullable_schema_combos[False-oneOf-None is not valid under any of the given schemas] PASSED
tests/integration/test_validators.py::TestOAS30ValidatorValidate::test_nullable_schema_combos[False-anyOf-None is not valid under any of the given schemas] PASSED
tests/integration/test_validators.py::TestOAS30ValidatorValidate::test_nullable_schema_combos[False-allOf-None for not nullable] PASSED
tests/integration/test_validators.py::TestOAS31Validator::test_format_checkers FAILED
tests/integration/test_validators.py::TestOAS30ReadWriteValidatorValidate::test_read_only PASSED
tests/integration/test_validators.py::TestOAS30ReadWriteValidatorValidate::test_write_only PASSED
tests/integration/test_validators.py::TestOAS30ReadWriteValidatorValidate::test_required_read_only PASSED
tests/integration/test_validators.py::TestOAS30ReadWriteValidatorValidate::test_required_write_only PASSED
tests/integration/test_validators.py::TestOAS31ValidatorValidate::test_null[boolean] PASSED
tests/integration/test_validators.py::TestOAS31ValidatorValidate::test_null[array] PASSED
tests/integration/test_validators.py::TestOAS31ValidatorValidate::test_null[integer] PASSED
tests/integration/test_validators.py::TestOAS31ValidatorValidate::test_null[number] PASSED
tests/integration/test_validators.py::TestOAS31ValidatorValidate::test_null[string] PASSED
tests/integration/test_validators.py::TestOAS31ValidatorValidate::test_nullable[boolean] PASSED
tests/integration/test_validators.py::TestOAS31ValidatorValidate::test_nullable[array] PASSED
tests/integration/test_validators.py::TestOAS31ValidatorValidate::test_nullable[integer] PASSED
tests/integration/test_validators.py::TestOAS31ValidatorValidate::test_nullable[number] PASSED
tests/integration/test_validators.py::TestOAS31ValidatorValidate::test_nullable[string] PASSED
tests/integration/test_validators.py::TestOAS31ValidatorValidate::test_schema_validation PASSED
tests/integration/test_validators.py::TestOAS31ValidatorValidate::test_schema_ref PASSED
tests/integration/test_validators.py::TestOAS31ValidatorValidate::test_array_prefixitems[value0] PASSED
tests/integration/test_validators.py::TestOAS31ValidatorValidate::test_array_prefixitems[value1] PASSED
tests/integration/test_validators.py::TestOAS31ValidatorValidate::test_array_prefixitems_invalid[value0] PASSED
tests/unit/test_shortcut.py::ValidateTest::test_validate_does_not_mutate_schema_adding_nullable_key PASSED

============================================================== FAILURES ===============================================================
_______________________________________________ TestOAS30Validator.test_format_checkers _______________________________________________

self = <test_validators.TestOAS30Validator object at 0x7f0996aea470>

    def test_format_checkers(self):
>       assert set(OAS30Validator.FORMAT_CHECKER.checkers.keys()) == set([
            # standard formats
            "int32", "int64", "float", "double", "byte", "binary",
            "date", "date-time", "password",
            # extra formats
            "uuid", "regex",
            "ipv4", "ipv6", "email", "idn-email", "time"
        ])
E       AssertionError: assert {'binary',\n 'byte',\n 'date',\n 'double',\n 'email',\n 'float',\n 'idn-email',\n 'idn-hostname',\n 'int32',\n 'int64',\n 'ipv4',\n 'ipv6',\n 'password',\n 'regex',\n 'time',\n 'uuid'} == {'binary',\n 'byte',\n 'date',\n 'date-time',\n 'double',\n 'email',\n 'float',\n 'idn-email',\n 'int32',\n 'int64',\n 'ipv4',\n 'ipv6',\n 'password',\n 'regex',\n 'time',\n 'uuid'}
E         Extra items in the left set:
E         'idn-hostname'
E         Extra items in the right set:
E         'date-time'
E         Full diff:
E           {
E            'binary',
E            'byte',
E            'date',
E         -  'date-time',
E            'double',
E            'email',
E            'float',
E            'idn-email',
E         +  'idn-hostname',
E            'int32',
E            'int64',
E            'ipv4',
E            'ipv6',
E            'password',
E            'regex',
E            'time',
E            'uuid',
E           }

self       = <test_validators.TestOAS30Validator object at 0x7f0996aea470>

tests/integration/test_validators.py:17: AssertionError
_______________________________________________ TestOAS31Validator.test_format_checkers _______________________________________________

self = <test_validators.TestOAS31Validator object at 0x7f09969c2920>

    def test_format_checkers(self):
>       assert set(OAS31Validator.FORMAT_CHECKER.checkers.keys()) == set([
            # standard formats
            "int32", "int64", "float", "double", "password",
            # extra formats
            "date", "date-time", "uuid", "regex",
            "ipv4", "ipv6", "email", "idn-email", "time"
        ])
E       AssertionError: assert {'date',\n 'double',\n 'email',\n 'float',\n 'idn-email',\n 'idn-hostname',\n 'int32',\n 'int64',\n 'ipv4',\n 'ipv6',\n 'password',\n 'regex',\n 'time',\n 'uuid'} == {'date',\n 'date-time',\n 'double',\n 'email',\n 'float',\n 'idn-email',\n 'int32',\n 'int64',\n 'ipv4',\n 'ipv6',\n 'password',\n 'regex',\n 'time',\n 'uuid'}
E         Extra items in the left set:
E         'idn-hostname'
E         Extra items in the right set:
E         'date-time'
E         Full diff:
E           {
E            'date',
E         -  'date-time',
E            'double',
E            'email',
E            'float',
E            'idn-email',
E         +  'idn-hostname',
E            'int32',
E            'int64',
E            'ipv4',
E            'ipv6',
E            'password',
E            'regex',
E            'time',
E            'uuid',
E           }

self       = <test_validators.TestOAS31Validator object at 0x7f09969c2920>

tests/integration/test_validators.py:474: AssertionError
--------------------------------- generated xml file: /tmp/openapi-schema-validator/reports/junit.xml ---------------------------------

---------- coverage: platform linux, python 3.10.9-final-0 -----------
Name                                      Stmts   Miss Branch BrPart  Cover   Missing
-------------------------------------------------------------------------------------
openapi_schema_validator/__init__.py         13      0      0      0   100%
openapi_schema_validator/_format.py          41      7      6      1    79%   19, 25, 29, 36, 43-46
openapi_schema_validator/_types.py           12      0      0      0   100%
openapi_schema_validator/_validators.py     125     25     72     10    73%   37, 47-53, 62, 131, 133->exit, 146-150, 160, 184, 186->185, 188->195, 205, 207->206, 209->213, 222-237
openapi_schema_validator/shortcuts.py        13      1      2      1    87%   23
openapi_schema_validator/validators.py       37      0      6      0   100%
-------------------------------------------------------------------------------------
TOTAL                                       241     33     86     12    80%
Coverage XML written to file reports/coverage.xml

======================================================= short test summary info =======================================================
FAILED tests/integration/test_validators.py::TestOAS30Validator::test_format_checkers - AssertionError: assert {'binary',\n 'byte',\...
FAILED tests/integration/test_validators.py::TestOAS31Validator::test_format_checkers - AssertionError: assert {'date',\n 'double',\...
==================================================== 2 failed, 56 passed in 0.55s =====================================================

I can reproduce it reliably in a venv via:

git clone https://github.com/p1c2u/openapi-schema-validator/
cd openapi-schema-validator/
git checkout 0.4.2
poetry install
pytest -vv

On a production system, I get even more mismatches:

_______________________________________________ TestOAS30Validator.test_format_checkers _______________________________________________

self = <test_validators.TestOAS30Validator object at 0x7f6ee63cceb0>

    def test_format_checkers(self):
>       assert set(OAS30Validator.FORMAT_CHECKER.checkers.keys()) == set([
            # standard formats
            "int32", "int64", "float", "double", "byte", "binary",
            "date", "date-time", "password",
            # extra formats
            "uuid", "regex",
            "ipv4", "ipv6", "email", "idn-email", "time"
        ])
E       AssertionError: assert {'color', 'byte', 'idn-hostname', 'email', 'hostname', 'float', 'ipv6', 'idn-email', 'time', 'double', 'uri-template', 'password', 'uri-reference', 'duration', 'regex', 'uri', 'date', 'ipv4', 'int32', 'iri-reference', 'date-time', 'relative-json-pointer', 'iri', 'binary', 'uuid', 'json-pointer', 'int64'} == {'byte', 'time', 'ipv6', 'date', 'regex', 'double', 'email', 'float', 'int32', 'binary', 'password', 'uuid', 'ipv4', 'date-time', 'in
t64', 'idn-email'}
E         Extra items in the left set:
E         'color'
E         'uri'
E         'relative-json-pointer'
E         'idn-hostname'
E         'hostname'
E         'iri'
E         'uri-template'
E         'uri-reference'
E         'iri-reference'
E         'duration'
E         'json-pointer'
E         Full diff:
E           {
E            'binary',
E            'byte',
E         +  'color',
E            'date',
E            'date-time',
E            'double',
E         +  'duration',
E            'email',
E            'float',
E         +  'hostname',
E            'idn-email',
E         +  'idn-hostname',
E            'int32',
E            'int64',
E            'ipv4',
E            'ipv6',
E         +  'iri',
E         +  'iri-reference',
E         +  'json-pointer',
E            'password',
E            'regex',
E         +  'relative-json-pointer',
E            'time',
E         +  'uri',
E         +  'uri-reference',
E         +  'uri-template',
E            'uuid',
E           }

self       = <test_validators.TestOAS30Validator object at 0x7f6ee63cceb0>

tests/integration/test_validators.py:17: AssertionError
_______________________________________________ TestOAS31Validator.test_format_checkers _______________________________________________

self = <test_validators.TestOAS31Validator object at 0x7f6ee642d6f0>

    def test_format_checkers(self):
>       assert set(OAS31Validator.FORMAT_CHECKER.checkers.keys()) == set([
            # standard formats
            "int32", "int64", "float", "double", "password",
            # extra formats
            "date", "date-time", "uuid", "regex",
            "ipv4", "ipv6", "email", "idn-email", "time"
        ])
E       AssertionError: assert {'color', 'idn-hostname', 'email', 'hostname', 'float', 'ipv6', 'idn-email', 'time', 'double', 'uri-template', 'password', 'uri-reference', 'duration', 'regex', 'uri', 'date', 'ipv4', 'int32', 'iri-reference', 'date-time', 'relative-json-pointer', 'iri', 'uuid', 'json-pointer', 'int64'} == {'time', 'ipv6', 'date', 'regex', 'double', 'email', 'float', 'password', 'int32', 'uuid', 'ipv4', 'date-time', 'int64', 'idn-email'}
E         Extra items in the left set:
E         'color'
E         'uri'
E         'relative-json-pointer'
E         'idn-hostname'
E         'hostname'
E         'iri'
E         'uri-template'
E         'uri-reference'
E         'iri-reference'
E         'duration'
E         'json-pointer'
E         Full diff:
E           {
E         +  'color',
E            'date',
E            'date-time',
E            'double',
E         +  'duration',
E            'email',
E            'float',
E         +  'hostname',
E            'idn-email',
E         +  'idn-hostname',
E            'int32',
E            'int64',
E            'ipv4',
E            'ipv6',
E         +  'iri',
E         +  'iri-reference',
E         +  'json-pointer',
E            'password',
E            'regex',
E         +  'relative-json-pointer',
E            'time',
E         +  'uri',
E         +  'uri-reference',
E         +  'uri-template',
E            'uuid',
E           }

self       = <test_validators.TestOAS31Validator object at 0x7f6ee642d6f0>

tests/integration/test_validators.py:474: AssertionError
-- generated xml file: /tmp/portage/dev-python/openapi-schema-validator-0.4.2/work/openapi-schema-validator-0.4.2/reports/junit.xml ---
======================================================= short test summary info =======================================================
@p1c2u
Copy link
Collaborator

p1c2u commented Feb 1, 2023

Looks like test environment has more packages installed than just required ones. I will make a fix for that.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants