Invalid serialization format when using FIPS provder raises InternalError

[dwoz]

Using an invalid serialization format when the FIPS provder is enable raises InternalError instead of ValueError. This changed between 41.0.7 and 42.0.0.

# cryptographybug.sh
TEST_SCRIPT='
from cryptography.hazmat.backends import default_backend
from cryptography.hazmat.primitives import serialization
from cryptography.hazmat.primitives.asymmetric import rsa
backend = default_backend()
assert backend._fips_enabled
size = 2048
e = 65537
passphrase = "abc123"
key = rsa.generate_private_key(e, size)
enc = serialization.BestAvailableEncryption(passphrase.encode())
key.private_bytes(
    encoding=serialization.Encoding.PEM,
    format=serialization.PrivateFormat.TraditionalOpenSSL,
    encryption_algorithm=enc,
)'
tdnf install -y python3-devel python3-virtualenv python3-xml gcc libffi-devel glibc-devel linux-api-headers binutils rust openssl-devel openssl-fips-provider
virtualenv venv
venv/bin/pip3 install --no-binary=':all:' --no-cache-dir cryptography==41.0.7
venv/bin/python3 -c "$TEST_SCRIPT"
venv/bin/pip3 install --no-binary=':all:' --no-cache-dir cryptography==42.0.0
venv/bin/python3 -c "$TEST_SCRIPT"

docker run -it photon:5.0 /bin/bash -c "$(cat cryptographybug.sh)"

cryptography 41.0.7 output

Traceback (most recent call last):
  File "<string>", line 12, in <module>
  File "//venv/lib/python3.11/site-packages/cryptography/hazmat/backends/openssl/rsa.py", line 496, in private_bytes
    return self._backend._private_key_bytes(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "//venv/lib/python3.11/site-packages/cryptography/hazmat/backends/openssl/backend.py", line 1322, in _private_key_bytes
    raise ValueError(

cryptography 42.0.0 output

Traceback (most recent call last):
  File "<string>", line 12, in <module>
cryptography.exceptions.InternalError: Unknown OpenSSL error. This error is commonly encountered
                        when another library is not cleaning up the OpenSSL error
                        stack. If you are using cryptography with another library
                        that uses OpenSSL try disabling it before reporting a bug.
                        Otherwise please file an issue at
                        https://github.com/pyca/cryptography/issues with
                        information on how to reproduce this. ([<OpenSSLError(code=50856204, lib=6, reason=524556, reason_text=unsupported)>, <OpenSSLError(code=50331782, lib=6, reason=134, reason_text=initialization error)>])

[reaperhulk]

Looks like we didn't port the ValueError branch for FIPS when we moved this code to rust (#9152). Perhaps we have a test that is too permissive (that branch was covered in tests).