Skip to content

RSA.VerifyData terminates with "Out of memory." on Linux when passing an all-zero modulus #78293

@Thealexbarney

Description

@Thealexbarney

Description

If an all-zero modulus is passed to RSA.VerifyData on Linux, the program will unexpectedly terminate instead of throwing an exception.

Reproduction Steps

using System.Security.Cryptography;

var param = new RSAParameters{Modulus = new byte[256], Exponent = new byte[3]};
using (var rsa = RSA.Create(param))
{
    rsa.VerifyData(new byte[512], new byte[256], HashAlgorithmName.SHA256, RSASignaturePadding.Pss);
}

Expected behavior

VerifyData throws a CryptographicException like it does on Windows.

Actual behavior

The program terminates with Out of memory. Command terminated by signal 6

Regression?

Issue occurs on .NET 6 and 7. Unsure about earlier versions.

Known Workarounds

No response

Configuration

I've tested running .NET 6.0 and 7.0 on both x64 Ubuntu 22.04 and Debian 11.

Other information

No response

Metadata

Metadata

Assignees

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions