OpenSSL: From FIPS 140-2 upstream to 140-3 downstream

Red Hat Enterprise Linux 9.0 and OpenSSL 3.0

During the development of Red Hat Enterprise Linux (RHEL) 9, we decided to switch to OpenSSL 3.0 even though we were not sure that it would be finalized early enough. This decision was made to significantly reduce our maintenance burden during the 10+ years of RHEL 9 support.

One of the anticipated changes in OpenSSL 3.0 was the new provider model. The provider that we were particularly interested in was the one implementing Federal Information Processing Standard (FIPS) algorithms and limitations. Extracting the FIPS-certified code to a separate module instead of instrumenting both libcrypto and libssl also significantly reduces maintenance efforts.

However, we came across a gap between the upstream solution and our productization requirements. The upstream was aiming for a FIPS 140-2 certification, but we needed the newer FIPS 140-3 certification with much stricter requirements.

Our patches: general approach

While we have to fix vulnerabilities and bugs, we avoid leaping to new major—or even minor—releases in the name of stability, backporting the most important fixes instead. We also have our own patches related to hardening so our distributions follow up-to-date recommendations about algorithms and protocols. All in all, OpenSSL in RHEL 9.0 currently carries more than 60 patches. Some of them are fixes for common vulnerabilities and exposures (CVEs) found after the release of the version we use. The rest are various hardening patches.

**Non-FIPS patches **

We have a set of requirements and features not related to FIPS to make RHEL more robust, consistent and up-to-date from a security point of view.

Crypto policies have been present in our distributions since the RHEL 8.0 release and allow consistent enforcement of safe crypto algorithms for all the low-level crypto libraries, so we patch OpenSSL to support them.

Also for RHEL 9, we’ve completed a system-wide project to stop using the SHA1 hash algorithm for digital signatures. The algorithm can’t be considered secure enough in the face of modern computer security threats, so we disable support of SHA1 in signatures by default. Unfortunately, we have to keep it for some legacy compatibility, but enabling it for digital signatures, both for signing and verification, now requires explicit setup by the end user. This also can be done via crypto policies.

FIPS 140-3 related patches

There are three categories of FIPS-related patches. Some simplify dealing with the FIPS provider for our purposes. We also introduced FIPS indicators—a new concept of the FIPS 140-3 series of standards. And the rest bridge the significant differences between FIPS-140-2 and FIPS 140-3 requirements and match the selection of the features we certify.

Loading and configuring the provider

First, applications may need to detect FIPS mode. In OpenSSL 1.1, we had a macro named FIPS_mode for RHEL 8, so now we reimplement it for RHEL 9. Relying on the new OpenSSL architecture, it retains the previous API signature.

We’ve implemented auto-activation of the FIPS provider when the system is in FIPS mode. The original upstream solution required some explicit actions for this purpose, and we decided to avoid them. It simplifies installers and helps avoid misconfiguration—the situation when the FIPS provider is not used in FIPS mode by accident becomes hardly possible.

FIPS requires checksum validation of the certified modules. Previous versions used designated files for these purposes, and upstream stores the hash sum in the configuration. Instead of this approach, we embed the checksum in the library itself. Just like the auto-activation change, we’re aiming for increased robustness here. It makes the system much more protected from accidentally garbling of the checksum files or omitting them from non-standard system images.

Having implemented the autodetection, we were able to get rid of the upstream fipsinstall utility—we just don’t need it for our builds because of these changes.

The patches described above cover the changes in bootstrapping the FIPS provider in libcrypto and libssl. Now we’ll talk about other changes.

Indicators

FIPS 140-3 requirements introduced the concept of service indicators. The concept implies that there are many ways of using or combining FIPS-approved algorithms and that some violate the scenarios that are approved by NIST. The library user (the application or library using OpenSSL) should be able to detect whether they’ve used not just the proper algorithm but also used it in the approved manner.

We identified two possible approaches to implement the indicators: implicit and explicit. The implicit approach assumes that only approved operations return successfully; all others return an error. For explicit indicators, the application should check the indicator, either once the parameters are set or after successful execution whether the operation was FIPS-approved.

Initially, we thought that it would be OK to use only implicit indicators. This would simply exclude the algorithms we didn’t want to certify from the module. Unfortunately, the real-life scenarios and some specific NIST requirements forced us to switch to a mixed approach. If a properly set up cryptographic operation fails, it means it is not permitted. But if it succeeded, the user would have to query the indicator state to check whether it was permitted or not. This approach leaves some time for application authors to rewrite the legacy applications without breaking them immediately after switching to the FIPS 140-3, making sure that the applications are not only “just working” with the FIPS module but are truly compliant.

Implementation details

We strictly follow the NIST recommendations, so we had to disable the algorithms and behaviors that are either not approved yet (Ed25519/Ed448 curves) or will be forbidden in the near future (RSA PKCS#1 v1.5 encryption, Triple DES symmetric encryption algorithm, etc.), even though they were present in the original upstream FIPS provider. We’ve also disabled SHA-1-based signatures completely in FIPS mode. Technically, FIPS 140-3 currently still allows verification of such signatures, but we don’t accept it by default in RHEL 9 in non-FIPS mode, so it would be inconsistent if it still worked in FIPS mode, and NIST is moving to deprecate SHA-1 anyway.

Several patches are dedicated to forbid the so-called explicit elliptic curve (EC) parameters. FIPS strictly limits the EC suitable for use in compliant systems. The standards that specify the usage of elliptic curves in X.509 certificates and private keys provide several ways to specify particular curve parameters. Using the named curves is safe. Processing the explicitly-specified parameters is potentially error-prone; a recent CVE in OpenSSL demonstrated the potential problems with the explicit parameters. We have disabled explicit parameters, even strictly matching those of named curves, in FIPS mode. These patches are aligned with upstream and will be eliminated in some future version of RHEL.

There was a lot of routine work to implement the FIPS 140-3 requirements. We now have to use deterministic known-answer tests (KATs) for non-deterministic signature algorithms, and we needed to develop an option to make the tests deterministic without undermining the security of the algorithm. Also, KATs must be run during the FIPS module initialization. Some original tests have been replaced with tests for other algorithm modifications, e.g RSA-OAEP instead of PKCS#1v1.5.

FIPS 140-3 introduces more strict requirements for cleaning the key data. Previous standards didn’t require the cleanup of the public keys, so we had to introduce it for new code. There are equally strict requirements for checking the newly generated key values, so we implemented extra checks for EC and DH keys.

Random number generators (RNGs) are a very important part of modern cryptography. Bad RNGs may cause the successful decipherment of the messages or leak of the private key, even if everything else is implemented properly. As most of the modern RNGs derive pseudo-random values from unpredictable seeds, there are new strict requirements around reseeding RNGs. We had to propagate the changes to get the entropy from the Linux kernel and modify the kernel itself to match these requirements. Also, according to the NIST requirements, we could use only a limited set of the digest algorithms for the hash-based RNGs.

There are also new restrictions that limit the length of seeds used for key derivation functions. As these functions are normally used for getting pseudo-random data for various protocols, using short seeds could undermine confidentiality of communications.

Even now, the certification preparations aren’t over, and the patch set and its description in this blog post are not final because of the duration and difficulty of the initial certification process. Still, we hope this post sheds some light on why we have to implement and maintain so many downstream patches for OpenSSL.