FreeIPA Identity Management planet - technical blogs

July 19, 2019

Fraser Tweedale

Designing revocation self-service for FreeIPA

Designing revocation self-service for FreeIPA

The FreeIPA team recently received a feature request for self-service certificate revocation. At the moment, revocation must be performed by a privileged user with the Revoke Certificate permission. The one exception to this is that a host principal can revoke certificates for the same hostname. There are no expections when it comes to user certificates.

In this post I’ll discuss revocation self-service and how it might work in FreeIPA.

Requirements and approaches for self-service

It is critical to avoid scenarios where a user could revoke a certificate they should not be able to revoke; this would constitute a Denial-of-Service (DoS) vulnerability. Therefore FreeIPA must establish that the principal issuing the revocation request has authority to revoke the nominated certificate. Conceptually, there are several ways we might establish that authority. Each scenario has trade-offs, either fundamental to the scenario or specific to FreeIPA.

Proof of possession

Proof of possession (PoP) establishes a cryptographic proof that the operator possess the private key for the certificate to be revoked. Either they are rightful subject of the certificate, in which case it is reasonable to service their revocation request. Or they have compromised the subject’s key, in which case it is reasonable to revoke it anyway.

There are challenges implementing a POP-based revocation system. A single request is not enough. The client must request a nonce (which the server must remember), and the subsequent message must contain a signature over that nonce. This complicates the command interface. And the user interface must consider how to access the key, i.e. it must learn arguments related to paths, passphrases, and so on.

Finally, there is an important use case this scenario does not handle: when the user no longer has control of their private key (they deleted it, forgot the passphrase, etc.)

Certificate inspection

The revocation command could inspect the certificate and decide if it “belongs to” the requestor. This must be done with extreme care, because a false-positive is equivalent to a DoS vulnerability. For example, merely checking that the UID or CN attribute in the certificate Subject DN corresponds to the requestor is inadequate.

It is hard to attain 100% certainty, especially considering administrators can create custom certificate profiles. But there are some options that seem safe enough to implement. It should be reasonable to authorise the revocation if:

  • The Subject Alternative Name (SAN) extension contains a KRB5PrincipalName or UPN value equal to the authenticated principal. FreeIPA supports such certificates out of the box, contingent on the CSR including these data.

  • The SAN contains a rfc822Name (email address) equal to one of the user’s email addresses. Again, FreeIPA supports this with the same CSR caveat.

  • The SAN contains a directoryName (DN) equal to the user’s full DN in the FreeIPA LDAP directory. Supported, with CSR caveat.

  • The certificate Subject DN is equal to the user’s full DN in the FreeIPA LDAP directory. Supported with a custom profile having subjectNameDefaultImpl configuration like (wrapped for display):

    policyset.serverCertSet.1.default.params.name=
      UID=$request.req_subject_name.cn$,
      CN=users,CN=accounts,DC=example,DC=com

The CSR caveat presents a burden to users: they must lovingly handcraft their CSR to include the relevant data. To say the tools have poor usability in this area is an understatement. But the SAN options are supported out of the box by the default user certificate profile IECUserRoles (don’t ask about the name).

On the other hand, the Subject DN approach requires a custom profile but nothing special needs to go in the CSR. A Subject DN of CN=username will suffice.

Audit-based approach

When issuing a certificate via ipa cert-request, there are two principals at play: the operator who is performing the request, and the subject principal who the certificate is for. (These could be the same principal). Subject to organisational security policy, it may be reasonable to revoke a certificate if either of these principals requests it.

Unfortunately, in FreeIPA today we do not record these data in a way that is useful to make a revocation authorisation decision. In the future, when FreeIPA authenticates to Dogtag using GSS-API and a Kerberos proxy credential for the operator (instead of the IPA RA agent credential we use today), we will be able to store the needed data. Then it may be feasible to implement this approach. Until then, forget about it.

The way forward

So, which way will we go? Nothing is decided yet (including whether to implement this at all). If we go ahead, I would like to implement the certificate inspection approach. Proof of possession is tractable, but a lot of extra complexity and probably a usability nightmare for users. The audit-based approach is infeasible at this time, though it is a solid option if/when the right pieces are in place. Certificate inspection carries a risk of DoS exposure through revocation of inappropriate certificates, but if we carefully choose which data to inspect and match, the risk is minimised while achieving satisfactory usability.

July 19, 2019 12:00 AM

June 01, 2019

Luc de Louw

OpenID and SAML authentication with Keycloak and FreeIPA

Not every web application can handle Kerberos SSO, but some provide OpenID and/or SAML. There is how Keycloak comes into the game. You can use Keycloak to federate users from different sources. This guide shows how to integrate Keyclock and … Continue reading

The post OpenID and SAML authentication with Keycloak and FreeIPA appeared first on Luc de Louw's Blog.

by Luc de Louw at June 01, 2019 08:43 PM

May 28, 2019

Fraser Tweedale

A Distinguished Name is not a string

A Distinguished Name is not a string

Distinguished Names (DNs) are used to identify entities in LDAP databases and X.509 certificates. Although DNs are often presented as strings, they have a complex structure. Because of the numerous formal and ad-hoc serialisations have been devised, and the prevalence of ad-hoc or buggy parsers, treating DNs as string in the interals of a program inevitably leads to errors. In fact, dangerous security issues can arise!

In this post I will explain the structure of DNs, review the common serialisation regimes, and review some DN-related bugs in projects I worked on. I’ll conclude with my best practices recommendations for working with DNs.

DN structure

DNs are defined by the ITU-T X.501 standard a ASN.1 objects:

Name ::= CHOICE {
  -- only one possibility for now --
  rdnSequence RDNSequence }

RDNSequence ::= SEQUENCE OF RelativeDistinguishedName

DistinguishedName ::= RDNSequence

RelativeDistinguishedName ::=
  SET SIZE (1..MAX) OF AttributeTypeAndValue

AttributeTypeAndValue ::= SEQUENCE {
  type  ATTRIBUTE.&id({SupportedAttributes}),
  value ATTRIBUTE.&Type({SupportedAttributes}{@type}),
  ... }

The AttributeTypeAndValue definition refers to some other definitions. It means that type is an object identifier (OID) of some supported attribute, and the syntax of value is determined by type. The term attribute-value assertion (AVA) is a common synonym for AttributeTypeAndValue.

Applications define a bounded set of supported attributes. For example the X.509 certificate standard suggests a minimal set of supported attributes, and an LDAP server’s schema defines all the attribute types understood by that server. Depending on the application, a program might fail to process a DN with an unrecognised attribute type, or it might process it just fine, treating the corresponding value as opaque data.

Whereas the order of AVAs within an RDN is insignificant (it is a SET), the order of RDNs within the DN is significant. If you view the list left-to-right, then the root is on the left. X.501 formalises it thus:

Each initial sub-sequence of the name of an object is also the name of an object. The sequence of objects so identified, starting with the root and ending with the object being named, is such that each is the immediate superior of that which follows it in the sequence.

This also means that the empty DN is a valid DN.

Comparing DNs

Testing DNs for equality is an important operation. For example, when constructing an X.509 certification path, we have to find a trusted CA certificate based on the certificate chain presented by an entity (e.g. a TLS server), then verify that the chain is complete by ensuring that each Issuer DN, starting from the end entity certificate, matches the Subject DN of the certificate “above” it, all the way up to a trusted CA certificate. (Then the signatures must be verified, and several more checks performed).

Continuing with this example, if an implementation falsely determines that two equal DNs (under X.500) are inequal, then it will fail to construct the certification path and reject the certificate. This is not good. But even worse would be if it decides that two unequal DNs are in fact equal! Similarly, if you are issuing certificates or creating LDAP objects or anything else, a user could exploit bugs in your DN handling code to cause you to issue certificates, or create objects, that you did not intend.

Having motivated the importance of correct DN comparison, well, how do you compare DNs correctly?

First, the program must represent the DNs according to their true structure: a list of sets (RDNs) of attribute-value pairs (AVAs). If the DNs are not already represented this way in the program, they must be parsed or processed—correctly.

Now that the structure is correct, AVAs can be compared for equality. Each attribute type defines an equality matching rule that says how values should be compared. In some cases this is just binary matching. In other cases, normalisation or other rules must be applied to the values. For example, some string types may be case insensitive.

A notable case is the DirectoryString syntax used by several attribute types in X.509:

DirectoryString ::= CHOICE {
    teletexString       TeletexString   (SIZE (1..MAX)),
    printableString     PrintableString (SIZE (1..MAX)),
    universalString     UniversalString (SIZE (1..MAX)),
    utf8String          UTF8String      (SIZE (1..MAX)),
    bmpString           BMPString       (SIZE (1..MAX)) }

DirectoryString supports a choice of string encodings. Values of use PrintableString orr UTF8String encoding must be preprocessed using the LDAP Internationalized String Preparation rules (RFC 4518), including case folding and insignificant whitespace compression.

Taking the DN as a whole, two DNs are equal if they have the same RDNs in the same order, and two RDNs are equal if they have the same AVAs in any order (i.e. sets of equal size, with each AVA in one set having a matching AVA in the other set).

Ultimately this means that, despite X.509 certificates using Distinguised Encoding Rules (DER) for serialisation, there can still be multiple ways to represent equivalent data (by using different string encodings). Therefore, binary matching of serialised DNs, or even binary matching of individual attribute values, is incorrect behaviour and may lead to failures.

String representations

Several string representations of DNs, both formally-specified and ad-hoc, are in widespread use. In this section I’ll list some of the more important ones.

Because DNs are ordered, one of the most obvious characteristics of a string representation is whether it lists the RDNs in forward or reverse order, i.e. with the root at the left or right. Some popular libraries and programs differ in this regard.

As we look at some of these common implementations, we’ll use the following DN as an example:

SEQUENCE (3 elem)
  SET (2 elem)
    SEQUENCE (2 elem)
      OBJECT IDENTIFIER 2.5.4.6 countryName
      PrintableString AU
    SEQUENCE (2 elem)
      OBJECT IDENTIFIER 2.5.4.8 stateOrProvinceName
      PrintableString Queensland
  SET (1 elem)
    SEQUENCE (2 elem)
      OBJECT IDENTIFIER 2.5.4.10 organizationName
      PrintableString Acme, Inc.
  SET (1 elem)
    SEQUENCE (2 elem)
      OBJECT IDENTIFIER 2.5.4.3 commonName
      PrintableString CA

RFC 4514

CN=CA,O=Acme\, Inc.,C=AU+ST=Queensland
CN=CA,O=Acme\2C Inc.,C=AU+ST=Queensland

RFC 4514 defines the string representation of distinguished names used in LDAP. As such, there is widespread library support for parsing and printing DNs in this format. The RDNs are in reverse order, separated by ,. Special characters are escaped using backslash (\), and can be represented using the escaped character itself (e.g. \,) or two hex nibbles (\2C). The AVAs within a multi-valued RDN are separated by +, in any order.

Due to the multiple ways of escaping special characters, this is not a distinguished encoding.

This format is used by GnuTLS, OpenLDAP and FreeIPA, among other projects.

RFC 1485

CN=CA,O="Acme, Inc.",C=AU+ST=Queensland

RFC 1485 is a predecessor of a predecessor (RFC 1779) of a predecessor (RFC 2253) of RFC 4514. There are some differences from RFC 4514. For example, special character escapes are not supported; quotes must be used. This format is still relevant today because NSS uses it for pretty-printing and parsing DNs.

OpenSSL

OpenSSL prints DNs in its own special way. Unlike most other implementations, it works with DNs in forward order (root at left). The pretty-print looks like:

C = AU + ST = Queensland, O = "Acme, Inc.", CN = CA

The format when parsing is different again. Some commands need a flag to enable support for multi-valued RDNs; e.g. openssl req -multivalue-rdn ....

/C=AU+ST=Queensland/O=Acme, Inc./CN=CA

OpenSSL can also read DNs from a config file where AVAs are given line by line (see config and x509v3_config(5)). But this is not a DN string representation per se so I won’t cover it here.

Bugs, bugs, bugs

Here are three interesting bugs I discovered, related to DN string encoding.

389 DS #49543: certmap fails when Issuer DN has comma in name

389 DS supports TLS certificate authentication for binding to LDAP. Different certificate mapping (certmap) policies can be defined for different CAs. The issuer DN in the client certificate is used to look up a certmap configuration. Unfortunately, a string comparison was used to perform this lookup. 389 uses NSS, which serialised the DN using RFC 1485 syntax. If this disagreed with how the DN in the certmap configuration appeared (after normalisation), the lookup—hence the LDAP bind—would fail. The normalisation function was also buggy.

The fix was to parse the certmap DN string into an a NSS CertNAME using the CERT_AsciiToName routine, then compare the Issuer DN from the certificate against it using the NSS DN comparison routine (CERT_CompareName). The buggy normalisation routine was deleted.

Certmonger #90: incorrect DN in CSR

Certmonger stores tracking request configuration in a flat text file. This configuration includes the string representation of the DN, ostensibly in RFC 4514 syntax. When constructing a CSR for the tracking request, it parsed the DN then used the result to construct an OpenSSL X509_NAME, which would be used in OpenSSL routines to create the CSR.

Unfortunately, the DN parsing implementation—a custom routine in Certmonger itself—was busted. A DN string like:

CN=IPA RA,O=Acme\, Inc.,ST=Massachusetts,C=US

Resulted in a CSR with the following DN:

CN=IPA RA,CN=Inc.,O=Acme\\,ST=Massachusetts,C=US

The fix was to remove the buggy parser and use the OpenLDAP ldap_str2dn routine instead. This was a joint effort between Rob Crittenden and myself.

FreeIPA #7750: invalid modlist when attribute encoding can vary

FreeIPA’s LDAP library, ipaldap, uses python-ldap for handling low-level stuff and provides a lot of useful stuff on top. One useful thing it does is keeps track of the original attribute values for an object, so that we can perform changes locally and efficiently produce a list of modifications (modlist) for when we want to update the object at the server.

ipaldap did not take into account the possibility of the attribute encoding returned by python-ldap differing from the attribute encoding produced by FreeIPA. A disagreement could arise when DN attribute values contained special characters requiring escaping. For example, python-ldap escaped characters using hex encoding:

CN=CA,O=Red Hat\2C Inc.,L=Brisbane,C=AU

The representation produced by python-ldap is recorded as the original value of the attribute. However, if you wrote the same attribute value back, it would pass through FreeIPA’s encoding routine, which might encode it differently and record it as a new value:

CN=CA,O=Red Hat\, Inc.,L=Brisbane,C=AU

When you go to update the object, the modlist would look like:

[ (ldap.MOD_ADD, 'ipacaissuerdn',
    [b'CN=CA,O=Red Hat\, Inc.,L=Brisbane,C=AU'])
, (ldap.MOD_DELETE, 'ipacaissuerdn',
    [b'CN=CA,O=Red Hat\2C Inc.,L=Brisbane,C=AU'])
]

Though encoded differently, these are the same value but that in itself is not a problem. The problem is that the server also has the same value, and processing the MOD_ADD first results in an attributeOrValueExists error. You can’t add a value that’s already there!

The ideal fix for this would be to update ipaldap to record all values as ASN.1 data or DER, rather than strings. But that would be a large and risky change. Instead, we work around the issue by always putting deletes before adds in the modlist. LDAP servers process changes in the order they are presented (389 DS does so atomically). So deleting an attribute value then adding it straight back is a safe, albeit inefficient, workaround.

Discussion

So you have to compare or handle some DNs. What do you do? My recommendations are:

  • If you need to print/parse DNs as strings, if possible use RFC 4514 because it has the most widespread library support.
  • Don’t write your own DN parsing code. This is where security vulnerabilities are most likely. Use existing library routines for parsing DNs. If you have no other choice, take extreme care and if possible use a parser combinator library or parser generator to make the definitions more declarative and reduce likelihood of error.
  • Always decode attribute values (if the DN parsing routine doesn’t do it for you). This avoids confusion where attribute values could be encoded in different ways (due to escaped characters or differing string encodings).
  • Use established library routines for comparing DNs using the internal DN structures, not strings.

Above all, just remember: a Distinguished Name is not a string, so don’t treat it like a string. For sure it’s more work, but DNs need special treatment or bugs will certainly arise.

That’s not to say that “native” DN parsing and comparison routines are bug-free. They are not. A common error is equal DNs comparing inequal due to differing attribute string encodings (e.g. PrintableString versus UTF8String). I have written about this in a previous post. In Dogtag we’ve enountered this kind of bug quite a few times. In these situations the DN comparison should be fixed, but it may be a satisfactory workaround to serialise both DNs and perform a string comparison.

Another common issue is lack of support for multi-valued RDNs. A few years ago we wanted to switch FreeIPA’s certificate handling from python-nss to the cryptography library. I had to add support for multi-valued RDNs before we could make the switch.

A final takeaway for authors of standards. Providing multiple ways to serialise the same value leads to incompatibilities and bugs. For sure, there is a tradeoff between usability, implementation complexity and risk of interoperability issues and bugs. RFC 4514 would be less human-friendly if it only permitted hex-escapes. But implementations would be simpler and the interop/bug risk would be reduced. It’s important to think about these tradeoffs and the consequences, especially for standards and protocols relating to security.

May 28, 2019 12:00 AM

May 24, 2019

Fraser Tweedale

Fixing expired system certificates in FreeIPA

Fixing expired system certificates in FreeIPA

In previous posts I outlined and demonstrated the pki-server cert-fix tool. This tool is part of Dogtag PKI. I also discussed what additional functionality would be needed to successfully use this tool in a FreeIPA environment.

This post details the result of the effort to make cert-fix useful for FreeIPA administrators. We implemented a wrapper program, ipa-cert-fix, which performs FreeIPA-specific steps in addition to executing pki-server cert-fix.

What does ipa-cert-fix do?

In brief, the steps performed by ipa-cert-fix are:

  1. Inspect deployment to work out which certificates need renewing. This includes both Dogtag system certificates, FreeIPA-specific certificates (HTTP, LDAP, KDC and IPA RA).
  2. Print intentions and await operator confirmation.
  3. Invoke pki-server cert-fix to renew expired certificates, including FreeIPA-specific certificates.
  4. Install renewed FreeIPA-specific certificates to their respective locations.
  5. If any shared certificates were renewed (Dogtag system certificates excluding HTTP, and IPA RA), import them to the LDAP ca_renewal subtree and set the caRenewalMaster configuration to be the current server. This allows CA replicas to pick up the renewed shared certificates.
  6. Restart FreeIPA (ipactl restart).

Demonstration

For this demonstration I used a deployment with the following characteristics:

  • Two servers, f29-0 and f29-1, with CA on both.
  • f29-0 is the current CA renewal master.
  • A KRA instance is installed on f29-1.
  • The deployment was created on 2019-05-24, so most of the certificates expire on or before 2021-05-24 (the exception being the CA certificate).

On both servers I disabled chronyd and put the clock forward 27 months, so that all the certificates (except the IPA CA itself) are expired:

[f29-1] ftweedal% sudo systemctl stop chronyd
[f29-1] ftweedal% date
Fri May 24 12:01:16 AEST 2019
[f29-1] ftweedal% sudo date 082412012021
Tue Aug 24 12:01:00 AEST 2021

We want to perform this step on all machines in the topology. After all, we are simulating the passage of time.

After ipactl restart the Dogtag CA did not start, and we cannot communicate with FreeIPA due to the expired HTTP certificate:

[f29-1] ftweedal% sudo ipactl status
Directory Service: RUNNING
krb5kdc Service: RUNNING
kadmin Service: RUNNING
httpd Service: RUNNING
ipa-custodia Service: RUNNING
pki-tomcatd Service: STOPPED
ipa-otpd Service: RUNNING
ipa: INFO: The ipactl command was successful

[f29-1] ftweedal% ipa user-find
ipa: ERROR: cannot connect to 'https://f29-1.ipa.local/ipa/json':
  [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed:
  certificate has expired (_ssl.c:1056)

Fixing the first server

I will repair f29-1 first, so that we can see why resetting the CA renewal master is an important step performed by ipa-cert-fix.

I ran ipa-cert-fix as root. It analyses the server, then prints a warning and the list of certificates to be renewed, and asks for confirmation:

[f29-1] ftweedal% sudo ipa-cert-fix

                          WARNING

ipa-cert-fix is intended for recovery when expired certificates
prevent the normal operation of FreeIPA.  It should ONLY be used
in such scenarios, and backup of the system, especially certificates
and keys, is STRONGLY RECOMMENDED.


The following certificates will be renewed:

Dogtag sslserver certificate:                                                                                                                                                                                [2/34]
  Subject: CN=f29-1.ipa.local,O=IPA.LOCAL 201905222205                                                                                                                                                             
  Serial:  13
  Expires: 2021-05-12 05:55:47

Dogtag subsystem certificate:
  Subject: CN=CA Subsystem,O=IPA.LOCAL 201905222205
  Serial:  4
  Expires: 2021-05-11 12:07:11

Dogtag ca_ocsp_signing certificate:
  Subject: CN=OCSP Subsystem,O=IPA.LOCAL 201905222205
  Serial:  2
  Expires: 2021-05-11 12:07:11

Dogtag ca_audit_signing certificate:
  Subject: CN=CA Audit,O=IPA.LOCAL 201905222205
  Serial:  5
  Expires: 2021-05-11 12:07:12

Dogtag kra_transport certificate:
  Subject: CN=KRA Transport Certificate,O=IPA.LOCAL 201905222205
  Serial:  268369921
  Expires: 2021-05-12 06:00:10

Dogtag kra_storage certificate:
  Subject: CN=KRA Storage Certificate,O=IPA.LOCAL 201905222205
  Serial:  268369922
  Expires: 2021-05-12 06:00:10

Dogtag kra_audit_signing certificate:
  Subject: CN=KRA Audit,O=IPA.LOCAL 201905222205
  Serial:  268369923
  Expires: 2021-05-12 06:00:11

IPA IPA RA certificate:
  Subject: CN=IPA RA,O=IPA.LOCAL 201905222205
  Serial:  7
  Expires: 2021-05-11 12:07:47

IPA Apache HTTPS certificate:
  Subject: CN=f29-1.ipa.local,O=IPA.LOCAL 201905222205
  Serial:  12
  Expires: 2021-05-23 05:54:11

IPA LDAP certificate:
  Subject: CN=f29-1.ipa.local,O=IPA.LOCAL 201905222205
  Serial:  11
  Expires: 2021-05-23 05:53:58

IPA KDC certificate:
  Subject: CN=f29-1.ipa.local,O=IPA.LOCAL 201905222205
  Serial:  14
  Expires: 2021-05-23 05:57:50

Enter "yes" to proceed:

Observe that the KRA certificates are included (we are on f29-1). I type “yes” and continue. After a few minutes the process has completed:

Proceeding.
Renewed Dogtag sslserver certificate:
  Subject: CN=f29-1.ipa.local,O=IPA.LOCAL 201905222205
  Serial:  268369925
  Expires: 2023-08-14 02:19:33

... (9 certificates elided)

Renewed IPA KDC certificate:
  Subject: CN=f29-1.ipa.local,O=IPA.LOCAL 201905222205
  Serial:  268369935
  Expires: 2023-08-25 02:19:42

Becoming renewal master.
The ipa-cert-fix command was successful

As suggested by the expiry dates, it took about 11 seconds to renew all 11 certifiates. So why did it take so long? The pki-server cert-fix command, which is part of Dogtag and invoked by ipa-cert-fix, restarts the Dogtag instance as its final step. Although a new LDAP certificate was issued, it is not yet been installed in 389’s certificate database. Dogtag fails to start; it cannot talk to LDAP because of the expired certificate, and the restart operation hangs for a while. ipa-cert-fix knows to expect this and ignores the pki-server cert-fix failure when the LDAP certificate needs renewal.

ipa-cert-fix also reported that it was setting the renewal master (because shared certificates were renewed). Let’s check the server status and verify the configuration.

[f29-1] ftweedal% sudo ipactl status
Directory Service: RUNNING
krb5kdc Service: RUNNING
kadmin Service: RUNNING
httpd Service: RUNNING
ipa-custodia Service: RUNNING
pki-tomcatd Service: RUNNING
ipa-otpd Service: RUNNING
ipa: INFO: The ipactl command was successful

The server is up and running.

[f29-1] ftweedal% kinit admin
Password for admin@IPA.LOCAL:
Password expired.  You must change it now.
Enter new password:
Enter it again:

Passwords have expired (due to time-travel).

[f29-1] ftweedal% ipa config-show |grep renewal
  IPA CA renewal master: f29-1.ipa.local

f29-1 has indeed become the renewal master. Oh, and the HTTP and LDAP certifiate have been fixed.

[f29-1] ftweedal% ipa cert-show 1 | grep Subject
  Subject: CN=Certificate Authority,O=IPA.LOCAL 201905222205

And the IPA framework can talk to Dogtag. This proves that the IPA RA and Dogtag HTTPS and subsystem certificates are valid.

Fixing subsequent servers

Jumping back onto f29-0, let’s look at the Certmonger request statuses:

[f29-0] ftweedal% sudo getcert list \
                  | egrep '^Request|status:|subject:'
Request ID '20190522120745':
        status: CA_UNREACHABLE
        subject: CN=IPA RA,O=IPA.LOCAL 201905222205
Request ID '20190522120831':
        status: CA_UNREACHABLE
        subject: CN=CA Audit,O=IPA.LOCAL 201905222205
Request ID '20190522120832':
        status: CA_UNREACHABLE
        subject: CN=OCSP Subsystem,O=IPA.LOCAL 201905222205
Request ID '20190522120833':
        status: CA_UNREACHABLE
        subject: CN=CA Subsystem,O=IPA.LOCAL 201905222205
Request ID '20190522120834':
        status: MONITORING
        subject: CN=Certificate Authority,O=IPA.LOCAL 201905222205
Request ID '20190522120835':
        status: CA_UNREACHABLE
        subject: CN=f29-0.ipa.local,O=IPA.LOCAL 201905222205
Request ID '20190522120903':
        status: CA_UNREACHABLE
        subject: CN=f29-0.ipa.local,O=IPA.LOCAL 201905222205
Request ID '20190522120932':
        status: CA_UNREACHABLE
        subject: CN=f29-0.ipa.local,O=IPA.LOCAL 201905222205
Request ID '20190522120940':
        status: CA_UNREACHABLE
        subject: CN=f29-0.ipa.local,O=IPA.LOCAL 201905222205

The MONITORING request is the CA certificate. All the other requests are stuck in CA_UNREACHABLE.

The Certmonger tracking requests need to communicate with LDAP to retrieve shared certificates. So we have to ipactl restart with --force to ignore individual service startup failures (Dogtag will fail):

[f29-0] ftweedal% sudo ipactl restart --force
Skipping version check
Starting Directory Service
Starting krb5kdc Service
Starting kadmin Service
Starting httpd Service
Starting ipa-custodia Service
Starting pki-tomcatd Service
Starting ipa-otpd Service
ipa: INFO: The ipactl command was successful

[f29-0] ftweedal% sudo ipactl status
Directory Service: RUNNING
krb5kdc Service: RUNNING
kadmin Service: RUNNING
httpd Service: RUNNING
ipa-custodia Service: RUNNING
pki-tomcatd Service: STOPPED
ipa-otpd Service: RUNNING
ipa: INFO: The ipactl command was successful

Now Certmonger is able to renew the shared certificates by retrieving the new certificate from LDAP. The IPA-managed certificates are also able to be renewed by falling back to requesting them from another CA server (the already repaired f29-1). After a short wait, getcert list shows that all but one of the certificates have been renewed:

[f29-0] ftweedal% sudo getcert list \
                  | egrep '^Request|status:|subject:'
Request ID '20190522120745':
        status: MONITORING
        subject: CN=IPA RA,O=IPA.LOCAL 201905222205
Request ID '20190522120831':
        status: MONITORING
        subject: CN=CA Audit,O=IPA.LOCAL 201905222205
Request ID '20190522120832':
        status: MONITORING
        subject: CN=OCSP Subsystem,O=IPA.LOCAL 201905222205
Request ID '20190522120833':
        status: MONITORING
        subject: CN=CA Subsystem,O=IPA.LOCAL 201905222205
Request ID '20190522120834':
        status: MONITORING
        subject: CN=Certificate Authority,O=IPA.LOCAL 201905222205
Request ID '20190522120835':
        status: CA_UNREACHABLE
        subject: CN=f29-0.ipa.local,O=IPA.LOCAL 201905222205
Request ID '20190522120903':
        status: MONITORING
        subject: CN=f29-0.ipa.local,O=IPA.LOCAL 201905222205
Request ID '20190522120932':
        status: MONITORING
        subject: CN=f29-0.ipa.local,O=IPA.LOCAL 201905222205
Request ID '20190522120940':
        status: MONITORING
        subject: CN=f29-0.ipa.local,O=IPA.LOCAL 201905222205

The final CA_UNREACHABLE request is the Dogtag HTTP certificate. We can now run ipa-cert-fix on f29-0 to repair this certificate:

[f29-0] ftweedal% sudo ipa-cert-fix

                          WARNING

ipa-cert-fix is intended for recovery when expired certificates
prevent the normal operation of FreeIPA.  It should ONLY be used
in such scenarios, and backup of the system, especially certificates
and keys, is STRONGLY RECOMMENDED.


The following certificates will be renewed:

Dogtag sslserver certificate:
  Subject: CN=f29-0.ipa.local,O=IPA.LOCAL 201905222205
  Serial:  3
  Expires: 2021-05-11 12:07:11

Enter "yes" to proceed: yes
Proceeding.
Renewed Dogtag sslserver certificate:
  Subject: CN=f29-0.ipa.local,O=IPA.LOCAL 201905222205
  Serial:  15
  Expires: 2023-08-14 04:25:05

The ipa-cert-fix command was successful

All done?

Yep. A subsequent execution of ipa-cert-fix shows that there is nothing to do, and exits:

[f29-0] ftweedal% sudo ipa-cert-fix
Nothing to do.
The ipa-cert-fix command was successful

Feature status

Against the usual procedure for FreeIPA (and Red Hat projects in general), ipa-cert-fix was developed “downstream-first”. It has been merged to the ipa-4-6 branch, but there might not even be another upstream release from that branch. But there might be a future RHEL release based on that branch (the savvy reader might infer a high degree of certainty, given we actually bothered to do that…)

In the meantime, work to forward-port the feature to master and newer branches is ongoing. I hope that it will be merged in the next week or so.

May 24, 2019 12:00 AM

May 07, 2019

Rob Crittenden

How do I revert back to using IPA-issued Web & LDAP certs?

For IPA v4.6.x.

So you have an IPA installation with a CA and you decided you don’t want your users to have to install the IPA CA certificate(s) so instead you want to use certificates for the Web and LDAP using some known 3rd party issuer. Sure, that works fine. You’d do something like:

Install the 3rd party CA chain and update your IPA master:

# ipa-cacert-manage install /path/to/root.pem -t CT,,
# ipa-cacert-manage install intermediate.cert.pem -t CT,,
# ipa-certupdate

Install the 3rd-party provided server certificate. In this case I have it as two separate files, the public cert and the private key.

# ipa-server-certinstall --dirman-password password -w -d --pin '' server.cert.pem server.cert.key root.pem \
intermediate.cert.pem

Great. IPA is working fine and my users don’t need to import the IPA CA.

Two years later…

My 3rd party certs are expiring soon and I don’t want to pay for new ones and I want to switch back to using IPA-issued certificates. We can use certmonger for that. This assumes that your CA is still up and functioning properly.

I’d start by backing up the two NSS databases. It is safest to do this offline (ipactl stop). You need to copy *.db from /etc/dirsrv/slapd-EXAMPLE-TEST and /etc/httpd/alias someplace safe, then restart the world (ipactl start).

First the web server:

# ipa-getcert request -d /etc/httpd/alias -n Server-Cert -K HTTP/`hostname` -D `hostname` -C /usr/libexec/ipa/certmonger/restart_httpd -p /etc/httpd/alias/pwdfile.txt

Edit /etc/httpd/conf.d/nss.conf and replace the value of NSSNickname with Server-Cert.

Wait a bit to be sure the cert is issued. You can run this to see the status:

# ipa-getcert list -d /etc/httpd/alias -n Server-Cert

Now the LDAP server:

# ipa-getcert request -d /etc/dirsrv/slapd-EXAMPLE-TEST -n Server-Cert -D `hostname` -K ldap/`hostname` -C "/usr/libexec/ipa/certmonger/restart_dirsrv EXAMPLE-TEST" -p /etc/dirsrv/slapd-EXAMPLE-TEST/pwdfile.txt

Similarly wait for it to be issued. To track the status:

# ipa-getcert list -d /etc/dirsrv/slapd-EXAMPLE-TEST -n Server-Cert

Once it is issued run:

# ipactl stop

Now edit /etc/dirsrv/slapd-EXAMPLE-TEST/dse.ldif. We could do this while the server is online but we need to restart anyway and your favorite editor is easier than ldapmodify. Replace the value of nsSSLPersonalitySSL with Server-Cert

Now restart the world:

# ipactl start

Connect to each port if you want to confirm that the certificate and chain are correct, e.g.

# openssl s_client -host `hostname` -port 443
CONNECTED(00000003)
depth=1 O = EXAMPLE.TEST, CN = Certificate Authority
verify return:1
depth=0 O = EXAMPLE.TEST, CN = ipa.example.test
verify return:1
---
Certificate chain
0 s:/O=EXAMPLE.TEST/CN=ipa.example.test
i:/O=EXAMPLE.TEST/CN=Certificate Authority
1 s:/O=EXAMPLE.TEST/CN=Certificate Authority
i:/O=EXAMPLE.TEST/CN=Certificate Authority
---
...

by rcritten at May 07, 2019 02:43 AM

Powered by Planet