FreeIPA Identity Management planet - technical blogs

September 23, 2019

Fraser Tweedale

Requesting certificates from FreeIPA on Active Directory clients

Requesting certificates from FreeIPA on Active Directory clients

In recent times I have seen some support cases and sales inquiries about getting certificates on Linux systems that are enrolled in Active Directory (AD). Linux hosts can be directly enrolled in AD via realmd or adcli. On AD-enrolled machines, SSSD can provide authentication services, access control and some Group Policy enforcement (see sssd-ad(5)). At Red Hat we call this approach direct integration. The alternative approach is to enrol hosts in a FreeIPA / IDM realm, and use cross-realm trusts to allow AD users/principals to authenticate to FreeIPA services, or vice-versa.

Unfortunately when it comes to getting certificates from AD-CS (the Active Directory certificate authority component) we don’t have a good story yet. Certmonger lacks an out-of-the-box capability to talk to AD-CS (except via SCEP, but that is not what we want). I do not know how much work would be involved in writing an AD-CS request helper for Certmonger. It might be a large effort or small; the little AD-CS enrolment documentation I have found is hard to penetrate. But even if we could write the AD-CS helper and ship it tomorrow, it would not help users and customers on older releases.

For now the best solution is to deploy FreeIPA / IDM, and use the IPA CA to issue certificates to AD-enrolled hosts. In this HOWTO-style post I walk through this scenario with a RHEL 6 client.

Environment

My local Windows Server 2012 R2 server defines the AD.LOCAL Active Directory (AD) realm. The DNS zone is ad.local. I configured a DNS Conditional Forwarder for ipa.local, delegating that namespace to the FreeIPA DNS server.

The FreeIPA server is rhel76-0.ipa.local. There is no AD trust. The integrated DNS is in use so KDC discovery will work.

In a typical scenario the IPA CA might be a subordinate of the AD CA. In my setup the IPA CA is self-signed. Operationally there is one additional step when the IPA CA is not subordinate to the AD CA: the IPA CA certificate has to be explicitly trusted. To trust the certificate copy it to a file under /etc/pki/ca-trust/source/anchors/ then execute update-ca-trust.

The RHEL 6 host is named rhel610-0.ipa.local The packages required were adcli and ipa-client. On RHEL 7 and later or Fedora the realm command, which is provided by the realmd package, is a better choice than adcli. /etc/resolv.conf points to the Windows Server.

Active Directory enrolment

I used the adcli command to enrol rhel610-0 in Active Directory:

[root@rhel610-0 ~]# adcli join ad.local --show-details
Password for Administrator@AD.LOCAL: XXXXXXXX
[domain]
domain-name = ad.local
domain-realm = AD.LOCAL
domain-controller = win-ppk015f9mdq.ad.local
domain-short = AD
domain-SID = S-1-5-21-3519545429-1027502194-913185514
naming-context = DC=ad,DC=local
domain-ou = (null)
[computer]
host-fqdn = rhel610-0.ipa.local
computer-name = RHEL610-0
computer-dn = CN=RHEL610-0,CN=Computers,DC=ad,DC=local
os-name = redhat-linux-gnu
[keytab]
kvno = 2
keytab = FILE:/etc/krb5.keytab

The output shows that the enrolment succeeded and prints information about the realm and enrolment. Inspecting the system keytab /etc/krb5.keytab shows the Kerberos keys:

[root@rhel610-0 ~]# ktutil
ktutil:  read_kt /etc/krb5.keytab
ktutil:  list
slot KVNO Principal
---- ---- ---------------------------------------------------------------------
   1    2                      RHEL610-0$@AD.LOCAL
   2    2                      RHEL610-0$@AD.LOCAL
   3    2                      RHEL610-0$@AD.LOCAL
   4    2                      RHEL610-0$@AD.LOCAL
   5    2                      RHEL610-0$@AD.LOCAL
   6    2                      RHEL610-0$@AD.LOCAL
   7    2                  host/RHEL610-0@AD.LOCAL
   8    2                  host/RHEL610-0@AD.LOCAL
   9    2                  host/RHEL610-0@AD.LOCAL
  10    2                  host/RHEL610-0@AD.LOCAL
  11    2                  host/RHEL610-0@AD.LOCAL
  12    2                  host/RHEL610-0@AD.LOCAL
  13    2        host/rhel610-0.ipa.local@AD.LOCAL
  14    2        host/rhel610-0.ipa.local@AD.LOCAL
  15    2        host/rhel610-0.ipa.local@AD.LOCAL
  16    2        host/rhel610-0.ipa.local@AD.LOCAL
  17    2        host/rhel610-0.ipa.local@AD.LOCAL
  18    2        host/rhel610-0.ipa.local@AD.LOCAL
  19    2     RestrictedKrbHost/RHEL610-0@AD.LOCAL
  20    2     RestrictedKrbHost/RHEL610-0@AD.LOCAL
  21    2     RestrictedKrbHost/RHEL610-0@AD.LOCAL
  22    2     RestrictedKrbHost/RHEL610-0@AD.LOCAL
  23    2     RestrictedKrbHost/RHEL610-0@AD.LOCAL
  24    2     RestrictedKrbHost/RHEL610-0@AD.LOCAL
  25    2 RestrictedKrbHost/rhel610-0.ipa.local@AD.LOCAL
  26    2 RestrictedKrbHost/rhel610-0.ipa.local@AD.LOCAL
  27    2 RestrictedKrbHost/rhel610-0.ipa.local@AD.LOCAL
  28    2 RestrictedKrbHost/rhel610-0.ipa.local@AD.LOCAL
  29    2 RestrictedKrbHost/rhel610-0.ipa.local@AD.LOCAL
  30    2 RestrictedKrbHost/rhel610-0.ipa.local@AD.LOCAL
ktutil:  quit

FreeIPA “enrolment”

Next I created a host princpial for rhel610-0.ipa.local in the FreeIPA realm:

[root@rhel76-0 ~]# ipa host-add rhel610-0.ipa.local
--------------------------------
Added host "rhel610-0.ipa.local"
--------------------------------
  Host name: rhel610-0.ipa.local
  Principal name: host/rhel610-0.ipa.local@IPA.LOCAL
  Principal alias: host/rhel610-0.ipa.local@IPA.LOCAL
  Password: False
  Keytab: False
  Managed by: rhel610-0.ipa.local

Because the integrated DNS is in use, we do not need to explicitly tell the Kerberos library about the IPA.LOCAL KDC. Instead you only need to ensure that /etc/krb5.conf does not contain:

[libdefaults]
  dns_lookup_kdc = false

When not using KDC discovery a section like the following is needed:

[realms]
 IPA.LOCAL = {
  kdc = rhel76-0.ipa.local
  admin_server = rhel76-0.ipa.local
 }

I also needed to add a [domain_realm] section to tell the Kerberos client library what realm to use when talking to the IPA server:

[domain_realm]
 .ipa.local = IPA.LOCAL

Reading krb5.conf(5), there is a [libdefaults] knob called realm_try_domains. From the description, it seems that using it could avoid the need for a [domain_realm] section. But it did not work for me, in the way I expected (on this RHEL 6 client at least).

Next I had to retrieve the host keys for the IPA.LOCAL realm into the system keytab. The Certmonger IPA helper will use those keys to authenticate to FreeIPA when requesting a certificate:

[root@rhel610-0 ~]# kinit admin@IPA.LOCAL
Password for admin@IPA.LOCAL: 
[root@rhel610-0 ~]# ipa-getkeytab -s rhel76-0.ipa.local \
    -p host/rhel610-0.ipa.local@IPA.LOCAL \
    -k /etc/krb5.keytab
Keytab successfully retrieved and stored in: /etc/krb5.keytab

Listing the keys in /etc/krb5.conf we now see that the IPA.LOCAL host keys have been appended:

[root@rhel610-0 ~]# ktutil
ktutil:  read_kt /etc/krb5.keytab
ktutil:  list
slot KVNO Principal
---- ---- ---------------------------------------------------------------------
   1    2                      RHEL610-0$@AD.LOCAL
    ...
  30    2 RestrictedKrbHost/rhel610-0.ipa.local@AD.LOCAL
  31    1       host/rhel610-0.ipa.local@IPA.LOCAL
  32    1       host/rhel610-0.ipa.local@IPA.LOCAL
  33    1       host/rhel610-0.ipa.local@IPA.LOCAL
  34    1       host/rhel610-0.ipa.local@IPA.LOCAL
ktutil:  quit

SELinux considerations

I will store certificates and keys under /etc/pki/tls/private/ because this directory has the correct SELinux context (and default context rules) for Certmonger to use it:

[root@rhel610-0 ~]# ls -l -d -Z /etc/pki/tls/private/
drwxr-xr-x. root root system_u:object_r:cert_t:s0 /etc/pki/tls/private/

If you want Certmonger to manage keys and certificates in other directories you need to ensure the files/directory have the cert_t type label. This can be achieved via the semanage(8) and restorecon(8), but I will not go into further detail here.

Certmonger IPA configuration

Certmonger comes out of the box with a request/renewal helper for an IPA CA. But it assumes that the client is an IPA-enrolled server, i.e. per ipa-client-install. In particular there are two files that must be manually set up. First, the IPA CA (and chain) must be present in /etc/ipa/ca.crt. It can be copied from the IPA server without changes. I have filed a ticket to make Certmonger use the system CA trust store.

The other file is /etc/ipa/default.conf. The Certmonger IPA helper reads several fields from this file to locate the IPA server and work out how to initialise Kerberos credentials. I used the following configuration:

[global]
server = rhel76-0.ipa.local
basedn = dc=ipa,dc=local
realm = IPA.LOCAL
domain = ipa.local
xmlrpc_uri = https://rhel76-0.ipa.local/ipa/xml
ldap_uri = ldaps://rhel76-0.ipa.local

Requesting the certificate

Now we can tell Certmonger to request a certificate using the IPA CA:

[root@rhel610-0 ~]# getcert request \
    -c IPA \
    -k /etc/pki/tls/private/cert.key \
    -f /etc/pki/tls/private/cert.pem \
    -K host/rhel610-0.ipa.local@IPA.LOCAL \
    -D rhel610-0.ipa.local
New signing request "20190920053226" added.

The options used are:

-c

Use the IPA CA request/renewal helper. To see a list of all the defined CA helpers execute getcert list-cas.

-k

Where to store the newly generated, or read the existing, private key.

-f

Where to store the issued certificate.

-K

Kerberos principal name, which will appear in the CSR’s Subject Alternative Name extension. The IPA request helper requires this parameter.

-D

DNS name to include in the Subject Alternative Name extension.

We can use the request ID to print the details of the certificate request:

[root@rhel610-0 ~]# getcert list -i 20190920053226
Number of certificates and requests being tracked: 1.
Request ID '20190920053226':
  status: MONITORING
  stuck: no
  key pair storage: type=FILE,location='/etc/pki/tls/private/cert.key'
  certificate: type=FILE,location='/etc/pki/tls/private/cert.pem'
  CA: IPA
  issuer: CN=Certificate Authority,O=IPA.LOCAL 201909191314
  subject: CN=rhel610-0.ipa.local,O=IPA.LOCAL 201909191314
  expires: 2021-09-23 09:34:49 UTC
  dns: rhel610-0.ipa.local
  principal name: host/rhel610-0.ipa.local@IPA.LOCAL
  key usage: digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment
  eku: id-kp-serverAuth,id-kp-clientAuth
  pre-save command: 
  post-save command: 
  track: yes
  auto-renew: yes

The MONITORING status shows that the initial certificate request was successful. Certmonger is now tracking the certificate and will attempt to renew it when its notAfter (expiration) time approaches. We can also pretty-print the certificate to see the gory details:

[root@rhel610-0 ~]# openssl x509 -text -noout \
    < /etc/pki/tls/private/cert.pem
Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number: 19 (0x13)
    Signature Algorithm: sha256WithRSAEncryption
        Issuer: O=IPA.LOCAL 201909191314, CN=Certificate Authority
        Validity
            Not Before: Sep 23 09:34:49 2019 GMT
            Not After : Sep 23 09:34:49 2021 GMT
        Subject: O=IPA.LOCAL 201909191314, CN=rhel610-0.ipa.local
        Subject Public Key Info:
            Public Key Algorithm: rsaEncryption
                Public-Key: (2048 bit)
                Modulus:
                    00:da:ca:ca:08:d5:da:d5:79:9e:46:49:85:3f:c9:
                    ... <snip>
                Exponent: 65537 (0x10001)
        X509v3 extensions:
            X509v3 Authority Key Identifier: 
                keyid:DB:24:C2:6B:51:FD:F7:6B:25:79:6B:37:23:02:51:05:07:52:2D:39

            Authority Information Access: 
                OCSP - URI:http://ipa-ca.ipa.local/ca/ocsp

            X509v3 Key Usage: critical
                Digital Signature, Non Repudiation, Key Encipherment, Data Encipherment
            X509v3 Extended Key Usage: 
                TLS Web Server Authentication, TLS Web Client Authentication
            X509v3 CRL Distribution Points: 

                Full Name:
                  URI:http://ipa-ca.ipa.local/ipa/crl/MasterCRL.bin
                CRL Issuer:
                  DirName: O = ipaca, CN = Certificate Authority

            X509v3 Subject Key Identifier: 
                87:71:B3:6C:1D:9B:B9:7E:9D:2E:25:B0:CC:68:A4:92:FA:EE:33:C3
            X509v3 Subject Alternative Name: 
                DNS:rhel610-0.ipa.local, othername:<unsupported>, othername:<unsupported>
    Signature Algorithm: sha256WithRSAEncryption
         5e:36:e3:21:c3:14:7f:d9:1c:c1:ac:7e:12:3e:6b:34:76:a6:
         ... <snip>

Conclusion

I have shown how AD-enrolled Linux hosts can request certificates from FreeIPA. Reviewing the major considerations and steps:

  1. Create a host principal in the FreeIPA realm
  2. Retrieve the keytab
  3. Adjust /etc/krb5.conf for the FreeIPA realm (DNS-based KDC discovery means there is less to do)
  4. Add IPA CA certificate to /etc/ipa/ca.crt and add /etc/ipa/default.conf; these are needed by the Certmonger request helper
  5. Request certificate (some SELinux-fu needed if storing certs/keys in non-default locations)

The exact steps were for a RHEL 6 machine. The procedure may differ for newer systems, but not in any big ways.

In the course of exploring the procedure for this post I found it helpful to invoke the Certmonger IPA helper directly, e.g.:

[root@rhel610-0 ~]# /usr/libexec/certmonger/ipa-submit \
    -P host/rhel610-0.ipa.local@IPA.LOCAL foo.req
Submitting request to "https://rhel76-0.ipa.local/ipa/xml".
Fault 3009: (RPC failed at server.  invalid 'csr': hostname in
  subject of request 'freebsd10-0.ipa.local' does not match name
  or aliases of principal 'host/rhel610-0.ipa.local@IPA.LOCAL').
Server at https://rhel76-0.ipa.local/ipa/xml denied our request,
  giving up: 3009 (RPC failed at server.  invalid 'csr': hostname
  in subject of request 'freebsd10-0.ipa.local' does not match
  name or aliases of principal 'host/rhel610-0.ipa.local@IPA.LOCAL').

In the preceding example, I invoked the helper directly, supplying a (bogus) CSR and specifying the subject principal. The goal was not to successfully request a certificate but to verify the Kerberos configuration. If you are trying to use the IPA helper on a non-IPA-enrolled system you may also find this approach helpful for diagnosing issues.

Newer releases of Certmonger added support for requesting certificates using a different certificate profile, or a different IPA (sub-)CA. On RHEL 6, it is not possible to request a different profile so the default profile (caIPAserviceCert) is always used. IPA server on RHEL 7 and later does support modifying profiles, including the default profile.

The Certmonger IPA request helper uses /etc/ipa/ca.crt as the trust store for the HTTPS requests it makes to the FreeIPA server. If the IPA CA certificate is updated, this file will have to be updated on clients. When there are systems not IPA-enrolled á la ipa-client-install, it may be worthwhile to use configuration management tools such as Ansible to do this.

As for getting certificates from AD-CS directly, there is interest from users and customers. I would like to see it implemented, but when or by whom, or whether we even will, has not been decided as of September 2019.

September 23, 2019 12:00 AM

August 02, 2019

Nathaniel McCallum

Do Not Use ring (or rustls)

Rust is a fantastic systems language. And without a runtime, it is an exciting new language for cryptography. Further, mapping existing cryptographic libraries, such as openssl and mbedtls, into the Rust landscape requires a variety of trade-offs that would not have to be made if we had native Rust cryptography. All of this makes the desire for a native Rust crypto library very high.

Enter ring. The ring project has all appearances of a serious native Rust cryptography project. It has thousands of commits over multiple years. It has a robust test framework. What could go wrong?

The ring project is now holding pull request reviews for ransom.

I’m not joking. If you file a pull request, you will be asked for money. And it isn’t the first time.

Might I also mention that ring’s implementation doesn’t use blinding during RSA signing? Nor have they merged the latest attack mitigations for pkcs1_encode() from BoringSSL. It is easy to be fast when you’re insecure.

Then there’s the fact that they don’t do security embargoes. All disclosures are zero days. Never mind the fact that GitHub gives the ability to do all this sanely.

I’m willing to work around (and patch) some of these issues. But if I can’t contribute without a shakedown, what’s the point?

Don’t use ring.

Unfortunately, this means that rustls is now stuck. They are built on top of ring and are widely used in the Rust community. So I can’t recommend rustls until ring fixes its problems.

Don’t use rustls.

August 02, 2019 01:38 PM

Fraser Tweedale

Certificates need not be limited to the CA’s validity period

Certificates need not be limited to the CA’s validity period

All X.509 certificates have a notBefore and notAfter date. These define the validity period of the certificate. When we talk about certificate expiry, we are talking about the notAfter date. The question often arises: can a certificate’s notAfter date exceed the notAfter date of the issuer’s certificate?

The naïve intuition says, surely a certificate’s validity period cannot exceed the CA’s. But let’s think it through, and look at what these fields actually mean. According to RFC 5280 §4.1.2.5:

The certificate validity period is the time interval
during which the CA warrants that it will maintain
information about the status of the certificate.

The whole section makes no mention of the issuer’s notAfter date or validity period. It only says that the CA must maintain status (i.e. revocation) information about the issued certificate until (at least) the notAfter date.

But what if the CA certificate expires before an issued certificate? One of two things happens:

  1. The CA certificate got renewed and the verifier has a copy of the new certificate. The certificate being verified is within its validity period and so is the CA certificate, so there is a certificate path and everything is fine.
  2. The CA certificate was not renewed (or the verifier doesn’t have the renewed certificate). The certificate being verified is within its validity period, but the issuer certificate is not. So there is no certificate path; the certificate being verified cannot not be trusted.

So it is fine for issued certificate to have expiry dates beyond that of the CA.

In fact, clamping the notAfter of issued certificates to the notAfter of the CA can cause operational challenges. At the same time as the CA needs renewal, so do potentially many issued certificates! You may end up with certificates with short validity periods if the CA certificate is renewed close to its notAfter time, and a flood of renewals to perform at the same time.

There is one situation where it is required to clamp the notAfter of issued certificates to the issuer notAfter. This is when it is known that the issuer, including its CRL and OCSP facilities, will be decommissioned shortly after the expiry of the issuer certificate. Otherwise, in light of the potential operational hazards, I recommend issuing certificates with whatever validity period is appropriate for the application, regardless of when the issuer certificate expires.

August 02, 2019 12:00 AM

July 29, 2019

Nathaniel McCallum

Foreign-Architecture Docker

Sometimes you need to run a Docker image for a CPU architecture you don’t own. The multiarch project has attempted to do this in the past. But the approach they have taken requires you to run modified images. This is a lot of maintenance overhead. And it means the images are perpetually out of date.

Enter npmccallum/qemu-register. This Docker image enables the host to run unmodified foreign-architecture Docker images. Want an example?

$ sudo docker run --rm --privileged npmccallum/qemu-register
$ sudo docker run --rm aarch64/busybox uname -m
aarch64

Yup! That’s it!

Requirements

  • The host must be running Linux 4.8 or later.

How it Works

Linux 4.8 added the F flag to binfmt_misc. This flag causes the kernel to load the interpreter for the binary as soon as the configuration is loaded rather than lazily on program execution.

This configuration really helps with containers because it means the interpreter from one container can be used in another container. For example, when you load an aarch64 binary from the aarch64/busybox Docker image above, the qemu-user-aarch64 emulation binary is used from the npmccallum/qemu-register Docker image. Therefore, foreign-architecture images can run unmodified with this strategy.

The npmccallum/qemu-register Docker image is built with Fedora since they are the only major distro that has chosen this configuration by default.

Security Implications

Any qemu vulnerabilities in npmccallum/qemu-register may impact your application. This could result in an escalation of privileges. But containers don’t contain anyway. You probably don’t need to be scared if you aren’t using this for public-facing services.

July 29, 2019 06:39 PM

July 26, 2019

Fraser Tweedale

Dogtag replica range management

Dogtag replica range management

Dogtag supports distributed deployment, with multiple replicas (also called clones) processing requests and issuing certificates. All replicas read and write a replicated LDAP database. A Dogtag server can create many kinds of objects: certificates, requests, archived keys or secrets. These objects need identifiers that are unique across the deployment.

How does a Dogtag clone choose an identifier for a new object? In this post I will explain Dogtag’s range management—how it works, how it can break, and what to do if it does.

Object types with managed ranges

There are several types of objects for which Dogtag manages identifier ranges. For example:

  • Certificate serial numbers; it is essential that these be unique. Collisions are a violation of X.509 and can lead to erroneous denial of service, or false positive validity, when revocation comes into play.
  • Certificate requests (including revocation and renewal requests) are stored in the database and must have a unique ID. Clobbering of requests objects due to range conflicts can lead to renewal request failures resulting in denial of service, or worse, issuance of a valid certificate with incorrect details, allowing impersonation attacks.
  • KRA request identifiers are assigned from a managed range.
  • KRA archived key and data objects are assigned from a managed range.
  • Clones themselves are assigned identifiers when they are created; these come from managed ranges.

The identifiers themselves are unbounded nonzero integers. All of the managed ranges are separate domains. That is, the same numbers exist in each range, and the ranges are managed independently.

Active and standby ranges

For each kind of range, each replica remembers up to two range assignments. The active range is the range from which identifiers are actively assigned. When the active range is exhausted, the standby range becomes the active range and the clone acquires a new range assignment, which will be the new standby range. A clone doesn’t necessarily have a standby range at all times. It only acquires a new allocation for the standby range when the unused amount of its active range falls below some configured low water mark.

Range assignments

Range assignments are recorded in LDAP. A clone’s active and standby ranges are also recorded in the clone’s CS.cfg configuration file. A range object looks like:

dn: cn=10000001,ou=certificateRepository,ou=ranges,o=ipaca
objectClass: top
objectClass: pkiRange
beginRange: 10000001
endRange: 20000000
cn: 10000001
host: f30-1.ipa.local
SecurePort: 443

This is a serial number range assignment. Host f30-1.ipa.local has been assigned the range 10000001..20000000. It is not apparent from this object, but these are actually hexadecimal numbers! Whether the numbers are decimal or hexadecimal varies among managed ranges.

The directives in the CS.cfg on f30-1.ipa.local reflect this assignment:

dbs.enableSerialManagement=true

dbs.beginSerialNumber=fff0001
dbs.endSerialNumber=10000000

dbs.nextBeginSerialNumber=10000001
dbs.nextEndSerialNumber=20000000

dbs.enableRandomSerialNumbers=false
dbs.randomSerialNumberCounter=-1

dbs.serialCloneTransferNumber=10000
dbs.serialIncrement=10000000
dbs.serialLowWaterMark=2000000

The active range is fff0001..10000000, and the standby range is 10000001..20000000, which corresponds to the LDAP entry shown above.

Range delegation

Why is f30-1’s active range so much smaller than its standby range? This is the result of how ranges are assigned during cloning. When creating a clone, the server being configured contacts an existing clone and asks it for some configuration values, including serial/request/replica ID ranges. The existing clone delegates to the new clone a small segment of either its active or standby range. It delegates from the end of its active range, but if there are not enough numbers left in the active range, it delegates from the end the standby range instead.

The size of the range delegation is configured in CS.cfg. For example, for serial numbers it is the dbs.serialCloneTransferNumber setting. I have never heard of anyone changing the default, and I can’t think of a reason to do so.

Because the delegation is a portion of an already-assigned range (with corresponding LDAP object), new LDAP range objects are not created for delegated ranges, and the existing range object is not modified in any way. Therefore, LDAP only ever shows the original range assignments.

This range delegation procedure has been a source of bugs. For example, issue 3055 was a cloning failure when creating two clones (call them C and D) from a server that is itself a clone (call it B). Because the delegation size is fixed (the dbs.serialCloneTransferNumber setting), creating C delegates B’s whole active range to C. Unless B had a chance to switch to its standby range (when didn’t happen during cloning), creating the second clone D would fail because B’s active range was exhausted. This issue was fixed, but a more robust solution is to do away with range delegation entirely; the server can create full range assignments for the new clone instead of delegating part of its own range assignment. Issue 3060 tracks this work.

Random serial numbers

Most repositories with range management yield numbers sequentially from the active ranges. For the certificate repository only, you can optionally enable random serial numbers. Numbers are chosen by a uniform random sample from the clone’s assigned range. Dogtag checks to make sure the number was not already used; if it was used, it tries again (and again, up to a limit).

Some additional configuration values come into play when using random serial numbers:

dbs.enableRandomSerialNumbers

Enable random serial numbers (default: off)

dbs.collisionRecoverySteps

How many retries when a collision is detected (default: 10)

dbs.minimumRandomBits

Minimum size of the range, in bits (default: 4 bits)

dbs.serialLowWaterMark

Switch to standby range when there are fewer than this many serials left in the range (default: 2000000)

Critically, The dbs.minimumRandomBits does not determine how much entry is in the serial number. If many serial numbers in the range have already been used, the actual number of serials left could be less than dbs.minimumRandomBits of entropy. When issuing random serial numbers, the server keeps a running count of how many serial numbers have been used in the active range. When the range size minus the current count falls below dbs.serialLowWaterMark, the server switches to the standby range. Therefore it is dbs.serialLowWaterMark, not dbs.minimumRandomBits, that actually controls the minimum amount of randomness in the serial number.

Switching to the standby range

The actions performed by the subroutine that switches to the next range are:

1. Set the active range start and end variables to the standby range

start and end

  1. Reset the standby range start and end variables to null
  2. Reset counters
  3. Persist these changes to CS.cfg.

The switchover procedure does not acquire a new standby range assignment. Immediately after switching to the standby range, there isn’t a standby range anymore.

Acquiring a new range assignment

As currently implemented, a new standby range is only acquired at system startup. Dogtag checks each repository to see if the amount of unused numbers in the active range has fallen below the low water mark. If it has, and if there is no standby range, it self-allocates a new range assignment in LDAP. The size of the allocation is determined by CS.cfg configurables, and its lower bound is the value of the nextRange attribute in the repository parent LDAP object. It adds a range object to the ranges subtree, and updates the nextRange attribute on the repository parent. See the appendix for a list of which subtree parents and range entries are involved for each repository.

This procedure is brittle under the possibliity of LDAP replication races or transient failures. Two clones could end up adding the same range, and a replication error will occur. This can lead to identifier collisions resulting in problems later (see earlier discussion).

Internals

Most of everything discussed so far lives in the Repository class, with CertificateRepository providing additional behaviour related to random serial numbers. Code for acquiring a new range assignment lives in DBSubsystem. Some methods of interest include:

Repository.getNextSerialNumber

Get the next number; calls checkRange before returning it

Repository.checkRange

Check if the range is exhausted; if so call switchToNextRange

Repository.switchToNextRange

Switches to next range (see discussion in earlier section)

Repository.checkRanges

Sanity checks the active and standby ranges; acquires new range allocation if necessary (by calling DBSubsystem.getNextRange) and persists the changes to CS.cfg.

DBSubsystem.getNextRange

This method creates the LDAP range object and updates the nextRange attribute, returning the range bounds to the caller.

Fixing range conflicts

If you have range conflicts, the following high-level steps can be followed to fix them:

  1. Stop all Dogtag servers.
  2. Resolve any replication issues or conflict entries.
  3. Examine active and standby ranges in CS.cfg on all replicas.
  4. If there are any conflicts (including between active and standby ranges), choose new ranges such that there are no conflicts. Update CS.cfg of each replica with its new ranges.
  5. Update the nextRange attribute for each repository object to a number greater than the highest number of any allocated range (max + 1 is fine). See appendix for the objects involved.
  6. (Optional) Update and add new range entries. This is not essential because nothing will break if the ranges entries don’t actually correspond to what’s in each replica’s CS.cfg. But is is still desirable that the LDAP entries reflect the configuration of each server.
  7. Start Dogtag servers. If some servers do not have a standby range, it is a good idea to stagger their startup. Otherwise there is a high risk of an immediate replication race causing range conflicts as servers acquire new range assignments.

Note that this procedure will not save your skin if, e.g., multiple certificates with the same serial number were issued. Renewal problems may be unavoidable when collisions have occurred. This is the main reason we are switching to profile-based renewal for FreeIPA system certificates. Renewal requests refer to existing certificate and requests by serial / request ID. Thus if there have been range conflicts they are susceptible to failure or issuance of certificates with incorrect attributes. Performing a “fresh enrolment” when renewing system certificates avoids these problems because the profile enrolment request does not refer to any existing certificates or requests.

Discussion

Dogtag is over 20 years old, and I suppose that sequential numbers with range management made sense at the time. Maybe a multi-server deployment with a replicated database was not foreseen, and range management was bolted on later when the requirement emerged. Maybe using random identifiers was seen as difficult to get write; UUIDs were not widespread back then. Or maybe using random numbers was seen as not user-friendly (and that is true, but when you have more than one replica the ranged identifiers aren’t much better).

On the fact of some ranges using base 16 (hexademical) and others using base 10: I cannot even imagine why this is so. Extra user and operator pain, for what gain? I cannot tell. The reasons are probably, like so many things in old programs, lost in time.

The random serial number configuration and behaviour is… not state of the art. The program logic is difficult to follow and it is not clear which configuration directives govern the (minimum) amount of entropy in the chosen numbers.

If I were designing a system like Dogtag today, I would use random UUIDs for everything, except possibly serial numbers. There are 122 bits of entropy in a Version 4 UUID. The current CA/Browser Forum Baseline Requirements (v1.6.5) require serial numbers with 64 bits of high-quality randomness, but if that is ever increased beyond 122 bits a UUID won’t cut it anymore. So I would just use very large random numbers for all serial numbers.

Can we move Dogtag from what we have now to something more robust? Of course it is possible, but it would be a big effort. So all that is likely to happen is smaller, well understood and bounded efforts with an obvious payoff, like avoiding range delegation (Issue 3060).

The new FreeIPA Health Check system provides pluggable checks for system health. There is an open ticket to implement Dogtag range conflict and sanity checking in the Health Check tool, so that problems can be detected before they cause major failures.

Appendix: range configuration directives and objects

In all LDAP DNs below, substitute o=ipaca with the relevant base DN.

Certificate serial numbers

Base: hexademical

CS.cfg attributes:

dbs.beginSerialNumber
dbs.endSerialNumber
dbs.nextBeginSerialNumber
dbs.nextEndSerialNumber
dbs.serialIncrement

LDAP repository object (nextRange attribute):

dn: ou=certificateRepository,ou=ca,o=ipaca

LDAP ranges subtree parent:

dn: ou=certificateRepository,ou=ranges,o=ipaca

CA requests

Base: demical

CS.cfg attributes:

dbs.beginRequestNumber
dbs.endRequestNumber
dbs.nextBeginRequestNumber
dbs.nextEndRequestNumber
dbs.requestIncrement

LDAP repository object (nextRange attribute):

dn: ou=ca,ou=requests,o=ipaca

LDAP ranges subtree parent:

dn: ou=requests,ou=ranges,o=ipaca

Replica numbers

Base: demical

CS.cfg attributes:

dbs.beginReplicaNumber
dbs.endReplicaNumber
dbs.nextBeginReplicaNumber
dbs.nextEndReplicaNumber
dbs.replicaIncrement

LDAP repository object (nextRange attribute):

dn: ou=replica,o=ipaca

LDAP ranges subtree parent:

dn: ou=replica,ou=ranges,o=ipaca

KRA keys

Base: hexademical

kra/CS.cfg attributes:

dbs.beginSerialNumber
dbs.endSerialNumber
dbs.nextBeginSerialNumber
dbs.nextEndSerialNumber
dbs.serialIncrement

LDAP repository object (nextRange attribute):

dn: ou=keyRepository,ou=kra,o=kra,o=ipaca

LDAP ranges subtree parent:

dn: ou=keyRepository,ou=ranges,o=kra,o=ipaca

KRA requests

Base: demical

kra/CS.cfg attributes:

dbs.beginRequestNumber
dbs.endRequestNumber
dbs.nextBeginRequestNumber
dbs.nextEndRequestNumber
dbs.requestIncrement

LDAP repository object (nextRange attribute):

dn: ou=kra,ou=requests,o=kra,o=ipaca

LDAP ranges subtree parent:

dn: ou=requests,ou=ranges,o=kra,o=ipaca

KRA replicas numbers

Base: demical

CS.cfg attributes:

dbs.beginReplicaNumber
dbs.endReplicaNumber
dbs.nextBeginReplicaNumber
dbs.nextEndReplicaNumber
dbs.replicaIncrement

LDAP repository object (nextRange attribute):

dn: ou=replica,o=kra,o=ipaca

LDAP ranges subtree parent:

dn: ou=replica,ou=ranges,o=kra,o=ipaca

July 26, 2019 12:00 AM

July 19, 2019

Fraser Tweedale

Designing revocation self-service for FreeIPA

Designing revocation self-service for FreeIPA

The FreeIPA team recently received a feature request for self-service certificate revocation. At the moment, revocation must be performed by a privileged user with the Revoke Certificate permission. The one exception to this is that a host principal can revoke certificates for the same hostname. There are no expections when it comes to user certificates.

In this post I’ll discuss revocation self-service and how it might work in FreeIPA.

Requirements and approaches for self-service

It is critical to avoid scenarios where a user could revoke a certificate they should not be able to revoke; this would constitute a Denial-of-Service (DoS) vulnerability. Therefore FreeIPA must establish that the principal issuing the revocation request has authority to revoke the nominated certificate. Conceptually, there are several ways we might establish that authority. Each scenario has trade-offs, either fundamental to the scenario or specific to FreeIPA.

Proof of possession

Proof of possession (PoP) establishes a cryptographic proof that the operator possess the private key for the certificate to be revoked. Either they are rightful subject of the certificate, in which case it is reasonable to service their revocation request. Or they have compromised the subject’s key, in which case it is reasonable to revoke it anyway.

A PoP-based revocation system must defeat replay attacks. Using a nonce would complicate the client-server interaction—the client would first have to request the nonce, and the server would have to remember it. Instead of a nonce, it would be sufficient for the client to sign a timestamped statement of intent to revoke.

The main issue with PoP-based revocation is that the user interface must consider how to access the key. The UI must learn options related to key or certificate database paths, passphrases, and so on. This is a significant burden for users.

Finally, there is an important use case this scenario does not handle: when the user no longer has control of their private key (they deleted it, forgot the passphrase, etc.)

Certificate inspection

The revocation command could inspect the certificate and decide if it “belongs to” the requestor. This must be done with extreme care, because a false-positive is equivalent to a DoS vulnerability. For example, merely checking that the UID or CN attribute in the certificate Subject DN corresponds to the requestor is inadequate.

It is hard to attain 100% certainty, especially considering administrators can create custom certificate profiles. But there are some options that seem safe enough to implement. It should be reasonable to authorise the revocation if:

  • The Subject Alternative Name (SAN) extension contains a KRB5PrincipalName or UPN value equal to the authenticated principal. FreeIPA supports such certificates out of the box, contingent on the CSR including these data.

  • The SAN contains a rfc822Name (email address) equal to one of the user’s email addresses. Again, FreeIPA supports this with the same CSR caveat.

  • The SAN contains a directoryName (DN) equal to the user’s full DN in the FreeIPA LDAP directory. Supported, with CSR caveat.

  • The certificate Subject DN is equal to the user’s full DN in the FreeIPA LDAP directory. Supported with a custom profile having subjectNameDefaultImpl configuration like (wrapped for display):

    policyset.serverCertSet.1.default.params.name=
      UID=$request.req_subject_name.cn$,
      CN=users,CN=accounts,DC=example,DC=com

The CSR caveat presents a burden to users: they must lovingly handcraft their CSR to include the relevant data. To say the tools have poor usability in this area is an understatement. But the SAN options are supported out of the box by the default user certificate profile IECUserRoles (don’t ask about the name).

On the other hand, the Subject DN approach requires a custom profile but nothing special needs to go in the CSR. A Subject DN of CN=username will suffice.

Audit-based approach

When issuing a certificate via ipa cert-request, there are two principals at play: the operator who is performing the request, and the subject principal who the certificate is for. (These could be the same principal). Subject to organisational security policy, it may be reasonable to revoke a certificate if either of these principals requests it.

Unfortunately, in FreeIPA today we do not record these data in a way that is useful to make a revocation authorisation decision. In the future, when FreeIPA authenticates to Dogtag using GSS-API and a Kerberos proxy credential for the operator (instead of the IPA RA agent credential we use today), we will be able to store the needed data. Then it may be feasible to implement this approach. Until then, forget about it.

The way forward

So, which way will we go? Nothing is decided yet (including whether to implement this at all). If we go ahead, I would like to implement the certificate inspection approach. Proof of possession is tractable, but a lot of extra complexity and probably a usability nightmare for users. The audit-based approach is infeasible at this time, though it is a solid option if/when the right pieces are in place. Certificate inspection carries a risk of DoS exposure through revocation of inappropriate certificates, but if we carefully choose which data to inspect and match, the risk is minimised while achieving satisfactory usability.

July 19, 2019 12:00 AM

Powered by Planet