Fix Lets Encrypt Issuer Long standing limitations - #1701
Merged
Conversation
The Let's Encrypt step templates held a hardcoded list of intermediate CA
common names, with a second list for staging. Let's Encrypt rotates its
intermediates on its own schedule. Once it does, any certificate issued by
the new intermediate is invisible to the next run of the step, because the
issuer common name is not in the list.
The step does not fail when that happens. It falls through to the "no
existing certificate" path and publishes a duplicate instead of renewing
the one already in the store.
The lists are already stale. Neither YR2 nor "(STAGING) Ersatz Emmer YR2"
appears in them, and both are in active use. E5, E6, E9 and R14 are
missing from the production list too.
This replaces both lists with two things that do not churn:
- the issuer organization, which has been "Let's Encrypt" since 2015
- the "(STAGING) " prefix on the issuer common name, which is what
separates the environments
The prefix carries the distinction. Generation Y staging intermediates use
the production organization, so the organization alone cannot tell the two
apart:
production CN=YR2,O=Let's Encrypt,C=US
staging CN=(STAGING) Ersatz Emmer YR2,O=Let's Encrypt,C=US
The step treats both "Let's Encrypt" and "(STAGING) Let's Encrypt" as
Let's Encrypt, because the 2024 generation of staging intermediates uses
the latter. The original "Fake LE Intermediate X1" carries no organization
at all, so the step matches that one on common name instead.
The step reads IssuerOrganization and IssuerCommonName from the
certificates API, and parses IssuerDistinguishedName when a server does
not populate them. The match only widens, so the step still finds every
certificate it found before, and no step parameters change.
All six templates in the lets-encrypt category get the same change.
The step checked $expiring_certificates to decide whether to renew, then replaced $certificates | Select-Object -First 1. Those are different sets. A domain can hold more than one certificate in the store, which is easy to end up with because the previous behaviour published duplicates whenever the issuer list went stale. When that happens, whether the step renews the right certificate comes down to the order the search API returns them in. Get it wrong and the step renews a healthy certificate while the expiring one lapses. Selects from $expiring_certificates instead, sorted so the certificate closest to expiry wins when several are due. Also logs which certificate Id is being replaced, which matters when a domain has more than one. Both versions filter to PKCS#12, so a store with no PKCS#12 certificate fails the same way it did before. The two differ only when an expiring certificate is PEM and a non-expiring PKCS#12 exists alongside it, where the old code renewed the healthy one. Failing there is the better outcome. All six templates in the lets-encrypt category get the same change.
|
Review this PR in Hyponome for a side-by-side diff of the step-template JSON and any embedded scripts. |
Contributor
Author
|
@benjimac93 - tagging you in case you wanted to take a look (I think you have used these in the past). Feedback welcome 👍 |
Contributor
|
@harrisonmeister this seems a LOT more sensible - have pulled this into our instance this morning, and is working as expected :) |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Background
The Let's Encrypt step templates matched certificates against a hardcoded list of intermediate CA common names, with a second list for staging. Let's Encrypt rotates its intermediates on its own schedule, so the lists go stale and the step stops recognising certificates it created.
The step doesn't fail when that happens. It falls through to the "no existing certificate" path and publishes a duplicate instead of renewing what's already in the store.
Both lists are stale today.
YR2and(STAGING) Ersatz Emmer YR2are in active use and appear in neither, andE5,E6,E9andR14are missing from the production list while their staging counterparts are present.Results
Two commits, applied to all six templates in the
lets-encryptcategory.Match on the issuer organization and the staging prefix. The organization has been
Let's Encryptsince 2015, and staging common names carry a documented(STAGING)prefix. Neither changes when an intermediate rotates.The prefix is what separates the environments, not the organization. Generation Y staging intermediates carry the production organization:
The step accepts both organization values, because the 2024 generation of staging intermediates uses
O=(STAGING) Let's Encrypt. The originalFake LE Intermediate X1has no organization at all, so the step matches that one on common name. It readsIssuerOrganizationandIssuerCommonNamefrom the certificates API, and parsesIssuerDistinguishedNamewhen a server doesn't populate them.Replace the certificate that's expiring. The step tested
$expiring_certificatesto decide whether to renew, then replaced$certificates | Select-Object -First 1. Those are different sets, so with more than one certificate for a domain the step could renew a healthy certificate and let the expiring one lapse, depending on the order the search returned them in. It now selects from$expiring_certificates, sorted so the closest to expiry wins, and logs which Id it replaced.No parameters change. Matching only widens, so every certificate found before is still found. This doesn't clean up duplicates already in a store, it stops new ones being created.
Before
A staging run against a domain whose certificate came from
(STAGING) Ersatz Emmer YR2finds nothing, because that name was never added to the staging list:A duplicate is published rather than the existing certificate being renewed.
After
The same store, filtered with the new matching. Both certificates share an organization and only the prefix separates them:
A production run selects the other one. From a live renewal where the domain had two certificates and only one was near expiry:
Tested against a live instance across two domains in both modes (staging vs live). A sample of what changes, and what doesn't:
YR2Let's Encrypt(STAGING) Ersatz Emmer YR2Let's Encrypt(STAGING) Ersatz Emmer YR2Let's EncryptR12Let's EncryptE5Let's Encrypt(STAGING) Wannabe Watercress R11(STAGING) Let's EncryptFake LE Intermediate X1ZQ7(hypothetical future intermediate)Let's EncryptDigiCert TLS RSA SHA256 2020 CA1DigiCert IncThe last row matters as much as the others: widening the match mustn't pull in certificates from other authorities that happen to share a subject. Rows three and nine confirm nothing that was previously ignored is now selected, and no row goes the other way, from renewing to missing.
The script body parses cleanly, and the matching was compared against every common name in the old arrays using Claude code: 480 identical verdicts, with no case where the old code matched and the new code doesn't.
Versionbumps:No new
Category, and no parameters added or changed.