Let's Encrypt Certificates - HTTP01 Challenge

Pre-Requisites

To make a HTTP challenge work, you’ll need a couple pre-requisites:

  • A Publicly registered domain (same as a DNS challenge, but you can be more flexible about who you are registered with).
  • A Public, Static IP address with an A record for your subdomain (like helloworld.gurucomputing.com.au) pointed to your public address.
  • Ports 80 and 443 forwarded to your Rancher Host

Obviously by doing this, you are exposing your ingress controller to the wide web. This is not inherently dangerous if you are using IP whitelisting. It is still less secure than not exposing your services publicly.

TL;DR

In this article, we will:

  • Set up staging certificates with let’s encrypt
  • Convert the staging certificates to trusted production certificates

What is an HTTP Challenge?

A HTTP Challenge is where you say to let’s encrypt “Hey, I have a web service running on this specific subdomain.” And let’s encrypt says “Oh yeah? Well can I access you on that domain? Stick this key there to prove it”. Let’s Encrypt then looks for that key on your webserver, and if it finds it, gives you a valid certificate.

That’s great, and it works well. But there are drawbacks. Many consumer grade ISPs won’t give you a public IP address, or allow you to host on ports 80 or 443. You also have to forward ports to your Rancher Host. You may have been planning to do that anyway, but it’s a security concern if not. You also have to generate a new certificate for every subdomain/service you want to expose (luckily Rancher will automate that for you).

However if you can’t use the DNS challenge for whatever reason then you’re going to want to use HTTP challenges instead.

Registering a Staging Issuer

Rancher does not have fillable forms for cert-manager based instructions. Instead, we will fall back to the default way of applying settings: Kubernetes Manifests!

Different providers have different structures for Kubernetes Manifests. You can read the documentation for cert-manager manifests on their documentation.

  • At the top of the rancher interface (regardless of what you have open in the sidebar) there is an Import YAML option. Open that now, and paste the following, changing the email to your email
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: letsencrypt-staging
spec:
  acme:
    server: https://acme-staging-v02.api.letsencrypt.org/directory
    email: bob@contosso.com
    privateKeySecretRef:
      name: letsencrypt-staging
    solvers:
    - http01:
        ingress:
          class: nginx

Once you have done that, you can clear your namespace filters, and find the status of our manifest in more resources→cert-manager→ClusterIssuers. You should see a registered status for your issuer:

Requesting the Staging Certificate

  • Now that we have an issuer configured, we need to create a certificate. Import the following YAML, changing:
    • mydomain to your domain
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: helloworld-mydomain-com-au-staging
  namespace: homelab
spec:
  secretName: helloworld-mydomain-com-au-staging
  issuerRef:
    name: letsencrypt-staging
    kind: ClusterIssuer
  dnsNames:
  - helloworld.mydomain.com.au

  • Alright! If all goes well, that’s enough to generate your certificate. You should get a nice active under more resources→cert manager→certificates after a while (give it up to 10 minutes, but typically within 30 seconds).

if you do not get active for your certificate, you can investigate what went wrong by looking up certificaterequests on the side panel.

  • Back in ingresses, you can now edit your ingress and choose your new staging certificate in certificates.

  • Once your ingress is saved with the new certificate, you can check if your certificate is issued by the staging authority correctly!

Moving from Staging to Production

  • Alright! Now we need to fix it up to not use the staging authority, and use the proper authority. Head back to Issuers and clone

  • On the certificates category, we can also clone the existing staging certificate:

  • Change the name to a descriptive name for the cert, and the issuerRef name and secretName from staging to production

  • Once the cert becomes active, we can change the secret in the ingress and test our website! You can see me do that below:

We now have successfully generated an automatically renewing cert for helloworld.gurucomputing.com.au. If you wish, you can also go delete the staging certificate from certficates, the staging issuer from issuers, and the staging secret from secrets. For other subdomains we just have to clone the certificate we have already created, rename it, and let cert-manager do the rest!

Creating a Let’s Encrypt Certificate for Rancher

We can also create a valid certificate for our rancher installation. However the process is a bit different (as rancher will immediately overwrite any certificate we manually change it to)

  • clear the namespace filters and navigate to More Resources→K3S→Helm Charts. Edit YAML on rancher

  • Add the following to the valuesContent section:
ingress:
  tls:
    source: secret
  extraAnnotations:
    nginx.ingress.kubernetes.io/whitelist-source-range: "127.0.0.0/8,10.0.0.0/8,172.16.0.0/12,192.168.0.0/16"

The extraAnnotations section is optional, but useful if you want to restrict who can access your rancher web interface. You may even want to lock down the interface to a smaller subnet.

  • Under cert manager→certificates, delete the existing tls-rancher-ingress certificate

apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: kub01-mydomain-com-au-production
  namespace: cattle-system
spec:
  secretName: tls-rancher-ingress
  issuerRef:
    name: letsencrypt-production
    kind: ClusterIssuer
  dnsNames:
  - kub01.mydomain.com.au

Finally under storage→secrets, delete the existing tls-rancher-ingress secret to regenerate a lets encrypt version:

  • If all goes well, after 30 seconds to 5 minutes, you can access your rancher interface with a let’s encrypt certificate!

As a side benefit, you have also added additional security measures for accessing your rancher web interface via the IP whitelist!

\