Kubernetes Part 6: Install and Configure Cert-manager for Let's Encrypt SSL certificates

When the nginx-ingress controller has been implemented (see here), the next step is to implement cert-manager. Using cert-manager is a great way of providing your apps (websites) with a legitimate  Let's Encrypt SSL certificate, automatically when configuring an ingress rule.

The installation procedure is as follows:

- Install cert-manager. 

SSH into the kubernetes master node. This will create a new namespace calle cert-manager.

# Download and create cert-manager installation yaml
curl -sL \
https://github.com/jetstack/cert-manager/releases/download/v1.3.1/cert-manager.yaml |\
sed -r 's/(image:.*):(v.*)$/\1-arm:\2/g' > cert-manager-arm.yaml

# Apply the cert-manager installation yaml
kubectl apply -f cert-manager-arm.yaml

- Public DNS & Portforwarding
- Add public DNS record for required certificate (for example nginx-1.mydomain.com).
- Portforward port 80, 443 from your public internet ip to the K8S external address via your (home) router Check my post on MetalLB here to find your external IP. For more information on how-to portforward ports from your router, this guide might provide you some information.

- Configure letsencrypt-staging.yaml

Next type the following command:

nano issuer-letsencrypt-staging.yml

Paste the text below and save the file (Ctrl-O, Ctrl-X). Change the red value into your own equivalent.

apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: letsencrypt-staging
spec:
  acme:
    # The ACME server URL
    server: https://acme-staging-v02.api.letsencrypt.org/directory
    # Email address used for ACME registration
    email: <your_email>@example.com
    # Name of a secret used to store the ACME account private key
    privateKeySecretRef:
      name: letsencrypt-staging
    # Enable the HTTP-01 challenge provider
    solvers:
    - http01:
        ingress:
          class: nginx

Apply the file in kubernetes with the command below

kubectl create -f issuer-letsencrypt-staging.yml -n cert-manager

 When the file is applied, you have created a ClusterIssuer for certificates in Kubernet. A ClusterIssuer can provide ssl certificates for all namespaces within the cluster. An Issuer can only provide ssl certificates for the namespace it resides.

- Check the staging ClusterIssuer installation

kubectl get clusterissuers

You should see that the letsencrypt-staging ClusterIssuer has been created.

We now have created a staging ClusterIssuer. It's best practice to first test cert-manager with a staging issuer, so you can check if everything is in place (firewall, network etc..) before configuring the production ClusterIssuer. 

- Test certificate staging ClusterIssuer

To the the ClusterIssuer, we create a new certificate request via yaml file, so do the following

nano nginx-1-test-certificate.yaml

Adjust and past the text. As in previous posts red are the example values that needs to be adjusted to your own values.

apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: nginx-1-mydomain-com
  namespace: default
spec:
  secretName: nginx-1-mydomain-com-tls
  issuerRef:
    name: letsencrypt-staging
    kind: ClusterIssuer
  commonName: nginx-1.mydomain.com
  dnsNames:
  - nginx-1.mydomain.com

Before you apply the yaml file, doublecheck if the portforwarding and DNS records are in place. The DNS record doesn't have to be and A record a CNAME record will work as well. Port 80 is required to be forwarded from the router to the external ip of the ingress contoller. Let's encrypt will check if the host is available via port 80, before generating a certificate. 

If all prerequistes are in place you can apply the yaml file and a certificate should be generated.

Apply the yaml file via the command:

kubectl apply -f nginx-1-test-certificate.yaml

You can check if the certificate has been generated via the command:

kubectl get certificates

You should see something similar like this (red values are example values)

erikdebont@k8s-master:~$ kubectl get certificates
NAME                       READY   SECRET                         AGE
nginx-1.mydomain-com-tls   True    nginx-1.mydomain.com-tls        3d2h
erikdebont@k8s-master:~$

If the READYstatus is FALSE you need to do some troubleshooting. If it's TRUE, the certificate has been generated and the staging proces went ok. It normally should not take longer than 10 minutes to generate a certificate, if it takes longer, than something went wrong. If the status is TRUE you can skip the trouble shooting guide and go the  Configure letsencrypt-prod.yaml section

- Troubleshooting certificate request issues

You can start the troubleshooting with the following command:

kubectl describe certificates nginx-1-mydomain-com-tls

This will return information about the certificates proces. There also should be reference to a certificate request. Something like  certificaterequest nginx-1-mydomain-com-12324563772 

So the next step it to describe the request:

kubectl describe certificaterequest nginx-1-mydomain-com-12324563772 

In this description you can find the order entry. This should be something like "Waiting on certificate issuance from order default/nginx-1-mydomain-com-12324563772-2342473670.

So let's describe the order

kubectl describe orders default/nginx-1-mydomain-com-12324563772-2342473670

In the order you look for the event that says Created Challenge resource "nginx-1-mydomain-com-12324563772-2342473670-1834560396" for domain "nginx-1.mydomain.com"

So let's describe the challenge:

kubectl describe challenge nginx-1-mydomain-com-12324563772-2342473670-1834560396

The last event returned from here is Presented challenge using http-01 challenge mechanism. That looks ok, so we scan up the describe output and see a message Waiting for http-01 challenge propagation: failed to perform self check GET request … no such host. Finally! We have found the problem! In this case, no such host means that the DNS lookup failed, so then we would go back and manually check our DNS settings and that our domain's DNS resolves correctly for us and make any changes needed.

(Thanks to https://opensource.com/article/20/3/ssl-letsencrypt-k3s for the excellent troubleshooting guide)

- Configure letsencrypt-prod.yaml

Before you start production configuration first cleanup the test request via the following command:

kubectl delete -f nginx-1-test-certificate.yaml

If creating a certificate via the letsencrypt-staging ClusterIssuer went ok, the backend is properly configured to install the producton ClusterIssuer. To implement the production ClusterIssuer create the following yaml-file. This is basically the same as the staging yaml file, with a couple of minor changes.

So type the following command:

nano issuer-letsencrypt-prod.yml

Paste the text below and save the file (Ctrl-O, Ctrl-X). Change the red value into your own equivalent.

apiVersion: cert-manager.io/v1alpha2
kind: ClusterIssuer
metadata:
  name: letsencrypt-prod
spec:
  acme:
    # The ACME server URL
    server: https://acme-v02.api.letsencrypt.org/directory
    # Email address used for ACME registration
    email:  <your_email>@example.com
    # Name of a secret used to store the ACME account private key
    privateKeySecretRef:
      name: letsencrypt-prod
    # Enable the HTTP-01 challenge provider
    solvers:
    - http01:
        ingress:
          class: nginx

Apply the file in kubernetes with the command below

kubectl create -f issuer-letsencrypt-prod.yml -n cert-manager

- Check the production ClusterIssuer installation

kubectl get clusterissuers

You now should two ClusterIssuers (staging and production)

Now certificates can be rolled with via Ingress in kubernetes by adding the follwing lines in the ingress.yaml file. Ingress will make sure there is a response when Let's Encrypt check the connection to host on port 80, before generating a valid ssl certificate.

apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
  name: nginx-1
  annotations:
    kubernetes.io/ingress.class: "nginx"
    cert-manager.io/cluster-issuer: "letsencrypt-prod" # < This line tells ingress which issues to use

spec:
  rules:
  - host: nginx-1.mydomain.com
    http:
      paths:
      - path: /
        backend:
          serviceName: nginx-1-service
          servicePort: 80
  tls: # < placing a host in the TLS config will indicate a cert should be created
  - hosts:
    - nginx-1.mydomain.com
    secretName: nginx-1-mydomain-com-cloud-tls # < cert-manager will store the created certificate in this secret.

You can do a full test with the yaml file below. This will create test website nginx-1 with a certificate generated by Let's Encrypt through Ingress. Therefore do the following:

nano deploy-nginx-1-k8s-ssl.yml

 Copy and past the text below in nano, and save the file

apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-1
  labels:
    app: nginx-1
spec:
  replicas: 1
  selector:
    matchLabels:
      app: nginx-1
  template:
    metadata:
      labels:
        app: nginx-1
    spec:
      containers:
      - image: nginx
        name: nginx-1
---
kind: Service
apiVersion: v1
metadata:
  name: nginx-1-service
spec:
  selector:
    app: nginx-1
  ports:
    - protocol: TCP
      port: 80
---
apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
  name: nginx-1
  annotations:
    kubernetes.io/ingress.class: "nginx"
    cert-manager.io/cluster-issuer: "letsencrypt-prod"

spec:
  rules:
  - host: nginx-1.mydomain.com
    http:
      paths:
      - path: /
        backend:
          serviceName: nginx-1-service
          servicePort: 80
  tls: 
  - hosts:
    - nginx-1.mydomain.com
    secretName: nginx-1-mydomain-com-tls 

Apply the script via the following command:

kubectl apply -f deploy-nginx-1-k8s-ssl.yml

To check if its working you should go to your browser and type https://nginx-1.mydomain.com. (mydomain.com should be replaced with your own domain) you should see default nginx html page, like the example below. You can check the certificate that has been generated.

If you have any questions, do not hesitate to leave a comment.

In the next post I will be writing how-to install and configure Kubernetes Dashboard