As of today, most websites are secured with TLS (https). With LetsEncrypt there is now a very affordable (free) way to get SSL certificates. Especially for us private users.

Because LetsEncrypt provides an API to retrieve and renew certificates, it is possible to add an automatic certificate service to our cluster: CertManager. With that component, we can automatically create and renew a certificate only by deploying an ingress resource with a specific annotation cert-manager.io/cluster-issuer.

Install the cert-manager.io

First we create the custom resource definitions for the cert-manager

kubectl apply --validate=false -f https://raw.githubusercontent.com/jetstack/cert-manager/v0.13.1/deploy/manifests/00-crds.yaml

Now we create a namespace where the cert-manager can live

kubectl create namespace cert-manager

Now we deploy the cert-manager to that namespace using helm

helm repo add jetstack https://charts.jetstack.io

helm repo update

helm install \
  cert-manager \
  --namespace cert-manager \
  --version v0.13.1 \
  --set webhook.enabled=false \
  jetstack/cert-manager

You may have noticed the webhook.enabled=false above. LetsEncrypt needs some kind of proof that you own a domain. To verify that, there are different challenges, which you’ll have to go through in order to get the certificate. This can be the http-01 challenge to answer over http to the LetsEncrypt challenge. But the more interesting one is the dns-01 challenge to answer the challenge request with DNS entries. With the dns-01 challenge you’ll have the possibility to create wildcard certificates for your domains, which wouldn’t be possible with the http-01 challenge. With Cert-Manager and the fact that we use CloudFlare for DNS this won’t be a problem. Cert-Manager supports CloudFlare out of the box. To enable the Cert-Manager to use CloudFlare we’ll create the Secret with the “Global API Key” from CloudFlare (in case if you wondering, no the API token from CloudFlare doesn’t work, even if that would be a more obvious choice).

cat <<EOF | kubectl apply -f-
apiVersion: v1
kind: Secret
metadata:
  name: cloudflare-api-key
  namespace: cert-manager
data:
   api-key: $(echo -n '<your-global-api-key-goes-here>' | base64)
EOF

On LetsEncrypt there are RateLimits to limit the amount of requests and certificates to be created for each week. If you’ll ever have a misconfiguration in an ingress resource or any other certificate request object (see https://cert-manager.io/docs/usage/certificate/), you will hit that limit pretty fast. I always recommend using the staging infrastructure of LetsEncrypt to test your configuration first, before using the LetsEncrypt production infrastructure. We will address that shortly.

CertManager needs an identity to request and get certificates from LetsEncrypt. We want to use both the staging and production infrastructure of LetsEncrypt, this is why we will create a ClusterIssuer for the LetsEncrypt production and the staging.

cat <<EOF | kubectl apply -f-
apiVersion: cert-manager.io/v1alpha2
kind: ClusterIssuer
metadata:
  name: letsencrypt-staging
spec:
  acme:
    # You must replace this email address with your own.
    # Let's Encrypt will use this to contact you about expiring
    # certificates, and issues related to your account.
    email: <your-email>
    server: https://acme-staging-v02.api.letsencrypt.org/directory
    privateKeySecretRef:
      # Secret resource used to store the account's private key.
      name: letsencrypt-staging-issuer-private-key
    # Add a single challenge solver, HTTP01 using nginx
    solvers:
    - dns01:
        cloudflare:
          email: <cloudflare-account-email>
          apiKeySecretRef:
            name: cloudflare-api-key
            key: api-key
---
apiVersion: cert-manager.io/v1alpha2
kind: ClusterIssuer
metadata:
  name: letsencrypt
spec:
  acme:
    # You must replace this email address with your own.
    # Let's Encrypt will use this to contact you about expiring
    # certificates, and issues related to your account.
    email: <your-email>
    server: https://acme-v02.api.letsencrypt.org/directory
    privateKeySecretRef:
      # Secret resource used to store the account's private key.
      name: letsencrypt-issuer-private-key
    # Add a single challenge solver, HTTP01 using nginx
    solvers:
    - dns01:
        cloudflare:
          email: <cloudflare-account-email>
          apiKeySecretRef:
            name: cloudflare-api-key
            key: api-key
EOF

This is it! Your cluster is now able to issue certificates for Ingress resources or any other certificate request on your cluster. Cert-Manager will renew certificates if they are about to expire.

See https://cert-manager.io/docs/usage/ingress/ for an example to make an ingress resource configure a certificate automatically. You can set the cert-manager.io/cluster-issuer to either letsencrypt-staging or letsencrypt to switch between both infrastructures.

Whats next?

I’m not that happy that the inter-container/node communication is not encrypted. The private network between the servers is already a better way than communicating over the Internet. But I might migrate away from the private network in favor of an encrypted WireGuard network between the servers.

Another topic could be to tackle the backup problem. I backup the Cluster or PersistedVolume only manually. This is not the way I want it to be. I’m writing a small controller for Kubernetes to backup the cluster on a remote ssh server using borg as backup utility, enabling me to create encrypted backups on a remote server. Since I’m writing all of it in my spare time, I’ll need a couple of weeks to get it done (maybe coding only a few (less than 3) hours a week).