Month: April 2024

Mozilla SOPS To Protect My cloudflared Secrets In Kubernetes

Summary

Aren’t these titles getting ridiculous? When talking about some of these stacks, you need a laundry list of names to drop. In this case I was working on publishing my CloudFlare Tunnels FTW work that houses my kind lab into my public GitHub Repository. I wanted to tie in FluxCD to it and essentially be able to easily blow away the cluster and recreate with secrets all through FluxCD.

I was able to successfully achieve that with all but the private key which needs to be manually loaded into the cluster so it can decrypt the sensitive information.

Why Do We Care About This?

While trying to go fully GitOps for Kubernetes, everything is stored in a Git Repository. This makes change management extremely simple and reduces complexities of compliance. Things like policy bots can automate change approval processes and document. But generally everything in Git is clear text.

Sure, there are private repositories but do all the the developers that work on the project need to read sensitive records like passwords for that project? Its best that they don’t and as a developer you really don’t want that responsibility!

Mozilla SOPS To The Rescue!

Mozilla SOPS is very well documented. In my case I’m using Flux which also has great documentation. For my lab, this work is focusing on “cluster3” which simply deploys my https://www.woohoosvcs.com and https://tools.woohoosvcs.com in my kind lab for local testing before pushing out to production.

Create Key with Age

Age appears to be the preferred encryption tool to use right now. It is pretty simple to use and going by the flux documentation we simply need to run

age-keygen -o age.agekey

This will create a file that contains both the public and private key. The public key will be in the comment and the command line will output the public key. We will need the private key later to add as a secret manually to decrypt. I’m sure there are ways of getting this into the cluster securely but for this blog article this is the only thing done outside of GitOps.

Let’s Get To the Details!

With Flux I have a bootstrap script to load flux into the environment. I also have a generate_cluster3.sh script that creates the yaml.

The pertinent lines to add to it above the standard are the following. The first line indicates that sops is a decryption provider. The second indicates the name of the secret to be stored. Flux requires this to be in the flux-system namespace

    --decryption-provider=sops \
    --decryption-secret=sops-age \

From there you simpley need to run the bootstrap_cluster3.sh which just loads the yaml manifests for flux. With flux you can do this on the command line but I preferred to have this generation and bootstrapping in Git. As you want to upgrade flux there’s also a upgrade_cluster3.sh script that is really a one liner.

flux install --export > ./clusters/cluster3/flux-system/gotk-components.yaml

This will update the components. If you’re already bootstrapped and running flux, you can run this and commit to push out the upgrades to use flux to upgrade itself!

In the root of the cluster3 folder I have .sops.yaml. This tells the kustomization module in flux what to decrypt and which public key to use.

Loading Private Key Via Secret

Once you have run the bootstrap_cluster3.sh you can then load the private key via

cat age.agekey | kubectl create secret generic sops-age \
  --namespace=flux-system --from-file=age.agekey=/dev/stdin

Caveat

This lab won’t work for you out of the box. This is because it requires a few confidential details

  1. My cloudflared secret is encrypted with my public key. You do not have my private key so you cannot load it into your cluster to decrypt it
  2. I have some private applications I am pushing into my kind cluster. You will have to clone and modify for your needs