How to synchronise your secrets and configMaps hosted on Git with your Kubernetes cluster using modern tools.
But I did not find an example that fits my needs and it seems like there are multiple ways to do it. Let’s make it simple !
The old dirty process before GitOps :
- Create the secret
- Apply it on the cluster
- Backup it somewhere safe (or use tools like velero )
- Make this process scheduled with a cron.
The GitOps way :
- Create the secret
- Save it on git safely
- Wait for it to be on each namespaces you want.
One app that I heard a lot about during my Kubernetes journey is SealedSecrets.
I liked this idea of syncing secrets on my cluster with git, because with flux, infrastructure is stored on the same repo and my coworkers can play with it safely with a dedicated repository.
So the GitOps way looks great, light and simple, but, you know, sometime this is hard to make things simple.
The tools we need to do it :
- Something to sync your config with GitOps approach (It looks like FluxV2 is a great app to do it).
- Something to encrypt your passwords : We could use SealedSecrets or Vault.
- Git repository (On this case we will use Git but you can do it wherever you want).
- A tool to sync you secrets on multiple namespaces. On this case we will use kubed which is perfect for our use case.
Now, we need to install all of this on our cluster.
FluxV2 could be used to sync our secrets but also to install kubed and the encryption app (SealedSecrets here).
SealedSecrets looks easier to install than Vault which looks like overkill on our use case.
Explanation attempt :
Let’s do it.
We will take a concrete example: "Docker registry Credentials" (also named regcred on this project), the private registry secret we use to allow K8S to access our private Docker registry.
The installation / configuration process
First of all, you need Flux on your computer.
Here we will install it on a mac on a kind Kubernetes Cluster.
Install Flux :
$ brew install fluxcd/tap/flux
We will create a cluster locally with kind (Kind documentation here : https://kind.sigs.k8s.io/docs/user/quick-start/
$ kind create cluster --name local-demo
Now we need a git repo and your token
Get your github token (created here) https://github.com/settings/tokens
Install FLux on your cluster :
This will install all flux components (AKA the GitOps toolkits) and will create a private git repo on your account with cluster configuration.
$ export GITHUB_TOKEN=YOUR_TOKEN# Then install flux on your local cluster :$ flux bootstrap github \
Create two namespaces :
# Create Kubed / Flux sync namespace :$ kubectl create namespace flux-resources
$ kubectl label --overwrite namespace flux-resources app=kubed# Create Apps namespace :$ kubectl create namespace apps
$ kubectl label --overwrite namespace apps app=kubed
Install Kubed and SealedSecrets :
Then we need to install the two apps we need : Kubed and SealedSecrets
We will create two helm releases from two different helm sources :
For SealedSecrets :
# The helm sources : $ flux create source helm appscode \
--url=https://charts.appscode.com/stable/$ flux create source helm sealed-secrets \
--url=https://bitnami-labs.github.io/sealed-secrets# First helmrelase, sealed-secrets : $ flux create helmrelease sealed-secrets \
# Create values file with kubed config :
$ cat << EOF > ./values-kubed.yaml
EOF# Then create kubed helmrelease : $ flux create helmrelease kubed \
Get the private-certificate generated by sealed secret and save it to the `sealed-secrets.pem` file.
$ kubeseal \
--fetch-cert > ./sealed-secrets.pem
If you can’t find your key, check the sealed-secrets pod logs !
Now we want the secrets to be encrypted by SealedSecrets.
Create the regcred secret :
$ kubectl create secret docker-registry regcred \
-oyaml > ./regcred.yaml
Add the following to the metadata part
I must say here that if you want to do it with configmaps, this is exactly the same process, you need this annotation.
Now this is the time to seal your secret using your private key :
$ kubeseal --format=yaml --cert=sealed-secrets.pem \
< regcred.yaml > regcred-sealed.yaml#Remove this good old secret : $ rm regcred.yaml
You can see that kubeseal keeps your annotation.
Deploy the sealedSecret to the kubed-sync :
kubectl apply -f ./regcred-sealed.yaml
SealedSecrets will decrypt it, add it to your cluster as a secret, and kubed will deploy it for every namespace having the metadata label `app=kubed` (thanks to the the `kubed.appscode.com/sync: app=kubed` annotation).
That’s “almost” it
OK But, my secret is not versioned, what was the point of all of this $!??
You’re right, but now it’s your turn to do the “hard” work with Flux / GitOps stuff.
Remember, few lines ago when we installed flux on our cluster. Flux created a private bucket on your git repository, you can use it to save all your secrets, and more !
But I will not leave you alone, you can clone the following Git repository, configure it to match your needs and get everything works automagically
Once your secrets / configMaps are safe on your git repo, you can share the sealed-secret key to your coworkers, give them correct permissions to the git repo and let them create / update secrets !
To know everything about all of this :
Thanks for reaching the end of this article, it’s my first one, I hope it’ll help some of you !
Have a nice day on your K8S Journey !