Sync your resources on Kubernetes with GitOps

5 min readMar 3, 2021

How to synchronise your secrets and configMaps hosted on Git with your Kubernetes cluster using modern tools.

We heard a lot about syncing secrets on Kubernetes using SealedSecrets, Vault, …

But I did not find an example that fits my needs and it seems like there are multiple ways to do it. Let’s make it simple !

The old dirty process before GitOps :

Create the secret
Apply it on the cluster
Backup it somewhere safe (or use tools like velero )
Make this process scheduled with a cron.

The GitOps way :

Create the secret
Save it on git safely
Wait for it to be on each namespaces you want.

One app that I heard a lot about during my Kubernetes journey is SealedSecrets.

I liked this idea of syncing secrets on my cluster with git, because with flux, infrastructure is stored on the same repo and my coworkers can play with it safely with a dedicated repository.

So the GitOps way looks great, light and simple, but, you know, sometime this is hard to make things simple.

The tools we need to do it :

Something to sync your config with GitOps approach (It looks like FluxV2 is a great app to do it).
Something to encrypt your passwords : We could use SealedSecrets or Vault.
Git repository (On this case we will use Git but you can do it wherever you want).
A tool to sync you secrets on multiple namespaces. On this case we will use kubed which is perfect for our use case.

Now, we need to install all of this on our cluster.
FluxV2 could be used to sync our secrets but also to install kubed and the encryption app (SealedSecrets here).
SealedSecrets looks easier to install than Vault which looks like overkill on our use case.

Explanation attempt :

Let’s do it.

We will take a concrete example: "Docker registry Credentials" (also named regcred on this project), the private registry secret we use to allow K8S to access our private Docker registry.

The installation / configuration process

First of all, you need Flux on your computer.

You should read the FluxV2 documentation here : https://toolkit.fluxcd.io/get-started/

Here we will install it on a mac on a kind Kubernetes Cluster.

Install Flux :

$ brew install fluxcd/tap/flux

We will create a cluster locally with kind (Kind documentation here : https://kind.sigs.k8s.io/docs/user/quick-start/

$ kind create cluster --name local-demo

Now we need a git repo and your token

Get your github token (created here) https://github.com/settings/tokens

Install FLux on your cluster :

This will install all flux components (AKA the GitOps toolkits) and will create a private git repo on your account with cluster configuration.

$ export GITHUB_TOKEN=YOUR_TOKEN# Then install flux on your local cluster :$ flux bootstrap github \
--owner=YOUR_USERNAME \
--repository=flux-infra \
--path=clusters/my-cluster \
--personal

Create two namespaces :

# Create Kubed / Flux sync namespace :$ kubectl create namespace flux-resources
$ kubectl label --overwrite namespace  flux-resources app=kubed# Create Apps namespace :$ kubectl create namespace apps
$ kubectl label --overwrite namespace  apps app=kubed

Install Kubed and SealedSecrets :

Then we need to install the two apps we need : Kubed and SealedSecrets

We will create two helm releases from two different helm sources :

For SealedSecrets :

# The helm sources : $ flux create source helm appscode \
--interval=1h \
--url=https://charts.appscode.com/stable/$ flux create source helm sealed-secrets \
--interval=1h \
--url=https://bitnami-labs.github.io/sealed-secrets# First helmrelase, sealed-secrets : $ flux create helmrelease sealed-secrets \
--interval=1h \
--release-name=sealed-secrets \
--target-namespace=flux-system \
--source=HelmRepository/sealed-secrets \
--chart=sealed-secrets \
--chart-version="1.13.x"

For kubed:

# Create values file with kubed config : 
$ cat << EOF > ./values-kubed.yaml
config:
  configSourceNamespace: kubed-sync
EOF# Then create kubed helmrelease : $ flux create helmrelease kubed \
--interval=1h \
--release-name=kubed \
--target-namespace=flux-system \
--source=HelmRepository/appscode \
--chart=kubed \
--values=./values-kubed.yaml

Flux will install SealedSecret and kubed to your cluster.

Get the private-certificate generated by sealed secret and save it to the `sealed-secrets.pem` file.

$ kubeseal \
 --controller-name=sealed-secrets \
 --controller-namespace=flux-system \
 --fetch-cert > ./sealed-secrets.pem

If you can’t find your key, check the sealed-secrets pod logs !

Create secrets

Now we want the secrets to be encrypted by SealedSecrets.

Create the regcred secret :

$ kubectl create secret docker-registry regcred \
—-docker-server=<your-registry-server> \
-—docker-username=<your-name> \
—-docker-password=<your-password> \
—-docker-email=<your-email> \
—-dry-run=client \
-oyaml > ./regcred.yaml

Add the following to the metadata part

I must say here that if you want to do it with configmaps, this is exactly the same process, you need this annotation.

annotations:
  kubed.appscode.com/sync: app=kubed

Now this is the time to seal your secret using your private key :

$ kubeseal --format=yaml --cert=sealed-secrets.pem \
< regcred.yaml > regcred-sealed.yaml#Remove this good old secret : $ rm regcred.yaml

You can see that kubeseal keeps your annotation.

Deploy the sealedSecret to the kubed-sync :

kubectl apply -f ./regcred-sealed.yaml

SealedSecrets will decrypt it, add it to your cluster as a secret, and kubed will deploy it for every namespace having the metadata label `app=kubed` (thanks to the the `kubed.appscode.com/sync: app=kubed` annotation).

That’s “almost” it

OK But, my secret is not versioned, what was the point of all of this $!??

You’re right, but now it’s your turn to do the “hard” work with Flux / GitOps stuff.

Remember, few lines ago when we installed flux on our cluster. Flux created a private bucket on your git repository, you can use it to save all your secrets, and more !

But I will not leave you alone, you can clone the following Git repository, configure it to match your needs and get everything works automagically

https://github.com/yoandl/fluxv2-sync-secrets.

Once your secrets / configMaps are safe on your git repo, you can share the sealed-secret key to your coworkers, give them correct permissions to the git repo and let them create / update secrets !

To know everything about all of this :

Thanks for reaching the end of this article, it’s my first one, I hope it’ll help some of you !

Have a nice day on your K8S Journey !