GitOps

An average simple deployment pipeline we have used and learned about is something like this.

1. Developer runs git push with modified code. E.g. to GitHub
2. This triggers a CI/CD service to start running. E.g. to GitHub actions
3. CI/CD service runs tests, builds an image, pushes the image to a registry and deploys the new image. E.g. to Kubernetes

This is called a push deployment. It is a descriptive name as everything is pushed forward by the previous step. There are some challenges with the push approach. For example, if we have a Kubernetes cluster that is unavailable for external connections i.e. the cluster on your local machine or any cluster we don't want to give outsiders access to. In those cases having CI/CD push the update to the cluster is not possible.

In a pull configuration the setup is reversed. We can have the cluster, running anywhere, pull the new image and deploy it automatically. The new image will still be tested and built by the CI/CD. We simply relieve the CI/CD of the burden of deployment and move it to another system that is doing the pulling.

GitOps is all about this reversal and promotes good practices for the operations side of things. This is achieved by having the state of the cluster be in a git repository. So besides handling the application deployment it will handle all changes to the cluster. This will require some additional configuration and rethinking past the tradition of server configuration. But when we get there GitOps will be the final nail in the coffin of imperative cluster management.

Flux will be the tool of choice. At the end our workflow should look like this:

1. Developer runs git push with modified code.
2. CI/CD service starts running.
3. CI/CD service builds and pushes new image and commits edit to "release" branch
4. Flux will take the state described in the release branch and set it as the state of our cluster.

To get started we'll need to get a GITHUB_TOKEN. You can follow the GitHub guide to click settings - developer settings - personal access tokens. And generate a token with all "repo" access rights. You may be able to avoid one or more of the access rights but we're okay with 100% access for a limited period. Save the variable for now.

Next step isn't a surprise at this point. As with most tools this time we will need to install the Flux CLI.

$ curl -s https://fluxcd.io/install.sh | sudo bash

or if that doesn't work read installation guide.

flux check will tell us if something is wrong with the cluster itself.

$ flux check --pre
  ► checking prerequisites
  ✔ Kubernetes 1.22.2+k3s2 >=1.19.0-0
  ✔ prerequisites checks passed

Everything looks green. Now we'll configure our cluster and our GitOps repository. We will need the token for the next step. CLI will read it from the environment so run export GITHUB_TOKEN=3dcb4daba731d77158cbac4dabe7ad1f2 with you own token now.

Now is a good time to make sure we are pointed at the right cluster. Bootstrapping flux to a cluster will install a lot of things. Read the following command instead of copy-pasting it. In this case, we use GitHub, the owner is your username and repository to be created is "kube-cluster-dwk". The cluster is personal (if omitted, we can set owner as organisation) and we don't need a private repo. There is a lot to configure and you can run flux bootstrap github --help if you're interested.

$ flux bootstrap github \
    --owner=<YOUR_USERNAME> \
    --repository=kube-cluster-dwk \
    --personal \
    --private=false

  ...

  ✔ bootstrap finished

That's it for flux CLI. That's it for kubectl apply as well. Do not use kubectl apply in this GitOps section. At least avoid using it since we should not need it.

Clone the new repository you just created and create two new files in it example-source.yaml and example-gitops-app.yaml. We will fill them now that we have a new CRDs GitRepository and Kustomization (not to be confused with kustomize) available to us.

example-source.yaml

apiVersion: source.toolkit.fluxcd.io/v1beta1
kind: GitRepository
metadata:
  name: example-repo
  namespace: flux-system
spec:
  interval: 10m
  url: https://github.com/kubernetes-hy/material-example
  ref:
    branch: master

This one is simply the repository. We'll want to observe the master branch. The fields used here can mostly be deduced from the context but read the documentation for other options and explanations.

example-gitops-app.yaml

apiVersion: kustomize.toolkit.fluxcd.io/v1beta1
kind: Kustomization
metadata:
  name: example-gitops-app
  namespace: flux-system
spec:
  sourceRef:
    kind: GitRepository
    name: example-repo
  interval: 10m
  path: ./4-gitops/manifests # Path tells where to find the files. Excellent for "monorepos" where you have multiple different applications in one repository, like the example repository.
  prune: true # This will make sure that deleting the file will delete the resource
  validation: client # Who validates the objects. Server or the client.

And this is the application within that repository and takes care of the manifests. Kustomization will either look for kustomization.yaml within the path or if none found generate one that contains all Kubernetes manifests in it. Now simply git add both of them, git commit and git push them to the repository. After a short while, you will have hashgenerator pod running

$ kubectl get pods
  NAME                                 READY   STATUS    RESTARTS   AGE
  hashgenerator-dep-558c84888d-qh4t9   1/1     Running   0          13m

Now we've done our first deploy. We can delete it simply by deleting the files from the repository, no kubectl delete required. To get it to follow our updates to the application repository we will need to do configure the CI/CD of the application to update the manifest files.

Time to create a plan and then open GitHub actions. This will be a lot simpler than the one we had to deal with previously. But first, create a kustomization.yaml to have easier access to the image name and tag. I will be showing the example using one of the apps (4-gitops) in the material-example repository. The repository also contains the directory structure. You can copy the application or use your own as you follow the example.

kustomization.yaml

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- manifests/deployment.yaml
images:
- name: IMAGE_PLACEHOLDER
  newName: jakousa/dwk-4-gitops-app
  newTag: fdafd7088d04892815ed037cf30ca4f61c7af2f7

After this and replacing the image in the deployment.yaml with IMAGE_PLACEHOLDER we're ready to automate updates with the following steps:

1. Build the image and push it to registry
2. Update the tag in the yaml to match the new version
3. Commit and push it to the repository

name: Release 4-gitops-app

on:
  push:
    branches:
      - master
    paths:
      - '4-gitops/app/**'
      - '.github/workflows/gitops-app.yml'

jobs:
  build:
    name: Build
    runs-on: ubuntu-latest
    steps:
    - uses: actions/checkout@v2

    - name: Login to Docker Hub
      uses: docker/login-action@v1
      with:
        username: ${{ secrets.DOCKER_USERNAME }}
        password: ${{ secrets.DOCKER_PASSWORD }}

    - name: Build and Push
      uses: docker/build-push-action@v2
      with:
        context: 4-gitops/app
        push: true
        tags: jakousa/dwk-4-gitops-app:${{ github.sha }}

  deploy:
    name: Deploy
    runs-on: ubuntu-latest
    needs: build

    steps:
    - uses: actions/checkout@v2

    # Set up kustomize
    - name: Set up Kustomize
      uses: imranismail/setup-kustomize@v1

    # Update yamls
    - name: Update yamls
      working-directory: 4-gitops/manifests
      run: |-
        kustomize edit set image IMAGE_PLACEHOLDER=jakousa/dwk-4-gitops-app:${{ github.sha }}

    # Commit and push
    - uses: EndBug/add-and-commit@v7
      with:
        add: '4-gitops/manifests/kustomization.yaml'
        message: New version release for gitops-app ${{ github.sha }}
      env:
        GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}

That should take care of that. Now any updates to the source code will automatically be released.

Anything you installed with Helm can also be moved to the infrastructure repository. Read through the documentation to know more.

With GitOps we achieved

• Better security

  • Nobody needs access to the cluster, not even CI/CD services. No need to share access to the cluster with collaborators; they will commit changes like everyone else.

• Better transparency

  • Everything is declared in the GitHub repository. When a new person joins the team they can check the repository; no need to pass ancient knowledge or hidden techniques as there are none.

• Better traceability

  • All changes to the cluster are version controlled. You will know exactly what was the state of the cluster and how it was changed and by whom.

• Risk reduction

  • If something breaks simply revert the cluster to a working commit. git revert and the whole cluster is in a previous state.

• Portability

  • Want to change to a new provider? Spin up a cluster and point it to the same repository - done your cluster is now there.

There are a few options for the GitOps setup. What we used here was having the configuration for the application in the same repository with the application itself. That required us to do some changes in the directory structure. Another option is to have the configuration separate from the source code. That approach also removes the risk of having a pipeline loop where your pipeline commits to the repository which then triggers the pipeline.