In this part we’ll learn more about management and maintenance as well as expand our knowledge on known features of Kubernetes. By the end of this part you will be able to
Label and namespace the resources inside a cluster.
Deploy stateful applications to Kubernetes
Monitor your application as well as the cluster with advanced tools
Networking Part 2
In part 1 we managed to setup networking configuration to enable routing traffic from outside of the cluster to a container inside a pod. In Part 2 we’ll focus on communication between applications.
Kubernetes includes a DNS service so communication between pods and containers in Kubernetes is as much of a challenge as it was with containers in docker-compose. Containers in a pod share the network. As such every other container inside a pod is accessible from
localhost. For communication between Pods a Service is used as they expose the Pods as a network service.
The following creates a cluster-internal IP which will enable other pods in the cluster to access the port 8080 of “example” application from http://example-service. ClusterIP is the default type for a Service.
apiVersion: v1 kind: Service metadata: name: example-service spec: type: ClusterIP selector: app: example ports: - name: http protocol: TCP port: 80 targetPort: 8080
Alternatively each Pod has an IP created by Kubernetes
Exercise 2.01: Connecting pods
Connect the main application and ping/pong application. Instead of sharing data via files use HTTP endpoints to respond with the number of pongs. Deprecate all of the volumes for the time being.
Create a Service and implement the communication.
The output will stay the same:
2020-03-30T12:15:17.705Z: 8523ecb1-c716-4cb6-a044-b9e83bb98e43. Ping / Pongs: 3
Exercise 2.02: Project v1.0
Create a new container for the backend of the TODO application.
You can use graphql or other solutions if you want.
Use ingress routing to enable access to the backend.
Create a POST /todos endpoint and a GET /todos endpoint where we can post a new todo and get all of the todos.
The todos can be saved into memory, we’ll add database later.
Frontend already has an input field. Connect it into our backend so that inputting data and pressing send will add a new todo into the list.
As you can imagine there may be a lot of resources inside a cluster. In fact, at the moment of writing this Kubernetes supports over 100 000 pods in a single cluster.
Namespaces are used to keep resources separated. A company which uses 1 cluster but has multiple projects can use namespaces to split the cluster into virtual clusters, one for each project. Most commonly they would be used to separate environments such as production, testing, staging. DNS entry for services includes the namespace so you can still have projects communicate with each other if needed through service.namespace address. e.g if the example-service from previous section was in a namespace “ns-test” it could be found from other namespaces via “http://example-service.ns-test”.
Accessing namespaces with kubectl is by using
-n flag. For example you can see what the namespace kube-system has with
$ kubectl get pods -n kube-system
To see everything you can use
$ kubectl get all --all-namespaces
Namespaces should be kept separate - you could run all of the examples and do the exercises of this course in a cluster that is shared with critical software. An administrator should set a ResourceQuota for that namespace so that you can safely run anything there. We’ll look into resource limits and requests later.
Defining a namespace is a oneliner, but requires the namespace to exist (
kubectl create namespace example-namespace):
... metadata: namespace: example-namespace name: example ...
If you’re using a namespace constantly you can set the namespace to be used by default with
kubectl config set-context --current --namespace=<name>.
Kubernetes Best Practices - Organizing Kubernetes with Namespaces
Labels are used to separate an application from others inside a namespace. They make it possible for having multiple applications as you’ve used in this course already.
Let’s look at the labels in Deployment yamls. This is the first yaml we created and you’ve copy pasted something similar:
apiVersion: apps/v1 kind: Deployment metadata: name: hashgenerator-dep spec: replicas: 1 selector: matchLabels: app: hashgenerator template: metadata: labels: app: hashgenerator spec: containers: - name: hashgenerator image: jakousa/dwk-app1:78031863af07c4c4cc3c96d07af68e8ce6e3afba
The selector and matchLabels reveal that the instructions of the deployment are directed to pods with the following label. matchLabels is a key-value pair but we could’ve used matchExpressions instead. While the template metadata includes a label with key-value pair app and hashgenerator. We can use the same label on multiple namespaces and the namespace would keep them from interfering with each other.
Exercise 2.03: Keep them separated
Create a namespace for the applications in the exercises. Move the applications to that namespace and use that in the future for all of the exercises. You can follow the material in the default namespace.
Exercise 2.04: Project v1.1
Move the project into its own namespace as well.
Configuration with Secrets and ConfigMaps
ConfigMaps help by keeping configuration separate from images.
Secrets help by keeping secrets secret.
Both can be used to introduce variables: Secrets for things like api keys and ConfigMaps for other things you would find as an environment variable.
Let’s use pixabay to display images on a simple web app. We will need to utilize authentication with api key. The api docs are good, we just need to log in to get ourselves a key here https://pixabay.com/api/docs/.
Here’s the app available. The application requires a API_KEY environment variable.
$ kubectl apply -f https://raw.githubusercontent.com/kubernetes-hy/material-example/master/app4/manifests/deployment.yaml \ -f https://raw.githubusercontent.com/kubernetes-hy/material-example/master/app4/manifests/ingress.yaml \ -f https://raw.githubusercontent.com/kubernetes-hy/material-example/master/app4/manifests/service.yaml
The requirement for an environment variable inside a secret is added to the deployment like so
... containers: - name: imageagain envFrom: - secretRef: name: pixabay-apikey
The application won’t run at first and we can see the reason with
kubectl get po and a more detailed with
kubectl describe pod imageapi-dep-....
Let’s use secret to pass the api key environment variable to the application.
Secrets use base64 encoding to avoid having to deal with special characters. We would like to use encryption to avoid printing our API_KEY for the world to see but for the sake of testing create and apply a new file secret.yaml with the following:
apiVersion: v1 kind: Secret metadata: name: pixabay-apikey data: API_KEY: aHR0cDovL3d3dy55b3V0dWJlLmNvbS93YXRjaD92PWRRdzR3OVdnWGNR # base64 encoded should look something like this, note that this won't work
As the containers are already instructed to use the environment from the secret using it happens automatically. We can now confirm that the app is working and then delete the old secret.
For encrypted secrets let’s use “Sealed Secrets”. It seems to be a solution until proven otherwise. We need to install it into our local machine as well as to our cluster. Install instructions are simple: apply the correct version to kube-system namespace.
$ kubectl apply -f https://github.com/bitnami-labs/sealed-secrets/releases/download/v0.12.1/controller.yaml
It may take a while to start but after that it’s ready to convert your secret into a sealed secret and apply it. Before that confirm that we didn’t forget to remove the old secret.
$ kubectl get secrets NAME TYPE DATA AGE default-token-jfr7n kubernetes.io/service-account-token 3 36m $ kubeseal -o yaml < secret.yaml > sealedsecret.yaml $ kubectl apply -f sealedsecret.yaml $ kubectl get secrets NAME TYPE DATA AGE default-token-jfr7n kubernetes.io/service-account-token 3 37m pixabay-apikey Opaque 1 2s
To confirm everything is working we can delete the pod and let it restart with the new environment variable
kubectl delete po imageapi-dep-.... Using SealedSecret was our first time using a custom resource - you can design your own with the help of the Kubernetes documentation.
ConfigMaps are similar but the data doesn’t have to be encoded and is not encrypted.
Exercise 2.05: Documentation and ConfigMaps
Use the official Kubernetes documentation for this exercise. https://kubernetes.io/docs/concepts/configuration/configmap/ and https://kubernetes.io/docs/tasks/configure-pod-container/configure-pod-configmap/ should contain everything you need.
Create a ConfigMap for a “dotenv file”. A file where you define environment variables that are loaded by the application. For this use an environment variable “MESSAGE” with value “Hello” to test and print the value. Implementation is up to you but the output should look like this:
Hello 2020-03-30T12:15:17.705Z: 8523ecb1-c716-4cb6-a044-b9e83bb98e43. Ping / Pongs: 3
In part 1 we learned how volumes are used with PersistentVolumes and PersistentVolumeClaims. We used Deployment with them and everything worked well enough for our testing purposes. The problem is that Deployment creates and scales pods that are replicas - they are a new copy of the same thing. With PersistentVolumeClaims, the method through which a pod reserves persistent storage, this creates a possibly non-desired effect as the claims are not pod specific. The claim is shared by all pods in that deployment.
StatefulSets are like Deployments except it makes sure that if a pod dies the replacement is identical, with the same network identity and name. In addition if the pod is scaled the copies will have their own storage. StatefulSets are for stateful applications. You could use StatefulSets to scale video game servers that require state, such as a Minecraft server. Or run a database. For data safety when deleted or scaled down StatefulSets will not delete the volumes they are associated with.
Deployment creates pods using a Resource called “ReplicaSet”. We’re using ReplicaSets through Deployments.
Let’s run redis and save some information there. We’re going to need a PersistentVolume as well as an application that utilizes the redis. In part 1 we jumped through a few hurdles to get ourselves a storage but k3s includes a helpful storageclass that will streamline local testing.
You can apply the statefulset from
apiVersion: v1 # Includes the Service for lazyness kind: Service metadata: name: redis-svc labels: app: redis spec: ports: - port: 6379 name: web clusterIP: None selector: app: redis --- apiVersion: apps/v1 kind: StatefulSet metadata: name: redis-ss spec: serviceName: redis replicas: 2 selector: matchLabels: app: redisapp template: metadata: labels: app: redisapp spec: containers: - name: redisfiller image: jakousa/dwk-app5:54203329200143875187753026f4e93a1305ae26 - name: redis image: redis:5.0 ports: - name: web containerPort: 6379 volumeMounts: - name: data mountPath: /data volumeClaimTemplates: - metadata: name: data spec: accessModes: ["ReadWriteOnce"] storageClassName: local-path resources: requests: storage: 100Mi
Looks a lot like Deployment but uses volumeClaimTemplate to claim a volume for each pod. StatefulSets require a “Headless Service” to be responsible for the network identity. We define a “Headless Service” with
clusterIP: None - this will instruct Kubernetes to not do proxying or load balancing and instead to allow access straight to the Pods.
You can now open two terminals and run
$ kubectl logs -f redis-ss-X redisfiller where X is 0 or 1. To confirm it’s working we can delete a pod and it will restart and continue right where you left off. In addition we can delete the statefulset and the volume will stay and bind back when you apply the statefulset back.
Exercise 2.06: Stateful applications
Let’s run a postgres database and save the ping/pong application counter into the database. The postgres database and ping/pong application should not be in the same pod. A single postgres database is enough and it may disappear with the cluster but it should survive even if all pods are taken down.
Exercise 2.07: Project v1.2
Create a database and save the todos there.
Use Secrets and/or ConfigMaps to have the backend access the database.
DaemonSets, like Deployments, define how to run Pods. DaemonSets differ from the other methods we’re now used to by enabling us to run Pods based on Node statuses specifically. A default use case for DaemonSet is to run a single pod on every node of the cluster. With nodeSelector proprety a DaemonSet can be targeted to specific nodes or ignore some nodes.
If you ever have a requirement to have a single pod on every node specifically you may need DaemonSets. Otherwise you would almost always use a Deployments instead. Due to this behavior a basic use case for DaemonSets is in monitoring and logging.
Our cluster and the apps in in have been pretty much a black box. We’ve thrown stuff in and then hoped that everything works all right. We’re going to use Prometheus to monitor the cluster and Grafana to view the data.
Before we can get started let’s look into how Kubernetes applications are managed more easily. Helm uses packaging format called charts to define the dependencies of an application. Among other things Helm Charts include information for the version of the chart, the requirements of the application such as the Kubernetes version as well as other charts that it may depend on.
Installation instructions are here. After that we can add the official charts repository:
$ helm repo add stable https://kubernetes-charts.storage.googleapis.com/
And after that we can install prometheus-operator. By default this would put everything to the default namespace.
$ kubectl create namespace prometheus $ helm install stable/prometheus-operator --generate-name --namespace prometheus
This added a lot of stuff to our cluster. You can remove almost everything with
helm delete [name] with the name found via
helm list. Custom resource definitions are left and have to be manually removed if the need arises.
Lets open a way into Grafana so we can see the data.
$ kubectl get po -n prometheus NAME READY STATUS RESTARTS AGE prometheus-operator-1587733290-kube-state-metrics-78dc98dc295tn 1/1 Running 0 53s prometheus-operator-1587733290-prometheus-node-exporter-ztsz8 1/1 Running 0 53s prometheus-operator-1587733290-prometheus-node-exporter-grpth 1/1 Running 0 53s prometheus-operator-1587733290-prometheus-node-exporter-sdc8b 1/1 Running 0 53s prometheus-operator-158773-operator-64dcc96864-c9svm 2/2 Running 0 53s alertmanager-prometheus-operator-158773-alertmanager-0 2/2 Running 0 34s prometheus-prometheus-operator-158773-prometheus-0 3/3 Running 1 23s prometheus-operator-1587733290-grafana-668cf4f5bb-k8xk7 1/2 Running 0 53s $ kubectl -n prometheus port-forward prometheus-operator-1587733290-grafana-668cf4f5bb-k8xk7 3000 Forwarding from 127.0.0.1:3000 -> 3000 Forwarding from [::1]:3000 -> 3000
Access http://localhost:3000 with browser and use the credentials admin / prom-operator. At the top left you can browse different dashboards.
The dashboards are nice but we’d like to know more about the apps we’re running as well. Let’s add Loki so that we can see logs. To confirm everything works for us let’s create a simple application that’ll output something to stdout.
Let’s run the redisapp from previously
https://raw.githubusercontent.com/kubernetes-hy/material-example/master/app5/manifests/statefulset.yaml. We can keep it running as it’ll generate a good amount of log output for us.
The Loki Chart includes almost everything:
$ helm repo add loki https://grafana.github.io/loki/charts $ helm repo update $ kubectl create namespace loki-stack namespace/loki-stack created $ helm upgrade --install loki --namespace=loki-stack loki/loki-stack $ kubectl get all -n loki-stack NAME READY STATUS RESTARTS AGE pod/loki-promtail-n2fgs 1/1 Running 0 18m pod/loki-promtail-h6xq2 1/1 Running 0 18m pod/loki-promtail-8l84g 1/1 Running 0 18m pod/loki-0 1/1 Running 0 18m NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE service/loki ClusterIP 10.43.170.68 <none> 3100/TCP 18m service/loki-headless ClusterIP None <none> 3100/TCP 18m NAME DESIRED CURRENT READY UP-TO-DATE AVAILABLE NODE SELECTOR AGE daemonset.apps/loki-promtail 3 3 3 3 3 <none> 18m NAME READY AGE statefulset.apps/loki 1/1 18m
Here we see that Loki is running in port 3100. Open Grafana and go to settings and “Add data source”. Choose Loki and then insert the correct URL. From the output above we can guess that the port should be 3100, namespace is loki-stack and the service is loki. So the answer would be: http://loki.loki-stack:3100. No other fields need to be changed.
Now we can use the Explore tab (compass) to explore the data.
Exercise 2.08: Project v1.3
The project could really use logging.
Add request logging so that you can monitor every todo that is sent to the backend.
Set the limit of 140 characters for todos into the backend as well. Use Postman or curl to test that too long todos are blocked by the backend and you can see the non-allowed messages in your grafana.
Submit your completed exercises through the submission application
We’re now at the state where we have the knowledge to deploy most software we’d develop into a Kubernetes cluster. Googling and reading documentation will still be necessary, as always, but we can confidently move from our own local Kubernetes cluster and start using Google Kubernetes Engine. Part 3