Run and Scale a Distributed Crossword Puzzle App with CI/CD on Kubernetes (Part 3)

Run and Scale a Distributed Crossword Puzzle App with CI/CD on Kubernetes (Part 3) - Printable Version

+- Sick Gaming (https://www.sickgaming.net)
+-- Forum: Computers (https://www.sickgaming.net/forum-86.html)
+--- Forum: Linux, FreeBSD, and Unix types (https://www.sickgaming.net/forum-88.html)
+--- Thread: Run and Scale a Distributed Crossword Puzzle App with CI/CD on Kubernetes (Part 3) (/thread-86927.html)

Run and Scale a Distributed Crossword Puzzle App with CI/CD on Kubernetes (Part 3) - xSicKxBot - 10-18-2018

Run and Scale a Distributed Crossword Puzzle App with CI/CD on Kubernetes (Part 3)

<div style="margin: 5px 5% 10px 5%;"><img src="http://www.sickgaming.net/blog/wp-content/uploads/2018/10/run-and-scale-a-distributed-crossword-puzzle-app-with-ci-cd-on-kubernetes-part-3.png" width="256" height="256" title="" alt="" /></div><div>In Part 2 of our series, we deployed a Jenkins pod into our Kubernetes cluster, and used Jenkins to set up a CI/CD pipeline that automated building and deploying our containerized Hello-Kenzan application in Kubernetes. 
In Part 3, we are going to set aside the Hello-Kenzan application and get to the main event: running our Kr8sswordz Puzzle application. We will showcase the built-in UI functionality to scale backend service pods up and down using the Kubernetes API, and also simulate a load test. We will also touch on showing caching in etcd and persistence in MongoDB. 
Before we start the install, it’s helpful to take a look at the pods we’ll run as part of the Kr8sswordz Puzzle app:
<ul>
<li>
kr8sswordz – A React container with our Node.js frontend UI. 
</li>
<li>
puzzle – The primary backend service that handles submitting and getting answers to the crossword puzzle via persistence in MongoDB and caching in ectd. 
</li>
<li>
mongo – A MongoDB container for persisting crossword answers. 
</li>
<li>
etcd – An etcd cluster for caching crossword answers (this is separate from the etcd cluster used by the K8s Control Plane). 
</li>
<li>
monitor-scale – A backend service that handles functionality for scaling the puzzle service up and down. This service also interacts with the UI by broadcasting websockets messages. 
</li>
</ul>
We will go into the main service endpoints and architecture in more detail after running the application. For now, let’s get going! 
Read all the articles in the series:
<div> 
<div>
<table>
<tbody>
<tr>
<td><img alt="3di6imeKV7hPtEx3cDcZM3dUG6aW4CWOPmdGOIFA" height="20" src="http://www.sickgaming.net/blog/wp-content/uploads/2018/10/run-and-scale-a-distributed-crossword-puzzle-app-with-ci-cd-on-kubernetes-part-3.png" width="20" /></td>
<td>
This tutorial only runs locally in Minikube and will not work on the cloud. You’ll need a computer running an up-to-date version of Linux or macOS. Optimally, it should have 16 GB of RAM. Minimally, it should have 8 GB of RAM. For best performance, reboot your computer and keep the number of running apps to a minimum. 
</td>
</tr>
</tbody>
</table>
</div>
</div>
<h2>Running the Kr8sswordz Puzzle App</h2>
First make sure you’ve run through the steps in Part 1 and Part 2, in which we set up our image repository and Jenkins pods—you will need these to proceed with Part 3 (to do so quickly, you can run the part1 and part2 <a href="https://docs.google.com/document/d/1oB6E036-UHJomLr3Na6Le63_w7fkaMaTucrWMVrYHeA/edit#heading=h.vuj44njqkr03">automated scripts detailed below</a>). If you previously stopped Minikube, you’ll need to start it up again. Enter the following terminal command, and wait for the cluster to start:
<pre>
minikube start</pre>
You can check the cluster status and view all the pods that are running. 
<pre>
kubectl cluster-info kubectl get pods --all-namespaces</pre>
<pre>
Make sure the registry and jenkins pods are up and running. 
</pre>
<div>
<table>
<tbody>
<tr>
<td><img alt="goO2T3gv5m_ehnoPadbP8Eww76Kgh9oUzsSGD7v9" height="20" src="http://www.sickgaming.net/blog/wp-content/uploads/2018/10/run-and-scale-a-distributed-crossword-puzzle-app-with-ci-cd-on-kubernetes-part-3.png" width="20" /></td>
<td>
So far we have been creating deployments directly using K8s manifests, and have not yet used <a href="https://helm.sh/">Helm</a>. Helm is a package manager that deploys a Chart (or package) onto a K8s cluster with all the resources and dependencies needed for the application. Underneath, the chart generates Kubernetes deployment manifests for the application using templates that replace environment configuration values. Charts are stored in a repository and versioned with releases so that cluster state can be maintained. Helm is very powerful because it allows you to templatize, version, reuse, and share the deployments you create for Kubernetes. See <a href="https://hub.kubeapps.com/">https://hub.kubeapps.com/</a> for a look at some of the open source charts available. We will be using Helm to install an etcd operator directly onto our cluster using a pre-built chart. 
</td>
</tr>
</tbody>
</table>
</div>
1. Initialize Helm. This will install Tiller (Helm’s server) into our Kubernetes cluster. 
<pre>
helm init --wait --debug; kubectl rollout status deploy/tiller-deploy -n kube-system</pre>
2. We will deploy an etcd operator onto the cluster using a Helm Chart. 
<pre>
helm install stable/etcd-operator --version 0.8.0 --name etcd-operator --debug --wait</pre>
<div>
<div>
<table>
<tbody>
<tr>
<td><img alt="goO2T3gv5m_ehnoPadbP8Eww76Kgh9oUzsSGD7v9" height="20" src="http://www.sickgaming.net/blog/wp-content/uploads/2018/10/run-and-scale-a-distributed-crossword-puzzle-app-with-ci-cd-on-kubernetes-part-3.png" width="20" /></td>
<td>
An operator is a custom controller for managing complex or stateful applications. As a separate watcher, it monitors the state of the application, and acts to align the application with a given specification as events occur. In the case of etcd, as nodes terminate, the operator will bring up replacement nodes using snapshot data. 
</td>
</tr>
</tbody>
</table>
</div>
</div>
3. Deploy the etcd cluster and K8s Services for accessing the cluster.
kubectl create -f manifests/etcd-cluster.yaml kubectl create -f manifests/etcd-service.yaml
You can see these new pods by entering kubectl get pods in a separate terminal window. The cluster runs as three pod instances for redundancy.
4. The crossword application is a multi-tier application whose services depend on each other. We will create three K8s Services so that the applications can communicate with one another.
kubectl apply -f manifests/all-services.yaml
5. Now we’re going to walk through an initial build of the monitor-scale application.
<pre>
docker build -t 127.0.0.1:30400/monitor-scale:`git rev-parse 
 --short HEAD` -f applications/monitor-scale/Dockerfile 
 applications/monitor-scale</pre>
<div>
<div>
<table>
<tbody>
<tr>
<td><img alt="goO2T3gv5m_ehnoPadbP8Eww76Kgh9oUzsSGD7v9" height="20" src="http://www.sickgaming.net/blog/wp-content/uploads/2018/10/run-and-scale-a-distributed-crossword-puzzle-app-with-ci-cd-on-kubernetes-part-3.png" width="20" /></td>
<td>
To simulate a real life scenario, we are leveraging the github commit id to tag all our service images, as shown in this command (git rev-parse –short HEAD). 
</td>
</tr>
</tbody>
</table>
</div>
</div>
6. Once again we’ll need to set up the Socat Registry proxy container to push the monitor-scale image to our registry, so let’s build it. Feel free to skip this step in case the socat-registry image already exists from Part 2 (to check, run docker images).
<pre>
docker build -t socat-registry -f applications/socat/Dockerfile 
 applications/socat</pre>
7. Run the proxy container from the newly created image.
<pre>
docker stop socat-registry; docker rm socat-registry; docker run 
 -d -e "REG_IP=`minikube ip`" -e "REG_PORT=30400" --name 
 socat-registry -p 30400:5000 socat-registry</pre>
<div>
<div>
<table>
<tbody>
<tr>
<td><img alt="goO2T3gv5m_ehnoPadbP8Eww76Kgh9oUzsSGD7v9" height="20" src="http://www.sickgaming.net/blog/wp-content/uploads/2018/10/run-and-scale-a-distributed-crossword-puzzle-app-with-ci-cd-on-kubernetes-part-3.png" width="20" /></td>
<td>
This step will fail if local port 30400 is currently in use by another process. You can check if there’s any process currently using this port by running the command lsof -i :30400
</td>
</tr>
</tbody>
</table>
</div>
</div>
8. Push the monitor-scale image to the registry.
<pre>
docker push 127.0.0.1:30400/monitor-scale:`git rev-parse --short HEAD`</pre>
9. The proxy’s work is done, so go ahead and stop it.
<pre>
docker stop socat-registry</pre>
10. Open the registry UI and verify that the monitor-scale image is in our local registry. 
<pre>
minikube service registry-ui</pre>
<div><img alt="_I4gSkKcakXTMxLSD_qfzVLlTlfLiabRf3fOZzrm" height="315" src="http://www.sickgaming.net/blog/wp-content/uploads/2018/10/run-and-scale-a-distributed-crossword-puzzle-app-with-ci-cd-on-kubernetes-part-3-1.png" width="624" /></div>
11. Monitor-scale has the functionality to let us scale our puzzle app up and down through the Kr8sswordz UI, therefore we’ll need to do some <a href="https://kubernetes.io/docs/reference/access-authn-authz/rbac/">RBAC</a> work in order to provide monitor-scale with the proper rights. 
<pre>
kubectl apply -f manifests/monitor-scale-serviceaccount.yaml</pre>
<div>
<div>
<table>
<tbody>
<tr>
<td><img alt="ANM4b9RSNsAb4CFeAbJNUYr6IlIzulAIb0sEvwVJ" height="20" src="http://www.sickgaming.net/blog/wp-content/uploads/2018/10/run-and-scale-a-distributed-crossword-puzzle-app-with-ci-cd-on-kubernetes-part-3.png" width="20" /></td>
<td>
In the manifests/monitor-scale-serviceaccount.yaml you’ll find the specs for the following K8s Objects. 
Role: The custom “puzzle-scaler” role allows “Update” and “Get” actions to be taken over the Deployments and Deployments/scale kinds of resources, specifically to the resource named “puzzle”. This is not a ClusterRole kind of object, which means it will only work on a specific namespace (in our case “default”) as opposed to being cluster-wide.
ServiceAccount: A “monitor-scale” ServiceAccount is assigned to the monitor-scale deployment.
RoleBinding: A “monitor-scale-puzzle-scaler” RoleBinding binds together the aforementioned objects. 
</td>
</tr>
</tbody>
</table>
</div>
</div>
12. Create the monitor-scale deployment and the Ingress defining the hostname by which this service will be accessible to the other services. 
<pre>
sed 's#127.0.0.1:30400/monitor-scale:$BUILD_TAG#127.0.0.1:30400/
 monitor-scale:'`git rev-parse --short HEAD`'#' 
 applications/monitor-scale/k8s/deployment.yaml | kubectl apply -f -</pre>
<div>
<div>
<table>
<tbody>
<tr>
<td><img alt="goO2T3gv5m_ehnoPadbP8Eww76Kgh9oUzsSGD7v9" height="20" src="http://www.sickgaming.net/blog/wp-content/uploads/2018/10/run-and-scale-a-distributed-crossword-puzzle-app-with-ci-cd-on-kubernetes-part-3.png" width="20" /></td>
<td>
The sed command is replacing the $BUILD_TAG substring from the manifest file with the actual build tag value used in the previous docker build command. We’ll see later how Jenkins plugin can do this automatically.
</td>
</tr>
</tbody>
</table>
</div>
</div>
13. Wait for the monitor-scale deployment to finish. 
<pre>
kubectl rollout status deployment/monitor-scale</pre>
14. View pods to see the monitor-scale pod running.
<pre>
kubectl get pods</pre>
15. View services to see the monitor-scale service. 
<pre>
kubectl get services</pre>
16. View ingress rules to see the monitor-scale ingress rule. 
<pre>
kubectl get ingress</pre>
17. View deployments to see the monitor-scale deployment. 
<pre>
kubectl get deployments</pre>
18. We will run a script to bootstrap the puzzle and mongo services, creating Docker images and storing them in the local registry. The puzzle.sh script runs through the same build, proxy, push, and deploy steps we just ran through manually for both services.
<pre>
scripts/puzzle.sh</pre>
19. Check to see if the puzzle and mongo services have been deployed. 
<pre>
kubectl rollout status deployment/puzzle
kubectl rollout status deployment/mongo</pre>
20. Bootstrap the kr8sswordz frontend web application. This script follows the same build proxy, push, and deploy steps that the other services followed. 
<pre>
scripts/kr8sswordz-pages.sh</pre>
21. Check to see if the frontend has been deployed.
<pre>
kubectl rollout status deployment/kr8sswordz</pre>
22. Check to see that all the pods are running.
<pre>
kubectl get pods</pre>
23. Start the web application in your default browser. 
<pre>
minikube service kr8sswordz
</pre>
<h2>Giving the Kr8sswordz Puzzle a Spin</h2>
Now that it’s up and running, let’s give the Kr8sswordz puzzle a try. We’ll also spin up several backend service instances and hammer it with a load test to see how Kubernetes automatically balances the load. 
1. Try filling out some of the answers to the puzzle. You’ll see that any wrong answers are automatically shown in red as letters are filled in. 
2. Click Submit. When you click Submit, your current answers for the puzzle are stored in MongoDB. 
<img alt="EfPr45Sz_JuXZDzxNUyRsfXnKCis5iwRZLGi3cSo" height="381" src="http://www.sickgaming.net/blog/wp-content/uploads/2018/10/run-and-scale-a-distributed-crossword-puzzle-app-with-ci-cd-on-kubernetes-part-3-2.png" width="624" /> 3. Try filling out the puzzle a bit more, then click Reload once. This will perform a GET which retrieves the last submitted puzzle answers in MongoDB.
Did you notice the green arrow on the right as you clicked Reload? The arrow indicates that the application is fetching the data from MongoDB. The GET also caches those same answers in etcd with a 30 sec TTL (time to live). If you immediately press Reload again, it will retrieve answers from etcd until the TTL expires, at which point answers are again retrieved from MongoDB and re-cached. Give it a try, and watch the arrows.
4. Scale the number of instances of the Kr8sswordz puzzle service up to 16 by dragging the upper slider all the way to the right, then click Scale. Notice the number of puzzle services increase.
<div>
<div>
<table>
<tbody>
<tr>
<td><img alt="goO2T3gv5m_ehnoPadbP8Eww76Kgh9oUzsSGD7v9" height="20" src="http://www.sickgaming.net/blog/wp-content/uploads/2018/10/run-and-scale-a-distributed-crossword-puzzle-app-with-ci-cd-on-kubernetes-part-3.png" width="20" /></td>
<td>
If you did not allocate 8 GB of memory to Minikube, we suggest not exceeding 6 scaled instances using the slider.
</td>
</tr>
</tbody>
</table>
</div>
</div>
<img alt="r5ShVJ4omRX9znIrPLlpBCwatys2yjjdHA2h2Dlq" height="467" src="https://lh3.googleusercontent.com/r5ShVJ4omRX9znIrPLlpBCwatys2yjjdHA2h2DlqTMmwEx2UOqKpWpNjrWAijGPUN48BpEC2ar5eD3Rk2M5yEPXPR0SnKr6Qn66unwO1oylxwZFReyN9QZHlzZ6cqvpufdMigc8C" width="423" />
In a terminal, run kubectl get pods to see the new replicas.
5. Now run a load test. Drag the lower slider to the right to 250 requests, and click Load Test. Notice how it very quickly hits several of the puzzle services (the ones that flash white) to manage the numerous requests. Kubernetes is automatically balancing the load across all available pod instances. Thanks, Kubernetes! <img alt="P4S5i1UdQg6LHo71fTFLfHiZa1IpGwmXDhg7nhZJ" height="382" src="https://lh5.googleusercontent.com/P4S5i1UdQg6LHo71fTFLfHiZa1IpGwmXDhg7nhZJzDZo2zPuVZX5vT8dsmI_G8w-tM8xEXR3A1cknH9f-CGcv9tMP-DAN-EJ3tu0t9kKiEIutBaHbmpNzw8O3MbWI23udVZKSCSm" width="564" />
6. Drag the middle slider back down to 1 and click Scale. In a terminal, run kubectl get pods to see the puzzle services terminating. 
<img alt="g5SHkVKTJQjiRvaG-huPf8aJmLWS19QGlmqgn2OI" height="119" src="https://lh3.googleusercontent.com/g5SHkVKTJQjiRvaG-huPf8aJmLWS19QGlmqgn2OI5ynzShJfN4oefZpQtBedQMXO6ZLrCMBAiTCRJSl5ISVJjDdeeLVqa3ODWCJzmitXO8k53GsK4_-WAhIY8lPGDPDKFLfuaPK4" width="481" />
7. Now let’s try deleting the puzzle pod to see Kubernetes restart a pod using its ability to automatically heal downed pods
a. In a terminal enter kubectl get pods to see all pods. Copy the puzzle pod name (similar to the one shown in the picture above). 
<pre>
 b. Enter the following command to delete the remaining puzzle pod. 
kubectl delete pod [puzzle podname]</pre>
c. Enter kubectl get pods to see the old pod terminating and the new pod starting. You should see the new puzzle pod appear in the Kr8sswordz Puzzle app. 
<h2>What’s Happening on the Backend</h2>
We’ve seen a bit of Kubernetes magic, showing how pods can be scaled for load, how Kubernetes automatically handles load balancing of requests, as well as how Pods are self-healed when they go down. Let’s take a closer look at what’s happening on the backend of the Kr8sswordz Puzzle app to make this functionality apparent. 
<img alt="Kr8sswordz.png" height="388" src="https://lh5.googleusercontent.com/aqi_8RG3l634TPbFDz9z2eGalvh3VemseUKKgYQl9dKSUClpcpLMTAjr8DtVi6Mst17vmzJhfFrisqjSWWAAe-PByS0MUghUOcutOJXN20qBHzEcnnS2ehTYtnWmTKRzXp7FFn0_" width="624" />
1. pod instance of the puzzle service. The puzzle service uses a LoopBack data source to store answers in MongoDB. When the Reload button is pressed, answers are retrieved with a GET request in MongoDB, and the etcd client is used to cache answers with a 30 second TTL. 
2. The monitor-scale pod handles scaling and load test functionality for the app. When the Scale button is pressed, the monitor-scale pod uses the Kubectl API to scale the number of puzzle pods up and down in Kubernetes. 
3. When the Load Test button is pressed, the monitor-scale pod handles the loadtest by sending several GET requests to the service pods based on the count sent from the front end. The puzzle service sends Hits to monitor-scale whenever it receives a request. Monitor-scale then uses websockets to broadcast to the UI to have pod instances light up green. 
4. When a puzzle pod instance goes up or down, the puzzle pod sends this information to the monitor-scale pod. The up and down states are configured as lifecycle hooks in the puzzle pod k8s deployment, which curls the same endpoint on monitor-scale (see kubernetes-ci-cd/applications/crossword/k8s/deployment.yml to view the hooks). Monitor-scale persists the list of available puzzle pods in etcd with set, delete, and get pod requests.
<div>
<div>
<table>
<tbody>
<tr>
<td><img alt="goO2T3gv5m_ehnoPadbP8Eww76Kgh9oUzsSGD7v9" height="20" src="http://www.sickgaming.net/blog/wp-content/uploads/2018/10/run-and-scale-a-distributed-crossword-puzzle-app-with-ci-cd-on-kubernetes-part-3.png" width="20" /></td>
<td>
We do not recommend stopping Minikube (minikube stop) before moving on to do the tutorial in <a href="https://www.linux.com/blog/learn/chapter/intro-to-kubernetes/2017/6/set-cicd-distributed-crossword-puzzle-app-kubernetes-part-4">Part 4</a>. Upon restart, it may create some issues with the etcd cluster.
</td>
</tr>
</tbody>
</table>
</div>
</div>
<div>
<table>
<tbody>
<tr>
<td>
<h3>Automated Scripts</h3>
If you need to walk through the steps we did again (or do so quickly), we’ve provided npm scripts that will automate running the same commands in a terminal. 
1. To use the automated scripts, you’ll need to install NodeJS and npm. 
On Linux, follow the <a href="https://nodejs.org/en/download/package-manager/">NodeJS installation steps</a> for your distribution. To quickly install NodeJS and npm on Ubuntu 16.04 or higher, use the following terminal commands. 
<pre>
 a. curl -sL https://deb.nodesource.com/setup_7.x | sudo -E bash -
 b. sudo apt-get install -y nodejs</pre>
On macOS, <a href="https://nodejs.org/en/">download the NodeJS installer</a>, and then double-click the .pkg file to install NodeJS and npm.
2. Change directories to the cloned repository and install the interactive tutorial script:
<pre>
 a. cd ~/kubernetes-ci-cd
 b. npm install</pre>
3. Start the script
<pre>
npm run part1 (or part2, part3, part4 of the blog series)</pre>
4. Press Enter to proceed running each command. 
</td>
</tr>
</tbody>
</table>
</div>
<h2>Up Next </h2>
Now that we’ve run our Kr8sswordz Puzzle app, the next step is to set up CI/CD for our app. Similar to what we did for the Hello-Kenzan app, Part 4 will cover creating a Jenkins pipeline for the Kr8sswordz Puzzle app so that it builds at the touch of a button. We will also modify a bit of code to enhance the application and enable our Submit button to show white hits on the puzzle service instances in the UI. 
Curious to learn more about Kubernetes? <a href="https://www.edx.org/course/introduction-kubernetes-linuxfoundationx-lfs158x">Enroll in Introduction to Kubernetes</a>, a FREE training course from The Linux Foundation, hosted on edX.org.
This article was revised and updated by David Zuluaga, a front end developer at Kenzan. He was born and raised in Colombia, where he studied his BE in Systems Engineering. After moving to the United States, he studied received his master’s degree in computer science at Maharishi University of Management. David has been working at Kenzan for four years, dynamically moving throughout a wide range of areas of technology, from front-end and back-end development to platform and cloud computing. David’s also helped design and deliver training sessions on Microservices for multiple client teams.
</div>