Note: This blog post may be out of date. Check out a more recent article “Get Started with Habitat on Kubernetes” from our partners at Kinvolk.
Habitat simplifies creating, managing, and running containers. Habitat allows you to package applications in a compact, atomic, and easily auditable manner that more fully delivers on the promise of containers. Habitat has a number of additional benefits, but one of the places it particularly shines is striking the right balance between manageability, portability, and consistency in managing a fleet of microservice applications. For most users, that means running in tandem with a scheduler and a container management platform. Today we’re going to explore how to run Habitat applications in Kubernetes.
For some additional background, the pattern for how habitat applications fit into a containerized infrastructure is explored in Habitat 201: Habitat in the ecosystem from ChefConf 2016. This blog post presumes you have a grasp of fundamental Habitat concepts and an understanding of Kubernetes basics.
For this example, we’ll be working with the Redis application used in the starter habitat demo and running it via Kubernetes (aka k8s) in Google Container Engine.
Setting up k8s on gcloud
Google Container Engine is a hosted Kubernetes environment. We’re going to use it to forego having to set up Kubernetes ourselves. We can use it to quickly deploy our containerized habitat applications. This tutorial presumes you have a Google Cloud account enabled, a working Google Cloud SDK installed, and kubectl installed and configured.
Start by opening the Google Cloud Console and select the drop down to “Create a New Project”. We’re going to call our project “my-k8s-toy”. For this project, you must enable both “Container Engine API” and “Google Compute API”. Once that setup is complete, we can proceed by using the SDK via command line.
Set the Compute Engine zone, project name, and create a new k8s cluster.
$ gcloud config set compute/zone us-central1-a $ export PROJECT_ID="my-k8s-toy" $ gcloud container clusters create redis $ gcloud container clusters get-credentials redis
Build a basic Habitat app
Habitat core maintainers provide an open-source repository of core habitat application plans anyone can use. We’re going to use those as a jumpstart for our purposes since one of the included plans is for the basic Redis app we want. Since we’re building our own custom habitat package, we’ll need our own origin key. In this example, our origin key is named my-k8s-toy. Clone the core-plans repo and enter the hab studio using your key with a source path for the Redis plan.
$ git clone https://github.com/habitat-sh/core-plans.git $ hab studio -k $PROJECT_ID -s core-plans/redis enter
Habitat automatically sets up your clean room build environment before you proceed. The habitat core plans we cloned have their pkg_origin value set to ‘core’. We need to edit our plan.sh and change that to the name of the origin key we’ll be using (my-k8s-toy) to sign this package.
[1][default:/src:0]# vi plan.sh
For the purposes of this walkthrough, we’re going to disable protected mode (client auth) on Redis just to keep things simple. Edit the default.toml file and set protected-mode="no". Once we’ve made those changes, we can build our Redis habitat app.
[2][default:/src:0]# vi default.toml [3][default:/src:0]# build
Redis is a surprisingly simple key/value store. The plan.sh for this app is pretty easy to understand and it’s a fast build with few dependencies. Habitat creates a habitat artifact file (.hart) that contains the redis binary, all of its dependencies (from glibc up), and a number of config parameters we can tell the habitat supervisor to alter when this package runs. You can examine the .hart file created in the results directory.
Creating a docker image to run in k8s
We want to run our Redis habitat app in Kubernetes, so we need to package it up in a Docker container. This command builds a minimal container and places into it both the .hart artifact we just built and the hab-sup binary.
[4][default:/src:0]# export PROJECT_ID="my-k8s-toy" [5][default:/src:0]# hab pkg export docker $PROJECT_ID/redis
For gcloud to push our container into the correct docker registry, we need to tag the container with the name of the google container registry (grc.io). We need to use the ‘docker tag’ command to do that, so we must first install docker into our clean room build environment.
[6][default:/src:0]# hab pkg install core/docker [7][default:/src:0]# hab pkg exec core/docker docker tag $PROJECT_ID/redis gcr.io/$PROJECT_ID/redis
We’re now done with the habitat studio and will be using k8s for the rest of our walkthrough.
[8][default:/src:0]# exit
Using our habitized docker image in k8s
We’ve created our docker image locally. In order to get it into k8s for gcloud, we need to push it to the Google Container Registry (gcr.io). We can do that with the gcloud SDK:
$ gcloud docker -- push gcr.io/$PROJECT_ID/redis:latest
With a successful push, you now have your habitized Redis container available to run via gcloud k8s. Start by making sure that our app runs as expected. First, we spin up an instance. Then we pull up the instance logs to verify our Redis app works.
$ kubectl run redis --image=gcr.io/$PROJECT_ID/redis:latest --port=6379 $ kubectl get pods $ kubectl logs PODNAME
Replace PODNAME with the pod that you started from the ‘get pods’ command. The logs should show you the typical Redis start screen. If so, everything works and we’re ready to start seeing the Habitat runtime in action.
Following the redis habitat demo that shows us how to alter configuration settings at runtime, we can do the same thing when running in k8s. You’ll see from your logs that we get an error about the configuration of our “tcp-backlog”. We can run another pod and tell the habitat supervisor to alter our process configuration on the fly.
$ kubectl run redis-tcpbacklog --env HAB_REDIS='tcp-backlog = 128' --image=gcr.io/$PROJECT_ID/redis:latest --port=6379 $ kubectl get pods $ kubectl logs PODNAME
This time, your logs should show that the previous error no longer appears. We’ve dynamically altered the behavior of our container at runtime. We can see that Habitat works. Now let’s delete those instances and look at a more useful example of what Habitat can do by using this same container to deploy Redis in a leader-follower topology.
$ kubectl delete deployment redis redis-tcpbacklog
Creating a highly available Redis cluster
You should never run Redis in standalone mode in production. With habitat, you can tell Redis to run in a leader-follower topology. K8s also gives us constructs to help scale and load balance across our instances. At present, there’s a bit of a shuffle we need to do in order for the hab-sup to work with k8s’ pod scaling methodology.
First, we have to start three separate pods in order to achieve a leader-follower quorum. Once we have that quorum initiated, we can then scale the instances within one of those pods to join the ring. At that point, the additional separate pods can be destroyed. That’s not optimal and implementing a better deployment pattern is a good candidate for future state work to improve the habitat+k8s experience.
Here’s how to make the two work together today. Start three pods. In order, we are going to name them redis1, redis2, and redis.
$ kubectl run redis1 --image=gcr.io/$PROJECT_ID/redis:latest --port=6379 -- --topology leader
This command starts redis and tells the habitat supervisor to configure redis to run in a leader-follower topology. If you inspect the running process, you’ll see redis waiting for a quorum of instances before it will run. View the pod log and you should see output like this:
hab-sup(MN): Starting my-k8s-toy/redis hab-sup(TP): Child process will run as user=hab, group=hab hab-sup(GS): Supervisor 10.0.1.15: 8415dcf6-3282-42a4-b5db-9c72c93b50a4 hab-sup(GS): Census redis.default: 8bd35f65-76fb-4dc7-bbe2-6bd1ff174979 hab-sup(GS): Starting inbound gossip listener hab-sup(GS): Starting outbound gossip distributor hab-sup(GS): Starting gossip failure detector hab-sup(CN): Starting census health adjuster hab-sup(SC): Updated redis.config hab-sup(TP): Restarting because the service config was updated via the census hab-sup(TL): 1 of 3 census entries; waiting for minimum quorum
This tells us where to find our first supervisor in the ring. We need to connect at least two more instances to it to get a quorum. We will tell subsequent instances where to find this supervisor with the “--peer” flag.
$ kubectl run redis2 --image=gcr.io/$PROJECT_ID/redis:latest --port=6379 -- --topology leader --peer 10.0.1.15 $ kubectl run redis --image=gcr.io/$PROJECT_ID/redis:latest --port=6379 -- --topology leader --peer 10.0.1.15
With this you should now have a working three node Redis cluster. Inspect your pod logs to verify this is true.
Creating a scalable load balanced Redis service for production
The deployment above has three pods running in separate deployments. The recommended way to run this in k8s is to have multiple pods within the same deployment and setup a “service” to load balance between them. Now that we have the quorum established from the process above, we need to get into our recommended architecture.
Scale the “redis” deployment to 4 replicas. As each replica comes up, it will reach out to the peer we specified earlier in order to join the ring.
$ kubectl scale deployment redis --replicas=4
If you inspect the individual pods, you’ll see the Redis cluster now has six total instances, distributed like so:
$ kubectl get pods NAME READY STATUS RESTARTS AGE redis-3578384135-1un51 1/1 Running 0 1m redis-3578384135-d9xzt 1/1 Running 0 1m redis-3578384135-s72wh 1/1 Running 0 2m redis-3578384135-ut9g0 1/1 Running 0 1m redis1-3792489326-0fa2s 1/1 Running 0 4m redis2-2558447467-62ybw 1/1 Running 0 3m $ kubectl get deployments NAME DESIRED CURRENT UP-TO-DATE AVAILABLE AGE redis 4 4 4 4 2m redis1 1 1 1 1 4m redis2 1 1 1 1 3m
At this point, you can delete redis2 if you desire. The awkward part (at present) is that we’re relying on redis1 to help any new pods coming up in the redis deployment discover where to find the habitat gossip ring.
Now that we have a healthy number of instances in our Redis cluster, we need to allow external network connectivity in order to use it. We’re going to clean up our deployments. Then we’re going to place a load balancer in front of the ‘redis’ deployment since it has most of our Redis instances behind it.
$ kubectl delete deployment redis2 $ kubectl expose deployment redis --type='LoadBalancer' $ kubectl get services NAME CLUSTER-IP EXTERNAL-IP PORT(S) AGE kubernetes 10.3.2.1 <none> 443/TCP 1h redis 10.3.2.75 8.34.210.140 6379/TCP 1m
We now have a working load balanced 5 node scalable Redis cluster ready for use by connecting to the external IP on 6379/tcp. We can also adjust the number of instances in our cluster by using the command “kubectl scale deployment redis --replicas=<number you want>”, which is both very easy and quite powerful.
If for any reason you don’t have a functional cluster at this point in the tutorial, please don’t hesitate to reach out to us via the Habitat Community Slack in the #kubernetes channel. We want to help!
Cleanup
What? You don’t just want all those instances you’re paying for to run indefinitely? Okay, let’s go ahead and clean up the demo.
First, delete the service and deployment.
$ kubectl delete service,deployment redis redis1
Next, remove the container clusters so that gcloud won’t spin them back up.
$ gcloud container clusters delete redis
Finally, remove the artifacts from the repository.
$ gsutil rm -r gs://artifacts.$PROJECT_ID.appspot.com/
Special thanks to George Miranda for co-authoring this post.