This is a follow up to the post Automated Rollback the Ultimate Application Release Insurance Policy in which we discuss key implementation considerations related to application rollback and rollforward. Chef Habitat is an open-source automation solution that enables you to define, package and deliver your applications to any environment without rewriting or refactoring. The software creates platform-independent build artifacts that are deployed and managed in the same way.
Application definition is the process of creating a codified operational runbook. It formalizes the process of describing in code everything an application needs to be built, run, and managed. As part of the definition process you have the opportunity to specify what runtime services you want to use. When you load a runtime service with Chef Habitat you select the appropriate behavior for your application. This includes the desired upgrade behavior like if your application is compatible with “rollback” or if only rollforward behavior is acceptable.
In this example we tell the Habitat Supervisor to load the “sample” application, to receive updates from the “prod” release channel, upgrade immediately, and finally that it can “rollback” with a demote or roll-forward with a “promote”. The Chef Habitat Supervisor is a light-weight agent that runs on/in a server, virtual machine, or container and manages the application according to the instructions defined in the Habitat Plan. Tasks are defined via pre-set scripts called lifecycle hooks that are included as part of the application plan.
With Chef Habitat you define everything you need to deliver your application as code, then you package your application into a Habitat Artifact(.hart). This artifact includes everything your application needs to deliver & run like the application codebase, any dependencies, and instructions on how to run the application.
The created Habitat Artifact is immutable, it includes your application, locked dependencies, the definition of health status for your application along with anything else your application may need.
When you load an application the Habitat Supervisor downloads the Habitat Artifact for your application from Builder, verifies the package then unpacks to an isolated place on the file system.
For the example above my application would be stored on a path similar to this:
/hab/pkgs/eric/sample/0.2.0/20200309020327`
0.2.0 is the Applications version and 20200309020327 is the applications build number.
Finally after the application is unpacked the Habitat Supervisor runs your application as you defined earlier. This is typically in your run hook which is a simple bash or powershell script for how your application should be run.
The running application will be found under the svc or services path, which would be something something similar to this:
/hab/svc/sample
After the Habitat Supervisor starts your application, the supervisor will report the health of the application to the Chef Automate dashboard. Here’s a visual showing the Habitat Supervisor downloading the Habitat Artifact from Builder, then reporting health status to Chef Automate.
Upgrading an Application with Chef Habitat
Now that we know how the Habitat Supervisor is used to run your application, let’s review how the Habitat Supervisor upgrades your application.
By default the Supervisor will check-in with Habitat Builder every 60 seconds looking for a new Habitat Artifact. When a new Habitat Artifact is detected the Habitat Supervisor downloads, verifies and unpacks the content to:
hab/pkg/applicationname/version/buildnumber
The Habitat Supervisor will also download and verify any application dependencies you’ve defined.
Once the Habitat Supervisor has put everything in place for your upgrade, it will stop the old version of your application, and start the service for the new version of your application.
Use the available hooks to perform any actions your application needs to clean up after itself, prepare the new version and run the new version. Commonly the init hook is used to prepare for your application like cleaning out old versions, setting environment variables or other tasks, then the run hook is responsible for starting your application.
Finally after your new application version has been started the health-check hook is used to validate the health of your application. By Default the Habitat Supervisor will run the health-check hook every 30 seconds. The results can be sent to Chef Automate to visualize a fleet of applications.
Next we will apply what we’ve learned to rollback & rollforward scenarios with Chef Habitat.
Automated Application Rollback with Chef Habitat
To implement rollback with Chef Habitat we start with our supervisor running Version A of our application. We promote a Version B of our application to the “prod” channel. The Habitat Supervisor will see the new version. It will then immediately begin the upgrade process. Once the upgrade is complete the Habitat Supervisor will start our health-check hook, which will detect the deployment failure.
hab pkg promote eric/sample/0.2.0/20200309020327 prod
To Rollback we will demote version B, which removes it from the “prod” channel. The Habitat Supervisor will see this demotion, it will then immediately revert back to Version A of our application. This is the exact same process we described in the upgrading section. Once the Rollback is complete the Habitat Supervisor will start our health-check which will validate service has been restored.
hab pkg demote eric/sample/0.2.0/20200309020327 prod
Automated Application Rollforward with Chef Habitat
In a rollforward scenario we use Chef Habitat’s upgrade capabilities slightly differently, with the same goal of quickly restoring services in the event of a failed deployment. Our application is either incompatible with rollback or we decide we want to continue forward with the new features found in our release.
We start with our supervisor running Version A of our application. We promote a Version B of our application to the “prod” channel. The Habitat Supervisor will see the new version, it will then immediately begin the upgrade process. After the upgrade the Habitat Supervisor will restart the health-check where we can see our release has failed.
We need to diagnose what caused Version B of our application to fail and correct the issue. Once we find the commit that caused the problem we can make the appropriate changes to resolve the issue. This may require us to add an additional commit or git revert the problem commit.
Whatever we decide we create a new pull request with our fixes. Our pull request should run our normal pipeline which will create a new build and run through our standard testing.
We now have a new tested version of our Application version C. We will promote version C to the “prod” channel. The Habitat Supervisor will see this promotion, it will then immediately rollforward to version C of our application. After the rollforward action is complete the Habitat Supervisor will restart our health-check which will validate service has been restored.
Conclusion
The promote or demote actions can be performed via the command line or via the habitat builder user interface bldr.habitat.sh. The cli is also a logical integration point with pipeline tools like Azure DevOps, Jenkins, Gitlab, etc.
Over time you may perform hundreds of releases with Chef Habitat. This can start to fill up your disk with unnecessary old Habitat Artifacts. You can configure the Habitat Supervisor to automatically cleanup old packages with the --keep-latest-packagesflag. The Habitat Supervisor will cleanup old packages only keeping the number of packages specified.
For example we can configure the Habitat Supervisor to keep the most recent three packages with --keep-latest-packages=3.
Technically how you “rollback” will depend on your application architecture, your constraints, and your processes. In some scenarios Rollback will be the most desirable and in others Rollforward will be the most appropriate.
What we do know is that failures will happen and when they do Chef Habitat has you covered with an insurance policy for any architecture.
Learn more about Chef Habitat here. Or if you would like to schedule a demo contact us.