This is a guest post by Michael Hüttermann. Michael is an expert in Continuous Delivery, DevOps and SCM/ALM. More information about him at http://huettermann.net, or follow him on Twitter: @huettermann.
This blog post introduces executable Software Configuration Management as a DevOps enabler, and shows what an implementation can look like with Artifactory as the caretaker to manage Chef Cookbooks.
The Path from executable Software Configuration Management to DevOps
Software configuration management (SCM) helps to improve the delivery process by tracking and controlling changes in the software. These processes are commonly based on the insight, that code and binaries should be handled differently, see Illustration 1. A central term for SCM is configuration item. Usually, the term configuration item refers to produced or consumed artifacts. SCM consists of four major functions:
- Configuration identification—Select and identify all configuration items and establish baselines on them so you can then control, audit, and report changes.
- Change control—Control changes to configuration items.
- Configuration audit—Ensure correctness, completeness, and consistency of baselines by examining the baselines, configuration items, and related processes.
- Status accounting—Report on the status of all configuration items throughout their lifecycle.
Although there are still use cases to follow different ways, many projects meanwhile make great success with documenting the configuration items in an executable way, and putting those descriptions into version control systems (VCS). This approach was coined to be Infrastructure as Code. One possible way to do so is via Chef, implementing Cookbooks, based on their Ruby-based domain-specific language (DSL). The distribution management tool, e.g. Artifactory, stores the packaged deliverables, i.e. the configuration items, which are often just called artifacts. Examples are JAR and EAR files, derived from Java source code, Docker images, derived from Dockerfiles, and tar.gz files derived from Cookbooks, the latter makes Cookbook deliverables to be Infrastructure as Binaries.
Using executable documentation and having a unified handling of configuration items is not only an excellent implementation of SCM, particularly its four building blocks configuration identification, change control, configuration audit and status accounting. It is also a prerequisite for implementing DevOps, see [1], that comprises development and operations using same processes and tools (e.g. Artifactory, and Chef, see [2][3][4][5][6]), to achieve shared goals, that often includes improving the cycle time. Broaden the scope from a sole technical to a functional perspective, leads to Agile Application Lifecycle Management (Agile ALM), see [7], that also has strong interfaces to SCM and DevOps.
Best-of-breed tools to implement SCM, and accelerate DevOps
Introducing both tools in depth, Chef and Artifactory, is out of scope here for this blog post, please consult the respective documentation for further information, see [8] and [9]. What is important to understand here is, since its version 5.1, see [10], Artifactory offers first class support for managing Chef Cookbooks. Artifactory helps to implement the four functions of SCM, as introduced above, and to underpin your DevOps initiative by supporting an unified process and providing single tools, across functions, including development and operations.
Above all, Artifactory’s Chef support includes support for:
- consistent and reliable access to Cookbooks on Chef Supermarket, see [11], via Artifactory Remote Repositories, see Illustration 2. Access can be channeled via Artifactory. This allows to implement Quality Gates.
- Artifactory Local Repositories to share internal Cookbooks. Also here Quality Gates can be injected.
- aggregation of Artifactory Remote and Local Repositories to Virtual Repositories, to have all different Cookbooks from all Repositories at your fingertips. From an end user consumer/producer perspective, a single URI can be used, in order to access more than one repository at a given time.
- compatibility with Knife command line tool to list, show and install Cookbooks, and compatibility with the Berkshelf command line to resolve Cookbook dependencies.
Since the universal binary repository manager Artifactory serves as a single point of entry, cross-cutting functionalities are backed by Artifactory, also for handling Chef Cookbooks, such as:
- Promoting and staging of artifacts.
- Assigning access permissions.
- Allowing checksum based storage, for efficient storage.
- Providing Searches, REST API, Wizards, context-sensitive help, and much more.
Note: Chef uses the concept of a Chef repository, to represent storing their own data objects on a workstation. This is different from the use of “repository” in Artifactory. Chef provides an official “supermarket” for cookbook packages, so Chef repositories in Artifactory are actually Chef supermarkets in Chef terminology.
After setting the stage, and describing the support, let’s now continue with an example.
A roundtrip, in a nutshell
First we configure the knife.rb, if needed. In our example, we customize the supermarket site, that is the URI of the Artifactory virtual repository, and the path where cookbooks are located, locally, on the workstation of the user.
Note: Please consult the respective documentation for further information how to install and use the tools.
<user-home-dir>/.chef/knife.rb knife[:supermarket_site] = 'http://localhost:8071/artifactory/api/chef/chef' knife[:cookbook_path] = '/Users/michaelh/work/data/chef'
In Artifactory, we can jump start by using the Chef Wizard to create a bunch of repos in one few steps, including Artifactory Local (for our own Chef Cookbooks) and Remote (pointing to Chef Supermarket and serving as a cache for it) and a Virtual to aggregate those under a single URI. In other words, after these few steps, we’ve created a Local Chef Supermarket, a Remote Chef Supermarket, and a Virtual Chef Supermarket, aligned with common good practices.
Given a local Chef Cookbook, named helloworld, we can now easily share it, i.e. uploading it to Artifactory for distributing it to the team, for further usage.
venus:chef michaelh$ knife artifactory share helloworld Generating metadata for helloworld from /var/folders/0j/r49k3d8n4ndczgj8wbd8x9jr270j68/T/chef-helloworld-build20170428-2022-rhut49/helloworld/metadata.rb Making tarball helloworld.tgz Upload complete
A second use case is to download an existing Chef Cookbook e.g. to install it afterwards, locally, or as part of a delivery pipeline. In our example we want to use the poise-python Cookbook, which is available on the community Chef Supermarket. To download its version 1.6.0, we trigger the following command:
venus:chef michaelh$ knife artifactory download poise-python 1.6.0 Downloading poise-python from Supermarket at version 1.6.0 to /Users/michaelh/work/data/chef/poise-python-1.6.0.tar.gz Cookbook saved: /Users/michaelh/work/data/chef/poise-python-1.6.0.tar.gz
After performing the command, the tar.gz is downloaded, and could be installed. Let’s now have a brief look into the situation in Artifactory. Opening the Artifact Repository Browser, you see the repositories. Illustration 3 shows the virtual repository, named “chef”, which lists all Cookbooks from Local and Remote Repositories. What you see here and what you are able to do is again aligned with the permissions granted to you in Artifactory, to access and operate on those targets, that was an example of a cross-cutting Artifactory functionality, see above.
The browser shows, that the home-made Cookbook was successfully uploaded to Artifactory. The screenshot also underlines, that also the downloaded Cookbook was processed and cached by Artifactory. In Illustration 4, you see a search performed on all artifacts, including Chef Cookbooks, that is another example of a cross-cutting Artifactory functionality. In our example, we perform a Property search, and look for artifacts having a defined maintainer.
This roundtrip delivered examples of the Artifactory support for Chef, aligned with the concept to use Artifactory as a private Supermarket. For more information, see [12].
Summary
In this blog post, we discussed the importance of executable Software Configuration Mangement, its path to DevOps, and how the leading tools Chef and Artifactory can be used in conjunction, to set up an effective and efficient tool chain to streamline your delivery process. I now wish you a lot of fun with working with Chef Cookbooks, and Artifactory.
References
[1] http://huettermann.net/devops
[2] https://www.chef.io/blog/devops-for-developers-wchef-guest-blog-series
[3] https://www.chef.io/blog/devops-for-developers-wchef-part-ii-guest-blog-series/
[4] https://www.chef.io/blog/devops-for-developers-wchef-part-iii-guest-blog-series/
[5] https://www.chef.io/blog/devops-for-developers-wchef-part-iv-guest-blog-series/
[6] https://www.chef.io/blog/devops-for-developers-wchef-part-v-guest-blog-series/
[7] http://huettermann.net/alm
[9] https://www.jfrog.com/artifactory
[10] https://www.jfrog.com/confluence/display/RTF/Release+Notes#ReleaseNotes-Artifactory5.1
[11] https://supermarket.chef.io
[12] https://www.jfrog.com/confluence/display/RTF/Chef+Cookbook+Repositories