Find Better Cookbooks on the Supermarket

The Chef Supermarket makes it easy to be successful with Chef through sharing the successes of a community of practitioners. Use the public Supermarket to collaborate with the community or install your own private Supermarket and collaborate with your co-workers.

The Supermarket is a clearinghouse of information about tools used within the Chef ecosystem.  The Supermarket is best known for the cookbooks that are shared there. Did you know that you can also find knife plugins, test kitchen drivers,  ohai plugins, and compliance profiles in the Tools & Plugins directory?

Anyone interested in publishing a cookbook to the Supermarket is welcome to do so. This openness is part of our core community beliefs. Cookbooks are always shared with the best intentions and likely met the requirements of the publisher. However, that does not necessarily mean that a given cookbook will work for your requirements. Additionally, many cookbooks may be published to help automate the same component of your infrastructure. For example, if you search the Supermarket for an nginx cookbook, thirty two results are returned.

We need a way to find and identify the best cookbooks. Quality Metrics are a set of objective attributes of a cookbook that may help find the cookbook that best meets your needs. Beware, however, that there really is no better measure of a cookbooks quality than the testing you do on that cookbook to validate that it meets your needs.

The community has identified a set of quality metrics that are objective metrics that can be measured automatically without human intervention. A number of these metrics are now available on the “Quality” tab of a cookbook. The quality metrics available today are listed below. At the end of this post you will see asummary of the pass / fail rate for the library of cookbooks currently available on the public Supermarket.

Published to the Supermarket

At first glance, this may seem like a pretty silly metric.  How can you view the qualities of a cookbook that is not published to the Supermarket?!  However, this metric will fail if a cookbook is deprecated or up for adoption.  A cookbook that has been deprecated should not be used.  A cookbook that is up for adoption no longer has an active maintainer.

Cookbook has Collaborators

Each cookbook on the Supermarket is published by an owner.  That owner may designate one or more other Supermarket members as a collaborator.  When the author of the cookbook wins the lottery and retires early, you want to be sure there others with permission to publish updates to the cookbook.

Cookbook has an Open Source license

The license dictates the terms under which a cookbook may be used.  It is a good idea to only use cookbooks from the Supermarket that include an open source license.

Cookbook supports at least one platform

A cookbook should declare the platform or platforms that are supported.  It is best to use cookbooks only on the operating systems that are listed as supported.

Cookbook passed Foodcritic evaluation

Foodcritic is a static analysis tool for cookbooks.  Foodcritic checks a cookbook for adherence to the agreed upon community practices.  Code that follows common standards is easier to read, write, and maintain.

Cookbook includes CONTRIBUTING guidelines

This metric checks the source repository of a cookbook for a file named CONTRIBUTING.md.  Proper contributing documentation is an important aid to ease the barrier to cookbook contribution.

Cookbook includes TESTING guidelines

This metric checks the source repository of a cookbook for a file named TESTING.md.  Proper testing documentation is an important aid to ease the barrier to cookbook contribution and improve cookbook quality.

Cookbook does not include binary files

Cookbook consumers should be able to examine the contents of the cookbook’s source code in order to evaluate its actions and security implications.  Community cookbooks that need to install binaries should obtain them from signed artifact repositories.

Cookbook repository has tags that match versions

To facilitate troubleshooting, consumers of a cookbook should be able to trace the version they are using to the state of the cookbook’s source at the time the cookbook build artifact was produced.

Future of Quality Metrics

Some additional quality metrics have been identified by the community in the cookbook quality metrics repository on GitHub.  You may read about these metrics and suggest additional metrics there.  You may also help implement additional quality metrics by submitting a pull request to the Supermarket on GitHub.

Evaluation of most of the quality metrics currently happens at a late stage of the development process, namely when a cookbook is published to the Supermarket.  Ideally, all quality metrics should be measured and accounted for before publishing a cookbook.  Foodcritic is used very early in the development of a cookbook.  This makes Foodcritic an ideal location to implement as many quality metrics as possible.  Overtime, some of the quality metrics on the Supermarket will be integrated into Foodcritic or Cookstyle, another tool for static analysis.

Summary of Quality Metric Data

A quick summary of the metrics is shown below.  The chef-client cookbook is an example of a cookbook that passes each of these while the broken_qualities_cookbook is an example that fails each of the quality metrics.

As a community we can work together to raise the pass rate of all of the cookbooks on the Supermarket!