Meet the Chef Analytics Platform

Update:  Learn more about Chef Analytics in our upcoming webinar on September 17, 2014 at 10am Pacific.

The Chef analytics platform is a premium feature of Chef. It provides real-time visibility into what is happening on your Chef server and is the latest in a growing suite of features that will provide a comprehensive view of your infrastructure. Chef analytics runs on separate hardware from the Chef server and consists of several components. Today, we are releasing the first version of the analytics platform, which includes the ability to log Chef actions. Over time, the analytics platform will provide additional capabilities for analyzing metrics and event data from Chef servers and clients.

What will the Chef analytics platform do for me?

Analytics give you visibility into your Chef server, a way to publish notifications when there is a change, and a way to verify compliance.

• Visibility. See what’s changing on your server in real time, see who made the change and see when they did it.
• Notifications. Integrate analytics with your favorite external systems, such as HipChat. Notify the relevant people or automated tools about changes to the Chef server so that they can react in real time.
• Compliance: Once you have visibility into your system changes, you can verify compliance against internal controls.

Action logs

Existing Enterprise Chef features include a management console, which provides an action-oriented view of your infrastructure, and reporting, which tracks what happened during chef-client runs. However, until now there has been no easy way to see what policies have been updated on a Chef server, or to monitor changes. For example, if a bug in a cookbook stopped node convergence, administrators couldn’t be sure what cookbooks had recently changed, or been uploaded, or by whom. Chef action logs solve this by providing a real-time feed of the operations that have changed the state of the server. Actions are policy and administrative changes made to the Chef server. The Chef server gathers a lot of data—each node object, the node run history for all nodes, the history of every cookbook and cookbook version, how policy settings, such as roles, environments, and data bags, are applied and to what they are applied, individual user data, and so on. Actions are changes to any of these objects. Actions can occur as the result of user interaction in the management console, as the result of knife commands, or by running the Chef client. No matter what the source, all actions are tracked by the analytics platform and recorded in the action log. The result is real-time tracking with an intuitive display. Action logs help you to answer the following questions:

• Which object changed?
• What type of change was made?
• When was this change made?
• Who changed it?

Examples are:

• The date and time on which a specific user uploaded a cookbook from their local workstation to the Chef server.
• The changes that were made to the system immediately before that cookbook stopped working.
• When a node was created for the first time, or when it was decommissioned and deleted from the Chef server.

This information can be used to quickly identify where in the overall system something changed, which in turn helps identify what went wrong, and then helps show your organization what the resolution should be. When errors occur, you’ll also be able to see what happened just before one (or more) chef-client runs started failing. Because Chef tracks all of this in real-time, your organization will now be able to react to events as they happen, and to more quickly resolve issues that arise.

Notify me!

Until now, in order to respond in real time to changes on your nodes, you needed to create custom client-side handlers embedded in the Chef client. With action logs, this logic moves to the server, augmenting and replacing these for you, giving you better, centralized control over those handlers. Today you might deploy a handler to Sensu, logstash to HipChat, or PagerDuty. These will be replaced by notifications, starting with HipChat notification in this release of the Chef analytics platform. And we’ll be iterating rapidly to bring you the notification features that you need.

Pipeline architecture

We use a publish-subscribe messaging platform for components to publish messages about interesting events that are happening within each public API. The pub/sub platform we use provides some standard consumers of the information, including a database archiving component and web visualization. The data is searchable and stored long term for after-the-fact investigation and audit purposes. Here’s how it works:

What can you do from the Chef action log UI?

Currently, the UI lets you:

• View actions on a Chef object. See changes to a node or role to track down bugs.
• View actions for an organization. See all the actions in a production organization to measure the level of change at different times of the day and week.
• Navigate to the Chef management console. There’s one-click to access an object in the management console from the Chef action log page.
• Fan-out messages for distribution. Send notifications from Chef to HipChat when cookbooks or roles have been uploaded.
• Distinguish between knife, chef-client and other clients. View the different types of client applications accessing the Chef server to spot unusual or unexpected behavior.
• Correlate calls from a single client invocation. Group all the items that happened during a single client (e.g. chef-client, knife) invocation.
• Browse actions for after-the-fact investigation. A persistent log can be saved on demand for auditing purposes or for analyzing information after an event has occurred. All messages are immutable. Non-modifiable messages are delivered reliably to the actions database. They can be used as a trusted source of data about changes to your infrastructure.

Share what you think

We’d love to hear from you! Chef action logs were demoed at #ChefConf2014, and we’ve had great feedback and suggestions since then. Share your ideas with your sales representative. Analytics is an exciting new direction for us, and your use cases are important. We plan to continually release new features–let us know what you would like to see.

Installation instructions

To allow you to scale Chef analytics independently from the Chef server in production, the analytics server should be deployed in a standalone configuration, on a different host from the Chef server. Follow the instructions on the Chef documentation site to install Chef action logs using the analytics package. See Install analytics prerequisites and Install analytics. Also, check out the documentation.