Sensu 2.0 and Habitat are both modern distributed systems, one for monitoring, the other for application packaging and deployment. Over the last few months, we at Indellient, a Chef partner, have been working on packaging Sensu in Habitat to take advantage of some of its features for dynamic service discovery and configuration updates, critical capabilities in a cloud world where monitoring agents come and go.
Sensu is a modern monitoring system which flips the architecture of a monitoring system on its head. In contrast to traditional monitoring systems, which are centralized and poll agents on a schedule, Sensu puts most of the logic at the edge (agent) and has the agents asynchronously push events to a centralized queue (backend). This allows Sensu to scale much better than older, datacenter-based monitoring systems, and makes it more resilient to failure of either the agents or the backend.
Sensu 2.0 is a complete rewrite in Go. This rewrite reduces the number of needed components in Sensu, as well as making it simpler to install. Sensu’s new architecture also has the side effect of being easier to package and deploy in a system like Habitat. The two required components for a Sensu deployment are:
You can find out more about Sensu on episode 122 of the Food Fight Show.
Since the Sensu components are both compiled Go binaries, the Habitat packages are minimal and straightforward. Both the backend and the agent are configured using a YAML file which is applied at runtime via a command line argument.
Of course, Habitat has some great features that lend itself well for installing Sensu: for example, dynamic service discovery to deal with ever-changing IP addresses for backends. We used Habitat’s service bindings feature to allow us to dynamically configure Sensu at runtime. Since Sensu agents require a Sensu backend to report data, we built in a service binding that will configure the Sensu agent to report to each backend in the service group.
backend-url:
{{~#each bind.backend.members as |member|}}
- "ws://{{member.sys.ip}}:{{member.cfg.port}}"
{{~/each}}
This service binding allows us to deploy one or more Sensu backends and, through Habitat, update this backend-url configuration at runtime. If the backends change, this fact is propagated through Habitat’s supervisor ring and are automatically communicated to the agents, who can reload their configurations through Habitat’s lifecycle hooks system — with no user intervention.
Both plans are available now from the unstable channel on Habitat Builder. When Sensu 2.0 reaches general availability, we will update the plan and release to the stable channel.