On Tuesday, we announced the release of the InSpec infrastructure testing framework as part of the release of Chef Compliance. Today we would like to dive into how and why we decided to create InSpec, from its journey as a Serverspec extension to its current state as a standalone project.

Serverspec for compliance testing

Our startup VulcanoSec, acquired by Chef earlier this year, was founded to solve a single problem: how can we bring the best of DevOps practices to the world of compliance? We were already deeply rooted in the world of DevOps and experienced the great shift it had brought to infrastructure management. We wanted to bring the same benefits and workflows to the domains of security and compliance.

Existing solutions for checking compliance can roughly be grouped into two areas: static analysis, and dynamic script-based tools. The first often comes with a highly portable syntax, where you describe your expected values and results (e.g. in XML). This is great for portability, but usually offers less flexibility, because these documents have highly-fixed schemas. On the other hand, script-driven solutions approach the problem from the opposite direction, by letting users freely define tests, often with minimal structure. But it comes at the cost of higher complexity and maintainability. Ultimately, we really wanted something that would provide a lightweight structure for checking compliance out of the box, but allow for easy customization.

We were inspired by the Serverspec project which brought acceptance testing of infrastructure, which are often at the root of security assertions. Serverspec had taken Ruby and RSpec to create a descriptive language for infrastructure tests. ServerSpec offers the right mix of static expectations (e.g. expect resource X to be in state Y) and programmatic flexibility.

With Serverspec at the core, we just needed to create the pieces necessary for compliance. We wanted to describe our tests as a collection of controls, which are grouped into compliance profiles. These had to be easily shareable and extendable and offer a mechanism to make simple adjustments from one project to the next. We wanted our tests to be more descriptive, adding in criticality (so that you know what to focus on first, when you get truckload of test failures) as well as descriptive metadata to help explain the meaning of the test.

The solution we created in the early days of VulcanoSec had a great stack for DevOps: An expressive Ruby-driven core, with Serverspec making up the structure, and compliance added all around it.

Learnings from the field, or the evolution into InSpec

With the core framework ready, we put it into the hands of compliance and security auditors. The initial contact was often one of curiosity about the world of DevOps, followed by some apprehension about the tight coupling with Ruby, and finally great excitement about the potential of how it could improve their daily work and maintenance of compliance.

Smart resources

In the hands of our test users, we came to learn a great deal about the way to approach their domain. We realized that we had to change our way of thinking and focus more on what our auditors actually wanted and what they were capable of. And they definitely didn’t want to write huge checks like this around their actual tests:

if os[:family] == 'redhat' or os[:family] == 'fedora'

if file('/etc/ssh/sshdconfig').exists? and
 file('/etc/ssh/sshdconfig').readable?

describe 'check ssh config values' do

...

end

else
 describe 'check ssh config values' do
 skip "Cannot read sshd configuration file."
 end
 end

else
 describe 'check ssh config values' do
 skip "It is not supported on your OS #{os[:family]}"
 end
 end

Instead, they wanted the work to be done by the resource, so that they could focus on just writing the test, with a minimum of ceremony:

describe sshd_config do
 # Do not fail the test if the resource is not readable or supported!
 # Instead, skip the test automatically if it is not supported here.
 ...
 end

The example above would register all tests we create and make sure that they are skipped with the right message if the resource isn’t supported or can’t be examined. For example, this lets you run compliance checks as a regular user, with checks requiring superuser access being skipped.

Custom resources

This gets us to the second lesson learned: the need to allow users to easily create custom resources. As I described in a previous article, you can create resources in Serverspec already. We wanted to take this concept to the next level.

Some auditors are very good at coding and want to create their own resources, while others only want to use them. In order to support the former, we needed to create a clear interface for creating custom resources and extending existing ones. It should be easy to talk to the backend, define what is supported, and offer values to actual tests. All of this had to be done in a way so that regular users who only want to consume resources can do so easily.

After many discussions and several rounds of development, we created the InSpec plugin system, with a clear backend and interface definition. Here is an example of how the SSH server configuration resource is created:

class SshConf < Inspec.resource(1)
 name 'ssh_config'
 ...
 end
 ```

This will register a new resource as `ssh_config`. You can now use it in your tests:

This will register a new resource as ssh_config. You can now use it in your tests:

describe ssh_config do
 ...
 end

Because it’s an InSpec resource, you get access to the backend and all other resources:

def read_file
 f = inspec.file('/etc/ssh/ssh_config')
 return skip_resource "Can't find SSH config" if !f.file?
 return skip_resource "Can't read SSH config" if f.content.empty? && f.size > 0
 @content = f.content
 end

All of this is only the tip of the iceberg. Resource inheritance, resource sharing and re-use, as well as defining supported operating systems are all included in this model. By making the interface for it very clear and explicit, we provide a structure for creating and sharing resources.

Backend execution

This part is more about InSpec internals but it ended up being really important to us.

When you call resources in Serverspec, your call is actually piped through 2 projects: Serverspec and Specinfra. This division separates the testing framework from the actual transport and execution layer. Within Specinfra, you can find a huge collection of calls, which are implemented for different operating systems that are supported.

We liked the clear-cut modular system but wanted to move a few things around. We were looking to have a minimal execution and transport layer that didn’t know anything about resources. This layer is designed to only provide a small set of functionality for running commands, provide basic OS info, and interact with files on the system. This project is now called train, the “transport interface”. As the name suggests, it will also take care of transport configuration, i.e. running your commands locally, on SSH, WinRM, or even Docker.

With the introduction of train, InSpec was able to focus on managing resources and actually executing tests. Also, the differentiation of operating systems was now done in the resource itself, i.e. inside InSpec. This approach drastically changed the interaction between resource execution and transport execution, and didn’t fit the previous model anymore.

CLI

The last step is for convenience and helps users who aren’t familiar with Ruby, RSpec, or Rake. Serverspec has a great way of configuring itself via RSpec right inside your code and uses either rspec or a Rake task to execute itself. In our early days, we created a common spec_helper.rb which took this approach as well. The user – or in our case the compliance testing backend – would call the test and put its configuration for SSH/WinRM into environment variables. These would be transferred to RSpec/Serverspec configuration by the helper script. In essence, it allowed you to run Serverspec tests without an additional command line utility.

When we created InSpec, we wanted to offer users a way to run their tests, without knowing anything about Rake or spec_helper.rb. We wanted to have a comfortable command line utility, which could configure everything. Moreover, users should be able to run tests, which are written in a plain file, without having to create a complex folder structure around it to mimic regular RSpec unit tests. This is geared towards quick testing and debugging.

With InSpec, running a test is now as simple as calling:

inspec exec test.rb

You can easily run this test on a remote machine using SSH/WinRM or a Docker container:

inspec exec test.rb --target ssh://user@hostname
 inspec exec test.rb --target winrm://Administrator@hostname
 inspec exec test.rb --target docker://container_id

We are also bringing back RSpec-based configuration and will add Rake tasks for users who like to drive their tests through a Ruby-driven workflow.

Summary

InSpec has emerged at the end of an exciting journey of server and compliance testing, from its initial version, which used the Serverspec project, to the current state as a standalone framework for infrastructure testing. We have added features around smart and extensible resources, a common execution framework for local/remote/container testing, and useful utilities.

We are excited to welcome contributors to the project. Join us at https://github.com/chef/inspec.