Writing Ruby test code to verify Unix/Linux systems for auditing purposes
Many organizations must adhere to PCI-DSS requirements, or similar standards. However, those standards are often not specific, so we cannot rely on them to give implementation details. The CIS Benchmarks provide technical recommendations with specific commands and scripts to audit systems, and remediate any discrepencies. While PCI-DSS is mentioned in this post it is not assumed that this is adequate for audit testing PCI-DSS compliance.
This post serves as an instructional exercise for the reader to adapt to their own implementation of auditing systems using Chef’s audit mode, whether for PCI-DSS or other compliance standards. For example purposes, I will use Ubuntu 14.04 (LTS) as my target platform, and the CIS Benchmark for Ubuntu 14.04.
PCI-DSS and other security compliance frameworks present a number of requirements. They often don’t have specifics, because they’re generally applicable across many different operating systems and services or products. For example, PCI-DSS v3.1 section 2.2.2 states:
Enable only necessary services, protocols, daemons, etc., as required for the function of the system.
It doesn’t tell us which services, protocols, daemons, etc. are required, or the ones that are not required. What we need to do as developers and operators is interpret and apply that to our systems. For example, after interviewing everyone appropriate in the company, and we determine that we should not be running xinetd, we might log into systems to assess their state:
root@localhost# initctl show-config xinetd
initctl: Unknown job: xinetd
We know xinetd is therefore not enabled, since the system doesn’t know about it. We would repeat this for every other service we’ve determined as unnecessary. We might even implement a shell script that we can run to get a report of all these services.
This is the kind of thing that the CIS Benchmarks do. The CIS Benchmarks are lists of recommendations for securing a specific operating system/distribution like Ubuntu Linux, or software package like Apache HTTPD. Each recommendation has an “audit” section that includes a command or a script that the user should run to determine if the system complies with the recommendation or not. In fact, the output above comes directly from recommendation 5.1.8 in the CIS Benchmark for Ubuntu 14.04. In this post, I will discuss several of these recommendations, the audit step, and how that can be written using Chef’s audit mode tests. These come from the audit-cis cookbook.
Before we begin, let’s recap Chef audit mode. It is a feature of chef-client that allows users to write control tests in recipes that can be verified during the Chef run. It is implemented in Serverspec, which in turn is based on RSpec. Serverspec provides a number of resource types that it can test, such as services, packages, commands, or users. It does this with matchers for the resource’s state, such as running, installed, etc.
Checking Services
It is quite common to test the state of services running on the system. These are basic test matchers in Serverspec, where we can write tests that check whether services are running or enabled. To expand on the example above about xinetd, the CIS Benchmark states:
5.1.8 Ensure xinetd is not enabled
The audit section says “Ensure no start conditions listed for xinetd“, with the command from above. In Chef audit mode, we write a control like this:
[ruby]
control_group ‘5 OS Services’ do
context ‘5.1 Ensure Legacy Services are not enabled’ do
control ‘5.1.8 Ensure xinetd is not enabled’ do
it ‘is not running the xinetd service’ do
expect(service(‘xinetd’)).to_not be_running
expect(service(‘xinetd’)).to_not be_enabled
expect(service(‘openbsd-inetd’)).to_not be_running
expect(service(‘openbsd-inetd’)).to_not be_enabled
end
end
end
end
[/ruby]
Let’s break that down.
First, audit mode requires grouping everything in a control_group block. When implementing the CIS benchmark in audit-cis, we’ve created a separate group for each recommendation section.
[ruby]
control_group ‘5 OS Services’ do
…
[/ruby]
Next, section “5” has subsection “5.1.” This is grouped under a context because it doesn’t really make sense to nest controls for audits. Contexts are a feature of RSpec to logically group tests with descriptions.
[ruby]
context ‘5.1 Ensure Legacy Services are not enabled’ do
…
[/ruby]
Then we have the actual control. This is the specific recommendation from the CIS benchmark.
[ruby]
control ‘5.1.8 Ensure xinetd is not enabled’ do
…
[/ruby]
The control is tested with an it block. Inside this block we write our expectations for the system state. The audit section of the CIS Benchmark for this recommendation simply has a query for the service using an initctl command; using audit mode means we write a Serverspec test.
[ruby]
it ‘is not running the xinetd service’ do
expect(service(‘xinetd’)).to_not be_running
expect(service(‘xinetd’)).to_not be_enabled
expect(service(‘openbsd-inetd’)).to_not be_running
expect(service(‘openbsd-inetd’)).to_not be_enabled
end
[/ruby]
The intention is that this reads like English. “It is not running the xinetd service.” “We expect service xinetd to not be running.” In this particular example, we cover both xinetd, and openbsd-inetd, because both implementations are packaged for Ubuntu 14.04, and we want to be sure neither is enabled or running. This is RSpec’s expectation syntax, which is what we expect (ahem) to use when writing tests in audit mode. Under the covers, Serverspec knows which OS it is running on and will use the appropriate commands to determine what the state of the services are.
Packages
Another basic test in audit mode built into Serverspec is testing that packages are installed, or not. For example, with our xinetd control we also expect the package not to be installed. This is also in section 5.1.8:
[ruby]
it ‘does not have xinetd package installed’ do
expect(package(‘xinetd’)).to_not be_installed
expect(package(‘openbsd-inetd’)).to_not be_installed
end
[/ruby]
Like the service resource type, Serverspec knows how to check the state of the packages based on which OS* it is running on.
(*) As long as the OS is supported. See a list in the Specinfra project. Note that while Windows isn’t present in the Specinfra platforms, it is supported for audit mode.
File Content
Commonly with audit compliance checks, the contents of configuration files need to be verified. In this section, we’ll look at using a let block to cache a value across multiple tests. In the example here we only use it once, but in the audit-cis cookbook, it’s used multiple times.
[ruby]
control_group ‘5 OS Services’ do
let(:inetd_exists) { File.exists?(‘/etc/inetd.conf’) }
let(:inetd_conf) { file(‘/etc/inetd.conf’) }
end
[/ruby]
In Chef audit mode, use let blocks inside the following blocks:
control_groupcontext
What we’ve done is set up two reusable local variables that we’re going to use in future tests in this control_group. These are available across all the controls in the group. Depending on the kind of tests being written, another block may be a more appropriate location for using let blocks. The first one,
[ruby]
let(:inetd-_exists) { File.exists?(‘/etc/inetd.conf’) }
[/ruby]
Uses Ruby’s File class method, #exists? which will return true or false depending on whether the target file exists. On Ubuntu 14.04, this file exists if the inetd packages are installed, so one might wonder why we’re still checking its existence, when we also have a test that checks if the package is installed. The reason of course is that someone might have created the file in this location outside package management control, or perhaps it had local changes and the package manager didn’t remove the file during uninstallation. It’s hard to know for sure, so we’re going to check its existence. Later on in our tests we can use inetd_exists like a local variable.
Second, we have,
[ruby]
let(:inetd-_conf) { file(‘/etc/inetd.conf’) }
[/ruby]
Here, the file method we’re using is from Serverspec. It loads an object that has state information about the target file, including the type, permissions, ownership, and its contents. In our tests we can use inetd_conf to refer to this instead of typing file('/etc/inetd.conf') everywhere.
Let’s take a look at one of the tests that works with both these.
[ruby]
control ‘5.1.6 Ensure telnet server is not enabled’ do
it ‘does not have telnet services in /etc/inetd.conf or /etc/inetd.conf does not exist’ do
if inetd_exists
expect(inetd_conf.content).to_not match(/^telnet/)
else
true
end
end
end
[/ruby]
This control is from CIS Benchmark recommendation 5.1.6, which requires that the telnet server is not enabled. This is a common compliance practice, where we want to ensure that login daemons that do not encrypt traffic are not running. Here we’re using inetd_exists in an if statement:
[ruby]
if inetd_exists
expect(inetd_conf.content).to_not match(/^telnet/)
else
true
end
[/ruby]
In other words, “if /etc/inetd.conf exists, check that it’s content doesn’t have a line beginning with telnet.” The audit section in the CIS Benchmark is:
root@localhost# grep ^telnet /etc/inetd.conf
It indicates that no results should be returned. Of course, if the file is not present, grep will fail because it won’t find the file.
In the case of the audit mode test, we simply return true, because we know that if the file isn’t there, telnet won’t be enabled.
Commands
Sometimes, we need to run a command and check its output or exit status. We can do that using the command resource in Serverspec. We’ll move on to another section of the CIS Benchmark, since #5 doesn’t have any commands that it processes. We’ll look at the very first recommendation, section 1.1 “Install Updates, Patches, and Additional Security Software.”
The entire section is only one control with two tests:
[ruby]
control_group ‘1 Patching and Software Updates’ do
control ‘1.1 Install Updates, Patches, and Additional Security Software’ do
let(:apt_get_upgrade) { command(‘apt-get -u upgrade –assume-no’) }
it ‘returns 1 when there are packages to upgrade’ do
expect(apt_get_upgrade.exit_status).to eql(1)
end
it ‘does not have packages to upgrade’ do
expect(apt_get_upgrade.stdout).to_not match(/^The following packages will be upgraded:/)
end
end
end
[/ruby]
Here, we’re using a let block that will capture the command state, which will have the exit status, and the output printed to STDOUT. The CIS recommendation’s audit section indicates using the command sudo apt-get --just-print upgrade, however I wanted to verify there was correctly a non-zero return code in addition to the output indicating there are packages to upgrade.
[ruby]
it ‘returns 1 when there are packages to upgrade’ do
expect(apt_get_upgrade.exit_status).to eql(1)
end
[/ruby]
Remember that let blocks are like local variables. They’re instances of the object returned by the block, so in this case we have a Serverspec command object, and we can set the exit_status method to it. The exit status in this case is going to be a 1 if there are packages to upgrade, because --assume-no aborts the apt-get command. The matcher, eql is built into RSpec.
In the second test,
[ruby]
it ‘does not have packages to upgrade’ do
expect(apt_get_upgrade.stdout).to_not match(/^The following packages will be upgraded:/)
end
[/ruby]
We’re going to set the stdout method to the apt_get_upgrade object, which will return all of the content from STDOUT when running the command. Depending on the system, this can be a few, or dozens of packages. However knowing how apt-get works, we know we can search for the specified regular expression. We use the match matcher, with the regular expression.
Users and Groups
The CIS Benchmark devotes an entire section (#13) to reviewing user and group settings. In this section, we’ll show how we can use the Etc module from the Ruby standard library to retrieve user and group information that we can use in our tests. The first thing we do in the control_group for section #13 is set up some local objects with let blocks:
[ruby]
control_group ’13 Review User and Group Settings’ do
let(:root_path) { command(‘su – root -c "echo $PATH"’) }
let(:passwd) { file(‘/etc/passwd’) }
let(:group) { file(‘/etc/group’) }
let(:shadow) { file(‘/etc/shadow’) }
let(:gshadow) { file(‘/etc/gshadow’) }
let(:passwd_uids) { Etc::Passwd.map {|u| u.uid} }
let(:passwd_names) { Etc::Passwd.map {|u| u.name} }
let(:passwd_gids) { Etc::Group.map {|g| g.gid} }
let(:group_names) { Etc::Group.map {|g| g.name} }
let(:shadow_gid) { Etc::Group.select {|g| g.gid if g.name == ‘shadow’} }
end
[/ruby]
The first five should be familiar from the earlier discussion about let blocks – we’re setting up a command and four files. We’re not looking at these files for their content. Instead, we’re going to assess the permissions as part of the recommendations. Let’s talk about the last five let blocks which use the Etc module. From the Ruby standard library documentation:
The Etc module provides access to information typically stored in files in the /etc directory on Unix systems.
The information accessible consists of the information found in the /etc/passwd and /etc/group files, plus information about the system’s temporary directory (/tmp) and configuration directory (/etc).
Etc::Passwd and Etc::Group return an array of struct objects for each entry in the file. These entries have a method for each of their properties: name, uid, gid, shell, and so on. In the let blocks, since they are arrays, we’ll use the #map method to collect the field we want for each of these cached objects.
The first,
[ruby]
let(:passwd_uids) { Etc::Passwd.map {|u| u.uid} }
[/ruby]
Collects the UID of all users on the system. Then we collect the names in a separate object with,
[ruby]
let(:passwd_names) { Etc::Passwd.map {|u| u.name} }
[/ruby]
These are used in different ways within the tests in this section. We do similar collections of all the GIDs for all groups, and all the group names. Finally, we also get the GID for the shadow group as it has a specific test, too.
[ruby]
let(:shadow_gid) { Etc::Group.select {|g| g.gid if g.name == ‘shadow’} }
[/ruby]
Now let’s take a look at some the tests that consume these objects. The first is 13.11, “Check Groups in /etc/passwd.” This recommendation is to check whether configuration drift over time has caused groups to be defined in /etc/passwd, but not in /etc/group. The recommendation’s audit says to create a script, and run it.
#!/bin/bash
for i in $(cut -s -d: -f4 /etc/passwd | sort -u) ; do
grep -q -P "^.*?:[^:]*:$i:" /etc/group
if [ $? -ne 0 ]; then
echo "Group $i is referenced in /etc/passwd but not in /etc/group"
fi
done
Certainly we could use the command resource type from Serverspec, but this is prone to copy/paste errors, and is really hard to read within an audit mode recipe. With the let block for :passwd_gids before, we can iterate over these GIDs with passwd_gids.each, and check the GIDs with the Etc module, using the getgrgid method. That method will raise an error if the specified group does not exist in /etc/group, making this very simple to test.
[ruby]
control ‘13.11 Check Groups in /etc/passwd’ do
it ‘has a group for all users’ do
passwd_gids.each do |group|
expect{ Etc.getgrgid(group) }.to_not raise_error
end
end
end
[/ruby]
The next recommenation is 13.16, “Check for duplicate user names.” This is pretty straightforward – we shouldn’t have a system that has the same username multiple times. The audit for this looks like,
#!/bin/bash
cat /etc/passwd | /usr/bin/cut -f1 -d":" | /usr/bin/sort -n | /usr/bin/uniq -c |
while read x ; do
[ -z "${x}" ] && break set - $x
if [ $1 -gt 1 ]; then
uids=`/usr/bin/awk -F: '($1 == n) { print $3 }' n=$2 /etc/passwd | xargs`
echo "Duplicate User Name ($2): ${uids}" fi
done
That’s fairly straightforward if you’re familiar with Bash and Linux/Unix commands. However, it’s pretty gnarly to read, even if you’re familiar. Here’s the test in audit mode,
[ruby]
control ‘13.16 Check for Duplicate User Names’ do
it ‘does not have duplicate user names’ do
expect(passwd_names.find_all {|u| passwd_names.count(u) > 1}).to be_empty
end
end
[/ruby]
Here, passwd_names is an array of all user names – root, bin, sys, etc. We iterate over this array using the find_all method. This method takes a block that will return true for each element that fulfills the condition. So we count the number of times root, bin, sys, etc appear, and if there’s more than one, we know there is a duplicate username. This will return an array of results, and we expect it to be empty – no duplicates.
The last example I want to show here is 13.20, “Ensure shadow group is empty.” On Ubuntu 14.04, the default GID for the shadow group is 42, however, it is possible that was changed for some reason. In the let block we ask the system what the GID is.
[ruby]
control ‘13.20 Ensure shadow group is empty’ do
it ‘does not have any users in the shadow group’ do
expect(Etc::Passwd.select {|u| u.name if u.gid == shadow_gid}).to be_empty
end
end
[/ruby]
We aren’t using the passwd_name object because that only has the names – not the GIDs. It was simpler to not try and manage a data structure for our tests. We iterate over all the entries in /etc/passwd, using the select method to only return the name of users that have a GID of the shadow group. This will return an array, which should be empty.
Conclusion
We wrote the audit-cis cookbook for a couple of reasons.
- Implement the CIS Benchmarks for various platforms so the Chef community can take it and assess their nodes using audit mode, and see how they do against the recommendations.
- Provide a comprehensive set of example audit mode tests that the community can use as inspiration and examples for implementing their own audit mode rules, for whatever policy requirements they may have.
Ruby provides a number of robust modules and classes in its standard library that we can use within our tests to verify the state of the system. Serverspec is a great project that allows us to easily test various resource types that are commonly managed using Chef. Combined, these create a powerful framework for assessing compliance at velocity with relatively easy to understand code.