When you are building a configuration management system, one of the things you learn quickly is that you are going to hit a ton of edge cases. Many of these edge cases are the result of applications that were not written with automation in mind. As but one example, many applications do not change their exit code when they have an error – making automation very challenging.
We hit one particularly pernicious edge case recently while developing the Chef service provider for Red Hat. Whenever we would call out to /sbin/service to start/stop/restart a service, Chef would block forever waiting to read the output of the command. The bug is not unique to Red Hat. Indeed, we saw a similar bug that we fixed in a much less elegant way with Ubuntu’s CouchDB package. This is that bugs’ story.
We were using a CentOS 5.2 system to develop the provider. For this particular issue, we used /etc/init.d/gpm restart as our test case – but it appeared to happen with every init script we tried.
Here is a simple test case in Ruby:
#!/usr/bin/ruby
output = IO.popen("/sbin/service gpm restart")
puts "Reading output..."
puts output.read
puts "Would be nice to get here, but never going to happen."
Run that code snippet on any CentOS 5.2 desktop (which will have gpm installed) and you’ll notice that it blocks forever.
When you examine the process table, you’ll notice that the process we spawned (/sbin/service) has gone Zombie:
root 12586 Z+ 02:29 0:00 [service] <defunct>
For those of you unfamiliar with Zombies: when a process is spawned, its parent is expected to clean up after it.
Typically, this is done by issuing one of the variations on the wait system call, or catching SIGCHLD. If the child has exited, but the parent has not done its duty in cleaning up after the child yet, the process is marked as a Zombie. If the parent exits, the child will be inherited by init and reaped immediately.
When a child is forked, it inherits the file handles of its parent, in particular STDIN, STDOUT, and STDERR. You can see this behavior with this test script:
#!/usr/bin/ruby
STDOUT.puts "I am the parent @ #{Process.pid}"
STDERR.puts "I am the parents stderr"
cid = fork do
STDOUT.puts "I am the child @ #{Process.pid}"
STDERR.puts "I am the childs stderr"
exit 42
end
cid, status = Process.waitpid2(cid)
puts "Child #{cid} had exit status #{status.exitstatus}"
All we are doing here is the simplest fork – we spawn a new process, and print to STDOUT and STDERR. The output gets to the terminal because the child is using the same file descriptors as the parent. Because we’re a good unix citizen, we are running waitpid2 in our parent – we care about our children!
When dealing with pipes (or file descriptors), its important to remember one thing:
The read end of a pipe will never issue an end of file if a write end is still open
In the case of our errant init script, the issue comes down to this: one of our children has spawned a grandchild, and that grandchild has inherited one or both of our file descriptors, and neither cleans up properly – they fail to close their end of the Pipe.
This leaves the child zombied, because the grand child has neither closed their end of the pipe or exited – and the parent blocks forever on read.
Ideally, everyone would be a good unix citizen and clean up after themselves when they spawn children – in practice, they often aren’t. One way to get around this issue would be to send our output to a temporary file, rather than read directly from a Pipe. The child won’t know the difference, and reading files is not the same as having the read end of a pipe open, we can then read the temp file at our leisure, knowing that if our child is dead, its output is in the file.
#!/usr/bin/ruby
require 'tempfile'
pin = IO.pipe
outfile = Tempfile.new("chef-exec")
errfile = Tempfile.new("chef-exec")
cid = fork
if cid
pin.last.close
outfile.close
errfile.close
cid, status = Process.waitpid2(cid)
puts "Child #{cid} has exit status #{status.exitstatus}"
puts IO.read(outfile.path)
puts IO.read(errfile.path)
# Tempfile cleans up automatically when the objects
# go out of scope
else
pin.last.close
STDIN.reopen pin.first
pin.first.close
STDOUT.reopen outfile
outfile.close
STDERR.reopen errfile
errfile.close
exec("/sbin/service gpm restart")
end
While this is straightforward, it is a hack. For every process we spawn, we create a pair of temp files (one for STDOUT and one for STDIN)… and, if we need the output, we have to open, read, and close those temp files, then clean up after ourselves. That’s a lot of work for such simple functionality.
A better answer is to flip the order in which we do things just a little bit, and sprinkle a bit of O_NONBLOCK on our pipes. Avoiding the use of IO.popen this time (because we need a bit more control than it provides), here is an example that will not block, and still return all the output:
#!/usr/bin/ruby
require 'fcntl'
require 'io/wait'
pin, pout, perr = IO.pipe, IO.pipe, IO.pipe
cid = fork
if cid
[pin.first, pout.last, perr.last].each{ |fd| fd.close }
pin.last.close
pout.first.fcntl(Fcntl::F_SETFL, pout.first.fcntl(Fcntl::F_GETFL) | Fcntl::O_NONBLOCK)
perr.first.fcntl(Fcntl::F_SETFL, perr.first.fcntl(Fcntl::F_GETFL) | Fcntl::O_NONBLOCK)
cid, status = Process.waitpid2(cid)
puts "Child #{cid} has exit status #{status.exitstatus}"
puts pout.first.read if pout.first.ready?
puts perr.first.read if perr.first.ready?
else
pin.last.close
STDIN.reopen pin.first
pin.first.close
pout.first.close
STDOUT.reopen pout.last
pout.last.close
perr.first.close
STDERR.reopen perr.last
perr.last.close
exec("/sbin/service gpm restart")
end
That code can seem a little magical, so here it is step by step:
STDIN, STDOUT and STDERR.STDOUT and STDERR pipes to be non-blocking.That’s quite a bit more code than just calling IO.popen! But it does not ever cause your application to block just to read the STDOUT and STDERR of another program, regardless of how badly behaved its children might be. It also avoids the significant overhead of streaming the output to temporary files just to open, read, and close them again.
I want to give a special thanks to the many kind souls who helped us debug this problem (you know who you are – thank you.) In particular, Benjamin Black endured no end of my griping about “how you maybe couldn’t fix it, and the temp file trick wasn’t that bad, right?” and Artur Bergman pointed us in the direction of changing the order you wait for the child in, which was crucial in making things work.