This post continues our bi-weekly blog series on Awesome Chef Paul Comtois’ DevOps Story. You can read part two in this series below, while part one is here.
Context: Pauly Comtois is the VP of DevOps at Hearst Business Media. Before that, he was Chef’s own VP of IT Operations. We recently caught up with Pauly and asked him about his time as Director of Operations and Application Support at a software company of about 200 people, where he introduced Chef and the DevOps culture. It’s a story that will be familiar to many who have made the journey away from manual processes and the negative effects of silos.
The Proposed Solution
We wanted to fix the situation, and decided to address the culture, process and tools. We began investigating automation tools to remove the manual steps from our deployments. I needed someone to run it, and I thought that the app team was the closest to success because they already knew how to code and had a strong operations backgrounds. It seemed that, no matter what form of automation was used, you needed either to know at least something about programming or have a willingness to learn.
We started with CFEngine, then tried Puppet and finally went to Chef because it fit our workflow best. Part of the reason, I think, was because Chefs approach brought people together and had a community that we felt fit our style and ops culture.
Unlike today, installing open source Chef back then was really tough. We almost didn’t do it, but everything else was just as hard with less reward. In the end, installing manually helped us to understand the underpinnings of Chef and how it was designed.
Leveling Up the App Team
Because we first started with the app team, we didn’t use Chef to manage hardware configurations. The app team used it for custom-built application configurations. They began by managing things like Oracle JVM and other tweaks and knobs that we needed for the application. That’s when we began to see some real wins.
There was one person on the app team who learned Chef inside and out. That person became my evangelist. He was a developer who had moved to ops. I haven’t had the pleasure of working with many folks that move from Dev to Ops (or vice-versa), but those individuals are usually remarkable human beings.
He already had a mindset for writing code, but he also had a great love for operations, which was wonderful because I had somebody at that point who could talk up and down the stack and more importantly, across teams. The focus on DevOps was successful largely due to his continued efforts to bring the developers to the table.
Leveling Up the Release Team
The next step was for the app team to train the release team to use Chef. We wanted to replace ControlTier and the homegrown automation scripts that we had for deploying code.
The release team was my first effort to try and get the Dev and Ops teams to work together. I needed a win to bring Dev and Ops together, and it turned out to be Chef. I started to bring in developers and operations folks in rotations. This got them working together and to start focusing on things like empathy, sympathy, compassion, and understanding what each team was doing. I was careful not to leave out the other critical teams: security, product management and quality assurance.
As you’d expect, there were some problems. A common objection from developers was that they were busy and didn’t have time to learn another language. I’d tell them, “You don’t really have to learn Ruby. You already know a language and the core concepts are close enough.”
Of course, everyone on the release team had a common pain point, which was they had to work weekends. Just the hope that Chef could fix that problem was enough motivation for people to give it a try.
With Chef we got the release time down from 12 hours and a team of around six people to one person in 15 minutes. This enabled us to move from quarterly deployments to deploying whenever we wanted. Most of that was just running the automated QA tests and deployment scripts. At that point, I was in love with Chef.
We never reached fully automated continuous delivery. We deployed manually, even though the process, the continuous integration through Jenkins, was there. Somebody still had to hit the button.
In part three of Pauly’s DevOps Story he’ll recall leveling up the Sys Admins and the “Big Win” he and his team accomplished. Stay tuned!