Six years ago, CSG International, the company that provides backend SaaS support to major telecom companies, started their DevOps transformation journey to modernize how they develop and run software. And for the past six years, CSG has made significant strides in transforming to a high-performing IT organization. Scott Prugh, SVP of Engineering and Chief Architect at CSG, says the key is to focus on process, people, organizational design, and culture.
Traditional to Agile
Historically, CSG has had a very traditional infrastructure with a ‘waterfall’ org structure. Scott’s vision was to have a 1×1 feature flow that was a 100% value add. In 2012, they began applying a new framework based on Lean practices that was optimized for developing efficiency. While some parts of the framework implementation were quite successful, CSG continued to struggle in the operational environment and not having the reliability they wanted. So in 2016, they implemented a DevOps reorg to create multi-disciplinary teams, combining developers and operations folks, so each team not only knew how to build what they were building, but how to manage that application in a production environment to ensure the most uptime with the least number of incidents. Scott coins the concept as ‘Service Orientation’ organized with flow and knowledge efficiency. And today, CSG continues to iterate on this model.
Success metrics
At ChefConf 2018, Erica Morrison, Executive Director of DevOps at CSG, spotlighted the successes of a couple teams that have gone through the transformation.
Increased speed
The Load Balancer team, which had traditionally been ops focused, added architects and developers to the team. Before the transformation, deployments were manual and change windows were up to six hours long which was a lot for a team that needed to make 20-30 production changes a week. By treating their infrastructure as code, they now have over 100 configs and a button-click ability to push code into their environments. Now that they have source code, they’re all speaking the same language and the future looks bright as this team continues to evolve to a self-service model.
Reduced risk
The Monitoring and Alerting platform team had been making good progress towards DevOps adoption maturity but still had a lot of room for growth. Giving visibility into their system usage was one of the first accomplishments of this team. They now have over 1,400 InSpec tests that verify CIS hardening standards and have also implemented a modified blue/green approach to improve patching.
With InSpec, CSG was able to significantly reduce their annual 20,000 hour effort for PCI compliance by 80%.
With Chef, CSG was able to demonstrate compliance of our configuration standards for multiple OS’s across our server population. This saved weeks of manual compliance checks for our PCI QSAs and far better assurance than same set checking. — Doran Stienike, CISO, CSG
Improved efficiency
When it comes to the business metrics, CSG has made significant improvements with their quality metrics with an 83% reduction in number of incidents caused by a release, which was extremely significant for their clients. That release impact score proved out that the move to agile practices in the new reorg structure, combining both dev and ops, to be a huge value to the business.
The future of DevOps at CSG
What does the future hold for CSG? A lot. Next is public and private cloud migration, a work/life balance initiative for CSG’s employees, and improving security and compliance. The key, according to Scott Prugh, is to involve all of the people in the value stream to help with these transformations. And as we can see, it works!
Learn more
Watch a recording of Scott and Erica’s main stage presentation from ChefConf 2018:
- Read the Continuous Automation for the Continuous Enterprise white paper.
- Try InSpec! See how InSpec can help you quickly identify potential compliance and security issues on your infrastructure.