This is the fourth entry in our ongoing, bi-weekly series examining our customer Standard Bank’s DevOps journey. You can read the first entry here, the second entry here and the third entry here. Continue below for part four.
This post discusses how Standard Bank’s adoption of DevOps and Chef has affected its approach to building infrastructure. We’ll also see that changes in culture and the use of automation had far-reaching effects on both operations and development.
Creating consistent environments had long been difficult for Standard Bank. Mike Murphy, Head of IT Operations for the Standard Bank Group, described the process.
“We could spin up VMs fairly quickly. That was never the real issue. The issue had more to do with creating the machines in a predictable, standard and consistent fashion. The machines spun up relied, to a degree, on humans doing the right thing and we know that, oftentimes, that doesn’t work. Also, spinning up a cluster of machines to create an environment was not something we contemplated. Machines were literally spun up one by one, on their own, and in their own ways. The consistency simply wasn’t there.
For example, if we had an application that was built from scratch and deployed onto a virtual environment in production (with its associated high-availability (HA) and disaster recovery (DR) elements), we’d sometimes encounter a problem when invoking either the HA or DR component. This was, more often than not, as a result of differences in the configuration of the three environments that was caused by reliance on manual work. We didn’t really have peace of mind that either the HA or DR capability would operate as designed.”
Another issue was time. Mike says, “If there was a request from a project to spin up a “full-stack” (OS, DB, middleware, etc.), it would start with the VM itself getting spun up and when that piece was working, it was handed over to the database team, then to the middleware team, then to the backup team and then to the monitoring team. All of these teams, whilst housed within IT Infrastructure, operated relatively independently of one another and you’d have to hand off from team to team to team to get the activity completed. Because the hand off was manual and the configuration and deployment was manual, you could end up waiting weeks or longer before the solution was working. ”
Dawie Olivier, Executive Head of Group Technology Build, said, “Handoffs are a killer. In any lean environment, they’ll tell you that you lose between 15% and 20% of efficiency at each hand off. It only takes four or five handoffs before I should start paying the business for the right to develop the app.”
The solution was both technical and cultural. The chop chop team, responsible for the DevSecOps pilot project at Standard Bank, is multidisciplinary. Everyone involved with the prepaid feature, from operational administrators to developers, sit together and work together. They benefit from each other’s insights and add to the discussion. For example, in the past, if someone made a configuration tweak to improve performance, people would have found out about it by accident, if at all. Now, the change is a deliberate part of the design because it was a part of the conversation.
Crossing boundaries also shortened the learning cycle. Knowledge went straight into the solution instead of existing as documents that were handed over to another group. Nothing was lost in translation. The prepaid app was developed much more quickly than with the traditional approach.
After automation, deployments also happened more frequently. Team members participated in creating Chef recipes for automating the deployments. The chop chop team can now cycle through three different versions of a deployment in a day, or even half a day, just because they don’t have to wait for something to manifest in an environment.
Under the old system, creating an environment could take weeks. Now, thanks to a DevOps culture and automation, the team can spin up the entire Internet banking environment in 17 minutes.
Dawie says, “The part that’s powerful is when I can spin up a preconfigured and certified environment that’s ready for my entire Internet banking solution to run in it. That requires a very significant change to our approach. That’s what Chef does.”
Mike stressed the importance of cultural change and empowering the people on the team. “Perhaps the biggest learning for me was if you get the right people in the room and you give them the space to operate and you focus on clearing away impediments to their progress then amazing things can happen.”
“Chop chop was conceived, planned and delivered by the team. My role and Dawie’s was really hands off. The only thing we did was give the team access to resources and clear obstacles out of their way. It was a self-discovery journey for them. They literally did everything themselves. That’s why we’re looking to rework some of the cultural elements in the organization. How do we shift the culture from being a management-led culture to an engineering-led culture? How do we get the guys who do the actual work to drive the direction we should be going in?”
Next: Meet the test team.