This is the sixth and final entry in our ongoing, bi-weekly series examining our customer Standard Bank’s DevOps journey. You can read the first entry here, the second entry here, the third entry here, the fourth entry here, and the fifth entry here. Continue below for part six.
On February 11, the Chop Chop team went live with their prepaid feature. Currently, it’s available internally, on the Standard Bank network, but anyone who’s a part of that network can use it. The next day, there was a large, internal IT conference at Standard Bank where executives discussed what they want to do in the coming year. Dawie Olivier gave a presentation, showing how the app was promoted through the different quality gates into production, and how a person could log on and use it.
The Numbers
The Chop Chop team gathered some metrics to demonstrate what they’ve accomplished with Chef and DevOps.
- Time to build the stack: 26 minutes to build for production nodes (2 web servers, 2 app servers, with end to end deployment and testing)
- Number of automated tests embedded into Bamboo:
o Test Environment – Total 209 Tests
- 31 Infrastructure
- 178 Application – already existed before Chop Chop
o Production Environment – Total 39 Tests
- 31 Infrastructure
- 8 Application – written by Chop Chop
- Number of cookbooks – 12 (5 custom, 7 community)
- Time to market for pilot Internet Bank Refresh deployment – 12 weeks
- Number of catalogued automated services: 3
o Redhat Linux VM
o Enterprise Application Platform (EAP)/JBoss on Redhat
o Apache on Redhat
Looking Forward
The demonstration was well received and there was much discussion about the implications of adopting a DevOps culture. The members of the Chop Chop team are now thinking about how to put what they’ve learned into broader practice and what else needs to happen to bring Chef and DevOps to Standard Bank.
Derek Chung, the iteration manager, says, “Whatever the organization is looking for, whatever we can automate, is basically what we’re trying to do. Hopefully, it will reduce the amount of administration the bank faces at the moment and the amount of governance. The big question is, ‘Can we replicate the success we’ve had in the last few months?'”
Marcus Talkien, the technical lead, wants to build on the team’s growing library of Chef cookbooks to create a service catalog. “We’ve written our own JBoss cookbook, which has a whole bunch of functionality. The next project that comes along that needs JBoss, we can easily deploy it. That’s true as well of Apache. The next project that needs Oracle, it may take us a while to get our heads around it, but going forward, once we’ve got that Chef cookbook filled out, it becomes relatively easy that any project that wants Oracle, we’ve got a cookbook. It’s just about putting in the right parameters.”
He also wants to make sure that they safeguard the tools they use, such as Chef and Bamboo. “These tools need high availability (HA) and disaster recovery (DR) environments. For example, we’ve put in the Chef server but it doesn’t have HA. We definitely want to have a backup server ready to go in the event we lost our Chef box.”
Another consideration is how to best take advantage of their enterprise CMDB. It contains a great deal of information, but how to use that information well is still under discussion.
The implications of DevOps are far reaching and affect many areas of the bank. People from a variety of teams want to know how automatically provisioning environments will affect what they do. For example, how will they know when an environment is provisioned?
Because the bank has, of course, many auditing and compliance requirements to meet, the question of how to integrate the bank’s IT service management suite, BMC Remedy, into a DevOps workflow looms large.
Derek says, “From an ITIL perspective, there’s a lot of considerations that need to be taken into account. I don’t think the bank was prepared for such a disruptive concept. Before, we accepted the whole ‘four weeks to build a server’ idea. Slowly, the information would trickle down to the configuration management database (CMDB) and, obviously, the cost would get charged to a specific client.”
“There have been a lot of questions asked by the team that’s looking after ITIL, primarily about how the ability to create an environment on demand will affect them. There’s a lot of detail around instant configuration changes and release management.
“Historically, we haven’t had a mature configuration management practice. We’re just setting that up. This is one of the big opportunities we have to influence things. It’s not just CMDB. It’s also total cost of ownership (TCO).”
We have a TCO model that only charges per core that you’re allocated. How do you charge on a consumption basis? At the moment, we don’t have answers. The way the bank charges back is very complicated and the financial chargeback models are kind of static. You’re only charged every three months for what you’ve used in the last three months for full allocation. How do you charge if I allocate you half a core for one day? How do you pay for it? We are coming to grips with it. It will be a part of a big theme we’re trying to drive.
As always, culture is an important factor. Mike Murphy, head of IT Operations, discusses how to move from a pilot project to a more broad-based initiative. “How do you take the continuous delivery, DevOps concepts and scale them across a much larger organization? How do you approach it? We proved that what we thought was provable really works and the benefits we suspected we would see we absolutely do see. How do we make this single instance and the practices and culture associated with it much more widespread? I don’t think it’s a case that one can scale this in a linear fashion. Scaling is a complex thing to contemplate. Technology scaling can certainly happen but how do you scale and influence culture in a much broader way?”
“I guess you could say that, at the moment, we’re in kind of a contemplative, information gathering phase. We don’t have the answer. We’re speaking to a number of companies, including Chef. ChefConf at the end of March is going to be a great opportunity to talk to people to get a sense of what is possible and what has and hasn’t worked.”
“One other thing that’s important in this new way of doing things is to actually ask the team itself to contemplate how they would do it, given their hands-on, personal experience. How would they influence their colleagues? We’re trying to get the answer from the bottom up as opposed to a top-down, management approach. That’s where we are at the moment. We certainly do not know. We’re trying to work out what’s next. Everyone was wowed by what they saw at the showcase. There’s a general, kind of corridor buy in, so how do we make this real?”