I recently wrote about the updates coming soon to Hosted Chef. In particular I described a brand-new permissions manager then scheduled to be deployed the evening of 5/8. This deploy did not occur. We chose to re-schedule due to the last minute discovery of a subtle and hard-to-isolate bug arising from the interaction between Hosted Chef back-end systems.
Delaying the deploy at the last minute was a major bummer for us. Even though the delay meant disappointing ourselves and our users we knew it was the right decision. We want to be as confident as possible of the code we deploy. We would’ve likely missed this bug entirely had it not been for the deploy team’s mantra of “test, verify, and test again”. Replacing existing systems is difficult especially when these systems contain complicated logic. Doubly so when a complex system is a component of a larger even more complicated system. Catching this single bug made the weeks spent rehearsing and validating test deploys worth it.
Since then we’ve been working hard to fix the bug. We’ve identified a minimally reproducible case and are focused on testing and validating a fix. As soon as we’re sure of the fix and we’re confident we have it right we’ll try the deploy again.
We apologize for the delay and unmet expectations. We remain committed to improving Hosted Chef. We promise the additional waiting will be worth it.
