Hey chefs,
On 2014/12/01 we have released a few gems in order to prepare for upcoming Chef Client 12.0.0 release. During these releases we’ve caused 2 distruptions:
1. `ohai 7.6.0` release broke gem installation of `chef 11.16.4` when using an older version of `rubygems`.
2. Yanking `mixlib-shellout 1.6.0` release from `rubygems.org` caused issues in projects with `Gemfile.lock` files that are pointing to this version.
We have resolved the first issue by yanking `ohai 7.6.0` from `rubygems.org`. The resolution to the second issue is to update the `mixlib-shellout` to version 1.6.1 by running `bundle update mixlib-shellout`.
This post summarizes the details of these issues and the fixes that are in flight. We will have a public postmortem on **2014/12/03 at 13:00 PT** to determine any further prevention mechanisms and corrective actions. We will tweet the link for the hangout and youtube livestream for folks to join or watch when we are getting started.
#### “gem install chef” temporarily broken for 11.16.4
Huge thanks to migurski for reporting this issue. You can read the details of the issue here. As migurski pointed out the gem dependencies of recent releases of chef, ohai and mixlib-shellout are as follows:
`chef 11.16.4 depends on mixlib-shellout ~> 1.4`
`chef 11.16.4 depends on ohai ~> 7.4`
`ohai 7.4.0 depends on mixlib-shellout ~> 1.2`
`ohai 7.6.0 depends on mixlib-shellout < 3.0, >= 2.0.0.rc.0`
In addition to this `mixlib-shellout 2.0.0` is not backwards compatible with `chef 11.16.4` due to a change around encoding environment variables.
These constraints caused `gem install chef` command to fail with earlier versions of `rubygems` since it was not able to find out a solution for the dependency tree even though one exists with using `ohai 7.4.0` & `mixlib-shellout 1.6.1`. This issue was not seen on the newer versions of `rubygems`
Amongst the many possible resolutions to this issue we have selected to yank `ohai 7.6.0` from `rubygems.org` and decided to release `ohai 8.0.0` to go with `chef 12.0.0`. The reason to select this choice was to ensure that we don’t leave any versions of our gems out there which can create issues with the dependency resolution logic of older `rubygems`.
This resolution also keeps the dependencies clear in between major versions of `chef` releases. `chef 12.X` will depend on `ohai 8.X` and `mixlib-shellout 2.X` where `chef 11.X` depends on `ohai 7.X` and `mixlib-shellout 1.X`.
The next steps remaining in our resolution are:
* Release `ohai 8.0.0`
* Release `chef 12.0.0` with dependencies `mixlib-shellout ~> 2.0` & `ohai ~> 8.0`
Existing release, `chef 11.16.4`, will continue to work with existing release of `ohai 7.4.0` and `mixlib-shellout 1.6.1`.
#### Broken Gemfile.lock files with mixlib-shellout 1.6.0
One of the frequently reported issues for `mixlib-shellout 1.6.0` has been the fact that it is broken on Ruby 1.8.7. This was caused by a change that went into `mixlib-shellout` that wasn’t supported in Ruby 1.8.7. Even though we had this change in `mixlib-shellout` we didn’t update the `required_ruby_version` setting for `mixlib-shellout`.
In order to recover from this issue we’ve yanked `mixlib-shellout 1.6.0` from rubygems.org and released `mixlib-shellout 1.6.1` with only change of including `s.required_ruby_version = “>= 1.9.3″` in the gemspec of `mixlib-shellout`. This made sure that in the future we will not have any failures in `mixlib-shellout` due to the incompatible Ruby 1.8.7 logic.
However, since `mixlib-shellout 1.6.0` has been out for a while (since October 7, 2014), it made its way into many `Gemfile.lock` files. Once this version was yanked, attempts to install this version of `mixlib-shellout` using already existing `Gemfile.lock` files started failing. This impacted all the build projects of Chef Software Inc and potentially other external projects.
We were not able to reserve this issue since rubygems does not allow re-pushing an already yanked gems with good reasons. Workaround for this issue remains to be to update the `mixlib-shellout` version in the gem bundle of the impacted projects with this command:
“`
$ bundle update mixlib-shellout
“`
## Postmortem
Join us on **2014/12/03 at 13:00 PT** in our postmortem to talk about any additional corrective actions in order not to run into this issue in the future. We will tweet the link for the hangout and youtube livestream for folks to join or watch when we are getting started.
Thanks for your help on getting these issues resolved and your patience while they are sorted out. Feel free to reach us with any questions regarding this in our mailing lists chef and chef-dev or on Github.
