We at Chef believe it is important to conduct public post mortems whenever possible. We recently conducted one around a Supermarket/Berkshelf incident that occurred on May 16, 2016. I was the incident commander for this incident and would like to share both the video and write up.

Video Recording

Write Up

Description

On May 16 we experienced a brief SSL issue between Supermarket and Berkshelf.

Timeline

This incident began at 21:56UTC on Monday, May 16, 2016. It was resolved at 22:49UTC that same day.

**Time to detect**: 13 minutes 21:56UTC - 22:09UTC on Monday, May 16, 2016
**Time to resolve**: 44 minutes 21:56UTC - 22:36UTC on Monday, May 16, 2016
All times UTC
21:56  -   Nell Shamrell-Harrington upgraded 2 of the 4 Supermarket Prod nodes from Supermarket 2.5.2 to Supermarket 2.6.0.  She also upgraded the cookbook versions of oc-omnibus-supermarket and supermarket-omnibus-cookbook
22:09  -   Nell Shamrell-Harrington ran berks install to pull cookbooks from the public Supermarket and received this error:
           OpenSSL::SSL::SSLError: hostname "community-files.opscode.com.s3.amazonaws.com" does not match the server certificate
          She asked in the internal Chef Slack if someone else would run berks install to confirm what she was seeing
22:24  -  Lamont Grandquist confirmed that he was seeing the same error in Travis builds
22:32  -  Nell Shamrell-Harrington declared an incident
22:36  -  Nell Shamrell-Harrington moved the two upgraded Supermarket prod nodes out of the Supermarket prod ELB and confirmed that she no longer saw the error when running berks install
22:38  -  SaintAardvark in the #chef IRC channel reported SSL issues with running Berks install, Noah Katrowitz mentioned that kisoku (#chef IRC handle) was reporting the same thing
22:39  -  Noah Kantrowitz DM'd Nell Shamrell-Harrington to let her know that users in the Chef IRC channel were reporting issues with berks and Supermarket
22:43  -  Lamont Grandquist reported that Travis runs were working again
22:46  -  Nell Shamrell-Harrington entered #chef IRC
22:47  -  kisoku reported that his CI jobs were working again in #chef IRC
22:50  -  SaintAardvark reported that his Jenkins jobs were working again in #chef IRC
22:49  -  Nell Shamrell-Harrington declared the incident closed

Contributing Factor(s)

The 2.6.0 release of Supermarket included a commit which changed the AWS S3 urls used to access cookbook artifacts in S3 storage. Prior to this change, Supermarket (through the Paperclip plug in) used a hosted-style S3 url. The one for public Supermarket looked like this: https://s3.amazonaws.com/community-files.opscode.com/

The problem was this URL style only worked if an S3 bucket was in N. Virginia. To fix this, we changed our config to use a path-style url like this: https://community-files.opscode.com.s3.amazonaws.com

When this change was merged and deployed, this error appeared when someone attempted to do a berks install using public Supermarket as the cookbook source:

OpenSSL::SSL::SSLError: hostname "community-files.opscode.com.s3.amazonaws.com" does not match the server certificate


This was due to there being “.” in the bucket name “community-files.opscode.com.s3.amazonaws.com.” Although the previous S3 url style worked with dots in the bucket name, it did not work for a path-style url

Stabilization Steps

We had fortunately only upgraded 2 of the 4 prod nodes, so we removed the 2 upgraded nodes from the ELB, then downgraded them back to Supermarket 2.5.2

Impact

For approximately 53 minutes, anyone using berks install saw the SSL error.

Corrective Actions

• Make S3 url style configurable in Supermarket
• Make sure staging bucket has similar formatted name to the production bucket
• Ensure that berks install is part of smoke tests in both staging and production
• Add documentation around considerations when naming an S3 bucket
• Investigate adding a monitor that does a simple berks install and executes on a regular basis