Ohai Chefs!
On December 1st we held the public post mortem for our force push to master issue. We briefly covered what happened, the timeline of events and then talked about ways we could prevent this in the future.
In the Chef repo a branch was accidentally force pushed to master, causing existing pull requests to include unnecessary diffs. We discovered the issue, rolled the git tree back to the correct state, and applied the commits which were originally meant to be included.
I emailed Github support and they have the ability to disable force pushes to a repo on all branches or the default branch. So we are going to disable force pushes to the default (master) branch of the Chef repo.
But this also seems like a good policy for more than just the Chef repo. Github can disable force pushes for all the repos in an organization, or multiple specific repos. We decided to disable it across all 221 repos in the Opscode org. No more force pushing to master, yay!
Another option we considered was using the Github webhook API to setup a Pagerduty alert notifying us when a force push occurred to an unwanted branch. This has the advantage that we could notify on more than just the default branch (12-stable, 11-stable, etc.). But we decided that if we can prevent the problem in the first place then we should do that.