On Tuesday, May 17th, I co-presented a live webinar covering a showcase of some of the latest Chef integrations with AWS. You can watch the recording below.
Most of the components we presented have been available to you in the Chef framework for quite some time. However, we worked with our friends at FastRobot to put together a code repository that does a lot of the boilerplate configuration you might typically do when getting started in AWS. That approach does two things: 1) it gives you a fully baked development environment as a quick-start for experimenting with Chef in AWS, 2) if you decide your development environment should be configured differently than what’s provided, you can reference the implementation for ideas on how to automatically customize your own.
During my time at Chef I’ve spent a good amount of time at AWS field events, run a variety of training sessions for both AWS engineers and AWS users, and spent time helping folks model solutions around approaching full stack automation and DevOps. I typically provide a pre-configured development environment during these sessions so that we can dive right into the bits that matter. The demo repo does that for you. The setup is opinionated and fairly simple. But if you’re just getting started in exploring Chef and AWS, it quickly gets you to the bits that matter.
In the webinar, we cover a basic Test Driven Development workflow with Chef. We manage a simple Sinatra application with nginx proxied to Unicorn. The app uses a Redis database to store data about the counter it displays. It’s a simple and contrived setup that’s intended as generic enough to be universally applicable, yet complex enough to show you all the moving pieces when considering testing vs. deployment patterns.
We cover a few new developments like the availability of Chef via the AWS Marketplace and the benefits of flexible consumption pricing. We show you some common deployment patterns you can incorporate into your own workflow. We also cover how to use the AWS Lambda service to enable a function that automatically, elegantly, and securely cleans up your Chef server any EC2 instance termination event.
At the end of this post you’ll find a summary of Q&A from the live presentation, including questions we didn’t have time to answer during the live event.
Special thanks to FastRobot. FastRobot is a San Francisco based consultancy, and Chef Certified Partner, focused on UNIX performance and automation. You can email them if you’d like to talk to them about how you can reach success together.
Q: Why is this demo set up to only use two AWS regions?
A: Flexible consumption pricing in AWS Marketplace currently has availability limited to the us-east-1 and us-west-2 regions. In order to enable quick start development environments that only pay for what you use, we have limited the demo repo to only work in these two regions.
Q: What is the scalability of a Chef Server. What is the best strategy to handle a large number of client nodes hitting the Chef Server?
A: The Chef Server in the Marketplace is built on a standalone instance that is reasonably vertically scalable. Performance benchmarks should be taken with a grain of salt (true performance is variable based on a number of workload factors), but our synthetic lab benchmarks against a single Chef server on a c3.2xlarge instance show support for more than 1,000 chef-client-runs/minute, or approximately a 30,000 node infrastructure using a standard chef-client configuration. Chef Server can scale much further beyond that mark (or provide HA for mission-critical applications) with multi-tiered configurations that provide options for either managing your own vertically-scalable data store clusters or that make use of AWS native services (RDS, Elasticsearch, etc) to provide a level of consistency and fault-tolerance that is appropriate for your particular setting.
Q: Currently to deploy Chef Server using AWS marketplace AMI we need to enter details like userid/org name etc. interactively. Is there a way to do this silently? Using some commands to configure Chef Server without any manual interaction?
A: You can run `chef-marketplace-ctl setup` with all the required parameters as a user-data script to do a non-interactive setup. See the chef-marketplace README for more information https://github.com/chef-partners/omnibus-marketplace#setup.
Q: IIS setup as well on Windows EC2 install for website deployment?
A: You can use the community IIS cookbook to configure IIS: https://supermarket.chef.io/cookbooks/iis
Q: How I can provision Windows EC2 install and install .Net framework and website using Chef?
A: There are several community .net cookbooks: https://supermarket.chef.io/cookbooks?utf8=%E2%9C%93&q=dotnet&platforms%5B%5D=
Q: I’m an AWS solutions architect and new to Chef. Is there a self paced tutorial series or document for me to understand Chef in detail.
A: The Learn Chef website has self paced tutorials to get you up and running with Chef.
Q: Does Chef support Bigdata frameworks like apache Hadoop or Spark?
A: Chef is not directly tied into any specific frameworks. However, available cookbooks allow you to use Chef for Hadoop and Spark configuration management.
Q: Can we install our Chef Server and use it with Supermarket cookbooks?
A: Yes. In the demonstration we used Berkshelf to automatically install and upload community cookbooks into our Marketplace Chef Server.
Q: What is the Chef Provisioning for ECS (Docker) Containers ?
A: Currently chef-provisioning-aws does not support ECS. However, many users use Chef to manage and create their docker container artifacts.
Q: What’s the update cycle for the Chef Server AMI on the Marketplace? I see that it is an older version.
A: New versions of Chef Server ship to the AWS Marketplace periodically, as necessary and may be slightly behind current as we go through the Marketplace release process. At any time you can update the embedded software to the latest version by SSHing into the Marketplace instance and running `chef-marketplace-ctl upgrade -y`. As per release cadence, we generally try to do a release when major components are released.
Q: Could you clarify if with Kitchen I can test that the infrastructure is working as I expect? Can I test things as firewall rules, networking rules among the nodes, etc.?
A: Test Kitchen is able to run InSpec tests that can verify many things, including firewall and networks rules that are available on the local instance. If you can query the data with the shell you can probably write an InSpec test to verify it.
Q: How can you keep your Chef Server in sync with a git repository?
A: In the demo repo, we did not include a Continuous Delivery tool since Chef Delivery is not yet available in the AWS Marketplace. In a typical deployment, all cookbook uploads are handled by a CI/CD tool. Using Chef Delivery, every cookbook is assigned its own build pipeline that automatically tests your new change(s), along with any upstream dependencies, provides visibility around code review and deployment, applies several layers of quality control phases/approval gates, and results in a releasable artifact in sync with commits you deem ready for review. You can also use other CI/CD tools to accomplish this same goal (Jenkins, Bamboo, Travis, etc), but you have to build those pieces yourself.
Q: Can you confirm the github link for that sample repo you used in the demo please?
Q: Is there an IDE for Chef cookbooks similar to Eclipse for Java?
A: Chef Cookbooks are pure Ruby so any IDE that has Ruby support will probably do fine. There is a Chef extension for Visual Studio Code that we’ve heard good things about. Several folks I know also like using Rubymine along with Chef.
Q: How does this confluence of tools allow one to quickly fallback to earlier configurations if we’re unhappy with the results in production?
A: Rollbacks are a mixed bag. Chef normally recommends rolling forward into a fixed state rather than rolling back to ensure consistent and accurate results. However, there are methods for falling back if that’s the approach that works for you. In cloudy environments, like AWS, it’s typically easy enough to revert back to the last know working set of cookbooks (found in a Berksfile.lock or a Chef environment file) and apply that to newly provisioned infrastructure (in a blue/green or phoenix upgrade pattern). For long-lived persistent infrastructure (why is that in AWS?), a rollback would likely mean modeling undo logic in your code and performing some dark unholy acts to get to point-in-time recoveries with less damage incurred than simply rolling forward. It’s doable, but it’s complex and requires significant maintenance on your part. At some point, it’s diminishing returns vs. just rolling forward into a fixed state.
Q: What will users feel when we apply changes using Chef and a user is in the middle on the website? How we will inform users that website is in maintenance?
A: Update strategies are very use-case specific depending on how you choose to automate the rollout of the application. Some customers will use Blue/Green, Phoenix, or Canary deployment patterns to roll out application updates (all of which are supported by Chef). There are many approaches, but the best one for you depends on the constraints of your application and operational environment.