At ChefConf 2015, members of the Chef community received an invitation to participate in the Chef Community Cookbooks Survey. The survey was designed by Dr. Nicole Forsgren, Chef’s Director of Organizational Performance and Analytics, who is also a recognized expert on tool use, DevOps, and IT performance, particularly among technical professionals like sysadmins and developers. The survey assessed what people feel about Chef community cookbooks and how those cookbooks influence their use of community cookbooks and Chef.
This is the first in a series of blog posts that tells the behind-the-scenes story of the community cookbooks survey—how the project began, the science behind it, and what the results showed. Along the way, there will be tips and suggestions for creating measurement tools for your own projects.
Since Nicole Forsgren came onboard one of her aims has been to make Chef a metrics-driven organization. This includes building analytics systems for internal improvement, consulting and training on measurement practices, and developing ways to visualize our efforts.
Chef’s hiring a researcher and data scientist inspired Cameron Haight, Vice President in Research at Gartner, to write a blog post called Bringing in the Science. His post discussed Nicole’s work and stressed how important it was for other companies to begin using analytic techniques to better understand how people collaborate and process data.
The community cookbooks survey is one of Nicole’s more visible projects. It began in March, when Sean O’Meara, a community software engineer here at Chef, contacted her. Sean had an intuitive sense that a good experience with community cookbooks led to people using cookbooks and Chef more, and he wanted to see if that was true.
Nicole suggested he write down what he believed to be true about community cookbooks. Sean and Nicole also decided to ask Nathen Harvey, Community Director at Chef, to contribute because of his deep understanding of the open-source community.
All three agreed that a community cookbook is any cookbook that is publicly available on the Internet. It didn’t matter if the cookbook was written by a community member or by Chef because any cookbook available to the community may be a part of a user’s experience with Chef.
Next, Sean brainstormed some ideas of what were important for cookbooks, and Nathen added comments and his own insights. Together, they came up with the following ideas.
Nicole read through their list of ideas to get a feel of what their experiences with cookbooks were, keeping an open mind about where a potential survey might go.
What really excited her, however, was that the points Sean and Nathen made fell into three broad categories:
Coming from their own experiences with the community and community cookbooks, Sean and Nathen had started to identify some of the components of the Technology Acceptance Model (TAM), which is the standard model for predicting adoption and use of a technology. The model was first proposed in a paper written by Fred D. Davis in 1989[1]. Here is the TAM model.
On the one hand, it was gratifying that practical experience agreed so well with theory. On the other hand, from a researcher’s point of view, the agreement was a foregone conclusion. Nicole said, “In my field, you need to create or extend theory. Theory is the interesting piece. A journal is going to say, ‘It’s TAM. Of course it’s going to work. Of course it’s going to predict use.”‘ In fact, the cookbook survey did offer new opportunities. TAM hadn’t been tested extensively with open-source projects – so this really was an exciting survey, for Sean and Nathen, for Nicole, and for the research community.
Nicole suggested that they put together a survey that measured:
The adapted TAM model looked like this:
According to the TAM model, perceived usefulness (PU) and perceived ease of use (EoU) lead to an intention to use more cookbooks, and then to increased use of Chef and community cookbooks. When thinking about these relationships in terms of a statistical model, you can consider PU and EoU as independent variables that predict the intention to use community cookbooks, which is the dependent variable. In this model, use of Chef is also a dependent variable. This type of model is called a structural equation model (SEM), where each arrow represents a testable hypothesis. If the dependent variable has a positive correlation with the independent variables, it’s evidence of the hypothesis that positive perceptions of community cookbook usefulness and ease of use cause increased usage.
After establishing a testable hypothesis, the task was to design a survey that measured the variables of the statistical model. The challenge was how to reliably gauge attributes like human perceptions and feelings, which cannot be directly measured. In the language of research, these are latent constructs, which can only be measured using indirect indicators carefully chosen by the researcher to represent the underlying ideas. (For example, blood pressure can be directly measured by a medical device, so it is not a latent construct.
Nicole said, “We wanted to understand, from more than an anecdotal standpoint, the role community cookbooks play in the Chef ecosystem. We wanted an understanding of the role they play based on metrics. If we have the numbers and metrics that show that cookbooks are important to the use and continued adoption of Chef, then we can prove their value.
“The survey will also provide a baseline moving forward if we ever decide to run this survey again. We’ll be able to say what the perception of cookbook quality is now as compared to when we started.”
Next: Developing and Validating the Community Cookbooks Survey
[1] Davis, F. D. (1989), “Perceived usefulness, perceived ease of use, and user acceptance of information technology”, MIS Quarterly 13 (3): 319–340,doi:10.2307/249008