You’ve just been given the responsibility to lead a DevOps transformation in your organization. Where do you begin? How will you approach the situation? What will you start or stop doing? What are your goals?
Luckily for you, many people and organizations, both large and small, have already experienced the growing pains associated with this process and have had great success. Now that others have worked through the challenges, there’s a basic playbook for moving your organization toward DevOps principles. It won’t be easy, but it can be done!
Let’s take Target, for example. They are the sixth largest retailer in the U.S., and they spend over $1 billion on technology each year. They rely on strong IT to compete online against competitors like Amazon and Walmart. The DevOps Handbook discusses Target’s transformation in various case studies. A case study on API enablement from 2015 discusses the organization’s shift to DevOps principles during 2012. It took almost a year to deploy new technologies, during which time Target adopted cross-functional teams, continuous integration, and increased automation. As a result of these changes, their digital sales increased by 42% during the 2014 holiday season, and they were able to introduce powerful business capabilities like Ship to Store and Gift Registry. Sounds like their investment in IT paid off, doesn’t it?
In this blog post, I review the best ideas and practices from The DevOps Handbook. This book provides concrete steps that you can take to transform your organization with DevOps. These aren’t pie in the sky ideas either. All of the practices presented in this resource have proven successful in practice and with focused case studies in larger organizations. Together, they provide a playbook for how to build a fast-moving, customer-facing organization.
DevOps, like all philosophies, requires adopting new assumptions. The first assumption deals with what your process should be optimized for. Generally speaking, your goal is to pick up the pace in presenting new ideas to your customers. The idea sounds simple enough, but it’s actually quite difficult in practice.
The first step in tackling this pacing objective is measuring your activity using what I call “lead times.” Ultimately, your goal is to shorten lead times, which in technical terms refers to how often production deployments happen. Utilizing this metric is critical. This metric alone tells us a great deal about an organization: how fast they turn business ideas into customer value, how quickly they can recover from bugs or failures, how reliable their deployment process is, etc.
Shortening lead times requires removing bottlenecks in your process — both technical and organizational. According to Dr. Eli Goldratt’s work, which can be found in The DevOps Handbook, “[t]o reduce lead times and increase throughput, we need to continually identify our system’s constraints and improve its work capacity. In Beyond the Goal, Dr. Goldratt states, ‘In any value stream, there is always a direction of flow, and there is always one and only one constraint; any improvement not made at that constraint is an illusion.’”
In essence, he’s saying that it isn’t enough to remove one constraint on a single occasion. You must continually assess and remove constraints from your process. This is the DevOps cycle, in which changes create positive feedback loops. These changes start at the top, which means they start with you.
Now that you have the power to reshape and direct your organization, you’ll be expected to put it to good use. Well-communicated and easily understood ideas help drive progress. The first order of business is communicating the idea that you’re going to give your customers more changes, more often.
That said, you must measure your lead times and make sure everyone in your organization sees the results. A simple measurement option might be the time from when an issue enters the backlog to the time when it’s verified in production, which captures everything in your process. The metric doesn’t measure all of the individual steps for putting XYZ into production since that’s irrelevant to your customer. What your customer cares about is how quickly they recieve your product or service. I recommend that you use this metric to better understand your customers’ and product team’s respective points of view — because they will undoubtedly feel differently.
Your first lead time numbers might be disappointing, but that shouldn’t be a surprise considering that’s the reason why you’re leading this transformation in the first place. To improve your lead time numbers, make sure that everyone in your organization sees the data and sees it often. Track and update it as much as possible. Put it on the screens around your offices. Use it to drive conversations at all stages in your process. Make it part of the daily conversation. Engrain it into your culture.
This is a powerful culture change on multiple fronts. First of all, it provides a simple, objective way to measure progress. Set a goal to drop lead times by X amount. You’ll be shocked at how powerful this direction is. Second, it promotes accountability in organizational processes through measurable metrics. It has a knock-on effect of creating a data-driven feedback loop in new places. Third, it forces your organization to continuously work on locating and fixing bottlenecks. Lastly, it will hopefully become personal for everyone in the organization — including you.
You’re leading this charge, but that doesn’t mean you can’t delegate certain tasks to other team members. Shortening lead times means increasing organizational autonomy. In other words, you need to give teams authority over and responsibility for their bit of the business. This kind of organizational change requires minimizing silos, reorganizing technical workers, and performing process checks.
Your organization needs cross-functional teams to succeed in the new IT landscape. Jeff Bezos did similar things when he reorganized Amazon. He set technical guidelines for how cross-functional teams should interact, and he established expectations for each team. These changes, along with the “you build it, you run it” philosophy, are now deeply embedded in Amazon’s culture, and they continue to drive the organization’s success. You’ll find similar stories from organizations like Target, in addition to some other unlikely IT success stories like Dixons Retail and Nationwide Insurance.
Organization and culture changes like the ones Bezos put in place empower knowledgeable team members to remove bottlenecks and shorten their own lead times. It also creates a technical framework for their activities.
Shortening lead times require changes in how your organization deploys its software. Your organization should strive for more deployments, so you should track that number continuously. Some DevOps organizations measure deployments per day, while others track deployments per developer per day. This may seem counterintuitive, but it results in fewer bugs and higher quality software. This practice is called continuous delivery (CD). CD is the technical embodiment of DevOps principles. Your goal is to push the organization in that direction. You can do that by requiring that each check-in must be deployable to production.
This check-in requirement forces your organization to adapt and enhance their existing practices. Generally speaking, this means running an automated test suite against each check-in and automated procedures to deploy each check-in. The automated test suite will ensure that everything works as expected. Automated deployment procedures remove human error and open up the deployments to more people. With automated test suites, your teams can be confident that new features will work correctly without breaking existing features, which is what your customers want.
However, constructing a CD pipeline isn’t straightforward, especially in brownfield projects. There’s probably a mix of manual and automated work. Think back to the quote from earlier. Start by eliminating manual, human effort in the process since these are the most likely bottlenecks. You can then expand outward from there. If the team is unsure of how to proceed, ask them what they can do to deploy more often and with more confidence.
The simple answer to this question might surprise you — work in smaller batches. As I have said, DevOps organizations work by shortening lead times. Generally, the two variables in this equation are how long it takes to deploy something and how big it is. If you can’t change the process, change the size. Smaller changes are usually easier to develop, test, deploy, and verify. For example, would you rather ship a major feature along with a banner color change, or just a banner color change? The answer is just a banner color change.
Having your teams work with the smallest possible batches will reduce lead times since less time will be spent on analyzing, reviewing, planning, and executing changes. Don’t assume that working in small batches means providing less value to the customer; it’s quite the opposite. Smaller batches with more, smaller changes get to the customers faster. What if the banner color changes would improve service for visually impaired customers? Why should they wait for other features? I recommend that you combine this practice with continuous integration for the best results.
Unfortunately, not everything fits into small, easily delivered batches. Some major business efforts require stages, customer verification, complex technical migrations, etc. This tends to happen with big bang deployments or extremely long release cycles. Deploying this way is unfavorable to say the least, but you’re probably familiar with it.
There is, however, a way around this problem — one that Facebook and many other organizations use. The solution is feature flippers, which are a fantastic alternative to big bang deployments. Using feature flippers, your team can control when certain product features are active. This is useful for beta testing among a preselected group of users or a random percentage of users, and even for disabling features when there are known problems.
Feature flippers provide engineers with more options in constrained environments, too. Maybe your team faces unique problems in production that can’t be replicated elsewhere. Feature flippers let your team test new features in stages rather than all at once. They also allow an “employee” mode so that everyone in the company can see the “next” version — all running in a production environment.
The technical and organizational practices presented in this post from The DevOps Handbook are just the tip of the iceberg. They’re all driven by feedback loops that are set up in code and in the organization itself. Creating these underlying feedback loops is your goal.
In order to succeed in your DevOps transformation, you need to hold everyone accountable to new, customer-facing value standards. Transformation happens in stages. You should consider walling off your DevOps transformation to a small team with complete buy-in. You must sell the ideas in order to be successful. Start small and expand outward when you’ve achieved desirable outcomes.
I recommend that you consider these pieces of advice and delve deeper into The DevOps Handbook for more guidance. Your efforts to shift your organization toward a DevOps transformation will be well worth it. The results of increased productivity and an adjusted culture will speak for themselves.