Part 1 of 4 in an analysis series sponsored by CenturyLink.
Making Development & Operations Partners
DevOps, while a big topic for most enterprises today, is often misunderstood and not approached well. This usually leads to a lack of success in implementing and using DevOps or not fully gaining the benefits DevOps brings. Many of the common approaches that fail involve putting a developer into an on-call rotation; this doesn’t work because the developer doesn’t have a stake in Operations.
One of the best ways to learn about DevOps is to look at an organization that has already successfully implemented it. CenturyLink’s Cloud team is an example of such an organization.
Why did CTL Cloud choose DevOps?
Jim Newkirk, VP of Engineering, joined Tier 3 in 2012 – prior to its acquisition by CenturyLink in 2013. Jim has been a Board Member of the Agile Alliance since 2009. Jim saw DevOps as a way to avoid fluctuations and chaotic behavior within the Engineering team whenever important bugs came in.
As a result, as new bugs are being reported, they are assigned to an individual developer — building empathy for customers and a better partnership with Operations. Developers stopped seeing themselves as performing Operations functions and Operations started seeing themselves as Developers too, since both sides are working on things that relate to the CenturyLink Cloud Platform.
Whilst good leadership can help organizations make any type of cultural shift deliver value, it is only a beginning. In this case, it required both the Development and Operations teams to work together. This initiative was started as a way to create developer accountability.
The Dynamic Duo – Batman & Robin
DevOps is the system of cooperation and participation by both the Development (Batman) and Operations (Robin) teams of an IT organization with the shared goal of delivering high quality, easily maintainable software to users. Batman and Robin (the characters from DC Comics fame) are an inspiration behind an analogy that CenturyLink uses to describe how they use DevOps internally.
Who is Batman?
Batman is a Developer on-call.
Batman is a detective and a problem solver.
Batman is also responsible for stability – to the point of asking for better logging and better data points in order to troubleshoot customer issues.
Batman now understands user issues and customer side use-cases.
Batman is driving innovation and the customer on-boarding process onto the CenturyLink Cloud platform.
Batman is not a specialist. He is a generalist. Batman has to work with many different technologies and learn many techniques in order to do his job.
Batman is constantly taking on new challenges that are thrown at him.
New Developers are put onto the front line call rotation after three weeks on the job. By giving Developers direct interaction with end users, Developers gain an appreciation for how their software is being used and where the operational pain is. This is what creates an empathetic cultural shift for Developers in understanding Operations by allowing the Developers to “get into the minds” of Operations. Developers also need to be accountable for their solutions in production, when the user is using them.
Developers need to be generalists as well, being able to debug code, run and understand SQL queries, and scripting, as examples, because without broad breadth, you cannot be Batman. This generalist approach allows many people to effectively solve problems across the entire stack, instead of creating a dependence on a single individual to solve a specific class of problems. The generalist can solve most problems because of their full understanding of how the system as a whole works, giving them the insight needed to get to the root cause of the problem and fix it.
Where does Robin fit in?
Robin is a customer service-centric thinker
Robin is a consumer of Development – a Systems Developer
Robin uses APIs – a functional programmer.
Operations figures out how to streamline the deployment and automation of applications written by the Development team. To do this, they write their own code against APIs supplied by the Development team, making them users of the code and platform as well. Because Operations is actually using the code and not just Operating the systems running code for end users, they are exposed to any weaknesses much earlier and understand how the applications and systems actually work. This is why Operations is Robin. Batman (Developers) can only do so much, often times Batman needs Robin to help him solve many of the problems that an individual can’t solve alone. Robin has the authority to make fixes live to solve customer issues.
What about the Utility Belt?
Batman relies on his skills, but he also takes full advantage of his Utility Belt. Batman’s Utility Belt is purpose built to solve a large number of general limitations that he faces as he tries to solve problems. In DevOps there are a large number of tools that can be used to automate and solve problems in both Development and Operations activities.
Here are some examples of items in CenturyLink’s Utility Belt:
Source: Gigaom Research
Benefits
The CenturyLink Cloud team already had good structure in place before adding DevOps practices into their organization. For the purposes of this report, we are considering a very small subset of the prior structure:
- Knowledge Base for support
- Deployment time of 5 hours over 3 data centers
Since adding the DevOps-Batman & Robin ethos:
- Knowledge Base has grown by 4x in 12 months
- 40% of tickets are solved immediately through the Knowledge Base
- Deployment time of 1 hour over 12 data centers
- Still 1 Batman/Dev on-call with 3x VM inventory in 12 months and 4x Cloud capacity
The Knowledge Base is fed Batman (Dev) & Robin (Ops) teams. Better operational efficiencies have been realized through this new process even with the CenturyLink Cloud’s VM inventory tripling in last 12 months and capacity has quadrupled.
This same team now also owns infrastructure engineering – further removing dependencies on a separate a team for the CenturyLink Cloud platform.
Both Batman and Robin are key to keeping the CenturyLink Cloud growing and running and its customers happy – regardless of whether they are internal groups such as Savvis, or external groups such as partners or a customer of the platform.
Everyone is an Engineer.
Join me and the CenturyLink Cloud team this week on a webinar where we will discuss this post and other findings.
