The DevOps Handbook – Create Organizational Learning

We wrap up our deep dive into The DevOps Handbook, while Allen ruined Halloween, Joe isn’t listening, and Michael failed to… forget it, it doesn’t even matter.

If you’re reading this via your podcast player, this episode’s full show notes can be found at https://www.codingblocks.net/episode145 where you can join the conversation, as well find past episode’s show notes.

Sponsors

• Command Line Heroes – A podcast that tells the epic true tales of developers, programmers, hackers, geeks, and open source rebels who are revolutionizing the technology landscape.
• Educative.io – Learn in-demand tech skills with hands-on courses using live developer environments. Visit educative.io/codingblocks to get an additional 10% off an Educative Unlimited annual subscription.
• xMatters – Sign up today to learn how to take control of your incident management workflow and get a free xMatters t-shirt.

Survey Says

News

• Thank you to everyone that left us a new review!
  • iTunes: AbhishekN12, Streichholzschächtelchen Mann
  • Stitcher: emirdev
• Zoom introduces the Podtrak P8. (zoomcorp.com)

Wrapping up The Third Way

Use Chat Rooms and Bots to Automate and Capture Organizational Knowledge

• Chat rooms have been increasingly used for triggering actions.
  • One of the first to do this was ChatOps at GitHub. By integrating automation tools within the chat, it was easy for people to see exactly how things were done.
• Everyone sees what’s happening.
• Onboarding is nice because people can look through the history and see how things work.
• This helps enable fast organizational learning.
• Another benefit is that typically chat rooms are public. so it creates an environment of transparency.
• One of the more beneficial outcomes was that ops engineers were able to discover problems quickly and help each other out more easily.

“Even when you’re new to the team, you can look in the chat logs and see how everything is done. It’s as if you were pair-programming with them all the time.”

Jesse Newland

Automate Standardized Processes in Software for Re-Use

• Often times developers document things in wikis, SharePoint systems, word documents, excel documents, etc., but other developers aren’t aware these documents exist so they do things a different way, and you end up with a bunch of disparate implementations.
• The solution is to put these processes and standards into executable code stored in a repository.

Create a Single, Shared Source Code Repository for Your Entire Organization

• This single repository enables quick of sharing amongst an entire organization.
• In 2015, Google had a single repository with over 1 billion files over 2 billion lines of code. This single repository is used by every software engineer and every product.
  • This repository doesn’t just include code, but also:
    • Configuration standards for libraries, infrastructure and environments like Chef, Ansible, etc.,
    • Deployment tools,
    • Testing standards and tools as well as security,
    • Deployment pipeline tools,
    • Monitoring and analysis tools, and
    • Tutorials and standards.
  • Whenever a commit goes in, everything is built from code in the shared repo: no dynamic linking. This ensures everything works with the latest code in the repository.
  • By building everything off a single source tree, Google eliminates the problems you encounter when you use external dependency management systems like Artifactory, Nuget, etc.

Spread Knowledge by Using Automated Tests as Documentation and Communities of Practice

• Sharing libraries throughout an organization means you need a good way of sharing expertise and improvements.
  • Automated tests are a great way to ensure things work with new commits and they are self-documenting.
  • TDD turns tests into up-to-date specifications of a system.
    • Want to know how to use the library? Take a look at the test suites.
• Ideally you want to have one group responsible for owning and supporting a library.
  • Ideally you only ever have one version of that code out in production. It will contain the current best collaborative knowledge of the organization.
  • The owner is also responsible for migrating each consumer of the library to the next version.
  • This requires the consumers to have a good suite of automated testing as well.
• Another great use of chat rooms is to have one for each library.

Design for Operations Through Codified Non-Functional Requirements

• When developers are responsible for incident response in their deployed applications, their applications become better designed for operations.
• As developers are involved in non-functional requirements, we design our systems for faster deployment, better reliability, the ability to detect problems, and allow for graceful degradation.
• Some of these non-functionals are:
  • Production telemetry,
  • Ability to track dependencies,
  • Resilient and gracefully degrading services,
  • Forward and backward compatibility between versions,
  • Ability to archive data to reduce size requirements,
  • Ability to search and understand log messages,
  • Ability to trace requests through multiple services, and
  • Centralized runtime configurations.

Build Reusable Operations User Stories into Development

• When there is ops works that needs to be done but can’t be fully automated, we need to make them as repeatable and deterministic as we can.
  • Automate as much as possible.
  • Document the rest for operations.
• Automation for handoffs is also helpful.
• By having these workflows and handoffs in place, it’s easier to measure and forecast future needs and ETAs.

Ensure Technology Choices Help Achieve Organizational Goals

• Any technology introduced, introduces more pressure on operations for support.
• If operations cannot support it, then the group that owns the service or library becomes a bottleneck, which could be a major problem.
• Always be identifying technologies that appear to be the problem areas. Maybe they:
  • Slow the flow of work,
  • Create high levels of unplanned work (i.e. fire fighting),
  • Unbalanced number of support requests, and/or
  • Don’t really meet organizational goals, such as stability, throughput, etc.
• This doesn’t mean don’t use new technologies or languages, but know that your level of support greatly diminishes as you go into uncharted territories.

Reserve time to Create Organizational Learning and Improvement

• Dedicate time over several days to attack and resolve a particular problem or issue.
• Use people outside the process to assist those inside the process.
• The most intense methodology is a 30 day focus group with coaches and engineers that focus on solving real company problems.
  • Not uncommon to solve in days what used to take months

Institutionalize Rituals to Pay Down Technical Debt

• Schedule time, a few days, a week, whatever, to fix problems you care about. No feature work allowed.
• Could be code problems, environment, configuration, etc.
• Usually want to include people from different teams working together, i.e. operations, developers, InfoSec, etc.
• Present accomplishments at the end of the blitz

Enable Everyone to Teach and Learn

• Everyone should be encouraged to teach and learn in their own ways.
• It’s becoming more important than ever for folks to have knowledge in more than just one area to be successful.
• Encourage cross functional pollination, i.e. have operations show developers how they do something, or vice versa.

Share Your Experiences from DevOps Conferences

• Organizations should encourage their employees to attend and/or speak at conferences.
• Hold your own company conference, even if it’s just for your small team.

Create Internal Consulting and Coaches to Spread Practices

• Encourage your SME’s to have office hours where they’ll answer technical questions.
• Create groups with missions to help the organization move forward.

Resources We Like

• The DevOps Handbook: How to Create World-Class Agility, Reliability, and Security in Technology Organizations (Amazon)
• The Phoenix Project: A Novel about IT, DevOps, and Helping Your Business Win (Amazon)
• The Unicorn Project: A Novel about Developers, Digital Disruption, and Thriving in the Age of Data (Amazon)
• Comparing Git Workflows (episode 90)
• Git Large File Storage (GitHub)
• Devopsdays (devopsdays.org)
• DevOps: Job Title or Job Responsibility? (episode 118)

Tip of the Week

• Diff syntax highlighting in Github Markdown (Stack Overflow)
• Code Chefs – Hungry Web Developer Podcast (Apple Podcasts)
• Maria, a coding environment for beginners (maria.cloud)
• CodeWorld, create drawings, animations, and games using math, shapes, colors, and transformations. (code.world)
• Generation numbers and preconditions – Apply preconditions to guarantee atomicity of multi-step transactions with object generation numbers to uniquely identify data resources. (cloud.google.com)
• helm search repo – Search repositories for a keyboard in charts. (helm.sh)
• Use -cur_console:p5 in your cmder WSL profile to ensure that the arrow keys work as expected on Windows 10 (GitHub)
  • cmder – A portable console emulator for Windows (cmder.net)

Episode source