Configmanagement Camp (CfgMgmtCamp for short, or YAML Camp), takes place after FOSDEM unless it’s 2021-2022, for obvious reasons. This year the 3-day fringe conference took place February 6-8, at the University College of Ghent, in Belgium.
Adam Jacob, former Co-Founder of Chef, in his keynote talked about “crisitunities”: Internet for everyone! Content on the internet! Applications on the internet! Social media! Digital transformation!
And in a “choose your own adventure” challenged us to pick rules for him to break, like Config as code, testing, and source control. To then reimagine them all, drastically, quoting American record producer Rick Rubin in saying that “rules direct us to average behavior”.
His new endeavor, System Initiative, is breaking all the rules and will be all open source, and we heard it first at Configmanagement Camp.
All that junk inside your trunk
Thierry de Pauw is a Senior IT Engineer at Abbove and the Founder of ThinkingLabs. In his talk he challenged my ideas about feature branches.
Apparently the only CI you achieve with feature branches is Continuous Isolation. “Feature branches delay feedback until changes are finally merged back into CI (the real CI, red).” But, but, but, what about automatic builds against branches? Those just prove that it builds, says Thierry, you won’t know how changes affect work done by other teams.
Feature branches hide work, work against refactoring. Maybe even adds to tech debt further. “It creates inventory. And inventory is money stuck in the system.” So what do we do?
- Reduce WIP branches
- Larger change sets and longer lived changes means context lost and finding root cause is harder, creates cognitive overload
- Practice extreme programming
- Your application is always in a releasable state on main line
- Trunk-based development
- Break changes into a set of small incremental changes
- TDD (Test Driven Development), and only commit when green
- Lots of fast tests (slow tests are at risk of not being run, or less frequently)
- Fixes are done ‘on prem’ and cherry picked into main multiple times a day
To audience members clutching their pearls / peer reviews, Thierry says that pairing can function as a way to do continuous code review. And he recommended everyone to read Accelerate (by Nicole Forsgren, Jez Humble, and Gene Kim), so naturally I’m halfway through the book (and late to the game).
Dylan Ratcliffe, in Unknown unknowns and how to know them: "There's a class of esoteric problems that we're not equipped to deal with."
Turns out that mental models from experience working in ops can't actually prepare us for all the ways a system can fail. Dylan recommends reading the STELLA report about dealing with anomalies. Which… I might, but first Accelerate!
“It’s easy to get into a tailspin of checking if the printer still works after changing something in a system that you thought was unrelated. Added process like that is what brings organizations to a screeching halt.”
- Outages always happen outside your mental model (unknown unknowns)
- Tools that require mental models therefore don't help
- Instead we need to make it easier to create mental models on the fly. This means making your inputs discoverable (for instance by dumping them in a Postgres, or a file you can easily grep)
And a shameless plug: Dylan’s company overmind helps companies understand unfamiliar systems, and needs feedback.
What do we want? Platforms!
What is an IDP? An Internal Developer Platform is built by a platform team to build golden paths and enable developer self-service. An IDP consists of many different techs and tools, glued together in a way that lowers cognitive load on developers without abstracting away context and underlying technologies.
Ulf Månsson has 10+ years of experience with IDPs and tells us that while there are companies that sell IDPs today, it is not something you can sell or buy. “It’s a combination of tools that is unique for your company."
Why would you go for an IDP from a business view? Speed of feature development and deployment, cost reduction, quality improvement, and the flexibility to adopt and roll-out infrastructure to meet new requirements. IDPs make devs self-serve, it’s standardization Ulf’s after: making it easy to do the right thing. Ulf says he’s lazy, by supporting the golden path he gets to spend more time on non-work stuff, like driving his tractor.
Mandi Walls, DevOps Advocate at PagerDuty, lists the benefits of automation, like the people in the room needed reminding, before moving to a Rundeck demo (the open source project Pagerduty recently acquired):
Dealing with complex systems and workflows
- Cope with change
- Reduce mistakes
- Reduce toil
Repetition is an opportunity for automation. Regarding what (else) we should automate, Mandi referenced that one XKCD cartoon.
Goals for any automation are that it should be testable, flexible (implemented for future improvement), reviewable by others, version controlled, repeatable and auditable.
Many a SRE, Support engineer, DBA, or developer would get huge value in being able to leverage automation, but can't because they miss knowledge, skills, or access. At the same time there is plenty of automation available in the cloud: scripts, APIs, IaC, platforms. You want to design self-service automation that support how your users work, provides consistency of experience, and includes documentation in context.
Your automation needs to ve safe to delegate as self-service actions to responders, for instance by authenticating users through PagerDuty, and by integrating with secrets management tools to access privileged actions.
Lastly Mandi talked about the evolution of automation:
- Identify automation opportunities
- Human initiated automation
- Automation with oversight
- Automation with fallback (humans notified if the automation fails)
- Monitor and evaluate (automation runs on its own and creates data)
- …
- Profit!!
On license dependencies
Dotan Horovits, Chief Tech Evangelist at Logz.io, talked about the daunting moment when the open source project(s) you’ve build your business on, go “bad”. Like when Elasticsearch changed their license to be more restrice, and in fact, not quite open source. The Open Source Initiative (OSI), the stewards of the Open Source Definition (OSD), confirmed that SSPL is in fact not an open source license.
AWS assuming the savior role in forking the version of Elasticsearch before the license change, while Elastic’s change was very much a response to AWS is… complicated. BUT: OpenSearch is an interesting project and it’s supported by Aiven.
If you’re building open source, Dotan will remind you that open source is not a business model. You will need to build a sustainable business, and create differentiation in your “Cloud” service. As an individual, don’t expect compensation.
Important for anyone using open source, is to manage your 3rd party dependencies’ licenses, be careful with automatic updates, and prefer extensions over modifying source. When vetting new technology, consider the license implications, consider who’s behind a project (foundations being the safer bet). Check the governance policy: who can review/approve PRs, and what's the growth path to becoming maintainers? And - and without serving the company that pays my bills too much - consider a software vendor or managed service. “Especially when that vendor contributes back upstream.”
Creating and keeping diverse teams
Feu Mourek, Developer Advocate at Icinga, and fellow Devopsdays organizer, talked about nurturing diverse teams. There’s internal, external, and professional diversity - the conferences you visited, skills learned on the job. Combined, those three shape your worldview (which can change over time). This means your team has a lot of different worldviews.
While all those worldviews combined offer a myriad of opportunities for collaboration, inclusiveness (and productivity, not having to worry about your appearance), employee satisfaction, and external representation (and thus talent acquisition), it also comes with challenges. Communication barriers, unconscious bias, systemic discrimination, and also change management is hard, yo.
Contributing to creating an inclusive environment as an individual is totally possible!
- Ask questions from a place of curiosity
- Accept if people don’t want to talk about something
- Listen to creators from different groups since they’re willingly putting in the labor of educating
- Assume people mean well
As an organization:
- Acknowledge (and learn from) the past
- Regular check-ins
- Give time and space for “experience exchange”
- Establish clear policies (CoCs)
- Hold people accountable
- Offer diversity and inclusion training
- Get professional help
Ultimately, Feu believes that “All conflict can be traced back to a mismatch in expectations”. Being intentionally excellent allows us to build and maintain diverse teams.
Top comments (0)