vNext Management and Automation of cloud and hybrid workloads

This post was created for the Azure Spring Clean 2023 event.

Introduction

If you happen to be responsible for running workloads on Azure in production, I bet you are familiar with Microsoft Monitoring Agent, Log Analytics, and Azure Automation services. They have been helping Azure customers with key management disciplines like monitoring, patching, inventory management, and process automation. We simply need to ensure our workloads are "well managed", protected, and watched for any 'abnormal' events that could impact the way our precious applications are available to their end-users.

With current trends like hybrid-cloud and multi-cloud and ever-changing business needs, our IT landscape is evolving with it, and what worked well and fit our needs a year ago might not be the best fit anymore.

Don't get me wrong, Azure Automation with its features is still a very robust and battle-tested service, but some specific design and implementation decisions make it difficult to stay relevant in the context of trends I mentioned.

Examples of such limitations are:

• dependency on Microsoft Monitoring Agent (MMA) and its centralized workspace-level configuration,
• Workspace mapping between Azure Automation and Log Analytics with specific regional pairs,
• support for non-Azure servers (on-premises and Third-Party cloud virtual machines),
• consistent onboarding experience.

One agent to rule them all

There were several major changes in Azure Management and Monitoring space, but I want to highlight two that are relevant to the topic.

The first one is Azure Monitor Agent (AMA) that aims (among other things) to consolidate the number of different agents that are available on Azure to monitor IaaS-based workloads and to decentralize its configuration by switching from workspace-level configuration to more distributed and granular Data Collection Rules (DCRs).

• AMA collects monitoring data from the guest operating system of Azure and hybrid virtual machines and delivers it to Azure Monitor for use by features, insights, and other services, such as Microsoft Sentinel and Microsoft Defender for Cloud.
• Azure Monitor Agent replaces all of Azure Monitor's legacy monitoring agents. There is a migration guide available for moving away from this 'legacy' agent.
• With DCRs you can completely decouple what is being collected (what logs, metrics, and distributed traces), where is it being stored (or streamed out of Azure), and from what sources (VMs).

Tip: Use the DCR generator to generate DCR resources (rules) from MMA workspace configurations.

The second change is the VM extension model that is now common for Azure VMs (with Azure VM Agent) and Arch-connected machines (with Connected Machine Agent). VM extensions are small applications that provide post-deployment configuration and automation tasks.

Disclaimer: Several capabilities I will be describing are still in preview, so make sure you understand Azure terms and conditions before you decide to use any of these in your production environments.

How to keep your VMs patched

Current

Azure Automation Update Management solution has been around for a long time and it allows for both assessing and deploying missing patches and other types of updates across your Azure and non-Azure server fleet. Onboarding to this service was based on MMA agent, its configuration, and target Log Analytics workspace that was linked to Azure Automation account that had the Update Management solution enabled.

vNext

Say hello to 'Azure Update Management Center' (UMC). This new service - currently in Preview - has several key features:

• it is a platform native functionality for Azure Compute and Azure Arc for Servers, globally available in all Azure Compute and Azure Arc regions
• there is no dependency on Log Analytics and Azure Automation services
• granular access control at resource level instead of access control at Automation account and Log Analytics workspace level
• improved onboarding experience at scale via Azure Portal and Azure Policy
• ability to enable on-demand and periodic assessments
• configuration of scheduled maintenance (update deployments)

Azure UMC relies on new VM Extensions (LinuxPatchExtension, WindowsPatchExtension) that works the same way on Azure VMs and Azure Arc-enabled servers:

• The extension is installed by either Azure VM agent or Connected Machine agent. No manual intervention required.
• This extension is automatically installed when you initiate any UMC operations such as check for updates, install one time update, periodic assessment on your machine.

Tip: For unattended onboarding of Azure VMs, it is required to set the following VM properties:

patchSettings.patchMode: AutomaticByPlatform
patchSettings.assessmentMode: AutomaticByPlatform
enableAutomaticUpdates: false

When the management of a VM is initiated, UMC pushes an extension to the agent. The extensions run locally and interact with the OS. They retrieve assessment information about the status of system updates and initiate download & installation of approved updates.

Machines assigned to UMC report how up to date they're based on what source they're configured to synchronize with, e.g., WSUS, Microsoft Update, a local or public YUM or APT package repository.

All assessment information and update installation results are reported to UMC from the extension and are available for analysis with Azure Resource Graph. You can view up to the last seven days of assessment data, and up to the last 30 days of update installation results.

The following diagram shows all key components and their properties:

Important: Check what OS versions and distros are supported during the preview in the support matrix.

The easiest path for enabling automated onboarding is by leveraging three Azure Policies:

1. Configure periodic check for missing system updates on Azure VMs: with Modify effect, it updates patchSettings.assessmentMode property for the deployed VM to be AutomaticByPlatform.
2. Schedule recurring updates using UMC: this DINE policy updates pathcSettings.patchMode property of the deployed VM to be AutomaticByPlatform and it deploys a 'Configuration Assignment' resource that links the VM with an existing 'Maintenance Configuration' object.
3. Machines should be configured to periodically check for missing system updates: this policy checks VM's patchSettings.assessmentMode property if it has AutomaticByPlatform value. It could either Audit or Deny the request. You can pick the effect you prefer upon assignment.

A word of caution: UMC can manage machines currently managed by Azure Automation UM feature without causing any disruptions in your update management process. Microsoft does not recommend customers to migrate from Update Management until UMC goes GA!

Track your software inventory and changes

Current

When you onboard your VMs to Azure Automation, you are also getting Change Tracking and Inventory features as a bonus, you just need to ensure you enabled them on your Automation account.

There are several items being tracked, like Windows software, Linux software (packages), Windows and Linux files, Windows registry keys, Windows services, and Linux daemons.

While there are still some limitations and the supported OS matrix is derived from Log Analytics agent, it is a robust service that can help you with e.g., Software Asset Management and configuration drift detection.

vNext

I guess it won't surprise you that there is a preview available for Change Tracking and Inventory using AMA agent. After all, we want to get rid of the MMA agent and its dependencies.

The main benefits are derived from the use of AMA agent: multi-homing support, granular management of 'data sources' using DCRs, and compatibility with the Change tracking (CT) extension deployed through the Azure Policy on the client's virtual machine. Once you switch to AMA, the CT extension pushes the software, files, and registry to AMA.

During the preview, there is an extensive list of limitations you need to take into account.

If you plan to onboard several VMs at once, consider using built-in policy initiatives for VMs, VM Scale Sets, or Arc-enabled servers. Simply go to the Azure Portal, open Azure Policy page, and search for initiative definitions that begin with [Preview]: Enable ChangeTracking and Inventory for string.

Let's have a closer look at the initiative for Azure VMs and figure out, what individual policies do:

• creates a User-Assigned Managed Identity and enables it for the VM (eventually you can bring your own / existing UAMI).
• deploys AMA extension for the VM (if it isn't deployed already)
• installs the ChangeTracking Extension (ChangeTracking-Windows or ChangeTracking-Linux) to enable File Integrity Monitoring (FIM) in Microsoft Defender.
• deploys 'DCR Association' to link the VM to specified DCR to enable ChangeTracking and Inventory

The missing piece is the deployment of a ChangeTracking-specific DCR. You can find the deployment template here.

Automate your IT processes

Unlike some of the previous areas, this one provides two mature generally available solutions you could choose from:

• Automation runbooks
• Azure Function Apps with PowerShell runtime

A couple of years ago, I had a session at Nordic Infrastructure Conference called 'Mortal Compat - Azure Automation vs. Functions', where I compared both services on several levels. I published the slides from this talk at Speaker Deck. Some info is outdated but surprisingly a lot of key facts are still valid.

I was a huge fan of runbooks and several improvements (like Managed Identity and PowerShell 7 support) rolled out in recent months, but Azure Functions allow you to apply event-driven approach and "plug in" additional components like Logic Apps and Event Grid. For me, this was a game-changer and I switched from runbooks to functions for most scenarios.

Modern in-guest configuration management

Current

The last piece in the puzzle is about being able to configure virtual machines and servers from "the inside" by declaring their desired state.

Azure Automation has a feature called Automation State Configuration. It has been around for a long time, and it allows you to write, manage, and compile PowerShell DSC (Desired State Configuration) configurations.

You could either enforce the configuration you want or use it as a report-only tool. And it supports non-Azure VMs as well (running on-premises or in "other clouds").

vNext

As you might have guessed, there is a newer and better version of DSC (generally) available, managed by a feature of Azure Policy named guest configuration. The guest configuration service combines features of DSC Extension and Azure Automation State Configuration. Guest configuration also includes hybrid machine support through Arc-enabled servers.

But wait, there has been a change in the name of this new service. Azure Policy Guest Configuration is now called Azure Automanage Machine Configuration. You didn't see that coming, did you? :)

Describing in detail how it works would require a separate blog post. You can find more info in the documentation.

My own conclusion

All those new management capabilities are in different stages of 'production readiness'; some are in preview, some GA, some available in limited regions or for limited OS versions.

Is it a suitable time to make the switch? I would say: "Not yet"!
Is this a good time to test them out, build proof-of-concept and get ready to migrate? Absolutely!

Remember: the clock is ticking, and August 2024 deadline is coming sooner than you think.