DEV Community

Alec Dutcher
Alec Dutcher

Posted on • Edited on

DP-203 Study Guide - Monitor data storage and data processing

Study guide

Implement logging used by Azure Monitor

  • Azure Monitor key features
    • Metrics - resource utilization, response time, etc
    • Logs - leverage Azure Log Analytics to store and query logs
    • Alerts - set up alerts based on specific metrics or log data
    • Service maps, analytics insights, workbooks, and more
  • Log Analytics Workspace
    • Central repository and analytics engine
    • Collects, stores, and analyzes log data and other telemetry
  • Diagnostic settings
    • Feature of Azure Monitor
    • Control and route diagnostic data from Azure resources to various destinations
    • Source (metrics, resource logs, activity logs)
    • Destination (Event Hubs, Log Analytics, Azure Storage)

Configure monitoring services

  • What can be monitored in Azure Monitor?
    • Applications
    • Infrastructure (containers, OS)
    • Azure Platform (resources, subscription, tenant)
    • Custom sources
  • Configure Monitor for Azure Resources
    • Monitoring section of any Azure resource
    • Select "Metrics"
    • Choose scope, metric, visual type, etc
    • Save as Azure Monitor workbook

Monitor stream processing

  • Monitor Stream Analytics jobs via Azure Monitor in the portal, Powershell, or .NET SDK
  • In the portal, select "Metrics" under the Monitoring section
  • Can save metrics to a dashboard or send to Azure Monitor workbook

Measure performance of data movement

  • In the details of a pipeline, there are records of the tasks performed
  • Click the eyeglasses symbol on a record to view details about the performance (duration, throughput, start/end time, etc)

Monitor and update statistics about data across a system

  • Statistics provide info about how data is distributed in a table and help the database figure out most efficient way to run a query
  • Important for
    • Query performance and optimization
    • Optimal execution plans
    • Index utilization
  • Update statistics daily or after loading/transforming data
  • Enabled at database level with ALTER DATABASE database_name SET AUTO_CREATE_STATISTICS ON
  • Querying stats data:
- Display Query Statistics information
sp_helpstats N'StatisticsTest', 'all'

- Display extra information
SELECT FROM sys.stats AS stat
CROSS APPLY sys.dm_db_stats_properties(stat.object_id, stat.stats_id) AS sp
WHERE stat.object_id = object_id('StatisticsTest');

- Display query details
SELECT * FROM dbo.StatisticsTest
Enter fullscreen mode Exit fullscreen mode

Monitor data pipeline performance

  • Monitor section in ADF or Synapse Studio
  • Displays pipeline runs
  • Within a pipeline run you can view
    • Consumption
    • Pipeline orchestration (activities performed)
    • Data flow (activity inputs, outputs, etc)
  • View in list view or Gantt view

Measure query performance

  • Tools to measure query performance
    • Query store
      • Identifies performance differences when query plan changes
    • Intelligent Insights
      • Uses AI to continuously monitor database usage to detect disruptive events
      • Detection metrics
        • Query duration
        • Timeout requests
        • Excessive wait time
        • Errored out requests
    • Dynamic Management Views (DMV)
      • Monitor server health
      • Diagnose problems
      • Tune performance
      • Available via SQL queries
        • sys.dm_pdw_exec_requests
        • sys.dm_exec_requests
        • sys.dm_pdw_request_steps
        • sys.dm_exec_query_plan
        • sys.dm_pdw_waits waits

Schedule and monitor pipeline tests

  • Using a scheduled trigger
    • Add a new trigger
    • Leave as Schedule type
    • Choose start time and frequency
    • Go to Manage section and view triggered runs

Interpret Azure Monitor metrics and logs

  • Azure Monitor Metrics
    • Collects numeric data from monitored resources and stores it in a time-series database
    • Allows point-in-time descriptions of resources
  • Resources that AMM pulls data from
    • Azure Resources
      • First party services
      • Access to metrics is available by default
    • Azure Monitor Agent - Collects data from OS
    • Application Insights - collects telemetry about specific application workloads
    • REST API - get data in and out of AMM
  • Azure Monitor Logs
    • Collect and organize logs and performance data from monitored resources
    • Log Analytics workspaces
      • Edit and run log queries
      • Create alerts and workbooks
      • Analyze logs with Kusto Query Language

Implement a pipeline alert strategy

  • Set up a pipeline alert in ADF
    • Provides ability to combine data and business process
    • Configured in the Monitor section under Alerts and Metrics
    • Alert
      • Set target criteria
      • Send out notification to an email or group
      • Send notifications via text, push notification, etc

Top comments (0)