DEV Community

Microsoft Azure profile image Abhishek Gupta
Abhishek Gupta for Microsoft Azure

Posted on • Edited on

How to use Azure Go SDK to manage Azure Data Explorer clusters

#azure #go #bigdata #tutorial

Getting started with Azure Data Explorer using the Go SDK covered how to use the Azure Data Explorer Go SDK to ingest and query data from azure data explorer to ingest and query data. In this blog you will the Azure Go SDK to manage Azure Data Explorer clusters and databases.

Azure Data Explorer (also known as Kusto) is a fast and scalable data exploration service for analyzing large volumes of diverse data from any data source, such as websites, applications, IoT devices, and more. This data can then be used for diagnostics, monitoring, reporting, machine learning, and additional analytics capabilities.

In case you're wondering, we are talking about two different SDKs here. The one covered in this blog is for resource administration (also known as the control plane SDK) and the the one I used in the other post is data plane SDK for interacting with the Azure Data Explorer service itself (ingestion, query etc.)

What's covered ?

A simple CLI application is used as an example to demonstrate how to use the Go SDK. We'll try out the application first and go through how to:

  • Create and list Azure Data Explorer clusters
  • Create and list databases in that cluster
  • Delete the database and cluster 
$ ./goadx --help
CLI to test sample program for Azure Data Explorer

Usage:
  goadx [command]

Available Commands:
  cluster     create, list and delete Azure Data Explorer clusters
  db          create, list and delete databases in an Azure Data Explorer cluster
  help        Help about any command

Flags:
  -h, --help   help for goadx

Use "goadx [command] --help" for more information about a command.
Enter fullscreen mode Exit fullscreen mode

Once that's done, we'll walk through the sample code to understand what's going on

The code is available on GitHub https://github.com/abhirockzz/azure-go-sdk-for-dataexplorer

Please note that this CLI based example is just meant to showcase how to use the Azure Go SDK (in the context of Azure Data Explorer) as a part of a larger application. It is not supposed to replace/substitute the Azure CLI which can be used to manage Azure Data Explorer resources

Pre-requisites

Install Go 1.13 or above

You will need a Microsoft Azure account. Go ahead and sign up for a free one!

Install the Azure CLI if you don't have it already (should be quick!)

Run the CLI application

Get the code and build it:

git clone https://github.com/abhirockzz/azure-go-sdk-for-dataexplorer
cd azure-go-sdk-for-dataexplorer
go build -o goadx

//to confirm
chmod a+x goadx && ./goadx
Enter fullscreen mode Exit fullscreen mode

To see the details for individual commands, e.g. for cluster creation:

./goadx cluster create --help

//output

Creates 1 instance of compute type DevNoSLAStandardD11V2 in Basic tier

Usage:
  goadx cluster create [flags]

Flags:
  -h, --help          help for create
      --loc string    ADX cluster location
      --name string   ADX cluster name

Global Flags:
      --rg string    Azure resource group
      --sub string   Azure subscription
Enter fullscreen mode Exit fullscreen mode

Few more steps before creating the cluster...

Create a resource group ....

... using az group create CLI command

az group create -l <region> -n <name>

e.g. az group create -l southeastasia -n my-adx-rg
Enter fullscreen mode Exit fullscreen mode

We need a Service Principal for the Go SDK to authenticate with Azure Data Explorer service to execute cluster and database operations.

Create a Service Principal...

...using az ad sp create-for-rbac

az ad sp create-for-rbac -n "test-datax-sp"
Enter fullscreen mode Exit fullscreen mode

You will get a JSON response as such

{
  "appId": "fe7280c7-5705-4789-b17f-71a472340429",
  "displayName": "test-datax-sp",
  "name": "http://test-datax-sp",
  "password": "29c719dd-f2b3-46de-b71c-4004fb6116ee",
  "tenant": "42f988bf-86f1-42af-91ab-2d7cd011db42"
}
Enter fullscreen mode Exit fullscreen mode

Set the service principal details as environment variables. During the code walk-through, you will see why these specific variables are required

export AZURE_CLIENT_ID="<appId>"
export AZURE_CLIENT_SECRET="<password>"
export AZURE_TENANT_ID="<tenant>"
Enter fullscreen mode Exit fullscreen mode

You are all set!

Create an Azure Data Explorer cluster 

./goadx cluster create --name <name of the cluster you want to create> --loc <azure region for the cluster> --rg <name of the resource group> --sub <azure subscription id>
Enter fullscreen mode Exit fullscreen mode

For example:

./goadx cluster create --name MyAdxCluster --loc "Southeast Asia" --rg my-adx-rg --sub 9a42a42f-ae42-4242-b6a7-eea0ea42d342
Enter fullscreen mode Exit fullscreen mode

This will create a single instance cluster with the DevNoSLAStandardD11V2 VM

this is hard coded for sake of simplicity/ease-of-use

The cluster creation will take some time, and the code blocks for that time period. Please be patient ;) Grab some coffee, check Twitter or do whatever keeps you busy for about ~10-15 mins. Once the cluster is created, you will see this message:

created cluster MyAdxCluster with ID /subscriptions/9a42a42f-ae42-4242-b6a7-eea0ea42d342/resourceGroups/my-adx-rg/providers/Microsoft.Kusto/Clusters/MyAdxCluster and type Microsoft.Kusto/Clusters
Enter fullscreen mode Exit fullscreen mode

List down all the clusters

You just created a cluster, let's make sure you can get its info (including any other cluster you might already have in your resource group)

./goadx cluster list --rg <name of the resource group> --sub <azure subscription id>
Enter fullscreen mode Exit fullscreen mode

for example

./goadx cluster list --rg my-adx-rg --sub 9a42a42f-ae42-4242-b6a7-eea0ea42d342
Enter fullscreen mode Exit fullscreen mode

You will get a tabular output as such:

+---------------+---------+----------------+-----------+-------------------------------------------------------+
|     NAME      |  STATE  |    LOCATION    | INSTANCES |                          URI                          |
+---------------+---------+----------------+-----------+-------------------------------------------------------+
| MyAdxCluster | Running | Southeast Asia |         1 | https://MyAdxCluster.southeastasia.kusto.windows.net |
+---------------+---------+----------------+-----------+-------------------------------------------------------+
Enter fullscreen mode Exit fullscreen mode

You have the cluster, its time to create a database

To create a database in the cluster

./goadx db create --name <name of the database you want to create> --cluster <nname of the adx cluster> --loc <azure region> --rg <resource group name> --sub <azure subcription id>
Enter fullscreen mode Exit fullscreen mode

for example

./goadx db create --name testadxdb --cluster MyAdxCluster --loc "Southeast Asia" --rg my-adx-rg --sub 9a42a42f-ae42-4242-b6a7-eea0ea42d342
Enter fullscreen mode Exit fullscreen mode

It shouldnt take a too long. Once the DB is created, you will see this message:

created DB MyAdxCluster/testadxdb with ID /subscriptions/9a42a42f-ae42-4242-b6a7-eea0ea42d342/resourceGroups/my-adx-rg/providers/Microsoft.Kusto/Clusters/MyAdxCluster/Databases/testadxdb and type Microsoft.Kusto/Clusters/Databases
Enter fullscreen mode Exit fullscreen mode

To check the DB you just created ... 

./goadx db list --cluster <name of the adx cluster> --rg <resource group name> --sub <azure subscription id>
Enter fullscreen mode Exit fullscreen mode

for example

./goadx db list --cluster MyAdxCluster --rg my-adx-rg --sub 9a42a42f-ae42-4242-b6a7-eea0ea42d342
Enter fullscreen mode Exit fullscreen mode

You will get a tabular output as such:

+---------------------------+-----------+----------------+------------------------------------+
|           NAME            |   STATE   |    LOCATION    |                TYPE                |
+---------------------------+-----------+----------------+------------------------------------+
| MyAdxCluster/testadxdb     | Succeeded | Southeast Asia | Microsoft.Kusto/Clusters/Databases |
+---------------------------+-----------+----------------+------------------------------------+
Enter fullscreen mode Exit fullscreen mode

You can continue experimenting further and create more clusters and/or databases

Time to clean up ...

You can delete the cluster directly as such:

./goadx cluster delete --name <name of adx cluster> --rg <resource group name> --sub <azure subcription id>
Enter fullscreen mode Exit fullscreen mode

example:

./goadx cluster delete --name MyAdxCluster --rg my-adx-rg --sub 9a42a42f-ae42-4242-b6a7-eea0ea42d342
Enter fullscreen mode Exit fullscreen mode

You should see a confirmation message indicating that the cluster has been deleted:

deleted ADX cluster MyAdxCluster from resource group my-adx-rg
Enter fullscreen mode Exit fullscreen mode

Or you could just delete the database itself

./goadx db delete --name <database name> --cluster <adx cluster name> --rg <resource group name> --sub <azure subscription id>
Enter fullscreen mode Exit fullscreen mode

For example

./goadx db delete --name testadxdb --cluster MyAdxCluster --rg my-adx-rg --sub 9a42a42f-ae42-4242-b6a7-eea0ea42d342
Enter fullscreen mode Exit fullscreen mode

You should see a confirmation message indicating that the database has been deleted:

deleted DB testadxdb from cluster MyAdxCluster
Enter fullscreen mode Exit fullscreen mode

Alright, now that you what can be done, it's time to see how its done!

Code walk through

At a high level, the application consists of two parts:

  • Azure Data Explorer operations (create, read, delete) - part of the ops package
  • CLI commands - part of the cli package

To be honest, the CLI bit is not important (at-least in the context of the blog). It uses the (ubiquitous) Cobra package to implement the cluster and cluster (top-level) commands and the sub-commands - create, list and delete. If you're interested, take a look at the following:

  • root.go - defines the root command for CLI entry point
  • cluster.go - defines and implements sub-commands for cluster operations
  • db.go - defines and implements sub-commands for database operations

The CLI part simply invokes CRUD oeprations on Azure Data Explorer resources which is the important part. So, lets go through them

Authentication

This is handled in ops/client.go. To get a handle for the client components to execute the required operations, we need to authenticate. For e.g. to get the kusto.ClustersClient:

func getClustersClient(subscription string) kusto.ClustersClient {
    client := kusto.NewClustersClient(subscription)
    authR, err := auth.NewAuthorizerFromEnvironment()
    if err != nil {
        log.Fatal(err)
    }
    client.Authorizer = authR

    return client
}
Enter fullscreen mode Exit fullscreen mode

The part to note is the auth.NewAuthorizerFromEnvironment() which looks for a set of pre-defined environment variables - this is User environment-based authentication

Cluster operations

ops/cluster.go is where the cluster create, list and delete is handled. Here is the snippet for create:

func CreateCluster(sub, clusterName, location, rgName string) {
    ...
    result, err := client.CreateOrUpdate(ctx, rgName, clusterName, kusto.Cluster{Location: &location, Sku: &kusto.AzureSku{Name: kusto.DevNoSLAStandardD11V2, Capacity: &numInstances, Tier: kusto.Basic}})
    err = result.WaitForCompletionRef(context.Background(), client.Client)
    r, err := result.Result(client)
    ...
}
Enter fullscreen mode Exit fullscreen mode

CreateOrUpdate is used to initiate the cluster creation. Since this is a long running op, it returns a ClustersCreateOrUpdateFuture which we them wait on (using WaitForCompletionRef)

Note that I have used context.Background(), you can use a different context e,g one with timeout or one which can be cancelled.

Once that's one, we check the future for the Result

Listing all the clusters in a resource group is quite simple and can be done using ListByResourceGroup

func ListClusters(sub, rgName string) kusto.ClusterListResult {
    ctx := context.Background()

    result, err := getClustersClient(sub).ListByResourceGroup(ctx, rgName)
    if err != nil {
        log.Fatal(err.Error())
    }
    return result
}
Enter fullscreen mode Exit fullscreen mode

Delete is similar to Create in the sense that it is a long running operation as well. Its the same drill - initiate the deletion (using Delete), wait for it to finish and then confirm the result

func DeleteCluster(sub, clusterName, rgName string) {
    ...
    result, err := client.Delete(ctx, rgName, clusterName)
    err = result.WaitForCompletionRef(context.Background(), client.Client)
    r, err := result.Result(client)
    ...
}
Enter fullscreen mode Exit fullscreen mode

Database operations

ops/db.go is where the database create, list and delete is handled. Here is the snippet for create:

func CreateDatabase(sub, rgName, clusterName, location, dbName string) {
    ...
    future, err := client.CreateOrUpdate(ctx, rgName, clusterName, dbName, kusto.ReadWriteDatabase{Kind: kusto.KindReadWrite, Location: &location})
    err = future.WaitForCompletionRef(context.Background(), client.Client)

    r, err := future.Result(client)
    kdb, _ := r.Value.AsReadWriteDatabase()
    ...
}
Enter fullscreen mode Exit fullscreen mode

Listing is straightforward and taken care of by ListByCluster

func ListDBs(sub, rgName, clusterName string) kusto.DatabaseListResult {
    ctx := context.Background()
    result, err := getDBClient(sub).ListByCluster(ctx, rgName, clusterName)
    if err != nil {
        log.Fatal("failed to get databases in cluster", err)
    }
    return result
}
Enter fullscreen mode Exit fullscreen mode

And finally, the deletion which is handled using Delete

func DeleteDB(sub, rgName, clusterName, dbName string) {
    ...
    future, err := getDBClient(sub).Delete(ctx, rgName, clusterName, dbName)
    err = future.WaitForCompletionRef(context.Background(), client.Client)
    r, err := future.Result(client)
    ...
}
Enter fullscreen mode Exit fullscreen mode

That's it for the code walkthrough!

Conclusion

Hopefully this was helpful in demonstrating how to use the Azure Go SDK to work with Azure Data Explorer resources. You can check the documentation for Python, .NET and other examples. Happy exploring!

Top comments (0)

Read next

aryankargwal profile image

Multi-Modality and Image Gen in a 1.3B Model!🔮

Aryan Kargwal -

1shubham7 profile image

My journey from a CNCF contributor to LFX mentee

Shubham Singh -

thevediwho profile image

LazyVStack vs List in iOS 18 - #30DaysOfSwift

Vaibhav Dwivedi -

jaber1028 profile image

Landing Your First CS Internship: A Strategic Guide

Jacob aberasturi -