Kusto01.log

I have heard 'Kusto' several times in my workplace or in some technical documentation about Azure. For sure, this concept was related to data as I heard about this word when I was doing something related to data. But I had so many questions as my understanding of this concept was shaky. So I decided to explore 'Kusto' a bit this time.

These are questions that I started with.
1) What is Kusto?
2) What is so special about Kusto?
3) When should I use Kusto?
4) How can I have hands-on experience with Kusto?

Please consider this post as one individual's personal note during the learning process. That being said, the information here is just my understanding and therefore, it might be inaccurate. Hopefully, as I develop a much better understanding of Kusto and start to see a bigger picture, I hope my future self can correct my past self's understanding.

So the first question first.

What is Kusto?

• At this point, I would say Kusto is a service that supports both data ingestion and data query. Kusto can get data from different sources, transform it and save it so that it can be queryable via SQL or SQL-like languages (ex: KQL). I do not have experience with the data ingestion part of Kusto yet. But in terms of querying data, I have seen people using some clients using Kusto Query Language (KQL, which I will cover in a bit) to query data to get some insights. I believe the entity that is responsible for data query service is Kusto.
• Kusto is a codename for Azure Data Explorer. So I believe I can interchangeably use Kusto with Azure Data Exporer.

What is so special about Kusto?

• I think Kusto service is used to explore the terabytes of real-time data to get insights. In other words, Kusto is the service that is optimized for streaming data such as logs or telemetry. To understand what it means to be optimized for streaming data, I had to understand what streaming data is. Streaming data is the data that is continuously generated by different sources. The amount of data is huge. As such, the services that query this type of data should be highly performant. And indeed, Kusto is one of those services.
• From the data ingestion perspective, my hypothesis is that Kusto is optimized to get any structure of data (structured, semi-structured or unstructured) from many different sources and can efficiently transform the data to save in their databases (Kusto databases). I am curious about what are these Kusto databases are. I haven't taken a look into this part this time, but I would explore later this week.

When should I use Kusto?

• In the practical world, I would use Kusto when I want to gain sights from logs or telemetry from my applications to debug any issues more effectively. This service capabilities are extended by other services such as Azure Monitor Logs, Application Insights, Time Series Insights, and Windows Defender Advanced Threat Protection. (Ref).

How can I have hands-on experience with Kusto?

• So now I have a better theoretical understanding of Kusto (relatively speaking), I would like to have some hands-on experience with Kusto. There are two options.
  • If my goal is getting familiar with Kusto Query Language itself, I think starting with Azure Data Explorer Web Client with the sample data is the right way to go.
  • If I want to create a testing cluster and testing database, ingest the data and then query that collected data, following the start guide will be the right choice.
  • I am wondering whether I can leverage my Azure Cosmos DB for my bot to see the data. I think this will be more interesting.

So today was my heavy research day for my first learning topic, Kusto.
Today learning gave me a good idea about what I want to prioritize for my learning.

1. Organize my notes about using Kusto for my work. *This will not be sharable. But I can keep it as my private notes.
2. Get Familiar with Kusto Query Language. Write down useful commands that would be used in my team. Apply those commands in Sample Data.
3. Learn how a cluster and a database need to be set up to be used for Kusto service, and check whether I can leverage my existing database. If it is possible, I can use KQL to get data from my bot. If not, simply follow the instructions.
4. Address remaining questions that I didn't have time to get to. a. From today's learning, these are the questions that I didn't have time to find answers.
   1. What sources can be connected to Kusto so Kusto can ingest the data from them?
   2. What happens when data is ingested in Kusto?
   3. What are the databases that Kusto uses (Kusto Database)?
   4. What's the structure of the Kusto database?