In this article we will explore .NET 6 with ELK stack including ElasticSearch, Kibana and Serilog
What are we covering in this article
- What is a Elastic Search
- What is Kibana
- Why use ElasticSearch
- How everything fits together
- Companies which use ELK
- Ingredients
- Code
You can watch the full video on Youtube
And you can find the link to the source code on GitHub
https://github.com/mohamadlawand087/Net6-ELK
What is Elastic Search
Elasticsearch is a distributed, open-source search and analytics engine, Elasticsearch allows you to store, search, and analyze huge volumes of data quickly and in near real-time and give back answers in milliseconds.
ElasticSearch utilisation
- Search your way
- Analyze at scale
What is Kibana
Kibana is a visual interface tool that allows you to explore, visualize, and build a dashboard over the log data massed in Elasticsearch Clusters.
The core feature of Kibana is data querying & analysis. In addition, Kibana’s visualization features allow you to visualize data in alternate ways
With Kibana, it is easy to understand big data, and you can quickly build and share dynamic dashboards that frame-out changes to the Elasticsearch query in real-time.
Why use ElasticSearch
- E*asy to get started*: Just fire up a docker compose file containing ElasticSearch and Kibana containers and you’re ready to start logging and searching.
- Easy to Query: ElasticSearch has a built-in full-text search engine that is based on Apache Lucene. Compared to other databases, Lucene is easy to query. Even non-technical people would be able to write common queries.
- Fast: Querying a large SQL database can easily take 10 or 20 seconds. It’s quite common for similar queries on a large ElasticSearch database to return results in under 10 milliseconds.
- Free: Well almost. The basic features are free, well mostly. If you need security and alerting features in Kibana, you can buy the commercial X-pack subscription for Kibana, or you can install some open source alternatives.
- Full Text Search Engine
- Analytical Engine
- RESTful API: ElasticSearch has a RESTful API. Query results are returned in JSON which means results are easy to work with. Querying and inserting data via the RESTful API means it’s easy to use any programming language to work with ElasticSearch.
- S*calable:* It’s easy to scale. Combined with the fact that it's open source means it's easy on the wallet too.
Companies which use ELK
- Netflix
- Ebay
- Walmart
- Adobe
- Stack Overflow
Code Time
We create a docker file
version: '3.1'
services:
elasticsearch:
container_name: els
image: docker.elastic.co/elasticsearch/elasticsearch:7.16.1
ports:
- 9200:9200
volumes:
- elasticsearch-data:/usr/share/elasticsearch/data
environment:
- xpack.monitoring.enabled=true
- xpack.watcher.enabled=false
- "ES_JAVA_OPTS=-Xms512m -Xmx512m"
- discovery.type=single-node
networks:
- elastic
kibana:
container_name: kibana
image: docker.elastic.co/kibana/kibana:7.16.1
ports:
- 5601:5601
depends_on:
- els
environment:
- ELASTICSEARCH_URL=http://localhost:9200
networks:
- elastic
networks:
elastic:
driver: bridge
volumes:
elasticsearch-data:
Execute the command
docker-compose up -d
To test they are running we visit the following URLs
- ELS: http://localhost:9200
- Kibana: http://localhost:5601
Now we create our web application
dotnet new webapi --no-https -n DotnetELK
Install the packages
dotnet add package Serilog.AspNetCore
dotnet add package Serilog.Enrichers.Environment
dotnet add package Serilog.Sinks.Debug
dotnet add package Serilog.Sinks.Elasticsearch
dotnet add package Serilog.Exceptions
dotnet restore
Update appsettings.json
{
"Serilog": {
"MinimumLevel": {
"Default": "Information",
"Override": {
"Microsoft": "Information",
"System": "Warning"
}
}
},
"ElasticConfiguration": {
"Uri": "http://localhost:9200"
},
"AllowedHosts": "*"
}
We need to update our program.cs
using System.Reflection;
using Serilog;
using Serilog.Sinks.Elasticsearch;
var builder = WebApplication.CreateBuilder(args);
ConfigureLogging();
builder.Host.UseSerilog();
// Add services to the container.
builder.Services.AddControllersWithViews();
var app = builder.Build();
// Configure the HTTP request pipeline.
if (!app.Environment.IsDevelopment())
{
app.UseExceptionHandler("/Home/Error");
}
app.UseStaticFiles();
app.UseRouting();
app.UseAuthorization();
app.MapControllerRoute(
name: "default",
pattern: "{controller=Home}/{action=Index}/{id?}");
app.Run();
void ConfigureLogging()
{
var environment = Environment.GetEnvironmentVariable("ASPNETCORE_ENVIRONMENT");
var configuration = new ConfigurationBuilder()
.AddJsonFile("appsettings.json", optional: false, reloadOnChange: true)
.AddJsonFile(
$"appsettings.{Environment.GetEnvironmentVariable("ASPNETCORE_ENVIRONMENT")}.json",
optional: true)
.Build();
Log.Logger = new LoggerConfiguration()
.Enrich.FromLogContext()
.Enrich.WithExceptionDetails()
.WriteTo.Debug()
.WriteTo.Console()
.WriteTo.Elasticsearch(ConfigureElasticSink(configuration, environment))
.Enrich.WithProperty("Environment", environment)
.ReadFrom.Configuration(configuration)
.CreateLogger();
}
ElasticsearchSinkOptions ConfigureElasticSink(IConfigurationRoot configuration, string environment)
{
return new ElasticsearchSinkOptions(new Uri(configuration["ElasticConfiguration:Uri"]))
{
AutoRegisterTemplate = true,
IndexFormat = $"{Assembly.GetExecutingAssembly().GetName().Name.ToLower().Replace(".", "-")}-{environment?.ToLower().Replace(".", "-")}-{DateTime.UtcNow:yyyy-MM}"
};
}
Now we run our application
dotnet run
Add integration
- click on menu
- Under Analytics click discover
- create index pattern
- in the Name type: dotnetelk-*
- in timestamp field choose @timestamp
- click on create index pattern
- Now go back to discover and we can see all of the logs so far there
Now let us create a custom message, inside our controller we update the following
[HttpGet(Name = "GetWeatherForecast")]
public IEnumerable<WeatherForecast> Get()
{
_logger.LogInformation("WeatherForecastController Get - this is a nice message to test the logs", DateTime.UtcNow);
return Enumerable.Range(1, 5).Select(index => new WeatherForecast
{
Date = DateTime.Now.AddDays(index),
TemperatureC = Random.Shared.Next(-20, 55),
Summary = Summaries[Random.Shared.Next(Summaries.Length)]
})
.ToArray();
}
Now we can search for it by typing this in the search box
message: "WeatherForecastController Get"
Now let us test errors, inside our controller
try
{
throw new Exception("Testing exceptions - ML");
}
catch (Exception ex)
{
_logger.LogError(ex, "An unknown error occurred on the Index action of the HomeController");
}
Top comments (1)
Is it still considered ELK stack without Logstash? or do we have a different stack. ESK maybe? ES + Serilog + Kibana?