As my recent posts might have hinted, the eventual aim of these posts is to be able to search personal photo database by content, categories or locations as in country, city or even more fine grained where possible.
So far the following has been covered in the recent posts:
- Geotagging images that do not have GPS locations out of the box.
- Summarising photographs using multimodal models
- Reverse geocoding (this post)
Overview
This post will start with a high level background on reverse geocoding, OpenStreetMap and Nominatim as a locally hosted geocoding and reverse geocoding API.
We will then cover how to build an Aspire Resource for Nominatim and go through the building blocks. Initially we will focus on the basics.
Then there will be a section to focus on container dependencies at startup to be able to perform startup tasks and avoid dependent applications throwing exceptions while the dependencies are getting ready. For instance, in the last post, the background worker would get an error connecting to RabbitMQ as the RabbitMq container was initialising and then the error would go away. While this was not a problem, there is an emerging more elegant way covered in this post.
In this post we also go through how the connection strings can be maintained correctly regardless of if we are running Docker locally or remotely.
Reverse Geocoding
Reverse geocoding is the process of converting geographical coordinates (latitude, longitude) into human readable information such ass address or location information.
For example latitude 47.378177 and longitude 8.540192 can be translated to the following using a geocoding service:
{
"other_properties": "omitted",
"display_name": "Maru, Hauptbahnhof, Plaza, City, Altstadt, Zürich, Bezirk Zürich, Zürich, 8001, Schweiz/Suisse/Svizzera/Svizra",
"address": {
"amenity": "Maru",
"locality": "Hauptbahnhof, Plaza",
"quarter": "City",
"suburb": "Altstadt",
"city": "Zürich",
"county": "Bezirk Zürich",
"state": "Zürich",
"ISO3166-2-lvl4": "CH-ZH",
"postcode": "8001",
"country": "Schweiz/Suisse/Svizzera/Svizra",
"country_code": "ch"
},
"more_properties": "omitted"
}
OpenStreetMap (OSM) is an open initiative that aims to create a free and open source editable map of the world in collaboration with volunteers across the globe. OSM project provides not only open data but also necessary software to process and serve geographical data, apis and maps.
This post is going to focus on project Nominatim which uses OpenStreetMap data to provide necessary APIs for geocoding and reverse geocoding. Relevant tools and servers can be installed using traditional approaches on various operating systems as well as running as containers.
We will be using Nominatim container and build a Nominatim resource for .Net Aspire. We will also get to define service dependencies so that when our container is downloading data, depending services can wait for the download to complete.
It is worth noting that, if the requirements are for production grade services that could have financial impact due to availability or quality of the data, using a Software as a Service provider will be more reliable and scalable and most likely to provide better value.
Speaking of location services such as address lookup, geocoding and reverse geocoding, there are several aspects to consider as following:
- Scalability of location services so that you don't worry about peak time readiness such as Black Friday.
- Availability of such services so that your customers do not get stuck at registration / checkout due to service issues.
- Data quality means can you trust the results and ship the products?
- Data freshness and coverage means would you get global coverage without missing information? And do you know that they will be doing all necessary updates to keep the data up to date?
For this reason there are services such as Loqate from GBG that provide accuracy, performance and availability at scale so the businesses can focus on their own core competencies. There is a good reason Loqate is trusted by thousands of companies worldwide.
Building a Nominatim Resource for .Net Aspire
In this project Nominatim is used for reverse geocoding when importing photographs from a directory into the database. Given the photographs have GPS coordinates, the importer will enrich the photo information by converting GPS coordinates for the photos into address information.
As this is an excuse to understand Aspire better, it was the perfect excuse to build a custom resource.
In this section we will ignore the Health check and cover the usual components as below:
- Resources
- ResourcebuilderExtensions
- LifecycleHooks
Defining our Nominatim Resource
The first step is to define what our resource is. In case of Nominatim there are not that many additional properties.
- We can specify an optional map download url so that when Nominatim is spinning up for the first time, it can download the data and import into the DB
- Hostname: given in this project we might be running Docker locally on localhost or might be using a remote host by IP, we will need to know the host information so that we can expose the correct end points and also create a valid connection string.
- Port: similar to other examples, we specify the port so that these are mapped correctly.
The code below is self explanatory. If we are specified a host other than "localhost", we use that. Otherwise we default to "localhost". This is then used when creating the ConnectionStringExpression.
This means, when we switch between local and remote Docker hosts, we do not need to adjust connection string via environment variables later as we do with RabbitMq Resource.
Then
public class NominatimResource(
string name,
string mapsDownloadUrl,
string externalHostIpAddress,
string port = "8180",
string? entrypoint = null)
: ContainerResource(name, entrypoint), IResourceWithConnectionString
{
private readonly string _host = string.IsNullOrWhiteSpace(externalHostIpAddress) ? "localhost"
: externalHostIpAddress;
private EndpointReference? _endpointReference;
private const string NominatimEndpointName = "http";
public string MapsDownloadUrl { get; } = mapsDownloadUrl;
public EndpointReference Endpoint =>
_endpointReference ??= new EndpointReference(this, NominatimEndpointName);
public ReferenceExpression ConnectionStringExpression =>
ReferenceExpression.Create(
$"http://{_host}:{port}"
);
}
Providing Extension Methods for Building our Resource and Registering Lifecycle Hook
At this point, we are providing convenience methods to ensure the container resource is crested correctly.
Key aspects here are:
- Injecting environment variables such as PBF_URL so that the container can take these into consideration during start-up time.
- If we are using remote Docker host then runtime arguments are passed so that the containers listen on the Host IP adores over local network as opposed to localhost.
- Registers NominatimResourceLifecycleHook high is responsible for checking the status of container resource and publishing updates about current status.
- Ensuring the downloaded data is persisted.
- This is crucial as the download size and the database size can lead to taking about 10 minutes or so for the first startup.
public static class NominatimResourceBuilderExtensions
{
private const int ContainerPort = 8080;
private const string NominatimImage = "mediagis/nominatim";
public static IResourceBuilder<NominatimResource> AddNominatim(this IDistributedApplicationBuilder builder,
string mapUrl,
string hostIpAddress = "",
string imageTag = "4.4",
string name = "Nominatim",
bool importUkPostcodes = false,
bool importWikipediaData = false,
int? hostPort = 8180)
{
var nominatimResource = new NominatimResource(name, mapUrl, hostIpAddress, hostPort.ToString()!);
var nominatimResourceBuilder = builder.AddResource(nominatimResource)
.WithAnnotation(new ContainerImageAnnotation { Image = NominatimImage, Tag = imageTag })
.PublishAsContainer()
.WithEnvironment("PBF_URL", mapUrl)
.WithEnvironment("IMPORT_WIKIPEDIA", importWikipediaData ? "true" : "false")
.WithEnvironment("IMPORT_GB_POSTCODES", importUkPostcodes ? "true" : "false")
.WithExternalHttpEndpoints();
if (!string.IsNullOrWhiteSpace(hostIpAddress))
{
nominatimResourceBuilder
.WithContainerRuntimeArgs("-p", $"0.0.0.0:{hostPort}:{ContainerPort}");
}
else
{
nominatimResourceBuilder.WithHttpEndpoint(hostPort, ContainerPort);
}
builder.Services.TryAddLifecycleHook<NominatimResourceLifecycleHook>();
return nominatimResourceBuilder;
}
public static IResourceBuilder<NominatimResource> WithPersistence(this IResourceBuilder<NominatimResource> builder,
string nominatimDataVolumeName="nominatim-data",
string nominatimFlatVolumeName = "nominatim-flat-node",
string nominatimPostgresqlVolumeName = "nominatim-postgres")
{
return builder
.WithVolume(nominatimDataVolumeName, "/nominatim/data")
.WithVolume(nominatimFlatVolumeName, "/nominatim/flatnode")
.WithVolume(nominatimPostgresqlVolumeName, "/var/lib/postgresql/14/main");
}
}
The lifecycle hook here is not doing much work at the moment and all it does it to regularly check if Nominatim container is ready and then publish the status via ResourceNotificationService so that in the dashboard, we can see the current status. In contrast, the lifecycle hook in Ollama Resource actually does more by calling Ollama API to download the model if it does not exist and then publish status updates. So there are certain use cases where custom lifecycle hooks might more relevant as with Ollama.
.Net Aspire Health Checks and Start-up Dependencies
With docker compose, it has been possible to define health checks and dependencies so that we could have similar functionality to init containers where we can do certain tasks such as running database migrations, downloading test data and similar tasks.
In docker compose it looks like as following:
services:
web:
build: .
depends_on:
db:
condition: service_healthy
restart: true
redis:
condition: service_started
redis:
image: redis
db:
image: postgres
healthcheck:
test: ["CMD-SHELL", "pg_isready -U ${POSTGRES_USER} -d ${POSTGRES_DB}"]
interval: 10s
retries: 5
start_period: 30s
timeout: 10s
This makes life convenient for local tests as well as local development when using docker compose to run external services.
So how can this be possible in .Net Aspire?
Fortunately, there is an active community sharing use cases as Github issues and a responsive Microsoft representatives that not only take feedback but also accept PRs from the community as well as sharing internal Spikes on Some of the use cases.
So this is what David Fowler has offered in one of the issues asking about similar functionality to docker compose depends-on concept. While this is a spike, it is. providing the expected functionality by following the code examples provided.
.Net Aspire Health Checks and WaitFor Resources
This is a flexible solution that works as following:
- Define health checks: This is where you define the rules of being ready.
- Is the server running?
- Is the server accepting connections?
- Are the necessary data downloads and setup completed?
- Add Health Checks to the resource you create
- In the dependent resource, call
.WaitFor(resource)
In essence this is all it takes.
Define the Health Check
In our hearth check, we are making two checks:
- Is the server accepting requests?
- Are we getting success result to our search query?
If these two conditions are true, then the health check will pass.
public class NominatimHealthCheck : IHealthCheck
{
private readonly HttpClient _httpClient;
public NominatimHealthCheck(string url)
{
_httpClient = HttpClientFactory.Create();
_httpClient.BaseAddress = new Uri(url);
}
public async Task<HealthCheckResult> CheckHealthAsync(HealthCheckContext context,
CancellationToken cancellationToken = new CancellationToken())
{
var ready = await IsServerReady(cancellationToken);
Console.WriteLine(ready ? "Nominatim container is ready." : "Nominatim container is not ready yet.");
return ready
? HealthCheckResult.Healthy("Nominatim is ready")
: HealthCheckResult.Unhealthy("Nominatim is not ready yet");
}
private async Task<bool> IsServerReady(CancellationToken cancellationToken = default)
{
const string searchUrl = "/search.php?q=avenue%20pasteur";
try
{
var status =
await _httpClient.GetFromJsonAsync<NominatimStatusResponse>("/status?format=json", cancellationToken);
if (status == null || status.Status != 0)
{
return false;
}
var searchResponse = await _httpClient.GetAsync(searchUrl, cancellationToken);
return searchResponse.IsSuccessStatusCode;
}
catch
{
// ignored
}
return false;
}
}
Register the Health Check and then define the dependency
var nominatimContainer =
builder.AddNominatim(name:"Nominatim", hostIpAddress: dockerHost, mapUrl: mapUrl!, hostPort: 8180, imageTag: "4.4")
.WithPersistence()
.WithHealthCheck();
var backgroundWorker = builder.AddProject<Projects.PhotoSearch_Worker>("backgroundservice")
.WithReference(ollamaContainer)
.WithReference(postgresDb)
.WithReference(flaskAppFlorenceApi)
.WithReference(nominatimContainer)
.WithReference(messaging)
.WaitFor(ollamaContainer)
.WaitFor(nominatimContainer)
.WaitFor(messaging);
As we can see above, we first use the .WithHealthCheck() extension method adapted from David Fowler's example
/// <summary>
///
/// Reference: https://github.dev/davidfowl/WaitForDependenciesAspire
/// </summary>
public static class NominatimHealthCheckExtensions
{
public static IResourceBuilder<NominatimResource> WithHealthCheck(
this IResourceBuilder<NominatimResource> builder)
{
return builder.WithAnnotation(HealthCheckAnnotation.Create(cs =>
{
Console.WriteLine(cs);
return new NominatimHealthCheck(cs);
}));
}
}
.WaitFor(dependency) is works as is from the original code example
The only change in our example is CreateResilliencyPipeline() method where we extend MaxRetry attempts. Some of the maps are about 2GB and also takes long time to extract and import into the DB so this can take longer and therefore we allow for max 500 retries with max wait period 60 seconds.
So the code provided in the spike repository works out of the box in conjunction with any extension methods you need to introduce.
Once this API is agreed on and approved, it will make local environment setup much more convenient. Until then, thanks to David Fowler for providing an example demonstrating a potential solution.
Final remarks
One thing I have noticed is the way .Waitfor() works has an impact on OllamaResourceLifecycleHook and other lifecycle hooks.
Given we are using something similar to readiness checks, our resource is not ready until these checks pass. However, OllamaResourceLifecycleHook relies on the following hook to check and download the models at startup:
Task AfterResourcesCreatedAsync(DistributedApplicationModel appModel, CancellationToken cancellationToken = default(CancellationToken))
{
return Task.CompletedTask;
}
This hook is called after readiness checks complete so in my case, health check was failing as models needed to be downloaded but the lifecycle hook was not downloading the models as AfterResourcesCreatedAsync would not be called until resource started (health checks passed)
So the solution was simple. I had to move the code to BeforStartAsync lifecycle method as below:
public async Task BeforeStartAsync(DistributedApplicationModel appModel, CancellationToken cancellationToken = default)
{
foreach (var resource in appModel.Resources.OfType<OllamaResource>())
{
Console.WriteLine($"Verifying if models are downloaded for model for {resource.Name}");
await notificationService.PublishUpdateAsync(resource, resource.Name,
state => state with
{
State = new ResourceStateSnapshot("Initialising", KnownResourceStateStyles.Info)
});
DownloadModel(resource, cancellationToken);
}
}
As a reminder, the original code for Ollama resourc is provided by Jason Fauchelle from Raygun
Conclusion
So far, Aspire has surpassed my expectations. While the documentation from Microsoft is often great, what I found really useful in case of Aspire is the GitHub Issues and Discussions.
The use cases and challenges discussed there helped me better understand. In addition, Aspire seems to be taking shape by community contributions and participation which means it provides even more value to engineers by focusing on use cases Software Engineers often come across in the local development scenario.
The next issue I am excited about seeing progress is Option to set up and keep resources (containers and executables) after AppHost shutdown.
This is another important use case where we would typically start docker compose and let the services run without being tied into our application debug cycles.
It makes sense to let local dependencies keep running even if we are not running our solution. So this will be another issue to keep an eye on.
On the other hand, I have also looked into the fist steps towards looking into how the UI could work out so next post will be on integrating Web Components (using StencilJS) and mapping components to be able to view the photos on map as well as the summaries which will then lead up to the topic of how to measure model performance when dealing with generative models.
Top comments (0)