Using app secrets in #dotnetcore console applications

I was writing a sample dotnetcore console application for a talk because why not! I felt using a sample aspnet core web app was overkill. The app was connecting to a bunch of Azure cloud and 3rd party services (think Twilio API for SMS or LaunchDarkly API for Feature Flags) and I had to deal with connection strings.

Now I have a nasty habit of "accidentally" checking in connection string and secrets into public GitHub repositories, so I wanted to do this right from the get go.

I started with this official documentation on adding configuration in a new .NET console application. To start with, add a package reference to Microsoft.Extensions.Hosting (example with dotnet cli)

dotnet add package Microsoft.Extensions.Hosting

Next, modify Program.cs to instantiate a new instance of the HostBuilder class with pre-configured defaults

using System.Threading.Tasks;
using Microsoft.Extensions.Hosting;

namespace Console.Example
{
    class Program
    {
        static async Task Main(string[] args)
        {
            using IHost host = CreateHostBuilder(args).Build();

            // Application code should start here.

            await host.RunAsync();
        }

        static IHostBuilder CreateHostBuilder(string[] args) =>
            Host.CreateDefaultBuilder(args);
    }
}

According to the Microsoft docs, The Host.CreateDefaultBuilder(String[]) method provides default configuration for the app in the following order:

1. ChainedConfigurationProvider : Adds an existing IConfiguration as a source.
2. appsettings.json using the JSON configuration provider.
3. appsettings.Environment.json using the JSON configuration provider. For example, appsettings.Production.json and appsettings.Development.json.
4. App secrets when the app runs in the Development environment.
5. Environment variables using the Environment Variables configuration provider.
6. Command-line arguments using the Command-line configuration provider.

While options 1, 2, 3, 5 and 6 sound like they would work, I would really like to use option 4 since it explicitly talks about app secrets and I believe secrets should be stored separate from config. Additionally, I really liked the app secrets experience, including support for POCO, when I worked with aspnetcore web applications. So, I decided to go with option 4.

But the Microsoft docs takes you a page that shows you Safe storage of app secrets in development in ASP.NET Core. This might work but I would like to keep my application as a simple console app not a web app. Still there are some instructions that would be useful on here.

The app secret values will be stored in a JSON file in the local machine’s user profile folder. For Windows this path will be %APPDATA%\Microsoft\UserSecrets\secrets.json and for Linux or MacOS this path will be ~/.microsoft/usersecrets//secrets.json.

Use the Secrets Manager command line to enable secrets for the project

dotnet user-secrets init

This puts a guid in your .csproj file. If you take this guid and plug it into the paths above instead of , you will get access to the the secrets.json file that will store the secrets for the application, in case you want to review them during debugging. Please note, the init function will not create the file. The file will be created and modified as you continue to add and modify secrets.

Next, we come up with POCO classes that can be used to map to the secrets. In this case, I split MyAppSecrets into AzureSecrets and ExternalSecrets and have dedicated sub-classes for each.

    class MyAppSecrets{
        public AzureSecrets CloudSecrets{get;set;}
        public ExternalSecrets UtilitySecrets{get;set;}
    }
    class AzureSecrets{
        public string SQLConnectionString{get;set;}
        public string CosmosConnectionString{get;set;}
    }

    class ExternalSecrets{
        public string TwilioApiKey{get;set;}
        public string LaunchDarklyApiKey{get;set;}
    }

Now to add the secrets themselves. The dotnet user secrets tool does not store nested properties as proper json. Instead it uses a : to separate the properties structure. Let’s take the above POCO classes, to access SQLConnectionString, we would use TopLevelClassName:PropertyClassName:PropertyName as the secret name, for instance, MyAppSecrets:CloudSecrets:SQLConnectionString. To set a secret, you would use a dotnet cli command as below

dotnet user-secrets set "MyAppSecrets:CloudSecrets:SQLConnectionString" "sqlconnstring"

Repeating this for all the sub-classes and properties you could end up with a secrets.json file that looks something like below – not very readable, I know

{
  "MyAppSecrets:UtilitySecrets:TwilioApiKey": "twilioconnstring",
  "MyAppSecrets:CloudSecrets:SQLConnectionString": "sqlconnstring",
  "MyAppSecrets:CloudSecrets:CosmosConnectionString": "cosmosdbconnstring",
  "MyAppSecrets:UtilitySecrets:LaunchDarklyApiKey": "launchdarklyconnstring"
}

Now that we have the secrets stored and the POCO class to retrieve it, app secrets should be enabled when we run the app in development environment. It turns out you can tell the HostBuilder to use a particular environment right after instantiating it.

Host.CreateDefaultBuilder(args).UseEnvironment("development");

Convenient, isn’t it? You can add and delete the UseEnvironment("development") code snippet and witness for yourself how Secrets are added and removed from the list of configuration sources by debug watching the sources variable in below code.

        static IHostBuilder CreateHostBuilder(string[] args) =>
            Host.CreateDefaultBuilder(args).UseEnvironment("development")
            .ConfigureAppConfiguration((hostingContext, configuration) => {
                var sources = configuration.Sources;
            });

But I really don’t want to have to deal with all these source since I’m only interested in the App Secrets source. I can do this using a configuration.Sources.Clear() call from the CreateHostBuilder method. Having done that I can build a configuration source for my app secrets and read it on to a static variable to use in the program as follow

Add a static variable for the Program class

static IConfiguration Configuration;

Then modify your HostBuilder method to build an app secrets configuration.

        static IHostBuilder CreateHostBuilder(string[] args) =>
            Host.CreateDefaultBuilder(args).UseEnvironment("development")
            .ConfigureAppConfiguration((hostingContext, configuration) => {
                configuration.Sources.Clear();
                Configuration = configuration.AddUserSecrets<MyAppSecrets>().Build(); 
            });

And boom goes the dynamite. Now you can use the static variable in your Main class to either access your secrets as a key value pair or by mapping it to a strongly-typed class as shown below

static async Task Main(string[] args)
{
    using IHost host = CreateHostBuilder(args).Build();
    //mapping static variable to strongly typed class
    var appSecrets = Configuration.GetSection(nameof(MyAppSecrets)).Get<MyAppSecrets>();
    Console.WriteLine($"Strongly typed mapping {appSecrets.CloudSecrets.SQLConnectionString}");
    //access secrets using key value pair
    Console.WriteLine($"Key value pair {Configuration["MyAppSecrets:CloudSecrets:SQLConnectionString"]}");
    await host.RunAsync();
}

But wait! There’s more. If you don’t want to have to deal with using the dotnetcore cli to enter awkwardly structured secrets like MyAppSecrets:CloudSecrets:SQLConnectionString, you can modify the secrets.json file directly with properly formatted JSON value and your strongly type mapping will ensure that these values are available to you either as strongly typed class properties or config key value pairs

{
    "MyAppSecrets":{
      "UtilitySecrets":{
          "LaunchDarklyApiKey": "launchdarklyconnstring",
          "TwilioApiKey": "twilioconnstring"
      },
      "CloudSecrets":{
          "SQLConnectionString": "sqlconnstring",
          "CosmosConnectionString": "cosmosdbconnstring"
      }
    }
}

Feel free to peruse the easy to read source code. I will add more instructions in the GitHub repo but this blog should suffice for now.