Spring Boot has been a very popular framework for building microservices in the cloud. Working with Spring Boot on Kubernetes has always been fun, but also comes with its own set of challenges as well as presents numerous architectural options ranging from application security, package management, containers, security, service configuration, and secrets management. More recently I have had a chance to evaluate and implement many of these patterns. In this post, I share some of my encounters and learnings in the service configuration and secret management space that I hope will help you on your K8s journey!
The Norm
A typical Spring Boot microservice configuration includes a bunch of profiles set up through application or bootstrap YAML files. One could better that with a pattern where we have profiles and application YAML files externalized through Spring Cloud Config. Now that we have configuration externalized, how about dynamically reloading configuration without application restart? Sure. One could do that with some kind of a job or a task that knows how to perform a /actuator/refresh once the updated configs have been deployed. Others may choose more of an automated approach with spring cloud bus over Kafka or RabbitMQ to publish state change events and reload applications appropriately.
Meet the k8s counterpart
At the core of the Kubernetes are concepts such as ConfigMaps and Secrets that provide a clean separation between sensitive and non-sensitive configurations for your services.
ConfigMaps and Secrets as Environment variables or files.
The twelve-factor app stores config in environment variables (often shortened to env vars or env). Env vars are easy to change between deploys without changing any code; unlike config files, there is little chance of them being checked into the code repo accidentally; and unlike custom config files, or other config mechanisms such as Java System Properties, they are a language- and OS-agnostic standard.
On Kubernetes we have a choice to make configurations sourced from ConfigMaps and Secrets available to the application as environment variables through deployment descriptors such as configMapkeyRef or secretKeyRef .
Another alternative to environment variables is to load Configmaps or Secrets as a file through k8s volumes and volumeMounts where applications also get the benefit of listening to file system change events as configurations may be updated.
Spring Cloud Kubernetes
Spring Cloud Kubernetes makes it a lot easier for Spring Boot applications on Kubernetes to leverage these features for service configuration as well as secret management.
Along with reading ConfigMaps and Secrets, the application can also watch for changes and be configured to reload itself if there were any changes. This is a pretty cool feature. However, this comes with a level of coupling as well as Kubernetes permission sets required (Such as access to the API Server) that may not be desired for the applications. Starting Spring Cloud 2020.0 release, this feature has been deprecated. Read the deprecation notice for more details. More details on the deprecation notice can be found here. What Next?
Config Watcher
There is now a config-watcher available that can be deployed as a side-car application that watches for any changes to ConfigMaps or Secrets and is capable of issuing a reload to your application across your replicaset. There are a couple of methods to achieve this.
HTTP — sidecar for config watcher
This integration is enabled through the DiscoveryClient where applications can enable this feature with an @EnableDiscoveryClient annotation. The config-watcher can discover pods that match the namespace or pod label configured and issue a refresh against the pods upon a configuration change event for a ConfigMap or Secret.
Spring Cloud Bus — sidecar for config watcher
Just like the HTTP option discussed above, the config-watcher also supports Spring Cloud Bus integration over RabbitMQ and Kafka (which I personally had the privilege to contribute to recently. See Support Kafka for Spring Cloud Kubernetes Configuration Watcher for more details). Upon Detecting a change the config-watcher publishes a ReloadEvent that the pods could listen to and perform an application Reload. See the documentation on cloud bus intergation for configuration details.
Another useful feature that I was also able to contribute to in this space is the ability to choose between enabling ConfigMaps and/or Secrets (See spring-cloud-kubernetes-config options and autoconfigure not working with reload for more details).
spring:
cloud:
kubernetes:
config:
enabled: true
secrets:
enabled: false
discovery:
enabled: false
reload:
enabled: true
monitoring-config-maps: true
Allows us that extra flexibility for applications that may only need to enable one or the other or both if you may wish.
You can already see the benefits here of using a sidecar container such as the config-watcher either over HTTP or Spring Cloud Bus. Gets us a level of decoupling that is more secured and saves the application from the finer details of watching for configuration changes. While this is a step forward this does not completely get rid of the dependencies on the K8s API Server. See Strongly discourage applications from talking to the Kubernetes API Server for more details.
In general, applications should not need to know that they are running inside Kubernetes. However, Spring Cloud Kubernetes allows you to detect this by automatically enabling a Kubernetes profile (adds onto existing active profiles if you have any already) that one could use to determine the deploy environment. This is useful if we had applications that are deployed on multiple platforms.
Consul
Consul from Hashicorp is a service mesh solution providing a full-featured control plane with service discovery, configuration, and segmentation functionality.
With the Consul KV, we could store plain key/value or files such as .yaml or .json. Spring Cloud Consul makes it a lot easier for Spring Boot applications to integrate with consul where .yaml files are automatically loaded as part of the spring bootstrap phase and attached to the appropriate profiles making it a great option for externalizing non-sensitive configuration and still benefit from spring profiles. In addition, it also supports an out of the box watcher that keeps track of any changes and automatically reloads any spring components that may have been @RefreshScope .
Consul service mesh enables service-to-service communication with authorization and encryption. Applications can use sidecar proxies in a service mesh configuration to automatically establish TLS connections for inbound and outbound connections without being aware of the network configuration and topology. A great way to secure your services with managed configurations or Intentions.
Vault
Vault is another product from HashiCorp that brings in capabilities for secrets management with a variety of secret engine backends such as consul (infra), database (infra), aws (cloud), and its own kv (generic) among others.
Vault KV can be very useful to store simple key/values and supports spring profiles for property sources.
The Database backend supports generating dynamic credentials for your applications and saves you from generating one and persisting it. This can be injected into your data source and also be eligible for lease rotation.
The AWS backend can be used to generate dynamic credentials based on IAM user or STS (assumed_role and federation_token) for AWS service integration.
The Consul backend can be used to secure application access to consul. For instance, applications would need a consul token to integrate with consul kv where the token itself is sensitive information and applications need a mechanism to securely request consul tokens and thus integrate with consul kv.
Spring Cloud Consul and Spring Cloud Vault
Spring Cloud Vault and Spring Cloud Consul can be used together and work pretty well and are great choices for Spring Boot applications integrating with Consul and Vault for service configuration and secrets management. There are some challenges at bootstrap to make sure the consul token is retrieved before asking for non-sensitive configuration from consul kv (Workaround available. See Consul Tokens from Spring Vault do not get picked up by Spring Cloud Config Consul for more details.).
Recently I did run into some issues with the spring cloud vault not supporting AWS STS for temporary credentials behind the AWS Secret Engine. This is being discussed under Support AWS STS for the vault secrets backend for aws and I have a pull request in the works!
Vault Agent Injector Sidecar
Another pattern that could be used with vault and pods on k8s is the sidecar injector option. Provides for an annotation-driven secret injection for your pods that could be a pretty handy way to decouple applications with secret management. This is also a great choice for applications and tech stacks that do not have a comprehensive SDK/library that supports vault integration by abstracting away the details and allowing pods to use mounts and provide secrets as files along with support for lease rotations. See Agent Sidecar Injectorfor more details.
Config file processing on Spring Boot 2.4
Starting Spring Boot 2.4 there is a significant change in how configs are loaded in spring boot. For example, Vault could now be used in conjunction with spring.config.import and provides a secured replacement to bootstrap.yaml.
spring:
config:
import: vault://secret/app/pres/dev
activate:
on-profile: "dev"
datasource:
username: ${username}
password: ${password}
Read config file processing for more details.
If you got this far, you would already know by now that there are a plethora of architectural patterns in this space. :-) There are not any rights or wrongs here and its’ mostly up to you to pick what's best for you/team/projects based on the capability, flexibility, and most importantly security you may be looking for with your configurations and secret management. I hope this post helps you a tiny bit along your k8s journey!
Good Luck and Stay Safe!
Thanks to Attila Vágó, Darragh Grace, and Francislâiny Campos for their feedback on this post!
Top comments (0)