We recently migrated our infrastructure from Kubernetes to HashiCorp Nomad.
Soon after, we encountered service discovery issues and integrated Consul to address them.
At this point, we were feeling a bit more relaxed, knowing that we could dedicate less time to infrastructure and focus more on product development, as we don’t have a separate resource dedicated to DevOps.
However, we encountered another challenge—a Nomad port was open on the Node, and a friendly person helped us identify this vulnerability.
They were able to access it directly using the server's IP address, without needing a domain.
As a result, we had to quickly rotate all our payment keys, AWS credentials, and other secrets.
The Problem: Hardcoded Secrets and Security Risks
In some areas, our team had hardcoded keys directly into the codebase.
This meant there wasn’t a single place to manage these secrets, and updating them across our systems was a manual, error-prone process.
It was clear we needed a solution that would allow us to change a secret in one place, and have it propagate automatically throughout the system.
Initially, we considered using AWS Secrets Manager alongside our GitLab CI/CD pipeline to inject environment variables during container deployments.
However, this approach required passing a large number of variables as arguments to nomad job run from the project's .gitlab-ci.yml, which cluttered our configuration and reduced system readability.
That’s when we thought, Why not go full HashiStack? We decided to explore Vault.
Exploring Secret Management Options
AWS Secrets Manager
AWS Secrets Manager offers capabilities for managing and rotating secrets such as database credentials, API keys, and other sensitive information.
- Access Management: AWS Secrets Manager relies on AWS IAM (Identity and Access Management) for managing permissions. For granular control, additional tools like Chamber maybe required. Chamber leverages AWS IAM roles and policies to restrict access by namespace, simplifying the process of partitioning secrets across different environments.
- Cross-Cloud Limitations: While effective within AWS (our existing infra), use in multi-cloud environments.
- Cost of service: $0.40 per secret per month.
HashiCorp Vault
Hashicorp Vault provides a robust, identity-based security solution designed to authenticate and authorize access to secrets automatically, supporting a wide range of infrastructures.
- Access Control: Vault offers RBAC (Role-Based Access Control) for user logins, enabling administrators to manage human access to secrets. For machine access, Vault provides policy-driven access with leases, allowing for precise control over which applications can access secrets and for how long. This includes options like automatic key rotation and expiration.
- Dynamic Secrets Generation: Vault can dynamically generate AWS IAM credentials, providing temporary access that’s tightly controlled by policies and easily revoked if needed.
- Cross-Environment Compatibility: Vault supports cross-region, cross-cloud, and cross-datacenter replication, making it a seamless choice for organizations with multi-cloud setups.
- Cost of service: Free.
Why We Chose HashiCorp Vault
Vault offered the ideal solution: it allowed Nomad to request sensitive information, such as payment secrets or AWS keys, directly from Vault.
Vault would verify if Nomad was authorized to access those secrets and promptly deliver the data.
This way, we could centralize key management, automate secret updates, and improve security by avoiding hardcoded keys.
We developed a proof of concept, where we demonstrated Vault's ability to manage key rotation and securely store secrets in one place, accessible across all our projects.
This not only simplified secret management but also ensured consistency and security for our entire infrastructure stack.
Top comments (0)