As your HashiCorp Vault cluster begins to handle more complex and varied tasks, it becomes essential to support multiple tenants. This approach helps centralize, scale, and standardize the management of your organization’s secrets lifecycle. To meet this need, Vault 0.11 introduced the concept of namespaces several years ago, which has since evolved.
However, the benefits and appropriate uses of namespaces are not always clear. This blog post aims to clarify these aspects, drawing from practical solutions engineering experience to explore what works and what doesn’t in multi-tenant deployments. We will explain when namespaces are suitable and when they are not, supported by real-world examples and insights.
What Namespaces Are Not
Vault namespaces are often misunderstood and misused. Here are some scenarios where namespaces should not be used.
A Security Boundary
Vault’s default policies deny access, meaning an empty policy results in no permissions within the system. Vault’s access control list (ACL) policies adhere to the principle of least privilege, which includes a deny-by-default stance. Therefore, namespaces don’t need to be used to manage these controls.
A more effective security boundary can be established through your infrastructure as code (IaC) workflow, which is crucial for preventing unauthorized lateral movement within your systems. For more details on securing your IaC, you can refer to HashiCorp’s guide.
An Onboarding Strategy
Namespaces should not be used as a consumption model where users can utilize them without any control. For secure self-service usage of Vault, you need a well-designed IaC pipeline and an API-based onboarding process, along with thoughtful Day 2+ management strategies.
Symptoms of an inadequate consumption model include empty namespaces or KV mounts. These should be avoided as they reduce the space available to other tenants. To prevent this, configure the secrets engine during the same provisioning process as the resource it provides access to, creating a seamless lifecycle management process.
When Should Namespaces Be Used?
Here are some typical use cases for namespaces.
Tenant Segregation
Namespaces help organize secrets engines, authentication mounts, policies, entities, and groups in a structured manner. When a client authenticates at the namespace level, their token can only have policies specified in that namespace, which is particularly useful with dynamic or templated policies to prevent incorrect path matching.
For cross-namespace access, you can add an entity from a different namespace to a group or move the authentication backend to a higher level, requiring a setting change in /sys/config/group-policy-application. However, such cross-namespace access is rare and may indicate a need to review your namespace strategy.
Granular Break-Glass Workflows
Administrators can use the namespace API to lock a specific namespace, similar to sealing Vault but without affecting multiple tenants or geographies. This granular approach allows for more targeted and controlled access management, reducing the risk of unintended consequences.
Setting Lease Limits
Lease count quotas help ensure cluster stability by protecting against unchecked lease generation that could overwhelm the system. This setting can be applied to a path or an entire namespace.
Setting Rate Limits
Vault allows operators to set resource quotas and apply rate limits to prevent system overloads and ensure consistent performance. This can be applied to any path, such as a mount or namespace. Rate limits manage the frequency of API requests, while lease count quotas control the number of active resource permissions, both essential for efficient resource management and system reliability.
Scoping Performance Replication
Namespaces can be used to scope what gets replicated to which cluster by configuring a paths-filter with an allow filter. For example, a primary cluster with namespaces defined by region can replicate each namespace to its own individual secondary cluster. Changes in a secondary cluster are also replicated back to the primary cluster unless a mount is created as local.
To ensure clients connect to the correct namespace on each cluster, consider using chroot in the TLS listener configuration.
Usage Cross-Charging
Implementing cross-charging of the Vault license allocates expenses based on actual usage, promoting accountability and encouraging resource optimization. This can be achieved by exporting the exact consumption per namespace and mount, which can then be used to generate detailed usage reports for better financial management and strategic planning.
Exploring Vault Namespace Structures
There are several approaches to structuring Vault namespaces, depending on your business requirements.
Geographical Region
Organizing namespaces by geographical region involves creating separate namespaces for each region (e.g., International, EU, US, APAC, on-premises).
Pros:
- Strong enforcement of data localization
- Clear communication between developers and the platform team about data hosting or management
- Customization of rate limits, policies, etc., by region
- Settings affect the entire cluster, not just the namespace within a cluster
Cons:
- Requires strict oversight for onboarding and configuration management
- Cross-charging can be challenging and needs to be processed per mount
- Changes affect all tenants in a namespace/cluster
Data Sensitivity
Organizing namespaces by data sensitivity involves creating separate namespaces for different levels of data sensitivity (e.g., Private, Internal, Confidential).
Pros:
- Aligns with common regulatory data-sensitivity classifications
- Can leverage replication path filters to link data sensitivity classification with geographical localization
- Allows easier enforcement of MFA per namespace path
- Simplifies limiting or disabling human access based on data sensitivity
Cons:
- Requires tight management during onboarding and configuration
- Cross-charging remains complex, requiring processing per mount
- Changes can affect multiple tenants
Business Unit
Organizing namespaces by business unit involves creating separate namespaces for each business unit (e.g., BU1, BU2, BU3). A strategy could involve counting four or five levels down the org chart starting with the CEO and making those teams the business units.
Pros:
- Clearer cross-charging
- Data residency issues can be addressed per business unit needs
- Simpler self-service onboarding process, as issues with one team’s namespace won’t affect others
Cons:
- Deciding the size of business unit namespaces considering namespace and mount limits is not trivial
- Business unit names and organizational structure changes can introduce technical debt
- Relying on namespace ID instead of a name requires documentation for developers to understand the Vault path
Application ID
Assigning a unique namespace to each application (e.g., App ID 1, App ID 2, App ID 3) is also a popular choice but requires several considerations to be scalable.
Pros:
- Clearly defines cross-charging and responsibility for each application
- Prevents accidental access between applications due to ACL policy errors
- Changes affect only a small number of services/applications
Cons:
- Managing a large number of App IDs may require sharding across multiple clusters due to namespace and mount limits
- Managing multiple namespaces across multiple IaC pipelines can be complex
What’s Next?
Carefully considering these factors and selecting the most suitable structure for your needs can help optimize your Vault usage for efficiency, security, and scalability. Regardless of the chosen approach, consider the following:
Moving to a New Structure
You have two options when moving to a new structure:
- Re-mount all with the mount migration API.
- Set up a new structure, migrate applications to the new paths, and decommission the previous structure.
Either way, the migration will generate new clients as entities are namespaced and cannot be moved. Consult your account team for planning and mitigating the impact.
Addressing the Limits
An estimated default limit is 7,000 namespaces and about 14,000 mounts, dependent on the storage backend used. Monitor the number of mount points and size of each mount table using Vault’s telemetry metrics. If using a raft/integrated storage backend, you can change the table entry size for mounts and namespaces, but read the guide on running Vault Enterprise with many namespaces to understand performance considerations. Avoid creating empty placeholder namespaces or mounts as they impact management of these limits.
How to Test Different Options
To experiment with different namespace approaches, start a development Vault cluster in the HashiCorp Cloud Platform (HCP) or request a non-production license to use Vault Enterprise locally. This allows you to test and optimize strategies before implementing them in production environments.
For more detailed information, refer to the namespaces documentation on the HashiCorp Developer website.
For more Information, Refer to this article.
