# Architecture (/docs/architecture)
BoltMCP is a platform for creating and managing custom Model Context Protocol (MCP) servers. A single Helm chart deploys the full stack - application services and optional database and identity provider - into a single Kubernetes namespace.
## Architecture [#architecture]
## Database Isolation [#database-isolation]
Each BoltMCP service connects as its own PostgreSQL login role scoped to a `boltmcp_*` schema:
| Service | Login role | Schema | Access |
| ----------- | ---------------------- | ------------------ | -------------------------- |
| Web | `boltmcp_web` | `boltmcp_core` | Read/write |
| REST API | `boltmcp_rest_api` | `boltmcp_core` | Read/write |
| MCP Servers | `boltmcp_mcp_server` | `boltmcp_core` | Read-only |
| Migrations | `boltmcp_migrate_core` | `boltmcp_core` | Owner with full privileges |
| Keycloak | `boltmcp_keycloak` | `boltmcp_keycloak` | Owner with full privileges |
| Vault | `boltmcp_vault` | `boltmcp_vault` | Owner with full privileges |
# Cluster Preparation (/docs/cluster-preparation)
BoltMCP's Helm chart and container images are both hosted in a private Google Artifact Registry. You should have received a `key.json` service account key file for access — the same key authenticates both Helm (to pull the chart) and your Kubernetes cluster (to pull images at runtime).
Before installing the chart you also need to pre-create three Kubernetes Secrets that BoltMCP reads at runtime (database passwords, OIDC client secrets, and auth tokens). The chart does not generate these — you control how they are populated.
## Locate BoltMCP key file [#locate-boltmcp-key-file]
Export the path to your key file as the shell variable `HELM_REGISTRY_CONFIG`:
```bash
export HELM_REGISTRY_CONFIG="$PWD/keys/boltmcp-key.json"
```
This variable is used explicitly by `kubectl` below, and implicitly by every `helm` command on the following pages.
## Create Namespace [#create-namespace]
This namespace is where BoltMCP will be installed:
```bash
kubectl create namespace boltmcp
```
## StorageClass [#storageclass]
BoltMCP's bundled PostgreSQL stores its data on a PersistentVolumeClaim (PVC). The cluster fulfils that claim by dynamically provisioning a volume through a StorageClass, so it must have an appropriate StorageClass available before you install the chart — otherwise the database PVC stays `Pending` and the install stalls.
List what the cluster already provides:
```bash
kubectl get storageclass
```
Most managed clusters ship a suitable class out of the box: **GKE** and **AKS** both come with a usable default, so this command will list one and there is nothing more to do here.
If no appropriate StorageClass exists, create one. The common case is **EKS Auto Mode**: a fresh cluster comes with only a non-default `gp2` class backed by the deprecated in-tree provisioner, so add a CSI-backed `gp3` class instead:
```yaml title="config/storageclass-gp3.yaml" extract="true"
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
name: gp3
provisioner: ebs.csi.eks.amazonaws.com
volumeBindingMode: WaitForFirstConsumer
allowVolumeExpansion: true
parameters:
type: gp3
```
```bash
kubectl apply -f ./config/storageclass-gp3.yaml
```
The `ebs.csi.eks.amazonaws.com` provisioner is the one built into **EKS Auto Mode**. On a classic managed-node-group EKS cluster, install the EBS CSI driver add-on instead and use its provisioner, `ebs.csi.aws.com`.
## Image Pull Secret [#image-pull-secret]
Create the pull secret in the `boltmcp` namespace so the cluster can pull images at runtime:
```bash
kubectl create secret docker-registry \
boltmcp-pull-secret \
-n boltmcp \
--from-file=.dockerconfigjson=$HELM_REGISTRY_CONFIG
```
The chart references this secret by name via `global.imagePullSecrets`, default value `boltmcp-pull-secret`.
## Application Secrets [#application-secrets]
BoltMCP reads passwords and tokens from three user-managed Kubernetes Secrets. The chart never creates them. Make sure all three Secrets exist with every required key populated before moving on to the deployment step.
The Secret names are derived from the Helm release name as:
* `-database`
* `-oidc`
* `-auth`
Assuming the default release name `boltmcp`, the secret names and their required keys are as follows:
### `boltmcp-database` [#boltmcp-database]
| Key | Used for |
| ----------------------- | ----------------------------------- |
| `superuser-password` | PostgreSQL superuser password |
| `migrate-core-password` | DB password for the migration role |
| `web-password` | DB password for the BoltMCP web app |
| `rest-api-password` | DB password for the REST API |
| `mcp-server-password` | DB password for the MCP servers |
| `keycloak-password` | DB password for Keycloak |
| `vault-password` | DB password for Vault's storage |
### `boltmcp-oidc` [#boltmcp-oidc]
| Key | Used for |
| ---------------------------------------- | --------------------------------------------------------- |
| `web-client-secret` | OIDC client secret for the web app |
| `mcp-server-client-secret` | OIDC client secret for the MCP server |
| `rest-api-resource-server-client-secret` | OIDC client secret used by the REST API to verify tokens. |
Other OIDC clients created by BoltMCP are either public or have their passwords automatically set by Keycloak.
### `boltmcp-auth` [#boltmcp-auth]
| Key | Used for |
| -------------------------------- | ------------------------------------------------------------------------- |
| `web-auth-secret` | Session signing key for the web app (≥ 32 chars) |
| `keycloak-admin-password` | Master-realm Keycloak operator password (break-glass admin-console login) |
| `boltmcp-admin-password` | Password for the first user in the BoltMCP Keycloak realm |
| `mcp-inspector-proxy-auth-token` | Proxy auth token |
### Create secrets manually [#create-secrets-manually]
Generate random values inline and create all three Secrets in one shot. This is the fastest path for evaluation installs and any environment where you don't already have a secrets manager. Update `RELEASE` below if you plan to install the chart with a custom release name.
```bash title="create-secrets.sh" extract="true" bucket="allow"
RELEASE=boltmcp
rand() { openssl rand -base64 48 | tr -d '\n=+/' | cut -c1-32; }
kubectl create secret generic ${RELEASE}-database -n boltmcp \
--from-literal=superuser-password="$(rand)" \
--from-literal=migrate-core-password="$(rand)" \
--from-literal=web-password="$(rand)" \
--from-literal=rest-api-password="$(rand)" \
--from-literal=mcp-server-password="$(rand)" \
--from-literal=keycloak-password="$(rand)" \
--from-literal=vault-password="$(rand)"
kubectl create secret generic ${RELEASE}-oidc -n boltmcp \
--from-literal=web-client-secret="$(rand)" \
--from-literal=mcp-server-client-secret="$(rand)" \
--from-literal=rest-api-resource-server-client-secret="$(rand)"
kubectl create secret generic ${RELEASE}-auth -n boltmcp \
--from-literal=web-auth-secret="$(rand)" \
--from-literal=keycloak-admin-password="$(rand)" \
--from-literal=boltmcp-admin-password="$(rand)" \
--from-literal=mcp-inspector-proxy-auth-token="$(rand)"
```
To retrieve a value later (assuming release name `boltmcp`):
```bash
kubectl get secret boltmcp-auth -n boltmcp \
-o jsonpath='{.data.boltmcp-admin-password}' | base64 -d; echo
```
### Alternatives for populating secrets [#alternatives-for-populating-secrets]
The manual approach above is the simplest path, but if you already run a secrets-management workflow you can populate the same three Secrets from it instead. With the *External Secrets Operator* you sync the values from HashiCorp Vault or a cloud secrets manager (AWS Secrets Manager, GCP Secret Manager, Azure Key Vault, 1Password) into Kubernetes Secrets; with *Sealed Secrets* or *SOPS* you keep the encrypted source material in Git and let an in-cluster controller (or your GitOps tool) materialise the plain Secrets.
# Cluster Provisioning (/docs/cluster-provisioning)
## Cluster Requirements [#cluster-requirements]
BoltMCP runs on any conformant Kubernetes cluster (v1.28+). Pick the tier that matches your deployment:
| Tier | Nodes | Per-node spec | Suitable for |
| ---------------------------- | ----- | ------------------ | -------------------------------------------------------- |
| **Evaluation** | 1 | 2 vCPU, 8 GiB RAM | Trying BoltMCP, demos, throwaway dev clusters |
| **Production (minimum)** | 3 | 2 vCPU, 8 GiB RAM | Internal use, small teams, single-region |
| **Production (recommended)** | 3 | 4 vCPU, 16 GiB RAM | Customer-facing, room to scale replicas and run upgrades |
## Create a Cluster [#create-a-cluster]
If you already have a suitable cluster, skip this page.
The cluster-creation commands below provision the **evaluation** tier (a single small node) and are intended for demos and exploration only. For production deployments, increase the node count and machine type to match an appropriate tier from the table above.
Make sure you've logged into the relevant CLI (`gcloud auth login` / `aws configure` / `az login`) before running these commands.
Create the cluster (5-10 minutes):
```bash
gcloud container clusters create boltmcp-cluster \
--zone europe-west2-a \
--num-nodes 1 \
--machine-type e2-standard-2
```
Connect kubectl:
```bash
gcloud container clusters get-credentials boltmcp-cluster \
--zone europe-west2-a
```
`kubectl` needs the `gke-gcloud-auth-plugin` to authenticate against GKE. Install it as a `gcloud` component:
```bash
gcloud components install gke-gcloud-auth-plugin
```
If you installed `gcloud` via Homebrew on macOS, the plugin binary lands in `/opt/homebrew/share/google-cloud-sdk/bin`, which is not on `PATH` by default. Add it to your shell config (`~/.zshrc` or `~/.bashrc`):
```bash
export PATH="/opt/homebrew/share/google-cloud-sdk/bin:$PATH"
```
Create the cluster (10-15 minutes):
```bash
eksctl create cluster \
--name boltmcp-cluster \
--region eu-west-2 \
--enable-auto-mode
```
Connect kubectl:
```bash
aws eks update-kubeconfig \
--name boltmcp-cluster \
--region eu-west-2
```
kubectl should get connected automatically if the cluster created successfully, but the command above is required after fixing a partial or failed run.
If this is the first time you're creating an AKS cluster in this subscription, register the `Microsoft.ContainerService` resource provider once (otherwise `az aks create` fails with `MissingSubscriptionRegistration`):
```bash
az provider register --namespace Microsoft.ContainerService
```
Registration runs in the background. Wait for it to report `Registered` before continuing:
```bash
az provider show -n Microsoft.ContainerService --query registrationState -o tsv
```
Create the resource group and cluster (5-10 minutes):
```bash
az group create --name boltmcp-rg --location westeurope
az aks create \
--resource-group boltmcp-rg \
--name boltmcp-cluster \
--node-count 1 \
--node-vm-size Standard_D2s_v3 \
--generate-ssh-keys
```
Connect kubectl:
```bash
az aks get-credentials \
--resource-group boltmcp-rg \
--name boltmcp-cluster
```
## Verify Cluster Access [#verify-cluster-access]
Confirm `kubectl` is pointing at the cluster you intend to install into:
```bash
kubectl config current-context
```
If it's not the right one, list the available contexts and switch:
```bash
kubectl config get-contexts
```
```bash
kubectl config use-context
```
Finally, verify the cluster is ready:
```bash
kubectl get nodes
```
All nodes should show `Ready` status.
On **EKS Auto Mode** there is no static node pool — nodes are provisioned on demand only when there are pods to schedule, so a freshly-created idle cluster has **zero nodes**. Don't use `kubectl get nodes` to verify access here; it returns `No resources found`, which looks like a failure but is expected. Confirm the API is reachable instead:
```bash
kubectl get ns
```
You should see the default namespaces (`default`, `kube-system`, …).
The first nodes appear a few minutes after you deploy a workload (e.g. during `helm install`), once Auto Mode's `general-purpose` NodePool reconciles and launches capacity. At that point `kubectl get nodes` will list `Ready` nodes.
```bash
kubectl get nodes
```
All nodes should show `Ready` status.
# Configuration Reference (/docs/configuration-reference)
{/* AUTO-GENERATED from charts/boltmcp/values.schema.json — do not edit. Run: pnpm --filter boltmcp-docs-install generate */}
## database [#database]
| Parameter | Type | Default | Description |
| ------------------------------------- | ------ | ------------------------ | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `database.affinity` | object | `{}` | Affinity rules. Pass-through, not strictly validated. |
| `database.coreSchema` | string | `"boltmcp_core"` | Schema holding the shared application tables. The ltree extension is namespaced here and the schema is owned by the migrate-core user; application roles have no dependency on public. |
| `database.host` | string | `""` | External database host (when internal.enabled=false) |
| `database.image.pullPolicy` | string | `"IfNotPresent"` | Image pull policy |
| `database.image.repository` | string | `"postgres"` | PostgreSQL image repository |
| `database.image.tag` | string | `"15-alpine"` | PostgreSQL image tag |
| `database.internal.enabled` | bool | `true` | Deploy internal PostgreSQL. Set false for external DB |
| `database.name` | string | `"boltmcp"` | Database name |
| `database.nodeSelector` | object | `{}` | Node selector |
| `database.persistence.accessModes` | array | `["ReadWriteOnce"]` | PVC access modes |
| `database.persistence.enabled` | bool | `true` | Enable persistent storage |
| `database.persistence.size` | string | `"10Gi"` | PVC size |
| `database.persistence.storageClass` | string | `""` | Storage class (falls back to global.storageClass) |
| `database.port` | int | `5432` | Database port |
| `database.resources` | object | `{}` | CPU/memory resource limits and requests. Pass-through, not strictly validated. |
| `database.superuser.username` | string | `"postgres"` | Superuser username (password lives in the database Secret as superuser-password) |
| `database.tolerations` | array | `[]` | Tolerations |
| `database.users.keycloak.schema` | string | `"boltmcp_keycloak"` | Keycloak schema name |
| `database.users.keycloak.username` | string | `"boltmcp_keycloak"` | Keycloak DB user (password: keycloak-password) |
| `database.users.mcpServer.username` | string | `"boltmcp_mcp_server"` | MCP Server DB user, read-only on the core schema (password: mcp-server-password) |
| `database.users.migrateCore.username` | string | `"boltmcp_migrate_core"` | Migration DB user — owns the core schema and is the only role that runs DDL/migrations (password: migrate-core-password) |
| `database.users.restApi.username` | string | `"boltmcp_rest_api"` | REST API DB user, read/write on the core schema (password: rest-api-password) |
| `database.users.vault.schema` | string | `"boltmcp_vault"` | Vault storage schema name |
| `database.users.vault.username` | string | `"boltmcp_vault"` | Vault storage DB user; owns the pre-created vault\_kv\_store table (password: vault-password) |
| `database.users.web.username` | string | `"boltmcp_web"` | Web app DB user, read/write on the core schema (password lives in the database Secret as web-password) |
## global [#global]
| Parameter | Type | Default | Description |
| ------------------------- | ------ | --------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `global.domain` | string | `""` | Apex domain that hosts BoltMCP. Used to derive per-service hostnames as `web.`, `auth.`, `server.`, `inspector.`. Required unless every per-service URL/hostname override is set explicitly. |
| `global.hostAliases` | array | `[]` | hostAliases injected into every BoltMCP pod. Use when the public BoltMCP hostnames don't resolve from inside the cluster (split-horizon DNS, local installs) so in-cluster OIDC discovery can reach Keycloak. |
| `global.imagePullSecrets` | array | `[{"name":"boltmcp-pull-secret"}]` | Image pull secrets for private registries. Matches the Secret name created in cluster prep. Set to \[] if your images come from a registry that doesn't require auth. |
| `global.imageRegistry` | string | `"europe-west2-docker.pkg.dev/boltmcp-platform/boltmcp-alpha/images"` | Default image registry for BoltMCP images |
| `global.storageClass` | string | `""` | Storage class for all BoltMCP persistent volumes. Empty uses the cluster default (fine on GKE/AKS). On EKS set it to a class you created. Overridable per-volume by database.persistence.storageClass. |
| `global.tls.enabled` | bool | `true` | Whether the public BoltMCP URLs are served over HTTPS. Declares the scheme the workloads advertise and expect — it does not provision TLS itself; certificates and termination remain the job of your ingress / load balancer (with the chart-managed Ingress, of cert-manager via ingress.annotations). Single source of the scheme for every derived URL (web/server/inspector/Keycloak base URLs, the OIDC issuer) and for whether the chart-managed Ingress carries a tls section. Set false only for evaluation installs served over plain HTTP. Explicit overrides such as web.baseUrl and keycloak.baseUrl always win, scheme included. |
## ingress [#ingress]
Optional chart-managed Ingress. Disabled by default: bring your own ingress / gateway / load balancer instead (see examples/ingress in the chart for a reference manifest).
| Parameter | Type | Default | Description |
| ------------------------ | ------ | --------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `ingress.annotations` | object | `{}` | Annotations for the Ingress. Controller- and cert-manager-specific settings go here: on ingress-nginx you almost always want cert-manager.io/cluster-issuer for TLS certificates, and on plain-HTTP installs nginx.ingress.kubernetes.io/ssl-redirect: "false". The chart applies two ingress-nginx defaults you do not need to repeat: proxy-body-size: "10m", which caps API spec uploads (ingress-nginx's own 1m default rejects them with 413), and proxy-buffer-size: "128k", required for Keycloak's large auth headers (the 4k default breaks login with "upstream sent too big header"). Setting either key here overrides the chart default; all other annotations you supply are merged alongside them. |
| `ingress.className` | string | `""` | IngressClass name (spec.ingressClassName). Empty omits the field so the cluster default class applies. |
| `ingress.enabled` | bool | `false` | Create a single Ingress routing the public BoltMCP hostnames (derived from global.domain / the per-service baseUrl overrides, so they can never drift from the URLs the services advertise) to the in-cluster services. |
| `ingress.tls.secretName` | string | `"boltmcp-tls"` | Name of the TLS Secret covering all BoltMCP hostnames (created by cert-manager when the cluster-issuer annotation is set). The Ingress carries a tls section only when global.tls.enabled is true. |
## keycloak [#keycloak]
| Parameter | Type | Default | Description |
| ----------------------------- | ------ | ----------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `keycloak.affinity` | object | `{}` | Affinity rules. Pass-through, not strictly validated. |
| `keycloak.baseUrl` | string | `""` | Base URL. If empty, defaults to `https://auth.` (http when global.tls.enabled is false). The OIDC issuer is `/realms/boltmcp` |
| `keycloak.enabled` | bool | `true` | Enable Keycloak deployment |
| `keycloak.image.pullPolicy` | string | `"IfNotPresent"` | Image pull policy |
| `keycloak.image.repository` | string | `"quay.io/keycloak/keycloak"` | Keycloak image |
| `keycloak.image.tag` | string | `"26.7.0-1"` | Keycloak image tag |
| `keycloak.nodeSelector` | object | `{}` | Node selector |
| `keycloak.production.enabled` | bool | `true` | Run in production mode (start vs start-dev). Production mode works over plain HTTP behind a proxy (the chart sets the proxy-headers env); set false only for evaluation installs with no ingress / reverse proxy in front of Keycloak at all |
| `keycloak.resources` | object | `{}` | CPU/memory resource limits and requests. Pass-through, not strictly validated. |
| `keycloak.service.healthPort` | int | `9000` | Health check port |
| `keycloak.service.port` | int | `8080` | Service port |
| `keycloak.service.type` | string | `"ClusterIP"` | Service type |
| `keycloak.tolerations` | array | `[]` | Tolerations |
## mcpInspector [#mcpinspector]
| Parameter | Type | Default | Description |
| -------------------------------- | ------ | ------------------------------------------ | ----------------------------------------------------------------------------------------------------------- |
| `mcpInspector.affinity` | object | `{}` | Affinity rules. Pass-through, not strictly validated. |
| `mcpInspector.baseUrl` | string | `""` | Base URL. If empty, defaults to `https://inspector.` (http when global.tls.enabled is false) |
| `mcpInspector.enabled` | bool | `true` | Enable MCP Inspector |
| `mcpInspector.image.pullPolicy` | string | `"IfNotPresent"` | Image pull policy |
| `mcpInspector.image.repository` | string | `"ghcr.io/modelcontextprotocol/inspector"` | Image repository |
| `mcpInspector.image.tag` | string | `"0.21.1"` | Image tag |
| `mcpInspector.nodeSelector` | object | `{}` | Node selector |
| `mcpInspector.resources` | object | `{}` | CPU/memory resource limits and requests. Pass-through, not strictly validated. |
| `mcpInspector.service.proxyPort` | int | `6277` | Proxy port |
| `mcpInspector.service.type` | string | `"ClusterIP"` | Service type |
| `mcpInspector.service.webPort` | int | `6274` | Web UI port |
| `mcpInspector.tolerations` | array | `[]` | Tolerations |
## mcpServer [#mcpserver]
| Parameter | Type | Default | Description |
| ---------------------------- | ------ | ---------------- | -------------------------------------------------------------------------------------------------------- |
| `mcpServer.affinity` | object | `{}` | Affinity rules. Pass-through, not strictly validated. |
| `mcpServer.baseUrl` | string | `""` | Base URL. If empty, defaults to `https://server.` (http when global.tls.enabled is false) |
| `mcpServer.extraEnv` | array | `[]` | Additional environment variables |
| `mcpServer.image.pullPolicy` | string | `"IfNotPresent"` | Image pull policy |
| `mcpServer.image.repository` | string | `""` | Image repository (defaults to global.imageRegistry/boltmcp-mcp-server) |
| `mcpServer.image.tag` | string | `""` | Image tag (defaults to .Chart.AppVersion if empty) |
| `mcpServer.nodeSelector` | object | `{}` | Node selector |
| `mcpServer.replicaCount` | int | `1` | Number of replicas |
| `mcpServer.resources` | object | `{}` | CPU/memory resource limits and requests. Pass-through, not strictly validated. |
| `mcpServer.service.port` | int | `3001` | Service port |
| `mcpServer.service.type` | string | `"ClusterIP"` | Service type |
| `mcpServer.tolerations` | array | `[]` | Tolerations |
## migrations [#migrations]
| Parameter | Type | Default | Description |
| ---------------------------------------- | ------ | ------- | ----------------------------------------------------------------------------------- |
| `migrations.backoffLimit` | int | `3` | Job retry limit |
| `migrations.coreMigrateImage.repository` | string | `""` | Core schema migration image (defaults to global.imageRegistry/boltmcp-migrate-core) |
| `migrations.coreMigrateImage.tag` | string | `""` | Tag (inherits from web.image.tag) |
| `migrations.ttlSecondsAfterFinished` | int | `300` | Cleanup delay after job completion |
## oidc [#oidc]
| Parameter | Type | Default | Description |
| ------------------------------------- | ------ | ---------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| `oidc.adminUser.email` | string | `""` | Required. Email for the first user in the boltmcp Keycloak realm. Provisioned on first install via --import-realm (username boltmcp\_admin, firstName Admin, emailVerified true, granted realm-management/realm-admin). Password lives in the auth Secret as boltmcp-admin-password. |
| `oidc.mcpServer.clientId` | string | `"boltmcp-mcp-server"` | MCP Server client ID (secret: mcp-server-client-secret) |
| `oidc.mcpServerToRestApi.clientId` | string | `"boltmcp-mcp-server-to-rest-api"` | Client used by the MCP server to obtain user tokens for the REST API (secret: mcp-server-to-rest-api-client-secret). PKCE-aware, audience-mapped to REST\_API\_BASE\_URL. |
| `oidc.provider` | string | `"keycloak"` | OIDC provider type |
| `oidc.restApiResourceServer.clientId` | string | `"boltmcp-rest-api"` | Client used by the REST API to authenticate to Keycloak's introspection endpoint (secret: rest-api-resource-server-client-secret). |
| `oidc.web.clientId` | string | `"boltmcp-web"` | Platform client ID (secret lives in the OIDC Secret as web-client-secret) |
## restApi [#restapi]
| Parameter | Type | Default | Description |
| -------------------------- | ------ | ---------------- | ------------------------------------------------------------------------------ |
| `restApi.affinity` | object | `{}` | Affinity rules. Pass-through, not strictly validated. |
| `restApi.extraEnv` | array | `[]` | Additional environment variables |
| `restApi.image.pullPolicy` | string | `"IfNotPresent"` | Image pull policy |
| `restApi.image.repository` | string | `""` | Image repository (defaults to global.imageRegistry/boltmcp-rest-api) |
| `restApi.image.tag` | string | `""` | Image tag (defaults to .Chart.AppVersion if empty) |
| `restApi.nodeSelector` | object | `{}` | Node selector |
| `restApi.replicaCount` | int | `1` | Number of replicas |
| `restApi.resources` | object | `{}` | CPU/memory resource limits and requests. Pass-through, not strictly validated. |
| `restApi.service.port` | int | `3003` | Service port |
| `restApi.service.type` | string | `"ClusterIP"` | Service type |
| `restApi.tolerations` | array | `[]` | Tolerations |
## secrets [#secrets]
Names of user-managed Kubernetes Secrets the chart reads. The chart never creates these.
| Parameter | Type | Default | Description |
| ----------------------- | ------ | ------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `secrets.auth.name` | string | `""` | Name of the user-managed Secret holding auth tokens, the master-realm Keycloak admin password (keycloak-admin-password), the BoltMCP-realm first-user password (boltmcp-admin-password), and the MCP Inspector token. If empty, defaults to `-auth`. |
| `secrets.database.name` | string | `""` | Name of the user-managed Secret holding database passwords. If empty, defaults to `-database`. |
| `secrets.oidc.name` | string | `""` | Name of the user-managed Secret holding OIDC client secrets. If empty, defaults to `-oidc`. |
## vault [#vault]
| Parameter | Type | Default | Description |
| --------------------------------------- | ------ | ------------------------------- | -------------------------------------------------------------------------------------------------------------------- |
| `vault.affinity` | object | `{}` | Affinity rules. Pass-through, not strictly validated. |
| `vault.enabled` | bool | `true` | Deploy the bundled Vault |
| `vault.image.pullPolicy` | string | `"IfNotPresent"` | Image pull policy |
| `vault.image.repository` | string | `"hashicorp/vault"` | Image repository |
| `vault.image.tag` | string | `"1.21.4"` | Image tag |
| `vault.kubernetesAuth.audience` | string | `"vault"` | Projected-token audience (must match the Vault role) |
| `vault.kubernetesAuth.authMountPath` | string | `"kubernetes"` | Mount path of Vault's Kubernetes auth method |
| `vault.kubernetesAuth.enabled` | bool | `true` | Wire boltmcp-rest-api to Vault via Kubernetes auth (ServiceAccount + projected token + env) |
| `vault.kubernetesAuth.kvMount` | string | `"secret"` | KV v2 secrets engine mount |
| `vault.kubernetesAuth.kvPathPrefix` | string | `"boltmcp/server-env-api-cred"` | Logical prefix the REST API stores secrets under |
| `vault.kubernetesAuth.mcpServer.policy` | string | `"boltmcp-mcp-server"` | Vault policy for the MCP Server's read-only access to the secret prefix |
| `vault.kubernetesAuth.mcpServer.role` | string | `"boltmcp-mcp-server"` | Vault role bound to the MCP Server ServiceAccount (the reader; shares the auth mount, KV mount/prefix, and audience) |
| `vault.kubernetesAuth.policy` | string | `"boltmcp-rest-api"` | Vault policy for the secret prefix |
| `vault.kubernetesAuth.role` | string | `"boltmcp-rest-api"` | Vault role bound to the REST API ServiceAccount |
| `vault.nodeSelector` | object | `{}` | Node selector |
| `vault.resources` | object | `{}` | CPU/memory resource limits and requests. Pass-through, not strictly validated. |
| `vault.seal.config` | object | `{}` | Key/value pairs rendered into the seal stanza (e.g. region, kms\_key\_id) |
| `vault.seal.extraEnv` | array | `[]` | Extra Vault container env for the seal (e.g. KMS credentials) |
| `vault.seal.type` | string | `""` | Seal type — "" is Shamir/manual unseal; awskms/gcpckms/azurekeyvault/transit enable auto-unseal |
| `vault.service.port` | int | `8200` | Service port |
| `vault.service.type` | string | `"ClusterIP"` | Service type |
| `vault.tolerations` | array | `[]` | Tolerations |
## web [#web]
| Parameter | Type | Default | Description |
| ---------------------- | ------ | ---------------- | ----------------------------------------------------------------------------------------------------- |
| `web.affinity` | object | `{}` | Affinity rules. Pass-through, not strictly validated. |
| `web.baseUrl` | string | `""` | Base URL. If empty, defaults to `https://web.` (http when global.tls.enabled is false) |
| `web.extraEnv` | array | `[]` | Additional environment variables |
| `web.image.pullPolicy` | string | `"IfNotPresent"` | Image pull policy |
| `web.image.repository` | string | `""` | Image repository (defaults to global.imageRegistry/boltmcp-web) |
| `web.image.tag` | string | `""` | Image tag (defaults to .Chart.AppVersion if empty) |
| `web.nodeSelector` | object | `{}` | Node selector |
| `web.replicaCount` | int | `1` | Number of replicas |
| `web.resources` | object | `{}` | CPU/memory resource limits and requests. Pass-through, not strictly validated. |
| `web.service.port` | int | `3000` | Service port |
| `web.service.type` | string | `"ClusterIP"` | Service type |
| `web.tolerations` | array | `[]` | Tolerations |
## Other values [#other-values]
| Parameter | Type | Default | Description |
| -------------------- | ------ | ------- | ------------------------------ |
| `fullnameOverride` | string | `""` | Override the full release name |
| `nameOverride` | string | `""` | Override the release name |
| `podAnnotations` | object | `{}` | Pod annotations |
| `podSecurityContext` | object | `{}` | Pod security context |
# Deployment (/docs/deployment)
## Values File Templates [#values-file-templates]
Create a values yaml file and update the values accordingly.
### Remote HTTPS [#remote-https]
Use this template if your deployment will sit behind a real domain with TLS enabled.
```yaml title="config/values-prod.yaml" extract="true"
global:
# Apex domain for BoltMCP. The chart derives per-service hostnames as
# web., auth., server., inspector..
# Recommended: choose the `boltmcp` subdomain of your company's domain.
domain: "boltmcp.example.com"
# Advertise https:// URLs everywhere.
tls:
enabled: true
# Storage class for the bundled database's persistent volume. Leave unset
# to use the cluster default (GKE/AKS ship one). On EKS, set this to the
# class you created in Cluster Prep.
# storageClass: gp3
oidc:
adminUser:
# Email for the first admin user (must be a real email address).
email: "admin@example.com"
```
Note that setting `global.tls.enabled: true` does not provision TLS itself.
### Local HTTP [#local-http]
Use this template if you're deploying locally e.g. Docker Desktop Kubernetes.
```yaml title="config/values-local.yaml" extract="true"
global:
# Domain your HTTP ingress serves, e.g. an /etc/hosts entry pointing
# web., auth., server. and inspector.
# at your ingress controller.
domain: "boltmcp.local"
# Advertise http:// URLs everywhere.
tls:
enabled: false
oidc:
adminUser:
# Email for the first admin user (must be a real email address).
email: "admin@example.com"
```
## Pin a Chart Version [#pin-a-chart-version]
Set a version once and reference it in subsequent commands. Use the latest release version you've been provided:
```bash
export BOLTMCP_VERSION=0.3.8
```
## Install the Chart [#install-the-chart]
Before installing BoltMCP, make sure you've completed [Cluster Preparation](./cluster-preparation)
If it isn't already, set the shell variable `HELM_REGISTRY_CONFIG` to point to your BoltMCP key file. This is used implicitly to authorize Helm commands.
```bash
export HELM_REGISTRY_CONFIG="$PWD/keys/boltmcp-key.json"
```
Deploy all services into the `boltmcp` namespace:
```bash
helm install boltmcp \
oci://europe-west2-docker.pkg.dev/boltmcp-platform/boltmcp-alpha/charts/boltmcp \
--version ${BOLTMCP_VERSION} \
-n boltmcp \
-f ./config/values-prod.yaml \
--timeout 15m
```
`--timeout 15m` is required on EKS Auto Mode. On a brand-new EKS Auto
Mode cluster the general-purpose NodePool starts with zero nodes, and bringing
up the first node can take longer than Helm's default hook timeout. For GKE/AKS deployments the default `--timeout 5m`
is generally sufficient.
## Verify the Deployment [#verify-the-deployment]
Watch pods until all show `Running` / `Completed`:
```bash
kubectl get pods -n boltmcp -w
```
Expected output:
```
NAME READY STATUS RESTARTS AGE
boltmcp-database-0 1/1 Running 0 2m52s
boltmcp-keycloak-xxxxxxxxxx-xxxxx 1/1 Running 0 2m52s
boltmcp-mcp-inspector-xxxxxxxxxx-xxxxx 1/1 Running 0 2m52s
boltmcp-mcp-server-xxxxxxxxxx-xxxxx 1/1 Running 0 2m52s
boltmcp-rest-api-xxxxxxxxxx-xxxxx 1/1 Running 0 2m52s
boltmcp-web-xxxxxxxxxx-xxxxx 1/1 Running 0 2m52s
boltmcp-migrate-core-xxxxx 0/1 Completed 0 2m14s
boltmcp-seed-xxxxx 0/1 Completed 0 2m14s
```
If a pod is not starting, check its logs and events:
```bash
kubectl logs -n boltmcp
kubectl describe pod -n boltmcp
```
### Retrying a failed install [#retrying-a-failed-install]
If the install fails, the release is left in `failed` state and another plain `helm install` will error with *"cannot re-use a name that is still in use"*. Check the failed job's pod logs to fix the underlying issue before deleting the job so a fresh one is created, then retry with `helm upgrade --install`:
```bash
kubectl delete job -n boltmcp
helm upgrade --install boltmcp \
oci://europe-west2-docker.pkg.dev/boltmcp-platform/boltmcp-alpha/charts/boltmcp \
--version ${BOLTMCP_VERSION} \
-n boltmcp \
-f ./config/values-prod.yaml \
--timeout 15m
```
# External Secrets Operator (/docs/external-secrets-operator)
This page expands on the [Alternatives](/docs/cluster-preparation#alternatives-for-populating-secrets) noted in Cluster Prep. It shows how to populate BoltMCP's three application Secrets (`boltmcp-database`, `boltmcp-oidc`, `boltmcp-auth`) from an external secrets manager instead of creating them manually with `kubectl`.
If your organisation already stores secrets in Vault (or AWS Secrets Manager, GCP Secret Manager, Azure Key Vault, 1Password), the [External Secrets Operator](https://external-secrets.io/) syncs them into Kubernetes `Secret` resources for the chart to read.
**One-time cluster setup** — install ESO and create a `ClusterSecretStore` pointing at your Vault:
```yaml title="cluster-secret-store-vault.yaml"
apiVersion: external-secrets.io/v1beta1
kind: ClusterSecretStore
metadata:
name: vault-backend
spec:
provider:
vault:
server: "https://vault.example.com:8200"
path: "secret"
version: "v2"
auth:
kubernetes:
mountPath: "kubernetes"
role: "boltmcp-eso"
serviceAccountRef:
name: "external-secrets"
namespace: "external-secrets"
```
```bash
kubectl apply -f cluster-secret-store-vault.yaml
```
**Per-release setup** — populate the three Vault entries (`secret/boltmcp/database`, `secret/boltmcp/oidc`, `secret/boltmcp/auth`) with the key names listed in [Cluster Prep](/docs/cluster-preparation#application-secrets) (`secret/boltmcp/oidc` carries all five client secrets), then apply three `ExternalSecret` resources. Here's the database one as a worked example:
```yaml title="external-secret-database.yaml"
apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
name: boltmcp-database
namespace: boltmcp
spec:
refreshInterval: 1h
secretStoreRef:
name: vault-backend
kind: ClusterSecretStore
target:
name: boltmcp-database
creationPolicy: Owner
data:
- secretKey: superuser-password
remoteRef: { key: boltmcp/database, property: superuser-password }
- secretKey: migrate-core-password
remoteRef: { key: boltmcp/database, property: migrate-core-password }
- secretKey: web-password
remoteRef: { key: boltmcp/database, property: web-password }
- secretKey: rest-api-password
remoteRef: { key: boltmcp/database, property: rest-api-password }
- secretKey: mcp-server-password
remoteRef: { key: boltmcp/database, property: mcp-server-password }
- secretKey: keycloak-password
remoteRef: { key: boltmcp/database, property: keycloak-password }
- secretKey: vault-password
remoteRef: { key: boltmcp/database, property: vault-password }
```
Full manifests for all three Secrets (database, OIDC, auth) live at `charts/boltmcp/examples/secrets/external-secrets-vault.yaml`. Wait for `kubectl get externalsecret -n boltmcp` to show `STATUS=SecretSynced` for all three before installing the chart.
For AWS Secrets Manager, GCP Secret Manager, or Azure Key Vault, swap only the `provider` block in the `ClusterSecretStore` — the `ExternalSecret` manifests stay the same.
# Installation (/docs)
## Install with Claude [#install-with-claude]
This documentation site is also bundled as an Agent Skill, which Claude Code can use to walk you through the installation process. Before carrying out the steps below, you might prefer to [install the prerequisites manually](/docs/prerequisites). If not, Claude will walk you through that too.
[Install Claude Code](https://code.claude.com/docs/) and log in
```bash
claude auth login
```
Clone boltmcp
```bash
git clone https://github.com/boltmcp/boltmcp.git
```
Move your BoltMCP access key to the `keys` directory
```bash
mv ~/Downloads/my-key.json boltmcp/keys/boltmcp-key.json
```
Replace `~/Downloads/my-key.json` with the path to your access key file
Open Claude Code from inside the `boltmcp` directory
```bash
cd boltmcp && claude
```
Invoke the Skill and Claude will walk you through the installation process
```
/install-boltmcp Install BoltMCP on a new cluster
```
Replace the text after "/install-boltmcp " with any instruction related to installing, updating or uninstalling BoltMCP.
***
## Install manually [#install-manually]
If you'd prefer to do the installation manually, continue reading the following pages.
# Ingress & TLS (/docs/ingress-tls)
By default the BoltMCP chart doesn't manage cluster ingress. It expects you to provision your own ingress / gateway / load balancer that terminates TLS and routes the BoltMCP hostnames to the in-cluster services.
If your platform team handles ingress, hand them the hostnames table below and skip the rest of this page. The walkthrough that follows is a reference setup using NGINX Ingress Controller, cert-manager, Let's Encrypt, and the chart-managed Ingress. Adapt it for your environment, or replace it entirely with whatever ingress your cluster already runs.
## What BoltMCP needs from your ingress [#what-boltmcp-needs-from-your-ingress]
For every value of `global.domain` you set in `values-prod.yaml`, the chart configures the workloads to expect these public hostnames, terminating TLS, routed to these in-cluster Services:
| Public hostname | Service | Port |
| -------------------- | ----------------------- | ---- |
| `web.` | `boltmcp-web` | 3000 |
| `auth.` | `boltmcp-keycloak` | 8080 |
| `server.` | `boltmcp-mcp-server` | 3001 |
| `inspector.` | `boltmcp-mcp-inspector` | 6274 |
If you've overridden any of `web.baseUrl`, `mcpServer.baseUrl`, `mcpInspector.baseUrl`, or `keycloak.baseUrl` in your values, mirror those changes in your ingress (the chart-managed Ingress picks them up automatically).
Two NGINX-specific annotations matter:
* `nginx.ingress.kubernetes.io/proxy-buffer-size: "128k"` - required for Keycloak's large authentication headers. NGINX's own default is 4k, which truncates them and breaks every login with `upstream sent too big header`.
* `nginx.ingress.kubernetes.io/proxy-body-size: "10m"` - caps API spec uploads. NGINX's own default is 1m, which rejects larger specs with `413 Request Entity Too Large`.
Both are BoltMCP requirements rather than site preferences, so **the chart-managed Ingress applies them for you** - don't repeat them in your values file. You only set them explicitly to choose a different limit, or when you're authoring your own Ingress manifest. To change one, set the same key in `ingress.annotations`: your value replaces the chart default, and any other annotations you supply are merged alongside it.
The equivalent settings exist on most ingress implementations; check your platform's docs.
If you're authoring your own Ingress manifest rather than using the chart-managed one, the chart package's `examples/ingress` directory contains a reference manifest to start from.
## Reference Ingress Setup [#reference-ingress-setup]
The rest of this page walks through one common setup: NGINX Ingress Controller, DNS-based routing, and the chart-managed Ingress tying them together, followed by cert-manager for automatic TLS in [Reference TLS Setup](#reference-tls-setup). Skip the controller and cert-manager steps if your cluster already has them.
### Install NGINX Ingress Controller [#install-nginx-ingress-controller]
```bash
helm repo add ingress-nginx https://kubernetes.github.io/ingress-nginx
helm repo update
```
```bash
helm install ingress-nginx ingress-nginx/ingress-nginx \
--namespace ingress-nginx \
--create-namespace \
--set controller.service.externalTrafficPolicy=Local
```
On EKS - and especially **EKS Auto Mode** - the integrated AWS Load Balancer Controller provisions a `LoadBalancer` Service as **`internal`** by default, which leaves the ingress unreachable from the public internet (and from Let's Encrypt's HTTP-01 validation). Set the `aws-load-balancer-scheme` annotation so the NLB is **internet-facing** from the start:
```bash
helm install ingress-nginx ingress-nginx/ingress-nginx \
--namespace ingress-nginx \
--create-namespace \
--set controller.service.externalTrafficPolicy=Local \
--set controller.service.annotations."service\.beta\.kubernetes\.io/aws-load-balancer-scheme"=internet-facing
```
The load balancer **scheme is immutable**. If you install without this annotation and add it later, the controller deletes the internal NLB and creates a fresh internet-facing one with a **different hostname** - so any DNS records you've already pointed at the old hostname must be updated, and you'll wait out their TTL while resolution converges. Setting it up front avoids that churn.
```bash
helm install ingress-nginx ingress-nginx/ingress-nginx \
--namespace ingress-nginx \
--create-namespace \
--set controller.service.externalTrafficPolicy=Local
```
```bash
helm install ingress-nginx ingress-nginx/ingress-nginx \
--namespace ingress-nginx \
--create-namespace
```
Wait for the external IP:
```bash
kubectl get svc ingress-nginx-controller -n ingress-nginx -w
```
Expected output:
```
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
ingress-nginx-controller LoadBalancer xx.x.xxx.xx x.xxx.xxx.xxx 80:32695/TCP,443:30691/TCP 2m8s
```
```
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
ingress-nginx-controller LoadBalancer xx.xxx.x.xx xx-xx.eu-west-2.elb.amazonaws.com 80:30454/TCP,443:31623/TCP 2m8s
```
```
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
ingress-nginx-controller LoadBalancer xx.x.xxx.xx x.xxx.xxx.xxx 80:32695/TCP,443:30691/TCP 2m8s
```
```
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
ingress-nginx-controller LoadBalancer xx.xx.xxx.xx xxx.xx.x.x 80:32127/TCP,443:31751/TCP 7s
```
Installing locally (e.g. Docker Desktop)? Skip to [Enable the Chart-managed Ingress](#enable-the-chart-managed-ingress).
### Reserve a Stable Address [#reserve-a-stable-address]
Give the load balancer a stable address so your DNS records stay valid across restarts. Whether you need to do anything here depends on your cloud:
```bash
REGION=$(gcloud container clusters describe \
--format="get(location)" | sed 's/-[a-z]$//')
CURRENT_IP=$(kubectl get svc ingress-nginx-controller \
-n ingress-nginx \
-o jsonpath='{.status.loadBalancer.ingress[0].ip}')
gcloud compute addresses create boltmcp-ingress-ip \
--addresses $CURRENT_IP \
--region $REGION
```
Run straight after the `EXTERNAL-IP` first appears, this can fail with `Invalid value for field 'resource.address': ... Specified IP address is already reserved.` GKE holds the address while it finishes provisioning the load balancer. Wait a minute and re-run the same command - it succeeds once provisioning settles.
Confirm the address is reserved before moving on - until it is, the IP is still ephemeral and recreating the ingress-nginx Service would change it, invalidating the DNS records you create next:
```bash
gcloud compute addresses describe boltmcp-ingress-ip \
--region $REGION \
--format='value(address,status)'
# Should print your EXTERNAL-IP and IN_USE
```
The default `ingress-nginx` install provisions a **Classic ELB**, and its DNS hostname is **already stable for the life of the Service** - it survives pod, node, and controller restarts. There is nothing to reserve: note the hostname and point DNS at it with a **CNAME** in the next step.
You only need a literal static **IP** on EKS if something external must allowlist a fixed address, or you've overridden a BoltMCP service hostname to a **zone apex** (where CNAMEs aren't permitted - see the DNS step). In that case, provision a **Network Load Balancer (NLB) with Elastic IPs** instead of the default Classic ELB: install the AWS Load Balancer Controller, then recreate the controller Service as an NLB with the annotations `service.beta.kubernetes.io/aws-load-balancer-type: external` and `service.beta.kubernetes.io/aws-load-balancer-eip-allocations` set to one pre-allocated Elastic IP **per public subnet (AZ)**. The EIP allocations must be set when the NLB is first created - they cannot be attached to an existing Classic ELB by annotation.
```bash
# Create a static public IP in the node resource group
NODE_RG=$(az aks show \
--resource-group boltmcp-rg \
--name boltmcp-cluster \
--query nodeResourceGroup -o tsv)
az network public-ip create \
--resource-group $NODE_RG \
--name boltmcp-ingress-ip \
--sku Standard \
--allocation-method Static \
--zone 1 2 3 \
--location westeurope
```
The `--location` should match the region of your AKS node resource group. The `create` command's JSON response includes the assigned `ipAddress` - note it down. If you lose it, look it up again with `az network public-ip show -g $NODE_RG -n boltmcp-ingress-ip --query ipAddress -o tsv`.
Set the IP on the ingress controller:
```bash
helm upgrade ingress-nginx ingress-nginx/ingress-nginx \
--namespace ingress-nginx \
--reuse-values \
--set controller.service.loadBalancerIP=
```
On AKS - unlike GKE and EKS - the load balancer's external IP **changes** when you run this upgrade: Azure swaps the ephemeral IP for your reserved one. Update any in-progress DNS work to use the new IP.
### Configure DNS [#configure-dns]
Point each BoltMCP subdomain at the load balancer. The **record type** depends on what the previous steps gave you, and the **hostname** values depend on whether your `global.domain` matches the DNS hosted zone.
* If your load balancer is an **IP address** (GKE, AKS, or an EKS NLB+EIP), create **A** records.
* If it's a **DNS hostname** (the default EKS Classic ELB), create **CNAME** records - a raw hostname can't go in an A record.
In the tables below, **``** is the `EXTERNAL-IP` you noted earlier (an IP *or* a hostname) and **``** is `A` for an IP or `CNAME` for a hostname.
BoltMCP's hostnames are always `web.`, `auth.`, `server.`, and `inspector.` prefixed, so they're never a zone apex - a wildcard or per-service **CNAME** is always valid for the EKS hostname case. The apex restriction only matters if you've overridden a service URL to a bare apex domain; there, use your provider's ALIAS/ANAME record (or a Route 53 *Alias*) instead of a CNAME, or switch to an IP and an A record.
You are updating DNS records in a hosted zone which sits above `global.domain`. For example:
* **DNS hosted zone**: *example.com*
* **BoltMCP global domain**: *boltmcp.example.com*
If your DNS provider supports wildcard records and a wildcard at this label won't clash with other records in the zone, add a single record:
| Hostname | Type | Value |
| ----------- | --------------- | --------------- |
| `*.boltmcp` | `` | `` |
Otherwise, create four explicit records:
| Hostname | Type | Value |
| ------------------- | --------------- | --------------- |
| `web.boltmcp` | `` | `` |
| `auth.boltmcp` | `` | `` |
| `server.boltmcp` | `` | `` |
| `inspector.boltmcp` | `` | `` |
If your subdomain is different, replace "boltmcp" in the Hostnames accordingly.
You are updating DNS records in a hosted zone which matches `global.domain`. For example:
* **DNS hosted zone**: *example.com*
* **BoltMCP global domain**: *example.com*
If your DNS provider supports wildcard records and a wildcard at the zone root won't clash with other records in the zone, add a single record:
| Hostname | Type | Value |
| -------- | --------------- | --------------- |
| `*` | `` | `` |
Otherwise, create four explicit records:
| Hostname | Type | Value |
| ----------- | --------------- | --------------- |
| `web` | `` | `` |
| `auth` | `` | `` |
| `server` | `` | `` |
| `inspector` | `` | `` |
Verify propagation across all four subdomains:
```bash
for h in web auth server inspector; do
printf "%s -> " "$h."
dig +short "$h." @8.8.8.8 || echo "(none)"
done
```
Each line should resolve to your load balancer - directly to the IP for **A** records, or to the load balancer hostname (and, below it, the IPs it points to) for **CNAME** records. Replace `` with your `global.domain` value.
### Enable the Chart-managed Ingress [#enable-the-chart-managed-ingress]
Add an `ingress` block to your values file. The annotations differ depending on whether or not you're terminating TLS:
```yaml title="config/values-prod.yaml"
ingress:
enabled: true
className: nginx
annotations:
cert-manager.io/cluster-issuer: letsencrypt-staging
```
The `cert-manager.io/cluster-issuer` annotation points at the issuer you create in [Reference TLS Setup](#reference-tls-setup) below. Until that issuer exists the certificate stays pending, which is expected.
```yaml title="config/values-local.yaml"
ingress:
enabled: true
className: nginx
annotations:
nginx.ingress.kubernetes.io/ssl-redirect: "false"
# Local install only (e.g. Docker Desktop): let in-cluster pods resolve the
# public hostnames to the ingress so server-side OIDC discovery reaches
# Keycloak. Set `ip` to the ingress controller's CLUSTER-IP.
global:
hostAliases:
- ip: ""
hostnames:
- web.boltmcp.local
- auth.boltmcp.local
- server.boltmcp.local
- inspector.boltmcp.local
```
For a local install, also add the browser-side hostnames to your workstation's `/etc/hosts`:
```
127.0.0.1 web.boltmcp.local auth.boltmcp.local server.boltmcp.local inspector.boltmcp.local
```
The chart's Ingress follows `global.tls.enabled`: on an HTTPS install it covers all BoltMCP hostnames with the `boltmcp-tls` Secret (name configurable via `ingress.tls.secretName`); on a plain-HTTP install it omits TLS entirely.
Apply the change using the same chart version and values file as the install (re-export `HELM_REGISTRY_CONFIG` and `BOLTMCP_VERSION` first if you're in a new shell):
```bash
helm upgrade boltmcp \
oci://europe-west2-docker.pkg.dev/boltmcp-platform/boltmcp-alpha/charts/boltmcp \
--version ${BOLTMCP_VERSION} \
-n boltmcp \
-f ./config/values-prod.yaml \
--timeout 15m
```
## Reference TLS Setup [#reference-tls-setup]
These steps add automatic HTTPS to the chart-managed Ingress via cert-manager and Let's Encrypt. Skip this section entirely on a plain-HTTP install (`global.tls.enabled: false`), or if your cluster already terminates TLS in front of the ingress.
### Install cert-manager [#install-cert-manager]
```bash
helm repo add jetstack https://charts.jetstack.io
helm repo update
helm install cert-manager jetstack/cert-manager \
--namespace cert-manager \
--create-namespace \
--set crds.enabled=true
```
Wait for all cert-manager pods to be running:
```bash
kubectl get pods -n cert-manager
```
### Create a ClusterIssuer [#create-a-clusterissuer]
Start with a **staging** issuer for testing (avoids Let's Encrypt rate limits):
```bash
kubectl apply -f ./config/cluster-issuer-staging.yaml
```
```yaml title="config/cluster-issuer-staging.yaml" extract="true"
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
name: letsencrypt-staging
spec:
acme:
server: https://acme-staging-v02.api.letsencrypt.org/directory
email: your-email@example.com
privateKeySecretRef:
name: letsencrypt-staging-account-key
solvers:
- http01:
ingress:
class: nginx
```
Replace `your-email@example.com` with a mailbox you actually monitor. Let's Encrypt uses this address to warn you when a certificate is approaching expiry without having auto-renewed - it's your only signal that renewal is broken before the cert goes down.
Verify:
```bash
kubectl get clusterissuer
# READY should be True
```
With the staging issuer in place, the certificate referenced by the chart-managed Ingress (applied in [Enable the Chart-managed Ingress](#enable-the-chart-managed-ingress) above) can now be provisioned. Watch it:
```bash
kubectl get certificates -n boltmcp -w
# Wait for READY to become True
```
### Switch to Production Certificates [#switch-to-production-certificates]
Once everything works with staging certificates, create a production issuer:
```bash
kubectl apply -f ./config/cluster-issuer-production.yaml
```
```yaml title="config/cluster-issuer-production.yaml" extract="true"
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
name: letsencrypt-production
spec:
acme:
server: https://acme-v02.api.letsencrypt.org/directory
email: your-email@example.com
privateKeySecretRef:
name: letsencrypt-production-account-key
solvers:
- http01:
ingress:
class: nginx
```
Again, replace `your-email@example.com` with a monitored mailbox so you receive Let's Encrypt's renewal-failure warnings.
Point the Ingress at the production issuer by updating the annotation in `values-prod.yaml`:
```yaml
ingress:
annotations:
cert-manager.io/cluster-issuer: letsencrypt-production
```
Run the same `helm upgrade` command as in the previous step, then delete the staging certificate Secret to trigger re-issuance:
```bash
kubectl delete secret boltmcp-tls -n boltmcp
```
Verify the new certificate:
```bash
kubectl get certificates -n boltmcp -w
# Wait for READY = True
```
Congratulations! You've successfully exposed BoltMCP to the public internet with TLS-secured ingress. Your cluster is now reachable at your configured hostnames with valid, auto-renewing certificates. Your browser should now show a trusted certificate without security warnings.
# Prerequisites (/docs/prerequisites)
## Kubernetes Client CLIs [#kubernetes-client-clis]
On the workstation you're installing from, you'll need:
* `kubectl v1.28+` to talk to your cluster · [Install kubectl on macOS](https://kubernetes.io/docs/tasks/tools/install-kubectl-macos/)
* `helm v3.12+` to install the BoltMCP chart on your cluster · [Install helm on macOS](https://helm.sh/docs/intro/install/#from-homebrew-macos)
* `kubectl v1.28+` to talk to your cluster · [Install kubectl on Linux](https://kubernetes.io/docs/tasks/tools/install-kubectl-linux/)
* `helm v3.12+` to install the BoltMCP chart on your cluster · [Install helm on Linux](https://helm.sh/docs/intro/install/#from-apt-debianubuntu)
* `kubectl v1.28+` to talk to your cluster · [Install kubectl on Windows](https://kubernetes.io/docs/tasks/tools/install-kubectl-windows/)
* `helm v3.12+` to install the BoltMCP chart on your cluster · [Install helm on Windows](https://helm.sh/docs/intro/install/#from-winget-windows)
Verify your kubectl version satisfies v1.28+:
```bash
kubectl version --client
```
Verify your helm version satisfies v3.12+:
```bash
helm version
```
## Cloud Provider CLI [#cloud-provider-cli]
If you're installing BoltMCP on one of the big three cloud providers, you'll need the appropriate CLI installed *and authenticated*:
| Cloud | CLI | Installation guides | Log in |
| ---------- | ---------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------- |
| Google GKE | `gcloud` | [Install gcloud](https://docs.cloud.google.com/sdk/docs/install-sdk) | `gcloud auth login` |
| Amazon EKS | `aws` + `eksctl` | [Install aws](https://docs.aws.amazon.com/cli/latest/userguide/getting-started-install.html) · [Install eksctl](https://docs.aws.amazon.com/eks/latest/eksctl/installation.html) | `aws login` |
| Azure AKS | `az` | [Install az](https://learn.microsoft.com/en-us/cli/azure/install-azure-cli) | `az login` |
# Sealed Secrets & SOPS (/docs/sealed-secrets-and-sops)
This page expands on the [Alternatives](/docs/cluster-preparation#alternatives-for-populating-secrets) noted in Cluster Prep. It covers GitOps-friendly ways to keep BoltMCP's three application Secrets (`boltmcp-database`, `boltmcp-oidc`, `boltmcp-auth`) encrypted at rest in version control.
Both approaches produce ordinary `Secret` resources that the chart consumes; the difference is how the source material lives at rest.
* **Sealed Secrets** — encrypt a `Secret` with `kubeseal` against the cluster's public key, commit the resulting `SealedSecret` to Git, and the in-cluster controller decrypts it back into a regular `Secret`. Good when you don't have an external secrets manager.
* **SOPS** — encrypt `Secret` YAML files with age / PGP / cloud KMS. Flux decrypts on apply natively; Argo CD supports SOPS via `argocd-vault-plugin` or `helm-secrets`.
In both cases the materialised `Secret` must end up named `boltmcp-database` / `boltmcp-oidc` / `boltmcp-auth` (or whatever you set `secrets..name` to) with the keys listed in [Cluster Prep](/docs/cluster-preparation#application-secrets).
# Security Hardening (/docs/security-hardening)
With public access enabled, consider:
* **Rate limiting** via ingress annotations:
```yaml
nginx.ingress.kubernetes.io/limit-rps: "10"
nginx.ingress.kubernetes.io/limit-connections: "5"
```
* **Brute force protection** in Keycloak: Realm Settings > Security Defenses > Enable Brute Force Detection
* **DDoS protection** via your cloud provider's WAF/shield service
* **Monitoring** with uptime checks on public endpoints
# Sign In (/docs/sign-in)
Once you've installed BoltMCP and set up ingress, run this script to retrieve your SSO login credentials. It reads the URLs straight from the deployed web workload, so it prints the right scheme and hostnames whatever your values (`global.domain`, `global.tls.enabled`, or any explicit overrides):
```bash title="get-login-details.sh" extract="true" bucket="deny"
#!/usr/bin/env bash
set -euo pipefail
env_of() {
kubectl get deployment boltmcp-web -n boltmcp \
-o jsonpath="{.spec.template.spec.containers[0].env[?(@.name=='$1')].value}"
}
WEB_BASE_URL=$(env_of WEB_BASE_URL)
OIDC_ISSUER_URL=$(env_of OIDC_ISSUER_URL)
KEYCLOAK_BASE_URL=${OIDC_ISSUER_URL%/realms/boltmcp}
echo "BoltMCP dashboard: ${WEB_BASE_URL}"
echo "BoltMCP docs: ${WEB_BASE_URL}/docs"
echo "Keycloak admin: ${KEYCLOAK_BASE_URL}/admin/boltmcp/console/"
echo "Username: boltmcp_admin"
printf "Password: "
kubectl get secret boltmcp-auth -n boltmcp \
-o jsonpath='{.data.boltmcp-admin-password}' | base64 -d
echo
```
This shows the initial password set at deployment time. If you've since updated the password in Keycloak, the value printed here will be stale.
The script output includes links to three websites from your deployment:
### 1. BoltMCP Dashboard [#1-boltmcp-dashboard]
Click the link to the BoltMCP dashboard in the script output and sign in via Keycloak with the given credentials. This dashboard is where you'll create and manage your MCP servers.
### 2. BoltMCP Documentation [#2-boltmcp-documentation]
The docs are linked from the dashboard's sidebar. They're distinct from these install docs. They give instruction on how to use the dashboard, how to manage authorization and how to connect MCP clients to your MCP servers.
### 3. Keycloak Admin Console [#3-keycloak-admin-console]
The admin console is also linked from the dashboard's sidebar. This is where you'll manage users, OAuth clients and connection to any other identity providers.
#### OAuth Clients [#oauth-clients]
As well as the Keycloak defaults, the BoltMCP Keycloak realm should already contain the following clients which are fundamental for the platform to function:
* `boltmcp-web` - for the dashboard to authenticate users
* `boltmcp-mcp-server` - for the MCP servers to authenticate users and agents
* `boltmcp-rest-api` - for the internal API to verify tokens
In addition, one client is pre-configured to streamline your first server setup:
* `boltmcp-mcp-server-to-rest-api` - for your first MCP server to authenticate users with the internal API, so that Claude can use the API
MCP clients such as Claude don't need a pre-configured client: they register themselves via the realm's CIMD client policies when connecting.
#### Realms [#realms]
Everything related to BoltMCP lives in the `boltmcp` Keycloak realm, including the clients above and the `boltmcp_admin` user.
The admin console base URL (e.g. `https://auth.`, or `http://` on a plain-HTTP install) automatically redirects to the master realm, not the BoltMCP realm. Use the link in the BoltMCP dashboard sidebar to open the BoltMCP realm.
For an evaluation deployment you shouldn't need access to the `master` realm. But for completeness, here are the master realm admin credentials:
```bash title="get-master-realm-creds.sh"
#!/usr/bin/env bash
set -eo pipefail
echo "Username: master_admin"
printf "Password: "
kubectl get secret boltmcp-auth -n boltmcp \
-o jsonpath='{.data.keycloak-admin-password}' | base64 -d
echo
```
# Troubleshooting (/docs/troubleshooting)
## Pods Stuck in Pending [#pods-stuck-in-pending]
The cluster may lack sufficient resources.
```bash
kubectl describe pod -n boltmcp
```
Look for events mentioning insufficient CPU or memory. Either scale the cluster or set resource requests in your values file.
## CreateContainerConfigError / Missing Secret [#createcontainerconfigerror--missing-secret]
The chart never creates the three application Secrets — pods fail with `CreateContainerConfigError: secret "boltmcp-database" not found` (or `-oidc` / `-auth`) until you create them. List what's actually present:
```bash
kubectl get secrets -n boltmcp
```
If any of `boltmcp-database`, `boltmcp-oidc`, `boltmcp-auth` is missing, create it per [Cluster Prep → Application Secrets](./cluster-preparation#application-secrets). Pods recover automatically on the next restart loop once the Secret exists.
## CrashLoopBackOff [#crashloopbackoff]
The application is crashing on startup. Check logs:
```bash
kubectl logs -n boltmcp
```
Common causes:
* **Wrong database password** — the password baked into PostgreSQL on first startup must match the `migrate-core-password` / `web-password` / `rest-api-password` / `mcp-server-password` / `keycloak-password` / `vault-password` in your `boltmcp-database` Secret. If you rotated a value in the Secret without resetting the corresponding DB user via `ALTER USER ... PASSWORD ...`, they'll diverge. Reset the password in the database or roll back the Secret value.
* **Missing key in a Secret** — if the chart references a key that doesn't exist in the user-managed Secret (e.g. `mcp-inspector-proxy-auth-token` while `mcpInspector.enabled=true`), pods fail to start. `kubectl describe pod` shows the missing key. Edit the Secret to add the key, then `kubectl rollout restart deployment/`.
* **Database not ready** — the init container should wait, but verify the database pod is healthy.
## ErrImagePull / ImagePullBackOff [#errimagepull--imagepullbackoff]
Kubernetes cannot pull the container images.
```bash
kubectl describe pod -n boltmcp
```
Verify the image pull secret exists:
```bash
kubectl get secrets -n boltmcp | grep boltmcp-pull-secret
```
If missing, recreate it:
```bash
kubectl create secret docker-registry boltmcp-pull-secret \
-n boltmcp \
--docker-server=europe-west2-docker.pkg.dev \
--docker-username=_json_key \
--docker-password="$(cat ./key.json)"
```
The chart's default `global.imagePullSecrets` is `[{ name: boltmcp-pull-secret }]`, so as long as the Secret exists under that name in the install namespace it will be picked up on the next pod restart (a `helm upgrade` is only required if you used a non-default Secret name and need to override the value).
## Connection Refused [#connection-refused]
* **Pod not ready** — check pod status with `kubectl get pods -n boltmcp`
* **Service not found** — verify services exist with `kubectl get svc -n boltmcp`
* **Ingress or DNS misconfigured** — bypass them with port-forwarding (see below) to confirm the pod itself is healthy
## Bypass Ingress with Port-Forwarding [#bypass-ingress-with-port-forwarding]
If the Ingress, DNS, or TLS layer is misbehaving, port-forward directly to a service to confirm the pod is responding. This is a diagnostic tool, not a normal access path.
```bash
# Web app
kubectl port-forward -n boltmcp svc/boltmcp-web 3000:3000
# Keycloak
kubectl port-forward -n boltmcp svc/boltmcp-keycloak 8080:8080
# MCP Server
kubectl port-forward -n boltmcp svc/boltmcp-mcp-server 3001:3001
```
Note: OIDC redirects will fail when accessed via `localhost`, since the issuer URL in the values file points at your public Keycloak hostname. Port-forwarding is useful for verifying a single service is up, not for an end-to-end auth flow.
Keycloak's admin UI is a special case: with `keycloak.production.enabled: true` (the default), `KC_HOSTNAME` is enforced, so the admin console will load briefly via `localhost:8080` and then redirect you to `https://auth.boltmcp.example.com`. Use the public Keycloak URL for admin work; port-forwarding to Keycloak is only useful for hitting `/health/ready` to confirm the pod is up.
## Authentication Not Working [#authentication-not-working]
### Issuer URL Mismatch [#issuer-url-mismatch]
The OIDC issuer URL must be identical in the browser and in the application pods:
```bash
kubectl describe pod -n boltmcp | grep OIDC
```
Ensure the issuer URL matches the Keycloak hostname exactly (protocol, host, port, path).
On plain-HTTP installs, set `global.tls.enabled: false` rather than overriding URLs one by one — a lone `https://` issuer (the default when only the base URLs are overridden to `http://`) silently breaks token validation.
### Public hostnames don't resolve inside the cluster (split-horizon DNS) [#public-hostnames-dont-resolve-inside-the-cluster-split-horizon-dns]
**Symptoms:** the browser reaches BoltMCP fine, but the web, rest-api, or seed pods fail with OIDC discovery errors against `auth.` — `ENOTFOUND`, connection timeouts, or `ECONNREFUSED 127.0.0.1` (a workstation-only `/etc/hosts` entry leaking into cluster DNS and resolving to the pod's own loopback).
**Cause:** the services fetch the OIDC discovery document from the *public* issuer URL server-side. If the public hostnames only resolve outside the cluster (local installs, split-horizon corporate DNS), those in-cluster requests fail even though everything works in the browser.
**Fix:** set `global.hostAliases` so every BoltMCP pod resolves the public hostnames to your ingress:
```yaml
global:
hostAliases:
- ip: ""
hostnames:
- auth.boltmcp.example.com
- web.boltmcp.example.com
- server.boltmcp.example.com
```
Then `helm upgrade` with the updated values. No CoreDNS changes are needed.
### Missing Email or Name [#missing-email-or-name]
BoltMCP requires users to have an **email** and **first name** to sign in. The auto-provisioned `boltmcp_admin` user gets both fields set at realm-import time (email from `oidc.adminUser.email`, firstName `Admin`). If you add more users later through the Keycloak admin console, make sure each has both fields populated before they try to sign into the BoltMCP web app.
### Client Secret Mismatch [#client-secret-mismatch]
The OIDC client secrets in the `boltmcp-oidc` Secret must match what's configured on the corresponding clients in Keycloak. There are five client secrets: `web-client-secret`, `mcp-server-client-secret`, `mcp-client-client-secret`, `mcp-server-to-rest-api-client-secret`, and `rest-api-resource-server-client-secret`. To rotate a value:
1. Edit the `boltmcp-oidc` Secret (`kubectl edit secret boltmcp-oidc -n boltmcp`, or re-apply via your secrets manager) so the new value is base64-encoded under the right key.
2. Update the same value on the matching client in the Keycloak admin console.
3. Restart deployments so they pick up the new value (Kubernetes does not auto-restart pods on Secret changes):
```bash
kubectl rollout restart -n boltmcp deployment/boltmcp-web
kubectl rollout restart -n boltmcp deployment/boltmcp-mcp-server
kubectl rollout restart -n boltmcp deployment/boltmcp-rest-api
```
Rotating `mcp-server-to-rest-api-client-secret` or `rest-api-resource-server-client-secret` also requires restarting Keycloak. Those values are interpolated into the realm JSON at `--import-realm` time from environment variables on the Keycloak Pod, so Keycloak holds the old secret in memory until it restarts — without a restart you will see introspection failures (401) persist after the rotation.
```bash
kubectl rollout restart -n boltmcp deployment/boltmcp-keycloak
```
### Redirect Loop [#redirect-loop]
Check that client redirect URIs in Keycloak match the web URLs. The redirect URI must include the full path pattern (e.g. `https://web.boltmcp.example.com/*`).
## Vault Secret Endpoints Failing (502 / 503) [#vault-secret-endpoints-failing-502--503]
The REST API's secret endpoints (`/api/v1/secrets/*`) depend on the bundled Vault being initialized, unsealed, and bootstrapped. The HTTP status tells you which step is missing:
* **503 (Vault not configured)** — `vault.kubernetesAuth.enabled` is off, or the bootstrap hasn't created the auth method/role yet. Run the [Vault bootstrap](./vault-secret-store).
* **502 (Vault unavailable)** — Vault is **sealed** or unreachable, or the Kubernetes login was rejected. Check the seal state and unseal if needed:
```bash
kubectl exec -it -n boltmcp deploy/boltmcp-vault -- vault status
```
Remember that with the default (Shamir) seal, **every Vault pod restart re-seals it** — re-run `vault operator unseal`, or configure [auto-unseal](./vault-secret-store).
If login is rejected even when Vault is unsealed and bootstrapped, the most common causes are a projected-token **audience** that doesn't match the Vault role's `audience`, or the Vault ServiceAccount lacking the `system:auth-delegator` binding (it can't run `TokenReview`). Both are wired by the chart, so check that `vault.kubernetesAuth.audience` was not overridden inconsistently and inspect the REST API logs:
```bash
kubectl logs -n boltmcp deploy/boltmcp-rest-api | grep -i vault
```
## Certificate Issues [#certificate-issues]
### Certificate Stuck in False State [#certificate-stuck-in-false-state]
```bash
kubectl describe certificate boltmcp-tls -n boltmcp
kubectl get challenges -n boltmcp
```
Common causes:
* **DNS not propagated** — verify with `nslookup web.boltmcp.example.com`
* **HTTP-01 challenge failed** — ensure NGINX ingress is running and accessible
* **Rate limited** — use the staging ClusterIssuer for testing
### Large Header Errors [#large-header-errors]
If Keycloak produces "upstream sent too big header" errors, the auth headers exceed the ingress buffer. The chart-managed Ingress allows 128k by default; raise it in your values file:
```yaml
ingress:
annotations:
nginx.ingress.kubernetes.io/proxy-buffer-size: "256k"
```
If you run your own Ingress manifest rather than the chart-managed one, set the same annotation there - without it NGINX applies a 4k default.
### 413 Errors Uploading an API Spec [#413-errors-uploading-an-api-spec]
If importing an OpenAPI spec from the **APIs** page fails with `413 Request Entity Too Large`, the file exceeds the ingress body-size limit. The chart-managed Ingress allows 10 MB by default; raise it in your values file:
```yaml
ingress:
annotations:
nginx.ingress.kubernetes.io/proxy-body-size: "25m"
```
If you run your own Ingress manifest rather than the chart-managed one, set the same annotation there - without it NGINX applies a 1 MB default. Note that NGINX buffers the whole upload before forwarding it, so very large values increase memory and disk use on the ingress controller.
## Database lost+found Error [#database-lostfound-error]
If the database pod logs show:
```
initdb: error: directory "/var/lib/postgresql/data" exists but is not empty
initdb: detail: It contains a lost+found directory
```
The PVC must be recreated:
```bash
helm uninstall boltmcp -n boltmcp
kubectl delete pvc data-boltmcp-database-0 -n boltmcp
helm install boltmcp \
oci://europe-west2-docker.pkg.dev/boltmcp-platform/boltmcp-alpha/charts/boltmcp \
--version ${BOLTMCP_VERSION} \
-n boltmcp \
-f ./config/values-prod.yaml
```
## Diagnostic Commands [#diagnostic-commands]
```bash
# Pod status
kubectl get pods -n boltmcp
# Pod logs
kubectl logs -n boltmcp
# Pod events and details
kubectl describe pod -n boltmcp
# Services and endpoints
kubectl get svc -n boltmcp
kubectl get endpoints -n boltmcp
# Secrets
kubectl get secrets -n boltmcp
# Helm release status
helm list -n boltmcp
helm status boltmcp -n boltmcp
# Certificate status (if using Ingress)
kubectl get certificates -n boltmcp
kubectl get challenges -n boltmcp
# Ingress status
kubectl get ingress -n boltmcp
# NGINX Ingress logs
kubectl logs -n ingress-nginx -l app.kubernetes.io/name=ingress-nginx
```
Any helm commands require having set shell variable `HELM_REGISTRY_CONFIG` to point to your boltmcp key.
# Uninstall & Cleanup (/docs/uninstall)
## Set Variables [#set-variables]
Export the Helm release name and namespace so the commands below can be pasted verbatim. Both default to `boltmcp` — adjust if you installed under different names (run `helm list -A` to look them up).
```bash
export RELEASE=boltmcp
export NAMESPACE=boltmcp
```
## Uninstall the Helm Release [#uninstall-the-helm-release]
```bash
helm uninstall ${RELEASE} -n ${NAMESPACE}
```
Depending on their delete policy, some Helm hook Jobs (`migrate-core`, `seed`) may survive `helm uninstall`. This only matters if you intend to keep the namespace — deleting the namespace (below) clears them.
## Delete Persistent Data [#delete-persistent-data]
This deletes all persistent volume claims in the namespace, including the database. **All data will be lost.**
```bash
kubectl delete pvc -n ${NAMESPACE} --all
```
Do this **before** deleting the cluster. The database PersistentVolume is provisioned with `reclaimPolicy: Delete`, so deleting the PVC releases the backing disk — on EKS, the underlying EBS volume. `eksctl delete cluster` does **not** delete CSI-provisioned EBS volumes (they aren't part of its CloudFormation stack), so skipping this step and jumping straight to cluster deletion leaks the volume.
## Delete Secrets [#delete-secrets]
The three application secrets and the image pull secret were created manually during [Cluster Prep](/docs/cluster-preparation) and are not managed by Helm, so `helm uninstall` does not remove them. Delete them explicitly:
```bash
kubectl delete secret \
${RELEASE}-auth \
${RELEASE}-database \
${RELEASE}-oidc \
boltmcp-pull-secret \
-n ${NAMESPACE}
```
The three application secrets default to `-auth`, `-database`, `-oidc`. The pull secret defaults to `boltmcp-pull-secret` — adjust the command above if you created it under a different name in cluster prep.
## Remove Ingress Resources [#remove-ingress-resources]
If you configured [Ingress & TLS](/docs/ingress-tls), remove those resources:
If other services in your cluster share the same NGINX Ingress Controller or cert-manager installation, skip those components below. In particular, deleting the cert-manager CRDs is **cluster-wide destructive** — it removes every `Certificate`, `Issuer`, and `ClusterIssuer` in the cluster, not just BoltMCP's.
```bash
# TLS secret (the chart-managed Ingress itself is already removed with the
# release; if you created your own Ingress resource instead, delete it too)
kubectl delete secret boltmcp-tls -n ${NAMESPACE}
# Certificate (may return NotFound if already cleaned up by cert-manager)
kubectl delete certificate boltmcp-tls -n ${NAMESPACE}
# ClusterIssuers
kubectl delete clusterissuer letsencrypt-staging
kubectl delete clusterissuer letsencrypt-production
# cert-manager (helm uninstall intentionally keeps CRDs — delete them manually)
helm uninstall cert-manager -n cert-manager
kubectl delete namespace cert-manager
# Deleting these CRDs is cluster-wide: it removes every cert-manager
# resource in the cluster, not just BoltMCP's. Skip if other workloads use cert-manager.
kubectl delete crd \
challenges.acme.cert-manager.io \
orders.acme.cert-manager.io \
certificaterequests.cert-manager.io \
certificates.cert-manager.io \
clusterissuers.cert-manager.io \
issuers.cert-manager.io
# NGINX Ingress Controller
helm uninstall ingress-nginx -n ingress-nginx
# The cloud load balancer is deleted asynchronously when the LoadBalancer
# Service is removed. Wait until this returns no controller Service before
# continuing — re-run as needed:
kubectl get svc -n ingress-nginx
kubectl delete namespace ingress-nginx
```
On EKS, do not move on to deleting the cluster until the load balancer is gone. An orphaned load balancer and its controller-managed security group will block `eksctl delete cluster` from tearing down the VPC. Confirm `kubectl get svc -n ingress-nginx` shows no controller Service (and, if in doubt, check the EC2 console) before continuing.
### Release Static IP [#release-static-ip]
```bash
REGION=$(gcloud container clusters describe \
--format="get(location)" | sed 's/-[a-z]$//')
gcloud compute addresses delete boltmcp-ingress-ip --region $REGION
```
The default ingress setup reserves nothing to release. The Classic ELB has no Elastic IP of its own, and the cluster's NAT gateway EIP is owned by the eksctl CloudFormation stack — `eksctl delete cluster` releases it for you. Skip this step.
**Only if** you provisioned the optional **NLB with Elastic IPs** (the apex/allowlist override described on the [Ingress & TLS](/docs/ingress-tls) page) do you need to release those addresses manually:
```bash
# Find the allocation ID(s) for the Elastic IP(s) you allocated
ALLOCATION_ID=$(aws ec2 describe-addresses \
--filters "Name=tag:Name,Values=boltmcp-*" \
--query 'Addresses[0].AllocationId' --output text)
aws ec2 release-address --allocation-id $ALLOCATION_ID
```
```bash
NODE_RG=$(az aks show \
--resource-group boltmcp-rg \
--name boltmcp-cluster \
--query nodeResourceGroup -o tsv)
az network public-ip delete \
--resource-group $NODE_RG \
--name boltmcp-ingress-ip
```
Remove the DNS records you created in [Configure DNS](/docs/ingress-tls#configure-dns) — either the single wildcard or the four `web`, `auth`, `server`, and `inspector` records for your `global.domain` (e.g. `web.`) — from your DNS provider. These are **A** records on GKE and AKS (and on an EKS NLB+EIP), or **CNAME** records pointing at the load balancer hostname on the default EKS Classic ELB.
## Delete the Namespace [#delete-the-namespace]
If you no longer need the namespace, delete it:
```bash
kubectl delete namespace ${NAMESPACE}
```
Skip this step if you plan to reinstall BoltMCP into the same namespace.
## Delete the StorageClass (EKS) [#delete-the-storageclass-eks]
On EKS you created a `gp3` StorageClass during [Cluster Prep](/docs/cluster-preparation#storageclass). It is cluster-scoped and was applied directly with `kubectl`, so neither `helm uninstall` nor deleting the namespace removes it. (GKE and AKS used their built-in default — there is nothing to delete.)
```bash
kubectl delete storageclass gp3
```
Safe to keep if you plan to reinstall BoltMCP into the same cluster — the next install will reuse it.
If you opted to mark this class the cluster default (`storageclass.kubernetes.io/is-default-class: "true"`) and other workloads have come to rely on it, deleting it changes their PVC behaviour — leave it in place, or mark another StorageClass default first.
## Delete the Cluster [#delete-the-cluster]
On EKS, deleting the cluster is **not** self-sufficient. Before running the command below, make sure you have already (1) uninstalled `ingress-nginx` and confirmed the cloud load balancer is gone (see [Remove Ingress Resources](#remove-ingress-resources)), and (2) deleted the PVCs (see [Delete Persistent Data](#delete-persistent-data)). Otherwise the load balancer and its security group can stall the VPC teardown, and the EBS volumes behind the PVCs will leak.
If you no longer need the Kubernetes cluster:
```bash
gcloud container clusters delete boltmcp-cluster --zone europe-west2-a
```
```bash
eksctl delete cluster --name boltmcp-cluster --region eu-west-2
```
If you'd rather keep the cluster for a future reinstall, there's no need to scale anything down: once the BoltMCP workloads are gone, EKS Auto Mode self-consolidates its nodes to zero, so no EC2 instances are left running.
```bash
az aks delete \
--resource-group boltmcp-rg \
--name boltmcp-cluster
# Optionally delete the resource group
az group delete --name boltmcp-rg
```
## Clean Up Local Workstation (Optional) [#clean-up-local-workstation-optional]
### kubeconfig [#kubeconfig]
Deleting the cluster in your cloud provider does not touch your local `~/.kube/config`. The stale context, cluster, and user entries will remain until you remove them explicitly:
```bash
kubectl config delete-context
kubectl config delete-cluster
kubectl config delete-user
```
Run `kubectl config get-contexts` first to find the exact names. If the deleted context was your active one, set a new active context afterwards:
```bash
kubectl config use-context
```
### Local YAML manifests [#local-yaml-manifests]
Remove any files you created locally during install — for example `values-prod.yaml`, `storageclass-gp3.yaml` (EKS), `cluster-issuer-staging.yaml`, and `cluster-issuer-production.yaml` — from the directory where you ran the install.
### Helm registry credentials [#helm-registry-credentials]
Following these docs, you pointed Helm at the BoltMCP key file via the `HELM_REGISTRY_CONFIG` environment variable rather than running `helm registry login`, so there is no cached credential to remove — just discard the env var (it disappears when you close the shell) and delete the key file if you no longer need it:
```bash
rm keys/boltmcp-key.json
```
Only if you explicitly ran `helm registry login` against the registry do you need to log out to clear the cached credential at `~/.config/helm/registry/config.json`:
```bash
helm registry logout europe-west2-docker.pkg.dev
```
# Manual Upgrade (/docs/upgrading)
To deploy a new release, first set the target version and update the paths to your values file and access key file:
```bash
export BOLTMCP_VERSION=0.3.8
export BOLTMCP_VALUES="$PWD/config/values-prod.yaml"
export HELM_REGISTRY_CONFIG="$PWD/keys/boltmcp-key.json"
```
Make sure your values file contains all required fields from the *Getting Started* pages, including the `ingress` block if you're using chart-managed ingress.
Now run the upgrade:
```bash
helm upgrade boltmcp \
oci://europe-west2-docker.pkg.dev/boltmcp-platform/boltmcp-alpha/charts/boltmcp \
--version ${BOLTMCP_VERSION} \
-n boltmcp \
-f ${BOLTMCP_VALUES} \
--timeout 15m
```
Run with `--dry-run=client` first to verify the rendered manifests and surface any schema errors before applying for real.
The upgrade blocks on the post-upgrade hook jobs (database migration and seed). As with [installation](./deployment), `--timeout 15m` is required on EKS Auto Mode, where scheduling the hook pods can trigger a node scale-up that exceeds Helm's default 5m timeout. A timed-out upgrade leaves the release stuck in `pending-upgrade` (see below).
## Verify the Upgrade [#verify-the-upgrade]
Confirm the new revision shows `deployed`:
```bash
helm history boltmcp -n boltmcp
```
Watch pods until all show `Running` / `Completed`:
```bash
kubectl get pods -n boltmcp -w
```
The `boltmcp-migrate-core` and `boltmcp-seed` pods created by the post-upgrade hooks should show `Completed`. If a pod is not starting, check its logs and events:
```bash
kubectl logs -n boltmcp
kubectl describe pod -n boltmcp
```
## Check the Vault Seal Status [#check-the-vault-seal-status]
If the upgrade caused the Vault pod to restart, and auto-unseal isn't configured, it will come back sealed. Check the `Sealed` status with:
```bash
kubectl exec -n boltmcp deploy/boltmcp-vault -- vault status
```
If it reads `true`, [unseal Vault again](./vault-secret-store), or configure [auto-unseal](./vault-auto-unseal) to avoid the manual step on future restarts.
## Values-Only Changes [#values-only-changes]
To apply a change to `values-prod.yaml` without a chart-version bump, re-run the same `helm upgrade` command above with the unchanged `--version`. Values render into the pod specs, so Helm rolls the affected deployments automatically — no manual restart is needed.
Editing the **contents** of the externally-managed application Secrets (`boltmcp-database`, `boltmcp-oidc`, `boltmcp-auth`) is different: that isn't a values change and Helm won't notice it. Pods read these Secrets only at startup, so restart the deployments that consume them:
```bash
kubectl rollout restart -n boltmcp deployment/
```
## Recovering a stuck `pending-upgrade` release [#recovering-a-stuck-pending-upgrade-release]
If a `helm upgrade` is interrupted (timeout, Ctrl-C, OOM, kube-apiserver hiccup) the release record can be left in `pending-upgrade` state. Every subsequent `helm upgrade` will then fail with:
```
Error: UPGRADE FAILED: another operation (install/upgrade/rollback) is in progress
```
Before doing anything, make sure the state is actually stale: confirm no `helm upgrade` is still running in another terminal or CI job, and let any in-flight hook jobs from the interrupted attempt finish (retrying while a migration job is mid-run would delete and recreate it):
```bash
kubectl get jobs -n boltmcp
```
To clear the stale record without losing the deployed resources, delete the stuck revision's Helm release Secret. Helm will fall back to the previous successful revision as the current state:
```bash
# Find the pending revision (column STATUS = pending-upgrade)
helm history boltmcp -n boltmcp
# Delete its release Secret
kubectl delete secret sh.helm.release.v1.boltmcp.v -n boltmcp
# Confirm the release no longer shows pending-upgrade
helm history boltmcp -n boltmcp
# Re-run the upgrade
helm upgrade boltmcp ... -f ./config/values-prod.yaml
```
`helm rollback` is the textbook recovery, but it can fail with `original object Secret with the name "boltmcp-auth" not found` when rolling back across a chart version that changed which resources are Helm-managed. If that happens, fall back to the manual `kubectl delete secret sh.helm.release.v1.boltmcp.v` approach above.
# Vault Auto-unseal (/docs/vault-auto-unseal)
## Auto-unseal (production) [#auto-unseal-production]
`vault.seal` makes the seal method pluggable. The default (`type: ""`) is Shamir / manual unseal — no external dependency, installable on any cluster, but it re-seals on every restart. Set `vault.seal.type` to a cloud KMS (`awskms`, `gcpckms`, `azurekeyvault`) or `transit` to auto-unseal on every pod start:
```yaml title="config/values-prod.yaml"
vault:
seal:
type: awskms
config:
region: us-east-1
kms_key_id: ""
# Credentials for the seal, injected as Vault container env (often a
# secretKeyRef). On AWS you can instead use IRSA and omit these.
extraEnv:
- name: AWS_ACCESS_KEY_ID
valueFrom:
secretKeyRef:
name: boltmcp-vault-seal
key: aws-access-key-id
- name: AWS_SECRET_ACCESS_KEY
valueFrom:
secretKeyRef:
name: boltmcp-vault-seal
key: aws-secret-access-key
```
Auto-unseal removes the *repeated* unseal-after-restart toil, but **not** the
one-time `vault operator init` — a fresh Vault still needs initializing once
(with auto-unseal, `init` emits *recovery* keys plus the initial root token).
# Vault Secret Store (/docs/vault-secret-store)
The BoltMCP chart deploys a bundled HashiCorp Vault backed by the shared PostgreSQL (schema `boltmcp_vault`). It serves as a secret store for things like sensitive upstream API credentials. It boots sealed and uninitialized, single-replica, and its data persists across restarts. It's never exposed outside the cluster.
## One-time setup [#one-time-setup]
After the chart is installed and the `boltmcp-vault` pod is `Running`:
**Initialize Vault** - this generates the unseal/recovery keys and the initial root token. **Store them securely**, they cannot be recovered.
```bash
kubectl exec -it -n boltmcp deploy/boltmcp-vault -- \
vault operator init
```
**Unseal Vault** - repeat this command three times, pasting a distinct unseal key each time you're prompted, until `sealed` flips to false:
```bash
kubectl exec -it -n boltmcp deploy/boltmcp-vault -- \
vault operator unseal
```
With the default Shamir seal, **every Vault pod restart re-seals it**, so you must repeat this step after restarts. For production,
configure [auto-unseal](./vault-auto-unseal).
**Run the bootstrap** - this gives BoltMCP services read/write access. Log in with the root (or a suitably privileged) token, then run the chart-mounted script:
```bash
kubectl exec -it -n boltmcp deploy/boltmcp-vault -- \
sh -c 'vault login && sh /bootstrap/vault-bootstrap.sh'
```
This is **idempotent**, and a one-time action per Vault data lifetime, so you don't need to re-run it on every helm upgrade.
## What the bootstrap creates [#what-the-bootstrap-creates]
The bootstrap script is rendered from your values, so its names match the chart. With defaults it:
* Enables the KV v2 secrets engine at `secret/`.
* Enables the Kubernetes auth method at `kubernetes/` and points it at the in-cluster API server (Vault uses its own pod token for `TokenReview`).
* Writes policies granting read/write under a specific path (and metadata).
* Writes roles binding each ServiceAccount + namespace + token audience to the policies.
## About Kubernetes auth [#about-kubernetes-auth]
The chart gives each consuming BoltMCP service a dedicated ServiceAccount and an audience-bound, auto-rotated projected token. At request time the service logs in to Vault's Kubernetes auth method with that token, receives a short-lived Vault token, and uses it to read/write secrets. Vault runs as its own ServiceAccount bound to `system:auth-delegator` so it can perform the `TokenReview` that validates the login.