Manual Upgrade

To deploy a new release, bump BOLTMCP_VERSION and run helm upgrade with your values file:

export BOLTMCP_VERSION=0.1.2

helm upgrade boltmcp \
  oci://europe-west2-docker.pkg.dev/boltmcp-platform/boltmcp-alpha/charts/boltmcp \
  --version ${BOLTMCP_VERSION} \
  -n boltmcp \
  -f ./config/values-prod.yaml

Image references in the chart are pinned to the chart's version, so an upgrade automatically rolls each Deployment without needing kubectl rollout restart.

Always run with --dry-run=client first and verify the rendered manifests / surface any schema errors before applying for real.

How values are merged

The -f ./config/values-prod.yaml form above uses the new chart's values.yaml as the base, with your file overlaid on top. The previous release's in-cluster values are not consulted. Anything you omit from your file falls back to the new chart's defaults.

This is almost always what you want. Two alternative strategies exist but are usually wrong:

• --reuse-values -f deltas.yaml — the previous in-cluster values become the base. Useful if your local file only contains the changes since last upgrade. Avoid when bumping chart versions — if the new chart's schema rejects any field present in the old values (renamed keys, fields moved into Secrets, etc.), the upgrade fails validation before applying anything.
• --reset-values — drops all custom values and uses pure chart defaults. Rarely the intended behaviour.

If you don't have a local values-prod.yaml yet, recover the current one with helm get values boltmcp -n boltmcp -o yaml > values-prod.yaml, then strip out any fields the new chart no longer accepts (see "Schema migrations" below).

Values-Only Changes

If you only changed values in values-prod.yaml (no chart-version bump) and the changed values flow into Secrets/ConfigMaps that your pods consume only at startup, restart the affected deployments so new values take effect:

kubectl rollout restart -n boltmcp deployment/boltmcp-web
kubectl rollout restart -n boltmcp deployment/boltmcp-mcp-server
kubectl rollout restart -n boltmcp deployment/boltmcp-rest-api

Rotating Application Secrets

The three application Secrets (boltmcp-database, boltmcp-oidc, boltmcp-auth) are managed outside Helm — rotation is a direct edit, no helm upgrade is involved:

1. Update the value in the source of truth (your secrets manager, kubeseal source, or the Secret itself via kubectl edit secret <name> -n boltmcp).

2. If the value lives in an external system (Vault, AWS SM, etc.) wait for ESO / your operator to sync, or force an immediate sync.

3. Restart the deployments that consume the changed key so they pick up the new value:

   kubectl rollout restart -n boltmcp deployment/boltmcp-web
   kubectl rollout restart -n boltmcp deployment/boltmcp-mcp-server

Rotating database passwords additionally requires updating the corresponding PostgreSQL users (ALTER USER ... PASSWORD ...) so the new Secret value matches what the server expects. The chart does not do this for you.

Schema migrations between chart versions

A new chart version may rename fields, remove ones the old chart accepted, or require new ones. The dry run surfaces this as values don't meet the specifications of the schema errors — read them carefully and edit your local values-prod.yaml to match.

Two patterns to watch for:

• Fields moved into user-managed Secrets. If the old chart accepted inline values like database.users.web.password or oidc.web.clientSecret and the new chart rejects them, the values have moved into one of the three Secrets (boltmcp-database, boltmcp-oidc, boltmcp-auth). See Cluster Prep → Application Secrets for the full key list. Pre-create the Secrets with the same values that were previously inline (otherwise pods will start with new credentials but the database/Keycloak still expect the old ones), then remove those fields from values-prod.yaml.
• Renamed fields with new required fields. E.g. a new required oidc.adminUser.email. Add the new field with a value matching the existing state (e.g. the email of whoever was provisioned as first admin on install).

Migrating from a chart version that templated the application Secrets

Chart versions prior to 0.0.81 generated boltmcp-database, boltmcp-oidc, and boltmcp-auth Secrets internally from inline password values. 0.0.81 expects them to exist as user-managed resources instead, and a naive upgrade will silently delete the existing Secrets via Helm's 3-way merge (the resources are in the previous revision's tracked manifest but not in the new one).

The deletion isn't immediately obvious because already-running pods cache their env vars and keep working; the breakage shows up in the post-upgrade hook Job (boltmcp-migrate-core) failing with Error: secret "boltmcp-database" not found, and the release getting stuck in pending-upgrade.

Before running helm upgrade, pre-create the three Secrets with the existing values and the helm.sh/resource-policy: keep annotation, which tells Helm to skip them during 3-way merge deletion:

apiVersion: v1
kind: Secret
metadata:
  name: boltmcp-database
  namespace: boltmcp
  annotations:
    helm.sh/resource-policy: keep
type: Opaque
stringData:
  superuser-password: "..."   # value previously in database.superuser.password
  web-password: "..."         # value previously in database.users.web.password
  mcp-server-password: "..."
  keycloak-password: "..."
---
apiVersion: v1
kind: Secret
metadata:
  name: boltmcp-oidc
  namespace: boltmcp
  annotations:
    helm.sh/resource-policy: keep
type: Opaque
stringData:
  web-client-secret: "..."           # value previously in oidc.web.clientSecret
  mcp-server-client-secret: "..."
  mcp-client-client-secret: "..."
  mcp-server-to-rest-api-client-secret: "..."    # new
  rest-api-resource-server-client-secret: "..."  # new
---
apiVersion: v1
kind: Secret
metadata:
  name: boltmcp-auth
  namespace: boltmcp
  annotations:
    helm.sh/resource-policy: keep
type: Opaque
stringData:
  web-auth-secret: "..."             # value previously in web.authSecret
  keycloak-admin-password: "..."     # value previously in keycloak.admin.password
  boltmcp-admin-password: "..."      # new — first admin user in the boltmcp realm
  mcp-inspector-proxy-auth-token: "..."  # value previously in mcpInspector.proxyAuthToken

Recover the existing values from helm get values boltmcp -n boltmcp -o yaml (or from your secrets manager). Apply the file, then run helm upgrade as normal.

Upgrading to a chart with the bundled Vault

Newer chart versions deploy a HashiCorp Vault (boltmcp-vault) backed by the shared PostgreSQL, enabled by default. Upgrading an existing release has two prerequisites — handle both before helm upgrade, or skip the component for now with vault.enabled: false.

1. Add the vault-password key to the boltmcp-database Secret. Without it the Vault pod fails to start with CreateContainerConfigError. Add the key with whatever mechanism manages that Secret (e.g. kubectl edit secret boltmcp-database -n boltmcp, or your ESO / SealedSecret / SOPS source), using a URL-safe value (no @ : / + #).

2. Create the Vault schema/role/table in the existing database. The init script that provisions these only runs on a fresh database, so an existing PVC won't have them and Vault will crash-loop on a missing vault_kv_store table. Run the following once against the existing database as a superuser, using the same password you put in the Secret:

   CREATE USER boltmcp_vault WITH PASSWORD '<value of vault-password>';
   CREATE SCHEMA IF NOT EXISTS boltmcp_vault AUTHORIZATION boltmcp_vault;
   GRANT ALL PRIVILEGES ON SCHEMA boltmcp_vault TO boltmcp_vault;
   ALTER ROLE boltmcp_vault SET search_path TO boltmcp_vault;
   
   SET ROLE boltmcp_vault;
   SET search_path TO boltmcp_vault;
   CREATE TABLE IF NOT EXISTS vault_kv_store (
     parent_path TEXT COLLATE "C" NOT NULL,
     path        TEXT COLLATE "C",
     key         TEXT COLLATE "C",
     value       BYTEA,
     CONSTRAINT pkey PRIMARY KEY (path, key)
   );
   CREATE INDEX IF NOT EXISTS parent_path_idx ON vault_kv_store (parent_path);
   RESET ROLE;

Vault boots sealed and uninitialized; initialize/unseal it out-of-band (kubectl exec ... -- vault operator init / ... unseal). boltmcp-rest-api consumes it via Kubernetes auth, but only after a one-time bootstrap — until you initialize, unseal, and run that bootstrap, the REST API's secret endpoints return 503/502 and the rest of the stack is unaffected. See the Vault page for the full init / unseal / bootstrap procedure and auto-unseal options.

Recovering a stuck pending-upgrade release

If a helm upgrade is interrupted (timeout, Ctrl-C, OOM, kube-apiserver hiccup) the release record can be left in pending-upgrade state. Every subsequent helm upgrade will then fail with:

Error: UPGRADE FAILED: another operation (install/upgrade/rollback) is in progress

To clear it without losing the deployed resources, delete the stuck revision's Helm release Secret. Helm will fall back to the previous successful revision as the current state:

# Find the pending revision (column STATUS = pending-upgrade)
helm history boltmcp -n boltmcp

# Delete its release Secret
kubectl delete secret sh.helm.release.v1.boltmcp.v<N> -n boltmcp

# Confirm the release no longer shows pending-upgrade
helm history boltmcp -n boltmcp

# Re-run the upgrade
helm upgrade boltmcp ... -f ./config/values-prod.yaml

helm rollback is the textbook recovery, but it can fail with original object Secret with the name "boltmcp-auth" not found when rolling back across a chart version that changed which resources are Helm-managed. If that happens, fall back to the manual kubectl delete secret sh.helm.release.v1.boltmcp.v<N> approach above.