Troubleshooting

Pods Stuck in Pending

The cluster may lack sufficient resources.

kubectl describe pod -n boltmcp <pod-name>

Look for events mentioning insufficient CPU or memory. Either scale the cluster or set resource requests in your values file.

CreateContainerConfigError / Missing Secret

The chart never creates the three application Secrets — pods fail with CreateContainerConfigError: secret "boltmcp-database" not found (or -oidc / -auth) until you create them. List what's actually present:

kubectl get secrets -n boltmcp

If any of boltmcp-database, boltmcp-oidc, boltmcp-auth is missing, create it per Cluster Prep → Application Secrets. Pods recover automatically on the next restart loop once the Secret exists.

CrashLoopBackOff

The application is crashing on startup. Check logs:

kubectl logs -n boltmcp <pod-name>

Common causes:

• Wrong database password — the password baked into PostgreSQL on first startup must match the migrate-core-password / web-password / rest-api-password / mcp-server-password / keycloak-password / vault-password in your boltmcp-database Secret. If you rotated a value in the Secret without resetting the corresponding DB user via ALTER USER ... PASSWORD ..., they'll diverge. Reset the password in the database or roll back the Secret value.
• Missing key in a Secret — if the chart references a key that doesn't exist in the user-managed Secret (e.g. mcp-inspector-api-token while mcpInspector.enabled=true), pods fail to start. kubectl describe pod shows the missing key. Edit the Secret to add the key, then kubectl rollout restart deployment/<service>.
• Database not ready — the init container should wait, but verify the database pod is healthy.

ErrImagePull / ImagePullBackOff

Kubernetes cannot pull the container images.

kubectl describe pod -n boltmcp <pod-name>

Verify the image pull secret exists:

kubectl get secrets -n boltmcp | grep boltmcp-pull-secret

If missing, recreate it:

kubectl create secret docker-registry boltmcp-pull-secret \
  -n boltmcp \
  --docker-server=europe-west2-docker.pkg.dev \
  --docker-username=_json_key \
  --docker-password="$(cat ./key.json)"

The chart's default global.imagePullSecrets is [{ name: boltmcp-pull-secret }], so as long as the Secret exists under that name in the install namespace it will be picked up on the next pod restart (a helm upgrade is only required if you used a non-default Secret name and need to override the value).

Connection Refused

• Pod not ready — check pod status with kubectl get pods -n boltmcp
• Service not found — verify services exist with kubectl get svc -n boltmcp
• Ingress or DNS misconfigured — bypass them with port-forwarding (see below) to confirm the pod itself is healthy

Bypass Ingress with Port-Forwarding

If the Ingress, DNS, or TLS layer is misbehaving, port-forward directly to a service to confirm the pod is responding. This is a diagnostic tool, not a normal access path.

# Web app
kubectl port-forward -n boltmcp svc/boltmcp-web 3000:3000

# Keycloak
kubectl port-forward -n boltmcp svc/boltmcp-keycloak 8080:8080

# MCP Server
kubectl port-forward -n boltmcp svc/boltmcp-mcp-server 3001:3001

Note: OIDC redirects will fail when accessed via localhost, since the issuer URL in the values file points at your public Keycloak hostname. Port-forwarding is useful for verifying a single service is up, not for an end-to-end auth flow.

Keycloak's admin UI is a special case: with keycloak.production.enabled: true (the default), KC_HOSTNAME is enforced, so the admin console will load briefly via localhost:8080 and then redirect you to https://auth.boltmcp.example.com. Use the public Keycloak URL for admin work; port-forwarding to Keycloak is only useful for hitting /health/ready to confirm the pod is up.

Authentication Not Working

Issuer URL Mismatch

The OIDC issuer URL must be identical in the browser and in the application pods:

kubectl describe pod <web-pod> -n boltmcp | grep OIDC

Ensure the issuer URL matches the Keycloak hostname exactly (protocol, host, port, path).

On plain-HTTP installs, set global.tls.enabled: false rather than overriding URLs one by one — a lone https:// issuer (the default when only the base URLs are overridden to http://) silently breaks token validation.

Public hostnames don't resolve inside the cluster (split-horizon DNS)

Symptoms: the browser reaches BoltMCP fine, but the web, rest-api, or keycloak-reconcile pods fail with OIDC discovery errors against auth.<domain> — ENOTFOUND, connection timeouts, or ECONNREFUSED 127.0.0.1 (a workstation-only /etc/hosts entry leaking into cluster DNS and resolving to the pod's own loopback).

Cause: the services fetch the OIDC discovery document from the public issuer URL server-side. If the public hostnames only resolve outside the cluster (local installs, split-horizon corporate DNS), those in-cluster requests fail even though everything works in the browser.

Fix: set global.hostAliases so every BoltMCP pod resolves the public hostnames to your ingress:

global:
  hostAliases:
    - ip: "<ingress IP reachable from inside the cluster>"
      hostnames:
        - auth.boltmcp.example.com
        - web.boltmcp.example.com
        - server.boltmcp.example.com

Then helm upgrade with the updated values. No CoreDNS changes are needed.

keycloak-reconcile hook fails with HTTPS required

Symptoms: helm install or helm upgrade fails on the boltmcp-keycloak-reconcile hook (pre-upgrade / post-install), and the hook pod's log ends with:

Keycloak reconcile failed: KeycloakUnavailableError: Keycloak refused admin credentials for master_admin (403): {"error":"invalid_request","error_description":"HTTPS required"}

Cause: the reconcile Job talks to Keycloak over plain HTTP on the in-cluster Service (http://boltmcp-keycloak:8080), because the public hostname may not be routable or have a certificate yet during a fresh install. In production mode (keycloak.production.enabled: true, the default) the master realm's Require SSL setting defaults to external, which waives HTTPS only for requests from loopback or RFC1918 addresses (10/8, 172.16/12, 192.168/16). If your cluster's pod CIDR lies outside those ranges (commonly 100.64.0.0/10), Keycloak treats the Job as an external client and refuses the token request. The boltmcp realm is unaffected: it is only reached through your HTTPS ingress.

Fix: set the master realm's Require SSL to None once. The setting is stored in Keycloak's database, so it survives upgrades and Pod restarts. Either use the admin console at https://auth.<domain>/admin (sign in as master_admin, switch to the master realm, then Realm settings → General → Require SSL → None), or do it from inside the Keycloak pod, which is always allowed to use plain HTTP over loopback:

KC_PASSWORD=$(kubectl get secret boltmcp-auth -n boltmcp -o jsonpath='{.data.keycloak-admin-password}' | base64 -d)
kubectl exec -n boltmcp deploy/boltmcp-keycloak -- \
  /opt/keycloak/bin/kcadm.sh config credentials --server http://127.0.0.1:8080 \
  --realm master --user master_admin --password "$KC_PASSWORD"
kubectl exec -n boltmcp deploy/boltmcp-keycloak -- \
  /opt/keycloak/bin/kcadm.sh update realms/master -s sslRequired=none

Replace boltmcp-auth with the name of your auth Secret if you set secrets.auth.name. Then re-run helm upgrade (or helm install after helm uninstall if the failed install left the release in failed). If you would rather keep Require SSL at external, the alternative is a pod CIDR inside the private ranges; there is no chart value that changes the Job's URL.

Missing Email or Name

BoltMCP requires users to have an email and first name to sign in. The auto-provisioned boltmcp_admin user gets both fields set at realm-import time (email from oidc.adminUser.email, firstName Admin). If you add more users later through the Keycloak admin console, make sure each has both fields populated before they try to sign into the BoltMCP web app.

Client Secret Mismatch

The OIDC client secrets in the boltmcp-oidc Secret must match what's configured on the corresponding clients in Keycloak. There are four client secrets: web-client-secret, mcp-server-client-secret, rest-api-to-keycloak-client-secret and keycloak-reconcile-client-secret (see Rotating Secrets for which client and Deployment each one belongs to). To rotate a value:

1. Edit the boltmcp-oidc Secret (kubectl edit secret boltmcp-oidc -n boltmcp, or re-apply via your secrets manager) so the new value is base64-encoded under the right key.
2. Run helm upgrade with your existing values (see Upgrading). Its keycloak-reconcile hook Job stamps the new secret onto the matching Keycloak client. Alternatively, update the client secret by hand in the Keycloak admin console.
3. Restart deployments so they pick up the new value (Kubernetes does not auto-restart pods on Secret changes):

kubectl rollout restart -n boltmcp deployment/boltmcp-web
kubectl rollout restart -n boltmcp deployment/boltmcp-mcp-server
kubectl rollout restart -n boltmcp deployment/boltmcp-rest-api

Redirect Loop

Check that client redirect URIs in Keycloak match the web URLs. The boltmcp-web client's root URL and redirect URI are derived from web.baseUrl / global.domain and re-applied by the keycloak-reconcile hook Job on every upgrade, so after changing those values run helm upgrade rather than editing the client by hand.

Secret store is unavailable

The REST API's secret endpoints (/api/v1/secrets/*) depend on the bundled Vault being initialized, unsealed, and bootstrapped. The HTTP status tells you which step is missing:

• 503 (Vault not configured) — vault.kubernetesAuth.enabled is off, or the bootstrap hasn't created the auth method/role yet. Run the Vault bootstrap.

• 502 (Vault unavailable) — Vault is sealed or unreachable, or the Kubernetes login was rejected. Check the seal state and unseal if needed:

  kubectl exec -it -n boltmcp deploy/boltmcp-vault -- vault status

  Remember that with the default (Shamir) seal, every Vault pod restart re-seals it — re-run vault operator unseal, or configure auto-unseal.

If login is rejected even when Vault is unsealed and bootstrapped, the most common causes are a projected-token audience that doesn't match the Vault role's audience, or the Vault ServiceAccount lacking the system:auth-delegator binding (it can't run TokenReview). Both are wired by the chart, so check that vault.kubernetesAuth.audience was not overridden inconsistently and inspect the REST API logs:

kubectl logs -n boltmcp deploy/boltmcp-rest-api | grep -i vault

Certificate Issues

Certificate Stuck in False State

kubectl describe certificate boltmcp-tls -n boltmcp
kubectl get challenges -n boltmcp

Common causes:

• DNS not propagated — verify with nslookup web.boltmcp.example.com
• HTTP-01 challenge failed — ensure NGINX ingress is running and accessible
• Rate limited — use the staging ClusterIssuer for testing

Large Header Errors

If Keycloak produces "upstream sent too big header" errors, the auth headers exceed the ingress buffer. The chart-managed Ingress allows 128k by default; raise it in your values file:

ingress:
  annotations:
    nginx.ingress.kubernetes.io/proxy-buffer-size: "256k"

If you run your own Ingress manifest rather than the chart-managed one, set the same annotation there - without it NGINX applies a 4k default.

413 Errors Uploading an API Spec

If importing an OpenAPI spec from the APIs page fails with a 413 error, the file exceeds the platform's 10 MB request-body limit. The BoltMCP REST API enforces this limit on every request (it answers with {"error":"Payload Too Large"}) and the chart-managed Ingress mirrors it with nginx.ingress.kubernetes.io/proxy-body-size: "10m". Raising the ingress annotation does not raise the limit - the fix is a smaller spec, for example by splitting it or removing unused paths.

If you run your own Ingress manifest rather than the chart-managed one, set the annotation there to at least 10m:

nginx.ingress.kubernetes.io/proxy-body-size: "10m"

Without it NGINX applies a 1 MB default and rejects uploads with 413 Request Entity Too Large before they reach the platform.

Database lost+found Error

If the database pod logs show:

initdb: error: directory "/var/lib/postgresql/data" exists but is not empty
initdb: detail: It contains a lost+found directory

The PVC must be recreated:

helm uninstall boltmcp -n boltmcp
kubectl delete pvc data-boltmcp-database-0 -n boltmcp

helm install boltmcp \
  oci://europe-west2-docker.pkg.dev/boltmcp-platform/boltmcp-alpha/charts/boltmcp \
  --version ${BOLTMCP_VERSION} \
  -n boltmcp \
  -f ./config/values-prod.yaml

Diagnostic Commands

# Pod status
kubectl get pods -n boltmcp

# Pod logs
kubectl logs -n boltmcp <pod-name>

# Pod events and details
kubectl describe pod -n boltmcp <pod-name>

# Services and endpoints
kubectl get svc -n boltmcp
kubectl get endpoints -n boltmcp

# Secrets
kubectl get secrets -n boltmcp

# Helm release status
helm list -n boltmcp
helm status boltmcp -n boltmcp

# Certificate status (if using Ingress)
kubectl get certificates -n boltmcp
kubectl get challenges -n boltmcp

# Ingress status
kubectl get ingress -n boltmcp

# NGINX Ingress logs
kubectl logs -n ingress-nginx -l app.kubernetes.io/name=ingress-nginx

Any helm commands require having set shell variable HELM_REGISTRY_CONFIG to point to your boltmcp key.