KuberCoin Docs

Secret Rotation Runbook

How to rotate HMAC keys, DB passwords, RPC tokens, and admin credentials.

Secret Rotation Runbook

This runbook documents the rotation procedure for every long-lived secret in the KuberCoin operational footprint. Routine rotations are performed quarterly; incident-driven rotations follow the same steps but compress timelines aggressively.

Secret inventory

KindWhere it livesRotation cadenceBlast radius
HMAC signing keys (alerter, webhook delivery)config.php on each surfaceQuarterlyOne subscriber pool
Database passwordsconfig.php; managed user in MariaDBQuarterlyOne surface's read/write access
RPC metrics_tokenrpc/config.phpQuarterlyPrometheus scrape access
Explorer admin bcrypt hashexplorer/config.phpOn admin departure or annuallyExplorer admin UI
TLS certificatesReverse proxy (caddy / nginx)Automated via ACME (Let's Encrypt, 60d)Surface availability

Rotating HMAC keys

  1. Generate a new 32-byte key: openssl rand -hex 32.
  2. Add it as $config['hmac_keys']['v<n+1>'] alongside the existing key. The dispatcher accepts any active key id; verifiers prefer the highest version.
  3. Deploy the dual-key configuration to every surface that signs or verifies, and wait one full delivery window (24h for webhooks).
  4. Switch the active signing key id to v<n+1>.
  5. After 7 days with no verification failures using the old key, remove v<n> from configuration.

Rotating database passwords

  1. Create a new MariaDB user with the same grants: CREATE USER 'svc_rw_v2'@'%' IDENTIFIED BY '...'; GRANT ... ON db.* TO 'svc_rw_v2'@'%';.
  2. Roll the surface to the new credentials via configuration management; verify /readyz reports db: ok.
  3. After 24h with no errors, DROP USER 'svc_rw_v1'@'%';.

Never edit the currently active user's password in place — the moment of rotation drops every in-flight connection and triggers a thundering herd of reconnects.

Rotating the RPC metrics_token

  1. Generate a new token: openssl rand -hex 24.
  2. Add it as $config['metrics_tokens'][] = '...';. The endpoint accepts any token in the array.
  3. Update Prometheus scrape configuration to use the new token.
  4. After one scrape interval (15s) with no 401 in the access log, remove the old token from the array.

Rotating the explorer admin credential

  1. Generate a new bcrypt hash: php -r 'echo password_hash("...", PASSWORD_BCRYPT, ["cost" => 12]);'.
  2. Replace $config['admin_bcrypt']; sessions tied to the old credential remain valid until expiry.
  3. Invalidate active sessions by rotating the session signing key in the same change.

Incident-driven rotation

When a secret is suspected to have leaked, skip the dual-key warm-up window and revoke the old credential immediately. Accept the short-lived availability impact as the cost of containment. File a post-mortem within seven days describing exposure window, mitigations and detection improvements.

Observability

Each surface should expose a secret_age_seconds{kind,name} gauge, populated from a Prometheus textfile collector seeded by the rotation tooling. Alert when any value exceeds the cadence in the inventory above by more than 10%.