Rotating a secret without breaking what it signed

Sign the request, not the session

The cross-product endpoint used to take a static bearer token. A bearer token leaks once and stays valid forever, and it tells you nothing about when it leaked or who used it.

Instead: HMAC-SHA256 over the timestamp and the request path, compared with hmac.compare_digest so the comparison takes the same time whether the first byte is wrong or the last one is. No database lookup, so verification stays O(1) on the hot path.

Give the signature an expiry

A signature is accepted only inside a five-minute window either side of now. Stale requests are refused, and so are future-dated ones — a detail that is easy to skip and that turns a clock-skew bug into a forgery vector.

The property this buys is the one a bearer token can never have: a leaked signature expires on its own, without anyone noticing the leak or revoking anything.

Two keys, briefly

During rotation we sign with the primary and verify against primary or next, so anything issued before the swap still validates while callers migrate across.

The subtle part is the error handling. The retry against the second key fires only on an invalid-signature error. An expired token is expired regardless of which key signed it, so expiry and missing-claim failures propagate untouched instead of being quietly papered over by a second decode attempt — which would turn a rotation helper into a way to accept expired credentials.

try:
    return jwt.decode(token, secret_current, **kw)
except jwt.InvalidSignatureError:
    # only a DIFFERENT KEY falls through to _NEXT.
    # expired is expired, whichever key signed it.
    if secret_next:
        return jwt.decode(token, secret_next, **kw)
    raise

Make it boring enough to run quarterly

A scheduled AWS Lambda performs the rotation. Its IAM policy grants exactly two actions — GetSecretValue and PutSecretValue — on one secret ARN prefix, and nothing else. Rotation state lives in Secrets Manager alongside the secrets it tracks.

The deploy script deliberately stops before enabling the schedule. You prove the loop with three manual invokes against a real low-stakes secret first, and turn the schedule on only once you trust it. Automation you cannot yet trust is worse than no automation.

Zero-downtime rotation on a quarterly cadence, leak or no leak. The verifier logs which key matched on every success, so you can confirm every caller has migrated before dropping the old secret — the step people skip and then cannot undo.