Issue

When running a Reverse ETL sync in the Segment Console, users may encounter failures with the error message:
googleapi: Error 400: Cloud KMS Error: projects/[project]/locations/[location]/keyRings/[keyRing]/cryptoKeys/[key]/cryptoKeyVersions/[version] is not enabled, current state is: DISABLED., invalid.
This issue can cause some models to fail while others succeed, leading to confusion about the root cause.

Product

Twilio Segment

Environment

Segment Console

Cause

Each Reverse ETL (rETL) model in Segment creates its own "State Table" in BigQuery. If a model was initialized when a specific version of a Google Cloud Platform (GCP) Customer-Managed Encryption Key (CMEK) was enabled, it will continue to require that version to decrypt its state. If that key version is later disabled or rotated, only models tied to the disabled version will fail with the above error, while models using newer, enabled key versions will continue to work.

Resolution

To resolve the "cryptoKeyVersion is not enabled" error for affected Reverse ETL models:

1. Identify the failing model(s):
   In the Segment Console, locate the Reverse ETL sync(s) showing the error.

2. Reset or Re-create the Model:

   • Reset the model: This will treat the next scheduled sync as a first sync, re-extracting all data and associating the model with the currently enabled key version.
   • Delete and re-create the model: If a reset is not sufficient or possible, delete the model and create it again. This will also bind it to the active key version.

3. Verify GCP Key Status:

   • In your GCP Console, navigate to Security > Key Management.
   • Ensure the required key version is enabled, or confirm that new models are using an active key version.

4. Check IAM Permissions:

   • Make sure the BigQuery Service Account has the cloudkms.cryptoKeyDecrypter role for the key in question.

5. Re-run the Sync:

   • After resetting or recreating the model, trigger the sync again to confirm the issue is resolved.

Additional Information

• This error does not affect all models because each model may use a different key version depending on when it was created or last reset.
• If you continue to see failures after following these steps, consult your GCP administrator to check for recent key rotations or policy changes.