Resolving Databricks Sync Failures Caused by Varchar Limit Exceedance – Twilio Support

Issue

Reverse ETL syncs from a Databricks source fail during the extraction phase. The sync details UI may display a generic message such as "Extract failed due to an unexpected error". Checking the internal backend logs reveals a [DELTA_EXCEED_CHAR_VARCHAR_LIMIT] error, indicating that a value "exceeds char/varchar type length limitation" and failed the check ((record_id IS NULL) OR (length(record_id) <= 512)).

Product

Segment

Environment

Segment Console

Cause

Segment uses a state table (typically named __segment_reverse_etl) in the data warehouse to track changes for record diffing. This table enforces a strict 512-character limit on the unique identifier column (record_id). If the unique identifier from the source data contains literal double-quotes, single-quotes, or complex JSON characters, Segment's extraction driver attempts to escape them during state serialization. This can trigger a recursive string-escaping loop, exponentially expanding the string with backslashes until it blows past the 512-character limit, causing the extraction to fail. Simply using a TRIM() or CAST() function in the query is insufficient if literal quotes are embedded within the string itself.

Resolution

To resolve this issue, you must sanitize the unique identifier in the model query and clear the corrupted diffing state.

Navigate to your Reverse ETL Model in the Segment Console and edit the SQL query.
Update the query to explicitly strip single and double quotes from the unique identifier column using the REPLACE() function. For example:

SELECT REPLACE(REPLACE(CAST(id AS STRING), '"', ''), '''', '') AS id

Save the updated model.
Clear the corrupted state in the warehouse by doing one of the following:
- Delete and re-create the Reverse ETL mapping in the Segment Console.
- Or, manually drop the __segment_reverse_etl staging schema directly within your Databricks environment.
Trigger a manual sync in the Segment Console to rebuild the state using the sanitized identifiers.

Additional Information

Because resetting the mapping or dropping the staging schema forces the system to abandon its previous diffing state, the subsequent sync will act as a full, initial sync. This means Segment will extract and send all rows returned by your data model to the destination. Be mindful of your destination's API rate limits and billing implications when triggering this full sync.