Issue

When syncing data to a Webhooks destination, you may experience delivery failures returning REQUESTTIMEOUTERROR, 503 Service Unavailable, or ETIMEDOUT error messages. 

If your Webhooks destination is connected to a Reverse ETL (rETL) source, you may notice a discrepancy on the Sync Details page. Specifically, the number of errors displayed in the 'Failed Reason' table might appear significantly larger than the total 'Extraction results' for that specific sync run.

Product

Twilio Segment

Environment

Segment Console

Cause

These errors are different symptoms of the same underlying problem: the destination endpoint's server is temporarily unable to handle the volume of requests, likely due to an overloaded server or scheduled maintenance.

• 503 Service Unavailable: This is an active response from the destination server (e.g., Mulesoft) indicating it is online but its processing queues are completely full.
• REQUESTTIMEOUTERROR and ETIMEDOUT: Segment enforces a strict maximum waiting period of about 10 seconds for Webhook deliveries. If the destination takes longer than 10 seconds to process the data and acknowledge receipt, Segment forcibly severs the connection to prevent a backlog and marks the delivery as failed.

For Reverse ETL connections, you may see metric discrepancies in the Sync Details UI because the 'Failed Reason' table logs every individual retry attempt rather than the number of unique records.

Because Segment rapidly retries failed records multiple times over the course of a single sync run, the total number of logged failure attempts can become significantly higher than the actual number of unique failed records extracted from your model.

Resolution

Because these limitations originate from the destination server, you must adjust the flow of data to accommodate the endpoint's capacity.

1. Investigate your destination server environment to identify the root cause of the overload or to confirm if there is ongoing maintenance.
2. Determine the specific rate limits and processing capacity of your webhook endpoint.

3. Reduce the volume of events passing through to your destination:

   • For Standard Event sources: Optimize your source-level tracking implementation to ensure you are only firing necessary event calls. Additionally, utilize destination-level filters or mappings within Segment to guarantee that only essential events are forwarded to the endpoint.

   • For Reverse ETL sources: Navigate to your Segment model, add filters to the current model query, and save it to limit the data extracted per sync.

4. Deploy the changes to prevent the destination server from becoming overwhelmed.

Additional Information 

For Reverse ETL connections, you can safely trust the 'Extraction results' and 'Load results' badges at the top of the Sync Details page for an accurate count of unique records. These badges provide the true total of processed records, completely unaffected by the individual retry attempts logged in the 'Failed Reason' table.