On 26 February 2021, Twilio suffered a set of broad-based connection issues. We first began seeing these issues around 5:25 AM PST. Services began recovering around 7:15 AM PST, and most new API traffic was restored to normal by about 7:30 AM PST.
The following products and services were affected:
- Phone Numbers
- Pricing API
- Programmable Chat
- Programmable Fax
- Programmable Messaging
- Super SIM
- Verify API
Please note that Programmable Voice, Elastic SIP Trunking (including voice calls), and Email on our platform were unaffected. For full details, please see this incident on our Status page.
Notice: We will be updating this document with additional product-specific impacts as we continue to investigate the root cause of this incident.
What was the effect on Authy?
One Touch approvals may not have been approved successfully during the impacted timeframe. Additionally, some push authentications may not have been delivered to end users.
What was the effect on Autopilot?
Customers would have experienced increased latency and sporadic API request failures during the impacted timeframe.
What was the effect on Console?
Customers were unable to load Console during the impacted timeframe.
Were customers able to create new Twilio accounts?
No, new account creation was down during the impacted timeframe. Customers who experienced issues should try signing up again.
How do I reach Twilio Customer Support if I am unable to create a ticket via the Twilio Console?
You can always email us at email@example.com. Customers with Business Support or Personalized Support plans can also reach Twilio Customer Support via their dedicated telephone number.
What was the effect on Flex?
During the impacted timeframe, a subset of agents were unable to log in. New chat sessions may have failed to be initiated, and existing chats may have experienced high latency and message failures.
What was the effect on Phone Numbers?
Users could not configure or manage phone numbers via Console during the impacted timeframe. Users may also not have been able to successfully search, buy, or manage phone numbers via the REST API.
What was the effect on Chat?
During the impacted timeframe, users may have experienced failures when attempting to create a new chat. Existing chats may have been subject to large latencies and failures.
What was the effect on Fax?
Around 14% of requests to send faxes did not receive a 2XX response during the impacted timeframe. Other requests do not appear to have been impacted.
What happened to outbound message API requests that were sent to Twilio during the outage?
For some message requests, Twilio’s API returned
HTTP 500 error responses (Twilio error code 20500). Some of these requests actually resulted in a message being successfully created and sent, sometimes with a delay.
Other Message API requests that returned
HTTP 201 were created as expected, and sent successfully.
My users reported receiving duplicate messages, why?
Duplicate messages may have occurred due to customers retrying messages that appeared to fail, but were actually sent.
Some message requests that returned an
HTTP 500 error from Twilio did result in a message being successfully sent. However, customer applications should have still received Twilio's error response. If your application is set up to retry failed API requests when a 500 comes back from Twilio, you may have inadvertently sent duplicate messages to end users.
What happened to inbound messages sent from end users to Twilio during this time?
The majority of inbound messages successfully reached Twilio accounts and customer applications, with some delay. At the time of this writing, our analysis indicates that approximately 30% of inbound messages sent to Twilio during the impacted timeframe failed to reach Twilio. These messages cannot be replayed.
Customers using inbound messaging may have experienced a spike in incoming messages hitting their Twilio accounts and applications toward the end of the timeframe.
What do affected outbound messages look like in my Twilio Console or at the Messages API?
As noted above, some outbound messages were delayed, and will reflect a long queue time in the Message Delivery Steps in Console. In cases where Twilio returned an
HTTP 500 to the API request, but successfully sent the message, the message record appears as usual in Console, and at the /Messages API resource.
Delivery statuses were not affected by this incident. Status Callbacks for tracking message delivery also occurred normally for these messages.
What was the effect on Studio Flows?
Messaging flows may have received multiple delayed messages in "Send and Wait for Reply" widgets. Customers who sent multiple "responses" could have seen Error 81002, as these responses would not have been handled by the appropriate widget.
What was the effect on Studio API?
A small subset of Studio API users may have encountered timeouts when creating new executions during the impacted timeframe.
What was the effect on Super SIM data connectivity?
Devices using Super SIM for data connectivity were able to use data throughout the impacted timeframe. Super SIMs that exceeded their data limit either prior to or during the impacted timeframe were not limited, and would have been able to continue to use data.
What was the effect on Super SIM SMS Commands?
SMS Commands sent from devices were dropped. These will not be persisted, and callbacks to notify you of a received SMS Command will not be resent. SMS Commands to your devices that were either queued or sent during the impacted timeframe that are stuck in those states will not be updated.
What was the effect on Verify?
Some verifications started by SMS may not have been delivered successfully during the impacted timeframe.