Cluster A – Login and Sending/Receiving Issues

Original incident notification from OpenSRS
These notifications affect registration, transfer or renewal of domain names, SSL certificates and enterprise email clients.

Affected Service: Cluster A - Login and Sending/Receiving Issues

May 3, 11:59 UTC
Resolved - This incident has been resolved.

May 2, 23:17 UTC
Monitoring - At approximately 5 PM ET (9 PM UTC), our system alerted us to a large-scale DDoS attack to Cluster A of our Hosted Email Service. We effectively mitigated the risks, however the unprecedented size of the attack exposed some limitations in the scalability of our authentication service. Our engineering team moved quickly to identify and address the root problem.

As of the time of this update, normal service operations have been restored, all users are now able to log in without issue, and all new send requests are being processed as expected. The remaining backlog of inbound email requests may take up to 12 hours to clear.

Having addressed the root issue, we are marking this incident as closed. We sincerely apologize for the impact this service interruption has had. In the coming weeks, we will release an Incident Report detailing the what happened, how we approached and resolved it, and how we will perform better in the future.

May 2, 19:02 UTC
Update - Our system is now actively processing backlogged email requests. Users may continue to experience intermittent login issues and delays in sending and receiving email over the next few hours. By 7 PM ET, we expect the majority of the backlog to be cleared and for our email service to be functioning normally.

We will provide another update by 7 PM ET.

May 2, 18:20 UTC
Update - Users are still experiencing login issues as well as delays and failures to send and receive email. We're continuing the recovery process to restore service.

We will provide another update by 3 PM ET.

May 2, 17:00 UTC
Update - Users are still experiencing login issues as well as delays and failures to send and receive email. We’re continuing the recovery process to restore service.

We will provide another update by 2 PM ET.

May 2, 16:00 UTC
Update - Users are still experiencing login issues as well as delays and failures to send and receive email. We’re continuing the recovery process to restore service.

We will provide another update by 1 PM ET.

May 2, 14:30 UTC
Update - On May 1, 2023, at approximately 5 PM ET, our system alerted us to an attempted large-scale DDoS attack to Cluster A of our Hosted Email Service. We safely mitigated this attack.

However, the large volume of requests caused an interruption to our authentication service. We are now gradually ramping up our email service in accordance with our recovery procedure. No data has been or will be lost.

We will provide another update by 12 PM ET.

May 2, 13:11 UTC
Update - Restoration efforts are underway. There will be no data or email loss. Both inbound and outbound emails will be delivered gradually upon full restoration of the service.

We will provide another update by 11 AM ET.

May 2, 12:20 UTC
Update - Our Email team continues their work towards a resolution.

We apologize again for the inconvenience while we try to get this issue resolved.

May 2, 10:57 UTC
Update - Our email team continues to work as quickly as possible to resolve the Cluster A issues. We currently are not able to offer an ETR at this time, but will continue to keep you updated. We appreciate your patience.

May 2, 09:46 UTC
Update - Our teams are still working hard to resolve the Cluster A issues. We will continue to update you with any new information or resolution.
We appreciate your patience.

May 2, 08:26 UTC
Update - We appreciate your patience as our team continues to work to resolve email on Cluster A. We are working as fast as possible to restore services and will continue to update with any new information.

May 2, 07:26 UTC
Update - Our teams are still working hard to resolve the Cluster A issues.
We will provide an update once we know more.

May 2, 06:37 UTC
Update - Our teams are still working hard to resolve the Cluster A issues. We will continue to update you with any new information or resolution.
We appreciate your patience.

May 2, 05:37 UTC
Update - Our team continues to work to resolve the Cluster A Email issues, we will continue to update you when we have more information or the issue has been resolved. We appreciate your patience.

May 2, 04:04 UTC
Update - Nothing new to report at the moment. Our team is still hard at work trying to solve the problem.

Next update: within 60 minutes

May 2, 02:47 UTC
Update - We are continuing to work on a fix for this issue.

May 2, 02:39 UTC
Update - our engineering team continues to resolve the issue, we appreciate your patience and will provide updates as they become available.

May 2, 00:22 UTC
Update - We are still working on this and will update again when there is more progress.

May 1, 23:01 UTC
Update - We apologize for the inconvenience and are working to resolve the issue as quickly as possible.

May 1, 22:07 UTC
Identified - We think we've found the problem, and we're working on a fix. Please stand by.

May 1, 21:10 UTC
Update - This impacts IMAP and POP connections as well.

May 1, 21:04 UTC
Investigating - Clients will have difficulties logging into the cluster A webmail, as well as sending/receiving emails. Our engineering team has been engaged.

We will provide an update once we have additional information.

Visit our product pages for details and methods of registration, transfer and renewal of your Domain name, SSL Certificate or Enterprise Email Service.

Have an issue thats being affected by this maintenance? Open a Ticket and we can help you find a solution.