We are aware of a potentially service impacting issue. Learn more
Fire in Datacenter, SBG (در حال گذراندن مراحل) ضروری

تاثیر سیستم - dnseu,mxeu

  • 09/03/2021 19:21
  • آخرین بروزرسانی 10/03/2021 02:32

Our servers in Europe, specifically Strasbourg in Central Europe caught fire and forced isolation procedures.

We currently maintain, DNS, MX, WebProxy, Load Balancer and vRack, those services are redundant to our primary clusters in Canada and the U.S.

This might cause performance issues, we are scheduling a move in another datacenter in Europe, but with the fire resources will be tight.

It would be safe to remove, dnseu.zenithmedia.net, mxeu.zenithmedia.net in your DNS but not required.

DNS and Mail are built to handle that failure.

Anycast, corporate email and other enterprise services are still working with no impact.

 

Message from our upstream vendor

```

http://travaux.ovh.net/?do=details&id=49484&

We are currently facing a major incident in our DataCenter of Strasbourg with a fire declared in the building SBG2. 

Firefighters were immediately on the scene but could not control the fire in SBG2.

The whole site has been isolated, which impacts all our services on SBG1, SBG2, SBG3 and SBG4. 

If your production is in Strasbourg, we recommend to activate your Disaster Recovery Plan.

All our teams are fully mobilized along with the firefighters.

We will keep you updated as more information becomes available.

```

 

We will update this notice when we have more information, schedule a migration. No dataloss on our end.

Database Error (در حال گذراندن مراحل) ضروری

تاثیر سرور - Adam - WEBCA

  • 07/03/2021 22:12 - 08/03/2021 00:40
  • آخرین بروزرسانی 08/03/2021 01:01

Issue

We received reports of a database connection error, we investigated and found that the issue was related to write.

In our tests we found that a possible issue was REDIS not writing to disk and all reports used our REDIS backend.

We then found a custom hung script that wrote to system disk filling it.

We deleted the data and stopped the script, we will be updating error correction and process monitoring for that custom solution and how its hooking into the backup system.

Who was effected?

Any application writing to disk. All services were able to server existing data. No actual system downtime, service downtime to redis/mariadb/ftp

How we will solve this issue?

We will offload the compression routine, storage and transfer of backups off the system disk into a dedicated attached storage. preventing the backup system from taking over system storage.

User storage is already offloaded to a cloud attached data system.

 

We already had plans to offload the creation of backups on the system to a separate disk, we had planned to implement this in early 2021. We will finalize our code and push it this week.

 

Offloading Progress

We will update when we finish the migration to the new backup structure.

Placeholder - General Maintenance (گزارش شده) کم

تاثیر سیستم - HaaS

  • 23/10/2018 22:57
  • آخرین بروزرسانی 08/03/2021 00:59

Our grid of general maintenance for your consideration, this is the default rules we follow. Exceptions per server are made based on client requirements.

 

Process Start Time End Time System Load / Priority Comments
Server Reboot Sunday 20:00 Sunday 24:00 Low kernel or security patch
Analytics Daily 1:00 Daily 2:00 Low Push of all analytics to hive
Server Backup 4:00 6:00 Medium Days depend on contract (Daily,Weekly,Monthly)
Server Backup Offsite 6:00 7:00 Medium Offloading to Cold Storage
Server Updates 7:00 7:30 Low Automatic Package Updates

 

This is a general rule of thumb based on average usage. Times are average based on system load.

Placeholder - Server Status (گزارش شده) کم

تاثیر سیستم - HaaS

  • 23/02/2016 01:41
  • آخرین بروزرسانی 21/01/2021 11:56

This is a placeholder. Please ignore. If nothing above this post is displayed, then no issues are currently reported.

You can view a live status of all our servers https://zenithmedia.ca/network-status