Gällande System - dnseu,mxeu
Our servers in Europe, specifically Strasbourg in Central Europe caught fire and forced isolation procedures.
We currently maintain, DNS, MX, WebProxy, Load Balancer and vRack, those services are redundant to our primary clusters in Canada and the U.S.
This might cause performance issues, we are scheduling a move in another datacenter in Europe, but with the fire resources will be tight.
It would be safe to remove, dnseu.zenithmedia.net, mxeu.zenithmedia.net in your DNS but not required.
DNS and Mail are built to handle that failure.
Anycast, corporate email and other enterprise services are still working with no impact.
Message from our upstream vendor
```
http://travaux.ovh.net/?do=details&id=49484&
We are currently facing a major incident in our DataCenter of Strasbourg with a fire declared in the building SBG2.
Firefighters were immediately on the scene but could not control the fire in SBG2.
The whole site has been isolated, which impacts all our services on SBG1, SBG2, SBG3 and SBG4.
If your production is in Strasbourg, we recommend to activate your Disaster Recovery Plan.
All our teams are fully mobilized along with the firefighters.
We will keep you updated as more information becomes available.
```
We will update this notice when we have more information, schedule a migration. No dataloss on our end.
Gällande Server - Adam - WEBCA
We received reports of a database connection error, we investigated and found that the issue was related to write.
In our tests we found that a possible issue was REDIS not writing to disk and all reports used our REDIS backend.
We then found a custom hung script that wrote to system disk filling it.
We deleted the data and stopped the script, we will be updating error correction and process monitoring for that custom solution and how its hooking into the backup system.
Any application writing to disk. All services were able to server existing data. No actual system downtime, service downtime to redis/mariadb/ftp
We will offload the compression routine, storage and transfer of backups off the system disk into a dedicated attached storage. preventing the backup system from taking over system storage.
User storage is already offloaded to a cloud attached data system.
We already had plans to offload the creation of backups on the system to a separate disk, we had planned to implement this in early 2021. We will finalize our code and push it this week.
We will update when we finish the migration to the new backup structure.
Gällande System - HaaS
Our grid of general maintenance for your consideration, this is the default rules we follow. Exceptions per server are made based on client requirements.
| Process | Start Time | End Time | System Load / Priority | Comments |
| Server Reboot | Sunday 20:00 | Sunday 24:00 | Low | kernel or security patch |
| Analytics | Daily 1:00 | Daily 2:00 | Low | Push of all analytics to hive |
| Server Backup | 4:00 | 6:00 | Medium | Days depend on contract (Daily,Weekly,Monthly) |
| Server Backup Offsite | 6:00 | 7:00 | Medium | Offloading to Cold Storage |
| Server Updates | 7:00 | 7:30 | Low | Automatic Package Updates |
This is a general rule of thumb based on average usage. Times are average based on system load.
Gällande System - HaaS
This is a placeholder. Please ignore. If nothing above this post is displayed, then no issues are currently reported.
You can view a live status of all our servers https://zenithmedia.ca/network-status
