Heart Internet Down 10/02/2016
We have a lot of servers based at Heart Internet and with the problems faced due to Heart Internet being down since 15:05 we’ve decided to provide an update for our clients as information comes available:-
Heart Internet Status 10/02/2016
Wed, 10 Feb 2016 15:05:44 +0000
We suffered a brief interruption in power during works in one of our data centre halls. This has now been restored and all servers are coming back online. We will provide further updates as work progresses every 20 minutes
Wed, 10 Feb 2016 15:50:00 +0000
While a top priority is to keep your websites and services online, our highest priority is the safety of our engineers. When working with equipment such as UPS (massive battery arrays), generators, switching equipment and multiple sources from the mains, we ensure our team are safe. In this case, a voltage sensor was due for replacement, we had switched over to bypass this unit, however for a reason yet unknown a safety warning was triggered which resulted in the shut down of power for about 9 minutes. The full reason is yet unknown, we have our expert data centre team, as well as the equipment providers, working to understand this fully. Right now we can assure you that power is flowing to the data centre as it should be.
Wed, 10 Feb 2016 16:01:00 +0000
Currently our priority is to restore service to infrastructure servers, this is in progress now and will allow us to being work restoring service to the shared platform.
Wed, 10 Feb 2016 16:07:00 +0000
Infrastructure servers are now showing as fully online. Work ongoing on the shared platform and the current ETA for restoration of service is 17:00
Wed, 10 Feb 2016 16:14:00 +0000
Hybrid VPS servers are now being powered on. A tentative ETA for restoration of hybrid servers is 17:30
Wed, 10 Feb 2016 16:38:00 +0000
VPS KVM host servers are now being powered on. Guests are still currently unavailable
Wed, 10 Feb 2016 17:08 +0000
Premium hosting front end webservers are now coming online. Databases are currently still powered down
Wed, 10 Feb 2016 17:37 +0000
Premium hosting database servers are now being powered on.
Wed, 10 Feb 2016 18:09 +0000
Apologies for the delays. Servers are being powered up in stepped process in order to prevent further complications. Once an individual server is powered it needs to be fully checked for unexpected behaviour due to the sudden power loss. Work is ongoing.
Wed, 10 Feb 2016 18:09 +0000
Apologies for the delays. Servers are being powered up in stepped process in order to prevent further complications. Once an individual server is powered it needs to be fully checked for unexpected behaviour due to the sudden power loss. Work is ongoing.
Wed, 10 Feb 2016 19:50 +0000
We have identified a failed switch serving a number of KVM hosts. If your VPS server is entirely inaccessible please await a further update.
Wed, 10 Feb 2016 19:57 +0000
A large portion of shared server sites are now online. Some clusters still have inaccessible databases and backend storage. Work is in progress on the remaining clusters.
Wed, 10 Feb 2016 20:16 +0000
We have identified a problem with webmail backend storage. No stored mail has been lost but the webmail service will be unavailable until restoration from backup is complete.
Wed, 10 Feb 2016 20:48 +0000
If your VPS or Hybrid server is entirely unresponsive (does not respond to ping) please await a further update, KVM hosts are still being powered on. If your server is accessible via SSH but the individual sites hosted on it are offline please contact support.
Wed, 10 Feb 2016 21:07 +0000
Mail delivery should now be completely restored. If you have any outstanding issues please contact support. Please note that while mail servers were offline delivery will depend on the retry time of the sender.
Wed, 10 Feb 2016 22:00 +0000
The following KVM hosts are still unavailable due to a failed switch caused by the power outage.
KVMhost107
KVMhost150 to 155
KVMhost159
KVMhost164 to 167
KVMhost196
KVMhost87
KVMhost89
We also have the following KVM servers in alternate cabinets down:
KVMhost102
KVMhost182
KVMhost249
KVMhost4
KVMhost74
KVMhost88
If your VPS is on a host not listed above but is still offline please contact support
Wed, 10 Feb 2016 22:22 +0000
The switch above has been replaced and all hosts apart from KVMhost88, KVMhost249 and KVMhost74 are now responsive. If your individual guest still has problems please contact support. Please note the support team are dealing with a large influx of tickets but we are working through them chronologically.
Wed, 10 Feb 2016 23:05 +0000
Around 60% of shared sites are now online. Work is continuing on backend storage.
Wed, 10 Feb 2016 23:31 +0000
KVMhost69 is still unresponsive. The sysadmin team are currently investigating the cause
Thu, 11 Feb 2016 00:55 +0000
The following kvmhosts still have an issue –
kvmhost153
kvmhost237
kvmhost4
kvmhost69
kvmhost70
kvmhost74
kvmhost248
kvmhost249
Thu, 11 Feb 2016 01:55 +0000
The following webservers still have an issue – web75, web76, web77 and web78. This should hopefully be resolved shortly. We also discovered problems with MySQL on web21, web22, web23 and web24 which has been resolved.
Thu, 11 Feb 2016 03:00 +0000
Webmail – unfortunately we still have problems with the webmail system, it is sadly likely to be unavailable for several hours. Apologies for the inconvenience caused again.
Thu, 11 Feb 2016 04:00 +0000
We are still working our very hardest to resolve the remaining kvmhost and webserver (web75, web76, web77 and web78) issues, many of our staff have been in the office since 8am yesterday and are still here now. We will continue to keep you updated here.
Thu, 11 Feb 2016 04:41 +0000
kmvhost88 which we discovered had some issues is now working.
Thu, 11 Feb 2016 06:05 +0000
We discovered a problem with the webservers web175, web176, web177 and web178. This has now been resolved, sites will be slow to load for a short while due to the sudden influx of traffic but this will calm down soon.
Thu, 11 Feb 2016 06:45 +0000
The problems with kvmhosts kvmhost4, kvmhost69, kvmhost70, kvmhost74, kvmhost153 (this was actually wrongly reported earlier and was fine), and kvmhost237 have now been resolved. There are still issues with kvmhost248 and kvmhost249 unfortunately that will sadly take at least a few hours to resolve.
Our normal information feed can be found here:-
http://www.chameleonhosting.co.uk/system_status.php
Heart Internet Uptime
In support to Heart Internet we thought we would share our uptime monitoring results that we have on clients sites. We have found that as a hosting company they have been very reliable much better than others we use ourselves and for clients with hosting at various companies. This issue is the same issue we all face. Last week BT was hit and went down, the week before HSBC was hit.
Our uptime monitoring on average across sites is as follows:-
2014 = 99.99%
2015 = 99.98%
2016 = 98.05% – increasing as the down time length continues.
Update for Heart Internet being down 11/02/2016 – http://www.chameleonwebservices.co.uk/heart-internet-still-down-11022016/
