Get in Tutch

A message to our customers

11th February 2016

On Wednesday 10th February, our hosting provider suffered an interruption in power to their data centre facility during emergency works to fix a fault. This has caused disruption to some customers’ services and our teams are working to resolve remaining issues.

Data centre

The UK-based data centre facility is one of the most efficient and resilient data centres in Europe. They carry out tests on a regular basis and test the generators every weekend to make sure that everything is in order. Additionally, they have expert technicians staffing the data centre 24/7, every day of the year. They have also have two UPS systems serving each data centre hall.

So, what happened?

  • On Wednesday, one of the data centre halls suffered a power loss which affected the facility for less than 9 minutes.
  • Each data centre hall has two UPS (uninterruptable power supplies) which feed into a LTM (load transfer module), which manages the feed of power from the UPS to the data centre hall, where our servers are housed.
  • This piece of hardware automatically switches the power between the two external supplies, should one fail. This is part of their redundancy infrastructure.
  • The LTM showed a fault on the primary power supply and was running on its backup. Their teams followed the guidelines and contacted the manufacturer and an expert in this piece of hardware was sent to the facility to investigate and fix.
  • The fault code indicated that there was a problem with the voltage monitor. As a safety procedure, this automatically shuts down primary power, even though the power supply itself is likely fine.
  • The engineer on site assisted with the fitting of the replacement part. The procedure was then followed to turn off the already disabled switch and to change the part safely. Unfortunately, a safety mechanism in the device triggered incorrectly, which led to the data centre losing power.

Will this happen again?

A fault of this nature is almost unheard of and they will be working with the manufacture of the data centre hardware to ensure that this cannot happen again. We will, of course, keep you updated on any changes following their reviews.

We would like to apologise to all customers who have been affected and we appreciate your patience and understand your frustrations. Their teams have been working around the clock to resolve all remaining issues, and will continue to until everything is restored to our normal high level of service.