Hi,
We are currently experiencing issues with our VZ1 Node, we are looking into this now however it is more than likely this machine will need to be rebooted as it appears to have hung on the console.
More Updates will be posted here as and when we have them.
Update @ 11:12 – The machine has been rebooted and has sucessfully booted back up, VPS are now starting up one by one.
vz1 is currently experiencing high load. We are working to get it under control now
Update: Load is now coming down
xn11 is currently suffering from some I/O issues again, this is currently being worked on. Further updates will follow
Update: The system has been rebooted, the RAID is in a degraded state but there are no signs of any disk issues. The RAID is now rebuilding, and the degraded disk has been noted as it may require replacement. As unlikely as it is, drives are around the same age so it could be a second faulty disk in a month. We will continue to keep a close eye on this server.
xn11 has crashed and is being checked
Update: A drive has failed in the RAID10 array, unfortunatley the RAID card was unable to handle the transfer resulting in the filesystem going to read only mode
Update 2: A replacement drive has been fitted, xn11 is now back online and vps are starting up one by one.
Please expect some reduction in I/O performance for the next few hours while the RAID array rebuilds to an optimal state.
Appologies for the inconvenience.
Planned maintainence work is currently being carried out on XN12, affecting a small portion of customers on the 95.154.207.xx range.
This is a mandatory maintainence window in order to resolve ongoing stability issues.
Update: This has been completed, total downtime approximatley 20 minutes.
There is currently a critical network outage affecting all of our equipment in Spectrum House. We are in contact with our Upstream now and further updates will follow.
Update: Connectivity has been restored, downtime less than 5 minutes. This appears to have been a blip from one of the Core Routers.
RFO From Our Upstream: On Friday 16/07/10 our monitoring systems detected a connectivity problem affecting elements of the network. The issue was quickly escalated and once traced to certain malicious activity, senior network engineers immediately took action to halt the offending traffic.
xn12 and xn13 are currently experiencing stability issues and are being worked on.
xn12 Update: This machine appears to be suffering from hardware issues, due to a number of recent crashes. We will be replacing the RAM in due course and customers will be notified by email in advance of this planned maintainence task.
xn13 Update: This has been isolated to a particular VPS which has been suspended pending investigation.
Our monitoring has just picked up that xn14 has gone down. This is being checked
Update: We are waiting for some onsite hands.
Update 2: Ping requests are intermittently coming through, its looking like a network based attack against the machine or more likely a VPS.
Update 3: Connectivity has been restored to the machine and all VPS, we are now investigating further.
We are currently aware of issues with the XN12 Server, we are working with onsite staff to get this resolved as quick as possible.
Updates will follow here when we have them.
Update @ 11:51 BST – This machine is now back online, it will be monitored very closely over the next few hours and a full investigation into the cause will follow.
Update @ 11:44 BST – This is looking like a hardware failure, its not confirmed yet however if so we have onsite stock of parts should the worst happen.
xn12 appears to have crashed about 5 minutes ago. We are just waiting for some remote hands and will provide updates shortly.
Update: The server is now up and VPS are starting up.