The Windows 2008 R2 hardware node VZWIN11 is exhibiting stability issues and rebooting frequently this morning.

Containers are mostly offline but are occasionally available for short periods.

We are investigating possible hardware issues and will update around 9:00

Affected VPS IP addresses are:

79.170.247.1
79.170.247.10
79.170.247.11
79.170.247.16
79.170.247.20
79.170.247.23
79.170.247.31
79.170.247.38
79.170.247.40
79.170.247.45
79.170.247.54
79.170.247.55
79.170.247.59

[Update 09:10]

We have restarted the node and run some hardware diagnostics that check out OK. We are bringing up containers one by one so all should be running normally over the next 15 minutes.

[Update 09:45]

All containers on the affected node should be running normally.

[Update 11:45]

We have seen a similar issue affect hardware node VZWIN12 within the last hour although it is now stable again.

We think this may be related to recent buggy patches released by Intel/Microsoft in relation to the recent Intel CPU security issues Meltdown/Spectre.

For now we will monitor the situation to understand if there is a specific set of hardware/software/other circumstances that trigger these resets and will apply updated patches as and when they become available for servers.

[Update 06/02/17 02:25]

We have applied new vendor supplied patches to all Windows VPS servers.

This week the industry has become aware of a number of vulnerabilities in modern CPU's manufactured by Intel, AMD and some other vendors that could allow software running on these devices to access protected memory on servers, desktops etc bypassing inbuilt security mechanisms that operating systems have been relying on.

This is a complicated issue affecting just about every laptop, desktop, server running Windows, Linux, iOS as well as many mobile devices. While it does not of itself directly allow a device to be compromised, it is rated as a very serious information disclosure especially in multi-tenant environments that could potentially allow sensitive in-memory information such as passwords, user data etc. be harvested.

• CVE-2017-5753: bounds check bypass
• CVE-2017-5715: branch target injection
• CVE-2017-5754: rogue data cache load

Operating system and virtualization vendors are working on emergency patches/releases for these issues that will be made available in the coming days. These will be applied to all services/servers as and when they become available and are shown to be stable. Updating/patching/rebooting will take place usually between 10pm and 4am.

Legacy Linux shared hosting is implemented on Virtuozzo Containers for Linux so updates will be applied out of hours as and when new kernel images are made available by Virtuozzo. Websites may be affected for periods of time as individual servers are rebooted.

NG shared hosting is implemented on Cloud Linux. Kernel upgrades when available will be applied to individual nodes in rotation. In general this will not affect services except for a short time when load balancing nodes are restarted.

Legacy Windows shares hosting is implemented on Virtuozzo Containers for Windows and will be updated out of hours as and when new WSUS updates are made available by Virtuozzo.

Cloud VPS Linux/Windows will be updated out of hours as and when new kernel images or WSUS updates are made available by Virtuozzo. There will be several short duration outages of about 15 minutes when patches applied and servers are rebooted (typically two server reboots are required for VPS node updates).

[Update 5/1/18]

Microsoft updates are available and have been applied to all Windows non-VPS servers.

[Update 6/1/18]

Cloud Linux have released patches for affected kernels and NG web hosting has been upgraded.

[Update 8/1/18]

Virtuozzo have released new kernel versions for Linux and these have been applied to all Cloud VPS Linux and Legacy shared hosting nodes.

[Update 9/1/18]

Virtuozzo have released Windows WSUS updates for some versions and these have been applied where relevant. Updates are still pending for some older Cloud VPS Windows servers and will be applied when available.

Note: Some patches are preliminary and designed to address the main known vulnerabilities however we expect edge cases to remain and further patches may be released by vendors over the coming days/weeks to address these. We will apply updates/patches as and when they become available (where possible they will be applied between 10pm and 4am).

[Update 11/1/18]

All recommended vendor supplied patches have been applied.

Due to maintenance on Data Center power systems, there will be a risk of interruption to some services between 22:00 on 29/11/17 and 06:00 30/11/2017

This could affect some web based services including Cloud Linux shared hosting and control panel access for periods of time.

Email, VPS and legacy Linux/Windows hosting services are not expected to be affected.

Update [04:30]

All maintenance is completed.

We are currently experiencing some network connectivity issues affecting all services.

These are under investigation and we will update here in about 45 minutes.

[Update 04:30]

Problem appears to be related to power outage on Data Center A side.

Normally this is not a problem but it seems to have caused issues on a couple of network devices that is affecting several services.

We expect to have connectivity restored to most services progressively over the next 30-45 minutes.

[Update 05:15]

Email, VPS and Legacy Windows/Linux hosting are all functioning normally, we expect remaining services to restore within the next few minutes.

[Update 05:50]