All systems are operational

About This Site

Check for ongoing status updates. Here you can find all information about host systems, data centers, API interfaces and more. Always stay up to date.

Maintenance
Restart - Düsseldorf infrastructure

We are rebooting our infrastructure in Düsseldorf and deploying new updates and security aspects.

The restart is scheduled to take 30 minutes and should not cause any irregularities. During this time, all services running on the infrastructure will be unavailable.

Start: 11.04.2025 - 23:00

Expected end: 11.04.2025 - 23:30

Expansion of the RPKI cluster

We are expanding the RPKI cluster and would like to integrate additional slaves into the cluster. So that we can continue to operate the RPKI certificates.

We currently have a single server in the cluster, which can result in rejection or unavailability of the individual IP networks in the event of failures by our provider. By expanding the cluster, we are addressing this problem and adding two more servers to the cluster - which are operated via multi-sites.

During this period, there may be interruptions to the RPKI certificates. We do not expect this, but we would still like to inform you of any circumstances. Should this be the case, we will always endeavour to deploy the RPKI certificates again immediately.

Start: 15.04.2025 - 01:00 a.m.

Expected end: 15.04.2025 - 03:00 a.m.

Updates for the mail cluster

We will update the mail cluster to the current status and integrate additional security mechanisms in the course of the past updates.

We expect half an hour per node, as we operate nine of them - we assume a maintenance window of five to seven hours.

We carry out the updates in such a way that the availability of a mail server is guaranteed at all times. This means that all important emails can be delivered and sent.

Accordingly, we do not expect any downtime overall.

Start: 17.04.2025 - 01:00 a.m.

Expected end: 17.04.2025 - 08:00 a.m.

Past Incidents

Friday 24th January 2025

No incidents reported

Thursday 23rd January 2025

No incidents reported

Wednesday 22nd January 2025

No incidents reported

Tuesday 21st January 2025

No incidents reported

Monday 20th January 2025

No incidents reported

Sunday 19th January 2025

No incidents reported

Saturday 18th January 2025

No incidents reported

Friday 17th January 2025

No incidents reported

Thursday 16th January 2025

No incidents reported

Wednesday 15th January 2025

No incidents reported

Tuesday 14th January 2025

No incidents reported

Monday 13th January 2025

No incidents reported

Sunday 12th January 2025

No incidents reported

Saturday 11th January 2025

No incidents reported

Friday 10th January 2025

TeamSpeak-Instances / -Servers Package loss in FRA

The ISP, we and our customers have noticed a partial packet loss on the TeamSpeak instances in Frankfurt.

The problem is already being monitored and analysed by the ISP. We currently have no further information.

  • The replacement was completed around 6 pm and switched from the redundancy on the main core.

    We have not noticed any further problems so far.

  • We have received information that the cause can be traced back to a defective module. An employee is already on his way to the data centre to replace it.

    The replacement should not pose a problem during operation and we therefore do not expect it to be affected. The process will probably be carried out within the next 20 minutes to an hour and a half.

    We will continue to keep you up to date.

  • Thursday 9th January 2025

    Schleyer-EDV Persistent problems with our Plesk server

    We have noticed for some time that Plesk has some problems that obviously cannot be solved.

    Accordingly, we will carry out a new installation next weekend. During this time, our websites and administrative interfaces will not be accessible.

    The process does not affect the operation of our services.

  • We have just reinstalled the Plesk instance.

    Why so late? We had made countless adjustments over the past weeks and days. Apparently, however, the server was misused for various illegal purposes, e.g. to run scans and logging services. And to distribute minimal TCP packets to various destinations. The search has taken some time, up to now we have recorded around 37 abuse reports. In addition, some errors were detected in all Plesk configurations in the early evening, which were no longer able to operate our web services.

    Now that the server has been reinstalled and additional security services have been put into operation, we are continuing to monitor the situation closely and are constantly endeavouring to make further changes to our infrastructure for the future. In doing so, it plays an enormous role to phase out the solution: Plesk for our internal purposes.

    The incident is over for the first time. Should we discover further problems or errors, we will initiate a new one.

  • We have just reinstalled the Plesk instance.

    Why so late? We had made countless adjustments over the past weeks and days. Apparently, however, the server was misused for various illegal purposes, e.g. to run scans and logging services. And to distribute minimal TCP packets to various destinations. The search has taken some time, up to now we have recorded around 37 abuse reports. In addition, some errors were detected in all Plesk configurations in the early evening, which were no longer able to operate our web services.

    Now that the server has been reinstalled and additional security services have been put into operation, we are continuing to monitor the situation closely and are constantly endeavouring to make further changes to our infrastructure for the future. In doing so, it plays an enormous role to phase out the solution: Plesk for our internal purposes.

    The incident is over for the first time. Should we discover further problems or errors, we will initiate a new one.

  • We were able to narrow down the problem using a network loop.

    Despite the existing possibility, we still assume a Plesk error and will endeavour to reinstall the entire Plesk instance in the coming night.

    The instance is currently running without any major problems.

  • Wednesday 8th January 2025

    No incidents reported

    Tuesday 7th January 2025

    No incidents reported

    Monday 6th January 2025

    No incidents reported

    Sunday 5th January 2025

    No incidents reported

    Saturday 4th January 2025

    No incidents reported

    Friday 3rd January 2025

    No incidents reported

    Thursday 2nd January 2025

    No incidents reported

    Wednesday 1st January 2025

    No incidents reported

    Tuesday 31st December 2024

    No incidents reported

    Monday 30th December 2024

    No incidents reported

    Sunday 29th December 2024

    No incidents reported

    Saturday 28th December 2024

    myLoc Datacenter (DUS) Failure of our infrastructure in Düsseldorf

    We detected a failure in our infrastructure at 13:57. As we also announced maintenance work from 12:00 noon, we did not assume that this was really serious. After enquiring whether our partners in the data centre would start the necessary work, we were told no a long time later. While we waited for an answer from our partner, we analysed the incident to prevent damage, for example.

    We could not find the direct fault, so we assumed something more serious. After a restart didn't make sense either, we continued to analyse the system. After some time, at around 16:10, we were able to reduce the error to the GRUB boot loader - which had obviously been damaged by the ZFS. In principle, we did not assume that this was the case, as the affected host system was a few days old.

    After a few more hours had passed, the necessary work was carried out by our partner at around 6.30 pm - which also only took about 15 minutes. We continued to analyse in order to fix the problem - which looked difficult after a few more hours in collaboration with various partners. At around 8.30pm we decided to look for a new host system - which we had originally planned to use earlier - and make enquiries. The host system was provisioned at around 10 p.m. - we then took the ZFS apart to find data remnants and compare them with our PBS backups. However, as all the data was still intact on the hard drives, the only option was to transfer the data. Which was done quite quickly.

    However, as the individual virtual server configuration files no longer existed, we had to recreate some of them using our PBS. This went quite quickly and smoothly. At around 02:05, all services were running smoothly again.

    We are taking further internal measures to minimise such cases in the future. Every affected customer will receive compensation from us; in the case of pre-paid products, a four-day credit for the duration of the service; contract and business customers will each receive a €30.00 credit on their next bill.

    As this incident took place between the public holidays and in the middle of the holiday season, we were understaffed and could not act as quickly as we normally do. We would like to apologise wholeheartedly for this.

    We will continue to closely monitor the host systems for a further 48 hours to prevent any incidents.

    Friday 27th December 2024

    No incidents reported

    Thursday 26th December 2024

    No incidents reported

    Wednesday 25th December 2024

    TeamSpeak-Instances / -Servers Packet loss to the TeamSpeak instances in Düsseldorf

    We notice that our two instance nodes have various network problems after a live migration to a larger host system. We have currently identified various route problems that are not arriving correctly - in addition, traceroutes and various MTRs are not producing any results apart from ICMP distortions in some cases.

    We are continuing to look into the problem.

  • We were unable to detect any further abnormalities over 24 hours. It was therefore due to the incorrectly set MAC addresses.

    We would like to apologise to you for this error, such things should not happen.

  • The problem has been found. We were able to trace it back to individual MAC addresses that were set the same in the respective interface.

    We regenerated and reassigned the MACs. We also restarted the affected VMs. Observations are still being made.

  • Tuesday 24th December 2024

    No incidents reported

    Monday 23rd December 2024

    NTT Global Datacenter FRA1 Network fault in NTT

    We do not yet have any information on the fault in the NTT.

  • The network was stable again at around 02:27. All pings and tests are in the normal range and services are no longer affected.

    We continue to monitor our part. An RCA / post mortem from our partner will follow.

  • It seems to be more long-lasting. So far there is no ETA, statement or similar.

    We are continuing to monitor excessively and are waiting for further results. Pings and various tests are going through, but not fully but only partially with a latency of over 250ms.

  • The connections have been partially working since 22:07. Fundamentally, there is no recognisable stability.

    We are still waiting for a statement and a solution to the problem.

  • Schleyer-EDV Plesk error

    Due to a Plesk error, our websites are currently not or only partially accessible.

    We are already investigating the problem.

  • We were able to fix the Plesk problem. However, due to the network problems in NTT, part of our database cluster is inaccessible, which is why some connections are still not working.

  • Sunday 22nd December 2024

    No incidents reported

    Saturday 21st December 2024

    No incidents reported

    Friday 20th December 2024

    No incidents reported

    Thursday 19th December 2024

    No incidents reported

    Wednesday 18th December 2024

    No incidents reported

    Tuesday 17th December 2024

    No incidents reported

    Monday 16th December 2024

    No incidents reported

    Sunday 15th December 2024

    No incidents reported

    Saturday 14th December 2024

    No incidents reported

    Friday 13th December 2024

    No incidents reported

    Thursday 12th December 2024

    No incidents reported

    Wednesday 11th December 2024

    No incidents reported

    Tuesday 10th December 2024

    No incidents reported

    Monday 9th December 2024

    No incidents reported

    Sunday 8th December 2024

    No incidents reported

    Saturday 7th December 2024

    No incidents reported

    Friday 6th December 2024

    No incidents reported

    Thursday 5th December 2024

    No incidents reported

    Wednesday 4th December 2024

    No incidents reported

    Tuesday 3rd December 2024

    No incidents reported

    Monday 2nd December 2024

    No incidents reported

    Sunday 1st December 2024

    No incidents reported

    Saturday 30th November 2024

    No incidents reported

    Friday 29th November 2024

    No incidents reported

    Thursday 28th November 2024

    No incidents reported

    Wednesday 27th November 2024

    No incidents reported

    Tuesday 26th November 2024

    No incidents reported