Intermittent downtime

[Fred]

Update: Our host have come back to me with the result of what happened.

*As this is a public thread I have redacted our hosts name, if any member would like to know who they are let me know.

Reason for Outage (RFO)

Quote:

---

Summary:
On the morning of Thursday, April 9, 2026, ********** services experienced a network disruption originating from a hardware failure at our upstream network provider's facility. All  **********-owned network equipment, servers, and storage operated normally throughout the event; the disruption was confined to connectivity between our infrastructure and the wider internet. We are issuing this RFO now that we have received the formal post-incident report from the upstream facility.

Service Impact:
From approximately 9:00 AM ET until approximately 11:30 AM ET, inbound and outbound network connectivity to  ********** services was intermittent or unavailable. Websites, email, and other hosted services became reachable again as upstream connectivity was restored. No data loss occurred on  ********** systems, and no  ********** hardware required recovery or replacement.

Root Cause:
At 8:58 AM ET, a core distribution switch at our upstream provider's facility suffered a hardware failure and entered a reboot loop, repeatedly crashing and restarting. This disrupted network traffic passing through that facility, which is the path by which our infrastructure reaches the internet. The repeated failures prevented a clean failover and caused the service interruption our clients experienced.

Resolution:
Facility engineers were engaged within minutes of the first alerts. At 11:02 AM ET, the faulty switch was isolated and powered off, which restored the majority of traffic. Remaining network paths were fully restored by 12:55 PM ET following a firewall uplink reconfiguration. The failed switch hardware was physically replaced on April 11.

On our side,  ********** staff were engaged throughout the event, monitoring service health, communicating with the upstream provider, and verifying that our systems returned to full function as connectivity was restored.

Going Forward:
While the underlying hardware failure occurred outside our network border, we understand that from a client's perspective the distinction matters less than the disruption itself. We take the reliability of the services we provide seriously, regardless of where in the path a fault occurs. We are reviewing the incident with our upstream provider and evaluating what, if anything, we can do on our end to reduce the client-visible impact of similar events in the future.

Thank you for your patience and for trusting us with your hosting.