Incident Summary
Cloudi-fi identified (December 3rd 20:00 CET) an incident - our primary EMEA DHCP node (87.98.189.202) is down.
Status
Resolved
Incident time window
Start: December 3rd 2023, at 20:00 CET
- Problem definition - DONE
- Operation to identify the root cause - DONE
- Workaround and development of the patch - DONE
End: December 4th 2023, at 11:50 CET
Impact
All IPSEC tunnels with our primary EMEA DHCP node (87.98.189.202) are down. We invite you to use our EMEA secondary node 135.125.2.85.
Root cause
The service reset algorithm was not optimised because an excessive number of concurrent configuration initialisations were executed during the service startup. This competition resulted in slowdowns across the entire system, causing timeouts for certain operations and rendering them only partially applied.
Actions plan
Action 1 - December 4th 2023, at 10:30 CET
The initialization process has been modified so that operations are performed sequentially, thereby avoiding the influx of concurrent requests. New optimizations will be implemented in the coming days.