Incident Summary
During the incident window, some of our nodes went down (so all VPN tunnels to these nodes went down and so DHCP service was not usable)
Status
Resolved
Incident time window
Start: May 11th 2023, at 06:07 UTC
- Problem definition
- Operation to identify the root cause
- Workaround and development of the patch
End: May 11th 2023, at 09:03 UTC
Root cause
This incident was due to an ID conflict (duplicate) in the routing table used by DHCP service and by our IPSEC service (for information - each packet is tagged with an ID to allow us to segment traffic and route it in the right tunnel). As a result, the IPSec traffic was not routed properly on nodes and all tunnels went down.
Impact
All locations using EMEA (node - 87.98.189.202), APAC (51.79.178.35, 139.99.69.206), US (51.81.66.49)
Actions plan
Action 1 - System - Done (11/05/2023, 09:00 UTC)
IPSEC - change the ID of the tunnel (that created the conflict)
Action 2 - System - Done (11/05/2023, 09:00 UTC)
IPSEC - remove the ID that created the conflict to prevent from its re-use