Incident Summary
Cloudi-fi identified (February 15th 11:30 UTC+1) an incident - authentication on captive portals (for Guest) and authentication on Admin platform (for Administrators) were not anymore available during 75 minutes before fix.
Status
Resolved
Incident time window
Start: February 15th 2023, at 11:30 UTC+1
- Problem definition
- Operation to identify the root cause
- Workaround and development of the patch
End: February 15th 2023, at 12:45 UTC+1
Root cause
Overload of our API - an unusual volume of API calls coming from multi-teams tenants. As a reminder, our APIs are used for processing our workflows as Guest registrations or Administration authentications.
Impact
Thus, these API calls produced a few queries that took unexpected duration to run, hogging database resources and preventing it from processing any other queries (such as Guests registration or administration authentication).
Actions plan
Action 1 - Monitoring - Done
Monitoring detected an overload on the database without producing a critical alert
Change the alert thresholds to raise a critical alert
Action 2 - Patch - In progress
Optimize queries performed by our Admin API on multi-teams enabled tenants to reduce database stress
Action 3 - WAF - In progress
Activate WAF to reduce unwanted traffic ( bot, monitoring, etc.)