π οΈ Incident Report
Date: Tuesday, September 9, 2025
Duration: 14:50 β 18:30 (Paris time)
Impact: ~23% of traffic
π Summary
On September 9, Fasterize performed a planned network maintenance to replace an obsolete private network. As part of this operation, a new private IP address was introduced for our load balancers while keeping the existing ones in place.
Later that afternoon, a DDoS attack led to the saturation and restart of one of the load balancers. Following this restart, the load balancer started using the new private IP address to communicate with the backend servers. Since this IP had not yet been registered as a trusted proxy, some requests were processed incorrectly.
π Impact
- ~23% of traffic was affected between 14:50 and 18:30.
- Some users encountered 403 errors.
- In certain cases, client IP addresses were not correctly identified in the headers.
π
Timeline
- 10:25 β New private IP address added to the load balancers.
- 14:49 β DDoS attack causes one load balancer to restart.
- 14:50 β First incorrect headers observed (undetected at this stage).
- 16:02 β Support receives a ticket mentioning 403 errors.
- 17:16 β Ticket escalated to the platform team.
- 18:16 β Root cause identified and linked to the network update.
- 18:30 β New IP address added to the trusted list; incident resolved.
Total duration of impact: 3h40
π Root Cause
The new private IP address introduced during the maintenance was not yet included in the trusted proxy configuration. When one load balancer restarted, it began using this IP, leading to incorrect handling of client headers.
π Next Steps
- All load balancer IPs have now been added to the trusted configuration across environments.
- Documentation has been updated to include this step in future network changes.
- Additional monitoring will be set up to detect unexpected private IPs in client headers.
- Escalation procedures between support and platform teams will be reviewed to ensure faster response times.