tag:status.fasterize.com,2005:/historyFasterize Status - Incident History2024-03-29T08:51:16+01:00Fasterizetag:status.fasterize.com,2005:Incident/193320312023-12-05T22:30:33+01:002023-12-05T22:40:52+01:00Deployment on the Europe platform<p><small>Dec <var data-var='date'> 5</var>, <var data-var='time'>22:40</var> CET</small><br><strong>Update</strong> - The scheduled maintenance has been completed.</p><p><small>Dec <var data-var='date'> 5</var>, <var data-var='time'>22:30</var> CET</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Dec <var data-var='date'> 5</var>, <var data-var='time'>22:25</var> CET</small><br><strong>Verifying</strong> - Verification is currently underway for the maintenance items.</p><p><small>Dec <var data-var='date'> 5</var>, <var data-var='time'>22:00</var> CET</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Dec <var data-var='date'> 5</var>, <var data-var='time'>12:33</var> CET</small><br><strong>Scheduled</strong> - At 22:00, update of the database holding page artifacts. We anticipate a maintenance period of 30 minutes.</p>tag:status.fasterize.com,2005:Incident/190024822023-11-03T01:06:14+01:002023-11-03T01:06:14+01:00Fasterize issue impacts website features<p><small>Nov <var data-var='date'> 3</var>, <var data-var='time'>01:06</var> CET</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Nov <var data-var='date'> 2</var>, <var data-var='time'>19:07</var> CET</small><br><strong>Monitoring</strong> - The rollback is done. We are still monitoring the platform.</p><p><small>Nov <var data-var='date'> 2</var>, <var data-var='time'>18:57</var> CET</small><br><strong>Investigating</strong> - Some customers notified us about broken features on their websites after 17h00 (Paris time). We correlate the incident with a release done this afternoon. We are rolling back the changes on our platform. We expected the end of rollback around 19h15.</p>tag:status.fasterize.com,2005:Incident/188554312023-10-20T09:22:01+02:002023-10-23T23:31:33+02:00Performance degradation<p><small>Oct <var data-var='date'>20</var>, <var data-var='time'>09:22</var> CEST</small><br><strong>Resolved</strong> - This incident has been resolved at 18h25 (Paris time). A post mortem will follow.</p><p><small>Oct <var data-var='date'>19</var>, <var data-var='time'>18:49</var> CEST</small><br><strong>Monitoring</strong> - We're monitoring the results but everything's fine. Seems to be related to a schema change in a storage component (to be confirmed after the RCA).</p><p><small>Oct <var data-var='date'>19</var>, <var data-var='time'>18:35</var> CEST</small><br><strong>Identified</strong> - We have mitigated the issue. Performance is back to normal. Still investigating for the root cause.</p><p><small>Oct <var data-var='date'>19</var>, <var data-var='time'>18:04</var> CEST</small><br><strong>Investigating</strong> - We currently have some issues on our european infrastructure. Being fixed. Slight impact on acceleration. Some pages can have some slowdowns. Some optimizations are disabled.</p>tag:status.fasterize.com,2005:Incident/186783912023-10-03T17:49:54+02:002023-10-04T18:26:59+02:00Traffic temporarily redirected to origin<p><small>Oct <var data-var='date'> 3</var>, <var data-var='time'>17:49</var> CEST</small><br><strong>Resolved</strong> - This incident has been resolved since 16h35. A postmortem is in construction and will be available tomorrow.</p><p><small>Oct <var data-var='date'> 3</var>, <var data-var='time'>16:22</var> CEST</small><br><strong>Monitoring</strong> - The situation is restored, all traffic is now routing through Fasterize as expected. A postmortem of the incident will shortly be provided</p><p><small>Oct <var data-var='date'> 3</var>, <var data-var='time'>16:16</var> CEST</small><br><strong>Identified</strong> - Traffic is temporarily redirected to the origin of customer websites, the situation will be restored very soon.</p>tag:status.fasterize.com,2005:Incident/181630942023-08-16T11:30:00+02:002023-08-16T16:11:59+02:00Unavailability of caches due to a DDOS attack<p><small>Aug <var data-var='date'>16</var>, <var data-var='time'>11:30</var> CEST</small><br><strong>Resolved</strong> - A DDOS attack made our cache layer for pages unavailable between 11:38 and 11:44.</p>tag:status.fasterize.com,2005:Incident/174396432023-06-01T09:19:09+02:002023-06-01T09:19:10+02:00Mise à jour du nouveau dashboard Fasterize<p><small>Jun <var data-var='date'> 1</var>, <var data-var='time'>09:19</var> CEST</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Jun <var data-var='date'> 1</var>, <var data-var='time'>09:13</var> CEST</small><br><strong>Update</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Jun <var data-var='date'> 1</var>, <var data-var='time'>09:03</var> CEST</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Jun <var data-var='date'> 1</var>, <var data-var='time'>09:01</var> CEST</small><br><strong>Scheduled</strong> - We will be undergoing scheduled maintenance during this time.</p>tag:status.fasterize.com,2005:Incident/172800152023-05-17T18:00:10+02:002023-05-17T18:00:10+02:00We are facing intermittent DDOS attacks<p><small>May <var data-var='date'>17</var>, <var data-var='time'>18:00</var> CEST</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>May <var data-var='date'>17</var>, <var data-var='time'>17:43</var> CEST</small><br><strong>Identified</strong> - At 5:05pm (CEST) and 5:20pm (CEST), we've seen performance degradation of our frontals servers due to multiple DDOS attacks.<br />Our network protection are mitigating them and we are adding several other rules in order to block them all.<br />If our protection is not sufficient enough, we will reroute impacted domain's traffic to their origin by bypassing fasterize.</p>tag:status.fasterize.com,2005:Incident/172267842023-05-12T12:52:13+02:002023-05-12T12:52:13+02:00Fasterize API unavailable<p><small>May <var data-var='date'>12</var>, <var data-var='time'>12:52</var> CEST</small><br><strong>Resolved</strong> - This incident is now resolved.</p><p><small>May <var data-var='date'>12</var>, <var data-var='time'>11:17</var> CEST</small><br><strong>Monitoring</strong> - The fix is now deployed for the purge API. Everything is back to normal but we're still monitoring the results.</p><p><small>May <var data-var='date'>12</var>, <var data-var='time'>11:14</var> CEST</small><br><strong>Update</strong> - We are continuing to work on a fix. Dashboards are now ok. Purge API still WIP.</p><p><small>May <var data-var='date'>12</var>, <var data-var='time'>10:58</var> CEST</small><br><strong>Identified</strong> - The issue has been identified and is being fixed. ETA < 1h</p><p><small>May <var data-var='date'>12</var>, <var data-var='time'>10:49</var> CEST</small><br><strong>Investigating</strong> - We currently have some issues on our API servers. Being investigated. It might be impossible to access dashboards, update rules or flush the cache. No impact on acceleration.</p>tag:status.fasterize.com,2005:Incident/172079662023-05-10T18:31:51+02:002023-05-10T18:31:51+02:00Issue on the image display<p><small>May <var data-var='date'>10</var>, <var data-var='time'>18:31</var> CEST</small><br><strong>Resolved</strong> - The rollback completed at 5:11 pm. We will conduct an investigation and improve our tests suite to improve coverage on the impacted feature. We are sorry for the inconvenience.</p><p><small>May <var data-var='date'>10</var>, <var data-var='time'>17:02</var> CEST</small><br><strong>Identified</strong> - We are investigating an issue on images not correctly displayed since 2pm. The issue seems related to the feature “Improve CLS with img dimensions attributes”. A rollback of changes introduced this afternoon on the engine is on going. As a temporaray workaround, we disabled this feature on impacted websites.</p>tag:status.fasterize.com,2005:Incident/171447512023-05-04T18:25:48+02:002023-05-05T13:12:08+02:00Configuration issues<p><small>May <var data-var='date'> 4</var>, <var data-var='time'>18:25</var> CEST</small><br><strong>Resolved</strong> - The rollback has been completed. We will publish a postmortem tomorrow (5th of May) to clarify the root cause of the incident.</p><p><small>May <var data-var='date'> 4</var>, <var data-var='time'>17:49</var> CEST</small><br><strong>Monitoring</strong> - No more errors are being generated since 5:42pm (UTC+2) and 90% less errors since 5:30pm. Impacts were limited to websites using config version 2.</p><p><small>May <var data-var='date'> 4</var>, <var data-var='time'>17:37</var> CEST</small><br><strong>Identified</strong> - The issue has been identified. A mitigation has been deployed to replace faulty proxies. The root cause is being diagnosed.</p><p><small>May <var data-var='date'> 4</var>, <var data-var='time'>17:36</var> CEST</small><br><strong>Investigating</strong> - Some proxies has loaded invalid configurations and cannot process incoming requests.<br />We are currently investigating this issue.</p>tag:status.fasterize.com,2005:Incident/167567162023-04-05T11:59:02+02:002023-04-05T11:59:02+02:00Deployment on the Europe platform<p><small>Apr <var data-var='date'> 5</var>, <var data-var='time'>11:59</var> CEST</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Apr <var data-var='date'> 5</var>, <var data-var='time'>09:45</var> CEST</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Apr <var data-var='date'> 5</var>, <var data-var='time'>09:18</var> CEST</small><br><strong>Scheduled</strong> - Various fix of the engine.</p>tag:status.fasterize.com,2005:Incident/164004212023-03-09T10:00:00+01:002023-03-09T12:56:13+01:00Purge issue on CDN via Dashboard and API<p><small>Mar <var data-var='date'> 9</var>, <var data-var='time'>10:00</var> CET</small><br><strong>Resolved</strong> - Between 10:00 AM (GMT+1) and 12h45AM (GMT+1), the purge on the CDN was not working as expected. We found the issue and apply a fix on the API. This is now resolved. Sorry for the inconvenience</p>tag:status.fasterize.com,2005:Incident/163905362023-03-08T07:00:00+01:002023-03-08T17:27:10+01:00Performance degradation<p><small>Mar <var data-var='date'> 8</var>, <var data-var='time'>07:00</var> CET</small><br><strong>Resolved</strong> - Starting from approximately 7am (CEST) and until 3pm (CEST), we've seen performance degradation of our engine due to a defect in the autoscaling process. Some users / websites might have noticed some slight slowdown on non-cacheable objects of the order of a couple hundred ms. This is now resolved. Sorry for the inconvenience</p>tag:status.fasterize.com,2005:Incident/162334662023-02-22T14:55:38+01:002023-02-22T14:55:38+01:00Security implementation causes 429 errors<p><small>Feb <var data-var='date'>22</var>, <var data-var='time'>14:55</var> CET</small><br><strong>Resolved</strong> - This incident is now closed. Sorry for the inconvenience.</p><p><small>Feb <var data-var='date'>22</var>, <var data-var='time'>12:16</var> CET</small><br><strong>Monitoring</strong> - The fix has been rolled out. There are no more 429 errors.</p><p><small>Feb <var data-var='date'>22</var>, <var data-var='time'>11:43</var> CET</small><br><strong>Identified</strong> - Several requests returns a 429 status code during a security implementation (rate limiting) in order to protect the platform against DDOS attacks.<br />We are currently fixing the issue.</p>tag:status.fasterize.com,2005:Incident/161660792023-02-16T18:00:38+01:002023-02-17T18:32:01+01:00Acceleration is disabled.<p><small>Feb <var data-var='date'>16</var>, <var data-var='time'>18:00</var> CET</small><br><strong>Resolved</strong> - This incident has been resolved at 17:42. We will provide a post mortem tomorrow (16/02/2023).</p><p><small>Feb <var data-var='date'>16</var>, <var data-var='time'>17:49</var> CET</small><br><strong>Monitoring</strong> - The issue is identified. It was related to an attack that has been mitigated.</p><p><small>Feb <var data-var='date'>16</var>, <var data-var='time'>17:12</var> CET</small><br><strong>Investigating</strong> - We currently have some issues on our european infrastructure. Being fixed. Speeding-up is disabled but trafic is ok.</p>tag:status.fasterize.com,2005:Incident/160792232023-02-10T09:00:51+01:002023-02-10T09:00:51+01:00Upgrade of the cluster supporting our API and dashboard.<p><small>Feb <var data-var='date'>10</var>, <var data-var='time'>09:00</var> CET</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Feb <var data-var='date'>10</var>, <var data-var='time'>08:30</var> CET</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Feb <var data-var='date'> 9</var>, <var data-var='time'>18:24</var> CET</small><br><strong>Scheduled</strong> - No code changes to expect, However, this will affect the use of our API and Dashboard for a few minutes</p>tag:status.fasterize.com,2005:Incident/159195422023-01-24T13:00:57+01:002023-01-24T13:00:57+01:00Deployment on the Europe platform (engine update)<p><small>Jan <var data-var='date'>24</var>, <var data-var='time'>13:00</var> CET</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Jan <var data-var='date'>24</var>, <var data-var='time'>11:00</var> CET</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Jan <var data-var='date'>24</var>, <var data-var='time'>10:22</var> CET</small><br><strong>Scheduled</strong> - An optimization engine update is scheduled for January 24 at 11 a.m. to apply improvements and fixes.<br />Une mise à jour du moteur d'optimisation est prévue le 24 janvier à 11h afin d'appliquer des améliorations et des correctifs.</p>tag:status.fasterize.com,2005:Incident/157648352023-01-05T12:42:57+01:002023-01-05T12:42:57+01:00Deployment on the Europe platform<p><small>Jan <var data-var='date'> 5</var>, <var data-var='time'>12:42</var> CET</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Jan <var data-var='date'> 5</var>, <var data-var='time'>09:00</var> CET</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Jan <var data-var='date'> 3</var>, <var data-var='time'>17:41</var> CET</small><br><strong>Scheduled</strong> - An update of our optimization engine is scheduled on Thursday 05/01 at 9am in order to deploy improvements and fixes.</p>tag:status.fasterize.com,2005:Incident/149811242022-12-12T15:00:07+01:002022-12-12T15:00:07+01:00Mise à jour du moteur d'optimisation<p><small>Dec <var data-var='date'>12</var>, <var data-var='time'>15:00</var> CET</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Dec <var data-var='date'>12</var>, <var data-var='time'>13:00</var> CET</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Dec <var data-var='date'>12</var>, <var data-var='time'>09:49</var> CET</small><br><strong>Scheduled</strong> - Une mise à jour du moteur d'optimisation est prévue aujourd'hui à 14h afin d'appliquer des améliorations et des correctifs.</p>tag:status.fasterize.com,2005:Incident/133405612022-11-18T11:59:18+01:002022-11-18T11:59:19+01:00Performance degradation<p><small>Nov <var data-var='date'>18</var>, <var data-var='time'>11:59</var> CET</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Nov <var data-var='date'>18</var>, <var data-var='time'>11:50</var> CET</small><br><strong>Monitoring</strong> - Fix has been deployed, acceleration is back to normal. We're still monitoring. Sorry for the inconvenience</p><p><small>Nov <var data-var='date'>18</var>, <var data-var='time'>11:36</var> CET</small><br><strong>Identified</strong> - Issue has been identified. Fix is being deployed.<br />ETA: 11:50 UTC+1</p><p><small>Nov <var data-var='date'>18</var>, <var data-var='time'>11:27</var> CET</small><br><strong>Investigating</strong> - We currently have some issues on our european infrastructure. Being fixed. Slight impact on acceleration. Some pages can have some slowdowns.</p>tag:status.fasterize.com,2005:Incident/113108682022-10-24T10:39:26+02:002022-10-24T10:39:27+02:00Errors 502 from the platform<p><small>Oct <var data-var='date'>24</var>, <var data-var='time'>10:39</var> CEST</small><br><strong>Resolved</strong> - This incident has been now resolved by progressively deploying the updated module and monitoring the results in the last few days.<br />Reminder: before that, a workaround was deployed to mitigate errors.</p><p><small>Sep <var data-var='date'>30</var>, <var data-var='time'>16:49</var> CEST</small><br><strong>Monitoring</strong> - The fix has been deployed. We have identified the source of the problem, which is an external integration provided by a partner. We will work with our partner to update/fix the offending module.</p><p><small>Sep <var data-var='date'>29</var>, <var data-var='time'>16:18</var> CEST</small><br><strong>Update</strong> - A workaround is being deployed. However, while we have identified the cause of the 502 increase, we have yet to determine the root cause. We're actively investigating this topic.</p><p><small>Sep <var data-var='date'>29</var>, <var data-var='time'>10:53</var> CEST</small><br><strong>Update</strong> - We are continuing to work on a fix for this issue.</p><p><small>Sep <var data-var='date'>29</var>, <var data-var='time'>10:44</var> CEST</small><br><strong>Identified</strong> - We are currently experiencing a platform issue. <br />We noticed an increase in the number of TCP Resets since ~10 days on our load balancer.<br /><br />This issue has generated an increase of the level of 502 errors. Within the last hours, it represents 0.03% of the total number of requests.<br /><br />Our technical team is massively working to resolve this critical issue and will update the status of the debugging steps until full resolution.</p>tag:status.fasterize.com,2005:Incident/115274422022-10-06T11:13:29+02:002022-10-06T11:13:30+02:00Slight slow down<p><small>Oct <var data-var='date'> 6</var>, <var data-var='time'>11:13</var> CEST</small><br><strong>Resolved</strong> - This incident is now resolved. A post-mortem will follow in the next few days.</p><p><small>Oct <var data-var='date'> 6</var>, <var data-var='time'>10:51</var> CEST</small><br><strong>Update</strong> - We are still seeing wait time optimizations but they will decrease until the patch is fully deployed.</p><p><small>Oct <var data-var='date'> 6</var>, <var data-var='time'>10:35</var> CEST</small><br><strong>Monitoring</strong> - From 9:54 GMT+2 until 10:25 GMT+2, we had a slight disruption in the optimization process, causing some pages to slow down. We apologize for the inconvenience. The problem is now under control and we are investigating to understand the root cause of the incident.</p>tag:status.fasterize.com,2005:Incident/106928362022-07-26T14:03:51+02:002022-07-26T14:03:51+02:00Issues on image optimization<p><small>Jul <var data-var='date'>26</var>, <var data-var='time'>14:03</var> CEST</small><br><strong>Resolved</strong> - Incident is now closed. Sorry for the inconvenience.</p><p><small>Jul <var data-var='date'>26</var>, <var data-var='time'>13:48</var> CEST</small><br><strong>Monitoring</strong> - Fix is being deployed. We're still monitoring.</p><p><small>Jul <var data-var='date'>26</var>, <var data-var='time'>13:09</var> CEST</small><br><strong>Identified</strong> - We currently have some issues on our european infrastructure: some images are not resized or not optimized anymore. Being fixed. Expected impact on images delivery (images delayed or/and bigger than expected).</p>tag:status.fasterize.com,2005:Incident/102657742022-06-27T09:20:58+02:002022-06-27T09:20:58+02:00Important: New IP range that must be whitelisted if applicable<p><small>Jun <var data-var='date'>27</var>, <var data-var='time'>09:20</var> CEST</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Jun <var data-var='date'>13</var>, <var data-var='time'>16:55</var> CEST</small><br><strong>Scheduled</strong> - We have decided to acquire new IP addresses Blocks that will help us extend the capability of our infrastructure. This change will be effective on June 21th 2022 8:00AM.<br />Please, make sure that the new following IP addresses are whitelisted in your firewall system(s) in order to prevent any 502 response codes from being generated unexpectedly:<br /><br />13.37.51.216<br />35.181.42.150<br />13.38.174.20<br />13.37.61.69<br />52.47.48.3<br />15.188.198.0<br />15.188.120.86<br />15.188.135.165<br />15.188.205.129<br />13.36.46.149<br />15.188.101.161<br /><br />The complete IP range can be found at this URL https://support.fasterize.com/en/support/solutions/articles/43000456888-list-of-fasterize-ip-addresses.<br /><br />Don’t hesitate to ask for any help by sending an e-mail to support@fasterize.com.<br /><br />Best regards,<br /><br />Fasterize team</p>tag:status.fasterize.com,2005:Incident/103686392022-06-21T15:10:25+02:002022-06-21T15:10:25+02:00Degraded performance of our API<p><small>Jun <var data-var='date'>21</var>, <var data-var='time'>15:10</var> CEST</small><br><strong>Resolved</strong> - The issue has been resolved, the API and dashboard are now fully operational.</p><p><small>Jun <var data-var='date'>21</var>, <var data-var='time'>13:51</var> CEST</small><br><strong>Monitoring</strong> - Issue has been identified and we are currently monitoring it</p><p><small>Jun <var data-var='date'>21</var>, <var data-var='time'>12:35</var> CEST</small><br><strong>Investigating</strong> - We are currently investigating a degradation of performance of our API that may affect the availability of our dashboard.</p>