Realtime latency

Updates timeline

1. Investigating Oct 12, 10:59 PM UTC

   We are experiencing increased latency with Realtime and are actively investigating.

2. Monitoring Oct 13, 12:02 AM UTC

   We have identified the cause and taken actions to remediate the high latency issues. We will continue to monitor.

3. Monitoring Oct 13, 06:38 AM UTC

   We have taken preventative steps to expand our realtime cluster capacity and are continuing to monitor our system stability for full recovery.

4. Identified Oct 13, 01:49 PM UTC

   We’re seeing increased Realtime latency and instability affecting the presence feature, which may impact join rates. This recurrence comes as we continue work on the remaining preventative measures. We are actively investigating and taking steps to mitigate the impact. We’ll share another update as soon as we have more information.

5. Identified Oct 13, 05:34 PM UTC

   We are continuing to investigate increased Realtime latency and instability affecting the presence feature, which may impact join rates. Our team is actively working on mitigation and we will share another update as soon as more information becomes available.

6. Identified Oct 13, 09:15 PM UTC

   We’ve added extra capacity to the system and are currently testing it. We expect to begin stabilizing the affected clusters shortly. We’ll continue to share further updates as more information becomes available.

7. Identified Oct 13, 11:27 PM UTC

   We’re seeing recovery for a large percentage of affected customers, though some issues remain in certain regions. We’re continuing to work on resolving the remaining issues and will share further updates as more information becomes available.

8. Monitoring Oct 14, 03:00 AM UTC

   We have stabilised impacted users and enabled improvements to our Realtime infrastructure. We will continue to monitor for full recovery.