Project status change failures in multiple regions

Updates timeline

1. Investigating Jun 30, 02:38 AM UTC

   We are currently investigating an issue affecting project creation in multiple regions.

2. Investigating Jun 30, 02:43 AM UTC

   Project creation is impacted across multiple regions, we are investigating the root cause.

3. Investigating Jun 30, 03:08 AM UTC

   We have identified the issue and confirmed that impact extends beyond project creation. Project creation, project resizing, branch provisioning, and project restarts are currently affected across multiple regions. We will share another update as soon as we have more information.

4. Identified Jun 30, 03:38 AM UTC

   We have identified the issue and are working on a fix. Impact is currently affecting project creation, project resizing, branch provisioning, and project restarts across multiple regions. We will provide another update as soon as we have more information.

5. Identified Jun 30, 04:32 AM UTC

   We are continuing to work on a fix for the issue and will provide updates as more information is available.

6. Identified Jun 30, 05:11 AM UTC

   We are still working to resolve the issue affecting project creation, project resizing, branch provisioning, and project restarts across multiple regions. We will continue to provide updates as our work progresses.

7. Identified Jun 30, 06:49 AM UTC

   We are still working to resolve the issue affecting project creation, project resizing, branch provisioning, and project restarts across multiple regions. We will continue to provide updates as our work progresses.

8. Identified Jun 30, 11:20 AM UTC

   Additional capacity has been made available in the majority of regions, and we're seeing reduced failure rates in terms of project creation, project resizing, branch provisioning, and project restarts. We are still experiencing limited instance capacity in ap-northeast-2, ap-south-1, and eu-north-1. As a result, project operations that require capacity in these regions, including project creation, resizing, branch provisioning, restarts, may continue to fail. We are actively working to restore full capacity and will provide another update as soon as more information is available.

9. Identified Jun 30, 06:58 PM UTC

   We are continuing to investigate the remaining capacity constraints affecting ap-northeast-2, ap-south-1, and eu-north-1. Project operations that require capacity in these regions may continue to fail. We will provide another update as soon as more information is available.

10. Identified Jun 30, 10:22 PM UTC

    We are continuing to investigate the remaining capacity constraints affecting ap-northeast-2, ap-south-1, and eu-north-1. Project operations that require capacity in these regions may continue to fail. We will provide another update as soon as more information is available.

11. Identified Jul 1, 03:23 AM UTC

    We are continuing to remediate the capacity constraints affecting multiple regions (ap-northeast-1, ap-northeast-2, ap-south-1, ap-southeast-1, ap-southeast-2, eu-central-2, eu-north-1, sa-east-1, us-east-1, us-east-2). Project operations that require capacity in these regions may continue to fail or require a re-try. We will provide another update as soon as more information is available.

12. Identified Jul 1, 10:12 AM UTC

    We continue to experience intermittent capacity related issues across all regions, impacting operations such as project creation, project resizing, branch provisioning, and project restarts. These operations may fail intermittently or require one or more retries to complete successfully. Our team is actively investigating ways to improve the situation and is implementing a number of mitigation measures to reduce the impact on affected operations. We will provide another update as soon as more information becomes available.

13. Identified Jul 1, 04:58 PM UTC

    We continue to experience widespread capacity constraints across most regions, affecting project creation, project resizing, project restarts, branch provisioning, and restores. As a result, some users may experience repeated failures when attempting these operations. Our team is actively implementing mitigation measures to increase capacity and restore normal service. We will provide further updates as the situation evolves. If you're attempting to create a new project, we recommend waiting until the incident has progressed and the status page indicates that additional capacity has been made available before retrying. This issue does not affect existing running projects or their availability.

14. Identified Jul 2, 01:30 AM UTC

    We continue to experience widespread capacity constraints across most regions, affecting project creation, project resizing, project restarts, branch provisioning, and restores. As a result, some users may experience repeated failures when attempting these operations. Our team is actively implementing mitigation measures to increase capacity and restore normal service. We will provide further updates as the situation evolves. If you're attempting to create a new project, we recommend waiting until the incident has progressed and the status page indicates that additional capacity has been made available before retrying. This issue does not affect existing projects or their availability unless the instance undergoes restarts, restores, or database upgrades.

15. Identified Jul 2, 05:53 AM UTC

    We are seeing capacity improvements across all regions, but are continuing to work on a full mitigation for all users. Project creation, project resizing, project restarts, branch provisioning, and restores are slowly recovering and we are asking users to re-try their database actions. Please reach out to our Support if you continue to fail. Running projects are unaffected and stay available. The issue has only surfaced during a restart, restore, or database upgrade. We are still addressing an issue with Read Replicas and continuing to increase capacity to restore normal service in full with highest priority.