Snapshot 232208
Raw (normalized text)
Scripts and page chrome removed; this is what change detection compares.
{
"incidents": [
{
"created_at": "2026-10-22T18:26:02Z",
"id": "01M5JDSVJXDAN1PQYRMY5CHP50",
"impact": "none",
"incident_updates": [
{
"body": "We can confirm that no PHP components are included in Astro.",
"created_at": "2026-11-07T13:39:56Z",
"display_at": "2026-11-07T13:39:56Z",
"id": "01M6V3SEZAJ6PBE5DXHXCBMW7M",
"incident_id": "01M5JDSVJXDAN1PQYRMY5CHP50",
"status": "resolved",
"updated_at": "2026-11-07T13:39:56Z"
},
{
"body": "**PHP Vulnerabilities (CVE-2026-109618 and CVE-2026-109619)\n** Astronomer has conducted a thorough internal review of its code and container images and confirmed that Astro and Astro Private Cloud do not include PHP and are therefore not affected by the recently disclosed PHP vulnerabilities.\n\n**Customer action:** Customers should review their own code and any modifications they have made to Astronomer-provided runtime images (for example, custom-installed packages or system dependencies) to determine whether PHP is present and whether they may be impacted. Please follow the guidance in the[ NCSC advisory](https://www.ncsc.gov.uk/news/critical-security-issue-affecting-php) and the PHP project’s security update to remediate any affected components.",
"created_at": "2026-10-22T18:26:03Z",
"display_at": "2026-10-22T18:26:02Z",
"id": "01M5JDSVJX3CJMG9CA2HW5XREE",
"incident_id": "01M5JDSVJXDAN1PQYRMY5CHP50",
"status": "monitoring",
"updated_at": "2026-10-22T18:26:03Z"
}
],
"name": "PHP Security Advisory: Astro and Astro Private Cloud Not Affected; Review Custom Images",
"page_id": "01KN858NJKF1NW5FX72VHGX8D1",
"resolved_at": "2026-11-07T13:39:56Z",
"status": "resolved",
"updated_at": "2026-11-07T13:39:56Z"
},
{
"created_at": "2026-10-22T00:01:42Z",
"id": "01M5GEKQW2J6J9QTP7185NK365",
"impact": "none",
"incident_updates": [
{
"body": "We have now released updated agent client patches for use in Remote Execution deployments that incorporate certificate trust updates to mitigate the vulnerability. These are 1.8.11, 1.7.7, and 1.6.5. See more details in the [Agent Release Notes](https://www.astronomer.io/docs/runtime/runtime-release-feed).\n\n \n\nAdditionally, we have released another update of the Runtime images to handle the following two changes related to this vulnerability.\n\n• New virtual environments created with `python -m venv` now get pip 26.2.2\n\n• Upgrade `grpcio` to 1.84.1, whose bundled CA list no longer includes the roots affected by the eMudhra compromise\n\nThe following Runtime image versions include these changes. For more details see [Runtime Release Notes](https://www.astronomer.io/docs/runtime/runtime-release-feed).\n\n• **Runtime 13:** 13.12.2\n\n• **Runtime 3.0:** 3.0-21\n\n• **Runtime 3.1:** 3.1-24\n\n• **Runtime 3.2:** 3.2-13\n\n• **Runtime 3.3:** 3.3-11",
"created_at": "2026-10-23T21:28:56Z",
"display_at": "2026-10-23T21:28:56Z",
"id": "01M5NANEZEZA6DSKZMG7B3RGGN",
"incident_id": "01M5GEKQW2J6J9QTP7185NK365",
"status": "resolved",
"updated_at": "2026-10-23T21:28:56Z"
},
{
"body": "Astronomer has released Runtime updates to address a publicly disclosed security vulnerability concerning private keys associated with eight eMudhra/emSign intermediate certificate authority (CA) certificates.\n\nFor additional background, visit \u003chttps://emudhra.we-have-a-problem.com/\u003e.\n\n**Scope and available updates**\n\nThis advisory stems from an issue with a third-party certificate authority. Action required for customer-managed components depends on their specific certificate trust stores and verification mechanisms.\n\nThe following Runtime releases incorporate certificate trust updates to mitigate this vulnerability:\n\n• **Runtime 13:** 13.12.1\n\n• **Runtime 3.0:** 3.0-20\n\n• **Runtime 3.1:** 3.1-23\n\n• **Runtime 3.2:** 3.2-12\n\n• **Runtime 3.3:** 3.3-10\n\nFor full details, please refer to the [Runtime release feed](https://www.astronomer.io/docs/runtime/runtime-release-feed).\n\n**Recommended customer actions**\n\n• Upgrade to the appropriate updated Runtime release for your environment.\n\n• Rebuild and redeploy custom images using the updated base image. Refer to our [mitigation guide](https://www.astronomer.io/docs/runtime/mitigate-emudhra-ca-compromise) for step-by-step instructions.\n\n• Audit customer-managed build pipelines and tooling, applying relevant vendor updates to certificate trust stores and signature verification components.\n\n• Ensure TLS and code-signature verification remain enabled. If updates cause validation failures, contact the endpoint owner or software publisher for replacement certificate or artifact guidance.\n\nPlease note that Runtime updates address the trust components included within those specific releases. Customer-added dependencies, custom certificate bundles, and external tooling may require separate updates.\n\n**Support and further guidance**\n\nIf you are unable to upgrade or require assistance determining the applicable next steps, please contact [**support@astronomer.io**](mailto:support@astronomer.io). Include your deployment model, Runtime version, and whether you maintain custom images or build pipelines.\n\nAdditional mitigation guidance is currently being finalized. We will update this advisory as further details and Remote Execution releases are confirmed.",
"created_at": "2026-10-22T00:01:42Z",
"display_at": "2026-10-22T00:01:42Z",
"id": "01M5GEKQW2RD3YK2Z5QRHBC5EY",
"incident_id": "01M5GEKQW2J6J9QTP7185NK365",
"status": "identified",
"updated_at": "2026-10-22T00:01:42Z"
}
],
"name": "Security Update: Runtime Releases and Guidance for the eMudhra/emSign Certificate Issue",
"page_id": "01KN858NJKF1NW5FX72VHGX8D1",
"resolved_at": "2026-10-23T21:28:56Z",
"status": "resolved",
"updated_at": "2026-10-23T21:28:56Z"
},
{
"created_at": "2026-10-20T06:40:27Z",
"id": "01M5C0MER2KMXS4ZVJ2E1G3QMP",
"impact": "major",
"incident_updates": [
{
"body": "Astro Alerts delivery has returned to normal, and this incident is resolved. We have applied the fix and fully processed the backlog of delayed alerts. Thank you for your patience while we worked to restore service.",
"created_at": "2026-10-21T08:49:38Z",
"display_at": "2026-10-21T08:49:38Z",
"id": "01M5ETDPTNR3RK6W0ZH8BAGADN",
"incident_id": "01M5C0MER2KMXS4ZVJ2E1G3QMP",
"status": "resolved",
"updated_at": "2026-10-21T08:49:38Z"
},
{
"body": "The backlog of delayed Astro Alerts has been fully processed, and we are monitoring recovery following the fix. A small number of alerts may still be briefly delayed as the fix completes on remaining systems.",
"created_at": "2026-10-20T12:38:47Z",
"display_at": "2026-10-20T12:38:47Z",
"id": "01M5CN4JERC0B1V74AW65YBA2K",
"incident_id": "01M5C0MER2KMXS4ZVJ2E1G3QMP",
"status": "monitoring",
"updated_at": "2026-10-20T12:38:47Z"
},
{
"body": "The fix has been applied to the majority of affected deployments, and delayed alerts continue to be delivered as we process the backlog. We are addressing a small number of remaining deployments and will provide another update once this is complete.",
"created_at": "2026-10-20T10:42:33Z",
"display_at": "2026-10-20T10:42:33Z",
"id": "01M5CEFQT66REEXWDHJJ4Y94H4",
"incident_id": "01M5C0MER2KMXS4ZVJ2E1G3QMP",
"status": "identified",
"updated_at": "2026-10-20T10:42:33Z"
},
{
"body": "We have identified the cause and are rolling out a fix to the affected deployments. Alerts held during the incident have started to be delivered, and some alerts may continue to be delayed until the rollout is complete. We will provide another update once the fix is fully rolled out.",
"created_at": "2026-10-20T07:55:03Z",
"display_at": "2026-10-20T07:55:03Z",
"id": "01M5C4X171Y1MHX8ACJB82RAG0",
"incident_id": "01M5C0MER2KMXS4ZVJ2E1G3QMP",
"status": "identified",
"updated_at": "2026-10-20T07:55:03Z"
},
{
"body": "We are investigating an issue where Astro Alerts notifications may be delayed for some customers. Our team is actively working to resolve the issue, and we will provide an update as soon as we have more information.",
"created_at": "2026-10-20T06:40:27Z",
"display_at": "2026-10-20T06:40:27Z",
"id": "01M5C0MER2VB5SNANA06TKP06K",
"incident_id": "01M5C0MER2KMXS4ZVJ2E1G3QMP",
"status": "investigating",
"updated_at": "2026-10-20T06:40:27Z"
}
],
"name": "Delayed delivery of Astro Alerts",
"page_id": "01KN858NJKF1NW5FX72VHGX8D1",
"resolved_at": "2026-10-21T08:49:38Z",
"status": "resolved",
"updated_at": "2026-10-21T08:49:38Z"
},
{
"created_at": "2026-10-19T09:37:06Z",
"id": "01M59RB6D3SW2MK83XPPGBT114",
"impact": "major",
"incident_updates": [
{
"body": "The issue has been resolved, and the Organization Dashboards are now working as expected.",
"created_at": "2026-10-19T11:00:06Z",
"display_at": "2026-10-19T11:00:06Z",
"id": "01M59X35MY04GFV2BSZG6RAE4V",
"incident_id": "01M59RB6D3SW2MK83XPPGBT114",
"status": "resolved",
"updated_at": "2026-10-19T11:00:06Z"
},
{
"body": "We have identified the source of the issue and are actively working on a fix. We’ll share further updates as we make progress.",
"created_at": "2026-10-19T10:28:45Z",
"display_at": "2026-10-19T10:28:45Z",
"id": "01M59V9R38MT3CZKEZ484RW4WC",
"incident_id": "01M59RB6D3SW2MK83XPPGBT114",
"status": "identified",
"updated_at": "2026-10-19T10:28:45Z"
},
{
"body": "We are currently investigating this issue. We will keep this page updated as we have more information.",
"created_at": "2026-10-19T09:37:07Z",
"display_at": "2026-10-19T09:37:06Z",
"id": "01M59RB6D3W4SA4RZ37KEYZ1DQ",
"incident_id": "01M59RB6D3SW2MK83XPPGBT114",
"status": "investigating",
"updated_at": "2026-10-19T09:37:07Z"
}
],
"name": "Organization Dashboards are failing to load with Invalid kid parameter error",
"page_id": "01KN858NJKF1NW5FX72VHGX8D1",
"resolved_at": "2026-10-19T11:00:06Z",
"status": "resolved",
"updated_at": "2026-10-19T11:00:07Z"
},
{
"created_at": "2026-10-06T03:33:51Z",
"id": "01M47MCQ7XP80T96R5JX6QGKP2",
"impact": "minor",
"incident_updates": [
{
"body": "We have seen improved capacity in the GCP us-central1 region across additional machine types, and error rates for worker node and pod scale-up have dropped.\n\nIf any tasks failed during this incident, you can clear and rerun them. To recover automatically from future capacity constraints, we recommend setting `retries` and `retry_delay` on your tasks or in `default_args`.\n\nIf you’re still seeing issues, please contact Astronomer Support.",
"created_at": "2026-10-14T00:34:25Z",
"display_at": "2026-10-14T00:34:25Z",
"id": "01M4VX9WWWWVA2A12NZ62XZDKX",
"incident_id": "01M47MCQ7XP80T96R5JX6QGKP2",
"status": "resolved",
"updated_at": "2026-10-14T00:34:25Z"
},
{
"body": "We are currently seeing a capacity shortage in the Google Cloud (GCP) `us-central1` region. As a result, some Deployments running in this region may experience delays when scaling up worker nodes and pods.\n\nSome tasks may remain in a queued state for longer than usual while waiting for compute capacity. In some cases, tasks that stay queued beyond their configured timeout may fail.\n\n**Recommended workaround:** Configuring automated task retries (for example, setting `retries` and a `retry_delay` on your tasks or in your DAG's `default_args`) will help tasks recover automatically once capacity becomes available. \u003chttps://www.astronomer.io/docs/learn/rerunning-dags#automatically-retry-tasks\u003e \n\nWe are working to address the issue and keep this page posted as we have more updates.",
"created_at": "2026-10-06T03:33:52Z",
"display_at": "2026-10-06T03:33:51Z",
"id": "01M47MCQ7XWBGVCVZ871CDM7DB",
"incident_id": "01M47MCQ7XP80T96R5JX6QGKP2",
"status": "investigating",
"updated_at": "2026-10-06T03:33:52Z"
}
],
"name": "Capacity constraints in GCP us-central1 may delay or fail some task executions",
"page_id": "01KN858NJKF1NW5FX72VHGX8D1",
"resolved_at": "2026-10-14T00:34:25Z",
"status": "resolved",
"updated_at": "2026-10-14T00:34:25Z"
},
{
"created_at": "2026-10-13T18:02:11Z",
"id": "01M4V6VPAPNE7RC6Q1KCXGMVYE",
"impact": "major",
"incident_updates": [
{
"body": "We have identified the proximate cause of the incident and are confident that the situation is stable. We will follow up with more detail in a postmortem at a later date.",
"created_at": "2026-10-13T19:53:11Z",
"display_at": "2026-10-13T19:53:11Z",
"id": "01M4VD6YCRKSM09RCEFB13WHFB",
"incident_id": "01M4V6VPAPNE7RC6Q1KCXGMVYE",
"status": "resolved",
"updated_at": "2026-10-13T19:53:11Z"
},
{
"body": "We have restarted one of our services that was experiencing higher than expected load. This has returned the Astro UI and API to functionality.\n\nWe are continuing to investigate root cause.",
"created_at": "2026-10-13T18:28:01Z",
"display_at": "2026-10-13T18:28:01Z",
"id": "01M4V8B03M3TQST0Q0J1ZZDRT7",
"incident_id": "01M4V6VPAPNE7RC6Q1KCXGMVYE",
"status": "monitoring",
"updated_at": "2026-10-13T18:28:01Z"
},
{
"body": "Our metrics indicate no impact on existing tasks and Dag Runs. They are continuing to run successfully. We are continuing to investigate what is causing the API and UI errors and are attempting a workaround.",
"created_at": "2026-10-13T18:13:41Z",
"display_at": "2026-10-13T18:13:41Z",
"id": "01M4V7GRFPKVNJ4RB5QRW4D1N4",
"incident_id": "01M4V6VPAPNE7RC6Q1KCXGMVYE",
"status": "investigating",
"updated_at": "2026-10-13T18:13:41Z"
},
{
"body": "The Astro Core API is showing an elevated number of errors. We are investigating actively and will update as soon as we know more. Next update will be within the next 30 minutes.",
"created_at": "2026-10-13T18:02:11Z",
"display_at": "2026-10-13T18:02:11Z",
"id": "01M4V6VPAPY519CG5NKD1RMPYK",
"incident_id": "01M4V6VPAPNE7RC6Q1KCXGMVYE",
"status": "investigating",
"updated_at": "2026-10-13T18:02:11Z"
}
],
"name": "Astro UI and API are appearing offline for some users",
"page_id": "01KN858NJKF1NW5FX72VHGX8D1",
"resolved_at": "2026-10-13T19:53:11Z",
"status": "resolved",
"updated_at": "2026-10-19T21:36:56Z"
},
{
"created_at": "2026-10-03T20:42:50Z",
"id": "01M41R2NQVA0KGBPGAT39RKD8R",
"impact": "minor",
"incident_updates": [
{
"body": "All Astro Alert emails for Dag Success/Failure now correctly include the deployment/workspace name.",
"created_at": "2026-10-03T21:39:27Z",
"display_at": "2026-10-03T21:39:27Z",
"id": "01M41VABGP2P4HPQFEZZ0W2XVP",
"incident_id": "01M41R2NQVA0KGBPGAT39RKD8R",
"status": "resolved",
"updated_at": "2026-10-03T21:39:27Z"
},
{
"body": "We have reverted the problematic email template changes and are continuing to monitor for full resolution.",
"created_at": "2026-10-03T21:13:03Z",
"display_at": "2026-10-03T21:13:03Z",
"id": "01M41SSZQTJGE3YR6WK3D331K9",
"incident_id": "01M41R2NQVA0KGBPGAT39RKD8R",
"status": "monitoring",
"updated_at": "2026-10-03T21:13:03Z"
},
{
"body": "Correction: Deployment and Workspace _names_ are not populated, earlier update cited IDs incorrectly.",
"created_at": "2026-10-03T21:03:35Z",
"display_at": "2026-10-03T21:03:34Z",
"id": "01M41S8N0WW3ZG6BV540XKMX4B",
"incident_id": "01M41R2NQVA0KGBPGAT39RKD8R",
"status": "identified",
"updated_at": "2026-10-03T21:03:35Z"
},
{
"body": "We have identified a recent change that broke the templating of the deploymentId and workspaceId fields used in the Astro Alert emails for Dag Success and Failure emails.\n\nWe are working on a fix and will provide updates shortly.",
"created_at": "2026-10-03T20:42:50Z",
"display_at": "2026-10-03T20:42:50Z",
"id": "01M41R2NQV01VPVSZGPREM2FBF",
"incident_id": "01M41R2NQVA0KGBPGAT39RKD8R",
"status": "identified",
"updated_at": "2026-10-03T20:42:50Z"
}
],
"name": "Dag success/failure Astro Alert emails missing deployment \u0026 workspace names",
"page_id": "01KN858NJKF1NW5FX72VHGX8D1",
"resolved_at": "2026-10-03T21:39:27Z",
"status": "resolved",
"updated_at": "2026-10-03T21:39:28Z"
},
{
"created_at": "2026-10-01T22:40:38Z",
"id": "01M3WT0XXCDDNQG63T7PA89QV0",
"impact": "critical",
"incident_updates": [
{
"body": "Our development team has concluded that this was a false alert. It is not affecting multiple deployments.",
"created_at": "2026-10-01T23:47:48Z",
"display_at": "2026-10-01T23:47:48Z",
"id": "01M3WXVX7VCFFZ7ARWZTSTDCTH",
"incident_id": "01M3WT0XXCDDNQG63T7PA89QV0",
"status": "resolved",
"updated_at": "2026-10-01T23:47:48Z"
},
{
"body": "We have identified several hosted deployments where tasks are getting stuck in the queued state and failing. This is happening the worker autoscaler is having issues with scaling up workers to run tasks. We are investigating the cause of this.",
"created_at": "2026-10-01T22:40:38Z",
"display_at": "2026-10-01T22:40:38Z",
"id": "01M3WT0XXCGB957S6BFY43NMKT",
"incident_id": "01M3WT0XXCDDNQG63T7PA89QV0",
"status": "investigating",
"updated_at": "2026-10-01T22:40:38Z"
}
],
"name": "Tasks getting stuck in the queued state and failing",
"page_id": "01KN858NJKF1NW5FX72VHGX8D1",
"resolved_at": "2026-10-01T23:47:48Z",
"status": "resolved",
"updated_at": "2026-10-01T23:47:48Z"
},
{
"created_at": "2026-09-24T14:52:45Z",
"id": "01M39YF61YAEX17RXNY8KM25ME",
"impact": "major",
"incident_updates": [
{
"body": "Issue has been resolved and error rates have returned to baseline. If you are still experiencing issues please refresh the page, and open a support ticket if that does not resolve the issue.",
"created_at": "2026-09-24T15:34:41Z",
"display_at": "2026-09-24T15:34:41Z",
"id": "01M3A0VZ6MNTPKWDWC48CH3PPY",
"incident_id": "01M39YF61YAEX17RXNY8KM25ME",
"status": "resolved",
"updated_at": "2026-09-24T15:34:41Z"
},
{
"body": "We have rolled back an identified breaking change, and are beginning to see error rates decrease. Please refresh the Astro Cloud UI in your browser.",
"created_at": "2026-09-24T15:19:04Z",
"display_at": "2026-09-24T15:19:03Z",
"id": "01M39ZZBG3RBG52851JV1F0BFS",
"incident_id": "01M39YF61YAEX17RXNY8KM25ME",
"status": "monitoring",
"updated_at": "2026-09-24T15:19:04Z"
},
{
"body": "We have confirmed that task execution is unaffected. We're continuing to investigate.",
"created_at": "2026-09-24T14:56:33Z",
"display_at": "2026-09-24T14:56:33Z",
"id": "01M39YP4P6D1NRN9GZJYY1AN72",
"incident_id": "01M39YF61YAEX17RXNY8KM25ME",
"status": "investigating",
"updated_at": "2026-09-24T14:56:33Z"
},
{
"body": "We are investigation reports that the Deployments view of the Astro Cloud UI is unavailable. We will update as we know more.",
"created_at": "2026-09-24T14:52:45Z",
"display_at": "2026-09-24T14:52:45Z",
"id": "01M39YF61YFQMQJY5N4KMPQPZ2",
"incident_id": "01M39YF61YAEX17RXNY8KM25ME",
"status": "investigating",
"updated_at": "2026-09-24T14:52:45Z"
}
],
"name": "Astro UI Deployment view unavailable",
"page_id": "01KN858NJKF1NW5FX72VHGX8D1",
"resolved_at": "2026-09-24T15:34:41Z",
"status": "resolved",
"updated_at": "2026-09-24T15:34:41Z"
},
{
"created_at": "2026-09-17T17:54:09Z",
"id": "01M2R82A38HH3X91BKSJH8WR22",
"impact": "major",
"incident_updates": [
{
"body": "No further issues have been found, so this is being marked as resolved.",
"created_at": "2026-09-17T21:50:28Z",
"display_at": "2026-09-17T21:50:28Z",
"id": "01M2RNK0F5MT3YNN87DHZ5YWFQ",
"incident_id": "01M2R82A38HH3X91BKSJH8WR22",
"status": "resolved",
"updated_at": "2026-09-17T21:50:28Z"
},
{
"body": "All custom roles have been restored to their previous functionality and full operation has been restored. ",
"created_at": "2026-09-17T19:29:20Z",
"display_at": "2026-09-17T19:29:20Z",
"id": "01M2RDGJWS6GWM6J5W54160Z8W",
"incident_id": "01M2R82A38HH3X91BKSJH8WR22",
"status": "monitoring",
"updated_at": "2026-09-17T19:29:20Z"
},
{
"body": "We have identified a regression affecting custom roles due to a change rolled out today. This resulted in permissions being dropped from custom roles silently. \n\nWe have received reports of deploys linked to tokens using custom roles not working. Until the fix has been rolled out, deploy failures can be mitigated by re-adding the `deployment.deploys.create` permission to your custom role. You may also manually restore any other permissions that were silently dropped.\n\nWe are working on fixing this regression for all affected custom roles.",
"created_at": "2026-09-17T17:54:09Z",
"display_at": "2026-09-17T17:54:09Z",
"id": "01M2R82A38KE6R7EJP94JTY5KE",
"incident_id": "01M2R82A38HH3X91BKSJH8WR22",
"status": "identified",
"updated_at": "2026-09-17T17:54:09Z"
}
],
"name": "Permissions silently dropped from custom roles",
"page_id": "01KN858NJKF1NW5FX72VHGX8D1",
"resolved_at": "2026-09-17T21:50:28Z",
"status": "resolved",
"updated_at": "2026-09-17T21:50:28Z"
},
{
"created_at": "2026-08-18T18:51:59Z",
"id": "01M0B3EMK58DTX2DW16S9RNHV5",
"impact": "none",
"incident_updates": [
{
"body": "This issue is now resolved.",
"created_at": "2026-09-10T22:19:21Z",
"display_at": "2026-09-10T22:19:21Z",
"id": "01M26PEWESA5BTGBNG0CJV5WV1",
"incident_id": "01M0B3EMK58DTX2DW16S9RNHV5",
"status": "resolved",
"updated_at": "2026-09-10T22:19:21Z"
},
{
"body": "We've identified the cause of the false availability alerts that have been showing the Astro UI. This is a degradation affecting dashboard and monitoring visibility only. Task execution, scheduling, and workflow processing have not been impacted. The issue is the result of a metrics reporting bug that has been intermittently flagging scheduler heartbeats as missed when they were not.\nWe are actively rolling out a fix that corrects how these metrics are reported.",
"created_at": "2026-09-08T18:31:44Z",
"display_at": "2026-09-08T18:31:44Z",
"id": "01M214MMYSNRBM93Q4W327SKG9",
"incident_id": "01M0B3EMK58DTX2DW16S9RNHV5",
"status": "identified",
"updated_at": "2026-09-08T18:31:44Z"
},
{
"body": "Our metrics pipeline is falsely displaying spikes in scheduler heartbeat metrics, despite underlying component health. ",
"created_at": "2026-08-18T18:51:59Z",
"display_at": "2026-08-18T18:51:59Z",
"id": "01M0B3EMK500NA5RQR70RJJ66W",
"incident_id": "01M0B3EMK58DTX2DW16S9RNHV5",
"status": "investigating",
"updated_at": "2026-08-18T18:51:59Z"
}
],
"name": "Incorrect scheduler heartbeat metrics being reported",
"page_id": "01KN858NJKF1NW5FX72VHGX8D1",
"resolved_at": "2026-09-10T22:19:21Z",
"status": "resolved",
"updated_at": "2026-09-10T22:19:21Z"
},
{
"created_at": "2026-09-08T12:33:58Z",
"id": "01M20G5JES5V16BF6T2MZE5JW3",
"impact": "major",
"incident_updates": [
{
"body": "The incident has been resolved.",
"created_at": "2026-09-08T20:09:46Z",
"display_at": "2026-09-08T20:09:46Z",
"id": "01M21A855E5YW2A13X8RF7SRGX",
"incident_id": "01M20G5JES5V16BF6T2MZE5JW3",
"status": "resolved",
"updated_at": "2026-09-08T20:09:46Z"
},
{
"body": "A fix has been implemented, and we are monitoring the results.",
"created_at": "2026-09-08T19:42:21Z",
"display_at": "2026-09-08T19:42:21Z",
"id": "01M218NYPRXAZKAVDRPG5C5HTN",
"incident_id": "01M20G5JES5V16BF6T2MZE5JW3",
"status": "monitoring",
"updated_at": "2026-09-08T19:42:21Z"
},
{
"body": "We have identified the cause.\n\nOn Astro, each worker queue is backed by a Kubernetes autoscaling resource whose name is built from a fixed platform prefix, the deployment release name, and the worker queue name.\n\nFor a very small number of customers, where that combined name exceeds the Kubernetes 63-character limit, the autoscaling resource is rejected at creation. As a result, workers do not scale for the affected queue and tasks routed there remain in the queued state.\n\nThis is why the limit can be reached even when the worker queue name itself looks short: the release name and platform prefix already consume a large portion of the 63 characters before the queue name is appended.\n\nWorkaround: If affected, until the permanent fix is deployed, please keep worker queue name at or below 10 characters.\n\nWe are preparing a permanent fix. Further updates to follow.",
"created_at": "2026-09-08T12:33:58Z",
"display_at": "2026-09-08T12:33:58Z",
"id": "01M20G5JES7JTG3Q60YE7H28HX",
"incident_id": "01M20G5JES5V16BF6T2MZE5JW3",
"status": "identified",
"updated_at": "2026-09-08T12:33:58Z"
}
],
"name": "Tasks remain in the queued state on some worker queues.",
"page_id": "01KN858NJKF1NW5FX72VHGX8D1",
"resolved_at": "2026-09-08T20:09:46Z",
"status": "resolved",
"updated_at": "2026-09-08T20:09:46Z"
},
{
"created_at": "2026-09-04T03:08:43Z",
"id": "01M1N67NVTBCAEETQPS70AQ8JR",
"impact": "major",
"incident_updates": [
{
"body": "The issue has been fully resolved. All services, including authentication, deployment access, and DAG visibility, are operating normally.\n\nWe apologize for the disruption and thank you for your patience.",
"created_at": "2026-09-04T06:07:24Z",
"display_at": "2026-09-04T06:07:24Z",
"id": "01M1NGEVZC26DJM7YSJ97HD3XS",
"incident_id": "01M1N67NVTBCAEETQPS70AQ8JR",
"status": "resolved",
"updated_at": "2026-09-04T06:07:24Z"
},
{
"body": "We identified and resolved an infrastructure configuration issue that was causing authentication failures and deployment access errors across the platform. The root cause was a network configuration change during an infrastructure upgrade that temporarily disrupted connectivity between our control plane and customer deployments.\nThe fix has been applied and initial validation confirms services are recovering\n\nWe are actively monitoring to confirm full recovery across all affected deployments. If you are still experiencing issues, please contact support.",
"created_at": "2026-09-04T05:53:34Z",
"display_at": "2026-09-04T05:53:34Z",
"id": "01M1NFNHDQJWAMG853XT407672",
"incident_id": "01M1N67NVTBCAEETQPS70AQ8JR",
"status": "monitoring",
"updated_at": "2026-09-04T05:53:34Z"
},
{
"body": "The potential root cause identified, and fix is underway",
"created_at": "2026-09-04T04:52:52Z",
"display_at": "2026-09-04T04:52:52Z",
"id": "01M1NC6CACGYRY5YG04JPDJ86J",
"incident_id": "01M1N67NVTBCAEETQPS70AQ8JR",
"status": "identified",
"updated_at": "2026-09-04T04:52:52Z"
},
{
"body": "We're continuing to investigate the issue affecting access/auth to some deployments and services. We will follow up with an update no later than 5:30 AM UTC on Friday, September 4, 2026, or sooner if we have something to share.",
"created_at": "2026-09-04T04:28:53Z",
"display_at": "2026-09-04T04:28:53Z",
"id": "01M1NATF9N373SMK1KE9JEGG6V",
"incident_id": "01M1N67NVTBCAEETQPS70AQ8JR",
"status": "investigating",
"updated_at": "2026-09-04T04:28:53Z"
},
{
"body": "We are currently investigating an issue affecting deployment access, authentication, and related services for multiple customers. Some users may experience 401 errors when using organization-based Astronomer URLs (ORG_ID.astronomer.run), as well as \"RBAC: access denied\" errors or difficulty viewing DAGs in the Astronomer UI.",
"created_at": "2026-09-04T03:08:43Z",
"display_at": "2026-09-04T03:08:43Z",
"id": "01M1N67NVTRF6D2GHF8Q5APZ0S",
"incident_id": "01M1N67NVTBCAEETQPS70AQ8JR",
"status": "investigating",
"updated_at": "2026-09-04T03:08:43Z"
}
],
"name": "Increased auth errors",
"page_id": "01KN858NJKF1NW5FX72VHGX8D1",
"resolved_at": "2026-09-04T06:07:24Z",
"status": "resolved",
"updated_at": "2026-09-04T06:07:24Z"
},
{
"created_at": "2026-09-01T18:40:02Z",
"id": "01M1F4ATPJ2SJ49XG8EXN9MTZB",
"impact": "minor",
"incident_updates": [
{
"body": "The GCP incident is marked as Closed officially",
"created_at": "2026-09-01T22:29:51Z",
"display_at": "2026-09-01T22:29:51Z",
"id": "01M1FHFMP4Z9G2TZXEJQ290XGP",
"incident_id": "01M1F4ATPJ2SJ49XG8EXN9MTZB",
"status": "resolved",
"updated_at": "2026-09-01T22:29:51Z"
},
{
"body": "Google Cloud has resolved the networking incident affecting us-central1. The incident ran from approximately 14:44 UTC to 18:52 UTC.\n\nDuring this period, some Astro deployments in us-central1 may have experienced intermittent networking connectivity disruptions, DNS resolution errors, or missing metrics from Airflow deployments.\n\nThis issue has since been resolved by Google, but we are continuing to monitor the situation internally to ensure it's resolution.",
"created_at": "2026-09-01T20:42:47Z",
"display_at": "2026-09-01T20:42:47Z",
"id": "01M1FBBJPQR7NXDH6J4JGPYRDR",
"incident_id": "01M1F4ATPJ2SJ49XG8EXN9MTZB",
"status": "monitoring",
"updated_at": "2026-09-01T20:42:47Z"
},
{
"body": "Google Cloud has resolved the networking incident affecting us-central1. The incident ran from approximately 14:44 UTC to 18:52 UTC.\n\nDuring this period, some Astro deployments in us-central1 may have experienced intermittent networking connectivity disruptions, DNS resolution errors, or missing metrics from Airflow deployments.",
"created_at": "2026-09-01T19:19:52Z",
"display_at": "2026-09-01T19:19:52Z",
"id": "01M1F6KRTD8M18DAP5TD2W48T7",
"incident_id": "01M1F4ATPJ2SJ49XG8EXN9MTZB",
"status": "monitoring",
"updated_at": "2026-09-01T19:19:52Z"
},
{
"body": "We are investigating a Google Cloud networking incident that began around 14:41 UTC and may affect Astro deployments running in the us-central1 region. Customers may see intermittent networking connectivity disruptions, DNS resolution errors, or missing metrics from Airflow deployments.\n\nFor more information, see the [Google Cloud status incident](https://status.cloud.google.com/incidents/J5ia5t9p3g9Q5Wi7r8Ev \"Google Cloud status incident\").",
"created_at": "2026-09-01T19:05:32Z",
"display_at": "2026-09-01T19:05:32Z",
"id": "01M1F5SGGFTD0FE8P24R3BXMY0",
"incident_id": "01M1F4ATPJ2SJ49XG8EXN9MTZB",
"status": "investigating",
"updated_at": "2026-09-01T19:05:32Z"
},
{
"body": "We are currently investigating a Google Cloud networking incident that began around 16:03 UTC and may affect Astro deployments running in the us-central1 region. Customers may see intermittent networking connectivity disruptions, DNS resolution errors, or missing metrics from Airflow deployments.\n\nFor more information, see the [Google Cloud status incident](https://status.cloud.google.com/incidents/J5ia5t9p3g9Q5Wi7r8Ev \"Google Cloud status incident\").",
"created_at": "2026-09-01T18:44:01Z",
"display_at": "2026-09-01T18:44:01Z",
"id": "01M1F4J41PBPB9VCEZJJK2DDDW",
"incident_id": "01M1F4ATPJ2SJ49XG8EXN9MTZB",
"status": "investigating",
"updated_at": "2026-09-01T18:44:01Z"
},
{
"body": "We are currently investigating a Google Cloud networking incident that began around 16:03 UTC and may affect Astro deployments running in the us-central1 region. Customers may see intermittent networking connectivity disruptions, DNS resolution errors, or missing metrics from Airflow deployments.\n\nFor more information, see the [Google Cloud status incident](https://status.cloud.google.com/incidents/J5ia5t9p3g9Q5Wi7r8Ev \"Google Cloud status incident\").",
"created_at": "2026-09-01T18:43:00Z",
"display_at": "2026-09-01T18:43:00Z",
"id": "01M1F4G89ZJ3T6FQ5BHD9JFZ5Q",
"incident_id": "01M1F4ATPJ2SJ49XG8EXN9MTZB",
"status": "investigating",
"updated_at": "2026-09-01T18:43:00Z"
},
{
"body": "We are currently investigating a Google Cloud networking incident that may affect Astro deployments running in the us-central1 region. Customers may see intermittent networking connectivity disruptions, DNS resolution errors, or missing metrics from Airflow deployments.\n\nFor more information, see the [Google Cloud status incident](https://status.cloud.google.com/incidents/J5ia5t9p3g9Q5Wi7r8Ev \"Google Cloud status incident\").",
"created_at": "2026-09-01T18:40:02Z",
"display_at": "2026-09-01T18:40:02Z",
"id": "01M1F4ATPJVVAMRN7CS6Q5AN1M",
"incident_id": "01M1F4ATPJ2SJ49XG8EXN9MTZB",
"status": "investigating",
"updated_at": "2026-09-01T18:40:02Z"
}
],
"name": "Potential impact to Astro deployments in GCP us-central1",
"page_id": "01KN858NJKF1NW5FX72VHGX8D1",
"resolved_at": "2026-09-01T22:29:51Z",
"status": "resolved",
"updated_at": "2026-09-01T22:29:52Z"
},
{
"created_at": "2026-08-18T04:17:17Z",
"id": "01M09HD09CG69RBJWCQ2ZHM4BP",
"impact": "major",
"incident_updates": [
{
"body": "Rollout has been completed",
"created_at": "2026-08-18T07:36:15Z",
"display_at": "2026-08-18T07:36:15Z",
"id": "01M09WSARX2RXDNCZQZN82R0WD",
"incident_id": "01M09HD09CG69RBJWCQ2ZHM4BP",
"status": "resolved",
"updated_at": "2026-08-18T07:36:15Z"
},
{
"body": "Fix is being rolled out across all astro deployments.",
"created_at": "2026-08-18T04:18:00Z",
"display_at": "2026-08-18T04:18:00Z",
"id": "01M09HEA6NTDS97SW5VR1JTEMD",
"incident_id": "01M09HD09CG69RBJWCQ2ZHM4BP",
"status": "monitoring",
"updated_at": "2026-08-18T04:18:00Z"
},
{
"body": "We have identified an issue affecting an internal rollout related to the `pre_execute` Airflow hook in Astro Hosted. We have identified a fix and are rolling it out across the fleet. We are monitoring the rollout and will provide further updates as it progresses.",
"created_at": "2026-08-18T04:17:17Z",
"display_at": "2026-08-18T04:17:17Z",
"id": "01M09HD09C8MJAZW71BEGKBSYA",
"incident_id": "01M09HD09CG69RBJWCQ2ZHM4BP",
"status": "investigating",
"updated_at": "2026-08-18T04:17:17Z"
}
],
"name": "Intermittent task instances failures.",
"page_id": "01KN858NJKF1NW5FX72VHGX8D1",
"resolved_at": "2026-08-18T07:36:15Z",
"status": "resolved",
"updated_at": "2026-08-18T07:36:15Z"
},
{
"created_at": "2026-08-04T15:11:19Z",
"id": "01KZ6N8GP1FVGTB8WPZ7WT98PR",
"impact": "minor",
"incident_updates": [
{
"body": "Mitigations have been successful. A Platform release that corrects this errant behavior is being worked on now. Issue is resolved.",
"created_at": "2026-08-04T16:40:54Z",
"display_at": "2026-08-04T16:40:54Z",
"id": "01KZ6TCJB93RDVSWK3STESYJSM",
"incident_id": "01KZ6N8GP1FVGTB8WPZ7WT98PR",
"status": "resolved",
"updated_at": "2026-08-04T16:40:54Z"
},
{
"body": "We have rolled back the changes that triggered this issue and are monitoring the issue for confirmation that this issue is resolved.",
"created_at": "2026-08-04T15:15:19Z",
"display_at": "2026-08-04T15:15:19Z",
"id": "01KZ6NFVKMV2W6PNMTPMRXJ5DY",
"incident_id": "01KZ6N8GP1FVGTB8WPZ7WT98PR",
"status": "monitoring",
"updated_at": "2026-08-04T15:15:19Z"
},
{
"body": "We've identified a change that enabled GPU support for KubernetesPodOperator pods had an unintended side effect when reading pod specs without proper limits set in the expected format.\n\nWe're working on a fix, if you notice any unexpected failures for KPO tasks in your Astro deployments, please open a support ticket.",
"created_at": "2026-08-04T15:11:19Z",
"display_at": "2026-08-04T15:11:19Z",
"id": "01KZ6N8GP1FHZG1JAEVDPZ8PT5",
"incident_id": "01KZ6N8GP1FVGTB8WPZ7WT98PR",
"status": "identified",
"updated_at": "2026-08-04T15:11:19Z"
}
],
"name": "KPO podspec changes for GPU-enabled nodes may cause KPO pod startup failures",
"page_id": "01KN858NJKF1NW5FX72VHGX8D1",
"resolved_at": "2026-08-04T16:40:54Z",
"status": "resolved",
"updated_at": "2026-08-04T16:40:54Z"
},
{
"created_at": "2026-08-04T12:57:52Z",
"id": "01KZ6DM59A3EPVFYXNV8G6CH6H",
"impact": "minor",
"incident_updates": [
{
"body": "This interruption has been resolved.",
"created_at": "2026-08-04T14:36:21Z",
"display_at": "2026-08-04T14:36:21Z",
"id": "01KZ6K8GEQB8PJYFZ69M736DVX",
"incident_id": "01KZ6DM59A3EPVFYXNV8G6CH6H",
"status": "resolved",
"updated_at": "2026-08-04T14:36:21Z"
},
{
"body": "Affected clusters have recovered and are operating normally. We are continuing to monitor the environment to ensure service stability.",
"created_at": "2026-08-04T13:34:04Z",
"display_at": "2026-08-04T13:34:04Z",
"id": "01KZ6FPF3D6W9014VZAX08AXRV",
"incident_id": "01KZ6DM59A3EPVFYXNV8G6CH6H",
"status": "monitoring",
"updated_at": "2026-08-04T13:34:04Z"
},
{
"body": "We are investigating intermittent connectivity issues affecting clusters in Azure Central US when calling Astronomer APIs. This may prevent or delay attempts to create, update, or delete Airflow Deployments. Our engineering team is actively investigating the issue and working to restore normal service.",
"created_at": "2026-08-04T12:57:52Z",
"display_at": "2026-08-04T12:57:52Z",
"id": "01KZ6DM59AGT1D4N45JYPKYM6P",
"incident_id": "01KZ6DM59A3EPVFYXNV8G6CH6H",
"status": "investigating",
"updated_at": "2026-08-04T12:57:52Z"
}
],
"name": "Intermittent Delays Managing Airflow Deployments in Azure Central US",
"page_id": "01KN858NJKF1NW5FX72VHGX8D1",
"resolved_at": "2026-08-04T14:36:21Z",
"status": "resolved",
"updated_at": "2026-08-04T14:36:21Z"
},
{
"created_at": "2026-07-30T16:02:58Z",
"id": "01KYSW7GC5H1S4AE1ADFAVKNQK",
"impact": "minor",
"incident_updates": [
{
"body": "Ingestion lag is back at baseline",
"created_at": "2026-07-30T22:38:19Z",
"display_at": "2026-07-30T22:38:19Z",
"id": "01KYTJVCW5FCG8553MERACXQ3Z",
"incident_id": "01KYSW7GC5H1S4AE1ADFAVKNQK",
"status": "resolved",
"updated_at": "2026-07-30T22:38:19Z"
},
{
"body": "We are continuing to see progress as the log injestions works through the backlog. Available logs are now about ~20mins old, and we expect to see that latency return to normal baseline in the next 2 hours.",
"created_at": "2026-07-30T21:47:17Z",
"display_at": "2026-07-30T21:47:17Z",
"id": "01KYTFXYZ9TG7PW94SFAX527CK",
"incident_id": "01KYSW7GC5H1S4AE1ADFAVKNQK",
"status": "monitoring",
"updated_at": "2026-07-30T21:47:17Z"
},
{
"body": "We continue to see log ingestion improving on the platform, and we're continuing to monitor the situation.",
"created_at": "2026-07-30T20:05:23Z",
"display_at": "2026-07-30T20:05:23Z",
"id": "01KYTA3CR21HT4N13QN9G0NS23",
"incident_id": "01KYSW7GC5H1S4AE1ADFAVKNQK",
"status": "monitoring",
"updated_at": "2026-07-30T20:05:23Z"
},
{
"body": "We are beginning to see ingestion catch up with backlog log load. Customers may begin to see some logs becoming available in the Cloud UI",
"created_at": "2026-07-30T19:06:23Z",
"display_at": "2026-07-30T19:06:23Z",
"id": "01KYT6QBK6MZ2AJKDMWS27NBV6",
"incident_id": "01KYSW7GC5H1S4AE1ADFAVKNQK",
"status": "monitoring",
"updated_at": "2026-07-30T19:06:23Z"
},
{
"body": "We are continuing to investigate and monitor this issue.",
"created_at": "2026-07-30T18:43:02Z",
"display_at": "2026-07-30T18:43:02Z",
"id": "01KYT5CJRZHVWHYX17N96N4SJY",
"incident_id": "01KYSW7GC5H1S4AE1ADFAVKNQK",
"status": "identified",
"updated_at": "2026-07-30T18:43:02Z"
},
{
"body": "We have identified the source of increased log generation volume, which is causing issues on the ingestion side. We're implementing mitigations now and are monitoring the results.",
"created_at": "2026-07-30T18:03:20Z",
"display_at": "2026-07-30T18:03:20Z",
"id": "01KYT33WRAW4143151Z0BXZ6XZ",
"incident_id": "01KYSW7GC5H1S4AE1ADFAVKNQK",
"status": "identified",
"updated_at": "2026-07-30T18:03:20Z"
},
{
"body": "We are continuing to investigate this issue, and will provide another update in 30mins or as new information is available.",
"created_at": "2026-07-30T16:33:47Z",
"display_at": "2026-07-30T16:33:47Z",
"id": "01KYSXZXZD093SSEQ1VR1FG1WV",
"incident_id": "01KYSW7GC5H1S4AE1ADFAVKNQK",
"status": "investigating",
"updated_at": "2026-07-30T16:33:47Z"
},
{
"body": "We are currently investigating an issue with platform log ingestion for Astro. Logs may not be available in the Cloud UI.",
"created_at": "2026-07-30T16:02:58Z",
"display_at": "2026-07-30T16:02:58Z",
"id": "01KYSW7GC5B704C0N29V3P2X8X",
"incident_id": "01KYSW7GC5H1S4AE1ADFAVKNQK",
"status": "investigating",
"updated_at": "2026-07-30T16:02:58Z"
}
],
"name": "Platform Log ingestion issue",
"page_id": "01KN858NJKF1NW5FX72VHGX8D1",
"resolved_at": "2026-07-30T22:38:19Z",
"status": "resolved",
"updated_at": "2026-07-30T22:38:19Z"
},
{
"created_at": "2026-07-07T22:18:46Z",
"id": "01KWZAK2QDJ50ZJ6B1ZP23C9Z9",
"impact": "minor",
"incident_updates": [
{
"body": "The incident is fully resolved and all Azure clusters in EastUS2 are functioning as expected.",
"created_at": "2026-07-07T22:47:19Z",
"display_at": "2026-07-07T22:47:19Z",
"id": "01KWZC7BQ4HPM304S061MRK7EC",
"incident_id": "01KWZAK2QDJ50ZJ6B1ZP23C9Z9",
"status": "resolved",
"updated_at": "2026-07-07T22:47:19Z"
},
{
"body": "Engineers observe that affected clusters are returning to \"healthy\" status.",
"created_at": "2026-07-07T22:46:15Z",
"display_at": "2026-07-07T22:46:15Z",
"id": "01KWZC5DEXW07TB7ETQCSMCJGY",
"incident_id": "01KWZAK2QDJ50ZJ6B1ZP23C9Z9",
"status": "monitoring",
"updated_at": "2026-07-07T22:46:15Z"
},
{
"body": "We have detected node scaling issues in the Azure EastUS2 region and are currently investigating.",
"created_at": "2026-07-07T22:18:46Z",
"display_at": "2026-07-07T22:18:46Z",
"id": "01KWZAK2QDHB55HDSY6T075MSD",
"incident_id": "01KWZAK2QDJ50ZJ6B1ZP23C9Z9",
"status": "investigating",
"updated_at": "2026-07-07T22:18:46Z"
}
],
"name": "Node Scaling Issue in Azure EastUS2 Region",
"page_id": "01KN858NJKF1NW5FX72VHGX8D1",
"resolved_at": "2026-07-07T22:47:19Z",
"status": "resolved",
"updated_at": "2026-07-07T22:47:19Z"
},
{
"created_at": "2026-07-06T11:35:27Z",
"id": "01KWVKCDM7BXXGN6JQHY1D8S4D",
"impact": "major",
"incident_updates": [
{
"body": "AWS has confirmed this incident is resolved.",
"created_at": "2026-07-06T16:06:06Z",
"display_at": "2026-07-06T16:06:06Z",
"id": "01KWW2W01J4GKFPXN49H2BJHM1",
"incident_id": "01KWVKCDM7BXXGN6JQHY1D8S4D",
"status": "resolved",
"updated_at": "2026-07-06T16:06:06Z"
},
{
"body": "We are no longer seeing any impacting clusters in the AWS us-east-1 region. We're continuing to monitor the situation.",
"created_at": "2026-07-06T15:00:37Z",
"display_at": "2026-07-06T15:00:37Z",
"id": "01KWVZ4330CYA8CC1PJQSCQQ6B",
"incident_id": "01KWVKCDM7BXXGN6JQHY1D8S4D",
"status": "monitoring",
"updated_at": "2026-07-06T15:00:37Z"
},
{
"body": "AWS has now acknowledged a us-east-1 EC2 Launch Template API incident impacting EKS node provisioning/scaling (incl. Karpenter). We are continuing to monitor the situation.",
"created_at": "2026-07-06T14:24:42Z",
"display_at": "2026-07-06T14:24:42Z",
"id": "01KWVX2AD4FY48BGYXF9T1Y76T",
"incident_id": "01KWVKCDM7BXXGN6JQHY1D8S4D",
"status": "monitoring",
"updated_at": "2026-07-06T14:24:42Z"
},
{
"body": "AWS has published a status update for this issue. We are actively monitoring their progress and will continue to share updates.\n\u003chttps://health.aws.amazon.com/health/status?path=service-history\u003e",
"created_at": "2026-07-06T13:00:36Z",
"display_at": "2026-07-06T13:00:36Z",
"id": "01KWVR8ATPK8N4E85PAS7A4HCK",
"incident_id": "01KWVKCDM7BXXGN6JQHY1D8S4D",
"status": "investigating",
"updated_at": "2026-07-06T13:00:36Z"
},
{
"body": "We are investigating node scaling issues in the AWS us-east-1 region. Customers may experience issues with worker scale-up and new task startup.",
"created_at": "2026-07-06T11:35:27Z",
"display_at": "2026-07-06T11:35:27Z",
"id": "01KWVKCDM7N1DJT92Y6FQ6N1XD",
"incident_id": "01KWVKCDM7BXXGN6JQHY1D8S4D",
"status": "investigating",
"updated_at": "2026-07-06T11:56:59Z"
}
],
"name": "Node Scaling issues in AWS us-east-1 region",
"page_id": "01KN858NJKF1NW5FX72VHGX8D1",
"resolved_at": "2026-07-06T16:06:06Z",
"status": "resolved",
"updated_at": "2026-07-06T16:06:06Z"
},
{
"created_at": "2026-06-16T00:05:20Z",
"id": "01KV6VYCRMY5QK0BAYGBEN9EZ4",
"impact": "minor",
"incident_updates": [
{
"body": "This incident has been resolved.",
"created_at": "2026-06-16T01:07:01Z",
"display_at": "2026-06-16T00:53:16Z",
"id": "01KV6ZFAW693GVWW1RM6FSPVZR",
"incident_id": "01KV6VYCRMY5QK0BAYGBEN9EZ4",
"status": "resolved",
"updated_at": "2026-06-30T19:06:50Z"
},
{
"body": "A fix has been implemented and we are monitoring the results.",
"created_at": "2026-06-16T00:05:20Z",
"display_at": "2026-06-15T23:34:39Z",
"id": "01KV6VYCRMD282DAWYDHHWRC0B",
"incident_id": "01KV6VYCRMY5QK0BAYGBEN9EZ4",
"status": "monitoring",
"updated_at": "2026-06-30T19:06:50Z"
},
{
"body": "**The issue, according to Azure status page**\n\nInvestigating a spike in 401 authentication errors We are actively investigating customer alerts of 401 authentication errors when pulling images from mcr.microsoft.com. We've identified a potential underlying factor related to a recent update made to a backed service and we are actively working on determining mitigation workstreams. More information will be provided shortly.\n\nThis message was last updated at 23:04 UTC on 15 June 2026",
"created_at": "2026-06-16T00:05:20Z",
"display_at": "2026-06-15T23:28:33Z",
"id": "01KV6VYCRM96J70ESPWJN95MHN",
"incident_id": "01KV6VYCRMY5QK0BAYGBEN9EZ4",
"status": "identified",
"updated_at": "2026-06-30T19:06:50Z"
},
{
"body": "We are currently investigating this issue.",
"created_at": "2026-06-16T00:05:20Z",
"display_at": "2026-06-15T23:27:23Z",
"id": "01KV6VYCRMPHJVRZNHRFP311DV",
"incident_id": "01KV6VYCRMY5QK0BAYGBEN9EZ4",
"status": "investigating",
"updated_at": "2026-06-30T19:06:50Z"
}
],
"name": "Azure clusters may experience issues with worker scale up and task pod scheduling",
"page_id": "01KN858NJKF1NW5FX72VHGX8D1",
"resolved_at": "2026-06-16T00:53:16Z",
"status": "resolved",
"updated_at": "2026-06-30T19:06:50Z"
},
{
"created_at": "2026-06-15T16:23:54Z",
"id": "01KV61HFQ96PDBQ6M3T569D9JA",
"impact": "none",
"incident_updates": [
{
"body": "Mitigation was effective and all affected clusters are healthy.",
"created_at": "2026-06-15T17:51:44Z",
"display_at": "2026-06-15T17:51:43Z",
"id": "01KV66JAKKC3CTPE2Q96WYG7RD",
"incident_id": "01KV61HFQ96PDBQ6M3T569D9JA",
"status": "resolved",
"updated_at": "2026-06-30T19:06:50Z"
},
{
"body": "We are monitoring the affect of the applied fix and are observing error rates coming down/ceasing for affected clusters.",
"created_at": "2026-06-15T16:54:24Z",
"display_at": "2026-06-15T16:54:22Z",
"id": "01KV639AKNMH4NMMMTG14HVAMA",
"incident_id": "01KV61HFQ96PDBQ6M3T569D9JA",
"status": "monitoring",
"updated_at": "2026-06-30T19:06:50Z"
},
{
"body": "We've identified an issue with `astro deploy`s not updating manifests in Astro Hosted clusters, and we're currently applying a workaround and are continuing to monitor the situation.",
"created_at": "2026-06-15T16:23:54Z",
"display_at": "2026-06-15T16:23:51Z",
"id": "01KV61HFQ9R7XKW6BHNQTYKPJA",
"incident_id": "01KV61HFQ96PDBQ6M3T569D9JA",
"status": "identified",
"updated_at": "2026-06-30T19:06:50Z"
}
],
"name": "astro deploys not reflecting on certain Astro clusters",
"page_id": "01KN858NJKF1NW5FX72VHGX8D1",
"resolved_at": "2026-06-15T17:51:43Z",
"status": "resolved",
"updated_at": "2026-06-30T19:06:50Z"
},
{
"created_at": "2026-05-08T08:06:11Z",
"id": "01KR39YTN08XR55ENR1MXD9421",
"impact": "minor",
"incident_updates": [
{
"body": "AWS has confirmed recovery in the affected Availability Zone (use1-az4), and system performance has returned to normal levels.\n\nWe are no longer observing elevated task failures or latency in the AWS us-east-1 region. Any previously impacted workloads should now be operating as expected.\n\nIf you continue to experience issues, please reach out to Astronomer Support for assistance.",
"created_at": "2026-05-09T02:07:26Z",
"display_at": "2026-05-09T01:43:53Z",
"id": "01KR57TNE8323VT0E1T2BM6C3Z",
"incident_id": "01KR39YTN08XR55ENR1MXD9421",
"status": "resolved",
"updated_at": "2026-06-30T19:06:50Z"
},
{
"body": "We are continuing to monitor the situation as AWS works toward full recovery in the affected Availability Zone. At this time, we are seeing signs of stabilization, and most transient failures should continue to self-resolve.\n\nWe will provide further updates once AWS confirms that the issue has been fully resolved.",
"created_at": "2026-05-08T08:06:11Z",
"display_at": "2026-05-08T07:45:35Z",
"id": "01KR39YTN0MJGMVBRPWGXHYZWP",
"incident_id": "01KR39YTN08XR55ENR1MXD9421",
"status": "monitoring",
"updated_at": "2026-06-30T19:06:50Z"
},
{
"body": "We are currently observing elevated task failures and latency for some deployments running in the AWS us-east-1 region. This is related to an ongoing AWS incident affecting a single Availability Zone (use1-az4), where EC2 and EBS resources have experienced impairments.\n\nWhat to expect: You may see intermittent task failures or retries in your deployments. In most cases, these failures are transient and should self-resolve automatically as AWS continues recovery.\n\nWhat you should do: No immediate action is required. However, if you notice consistent or prolonged failures, please reach out to Astronomer Support, and we’ll help investigate further.\n\nWe are continuing to monitor the situation closely and will share updates as needed.\n\nRefer: https://health.aws.amazon.com/health/status",
"created_at": "2026-05-08T08:06:11Z",
"display_at": "2026-05-08T07:43:41Z",
"id": "01KR39YTN0N2WF3J7YDY77HT7Q",
"incident_id": "01KR39YTN08XR55ENR1MXD9421",
"status": "investigating",
"updated_at": "2026-06-30T19:06:50Z"
}
],
"name": "Increased Task Failures in US-East-1 (AWS) - AWS Incident",
"page_id": "01KN858NJKF1NW5FX72VHGX8D1",
"resolved_at": "2026-05-09T01:43:53Z",
"status": "resolved",
"updated_at": "2026-06-30T19:06:50Z"
},
{
"created_at": "2026-05-07T11:08:27Z",
"id": "01KR11ZVM86WTS15YSNYQKGY0B",
"impact": "minor",
"incident_updates": [
{
"body": "The incident has been resolved.",
"created_at": "2026-05-07T15:08:36Z",
"display_at": "2026-05-07T14:24:42Z",
"id": "01KR1FQK1RK53WHSAF6FXR0D90",
"incident_id": "01KR11ZVM86WTS15YSNYQKGY0B",
"status": "resolved",
"updated_at": "2026-06-30T19:06:50Z"
},
{
"body": "A fix has been implemented, and we are monitoring the results.",
"created_at": "2026-05-07T14:07:28Z",
"display_at": "2026-05-07T13:49:16Z",
"id": "01KR1C7MJ08GJ9W9WPQS1PQPK6",
"incident_id": "01KR11ZVM86WTS15YSNYQKGY0B",
"status": "monitoring",
"updated_at": "2026-06-30T19:06:50Z"
},
{
"body": "A fix has been prepared and is moving through deployment. We will provide another update once the rollout is complete and validation is underway.",
"created_at": "2026-05-07T13:06:43Z",
"display_at": "2026-05-07T12:57:33Z",
"id": "01KR18RDD6DKSGTZ599FM04W5P",
"incident_id": "01KR11ZVM86WTS15YSNYQKGY0B",
"status": "identified",
"updated_at": "2026-06-30T19:06:50Z"
},
{
"body": "We have identified the cause of the delayed delivery for some time-based alerts and are implementing a fix. We will provide another update once the fix has been deployed and validated.",
"created_at": "2026-05-07T12:08:58Z",
"display_at": "2026-05-07T11:15:03Z",
"id": "01KR15EMZEYFTZN6JJX555CVAJ",
"incident_id": "01KR11ZVM86WTS15YSNYQKGY0B",
"status": "identified",
"updated_at": "2026-06-30T19:06:50Z"
},
{
"body": "We are investigating delayed delivery for some time-based alerts. The delay is primarily visible for DAG Timeliness alerts, and may also affect Observe-related SLA, Proactive SLA, and Data Quality monitor alerts. DAG Duration and Task Duration alerts do not appear to be affected at this time.",
"created_at": "2026-05-07T11:08:27Z",
"display_at": "2026-05-07T10:31:36Z",
"id": "01KR11ZVM8B17H5R6Z0BH6GX5J",
"incident_id": "01KR11ZVM86WTS15YSNYQKGY0B",
"status": "investigating",
"updated_at": "2026-06-30T19:06:50Z"
}
],
"name": "Delayed delivery for some time-based alerts",
"page_id": "01KN858NJKF1NW5FX72VHGX8D1",
"resolved_at": "2026-05-07T14:24:42Z",
"status": "resolved",
"updated_at": "2026-06-30T19:06:50Z"
},
{
"created_at": "2026-05-06T20:05:31Z",
"id": "01KQZEAH1H8T5FGV8C60DQKMP9",
"impact": "major",
"incident_updates": [
{
"body": "Our team has determined this only affects internal access, Astro end users are unaffected.",
"created_at": "2026-05-06T21:05:54Z",
"display_at": "2026-05-06T20:05:27Z",
"id": "01KQZHS2WFZ1VV18ZSGSBZEXR6",
"incident_id": "01KQZEAH1H8T5FGV8C60DQKMP9",
"status": "resolved",
"updated_at": "2026-06-30T19:06:50Z"
},
{
"body": "Attempting to access the dags page in the Airflow UI results in a 403 Forbidden error. This should not be affecting task execution.",
"created_at": "2026-05-06T20:05:31Z",
"display_at": "2026-05-06T19:56:05Z",
"id": "01KQZEAH1HBXM7VPJNWQM8GYD1",
"incident_id": "01KQZEAH1H8T5FGV8C60DQKMP9",
"status": "investigating",
"updated_at": "2026-06-30T19:06:50Z"
}
],
"name": "Deployments on Astronomer Runtime 3.2-3 are returning 403 errors when accessing dags page.",
"page_id": "01KN858NJKF1NW5FX72VHGX8D1",
"resolved_at": "2026-05-06T20:05:27Z",
"status": "resolved",
"updated_at": "2026-06-30T19:06:50Z"
}
],
"page": {
"id": "01KN858NJKF1NW5FX72VHGX8D1",
"name": "Astro",
"updated_at": "2026-11-07T13:39:56Z",
"url": "https://status.astronomer.io/"
}
}