Snapshot 97190
Raw (normalized text)
Scripts and page chrome removed; this is what change detection compares.
{
"incidents": [
{
"created_at": "2026-04-27T07:46:49Z",
"id": "01KZ94ZGN0YAAZZ2BDP7BJXNS7",
"impact": "critical",
"incident_updates": [
{
"body": "On Monday, we experienced a number of issues related to expired certificates on humanitec.dev subdomains.\n\nThe issue, which began just before midnight CEST (technically, Sunday), was caused by conflicting ACME challenge records for these domains. Specifically, the domains were originally set up using a system that required TXT records for ACME challenge verification. Subsequently, as part of work to move our internal Kubernetes ingress away from ingress-nginx, a number of domains were updated to another system, which uses CNAME records for the same verifications. Although the initial certificate generation was not problematic, upon attempted renewal, the system errored out.\n\nIn our post-mortem, we discussed and have added to our roadmap the following items to mitigate this from happening in the future, across all of our domains:\n\n• Add monitoring and alerting on “time to expiration” (where it does not already exist), so that this issues are flagged well before certificate expiration.\n\n• Shorten expiration lengths in development and staging environments, so similar errors that could affect production would happen to those other environments first.\n\n• Review run book to ensure other systems migrating off of ingress-nginx do not experience similar issues.\n\nAs always, please reach out with any questions.",
"created_at": "2026-04-30T18:07:34Z",
"display_at": "2026-04-30T18:07:34Z",
"id": "01KZ94ZGN0PPMG6SXSA90ET86P",
"incident_id": "01KZ94ZGN0YAAZZ2BDP7BJXNS7",
"status": "resolved",
"updated_at": "2026-04-30T18:07:34Z"
},
{
"body": "The system is once again fully functional.",
"created_at": "2026-04-27T10:24:40Z",
"display_at": "2026-04-27T10:24:40Z",
"id": "01KZ94ZGN0E51H19FMSY224YYD",
"incident_id": "01KZ94ZGN0YAAZZ2BDP7BJXNS7",
"status": "resolved",
"updated_at": "2026-04-27T10:24:40Z"
},
{
"body": "This morning, we have experienced a number of issues related to expired certificates on the [humanitec.dev](http://humanitec.dev \"humanitec.dev\") domain.\n\nThe issue is now resolved and propagating. We are now monitoring this will, and share a detailed summary of the cause of the issue after a post-mortem meeting later this week.",
"created_at": "2026-04-27T10:15:18Z",
"display_at": "2026-04-27T10:15:17Z",
"id": "01KZ94ZGN0SQB2EMG5HHAEX15V",
"incident_id": "01KZ94ZGN0YAAZZ2BDP7BJXNS7",
"status": "monitoring",
"updated_at": "2026-04-27T10:15:18Z"
},
{
"body": "We are actively working on a fix for the certificate issue with `humanitec.dev` domains.",
"created_at": "2026-04-27T08:14:06Z",
"display_at": "2026-04-27T08:14:06Z",
"id": "01KZ94ZGN0G08VG5YYZEPP4W37",
"incident_id": "01KZ94ZGN0YAAZZ2BDP7BJXNS7",
"status": "identified",
"updated_at": "2026-04-27T08:14:06Z"
},
{
"body": "We are experiencing issues with Humanitec V2, due to issues with several TLS certificates. W are investigating",
"created_at": "2026-04-27T07:46:50Z",
"display_at": "2026-04-27T07:46:49Z",
"id": "01KZ94ZGN0JS0ZEHGJWC503ZBE",
"incident_id": "01KZ94ZGN0YAAZZ2BDP7BJXNS7",
"status": "identified",
"updated_at": "2026-04-27T07:46:50Z"
}
],
"name": "Expired certificates on *.humanitec.dev domain",
"page_id": "01KZ94ZGN0KZBMW0JREG5FDF1H",
"resolved_at": "2026-04-27T10:24:40Z",
"status": "resolved",
"updated_at": "2026-04-30T18:07:34Z"
},
{
"created_at": "2026-04-22T08:03:07Z",
"id": "01KZ94ZGN0KN45B3HNWSBZQZ85",
"impact": "major",
"incident_updates": [
{
"body": "On Wednesday, the Humanitec Platform Orchestrator was unable to deploy for customers using temporary credentials in AWS. I am reaching out to provide an overview of what happened, and steps we are taking to prevent future issues going forward.\n\n**Summary**\n\nAt approximately 9:22 CEST, as part of an infrastructure clean-up project, the AWS account that hosts the trusted user used for temporary authentication to Humanitec was destroyed, alongside a number of other no-longer-used accounts.\n\nOur team was notified of the error almost immediately, quickly identified its source, and pursued a strategy to restore the specific account through our third-party provider. (See below.) When it was clear that the response from said provider was causing undue delay, our team prepared a separate remedy, changing the trusted user associated with our temporary credentials to a new account. As there can only be one trusted user associated with temporary credentials via STS, we knew this was a backwards-incompatible operation.\n\nAt 15:10 CEST, after confirming the fix functioned as planned, we updated our production systems to reference the new user in our systems. Shortly thereafter, we began reaching out to affected customers with instructions on how to update. With this change, the issue was effectively resolved.\n\nHumanitec works with a third-party reseller to simplify management for several cloud providers. While providing some benefits to cost reporting and support, it also adds a layer of indirection when performing less-common operations like restoring a deleted account. In the end, the process from contacting the vendor to having the accoung restored took approximately 24 hours.\n\n**Factors that contributed to the issue, and associated remedies**\n\nIn review of the incident, our team identified a number of contributing factors, and associated remedies to mitigate future incidents:\n\n**Factor:** The oversight was due to a misleading name for the account – which referenced the term “demo” – and insufficient checks during planning for the deletions.\n**Remedies:** We will audit and rename all existing cloud accounts, to denote usage clearly.\n\n**Factor:** Our ability to react to this issue, in particular, was clearly hampered by having an intermediary between us and AWS.\n**Remedy:** We will shift to a direct relationship with AWS this quarter.\n\n**Factor:** In supporting temporary credentials, our set-up for AWS differs from what we do in Azure and GCP, and is less resilient. Our set-ups for Azure and GCP both use OIDC.\n**Remedy:** We have prioritized work to support using OIDC on AWS in our backlog, anticipating completion in the coming weeks. We will follow up with details on moving to the updated methodology at that time.\n\n**Factor:** Simply put, we took too long to decide to address the issue by making a breaking change. This amount of downtime is hard to stomach, regardless of source.\n**Remedy:** We are updating our incident run books to ensure they factor in time to resolution alongside impact and risk. The Incident Commander will be responsible for driving forward to resolution, alongside their existing responsibilities.\n\nWe are deeply sorry about the impact this incident may have had on your organization. Please reach out to me or your customer success team with any follow-up questions.\n\nLouis Bennett\nVP of Engineering @ Humanitec",
"created_at": "2026-04-24T12:57:58Z",
"display_at": "2026-04-24T12:57:58Z",
"id": "01KZ94ZGN0TXK7PFX1YJADMW6H",
"incident_id": "01KZ94ZGN0KN45B3HNWSBZQZ85",
"status": "resolved",
"updated_at": "2026-04-24T12:57:58Z"
},
{
"body": "We have confirmed that the change of trusted account has resolved the issue for a number of affected customers, so are marking this incident resolved.\n\nAs before, please reach out to your customer success team if you have any questions.",
"created_at": "2026-04-22T14:57:26Z",
"display_at": "2026-04-22T14:57:26Z",
"id": "01KZ94ZGN062BENTZ1XS4BKN79",
"incident_id": "01KZ94ZGN0KN45B3HNWSBZQZ85",
"status": "resolved",
"updated_at": "2026-04-22T14:57:26Z"
},
{
"body": "Earlier today, the AWS organization that hosts the trusted user used for temporary authentication to Humanitec was inadvertently destroyed. We have escalated with our AWS partner to restore the deleted organization, but rather than continue to wait – blocking deployments for customers like you – we are simply changing to a new trusted user on our end.\n\nAs a customer, you can find specifics on what you need to do in order to update things on your side here: [https://www.notion.so/humanitec/Updating-AWS-temporary-credentials-34a000565fc68066a060ded9ca275708](https://www.notion.so/humanitec/Updating-AWS-temporary-credentials-34a000565fc68066a060ded9ca275708 \"https://www.notion.so/humanitec/Updating-AWS-temporary-credentials-34a000565fc68066a060ded9ca275708\")\n\nWe are deeply sorry about the downtime and any impact that it has had on your organization. We'll be in touch in the coming days with any relevant updates as well as a summary of our post-mortem for this incident.",
"created_at": "2026-04-22T13:52:50Z",
"display_at": "2026-04-22T13:52:50Z",
"id": "01KZ94ZGN0Y24NSD70KF7TFNY6",
"incident_id": "01KZ94ZGN0KN45B3HNWSBZQZ85",
"status": "monitoring",
"updated_at": "2026-04-22T13:52:50Z"
},
{
"body": "We are currently experiencing an issue with temporary credentials use in AWS, as the AWS organization associated with the primary account used was inadvertently deleted. We have escalated with our AWS partner to restore the deleted organization, but are not able to provide a specific estimate for whether or when they will be able to restore the organization.\n\nAccordingly, we are currently testing a remediation for the issue, which should be shared with customers by 16:00 CEST.",
"created_at": "2026-04-22T12:50:07Z",
"display_at": "2026-04-22T12:50:07Z",
"id": "01KZ94ZGN05E8A7PTN47KC8E70",
"incident_id": "01KZ94ZGN0KN45B3HNWSBZQZ85",
"status": "identified",
"updated_at": "2026-04-22T12:50:07Z"
},
{
"body": "We are currently experiencing an issue with temporary credentials use in AWS, as the AWS organization associated with the primary account used was inadvertently deleted. :grimacing:\n\nWe are working with our AWS partner to restore the deleted organization, and are prepared to have customers switch to a new organization if restoring is not an option.\n\nThank for you for your patience as we address this issue, and continued apologies for the associated disruption in service.",
"created_at": "2026-04-22T10:31:50Z",
"display_at": "2026-04-22T10:31:50Z",
"id": "01KZ94ZGN0NS4HEE1PS6M8T3B3",
"incident_id": "01KZ94ZGN0KN45B3HNWSBZQZ85",
"status": "identified",
"updated_at": "2026-04-22T10:31:50Z"
},
{
"body": "We continue to pursue a remediation strategy for this issue. Our sincere apologies to those affected.",
"created_at": "2026-04-22T09:01:38Z",
"display_at": "2026-04-22T09:01:38Z",
"id": "01KZ94ZGN0SNYR2QQ5FW9NY72E",
"incident_id": "01KZ94ZGN0KN45B3HNWSBZQZ85",
"status": "identified",
"updated_at": "2026-04-22T09:01:38Z"
},
{
"body": "We have identified the probable cause of this incident, and are working to remediate it.\n\nAnother update will be provided before 11:00 CEST.",
"created_at": "2026-04-22T08:20:50Z",
"display_at": "2026-04-22T08:20:50Z",
"id": "01KZ94ZGN0P8B9SRWV586PEEP3",
"incident_id": "01KZ94ZGN0KN45B3HNWSBZQZ85",
"status": "identified",
"updated_at": "2026-04-22T08:20:50Z"
},
{
"body": "\u003e We are currently investigating an issue. It appears to be isolated to customers using AWS temporary credentials.\n\n ",
"created_at": "2026-04-22T08:03:07Z",
"display_at": "2026-04-22T08:03:07Z",
"id": "01KZ94ZGN0FZW78AEWHSTNXHZR",
"incident_id": "01KZ94ZGN0KN45B3HNWSBZQZ85",
"status": "investigating",
"updated_at": "2026-04-22T08:03:07Z"
}
],
"name": "Issues with AWS temporary credentials",
"page_id": "01KZ94ZGN0KZBMW0JREG5FDF1H",
"resolved_at": "2026-04-22T14:57:26Z",
"status": "resolved",
"updated_at": "2026-04-24T12:57:58Z"
},
{
"created_at": "2026-01-29T22:20:06Z",
"id": "01KZ94ZGN0N04JYB5EJ2SDKRTT",
"impact": "major",
"incident_updates": [
{
"body": "A change applied to our infrastructure (v1) earlier today introduced an adverse effect when the system was put under load. We have adapted the change accordingly, and consider the issue to be resolved.\n\nWe apologize for the downtime. A full post-mortem on the issue will occur in the coming week.",
"created_at": "2026-01-29T23:34:33Z",
"display_at": "2026-01-29T23:34:33Z",
"id": "01KZ94ZGN0JF8DA2MKR0D7DE13",
"incident_id": "01KZ94ZGN0N04JYB5EJ2SDKRTT",
"status": "resolved",
"updated_at": "2026-01-29T23:34:33Z"
},
{
"body": "We have identified the likely issue, and have deployed a fix to mitigate. We are currently observing to assess its impact.\n\n \n\nAt this point, service should be considered restored.",
"created_at": "2026-01-29T23:20:32Z",
"display_at": "2026-01-29T23:20:32Z",
"id": "01KZ94ZGN085AP0GPPGB3AXPFH",
"incident_id": "01KZ94ZGN0N04JYB5EJ2SDKRTT",
"status": "monitoring",
"updated_at": "2026-01-29T23:20:32Z"
},
{
"body": "We are actively investigating and experimenting with ways to mitigate the issue.",
"created_at": "2026-01-29T23:04:58Z",
"display_at": "2026-01-29T23:04:58Z",
"id": "01KZ94ZGN0KG3T7S47ZFFCAPGX",
"incident_id": "01KZ94ZGN0N04JYB5EJ2SDKRTT",
"status": "investigating",
"updated_at": "2026-01-29T23:04:58Z"
},
{
"body": "We are investigating multiple reports of availability issues in our v1 product.",
"created_at": "2026-01-29T22:20:06Z",
"display_at": "2026-01-29T22:20:06Z",
"id": "01KZ94ZGN0DE6F6KQ3W9XTQ2KP",
"incident_id": "01KZ94ZGN0N04JYB5EJ2SDKRTT",
"status": "investigating",
"updated_at": "2026-01-29T22:20:06Z"
}
],
"name": "Availability issues - v1 product",
"page_id": "01KZ94ZGN0KZBMW0JREG5FDF1H",
"resolved_at": "2026-01-29T23:34:33Z",
"status": "resolved",
"updated_at": "2026-03-12T18:16:04Z"
},
{
"created_at": "2025-11-18T12:55:33Z",
"id": "01KZ94ZGN04K2BFT2T603Y888T",
"impact": "minor",
"incident_updates": [
{
"body": "The Cloudflare incident has been resolved.",
"created_at": "2025-11-18T21:47:16Z",
"display_at": "2025-11-18T21:47:16Z",
"id": "01KZ94ZGN0YP1ZNMQ2H1Q0C90Q",
"incident_id": "01KZ94ZGN04K2BFT2T603Y888T",
"status": "resolved",
"updated_at": "2025-11-18T21:47:16Z"
},
{
"body": "The Cloudflare incident has been resolved, and their cache is coming back online. We expect sporadic issues, as with any large outage, but are considering the issue resolved on our side, as well.",
"created_at": "2025-11-18T15:16:52Z",
"display_at": "2025-11-18T15:16:52Z",
"id": "01KZ94ZGN04FYPQVCPKG3XVWC3",
"incident_id": "01KZ94ZGN04K2BFT2T603Y888T",
"status": "monitoring",
"updated_at": "2025-11-18T15:16:52Z"
},
{
"body": "Our developer and API documentation sites are currently experiencing degraded performance due to an ongoing issue with CloudFlare. We are monitoring the issue, described here: [https://www.cloudflarestatus.com/incidents/8gmgl950y3h7](https://www.cloudflarestatus.com/incidents/8gmgl950y3h7 \"https://www.cloudflarestatus.com/incidents/8gmgl950y3h7\")",
"created_at": "2025-11-18T12:55:33Z",
"display_at": "2025-11-18T12:55:33Z",
"id": "01KZ94ZGN0M822CEC9QT6KYQ92",
"incident_id": "01KZ94ZGN04K2BFT2T603Y888T",
"status": "identified",
"updated_at": "2025-11-18T12:55:33Z"
}
],
"name": "Documentation site availability degraded",
"page_id": "01KZ94ZGN0KZBMW0JREG5FDF1H",
"resolved_at": "2025-11-18T21:47:16Z",
"status": "resolved",
"updated_at": "2026-03-12T18:16:04Z"
},
{
"created_at": "2025-09-09T13:15:45Z",
"id": "01KZ94ZGN0S1H086PF2KY0N7GP",
"impact": "major",
"incident_updates": [
{
"body": "While we continue to work with out cloud vendor to troubleshoot the underlying issue introduced yesterday, we have resolved this issue from the customer’s perspective – deployments using Terraform Driver and the Humanitec-hosted runner should function as normal.",
"created_at": "2025-09-09T13:15:45Z",
"display_at": "2025-08-07T06:46:00Z",
"id": "01KZ94ZGN0ERXWB6B2V8CQXPP0",
"incident_id": "01KZ94ZGN0S1H086PF2KY0N7GP",
"status": "resolved",
"updated_at": "2025-09-11T07:07:14Z"
},
{
"body": "We are continuing to investigate the root cause with our cloud provider. In the meantime, we have established a functional alternative and are monitoring its performance.",
"created_at": "2025-09-09T13:15:45Z",
"display_at": "2025-08-06T20:37:17Z",
"id": "01KZ94ZGN0JTK0TC8V9CX9T50R",
"incident_id": "01KZ94ZGN0S1H086PF2KY0N7GP",
"status": "monitoring",
"updated_at": "2025-09-11T07:07:14Z"
},
{
"body": "We are aware of problems with deployments from the Terraform Runner that use Humanitec runners.\n\nThe issue is in our cloud provider. We are working on a fix. In the interim, customers can use self-hosted runners without issue.",
"created_at": "2025-09-09T13:15:45Z",
"display_at": "2025-08-06T19:17:09Z",
"id": "01KZ94ZGN0G43Q59W6J76FVX5Y",
"incident_id": "01KZ94ZGN0S1H086PF2KY0N7GP",
"status": "identified",
"updated_at": "2025-09-11T07:07:14Z"
}
],
"name": "Issues deploying workloads with Terraform Driver",
"page_id": "01KZ94ZGN0KZBMW0JREG5FDF1H",
"resolved_at": "2025-08-07T06:46:00Z",
"status": "resolved",
"updated_at": "2026-03-12T18:16:04Z"
},
{
"created_at": "2025-09-09T13:15:46Z",
"id": "01KZ94ZGN0MR6Z3YE0C7DVHSG0",
"impact": "critical",
"incident_updates": [
{
"body": "This incident has been resolved.",
"created_at": "2025-09-09T13:15:47Z",
"display_at": "2025-06-17T09:12:20Z",
"id": "01KZ94ZGN00M72CXP7G4QEVC2X",
"incident_id": "01KZ94ZGN0MR6Z3YE0C7DVHSG0",
"status": "resolved",
"updated_at": "2025-09-11T07:07:15Z"
},
{
"body": "We experienced an issue with automatic certificate renewal for several domains due to an issue with tokens associated with our ACME process.\n\nAs always, a post-mortem will follow to solidify fixes and process improvements to mitigate against similar issues in the future.",
"created_at": "2025-09-09T13:15:46Z",
"display_at": "2025-06-17T09:11:55Z",
"id": "01KZ94ZGN00SRTS31B7QW641QN",
"incident_id": "01KZ94ZGN0MR6Z3YE0C7DVHSG0",
"status": "monitoring",
"updated_at": "2025-09-11T07:07:15Z"
},
{
"body": "API Documentation is operational again. We are still investigating the general Documentation issues.",
"created_at": "2025-09-09T13:15:46Z",
"display_at": "2025-06-17T08:45:16Z",
"id": "01KZ94ZGN0GT9WQPQN4AMA4DNB",
"incident_id": "01KZ94ZGN0MR6Z3YE0C7DVHSG0",
"status": "identified",
"updated_at": "2025-09-11T07:07:15Z"
},
{
"body": "We are continuing to work on a fix for this issue.",
"created_at": "2025-09-09T13:15:46Z",
"display_at": "2025-06-17T08:44:44Z",
"id": "01KZ94ZGN050Y5D62EEKDAK7Z5",
"incident_id": "01KZ94ZGN0MR6Z3YE0C7DVHSG0",
"status": "identified",
"updated_at": "2025-09-11T07:07:15Z"
},
{
"body": "The issue has been identified and a fix is being implemented.",
"created_at": "2025-09-09T13:15:46Z",
"display_at": "2025-06-17T08:42:35Z",
"id": "01KZ94ZGN0NP35HKEVG4BEB6MG",
"incident_id": "01KZ94ZGN0MR6Z3YE0C7DVHSG0",
"status": "identified",
"updated_at": "2025-09-11T07:07:15Z"
},
{
"body": "We are investigating an issue with the SSL certificate for our documentation page.",
"created_at": "2025-09-09T13:15:46Z",
"display_at": "2025-06-17T06:43:11Z",
"id": "01KZ94ZGN0XJ78P8W7GQ0298CB",
"incident_id": "01KZ94ZGN0MR6Z3YE0C7DVHSG0",
"status": "investigating",
"updated_at": "2025-09-11T07:07:15Z"
}
],
"name": "Invalid SSL certificate",
"page_id": "01KZ94ZGN0KZBMW0JREG5FDF1H",
"resolved_at": "2025-06-17T09:12:20Z",
"status": "resolved",
"updated_at": "2026-03-12T18:16:04Z"
},
{
"created_at": "2025-09-09T13:15:47Z",
"id": "01KZ94ZGN0YTSHXSYRHMMVZ6WB",
"impact": "none",
"incident_updates": [
{
"body": "This incident has been resolved.",
"created_at": "2025-09-09T13:15:47Z",
"display_at": "2025-05-12T07:14:56Z",
"id": "01KZ94ZGN0V35H5M57P1761MFH",
"incident_id": "01KZ94ZGN0YTSHXSYRHMMVZ6WB",
"status": "resolved",
"updated_at": "2025-09-11T07:07:15Z"
},
{
"body": "We are continuing to investigate this issue. We have identified its probable cause and are mitigating the issue.",
"created_at": "2025-09-09T13:15:47Z",
"display_at": "2025-05-12T07:05:55Z",
"id": "01KZ94ZGN0PE05H2SC3X6N8MKC",
"incident_id": "01KZ94ZGN0YTSHXSYRHMMVZ6WB",
"status": "investigating",
"updated_at": "2025-09-11T07:07:15Z"
},
{
"body": "We are currently investigating this issue.",
"created_at": "2025-09-09T13:15:47Z",
"display_at": "2025-05-12T06:43:19Z",
"id": "01KZ94ZGN0NQHTS16YBF7FT39W",
"incident_id": "01KZ94ZGN0YTSHXSYRHMMVZ6WB",
"status": "investigating",
"updated_at": "2025-09-11T07:07:15Z"
}
],
"name": "Reports of deployment errors",
"page_id": "01KZ94ZGN0KZBMW0JREG5FDF1H",
"resolved_at": "2025-05-12T07:14:56Z",
"status": "resolved",
"updated_at": "2026-03-12T18:16:04Z"
},
{
"created_at": "2025-09-09T13:15:47Z",
"id": "01KZ94ZGN04AHD45KPAH36A55W",
"impact": "minor",
"incident_updates": [
{
"body": "This incident has been resolved.\n\nWe experienced an issue with our underlying cloud infrastructure triggered a failure in our internal DNS, which affected deployments. We will complete a full post-mortem (standard procedure), and follow-up with our cloud provider, as this is not an expected behavior.",
"created_at": "2025-09-09T13:15:47Z",
"display_at": "2025-05-05T07:36:28Z",
"id": "01KZ94ZGN033N7YT1YJCV0RBN5",
"incident_id": "01KZ94ZGN04AHD45KPAH36A55W",
"status": "resolved",
"updated_at": "2025-09-11T07:07:15Z"
},
{
"body": "A fix has been implemented and we are monitoring the results.",
"created_at": "2025-09-09T13:15:47Z",
"display_at": "2025-05-05T07:26:59Z",
"id": "01KZ94ZGN00E67J4KJD8ZSYDSQ",
"incident_id": "01KZ94ZGN04AHD45KPAH36A55W",
"status": "monitoring",
"updated_at": "2025-09-11T07:07:15Z"
},
{
"body": "We are continuing to investigate this issue.",
"created_at": "2025-09-09T13:15:47Z",
"display_at": "2025-05-05T06:54:48Z",
"id": "01KZ94ZGN0KFHHYN7BKC5PKC34",
"incident_id": "01KZ94ZGN04AHD45KPAH36A55W",
"status": "investigating",
"updated_at": "2025-09-11T07:07:15Z"
},
{
"body": "We are currently reporting reports of issues deploying",
"created_at": "2025-09-09T13:15:47Z",
"display_at": "2025-05-05T06:33:09Z",
"id": "01KZ94ZGN0PS3944HSZ7EY9R2J",
"incident_id": "01KZ94ZGN04AHD45KPAH36A55W",
"status": "investigating",
"updated_at": "2025-09-11T07:07:15Z"
}
],
"name": "Reports of errors deploying",
"page_id": "01KZ94ZGN0KZBMW0JREG5FDF1H",
"resolved_at": "2025-05-05T07:36:28Z",
"status": "resolved",
"updated_at": "2026-03-12T18:16:04Z"
},
{
"created_at": "2025-09-09T13:15:47Z",
"id": "01KZ94ZGN03XKK8Y2VBHXKW4SA",
"impact": "none",
"incident_updates": [
{
"body": "Customers using both Humanitec-hosted Terraform Driver with the Terraform Runner may be experiencing issues with deployments.\n\nThis issue was mitigated and resolved 2025-04-16 10:51 CEST.",
"created_at": "2025-09-09T13:15:47Z",
"display_at": "2025-04-15T16:36:00Z",
"id": "01KZ94ZGN0AY2ZE4NCF3AWC7NQ",
"incident_id": "01KZ94ZGN03XKK8Y2VBHXKW4SA",
"status": "resolved",
"updated_at": "2025-09-11T07:07:15Z"
}
],
"name": "Issues with deployments by Terraform Runner",
"page_id": "01KZ94ZGN0KZBMW0JREG5FDF1H",
"resolved_at": "2025-04-15T16:36:00Z",
"status": "resolved",
"updated_at": "2026-03-12T18:16:04Z"
},
{
"created_at": "2025-09-09T13:15:47Z",
"id": "01KZ94ZGN0H6GV1XH88RE39AAN",
"impact": "major",
"incident_updates": [
{
"body": "This incident was effectively mitigated at 15:34. Post-incident post-mortem + debrief will happen over the next two days.",
"created_at": "2025-09-09T13:15:48Z",
"display_at": "2025-03-31T15:27:54Z",
"id": "01KZ94ZGN0W5TPHN0YCZSHT9X5",
"incident_id": "01KZ94ZGN0H6GV1XH88RE39AAN",
"status": "resolved",
"updated_at": "2025-09-11T07:07:15Z"
},
{
"body": "We continue to monitoring the results of our mitigations.\n\nPlease continue to flag any issues you see with our Customer Success team.",
"created_at": "2025-09-09T13:15:48Z",
"display_at": "2025-03-31T14:32:23Z",
"id": "01KZ94ZGN0NY1GWF3X8ZC658HQ",
"incident_id": "01KZ94ZGN0H6GV1XH88RE39AAN",
"status": "monitoring",
"updated_at": "2025-09-11T07:07:15Z"
},
{
"body": "We are continuing to work on a fix for this issue.",
"created_at": "2025-09-09T13:15:48Z",
"display_at": "2025-03-31T13:36:22Z",
"id": "01KZ94ZGN03PX3RC8690KCKDC1",
"incident_id": "01KZ94ZGN0H6GV1XH88RE39AAN",
"status": "identified",
"updated_at": "2025-09-11T07:07:15Z"
},
{
"body": "We have mitigated the issue as we continue to investigate its root cause.\n\nAs a customer, you can expect deploy to fail as they timeout. Please re-run any deployments of environments which are inconsistent or failed.",
"created_at": "2025-09-09T13:15:48Z",
"display_at": "2025-03-31T13:36:01Z",
"id": "01KZ94ZGN0RY64ER8T6ZDSZXGX",
"incident_id": "01KZ94ZGN0H6GV1XH88RE39AAN",
"status": "identified",
"updated_at": "2025-09-11T07:07:15Z"
},
{
"body": "We are currently investigating an issue related to deployments made with Humanitec.",
"created_at": "2025-09-09T13:15:47Z",
"display_at": "2025-03-31T08:33:36Z",
"id": "01KZ94ZGN0FP7X94G85Z4PPTVX",
"incident_id": "01KZ94ZGN0H6GV1XH88RE39AAN",
"status": "investigating",
"updated_at": "2025-09-11T07:07:15Z"
}
],
"name": "Issue with new deployments",
"page_id": "01KZ94ZGN0KZBMW0JREG5FDF1H",
"resolved_at": "2025-03-31T15:27:54Z",
"status": "resolved",
"updated_at": "2026-03-12T18:16:04Z"
},
{
"created_at": "2025-09-09T13:15:48Z",
"id": "01KZ94ZGN0DEHYT5WT50JRAV81",
"impact": "none",
"incident_updates": [
{
"body": "We had an outage for deployments using secrets due to a deployed misconfiguartion from our side. The issue was introduced at 2:45 PM CET and was successfuly fixed at 3:15 PM CET.",
"created_at": "2025-09-09T13:15:49Z",
"display_at": "2024-01-08T15:00:04Z",
"id": "01KZ94ZGN06MGTSPX75Y5PRZGA",
"incident_id": "01KZ94ZGN0DEHYT5WT50JRAV81",
"status": "resolved",
"updated_at": "2025-09-11T07:07:15Z"
},
{
"body": "The issue has been identified and solved.",
"created_at": "2025-09-09T13:15:48Z",
"display_at": "2024-01-08T14:59:05Z",
"id": "01KZ94ZGN0NDBS8AFRXN6SW5BX",
"incident_id": "01KZ94ZGN0DEHYT5WT50JRAV81",
"status": "resolved",
"updated_at": "2025-09-11T07:07:15Z"
},
{
"body": "Due to an outage, certain deployments are experiencing failures. The issue has been identified and a fix is being implemented.",
"created_at": "2025-09-09T13:15:48Z",
"display_at": "2024-01-08T14:23:31Z",
"id": "01KZ94ZGN0AGW9J09H5YB6JQ6S",
"incident_id": "01KZ94ZGN0DEHYT5WT50JRAV81",
"status": "identified",
"updated_at": "2025-09-11T07:07:15Z"
}
],
"name": "Failing deployments",
"page_id": "01KZ94ZGN0KZBMW0JREG5FDF1H",
"resolved_at": "2024-01-08T14:59:05Z",
"status": "resolved",
"updated_at": "2026-03-12T18:16:04Z"
},
{
"created_at": "2025-09-09T13:15:49Z",
"id": "01KZ94ZGN0ZA4GXEC4E6WKFZBA",
"impact": "none",
"incident_updates": [
{
"body": "We had an outage for deployments using secrets due to a deployed misconfiguartion from our side. The issue was introduced at 2:45 PM CET and was successfuly fixed at 3:15 PM CET.",
"created_at": "2025-09-09T13:15:49Z",
"display_at": "2024-01-08T13:30:00Z",
"id": "01KZ94ZGN0MGA5SZQ3661VP11R",
"incident_id": "01KZ94ZGN0ZA4GXEC4E6WKFZBA",
"status": "resolved",
"updated_at": "2025-09-11T07:07:15Z"
}
],
"name": "Deployment outage",
"page_id": "01KZ94ZGN0KZBMW0JREG5FDF1H",
"resolved_at": "2024-01-08T13:30:00Z",
"status": "resolved",
"updated_at": "2026-03-12T18:16:04Z"
},
{
"created_at": "2025-09-09T13:15:49Z",
"id": "01KZ94ZGN0P6FZ7WNYX81THW3B",
"impact": "none",
"incident_updates": [
{
"body": "This incident has been resolved.",
"created_at": "2025-09-09T13:15:49Z",
"display_at": "2023-07-21T15:03:23Z",
"id": "01KZ94ZGN090AAXQ8W5RJT7H90",
"incident_id": "01KZ94ZGN0P6FZ7WNYX81THW3B",
"status": "resolved",
"updated_at": "2025-09-11T07:07:15Z"
},
{
"body": "The issue has been identified and a fix is being implemented.",
"created_at": "2025-09-09T13:15:49Z",
"display_at": "2023-07-21T11:58:33Z",
"id": "01KZ94ZGN0C6RTB3S9S05F5M05",
"incident_id": "01KZ94ZGN0P6FZ7WNYX81THW3B",
"status": "identified",
"updated_at": "2025-09-11T07:07:15Z"
}
],
"name": "Degraded performance retrieving apps for some organizations",
"page_id": "01KZ94ZGN0KZBMW0JREG5FDF1H",
"resolved_at": "2023-07-21T15:03:23Z",
"status": "resolved",
"updated_at": "2026-03-12T18:16:04Z"
},
{
"created_at": "2025-09-09T13:15:49Z",
"id": "01KZ94ZGN0GRXY7SRFZZB203MM",
"impact": "minor",
"incident_updates": [
{
"body": "This incident has been resolved.",
"created_at": "2025-09-09T13:15:49Z",
"display_at": "2023-07-21T11:56:36Z",
"id": "01KZ94ZGN09MX4GFGYMJ7B9HQE",
"incident_id": "01KZ94ZGN0GRXY7SRFZZB203MM",
"status": "resolved",
"updated_at": "2025-09-11T07:07:15Z"
},
{
"body": "The issue has been identified and a fix is being implemented.",
"created_at": "2025-09-09T13:15:49Z",
"display_at": "2023-07-21T10:41:07Z",
"id": "01KZ94ZGN0J2A4MC0C1HZFPQW2",
"incident_id": "01KZ94ZGN0GRXY7SRFZZB203MM",
"status": "identified",
"updated_at": "2025-09-11T07:07:15Z"
}
],
"name": "Degraded performance retrieving apps for some organizations",
"page_id": "01KZ94ZGN0KZBMW0JREG5FDF1H",
"resolved_at": "2023-07-21T11:56:36Z",
"status": "resolved",
"updated_at": "2026-03-12T18:16:04Z"
},
{
"created_at": "2025-09-09T13:15:49Z",
"id": "01KZ94ZGN0AA4VGC6TM7S8XDY1",
"impact": "major",
"incident_updates": [
{
"body": "This incident has been resolved. \nThe root cause has been a faulty ingress configuration which has been removed now.",
"created_at": "2025-09-09T13:15:49Z",
"display_at": "2022-10-25T07:33:55Z",
"id": "01KZ94ZGN04XWSR2FJVHP2W197",
"incident_id": "01KZ94ZGN0AA4VGC6TM7S8XDY1",
"status": "resolved",
"updated_at": "2025-09-11T07:07:15Z"
},
{
"body": "We are currently investigating an issue with our ingress controller, which seems to cause unavailability of our platform in some cases.",
"created_at": "2025-09-09T13:15:49Z",
"display_at": "2022-10-25T07:00:40Z",
"id": "01KZ94ZGN0NX5GDS5KF015BMH6",
"incident_id": "01KZ94ZGN0AA4VGC6TM7S8XDY1",
"status": "investigating",
"updated_at": "2025-09-11T07:07:15Z"
}
],
"name": "Issues with ingress controller",
"page_id": "01KZ94ZGN0KZBMW0JREG5FDF1H",
"resolved_at": "2022-10-25T07:33:55Z",
"status": "resolved",
"updated_at": "2026-03-12T18:16:04Z"
},
{
"created_at": "2025-09-09T13:15:51Z",
"id": "01KZ94ZGN0AYE84FQQTBVKCE4K",
"impact": "major",
"incident_updates": [
{
"body": "The incident has been resolved.",
"created_at": "2025-09-09T13:15:51Z",
"display_at": "2022-07-25T08:21:07Z",
"id": "01KZ94ZGN08KY1R2QB8CGBYFTK",
"incident_id": "01KZ94ZGN0AYE84FQQTBVKCE4K",
"status": "resolved",
"updated_at": "2025-09-11T07:07:15Z"
},
{
"body": "The issue has been identified and a fix is being implemented.",
"created_at": "2025-09-09T13:15:51Z",
"display_at": "2022-07-25T07:36:32Z",
"id": "01KZ94ZGN0N8N48YSVZYNZSGWY",
"incident_id": "01KZ94ZGN0AYE84FQQTBVKCE4K",
"status": "identified",
"updated_at": "2025-09-11T07:07:15Z"
}
],
"name": "Issues with deployments",
"page_id": "01KZ94ZGN0KZBMW0JREG5FDF1H",
"resolved_at": "2022-07-25T08:21:07Z",
"status": "resolved",
"updated_at": "2026-03-12T18:16:04Z"
},
{
"created_at": "2025-09-09T13:15:51Z",
"id": "01KZ94ZGN0CQMJR5PTSP7KHEST",
"impact": "none",
"incident_updates": [
{
"body": "At 3:14 PM CEST we experienced a short downtime due to a deployment causing one of our backend services to go into a crashloop. The deployment was rolled back and all system functionality was restored after 2 mins at 3:16 PM CEST",
"created_at": "2025-09-09T13:15:51Z",
"display_at": "2022-05-11T13:00:00Z",
"id": "01KZ94ZGN0GVHGBBXRPV835Y7M",
"incident_id": "01KZ94ZGN0CQMJR5PTSP7KHEST",
"status": "resolved",
"updated_at": "2025-09-11T07:07:15Z"
}
],
"name": "Downtime due to faulty deployment and connected rollback",
"page_id": "01KZ94ZGN0KZBMW0JREG5FDF1H",
"resolved_at": "2022-05-11T13:00:00Z",
"status": "resolved",
"updated_at": "2026-03-12T18:16:04Z"
},
{
"created_at": "2025-09-09T13:15:52Z",
"id": "01KZ94ZGN0BMASJYM0XM761N1X",
"impact": "none",
"incident_updates": [
{
"body": "The backend latency issues have been mitigated. Response times are back to normal.",
"created_at": "2025-09-09T13:15:52Z",
"display_at": "2022-03-31T13:02:35Z",
"id": "01KZ94ZGN028XC406D8GXECD1R",
"incident_id": "01KZ94ZGN0BMASJYM0XM761N1X",
"status": "resolved",
"updated_at": "2025-09-11T07:07:15Z"
},
{
"body": "We currently experience high latency of our backend systems and are investigating the root cause.",
"created_at": "2025-09-09T13:15:52Z",
"display_at": "2022-03-31T09:50:01Z",
"id": "01KZ94ZGN0VW4KCREDDTH9B7KZ",
"incident_id": "01KZ94ZGN0BMASJYM0XM761N1X",
"status": "identified",
"updated_at": "2025-09-11T07:07:15Z"
}
],
"name": "Issue with login to app.humanitec.io",
"page_id": "01KZ94ZGN0KZBMW0JREG5FDF1H",
"resolved_at": "2022-03-31T13:02:35Z",
"status": "resolved",
"updated_at": "2026-03-12T18:16:04Z"
},
{
"created_at": "2025-09-09T13:15:52Z",
"id": "01KZ94ZGN0A5A8GKX339JHJ7ZV",
"impact": "critical",
"incident_updates": [
{
"body": "The user login for app.humanitec.io is back to normal. Our investigation for the root cause to this issue continues.",
"created_at": "2025-09-09T13:15:52Z",
"display_at": "2022-02-24T10:49:09Z",
"id": "01KZ94ZGN0ZR5G3VS2EQXTEA88",
"incident_id": "01KZ94ZGN0A5A8GKX339JHJ7ZV",
"status": "resolved",
"updated_at": "2025-09-11T07:07:15Z"
},
{
"body": "We are currently investigating this issue.",
"created_at": "2025-09-09T13:15:52Z",
"display_at": "2022-02-24T10:22:01Z",
"id": "01KZ94ZGN0GSGPPZZATJ7NTADX",
"incident_id": "01KZ94ZGN0A5A8GKX339JHJ7ZV",
"status": "investigating",
"updated_at": "2025-09-11T07:07:15Z"
}
],
"name": "Continuous issue with login to app.humanitec.io",
"page_id": "01KZ94ZGN0KZBMW0JREG5FDF1H",
"resolved_at": "2022-02-24T10:49:09Z",
"status": "resolved",
"updated_at": "2026-03-12T18:16:04Z"
},
{
"created_at": "2025-09-09T13:15:52Z",
"id": "01KZ94ZGN03FWTQ826FA2E896B",
"impact": "none",
"incident_updates": [
{
"body": "We were able to solve the underlying issues at 10:53am CET. For now we still see slow response times, but expect those to turn back to normal in the next minutes.",
"created_at": "2025-09-09T13:15:52Z",
"display_at": "2022-02-24T09:54:24Z",
"id": "01KZ94ZGN087XPWKQTRKRM8ME6",
"incident_id": "01KZ94ZGN03FWTQ826FA2E896B",
"status": "resolved",
"updated_at": "2025-09-11T07:07:16Z"
},
{
"body": "We currently experience issues with the user login to our platform.",
"created_at": "2025-09-09T13:15:52Z",
"display_at": "2022-02-24T09:30:00Z",
"id": "01KZ94ZGN0RZ0PS7VHJ8YFMXTE",
"incident_id": "01KZ94ZGN03FWTQ826FA2E896B",
"status": "resolved",
"updated_at": "2025-09-11T07:07:16Z"
}
],
"name": "Issues with user login to app.humanitec.io",
"page_id": "01KZ94ZGN0KZBMW0JREG5FDF1H",
"resolved_at": "2022-02-24T09:30:00Z",
"status": "resolved",
"updated_at": "2026-03-12T18:16:04Z"
},
{
"created_at": "2025-09-09T13:15:53Z",
"id": "01KZ94ZGN04P24RYFT9NH5RC8B",
"impact": "none",
"incident_updates": [
{
"body": "This incident has been resolved.",
"created_at": "2025-09-09T13:15:53Z",
"display_at": "2022-01-27T19:21:24Z",
"id": "01KZ94ZGN078P9GPRR5DGEZDSK",
"incident_id": "01KZ94ZGN04P24RYFT9NH5RC8B",
"status": "resolved",
"updated_at": "2025-09-11T07:07:16Z"
},
{
"body": "Accepting an invite to an existing Organization might fail. The issue has been identified and a fix is being implemented.",
"created_at": "2025-09-09T13:15:53Z",
"display_at": "2022-01-27T08:30:40Z",
"id": "01KZ94ZGN0KXEA4CA3MDQFAVKF",
"incident_id": "01KZ94ZGN04P24RYFT9NH5RC8B",
"status": "identified",
"updated_at": "2025-09-11T07:07:16Z"
}
],
"name": "Failing user invites",
"page_id": "01KZ94ZGN0KZBMW0JREG5FDF1H",
"resolved_at": "2022-01-27T19:21:24Z",
"status": "resolved",
"updated_at": "2026-03-12T18:16:04Z"
},
{
"created_at": "2025-09-09T13:15:54Z",
"id": "01KZ94ZGN0C0WGHSZVBRNDXFB6",
"impact": "none",
"incident_updates": [
{
"body": "This incident has been resolved.",
"created_at": "2025-09-09T13:15:54Z",
"display_at": "2022-01-17T13:43:49Z",
"id": "01KZ94ZGN0Q9YX7PQ31BCXNWDQ",
"incident_id": "01KZ94ZGN0C0WGHSZVBRNDXFB6",
"status": "resolved",
"updated_at": "2025-09-11T07:07:16Z"
},
{
"body": "We are facing an issue which is causing deployments to look like they are stuck in deploying state. Our team is currently looking into this.",
"created_at": "2025-09-09T13:15:54Z",
"display_at": "2022-01-17T13:05:03Z",
"id": "01KZ94ZGN0G78JDE9BBRQWC8SB",
"incident_id": "01KZ94ZGN0C0WGHSZVBRNDXFB6",
"status": "identified",
"updated_at": "2025-09-11T07:07:16Z"
}
],
"name": "Deployments stuck in deploying state",
"page_id": "01KZ94ZGN0KZBMW0JREG5FDF1H",
"resolved_at": "2022-01-17T13:43:49Z",
"status": "resolved",
"updated_at": "2026-03-12T18:16:04Z"
},
{
"created_at": "2025-09-09T13:15:54Z",
"id": "01KZ94ZGN03RHJFRP9FHRJNM39",
"impact": "major",
"incident_updates": [
{
"body": "This incident has been resolved.",
"created_at": "2025-09-09T13:15:54Z",
"display_at": "2021-10-25T16:05:46Z",
"id": "01KZ94ZGN0AYE12B745KCX9VHF",
"incident_id": "01KZ94ZGN03RHJFRP9FHRJNM39",
"status": "resolved",
"updated_at": "2025-09-11T07:07:16Z"
},
{
"body": "A fix has been implemented and we are monitoring the results.",
"created_at": "2025-09-09T13:15:54Z",
"display_at": "2021-10-25T12:22:47Z",
"id": "01KZ94ZGN0Y3Y59G6F2BXGEYQ5",
"incident_id": "01KZ94ZGN03RHJFRP9FHRJNM39",
"status": "monitoring",
"updated_at": "2025-09-11T07:07:16Z"
},
{
"body": "Container logs are not being delivered successfully to Humanitec Backend. The backend service in charge of collection customer container logs is responding with 500 errors. We are looking into the issue. \nWe can not show any container logs of running applications on the Humanitec UI until this issue is solved.",
"created_at": "2025-09-09T13:15:54Z",
"display_at": "2021-10-25T08:13:59Z",
"id": "01KZ94ZGN00445A83Z5PBPD3FJ",
"incident_id": "01KZ94ZGN03RHJFRP9FHRJNM39",
"status": "investigating",
"updated_at": "2025-09-11T07:07:16Z"
}
],
"name": "Container logs of running applications are not shown anymore",
"page_id": "01KZ94ZGN0KZBMW0JREG5FDF1H",
"resolved_at": "2021-10-25T16:05:46Z",
"status": "resolved",
"updated_at": "2026-03-12T18:16:04Z"
},
{
"created_at": "2025-09-09T13:15:55Z",
"id": "01KZ94ZGN0WZ01TBPEJQYAXC1G",
"impact": "major",
"incident_updates": [
{
"body": "This incident has been resolved.",
"created_at": "2025-09-09T13:15:55Z",
"display_at": "2021-10-19T07:09:07Z",
"id": "01KZ94ZGN0R3BRZ7QVR6GHCWC9",
"incident_id": "01KZ94ZGN0WZ01TBPEJQYAXC1G",
"status": "resolved",
"updated_at": "2025-09-11T07:07:16Z"
},
{
"body": "The Humanitec Docs search currently throws an error for certain search terms. The issue has been reported to GitBook.",
"created_at": "2025-09-09T13:15:55Z",
"display_at": "2021-10-18T14:18:43Z",
"id": "01KZ94ZGN0BJNP6BKH1Y83V53K",
"incident_id": "01KZ94ZGN0WZ01TBPEJQYAXC1G",
"status": "monitoring",
"updated_at": "2025-09-11T07:07:16Z"
}
],
"name": "Search Issues on docs.humanitec.com",
"page_id": "01KZ94ZGN0KZBMW0JREG5FDF1H",
"resolved_at": "2021-10-19T07:09:07Z",
"status": "resolved",
"updated_at": "2026-03-12T18:16:04Z"
},
{
"created_at": "2025-09-09T13:15:55Z",
"id": "01KZ94ZGN03QV51CVEXS6P4G2V",
"impact": "major",
"incident_updates": [
{
"body": "The incident has been resolved.",
"created_at": "2025-09-09T13:15:55Z",
"display_at": "2021-10-14T15:34:38Z",
"id": "01KZ94ZGN0CQB6MW86603NG9WM",
"incident_id": "01KZ94ZGN03QV51CVEXS6P4G2V",
"status": "resolved",
"updated_at": "2025-09-11T07:07:16Z"
},
{
"body": "We are currently experiencing issues with deployments due to an update of the resources management in Humanitec",
"created_at": "2025-09-09T13:15:55Z",
"display_at": "2021-10-14T15:21:33Z",
"id": "01KZ94ZGN0VEFS4YQGMBP093NK",
"incident_id": "01KZ94ZGN03QV51CVEXS6P4G2V",
"status": "identified",
"updated_at": "2025-09-11T07:07:16Z"
}
],
"name": "Deployment Issues",
"page_id": "01KZ94ZGN0KZBMW0JREG5FDF1H",
"resolved_at": "2021-10-14T15:34:38Z",
"status": "resolved",
"updated_at": "2026-03-12T18:16:04Z"
}
],
"page": {
"id": "01KZ94ZGN0KZBMW0JREG5FDF1H",
"name": "Humanitec",
"updated_at": "2026-08-06T10:09:48Z",
"url": "https://status.humanitec.com/"
}
}