Snapshot 199917
Raw (normalized text)
Scripts and page chrome removed; this is what change detection compares.
<?xml version="1.0" encoding="UTF-8"?> <rss version="2.0" xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/"> <channel> <title>Vespa Cloud Status - Incident history</title> <link>https://status.vespa.ai</link> <description>Vespa Cloud</description> <pubDate>Mon, 2 Nov 2026 13:43:50 +0000</pubDate> <item> <title>Vespa zone impaired: dev.aws-euw1-az1</title> <description> Type: Incident Duration: 1 hour and 19 minutes Affected Components: dev.aws-euw1-az1 Nov 2, 15:02:59 GMT+0 - Resolved - This incident has been resolved. Nov 2, 14:05:09 GMT+0 - Monitoring - We have restored the infrastructure in the affected Vespa Zone, we are monitoring the situation. If you encounter any issue deploying in this zone please contact support through the [Vespa Cloud Support oportal](https://support.vespa.ai/). Nov 2, 13:43:50 GMT+0 - Investigating - At the current time we are experiencing issues with deployments in the Vespa Zone: `dev.aws-euw1-az1` We are currently investigating this incident. Running applications are unaffected. **All other zones are unaffected.** </description> <content:encoded> <..</p> <p><small>Nov <var data-var='date'> 2</var>, <var data-var='time'>13:43:50</var> GMT+0</small><br><strong>Investigating</strong> - At the current time we are experiencing issues with deployments in the Vespa Zone: `dev.aws-euw1-az1` We are currently investigating this incident. Running applications are unaffected. **All other zones are unaffected.**.</p> ]]> </content:encoded> <pubDate>Mon, 2 Nov 2026 13:43:50 +0000</pubDate> <link>https://status.vespa.ai/incident/cmvzb82fy00fm1bljm5w9sc35</link> <guid>https://status.vespa.ai/incident/cmvzb82fy00fm1bljm5w9sc35</guid> </item> <item> <title>Query timeouts for some applications</title> <description> Type: Incident Duration: 1 day, 3 hours and 24 minutes Affected Components: Infrastructure Oct 15, 19:10:02 GMT+0 - Resolved - This incident has been resolved. Oct 15, 12:48:58 GMT+0 - Monitoring - Service fully restored; a small share of metrics for some applications returns by about 16:00 UTC Oct 15, 07:57:11 GMT+0 - Monitoring - We still observe \~1.4% loss of metrics across some applications. Besides that, infrastructure has been recovered and no traffic/data loss was observed. We will publish next update at 12:00 UTC. Oct 14, 21:07:32 GMT+0 - Monitoring - Some metrics might be still missing in Grafana for some applications. We will keep monitoring the system until further update. Oct 14, 16:41:54 GMT+0 - Monitoring - The change has been rolled back and affected nodes have received new certificates. We are monitoring while everything returns to normal. Oct 14, 15:46:09 GMT+0 - Investigating - We are investigating elevated query latency and timeouts (HTTP 504) affecting some applications in Vespa Cloud. A change to internal certificate handling caused some nodes to lose connectivity within their application. We have rolled back the change and are restoring the affected nodes. Feeding and stored data are not affected. </description> <content:encoded> <![CDATA[<p><strong>Type:</strong> Incident</p> <p><strong>Duration:</strong> 1 day, 3 hours and 24 minutes</p> <p><strong>Affected Components:</strong> </p> <p><small>Oct <var data-var='date'> 15</var>, <var data-var='time'>19:10:02</var> GMT+0</small><br><strong>Resolved</strong> - This incident has been resolved..</p> <p><small>Oct <var data-var='date'> 15</var>, <var data-var='time'>12:48:58</var> GMT+0</small><br><strong>Monitoring</strong> - Service fully restored; a small share of metrics for some applications returns by about 16:00 UTC.</p> <p><small>Oct <var data-var='date'> 15</var>, <var data-var='time'>07:57:11</var> GMT+0</small><br><strong>Monitoring</strong> - We still observe \~1.4% loss of metrics across some applications. Besides that, infrastructure has been recovered and no traffic/data loss was observed. We will publish next update at 12:00 UTC..</p> <p><small>Oct <var data-var='date'> 14</var>, <var data-var='time'>21:07:32</var> GMT+0</small><br><strong>Monitoring</strong> - Some metrics might be still missing in Grafana for some applications. We will keep monitoring the system until further update..</p> <p><small>Oct <var data-var='date'> 14</var>, <var data-var='time'>16:41:54</var> GMT+0</small><br><strong>Monitoring</strong> - The change has been rolled back and affected nodes have received new certificates. We are monitoring while everything returns to normal..</p> <p><small>Oct <var data-var='date'> 14</var>, <var data-var='time'>15:46:09</var> GMT+0</small><br><strong>Investigating</strong> - We are investigating elevated query latency and timeouts (HTTP 504) affecting some applications in Vespa Cloud. A change to internal certificate handling caused some nodes to lose connectivity within their application. We have rolled back the change and are restoring the affected nodes. Feeding and stored data are not affected..</p> ]]> </content:encoded> <pubDate>Wed, 14 Oct 2026 15:46:09 +0000</pubDate> <link>https://status.vespa.ai/incident/cmv8a86n5019g0wo48nx3pyu3</link> <guid>https://status.vespa.ai/incident/cmv8a86n5019g0wo48nx3pyu3</guid> </item> <item> <title>Intermittent “Node Allocation Failure” in GCP test and staging environments</title> <description> Type: Incident Duration: 27 days, 2 hours and 1 minute Affected Components: staging.gcp-us-central1-f, prod.gcp-us-central1-a, prod.gcp-us-central1-c, prod.gcp-us-central1-f, prod.gcp-us-central1-b, test.gcp-us-central1-f, dev.gcp-us-central1-f Oct 28, 09:01:19 GMT+0 - Resolved - **What have we done?** We made several provisioning changes that reduce how often GCP test and staging deployments fail with “Node Allocation Failure”. We also added new Vespa Zones in Europe for GCP test and staging, which have much better capacity than the US zone we used before. The new zones are enabled by default for all _Vespa Cloud_ customers. We are also working with Google to resolve the capacity limits in the original zone long term. **What should you do?** For most customers, nothing. _Vespa Cloud: Enclave_ customers who haven’t yet updated the Terraform module and contacted us to start using the new zone should do so. If you still see this error, retry the run or contact support at [support@vespa.ai](mailto:support@vespa.ai). Production applications that were already running were not affected at any point. This issue applied to GCP only. We’re sorry for the disruption. Oct 21, 14:24:02 GMT+0 - Monitoring - **A quick update from the team on this issue.** ### What have we done? We have added new Vespa Zones in Europe for GCP test and staging environments. The new zones appear to have much better capacity than the US Zone we used before, so “Node Allocation Failure” errors should be much rarer. ### What should you do? Good news, for most customers: nothing! The new zone is enabled by default for all _Vespa Cloud_ customers, so you don’t need to do anything. _Vespa Cloud: Enclave_ customers should update the Terraform module and contact us through the support portal to start using the zone for test and staging. --- **This issue applies to GCP only.** We’ll keep monitoring and will mark this resolved once we’re confident the failures have stopped. If you still see this error, retry the run or contact support. Oct 1, 07:00:00 GMT+0 - Identified - Some deployments to our GCP (Google Compute Cloud) test and staging environments have been failing now and then with “Node Allocation Failure”. Google Cloud has limited compute capacity in the affected availability zone, so new machines sometimes can’t be provisioned in time. Your application isn’t causing these failures, and production applications that are already running are not affected. We have made several provisioning improvements that reduce how often this happens. We are also setting up new test and staging zones with more capacity, which we expect to be ready soon. We will update this page when they are. Until then, retrying a failed run usually works. Using the default resources for test and staging, rather than pinning a specific architecture or machine size, also lowers the chance of hitting the error. If a change is urgent, you can use “Skip tests” in the Console to deploy to production. We’re sorry for the disruption. If you are still blocked, contact [support@vespa.ai](mailto:support@vespa.ai). </description> <content:encoded> <. Production applications that were already running were not affected at any point. This issue applied to GCP only. We’re sorry for the disruption..</p> <p><small>Oct <var data-var='date'> 21</var>, <var data-var='time'>14:24:02</var> GMT+0</small><br><strong>Monitoring</strong> - **A quick update from the team on this issue.** ### What have we done? We have added new Vespa Zones in Europe for GCP test and staging environments. The new zones appear to have much better capacity than the US Zone we used before, so “Node Allocation Failure” errors should be much rarer. ### What should you do? Good news, for most customers: nothing! The new zone is enabled by default for all _Vespa Cloud_ customers, so you don’t need to do anything. _Vespa Cloud: Enclave_ customers should update the Terraform module and contact us through the support portal to start using the zone for test and staging. --- **This issue applies to GCP only.** We’ll keep monitoring and will mark this resolved once we’re confident the failures have stopped. If you still see this error, retry the run or contact support..</p> <p><small>Oct <var data-var='date'> 1</var>, <var data-var='time'>07:00:00</var> GMT+0</small><br><strong>Identified</strong> - Some deployments to our GCP (Google Compute Cloud) test and staging environments have been failing now and then with “Node Allocation Failure”. Google Cloud has limited compute capacity in the affected availability zone, so new machines sometimes can’t be provisioned in time. Your application isn’t causing these failures, and production applications that are already running are not affected. We have made several provisioning improvements that reduce how often this happens. We are also setting up new test and staging zones with more capacity, which we expect to be ready soon. We will update this page when they are. Until then, retrying a failed run usually works. Using the default resources for test and staging, rather than pinning a specific architecture or machine size, also lowers the chance of hitting the error. If a change is urgent, you can use “Skip tests” in the Console to deploy to production. We’re sorry for the disruption. If you are still blocked, contact [support@vespa.ai](mailto:support@vespa.ai)..</p> ]]> </content:encoded> <pubDate>Thu, 1 Oct 2026 07:00:00 +0000</pubDate> <link>https://status.vespa.ai/incident/cmvaxbdao08w30wp40igcph0y</link> <guid>https://status.vespa.ai/incident/cmvaxbdao08w30wp40igcph0y</guid> </item> <item> <title>Failing staging tests in AWS us-east-1c</title> <description> Type: Incident Duration: 7 hours and 27 minutes Affected Components: AWS, staging.aws-us-east-1c, Infrastructure, test.aws-us-east-1c Aug 19, 13:27:07 GMT+0 - Resolved - This incident has been resolved. Aug 19, 09:43:18 GMT+0 - Monitoring - We implemented a fix and are currently monitoring the result. Aug 19, 08:56:30 GMT+0 - Identified - We are continuing to work on a fix for this incident. Aug 19, 07:42:49 GMT+0 - Investigating - We are currently investigating this incident. Aug 19, 07:26:21 GMT+0 - Investigating - We are currently investigating this incident. Aug 19, 06:00:00 GMT+0 - Investigating - We are currently investigating this incident. </description> <content:encoded> <![CDATA[<p><strong>Type:</strong> Incident</p> <p><strong>Duration:</strong> 7 hours and 27 minutes</p> <p><strong>Affected Components:</strong> </p> <p><small>Aug <var data-var='date'> 19</var>, <var data-var='time'>13:27:07</var> GMT+0</small><br><strong>Resolved</strong> - This incident has been resolved..</p> <p><small>Aug <var data-var='date'> 19</var>, <var data-var='time'>09:43:18</var> GMT+0</small><br><strong>Monitoring</strong> - We implemented a fix and are currently monitoring the result..</p> <p><small>Aug <var data-var='date'> 19</var>, <var data-var='time'>08:56:30</var> GMT+0</small><br><strong>Identified</strong> - We are continuing to work on a fix for this incident..</p> <p><small>Aug <var data-var='date'> 19</var>, <var data-var='time'>07:42:49</var> GMT+0</small><br><strong>Investigating</strong> - We are currently investigating this incident..</p> <p><small>Aug <var data-var='date'> 19</var>, <var data-var='time'>07:26:21</var> GMT+0</small><br><strong>Investigating</strong> - We are currently investigating this incident..</p> <p><small>Aug <var data-var='date'> 19</var>, <var data-var='time'>06:00:00</var> GMT+0</small><br><strong>Investigating</strong> - We are currently investigating this incident..</p> ]]> </content:encoded> <pubDate>Wed, 19 Aug 2026 06:00:00 +0000</pubDate> <link>https://status.vespa.ai/incident/cmszrmdaq0a9d1ao03b6yj0j0</link> <guid>https://status.vespa.ai/incident/cmszrmdaq0a9d1ao03b6yj0j0</guid> </item> <item> <title>Feed blocked messages displayed in Vespa Cloud console is fake news.</title> <description> Type: Incident Duration: 1 day, 3 hours and 45 minutes Affected Components: Console Jul 4, 13:44:30 GMT+0 - Resolved - This incident has been resolved. Jul 4, 09:44:32 GMT+0 - Identified - We have startet the process to release the build with the fix to our systems Jul 3, 21:04:45 GMT+0 - Identified - We are continuing to work on a fix for this incident. ETA is unknown at the moment Jul 3, 10:00:00 GMT+0 - Identified - Please confirm with the metrics view </description> <content:encoded> <![CDATA[<p><strong>Type:</strong> Incident</p> <p><strong>Duration:</strong> 1 day, 3 hours and 45 minutes</p> <p><strong>Affected Components:</strong> </p> <p><small>Jul <var data-var='date'> 4</var>, <var data-var='time'>13:44:30</var> GMT+0</small><br><strong>Resolved</strong> - This incident has been resolved..</p> <p><small>Jul <var data-var='date'> 4</var>, <var data-var='time'>09:44:32</var> GMT+0</small><br><strong>Identified</strong> - We have startet the process to release the build with the fix to our systems.</p> <p><small>Jul <var data-var='date'> 3</var>, <var data-var='time'>21:04:45</var> GMT+0</small><br><strong>Identified</strong> - We are continuing to work on a fix for this incident. ETA is unknown at the moment.</p> <p><small>Jul <var data-var='date'> 3</var>, <var data-var='time'>10:00:00</var> GMT+0</small><br><strong>Identified</strong> - Please confirm with the metrics view.</p> ]]> </content:encoded> <pubDate>Fri, 3 Jul 2026 10:00:00 +0000</pubDate> <link>https://status.vespa.ai/incident/cmr55d2ux0g0r0kqf3vmu5ezq</link> <guid>https://status.vespa.ai/incident/cmr55d2ux0g0r0kqf3vmu5ezq</guid> </item> <item> <title>Unable to provision new endpoint certificates - affects new deployments with legacy endpoints</title> <description> Type: Incident Duration: 5 hours and 1 minute Affected Components: Infrastructure Jul 2, 23:05:46 GMT+0 - Resolved - This incident has been resolved. Jul 2, 18:05:00 GMT+0 - Identified - We are waiting for an update from LetsEncrypt at https://letsencrypt.status.io/ </description> <content:encoded> <![CDATA[<p><strong>Type:</strong> Incident</p> <p><strong>Duration:</strong> 5 hours and 1 minute</p> <p><strong>Affected Components:</strong> </p> <p><small>Jul <var data-var='date'> 2</var>, <var data-var='time'>23:05:46</var> GMT+0</small><br><strong>Resolved</strong> - This incident has been resolved..</p> <p><small>Jul <var data-var='date'> 2</var>, <var data-var='time'>18:05:00</var> GMT+0</small><br><strong>Identified</strong> - We are waiting for an update from LetsEncrypt at https://letsencrypt.status.io/.</p> ]]> </content:encoded> <pubDate>Thu, 2 Jul 2026 18:05:00 +0000</pubDate> <link>https://status.vespa.ai/incident/cmr4333r703dw0zleszzeg7ex</link> <guid>https://status.vespa.ai/incident/cmr4333r703dw0zleszzeg7ex</guid> </item> <item> <title>Authentication issues on Vespa Cloud console</title> <description> Type: Incident Duration: 42 minutes Affected Components: Console Jun 26, 07:42:07 GMT+0 - Resolved - This incident has been resolved. Jun 26, 07:22:42 GMT+0 - Monitoring - We have not observed any login issues in the past few minutes, but we are still monitoring the status. Jun 26, 07:11:38 GMT+0 - Identified - Follow the status of the issue on our service provider: <https://status.auth0.com/incidents/z8sc9m1gzqb7> Jun 26, 07:00:34 GMT+0 - Investigating - We have are received reports of elevated authentication issues and we are onvestigating. </description> <content:encoded> <![CDATA[<p><strong>Type:</strong> Incident</p> <p><strong>Duration:</strong> 42 minutes</p> <p><strong>Affected Components:</strong> </p> <p><small>Jun <var data-var='date'> 26</var>, <var data-var='time'>07:42:07</var> GMT+0</small><br><strong>Resolved</strong> - This incident has been resolved..</p> <p><small>Jun <var data-var='date'> 26</var>, <var data-var='time'>07:22:42</var> GMT+0</small><br><strong>Monitoring</strong> - We have not observed any login issues in the past few minutes, but we are still monitoring the status..</p> <p><small>Jun <var data-var='date'> 26</var>, <var data-var='time'>07:11:38</var> GMT+0</small><br><strong>Identified</strong> - Follow the status of the issue on our service provider: <https://status.auth0.com/incidents/z8sc9m1gzqb7>.</p> <p><small>Jun <var data-var='date'> 26</var>, <var data-var='time'>07:00:34</var> GMT+0</small><br><strong>Investigating</strong> - We have are received reports of elevated authentication issues and we are onvestigating..</p> ]]> </content:encoded> <pubDate>Fri, 26 Jun 2026 07:00:34 +0000</pubDate> <link>https://status.vespa.ai/incident/cmqul0kwr07nd2onxlhxs47yf</link> <guid>https://status.vespa.ai/incident/cmqul0kwr07nd2onxlhxs47yf</guid> </item> <item> <title>Vespa Cloud Console Maintenance</title> <description> Type: Maintenance Duration: 57 minutes Affected Components: Console Apr 17, 09:57:01 GMT+0 - Completed - Maintenance has completed successfully. Apr 17, 09:57:00 GMT+0 - Completed - Maintenance has completed successfully. Apr 17, 09:00:01 GMT+0 - Identified - Maintenance is now in progress Apr 17, 09:00:00 GMT+0 - Identified - We are planning for a scheduled maintenance during that time. </description> <content:encoded> <![CDATA[<p><strong>Type:</strong> Maintenance</p> <p><strong>Duration:</strong> 57 minutes</p> <p><strong>Affected Components:</strong> </p> <p><small>Apr <var data-var='date'> 17</var>, <var data-var='time'>09:57:01</var> GMT+0</small><br><strong>Completed</strong> - Maintenance has completed successfully..</p> <p><small>Apr <var data-var='date'> 17</var>, <var data-var='time'>09:57:00</var> GMT+0</small><br><strong>Completed</strong> - Maintenance has completed successfully..</p> <p><small>Apr <var data-var='date'> 17</var>, <var data-var='time'>09:00:01</var> GMT+0</small><br><strong>Identified</strong> - Maintenance is now in progress.</p> <p><small>Apr <var data-var='date'> 17</var>, <var data-var='time'>09:00:00</var> GMT+0</small><br><strong>Identified</strong> - We are planning for a scheduled maintenance during that time..</p> ]]> </content:encoded> <pubDate>Fri, 17 Apr 2026 09:00:00 +0000</pubDate> <link>https://status.vespa.ai/maintenance/cmo2n1d7q0cmwz3zj3cax0qhx</link> <guid>https://status.vespa.ai/maintenance/cmo2n1d7q0cmwz3zj3cax0qhx</guid> </item> <item> <title>Vespa Cloud metrics view is unaccessible</title> <description> Type: Incident Duration: 3 hours and 38 minutes Affected Components: Console Apr 9, 16:22:23 GMT+0 - Resolved - This incident has been resolved. Apr 9, 14:50:18 GMT+0 - Investigating - We are currently investigating this incident. Apr 9, 12:44:00 GMT+0 - Investigating - We are currently investigating this incident. </description> <content:encoded> <![CDATA[<p><strong>Type:</strong> Incident</p> <p><strong>Duration:</strong> 3 hours and 38 minutes</p> <p><strong>Affected Components:</strong> </p> <p><small>Apr <var data-var='date'> 9</var>, <var data-var='time'>16:22:23</var> GMT+0</small><br><strong>Resolved</strong> - This incident has been resolved..</p> <p><small>Apr <var data-var='date'> 9</var>, <var data-var='time'>14:50:18</var> GMT+0</small><br><strong>Investigating</strong> - We are currently investigating this incident..</p> <p><small>Apr <var data-var='date'> 9</var>, <var data-var='time'>12:44:00</var> GMT+0</small><br><strong>Investigating</strong> - We are currently investigating this incident..</p> ]]> </content:encoded> <pubDate>Thu, 9 Apr 2026 12:44:00 +0000</pubDate> <link>https://status.vespa.ai/incident/cmnrmmmtk00ccx4z76glkb4md</link> <guid>https://status.vespa.ai/incident/cmnrmmmtk00ccx4z76glkb4md</guid> </item> <item> <title>Authentication issues with Vespa Console</title> <description> Type: Incident Affected Components: Console Apr 8, 09:28:00 GMT+0 - Resolved - This incident has been resolved. Apr 8, 09:25:00 GMT+0 - Identified - We are have identified the issue and proceeding with a fix Apr 8, 09:13:00 GMT+0 - Investigating - We are currently investigating issues with authentication in Vespa Console </description> <content:encoded> <![CDATA[<p><strong>Type:</strong> Incident</p> <p><strong>Affected Components:</strong> </p> <p><small>Apr <var data-var='date'> 8</var>, <var data-var='time'>09:28:00</var> GMT+0</small><br><strong>Resolved</strong> - This incident has been resolved..</p> <p><small>Apr <var data-var='date'> 8</var>, <var data-var='time'>09:25:00</var> GMT+0</small><br><strong>Identified</strong> - We are have identified the issue and proceeding with a fix.</p> <p><small>Apr <var data-var='date'> 8</var>, <var data-var='time'>09:13:00</var> GMT+0</small><br><strong>Investigating</strong> - We are currently investigating issues with authentication in Vespa Console.</p> ]]> </content:encoded> <pubDate>Wed, 8 Apr 2026 09:13:00 +0000</pubDate> <link>https://status.vespa.ai/incident/cmnpulhk402xcx917ddu0xjpn</link> <guid>https://status.vespa.ai/incident/cmnpulhk402xcx917ddu0xjpn</guid> </item> <item> <title>Metrics pipeline broken — Grafana dashboards reporting no incoming data</title> <description> Type: Incident Duration: 2 hours and 6 minutes Affected Components: AWS, staging.gcp-us-central1-f, prod.aws-eu-west-1a, prod.aws-us-west-2a, perf.aws-us-east-1c, Infrastructure, prod.aws-us-east-1c, prod.gcp-us-central1-f, test.aws-us-east-1c, dev.aws-euw1-az1, prod.aws-apne1-az1, prod.aws-cac1-az1, prod.aws-euw1-az1, prod.gcp-us-central1-c, prod.gcp-us-central1-a, GCP, staging.aws-us-east-1c, prod.aws-euc1-az1, prod.gcp-europe-west3-b, prod.gcp-us-central1-b, prod.aws-ap-northeast-1a, prod.aws-aps1-az1, prod.aws-euc1-az3, prod.aws-use1-az4, test.gcp-us-central1-f, prod.aws-cac1-az2, prod.aws-use2-az1, prod.aws-use2-az3, prod.aws-usw2-az3, dev.aws-us-east-1c, dev.gcp-us-central1-f, Enclave → dev → prod → test → perf → Zones → Apr 1, 02:56:04 GMT+0 - Resolved - This incident has been resolved. Apr 1, 02:22:24 GMT+0 - Monitoring - We implemented a fix and are currently monitoring the result. Apr 1, 00:50:14 GMT+0 - Investigating - We are currently investigating this incident. </description> <content:encoded> <![CDATA[<p><strong>Type:</strong> Incident</p> <p><strong>Duration:</strong> 2 hours and 6 minutes</p> <p><strong>Affected Components:</strong> </p> <p><small>Apr <var data-var='date'> 1</var>, <var data-var='time'>02:56:04</var> GMT+0</small><br><strong>Resolved</strong> - This incident has been resolved..</p> <p><small>Apr <var data-var='date'> 1</var>, <var data-var='time'>02:22:24</var> GMT+0</small><br><strong>Monitoring</strong> - We implemented a fix and are currently monitoring the result..</p> <p><small>Apr <var data-var='date'> 1</var>, <var data-var='time'>00:50:14</var> GMT+0</small><br><strong>Investigating</strong> - We are currently investigating this incident..</p> ]]> </content:encoded> <pubDate>Wed, 1 Apr 2026 00:50:14 +0000</pubDate> <link>https://status.vespa.ai/incident/cmnfbx2ce0vr213atwrrjzslo</link> <guid>https://status.vespa.ai/incident/cmnfbx2ce0vr213atwrrjzslo</guid> </item> <item> <title>prod - prod.gcp-europe-west3-b isn't accessible at the moment (automated)</title> <description> Type: Incident Duration: 1 hour and 53 minutes Affected Components: prod.gcp-europe-west3-b Mar 27, 05:40:43 GMT+0 - Resolved - This incident has been resolved. Mar 27, 03:47:23 GMT+0 - Investigating - prod - prod.gcp-europe-west3-b cannot be accessed at the moment. This incident was created by an automated monitoring service. </description> <content:encoded> <![CDATA[<p><strong>Type:</strong> Incident</p> <p><strong>Duration:</strong> 1 hour and 53 minutes</p> <p><strong>Affected Components:</strong> </p> <p><small>Mar <var data-var='date'> 27</var>, <var data-var='time'>05:40:43</var> GMT+0</small><br><strong>Resolved</strong> - This incident has been resolved..</p> <p><small>Mar <var data-var='date'> 27</var>, <var data-var='time'>03:47:23</var> GMT+0</small><br><strong>Investigating</strong> - prod - prod.gcp-europe-west3-b cannot be accessed at the moment. This incident was created by an automated monitoring service..</p> ]]> </content:encoded> <pubDate>Fri, 27 Mar 2026 03:47:23 +0000</pubDate> <link>https://status.vespa.ai/incident/cmn8d1meu04a2k9ymoajvolsc</link> <guid>https://status.vespa.ai/incident/cmn8d1meu04a2k9ymoajvolsc</guid> </item> <item> <title>New nodes can fail to provision -</title> <description> Type: Incident Duration: 6 hours and 33 minutes Affected Components: AWS, prod.aws-eu-west-1a, prod.aws-us-west-2a, perf.aws-us-east-1c, prod.aws-us-east-1c, test.aws-us-east-1c, dev.aws-euw1-az1, prod.aws-apne1-az1, prod.aws-cac1-az1, prod.aws-euw1-az1, staging.aws-us-east-1c, prod.aws-euc1-az1, prod.aws-ap-northeast-1a, prod.aws-aps1-az1, prod.aws-euc1-az3, prod.aws-use1-az4, prod.aws-cac1-az2, prod.aws-use2-az1, prod.aws-use2-az3, prod.aws-usw2-az3, dev.aws-us-east-1c Mar 6, 14:03:22 GMT+0 - Resolved - This incident has been resolved. Mar 6, 12:47:19 GMT+0 - Monitoring - The rollout is compete and new nodes are able to provision. We are monitoring the situation and will post a final update later today. Mar 6, 11:43:09 GMT+0 - Identified - We have found an edge-case which has been slowing down the rollout, we're in the process of mitigating this issue. For now **new hosts** are still impacted. New update at 12:15 or earlier. There is **no impact for existing nodes.** Mar 6, 11:09:03 GMT+0 - Identified - Rollout is still in progress, still impacting **new nodes on AWS**. **Existing nodes are not impacted.** Mar 6, 10:35:34 GMT+0 - Identified - We have implemented a fix and it is currently rolling out fleet-wide. Provision of **new nodes** is still impacted, but we should resolve within 30 minutes, we will post an update as soon as possible, or by 11:00 UTC in any case. **Existing nodes are not affected.** Mar 6, 10:06:58 GMT+0 - Identified - We are continuing to work on a fix for this incident, we have noticed failures in more zones for new hosts. _Ticket updated with the new Zones._ **Existing nodes are unaffected, only provisioning of new nodes is impacted.** Mar 6, 09:07:35 GMT+0 - Identified - We have identified the root cause and started working on a workaround. Mar 6, 07:30:00 GMT+0 - Investigating - We have observed that new nodes are failing to provision. **Existing nodes are unaffected.** </description> <content:encoded> <![CDATA[<p><strong>Type:</strong> Incident</p> <p><strong>Duration:</strong> 6 hours and 33 minutes</p> <p><strong>Affected Components:</strong> </p> <p><small>Mar <var data-var='date'> 6</var>, <var data-var='time'>14:03:22</var> GMT+0</small><br><strong>Resolved</strong> - This incident has been resolved..</p> <p><small>Mar <var data-var='date'> 6</var>, <var data-var='time'>12:47:19</var> GMT+0</small><br><strong>Monitoring</strong> - The rollout is compete and new nodes are able to provision. We are monitoring the situation and will post a final update later today..</p> <p><small>Mar <var data-var='date'> 6</var>, <var data-var='time'>11:43:09</var> GMT+0</small><br><strong>Identified</strong> - We have found an edge-case which has been slowing down the rollout, we're in the process of mitigating this issue. For now **new hosts** are still impacted. New update at 12:15 or earlier. There is **no impact for existing nodes.**.</p> <p><small>Mar <var data-var='date'> 6</var>, <var data-var='time'>11:09:03</var> GMT+0</small><br><strong>Identified</strong> - Rollout is still in progress, still impacting **new nodes on AWS**. **Existing nodes are not impacted.**.</p> <p><small>Mar <var data-var='date'> 6</var>, <var data-var='time'>10:35:34</var> GMT+0</small><br><strong>Identified</strong> - We have implemented a fix and it is currently rolling out fleet-wide. Provision of **new nodes** is still impacted, but we should resolve within 30 minutes, we will post an update as soon as possible, or by 11:00 UTC in any case. **Existing nodes are not affected.**.</p> <p><small>Mar <var data-var='date'> 6</var>, <var data-var='time'>10:06:58</var> GMT+0</small><br><strong>Identified</strong> - We are continuing to work on a fix for this incident, we have noticed failures in more zones for new hosts. _Ticket updated with the new Zones._ **Existing nodes are unaffected, only provisioning of new nodes is impacted.**.</p> <p><small>Mar <var data-var='date'> 6</var>, <var data-var='time'>09:07:35</var> GMT+0</small><br><strong>Identified</strong> - We have identified the root cause and started working on a workaround..</p> <p><small>Mar <var data-var='date'> 6</var>, <var data-var='time'>07:30:00</var> GMT+0</small><br><strong>Investigating</strong> - We have observed that new nodes are failing to provision. **Existing nodes are unaffected.**.</p> ]]> </content:encoded> <pubDate>Fri, 6 Mar 2026 07:30:00 +0000</pubDate> <link>https://status.vespa.ai/incident/cmmepi8z4000d3xjzgbtmlg49</link> <guid>https://status.vespa.ai/incident/cmmepi8z4000d3xjzgbtmlg49</guid> </item> <item> <title>Issues with staging/system test deployments</title> <description> Type: Incident Duration: 22 days, 8 hours and 28 minutes Affected Components: staging.gcp-us-central1-f, test.aws-us-east-1c, staging.aws-us-east-1c, test.gcp-us-central1-f Mar 27, 07:43:29 GMT+0 - Resolved - This incident has been resolved. Mar 5, 05:45:00 GMT+0 - Monitoring - We deployed a fix and are currently monitoring the result. Mar 4, 23:15:00 GMT+0 - Identified - The root cause has been identified </description> <content:encoded> <![CDATA[<p><strong>Type:</strong> Incident</p> <p><strong>Duration:</strong> 22 days, 8 hours and 28 minutes</p> <p><strong>Affected Components:</strong> </p> <p><small>Mar <var data-var='date'> 27</var>, <var data-var='time'>07:43:29</var> GMT+0</small><br><strong>Resolved</strong> - This incident has been resolved..</p> <p><small>Mar <var data-var='date'> 5</var>, <var data-var='time'>05:45:00</var> GMT+0</small><br><strong>Monitoring</strong> - We deployed a fix and are currently monitoring the result..</p> <p><small>Mar <var data-var='date'> 4</var>, <var data-var='time'>23:15:00</var> GMT+0</small><br><strong>Identified</strong> - The root cause has been identified.</p> ]]> </content:encoded> <pubDate>Wed, 4 Mar 2026 23:15:00 +0000</pubDate> <link>https://status.vespa.ai/incident/cmmdbpczm02cpx7bgt52w6ujt</link> <guid>https://status.vespa.ai/incident/cmmdbpczm02cpx7bgt52w6ujt</guid> </item> <item> <title>Node provisioning is failing, impacting all test and staging test runs, and node changes in deployments</title> <description> Type: Incident Duration: 5 hours and 41 minutes Affected Components: Infrastructure Mar 4, 12:01:01 GMT+0 - Resolved - This incident has been resolved. Mar 4, 10:53:07 GMT+0 - Monitoring - We implemented a fix and are currently monitoring the result. Mar 4, 10:43:31 GMT+0 - Monitoring - We implemented a fix and are currently monitoring the result. Mar 4, 10:41:18 GMT+0 - Identified - We are continuing to work on a fix for this incident. We are currently testing an attempted mitigation Mar 4, 06:20:00 GMT+0 - Identified - We are investigating </description> <content:encoded> <![CDATA[<p><strong>Type:</strong> Incident</p> <p><strong>Duration:</strong> 5 hours and 41 minutes</p> <p><strong>Affected Components:</strong> </p> <p><small>Mar <var data-var='date'> 4</var>, <var data-var='time'>12:01:01</var> GMT+0</small><br><strong>Resolved</strong> - This incident has been resolved..</p> <p><small>Mar <var data-var='date'> 4</var>, <var data-var='time'>10:53:07</var> GMT+0</small><br><strong>Monitoring</strong> - We implemented a fix and are currently monitoring the result..</p> <p><small>Mar <var data-var='date'> 4</var>, <var data-var='time'>10:43:31</var> GMT+0</small><br><strong>Monitoring</strong> - We implemented a fix and are currently monitoring the result..</p> <p><small>Mar <var data-var='date'> 4</var>, <var data-var='time'>10:41:18</var> GMT+0</small><br><strong>Identified</strong> - We are continuing to work on a fix for this incident. We are currently testing an attempted mitigation.</p> <p><small>Mar <var data-var='date'> 4</var>, <var data-var='time'>06:20:00</var> GMT+0</small><br><strong>Identified</strong> - We are investigating.</p> ]]> </content:encoded> <pubDate>Wed, 4 Mar 2026 06:20:00 +0000</pubDate> <link>https://status.vespa.ai/incident/cmmbs9mim00nl10ssrdpw94vi</link> <guid>https://status.vespa.ai/incident/cmmbs9mim00nl10ssrdpw94vi</guid> </item> <item> <title>Issues with staging/system test deployments</title> <description> Type: Incident Duration: 7 hours and 40 minutes Affected Components: test.aws-us-east-1c, staging.gcp-us-central1-f, staging.aws-us-east-1c, test.gcp-us-central1-f Mar 4, 06:55:00 GMT+0 - Resolved - The root cause has been identified and mitigated. Mar 3, 23:15:00 GMT+0 - Investigating - There was a period where some staging and system test deployments failed. </description> <content:encoded> <![CDATA[<p><strong>Type:</strong> Incident</p> <p><strong>Duration:</strong> 7 hours and 40 minutes</p> <p><strong>Affected Components:</strong> </p> <p><small>Mar <var data-var='date'> 4</var>, <var data-var='time'>06:55:00</var> GMT+0</small><br><strong>Resolved</strong> - The root cause has been identified and mitigated..</p> <p><small>Mar <var data-var='date'> 3</var>, <var data-var='time'>23:15:00</var> GMT+0</small><br><strong>Investigating</strong> - There was a period where some staging and system test deployments failed..</p> ]]> </content:encoded> <pubDate>Tue, 3 Mar 2026 23:15:00 +0000</pubDate> <link>https://status.vespa.ai/incident/cmmbwbt670n5p50p1wokttahl</link> <guid>https://status.vespa.ai/incident/cmmbwbt670n5p50p1wokttahl</guid> </item> <item> <title>Issues with staging/system test deployments</title> <description> Type: Incident Duration: 1 day and 1 hour Affected Components: test.aws-us-east-1c, staging.gcp-us-central1-f, staging.aws-us-east-1c, test.gcp-us-central1-f Mar 1, 15:00:00 GMT+0 - Resolved - This incident has been resolved. Feb 28, 14:00:00 GMT+0 - Investigating - There was a period where some staging and system test deployments failed. The root cause has been identified and fixed. </description> <content:encoded> <![CDATA[<p><strong>Type:</strong> Incident</p> <p><strong>Duration:</strong> 1 day and 1 hour</p> <p><strong>Affected Components:</strong> </p> <p><small>Mar <var data-var='date'> 1</var>, <var data-var='time'>15:00:00</var> GMT+0</small><br><strong>Resolved</strong> - This incident has been resolved..</p> <p><small>Feb <var data-var='date'> 28</var>, <var data-var='time'>14:00:00</var> GMT+0</small><br><strong>Investigating</strong> - There was a period where some staging and system test deployments failed. The root cause has been identified and fixed..</p> ]]> </content:encoded> <pubDate>Sat, 28 Feb 2026 14:00:00 +0000</pubDate> <link>https://status.vespa.ai/incident/cmm9dyv1w0af46l72zsgjqgfs</link> <guid>https://status.vespa.ai/incident/cmm9dyv1w0af46l72zsgjqgfs</guid> </item> <item> <title>Vespa Cloud Console Maintenance</title> <description> Type: Maintenance Duration: 12 minutes Affected Components: Console Feb 20, 11:11:58 GMT+0 - Completed - Maintenance has completed successfully. Feb 20, 11:00:01 GMT+0 - Identified - Maintenance is now in progress Feb 20, 11:00:00 GMT+0 - Identified - We are planning for a scheduled maintenance during that time. No downtime is expected. </description> <content:encoded> <![CDATA[<p><strong>Type:</strong> Maintenance</p> <p><strong>Duration:</strong> 12 minutes</p> <p><strong>Affected Components:</strong> </p> <p><small>Feb <var data-var='date'> 20</var>, <var data-var='time'>11:11:58</var> GMT+0</small><br><strong>Completed</strong> - Maintenance has completed successfully..</p> <p><small>Feb <var data-var='date'> 20</var>, <var data-var='time'>11:00:01</var> GMT+0</small><br><strong>Identified</strong> - Maintenance is now in progress.</p> <p><small>Feb <var data-var='date'> 20</var>, <var data-var='time'>11:00:00</var> GMT+0</small><br><strong>Identified</strong> - We are planning for a scheduled maintenance during that time. No downtime is expected..</p> ]]> </content:encoded> <pubDate>Fri, 20 Feb 2026 11:00:00 +0000</pubDate> <link>https://status.vespa.ai/maintenance/cmlthtoio04qj24hsd6x76jaq</link> <guid>https://status.vespa.ai/maintenance/cmlthtoio04qj24hsd6x76jaq</guid> </item> <item> <title>Issues with new host provisioning</title> <description> Type: Incident Duration: 3 hours and 48 minutes Affected Components: AWS, prod.gcp-europe-west3-b, test.aws-us-east-1c, staging.gcp-us-central1-f, staging.aws-us-east-1c, prod.aws-eu-west-1a, prod.aws-us-west-2a, perf.aws-us-east-1c, GCP, Infrastructure, prod.aws-us-east-1c, prod.gcp-us-central1-f, test.gcp-us-central1-f, dev.aws-us-east-1c, dev.gcp-us-central1-f, Console Feb 10, 15:15:59 GMT+0 - Resolved - This incident has been resolved. Feb 10, 12:59:35 GMT+0 - Monitoring - We implemented a fix and are currently monitoring the result. Feb 10, 12:39:50 GMT+0 - Identified - We have identified the root cause and are taking the necessary steps to resolve the issue. Feb 10, 11:28:07 GMT+0 - Investigating - We are experiencing a technical issue that is impacting provisioning of new Vespa hosts. Provisioning requests may fail or be delayed while we work to resolve the problem. **Existing deployments remain unaffected.** We are currently investigating this incident. </description> <content:encoded> <![CDATA[<p><strong>Type:</strong> Incident</p> <p><strong>Duration:</strong> 3 hours and 48 minutes</p> <p><strong>Affected Components:</strong> </p> <p><small>Feb <var data-var='date'> 10</var>, <var data-var='time'>15:15:59</var> GMT+0</small><br><strong>Resolved</strong> - This incident has been resolved..</p> <p><small>Feb <var data-var='date'> 10</var>, <var data-var='time'>12:59:35</var> GMT+0</small><br><strong>Monitoring</strong> - We implemented a fix and are currently monitoring the result..</p> <p><small>Feb <var data-var='date'> 10</var>, <var data-var='time'>12:39:50</var> GMT+0</small><br><strong>Identified</strong> - We have identified the root cause and are taking the necessary steps to resolve the issue..</p> <p><small>Feb <var data-var='date'> 10</var>, <var data-var='time'>11:28:07</var> GMT+0</small><br><strong>Investigating</strong> - We are experiencing a technical issue that is impacting provisioning of new Vespa hosts. Provisioning requests may fail or be delayed while we work to resolve the problem. **Existing deployments remain unaffected.** We are currently investigating this incident..</p> ]]> </content:encoded> <pubDate>Tue, 10 Feb 2026 11:28:07 +0000</pubDate> <link>https://status.vespa.ai/incident/cmlgiost217l05p7vd6jowrgy</link> <guid>https://status.vespa.ai/incident/cmlgiost217l05p7vd6jowrgy</guid> </item> <item> <title>Vespa Cloud Console Slow or Unavailable</title> <description> Type: Incident Duration: 43 minutes Affected Components: Console Feb 5, 21:28:21 GMT+0 - Resolved - This incident has been resolved, we have had a misconfiguration in autoscaling which prevented an immediate automatic resolution. We have implemented a workaround and will implement a permanent fix in the next days. Feb 5, 21:07:46 GMT+0 - Identified - We have identified the issue and are working on a workaround. Feb 5, 20:45:00 GMT+0 - Investigating - We are aware of slowdowns and downtime in the Vespa Console, we are currently investigating this incident. </description> <content:encoded> <![CDATA[<p><strong>Type:</strong> Incident</p> <p><strong>Duration:</strong> 43 minutes</p> <p><strong>Affected Components:</strong> </p> <p><small>Feb <var data-var='date'> 5</var>, <var data-var='time'>21:28:21</var> GMT+0</small><br><strong>Resolved</strong> - This incident has been resolved, we have had a misconfiguration in autoscaling which prevented an immediate automatic resolution. We have implemented a workaround and will implement a permanent fix in the next days..</p> <p><small>Feb <var data-var='date'> 5</var>, <var data-var='time'>21:07:46</var> GMT+0</small><br><strong>Identified</strong> - We have identified the issue and are working on a workaround..</p> <p><small>Feb <var data-var='date'> 5</var>, <var data-var='time'>20:45:00</var> GMT+0</small><br><strong>Investigating</strong> - We are aware of slowdowns and downtime in the Vespa Console, we are currently investigating this incident..</p> ]]> </content:encoded> <pubDate>Thu, 5 Feb 2026 20:45:00 +0000</pubDate> <link>https://status.vespa.ai/incident/cml9y5r7v00033m8w7z1ezl9u</link> <guid>https://status.vespa.ai/incident/cml9y5r7v00033m8w7z1ezl9u</guid> </item> <item> <title>Missing metrics from GCP and enclave</title> <description> Type: Incident Duration: 39 minutes Affected Components: Console Jan 17, 10:31:32 GMT+0 - Resolved - This incident has been resolved. Jan 17, 10:16:09 GMT+0 - Monitoring - We implemented a fix and are currently monitoring the result. Jan 17, 09:52:58 GMT+0 - Investigating - The metrics shown in console are incomplete for applications running in GCP and enclave. We are currently investigating this incident. </description> <content:encoded> <![CDATA[<p><strong>Type:</strong> Incident</p> <p><strong>Duration:</strong> 39 minutes</p> <p><strong>Affected Components:</strong> </p> <p><small>Jan <var data-var='date'> 17</var>, <var data-var='time'>10:31:32</var> GMT+0</small><br><strong>Resolved</strong> - This incident has been resolved..</p> <p><small>Jan <var data-var='date'> 17</var>, <var data-var='time'>10:16:09</var> GMT+0</small><br><strong>Monitoring</strong> - We implemented a fix and are currently monitoring the result..</p> <p><small>Jan <var data-var='date'> 17</var>, <var data-var='time'>09:52:58</var> GMT+0</small><br><strong>Investigating</strong> - The metrics shown in console are incomplete for applications running in GCP and enclave. We are currently investigating this incident..</p> ]]> </content:encoded> <pubDate>Sat, 17 Jan 2026 09:52:58 +0000</pubDate> <link>https://status.vespa.ai/incident/cmki4pzur073e8nn6utsrf1x3</link> <guid>https://status.vespa.ai/incident/cmki4pzur073e8nn6utsrf1x3</guid> </item> <item> <title>Missing Metrics for GCP and Enclave Accounts</title> <description> Type: Incident Duration: 8 hours and 39 minutes Affected Components: prod.gcp-europe-west3-b, staging.gcp-us-central1-f, GCP, prod.gcp-us-central1-f, test.gcp-us-central1-f Jan 4, 20:58:36 GMT+0 - Resolved - This incident has been resolved. Jan 4, 19:57:49 GMT+0 - Monitoring - We have applied a fix and are currently monitoring the result. Jan 4, 12:20:00 GMT+0 - Investigating - We are currently investigating this incident. </description> <content:encoded> <![CDATA[<p><strong>Type:</strong> Incident</p> <p><strong>Duration:</strong> 8 hours and 39 minutes</p> <p><strong>Affected Components:</strong> </p> <p><small>Jan <var data-var='date'> 4</var>, <var data-var='time'>20:58:36</var> GMT+0</small><br><strong>Resolved</strong> - This incident has been resolved..</p> <p><small>Jan <var data-var='date'> 4</var>, <var data-var='time'>19:57:49</var> GMT+0</small><br><strong>Monitoring</strong> - We have applied a fix and are currently monitoring the result..</p> <p><small>Jan <var data-var='date'> 4</var>, <var data-var='time'>12:20:00</var> GMT+0</small><br><strong>Investigating</strong> - We are currently investigating this incident..</p> ]]> </content:encoded> <pubDate>Sun, 4 Jan 2026 12:20:00 +0000</pubDate> <link>https://status.vespa.ai/incident/cmk03c8690123et5dbqlkmarl</link> <guid>https://status.vespa.ai/incident/cmk03c8690123et5dbqlkmarl</guid> </item> <item> <title>Missing Metrics for GCP and Enclave (All)</title> <description> Type: Incident Duration: 5 days and 30 minutes Affected Components: AWS, prod.gcp-europe-west3-b, staging.gcp-us-central1-f, GCP, prod.gcp-us-central1-f, dev.gcp-us-central1-f Dec 17, 11:39:08 GMT+0 - Resolved - This incident has been resolved. Dec 16, 16:47:27 GMT+0 - Monitoring - We have applied a fix and are currently monitoring the result. Dec 16, 10:24:22 GMT+0 - Identified - The first attempts to resolve this failed We are continuing to work on a fix for this incident. Dec 12, 15:00:00 GMT+0 - Monitoring - We implemented a fix and are currently monitoring the progress. Dec 12, 11:09:00 GMT+0 - Investigating - We identified an issue which is preventing metrics to be published to our systems.. We are investigating the issue. </description> <content:encoded> <![CDATA[<p><strong>Type:</strong> Incident</p> <p><strong>Duration:</strong> 5 days and 30 minutes</p> <p><strong>Affected Components:</strong> </p> <p><small>Dec <var data-var='date'> 17</var>, <var data-var='time'>11:39:08</var> GMT+0</small><br><strong>Resolved</strong> - This incident has been resolved..</p> <p><small>Dec <var data-var='date'> 16</var>, <var data-var='time'>16:47:27</var> GMT+0</small><br><strong>Monitoring</strong> - We have applied a fix and are currently monitoring the result..</p> <p><small>Dec <var data-var='date'> 16</var>, <var data-var='time'>10:24:22</var> GMT+0</small><br><strong>Identified</strong> - The first attempts to resolve this failed We are continuing to work on a fix for this incident..</p> <p><small>Dec <var data-var='date'> 12</var>, <var data-var='time'>15:00:00</var> GMT+0</small><br><strong>Monitoring</strong> - We implemented a fix and are currently monitoring the progress..</p> <p><small>Dec <var data-var='date'> 12</var>, <var data-var='time'>11:09:00</var> GMT+0</small><br><strong>Investigating</strong> - We identified an issue which is preventing metrics to be published to our systems.. We are investigating the issue..</p> ]]> </content:encoded> <pubDate>Fri, 12 Dec 2025 11:09:00 +0000</pubDate> <link>https://status.vespa.ai/incident/cmj32ve7r070j12dnedon30sv</link> <guid>https://status.vespa.ai/incident/cmj32ve7r070j12dnedon30sv</guid> </item> <item> <title>Temporary downtime of sign in on Vespa Cloud Console</title> <description> Type: Maintenance Duration: 30 minutes Affected Components: Console Dec 9, 13:30:00 GMT+0 - Completed - Maintenance has completed successfully Dec 9, 13:00:01 GMT+0 - Identified - Maintenance is now in progress Dec 9, 13:00:00 GMT+0 - Identified - We are preparing to perform maintenance on the sign-in functionality on the Vespa Cloud Console. Sign in functionality may be sporadically unavailable for up to 30 minutes. This will affect only the control plane login, data plane authentication will remain functional during this maintenance window. </description> <content:encoded> <![CDATA[<p><strong>Type:</strong> Maintenance</p> <p><strong>Duration:</strong> 30 minutes</p> <p><strong>Affected Components:</strong> </p> <p><small>Dec <var data-var='date'> 9</var>, <var data-var='time'>13:30:00</var> GMT+0</small><br><strong>Completed</strong> - Maintenance has completed successfully.</p> <p><small>Dec <var data-var='date'> 9</var>, <var data-var='time'>13:00:01</var> GMT+0</small><br><strong>Identified</strong> - Maintenance is now in progress.</p> <p><small>Dec <var data-var='date'> 9</var>, <var data-var='time'>13:00:00</var> GMT+0</small><br><strong>Identified</strong> - We are preparing to perform maintenance on the sign-in functionality on the Vespa Cloud Console. Sign in functionality may be sporadically unavailable for up to 30 minutes. This will affect only the control plane login, data plane authentication will remain functional during this maintenance window..</p> ]]> </content:encoded> <pubDate>Tue, 9 Dec 2025 13:00:00 +0000</pubDate> <link>https://status.vespa.ai/maintenance/cmiyl2l2d07k9c8n9oaf29fu3</link> <guid>https://status.vespa.ai/maintenance/cmiyl2l2d07k9c8n9oaf29fu3</guid> </item> <item> <title>AWS experiences issues in us-east-1</title> <description> Type: Incident Duration: 23 hours and 36 minutes Affected Components: AWS, dev.aws-us-east-1c, test.aws-us-east-1c, staging.aws-us-east-1c, perf.aws-us-east-1c, Infrastructure, prod.aws-us-east-1c Oct 21, 06:47:23 GMT+0 - Resolved - This incident has been resolved. Oct 20, 20:58:27 GMT+0 - Monitoring - Infrastructure services are now opertional. We continue to monitor our system performance to detect residual issues. Oct 20, 16:50:39 GMT+0 - Monitoring - We are still experiencing the failures with provisioning new hosts on AWS us-east region. Therefore, we recommend to avoid applying changes to the applications on this region until the issue is resolved on AWS side. Oct 20, 10:26:36 GMT+0 - Monitoring - Infrastructure services restored, we will continue to monitor the situation for any residual issues. Oct 20, 07:11:00 GMT+0 - Identified - Track status on <https://health.aws.amazon.com/health/status> Existing applications are not affected by outage. Only maintenance operations are degraded. </description> <content:encoded> <![CDATA[<p><strong>Type:</strong> Incident</p> <p><strong>Duration:</strong> 23 hours and 36 minutes</p> <p><strong>Affected Components:</strong> </p> <p><small>Oct <var data-var='date'> 21</var>, <var data-var='time'>06:47:23</var> GMT+0</small><br><strong>Resolved</strong> - This incident has been resolved..</p> <p><small>Oct <var data-var='date'> 20</var>, <var data-var='time'>20:58:27</var> GMT+0</small><br><strong>Monitoring</strong> - Infrastructure services are now opertional. We continue to monitor our system performance to detect residual issues..</p> <p><small>Oct <var data-var='date'> 20</var>, <var data-var='time'>16:50:39</var> GMT+0</small><br><strong>Monitoring</strong> - We are still experiencing the failures with provisioning new hosts on AWS us-east region. Therefore, we recommend to avoid applying changes to the applications on this region until the issue is resolved on AWS side. .</p> <p><small>Oct <var data-var='date'> 20</var>, <var data-var='time'>10:26:36</var> GMT+0</small><br><strong>Monitoring</strong> - Infrastructure services restored, we will continue to monitor the situation for any residual issues..</p> <p><small>Oct <var data-var='date'> 20</var>, <var data-var='time'>07:11:00</var> GMT+0</small><br><strong>Identified</strong> - Track status on <https://health.aws.amazon.com/health/status> Existing applications are not affected by outage. Only maintenance operations are degraded..</p> ]]> </content:encoded> <pubDate>Mon, 20 Oct 2025 07:11:00 +0000</pubDate> <link>https://status.vespa.ai/incident/cmgytgo3p00mlmx24ihc87kdx</link> <guid>https://status.vespa.ai/incident/cmgytgo3p00mlmx24ihc87kdx</guid> </item> <item> <title>Issues with metric dashboards in Vespa Console</title> <description> Type: Incident Duration: 3 hours and 13 minutes Affected Components: Console Aug 22, 15:08:28 GMT+0 - Resolved - This incident on Grafana is still open, but we don't observe any effect on our side. Aug 22, 11:55:00 GMT+0 - Monitoring - We have identified an issue in our vespa Console, when clicking on "Metrics" for an application we you might experience "Failed to load dashboard - Internal Server Error" This is caused by an outage in Grafana, for more details please refer to \[the Grafana Status Page\](https://status.grafana.com/incidents/z42xrjqydny8) updates. </description> <content:encoded> < updates..</p> ]]> </content:encoded> <pubDate>Fri, 22 Aug 2025 11:55:00 +0000</pubDate> <link>https://status.vespa.ai/incident/cmems9mnk0001107un22ahtkn</link> <guid>https://status.vespa.ai/incident/cmems9mnk0001107un22ahtkn</guid> </item> <item> <title>Provisioning new nodes might fail</title> <description> Type: Incident Duration: 1 hour and 12 minutes Affected Components: perf.aws-us-east-1c, test.gcp-us-central1-f, prod.aws-us-east-1c, prod.gcp-europe-west3-b, prod.gcp-us-central1-f, staging.gcp-us-central1-f, test.aws-us-east-1c, staging.aws-us-east-1c, dev.aws-us-east-1c, prod.aws-eu-west-1a, prod.aws-us-west-2a, dev.gcp-us-central1-f, Zones → Jul 2, 07:43:51 GMT+0 - Resolved - This incident has been resolved. Jul 2, 07:00:57 GMT+0 - Monitoring - Workaround is in place, so it should be possible to provision new nodes. If you encounter issues don't hesitate to contact support. Jul 2, 06:47:51 GMT+0 - Identified - We have identified the root cause and we are applying a mitigation to allow nodes to be provisioned. Jul 2, 06:31:58 GMT+0 - Investigating - We are currently investigating this incident. </description> <content:encoded> <![CDATA[<p><strong>Type:</strong> Incident</p> <p><strong>Duration:</strong> 1 hour and 12 minutes</p> <p><strong>Affected Components:</strong> </p> <p><small>Jul <var data-var='date'> 2</var>, <var data-var='time'>07:43:51</var> GMT+0</small><br><strong>Resolved</strong> - This incident has been resolved..</p> <p><small>Jul <var data-var='date'> 2</var>, <var data-var='time'>07:00:57</var> GMT+0</small><br><strong>Monitoring</strong> - Workaround is in place, so it should be possible to provision new nodes. If you encounter issues don't hesitate to contact support..</p> <p><small>Jul <var data-var='date'> 2</var>, <var data-var='time'>06:47:51</var> GMT+0</small><br><strong>Identified</strong> - We have identified the root cause and we are applying a mitigation to allow nodes to be provisioned..</p> <p><small>Jul <var data-var='date'> 2</var>, <var data-var='time'>06:31:58</var> GMT+0</small><br><strong>Investigating</strong> - We are currently investigating this incident..</p> ]]> </content:encoded> <pubDate>Wed, 2 Jul 2025 06:31:58 +0000</pubDate> <link>https://status.vespa.ai/incident/cmclkwzow01fxt1chtevzvasm</link> <guid>https://status.vespa.ai/incident/cmclkwzow01fxt1chtevzvasm</guid> </item> <item> <title>Metrics dashboard not working for GCP and enclave applications</title> <description> Type: Incident Duration: 1 hour and 1 minute Affected Components: Console Jun 26, 13:18:09 GMT+0 - Resolved - This incident has been resolved. Jun 26, 12:17:20 GMT+0 - Investigating - New data will not be populated in the dashboard, while historical data is still visible. We are currently investigating this incident. </description> <content:encoded> <![CDATA[<p><strong>Type:</strong> Incident</p> <p><strong>Duration:</strong> 1 hour and 1 minute</p> <p><strong>Affected Components:</strong> </p> <p><small>Jun <var data-var='date'> 26</var>, <var data-var='time'>13:18:09</var> GMT+0</small><br><strong>Resolved</strong> - This incident has been resolved..</p> <p><small>Jun <var data-var='date'> 26</var>, <var data-var='time'>12:17:20</var> GMT+0</small><br><strong>Investigating</strong> - New data will not be populated in the dashboard, while historical data is still visible. We are currently investigating this incident. .</p> ]]> </content:encoded> <pubDate>Thu, 26 Jun 2025 12:17:20 +0000</pubDate> <link>https://status.vespa.ai/incident/cmcdcm03p002f7gde7sg7nl6g</link> <guid>https://status.vespa.ai/incident/cmcdcm03p002f7gde7sg7nl6g</guid> </item> <item> <title>Vespa Cloud console maintenance</title> <description> Type: Maintenance Duration: 5 minutes Affected Components: Console Sep 4, 07:20:00 GMT+0 - Completed - Maintenance has completed successfully Sep 4, 07:15:01 GMT+0 - Identified - Maintenance is now in progress Sep 4, 07:15:00 GMT+0 - Identified - We are planning a Vespa Cloud console maintenance. The console will be unavailable for a few minutes </description> <content:encoded> <![CDATA[<p><strong>Type:</strong> Maintenance</p> <p><strong>Duration:</strong> 5 minutes</p> <p><strong>Affected Components:</strong> </p> <p><small>Sep <var data-var='date'> 4</var>, <var data-var='time'>07:20:00</var> GMT+0</small><br><strong>Completed</strong> - Maintenance has completed successfully.</p> <p><small>Sep <var data-var='date'> 4</var>, <var data-var='time'>07:15:01</var> GMT+0</small><br><strong>Identified</strong> - Maintenance is now in progress.</p> <p><small>Sep <var data-var='date'> 4</var>, <var data-var='time'>07:15:00</var> GMT+0</small><br><strong>Identified</strong> - We are planning a Vespa Cloud console maintenance. The console will be unavailable for a few minutes.</p> ]]> </content:encoded> <pubDate>Wed, 4 Sep 2024 07:15:00 +0000</pubDate> <link>https://status.vespa.ai/maintenance/cm0niqq02004vl3c8tq3x35xq</link> <guid>https://status.vespa.ai/maintenance/cm0niqq02004vl3c8tq3x35xq</guid> </item> </channel> </rss>