Snapshot 316479
Raw (normalized text)
Scripts and page chrome removed; this is what change detection compares.
<?xml version="1.0" encoding="UTF-8"?> <rss version="2.0" xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/"> <channel> <title>Ubicloud Status - Incident history</title> <link>https://status.ubicloud.com</link> <description>Ubicloud</description> <pubDate>Mon, 16 Nov 2026 11:59:40 +0000</pubDate> <item> <title>Incident with Pull Requests</title> <description> Type: Incident Duration: 11 days, 2 hours and 45 minutes Affected Components: GitHub → Actions Nov 16, 16:26:26 GMT+0 - Resolved - On November 16, 2026, between 09:30 and 15:55 UTC, some pull request merges on GitHub timed out. On average, 2.9% of merge attempts timed out, with a peak of 7.1% at 15:17 UTC. Some GitHub Actions workflow runs triggered by pull requests were also delayed. Git operations such as push and fetch stayed available, with higher latency. This was a repeat of a shorter event on November 13, between 13:15 and 17:50 UTC, when 1.7% of merges timed out.<br /><br />Both events had the same cause: some of the storage servers that hold Git repository data reached their disk read bandwidth limits. A merge must be written to every copy of a repository, so one overloaded server slowed merges for every repository it hosts. Read load on these servers had been growing as part of a planned move of repositories to new infrastructure. A disk configuration setting on those servers also made them read far more data than necessary, which cut their usable bandwidth roughly in half. We mitigated the incident by correcting that setting and moving some load away from the busiest servers.<br /><br />The corrected setting is now applied permanently across our storage fleet. We are adding storage capacity in the affected region and alerting on per-server disk bandwidth so we catch saturation before it affects customers. Sustained merge timeouts will now page our on-call engineers directly. We are also adding capacity checks before moving repositories or traffic onto new infrastructure. Nov 16, 16:16:07 GMT+0 - Monitoring - We’ve mitigated the issue affecting pull request merges and Actions workflow start times by reducing load on affected systems. Pull Requests and Actions have recovered, and we’re continuing to monitor their performance. Nov 16, 16:07:32 GMT+0 - Monitoring - The degradation affecting Actions and Pull Requests has been mitigated. We are monitoring to ensure stability. Nov 16, 15:55:07 GMT+0 - Investigating - We’re seeing early signs of improvement in pull request merge times & workflow start times, but recovery is not yet confirmed. Users may still have difficulty merging pull requests. We’re reducing load on affected systems and monitoring the impact. We’ll share another update as we learn more. Nov 16, 15:46:44 GMT+0 - Investigating - We’re seeing early signs of improvement in pull request merge times, but recovery is not yet confirmed. The last confirmed timeout rate was 2.5%, compared with a baseline near 0.2%, so some users may still have difficulty merging pull requests. We’ll share another update as we learn more. Nov 16, 15:33:14 GMT+0 - Investigating - Actions is experiencing degraded performance. We are continuing to investigate. </description> <content:encoded> <![CDATA[<p><strong>Type:</strong> Incident</p> <p><strong>Duration:</strong> 11 days, 2 hours and 45 minutes</p> <p><strong>Affected Components:</strong> </p> <p><small>Nov <var data-var='date'> 16</var>, <var data-var='time'>16:26:26</var> GMT+0</small><br><strong>Resolved</strong> - On November 16, 2026, between 09:30 and 15:55 UTC, some pull request merges on GitHub timed out. On average, 2.9% of merge attempts timed out, with a peak of 7.1% at 15:17 UTC. Some GitHub Actions workflow runs triggered by pull requests were also delayed. Git operations such as push and fetch stayed available, with higher latency. This was a repeat of a shorter event on November 13, between 13:15 and 17:50 UTC, when 1.7% of merges timed out.<br /><br />Both events had the same cause: some of the storage servers that hold Git repository data reached their disk read bandwidth limits. A merge must be written to every copy of a repository, so one overloaded server slowed merges for every repository it hosts. Read load on these servers had been growing as part of a planned move of repositories to new infrastructure. A disk configuration setting on those servers also made them read far more data than necessary, which cut their usable bandwidth roughly in half. We mitigated the incident by correcting that setting and moving some load away from the busiest servers.<br /><br />The corrected setting is now applied permanently across our storage fleet. We are adding storage capacity in the affected region and alerting on per-server disk bandwidth so we catch saturation before it affects customers. Sustained merge timeouts will now page our on-call engineers directly. We are also adding capacity checks before moving repositories or traffic onto new infrastructure..</p> <p><small>Nov <var data-var='date'> 16</var>, <var data-var='time'>16:16:07</var> GMT+0</small><br><strong>Monitoring</strong> - We’ve mitigated the issue affecting pull request merges and Actions workflow start times by reducing load on affected systems. Pull Requests and Actions have recovered, and we’re continuing to monitor their performance..</p> <p><small>Nov <var data-var='date'> 16</var>, <var data-var='time'>16:07:32</var> GMT+0</small><br><strong>Monitoring</strong> - The degradation affecting Actions and Pull Requests has been mitigated. We are monitoring to ensure stability..</p> <p><small>Nov <var data-var='date'> 16</var>, <var data-var='time'>15:55:07</var> GMT+0</small><br><strong>Investigating</strong> - We’re seeing early signs of improvement in pull request merge times & workflow start times, but recovery is not yet confirmed. Users may still have difficulty merging pull requests. We’re reducing load on affected systems and monitoring the impact. We’ll share another update as we learn more..</p> <p><small>Nov <var data-var='date'> 16</var>, <var data-var='time'>15:46:44</var> GMT+0</small><br><strong>Investigating</strong> - We’re seeing early signs of improvement in pull request merge times, but recovery is not yet confirmed. The last confirmed timeout rate was 2.5%, compared with a baseline near 0.2%, so some users may still have difficulty merging pull requests. We’ll share another update as we learn more..</p> <p><small>Nov <var data-var='date'> 16</var>, <var data-var='time'>15:33:14</var> GMT+0</small><br><strong>Investigating</strong> - Actions is experiencing degraded performance. We are continuing to investigate..</p> ]]> </content:encoded> <pubDate>Mon, 16 Nov 2026 11:59:40 +0000</pubDate> <link>https://status.ubicloud.com/incident/cmwjfbsvp02eh0kpbkhexitrw</link> <guid>https://status.ubicloud.com/incident/cmwjfbsvp02eh0kpbkhexitrw</guid> </item> <item> <title>Slow or stalled git checkouts on GitHub Actions runners</title> <description> Type: Incident Duration: 8 hours and 27 minutes Affected Components: Ubicloud Runners Nov 16, 15:27:09 GMT+0 - Resolved - Git checkouts on GitHub Actions runners are back to normal. The fix is deployed to all runner hosts. Cause: Runners in our European data centers were sometimes sent to GitHub servers in the US instead of the nearby ones in Europe. Some of those long-distance connections were losing packets, so git fetches became very slow. Runners now look up [github.com](http://github.com) through a DNS provider that picks GitHub servers based on where the request comes from, so each runner connects to the GitHub servers closest to it. Nov 16, 07:00:00 GMT+0 - Identified - Some GitHub Actions jobs on Ubicloud runners hang during actions/checkout or other git fetches from github.com. The download starts but then runs very slowly, often for 15 to 60 minutes, until the job times out. We have identified the cause and are rolling out a fix. </description> <content:encoded> < through a DNS provider that picks GitHub servers based on where the request comes from, so each runner connects to the GitHub servers closest to it..</p> <p><small>Nov <var data-var='date'> 16</var>, <var data-var='time'>07:00:00</var> GMT+0</small><br><strong>Identified</strong> - Some GitHub Actions jobs on Ubicloud runners hang during actions/checkout or other git fetches from github.com. The download starts but then runs very slowly, often for 15 to 60 minutes, until the job times out. We have identified the cause and are rolling out a fix..</p> ]]> </content:encoded> <pubDate>Mon, 16 Nov 2026 07:00:00 +0000</pubDate> <link>https://status.ubicloud.com/incident/cmwj62esz0iui13rt4wr0g7n5</link> <guid>https://status.ubicloud.com/incident/cmwj62esz0iui13rt4wr0g7n5</guid> </item> <item> <title>Incident with Webhooks API</title> <description> Type: Incident Duration: 2 days, 2 hours and 40 minutes Affected Components: GitHub → Webhooks Nov 12, 19:33:54 GMT+0 - Resolved - On November 12, 2026, between 17:59 UTC and 19:05 UTC, the GitHub Webhooks API experienced elevated latency and timeouts. During this time, about 1.23% of Webhooks API requests were slower than normal or timed out, peaking at about 3.63% of requests for a short period. 0.14% of webhook API requests failed during the incident. Webhook deliveries were unaffected. We traced the slow responses to elevated latency in an upstream database service that the Webhooks API depends on. Latency returned to normal at 19:05 UTC, and we continued to monitor until we resolved the incident at 19:33 UTC. We are working with our database provider to understand the cause and are improving our playbooks to detect degradations more quickly. Nov 12, 19:20:14 GMT+0 - Monitoring - The degradation affecting Webhooks has been mitigated. We are monitoring to ensure stability. Nov 12, 19:20:08 GMT+0 - Investigating - Webhooks API traffic has recovered. We will continue to monitor for stability. Nov 12, 18:52:18 GMT+0 - Investigating - Webhooks API is experiencing timeouts and errors impacting about 1% of customer traffic. We are working through possible mitigation strategies. Nov 12, 18:43:13 GMT+0 - Investigating - We are investigating reports of degraded performance for Webhooks </description> <content:encoded> <![CDATA[<p><strong>Type:</strong> Incident</p> <p><strong>Duration:</strong> 2 days, 2 hours and 40 minutes</p> <p><strong>Affected Components:</strong> </p> <p><small>Nov <var data-var='date'> 12</var>, <var data-var='time'>19:33:54</var> GMT+0</small><br><strong>Resolved</strong> - On November 12, 2026, between 17:59 UTC and 19:05 UTC, the GitHub Webhooks API experienced elevated latency and timeouts. During this time, about 1.23% of Webhooks API requests were slower than normal or timed out, peaking at about 3.63% of requests for a short period. 0.14% of webhook API requests failed during the incident. Webhook deliveries were unaffected. We traced the slow responses to elevated latency in an upstream database service that the Webhooks API depends on. Latency returned to normal at 19:05 UTC, and we continued to monitor until we resolved the incident at 19:33 UTC. We are working with our database provider to understand the cause and are improving our playbooks to detect degradations more quickly..</p> <p><small>Nov <var data-var='date'> 12</var>, <var data-var='time'>19:20:14</var> GMT+0</small><br><strong>Monitoring</strong> - The degradation affecting Webhooks has been mitigated. We are monitoring to ensure stability..</p> <p><small>Nov <var data-var='date'> 12</var>, <var data-var='time'>19:20:08</var> GMT+0</small><br><strong>Investigating</strong> - Webhooks API traffic has recovered. We will continue to monitor for stability..</p> <p><small>Nov <var data-var='date'> 12</var>, <var data-var='time'>18:52:18</var> GMT+0</small><br><strong>Investigating</strong> - Webhooks API is experiencing timeouts and errors impacting about 1% of customer traffic. We are working through possible mitigation strategies..</p> <p><small>Nov <var data-var='date'> 12</var>, <var data-var='time'>18:43:13</var> GMT+0</small><br><strong>Investigating</strong> - We are investigating reports of degraded performance for Webhooks.</p> ]]> </content:encoded> <pubDate>Thu, 12 Nov 2026 18:43:13 +0000</pubDate> <link>https://status.ubicloud.com/incident/cmwdwd90001bv0krxbh87rjtv</link> <guid>https://status.ubicloud.com/incident/cmwdwd90001bv0krxbh87rjtv</guid> </item> <item> <title>Console is back up</title> <description> Type: Incident Affected Components: Console Nov 12, 02:17:23 GMT+0 - Resolved - Console is back up. This incident was automatically resolved by Instatus monitoring. Nov 12, 02:16:39 GMT+0 - Investigating - Console is down at the moment. This incident was automatically created by Instatus monitoring. </description> <content:encoded> <![CDATA[<p><strong>Type:</strong> Incident</p> <p><strong>Affected Components:</strong> </p> <p><small>Nov <var data-var='date'> 12</var>, <var data-var='time'>02:17:23</var> GMT+0</small><br><strong>Resolved</strong> - Console is back up. This incident was automatically resolved by Instatus monitoring..</p> <p><small>Nov <var data-var='date'> 12</var>, <var data-var='time'>02:16:39</var> GMT+0</small><br><strong>Investigating</strong> - Console is down at the moment. This incident was automatically created by Instatus monitoring..</p> ]]> </content:encoded> <pubDate>Thu, 12 Nov 2026 02:16:39 +0000</pubDate> <link>https://status.ubicloud.com/incident/cmwcx2vqg00c607qv1lizof1f</link> <guid>https://status.ubicloud.com/incident/cmwcx2vqg00c607qv1lizof1f</guid> </item> <item> <title>API is back up</title> <description> Type: Incident Affected Components: API Nov 12, 02:17:22 GMT+0 - Resolved - API is back up. This incident was automatically resolved by Instatus monitoring. Nov 12, 02:16:28 GMT+0 - Investigating - API is down at the moment. This incident was automatically created by Instatus monitoring. </description> <content:encoded> <![CDATA[<p><strong>Type:</strong> Incident</p> <p><strong>Affected Components:</strong> </p> <p><small>Nov <var data-var='date'> 12</var>, <var data-var='time'>02:17:22</var> GMT+0</small><br><strong>Resolved</strong> - API is back up. This incident was automatically resolved by Instatus monitoring..</p> <p><small>Nov <var data-var='date'> 12</var>, <var data-var='time'>02:16:28</var> GMT+0</small><br><strong>Investigating</strong> - API is down at the moment. This incident was automatically created by Instatus monitoring..</p> ]]> </content:encoded> <pubDate>Thu, 12 Nov 2026 02:16:28 +0000</pubDate> <link>https://status.ubicloud.com/incident/cmwcx2ngm00dt07mxqmoff0hg</link> <guid>https://status.ubicloud.com/incident/cmwcx2ngm00dt07mxqmoff0hg</guid> </item> <item> <title>Impact across multiple GitHub products including Actions, Pages, and Copilot products.</title> <description> Type: Incident Duration: 1 day, 9 hours and 27 minutes Affected Components: GitHub → Webhooks, GitHub → Actions Nov 11, 21:21:39 GMT+0 - Resolved - On November 11, 2026, from 20:05 to 21:02 UTC, customers experienced delayed or failed Actions workflow starts, webhook errors and delivery delays, slower GitHub Apps API requests, and reduced availability of Copilot features, including Code Review. GitHub Pages builds were also delayed.<br /><br />Nearly all GitHub Actions runs triggered between 20:18 and 20:55 UTC were delayed or failed, leaving approximately 3.4 million partially created runs that are being canceled. No GitHub Pages deployments completed during the incident. In the Central US data residency stamp, between 20:06 and 20:50 UTC, about 63% of webhook API requests failed and 27% of deliveries were not recorded; those deliveries reached their destinations but remain unavailable to view or redeliver. Between approximately 20:20 and 21:00 UTC, Copilot code review on github.com completed 87% fewer reviews than normal and some other Copilot features were also unavailable.<br /><br />The disruption was caused by an Azure Cosmos DB regional outage affecting database accounts used by these services. Per Cosmos's RCA, a configuration change to the Central US gateways exposed a latent race condition under a specific high-load traffic pattern, causing gateway crashes and health-probe failures. Cosmos mitigated by rolling back the change. Service health recovered around 21:02 UTC, and the queued Actions backlog finished processing by 21:21 UTC.<br /><br />Our follow-up work focuses on validating regional database failover across affected services and developing tested recovery procedures. We are also improving recovery tools to quickly clear stuck Actions checks and runs after incidents so they do not remain uncancelable or block pull request merges. Nov 11, 21:21:32 GMT+0 - Monitoring - The backlog of Actions jobs has been fully processed. We’re observing full recovery across all impacted services. Nov 11, 21:07:26 GMT+0 - Monitoring - We’re seeing recovery across all services and continuing to monitor. The backlog of Actions jobs are being processed. We’ll update when we see full recovery. Nov 11, 21:07:24 GMT+0 - Monitoring - The degradation affecting Actions, Copilot, Pages and Webhooks has been mitigated. We are monitoring to ensure stability. Nov 11, 20:57:40 GMT+0 - Investigating - Copilot is experiencing degraded availability. We are continuing to investigate. Nov 11, 20:56:49 GMT+0 - Investigating - We’re investigating issues impacting multiple GitHub products including Actions, Pages, and Copilot products. Nov 11, 20:56:26 GMT+0 - Investigating - Pages is experiencing degraded availability. We are continuing to investigate. Nov 11, 20:48:46 GMT+0 - Investigating - Webhooks is experiencing degraded performance. We are continuing to investigate. Nov 11, 20:48:25 GMT+0 - Investigating - Actions is experiencing degraded availability. We are continuing to investigate. Nov 11, 20:48:12 GMT+0 - Investigating - We are investigating reports of degraded availability for Pages </description> <content:encoded> <![CDATA[<p><strong>Type:</strong> Incident</p> <p><strong>Duration:</strong> 1 day, 9 hours and 27 minutes</p> <p><strong>Affected Components:</strong> </p> <p><small>Nov <var data-var='date'> 11</var>, <var data-var='time'>21:21:39</var> GMT+0</small><br><strong>Resolved</strong> - On November 11, 2026, from 20:05 to 21:02 UTC, customers experienced delayed or failed Actions workflow starts, webhook errors and delivery delays, slower GitHub Apps API requests, and reduced availability of Copilot features, including Code Review. GitHub Pages builds were also delayed.<br /><br />Nearly all GitHub Actions runs triggered between 20:18 and 20:55 UTC were delayed or failed, leaving approximately 3.4 million partially created runs that are being canceled. No GitHub Pages deployments completed during the incident. In the Central US data residency stamp, between 20:06 and 20:50 UTC, about 63% of webhook API requests failed and 27% of deliveries were not recorded; those deliveries reached their destinations but remain unavailable to view or redeliver. Between approximately 20:20 and 21:00 UTC, Copilot code review on github.com completed 87% fewer reviews than normal and some other Copilot features were also unavailable.<br /><br />The disruption was caused by an Azure Cosmos DB regional outage affecting database accounts used by these services. Per Cosmos's RCA, a configuration change to the Central US gateways exposed a latent race condition under a specific high-load traffic pattern, causing gateway crashes and health-probe failures. Cosmos mitigated by rolling back the change. Service health recovered around 21:02 UTC, and the queued Actions backlog finished processing by 21:21 UTC.<br /><br />Our follow-up work focuses on validating regional database failover across affected services and developing tested recovery procedures. We are also improving recovery tools to quickly clear stuck Actions checks and runs after incidents so they do not remain uncancelable or block pull request merges..</p> <p><small>Nov <var data-var='date'> 11</var>, <var data-var='time'>21:21:32</var> GMT+0</small><br><strong>Monitoring</strong> - The backlog of Actions jobs has been fully processed. We’re observing full recovery across all impacted services..</p> <p><small>Nov <var data-var='date'> 11</var>, <var data-var='time'>21:07:26</var> GMT+0</small><br><strong>Monitoring</strong> - We’re seeing recovery across all services and continuing to monitor. The backlog of Actions jobs are being processed. We’ll update when we see full recovery..</p> <p><small>Nov <var data-var='date'> 11</var>, <var data-var='time'>21:07:24</var> GMT+0</small><br><strong>Monitoring</strong> - The degradation affecting Actions, Copilot, Pages and Webhooks has been mitigated. We are monitoring to ensure stability..</p> <p><small>Nov <var data-var='date'> 11</var>, <var data-var='time'>20:57:40</var> GMT+0</small><br><strong>Investigating</strong> - Copilot is experiencing degraded availability. We are continuing to investigate..</p> <p><small>Nov <var data-var='date'> 11</var>, <var data-var='time'>20:56:49</var> GMT+0</small><br><strong>Investigating</strong> - We’re investigating issues impacting multiple GitHub products including Actions, Pages, and Copilot products..</p> <p><small>Nov <var data-var='date'> 11</var>, <var data-var='time'>20:56:26</var> GMT+0</small><br><strong>Investigating</strong> - Pages is experiencing degraded availability. We are continuing to investigate..</p> <p><small>Nov <var data-var='date'> 11</var>, <var data-var='time'>20:48:46</var> GMT+0</small><br><strong>Investigating</strong> - Webhooks is experiencing degraded performance. We are continuing to investigate..</p> <p><small>Nov <var data-var='date'> 11</var>, <var data-var='time'>20:48:25</var> GMT+0</small><br><strong>Investigating</strong> - Actions is experiencing degraded availability. We are continuing to investigate..</p> <p><small>Nov <var data-var='date'> 11</var>, <var data-var='time'>20:48:12</var> GMT+0</small><br><strong>Investigating</strong> - We are investigating reports of degraded availability for Pages.</p> ]]> </content:encoded> <pubDate>Wed, 11 Nov 2026 20:48:12 +0000</pubDate> <link>https://status.ubicloud.com/incident/cmwcldnjt0qfx0knxje19mjz0</link> <guid>https://status.ubicloud.com/incident/cmwcldnjt0qfx0knxje19mjz0</guid> </item> <item> <title>Incident with Webhooks</title> <description> Type: Incident Duration: 4 hours and 3 minutes Affected Components: GitHub → Webhooks Nov 11, 20:26:23 GMT+0 - Resolved - This incident has been resolved. Thank you for your patience and understanding as we addressed this issue. A detailed root cause analysis will be shared as soon as it is available. Nov 11, 20:26:20 GMT+0 - Monitoring - Webhooks is operating normally. Nov 11, 20:22:20 GMT+0 - Investigating - We are investigating reports of degraded performance for Webhooks </description> <content:encoded> <![CDATA[<p><strong>Type:</strong> Incident</p> <p><strong>Duration:</strong> 4 hours and 3 minutes</p> <p><strong>Affected Components:</strong> </p> <p><small>Nov <var data-var='date'> 11</var>, <var data-var='time'>20:26:23</var> GMT+0</small><br><strong>Resolved</strong> - This incident has been resolved. Thank you for your patience and understanding as we addressed this issue. A detailed root cause analysis will be shared as soon as it is available..</p> <p><small>Nov <var data-var='date'> 11</var>, <var data-var='time'>20:26:20</var> GMT+0</small><br><strong>Monitoring</strong> - Webhooks is operating normally..</p> <p><small>Nov <var data-var='date'> 11</var>, <var data-var='time'>20:22:20</var> GMT+0</small><br><strong>Investigating</strong> - We are investigating reports of degraded performance for Webhooks.</p> ]]> </content:encoded> <pubDate>Wed, 11 Nov 2026 20:22:20 +0000</pubDate> <link>https://status.ubicloud.com/incident/cmwckhi3v0pey0knxyhtv2xgp</link> <guid>https://status.ubicloud.com/incident/cmwckhi3v0pey0knxyhtv2xgp</guid> </item> <item> <title>Incident with Git Operations and Pull Requests</title> <description> Type: Incident Duration: 4 days, 8 hours and 11 minutes Affected Components: GitHub → Actions Nov 10, 17:55:03 GMT+0 - Resolved - On November 10, 2026, from approximately 15:00 UTC to 17:55 UTC, GitHub experienced degraded performance affecting repository pages, pull requests, and GitHub Actions. Customers may have seen slow or failed page loads and delays processing pull request updates and merges. Some pull requests continued to display an earlier source-branch commit after the broader service degradation ended. GitHub Actions workflows were delayed for approximately 1.8% of events, peaking at about 7% between 16:30 and 16:50 UTC. Delays were typically 1.5 to 3.5 minutes, up to 10 minutes. Approximately 0.09% of pull request events did not start their workflows and needed to be re-triggered. <br /><br />The degradation resulted from overload in the shared infrastructure that supports Git operations. Recent changes increased background processing and the work required to compute pull request merges, adding pressure to the same systems used by customer-facing requests. We mitigated the incident by disabling a recent merge-processing optimization, pausing background analysis, and temporarily limiting lower-priority traffic. The temporary limits also restricted access to commit pages for users who were not signed in. Background processing continued to catch up after service health recovered. <br /><br />We have reduced repeated repository-analysis work and are adding workload-level concurrency controls to better protect customer-facing requests. We are also improving our procedures for safely reducing background load and making pull request synchronization more resilient to temporary failures. Nov 10, 17:46:33 GMT+0 - Investigating - We continue to investigate elevated response times for Git operations and pull requests. Actions workflows are recovering. Pull request processing has shown some improvement, but users continue to experience delays and failures. We’re monitoring the impact and working to restore normal service. Nov 10, 17:28:17 GMT+0 - Investigating - Actions is operating normally. Nov 10, 17:12:54 GMT+0 - Investigating - Pull Requests is experiencing degraded performance. We are continuing to investigate. Nov 10, 17:07:48 GMT+0 - Investigating - We continue to investigate elevated response times for Git operations and pull requests. Some pull-request processing and Actions workflows are improving, but delays and failures persist. We're monitoring the impact and working to restore normal service. Nov 10, 16:39:58 GMT+0 - Investigating - Actions is experiencing degraded availability. We are continuing to investigate. Nov 10, 16:31:01 GMT+0 - Investigating - We're investigating elevated response times for Git operations and pull requests. GitHub Actions workflows that are triggered from Pull Requests are delayed. We're working to mitigate the impact and will share updates as we learn more. Nov 10, 16:26:37 GMT+0 - Investigating - Actions and Pull Requests are experiencing degraded availability. We are continuing to investigate. Nov 10, 16:10:52 GMT+0 - Investigating - We are investigating reports of degraded performance for Actions and Pull Requests </description> <content:encoded> <![CDATA[<p><strong>Type:</strong> Incident</p> <p><strong>Duration:</strong> 4 days, 8 hours and 11 minutes</p> <p><strong>Affected Components:</strong> </p> <p><small>Nov <var data-var='date'> 10</var>, <var data-var='time'>17:55:03</var> GMT+0</small><br><strong>Resolved</strong> - On November 10, 2026, from approximately 15:00 UTC to 17:55 UTC, GitHub experienced degraded performance affecting repository pages, pull requests, and GitHub Actions. Customers may have seen slow or failed page loads and delays processing pull request updates and merges. Some pull requests continued to display an earlier source-branch commit after the broader service degradation ended. GitHub Actions workflows were delayed for approximately 1.8% of events, peaking at about 7% between 16:30 and 16:50 UTC. Delays were typically 1.5 to 3.5 minutes, up to 10 minutes. Approximately 0.09% of pull request events did not start their workflows and needed to be re-triggered. <br /><br />The degradation resulted from overload in the shared infrastructure that supports Git operations. Recent changes increased background processing and the work required to compute pull request merges, adding pressure to the same systems used by customer-facing requests. We mitigated the incident by disabling a recent merge-processing optimization, pausing background analysis, and temporarily limiting lower-priority traffic. The temporary limits also restricted access to commit pages for users who were not signed in. Background processing continued to catch up after service health recovered. <br /><br />We have reduced repeated repository-analysis work and are adding workload-level concurrency controls to better protect customer-facing requests. We are also improving our procedures for safely reducing background load and making pull request synchronization more resilient to temporary failures..</p> <p><small>Nov <var data-var='date'> 10</var>, <var data-var='time'>17:46:33</var> GMT+0</small><br><strong>Investigating</strong> - We continue to investigate elevated response times for Git operations and pull requests. Actions workflows are recovering. Pull request processing has shown some improvement, but users continue to experience delays and failures. We’re monitoring the impact and working to restore normal service..</p> <p><small>Nov <var data-var='date'> 10</var>, <var data-var='time'>17:28:17</var> GMT+0</small><br><strong>Investigating</strong> - Actions is operating normally..</p> <p><small>Nov <var data-var='date'> 10</var>, <var data-var='time'>17:12:54</var> GMT+0</small><br><strong>Investigating</strong> - Pull Requests is experiencing degraded performance. We are continuing to investigate..</p> <p><small>Nov <var data-var='date'> 10</var>, <var data-var='time'>17:07:48</var> GMT+0</small><br><strong>Investigating</strong> - We continue to investigate elevated response times for Git operations and pull requests. Some pull-request processing and Actions workflows are improving, but delays and failures persist. We're monitoring the impact and working to restore normal service..</p> <p><small>Nov <var data-var='date'> 10</var>, <var data-var='time'>16:39:58</var> GMT+0</small><br><strong>Investigating</strong> - Actions is experiencing degraded availability. We are continuing to investigate..</p> <p><small>Nov <var data-var='date'> 10</var>, <var data-var='time'>16:31:01</var> GMT+0</small><br><strong>Investigating</strong> - We're investigating elevated response times for Git operations and pull requests. GitHub Actions workflows that are triggered from Pull Requests are delayed. We're working to mitigate the impact and will share updates as we learn more..</p> <p><small>Nov <var data-var='date'> 10</var>, <var data-var='time'>16:26:37</var> GMT+0</small><br><strong>Investigating</strong> - Actions and Pull Requests are experiencing degraded availability. We are continuing to investigate..</p> <p><small>Nov <var data-var='date'> 10</var>, <var data-var='time'>16:10:52</var> GMT+0</small><br><strong>Investigating</strong> - We are investigating reports of degraded performance for Actions and Pull Requests.</p> ]]> </content:encoded> <pubDate>Tue, 10 Nov 2026 16:10:52 +0000</pubDate> <link>https://status.ubicloud.com/incident/cmwaw44rs27qd0rrhlxsiazma</link> <guid>https://status.ubicloud.com/incident/cmwaw44rs27qd0rrhlxsiazma</guid> </item> <item> <title>Incident with Actions and Checks API</title> <description> Type: Incident Duration: 1 day, 22 hours and 55 minutes Affected Components: GitHub → Actions, GitHub → Webhooks Nov 9, 18:59:32 GMT+0 - Resolved - On November 9, 2026, between 18:07 and 18:59 UTC, GitHub Actions experienced degraded performance. During this time, some customers saw workflow runs fail to start, update or report status correctly. An average of 28% of workflow runs were affected, peaking at 65% during the worst period of the incident. Once Actions services recovered, some jobs were left in a state that limited available concurrency until the runs were released. <br /><br />The incident occurred during a planned migration of a core Actions database from GitHub’s datacenter to a new cloud region. After the migration completed, we shifted Actions service traffic to the new region. The volume of requests exceeded the capacity and rate limits configured there, so a portion of requests were rejected and workflow updates failed. Dependent services, including Webhooks, Pages and Copilot code review, saw secondary impact. <br /><br />We mitigated impact by shifting some of the traffic back to the original region, allowing service traffic to remain within the configured limits. We then increased capacity and rate limits in the new region before gradually shifting traffic back. While this was a one-time migration event for this cluster that is now complete, we are improving how we validate capacity and limits before shifting traffic during remaining migrations, including shifting traffic in smaller steps. To address the stuck runs left after the incident, we are improving our ability to detect and mitigate these runs for future cases as well as readying an API to enable customers to address this themselves. Nov 9, 18:43:24 GMT+0 - Monitoring - The degradation affecting Actions, Pages and Webhooks has been mitigated. We are monitoring to ensure stability. Nov 9, 18:30:17 GMT+0 - Investigating - Pages is experiencing degraded performance. We are continuing to investigate. Nov 9, 18:24:13 GMT+0 - Investigating - We're investigating elevated errors affecting GitHub Actions and the Checks API. We'll share another update as we learn more. Nov 9, 18:23:13 GMT+0 - Investigating - Webhooks is experiencing degraded performance. We are continuing to investigate. Nov 9, 18:14:58 GMT+0 - Investigating - Actions is experiencing degraded availability. We are continuing to investigate. Nov 9, 18:12:37 GMT+0 - Investigating - We are investigating reports of degraded availability for Actions </description> <content:encoded> <![CDATA[<p><strong>Type:</strong> Incident</p> <p><strong>Duration:</strong> 1 day, 22 hours and 55 minutes</p> <p><strong>Affected Components:</strong> </p> <p><small>Nov <var data-var='date'> 9</var>, <var data-var='time'>18:59:32</var> GMT+0</small><br><strong>Resolved</strong> - On November 9, 2026, between 18:07 and 18:59 UTC, GitHub Actions experienced degraded performance. During this time, some customers saw workflow runs fail to start, update or report status correctly. An average of 28% of workflow runs were affected, peaking at 65% during the worst period of the incident. Once Actions services recovered, some jobs were left in a state that limited available concurrency until the runs were released. <br /><br />The incident occurred during a planned migration of a core Actions database from GitHub’s datacenter to a new cloud region. After the migration completed, we shifted Actions service traffic to the new region. The volume of requests exceeded the capacity and rate limits configured there, so a portion of requests were rejected and workflow updates failed. Dependent services, including Webhooks, Pages and Copilot code review, saw secondary impact. <br /><br />We mitigated impact by shifting some of the traffic back to the original region, allowing service traffic to remain within the configured limits. We then increased capacity and rate limits in the new region before gradually shifting traffic back. While this was a one-time migration event for this cluster that is now complete, we are improving how we validate capacity and limits before shifting traffic during remaining migrations, including shifting traffic in smaller steps. To address the stuck runs left after the incident, we are improving our ability to detect and mitigate these runs for future cases as well as readying an API to enable customers to address this themselves..</p> <p><small>Nov <var data-var='date'> 9</var>, <var data-var='time'>18:43:24</var> GMT+0</small><br><strong>Monitoring</strong> - The degradation affecting Actions, Pages and Webhooks has been mitigated. We are monitoring to ensure stability..</p> <p><small>Nov <var data-var='date'> 9</var>, <var data-var='time'>18:30:17</var> GMT+0</small><br><strong>Investigating</strong> - Pages is experiencing degraded performance. We are continuing to investigate..</p> <p><small>Nov <var data-var='date'> 9</var>, <var data-var='time'>18:24:13</var> GMT+0</small><br><strong>Investigating</strong> - We're investigating elevated errors affecting GitHub Actions and the Checks API. We'll share another update as we learn more..</p> <p><small>Nov <var data-var='date'> 9</var>, <var data-var='time'>18:23:13</var> GMT+0</small><br><strong>Investigating</strong> - Webhooks is experiencing degraded performance. We are continuing to investigate..</p> <p><small>Nov <var data-var='date'> 9</var>, <var data-var='time'>18:14:58</var> GMT+0</small><br><strong>Investigating</strong> - Actions is experiencing degraded availability. We are continuing to investigate..</p> <p><small>Nov <var data-var='date'> 9</var>, <var data-var='time'>18:12:37</var> GMT+0</small><br><strong>Investigating</strong> - We are investigating reports of degraded availability for Actions.</p> ]]> </content:encoded> <pubDate>Mon, 9 Nov 2026 18:12:37 +0000</pubDate> <link>https://status.ubicloud.com/incident/cmw9kylzp022o0rrhp7nvgbcm</link> <guid>https://status.ubicloud.com/incident/cmw9kylzp022o0rrhp7nvgbcm</guid> </item> <item> <title>Delayed processing of requests</title> <description> Type: Incident Duration: 50 minutes Affected Components: Ubicloud Runners, Ubicloud Cache Nov 9, 15:35:00 GMT+0 - Resolved - This incident has been resolved. Nov 9, 15:20:00 GMT+0 - Monitoring - The database load is back to normal and the waiting requests are being processed. Nov 9, 14:45:00 GMT+0 - Identified - Delayed processing of requests due to unexpected load on our production database. </description> <content:encoded> <![CDATA[<p><strong>Type:</strong> Incident</p> <p><strong>Duration:</strong> 50 minutes</p> <p><strong>Affected Components:</strong> </p> <p><small>Nov <var data-var='date'> 9</var>, <var data-var='time'>15:35:00</var> GMT+0</small><br><strong>Resolved</strong> - This incident has been resolved..</p> <p><small>Nov <var data-var='date'> 9</var>, <var data-var='time'>15:20:00</var> GMT+0</small><br><strong>Monitoring</strong> - The database load is back to normal and the waiting requests are being processed..</p> <p><small>Nov <var data-var='date'> 9</var>, <var data-var='time'>14:45:00</var> GMT+0</small><br><strong>Identified</strong> - Delayed processing of requests due to unexpected load on our production database..</p> ]]> </content:encoded> <pubDate>Mon, 9 Nov 2026 14:45:00 +0000</pubDate> <link>https://status.ubicloud.com/incident/cmwdcc7n10ck71mjyh1bjdyz8</link> <guid>https://status.ubicloud.com/incident/cmwdcc7n10ck71mjyh1bjdyz8</guid> </item> <item> <title>Incident with Webhooks and Actions</title> <description> Type: Incident Duration: 4 days and 14 hours Affected Components: GitHub → Webhooks, GitHub → Actions Nov 9, 11:50:35 GMT+0 - Resolved - On November 9, 2026, from 09:23 to 11:40 UTC, one of our job queue clusters was degraded, delaying webhook deliveries, Actions workflow runs, and Dependabot and dependency graph updates. Until mitigation at 10:37 UTC, an average of 26% of webhook deliveries were delayed, peaking at 30%, and the backlog cleared by 11:40 UTC. Some deliveries were delayed by up to 3.5 hours, and delays for a small share began as early as 01:00 UTC. Actions skips events older than 30 minutes, so some workflow runs did not start. <br /><br />The incident was caused by a degraded host that had slowed one of the cluster's Kafka brokers since November 7. The broker stopped reporting the metrics our alerts rely on, so we did not identify it until it stalled the cluster on November 9. <br /><br />We mitigated the incident by removing the broker from service. We have since added an alert that pages us within 30 minutes when a host slows a broker this way, so we can remove the broker before it affects customers. Nov 9, 11:45:07 GMT+0 - Monitoring - The degradation affecting Webhooks has been mitigated. We are monitoring to ensure stability. Nov 9, 11:42:04 GMT+0 - Investigating - Webhooks is experiencing degraded performance. We are continuing to investigate. Nov 9, 11:17:28 GMT+0 - Investigating - Webhooks is experiencing degraded availability. We are continuing to investigate. Nov 9, 11:06:35 GMT+0 - Investigating - The degradation affecting Actions has been mitigated. We are monitoring to ensure stability. Nov 9, 10:47:25 GMT+0 - Investigating - We've applied a mitigation and are monitoring for recovery. Nov 9, 10:28:19 GMT+0 - Investigating - We're investigating delays in Actions runs start times and webhook deliveries. Nov 9, 10:00:35 GMT+0 - Investigating - We are investigating reports of degraded performance for Actions and Webhooks </description> <content:encoded> <![CDATA[<p><strong>Type:</strong> Incident</p> <p><strong>Duration:</strong> 4 days and 14 hours</p> <p><strong>Affected Components:</strong> </p> <p><small>Nov <var data-var='date'> 9</var>, <var data-var='time'>11:50:35</var> GMT+0</small><br><strong>Resolved</strong> - On November 9, 2026, from 09:23 to 11:40 UTC, one of our job queue clusters was degraded, delaying webhook deliveries, Actions workflow runs, and Dependabot and dependency graph updates. Until mitigation at 10:37 UTC, an average of 26% of webhook deliveries were delayed, peaking at 30%, and the backlog cleared by 11:40 UTC. Some deliveries were delayed by up to 3.5 hours, and delays for a small share began as early as 01:00 UTC. Actions skips events older than 30 minutes, so some workflow runs did not start. <br /><br />The incident was caused by a degraded host that had slowed one of the cluster's Kafka brokers since November 7. The broker stopped reporting the metrics our alerts rely on, so we did not identify it until it stalled the cluster on November 9. <br /><br />We mitigated the incident by removing the broker from service. We have since added an alert that pages us within 30 minutes when a host slows a broker this way, so we can remove the broker before it affects customers..</p> <p><small>Nov <var data-var='date'> 9</var>, <var data-var='time'>11:45:07</var> GMT+0</small><br><strong>Monitoring</strong> - The degradation affecting Webhooks has been mitigated. We are monitoring to ensure stability..</p> <p><small>Nov <var data-var='date'> 9</var>, <var data-var='time'>11:42:04</var> GMT+0</small><br><strong>Investigating</strong> - Webhooks is experiencing degraded performance. We are continuing to investigate..</p> <p><small>Nov <var data-var='date'> 9</var>, <var data-var='time'>11:17:28</var> GMT+0</small><br><strong>Investigating</strong> - Webhooks is experiencing degraded availability. We are continuing to investigate..</p> <p><small>Nov <var data-var='date'> 9</var>, <var data-var='time'>11:06:35</var> GMT+0</small><br><strong>Investigating</strong> - The degradation affecting Actions has been mitigated. We are monitoring to ensure stability..</p> <p><small>Nov <var data-var='date'> 9</var>, <var data-var='time'>10:47:25</var> GMT+0</small><br><strong>Investigating</strong> - We've applied a mitigation and are monitoring for recovery..</p> <p><small>Nov <var data-var='date'> 9</var>, <var data-var='time'>10:28:19</var> GMT+0</small><br><strong>Investigating</strong> - We're investigating delays in Actions runs start times and webhook deliveries..</p> <p><small>Nov <var data-var='date'> 9</var>, <var data-var='time'>10:00:35</var> GMT+0</small><br><strong>Investigating</strong> - We are investigating reports of degraded performance for Actions and Webhooks.</p> ]]> </content:encoded> <pubDate>Mon, 9 Nov 2026 10:00:35 +0000</pubDate> <link>https://status.ubicloud.com/incident/cmw93ghbt1ppa0kld2xgfkdhl</link> <guid>https://status.ubicloud.com/incident/cmw93ghbt1ppa0kld2xgfkdhl</guid> </item> <item> <title>Increase in timeouts for some unauthenticated users</title> <description> Type: Incident Duration: 4 days, 20 hours and 30 minutes Affected Components: GitHub → API Requests Nov 8, 12:32:35 GMT+0 - Resolved - On November 8, 2026, between 08:43 and 12:29 UTC, unauthenticated requests to github.com and the GitHub REST API experienced elevated error rates. During this period, 11.3% of unauthenticated requests and 46.1% of release asset download requests failed. Authenticated requests were largely unaffected.<br /><br />Initial errors occurred in a backend service handling unauthenticated web traffic. A subsequent traffic spike caused high CPU utilization on servers in our caching layer. Some servers remained unhealthy after the spike ended, prolonging the disruption. We restored service by restarting caching software and rebooting affected servers.<br /><br />We are improving monitoring to detect these failures sooner, investigating why affected servers did not recover automatically, and reviewing caching behavior to better protect backend services during failures. Nov 8, 12:29:30 GMT+0 - Monitoring - The degradation has been mitigated. We are monitoring to ensure stability. Nov 8, 11:56:20 GMT+0 - Investigating - The degradation affecting API Requests has been mitigated. We are monitoring to ensure stability. Nov 8, 11:50:49 GMT+0 - Investigating - We've identified a problematic component and are working to redeploy it. Nov 8, 11:20:57 GMT+0 - Investigating - API Requests is experiencing degraded performance. We are continuing to investigate. Nov 8, 11:16:19 GMT+0 - Investigating - We are continuing to investigate degraded performance primarily for unauthenticated users. </description> <content:encoded> <![CDATA[<p><strong>Type:</strong> Incident</p> <p><strong>Duration:</strong> 4 days, 20 hours and 30 minutes</p> <p><strong>Affected Components:</strong> </p> <p><small>Nov <var data-var='date'> 8</var>, <var data-var='time'>12:32:35</var> GMT+0</small><br><strong>Resolved</strong> - On November 8, 2026, between 08:43 and 12:29 UTC, unauthenticated requests to github.com and the GitHub REST API experienced elevated error rates. During this period, 11.3% of unauthenticated requests and 46.1% of release asset download requests failed. Authenticated requests were largely unaffected.<br /><br />Initial errors occurred in a backend service handling unauthenticated web traffic. A subsequent traffic spike caused high CPU utilization on servers in our caching layer. Some servers remained unhealthy after the spike ended, prolonging the disruption. We restored service by restarting caching software and rebooting affected servers.<br /><br />We are improving monitoring to detect these failures sooner, investigating why affected servers did not recover automatically, and reviewing caching behavior to better protect backend services during failures..</p> <p><small>Nov <var data-var='date'> 8</var>, <var data-var='time'>12:29:30</var> GMT+0</small><br><strong>Monitoring</strong> - The degradation has been mitigated. We are monitoring to ensure stability..</p> <p><small>Nov <var data-var='date'> 8</var>, <var data-var='time'>11:56:20</var> GMT+0</small><br><strong>Investigating</strong> - The degradation affecting API Requests has been mitigated. We are monitoring to ensure stability..</p> <p><small>Nov <var data-var='date'> 8</var>, <var data-var='time'>11:50:49</var> GMT+0</small><br><strong>Investigating</strong> - We've identified a problematic component and are working to redeploy it..</p> <p><small>Nov <var data-var='date'> 8</var>, <var data-var='time'>11:20:57</var> GMT+0</small><br><strong>Investigating</strong> - API Requests is experiencing degraded performance. We are continuing to investigate..</p> <p><small>Nov <var data-var='date'> 8</var>, <var data-var='time'>11:16:19</var> GMT+0</small><br><strong>Investigating</strong> - We are continuing to investigate degraded performance primarily for unauthenticated users..</p> ]]> </content:encoded> <pubDate>Sun, 8 Nov 2026 10:36:05 +0000</pubDate> <link>https://status.ubicloud.com/incident/cmw7qvhkn6nac0kp3wqv0b7do</link> <guid>https://status.ubicloud.com/incident/cmw7qvhkn6nac0kp3wqv0b7do</guid> </item> <item> <title>Degraded performance for multiple GitHub services</title> <description> Type: Incident Duration: 1 day, 20 hours and 43 minutes Affected Components: GitHub → Actions Nov 6, 00:20:49 GMT+0 - Resolved - On November 5, 2026, between 23:21 and approximately 23:26 UTC, several GitHub services experienced elevated errors, including Git operations, Issues, Actions, authentication, and API requests. Customers may have encountered failed requests and Git pushes, and some Actions workflow runs were delayed. <br /><br />This incident was caused by a bug in our database failover automation during planned maintenance. The automation stopped before completing the switch to a replacement database primary, leaving the existing primary unable to accept writes. We mitigated the disruption by restoring writes on the original primary. We subsequently completed the maintenance and monitored services to confirm recovery. <br /><br />We are fixing the automation bug and strengthening our pre-maintenance checks and recovery procedures to prevent similar disruptions and restore service more quickly if a failure occurs. Nov 6, 00:19:59 GMT+0 - Monitoring - This incident has been resolved. All affected services, including Git operations, Issues, Actions, and API access, have recovered and remained stable. Thank you for your patience. We'll share more details in our availability report. Nov 6, 00:19:21 GMT+0 - Investigating - Git Operations is operating normally. Nov 6, 00:06:28 GMT+0 - Monitoring - We've applied a mitigation and are seeing recovery across affected services, including Git operations, Issues, Actions, and API access. We're monitoring to confirm stability and will post another update by 00:35 UTC. Nov 6, 00:06:11 GMT+0 - Monitoring - The degradation affecting Actions, Issues and Pages has been mitigated. We are monitoring to ensure stability. Nov 5, 23:58:28 GMT+0 - Investigating - Issues is experiencing degraded performance. We are continuing to investigate. Nov 5, 23:36:51 GMT+0 - Investigating - We're investigating degraded performance and errors affecting several GitHub services, including Git operations, Issues, Actions, and API access. Some users may see failed requests or slow responses. We'll post another update by 00:05 UTC. Nov 5, 23:36:06 GMT+0 - Investigating - We are investigating reports of degraded performance for Actions and Pages </description> <content:encoded> <![CDATA[<p><strong>Type:</strong> Incident</p> <p><strong>Duration:</strong> 1 day, 20 hours and 43 minutes</p> <p><strong>Affected Components:</strong> </p> <p><small>Nov <var data-var='date'> 6</var>, <var data-var='time'>00:20:49</var> GMT+0</small><br><strong>Resolved</strong> - On November 5, 2026, between 23:21 and approximately 23:26 UTC, several GitHub services experienced elevated errors, including Git operations, Issues, Actions, authentication, and API requests. Customers may have encountered failed requests and Git pushes, and some Actions workflow runs were delayed. <br /><br />This incident was caused by a bug in our database failover automation during planned maintenance. The automation stopped before completing the switch to a replacement database primary, leaving the existing primary unable to accept writes. We mitigated the disruption by restoring writes on the original primary. We subsequently completed the maintenance and monitored services to confirm recovery. <br /><br />We are fixing the automation bug and strengthening our pre-maintenance checks and recovery procedures to prevent similar disruptions and restore service more quickly if a failure occurs..</p> <p><small>Nov <var data-var='date'> 6</var>, <var data-var='time'>00:19:59</var> GMT+0</small><br><strong>Monitoring</strong> - This incident has been resolved. All affected services, including Git operations, Issues, Actions, and API access, have recovered and remained stable. Thank you for your patience. We'll share more details in our availability report..</p> <p><small>Nov <var data-var='date'> 6</var>, <var data-var='time'>00:19:21</var> GMT+0</small><br><strong>Investigating</strong> - Git Operations is operating normally..</p> <p><small>Nov <var data-var='date'> 6</var>, <var data-var='time'>00:06:28</var> GMT+0</small><br><strong>Monitoring</strong> - We've applied a mitigation and are seeing recovery across affected services, including Git operations, Issues, Actions, and API access. We're monitoring to confirm stability and will post another update by 00:35 UTC..</p> <p><small>Nov <var data-var='date'> 6</var>, <var data-var='time'>00:06:11</var> GMT+0</small><br><strong>Monitoring</strong> - The degradation affecting Actions, Issues and Pages has been mitigated. We are monitoring to ensure stability..</p> <p><small>Nov <var data-var='date'> 5</var>, <var data-var='time'>23:58:28</var> GMT+0</small><br><strong>Investigating</strong> - Issues is experiencing degraded performance. We are continuing to investigate..</p> <p><small>Nov <var data-var='date'> 5</var>, <var data-var='time'>23:36:51</var> GMT+0</small><br><strong>Investigating</strong> - We're investigating degraded performance and errors affecting several GitHub services, including Git operations, Issues, Actions, and API access. Some users may see failed requests or slow responses. We'll post another update by 00:05 UTC..</p> <p><small>Nov <var data-var='date'> 5</var>, <var data-var='time'>23:36:06</var> GMT+0</small><br><strong>Investigating</strong> - We are investigating reports of degraded performance for Actions and Pages.</p> ]]> </content:encoded> <pubDate>Thu, 5 Nov 2026 23:36:06 +0000</pubDate> <link>https://status.ubicloud.com/incident/cmw46t5lp0io70rp3zv38xoy3</link> <guid>https://status.ubicloud.com/incident/cmw46t5lp0io70rp3zv38xoy3</guid> </item> <item> <title>Incident with Actions</title> <description> Type: Incident Duration: 2 days, 4 hours and 24 minutes Affected Components: GitHub → Actions Nov 5, 17:42:35 GMT+0 - Resolved - This incident has been resolved. Thank you for your patience and understanding as we addressed this issue. A detailed root cause analysis will be shared as soon as it is available. Nov 5, 17:04:46 GMT+0 - Monitoring - The degradation affecting Actions has been mitigated. We are monitoring to ensure stability. Nov 5, 16:52:52 GMT+0 - Investigating - Some GitHub Actions workflow runs may experience delays or errors when starting. We’re investigating and preparing a mitigation. Nov 5, 16:50:11 GMT+0 - Investigating - We are investigating reports of degraded performance for Actions </description> <content:encoded> <![CDATA[<p><strong>Type:</strong> Incident</p> <p><strong>Duration:</strong> 2 days, 4 hours and 24 minutes</p> <p><strong>Affected Components:</strong> </p> <p><small>Nov <var data-var='date'> 5</var>, <var data-var='time'>17:42:35</var> GMT+0</small><br><strong>Resolved</strong> - This incident has been resolved. Thank you for your patience and understanding as we addressed this issue. A detailed root cause analysis will be shared as soon as it is available..</p> <p><small>Nov <var data-var='date'> 5</var>, <var data-var='time'>17:04:46</var> GMT+0</small><br><strong>Monitoring</strong> - The degradation affecting Actions has been mitigated. We are monitoring to ensure stability..</p> <p><small>Nov <var data-var='date'> 5</var>, <var data-var='time'>16:52:52</var> GMT+0</small><br><strong>Investigating</strong> - Some GitHub Actions workflow runs may experience delays or errors when starting. We’re investigating and preparing a mitigation..</p> <p><small>Nov <var data-var='date'> 5</var>, <var data-var='time'>16:50:11</var> GMT+0</small><br><strong>Investigating</strong> - We are investigating reports of degraded performance for Actions.</p> ]]> </content:encoded> <pubDate>Thu, 5 Nov 2026 16:50:11 +0000</pubDate> <link>https://status.ubicloud.com/incident/cmw3scbcx00lm0ro0ky0ge8p9</link> <guid>https://status.ubicloud.com/incident/cmw3scbcx00lm0ro0ky0ge8p9</guid> </item> <item> <title>Incident with Actions</title> <description> Type: Incident Duration: 1 day, 11 hours and 34 minutes Affected Components: GitHub → Actions Nov 4, 18:16:08 GMT+0 - Resolved - Please refer to the combined summary in this related incident: https://www.githubstatus.com/incidents/xr3myzj7ms51 Nov 4, 18:07:51 GMT+0 - Monitoring - The degradation affecting Actions has been mitigated. We are monitoring to ensure stability. Nov 4, 17:53:11 GMT+0 - Investigating - We have identified an issue with events that might trigger Actions. We have applied mitigations and are monitoring for recovery at this point. Customers may observe delayed status updates for their jobs. Nov 4, 17:40:33 GMT+0 - Investigating - We are investigating reports of degraded availability for Actions </description> <content:encoded> <![CDATA[<p><strong>Type:</strong> Incident</p> <p><strong>Duration:</strong> 1 day, 11 hours and 34 minutes</p> <p><strong>Affected Components:</strong> </p> <p><small>Nov <var data-var='date'> 4</var>, <var data-var='time'>18:16:08</var> GMT+0</small><br><strong>Resolved</strong> - Please refer to the combined summary in this related incident: https://www.githubstatus.com/incidents/xr3myzj7ms51.</p> <p><small>Nov <var data-var='date'> 4</var>, <var data-var='time'>18:07:51</var> GMT+0</small><br><strong>Monitoring</strong> - The degradation affecting Actions has been mitigated. We are monitoring to ensure stability..</p> <p><small>Nov <var data-var='date'> 4</var>, <var data-var='time'>17:53:11</var> GMT+0</small><br><strong>Investigating</strong> - We have identified an issue with events that might trigger Actions. We have applied mitigations and are monitoring for recovery at this point. Customers may observe delayed status updates for their jobs..</p> <p><small>Nov <var data-var='date'> 4</var>, <var data-var='time'>17:40:33</var> GMT+0</small><br><strong>Investigating</strong> - We are investigating reports of degraded availability for Actions.</p> ]]> </content:encoded> <pubDate>Wed, 4 Nov 2026 17:40:33 +0000</pubDate> <link>https://status.ubicloud.com/incident/cmw2eormz05fw0rqevfx6wko3</link> <guid>https://status.ubicloud.com/incident/cmw2eormz05fw0rqevfx6wko3</guid> </item> <item> <title>Incident with Actions and Pages</title> <description> Type: Incident Duration: 2 days, 7 hours and 39 minutes Affected Components: GitHub → Actions, GitHub → Webhooks Nov 4, 17:01:11 GMT+0 - Resolved - On November 4, 2026, between 15:49 UTC and 16:48 UTC, and again between 17:20 UTC and 17:55 UTC, GitHub Actions, Pages, Webhooks, Code Scanning, and Copilot coding agent experienced delays. During this time, Actions workflow runs and job status updates were delayed by up to 25 minutes. Pages deployments dropped to about 4% of normal volume, webhook deliveries dropped to about 58% of normal volume, and Code Scanning analyses and Copilot coding agent sessions were delayed. Delayed events were processed after mitigation. <br /><br />This incident was caused by a configuration change to an internal event-routing service that unexpectedly paused delivery of several event streams. The change depended on feature flags that were not set up as intended, which the service did not handle gracefully. We mitigated the first period by rolling back the configuration change at 16:14 UTC. The code change itself was not reverted by the rollback, so a subsequent deployment reintroduced it. We mitigated the second period at 17:40 UTC by rolling back again, then processed queued events and cleared backlogs. <br /><br />We are taking several steps to prevent a repeat. The routing service will keep its existing behavior when a feature toggle is missing, so a misconfigured flag can no longer pause delivery. We are adding a pre-deploy check that each routing flag exists and is registered under the correct owning service, and we are making flag ownership less error-prone to set. We are also improving alerting so we detect paused event delivery sooner. Nov 4, 16:49:27 GMT+0 - Monitoring - The degradation has been mitigated. We are monitoring to ensure stability. Nov 4, 16:48:03 GMT+0 - Monitoring - The degradation affecting Actions and Pages has been mitigated. We are monitoring to ensure stability. Nov 4, 16:45:48 GMT+0 - Investigating - The degradation affecting Webhooks has been mitigated. We are monitoring to ensure stability. Nov 4, 16:34:45 GMT+0 - Investigating - We have identified an issue with events that might trigger Actions. We have applied mitigations and are monitoring for recovery at this point. Customers may observe delayed status updates for their jobs. Nov 4, 16:17:29 GMT+0 - Investigating - Actions is experiencing degraded availability. We are continuing to investigate. Nov 4, 16:12:55 GMT+0 - Investigating - Webhooks is experiencing degraded performance. We are continuing to investigate. Nov 4, 16:11:38 GMT+0 - Investigating - We are investigating reports of issues with Actions and Pages. We will continue to keep users updated on progress towards mitigation. Nov 4, 16:05:31 GMT+0 - Investigating - We are investigating reports of degraded performance for Actions and Pages </description> <content:encoded> <![CDATA[<p><strong>Type:</strong> Incident</p> <p><strong>Duration:</strong> 2 days, 7 hours and 39 minutes</p> <p><strong>Affected Components:</strong> </p> <p><small>Nov <var data-var='date'> 4</var>, <var data-var='time'>17:01:11</var> GMT+0</small><br><strong>Resolved</strong> - On November 4, 2026, between 15:49 UTC and 16:48 UTC, and again between 17:20 UTC and 17:55 UTC, GitHub Actions, Pages, Webhooks, Code Scanning, and Copilot coding agent experienced delays. During this time, Actions workflow runs and job status updates were delayed by up to 25 minutes. Pages deployments dropped to about 4% of normal volume, webhook deliveries dropped to about 58% of normal volume, and Code Scanning analyses and Copilot coding agent sessions were delayed. Delayed events were processed after mitigation. <br /><br />This incident was caused by a configuration change to an internal event-routing service that unexpectedly paused delivery of several event streams. The change depended on feature flags that were not set up as intended, which the service did not handle gracefully. We mitigated the first period by rolling back the configuration change at 16:14 UTC. The code change itself was not reverted by the rollback, so a subsequent deployment reintroduced it. We mitigated the second period at 17:40 UTC by rolling back again, then processed queued events and cleared backlogs. <br /><br />We are taking several steps to prevent a repeat. The routing service will keep its existing behavior when a feature toggle is missing, so a misconfigured flag can no longer pause delivery. We are adding a pre-deploy check that each routing flag exists and is registered under the correct owning service, and we are making flag ownership less error-prone to set. We are also improving alerting so we detect paused event delivery sooner..</p> <p><small>Nov <var data-var='date'> 4</var>, <var data-var='time'>16:49:27</var> GMT+0</small><br><strong>Monitoring</strong> - The degradation has been mitigated. We are monitoring to ensure stability..</p> <p><small>Nov <var data-var='date'> 4</var>, <var data-var='time'>16:48:03</var> GMT+0</small><br><strong>Monitoring</strong> - The degradation affecting Actions and Pages has been mitigated. We are monitoring to ensure stability..</p> <p><small>Nov <var data-var='date'> 4</var>, <var data-var='time'>16:45:48</var> GMT+0</small><br><strong>Investigating</strong> - The degradation affecting Webhooks has been mitigated. We are monitoring to ensure stability..</p> <p><small>Nov <var data-var='date'> 4</var>, <var data-var='time'>16:34:45</var> GMT+0</small><br><strong>Investigating</strong> - We have identified an issue with events that might trigger Actions. We have applied mitigations and are monitoring for recovery at this point. Customers may observe delayed status updates for their jobs..</p> <p><small>Nov <var data-var='date'> 4</var>, <var data-var='time'>16:17:29</var> GMT+0</small><br><strong>Investigating</strong> - Actions is experiencing degraded availability. We are continuing to investigate..</p> <p><small>Nov <var data-var='date'> 4</var>, <var data-var='time'>16:12:55</var> GMT+0</small><br><strong>Investigating</strong> - Webhooks is experiencing degraded performance. We are continuing to investigate..</p> <p><small>Nov <var data-var='date'> 4</var>, <var data-var='time'>16:11:38</var> GMT+0</small><br><strong>Investigating</strong> - We are investigating reports of issues with Actions and Pages. We will continue to keep users updated on progress towards mitigation..</p> <p><small>Nov <var data-var='date'> 4</var>, <var data-var='time'>16:05:31</var> GMT+0</small><br><strong>Investigating</strong> - We are investigating reports of degraded performance for Actions and Pages.</p> ]]> </content:encoded> <pubDate>Wed, 4 Nov 2026 16:05:31 +0000</pubDate> <link>https://status.ubicloud.com/incident/cmw2bap2g01fa0rqedvttcnqp</link> <guid>https://status.ubicloud.com/incident/cmw2bap2g01fa0rqedvttcnqp</guid> </item> <item> <title>Incident with Pull Requests, Git Operations and Issues</title> <description> Type: Incident Duration: 5 days and 10 minutes Affected Components: GitHub → Actions, GitHub → API Requests Nov 4, 11:50:06 GMT+0 - Resolved - n November 4, 2026, between 09:35 UTC and 11:43 UTC, customers experienced intermittent errors and timeouts across GitHub.com, primarily on actions that write data. At peak, about 20% of pull request write operations (such as merging, commenting, and reviewing) failed, along with about 10% of issue creates and edits and 10–15% of project item edits. Git pushes over SSH, REST and GraphQL API requests, code scanning uploads, releases, and package operations were also affected, and about 1% of Actions runs were delayed by more than five minutes or failed to start. Overall, nearly 4% of all requests to GitHub.com failed at the worst point. Reads were largely unaffected, and retrying a failed action a short time later usually succeeded.<br /><br />The incident was triggered by a burst of concurrent updates to the same database record, which tied up all available connections to one of our primary databases. Although that contention cleared, the backlog of waiting requests kept our database connection layer overloaded, and it did not recover on its own. We mitigated the incident by failing over to a new primary database host. Most services recovered by 11:31 UTC, and Actions recovered by 11:43 UTC.<br /><br />Since the incident, we have disabled the code path that triggered the contention, increased the processing capacity of our database connection layer by 50%, and added alerting to detect this type of overload earlier. We are also improving our procedures so we can fail over sooner in similar situations. Nov 4, 11:43:52 GMT+0 - Monitoring - The degradation affecting Actions has been mitigated. We are monitoring to ensure stability. Nov 4, 11:30:42 GMT+0 - Investigating - The degradation affecting API Requests, Git Operations, Issues and Pull Requests has been mitigated. We are monitoring to ensure stability. Nov 4, 11:17:52 GMT+0 - Investigating - We're currently working on mitigating elevated errors and intermittent failures affecting GitHub, including Pull Requests, Git Operations, and Issues. Some requests may fail or be delayed while we work to restore service. We'll provide another update as more information becomes available. Nov 4, 11:11:30 GMT+0 - Investigating - API Requests is experiencing degraded performance. We are continuing to investigate. Nov 4, 10:38:49 GMT+0 - Investigating - We're investigating elevated errors and intermittent failures affecting GitHub including Pull Requests, Git Operations, and Issues. We'll continue to keep users updated as we work toward mitigation. Nov 4, 09:53:38 GMT+0 - Investigating - Pull Requests is experiencing degraded availability. We are continuing to investigate. Nov 4, 09:50:47 GMT+0 - Investigating - Actions is experiencing degraded performance. We are continuing to investigate. Nov 4, 09:49:55 GMT+0 - Investigating - We are investigating reports of degraded performance for Git Operations, Issues and Pull Requests </description> <content:encoded> <![CDATA[<p><strong>Type:</strong> Incident</p> <p><strong>Duration:</strong> 5 days and 10 minutes</p> <p><strong>Affected Components:</strong> </p> <p><small>Nov <var data-var='date'> 4</var>, <var data-var='time'>11:50:06</var> GMT+0</small><br><strong>Resolved</strong> - n November 4, 2026, between 09:35 UTC and 11:43 UTC, customers experienced intermittent errors and timeouts across GitHub.com, primarily on actions that write data. At peak, about 20% of pull request write operations (such as merging, commenting, and reviewing) failed, along with about 10% of issue creates and edits and 10–15% of project item edits. Git pushes over SSH, REST and GraphQL API requests, code scanning uploads, releases, and package operations were also affected, and about 1% of Actions runs were delayed by more than five minutes or failed to start. Overall, nearly 4% of all requests to GitHub.com failed at the worst point. Reads were largely unaffected, and retrying a failed action a short time later usually succeeded.<br /><br />The incident was triggered by a burst of concurrent updates to the same database record, which tied up all available connections to one of our primary databases. Although that contention cleared, the backlog of waiting requests kept our database connection layer overloaded, and it did not recover on its own. We mitigated the incident by failing over to a new primary database host. Most services recovered by 11:31 UTC, and Actions recovered by 11:43 UTC.<br /><br />Since the incident, we have disabled the code path that triggered the contention, increased the processing capacity of our database connection layer by 50%, and added alerting to detect this type of overload earlier. We are also improving our procedures so we can fail over sooner in similar situations..</p> <p><small>Nov <var data-var='date'> 4</var>, <var data-var='time'>11:43:52</var> GMT+0</small><br><strong>Monitoring</strong> - The degradation affecting Actions has been mitigated. We are monitoring to ensure stability..</p> <p><small>Nov <var data-var='date'> 4</var>, <var data-var='time'>11:30:42</var> GMT+0</small><br><strong>Investigating</strong> - The degradation affecting API Requests, Git Operations, Issues and Pull Requests has been mitigated. We are monitoring to ensure stability..</p> <p><small>Nov <var data-var='date'> 4</var>, <var data-var='time'>11:17:52</var> GMT+0</small><br><strong>Investigating</strong> - We're currently working on mitigating elevated errors and intermittent failures affecting GitHub, including Pull Requests, Git Operations, and Issues. Some requests may fail or be delayed while we work to restore service. We'll provide another update as more information becomes available..</p> <p><small>Nov <var data-var='date'> 4</var>, <var data-var='time'>11:11:30</var> GMT+0</small><br><strong>Investigating</strong> - API Requests is experiencing degraded performance. We are continuing to investigate..</p> <p><small>Nov <var data-var='date'> 4</var>, <var data-var='time'>10:38:49</var> GMT+0</small><br><strong>Investigating</strong> - We're investigating elevated errors and intermittent failures affecting GitHub including Pull Requests, Git Operations, and Issues. We'll continue to keep users updated as we work toward mitigation..</p> <p><small>Nov <var data-var='date'> 4</var>, <var data-var='time'>09:53:38</var> GMT+0</small><br><strong>Investigating</strong> - Pull Requests is experiencing degraded availability. We are continuing to investigate..</p> <p><small>Nov <var data-var='date'> 4</var>, <var data-var='time'>09:50:47</var> GMT+0</small><br><strong>Investigating</strong> - Actions is experiencing degraded performance. We are continuing to investigate..</p> <p><small>Nov <var data-var='date'> 4</var>, <var data-var='time'>09:49:55</var> GMT+0</small><br><strong>Investigating</strong> - We are investigating reports of degraded performance for Git Operations, Issues and Pull Requests.</p> ]]> </content:encoded> <pubDate>Wed, 4 Nov 2026 09:49:55 +0000</pubDate> <link>https://status.ubicloud.com/incident/cmw1xwc000yu60rofi7yazw9t</link> <guid>https://status.ubicloud.com/incident/cmw1xwc000yu60rofi7yazw9t</guid> </item> <item> <title>Network connectivity issues in Virgina, US</title> <description> Type: Incident Affected Components: PostgreSQL, Networking, Compute Nov 4, 08:00:00 GMT+0 - Resolved - This incident has been resolved. </description> <content:encoded> <![CDATA[<p><strong>Type:</strong> Incident</p> <p><strong>Affected Components:</strong> </p> <p><small>Nov <var data-var='date'> 4</var>, <var data-var='time'>08:00:00</var> GMT+0</small><br><strong>Resolved</strong> - This incident has been resolved..</p> ]]> </content:encoded> <pubDate>Wed, 4 Nov 2026 08:00:00 +0000</pubDate> <link>https://status.ubicloud.com/incident/cmw1stz3906le13pk3jojpaq1</link> <guid>https://status.ubicloud.com/incident/cmw1stz3906le13pk3jojpaq1</guid> </item> <item> <title>Incident with API, Git, and Copilot requests</title> <description> Type: Incident Duration: 1 day, 6 hours and 59 minutes Affected Components: GitHub → API Requests Nov 3, 13:56:18 GMT+0 - Resolved - On November 3, 2026, between 12:34 and 13:51 UTC, some users experienced authentication failures and errors with the REST and GraphQL APIs, pull requests, Issues, and GitHub Copilot. On average, the error rate was 0.46% and peaked at 0.98% of API requests. Copilot API errors averaged 2.0% and peaked at 3.1%.<br /><br />This was caused by a bug in our process for replacing control-plane servers in one of our compute clusters. When a replacement server did not receive a timely response from the existing cluster, it incorrectly created a separate cluster. This stopped traffic from reaching much of the cluster's capacity and overloaded the authentication servers that were still handling requests.<br /><br />We mitigated the incident by removing the incorrectly configured server.<br />We are also fixing the replacement logic and adding safeguards to prevent servers in this state from serving traffic. Nov 3, 13:54:20 GMT+0 - Monitoring - The degradation has been mitigated. We are monitoring to ensure stability. Nov 3, 13:52:21 GMT+0 - Investigating - API Requests is operating normally. Nov 3, 13:51:46 GMT+0 - Monitoring - The degradation affecting Copilot has been mitigated. We are monitoring to ensure stability. </description> <content:encoded> <![CDATA[<p><strong>Type:</strong> Incident</p> <p><strong>Duration:</strong> 1 day, 6 hours and 59 minutes</p> <p><strong>Affected Components:</strong> </p> <p><small>Nov <var data-var='date'> 3</var>, <var data-var='time'>13:56:18</var> GMT+0</small><br><strong>Resolved</strong> - On November 3, 2026, between 12:34 and 13:51 UTC, some users experienced authentication failures and errors with the REST and GraphQL APIs, pull requests, Issues, and GitHub Copilot. On average, the error rate was 0.46% and peaked at 0.98% of API requests. Copilot API errors averaged 2.0% and peaked at 3.1%.<br /><br />This was caused by a bug in our process for replacing control-plane servers in one of our compute clusters. When a replacement server did not receive a timely response from the existing cluster, it incorrectly created a separate cluster. This stopped traffic from reaching much of the cluster's capacity and overloaded the authentication servers that were still handling requests.<br /><br />We mitigated the incident by removing the incorrectly configured server.<br />We are also fixing the replacement logic and adding safeguards to prevent servers in this state from serving traffic..</p> <p><small>Nov <var data-var='date'> 3</var>, <var data-var='time'>13:54:20</var> GMT+0</small><br><strong>Monitoring</strong> - The degradation has been mitigated. We are monitoring to ensure stability..</p> <p><small>Nov <var data-var='date'> 3</var>, <var data-var='time'>13:52:21</var> GMT+0</small><br><strong>Investigating</strong> - API Requests is operating normally..</p> <p><small>Nov <var data-var='date'> 3</var>, <var data-var='time'>13:51:46</var> GMT+0</small><br><strong>Monitoring</strong> - The degradation affecting Copilot has been mitigated. We are monitoring to ensure stability..</p> ]]> </content:encoded> <pubDate>Tue, 3 Nov 2026 13:25:19 +0000</pubDate> <link>https://status.ubicloud.com/incident/cmw0r14r600lz0krtcvtlmjoo</link> <guid>https://status.ubicloud.com/incident/cmw0r14r600lz0krtcvtlmjoo</guid> </item> <item> <title>Incident with Actions, Pages, and Webhooks</title> <description> Type: Incident Duration: 3 days, 20 hours and 45 minutes Affected Components: GitHub → Webhooks, GitHub → Actions Oct 30, 19:23:40 GMT+0 - Resolved - On October 30, 2026, from 17:50 to 18:32 UTC, Actions, Pages, and Webhooks experienced degraded performance. Actions workflow processing, Pages deployments, and webhook deliveries were delayed, and code search and live updates returned errors. Some Actions queues took additional time to drain after mitigation, and the incident was resolved at 19:23 UTC. <br /><br />The impact was triggered by maintenance on a cloud network gateway connecting an Actions service in the GitHub data center to Redis clusters in the Central US Azure region. The service's Redis client retried aggressively and redialed those connections. That produced a connection storm that saturated the gateway, with about 90% of connection attempts failing. Other services sharing the same network path were degraded as well. <br /><br />We mitigated the incident by removing the affected resources from rotation, which ended the connection storm and allowed the gateway to recover. We have already reduced the number of connection retries in the impacted service and are further improving retry behavior with jittered backoff, shorter connect timeouts, and retry budgets. We are also working with our infrastructure team to make GitHub more resilient to gateway maintenance. Oct 30, 18:32:39 GMT+0 - Monitoring - The degradation affecting Actions, Pages and Webhooks has been mitigated. We are monitoring to ensure stability. Oct 30, 18:32:23 GMT+0 - Investigating - All affected services have recovered following mitigation. We continue to investigate root cause. Oct 30, 18:06:21 GMT+0 - Investigating - We continue to investigate delays affecting Actions workflows, Pages deployments, and webhook events, along with errors affecting code search and live updates. We have taken mitigation steps. Actions processing and Pages deployments are showing signs of recovery, but impact has not fully resolved. We are monitoring recovery and investigating the underlying cause. Oct 30, 18:00:58 GMT+0 - Investigating - Pages is experiencing degraded performance. We are continuing to investigate. Oct 30, 17:50:54 GMT+0 - Investigating - We are investigating reports of degraded performance for Actions and Webhooks </description> <content:encoded> <![CDATA[<p><strong>Type:</strong> Incident</p> <p><strong>Duration:</strong> 3 days, 20 hours and 45 minutes</p> <p><strong>Affected Components:</strong> </p> <p><small>Oct <var data-var='date'> 30</var>, <var data-var='time'>19:23:40</var> GMT+0</small><br><strong>Resolved</strong> - On October 30, 2026, from 17:50 to 18:32 UTC, Actions, Pages, and Webhooks experienced degraded performance. Actions workflow processing, Pages deployments, and webhook deliveries were delayed, and code search and live updates returned errors. Some Actions queues took additional time to drain after mitigation, and the incident was resolved at 19:23 UTC. <br /><br />The impact was triggered by maintenance on a cloud network gateway connecting an Actions service in the GitHub data center to Redis clusters in the Central US Azure region. The service's Redis client retried aggressively and redialed those connections. That produced a connection storm that saturated the gateway, with about 90% of connection attempts failing. Other services sharing the same network path were degraded as well. <br /><br />We mitigated the incident by removing the affected resources from rotation, which ended the connection storm and allowed the gateway to recover. We have already reduced the number of connection retries in the impacted service and are further improving retry behavior with jittered backoff, shorter connect timeouts, and retry budgets. We are also working with our infrastructure team to make GitHub more resilient to gateway maintenance..</p> <p><small>Oct <var data-var='date'> 30</var>, <var data-var='time'>18:32:39</var> GMT+0</small><br><strong>Monitoring</strong> - The degradation affecting Actions, Pages and Webhooks has been mitigated. We are monitoring to ensure stability..</p> <p><small>Oct <var data-var='date'> 30</var>, <var data-var='time'>18:32:23</var> GMT+0</small><br><strong>Investigating</strong> - All affected services have recovered following mitigation. We continue to investigate root cause..</p> <p><small>Oct <var data-var='date'> 30</var>, <var data-var='time'>18:06:21</var> GMT+0</small><br><strong>Investigating</strong> - We continue to investigate delays affecting Actions workflows, Pages deployments, and webhook events, along with errors affecting code search and live updates. We have taken mitigation steps. Actions processing and Pages deployments are showing signs of recovery, but impact has not fully resolved. We are monitoring recovery and investigating the underlying cause..</p> <p><small>Oct <var data-var='date'> 30</var>, <var data-var='time'>18:00:58</var> GMT+0</small><br><strong>Investigating</strong> - Pages is experiencing degraded performance. We are continuing to investigate..</p> <p><small>Oct <var data-var='date'> 30</var>, <var data-var='time'>17:50:54</var> GMT+0</small><br><strong>Investigating</strong> - We are investigating reports of degraded performance for Actions and Webhooks.</p> ]]> </content:encoded> <pubDate>Fri, 30 Oct 2026 17:50:54 +0000</pubDate> <link>https://status.ubicloud.com/incident/cmvv9ud1g1i080rp2mxkdw68p</link> <guid>https://status.ubicloud.com/incident/cmvv9ud1g1i080rp2mxkdw68p</guid> </item> <item> <title>Increased provisioning times for GitHub runners</title> <description> Type: Incident Duration: 1 hour and 57 minutes Affected Components: Ubicloud Runners Oct 30, 13:17:38 GMT+0 - Resolved - We investigated the cache slowness. On October 29, Cloudflare had issues on several of their services. From approximately 09:00 to 22:00 UTC, Cloudflare sent our cache traffic to its Prague location instead of Frankfurt, and Cloudflare R2 had increased latency. Because of this, cache upload and download speeds were much lower than normal. Cache performance has been normal since approximately 23:00 UTC on October 29. We are working with Cloudflare to find the root cause, and to prevent and detect similar problems earlier in the future. Oct 29, 19:17:27 GMT+0 - Resolved - Wait times for GitHub Actions runners are back to normal. Cause: Today we saw a surge in demand that put pressure on our capacity at peak times, especially while EU evenings and US mornings overlapped. This week the overlap is an hour longer than usual because Europe has already switched to winter time and the US hasn't yet. Slowness in Cloudflare R2, which we use for cache storage, also made jobs run longer. This further added to queue times. We're adding capacity to handle a continuous increase in overall demand. We apologize for the delay in processing times. Oct 29, 18:23:28 GMT+0 - Monitoring - Wait times are improving. We're processing the remaining backlog and monitoring until queue times are back to normal. Oct 29, 17:20:35 GMT+0 - Investigating - We're seeing longer than usual wait times for GitHub Actions runners to start, due to high demand. We're investigating and will post updates here. </description> <content:encoded> <![CDATA[<p><strong>Type:</strong> Incident</p> <p><strong>Duration:</strong> 1 hour and 57 minutes</p> <p><strong>Affected Components:</strong> </p> <p><small>Oct <var data-var='date'> 30</var>, <var data-var='time'>13:17:38</var> GMT+0</small><br><strong>Resolved</strong> - We investigated the cache slowness. On October 29, Cloudflare had issues on several of their services. From approximately 09:00 to 22:00 UTC, Cloudflare sent our cache traffic to its Prague location instead of Frankfurt, and Cloudflare R2 had increased latency. Because of this, cache upload and download speeds were much lower than normal. Cache performance has been normal since approximately 23:00 UTC on October 29. We are working with Cloudflare to find the root cause, and to prevent and detect similar problems earlier in the future..</p> <p><small>Oct <var data-var='date'> 29</var>, <var data-var='time'>19:17:27</var> GMT+0</small><br><strong>Resolved</strong> - Wait times for GitHub Actions runners are back to normal. Cause: Today we saw a surge in demand that put pressure on our capacity at peak times, especially while EU evenings and US mornings overlapped. This week the overlap is an hour longer than usual because Europe has already switched to winter time and the US hasn't yet. Slowness in Cloudflare R2, which we use for cache storage, also made jobs run longer. This further added to queue times. We're adding capacity to handle a continuous increase in overall demand. We apologize for the delay in processing times..</p> <p><small>Oct <var data-var='date'> 29</var>, <var data-var='time'>18:23:28</var> GMT+0</small><br><strong>Monitoring</strong> - Wait times are improving. We're processing the remaining backlog and monitoring until queue times are back to normal..</p> <p><small>Oct <var data-var='date'> 29</var>, <var data-var='time'>17:20:35</var> GMT+0</small><br><strong>Investigating</strong> - We're seeing longer than usual wait times for GitHub Actions runners to start, due to high demand. We're investigating and will post updates here..</p> ]]> </content:encoded> <pubDate>Thu, 29 Oct 2026 17:20:35 +0000</pubDate> <link>https://status.ubicloud.com/incident/cmvtt7f1k02sm1mphc3uplj42</link> <guid>https://status.ubicloud.com/incident/cmvtt7f1k02sm1mphc3uplj42</guid> </item> <item> <title>API is back up</title> <description> Type: Incident Affected Components: API Oct 28, 01:02:25 GMT+0 - Resolved - API is back up. This incident was automatically resolved by Instatus monitoring. Oct 28, 01:01:52 GMT+0 - Investigating - API is down at the moment. This incident was automatically created by Instatus monitoring. </description> <content:encoded> <![CDATA[<p><strong>Type:</strong> Incident</p> <p><strong>Affected Components:</strong> </p> <p><small>Oct <var data-var='date'> 28</var>, <var data-var='time'>01:02:25</var> GMT+0</small><br><strong>Resolved</strong> - API is back up. This incident was automatically resolved by Instatus monitoring..</p> <p><small>Oct <var data-var='date'> 28</var>, <var data-var='time'>01:01:52</var> GMT+0</small><br><strong>Investigating</strong> - API is down at the moment. This incident was automatically created by Instatus monitoring..</p> ]]> </content:encoded> <pubDate>Wed, 28 Oct 2026 01:01:52 +0000</pubDate> <link>https://status.ubicloud.com/incident/cmvreswkz004x07nqbibxgjpf</link> <guid>https://status.ubicloud.com/incident/cmvreswkz004x07nqbibxgjpf</guid> </item> <item> <title>Console is back up</title> <description> Type: Incident Affected Components: Console Oct 28, 01:02:31 GMT+0 - Resolved - Console is back up. This incident was automatically resolved by Instatus monitoring. Oct 28, 01:01:51 GMT+0 - Investigating - Console is down at the moment. This incident was automatically created by Instatus monitoring. </description> <content:encoded> <![CDATA[<p><strong>Type:</strong> Incident</p> <p><strong>Affected Components:</strong> </p> <p><small>Oct <var data-var='date'> 28</var>, <var data-var='time'>01:02:31</var> GMT+0</small><br><strong>Resolved</strong> - Console is back up. This incident was automatically resolved by Instatus monitoring..</p> <p><small>Oct <var data-var='date'> 28</var>, <var data-var='time'>01:01:51</var> GMT+0</small><br><strong>Investigating</strong> - Console is down at the moment. This incident was automatically created by Instatus monitoring..</p> ]]> </content:encoded> <pubDate>Wed, 28 Oct 2026 01:01:51 +0000</pubDate> <link>https://status.ubicloud.com/incident/cmvresy6j006w07o9m25m1q6f</link> <guid>https://status.ubicloud.com/incident/cmvresy6j006w07o9m25m1q6f</guid> </item> <item> <title>Incident with Actions, Issues, API Requests and Pull Requests</title> <description> Type: Incident Duration: 8 hours and 56 minutes Affected Components: GitHub → API Requests, GitHub → Actions Oct 26, 18:57:22 GMT+0 - Resolved - On October 26, 2026, between 17:52 and 18:49 UTC, some customers experienced slower response times across the Checks API, Issues, Pull Requests, GraphQL, Notifications, and related services. Impact was limited to requests served from one of our datacenter sites. Some GraphQL requests failed, and some notifications were delayed. Among GraphQL requests served from the affected site, the average error rate was 5.68%, with a peak five-minute error rate of 14.32%. <br /><br />Unusually large requests for contribution history overwhelmed part of our database infrastructure, causing a backlog. Requests continued reaching the slow component rather than being redirected to healthy systems. Restarting the component restored its performance. The incident was marked mitigated at 18:46 UTC and resolved at 18:57 UTC. <br /><br />We are adding limits to these requests, improving alerts to detect overload sooner, and strengthening automatic redirection to healthy systems. Oct 26, 18:54:04 GMT+0 - Monitoring - We saw elevated latency affecting REST API requests, including Checks, and some Actions, Issues, Pull Requests, and GraphQL functionality. Some requests may have been slow or timed out, and notifications were delayed from 17:59 UTC - 18:40 UTC.<br /><br />We mitigated the issue and all services have recovered. Oct 26, 18:50:58 GMT+0 - Monitoring - The degradation has been mitigated. We are monitoring to ensure stability. Oct 26, 18:48:26 GMT+0 - Investigating - We are investigating reports of degraded performance for API Requests, Actions, Issues and Pull Requests </description> <content:encoded> <![CDATA[<p><strong>Type:</strong> Incident</p> <p><strong>Duration:</strong> 8 hours and 56 minutes</p> <p><strong>Affected Components:</strong> </p> <p><small>Oct <var data-var='date'> 26</var>, <var data-var='time'>18:57:22</var> GMT+0</small><br><strong>Resolved</strong> - On October 26, 2026, between 17:52 and 18:49 UTC, some customers experienced slower response times across the Checks API, Issues, Pull Requests, GraphQL, Notifications, and related services. Impact was limited to requests served from one of our datacenter sites. Some GraphQL requests failed, and some notifications were delayed. Among GraphQL requests served from the affected site, the average error rate was 5.68%, with a peak five-minute error rate of 14.32%. <br /><br />Unusually large requests for contribution history overwhelmed part of our database infrastructure, causing a backlog. Requests continued reaching the slow component rather than being redirected to healthy systems. Restarting the component restored its performance. The incident was marked mitigated at 18:46 UTC and resolved at 18:57 UTC. <br /><br />We are adding limits to these requests, improving alerts to detect overload sooner, and strengthening automatic redirection to healthy systems..</p> <p><small>Oct <var data-var='date'> 26</var>, <var data-var='time'>18:54:04</var> GMT+0</small><br><strong>Monitoring</strong> - We saw elevated latency affecting REST API requests, including Checks, and some Actions, Issues, Pull Requests, and GraphQL functionality. Some requests may have been slow or timed out, and notifications were delayed from 17:59 UTC - 18:40 UTC.<br /><br />We mitigated the issue and all services have recovered..</p> <p><small>Oct <var data-var='date'> 26</var>, <var data-var='time'>18:50:58</var> GMT+0</small><br><strong>Monitoring</strong> - The degradation has been mitigated. We are monitoring to ensure stability..</p> <p><small>Oct <var data-var='date'> 26</var>, <var data-var='time'>18:48:26</var> GMT+0</small><br><strong>Investigating</strong> - We are investigating reports of degraded performance for API Requests, Actions, Issues and Pull Requests.</p> ]]> </content:encoded> <pubDate>Mon, 26 Oct 2026 18:48:26 +0000</pubDate> <link>https://status.ubicloud.com/incident/cmvpm2513044p0rnr0t0v76xf</link> <guid>https://status.ubicloud.com/incident/cmvpm2513044p0rnr0t0v76xf</guid> </item> <item> <title>Delayed Actions status check results</title> <description> Type: Incident Duration: 1 day, 12 hours and 24 minutes Affected Components: GitHub → Actions Oct 23, 08:05:27 GMT+0 - Resolved - On October 23, 2026, between 07:06 UTC and 07:27 UTC, GitHub Actions workflow run results were delayed. Job status, check run updates, and logs in the GitHub UI and API updated with an average delay of about 2 minutes, and some updates were delayed by up to about 7 minutes. <br /><br />During this window, a storage dependency that Actions uses to process workflow logs and artifacts throttled our requests. This slowed result processing, causing updates to queue up. The queue was fully processed by 07:27 UTC, shortly after the throttling stopped. <br /><br />To reduce the risk of recurrence, we have added several storage accounts to increase capacity, and we are working on additional steps to continue to scale. We are also working with our storage provider to understand why the throttling occurred, and improving how result processing handles storage throttling and how quickly we detect it. Oct 23, 07:56:35 GMT+0 - Monitoring - As of 07:27 UTC, Actions results are delivering within normal SLOs again.<br /><br />Thank you for this patience during this period of delayed Actions results delivery. Our team continues to monitor system health and explore refinements to reduce impact to customers in the future. Oct 23, 07:53:40 GMT+0 - Monitoring - The degradation affecting Actions has been mitigated. We are monitoring to ensure stability. Oct 23, 07:31:54 GMT+0 - Investigating - We are seeing delays in results from Actions workflows being reported within GitHub starting from 07:07 UTC.<br /><br />Recovery is already being seen but our team is continuing to investigate and monitor performance. Oct 23, 07:29:03 GMT+0 - Investigating - We are investigating reports of degraded performance for Actions </description> <content:encoded> <![CDATA[<p><strong>Type:</strong> Incident</p> <p><strong>Duration:</strong> 1 day, 12 hours and 24 minutes</p> <p><strong>Affected Components:</strong> </p> <p><small>Oct <var data-var='date'> 23</var>, <var data-var='time'>08:05:27</var> GMT+0</small><br><strong>Resolved</strong> - On October 23, 2026, between 07:06 UTC and 07:27 UTC, GitHub Actions workflow run results were delayed. Job status, check run updates, and logs in the GitHub UI and API updated with an average delay of about 2 minutes, and some updates were delayed by up to about 7 minutes. <br /><br />During this window, a storage dependency that Actions uses to process workflow logs and artifacts throttled our requests. This slowed result processing, causing updates to queue up. The queue was fully processed by 07:27 UTC, shortly after the throttling stopped. <br /><br />To reduce the risk of recurrence, we have added several storage accounts to increase capacity, and we are working on additional steps to continue to scale. We are also working with our storage provider to understand why the throttling occurred, and improving how result processing handles storage throttling and how quickly we detect it..</p> <p><small>Oct <var data-var='date'> 23</var>, <var data-var='time'>07:56:35</var> GMT+0</small><br><strong>Monitoring</strong> - As of 07:27 UTC, Actions results are delivering within normal SLOs again.<br /><br />Thank you for this patience during this period of delayed Actions results delivery. Our team continues to monitor system health and explore refinements to reduce impact to customers in the future..</p> <p><small>Oct <var data-var='date'> 23</var>, <var data-var='time'>07:53:40</var> GMT+0</small><br><strong>Monitoring</strong> - The degradation affecting Actions has been mitigated. We are monitoring to ensure stability..</p> <p><small>Oct <var data-var='date'> 23</var>, <var data-var='time'>07:31:54</var> GMT+0</small><br><strong>Investigating</strong> - We are seeing delays in results from Actions workflows being reported within GitHub starting from 07:07 UTC.<br /><br />Recovery is already being seen but our team is continuing to investigate and monitor performance..</p> <p><small>Oct <var data-var='date'> 23</var>, <var data-var='time'>07:29:03</var> GMT+0</small><br><strong>Investigating</strong> - We are investigating reports of degraded performance for Actions.</p> ]]> </content:encoded> <pubDate>Fri, 23 Oct 2026 07:29:03 +0000</pubDate> <link>https://status.ubicloud.com/incident/cmvknm0xe13660ro2mjyr3c5w</link> <guid>https://status.ubicloud.com/incident/cmvknm0xe13660ro2mjyr3c5w</guid> </item> </channel> </rss>