Snapshot 315491
Raw (normalized text)
Scripts and page chrome removed; this is what change detection compares.
<?xml version="1.0" encoding="UTF-8"?> <rss version="2.0" xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/"> <channel> <title>Macroscope Status - Incident history</title> <link>https://status.macroscope.com</link> <description>Macroscope</description> <pubDate>Mon, 16 Nov 2026 11:59:40 +0000</pubDate> <item> <title>Incident with Pull Requests</title> <description> Type: Incident Duration: 11 days, 2 hours and 45 minutes Affected Components: GitHub Actions Nov 16, 16:26:26 GMT+0 - Resolved - On November 16, 2026, between 09:30 and 15:55 UTC, some pull request merges on GitHub timed out. On average, 2.9% of merge attempts timed out, with a peak of 7.1% at 15:17 UTC. Some GitHub Actions workflow runs triggered by pull requests were also delayed. Git operations such as push and fetch stayed available, with higher latency. This was a repeat of a shorter event on November 13, between 13:15 and 17:50 UTC, when 1.7% of merges timed out.<br /><br />Both events had the same cause: some of the storage servers that hold Git repository data reached their disk read bandwidth limits. A merge must be written to every copy of a repository, so one overloaded server slowed merges for every repository it hosts. Read load on these servers had been growing as part of a planned move of repositories to new infrastructure. A disk configuration setting on those servers also made them read far more data than necessary, which cut their usable bandwidth roughly in half. We mitigated the incident by correcting that setting and moving some load away from the busiest servers.<br /><br />The corrected setting is now applied permanently across our storage fleet. We are adding storage capacity in the affected region and alerting on per-server disk bandwidth so we catch saturation before it affects customers. Sustained merge timeouts will now page our on-call engineers directly. We are also adding capacity checks before moving repositories or traffic onto new infrastructure. Nov 16, 16:16:07 GMT+0 - Monitoring - We’ve mitigated the issue affecting pull request merges and Actions workflow start times by reducing load on affected systems. Pull Requests and Actions have recovered, and we’re continuing to monitor their performance. Nov 16, 16:07:32 GMT+0 - Monitoring - The degradation affecting Actions and Pull Requests has been mitigated. We are monitoring to ensure stability. Nov 16, 15:55:07 GMT+0 - Investigating - We’re seeing early signs of improvement in pull request merge times & workflow start times, but recovery is not yet confirmed. Users may still have difficulty merging pull requests. We’re reducing load on affected systems and monitoring the impact. We’ll share another update as we learn more. Nov 16, 15:46:44 GMT+0 - Investigating - We’re seeing early signs of improvement in pull request merge times, but recovery is not yet confirmed. The last confirmed timeout rate was 2.5%, compared with a baseline near 0.2%, so some users may still have difficulty merging pull requests. We’ll share another update as we learn more. Nov 16, 15:33:14 GMT+0 - Investigating - Actions is experiencing degraded performance. We are continuing to investigate. </description> <content:encoded> <![CDATA[<p><strong>Type:</strong> Incident</p> <p><strong>Duration:</strong> 11 days, 2 hours and 45 minutes</p> <p><strong>Affected Components:</strong> </p> <p><small>Nov <var data-var='date'> 16</var>, <var data-var='time'>16:26:26</var> GMT+0</small><br><strong>Resolved</strong> - On November 16, 2026, between 09:30 and 15:55 UTC, some pull request merges on GitHub timed out. On average, 2.9% of merge attempts timed out, with a peak of 7.1% at 15:17 UTC. Some GitHub Actions workflow runs triggered by pull requests were also delayed. Git operations such as push and fetch stayed available, with higher latency. This was a repeat of a shorter event on November 13, between 13:15 and 17:50 UTC, when 1.7% of merges timed out.<br /><br />Both events had the same cause: some of the storage servers that hold Git repository data reached their disk read bandwidth limits. A merge must be written to every copy of a repository, so one overloaded server slowed merges for every repository it hosts. Read load on these servers had been growing as part of a planned move of repositories to new infrastructure. A disk configuration setting on those servers also made them read far more data than necessary, which cut their usable bandwidth roughly in half. We mitigated the incident by correcting that setting and moving some load away from the busiest servers.<br /><br />The corrected setting is now applied permanently across our storage fleet. We are adding storage capacity in the affected region and alerting on per-server disk bandwidth so we catch saturation before it affects customers. Sustained merge timeouts will now page our on-call engineers directly. We are also adding capacity checks before moving repositories or traffic onto new infrastructure..</p> <p><small>Nov <var data-var='date'> 16</var>, <var data-var='time'>16:16:07</var> GMT+0</small><br><strong>Monitoring</strong> - We’ve mitigated the issue affecting pull request merges and Actions workflow start times by reducing load on affected systems. Pull Requests and Actions have recovered, and we’re continuing to monitor their performance..</p> <p><small>Nov <var data-var='date'> 16</var>, <var data-var='time'>16:07:32</var> GMT+0</small><br><strong>Monitoring</strong> - The degradation affecting Actions and Pull Requests has been mitigated. We are monitoring to ensure stability..</p> <p><small>Nov <var data-var='date'> 16</var>, <var data-var='time'>15:55:07</var> GMT+0</small><br><strong>Investigating</strong> - We’re seeing early signs of improvement in pull request merge times & workflow start times, but recovery is not yet confirmed. Users may still have difficulty merging pull requests. We’re reducing load on affected systems and monitoring the impact. We’ll share another update as we learn more..</p> <p><small>Nov <var data-var='date'> 16</var>, <var data-var='time'>15:46:44</var> GMT+0</small><br><strong>Investigating</strong> - We’re seeing early signs of improvement in pull request merge times, but recovery is not yet confirmed. The last confirmed timeout rate was 2.5%, compared with a baseline near 0.2%, so some users may still have difficulty merging pull requests. We’ll share another update as we learn more..</p> <p><small>Nov <var data-var='date'> 16</var>, <var data-var='time'>15:33:14</var> GMT+0</small><br><strong>Investigating</strong> - Actions is experiencing degraded performance. We are continuing to investigate..</p> ]]> </content:encoded> <pubDate>Mon, 16 Nov 2026 11:59:40 +0000</pubDate> <link>https://status.macroscope.com/incident/cmwjfbsvy02ek0kpbema88tiv</link> <guid>https://status.macroscope.com/incident/cmwjfbsvy02ek0kpbema88tiv</guid> </item> <item> <title>Impact across multiple GitHub products including Actions, Pages, and Copilot products.</title> <description> Type: Incident Duration: 1 day, 9 hours and 27 minutes Affected Components: GitHub Actions Nov 11, 21:21:39 GMT+0 - Resolved - On November 11, 2026, from 20:05 to 21:02 UTC, customers experienced delayed or failed Actions workflow starts, webhook errors and delivery delays, slower GitHub Apps API requests, and reduced availability of Copilot features, including Code Review. GitHub Pages builds were also delayed.<br /><br />Nearly all GitHub Actions runs triggered between 20:18 and 20:55 UTC were delayed or failed, leaving approximately 3.4 million partially created runs that are being canceled. No GitHub Pages deployments completed during the incident. In the Central US data residency stamp, between 20:06 and 20:50 UTC, about 63% of webhook API requests failed and 27% of deliveries were not recorded; those deliveries reached their destinations but remain unavailable to view or redeliver. Between approximately 20:20 and 21:00 UTC, Copilot code review on github.com completed 87% fewer reviews than normal and some other Copilot features were also unavailable.<br /><br />The disruption was caused by an Azure Cosmos DB regional outage affecting database accounts used by these services. Per Cosmos's RCA, a configuration change to the Central US gateways exposed a latent race condition under a specific high-load traffic pattern, causing gateway crashes and health-probe failures. Cosmos mitigated by rolling back the change. Service health recovered around 21:02 UTC, and the queued Actions backlog finished processing by 21:21 UTC.<br /><br />Our follow-up work focuses on validating regional database failover across affected services and developing tested recovery procedures. We are also improving recovery tools to quickly clear stuck Actions checks and runs after incidents so they do not remain uncancelable or block pull request merges. Nov 11, 21:21:32 GMT+0 - Monitoring - The backlog of Actions jobs has been fully processed. We’re observing full recovery across all impacted services. Nov 11, 21:07:26 GMT+0 - Monitoring - We’re seeing recovery across all services and continuing to monitor. The backlog of Actions jobs are being processed. We’ll update when we see full recovery. Nov 11, 21:07:24 GMT+0 - Monitoring - The degradation affecting Actions, Copilot, Pages and Webhooks has been mitigated. We are monitoring to ensure stability. Nov 11, 20:57:40 GMT+0 - Investigating - Copilot is experiencing degraded availability. We are continuing to investigate. Nov 11, 20:56:49 GMT+0 - Investigating - We’re investigating issues impacting multiple GitHub products including Actions, Pages, and Copilot products. Nov 11, 20:56:26 GMT+0 - Investigating - Pages is experiencing degraded availability. We are continuing to investigate. Nov 11, 20:48:46 GMT+0 - Investigating - Webhooks is experiencing degraded performance. We are continuing to investigate. Nov 11, 20:48:25 GMT+0 - Investigating - Actions is experiencing degraded availability. We are continuing to investigate. Nov 11, 20:48:12 GMT+0 - Investigating - We are investigating reports of degraded availability for Pages </description> <content:encoded> <![CDATA[<p><strong>Type:</strong> Incident</p> <p><strong>Duration:</strong> 1 day, 9 hours and 27 minutes</p> <p><strong>Affected Components:</strong> </p> <p><small>Nov <var data-var='date'> 11</var>, <var data-var='time'>21:21:39</var> GMT+0</small><br><strong>Resolved</strong> - On November 11, 2026, from 20:05 to 21:02 UTC, customers experienced delayed or failed Actions workflow starts, webhook errors and delivery delays, slower GitHub Apps API requests, and reduced availability of Copilot features, including Code Review. GitHub Pages builds were also delayed.<br /><br />Nearly all GitHub Actions runs triggered between 20:18 and 20:55 UTC were delayed or failed, leaving approximately 3.4 million partially created runs that are being canceled. No GitHub Pages deployments completed during the incident. In the Central US data residency stamp, between 20:06 and 20:50 UTC, about 63% of webhook API requests failed and 27% of deliveries were not recorded; those deliveries reached their destinations but remain unavailable to view or redeliver. Between approximately 20:20 and 21:00 UTC, Copilot code review on github.com completed 87% fewer reviews than normal and some other Copilot features were also unavailable.<br /><br />The disruption was caused by an Azure Cosmos DB regional outage affecting database accounts used by these services. Per Cosmos's RCA, a configuration change to the Central US gateways exposed a latent race condition under a specific high-load traffic pattern, causing gateway crashes and health-probe failures. Cosmos mitigated by rolling back the change. Service health recovered around 21:02 UTC, and the queued Actions backlog finished processing by 21:21 UTC.<br /><br />Our follow-up work focuses on validating regional database failover across affected services and developing tested recovery procedures. We are also improving recovery tools to quickly clear stuck Actions checks and runs after incidents so they do not remain uncancelable or block pull request merges..</p> <p><small>Nov <var data-var='date'> 11</var>, <var data-var='time'>21:21:32</var> GMT+0</small><br><strong>Monitoring</strong> - The backlog of Actions jobs has been fully processed. We’re observing full recovery across all impacted services..</p> <p><small>Nov <var data-var='date'> 11</var>, <var data-var='time'>21:07:26</var> GMT+0</small><br><strong>Monitoring</strong> - We’re seeing recovery across all services and continuing to monitor. The backlog of Actions jobs are being processed. We’ll update when we see full recovery..</p> <p><small>Nov <var data-var='date'> 11</var>, <var data-var='time'>21:07:24</var> GMT+0</small><br><strong>Monitoring</strong> - The degradation affecting Actions, Copilot, Pages and Webhooks has been mitigated. We are monitoring to ensure stability..</p> <p><small>Nov <var data-var='date'> 11</var>, <var data-var='time'>20:57:40</var> GMT+0</small><br><strong>Investigating</strong> - Copilot is experiencing degraded availability. We are continuing to investigate..</p> <p><small>Nov <var data-var='date'> 11</var>, <var data-var='time'>20:56:49</var> GMT+0</small><br><strong>Investigating</strong> - We’re investigating issues impacting multiple GitHub products including Actions, Pages, and Copilot products..</p> <p><small>Nov <var data-var='date'> 11</var>, <var data-var='time'>20:56:26</var> GMT+0</small><br><strong>Investigating</strong> - Pages is experiencing degraded availability. We are continuing to investigate..</p> <p><small>Nov <var data-var='date'> 11</var>, <var data-var='time'>20:48:46</var> GMT+0</small><br><strong>Investigating</strong> - Webhooks is experiencing degraded performance. We are continuing to investigate..</p> <p><small>Nov <var data-var='date'> 11</var>, <var data-var='time'>20:48:25</var> GMT+0</small><br><strong>Investigating</strong> - Actions is experiencing degraded availability. We are continuing to investigate..</p> <p><small>Nov <var data-var='date'> 11</var>, <var data-var='time'>20:48:12</var> GMT+0</small><br><strong>Investigating</strong> - We are investigating reports of degraded availability for Pages.</p> ]]> </content:encoded> <pubDate>Wed, 11 Nov 2026 20:48:12 +0000</pubDate> <link>https://status.macroscope.com/incident/cmwcldnhn0qfq0knxqojzy86g</link> <guid>https://status.macroscope.com/incident/cmwcldnhn0qfq0knxqojzy86g</guid> </item> <item> <title>Incident with Git Operations and Pull Requests</title> <description> Type: Incident Duration: 4 days, 8 hours and 11 minutes Affected Components: GitHub Actions Nov 10, 17:55:03 GMT+0 - Resolved - On November 10, 2026, from approximately 15:00 UTC to 17:55 UTC, GitHub experienced degraded performance affecting repository pages, pull requests, and GitHub Actions. Customers may have seen slow or failed page loads and delays processing pull request updates and merges. Some pull requests continued to display an earlier source-branch commit after the broader service degradation ended. GitHub Actions workflows were delayed for approximately 1.8% of events, peaking at about 7% between 16:30 and 16:50 UTC. Delays were typically 1.5 to 3.5 minutes, up to 10 minutes. Approximately 0.09% of pull request events did not start their workflows and needed to be re-triggered. <br /><br />The degradation resulted from overload in the shared infrastructure that supports Git operations. Recent changes increased background processing and the work required to compute pull request merges, adding pressure to the same systems used by customer-facing requests. We mitigated the incident by disabling a recent merge-processing optimization, pausing background analysis, and temporarily limiting lower-priority traffic. The temporary limits also restricted access to commit pages for users who were not signed in. Background processing continued to catch up after service health recovered. <br /><br />We have reduced repeated repository-analysis work and are adding workload-level concurrency controls to better protect customer-facing requests. We are also improving our procedures for safely reducing background load and making pull request synchronization more resilient to temporary failures. Nov 10, 17:46:33 GMT+0 - Investigating - We continue to investigate elevated response times for Git operations and pull requests. Actions workflows are recovering. Pull request processing has shown some improvement, but users continue to experience delays and failures. We’re monitoring the impact and working to restore normal service. Nov 10, 17:28:17 GMT+0 - Investigating - Actions is operating normally. Nov 10, 17:12:54 GMT+0 - Investigating - Pull Requests is experiencing degraded performance. We are continuing to investigate. Nov 10, 17:07:48 GMT+0 - Investigating - We continue to investigate elevated response times for Git operations and pull requests. Some pull-request processing and Actions workflows are improving, but delays and failures persist. We're monitoring the impact and working to restore normal service. Nov 10, 16:39:58 GMT+0 - Investigating - Actions is experiencing degraded availability. We are continuing to investigate. Nov 10, 16:31:01 GMT+0 - Investigating - We're investigating elevated response times for Git operations and pull requests. GitHub Actions workflows that are triggered from Pull Requests are delayed. We're working to mitigate the impact and will share updates as we learn more. Nov 10, 16:26:37 GMT+0 - Investigating - Actions and Pull Requests are experiencing degraded availability. We are continuing to investigate. Nov 10, 16:10:52 GMT+0 - Investigating - We are investigating reports of degraded performance for Actions and Pull Requests </description> <content:encoded> <![CDATA[<p><strong>Type:</strong> Incident</p> <p><strong>Duration:</strong> 4 days, 8 hours and 11 minutes</p> <p><strong>Affected Components:</strong> </p> <p><small>Nov <var data-var='date'> 10</var>, <var data-var='time'>17:55:03</var> GMT+0</small><br><strong>Resolved</strong> - On November 10, 2026, from approximately 15:00 UTC to 17:55 UTC, GitHub experienced degraded performance affecting repository pages, pull requests, and GitHub Actions. Customers may have seen slow or failed page loads and delays processing pull request updates and merges. Some pull requests continued to display an earlier source-branch commit after the broader service degradation ended. GitHub Actions workflows were delayed for approximately 1.8% of events, peaking at about 7% between 16:30 and 16:50 UTC. Delays were typically 1.5 to 3.5 minutes, up to 10 minutes. Approximately 0.09% of pull request events did not start their workflows and needed to be re-triggered. <br /><br />The degradation resulted from overload in the shared infrastructure that supports Git operations. Recent changes increased background processing and the work required to compute pull request merges, adding pressure to the same systems used by customer-facing requests. We mitigated the incident by disabling a recent merge-processing optimization, pausing background analysis, and temporarily limiting lower-priority traffic. The temporary limits also restricted access to commit pages for users who were not signed in. Background processing continued to catch up after service health recovered. <br /><br />We have reduced repeated repository-analysis work and are adding workload-level concurrency controls to better protect customer-facing requests. We are also improving our procedures for safely reducing background load and making pull request synchronization more resilient to temporary failures..</p> <p><small>Nov <var data-var='date'> 10</var>, <var data-var='time'>17:46:33</var> GMT+0</small><br><strong>Investigating</strong> - We continue to investigate elevated response times for Git operations and pull requests. Actions workflows are recovering. Pull request processing has shown some improvement, but users continue to experience delays and failures. We’re monitoring the impact and working to restore normal service..</p> <p><small>Nov <var data-var='date'> 10</var>, <var data-var='time'>17:28:17</var> GMT+0</small><br><strong>Investigating</strong> - Actions is operating normally..</p> <p><small>Nov <var data-var='date'> 10</var>, <var data-var='time'>17:12:54</var> GMT+0</small><br><strong>Investigating</strong> - Pull Requests is experiencing degraded performance. We are continuing to investigate..</p> <p><small>Nov <var data-var='date'> 10</var>, <var data-var='time'>17:07:48</var> GMT+0</small><br><strong>Investigating</strong> - We continue to investigate elevated response times for Git operations and pull requests. Some pull-request processing and Actions workflows are improving, but delays and failures persist. We're monitoring the impact and working to restore normal service..</p> <p><small>Nov <var data-var='date'> 10</var>, <var data-var='time'>16:39:58</var> GMT+0</small><br><strong>Investigating</strong> - Actions is experiencing degraded availability. We are continuing to investigate..</p> <p><small>Nov <var data-var='date'> 10</var>, <var data-var='time'>16:31:01</var> GMT+0</small><br><strong>Investigating</strong> - We're investigating elevated response times for Git operations and pull requests. GitHub Actions workflows that are triggered from Pull Requests are delayed. We're working to mitigate the impact and will share updates as we learn more..</p> <p><small>Nov <var data-var='date'> 10</var>, <var data-var='time'>16:26:37</var> GMT+0</small><br><strong>Investigating</strong> - Actions and Pull Requests are experiencing degraded availability. We are continuing to investigate..</p> <p><small>Nov <var data-var='date'> 10</var>, <var data-var='time'>16:10:52</var> GMT+0</small><br><strong>Investigating</strong> - We are investigating reports of degraded performance for Actions and Pull Requests.</p> ]]> </content:encoded> <pubDate>Tue, 10 Nov 2026 16:10:52 +0000</pubDate> <link>https://status.macroscope.com/incident/cmwaw44t427qi0rrhfl3dkorz</link> <guid>https://status.macroscope.com/incident/cmwaw44t427qi0rrhfl3dkorz</guid> </item> <item> <title>Incident with Actions and Checks API</title> <description> Type: Incident Duration: 1 day, 22 hours and 55 minutes Affected Components: GitHub Actions Nov 9, 18:59:32 GMT+0 - Resolved - On November 9, 2026, between 18:07 and 18:59 UTC, GitHub Actions experienced degraded performance. During this time, some customers saw workflow runs fail to start, update or report status correctly. An average of 28% of workflow runs were affected, peaking at 65% during the worst period of the incident. Once Actions services recovered, some jobs were left in a state that limited available concurrency until the runs were released. <br /><br />The incident occurred during a planned migration of a core Actions database from GitHub’s datacenter to a new cloud region. After the migration completed, we shifted Actions service traffic to the new region. The volume of requests exceeded the capacity and rate limits configured there, so a portion of requests were rejected and workflow updates failed. Dependent services, including Webhooks, Pages and Copilot code review, saw secondary impact. <br /><br />We mitigated impact by shifting some of the traffic back to the original region, allowing service traffic to remain within the configured limits. We then increased capacity and rate limits in the new region before gradually shifting traffic back. While this was a one-time migration event for this cluster that is now complete, we are improving how we validate capacity and limits before shifting traffic during remaining migrations, including shifting traffic in smaller steps. To address the stuck runs left after the incident, we are improving our ability to detect and mitigate these runs for future cases as well as readying an API to enable customers to address this themselves. Nov 9, 18:43:24 GMT+0 - Monitoring - The degradation affecting Actions, Pages and Webhooks has been mitigated. We are monitoring to ensure stability. Nov 9, 18:30:17 GMT+0 - Investigating - Pages is experiencing degraded performance. We are continuing to investigate. Nov 9, 18:24:13 GMT+0 - Investigating - We're investigating elevated errors affecting GitHub Actions and the Checks API. We'll share another update as we learn more. Nov 9, 18:23:13 GMT+0 - Investigating - Webhooks is experiencing degraded performance. We are continuing to investigate. Nov 9, 18:14:58 GMT+0 - Investigating - Actions is experiencing degraded availability. We are continuing to investigate. Nov 9, 18:12:37 GMT+0 - Investigating - We are investigating reports of degraded availability for Actions </description> <content:encoded> <![CDATA[<p><strong>Type:</strong> Incident</p> <p><strong>Duration:</strong> 1 day, 22 hours and 55 minutes</p> <p><strong>Affected Components:</strong> </p> <p><small>Nov <var data-var='date'> 9</var>, <var data-var='time'>18:59:32</var> GMT+0</small><br><strong>Resolved</strong> - On November 9, 2026, between 18:07 and 18:59 UTC, GitHub Actions experienced degraded performance. During this time, some customers saw workflow runs fail to start, update or report status correctly. An average of 28% of workflow runs were affected, peaking at 65% during the worst period of the incident. Once Actions services recovered, some jobs were left in a state that limited available concurrency until the runs were released. <br /><br />The incident occurred during a planned migration of a core Actions database from GitHub’s datacenter to a new cloud region. After the migration completed, we shifted Actions service traffic to the new region. The volume of requests exceeded the capacity and rate limits configured there, so a portion of requests were rejected and workflow updates failed. Dependent services, including Webhooks, Pages and Copilot code review, saw secondary impact. <br /><br />We mitigated impact by shifting some of the traffic back to the original region, allowing service traffic to remain within the configured limits. We then increased capacity and rate limits in the new region before gradually shifting traffic back. While this was a one-time migration event for this cluster that is now complete, we are improving how we validate capacity and limits before shifting traffic during remaining migrations, including shifting traffic in smaller steps. To address the stuck runs left after the incident, we are improving our ability to detect and mitigate these runs for future cases as well as readying an API to enable customers to address this themselves..</p> <p><small>Nov <var data-var='date'> 9</var>, <var data-var='time'>18:43:24</var> GMT+0</small><br><strong>Monitoring</strong> - The degradation affecting Actions, Pages and Webhooks has been mitigated. We are monitoring to ensure stability..</p> <p><small>Nov <var data-var='date'> 9</var>, <var data-var='time'>18:30:17</var> GMT+0</small><br><strong>Investigating</strong> - Pages is experiencing degraded performance. We are continuing to investigate..</p> <p><small>Nov <var data-var='date'> 9</var>, <var data-var='time'>18:24:13</var> GMT+0</small><br><strong>Investigating</strong> - We're investigating elevated errors affecting GitHub Actions and the Checks API. We'll share another update as we learn more..</p> <p><small>Nov <var data-var='date'> 9</var>, <var data-var='time'>18:23:13</var> GMT+0</small><br><strong>Investigating</strong> - Webhooks is experiencing degraded performance. We are continuing to investigate..</p> <p><small>Nov <var data-var='date'> 9</var>, <var data-var='time'>18:14:58</var> GMT+0</small><br><strong>Investigating</strong> - Actions is experiencing degraded availability. We are continuing to investigate..</p> <p><small>Nov <var data-var='date'> 9</var>, <var data-var='time'>18:12:37</var> GMT+0</small><br><strong>Investigating</strong> - We are investigating reports of degraded availability for Actions.</p> ]]> </content:encoded> <pubDate>Mon, 9 Nov 2026 18:12:37 +0000</pubDate> <link>https://status.macroscope.com/incident/cmw9kym17022r0rrh5in39n3q</link> <guid>https://status.macroscope.com/incident/cmw9kym17022r0rrh5in39n3q</guid> </item> <item> <title>Incident with Webhooks and Actions</title> <description> Type: Incident Duration: 4 days and 14 hours Affected Components: GitHub Actions Nov 9, 11:50:35 GMT+0 - Resolved - On November 9, 2026, from 09:23 to 11:40 UTC, one of our job queue clusters was degraded, delaying webhook deliveries, Actions workflow runs, and Dependabot and dependency graph updates. Until mitigation at 10:37 UTC, an average of 26% of webhook deliveries were delayed, peaking at 30%, and the backlog cleared by 11:40 UTC. Some deliveries were delayed by up to 3.5 hours, and delays for a small share began as early as 01:00 UTC. Actions skips events older than 30 minutes, so some workflow runs did not start. <br /><br />The incident was caused by a degraded host that had slowed one of the cluster's Kafka brokers since November 7. The broker stopped reporting the metrics our alerts rely on, so we did not identify it until it stalled the cluster on November 9. <br /><br />We mitigated the incident by removing the broker from service. We have since added an alert that pages us within 30 minutes when a host slows a broker this way, so we can remove the broker before it affects customers. Nov 9, 11:45:07 GMT+0 - Monitoring - The degradation affecting Webhooks has been mitigated. We are monitoring to ensure stability. Nov 9, 11:42:04 GMT+0 - Investigating - Webhooks is experiencing degraded performance. We are continuing to investigate. Nov 9, 11:17:28 GMT+0 - Investigating - Webhooks is experiencing degraded availability. We are continuing to investigate. Nov 9, 11:06:35 GMT+0 - Investigating - The degradation affecting Actions has been mitigated. We are monitoring to ensure stability. Nov 9, 10:47:25 GMT+0 - Investigating - We've applied a mitigation and are monitoring for recovery. Nov 9, 10:28:19 GMT+0 - Investigating - We're investigating delays in Actions runs start times and webhook deliveries. Nov 9, 10:00:35 GMT+0 - Investigating - We are investigating reports of degraded performance for Actions and Webhooks </description> <content:encoded> <![CDATA[<p><strong>Type:</strong> Incident</p> <p><strong>Duration:</strong> 4 days and 14 hours</p> <p><strong>Affected Components:</strong> </p> <p><small>Nov <var data-var='date'> 9</var>, <var data-var='time'>11:50:35</var> GMT+0</small><br><strong>Resolved</strong> - On November 9, 2026, from 09:23 to 11:40 UTC, one of our job queue clusters was degraded, delaying webhook deliveries, Actions workflow runs, and Dependabot and dependency graph updates. Until mitigation at 10:37 UTC, an average of 26% of webhook deliveries were delayed, peaking at 30%, and the backlog cleared by 11:40 UTC. Some deliveries were delayed by up to 3.5 hours, and delays for a small share began as early as 01:00 UTC. Actions skips events older than 30 minutes, so some workflow runs did not start. <br /><br />The incident was caused by a degraded host that had slowed one of the cluster's Kafka brokers since November 7. The broker stopped reporting the metrics our alerts rely on, so we did not identify it until it stalled the cluster on November 9. <br /><br />We mitigated the incident by removing the broker from service. We have since added an alert that pages us within 30 minutes when a host slows a broker this way, so we can remove the broker before it affects customers..</p> <p><small>Nov <var data-var='date'> 9</var>, <var data-var='time'>11:45:07</var> GMT+0</small><br><strong>Monitoring</strong> - The degradation affecting Webhooks has been mitigated. We are monitoring to ensure stability..</p> <p><small>Nov <var data-var='date'> 9</var>, <var data-var='time'>11:42:04</var> GMT+0</small><br><strong>Investigating</strong> - Webhooks is experiencing degraded performance. We are continuing to investigate..</p> <p><small>Nov <var data-var='date'> 9</var>, <var data-var='time'>11:17:28</var> GMT+0</small><br><strong>Investigating</strong> - Webhooks is experiencing degraded availability. We are continuing to investigate..</p> <p><small>Nov <var data-var='date'> 9</var>, <var data-var='time'>11:06:35</var> GMT+0</small><br><strong>Investigating</strong> - The degradation affecting Actions has been mitigated. We are monitoring to ensure stability..</p> <p><small>Nov <var data-var='date'> 9</var>, <var data-var='time'>10:47:25</var> GMT+0</small><br><strong>Investigating</strong> - We've applied a mitigation and are monitoring for recovery..</p> <p><small>Nov <var data-var='date'> 9</var>, <var data-var='time'>10:28:19</var> GMT+0</small><br><strong>Investigating</strong> - We're investigating delays in Actions runs start times and webhook deliveries..</p> <p><small>Nov <var data-var='date'> 9</var>, <var data-var='time'>10:00:35</var> GMT+0</small><br><strong>Investigating</strong> - We are investigating reports of degraded performance for Actions and Webhooks.</p> ]]> </content:encoded> <pubDate>Mon, 9 Nov 2026 10:00:35 +0000</pubDate> <link>https://status.macroscope.com/incident/cmw93gh8q1pp30kldxrx2tuyk</link> <guid>https://status.macroscope.com/incident/cmw93gh8q1pp30kldxrx2tuyk</guid> </item> <item> <title>Increase in timeouts for some unauthenticated users</title> <description> Type: Incident Duration: 4 days, 20 hours and 30 minutes Affected Components: GitHub API Requests Nov 8, 12:32:35 GMT+0 - Resolved - On November 8, 2026, between 08:43 and 12:29 UTC, unauthenticated requests to github.com and the GitHub REST API experienced elevated error rates. During this period, 11.3% of unauthenticated requests and 46.1% of release asset download requests failed. Authenticated requests were largely unaffected.<br /><br />Initial errors occurred in a backend service handling unauthenticated web traffic. A subsequent traffic spike caused high CPU utilization on servers in our caching layer. Some servers remained unhealthy after the spike ended, prolonging the disruption. We restored service by restarting caching software and rebooting affected servers.<br /><br />We are improving monitoring to detect these failures sooner, investigating why affected servers did not recover automatically, and reviewing caching behavior to better protect backend services during failures. Nov 8, 12:29:30 GMT+0 - Monitoring - The degradation has been mitigated. We are monitoring to ensure stability. Nov 8, 11:56:20 GMT+0 - Investigating - The degradation affecting API Requests has been mitigated. We are monitoring to ensure stability. Nov 8, 11:50:49 GMT+0 - Investigating - We've identified a problematic component and are working to redeploy it. Nov 8, 11:20:57 GMT+0 - Investigating - API Requests is experiencing degraded performance. We are continuing to investigate. Nov 8, 11:16:19 GMT+0 - Investigating - We are continuing to investigate degraded performance primarily for unauthenticated users. </description> <content:encoded> <![CDATA[<p><strong>Type:</strong> Incident</p> <p><strong>Duration:</strong> 4 days, 20 hours and 30 minutes</p> <p><strong>Affected Components:</strong> </p> <p><small>Nov <var data-var='date'> 8</var>, <var data-var='time'>12:32:35</var> GMT+0</small><br><strong>Resolved</strong> - On November 8, 2026, between 08:43 and 12:29 UTC, unauthenticated requests to github.com and the GitHub REST API experienced elevated error rates. During this period, 11.3% of unauthenticated requests and 46.1% of release asset download requests failed. Authenticated requests were largely unaffected.<br /><br />Initial errors occurred in a backend service handling unauthenticated web traffic. A subsequent traffic spike caused high CPU utilization on servers in our caching layer. Some servers remained unhealthy after the spike ended, prolonging the disruption. We restored service by restarting caching software and rebooting affected servers.<br /><br />We are improving monitoring to detect these failures sooner, investigating why affected servers did not recover automatically, and reviewing caching behavior to better protect backend services during failures..</p> <p><small>Nov <var data-var='date'> 8</var>, <var data-var='time'>12:29:30</var> GMT+0</small><br><strong>Monitoring</strong> - The degradation has been mitigated. We are monitoring to ensure stability..</p> <p><small>Nov <var data-var='date'> 8</var>, <var data-var='time'>11:56:20</var> GMT+0</small><br><strong>Investigating</strong> - The degradation affecting API Requests has been mitigated. We are monitoring to ensure stability..</p> <p><small>Nov <var data-var='date'> 8</var>, <var data-var='time'>11:50:49</var> GMT+0</small><br><strong>Investigating</strong> - We've identified a problematic component and are working to redeploy it..</p> <p><small>Nov <var data-var='date'> 8</var>, <var data-var='time'>11:20:57</var> GMT+0</small><br><strong>Investigating</strong> - API Requests is experiencing degraded performance. We are continuing to investigate..</p> <p><small>Nov <var data-var='date'> 8</var>, <var data-var='time'>11:16:19</var> GMT+0</small><br><strong>Investigating</strong> - We are continuing to investigate degraded performance primarily for unauthenticated users..</p> ]]> </content:encoded> <pubDate>Sun, 8 Nov 2026 10:36:05 +0000</pubDate> <link>https://status.macroscope.com/incident/cmw7qvhl46naf0kp3d9qngkgj</link> <guid>https://status.macroscope.com/incident/cmw7qvhl46naf0kp3d9qngkgj</guid> </item> <item> <title>Degraded performance for multiple GitHub services</title> <description> Type: Incident Duration: 1 day, 20 hours and 43 minutes Affected Components: GitHub Actions Nov 6, 00:20:49 GMT+0 - Resolved - On November 5, 2026, between 23:21 and approximately 23:26 UTC, several GitHub services experienced elevated errors, including Git operations, Issues, Actions, authentication, and API requests. Customers may have encountered failed requests and Git pushes, and some Actions workflow runs were delayed. <br /><br />This incident was caused by a bug in our database failover automation during planned maintenance. The automation stopped before completing the switch to a replacement database primary, leaving the existing primary unable to accept writes. We mitigated the disruption by restoring writes on the original primary. We subsequently completed the maintenance and monitored services to confirm recovery. <br /><br />We are fixing the automation bug and strengthening our pre-maintenance checks and recovery procedures to prevent similar disruptions and restore service more quickly if a failure occurs. Nov 6, 00:19:59 GMT+0 - Monitoring - This incident has been resolved. All affected services, including Git operations, Issues, Actions, and API access, have recovered and remained stable. Thank you for your patience. We'll share more details in our availability report. Nov 6, 00:19:21 GMT+0 - Investigating - Git Operations is operating normally. Nov 6, 00:06:28 GMT+0 - Monitoring - We've applied a mitigation and are seeing recovery across affected services, including Git operations, Issues, Actions, and API access. We're monitoring to confirm stability and will post another update by 00:35 UTC. Nov 6, 00:06:11 GMT+0 - Monitoring - The degradation affecting Actions, Issues and Pages has been mitigated. We are monitoring to ensure stability. Nov 5, 23:58:28 GMT+0 - Investigating - Issues is experiencing degraded performance. We are continuing to investigate. Nov 5, 23:36:51 GMT+0 - Investigating - We're investigating degraded performance and errors affecting several GitHub services, including Git operations, Issues, Actions, and API access. Some users may see failed requests or slow responses. We'll post another update by 00:05 UTC. Nov 5, 23:36:06 GMT+0 - Investigating - We are investigating reports of degraded performance for Actions and Pages </description> <content:encoded> <![CDATA[<p><strong>Type:</strong> Incident</p> <p><strong>Duration:</strong> 1 day, 20 hours and 43 minutes</p> <p><strong>Affected Components:</strong> </p> <p><small>Nov <var data-var='date'> 6</var>, <var data-var='time'>00:20:49</var> GMT+0</small><br><strong>Resolved</strong> - On November 5, 2026, between 23:21 and approximately 23:26 UTC, several GitHub services experienced elevated errors, including Git operations, Issues, Actions, authentication, and API requests. Customers may have encountered failed requests and Git pushes, and some Actions workflow runs were delayed. <br /><br />This incident was caused by a bug in our database failover automation during planned maintenance. The automation stopped before completing the switch to a replacement database primary, leaving the existing primary unable to accept writes. We mitigated the disruption by restoring writes on the original primary. We subsequently completed the maintenance and monitored services to confirm recovery. <br /><br />We are fixing the automation bug and strengthening our pre-maintenance checks and recovery procedures to prevent similar disruptions and restore service more quickly if a failure occurs..</p> <p><small>Nov <var data-var='date'> 6</var>, <var data-var='time'>00:19:59</var> GMT+0</small><br><strong>Monitoring</strong> - This incident has been resolved. All affected services, including Git operations, Issues, Actions, and API access, have recovered and remained stable. Thank you for your patience. We'll share more details in our availability report..</p> <p><small>Nov <var data-var='date'> 6</var>, <var data-var='time'>00:19:21</var> GMT+0</small><br><strong>Investigating</strong> - Git Operations is operating normally..</p> <p><small>Nov <var data-var='date'> 6</var>, <var data-var='time'>00:06:28</var> GMT+0</small><br><strong>Monitoring</strong> - We've applied a mitigation and are seeing recovery across affected services, including Git operations, Issues, Actions, and API access. We're monitoring to confirm stability and will post another update by 00:35 UTC..</p> <p><small>Nov <var data-var='date'> 6</var>, <var data-var='time'>00:06:11</var> GMT+0</small><br><strong>Monitoring</strong> - The degradation affecting Actions, Issues and Pages has been mitigated. We are monitoring to ensure stability..</p> <p><small>Nov <var data-var='date'> 5</var>, <var data-var='time'>23:58:28</var> GMT+0</small><br><strong>Investigating</strong> - Issues is experiencing degraded performance. We are continuing to investigate..</p> <p><small>Nov <var data-var='date'> 5</var>, <var data-var='time'>23:36:51</var> GMT+0</small><br><strong>Investigating</strong> - We're investigating degraded performance and errors affecting several GitHub services, including Git operations, Issues, Actions, and API access. Some users may see failed requests or slow responses. We'll post another update by 00:05 UTC..</p> <p><small>Nov <var data-var='date'> 5</var>, <var data-var='time'>23:36:06</var> GMT+0</small><br><strong>Investigating</strong> - We are investigating reports of degraded performance for Actions and Pages.</p> ]]> </content:encoded> <pubDate>Thu, 5 Nov 2026 23:36:06 +0000</pubDate> <link>https://status.macroscope.com/incident/cmw46t5oz0ioc0rp3opum1p16</link> <guid>https://status.macroscope.com/incident/cmw46t5oz0ioc0rp3opum1p16</guid> </item> <item> <title>Incident with Actions</title> <description> Type: Incident Duration: 2 days, 4 hours and 24 minutes Affected Components: GitHub Actions Nov 5, 17:42:35 GMT+0 - Resolved - This incident has been resolved. Thank you for your patience and understanding as we addressed this issue. A detailed root cause analysis will be shared as soon as it is available. Nov 5, 17:04:46 GMT+0 - Monitoring - The degradation affecting Actions has been mitigated. We are monitoring to ensure stability. Nov 5, 16:52:52 GMT+0 - Investigating - Some GitHub Actions workflow runs may experience delays or errors when starting. We’re investigating and preparing a mitigation. Nov 5, 16:50:11 GMT+0 - Investigating - We are investigating reports of degraded performance for Actions </description> <content:encoded> <![CDATA[<p><strong>Type:</strong> Incident</p> <p><strong>Duration:</strong> 2 days, 4 hours and 24 minutes</p> <p><strong>Affected Components:</strong> </p> <p><small>Nov <var data-var='date'> 5</var>, <var data-var='time'>17:42:35</var> GMT+0</small><br><strong>Resolved</strong> - This incident has been resolved. Thank you for your patience and understanding as we addressed this issue. A detailed root cause analysis will be shared as soon as it is available..</p> <p><small>Nov <var data-var='date'> 5</var>, <var data-var='time'>17:04:46</var> GMT+0</small><br><strong>Monitoring</strong> - The degradation affecting Actions has been mitigated. We are monitoring to ensure stability..</p> <p><small>Nov <var data-var='date'> 5</var>, <var data-var='time'>16:52:52</var> GMT+0</small><br><strong>Investigating</strong> - Some GitHub Actions workflow runs may experience delays or errors when starting. We’re investigating and preparing a mitigation..</p> <p><small>Nov <var data-var='date'> 5</var>, <var data-var='time'>16:50:11</var> GMT+0</small><br><strong>Investigating</strong> - We are investigating reports of degraded performance for Actions.</p> ]]> </content:encoded> <pubDate>Thu, 5 Nov 2026 16:50:11 +0000</pubDate> <link>https://status.macroscope.com/incident/cmw3scbj500lr0ro0ifo7jeqo</link> <guid>https://status.macroscope.com/incident/cmw3scbj500lr0ro0ifo7jeqo</guid> </item> <item> <title>Incident with Actions</title> <description> Type: Incident Duration: 1 day, 11 hours and 34 minutes Affected Components: GitHub Actions Nov 4, 18:16:08 GMT+0 - Resolved - Please refer to the combined summary in this related incident: https://www.githubstatus.com/incidents/xr3myzj7ms51 Nov 4, 18:07:51 GMT+0 - Monitoring - The degradation affecting Actions has been mitigated. We are monitoring to ensure stability. Nov 4, 17:53:11 GMT+0 - Investigating - We have identified an issue with events that might trigger Actions. We have applied mitigations and are monitoring for recovery at this point. Customers may observe delayed status updates for their jobs. Nov 4, 17:40:33 GMT+0 - Investigating - We are investigating reports of degraded availability for Actions </description> <content:encoded> <![CDATA[<p><strong>Type:</strong> Incident</p> <p><strong>Duration:</strong> 1 day, 11 hours and 34 minutes</p> <p><strong>Affected Components:</strong> </p> <p><small>Nov <var data-var='date'> 4</var>, <var data-var='time'>18:16:08</var> GMT+0</small><br><strong>Resolved</strong> - Please refer to the combined summary in this related incident: https://www.githubstatus.com/incidents/xr3myzj7ms51.</p> <p><small>Nov <var data-var='date'> 4</var>, <var data-var='time'>18:07:51</var> GMT+0</small><br><strong>Monitoring</strong> - The degradation affecting Actions has been mitigated. We are monitoring to ensure stability..</p> <p><small>Nov <var data-var='date'> 4</var>, <var data-var='time'>17:53:11</var> GMT+0</small><br><strong>Investigating</strong> - We have identified an issue with events that might trigger Actions. We have applied mitigations and are monitoring for recovery at this point. Customers may observe delayed status updates for their jobs..</p> <p><small>Nov <var data-var='date'> 4</var>, <var data-var='time'>17:40:33</var> GMT+0</small><br><strong>Investigating</strong> - We are investigating reports of degraded availability for Actions.</p> ]]> </content:encoded> <pubDate>Wed, 4 Nov 2026 17:40:33 +0000</pubDate> <link>https://status.macroscope.com/incident/cmw2eorn805fx0rqeww7bsw09</link> <guid>https://status.macroscope.com/incident/cmw2eorn805fx0rqeww7bsw09</guid> </item> <item> <title>Incident with Actions and Pages</title> <description> Type: Incident Duration: 2 days, 7 hours and 39 minutes Affected Components: GitHub Actions Nov 4, 17:01:11 GMT+0 - Resolved - On November 4, 2026, between 15:49 UTC and 16:48 UTC, and again between 17:20 UTC and 17:55 UTC, GitHub Actions, Pages, Webhooks, Code Scanning, and Copilot coding agent experienced delays. During this time, Actions workflow runs and job status updates were delayed by up to 25 minutes. Pages deployments dropped to about 4% of normal volume, webhook deliveries dropped to about 58% of normal volume, and Code Scanning analyses and Copilot coding agent sessions were delayed. Delayed events were processed after mitigation. <br /><br />This incident was caused by a configuration change to an internal event-routing service that unexpectedly paused delivery of several event streams. The change depended on feature flags that were not set up as intended, which the service did not handle gracefully. We mitigated the first period by rolling back the configuration change at 16:14 UTC. The code change itself was not reverted by the rollback, so a subsequent deployment reintroduced it. We mitigated the second period at 17:40 UTC by rolling back again, then processed queued events and cleared backlogs. <br /><br />We are taking several steps to prevent a repeat. The routing service will keep its existing behavior when a feature toggle is missing, so a misconfigured flag can no longer pause delivery. We are adding a pre-deploy check that each routing flag exists and is registered under the correct owning service, and we are making flag ownership less error-prone to set. We are also improving alerting so we detect paused event delivery sooner. Nov 4, 16:49:27 GMT+0 - Monitoring - The degradation has been mitigated. We are monitoring to ensure stability. Nov 4, 16:48:03 GMT+0 - Monitoring - The degradation affecting Actions and Pages has been mitigated. We are monitoring to ensure stability. Nov 4, 16:45:48 GMT+0 - Investigating - The degradation affecting Webhooks has been mitigated. We are monitoring to ensure stability. Nov 4, 16:34:45 GMT+0 - Investigating - We have identified an issue with events that might trigger Actions. We have applied mitigations and are monitoring for recovery at this point. Customers may observe delayed status updates for their jobs. Nov 4, 16:17:29 GMT+0 - Investigating - Actions is experiencing degraded availability. We are continuing to investigate. Nov 4, 16:12:55 GMT+0 - Investigating - Webhooks is experiencing degraded performance. We are continuing to investigate. Nov 4, 16:11:38 GMT+0 - Investigating - We are investigating reports of issues with Actions and Pages. We will continue to keep users updated on progress towards mitigation. Nov 4, 16:05:31 GMT+0 - Investigating - We are investigating reports of degraded performance for Actions and Pages </description> <content:encoded> <![CDATA[<p><strong>Type:</strong> Incident</p> <p><strong>Duration:</strong> 2 days, 7 hours and 39 minutes</p> <p><strong>Affected Components:</strong> </p> <p><small>Nov <var data-var='date'> 4</var>, <var data-var='time'>17:01:11</var> GMT+0</small><br><strong>Resolved</strong> - On November 4, 2026, between 15:49 UTC and 16:48 UTC, and again between 17:20 UTC and 17:55 UTC, GitHub Actions, Pages, Webhooks, Code Scanning, and Copilot coding agent experienced delays. During this time, Actions workflow runs and job status updates were delayed by up to 25 minutes. Pages deployments dropped to about 4% of normal volume, webhook deliveries dropped to about 58% of normal volume, and Code Scanning analyses and Copilot coding agent sessions were delayed. Delayed events were processed after mitigation. <br /><br />This incident was caused by a configuration change to an internal event-routing service that unexpectedly paused delivery of several event streams. The change depended on feature flags that were not set up as intended, which the service did not handle gracefully. We mitigated the first period by rolling back the configuration change at 16:14 UTC. The code change itself was not reverted by the rollback, so a subsequent deployment reintroduced it. We mitigated the second period at 17:40 UTC by rolling back again, then processed queued events and cleared backlogs. <br /><br />We are taking several steps to prevent a repeat. The routing service will keep its existing behavior when a feature toggle is missing, so a misconfigured flag can no longer pause delivery. We are adding a pre-deploy check that each routing flag exists and is registered under the correct owning service, and we are making flag ownership less error-prone to set. We are also improving alerting so we detect paused event delivery sooner..</p> <p><small>Nov <var data-var='date'> 4</var>, <var data-var='time'>16:49:27</var> GMT+0</small><br><strong>Monitoring</strong> - The degradation has been mitigated. We are monitoring to ensure stability..</p> <p><small>Nov <var data-var='date'> 4</var>, <var data-var='time'>16:48:03</var> GMT+0</small><br><strong>Monitoring</strong> - The degradation affecting Actions and Pages has been mitigated. We are monitoring to ensure stability..</p> <p><small>Nov <var data-var='date'> 4</var>, <var data-var='time'>16:45:48</var> GMT+0</small><br><strong>Investigating</strong> - The degradation affecting Webhooks has been mitigated. We are monitoring to ensure stability..</p> <p><small>Nov <var data-var='date'> 4</var>, <var data-var='time'>16:34:45</var> GMT+0</small><br><strong>Investigating</strong> - We have identified an issue with events that might trigger Actions. We have applied mitigations and are monitoring for recovery at this point. Customers may observe delayed status updates for their jobs..</p> <p><small>Nov <var data-var='date'> 4</var>, <var data-var='time'>16:17:29</var> GMT+0</small><br><strong>Investigating</strong> - Actions is experiencing degraded availability. We are continuing to investigate..</p> <p><small>Nov <var data-var='date'> 4</var>, <var data-var='time'>16:12:55</var> GMT+0</small><br><strong>Investigating</strong> - Webhooks is experiencing degraded performance. We are continuing to investigate..</p> <p><small>Nov <var data-var='date'> 4</var>, <var data-var='time'>16:11:38</var> GMT+0</small><br><strong>Investigating</strong> - We are investigating reports of issues with Actions and Pages. We will continue to keep users updated on progress towards mitigation..</p> <p><small>Nov <var data-var='date'> 4</var>, <var data-var='time'>16:05:31</var> GMT+0</small><br><strong>Investigating</strong> - We are investigating reports of degraded performance for Actions and Pages.</p> ]]> </content:encoded> <pubDate>Wed, 4 Nov 2026 16:05:31 +0000</pubDate> <link>https://status.macroscope.com/incident/cmw2bap1s01f80rqejd98ktqs</link> <guid>https://status.macroscope.com/incident/cmw2bap1s01f80rqejd98ktqs</guid> </item> <item> <title>Incident with Pull Requests, Git Operations and Issues</title> <description> Type: Incident Duration: 5 days and 10 minutes Affected Components: GitHub Actions, GitHub API Requests Nov 4, 11:50:06 GMT+0 - Resolved - n November 4, 2026, between 09:35 UTC and 11:43 UTC, customers experienced intermittent errors and timeouts across GitHub.com, primarily on actions that write data. At peak, about 20% of pull request write operations (such as merging, commenting, and reviewing) failed, along with about 10% of issue creates and edits and 10–15% of project item edits. Git pushes over SSH, REST and GraphQL API requests, code scanning uploads, releases, and package operations were also affected, and about 1% of Actions runs were delayed by more than five minutes or failed to start. Overall, nearly 4% of all requests to GitHub.com failed at the worst point. Reads were largely unaffected, and retrying a failed action a short time later usually succeeded.<br /><br />The incident was triggered by a burst of concurrent updates to the same database record, which tied up all available connections to one of our primary databases. Although that contention cleared, the backlog of waiting requests kept our database connection layer overloaded, and it did not recover on its own. We mitigated the incident by failing over to a new primary database host. Most services recovered by 11:31 UTC, and Actions recovered by 11:43 UTC.<br /><br />Since the incident, we have disabled the code path that triggered the contention, increased the processing capacity of our database connection layer by 50%, and added alerting to detect this type of overload earlier. We are also improving our procedures so we can fail over sooner in similar situations. Nov 4, 11:43:52 GMT+0 - Monitoring - The degradation affecting Actions has been mitigated. We are monitoring to ensure stability. Nov 4, 11:30:42 GMT+0 - Investigating - The degradation affecting API Requests, Git Operations, Issues and Pull Requests has been mitigated. We are monitoring to ensure stability. Nov 4, 11:17:52 GMT+0 - Investigating - We're currently working on mitigating elevated errors and intermittent failures affecting GitHub, including Pull Requests, Git Operations, and Issues. Some requests may fail or be delayed while we work to restore service. We'll provide another update as more information becomes available. Nov 4, 11:11:30 GMT+0 - Investigating - API Requests is experiencing degraded performance. We are continuing to investigate. Nov 4, 10:38:49 GMT+0 - Investigating - We're investigating elevated errors and intermittent failures affecting GitHub including Pull Requests, Git Operations, and Issues. We'll continue to keep users updated as we work toward mitigation. Nov 4, 09:53:38 GMT+0 - Investigating - Pull Requests is experiencing degraded availability. We are continuing to investigate. Nov 4, 09:50:47 GMT+0 - Investigating - Actions is experiencing degraded performance. We are continuing to investigate. Nov 4, 09:49:55 GMT+0 - Investigating - We are investigating reports of degraded performance for Git Operations, Issues and Pull Requests </description> <content:encoded> <![CDATA[<p><strong>Type:</strong> Incident</p> <p><strong>Duration:</strong> 5 days and 10 minutes</p> <p><strong>Affected Components:</strong> </p> <p><small>Nov <var data-var='date'> 4</var>, <var data-var='time'>11:50:06</var> GMT+0</small><br><strong>Resolved</strong> - n November 4, 2026, between 09:35 UTC and 11:43 UTC, customers experienced intermittent errors and timeouts across GitHub.com, primarily on actions that write data. At peak, about 20% of pull request write operations (such as merging, commenting, and reviewing) failed, along with about 10% of issue creates and edits and 10–15% of project item edits. Git pushes over SSH, REST and GraphQL API requests, code scanning uploads, releases, and package operations were also affected, and about 1% of Actions runs were delayed by more than five minutes or failed to start. Overall, nearly 4% of all requests to GitHub.com failed at the worst point. Reads were largely unaffected, and retrying a failed action a short time later usually succeeded.<br /><br />The incident was triggered by a burst of concurrent updates to the same database record, which tied up all available connections to one of our primary databases. Although that contention cleared, the backlog of waiting requests kept our database connection layer overloaded, and it did not recover on its own. We mitigated the incident by failing over to a new primary database host. Most services recovered by 11:31 UTC, and Actions recovered by 11:43 UTC.<br /><br />Since the incident, we have disabled the code path that triggered the contention, increased the processing capacity of our database connection layer by 50%, and added alerting to detect this type of overload earlier. We are also improving our procedures so we can fail over sooner in similar situations..</p> <p><small>Nov <var data-var='date'> 4</var>, <var data-var='time'>11:43:52</var> GMT+0</small><br><strong>Monitoring</strong> - The degradation affecting Actions has been mitigated. We are monitoring to ensure stability..</p> <p><small>Nov <var data-var='date'> 4</var>, <var data-var='time'>11:30:42</var> GMT+0</small><br><strong>Investigating</strong> - The degradation affecting API Requests, Git Operations, Issues and Pull Requests has been mitigated. We are monitoring to ensure stability..</p> <p><small>Nov <var data-var='date'> 4</var>, <var data-var='time'>11:17:52</var> GMT+0</small><br><strong>Investigating</strong> - We're currently working on mitigating elevated errors and intermittent failures affecting GitHub, including Pull Requests, Git Operations, and Issues. Some requests may fail or be delayed while we work to restore service. We'll provide another update as more information becomes available..</p> <p><small>Nov <var data-var='date'> 4</var>, <var data-var='time'>11:11:30</var> GMT+0</small><br><strong>Investigating</strong> - API Requests is experiencing degraded performance. We are continuing to investigate..</p> <p><small>Nov <var data-var='date'> 4</var>, <var data-var='time'>10:38:49</var> GMT+0</small><br><strong>Investigating</strong> - We're investigating elevated errors and intermittent failures affecting GitHub including Pull Requests, Git Operations, and Issues. We'll continue to keep users updated as we work toward mitigation..</p> <p><small>Nov <var data-var='date'> 4</var>, <var data-var='time'>09:53:38</var> GMT+0</small><br><strong>Investigating</strong> - Pull Requests is experiencing degraded availability. We are continuing to investigate..</p> <p><small>Nov <var data-var='date'> 4</var>, <var data-var='time'>09:50:47</var> GMT+0</small><br><strong>Investigating</strong> - Actions is experiencing degraded performance. We are continuing to investigate..</p> <p><small>Nov <var data-var='date'> 4</var>, <var data-var='time'>09:49:55</var> GMT+0</small><br><strong>Investigating</strong> - We are investigating reports of degraded performance for Git Operations, Issues and Pull Requests.</p> ]]> </content:encoded> <pubDate>Wed, 4 Nov 2026 09:49:55 +0000</pubDate> <link>https://status.macroscope.com/incident/cmw1xwbww0yu30rofgqgj2gmb</link> <guid>https://status.macroscope.com/incident/cmw1xwbww0yu30rofgqgj2gmb</guid> </item> <item> <title>Incident with API, Git, and Copilot requests</title> <description> Type: Incident Duration: 1 day, 6 hours and 59 minutes Affected Components: GitHub API Requests Nov 3, 13:56:18 GMT+0 - Resolved - On November 3, 2026, between 12:34 and 13:51 UTC, some users experienced authentication failures and errors with the REST and GraphQL APIs, pull requests, Issues, and GitHub Copilot. On average, the error rate was 0.46% and peaked at 0.98% of API requests. Copilot API errors averaged 2.0% and peaked at 3.1%.<br /><br />This was caused by a bug in our process for replacing control-plane servers in one of our compute clusters. When a replacement server did not receive a timely response from the existing cluster, it incorrectly created a separate cluster. This stopped traffic from reaching much of the cluster's capacity and overloaded the authentication servers that were still handling requests.<br /><br />We mitigated the incident by removing the incorrectly configured server.<br />We are also fixing the replacement logic and adding safeguards to prevent servers in this state from serving traffic. Nov 3, 13:54:20 GMT+0 - Monitoring - The degradation has been mitigated. We are monitoring to ensure stability. Nov 3, 13:52:21 GMT+0 - Investigating - API Requests is operating normally. Nov 3, 13:51:46 GMT+0 - Monitoring - The degradation affecting Copilot has been mitigated. We are monitoring to ensure stability. </description> <content:encoded> <![CDATA[<p><strong>Type:</strong> Incident</p> <p><strong>Duration:</strong> 1 day, 6 hours and 59 minutes</p> <p><strong>Affected Components:</strong> </p> <p><small>Nov <var data-var='date'> 3</var>, <var data-var='time'>13:56:18</var> GMT+0</small><br><strong>Resolved</strong> - On November 3, 2026, between 12:34 and 13:51 UTC, some users experienced authentication failures and errors with the REST and GraphQL APIs, pull requests, Issues, and GitHub Copilot. On average, the error rate was 0.46% and peaked at 0.98% of API requests. Copilot API errors averaged 2.0% and peaked at 3.1%.<br /><br />This was caused by a bug in our process for replacing control-plane servers in one of our compute clusters. When a replacement server did not receive a timely response from the existing cluster, it incorrectly created a separate cluster. This stopped traffic from reaching much of the cluster's capacity and overloaded the authentication servers that were still handling requests.<br /><br />We mitigated the incident by removing the incorrectly configured server.<br />We are also fixing the replacement logic and adding safeguards to prevent servers in this state from serving traffic..</p> <p><small>Nov <var data-var='date'> 3</var>, <var data-var='time'>13:54:20</var> GMT+0</small><br><strong>Monitoring</strong> - The degradation has been mitigated. We are monitoring to ensure stability..</p> <p><small>Nov <var data-var='date'> 3</var>, <var data-var='time'>13:52:21</var> GMT+0</small><br><strong>Investigating</strong> - API Requests is operating normally..</p> <p><small>Nov <var data-var='date'> 3</var>, <var data-var='time'>13:51:46</var> GMT+0</small><br><strong>Monitoring</strong> - The degradation affecting Copilot has been mitigated. We are monitoring to ensure stability..</p> ]]> </content:encoded> <pubDate>Tue, 3 Nov 2026 13:25:19 +0000</pubDate> <link>https://status.macroscope.com/incident/cmw0r14sh00m00krtya6hnzbv</link> <guid>https://status.macroscope.com/incident/cmw0r14sh00m00krtya6hnzbv</guid> </item> <item> <title>Incident with Actions, Pages, and Webhooks</title> <description> Type: Incident Duration: 3 days, 20 hours and 45 minutes Affected Components: GitHub Actions Oct 30, 19:23:40 GMT+0 - Resolved - On October 30, 2026, from 17:50 to 18:32 UTC, Actions, Pages, and Webhooks experienced degraded performance. Actions workflow processing, Pages deployments, and webhook deliveries were delayed, and code search and live updates returned errors. Some Actions queues took additional time to drain after mitigation, and the incident was resolved at 19:23 UTC. <br /><br />The impact was triggered by maintenance on a cloud network gateway connecting an Actions service in the GitHub data center to Redis clusters in the Central US Azure region. The service's Redis client retried aggressively and redialed those connections. That produced a connection storm that saturated the gateway, with about 90% of connection attempts failing. Other services sharing the same network path were degraded as well. <br /><br />We mitigated the incident by removing the affected resources from rotation, which ended the connection storm and allowed the gateway to recover. We have already reduced the number of connection retries in the impacted service and are further improving retry behavior with jittered backoff, shorter connect timeouts, and retry budgets. We are also working with our infrastructure team to make GitHub more resilient to gateway maintenance. Oct 30, 18:32:39 GMT+0 - Monitoring - The degradation affecting Actions, Pages and Webhooks has been mitigated. We are monitoring to ensure stability. Oct 30, 18:32:23 GMT+0 - Investigating - All affected services have recovered following mitigation. We continue to investigate root cause. Oct 30, 18:06:21 GMT+0 - Investigating - We continue to investigate delays affecting Actions workflows, Pages deployments, and webhook events, along with errors affecting code search and live updates. We have taken mitigation steps. Actions processing and Pages deployments are showing signs of recovery, but impact has not fully resolved. We are monitoring recovery and investigating the underlying cause. Oct 30, 18:00:58 GMT+0 - Investigating - Pages is experiencing degraded performance. We are continuing to investigate. Oct 30, 17:50:54 GMT+0 - Investigating - We are investigating reports of degraded performance for Actions and Webhooks </description> <content:encoded> <![CDATA[<p><strong>Type:</strong> Incident</p> <p><strong>Duration:</strong> 3 days, 20 hours and 45 minutes</p> <p><strong>Affected Components:</strong> </p> <p><small>Oct <var data-var='date'> 30</var>, <var data-var='time'>19:23:40</var> GMT+0</small><br><strong>Resolved</strong> - On October 30, 2026, from 17:50 to 18:32 UTC, Actions, Pages, and Webhooks experienced degraded performance. Actions workflow processing, Pages deployments, and webhook deliveries were delayed, and code search and live updates returned errors. Some Actions queues took additional time to drain after mitigation, and the incident was resolved at 19:23 UTC. <br /><br />The impact was triggered by maintenance on a cloud network gateway connecting an Actions service in the GitHub data center to Redis clusters in the Central US Azure region. The service's Redis client retried aggressively and redialed those connections. That produced a connection storm that saturated the gateway, with about 90% of connection attempts failing. Other services sharing the same network path were degraded as well. <br /><br />We mitigated the incident by removing the affected resources from rotation, which ended the connection storm and allowed the gateway to recover. We have already reduced the number of connection retries in the impacted service and are further improving retry behavior with jittered backoff, shorter connect timeouts, and retry budgets. We are also working with our infrastructure team to make GitHub more resilient to gateway maintenance..</p> <p><small>Oct <var data-var='date'> 30</var>, <var data-var='time'>18:32:39</var> GMT+0</small><br><strong>Monitoring</strong> - The degradation affecting Actions, Pages and Webhooks has been mitigated. We are monitoring to ensure stability..</p> <p><small>Oct <var data-var='date'> 30</var>, <var data-var='time'>18:32:23</var> GMT+0</small><br><strong>Investigating</strong> - All affected services have recovered following mitigation. We continue to investigate root cause..</p> <p><small>Oct <var data-var='date'> 30</var>, <var data-var='time'>18:06:21</var> GMT+0</small><br><strong>Investigating</strong> - We continue to investigate delays affecting Actions workflows, Pages deployments, and webhook events, along with errors affecting code search and live updates. We have taken mitigation steps. Actions processing and Pages deployments are showing signs of recovery, but impact has not fully resolved. We are monitoring recovery and investigating the underlying cause..</p> <p><small>Oct <var data-var='date'> 30</var>, <var data-var='time'>18:00:58</var> GMT+0</small><br><strong>Investigating</strong> - Pages is experiencing degraded performance. We are continuing to investigate..</p> <p><small>Oct <var data-var='date'> 30</var>, <var data-var='time'>17:50:54</var> GMT+0</small><br><strong>Investigating</strong> - We are investigating reports of degraded performance for Actions and Webhooks.</p> ]]> </content:encoded> <pubDate>Fri, 30 Oct 2026 17:50:54 +0000</pubDate> <link>https://status.macroscope.com/incident/cmvv9ud2y1i0c0rp2vnsegyxw</link> <guid>https://status.macroscope.com/incident/cmvv9ud2y1i0c0rp2vnsegyxw</guid> </item> <item> <title>Incident with Actions, Issues, API Requests and Pull Requests</title> <description> Type: Incident Duration: 8 hours and 56 minutes Affected Components: GitHub API Requests, GitHub Actions Oct 26, 18:57:22 GMT+0 - Resolved - On October 26, 2026, between 17:52 and 18:49 UTC, some customers experienced slower response times across the Checks API, Issues, Pull Requests, GraphQL, Notifications, and related services. Impact was limited to requests served from one of our datacenter sites. Some GraphQL requests failed, and some notifications were delayed. Among GraphQL requests served from the affected site, the average error rate was 5.68%, with a peak five-minute error rate of 14.32%. <br /><br />Unusually large requests for contribution history overwhelmed part of our database infrastructure, causing a backlog. Requests continued reaching the slow component rather than being redirected to healthy systems. Restarting the component restored its performance. The incident was marked mitigated at 18:46 UTC and resolved at 18:57 UTC. <br /><br />We are adding limits to these requests, improving alerts to detect overload sooner, and strengthening automatic redirection to healthy systems. Oct 26, 18:54:04 GMT+0 - Monitoring - We saw elevated latency affecting REST API requests, including Checks, and some Actions, Issues, Pull Requests, and GraphQL functionality. Some requests may have been slow or timed out, and notifications were delayed from 17:59 UTC - 18:40 UTC.<br /><br />We mitigated the issue and all services have recovered. Oct 26, 18:50:58 GMT+0 - Monitoring - The degradation has been mitigated. We are monitoring to ensure stability. Oct 26, 18:48:26 GMT+0 - Investigating - We are investigating reports of degraded performance for API Requests, Actions, Issues and Pull Requests </description> <content:encoded> <![CDATA[<p><strong>Type:</strong> Incident</p> <p><strong>Duration:</strong> 8 hours and 56 minutes</p> <p><strong>Affected Components:</strong> </p> <p><small>Oct <var data-var='date'> 26</var>, <var data-var='time'>18:57:22</var> GMT+0</small><br><strong>Resolved</strong> - On October 26, 2026, between 17:52 and 18:49 UTC, some customers experienced slower response times across the Checks API, Issues, Pull Requests, GraphQL, Notifications, and related services. Impact was limited to requests served from one of our datacenter sites. Some GraphQL requests failed, and some notifications were delayed. Among GraphQL requests served from the affected site, the average error rate was 5.68%, with a peak five-minute error rate of 14.32%. <br /><br />Unusually large requests for contribution history overwhelmed part of our database infrastructure, causing a backlog. Requests continued reaching the slow component rather than being redirected to healthy systems. Restarting the component restored its performance. The incident was marked mitigated at 18:46 UTC and resolved at 18:57 UTC. <br /><br />We are adding limits to these requests, improving alerts to detect overload sooner, and strengthening automatic redirection to healthy systems..</p> <p><small>Oct <var data-var='date'> 26</var>, <var data-var='time'>18:54:04</var> GMT+0</small><br><strong>Monitoring</strong> - We saw elevated latency affecting REST API requests, including Checks, and some Actions, Issues, Pull Requests, and GraphQL functionality. Some requests may have been slow or timed out, and notifications were delayed from 17:59 UTC - 18:40 UTC.<br /><br />We mitigated the issue and all services have recovered..</p> <p><small>Oct <var data-var='date'> 26</var>, <var data-var='time'>18:50:58</var> GMT+0</small><br><strong>Monitoring</strong> - The degradation has been mitigated. We are monitoring to ensure stability..</p> <p><small>Oct <var data-var='date'> 26</var>, <var data-var='time'>18:48:26</var> GMT+0</small><br><strong>Investigating</strong> - We are investigating reports of degraded performance for API Requests, Actions, Issues and Pull Requests.</p> ]]> </content:encoded> <pubDate>Mon, 26 Oct 2026 18:48:26 +0000</pubDate> <link>https://status.macroscope.com/incident/cmvpm24xo044l0rnrk1oz9c5y</link> <guid>https://status.macroscope.com/incident/cmvpm24xo044l0rnrk1oz9c5y</guid> </item> <item> <title>Delayed Actions status check results</title> <description> Type: Incident Duration: 1 day, 12 hours and 24 minutes Affected Components: GitHub Actions Oct 23, 08:05:27 GMT+0 - Resolved - On October 23, 2026, between 07:06 UTC and 07:27 UTC, GitHub Actions workflow run results were delayed. Job status, check run updates, and logs in the GitHub UI and API updated with an average delay of about 2 minutes, and some updates were delayed by up to about 7 minutes. <br /><br />During this window, a storage dependency that Actions uses to process workflow logs and artifacts throttled our requests. This slowed result processing, causing updates to queue up. The queue was fully processed by 07:27 UTC, shortly after the throttling stopped. <br /><br />To reduce the risk of recurrence, we have added several storage accounts to increase capacity, and we are working on additional steps to continue to scale. We are also working with our storage provider to understand why the throttling occurred, and improving how result processing handles storage throttling and how quickly we detect it. Oct 23, 07:56:35 GMT+0 - Monitoring - As of 07:27 UTC, Actions results are delivering within normal SLOs again.<br /><br />Thank you for this patience during this period of delayed Actions results delivery. Our team continues to monitor system health and explore refinements to reduce impact to customers in the future. Oct 23, 07:53:40 GMT+0 - Monitoring - The degradation affecting Actions has been mitigated. We are monitoring to ensure stability. Oct 23, 07:31:54 GMT+0 - Investigating - We are seeing delays in results from Actions workflows being reported within GitHub starting from 07:07 UTC.<br /><br />Recovery is already being seen but our team is continuing to investigate and monitor performance. Oct 23, 07:29:03 GMT+0 - Investigating - We are investigating reports of degraded performance for Actions </description> <content:encoded> <![CDATA[<p><strong>Type:</strong> Incident</p> <p><strong>Duration:</strong> 1 day, 12 hours and 24 minutes</p> <p><strong>Affected Components:</strong> </p> <p><small>Oct <var data-var='date'> 23</var>, <var data-var='time'>08:05:27</var> GMT+0</small><br><strong>Resolved</strong> - On October 23, 2026, between 07:06 UTC and 07:27 UTC, GitHub Actions workflow run results were delayed. Job status, check run updates, and logs in the GitHub UI and API updated with an average delay of about 2 minutes, and some updates were delayed by up to about 7 minutes. <br /><br />During this window, a storage dependency that Actions uses to process workflow logs and artifacts throttled our requests. This slowed result processing, causing updates to queue up. The queue was fully processed by 07:27 UTC, shortly after the throttling stopped. <br /><br />To reduce the risk of recurrence, we have added several storage accounts to increase capacity, and we are working on additional steps to continue to scale. We are also working with our storage provider to understand why the throttling occurred, and improving how result processing handles storage throttling and how quickly we detect it..</p> <p><small>Oct <var data-var='date'> 23</var>, <var data-var='time'>07:56:35</var> GMT+0</small><br><strong>Monitoring</strong> - As of 07:27 UTC, Actions results are delivering within normal SLOs again.<br /><br />Thank you for this patience during this period of delayed Actions results delivery. Our team continues to monitor system health and explore refinements to reduce impact to customers in the future..</p> <p><small>Oct <var data-var='date'> 23</var>, <var data-var='time'>07:53:40</var> GMT+0</small><br><strong>Monitoring</strong> - The degradation affecting Actions has been mitigated. We are monitoring to ensure stability..</p> <p><small>Oct <var data-var='date'> 23</var>, <var data-var='time'>07:31:54</var> GMT+0</small><br><strong>Investigating</strong> - We are seeing delays in results from Actions workflows being reported within GitHub starting from 07:07 UTC.<br /><br />Recovery is already being seen but our team is continuing to investigate and monitor performance..</p> <p><small>Oct <var data-var='date'> 23</var>, <var data-var='time'>07:29:03</var> GMT+0</small><br><strong>Investigating</strong> - We are investigating reports of degraded performance for Actions.</p> ]]> </content:encoded> <pubDate>Fri, 23 Oct 2026 07:29:03 +0000</pubDate> <link>https://status.macroscope.com/incident/cmvknm0z513680ro2edgnuzhh</link> <guid>https://status.macroscope.com/incident/cmvknm0z513680ro2edgnuzhh</guid> </item> <item> <title>Project updates delayed</title> <description> Type: Incident Duration: 27 days, 14 hours and 21 minutes Affected Components: GitHub Actions, GitHub API Requests Oct 23, 02:30:25 GMT+0 - Resolved - On October 22, 2026, between 14:55 and 17:28 UTC, search on GitHub.com was degraded. Search-backed features in Issues, Pull Requests, Projects, Actions, Packages, and some REST API endpoints returned errors or incomplete results. On average, 21% of search requests failed, peaking at 52%. Running Actions workflows, uploading and downloading packages, and Git operations were not affected. <br /><br />This was due to the proxies that route requests to our search clusters running out of CPU while setting up encrypted connections. Requests that timed out were retried, which added more load and kept the proxies overloaded. The search clusters themselves remained healthy. <br /><br />We mitigated the incident by adding proxy capacity, after which search recovered at 17:28 UTC. <br /><br />We have started reducing the CPU cost of these encrypted connections. We are also working to make search retries adjustable during an incident without a deployment, and to alert on search failures sooner. Together these will reduce our time to detection and mitigation of issues like this one in the future. Oct 23, 00:55:18 GMT+0 - Monitoring - The degradation has been mitigated. We are monitoring to ensure stability. Oct 22, 23:48:05 GMT+0 - Investigating - GitHub Projects update processing has improved, but newly added items may still take longer than expected to appear. <br /><br />We're continuing to work on the remaining delays and monitor recovery. Oct 22, 22:07:36 GMT+0 - Investigating - Some GitHub Projects updates, including newly added items and changes to issues and pull requests, are still delayed.<br /><br />We have seen some recovery, and are working to add more capacity to accelerate the recovery. Oct 22, 20:46:26 GMT+0 - Investigating - Some GitHub Projects updates, including newly added items and changes to issues and pull requests, remain delayed.<br /><br />We've identified interruptions in update processing and are evaluating mitigations to restore normal operation. Oct 22, 19:56:09 GMT+0 - Investigating - Some GitHub Projects updates related to issue changes and merged pull requests are still delayed. We're investigating the remaining delays and working to restore normal processing. Oct 22, 19:04:03 GMT+0 - Investigating - Some updates in GitHub Projects are still delayed. We're continuing mitigation work and monitoring. Oct 22, 18:26:31 GMT+0 - Investigating - The other affected services have recovered. <br /><br />We're continuing to investigate GitHub Projects and verify that normal operation has been restored. Oct 22, 17:43:44 GMT+0 - Investigating - We're seeing recovery across the affected services. GitHub Projects is still recovering as queued updates are processed, and we're continuing to monitor for stability. Oct 22, 17:41:38 GMT+0 - Investigating - Actions, Issues, Packages and Pull Requests are operating normally. Oct 22, 17:16:06 GMT+0 - Investigating - We're seeing recovery after adding capacity, with errors decreasing. Some users may still experience errors loading Issues, Pull Requests and search results on github.com. <br /><br />We're continuing to work towards full recovery and monitoring for stability. Oct 22, 16:41:52 GMT+0 - Investigating - Some users continue to experience errors loading Issues, Pull Requests, package listings and search results on github.com. <br /><br />We're working through multiple mitigation efforts and continuing to investigate. Oct 22, 16:35:07 GMT+0 - Investigating - API Requests is operating normally. Oct 22, 16:28:56 GMT+0 - Investigating - Packages is experiencing degraded performance. We are continuing to investigate. Oct 22, 16:26:16 GMT+0 - Investigating - API Requests is experiencing degraded performance. We are continuing to investigate. Oct 22, 16:08:04 GMT+0 - Investigating - Some users continue to experience intermittent errors loading Issues, Pull Requests and search results on github.com. <br /><br />We're investigating and evaluating mitigation options to restore normal service. Oct 22, 15:33:52 GMT+0 - Investigating - Some users may experience intermittent errors loading Issues, Pull Requests and search results on github.com. <br /><br />We're investigating and working to restore normal service. Oct 22, 15:28:04 GMT+0 - Investigating - We are investigating reports of degraded performance for Actions, Issues and Pull Requests </description> <content:encoded> <![CDATA[<p><strong>Type:</strong> Incident</p> <p><strong>Duration:</strong> 27 days, 14 hours and 21 minutes</p> <p><strong>Affected Components:</strong> </p> <p><small>Oct <var data-var='date'> 23</var>, <var data-var='time'>02:30:25</var> GMT+0</small><br><strong>Resolved</strong> - On October 22, 2026, between 14:55 and 17:28 UTC, search on GitHub.com was degraded. Search-backed features in Issues, Pull Requests, Projects, Actions, Packages, and some REST API endpoints returned errors or incomplete results. On average, 21% of search requests failed, peaking at 52%. Running Actions workflows, uploading and downloading packages, and Git operations were not affected. <br /><br />This was due to the proxies that route requests to our search clusters running out of CPU while setting up encrypted connections. Requests that timed out were retried, which added more load and kept the proxies overloaded. The search clusters themselves remained healthy. <br /><br />We mitigated the incident by adding proxy capacity, after which search recovered at 17:28 UTC. <br /><br />We have started reducing the CPU cost of these encrypted connections. We are also working to make search retries adjustable during an incident without a deployment, and to alert on search failures sooner. Together these will reduce our time to detection and mitigation of issues like this one in the future..</p> <p><small>Oct <var data-var='date'> 23</var>, <var data-var='time'>00:55:18</var> GMT+0</small><br><strong>Monitoring</strong> - The degradation has been mitigated. We are monitoring to ensure stability..</p> <p><small>Oct <var data-var='date'> 22</var>, <var data-var='time'>23:48:05</var> GMT+0</small><br><strong>Investigating</strong> - GitHub Projects update processing has improved, but newly added items may still take longer than expected to appear. <br /><br />We're continuing to work on the remaining delays and monitor recovery..</p> <p><small>Oct <var data-var='date'> 22</var>, <var data-var='time'>22:07:36</var> GMT+0</small><br><strong>Investigating</strong> - Some GitHub Projects updates, including newly added items and changes to issues and pull requests, are still delayed.<br /><br />We have seen some recovery, and are working to add more capacity to accelerate the recovery..</p> <p><small>Oct <var data-var='date'> 22</var>, <var data-var='time'>20:46:26</var> GMT+0</small><br><strong>Investigating</strong> - Some GitHub Projects updates, including newly added items and changes to issues and pull requests, remain delayed.<br /><br />We've identified interruptions in update processing and are evaluating mitigations to restore normal operation..</p> <p><small>Oct <var data-var='date'> 22</var>, <var data-var='time'>19:56:09</var> GMT+0</small><br><strong>Investigating</strong> - Some GitHub Projects updates related to issue changes and merged pull requests are still delayed. We're investigating the remaining delays and working to restore normal processing..</p> <p><small>Oct <var data-var='date'> 22</var>, <var data-var='time'>19:04:03</var> GMT+0</small><br><strong>Investigating</strong> - Some updates in GitHub Projects are still delayed. We're continuing mitigation work and monitoring..</p> <p><small>Oct <var data-var='date'> 22</var>, <var data-var='time'>18:26:31</var> GMT+0</small><br><strong>Investigating</strong> - The other affected services have recovered. <br /><br />We're continuing to investigate GitHub Projects and verify that normal operation has been restored..</p> <p><small>Oct <var data-var='date'> 22</var>, <var data-var='time'>17:43:44</var> GMT+0</small><br><strong>Investigating</strong> - We're seeing recovery across the affected services. GitHub Projects is still recovering as queued updates are processed, and we're continuing to monitor for stability..</p> <p><small>Oct <var data-var='date'> 22</var>, <var data-var='time'>17:41:38</var> GMT+0</small><br><strong>Investigating</strong> - Actions, Issues, Packages and Pull Requests are operating normally..</p> <p><small>Oct <var data-var='date'> 22</var>, <var data-var='time'>17:16:06</var> GMT+0</small><br><strong>Investigating</strong> - We're seeing recovery after adding capacity, with errors decreasing. Some users may still experience errors loading Issues, Pull Requests and search results on github.com. <br /><br />We're continuing to work towards full recovery and monitoring for stability..</p> <p><small>Oct <var data-var='date'> 22</var>, <var data-var='time'>16:41:52</var> GMT+0</small><br><strong>Investigating</strong> - Some users continue to experience errors loading Issues, Pull Requests, package listings and search results on github.com. <br /><br />We're working through multiple mitigation efforts and continuing to investigate..</p> <p><small>Oct <var data-var='date'> 22</var>, <var data-var='time'>16:35:07</var> GMT+0</small><br><strong>Investigating</strong> - API Requests is operating normally..</p> <p><small>Oct <var data-var='date'> 22</var>, <var data-var='time'>16:28:56</var> GMT+0</small><br><strong>Investigating</strong> - Packages is experiencing degraded performance. We are continuing to investigate..</p> <p><small>Oct <var data-var='date'> 22</var>, <var data-var='time'>16:26:16</var> GMT+0</small><br><strong>Investigating</strong> - API Requests is experiencing degraded performance. We are continuing to investigate..</p> <p><small>Oct <var data-var='date'> 22</var>, <var data-var='time'>16:08:04</var> GMT+0</small><br><strong>Investigating</strong> - Some users continue to experience intermittent errors loading Issues, Pull Requests and search results on github.com. <br /><br />We're investigating and evaluating mitigation options to restore normal service..</p> <p><small>Oct <var data-var='date'> 22</var>, <var data-var='time'>15:33:52</var> GMT+0</small><br><strong>Investigating</strong> - Some users may experience intermittent errors loading Issues, Pull Requests and search results on github.com. <br /><br />We're investigating and working to restore normal service..</p> <p><small>Oct <var data-var='date'> 22</var>, <var data-var='time'>15:28:04</var> GMT+0</small><br><strong>Investigating</strong> - We are investigating reports of degraded performance for Actions, Issues and Pull Requests.</p> ]]> </content:encoded> <pubDate>Thu, 22 Oct 2026 15:28:03 +0000</pubDate> <link>https://status.macroscope.com/incident/cmvjp54y10jcs0rp4a64aa93b</link> <guid>https://status.macroscope.com/incident/cmvjp54y10jcs0rp4a64aa93b</guid> </item> <item> <title>Incident with Actions</title> <description> Type: Incident Duration: 2 days, 12 hours and 30 minutes Affected Components: GitHub Actions Oct 22, 15:18:46 GMT+0 - Resolved - On October 22, 2026, from 13:05 to 15:20 UTC, our event streaming platform was degraded resulting in elevated job processing latency spikes of 19-35 seconds and elevated job processing times of 4m – this resulted in delayed Actions workflow and step results (up to 13m), notifications (20m) and repository push events (12m). Recovery for repositories began at 14:32, with Actions recovery beginning at 14:59. The issue reoccurred at 16:26 again impacting Actions runs, repo push events, and notifications. Recovery began at 17:16 and completed at 2:07. <br /><br />The incident was caused by a significant increase in load coupled with client connection storms during a deployment overloading our Kafka event streaming brokers. <br /><br />To prevent future recurrence, we have expanded the affected cluster, shifted traffic to less loaded clusters, and implemented tunable client reconnect backoff. Oct 22, 15:18:41 GMT+0 - Monitoring - GitHub Actions workflow and step result processing has recovered. Repository update processing has also returned to normal. Oct 22, 15:17:23 GMT+0 - Monitoring - The degradation affecting Actions has been mitigated. We are monitoring to ensure stability. Oct 22, 15:06:35 GMT+0 - Investigating - We’re seeing recovery, and repository update processing has returned to normal. Delays affecting GitHub Actions workflow and step results are continuing to clear as we scale processing capacity; the remaining backlog is expected to take about 20 minutes to drain. Oct 22, 14:35:29 GMT+0 - Investigating - Repository update processing has recovered. We’re continuing to work on delays affecting GitHub Actions workflow and step results. We’ll share another update as we make progress. Oct 22, 14:19:32 GMT+0 - Investigating - We’re investigating delays in processing GitHub Actions workflow and step results, along with some repository updates. Oct 22, 14:18:15 GMT+0 - Investigating - We are investigating reports of degraded performance for Actions </description> <content:encoded> <![CDATA[<p><strong>Type:</strong> Incident</p> <p><strong>Duration:</strong> 2 days, 12 hours and 30 minutes</p> <p><strong>Affected Components:</strong> </p> <p><small>Oct <var data-var='date'> 22</var>, <var data-var='time'>15:18:46</var> GMT+0</small><br><strong>Resolved</strong> - On October 22, 2026, from 13:05 to 15:20 UTC, our event streaming platform was degraded resulting in elevated job processing latency spikes of 19-35 seconds and elevated job processing times of 4m – this resulted in delayed Actions workflow and step results (up to 13m), notifications (20m) and repository push events (12m). Recovery for repositories began at 14:32, with Actions recovery beginning at 14:59. The issue reoccurred at 16:26 again impacting Actions runs, repo push events, and notifications. Recovery began at 17:16 and completed at 2:07. <br /><br />The incident was caused by a significant increase in load coupled with client connection storms during a deployment overloading our Kafka event streaming brokers. <br /><br />To prevent future recurrence, we have expanded the affected cluster, shifted traffic to less loaded clusters, and implemented tunable client reconnect backoff..</p> <p><small>Oct <var data-var='date'> 22</var>, <var data-var='time'>15:18:41</var> GMT+0</small><br><strong>Monitoring</strong> - GitHub Actions workflow and step result processing has recovered. Repository update processing has also returned to normal..</p> <p><small>Oct <var data-var='date'> 22</var>, <var data-var='time'>15:17:23</var> GMT+0</small><br><strong>Monitoring</strong> - The degradation affecting Actions has been mitigated. We are monitoring to ensure stability..</p> <p><small>Oct <var data-var='date'> 22</var>, <var data-var='time'>15:06:35</var> GMT+0</small><br><strong>Investigating</strong> - We’re seeing recovery, and repository update processing has returned to normal. Delays affecting GitHub Actions workflow and step results are continuing to clear as we scale processing capacity; the remaining backlog is expected to take about 20 minutes to drain..</p> <p><small>Oct <var data-var='date'> 22</var>, <var data-var='time'>14:35:29</var> GMT+0</small><br><strong>Investigating</strong> - Repository update processing has recovered. We’re continuing to work on delays affecting GitHub Actions workflow and step results. We’ll share another update as we make progress..</p> <p><small>Oct <var data-var='date'> 22</var>, <var data-var='time'>14:19:32</var> GMT+0</small><br><strong>Investigating</strong> - We’re investigating delays in processing GitHub Actions workflow and step results, along with some repository updates..</p> <p><small>Oct <var data-var='date'> 22</var>, <var data-var='time'>14:18:15</var> GMT+0</small><br><strong>Investigating</strong> - We are investigating reports of degraded performance for Actions.</p> ]]> </content:encoded> <pubDate>Thu, 22 Oct 2026 14:18:15 +0000</pubDate> <link>https://status.macroscope.com/incident/cmvjmn0ks0epb0rp4j2hu08kx</link> <guid>https://status.macroscope.com/incident/cmvjmn0ks0epb0rp4j2hu08kx</guid> </item> <item> <title>Delays for Pull Requests and Webhook deliveries</title> <description> Type: Incident Duration: 11 days, 6 hours and 2 minutes Affected Components: GitHub Actions Oct 21, 19:22:52 GMT+0 - Resolved - On October 21, 2026, between 13:22 UTC and 19:22 UTC, several GitHub services were delayed due to degradation of our internal event streaming platform. <br /><br />Pull request updates, push webhooks, and push-triggered Actions workflows were delayed by up to 33 minutes until 15:36 UTC. Actions status updates were delayed by up to 33 minutes until 16:16 UTC, holding up some merges and deployments that wait on checks. Webhook deliveries and live updates were delayed by up to 25 minutes until 16:52 UTC. Audit log events were delayed by up to two hours. <br /><br />This was due to servers in the platform running out of memory caused by client reconnections without sufficient retry backoff. We mitigated the incident by moving the busiest workloads to less-loaded servers and adding capacity. <br /><br />We have since added reconnection backoff and added server capacity. We are building a solution for auto-rebalancing connections across servers, and splitting the platform into smaller, isolated clusters to reduce blast radius and time to mitigation of issues like this one in the future. Oct 21, 19:22:38 GMT+0 - Investigating - Pull request updates, Actions, and webhook deliveries have recovered. Audit log delays are now below 40 minutes, within the healthy threshold, and continue to decrease as the remaining backlog is processed. Oct 21, 18:52:50 GMT+0 - Investigating - Audit log events continue to recover, with delays now below 70 minutes. We’ve applied mitigations to speed recovery and expect audit logs to fully recover in about 40 minutes. Pull request updates, Actions, and webhook deliveries remain recovered. We are monitoring progress and will share further updates. Oct 21, 18:18:11 GMT+0 - Investigating - Audit log events are recovering, with delays now below 90 minutes as the remaining backlog is processed. We’ve reduced load on the event-streaming platform and are monitoring recovery. Further updates will follow. Oct 21, 17:09:35 GMT+0 - Investigating - Pull request updates, Actions, and webhook deliveries have recovered. Audit log events are recovering and delays are decreasing, but we expect the remaining backlog to take approximately two to three hours to clear. We’re monitoring recovery and will share further updates. Oct 21, 16:59:45 GMT+0 - Investigating - Actions and webhook deliveries have recovered. Audit log events are still delayed by about 100 minutes. We are working through the backlog, and full recovery will take some time. We will share more updates as recovery continues. Oct 21, 16:57:42 GMT+0 - Investigating - Actions and Webhooks are operating normally. Oct 21, 16:34:49 GMT+0 - Investigating - Audit log events continue to be delayed by up to 110 minutes. Pull request updates have recovered. Actions is recovering, and users should no longer experience impact in Actions. Webhook deliveries continue to recover as we work through backlogs. We are monitoring recovery and will share further updates as we learn more. Oct 21, 16:33:10 GMT+0 - Investigating - Pull Requests is operating normally. Oct 21, 16:25:30 GMT+0 - Investigating - Audit log events are delayed by up to 100 minutes, and Actions workflow status updates continue to experience extended delays. Pull request updates and webhook deliveries are recovering as we restore event processing and work through backlogs. We’re monitoring recovery and will share further updates as we learn more. Oct 21, 16:17:23 GMT+0 - Investigating - Audit log events and Actions workflow status updates continue to be delayed. Pull request updates and webhook deliveries are recovering as we restore event processing and work through backlogs. We’re monitoring the recovery and will share further updates as we learn more. Oct 21, 15:58:38 GMT+0 - Investigating - We continue to see delays affecting pull request updates, webhook deliveries, and audit log events. The event streaming platform is showing signs of recovery, and we are monitoring as the impact continues. Oct 21, 15:36:47 GMT+0 - Investigating - We are investigating delays affecting pull request updates and webhook deliveries. Increased load on our event streaming platform is contributing to hotspots. We are shifting traffic away from affected areas and adding processing capacity; impact is ongoing while these mitigations take effect. Oct 21, 15:15:04 GMT+0 - Investigating - Actions is experiencing degraded performance. We are continuing to investigate. Oct 21, 14:52:50 GMT+0 - Investigating - We are investigating reports of degraded performance for Actions, Pull Requests and Webhooks </description> <content:encoded> <![CDATA[<p><strong>Type:</strong> Incident</p> <p><strong>Duration:</strong> 11 days, 6 hours and 2 minutes</p> <p><strong>Affected Components:</strong> </p> <p><small>Oct <var data-var='date'> 21</var>, <var data-var='time'>19:22:52</var> GMT+0</small><br><strong>Resolved</strong> - On October 21, 2026, between 13:22 UTC and 19:22 UTC, several GitHub services were delayed due to degradation of our internal event streaming platform. <br /><br />Pull request updates, push webhooks, and push-triggered Actions workflows were delayed by up to 33 minutes until 15:36 UTC. Actions status updates were delayed by up to 33 minutes until 16:16 UTC, holding up some merges and deployments that wait on checks. Webhook deliveries and live updates were delayed by up to 25 minutes until 16:52 UTC. Audit log events were delayed by up to two hours. <br /><br />This was due to servers in the platform running out of memory caused by client reconnections without sufficient retry backoff. We mitigated the incident by moving the busiest workloads to less-loaded servers and adding capacity. <br /><br />We have since added reconnection backoff and added server capacity. We are building a solution for auto-rebalancing connections across servers, and splitting the platform into smaller, isolated clusters to reduce blast radius and time to mitigation of issues like this one in the future..</p> <p><small>Oct <var data-var='date'> 21</var>, <var data-var='time'>19:22:38</var> GMT+0</small><br><strong>Investigating</strong> - Pull request updates, Actions, and webhook deliveries have recovered. Audit log delays are now below 40 minutes, within the healthy threshold, and continue to decrease as the remaining backlog is processed..</p> <p><small>Oct <var data-var='date'> 21</var>, <var data-var='time'>18:52:50</var> GMT+0</small><br><strong>Investigating</strong> - Audit log events continue to recover, with delays now below 70 minutes. We’ve applied mitigations to speed recovery and expect audit logs to fully recover in about 40 minutes. Pull request updates, Actions, and webhook deliveries remain recovered. We are monitoring progress and will share further updates..</p> <p><small>Oct <var data-var='date'> 21</var>, <var data-var='time'>18:18:11</var> GMT+0</small><br><strong>Investigating</strong> - Audit log events are recovering, with delays now below 90 minutes as the remaining backlog is processed. We’ve reduced load on the event-streaming platform and are monitoring recovery. Further updates will follow..</p> <p><small>Oct <var data-var='date'> 21</var>, <var data-var='time'>17:09:35</var> GMT+0</small><br><strong>Investigating</strong> - Pull request updates, Actions, and webhook deliveries have recovered. Audit log events are recovering and delays are decreasing, but we expect the remaining backlog to take approximately two to three hours to clear. We’re monitoring recovery and will share further updates..</p> <p><small>Oct <var data-var='date'> 21</var>, <var data-var='time'>16:59:45</var> GMT+0</small><br><strong>Investigating</strong> - Actions and webhook deliveries have recovered. Audit log events are still delayed by about 100 minutes. We are working through the backlog, and full recovery will take some time. We will share more updates as recovery continues..</p> <p><small>Oct <var data-var='date'> 21</var>, <var data-var='time'>16:57:42</var> GMT+0</small><br><strong>Investigating</strong> - Actions and Webhooks are operating normally..</p> <p><small>Oct <var data-var='date'> 21</var>, <var data-var='time'>16:34:49</var> GMT+0</small><br><strong>Investigating</strong> - Audit log events continue to be delayed by up to 110 minutes. Pull request updates have recovered. Actions is recovering, and users should no longer experience impact in Actions. Webhook deliveries continue to recover as we work through backlogs. We are monitoring recovery and will share further updates as we learn more..</p> <p><small>Oct <var data-var='date'> 21</var>, <var data-var='time'>16:33:10</var> GMT+0</small><br><strong>Investigating</strong> - Pull Requests is operating normally..</p> <p><small>Oct <var data-var='date'> 21</var>, <var data-var='time'>16:25:30</var> GMT+0</small><br><strong>Investigating</strong> - Audit log events are delayed by up to 100 minutes, and Actions workflow status updates continue to experience extended delays. Pull request updates and webhook deliveries are recovering as we restore event processing and work through backlogs. We’re monitoring recovery and will share further updates as we learn more..</p> <p><small>Oct <var data-var='date'> 21</var>, <var data-var='time'>16:17:23</var> GMT+0</small><br><strong>Investigating</strong> - Audit log events and Actions workflow status updates continue to be delayed. Pull request updates and webhook deliveries are recovering as we restore event processing and work through backlogs. We’re monitoring the recovery and will share further updates as we learn more..</p> <p><small>Oct <var data-var='date'> 21</var>, <var data-var='time'>15:58:38</var> GMT+0</small><br><strong>Investigating</strong> - We continue to see delays affecting pull request updates, webhook deliveries, and audit log events. The event streaming platform is showing signs of recovery, and we are monitoring as the impact continues..</p> <p><small>Oct <var data-var='date'> 21</var>, <var data-var='time'>15:36:47</var> GMT+0</small><br><strong>Investigating</strong> - We are investigating delays affecting pull request updates and webhook deliveries. Increased load on our event streaming platform is contributing to hotspots. We are shifting traffic away from affected areas and adding processing capacity; impact is ongoing while these mitigations take effect..</p> <p><small>Oct <var data-var='date'> 21</var>, <var data-var='time'>15:15:04</var> GMT+0</small><br><strong>Investigating</strong> - Actions is experiencing degraded performance. We are continuing to investigate..</p> <p><small>Oct <var data-var='date'> 21</var>, <var data-var='time'>14:52:50</var> GMT+0</small><br><strong>Investigating</strong> - We are investigating reports of degraded performance for Actions, Pull Requests and Webhooks.</p> ]]> </content:encoded> <pubDate>Wed, 21 Oct 2026 14:52:49 +0000</pubDate> <link>https://status.macroscope.com/incident/cmvi8inhf0ibb0rqtjc9om7jq</link> <guid>https://status.macroscope.com/incident/cmvi8inhf0ibb0rqtjc9om7jq</guid> </item> <item> <title>Degraded availability for GitHub Actions, GitHub Pages and Webhooks</title> <description> Type: Incident Duration: 2 days, 19 hours and 32 minutes Affected Components: GitHub Actions Oct 19, 16:56:58 GMT+0 - Resolved - On October 19, 2026, between 15:24 and 16:52 UTC, GitHub Actions, GitHub Pages, and webhooks generated by Actions were degraded. Live updates on GitHub.com and code search were also degraded for part of this time. No data was lost. <br /><br />Impact <br /><br /> - Workflow status updates, logs, and checks were delayed. <br /> - 5.13% of jobs on GitHub-hosted runners were delayed in starting, and 0.02% of workflow runs failed. <br /> - During the first 30 minutes, 8.47% of requests to the Actions jobs REST API returned errors. <br /><br />Pages deployments and webhooks generated by Actions runs were delayed. <br /><br />An internal Actions service that stores logs and run status lost connections to some of its cache servers in one cloud region and immediately retried them from every server at once. The resulting surge of new connections overloaded a shared network link between our data center and that region, which also disrupted other services that use the same link. We mitigated the incident by moving the Actions service off the affected cache servers and reducing connections from other services on the link. We then processed the delayed Actions work. We are continuing to investigate the cause of the initial connection timeouts. <br /><br />We were already in the process of rolling out a limit on how quickly this service can open new connections, but it was still in shadow mode (logging only, not enforcing) at the time of the incident. We have since fully enforced it and capped the service's concurrent connection attempts. We are extending these limits to other Actions services, adding backoff to retries, limiting how much shared network capacity a single service can use, and improving alerting so we can find the source of connection surges faster. Oct 19, 16:56:54 GMT+0 - Monitoring - Actions, Pages and Webhooks are operating normally. Oct 19, 16:30:29 GMT+0 - Investigating - We’re seeing recovery following mitigation and are continuing to monitor. <br /><br />Customers may still experience delays in GitHub Actions workflow status updates, GitHub Pages deployments, and check and workflow webhook events generated by Actions runs. Oct 19, 15:56:34 GMT+0 - Investigating - We’re are currently investigating degraded availability affecting GitHub Actions, GitHub Pages and Webhooks. <br /><br />Customers may experience delays in workflow processing, site builds and webhook delivery.<br /> Oct 19, 15:50:10 GMT+0 - Investigating - Webhooks is experiencing degraded performance. We are continuing to investigate. Oct 19, 15:49:25 GMT+0 - Investigating - We are investigating reports of degraded performance for Actions and Pages </description> <content:encoded> <![CDATA[<p><strong>Type:</strong> Incident</p> <p><strong>Duration:</strong> 2 days, 19 hours and 32 minutes</p> <p><strong>Affected Components:</strong> </p> <p><small>Oct <var data-var='date'> 19</var>, <var data-var='time'>16:56:58</var> GMT+0</small><br><strong>Resolved</strong> - On October 19, 2026, between 15:24 and 16:52 UTC, GitHub Actions, GitHub Pages, and webhooks generated by Actions were degraded. Live updates on GitHub.com and code search were also degraded for part of this time. No data was lost. <br /><br />Impact <br /><br /> - Workflow status updates, logs, and checks were delayed. <br /> - 5.13% of jobs on GitHub-hosted runners were delayed in starting, and 0.02% of workflow runs failed. <br /> - During the first 30 minutes, 8.47% of requests to the Actions jobs REST API returned errors. <br /><br />Pages deployments and webhooks generated by Actions runs were delayed. <br /><br />An internal Actions service that stores logs and run status lost connections to some of its cache servers in one cloud region and immediately retried them from every server at once. The resulting surge of new connections overloaded a shared network link between our data center and that region, which also disrupted other services that use the same link. We mitigated the incident by moving the Actions service off the affected cache servers and reducing connections from other services on the link. We then processed the delayed Actions work. We are continuing to investigate the cause of the initial connection timeouts. <br /><br />We were already in the process of rolling out a limit on how quickly this service can open new connections, but it was still in shadow mode (logging only, not enforcing) at the time of the incident. We have since fully enforced it and capped the service's concurrent connection attempts. We are extending these limits to other Actions services, adding backoff to retries, limiting how much shared network capacity a single service can use, and improving alerting so we can find the source of connection surges faster..</p> <p><small>Oct <var data-var='date'> 19</var>, <var data-var='time'>16:56:54</var> GMT+0</small><br><strong>Monitoring</strong> - Actions, Pages and Webhooks are operating normally..</p> <p><small>Oct <var data-var='date'> 19</var>, <var data-var='time'>16:30:29</var> GMT+0</small><br><strong>Investigating</strong> - We’re seeing recovery following mitigation and are continuing to monitor. <br /><br />Customers may still experience delays in GitHub Actions workflow status updates, GitHub Pages deployments, and check and workflow webhook events generated by Actions runs..</p> <p><small>Oct <var data-var='date'> 19</var>, <var data-var='time'>15:56:34</var> GMT+0</small><br><strong>Investigating</strong> - We’re are currently investigating degraded availability affecting GitHub Actions, GitHub Pages and Webhooks. <br /><br />Customers may experience delays in workflow processing, site builds and webhook delivery.<br />.</p> <p><small>Oct <var data-var='date'> 19</var>, <var data-var='time'>15:50:10</var> GMT+0</small><br><strong>Investigating</strong> - Webhooks is experiencing degraded performance. We are continuing to investigate..</p> <p><small>Oct <var data-var='date'> 19</var>, <var data-var='time'>15:49:25</var> GMT+0</small><br><strong>Investigating</strong> - We are investigating reports of degraded performance for Actions and Pages.</p> ]]> </content:encoded> <pubDate>Mon, 19 Oct 2026 15:49:25 +0000</pubDate> <link>https://status.macroscope.com/incident/cmvhfer9c01yb0ws8b5ljygfh</link> <guid>https://status.macroscope.com/incident/cmvhfer9c01yb0ws8b5ljygfh</guid> </item> <item> <title>Incident with Actions, Pull Requests and Issues</title> <description> Type: Incident Duration: 19 days, 18 hours and 19 minutes Affected Components: GitHub Actions Oct 12, 20:16:07 GMT+0 - Resolved - On October 12th 2026 11:51 UTC GitHub experienced elevated 500 responses across several application pages. This caused failures on Issues and Pull Requests as well as Projects and Actions. Customers also experienced delayed notifications in their web inbox and via email. <br /><br />The immediate impact to Issues, Pull Requests, Projects, and Actions was mitigated by 12:41 UTC on October 12th. Notifications deliveries to the web inbox and via email were recovering at 15:27 UTC and fully mitigated by 20:16 UTC. <br /><br />The incident was triggered through a latent condition in our notifications delivery pipeline that put a high volume of an expensive query on our sharded database replicas. This reduced available database capacity to serve requests for Issues, Pull Requests, Projects, and Actions.<br /><br />We have fixed the notifications pipeline’s code path that issued the expensive query. We are also improving resilience of the database layer to reduce the likelihood and impact of similar incidents. Oct 12, 19:28:42 GMT+0 - Investigating - Web and email notifications will be delayed for 1 more hour as we are close to completing the backlog of notifications. Oct 12, 17:34:24 GMT+0 - Investigating - Web and email notifications continue to be delayed as we work through a backlog of notifications. Oct 12, 15:50:39 GMT+0 - Investigating - Notifications have resumed but will continue to be delayed as we work through a backlog. Oct 12, 15:19:00 GMT+0 - Investigating - We have successfully applied a mitigation for notifications, and notifications have resumed but will continue to be delayed as we work through a backlog. Oct 12, 14:37:49 GMT+0 - Investigating - We have mitigated the issues with Actions, issues, and projects, and have a mitigation in progress to address the delay with notifications. Oct 12, 14:36:09 GMT+0 - Investigating - The degradation affecting Actions has been mitigated. We are monitoring to ensure stability. Oct 12, 14:35:49 GMT+0 - Investigating - The degradation affecting Issues has been mitigated. We are monitoring to ensure stability. Oct 12, 13:38:44 GMT+0 - Investigating - Our initial mitigation was not fully successful, and users may still experience delays and errors with Actions, issues and projects, and notifications are delayed. We're continuing to investigate the root cause and work on additional mitigations. Oct 12, 13:02:17 GMT+0 - Investigating - Pull Requests is operating normally. Oct 12, 12:57:44 GMT+0 - Investigating - We have a mitigation in progress and are monitoring for recovery. Users may also experience delayed web notifications as we mitigate. Oct 12, 12:33:21 GMT+0 - Investigating - Users may experience issues loading the Actions tab UI, interacting with the Checks API, queueing new Actions runs, creating and updating issues, and requesting PR reviews, including via REST and GraphQL APIs. Oct 12, 12:21:48 GMT+0 - Investigating - We are investigating reports of degraded performance for Actions, Issues and Pull Requests </description> <content:encoded> <![CDATA[<p><strong>Type:</strong> Incident</p> <p><strong>Duration:</strong> 19 days, 18 hours and 19 minutes</p> <p><strong>Affected Components:</strong> </p> <p><small>Oct <var data-var='date'> 12</var>, <var data-var='time'>20:16:07</var> GMT+0</small><br><strong>Resolved</strong> - On October 12th 2026 11:51 UTC GitHub experienced elevated 500 responses across several application pages. This caused failures on Issues and Pull Requests as well as Projects and Actions. Customers also experienced delayed notifications in their web inbox and via email. <br /><br />The immediate impact to Issues, Pull Requests, Projects, and Actions was mitigated by 12:41 UTC on October 12th. Notifications deliveries to the web inbox and via email were recovering at 15:27 UTC and fully mitigated by 20:16 UTC. <br /><br />The incident was triggered through a latent condition in our notifications delivery pipeline that put a high volume of an expensive query on our sharded database replicas. This reduced available database capacity to serve requests for Issues, Pull Requests, Projects, and Actions.<br /><br />We have fixed the notifications pipeline’s code path that issued the expensive query. We are also improving resilience of the database layer to reduce the likelihood and impact of similar incidents..</p> <p><small>Oct <var data-var='date'> 12</var>, <var data-var='time'>19:28:42</var> GMT+0</small><br><strong>Investigating</strong> - Web and email notifications will be delayed for 1 more hour as we are close to completing the backlog of notifications..</p> <p><small>Oct <var data-var='date'> 12</var>, <var data-var='time'>17:34:24</var> GMT+0</small><br><strong>Investigating</strong> - Web and email notifications continue to be delayed as we work through a backlog of notifications..</p> <p><small>Oct <var data-var='date'> 12</var>, <var data-var='time'>15:50:39</var> GMT+0</small><br><strong>Investigating</strong> - Notifications have resumed but will continue to be delayed as we work through a backlog..</p> <p><small>Oct <var data-var='date'> 12</var>, <var data-var='time'>15:19:00</var> GMT+0</small><br><strong>Investigating</strong> - We have successfully applied a mitigation for notifications, and notifications have resumed but will continue to be delayed as we work through a backlog..</p> <p><small>Oct <var data-var='date'> 12</var>, <var data-var='time'>14:37:49</var> GMT+0</small><br><strong>Investigating</strong> - We have mitigated the issues with Actions, issues, and projects, and have a mitigation in progress to address the delay with notifications..</p> <p><small>Oct <var data-var='date'> 12</var>, <var data-var='time'>14:36:09</var> GMT+0</small><br><strong>Investigating</strong> - The degradation affecting Actions has been mitigated. We are monitoring to ensure stability..</p> <p><small>Oct <var data-var='date'> 12</var>, <var data-var='time'>14:35:49</var> GMT+0</small><br><strong>Investigating</strong> - The degradation affecting Issues has been mitigated. We are monitoring to ensure stability..</p> <p><small>Oct <var data-var='date'> 12</var>, <var data-var='time'>13:38:44</var> GMT+0</small><br><strong>Investigating</strong> - Our initial mitigation was not fully successful, and users may still experience delays and errors with Actions, issues and projects, and notifications are delayed. We're continuing to investigate the root cause and work on additional mitigations..</p> <p><small>Oct <var data-var='date'> 12</var>, <var data-var='time'>13:02:17</var> GMT+0</small><br><strong>Investigating</strong> - Pull Requests is operating normally..</p> <p><small>Oct <var data-var='date'> 12</var>, <var data-var='time'>12:57:44</var> GMT+0</small><br><strong>Investigating</strong> - We have a mitigation in progress and are monitoring for recovery. Users may also experience delayed web notifications as we mitigate..</p> <p><small>Oct <var data-var='date'> 12</var>, <var data-var='time'>12:33:21</var> GMT+0</small><br><strong>Investigating</strong> - Users may experience issues loading the Actions tab UI, interacting with the Checks API, queueing new Actions runs, creating and updating issues, and requesting PR reviews, including via REST and GraphQL APIs..</p> <p><small>Oct <var data-var='date'> 12</var>, <var data-var='time'>12:21:48</var> GMT+0</small><br><strong>Investigating</strong> - We are investigating reports of degraded performance for Actions, Issues and Pull Requests.</p> ]]> </content:encoded> <pubDate>Mon, 12 Oct 2026 12:21:47 +0000</pubDate> <link>https://status.macroscope.com/incident/cmvhferca01yd0ws81rcqt9v8</link> <guid>https://status.macroscope.com/incident/cmvhferca01yd0ws81rcqt9v8</guid> </item> <item> <title>[automated] Macroscope API — Degraded Performance</title> <description> Type: Incident Oct 11, 05:55:04 GMT+0 - Resolved - This issue has been resolved. The Macroscope API is operating normally. We apologize for any inconvenience. Oct 11, 05:50:12 GMT+0 - Investigating - The Macroscope API is experiencing degraded performance. We are investigating and will provide updates. </description> <content:encoded> <![CDATA[<p><strong>Type:</strong> Incident</p> <p><small>Oct <var data-var='date'> 11</var>, <var data-var='time'>05:55:04</var> GMT+0</small><br><strong>Resolved</strong> - This issue has been resolved. The Macroscope API is operating normally. We apologize for any inconvenience..</p> <p><small>Oct <var data-var='date'> 11</var>, <var data-var='time'>05:50:12</var> GMT+0</small><br><strong>Investigating</strong> - The Macroscope API is experiencing degraded performance. We are investigating and will provide updates..</p> ]]> </content:encoded> <pubDate>Sun, 11 Oct 2026 05:50:12 +0000</pubDate> <link>https://status.macroscope.com/incident/cmvhf29im048u1cplbkjy6v4f-k6w10rl292j2</link> <guid>https://status.macroscope.com/incident/cmvhf29im048u1cplbkjy6v4f-k6w10rl292j2</guid> </item> <item> <title>[automated] Macroscope API — Degraded Performance</title> <description> Type: Incident Duration: 5 minutes Affected Components: Agent, Status Oct 11, 05:55:00 GMT+0 - Resolved - We are continuing to work on a fix for this incident. Oct 11, 05:50:00 GMT+0 - Investigating - The Macroscope API is experiencing degraded performance. We are investigating and will provide updates. </description> <content:encoded> <![CDATA[<p><strong>Type:</strong> Incident</p> <p><strong>Duration:</strong> 5 minutes</p> <p><strong>Affected Components:</strong> </p> <p><small>Oct <var data-var='date'> 11</var>, <var data-var='time'>05:55:00</var> GMT+0</small><br><strong>Resolved</strong> - We are continuing to work on a fix for this incident..</p> <p><small>Oct <var data-var='date'> 11</var>, <var data-var='time'>05:50:00</var> GMT+0</small><br><strong>Investigating</strong> - The Macroscope API is experiencing degraded performance. We are investigating and will provide updates..</p> ]]> </content:encoded> <pubDate>Sun, 11 Oct 2026 05:50:00 +0000</pubDate> <link>https://status.macroscope.com/incident/cmvl8diwf0b2s1mlnuqw3idf5</link> <guid>https://status.macroscope.com/incident/cmvl8diwf0b2s1mlnuqw3idf5</guid> </item> <item> <title>Incident with Git Operations, Issues, Actions and Pull Requests</title> <description> Type: Incident Duration: 1 day, 23 hours and 43 minutes Affected Components: GitHub Actions Oct 7, 18:04:49 GMT+0 - Resolved - On October 7, 2026, from 16:52 to 17:01 UTC, GitHub experienced a recurrence of the write-operation outage from earlier that day. Both outages shared the same root cause. Please refer to the combined summary covering both incidents, their impact, and our prevention measures: https://www.githubstatus.com/incidents/djlmxz2zd0j7 Oct 7, 18:00:49 GMT+0 - Monitoring - All systems have recovered. We have applied mitigations and do not anticipate another recurrence. Oct 7, 17:27:12 GMT+0 - Monitoring - The degradation has been mitigated. We are monitoring to ensure stability. Oct 7, 17:25:26 GMT+0 - Investigating - We observed widespread impact across GitHub services from 16:52 UTC to 17:01 UTC, and are investigating the root cause. All systems have recovered. This incident is a reoccurrence of the incident from earlier in the day https://www.githubstatus.com/incidents/djlmxz2zd0j7. Durable mitigation is in progress. We will provide another update in 30 minutes. Oct 7, 17:17:05 GMT+0 - Investigating - We are investigating reports of degraded performance for Actions, Git Operations, Issues and Pull Requests </description> <content:encoded> <![CDATA[<p><strong>Type:</strong> Incident</p> <p><strong>Duration:</strong> 1 day, 23 hours and 43 minutes</p> <p><strong>Affected Components:</strong> </p> <p><small>Oct <var data-var='date'> 7</var>, <var data-var='time'>18:04:49</var> GMT+0</small><br><strong>Resolved</strong> - On October 7, 2026, from 16:52 to 17:01 UTC, GitHub experienced a recurrence of the write-operation outage from earlier that day. Both outages shared the same root cause. Please refer to the combined summary covering both incidents, their impact, and our prevention measures: https://www.githubstatus.com/incidents/djlmxz2zd0j7.</p> <p><small>Oct <var data-var='date'> 7</var>, <var data-var='time'>18:00:49</var> GMT+0</small><br><strong>Monitoring</strong> - All systems have recovered. We have applied mitigations and do not anticipate another recurrence..</p> <p><small>Oct <var data-var='date'> 7</var>, <var data-var='time'>17:27:12</var> GMT+0</small><br><strong>Monitoring</strong> - The degradation has been mitigated. We are monitoring to ensure stability..</p> <p><small>Oct <var data-var='date'> 7</var>, <var data-var='time'>17:25:26</var> GMT+0</small><br><strong>Investigating</strong> - We observed widespread impact across GitHub services from 16:52 UTC to 17:01 UTC, and are investigating the root cause. All systems have recovered. This incident is a reoccurrence of the incident from earlier in the day https://www.githubstatus.com/incidents/djlmxz2zd0j7. Durable mitigation is in progress. We will provide another update in 30 minutes..</p> <p><small>Oct <var data-var='date'> 7</var>, <var data-var='time'>17:17:05</var> GMT+0</small><br><strong>Investigating</strong> - We are investigating reports of degraded performance for Actions, Git Operations, Issues and Pull Requests.</p> ]]> </content:encoded> <pubDate>Wed, 7 Oct 2026 17:17:05 +0000</pubDate> <link>https://status.macroscope.com/incident/cmvhfer1v01y90ws8fij58zcm</link> <guid>https://status.macroscope.com/incident/cmvhfer1v01y90ws8fij58zcm</guid> </item> <item> <title>Incident with Git Operations, Pull Requests and Actions</title> <description> Type: Incident Affected Components: GitHub Actions Oct 7, 16:25:14 GMT+0 - Resolved - On October 7, 2026, GitHub had two outages of write operations with the same cause, totaling about 19 minutes. The first ran from 15:06 to 15:16 UTC (about 10 minutes) and the second from 16:52 to 17:01 UTC (about 9 minutes). In both windows, actions that save or change data failed across Git operations, pull requests, issues, GitHub Actions, the API, webhooks, sign-in and authentication, GitHub Apps and OAuth, code scanning, Codespaces, and Copilot. Reading data was largely unaffected. In the first outage, 5.6% of all web and API requests failed, peaking at 8.9%, and more than 12,000 organizations saw failed requests. Up to 26% of Git authentication requests failed, and 17% of Actions workflow runs failed with infrastructure errors. In the second outage, 6.2% of web and API requests failed, peaking at 9.3%, and more than 11,000 organizations were affected. Up to 25% of Git authentication requests failed, and 19% of Actions workflow runs failed. No databases experienced data loss, and operations retried after recovery succeeded. GitHub relies on an internal service that keeps track of which database servers should receive new and updated data. While replacing machines that run this service as part of regular maintenance, one of the new machines started with an outdated copy of its records, which still listed database servers we had already retired. When this node came online, it incorrectly modified routing configuration with the outdated records, routing write requests to non-existent database servers. Each outage ended when one of our established machines with correct configuration became the leader of the quorum and sent the correct configuration. After the first outage, we identified the new machine as the source and paused automated maintenance to stop it from being leader again. At 16:52 UTC, an automated maintenance task that did not honor the pause put it back in charge, and the outage happened again. We shut that machine down at 17:02 UTC and have since removed it, and the automation now honors the pause, which was a new regression due to an update of our puppet infrastructure. To prevent recurrence, we are clearing retired database servers from the service's records before any new machine is added, and we are keeping new machines from being elected leader until their records have been checked. We are also changing our systems to reject configuration that points to servers that don't exist and to keep using the last known-good settings instead. Finally, we are giving responders a faster way to take a misbehaving machine out of service, and adding alerts so changes like this are spotted within seconds. We will share a more detailed write-up of these incidents in our monthly availability report, along with the status of the repairs we are making to prevent issues like these in the future. Oct 7, 16:23:44 GMT+0 - Monitoring - Git Operations, Pull Requests and Actions are fully recovered and stable. The root cause has been identified and mitigations are being applied to prevent a reoccurrence. Oct 7, 15:58:35 GMT+0 - Monitoring - We are seeing full recovery across all systems including Git Operations, Pull Requests and Actions. We are continuing to investigate the root cause of the disruption between 15:06 UTC and 15:16 UTC, and will share another update shortly. Oct 7, 15:56:17 GMT+0 - Monitoring - The degradation has been mitigated. We are monitoring to ensure stability. Oct 7, 15:49:57 GMT+0 - Monitoring - The degradation has been mitigated. We are monitoring to ensure stability. Oct 7, 15:49:53 GMT+0 - Investigating - The degradation affecting Git Operations and Webhooks has been mitigated. We are monitoring to ensure stability. Oct 7, 15:47:07 GMT+0 - Investigating - The degradation affecting Issues has been mitigated. We are monitoring to ensure stability. Oct 7, 15:41:29 GMT+0 - Investigating - We are observing partial recovery across all systems. We are continuing to monitor and investigate the root cause of the disruption. Oct 7, 15:35:30 GMT+0 - Investigating - Git Operations is experiencing degraded performance. We are continuing to investigate. Oct 7, 15:32:02 GMT+0 - Investigating - Actions is operating normally. Oct 7, 15:31:55 GMT+0 - Investigating - The degradation affecting Pull Requests has been mitigated. We are monitoring to ensure stability. Oct 7, 15:27:28 GMT+0 - Investigating - Webhooks is experiencing degraded performance. We are continuing to investigate. Oct 7, 15:27:16 GMT+0 - Investigating - Git Operations is operating normally. Oct 7, 15:25:42 GMT+0 - Investigating - Pull Requests is experiencing degraded performance. We are continuing to investigate. Oct 7, 15:24:11 GMT+0 - Investigating - We observed widespread impact across GitHub services between 15:06 UTC and 15:16 UTC, and are investigating the root cause. We are seeing all systems recover, and will provide another update shortly. Oct 7, 15:21:36 GMT+0 - Investigating - Git Operations is experiencing degraded performance. We are continuing to investigate. Oct 7, 15:16:03 GMT+0 - Investigating - Webhooks is experiencing degraded availability. We are continuing to investigate. Oct 7, 15:15:18 GMT+0 - Investigating - Issues is experiencing degraded performance. We are continuing to investigate. Oct 7, 15:14:55 GMT+0 - Investigating - Webhooks is experiencing degraded performance. We are continuing to investigate. Oct 7, 15:14:45 GMT+0 - Investigating - We are investigating reports of degraded availability for Actions, Git Operations and Pull Requests </description> <content:encoded> <![CDATA[<p><strong>Type:</strong> Incident</p> <p><strong>Affected Components:</strong> </p> <p><small>Oct <var data-var='date'> 7</var>, <var data-var='time'>16:25:14</var> GMT+0</small><br><strong>Resolved</strong> - On October 7, 2026, GitHub had two outages of write operations with the same cause, totaling about 19 minutes. The first ran from 15:06 to 15:16 UTC (about 10 minutes) and the second from 16:52 to 17:01 UTC (about 9 minutes). In both windows, actions that save or change data failed across Git operations, pull requests, issues, GitHub Actions, the API, webhooks, sign-in and authentication, GitHub Apps and OAuth, code scanning, Codespaces, and Copilot. Reading data was largely unaffected. In the first outage, 5.6% of all web and API requests failed, peaking at 8.9%, and more than 12,000 organizations saw failed requests. Up to 26% of Git authentication requests failed, and 17% of Actions workflow runs failed with infrastructure errors. In the second outage, 6.2% of web and API requests failed, peaking at 9.3%, and more than 11,000 organizations were affected. Up to 25% of Git authentication requests failed, and 19% of Actions workflow runs failed. No databases experienced data loss, and operations retried after recovery succeeded. GitHub relies on an internal service that keeps track of which database servers should receive new and updated data. While replacing machines that run this service as part of regular maintenance, one of the new machines started with an outdated copy of its records, which still listed database servers we had already retired. When this node came online, it incorrectly modified routing configuration with the outdated records, routing write requests to non-existent database servers. Each outage ended when one of our established machines with correct configuration became the leader of the quorum and sent the correct configuration. After the first outage, we identified the new machine as the source and paused automated maintenance to stop it from being leader again. At 16:52 UTC, an automated maintenance task that did not honor the pause put it back in charge, and the outage happened again. We shut that machine down at 17:02 UTC and have since removed it, and the automation now honors the pause, which was a new regression due to an update of our puppet infrastructure. To prevent recurrence, we are clearing retired database servers from the service's records before any new machine is added, and we are keeping new machines from being elected leader until their records have been checked. We are also changing our systems to reject configuration that points to servers that don't exist and to keep using the last known-good settings instead. Finally, we are giving responders a faster way to take a misbehaving machine out of service, and adding alerts so changes like this are spotted within seconds. We will share a more detailed write-up of these incidents in our monthly availability report, along with the status of the repairs we are making to prevent issues like these in the future..</p> <p><small>Oct <var data-var='date'> 7</var>, <var data-var='time'>16:23:44</var> GMT+0</small><br><strong>Monitoring</strong> - Git Operations, Pull Requests and Actions are fully recovered and stable. The root cause has been identified and mitigations are being applied to prevent a reoccurrence..</p> <p><small>Oct <var data-var='date'> 7</var>, <var data-var='time'>15:58:35</var> GMT+0</small><br><strong>Monitoring</strong> - We are seeing full recovery across all systems including Git Operations, Pull Requests and Actions. We are continuing to investigate the root cause of the disruption between 15:06 UTC and 15:16 UTC, and will share another update shortly..</p> <p><small>Oct <var data-var='date'> 7</var>, <var data-var='time'>15:56:17</var> GMT+0</small><br><strong>Monitoring</strong> - The degradation has been mitigated. We are monitoring to ensure stability..</p> <p><small>Oct <var data-var='date'> 7</var>, <var data-var='time'>15:49:57</var> GMT+0</small><br><strong>Monitoring</strong> - The degradation has been mitigated. We are monitoring to ensure stability..</p> <p><small>Oct <var data-var='date'> 7</var>, <var data-var='time'>15:49:53</var> GMT+0</small><br><strong>Investigating</strong> - The degradation affecting Git Operations and Webhooks has been mitigated. We are monitoring to ensure stability..</p> <p><small>Oct <var data-var='date'> 7</var>, <var data-var='time'>15:47:07</var> GMT+0</small><br><strong>Investigating</strong> - The degradation affecting Issues has been mitigated. We are monitoring to ensure stability..</p> <p><small>Oct <var data-var='date'> 7</var>, <var data-var='time'>15:41:29</var> GMT+0</small><br><strong>Investigating</strong> - We are observing partial recovery across all systems. We are continuing to monitor and investigate the root cause of the disruption..</p> <p><small>Oct <var data-var='date'> 7</var>, <var data-var='time'>15:35:30</var> GMT+0</small><br><strong>Investigating</strong> - Git Operations is experiencing degraded performance. We are continuing to investigate..</p> <p><small>Oct <var data-var='date'> 7</var>, <var data-var='time'>15:32:02</var> GMT+0</small><br><strong>Investigating</strong> - Actions is operating normally..</p> <p><small>Oct <var data-var='date'> 7</var>, <var data-var='time'>15:31:55</var> GMT+0</small><br><strong>Investigating</strong> - The degradation affecting Pull Requests has been mitigated. We are monitoring to ensure stability..</p> <p><small>Oct <var data-var='date'> 7</var>, <var data-var='time'>15:27:28</var> GMT+0</small><br><strong>Investigating</strong> - Webhooks is experiencing degraded performance. We are continuing to investigate..</p> <p><small>Oct <var data-var='date'> 7</var>, <var data-var='time'>15:27:16</var> GMT+0</small><br><strong>Investigating</strong> - Git Operations is operating normally..</p> <p><small>Oct <var data-var='date'> 7</var>, <var data-var='time'>15:25:42</var> GMT+0</small><br><strong>Investigating</strong> - Pull Requests is experiencing degraded performance. We are continuing to investigate..</p> <p><small>Oct <var data-var='date'> 7</var>, <var data-var='time'>15:24:11</var> GMT+0</small><br><strong>Investigating</strong> - We observed widespread impact across GitHub services between 15:06 UTC and 15:16 UTC, and are investigating the root cause. We are seeing all systems recover, and will provide another update shortly..</p> <p><small>Oct <var data-var='date'> 7</var>, <var data-var='time'>15:21:36</var> GMT+0</small><br><strong>Investigating</strong> - Git Operations is experiencing degraded performance. We are continuing to investigate..</p> <p><small>Oct <var data-var='date'> 7</var>, <var data-var='time'>15:16:03</var> GMT+0</small><br><strong>Investigating</strong> - Webhooks is experiencing degraded availability. We are continuing to investigate..</p> <p><small>Oct <var data-var='date'> 7</var>, <var data-var='time'>15:15:18</var> GMT+0</small><br><strong>Investigating</strong> - Issues is experiencing degraded performance. We are continuing to investigate..</p> <p><small>Oct <var data-var='date'> 7</var>, <var data-var='time'>15:14:55</var> GMT+0</small><br><strong>Investigating</strong> - Webhooks is experiencing degraded performance. We are continuing to investigate..</p> <p><small>Oct <var data-var='date'> 7</var>, <var data-var='time'>15:14:45</var> GMT+0</small><br><strong>Investigating</strong> - We are investigating reports of degraded availability for Actions, Git Operations and Pull Requests.</p> ]]> </content:encoded> <pubDate>Wed, 7 Oct 2026 16:25:14 +0000</pubDate> <link>https://status.macroscope.com/incident/cmvhfeqza01y80ws8dadq0yze</link> <guid>https://status.macroscope.com/incident/cmvhfeqza01y80ws8dadq0yze</guid> </item> <item> <title>Incident with Actions</title> <description> Type: Incident Duration: 9 days, 1 hour and 44 minutes Affected Components: GitHub Actions Oct 5, 22:49:42 GMT+0 - Resolved - On October 5, 2026, between 18:48 and 22:49 UTC, GitHub Actions and Hosted Runners experienced degraded performance. Over the full incident, 14.3% of workflow runs using GitHub-hosted runners and 26.5% of individual GitHub-hosted jobs did not start within five minutes. <br /><br /> <br />During the roughly 90-minute period of highest impact, 47.0% of workflow runs using GitHub-hosted runners failed and 71.9% of GitHub-hosted jobs did not start within five minutes, while existing Pages sites continued to serve normally. A portion of customers also encountered intermittent errors in repository lists, licensing and billing pages, and some Copilot features. Self-hosted runners were not affected. <br /><br /><br />The incident was caused by a partial networking failure that disrupted connectivity between one of our on-premises datacenter regions and a subset of cloud-hosted regional database and storage services. <br /><br /><br />We mitigated the incident by shifting database and storage traffic to healthy regions and endpoints and increasing capacity for affected compute services. GitHub Actions and Hosted Runners recovered by 21:54 UTC, and all affected services recovered by 22:49 UTC. <br /><br /> <br />We are partnering with our cloud provider to understand the network failure and identify improvements to network-path resilience, detection, and recovery for similar incidents. Oct 5, 22:40:41 GMT+0 - Investigating - Pages is operating normally. Oct 5, 21:54:23 GMT+0 - Investigating - Actions is operating normally. Oct 5, 21:32:31 GMT+0 - Investigating - We have applied mitigations to address GitHub Actions failures. Queued jobs are clearing, new jobs are not delayed, and we expect full runner recovery shortly. We continue to monitor for full recovery and are working to mitigate ongoing issues affecting other services, including access to repository lists, licensing, and billing pages. Oct 5, 21:31:18 GMT+0 - Investigating - Actions is experiencing degraded performance. We are continuing to investigate. Oct 5, 21:22:39 GMT+0 - Investigating - Pages is experiencing degraded performance. We are continuing to investigate. Oct 5, 21:09:15 GMT+0 - Investigating - In addition to ongoing issues with Actions and Hosted Runners, some customers are also unable to access repository list, licensing and billing pages within their GitHub accounts. We continue to investigate and are working on mitigation. Oct 5, 20:47:22 GMT+0 - Investigating - Actions is experiencing degraded availability. We are continuing to investigate. Oct 5, 20:39:27 GMT+0 - Investigating - We’re continuing to investigate job failures and delays affecting GitHub-hosted runner assignment and workflow start times. Our teams are actively working to mitigate the impact and will provide another update as we learn more. Oct 5, 19:50:50 GMT+0 - Investigating - We’re still investigating delays in assigning GitHub-hosted runners, affecting workflow start times across multiple runner configurations. Our teams are working to mitigate the impact; we’ll share another update as we learn more. Oct 5, 19:15:17 GMT+0 - Investigating - We’re investigating an issue causing delays when assigning GitHub-hosted runners to Actions jobs. Some workflows may take longer to start across runner configurations. Our teams are working to mitigate the issue, and we’ll provide an update as we learn more. Oct 5, 19:11:58 GMT+0 - Investigating - We are investigating reports of degraded performance for Actions </description> <content:encoded> <![CDATA[<p><strong>Type:</strong> Incident</p> <p><strong>Duration:</strong> 9 days, 1 hour and 44 minutes</p> <p><strong>Affected Components:</strong> </p> <p><small>Oct <var data-var='date'> 5</var>, <var data-var='time'>22:49:42</var> GMT+0</small><br><strong>Resolved</strong> - On October 5, 2026, between 18:48 and 22:49 UTC, GitHub Actions and Hosted Runners experienced degraded performance. Over the full incident, 14.3% of workflow runs using GitHub-hosted runners and 26.5% of individual GitHub-hosted jobs did not start within five minutes. <br /><br /> <br />During the roughly 90-minute period of highest impact, 47.0% of workflow runs using GitHub-hosted runners failed and 71.9% of GitHub-hosted jobs did not start within five minutes, while existing Pages sites continued to serve normally. A portion of customers also encountered intermittent errors in repository lists, licensing and billing pages, and some Copilot features. Self-hosted runners were not affected. <br /><br /><br />The incident was caused by a partial networking failure that disrupted connectivity between one of our on-premises datacenter regions and a subset of cloud-hosted regional database and storage services. <br /><br /><br />We mitigated the incident by shifting database and storage traffic to healthy regions and endpoints and increasing capacity for affected compute services. GitHub Actions and Hosted Runners recovered by 21:54 UTC, and all affected services recovered by 22:49 UTC. <br /><br /> <br />We are partnering with our cloud provider to understand the network failure and identify improvements to network-path resilience, detection, and recovery for similar incidents..</p> <p><small>Oct <var data-var='date'> 5</var>, <var data-var='time'>22:40:41</var> GMT+0</small><br><strong>Investigating</strong> - Pages is operating normally..</p> <p><small>Oct <var data-var='date'> 5</var>, <var data-var='time'>21:54:23</var> GMT+0</small><br><strong>Investigating</strong> - Actions is operating normally..</p> <p><small>Oct <var data-var='date'> 5</var>, <var data-var='time'>21:32:31</var> GMT+0</small><br><strong>Investigating</strong> - We have applied mitigations to address GitHub Actions failures. Queued jobs are clearing, new jobs are not delayed, and we expect full runner recovery shortly. We continue to monitor for full recovery and are working to mitigate ongoing issues affecting other services, including access to repository lists, licensing, and billing pages..</p> <p><small>Oct <var data-var='date'> 5</var>, <var data-var='time'>21:31:18</var> GMT+0</small><br><strong>Investigating</strong> - Actions is experiencing degraded performance. We are continuing to investigate..</p> <p><small>Oct <var data-var='date'> 5</var>, <var data-var='time'>21:22:39</var> GMT+0</small><br><strong>Investigating</strong> - Pages is experiencing degraded performance. We are continuing to investigate..</p> <p><small>Oct <var data-var='date'> 5</var>, <var data-var='time'>21:09:15</var> GMT+0</small><br><strong>Investigating</strong> - In addition to ongoing issues with Actions and Hosted Runners, some customers are also unable to access repository list, licensing and billing pages within their GitHub accounts. We continue to investigate and are working on mitigation..</p> <p><small>Oct <var data-var='date'> 5</var>, <var data-var='time'>20:47:22</var> GMT+0</small><br><strong>Investigating</strong> - Actions is experiencing degraded availability. We are continuing to investigate..</p> <p><small>Oct <var data-var='date'> 5</var>, <var data-var='time'>20:39:27</var> GMT+0</small><br><strong>Investigating</strong> - We’re continuing to investigate job failures and delays affecting GitHub-hosted runner assignment and workflow start times. Our teams are actively working to mitigate the impact and will provide another update as we learn more..</p> <p><small>Oct <var data-var='date'> 5</var>, <var data-var='time'>19:50:50</var> GMT+0</small><br><strong>Investigating</strong> - We’re still investigating delays in assigning GitHub-hosted runners, affecting workflow start times across multiple runner configurations. Our teams are working to mitigate the impact; we’ll share another update as we learn more..</p> <p><small>Oct <var data-var='date'> 5</var>, <var data-var='time'>19:15:17</var> GMT+0</small><br><strong>Investigating</strong> - We’re investigating an issue causing delays when assigning GitHub-hosted runners to Actions jobs. Some workflows may take longer to start across runner configurations. Our teams are working to mitigate the issue, and we’ll provide an update as we learn more..</p> <p><small>Oct <var data-var='date'> 5</var>, <var data-var='time'>19:11:58</var> GMT+0</small><br><strong>Investigating</strong> - We are investigating reports of degraded performance for Actions.</p> ]]> </content:encoded> <pubDate>Mon, 5 Oct 2026 19:11:58 +0000</pubDate> <link>https://status.macroscope.com/incident/cmvhfeqvv01y50ws8h7299lvs</link> <guid>https://status.macroscope.com/incident/cmvhfeqvv01y50ws8h7299lvs</guid> </item> </channel> </rss>