Snapshot 17317
Normalized text
Scripts and page chrome removed; this is what change detection compares.
<?xml version="1.0" encoding="UTF-8"?> <rss version="2.0" xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/"> <channel> <title>blacksmith Status - Incident history</title> <link>https://status.blacksmith.sh</link> <description>blacksmith</description> <pubDate>Tue, 29 Sep 2026 17:00:21 +0000</pubDate> <item> <title>US west cache failure</title> <description> Type: Incident Duration: 1 hour and 32 minutes Affected Components: US West Cache Sep 29, 17:09:28 GMT+0 - Identified - We have identified the cause of the failures, and we are implementing a fix to restore cache service. Customers with jobs in us-west may still see cache restores and saves fail and fall back to a full install, which can make those jobs run longer or fail, while jobs in other regions are not affected. Sep 29, 17:00:21 GMT+0 - Investigating - We are investigating failing GitHub Actions cache operations for jobs running in us-west since approximately 16:10 UTC. Affected jobs may see cache restores and saves fail and fall back to a full install, which can make those jobs run longer or fail, while jobs in other regions are not affected. Sep 29, 17:35:30 GMT+0 - Monitoring - We have restored the cache infrastructure in us-west, and cache failures have decreased but are not yet back to normal. Jobs in us-west may still see some cache restores and saves fail and fall back to a full install, while the earlier delayed job starts have cleared and other regions are unaffected. We are bringing additional cache capacity online to fully restore service. Sep 29, 18:06:52 GMT+0 - Monitoring - Cache errors for jobs in us-west have largely subsided. As we complete recovery, some jobs may see a one-time cache miss on their next run. We are monitoring and will provide an update within the next hour. Sep 29, 18:32:38 GMT+0 - Resolved - This incident is resolved. GitHub Actions cache operations for jobs in us-west have returned to normal, and any jobs that failed during the incident can be re-run. </description> <content:encoded> <![CDATA[<p><strong>Type:</strong> Incident</p> <p><strong>Duration:</strong> 1 hour and 32 minutes</p> <p><strong>Affected Components:</strong> </p> <p><small>Sep <var data-var='date'> 29</var>, <var data-var='time'>17:09:28</var> GMT+0</small><br><strong>Identified</strong> - We have identified the cause of the failures, and we are implementing a fix to restore cache service. Customers with jobs in us-west may still see cache restores and saves fail and fall back to a full install, which can make those jobs run longer or fail, while jobs in other regions are not affected..</p> <p><small>Sep <var data-var='date'> 29</var>, <var data-var='time'>17:00:21</var> GMT+0</small><br><strong>Investigating</strong> - We are investigating failing GitHub Actions cache operations for jobs running in us-west since approximately 16:10 UTC. Affected jobs may see cache restores and saves fail and fall back to a full install, which can make those jobs run longer or fail, while jobs in other regions are not affected. .</p> <p><small>Sep <var data-var='date'> 29</var>, <var data-var='time'>17:35:30</var> GMT+0</small><br><strong>Monitoring</strong> - We have restored the cache infrastructure in us-west, and cache failures have decreased but are not yet back to normal. Jobs in us-west may still see some cache restores and saves fail and fall back to a full install, while the earlier delayed job starts have cleared and other regions are unaffected. We are bringing additional cache capacity online to fully restore service..</p> <p><small>Sep <var data-var='date'> 29</var>, <var data-var='time'>18:06:52</var> GMT+0</small><br><strong>Monitoring</strong> - Cache errors for jobs in us-west have largely subsided. As we complete recovery, some jobs may see a one-time cache miss on their next run. We are monitoring and will provide an update within the next hour..</p> <p><small>Sep <var data-var='date'> 29</var>, <var data-var='time'>18:32:38</var> GMT+0</small><br><strong>Resolved</strong> - This incident is resolved. GitHub Actions cache operations for jobs in us-west have returned to normal, and any jobs that failed during the incident can be re-run..</p> ]]> </content:encoded> <pubDate>Tue, 29 Sep 2026 17:00:21 +0000</pubDate> <link>https://status.blacksmith.sh/incident/cmumx9ti905cd0wo4apnryc73</link> <guid>https://status.blacksmith.sh/incident/cmumx9ti905cd0wo4apnryc73</guid> </item> <item> <title>GitHub connectivity degraded in us-west</title> <description> Type: Incident Duration: 33 minutes Affected Components: us-west x86, us-west ARM Sep 24, 19:31:46 GMT+0 - Monitoring - After rerouting traffic, congestion on the upstream network link has subsided, and git checkout performance in us-west has returned to normal. We are monitoring and will resolve this incident once performance has continued to stay stable. Sep 24, 19:16:56 GMT+0 - Investigating - We are investigating degraded network connectivity between our us-west region and GitHub. Customers running jobs in us-west may see slower-than-normal git checkouts, while jobs in other regions are not affected. Workflows using the Blacksmith checkout action with git checkout caching are less affected, and setup instructions are in the documentation below. Git checkout caching documentation: <https://docs.blacksmith.sh/blacksmith-caching/git-checkout-caching> Sep 24, 19:50:10 GMT+0 - Resolved - Git checkout performance in us-west has remained stable since we rerouted traffic away from the congested upstream network link, and this incident is now resolved. </description> <content:encoded> <![CDATA[<p><strong>Type:</strong> Incident</p> <p><strong>Duration:</strong> 33 minutes</p> <p><strong>Affected Components:</strong> , </p> <p><small>Sep <var data-var='date'> 24</var>, <var data-var='time'>19:31:46</var> GMT+0</small><br><strong>Monitoring</strong> - After rerouting traffic, congestion on the upstream network link has subsided, and git checkout performance in us-west has returned to normal. We are monitoring and will resolve this incident once performance has continued to stay stable. .</p> <p><small>Sep <var data-var='date'> 24</var>, <var data-var='time'>19:16:56</var> GMT+0</small><br><strong>Investigating</strong> - We are investigating degraded network connectivity between our us-west region and GitHub. Customers running jobs in us-west may see slower-than-normal git checkouts, while jobs in other regions are not affected. Workflows using the Blacksmith checkout action with git checkout caching are less affected, and setup instructions are in the documentation below. Git checkout caching documentation: <https://docs.blacksmith.sh/blacksmith-caching/git-checkout-caching>.</p> <p><small>Sep <var data-var='date'> 24</var>, <var data-var='time'>19:50:10</var> GMT+0</small><br><strong>Resolved</strong> - Git checkout performance in us-west has remained stable since we rerouted traffic away from the congested upstream network link, and this incident is now resolved..</p> ]]> </content:encoded> <pubDate>Thu, 24 Sep 2026 19:16:56 +0000</pubDate> <link>https://status.blacksmith.sh/incident/cmufwy7mx0ou71mqm2si72dxw</link> <guid>https://status.blacksmith.sh/incident/cmufwy7mx0ou71mqm2si72dxw</guid> </item> <item> <title>Elevated error rates in Blacksmith APIs</title> <description> Type: Incident Duration: 9 minutes Affected Components: us-east x86, US East Cache, eu-west Storage Cluster, eu-central x86, eu-west ARM, us-west Storage Cluster, EU Central Cache, us-west x86, eu-west x86, EU West Cache, eu-west Storage Cluster, eu-west Storage Cluster, API, eu-central Storage Cluster, us-west ARM, EU West Runtime Build Cache, eu-central Storage Cluster, eu-central Storage Cluster, EU Central Runtime Build Cache, US West Runtime Build Cache, US West Cache, us-central MacOS, eu-central ARM, us-west Storage Cluster, us-west Storage Cluster Sep 21, 18:37:44 GMT+0 - Identified - We are seeing elevated errors in job adoption and other control plane APIs. This is yielding higher latencies for your job to be adopted. You may also see certain actions fail, such as actions caching and sticky disks. We have identified the issue and are rolling out a mitigation. Sep 21, 18:47:13 GMT+0 - Resolved - This issue has been resolved, and all services have operating normally. If any of your jobs failed during this time, please re-run them. </description> <content:encoded> <![CDATA[<p><strong>Type:</strong> Incident</p> <p><strong>Duration:</strong> 9 minutes</p> <p><strong>Affected Components:</strong> , , , , , , , , , , , , , , , , , , , , , , , , </p> <p><small>Sep <var data-var='date'> 21</var>, <var data-var='time'>18:37:44</var> GMT+0</small><br><strong>Identified</strong> - We are seeing elevated errors in job adoption and other control plane APIs. This is yielding higher latencies for your job to be adopted. You may also see certain actions fail, such as actions caching and sticky disks. We have identified the issue and are rolling out a mitigation..</p> <p><small>Sep <var data-var='date'> 21</var>, <var data-var='time'>18:47:13</var> GMT+0</small><br><strong>Resolved</strong> - This issue has been resolved, and all services have operating normally. If any of your jobs failed during this time, please re-run them..</p> ]]> </content:encoded> <pubDate>Mon, 21 Sep 2026 18:37:44 +0000</pubDate> <link>https://status.blacksmith.sh/incident/cmubl88jr16480wmrgp8hu01n</link> <guid>https://status.blacksmith.sh/incident/cmubl88jr16480wmrgp8hu01n</guid> </item> <item> <title>Degraded US East AWS connectivity</title> <description> Type: Incident Duration: 47 minutes Affected Components: us-east x86 Sep 13, 01:11:55 GMT+0 - Investigating - We are currently observing network stalling in our US East region with runner connectivity against AWS's us-east-1 region. Uploads and downloads to and from ECR and S3 may experience occasional stalling. We are investigating. Sep 13, 01:40:50 GMT+0 - Monitoring - We have identified a small subset of runners that are exhibiting this network stalling and have isolated them. We're continuing to monitor recovery. Sep 13, 01:58:30 GMT+0 - Resolved - This incident has been resolved. </description> <content:encoded> <![CDATA[<p><strong>Type:</strong> Incident</p> <p><strong>Duration:</strong> 47 minutes</p> <p><strong>Affected Components:</strong> </p> <p><small>Sep <var data-var='date'> 13</var>, <var data-var='time'>01:11:55</var> GMT+0</small><br><strong>Investigating</strong> - We are currently observing network stalling in our US East region with runner connectivity against AWS's us-east-1 region. Uploads and downloads to and from ECR and S3 may experience occasional stalling. We are investigating..</p> <p><small>Sep <var data-var='date'> 13</var>, <var data-var='time'>01:40:50</var> GMT+0</small><br><strong>Monitoring</strong> - We have identified a small subset of runners that are exhibiting this network stalling and have isolated them. We're continuing to monitor recovery..</p> <p><small>Sep <var data-var='date'> 13</var>, <var data-var='time'>01:58:30</var> GMT+0</small><br><strong>Resolved</strong> - This incident has been resolved..</p> ]]> </content:encoded> <pubDate>Sun, 13 Sep 2026 01:11:55 +0000</pubDate> <link>https://status.blacksmith.sh/incident/cmtz4cib8018b1mmvp68b8fy4</link> <guid>https://status.blacksmith.sh/incident/cmtz4cib8018b1mmvp68b8fy4</guid> </item> <item> <title>Upstream Ubuntu package mirror outage affecting apt </title> <description> Type: Incident Duration: 9 hours and 30 minutes Sep 11, 05:30:00 GMT+0 - Monitoring - Since 06:30 UTC Canonical has had three outages affecting [archive.ubuntu.com](http://archive.ubuntu.com), [us.archive.ubuntu.com](http://us.archive.ubuntu.com) and [security.ubuntu.com](http://security.ubuntu.com). Although they have marked these as resolved, we are still seeing intermittent connection failures to those mirrors, which can cause apt-get steps to fail or hang. In the meantime, pointing apt at [azure.archive.ubuntu.com](http://azure.archive.ubuntu.com) instead of [archive.ubuntu.com](http://archive.ubuntu.com) and [security.ubuntu.com](http://security.ubuntu.com) will unblock affected jobs. Sep 11, 14:21:02 GMT+0 - Monitoring - We are deploying a mitigation on our side that routes apt around the affected Canonical mirrors. For the time being pointing apt at [azure.archive.ubuntu.com](http://azure.archive.ubuntu.com) in place of [archive.ubuntu.com](http://archive.ubuntu.com) and [security.ubuntu.com](http://security.ubuntu.com) will unblock affected jobs. Sep 11, 14:59:46 GMT+0 - Resolved - This incident has been resolved - Canonical's Ubuntu apt mirrors have recovered and apt installs are succeeding again. We are also rolling out an image change to reduce the impact of upstream mirror outages in future. </description> <content:encoded> <, [us.archive.ubuntu.com](http://us.archive.ubuntu.com) and [security.ubuntu.com](http://security.ubuntu.com). Although they have marked these as resolved, we are still seeing intermittent connection failures to those mirrors, which can cause apt-get steps to fail or hang. In the meantime, pointing apt at [azure.archive.ubuntu.com](http://azure.archive.ubuntu.com) instead of [archive.ubuntu.com](http://archive.ubuntu.com) and [security.ubuntu.com](http://security.ubuntu.com) will unblock affected jobs..</p> <p><small>Sep <var data-var='date'> 11</var>, <var data-var='time'>14:21:02</var> GMT+0</small><br><strong>Monitoring</strong> - We are deploying a mitigation on our side that routes apt around the affected Canonical mirrors. For the time being pointing apt at [azure.archive.ubuntu.com](http://azure.archive.ubuntu.com) in place of [archive.ubuntu.com](http://archive.ubuntu.com) and [security.ubuntu.com](http://security.ubuntu.com) will unblock affected jobs..</p> <p><small>Sep <var data-var='date'> 11</var>, <var data-var='time'>14:59:46</var> GMT+0</small><br><strong>Resolved</strong> - This incident has been resolved - Canonical's Ubuntu apt mirrors have recovered and apt installs are succeeding again. We are also rolling out an image change to reduce the impact of upstream mirror outages in future..</p> ]]> </content:encoded> <pubDate>Fri, 11 Sep 2026 05:30:00 +0000</pubDate> <link>https://status.blacksmith.sh/incident/cmtx0883205f40wn0gg834kz2</link> <guid>https://status.blacksmith.sh/incident/cmtx0883205f40wn0gg834kz2</guid> </item> <item> <title>Major outage with our caching infrastructure and our website</title> <description> Type: Incident Duration: 1 hour and 20 minutes Affected Components: eu-west Storage Cluster, us-west Storage Cluster, EU Central Cache, EU West Cache, eu-west Storage Cluster, eu-west Storage Cluster, eu-central Storage Cluster, US East Cache, EU West Runtime Build Cache, Dashboard, eu-central Storage Cluster, eu-central Storage Cluster, EU Central Runtime Build Cache, US West Runtime Build Cache, US West Cache, us-west Storage Cluster, us-west Storage Cluster Sep 9, 16:46:00 GMT+0 - Investigating - We are currently experiencing a major outage with both our caching infrastructure and our dashboards. These components are affected across all regions. We are investigating the issue and will provide an update soon. Sep 9, 17:21:48 GMT+0 - Monitoring - We are seeing recovery of our caching infrastructure and our dashboard. We will continue to monitor this situation closely. Sep 9, 18:06:25 GMT+0 - Resolved - This incident has been resolved. </description> <content:encoded> <![CDATA[<p><strong>Type:</strong> Incident</p> <p><strong>Duration:</strong> 1 hour and 20 minutes</p> <p><strong>Affected Components:</strong> , , , , , , , , , , , , , , , , </p> <p><small>Sep <var data-var='date'> 9</var>, <var data-var='time'>16:46:00</var> GMT+0</small><br><strong>Investigating</strong> - We are currently experiencing a major outage with both our caching infrastructure and our dashboards. These components are affected across all regions. We are investigating the issue and will provide an update soon..</p> <p><small>Sep <var data-var='date'> 9</var>, <var data-var='time'>17:21:48</var> GMT+0</small><br><strong>Monitoring</strong> - We are seeing recovery of our caching infrastructure and our dashboard. We will continue to monitor this situation closely..</p> <p><small>Sep <var data-var='date'> 9</var>, <var data-var='time'>18:06:25</var> GMT+0</small><br><strong>Resolved</strong> - This incident has been resolved..</p> ]]> </content:encoded> <pubDate>Wed, 9 Sep 2026 16:46:00 +0000</pubDate> <link>https://status.blacksmith.sh/incident/cmtuczghn0lh52cpq34fkf3zt</link> <guid>https://status.blacksmith.sh/incident/cmtuczghn0lh52cpq34fkf3zt</guid> </item> <item> <title>Job failures in EU West</title> <description> Type: Incident Duration: 1 hour and 44 minutes Affected Components: Github → API Requests Sep 8, 15:00:33 GMT+0 - Investigating - Since approximately 12:20 UTC, some jobs in our eu-west region have been failing with the GitHub error "The self-hosted runner lost communication with the server"; other regions are unaffected. Affected jobs can be safely re-run while we continue to investigate the underlying cause. Sep 8, 15:31:33 GMT+0 - Monitoring - Between approximately 12:20 and 15:15 UTC, some jobs in our eu-west region failed with the GitHub error "The self-hosted runner lost communication with the server"; other regions were unaffected. Job failures in eu-west returned to normal levels as of 15:15 UTC, and any affected jobs can be safely re-run while we continue to investigate the underlying cause. Sep 8, 16:44:09 GMT+0 - Resolved - This incident is resolved. Job failures in our eu-west region returned to normal levels at 15:15 UTC and have remained there since; any jobs that failed between approximately 12:20 and 15:15 UTC with the error "The self-hosted runner lost communication with the server" can be safely re-run. </description> <content:encoded> <![CDATA[<p><strong>Type:</strong> Incident</p> <p><strong>Duration:</strong> 1 hour and 44 minutes</p> <p><strong>Affected Components:</strong> </p> <p><small>Sep <var data-var='date'> 8</var>, <var data-var='time'>15:00:33</var> GMT+0</small><br><strong>Investigating</strong> - Since approximately 12:20 UTC, some jobs in our eu-west region have been failing with the GitHub error "The self-hosted runner lost communication with the server"; other regions are unaffected. Affected jobs can be safely re-run while we continue to investigate the underlying cause. .</p> <p><small>Sep <var data-var='date'> 8</var>, <var data-var='time'>15:31:33</var> GMT+0</small><br><strong>Monitoring</strong> - Between approximately 12:20 and 15:15 UTC, some jobs in our eu-west region failed with the GitHub error "The self-hosted runner lost communication with the server"; other regions were unaffected. Job failures in eu-west returned to normal levels as of 15:15 UTC, and any affected jobs can be safely re-run while we continue to investigate the underlying cause..</p> <p><small>Sep <var data-var='date'> 8</var>, <var data-var='time'>16:44:09</var> GMT+0</small><br><strong>Resolved</strong> - This incident is resolved. Job failures in our eu-west region returned to normal levels at 15:15 UTC and have remained there since; any jobs that failed between approximately 12:20 and 15:15 UTC with the error "The self-hosted runner lost communication with the server" can be safely re-run..</p> ]]> </content:encoded> <pubDate>Tue, 8 Sep 2026 15:00:33 +0000</pubDate> <link>https://status.blacksmith.sh/incident/cmtsv10ho09c30wpqqz4dg4o8</link> <guid>https://status.blacksmith.sh/incident/cmtsv10ho09c30wpqqz4dg4o8</guid> </item> <item> <title>US West storage cluster degradation</title> <description> Type: Incident Duration: 3 hours and 10 minutes Affected Components: us-west Storage Cluster, us-west Storage Cluster, us-west Storage Cluster Sep 2, 16:13:47 GMT+0 - Investigating - We are investigating reports of higher latency on sticky disk operations in our us-west region. Customers running jobs in us-west may see slower incremental Docker builds, Git caching, and container caching, so affected jobs can take longer than usual to complete. Sep 2, 16:48:35 GMT+0 - Investigating - Sticky disk storage in our us-west region is experiencing higher latency; our other regions are not affected. Customers running jobs in us-west may see slower Git-cached checkouts, incremental Docker builds, and container caching. We are continuing to investigate the underlying cause and will provide another update within the next 30 minutes. Sep 2, 17:22:52 GMT+0 - Investigating - We are still working to resolve elevated latency on sticky disk storage in our us-west region; other regions are not affected. Customers running jobs in us-west may continue to see slower Git-cached checkouts, incremental Docker builds, and container caching. Our investigation into the underlying cause is ongoing and we will provide another update within the next 30 minutes. Sep 2, 18:10:23 GMT+0 - Monitoring - Our us-west region storage cluster's latency and error rates have returned to baseline as of approximately 17:15 UTC, following mitigations that reduce load on the affected storage. Git-cached checkouts, incremental Docker builds, and container caching in us-west are back to normal, and we are monitoring to confirm the recovery holds while we continue to investigate the underlying cause. We will provide a final update within the next hour. Sep 2, 19:23:31 GMT+0 - Resolved - This incident is resolved. Sticky disk storage in our us-west region came under more read load than it could serve at normal latency, which slowed and produced higher error rates for Git-cached checkouts, incremental Docker builds, and container caching. We reduced and redistributed that load, and performance has been normal since approximately 17:15 UTC. Jobs that failed during the incident can be safely re-run. </description> <content:encoded> <![CDATA[<p><strong>Type:</strong> Incident</p> <p><strong>Duration:</strong> 3 hours and 10 minutes</p> <p><strong>Affected Components:</strong> , , </p> <p><small>Sep <var data-var='date'> 2</var>, <var data-var='time'>16:13:47</var> GMT+0</small><br><strong>Investigating</strong> - We are investigating reports of higher latency on sticky disk operations in our us-west region. Customers running jobs in us-west may see slower incremental Docker builds, Git caching, and container caching, so affected jobs can take longer than usual to complete. .</p> <p><small>Sep <var data-var='date'> 2</var>, <var data-var='time'>16:48:35</var> GMT+0</small><br><strong>Investigating</strong> - Sticky disk storage in our us-west region is experiencing higher latency; our other regions are not affected. Customers running jobs in us-west may see slower Git-cached checkouts, incremental Docker builds, and container caching. We are continuing to investigate the underlying cause and will provide another update within the next 30 minutes..</p> <p><small>Sep <var data-var='date'> 2</var>, <var data-var='time'>17:22:52</var> GMT+0</small><br><strong>Investigating</strong> - We are still working to resolve elevated latency on sticky disk storage in our us-west region; other regions are not affected. Customers running jobs in us-west may continue to see slower Git-cached checkouts, incremental Docker builds, and container caching. Our investigation into the underlying cause is ongoing and we will provide another update within the next 30 minutes. .</p> <p><small>Sep <var data-var='date'> 2</var>, <var data-var='time'>18:10:23</var> GMT+0</small><br><strong>Monitoring</strong> - Our us-west region storage cluster's latency and error rates have returned to baseline as of approximately 17:15 UTC, following mitigations that reduce load on the affected storage. Git-cached checkouts, incremental Docker builds, and container caching in us-west are back to normal, and we are monitoring to confirm the recovery holds while we continue to investigate the underlying cause. We will provide a final update within the next hour..</p> <p><small>Sep <var data-var='date'> 2</var>, <var data-var='time'>19:23:31</var> GMT+0</small><br><strong>Resolved</strong> - This incident is resolved. Sticky disk storage in our us-west region came under more read load than it could serve at normal latency, which slowed and produced higher error rates for Git-cached checkouts, incremental Docker builds, and container caching. We reduced and redistributed that load, and performance has been normal since approximately 17:15 UTC. Jobs that failed during the incident can be safely re-run. .</p> ]]> </content:encoded> <pubDate>Wed, 2 Sep 2026 16:13:47 +0000</pubDate> <link>https://status.blacksmith.sh/incident/cmtkapxle0jl313mfiqqeqjud</link> <guid>https://status.blacksmith.sh/incident/cmtkapxle0jl313mfiqqeqjud</guid> </item> <item> <title>Service degradation in upstream GHCR registries</title> <description> Type: Incident Duration: 2 hours and 22 minutes Affected Components: Github → API Requests Sep 2, 14:09:09 GMT+0 - Investigating - We are currently observing higher rates of errors for the upstream GHCR registries which may indicate an undeclared GitHub incident. We are currently monitoring. Sep 2, 14:20:19 GMT+0 - Investigating - We are seeing evidence that Git Checkouts are also affected by this. We are seeing high TCP retransmit rates into the EU GitHub loadbalancer. Checkouts in the EU West, EU Central, US East regions are affected. Sep 2, 15:43:34 GMT+0 - Monitoring - Connections to [github.com](http://github.com) and [ghcr.io](http://ghcr.io) from our eu-west, eu-central, and us-east regions have returned to normal as of approximately 15:00 UTC, so checkouts are completing at normal speed and login failures to GitHub Container Registry have dropped to baseline levels. We are continuing to monitor, and re-running any jobs that failed during this period should succeed. Sep 2, 16:30:55 GMT+0 - Resolved - This incident has been resolved. GitHub connectivity from our EU regions was degraded from \~12:45 to 16:15 UTC, affecting checkouts, [ghcr.io](http://ghcr.io), and some eu-west job starts. Normal since 16:15\. Affected jobs can be re-run. </description> <content:encoded> < and [ghcr.io](http://ghcr.io) from our eu-west, eu-central, and us-east regions have returned to normal as of approximately 15:00 UTC, so checkouts are completing at normal speed and login failures to GitHub Container Registry have dropped to baseline levels. We are continuing to monitor, and re-running any jobs that failed during this period should succeed..</p> <p><small>Sep <var data-var='date'> 2</var>, <var data-var='time'>16:30:55</var> GMT+0</small><br><strong>Resolved</strong> - This incident has been resolved. GitHub connectivity from our EU regions was degraded from \~12:45 to 16:15 UTC, affecting checkouts, [ghcr.io](http://ghcr.io), and some eu-west job starts. Normal since 16:15\. Affected jobs can be re-run..</p> ]]> </content:encoded> <pubDate>Wed, 2 Sep 2026 14:09:09 +0000</pubDate> <link>https://status.blacksmith.sh/incident/cmtk69nmb002c13pmm1yfslre</link> <guid>https://status.blacksmith.sh/incident/cmtk69nmb002c13pmm1yfslre</guid> </item> <item> <title>Requests to GitHub failing due to upstream incident</title> <description> Type: Incident Duration: 2 hours and 29 minutes Aug 18, 20:45:03 GMT+0 - Monitoring - Jobs failure rate at an elevated rate caused by an upstream GitHub being rejected by 429 rate-limit errors. We are seeing a single-digit percentage increase in job failure rate across all organizations and are seeing similar failures on non-Blacksmith infrastructure. Aug 18, 23:14:25 GMT+0 - Resolved - This incident has been resolved. </description> <content:encoded> <![CDATA[<p><strong>Type:</strong> Incident</p> <p><strong>Duration:</strong> 2 hours and 29 minutes</p> <p><small>Aug <var data-var='date'> 18</var>, <var data-var='time'>20:45:03</var> GMT+0</small><br><strong>Monitoring</strong> - Jobs failure rate at an elevated rate caused by an upstream GitHub being rejected by 429 rate-limit errors. We are seeing a single-digit percentage increase in job failure rate across all organizations and are seeing similar failures on non-Blacksmith infrastructure..</p> <p><small>Aug <var data-var='date'> 18</var>, <var data-var='time'>23:14:25</var> GMT+0</small><br><strong>Resolved</strong> - This incident has been resolved..</p> ]]> </content:encoded> <pubDate>Tue, 18 Aug 2026 20:45:03 +0000</pubDate> <link>https://status.blacksmith.sh/incident/cmsz4t08d056h0yo0nkcxi1hl</link> <guid>https://status.blacksmith.sh/incident/cmsz4t08d056h0yo0nkcxi1hl</guid> </item> <item> <title>Job adoption delays due to missed webhooks</title> <description> Type: Incident Duration: 2 minutes Affected Components: us-east x86, eu-central x86, eu-west ARM, us-west x86, eu-west x86, us-west ARM, us-central MacOS, eu-central ARM Aug 17, 18:58:04 GMT+0 - Investigating - We are seeing an increased error rate processing webhook events which will lead to delays in adoption jobs. We are actively investigating. Aug 17, 19:00:00 GMT+0 - Resolved - This incident has been resolved. </description> <content:encoded> <![CDATA[<p><strong>Type:</strong> Incident</p> <p><strong>Duration:</strong> 2 minutes</p> <p><strong>Affected Components:</strong> , , , , , , , </p> <p><small>Aug <var data-var='date'> 17</var>, <var data-var='time'>18:58:04</var> GMT+0</small><br><strong>Investigating</strong> - We are seeing an increased error rate processing webhook events which will lead to delays in adoption jobs. We are actively investigating..</p> <p><small>Aug <var data-var='date'> 17</var>, <var data-var='time'>19:00:00</var> GMT+0</small><br><strong>Resolved</strong> - This incident has been resolved..</p> ]]> </content:encoded> <pubDate>Mon, 17 Aug 2026 18:58:04 +0000</pubDate> <link>https://status.blacksmith.sh/incident/cmsxljkl317c80kqxuyyjqpg9</link> <guid>https://status.blacksmith.sh/incident/cmsxljkl317c80kqxuyyjqpg9</guid> </item> <item> <title>GitHub outage affecting job failures and dashboard errors</title> <description> Type: Incident Duration: 6 hours and 57 minutes Affected Components: Github → API Requests, Dashboard, Github → Webhooks Aug 17, 13:43:37 GMT+0 - Identified - The Blacksmith Dashboard is unable to load. We've identified the root cause to upstream 503s being returned from GitHub on permission-check requests. Aug 17, 13:46:46 GMT+0 - Identified - Upstream incident has been declared: <https://www.githubstatus.com/incidents/zkxwbgr0cnmx> We are also seeing elevated error rates in jobs as they hit upstream GitHub errors. Job adoption times are also affected and are delayed. We are monitoring and are looking at potential mitigations. Aug 17, 16:56:43 GMT+0 - Monitoring - We're seeing signs of GitHub recovery. The dashboard is now loading and jobs should be running again. We are monitoring the recovery. Aug 17, 17:52:36 GMT+0 - Monitoring - We are still seeing intermitting GitHub API errors at a low rate. Aug 17, 19:32:23 GMT+0 - Monitoring - We are no longer seeing upstream errors, and are continuing to monitor impact of the upstream outage. Aug 17, 20:40:52 GMT+0 - Resolved - This incident has been resolved. </description> <content:encoded> <![CDATA[<p><strong>Type:</strong> Incident</p> <p><strong>Duration:</strong> 6 hours and 57 minutes</p> <p><strong>Affected Components:</strong> , , </p> <p><small>Aug <var data-var='date'> 17</var>, <var data-var='time'>13:43:37</var> GMT+0</small><br><strong>Identified</strong> - The Blacksmith Dashboard is unable to load. We've identified the root cause to upstream 503s being returned from GitHub on permission-check requests..</p> <p><small>Aug <var data-var='date'> 17</var>, <var data-var='time'>13:46:46</var> GMT+0</small><br><strong>Identified</strong> - Upstream incident has been declared: <https://www.githubstatus.com/incidents/zkxwbgr0cnmx> We are also seeing elevated error rates in jobs as they hit upstream GitHub errors. Job adoption times are also affected and are delayed. We are monitoring and are looking at potential mitigations..</p> <p><small>Aug <var data-var='date'> 17</var>, <var data-var='time'>16:56:43</var> GMT+0</small><br><strong>Monitoring</strong> - We're seeing signs of GitHub recovery. The dashboard is now loading and jobs should be running again. We are monitoring the recovery..</p> <p><small>Aug <var data-var='date'> 17</var>, <var data-var='time'>17:52:36</var> GMT+0</small><br><strong>Monitoring</strong> - We are still seeing intermitting GitHub API errors at a low rate. .</p> <p><small>Aug <var data-var='date'> 17</var>, <var data-var='time'>19:32:23</var> GMT+0</small><br><strong>Monitoring</strong> - We are no longer seeing upstream errors, and are continuing to monitor impact of the upstream outage..</p> <p><small>Aug <var data-var='date'> 17</var>, <var data-var='time'>20:40:52</var> GMT+0</small><br><strong>Resolved</strong> - This incident has been resolved..</p> ]]> </content:encoded> <pubDate>Mon, 17 Aug 2026 13:43:37 +0000</pubDate> <link>https://status.blacksmith.sh/incident/cmsxab79p16kz0rqxaiw9jr56</link> <guid>https://status.blacksmith.sh/incident/cmsxab79p16kz0rqxaiw9jr56</guid> </item> <item> <title>Stickydisk Storage cluster maintenance</title> <description> Type: Maintenance Duration: 4 hours Affected Components: us-west Storage Cluster, us-west Storage Cluster, us-west Storage Cluster Aug 16, 11:47:45 GMT+0 - Identified - We are planning for a scheduled maintenance during that time. Aug 16, 12:00:01 GMT+0 - Identified - Maintenance is now in progress Aug 16, 16:00:00 GMT+0 - Completed - Maintenance has completed successfully </description> <content:encoded> <![CDATA[<p><strong>Type:</strong> Maintenance</p> <p><strong>Duration:</strong> 4 hours</p> <p><strong>Affected Components:</strong> , , </p> <p><small>Aug <var data-var='date'> 16</var>, <var data-var='time'>11:47:45</var> GMT+0</small><br><strong>Identified</strong> - We are planning for a scheduled maintenance during that time..</p> <p><small>Aug <var data-var='date'> 16</var>, <var data-var='time'>12:00:01</var> GMT+0</small><br><strong>Identified</strong> - Maintenance is now in progress.</p> <p><small>Aug <var data-var='date'> 16</var>, <var data-var='time'>16:00:00</var> GMT+0</small><br><strong>Completed</strong> - Maintenance has completed successfully.</p> ]]> </content:encoded> <pubDate>Sun, 16 Aug 2026 12:00:00 +0000</pubDate> <link>https://status.blacksmith.sh/maintenance/cmsvqqbo80tv01aqx9v4rr5j5</link> <guid>https://status.blacksmith.sh/maintenance/cmsvqqbo80tv01aqx9v4rr5j5</guid> </item> <item> <title>Storage degradation in us-west</title> <description> Type: Incident Duration: 1 day, 6 hours and 33 minutes Affected Components: us-east x86, eu-central x86, eu-west ARM, us-west Storage Cluster, EU Central Cache, us-west x86, eu-west x86, EU West Cache, us-west ARM, US East Cache, US West Cache, eu-central ARM, Github → Webhooks, us-west Storage Cluster, us-west Storage Cluster Aug 13, 16:32:39 GMT+0 - Identified - Our datacenter provider has begun deploying mitigations at the affected us-west facility, and GitHub has resolved its separate webhooks incident, though some knock-on delays may persist while affected jobs requeue. The us-west region remains heavily affected and with capacity reduced while we rebalance traffic, jobs in all regions may take longer than usual to start. Aug 13, 18:26:30 GMT+0 - Identified - Our provider has begun powering servers back on at the us-west facility, and a portion of our us-west capacity is back online. We have started moving some customers back to us-west to spread load across regions. Jobs in all regions may still take longer than usual to start. Aug 13, 13:59:44 GMT+0 - Identified - Conditions at our provider's us-west facility have not yet improved, and we are continuing to work around the issue to keep customer impact minimal. Such mitigations may include rebalancing customers to unaffected regions. Aug 13, 14:40:13 GMT+0 - Identified - Conditions at our provider's us-west facility have not yet improved. We are continuing to issue mitigations to ensure that customer impact is minimal. Aug 13, 19:15:14 GMT+0 - Identified - Our provider is continuing to bring us-west servers back online, and we are gradually moving customers back to us-west as capacity returns. The backlog has not yet cleared, so jobs in all regions may still take longer than usual to start. We will provide another update within the next hour. Aug 13, 20:44:17 GMT+0 - Identified - Queues in us-west have come down substantially as restored capacity comes online and we continue moving customers back. Cache operations, including sticky disks, remain unavailable in us-west, and jobs in us-east may still take significantly longer than usual to start while we work through the remaining backlog. We will provide another update within the next hour. Aug 13, 17:00:03 GMT+0 - Identified - Our provider has partially restored cooling at the us-west facility and expects to reach safe operating temperatures within the next few hours, at which point servers will be powered back on in phases. In the meantime jobs in all regions may still take significantly longer than usual to start. We will provide another update within the next hour. Aug 13, 12:06:56 GMT+0 - Identified - Caching performance in US West is currently degraded. Our upstream provider has lost cooling power in their data center, leading to failure of some storage systems in US West. As a result jobs may take longer to complete. We are taking action to mitigate the issue. Aug 13, 11:40:58 GMT+0 - Investigating - Our cloud provider is experiencing a thermal event in their us-west datacenter that is affecting caching in the region. Sticky Disk, Actions Cache, Docker Container Cache, and Bazel Build Caching are impacted: cache operations may be slow or unavailable which may result in some builds running slower than usual. Aug 13, 19:59:27 GMT+0 - Identified - More us-west capacity has come back online and queues in the region are steadily draining as we move customers back. Cache operations, including sticky disks, remain unavailable in us-west while our provider works to restore the storage portion of the facility, and because we shifted us-west traffic to other regions earlier today, jobs elsewhere, particularly in us-east, may still take longer than usual to start. We will provide another update within the next hour. Aug 13, 12:36:49 GMT+0 - Identified - We are currently applying mitigations for affected customers in the region. Aug 13, 17:42:15 GMT+0 - Identified - Our provider is continuing to restore cooling at the us-west facility; temperatures have not yet reached safe levels for servers to be powered back on, and we are staged to begin restoring runner capacity as soon as they are. The us-west region remains in a major outage, and jobs in all regions may still take longer than usual to start. We will provide another update within the next hour. Aug 13, 13:27:12 GMT+0 - Identified - Conditions at our provider's us-west facility have not yet improved. As we rebalance customers away from us-west, other regions may see slightly longer pickup times than usual as a side effect Aug 13, 15:11:27 GMT+0 - Identified - Conditions in our us-west region have not yet improved. We're actively working with our datacenter provider on the issue, and will provide updates as they occur. We are also manually rebalancing traffic out of the us-west region to aid with recovery. Github have also declared an incident affecting webhooks - <https://www.githubstatus.com/incidents/k8vbzwqjkxzn> which may cause some job adoption delays Aug 13, 13:02:39 GMT+0 - Identified - Conditions at our provider's us-west facility have not yet improved, and we are continuing to work around the issue to keep customer impact minimal. Such mitigations may include rebalancing customers to unaffected regions. Aug 13, 21:48:02 GMT+0 - Identified - Queue times in us-west have returned to near-normal levels, while us-east, eu-west, and eu-central are still working through their remaining backlogs. Cache operations, including sticky disks, remain unavailable in us-west while our provider works to restore the storage systems there. We will provide another update within the next hour. Aug 13, 22:54:06 GMT+0 - Identified - Queue times are improving across all regions as capacity comes back online, but jobs may still take longer than usual to start. Cache operations in us-west, including sticky disks, remain unavailable while our provider restores the storage systems, and jobs on our largest runner sizes may see the longest delays. We will provide another update within the next hour. Aug 14, 00:00:40 GMT+0 - Identified - Runner capacity in us-west continues to recover and queues in all regions are draining. Our provider has restored the access we need to begin bringing our us-west storage systems back online. Cache operations, including sticky disks, remain unavailable while that work completes, and jobs may still take longer than usual to start. Aug 14, 01:15:24 GMT+0 - Monitoring - Queue times for all runner sizes have returned to normal in all regions, and we are monitoring closely while eu-west clears the last of its backlog. Cache operations, including sticky disks, remain unavailable in us-west while we bring the restored storage hardware back online. We will provide another update within the next hour. Aug 14, 03:06:21 GMT+0 - Monitoring - Our us-west compute provider has restored power and cooling in their datacenter and the majority of our capacity has returned. Github Actions caching has been reenabled in the region but we are still working to restore Sticky Disks, Docker Container caching, and Incremental Docker Builders. We will post an update once these components are restored. Aug 14, 10:57:11 GMT+0 - Monitoring - We are continuing to work with our compute provider to restore the storage cluster for Sticky Disks, Docker Container Caching, and Incremental Docker Builders in us-west. This requires hands-on recovery work by our provider's team and may take a few more hours to fully restore. Workflows using the affected features in us-west may see slow or stalled Docker builds in the meantime. Aug 14, 02:08:56 GMT+0 - Monitoring - Job queues have fully recovered, and runners in all regions are operating normally for all runner sizes. Cache operations, including sticky disks, remain unavailable in us-west while we bring the restored storage hardware back online. We will provide another update within the next hour. Aug 14, 03:26:24 GMT+0 - Monitoring - We are noticing some issues with the storage cluster backing the Github Actions cache after it was restored. We are working on restoring this alongside the storage cluster backing Sticky Disks, Incremental Docker Builders, and Docker Container Caching. Aug 14, 15:02:08 GMT+0 - Monitoring - The storage cluster for Sticky Disks, Docker Container Caching, and Incremental Docker Builders in us-west has been restored, and disks are mounting and operating normally. As part of the recovery, a small amount of recently written cache data may need to be rebuilt, so some Docker builds may run slower over their first few runs while caches rehydrate. We are monitoring closely. Aug 14, 04:09:30 GMT+0 - Monitoring - We are still working with our compute provider to recover the storage clusters. Aug 14, 06:06:54 GMT+0 - Monitoring - We have restored the Github Actions Cache in us-west. We are still working with our compute provider to restore our remaining storage cluster for Sticky Disks, Docker Container Caching, and Incremental Docker Builders. Aug 14, 16:16:38 GMT+0 - Monitoring - All services have been restored, including sticky disks, Docker container caching, and incremental Docker builders in us-west, and job queues are operating normally in all regions. We are continuing to monitor the stability of the recovered storage cluster as it ramps back up with traffic, and some builds may run slower on their first runs while recently written cache data rebuilds. We will post a final update once we have confirmed stability. Aug 14, 18:14:17 GMT+0 - Resolved - All services have been fully restored, including sticky disks, Docker container caching, and incremental Docker builders in us-west, and job queues are operating normally in all regions. A small amount of recently written cache data could not be recovered during storage repair, so some builds may run slower on their first runs while caches rebuild. This incident is now resolved, and we will publish a detailed post-incident report in the coming days. </description> <content:encoded> <![CDATA[<p><strong>Type:</strong> Incident</p> <p><strong>Duration:</strong> 1 day, 6 hours and 33 minutes</p> <p><strong>Affected Components:</strong> , , , , , , , , , , , , , , </p> <p><small>Aug <var data-var='date'> 13</var>, <var data-var='time'>16:32:39</var> GMT+0</small><br><strong>Identified</strong> - Our datacenter provider has begun deploying mitigations at the affected us-west facility, and GitHub has resolved its separate webhooks incident, though some knock-on delays may persist while affected jobs requeue. The us-west region remains heavily affected and with capacity reduced while we rebalance traffic, jobs in all regions may take longer than usual to start..</p> <p><small>Aug <var data-var='date'> 13</var>, <var data-var='time'>18:26:30</var> GMT+0</small><br><strong>Identified</strong> - Our provider has begun powering servers back on at the us-west facility, and a portion of our us-west capacity is back online. We have started moving some customers back to us-west to spread load across regions. Jobs in all regions may still take longer than usual to start..</p> <p><small>Aug <var data-var='date'> 13</var>, <var data-var='time'>13:59:44</var> GMT+0</small><br><strong>Identified</strong> - Conditions at our provider's us-west facility have not yet improved, and we are continuing to work around the issue to keep customer impact minimal. Such mitigations may include rebalancing customers to unaffected regions..</p> <p><small>Aug <var data-var='date'> 13</var>, <var data-var='time'>14:40:13</var> GMT+0</small><br><strong>Identified</strong> - Conditions at our provider's us-west facility have not yet improved. We are continuing to issue mitigations to ensure that customer impact is minimal. .</p> <p><small>Aug <var data-var='date'> 13</var>, <var data-var='time'>19:15:14</var> GMT+0</small><br><strong>Identified</strong> - Our provider is continuing to bring us-west servers back online, and we are gradually moving customers back to us-west as capacity returns. The backlog has not yet cleared, so jobs in all regions may still take longer than usual to start. We will provide another update within the next hour..</p> <p><small>Aug <var data-var='date'> 13</var>, <var data-var='time'>20:44:17</var> GMT+0</small><br><strong>Identified</strong> - Queues in us-west have come down substantially as restored capacity comes online and we continue moving customers back. Cache operations, including sticky disks, remain unavailable in us-west, and jobs in us-east may still take significantly longer than usual to start while we work through the remaining backlog. We will provide another update within the next hour..</p> <p><small>Aug <var data-var='date'> 13</var>, <var data-var='time'>17:00:03</var> GMT+0</small><br><strong>Identified</strong> - Our provider has partially restored cooling at the us-west facility and expects to reach safe operating temperatures within the next few hours, at which point servers will be powered back on in phases. In the meantime jobs in all regions may still take significantly longer than usual to start. We will provide another update within the next hour..</p> <p><small>Aug <var data-var='date'> 13</var>, <var data-var='time'>12:06:56</var> GMT+0</small><br><strong>Identified</strong> - Caching performance in US West is currently degraded. Our upstream provider has lost cooling power in their data center, leading to failure of some storage systems in US West. As a result jobs may take longer to complete. We are taking action to mitigate the issue..</p> <p><small>Aug <var data-var='date'> 13</var>, <var data-var='time'>11:40:58</var> GMT+0</small><br><strong>Investigating</strong> - Our cloud provider is experiencing a thermal event in their us-west datacenter that is affecting caching in the region. Sticky Disk, Actions Cache, Docker Container Cache, and Bazel Build Caching are impacted: cache operations may be slow or unavailable which may result in some builds running slower than usual..</p> <p><small>Aug <var data-var='date'> 13</var>, <var data-var='time'>19:59:27</var> GMT+0</small><br><strong>Identified</strong> - More us-west capacity has come back online and queues in the region are steadily draining as we move customers back. Cache operations, including sticky disks, remain unavailable in us-west while our provider works to restore the storage portion of the facility, and because we shifted us-west traffic to other regions earlier today, jobs elsewhere, particularly in us-east, may still take longer than usual to start. We will provide another update within the next hour..</p> <p><small>Aug <var data-var='date'> 13</var>, <var data-var='time'>12:36:49</var> GMT+0</small><br><strong>Identified</strong> - We are currently applying mitigations for affected customers in the region..</p> <p><small>Aug <var data-var='date'> 13</var>, <var data-var='time'>17:42:15</var> GMT+0</small><br><strong>Identified</strong> - Our provider is continuing to restore cooling at the us-west facility; temperatures have not yet reached safe levels for servers to be powered back on, and we are staged to begin restoring runner capacity as soon as they are. The us-west region remains in a major outage, and jobs in all regions may still take longer than usual to start. We will provide another update within the next hour..</p> <p><small>Aug <var data-var='date'> 13</var>, <var data-var='time'>13:27:12</var> GMT+0</small><br><strong>Identified</strong> - Conditions at our provider's us-west facility have not yet improved. As we rebalance customers away from us-west, other regions may see slightly longer pickup times than usual as a side effect.</p> <p><small>Aug <var data-var='date'> 13</var>, <var data-var='time'>15:11:27</var> GMT+0</small><br><strong>Identified</strong> - Conditions in our us-west region have not yet improved. We're actively working with our datacenter provider on the issue, and will provide updates as they occur. We are also manually rebalancing traffic out of the us-west region to aid with recovery. Github have also declared an incident affecting webhooks - <https://www.githubstatus.com/incidents/k8vbzwqjkxzn> which may cause some job adoption delays.</p> <p><small>Aug <var data-var='date'> 13</var>, <var data-var='time'>13:02:39</var> GMT+0</small><br><strong>Identified</strong> - Conditions at our provider's us-west facility have not yet improved, and we are continuing to work around the issue to keep customer impact minimal. Such mitigations may include rebalancing customers to unaffected regions..</p> <p><small>Aug <var data-var='date'> 13</var>, <var data-var='time'>21:48:02</var> GMT+0</small><br><strong>Identified</strong> - Queue times in us-west have returned to near-normal levels, while us-east, eu-west, and eu-central are still working through their remaining backlogs. Cache operations, including sticky disks, remain unavailable in us-west while our provider works to restore the storage systems there. We will provide another update within the next hour..</p> <p><small>Aug <var data-var='date'> 13</var>, <var data-var='time'>22:54:06</var> GMT+0</small><br><strong>Identified</strong> - Queue times are improving across all regions as capacity comes back online, but jobs may still take longer than usual to start. Cache operations in us-west, including sticky disks, remain unavailable while our provider restores the storage systems, and jobs on our largest runner sizes may see the longest delays. We will provide another update within the next hour..</p> <p><small>Aug <var data-var='date'> 14</var>, <var data-var='time'>00:00:40</var> GMT+0</small><br><strong>Identified</strong> - Runner capacity in us-west continues to recover and queues in all regions are draining. Our provider has restored the access we need to begin bringing our us-west storage systems back online. Cache operations, including sticky disks, remain unavailable while that work completes, and jobs may still take longer than usual to start..</p> <p><small>Aug <var data-var='date'> 14</var>, <var data-var='time'>01:15:24</var> GMT+0</small><br><strong>Monitoring</strong> - Queue times for all runner sizes have returned to normal in all regions, and we are monitoring closely while eu-west clears the last of its backlog. Cache operations, including sticky disks, remain unavailable in us-west while we bring the restored storage hardware back online. We will provide another update within the next hour..</p> <p><small>Aug <var data-var='date'> 14</var>, <var data-var='time'>03:06:21</var> GMT+0</small><br><strong>Monitoring</strong> - Our us-west compute provider has restored power and cooling in their datacenter and the majority of our capacity has returned. Github Actions caching has been reenabled in the region but we are still working to restore Sticky Disks, Docker Container caching, and Incremental Docker Builders. We will post an update once these components are restored..</p> <p><small>Aug <var data-var='date'> 14</var>, <var data-var='time'>10:57:11</var> GMT+0</small><br><strong>Monitoring</strong> - We are continuing to work with our compute provider to restore the storage cluster for Sticky Disks, Docker Container Caching, and Incremental Docker Builders in us-west. This requires hands-on recovery work by our provider's team and may take a few more hours to fully restore. Workflows using the affected features in us-west may see slow or stalled Docker builds in the meantime..</p> <p><small>Aug <var data-var='date'> 14</var>, <var data-var='time'>02:08:56</var> GMT+0</small><br><strong>Monitoring</strong> - Job queues have fully recovered, and runners in all regions are operating normally for all runner sizes. Cache operations, including sticky disks, remain unavailable in us-west while we bring the restored storage hardware back online. We will provide another update within the next hour..</p> <p><small>Aug <var data-var='date'> 14</var>, <var data-var='time'>03:26:24</var> GMT+0</small><br><strong>Monitoring</strong> - We are noticing some issues with the storage cluster backing the Github Actions cache after it was restored. We are working on restoring this alongside the storage cluster backing Sticky Disks, Incremental Docker Builders, and Docker Container Caching..</p> <p><small>Aug <var data-var='date'> 14</var>, <var data-var='time'>15:02:08</var> GMT+0</small><br><strong>Monitoring</strong> - The storage cluster for Sticky Disks, Docker Container Caching, and Incremental Docker Builders in us-west has been restored, and disks are mounting and operating normally. As part of the recovery, a small amount of recently written cache data may need to be rebuilt, so some Docker builds may run slower over their first few runs while caches rehydrate. We are monitoring closely..</p> <p><small>Aug <var data-var='date'> 14</var>, <var data-var='time'>04:09:30</var> GMT+0</small><br><strong>Monitoring</strong> - We are still working with our compute provider to recover the storage clusters..</p> <p><small>Aug <var data-var='date'> 14</var>, <var data-var='time'>06:06:54</var> GMT+0</small><br><strong>Monitoring</strong> - We have restored the Github Actions Cache in us-west. We are still working with our compute provider to restore our remaining storage cluster for Sticky Disks, Docker Container Caching, and Incremental Docker Builders..</p> <p><small>Aug <var data-var='date'> 14</var>, <var data-var='time'>16:16:38</var> GMT+0</small><br><strong>Monitoring</strong> - All services have been restored, including sticky disks, Docker container caching, and incremental Docker builders in us-west, and job queues are operating normally in all regions. We are continuing to monitor the stability of the recovered storage cluster as it ramps back up with traffic, and some builds may run slower on their first runs while recently written cache data rebuilds. We will post a final update once we have confirmed stability..</p> <p><small>Aug <var data-var='date'> 14</var>, <var data-var='time'>18:14:17</var> GMT+0</small><br><strong>Resolved</strong> - All services have been fully restored, including sticky disks, Docker container caching, and incremental Docker builders in us-west, and job queues are operating normally in all regions. A small amount of recently written cache data could not be recovered during storage repair, so some builds may run slower on their first runs while caches rebuild. This incident is now resolved, and we will publish a detailed post-incident report in the coming days..</p> ]]> </content:encoded> <pubDate>Thu, 13 Aug 2026 11:40:58 +0000</pubDate> <link>https://status.blacksmith.sh/incident/cmsrg61yg09vd0lo9s1eagibm</link> <guid>https://status.blacksmith.sh/incident/cmsrg61yg09vd0lo9s1eagibm</guid> </item> <item> <title>Elevated failures downloading GitHub release assets</title> <description> Type: Incident Duration: 6 hours and 39 minutes Affected Components: Github → Actions Aug 12, 21:31:36 GMT+0 - Identified - Workflows downloading GitHub release assets remain affected across all regions because of an upstream issue at GitHub, and retrying failed jobs remains the most effective workaround. Aug 12, 18:41:51 GMT+0 - Investigating - We are investigating reports of socket hangups when attempting to fetch GitHub release assets. There are not any current upstream incidents, but we are continuing to investigate. Aug 12, 19:25:16 GMT+0 - Identified - We are still experiencing an elevated rate of socket hangup failures when workflows download GitHub release assets, across all regions. The failures have been isolated to GitHub upstream, and we are monitoring closely while the issue persists. We have observed multiple public reports that GitHub is experiencing a partial outage. Aug 12, 20:23:55 GMT+0 - Identified - We have isolated the cause to a subset of GitHub's edge servers and verified the failures occur from networks outside Blacksmith as well. Workflows downloading GitHub release assets remain affected across all regions, and retrying failed jobs is the most effective workaround until the upstream issue is resolved. We will provide another update within the next 30 minutes. Aug 12, 19:55:41 GMT+0 - Identified - We are still experiencing an elevated rate of socket hangup failures when workflows download GitHub release assets, across all regions. The failures remain isolated to GitHub upstream, and we are continuing to monitor for recovery; other GitHub interactions, including repository checkout, remain unaffected. We will provide another update within the next 30 minutes. Aug 12, 22:03:11 GMT+0 - Identified - GitHub has confirmed an incident affecting release asset downloads, consistent with our findings (githubstatus.com/incidents/lsvy8xsf0gxv). Workflows downloading GitHub release assets remain affected across all regions while GitHub works toward mitigation, and retrying failed jobs remains the most effective workaround. Aug 12, 20:52:19 GMT+0 - Identified - Workflows downloading GitHub release assets remain affected across all regions, and retrying failed jobs remains the most effective workaround. The failures remain isolated to a subset of GitHub's edge servers upstream of our infrastructure, and we are continuing to monitor for recovery. Aug 12, 23:06:43 GMT+0 - Monitoring - GitHub has implemented a fix and we are beginning to see recovery. We will continue to monitor the issue. <https://www.githubstatus.com/incidents/lsvy8xsf0gxv> Aug 13, 01:20:46 GMT+0 - Resolved - This incident has been resolved. </description> <content:encoded> <![CDATA[<p><strong>Type:</strong> Incident</p> <p><strong>Duration:</strong> 6 hours and 39 minutes</p> <p><strong>Affected Components:</strong> </p> <p><small>Aug <var data-var='date'> 12</var>, <var data-var='time'>21:31:36</var> GMT+0</small><br><strong>Identified</strong> - Workflows downloading GitHub release assets remain affected across all regions because of an upstream issue at GitHub, and retrying failed jobs remains the most effective workaround..</p> <p><small>Aug <var data-var='date'> 12</var>, <var data-var='time'>18:41:51</var> GMT+0</small><br><strong>Investigating</strong> - We are investigating reports of socket hangups when attempting to fetch GitHub release assets. There are not any current upstream incidents, but we are continuing to investigate..</p> <p><small>Aug <var data-var='date'> 12</var>, <var data-var='time'>19:25:16</var> GMT+0</small><br><strong>Identified</strong> - We are still experiencing an elevated rate of socket hangup failures when workflows download GitHub release assets, across all regions. The failures have been isolated to GitHub upstream, and we are monitoring closely while the issue persists. We have observed multiple public reports that GitHub is experiencing a partial outage..</p> <p><small>Aug <var data-var='date'> 12</var>, <var data-var='time'>20:23:55</var> GMT+0</small><br><strong>Identified</strong> - We have isolated the cause to a subset of GitHub's edge servers and verified the failures occur from networks outside Blacksmith as well. Workflows downloading GitHub release assets remain affected across all regions, and retrying failed jobs is the most effective workaround until the upstream issue is resolved. We will provide another update within the next 30 minutes..</p> <p><small>Aug <var data-var='date'> 12</var>, <var data-var='time'>19:55:41</var> GMT+0</small><br><strong>Identified</strong> - We are still experiencing an elevated rate of socket hangup failures when workflows download GitHub release assets, across all regions. The failures remain isolated to GitHub upstream, and we are continuing to monitor for recovery; other GitHub interactions, including repository checkout, remain unaffected. We will provide another update within the next 30 minutes..</p> <p><small>Aug <var data-var='date'> 12</var>, <var data-var='time'>22:03:11</var> GMT+0</small><br><strong>Identified</strong> - GitHub has confirmed an incident affecting release asset downloads, consistent with our findings (githubstatus.com/incidents/lsvy8xsf0gxv). Workflows downloading GitHub release assets remain affected across all regions while GitHub works toward mitigation, and retrying failed jobs remains the most effective workaround. .</p> <p><small>Aug <var data-var='date'> 12</var>, <var data-var='time'>20:52:19</var> GMT+0</small><br><strong>Identified</strong> - Workflows downloading GitHub release assets remain affected across all regions, and retrying failed jobs remains the most effective workaround. The failures remain isolated to a subset of GitHub's edge servers upstream of our infrastructure, and we are continuing to monitor for recovery..</p> <p><small>Aug <var data-var='date'> 12</var>, <var data-var='time'>23:06:43</var> GMT+0</small><br><strong>Monitoring</strong> - GitHub has implemented a fix and we are beginning to see recovery. We will continue to monitor the issue. <https://www.githubstatus.com/incidents/lsvy8xsf0gxv>.</p> <p><small>Aug <var data-var='date'> 13</var>, <var data-var='time'>01:20:46</var> GMT+0</small><br><strong>Resolved</strong> - This incident has been resolved..</p> ]]> </content:encoded> <pubDate>Wed, 12 Aug 2026 18:41:51 +0000</pubDate> <link>https://status.blacksmith.sh/incident/cmsqfrgv9022q0rqidjkjxqrz</link> <guid>https://status.blacksmith.sh/incident/cmsqfrgv9022q0rqidjkjxqrz</guid> </item> <item> <title>Hanging apt package installs on us-west runners due to an upstream mirror issue</title> <description> Type: Incident Duration: 8 hours and 21 minutes Affected Components: us-west x86 Aug 12, 03:30:00 GMT+0 - Investigating - We are currently investigating this incident. Aug 12, 09:10:10 GMT+0 - Identified - We have identified the affected mirror and are implementing a fix Aug 12, 10:33:01 GMT+0 - Identified - We are currently deploying a mitigation Aug 12, 11:26:10 GMT+0 - Monitoring - We implemented a fix and are seeing improvements. We are continuing to monitor the result. Aug 12, 11:50:47 GMT+0 - Resolved - This incident has been resolved. </description> <content:encoded> <![CDATA[<p><strong>Type:</strong> Incident</p> <p><strong>Duration:</strong> 8 hours and 21 minutes</p> <p><strong>Affected Components:</strong> </p> <p><small>Aug <var data-var='date'> 12</var>, <var data-var='time'>03:30:00</var> GMT+0</small><br><strong>Investigating</strong> - We are currently investigating this incident..</p> <p><small>Aug <var data-var='date'> 12</var>, <var data-var='time'>09:10:10</var> GMT+0</small><br><strong>Identified</strong> - We have identified the affected mirror and are implementing a fix.</p> <p><small>Aug <var data-var='date'> 12</var>, <var data-var='time'>10:33:01</var> GMT+0</small><br><strong>Identified</strong> - We are currently deploying a mitigation.</p> <p><small>Aug <var data-var='date'> 12</var>, <var data-var='time'>11:26:10</var> GMT+0</small><br><strong>Monitoring</strong> - We implemented a fix and are seeing improvements. We are continuing to monitor the result..</p> <p><small>Aug <var data-var='date'> 12</var>, <var data-var='time'>11:50:47</var> GMT+0</small><br><strong>Resolved</strong> - This incident has been resolved..</p> ]]> </content:encoded> <pubDate>Wed, 12 Aug 2026 03:30:00 +0000</pubDate> <link>https://status.blacksmith.sh/incident/cmspwjrqs01ar0zpcetcwi50r</link> <guid>https://status.blacksmith.sh/incident/cmspwjrqs01ar0zpcetcwi50r</guid> </item> <item> <title>Increased action cache miss rate for certain customers</title> <description> Type: Incident Duration: 1 hour Affected Components: , Actions Cache → Aug 11, 19:00:10 GMT+0 - Investigating - We are observing certain customers experiencing higher than baseline cache miss rates. Aug 11, 20:00:00 GMT+0 - Resolved - We resolved the configuration error leading to the increased rate of cache misses. Cache hit rates are now at the baseline. </description> <content:encoded> <![CDATA[<p><strong>Type:</strong> Incident</p> <p><strong>Duration:</strong> 1 hour</p> <p><strong>Affected Components:</strong> </p> <p><small>Aug <var data-var='date'> 11</var>, <var data-var='time'>19:00:10</var> GMT+0</small><br><strong>Investigating</strong> - We are observing certain customers experiencing higher than baseline cache miss rates..</p> <p><small>Aug <var data-var='date'> 11</var>, <var data-var='time'>20:00:00</var> GMT+0</small><br><strong>Resolved</strong> - We resolved the configuration error leading to the increased rate of cache misses. Cache hit rates are now at the baseline..</p> ]]> </content:encoded> <pubDate>Tue, 11 Aug 2026 19:00:10 +0000</pubDate> <link>https://status.blacksmith.sh/incident/cmsp3cl3306pj10ue6i1vap76</link> <guid>https://status.blacksmith.sh/incident/cmsp3cl3306pj10ue6i1vap76</guid> </item> <item> <title>Degraded job performance, metrics, and log ingestion in us-west</title> <description> Type: Incident Duration: 1 hour and 27 minutes Affected Components: us-west x86, us-west ARM, Dashboard Aug 10, 19:17:34 GMT+0 - Investigating - We are experiencing degradation in our metrics and log ingestion services in our us-west region. We are actively investigating the issue. Aug 10, 20:44:19 GMT+0 - Resolved - This incident is resolved, with job performance and the ingestion of metrics and logs in our us-west region stable for the past 30 minutes. Jobs that failed or timed out during the incident can be safely re-run. Aug 10, 19:44:03 GMT+0 - Investigating - We are continuing to investigate degraded metrics and log ingestion in our us-west region. Customers may still see metrics and logs for their jobs appear missing or delayed in the Blacksmith dashboard, while other regions remain unaffected. We will provide another update within the next 30 minutes. Aug 10, 19:45:37 GMT+0 - Investigating - We are experiencing degraded network performance in our us-west region, affecting metrics and log ingestion as well as job performance. Jobs in us-west that upload artifacts or transfer large amounts of data may run slower than normal and in some cases hit their configured timeouts and fail. Other regions are not affected, and we are actively investigating the issue. Aug 10, 20:18:51 GMT+0 - Monitoring - Metrics and log ingestion in our us-west region is recovering, and job performance in the region has returned to normal. We are monitoring to confirm the recovery holds and are continuing to investigate the underlying cause. We will provide an update shortly. </description> <content:encoded> <![CDATA[<p><strong>Type:</strong> Incident</p> <p><strong>Duration:</strong> 1 hour and 27 minutes</p> <p><strong>Affected Components:</strong> , , </p> <p><small>Aug <var data-var='date'> 10</var>, <var data-var='time'>19:17:34</var> GMT+0</small><br><strong>Investigating</strong> - We are experiencing degradation in our metrics and log ingestion services in our us-west region. We are actively investigating the issue..</p> <p><small>Aug <var data-var='date'> 10</var>, <var data-var='time'>20:44:19</var> GMT+0</small><br><strong>Resolved</strong> - This incident is resolved, with job performance and the ingestion of metrics and logs in our us-west region stable for the past 30 minutes. Jobs that failed or timed out during the incident can be safely re-run. .</p> <p><small>Aug <var data-var='date'> 10</var>, <var data-var='time'>19:44:03</var> GMT+0</small><br><strong>Investigating</strong> - We are continuing to investigate degraded metrics and log ingestion in our us-west region. Customers may still see metrics and logs for their jobs appear missing or delayed in the Blacksmith dashboard, while other regions remain unaffected. We will provide another update within the next 30 minutes..</p> <p><small>Aug <var data-var='date'> 10</var>, <var data-var='time'>19:45:37</var> GMT+0</small><br><strong>Investigating</strong> - We are experiencing degraded network performance in our us-west region, affecting metrics and log ingestion as well as job performance. Jobs in us-west that upload artifacts or transfer large amounts of data may run slower than normal and in some cases hit their configured timeouts and fail. Other regions are not affected, and we are actively investigating the issue..</p> <p><small>Aug <var data-var='date'> 10</var>, <var data-var='time'>20:18:51</var> GMT+0</small><br><strong>Monitoring</strong> - Metrics and log ingestion in our us-west region is recovering, and job performance in the region has returned to normal. We are monitoring to confirm the recovery holds and are continuing to investigate the underlying cause. We will provide an update shortly..</p> ]]> </content:encoded> <pubDate>Mon, 10 Aug 2026 19:17:34 +0000</pubDate> <link>https://status.blacksmith.sh/incident/cmsnm5osa017t0kloby5i3lgn</link> <guid>https://status.blacksmith.sh/incident/cmsnm5osa017t0kloby5i3lgn</guid> </item> <item> <title>Stickydisk Storage cluster maintenance</title> <description> Type: Maintenance Duration: 20 days, 14 hours and 54 minutes Affected Components: us-west Storage Cluster, eu-central Storage Cluster Aug 8, 11:45:06 GMT+0 - Identified - We are increasing Ceph background ops activity, elevating the overall load. No downtime expected. Aug 8, 12:00:01 GMT+0 - Identified - Maintenance is now in progress Aug 8, 20:00:00 GMT+0 - Completed - Maintenance has completed successfully </description> <content:encoded> <![CDATA[<p><strong>Type:</strong> Maintenance</p> <p><strong>Duration:</strong> 20 days, 14 hours and 54 minutes</p> <p><strong>Affected Components:</strong> , </p> <p><small>Aug <var data-var='date'> 8</var>, <var data-var='time'>11:45:06</var> GMT+0</small><br><strong>Identified</strong> - We are increasing Ceph background ops activity, elevating the overall load. No downtime expected..</p> <p><small>Aug <var data-var='date'> 8</var>, <var data-var='time'>12:00:01</var> GMT+0</small><br><strong>Identified</strong> - Maintenance is now in progress.</p> <p><small>Aug <var data-var='date'> 8</var>, <var data-var='time'>20:00:00</var> GMT+0</small><br><strong>Completed</strong> - Maintenance has completed successfully.</p> ]]> </content:encoded> <pubDate>Sat, 8 Aug 2026 12:00:00 +0000</pubDate> <link>https://status.blacksmith.sh/maintenance/cmskb43xa0ntf0ypb7jnzs0dy</link> <guid>https://status.blacksmith.sh/maintenance/cmskb43xa0ntf0ypb7jnzs0dy</guid> </item> <item> <title>Github → Actions experiencing degraded performance</title> <description> Type: Incident Duration: 9 hours and 27 minutes Affected Components: Github → Webhooks, Github → Actions Aug 6, 15:30:48 GMT+0 - Monitoring - GitHub has reported degraded performance for Actions. Jobs may be delayed to start, or fail during the set up step. We are monitoring this incident. <https://www.githubstatus.com/incidents/qcvjkzcs7j74> Aug 7, 00:57:44 GMT+0 - Resolved - GitHub job success and adoption rates are now back at normal levels. Jobs that were not adopted during the incident are being requeued by our team and should be picked up shortly. </description> <content:encoded> <![CDATA[<p><strong>Type:</strong> Incident</p> <p><strong>Duration:</strong> 9 hours and 27 minutes</p> <p><strong>Affected Components:</strong> , </p> <p><small>Aug <var data-var='date'> 6</var>, <var data-var='time'>15:30:48</var> GMT+0</small><br><strong>Monitoring</strong> - GitHub has reported degraded performance for Actions. Jobs may be delayed to start, or fail during the set up step. We are monitoring this incident. <https://www.githubstatus.com/incidents/qcvjkzcs7j74>.</p> <p><small>Aug <var data-var='date'> 7</var>, <var data-var='time'>00:57:44</var> GMT+0</small><br><strong>Resolved</strong> - GitHub job success and adoption rates are now back at normal levels. Jobs that were not adopted during the incident are being requeued by our team and should be picked up shortly..</p> ]]> </content:encoded> <pubDate>Thu, 6 Aug 2026 15:30:48 +0000</pubDate> <link>https://status.blacksmith.sh/incident/cmshoanyw016h0kpq84ulbqw7</link> <guid>https://status.blacksmith.sh/incident/cmshoanyw016h0kpq84ulbqw7</guid> </item> <item> <title>Jobs not getting picked up</title> <description> Type: Incident Duration: 1 hour and 34 minutes Affected Components: eu-west Storage Cluster, eu-central x86, eu-west ARM, us-west x86, us-west Storage Cluster, , eu-west x86, eu-west Storage Cluster, eu-west Storage Cluster, , us-west ARM, eu-central Storage Cluster, , Codesmith, eu-central Storage Cluster, eu-central Storage Cluster, https://blacksmith.sh, us-central MacOS, eu-central ARM, us-west Storage Cluster, us-west Storage Cluster, Runtime Build Caching → Actions Cache → Website → Aug 6, 01:18:55 GMT+0 - Investigating - We're currently experiencing an outage of our control plane. GitHub job adoption and execution are affected, as well as dashboard access. Aug 6, 01:36:40 GMT+0 - Identified - We've mitigated the root cause and jobs are resuming to run. Some jobs may still be delayed to start as we catch up with the job backlog. Aug 6, 01:51:15 GMT+0 - Monitoring - Jobs adoption has recovered and are operational. Aug 6, 02:11:10 GMT+0 - Monitoring - There is still a delay in job adoption times as we continue to recover. Aug 6, 02:45:02 GMT+0 - Monitoring - Job adoption is now fully operational. Aug 6, 02:53:11 GMT+0 - Resolved - This incident has been resolved. </description> <content:encoded> <![CDATA[<p><strong>Type:</strong> Incident</p> <p><strong>Duration:</strong> 1 hour and 34 minutes</p> <p><strong>Affected Components:</strong> , , , , , , , , , , , , , , , , , , , , </p> <p><small>Aug <var data-var='date'> 6</var>, <var data-var='time'>01:18:55</var> GMT+0</small><br><strong>Investigating</strong> - We're currently experiencing an outage of our control plane. GitHub job adoption and execution are affected, as well as dashboard access..</p> <p><small>Aug <var data-var='date'> 6</var>, <var data-var='time'>01:36:40</var> GMT+0</small><br><strong>Identified</strong> - We've mitigated the root cause and jobs are resuming to run. Some jobs may still be delayed to start as we catch up with the job backlog..</p> <p><small>Aug <var data-var='date'> 6</var>, <var data-var='time'>01:51:15</var> GMT+0</small><br><strong>Monitoring</strong> - Jobs adoption has recovered and are operational..</p> <p><small>Aug <var data-var='date'> 6</var>, <var data-var='time'>02:11:10</var> GMT+0</small><br><strong>Monitoring</strong> - There is still a delay in job adoption times as we continue to recover..</p> <p><small>Aug <var data-var='date'> 6</var>, <var data-var='time'>02:45:02</var> GMT+0</small><br><strong>Monitoring</strong> - Job adoption is now fully operational..</p> <p><small>Aug <var data-var='date'> 6</var>, <var data-var='time'>02:53:11</var> GMT+0</small><br><strong>Resolved</strong> - This incident has been resolved..</p> ]]> </content:encoded> <pubDate>Thu, 6 Aug 2026 01:18:55 +0000</pubDate> <link>https://status.blacksmith.sh/incident/cmsgtv53n01de0koyjrghkms7</link> <guid>https://status.blacksmith.sh/incident/cmsgtv53n01de0koyjrghkms7</guid> </item> <item> <title>Elevated error rates in Sticky Disk and Actions Cache requests</title> <description> Type: Incident Duration: 53 minutes Affected Components: eu-west Storage Cluster, us-west Storage Cluster, , eu-west Storage Cluster, eu-west Storage Cluster, , eu-central Storage Cluster, eu-central Storage Cluster, eu-central Storage Cluster, us-west Storage Cluster, us-west Storage Cluster, Runtime Build Caching → Actions Cache → Aug 5, 22:45:00 GMT+0 - Investigating - We are seeing elevated error rates for Sticky Disk and Actions Cache requests. We are investigating. Aug 5, 22:55:00 GMT+0 - Monitoring - We implemented a fix and are currently monitoring the result. Aug 5, 23:30:42 GMT+0 - Monitoring - We are still monitoring recovery. Aug 5, 23:37:56 GMT+0 - Resolved - This incident has been resolved. </description> <content:encoded> <![CDATA[<p><strong>Type:</strong> Incident</p> <p><strong>Duration:</strong> 53 minutes</p> <p><strong>Affected Components:</strong> , , , , , , , , , , </p> <p><small>Aug <var data-var='date'> 5</var>, <var data-var='time'>22:45:00</var> GMT+0</small><br><strong>Investigating</strong> - We are seeing elevated error rates for Sticky Disk and Actions Cache requests. We are investigating..</p> <p><small>Aug <var data-var='date'> 5</var>, <var data-var='time'>22:55:00</var> GMT+0</small><br><strong>Monitoring</strong> - We implemented a fix and are currently monitoring the result..</p> <p><small>Aug <var data-var='date'> 5</var>, <var data-var='time'>23:30:42</var> GMT+0</small><br><strong>Monitoring</strong> - We are still monitoring recovery..</p> <p><small>Aug <var data-var='date'> 5</var>, <var data-var='time'>23:37:56</var> GMT+0</small><br><strong>Resolved</strong> - This incident has been resolved..</p> ]]> </content:encoded> <pubDate>Wed, 5 Aug 2026 22:45:00 +0000</pubDate> <link>https://status.blacksmith.sh/incident/cmsgpt203009k0zpm7wgqvszx</link> <guid>https://status.blacksmith.sh/incident/cmsgpt203009k0zpm7wgqvszx</guid> </item> <item> <title>Delay in Job Adoption</title> <description> Type: Incident Duration: 6 minutes Affected Components: eu-central x86, eu-west ARM, us-west x86, eu-west x86, us-west ARM, us-central MacOS, eu-central ARM Aug 3, 15:37:05 GMT+0 - Investigating - We're seeing a number of cases where jobs may not be adopted promptly. We're currently investigating. Aug 3, 15:42:46 GMT+0 - Resolved - This incident has been resolved. </description> <content:encoded> <![CDATA[<p><strong>Type:</strong> Incident</p> <p><strong>Duration:</strong> 6 minutes</p> <p><strong>Affected Components:</strong> , , , , , , </p> <p><small>Aug <var data-var='date'> 3</var>, <var data-var='time'>15:37:05</var> GMT+0</small><br><strong>Investigating</strong> - We're seeing a number of cases where jobs may not be adopted promptly. We're currently investigating. .</p> <p><small>Aug <var data-var='date'> 3</var>, <var data-var='time'>15:42:46</var> GMT+0</small><br><strong>Resolved</strong> - This incident has been resolved..</p> ]]> </content:encoded> <pubDate>Mon, 3 Aug 2026 15:37:05 +0000</pubDate> <link>https://status.blacksmith.sh/incident/cmsde76cm00ws0zmsg1kanljn</link> <guid>https://status.blacksmith.sh/incident/cmsde76cm00ws0zmsg1kanljn</guid> </item> <item> <title>US-West storage cluster degraded</title> <description> Type: Incident Duration: 21 minutes Affected Components: us-west Storage Cluster, us-west Storage Cluster, us-west Storage Cluster Aug 2, 06:16:09 GMT+0 - Investigating - We are seeing some failures with sticky disk availability and increased latency with the US-West storage cluster. Aug 2, 06:25:32 GMT+0 - Monitoring - We've applied a fix and are seeing failure rates and latency starting to come down. Aug 2, 06:37:06 GMT+0 - Resolved - This incident has been resolved. </description> <content:encoded> <![CDATA[<p><strong>Type:</strong> Incident</p> <p><strong>Duration:</strong> 21 minutes</p> <p><strong>Affected Components:</strong> , , </p> <p><small>Aug <var data-var='date'> 2</var>, <var data-var='time'>06:16:09</var> GMT+0</small><br><strong>Investigating</strong> - We are seeing some failures with sticky disk availability and increased latency with the US-West storage cluster..</p> <p><small>Aug <var data-var='date'> 2</var>, <var data-var='time'>06:25:32</var> GMT+0</small><br><strong>Monitoring</strong> - We've applied a fix and are seeing failure rates and latency starting to come down..</p> <p><small>Aug <var data-var='date'> 2</var>, <var data-var='time'>06:37:06</var> GMT+0</small><br><strong>Resolved</strong> - This incident has been resolved..</p> ]]> </content:encoded> <pubDate>Sun, 2 Aug 2026 06:16:09 +0000</pubDate> <link>https://status.blacksmith.sh/incident/cmsbepz4803j81amr292ghbj8</link> <guid>https://status.blacksmith.sh/incident/cmsbepz4803j81amr292ghbj8</guid> </item> <item> <title>Runtime Build Caching</title> <description> Type: Maintenance Duration: 1 hour and 44 minutes Affected Components: , Runtime Build Caching → Jul 30, 01:15:54 GMT+0 - Identified - We will be performing maintenance on our Runtime Build Caching infrastructure. Jobs will continue to run, but some may run without remote caching and take longer than usual during this window. No customer action is required. We expect maintenance to last approximately two hours. Jul 30, 01:30:01 GMT+0 - Identified - Maintenance is now in progress Jul 30, 03:13:57 GMT+0 - Completed - Maintenance has completed successfully. Runtime Build Caching has been fully restored and is operating normally. </description> <content:encoded> <![CDATA[<p><strong>Type:</strong> Maintenance</p> <p><strong>Duration:</strong> 1 hour and 44 minutes</p> <p><strong>Affected Components:</strong> </p> <p><small>Jul <var data-var='date'> 30</var>, <var data-var='time'>01:15:54</var> GMT+0</small><br><strong>Identified</strong> - We will be performing maintenance on our Runtime Build Caching infrastructure. Jobs will continue to run, but some may run without remote caching and take longer than usual during this window. No customer action is required. We expect maintenance to last approximately two hours..</p> <p><small>Jul <var data-var='date'> 30</var>, <var data-var='time'>01:30:01</var> GMT+0</small><br><strong>Identified</strong> - Maintenance is now in progress.</p> <p><small>Jul <var data-var='date'> 30</var>, <var data-var='time'>03:13:57</var> GMT+0</small><br><strong>Completed</strong> - Maintenance has completed successfully. Runtime Build Caching has been fully restored and is operating normally..</p> ]]> </content:encoded> <pubDate>Thu, 30 Jul 2026 01:30:00 +0000</pubDate> <link>https://status.blacksmith.sh/maintenance/cms6toa2t08re0zpa4t4fdo76</link> <guid>https://status.blacksmith.sh/maintenance/cms6toa2t08re0zpa4t4fdo76</guid> </item> <item> <title>Increased Error Rate on Codesmith Sandbox Startup</title> <description> Type: Incident Duration: 26 minutes Affected Components: Codesmith Jul 29, 16:05:10 GMT+0 - Identified - We're seeing increased error rates from an upstream provider in spinning up sandbox environments in Codesmith. We are looking into remediation options. Jul 29, 16:08:02 GMT+0 - Monitoring - The upstream issue is resolved, we are monitoring the sandbox creation rate recovering. Jul 29, 16:31:17 GMT+0 - Resolved - This incident has been resolved. </description> <content:encoded> <![CDATA[<p><strong>Type:</strong> Incident</p> <p><strong>Duration:</strong> 26 minutes</p> <p><strong>Affected Components:</strong> </p> <p><small>Jul <var data-var='date'> 29</var>, <var data-var='time'>16:05:10</var> GMT+0</small><br><strong>Identified</strong> - We're seeing increased error rates from an upstream provider in spinning up sandbox environments in Codesmith. We are looking into remediation options..</p> <p><small>Jul <var data-var='date'> 29</var>, <var data-var='time'>16:08:02</var> GMT+0</small><br><strong>Monitoring</strong> - The upstream issue is resolved, we are monitoring the sandbox creation rate recovering..</p> <p><small>Jul <var data-var='date'> 29</var>, <var data-var='time'>16:31:17</var> GMT+0</small><br><strong>Resolved</strong> - This incident has been resolved..</p> ]]> </content:encoded> <pubDate>Wed, 29 Jul 2026 16:05:10 +0000</pubDate> <link>https://status.blacksmith.sh/incident/cms6a01k304aa0rpff68zzajg</link> <guid>https://status.blacksmith.sh/incident/cms6a01k304aa0rpff68zzajg</guid> </item> <item> <title>Jobs not being adopted in all regions</title> <description> Type: Incident Duration: 34 minutes Affected Components: eu-central x86, us-west x86, eu-west ARM, eu-west x86, us-west ARM, us-central MacOS, eu-central ARM Jul 24, 19:17:34 GMT+0 - Investigating - We are seeing webhooks not being accepted by our control plane resulting in jobs not being adopted. We are currently investigating. Jul 24, 19:20:01 GMT+0 - Monitoring - We have deployed a fix and are monitoring recovery, and will provide another update within the next 30 minutes. Jul 24, 19:23:36 GMT+0 - Monitoring - We have requeued the webhooks that were not processed during the incident, and any affected jobs should start shortly. Jul 24, 19:51:29 GMT+0 - Resolved - This incident has been resolved, and job pickup has returned to normal. </description> <content:encoded> <![CDATA[<p><strong>Type:</strong> Incident</p> <p><strong>Duration:</strong> 34 minutes</p> <p><strong>Affected Components:</strong> , , , , , , </p> <p><small>Jul <var data-var='date'> 24</var>, <var data-var='time'>19:17:34</var> GMT+0</small><br><strong>Investigating</strong> - We are seeing webhooks not being accepted by our control plane resulting in jobs not being adopted. We are currently investigating..</p> <p><small>Jul <var data-var='date'> 24</var>, <var data-var='time'>19:20:01</var> GMT+0</small><br><strong>Monitoring</strong> - We have deployed a fix and are monitoring recovery, and will provide another update within the next 30 minutes..</p> <p><small>Jul <var data-var='date'> 24</var>, <var data-var='time'>19:23:36</var> GMT+0</small><br><strong>Monitoring</strong> - We have requeued the webhooks that were not processed during the incident, and any affected jobs should start shortly..</p> <p><small>Jul <var data-var='date'> 24</var>, <var data-var='time'>19:51:29</var> GMT+0</small><br><strong>Resolved</strong> - This incident has been resolved, and job pickup has returned to normal..</p> ]]> </content:encoded> <pubDate>Fri, 24 Jul 2026 19:17:34 +0000</pubDate> <link>https://status.blacksmith.sh/incident/cmrzbo7j501hn1amtrn4aj12k</link> <guid>https://status.blacksmith.sh/incident/cmrzbo7j501hn1amtrn4aj12k</guid> </item> <item> <title>Sticky disk degradation in EU-West</title> <description> Type: Incident Duration: 43 minutes Affected Components: eu-west Storage Cluster, eu-west Storage Cluster Jul 23, 15:00:51 GMT+0 - Investigating - We're seeing large volumes of traffic in our storage cluster in EU-West and this is causing a number of sticky disk related requests to time out and are actively investigating. Jul 23, 15:18:56 GMT+0 - Monitoring - We implemented a fix and are currently monitoring the result. Jul 23, 15:44:06 GMT+0 - Resolved - This incident has been resolved. </description> <content:encoded> <![CDATA[<p><strong>Type:</strong> Incident</p> <p><strong>Duration:</strong> 43 minutes</p> <p><strong>Affected Components:</strong> , </p> <p><small>Jul <var data-var='date'> 23</var>, <var data-var='time'>15:00:51</var> GMT+0</small><br><strong>Investigating</strong> - We're seeing large volumes of traffic in our storage cluster in EU-West and this is causing a number of sticky disk related requests to time out and are actively investigating..</p> <p><small>Jul <var data-var='date'> 23</var>, <var data-var='time'>15:18:56</var> GMT+0</small><br><strong>Monitoring</strong> - We implemented a fix and are currently monitoring the result..</p> <p><small>Jul <var data-var='date'> 23</var>, <var data-var='time'>15:44:06</var> GMT+0</small><br><strong>Resolved</strong> - This incident has been resolved..</p> ]]> </content:encoded> <pubDate>Thu, 23 Jul 2026 15:00:51 +0000</pubDate> <link>https://status.blacksmith.sh/incident/cmrxn27tr01xz0sk0ywgsfxlb</link> <guid>https://status.blacksmith.sh/incident/cmrxn27tr01xz0sk0ywgsfxlb</guid> </item> <item> <title>Stickydisk Storage cluster maintenance</title> <description> Type: Maintenance Duration: 4 hours and 50 minutes Affected Components: us-west Storage Cluster Jul 12, 15:05:49 GMT+0 - Identified - We will be performing Ceph major version upgrade. No downtime expected. Jul 12, 20:00:01 GMT+0 - Identified - Maintenance is now in progress Jul 13, 00:50:27 GMT+0 - Completed - Maintenance has completed successfully. </description> <content:encoded> <![CDATA[<p><strong>Type:</strong> Maintenance</p> <p><strong>Duration:</strong> 4 hours and 50 minutes</p> <p><strong>Affected Components:</strong> </p> <p><small>Jul <var data-var='date'> 12</var>, <var data-var='time'>15:05:49</var> GMT+0</small><br><strong>Identified</strong> - We will be performing Ceph major version upgrade. No downtime expected..</p> <p><small>Jul <var data-var='date'> 12</var>, <var data-var='time'>20:00:01</var> GMT+0</small><br><strong>Identified</strong> - Maintenance is now in progress.</p> <p><small>Jul <var data-var='date'> 13</var>, <var data-var='time'>00:50:27</var> GMT+0</small><br><strong>Completed</strong> - Maintenance has completed successfully..</p> ]]> </content:encoded> <pubDate>Sun, 12 Jul 2026 20:00:00 +0000</pubDate> <link>https://status.blacksmith.sh/maintenance/cmrhxe8eb0wwz0rq609hnsfw9</link> <guid>https://status.blacksmith.sh/maintenance/cmrhxe8eb0wwz0rq609hnsfw9</guid> </item> <item> <title>Stickydisk Storage clusters maintenance</title> <description> Type: Maintenance Duration: 2 hours and 34 minutes Affected Components: eu-west Storage Cluster, eu-central Storage Cluster Jul 12, 12:25:50 GMT+0 - Identified - We will be performing Ceph major version upgrade. No downtime expected. Jul 12, 12:30:01 GMT+0 - Identified - Maintenance is now in progress Jul 12, 15:03:54 GMT+0 - Completed - Maintenance has completed successfully. </description> <content:encoded> <![CDATA[<p><strong>Type:</strong> Maintenance</p> <p><strong>Duration:</strong> 2 hours and 34 minutes</p> <p><strong>Affected Components:</strong> , </p> <p><small>Jul <var data-var='date'> 12</var>, <var data-var='time'>12:25:50</var> GMT+0</small><br><strong>Identified</strong> - We will be performing Ceph major version upgrade. No downtime expected..</p> <p><small>Jul <var data-var='date'> 12</var>, <var data-var='time'>12:30:01</var> GMT+0</small><br><strong>Identified</strong> - Maintenance is now in progress.</p> <p><small>Jul <var data-var='date'> 12</var>, <var data-var='time'>15:03:54</var> GMT+0</small><br><strong>Completed</strong> - Maintenance has completed successfully..</p> ]]> </content:encoded> <pubDate>Sun, 12 Jul 2026 12:30:00 +0000</pubDate> <link>https://status.blacksmith.sh/maintenance/cmrhrohyl0tje1aq69fycs87h</link> <guid>https://status.blacksmith.sh/maintenance/cmrhrohyl0tje1aq69fycs87h</guid> </item> <item> <title>Maintenance on our Actions Cache storage cluster</title> <description> Type: Maintenance Duration: 1 hour Affected Components: , Actions Cache → Jul 2, 23:58:28 GMT+0 - Identified - We are planning for a scheduled maintenance on our Actions Cache storage cluster. This is to address a kernel bug that we have discovered resulting in sporadic instability on the machines hosted this cluster. Customers can expect to see blips in their Actions Cache reads and writes during this period but they should not be fatal to the job. At the end of the maintenance all cached data will be reset so your jobs may run slower as they hydrate this cache. Jul 3, 00:00:01 GMT+0 - Identified - Maintenance is now in progress Jul 3, 01:00:00 GMT+0 - Completed - Maintenance has completed successfully </description> <content:encoded> <![CDATA[<p><strong>Type:</strong> Maintenance</p> <p><strong>Duration:</strong> 1 hour</p> <p><strong>Affected Components:</strong> </p> <p><small>Jul <var data-var='date'> 2</var>, <var data-var='time'>23:58:28</var> GMT+0</small><br><strong>Identified</strong> - We are planning for a scheduled maintenance on our Actions Cache storage cluster. This is to address a kernel bug that we have discovered resulting in sporadic instability on the machines hosted this cluster. Customers can expect to see blips in their Actions Cache reads and writes during this period but they should not be fatal to the job. At the end of the maintenance all cached data will be reset so your jobs may run slower as they hydrate this cache..</p> <p><small>Jul <var data-var='date'> 3</var>, <var data-var='time'>00:00:01</var> GMT+0</small><br><strong>Identified</strong> - Maintenance is now in progress.</p> <p><small>Jul <var data-var='date'> 3</var>, <var data-var='time'>01:00:00</var> GMT+0</small><br><strong>Completed</strong> - Maintenance has completed successfully.</p> ]]> </content:encoded> <pubDate>Fri, 3 Jul 2026 00:00:00 +0000</pubDate> <link>https://status.blacksmith.sh/maintenance/cmr460po500bz1amgcft4fzow</link> <guid>https://status.blacksmith.sh/maintenance/cmr460po500bz1amgcft4fzow</guid> </item> <item> <title>Actions cache cluster expansion</title> <description> Type: Maintenance Duration: 1 hour Affected Components: , Actions Cache → Jul 2, 00:30:00 GMT+0 - Identified - We are planning for a scheduled maintenance during that time. Cache states will be reset upon completion. Jul 2, 01:00:00 GMT+0 - Completed - Maintenance has completed successfully. </description> <content:encoded> <![CDATA[<p><strong>Type:</strong> Maintenance</p> <p><strong>Duration:</strong> 1 hour</p> <p><strong>Affected Components:</strong> </p> <p><small>Jul <var data-var='date'> 2</var>, <var data-var='time'>00:30:00</var> GMT+0</small><br><strong>Identified</strong> - We are planning for a scheduled maintenance during that time. Cache states will be reset upon completion..</p> <p><small>Jul <var data-var='date'> 2</var>, <var data-var='time'>01:00:00</var> GMT+0</small><br><strong>Completed</strong> - Maintenance has completed successfully..</p> ]]> </content:encoded> <pubDate>Thu, 2 Jul 2026 00:30:00 +0000</pubDate> <link>https://status.blacksmith.sh/maintenance/cmr2ts91s00fe1ap42jq7t564</link> <guid>https://status.blacksmith.sh/maintenance/cmr2ts91s00fe1ap42jq7t564</guid> </item> <item> <title>Emergency Maintenance on EU Storage Cluster</title> <description> Type: Maintenance Affected Components: eu-central Storage Cluster, eu-central Storage Cluster, eu-central Storage Cluster Jun 11, 15:58:54 GMT+0 - Identified - We are planning for a scheduled maintenance during that time. Jun 11, 15:59:04 GMT+0 - Completed - Maintenance has completed successfully. </description> <content:encoded> <![CDATA[<p><strong>Type:</strong> Maintenance</p> <p><strong>Affected Components:</strong> , , </p> <p><small>Jun <var data-var='date'> 11</var>, <var data-var='time'>15:58:54</var> GMT+0</small><br><strong>Identified</strong> - We are planning for a scheduled maintenance during that time..</p> <p><small>Jun <var data-var='date'> 11</var>, <var data-var='time'>15:59:04</var> GMT+0</small><br><strong>Completed</strong> - Maintenance has completed successfully..</p> ]]> </content:encoded> <pubDate>Thu, 11 Jun 2026 14:30:00 +0000</pubDate> <link>https://status.blacksmith.sh/maintenance/cmq9on339033zo7a0id0d4y0x</link> <guid>https://status.blacksmith.sh/maintenance/cmq9on339033zo7a0id0d4y0x</guid> </item> <item> <title>Emergency Maintenance on US Storage Cluster</title> <description> Type: Maintenance Duration: 1 hour and 15 minutes Affected Components: us-west Storage Cluster, us-west Storage Cluster, us-west Storage Cluster Jun 11, 02:55:02 GMT+0 - Identified - We are planning for a scheduled maintenance during that time. Jun 11, 02:55:19 GMT+0 - Identified - Maintenance is now in progress. Jun 11, 04:10:37 GMT+0 - Completed - Maintenance has completed successfully. </description> <content:encoded> <![CDATA[<p><strong>Type:</strong> Maintenance</p> <p><strong>Duration:</strong> 1 hour and 15 minutes</p> <p><strong>Affected Components:</strong> , , </p> <p><small>Jun <var data-var='date'> 11</var>, <var data-var='time'>02:55:02</var> GMT+0</small><br><strong>Identified</strong> - We are planning for a scheduled maintenance during that time..</p> <p><small>Jun <var data-var='date'> 11</var>, <var data-var='time'>02:55:19</var> GMT+0</small><br><strong>Identified</strong> - Maintenance is now in progress..</p> <p><small>Jun <var data-var='date'> 11</var>, <var data-var='time'>04:10:37</var> GMT+0</small><br><strong>Completed</strong> - Maintenance has completed successfully..</p> ]]> </content:encoded> <pubDate>Thu, 11 Jun 2026 03:00:00 +0000</pubDate> <link>https://status.blacksmith.sh/maintenance/cmq8wn14603k7pdmmh8j0nnjo</link> <guid>https://status.blacksmith.sh/maintenance/cmq8wn14603k7pdmmh8j0nnjo</guid> </item> <item> <title>Emergency Maintenance on US Storage Cluster</title> <description> Type: Maintenance Duration: 1 hour Affected Components: us-west Storage Cluster, us-west Storage Cluster, us-west Storage Cluster Jun 10, 01:00:00 GMT+0 - Identified - We're seeing some service degradation post maintenance so we need to extend the maintenance window. Jun 10, 01:00:01 GMT+0 - Identified - Maintenance is now in progress Jun 10, 02:00:00 GMT+0 - Completed - Maintenance has completed successfully </description> <content:encoded> <![CDATA[<p><strong>Type:</strong> Maintenance</p> <p><strong>Duration:</strong> 1 hour</p> <p><strong>Affected Components:</strong> , , </p> <p><small>Jun <var data-var='date'> 10</var>, <var data-var='time'>01:00:00</var> GMT+0</small><br><strong>Identified</strong> - We're seeing some service degradation post maintenance so we need to extend the maintenance window..</p> <p><small>Jun <var data-var='date'> 10</var>, <var data-var='time'>01:00:01</var> GMT+0</small><br><strong>Identified</strong> - Maintenance is now in progress.</p> <p><small>Jun <var data-var='date'> 10</var>, <var data-var='time'>02:00:00</var> GMT+0</small><br><strong>Completed</strong> - Maintenance has completed successfully.</p> ]]> </content:encoded> <pubDate>Wed, 10 Jun 2026 01:00:00 +0000</pubDate> <link>https://status.blacksmith.sh/maintenance/cmq7ctwim01g5p4uo3ixg5qms</link> <guid>https://status.blacksmith.sh/maintenance/cmq7ctwim01g5p4uo3ixg5qms</guid> </item> <item> <title>US Storage Cluster</title> <description> Type: Maintenance Duration: 20 minutes Affected Components: us-west Storage Cluster, us-west Storage Cluster, us-west Storage Cluster Jun 9, 23:30:00 GMT+0 - Identified - We're planning an emergency maintenance for the US Storage Cluster. Jun 9, 23:30:01 GMT+0 - Identified - Maintenance is now in progress Jun 9, 23:50:10 GMT+0 - Completed - Maintenance has completed successfully. </description> <content:encoded> <![CDATA[<p><strong>Type:</strong> Maintenance</p> <p><strong>Duration:</strong> 20 minutes</p> <p><strong>Affected Components:</strong> , , </p> <p><small>Jun <var data-var='date'> 9</var>, <var data-var='time'>23:30:00</var> GMT+0</small><br><strong>Identified</strong> - We're planning an emergency maintenance for the US Storage Cluster..</p> <p><small>Jun <var data-var='date'> 9</var>, <var data-var='time'>23:30:01</var> GMT+0</small><br><strong>Identified</strong> - Maintenance is now in progress.</p> <p><small>Jun <var data-var='date'> 9</var>, <var data-var='time'>23:50:10</var> GMT+0</small><br><strong>Completed</strong> - Maintenance has completed successfully..</p> ]]> </content:encoded> <pubDate>Tue, 9 Jun 2026 23:30:00 +0000</pubDate> <link>https://status.blacksmith.sh/maintenance/cmq78r1nc00kzqjwx5a6p7s6k</link> <guid>https://status.blacksmith.sh/maintenance/cmq78r1nc00kzqjwx5a6p7s6k</guid> </item> <item> <title>Emergency Maintenance on EU Storage Cluster</title> <description> Type: Maintenance Duration: 1 hour and 19 minutes Affected Components: eu-central Storage Cluster, eu-central Storage Cluster, eu-central Storage Cluster May 1, 20:30:00 GMT+0 - Identified - We are planning for a emergency maintenance at this time. During this maintenance the Storage Cluster components will have degraded performance. Existing caches will be wiped upon maintenance completion. May 1, 20:30:00 GMT+0 - Identified - We are planning for a emergency maintenance at this time. During this maintenance the Storage Cluster components will have degraded performance. Existing caches will be wiped upon maintenance completion. May 1, 21:49:24 GMT+0 - Completed - Maintenance has completed successfully. </description> <content:encoded> <![CDATA[<p><strong>Type:</strong> Maintenance</p> <p><strong>Duration:</strong> 1 hour and 19 minutes</p> <p><strong>Affected Components:</strong> , , </p> <p><small>May <var data-var='date'> 1</var>, <var data-var='time'>20:30:00</var> GMT+0</small><br><strong>Identified</strong> - We are planning for a emergency maintenance at this time. During this maintenance the Storage Cluster components will have degraded performance. Existing caches will be wiped upon maintenance completion..</p> <p><small>May <var data-var='date'> 1</var>, <var data-var='time'>20:30:00</var> GMT+0</small><br><strong>Identified</strong> - We are planning for a emergency maintenance at this time. During this maintenance the Storage Cluster components will have degraded performance. Existing caches will be wiped upon maintenance completion..</p> <p><small>May <var data-var='date'> 1</var>, <var data-var='time'>21:49:24</var> GMT+0</small><br><strong>Completed</strong> - Maintenance has completed successfully..</p> ]]> </content:encoded> <pubDate>Fri, 1 May 2026 20:30:00 +0000</pubDate> <link>https://status.blacksmith.sh/maintenance/cmond3m6703tc3j9pctepw6ab</link> <guid>https://status.blacksmith.sh/maintenance/cmond3m6703tc3j9pctepw6ab</guid> </item> <item> <title>Storage cluster maintenance</title> <description> Type: Maintenance Duration: 1 hour Affected Components: , , , Sticky Disks → Docker Container Cache → Incremental Docker Builders → Apr 18, 16:00:00 GMT+0 - Identified - We are planning for a scheduled maintenance during that time. Apr 18, 16:00:01 GMT+0 - Identified - Maintenance is now in progress Apr 18, 17:00:00 GMT+0 - Completed - Maintenance has completed successfully </description> <content:encoded> <![CDATA[<p><strong>Type:</strong> Maintenance</p> <p><strong>Duration:</strong> 1 hour</p> <p><strong>Affected Components:</strong> , , </p> <p><small>Apr <var data-var='date'> 18</var>, <var data-var='time'>16:00:00</var> GMT+0</small><br><strong>Identified</strong> - We are planning for a scheduled maintenance during that time..</p> <p><small>Apr <var data-var='date'> 18</var>, <var data-var='time'>16:00:01</var> GMT+0</small><br><strong>Identified</strong> - Maintenance is now in progress.</p> <p><small>Apr <var data-var='date'> 18</var>, <var data-var='time'>17:00:00</var> GMT+0</small><br><strong>Completed</strong> - Maintenance has completed successfully.</p> ]]> </content:encoded> <pubDate>Sat, 18 Apr 2026 16:00:00 +0000</pubDate> <link>https://status.blacksmith.sh/maintenance/cmo3v7gpf05zh113vmek4t8cd</link> <guid>https://status.blacksmith.sh/maintenance/cmo3v7gpf05zh113vmek4t8cd</guid> </item> <item> <title>Storage cluster expansion in US-West</title> <description> Type: Maintenance Duration: 1 hour Affected Components: , Incremental Docker Builders → Apr 12, 04:00:00 GMT+0 - Identified - We are planning for a scheduled maintenance during that time. Apr 12, 04:00:01 GMT+0 - Identified - Maintenance is now in progress Apr 12, 05:00:00 GMT+0 - Completed - Maintenance has completed successfully </description> <content:encoded> <![CDATA[<p><strong>Type:</strong> Maintenance</p> <p><strong>Duration:</strong> 1 hour</p> <p><strong>Affected Components:</strong> </p> <p><small>Apr <var data-var='date'> 12</var>, <var data-var='time'>04:00:00</var> GMT+0</small><br><strong>Identified</strong> - We are planning for a scheduled maintenance during that time..</p> <p><small>Apr <var data-var='date'> 12</var>, <var data-var='time'>04:00:01</var> GMT+0</small><br><strong>Identified</strong> - Maintenance is now in progress.</p> <p><small>Apr <var data-var='date'> 12</var>, <var data-var='time'>05:00:00</var> GMT+0</small><br><strong>Completed</strong> - Maintenance has completed successfully.</p> ]]> </content:encoded> <pubDate>Sun, 12 Apr 2026 04:00:00 +0000</pubDate> <link>https://status.blacksmith.sh/maintenance/cmntoekwg05r1lkmpl5bgu89o</link> <guid>https://status.blacksmith.sh/maintenance/cmntoekwg05r1lkmpl5bgu89o</guid> </item> <item> <title>Running emergency maintenance on our storage cluster to size it up and eliminate degraded interactions</title> <description> Type: Maintenance Duration: 20 minutes Affected Components: , Incremental Docker Builders → Apr 8, 18:45:00 GMT+0 - Identified - Maintenance is now in progress. Apr 8, 18:35:00 GMT+0 - Identified - We are planning for an emergency maintenance on our storage cluster to size it up to eliminate degraded interactions. All sticky disk cache entries will be wiped as part of this maintenance. Apr 8, 18:55:38 GMT+0 - Completed - Maintenance has completed successfully. </description> <content:encoded> <![CDATA[<p><strong>Type:</strong> Maintenance</p> <p><strong>Duration:</strong> 20 minutes</p> <p><strong>Affected Components:</strong> </p> <p><small>Apr <var data-var='date'> 8</var>, <var data-var='time'>18:45:00</var> GMT+0</small><br><strong>Identified</strong> - Maintenance is now in progress..</p> <p><small>Apr <var data-var='date'> 8</var>, <var data-var='time'>18:35:00</var> GMT+0</small><br><strong>Identified</strong> - We are planning for an emergency maintenance on our storage cluster to size it up to eliminate degraded interactions. All sticky disk cache entries will be wiped as part of this maintenance..</p> <p><small>Apr <var data-var='date'> 8</var>, <var data-var='time'>18:55:38</var> GMT+0</small><br><strong>Completed</strong> - Maintenance has completed successfully..</p> ]]> </content:encoded> <pubDate>Wed, 8 Apr 2026 18:35:00 +0000</pubDate> <link>https://status.blacksmith.sh/maintenance/cmnqe334y014214aryjb4qz5n</link> <guid>https://status.blacksmith.sh/maintenance/cmnqe334y014214aryjb4qz5n</guid> </item> <item> <title>Actions Cache Capacity Upgrade</title> <description> Type: Maintenance Duration: 15 minutes Affected Components: , Actions Cache → Mar 20, 01:00:00 GMT+0 - Identified - We are upgrading the Actions Cache capacity, cache performance may be degraded during this maintenance window. Mar 20, 01:00:01 GMT+0 - Identified - Maintenance is now in progress Mar 20, 01:15:00 GMT+0 - Completed - Maintenance has completed successfully </description> <content:encoded> <![CDATA[<p><strong>Type:</strong> Maintenance</p> <p><strong>Duration:</strong> 15 minutes</p> <p><strong>Affected Components:</strong> </p> <p><small>Mar <var data-var='date'> 20</var>, <var data-var='time'>01:00:00</var> GMT+0</small><br><strong>Identified</strong> - We are upgrading the Actions Cache capacity, cache performance may be degraded during this maintenance window..</p> <p><small>Mar <var data-var='date'> 20</var>, <var data-var='time'>01:00:01</var> GMT+0</small><br><strong>Identified</strong> - Maintenance is now in progress.</p> <p><small>Mar <var data-var='date'> 20</var>, <var data-var='time'>01:15:00</var> GMT+0</small><br><strong>Completed</strong> - Maintenance has completed successfully.</p> ]]> </content:encoded> <pubDate>Fri, 20 Mar 2026 01:00:00 +0000</pubDate> <link>https://status.blacksmith.sh/maintenance/cmmxvkdzi01z8ozdt5iktinib</link> <guid>https://status.blacksmith.sh/maintenance/cmmxvkdzi01z8ozdt5iktinib</guid> </item> <item> <title>Scheduled maintenance of our database </title> <description> Type: Maintenance Duration: 4 minutes Affected Components: us-west ARM, eu-central x86, us-west x86, eu-central ARM, , Blacksmith Managed Runners → Feb 28, 18:07:00 GMT+0 - Identified - We are running an operation on our database that may result in a short blip in service. Customers may notice longer queue times during this window. Feb 28, 18:05:51 GMT+0 - Identified - Maintenance is now in progress. Feb 28, 18:10:10 GMT+0 - Completed - Maintenance has completed successfully. </description> <content:encoded> <![CDATA[<p><strong>Type:</strong> Maintenance</p> <p><strong>Duration:</strong> 4 minutes</p> <p><strong>Affected Components:</strong> , , , , </p> <p><small>Feb <var data-var='date'> 28</var>, <var data-var='time'>18:07:00</var> GMT+0</small><br><strong>Identified</strong> - We are running an operation on our database that may result in a short blip in service. Customers may notice longer queue times during this window..</p> <p><small>Feb <var data-var='date'> 28</var>, <var data-var='time'>18:05:51</var> GMT+0</small><br><strong>Identified</strong> - Maintenance is now in progress..</p> <p><small>Feb <var data-var='date'> 28</var>, <var data-var='time'>18:10:10</var> GMT+0</small><br><strong>Completed</strong> - Maintenance has completed successfully..</p> ]]> </content:encoded> <pubDate>Sat, 28 Feb 2026 18:07:00 +0000</pubDate> <link>https://status.blacksmith.sh/maintenance/cmm6mtfip0dlxbpwwxx888gdk</link> <guid>https://status.blacksmith.sh/maintenance/cmm6mtfip0dlxbpwwxx888gdk</guid> </item> <item> <title>Running emergency maintenance on our storage cluster to size it up and eliminate degraded interactions</title> <description> Type: Maintenance Duration: 9 minutes Affected Components: , Incremental Docker Builders → Feb 25, 14:00:00 GMT+0 - Identified - We are planning for an emergency maintenance on our storage cluster to size it up to eliminate degraded interactions. All sticky disk cache entries will be wiped as part of this maintenance. Feb 26, 00:28:40 GMT+0 - Identified - Maintenance is now in progress. Feb 26, 00:37:41 GMT+0 - Completed - Maintenance has completed successfully the storage cluster is back to serving requests normally. </description> <content:encoded> <![CDATA[<p><strong>Type:</strong> Maintenance</p> <p><strong>Duration:</strong> 9 minutes</p> <p><strong>Affected Components:</strong> </p> <p><small>Feb <var data-var='date'> 25</var>, <var data-var='time'>14:00:00</var> GMT+0</small><br><strong>Identified</strong> - We are planning for an emergency maintenance on our storage cluster to size it up to eliminate degraded interactions. All sticky disk cache entries will be wiped as part of this maintenance..</p> <p><small>Feb <var data-var='date'> 26</var>, <var data-var='time'>00:28:40</var> GMT+0</small><br><strong>Identified</strong> - Maintenance is now in progress..</p> <p><small>Feb <var data-var='date'> 26</var>, <var data-var='time'>00:37:41</var> GMT+0</small><br><strong>Completed</strong> - Maintenance has completed successfully the storage cluster is back to serving requests normally..</p> ]]> </content:encoded> <pubDate>Wed, 25 Feb 2026 14:00:00 +0000</pubDate> <link>https://status.blacksmith.sh/maintenance/cmm2q3x9r0i1tg3s9gyg409u3</link> <guid>https://status.blacksmith.sh/maintenance/cmm2q3x9r0i1tg3s9gyg409u3</guid> </item> <item> <title>Upgrading storage fleet in our EU region</title> <description> Type: Maintenance Duration: 1 hour Affected Components: , Incremental Docker Builders → Aug 6, 18:30:00 GMT+0 - Identified - We are going to be running an important upgrade to our storage fleet cluster that may cause interruptions in stickydisk and docker caching. The upgrade will cause a reset of all stickydisk entities causing runs to be uncached for the first time until the cache is rehydrated. Aug 6, 19:30:00 GMT+0 - Completed - Maintenance has completed successfully. </description> <content:encoded> <![CDATA[<p><strong>Type:</strong> Maintenance</p> <p><strong>Duration:</strong> 1 hour</p> <p><strong>Affected Components:</strong> </p> <p><small>Aug <var data-var='date'> 6</var>, <var data-var='time'>18:30:00</var> GMT+0</small><br><strong>Identified</strong> - We are going to be running an important upgrade to our storage fleet cluster that may cause interruptions in stickydisk and docker caching. The upgrade will cause a reset of all stickydisk entities causing runs to be uncached for the first time until the cache is rehydrated..</p> <p><small>Aug <var data-var='date'> 6</var>, <var data-var='time'>19:30:00</var> GMT+0</small><br><strong>Completed</strong> - Maintenance has completed successfully..</p> ]]> </content:encoded> <pubDate>Wed, 6 Aug 2025 18:30:00 +0000</pubDate> <link>https://status.blacksmith.sh/maintenance/cme0amp1c01moptag4xxn1qh0</link> <guid>https://status.blacksmith.sh/maintenance/cme0amp1c01moptag4xxn1qh0</guid> </item> <item> <title>Landing page undergoing maintenance</title> <description> Type: Maintenance Duration: 47 minutes Affected Components: , Website → Jun 30, 21:30:00 GMT+0 - Identified - There is ongoing maintenance on our landing page ([blacksmith.sh](http://blacksmith.sh)), but our dashboard and support portals are unaffected <https://app.blacksmith.sh/> Jun 30, 22:16:52 GMT+0 - Completed - Maintenance has completed successfully. </description> <content:encoded> <), but our dashboard and support portals are unaffected <https://app.blacksmith.sh/>.</p> <p><small>Jun <var data-var='date'> 30</var>, <var data-var='time'>22:16:52</var> GMT+0</small><br><strong>Completed</strong> - Maintenance has completed successfully..</p> ]]> </content:encoded> <pubDate>Mon, 30 Jun 2025 21:30:00 +0000</pubDate> <link>https://status.blacksmith.sh/maintenance/cmcjmx54k0003ac3okuvr9tv3</link> <guid>https://status.blacksmith.sh/maintenance/cmcjmx54k0003ac3okuvr9tv3</guid> </item> </channel> </rss>