Snapshot 302592
Raw (normalized text)
Scripts and page chrome removed; this is what change detection compares.
{
"incidents": [
{
"created_at": "2026-11-18T15:40:44Z",
"id": "01M7QN2JMC1XD44C22VB124Z17",
"impact": "minor",
"incident_updates": [
{
"body": "The issue at ElevenLabs is resolved. Voice agent replies on calls and requests to create or update agents through the dashboard and API are back to normal. The problem affected agents using ElevenLabs voices from about 15:15 UTC to 21:10 UTC.\n\nTo protect your agents from a provider issue like this, we recommend setting a fallback voice from a different provider. Open your agent, go to **Voice**, set **Fallback Voice Model** to a different provider (for example, **Synthflow TTS**), and choose a **Fallback Voice** close to your main one. If you switched your voice model to Synthflow TTS during the incident, you can switch back now.\n\n\u003e If you have any questions or are still seeing issues, please contact Synthflow Support by submitting a ticket at \u003chttps://helpcenter.synthflow.ai\u003e.",
"created_at": "2026-11-18T21:12:28Z",
"display_at": "2026-11-18T21:12:28Z",
"id": "01M7R81ZM6CHQDY6A7AHTSY9CX",
"incident_id": "01M7QN2JMC1XD44C22VB124Z17",
"status": "resolved",
"updated_at": "2026-11-18T21:12:28Z"
},
{
"body": "Because of an issue at ElevenLabs, some customers’ voice agents are slow to reply on calls, and some replies don’t come through. Until ElevenLabs has fixed the issue, you can avoid it in one of two ways:\n\n1. **Switch voice model:** open your agent, go to **Voice**, set **Voice Model** to **Synthflow TTS**, choose a voice and save.\n2. **Add a fallback (Enterprise plans):** in the same **Voice** section, set **Fallback Voice Model** to **Synthflow TTS** and choose a **Fallback Voice** close to your current one. ",
"created_at": "2026-11-18T17:20:21Z",
"display_at": "2026-11-18T17:20:21Z",
"id": "01M7QTRZDWMWPFG4754QP7YZZ7",
"incident_id": "01M7QN2JMC1XD44C22VB124Z17",
"status": "identified",
"updated_at": "2026-11-18T17:20:21Z"
},
{
"body": "Since about 15:16 UTC, some customers have had slow responses when creating new voice agents in the dashboard or through the API. Some requests take more than 10 seconds to finish. \n\nWe have identified this issue is due to an incident that a downstream vendor ElevenLabs is having: \u003chttps://status.elevenlabs.io/united-states\u003e. This incident only affect agents using Elevenlabs text to speech. \n\nWe are working with the vendor to resolve the issue. ",
"created_at": "2026-11-18T15:40:44Z",
"display_at": "2026-11-18T15:40:44Z",
"id": "01M7QN2JMCXHBP5JHNV5G2797A",
"incident_id": "01M7QN2JMC1XD44C22VB124Z17",
"status": "identified",
"updated_at": "2026-11-18T15:40:44Z"
}
],
"name": "Degraded calls and API calls for agents using ElevenLabs voices",
"page_id": "01KX5ZM3M45N00Z1PM4V3P76WJ",
"resolved_at": "2026-11-18T21:12:28Z",
"status": "resolved",
"updated_at": "2026-11-18T21:12:28Z"
},
{
"created_at": "2026-11-17T15:20:33Z",
"id": "01M7N1GWX7Z8DTRNZS8JSNTQTC",
"impact": "minor",
"incident_updates": [
{
"body": "**Summary**\nFrom 13:02 to 14:48 GMT on 17 November, some inbound calls on our US region failed to start. This affected voice agents that have a call transfer action set up. We rolled back the release that caused it, and calls have connected normally since then.\n\n**Impact**\nFor about one hour and 48 minutes, some inbound calls to voice agents with a call transfer action did not start. This happened whether or not the “transfer availability” setting was on. Agents without a transfer action were not affected. Customers on our EU region or Global region were not affected either.\n\n**Data impact**\nNo customer data was lost or changed. Calls that failed during this window never started, so there are no recordings or transcripts for them.\n\n**Root cause**\nWe released an update to the service that handles live calls, and it was built on a new Docker base image. That image was missing time zone information. Agents with a transfer action check the current time to decide whether a transfer is available. Without time zone information that check failed, and the call could not start. Our automated pre-release tests did not cover this case, so the problem reached production.\n\n**Timeline (GMT)**\n\n- **17 Nov, 13:02:** We released the update to the US region.\n- **17 Nov, 13:32:** We received reports of inbound calls failing to start.\n- **17 Nov, 14:30:** We traced the failures to the new release and the missing time zone information.\n- **17 Nov, 14:35:** We started rolling back the release.\n- **17 Nov, 14:48:** The rollback finished and inbound calls were connecting normally again.\n\n**What we’re doing to prevent this**\n\n- We’re fixing the missing time zone information before we release this update again.\n- We’re adding call transfer actions to the automated tests that every release must pass.\n- We’re looking into why our existing pre-release checks and alerts didn’t catch the failure.\n\nWe’re sorry for the disruption.\n\n\u003e If you have any questions or are still seeing issues, please contact Synthflow Support by submitting a ticket at \u003chttps://helpcenter.synthflow.ai\u003e.",
"created_at": "2026-11-17T15:20:33Z",
"display_at": "2026-11-17T15:15:00Z",
"id": "01M7N1GWX7QEVH5YBV7MGMCGK4",
"incident_id": "01M7N1GWX7Z8DTRNZS8JSNTQTC",
"status": "resolved",
"updated_at": "2026-11-17T15:23:25Z"
}
],
"name": "Inbound calls failing to connect for some agents with call transfers",
"page_id": "01KX5ZM3M45N00Z1PM4V3P76WJ",
"resolved_at": "2026-11-17T15:15:00Z",
"status": "resolved",
"updated_at": "2026-11-17T15:20:33Z"
},
{
"created_at": "2026-10-28T10:49:22Z",
"id": "01M6121Z1A9BDRATFCJCJ2C3B8",
"impact": "minor",
"incident_updates": [
{
"body": "**Summary**\nFrom Tuesday 27th October, some customers who use sub-accounts couldn’t see some or all of their sub-account workspaces in the dashboard. Access has been restored for every affected workspace.\n\n**Impact**\nAffected users couldn’t see or open some of their sub-account workspaces from their main workspace. Calls and voice agents kept running as normal.\n\n**Data impact**\nNo data was lost. The sub-account workspaces and everything in them stayed intact. Only the link that lets the main workspace see them was affected. We contacted each affected customer directly.\n\n**Root cause**\nThis was caused by the same issue with one of our third-party providers behind the Settings and SSO login incident on 27–28 October. That issue cleared the information that links some sub-account workspaces to their main workspace. Without that link, the sub-accounts stopped showing in the dashboard. We found this while checking for any further effects of the earlier incident.\n\n**What we’re doing to prevent this**\n\n• The safeguards we added on 28 October now also stop the link between a main workspace and its sub-accounts from being cleared if a third-party provider fails.\n\n• We’re moving off this provider so this can’t happen again.",
"created_at": "2026-11-02T10:30:32Z",
"display_at": "2026-11-02T10:30:32Z",
"id": "01M6DWZ2JZF5QJP0CDE7FG1CF9",
"incident_id": "01M6121Z1A9BDRATFCJCJ2C3B8",
"status": "resolved",
"updated_at": "2026-11-02T10:30:32Z"
},
{
"body": "**Summary**\nFrom 17:15 UTC on 27 October to 08:42 UTC on 28 October, some customers in our US and EU regions could not open the Settings page, and some could not log in with SSO. Both problems are now fixed and have not come back.\n\n**Impact**\nAffected users saw the Settings page fail to load, and some SSO logins failed. Calls and voice agents kept running as normal.\n\n**Data impact**\nA small number of workspaces may have lost some workspace-level settings, such as the workspace name. No customer data was lost. We will contact each affected workspace directly to help restore these settings.\n\n**Root cause**\nAn issue with one of our third-party providers caused some workspace settings, including the workspace’s region, to be cleared. Without these settings, the Settings page and SSO login stopped working for affected workspaces. Our first fix restored the settings, but they were cleared again until we found and fixed the underlying cause.\n\n**Timeline (UTC)**\n\n• **27 Oct, 17:15:** We started investigating reports of Settings and SSO login failures.\n\n• **27 Oct, 18:33:** We restored settings for the affected workspaces.\n\n• **27 Oct, 21:11:** Some settings were being cleared again. We found the cause and began restoring all affected workspaces.\n\n• **28 Oct, 08:42:** We finished restoring every affected workspace and added safeguards. Our checks found no affected workspaces after this.\n\n**What we’re doing to prevent this**\n\n• We’ve added safeguards that stop workspace settings from being cleared if a third-party provider fails.\n\n• To migrate off this provider to prevent this issue from happening again in the future\n\nWe’re sorry for the disruption. If you still see problems with Settings or SSO login, please contact support at support@synthflow.ai, or reach out to your account representative. ",
"created_at": "2026-10-28T10:49:22Z",
"display_at": "2026-10-27T11:27:00Z",
"id": "01M6121Z1ACPNKBSC2WJJQJMRS",
"incident_id": "01M6121Z1A9BDRATFCJCJ2C3B8",
"status": "resolved",
"updated_at": "2026-10-28T10:49:22Z"
}
],
"name": "Settings page and SSO login unavailable for some workspaces",
"page_id": "01KX5ZM3M45N00Z1PM4V3P76WJ",
"resolved_at": "2026-10-27T11:27:00Z",
"status": "resolved",
"updated_at": "2026-11-02T10:30:32Z"
},
{
"created_at": "2026-10-26T21:00:21Z",
"id": "01M5X078ZQG4XJK6YY6CFGG4ES",
"impact": "critical",
"incident_updates": [
{
"body": "Access to the Synthflow dashboard and normal API performance have been restored. The issue was caused by a downstream vendor and has now been fixed.\n\nLive calling remained operational throughout the incident. We apologize for the disruption.",
"created_at": "2026-10-26T22:11:55Z",
"display_at": "2026-10-26T22:11:55Z",
"id": "01M5X4AA85FF9FKV87QJN5P2BG",
"incident_id": "01M5X078ZQG4XJK6YY6CFGG4ES",
"status": "resolved",
"updated_at": "2026-10-26T22:11:55Z"
},
{
"body": "We are aware of an issue affecting access to the Synthflow dashboard. Some users may also see degraded API performance, such as slower response times or intermittent errors. Our team is working to restore full service as quickly as possible. \n\nLive calling is operating normally. Agents continue to place and receive calls without interruption.\n\nWe will post another update within 30 minutes, or sooner as new information becomes available. We apologise for the inconvenience.",
"created_at": "2026-10-26T21:00:21Z",
"display_at": "2026-10-26T21:00:21Z",
"id": "01M5X078ZQHCMER5C9EM0BHJKT",
"incident_id": "01M5X078ZQG4XJK6YY6CFGG4ES",
"status": "identified",
"updated_at": "2026-10-26T21:00:21Z"
}
],
"name": "Platform ",
"page_id": "01KX5ZM3M45N00Z1PM4V3P76WJ",
"resolved_at": "2026-10-26T22:11:55Z",
"status": "resolved",
"updated_at": "2026-10-26T22:11:55Z"
},
{
"created_at": "2026-10-26T16:24:27Z",
"id": "01M5WGE2S5W2GYYSC80H0KYG1A",
"impact": "minor",
"incident_updates": [
{
"body": "**Summary**\n\nBetween 12:00 UTC 23th October and 16:00 26th October UTC, some requests to update a subaccount through our public API failed with a 400 error if they included the `operation_type` or `products` fields. The cause was an internal infrastructure change that moved some accounts to a new system for subaccount settings before that system supported these two fields. We reverted the change for all affected accounts, and updates are working normally again.\n\n**Impact**\n\n• **Affected endpoint:** PUT /v2/subaccounts/{subaccount_id}\n\n• **Symptom:** requests that included `operation_type` or `products` returned 400 \"Not supported for this workspace.\" No changes from those requests were saved, including other fields sent in the same request.\n\n• **Not affected:** creating, listing, retrieving and deleting subaccounts; updates that did not include these two fields; live agents and calls.\n\n**Root cause**\n\nA recent update to how we process subaccount changes did not support the `operation_type` and `products` fields, so requests that included them were rejected. Our pre-release checks did not catch the change in API behaviour.\n\n**Resolution**\n\nWe reverted the change for all affected accounts. Subaccount updates that include `operation_type` and `products` are now accepted and saved as expected.\n\n**Prevention**\n\nWe are strengthening our checks so that changes cannot alter documented API behaviour without being caught before release. We are also reviewing the other subaccount fields to confirm they are not affected.\n\n**What you need to do**\n\nIf you received a 400 error on a subaccount update during this window, please retry the request. The failed request saved nothing, so a retry is safe.\n\nWe apologise for the disruption. If you have questions, please contact support@synthflow.ai.",
"created_at": "2026-10-26T16:24:27Z",
"display_at": "2026-10-26T15:17:00Z",
"id": "01M5WGE2S56RMDZAQ17ZBJMPJJ",
"incident_id": "01M5WGE2S5W2GYYSC80H0KYG1A",
"status": "resolved",
"updated_at": "2026-10-26T16:27:23Z"
}
],
"name": "Errors when updating subaccounts via the API",
"page_id": "01KX5ZM3M45N00Z1PM4V3P76WJ",
"resolved_at": "2026-10-26T15:17:00Z",
"status": "resolved",
"updated_at": "2026-10-26T16:24:27Z"
},
{
"created_at": "2026-09-07T13:15:09Z",
"id": "01M1Y0482504EYAHVES8T1X034",
"impact": "minor",
"incident_updates": [
{
"body": "This issue is now resolved. **\n\nRCA:** \n\n**Impact:** Users were unable to log in to white-label workspaces using custom domains. Standard workspaces were unaffected.\n\n**Root cause:** A change introduced during an internal migration caused an expected issue affecting only custom domain workspaces. \n\n**Resolution:** We have applied the fix, and introduced enhanced QA processes specific for custom domain workspaces \n\nWe apologise for the disruption.",
"created_at": "2026-09-07T15:42:19Z",
"display_at": "2026-09-07T15:42:19Z",
"id": "01M1Y8HQ0VTYPTRJYFW9D5W49D",
"incident_id": "01M1Y0482504EYAHVES8T1X034",
"status": "resolved",
"updated_at": "2026-09-07T15:42:19Z"
},
{
"body": "We are aware of an issue preventing end users from logging in to white-label workspaces, those configured with a customer’s own domain and branding. This issue is limited to custom-domain workspaces. All other services are operating normally.",
"created_at": "2026-09-07T13:15:09Z",
"display_at": "2026-09-07T13:15:09Z",
"id": "01M1Y04825WPNV9AC27CB928Y6",
"incident_id": "01M1Y0482504EYAHVES8T1X034",
"status": "investigating",
"updated_at": "2026-09-07T13:15:09Z"
}
],
"name": "Login issue affecting white-label workspaces",
"page_id": "01KX5ZM3M45N00Z1PM4V3P76WJ",
"resolved_at": "2026-09-07T15:42:19Z",
"status": "resolved",
"updated_at": "2026-09-07T15:42:19Z"
},
{
"created_at": "2026-08-17T11:40:32Z",
"id": "01M07RBXMMS1XMKKJY2MX3CK8W",
"impact": "minor",
"incident_updates": [
{
"body": "**What happened**\n\nA change to our internal telephony configuration was rolled out to production and introduced an unexpected fault in call routing.\n\n**What was affected**\n\n• **EU region:** calls failed from 10:32–10:44 UTC (12:32–12:44 CEST) — 12 minutes.\n\n• **US region:** calls failed from 10:33–10:42 UTC (12:33–12:42 CEST) — 9 minutes.\n\n• **All regions:** warm transfers failed intermittently from 07:00–10:42 UTC (09:00–12:42 CEST) — 3 hours 42 minutes.\n\nAll services were fully restored by 10:44 UTC. No customer data was affected.\n\n**What we're doing next**\n\nWe have rolled back the change and added a pre-deployment check to catch this class of routing fault before it reaches production. A full root cause analysis will be published here shortly. \n\nWe're sorry for the disruption.",
"created_at": "2026-08-17T11:40:32Z",
"display_at": "2026-08-17T08:17:00Z",
"id": "01M07RBXMMYP2JHD7PJ7YXXVZS",
"incident_id": "01M07RBXMMS1XMKKJY2MX3CK8W",
"status": "resolved",
"updated_at": "2026-08-17T11:40:32Z"
}
],
"name": "Call failures and degraded performance on call transfers ",
"page_id": "01KX5ZM3M45N00Z1PM4V3P76WJ",
"resolved_at": "2026-08-17T08:17:00Z",
"status": "resolved",
"updated_at": "2026-08-17T11:40:32Z"
},
{
"created_at": "2026-08-07T13:57:01Z",
"id": "01KZE86M0Z0B0Y602D5H2DBK7V",
"impact": "minor",
"incident_updates": [
{
"body": "**What happened**\n\nAn internal configuration change restricted which AI models the Synthflow platform was permitted to call. The restriction was applied more broadly than intended and blocked the models used by our post-call processing pipeline.\n\nAs a result, AI-generated post-call outputs failed to produce calls completed during the window. The configuration was reverted at 11:10 UTC and all functionality returned to normal immediately.\n\n**What was affected**\n\nCalls placed and received during this window connected, ran, and completed normally. The failures were confined to AI processing within the post call webhook that runs **after** a call ends.\n\nThe following were affected:\n\n_Info extractors_: Extracted variables not populated\n\n_Call summary_: Not generated\n\n_LLM-as-a-judge_: Did not run\n\n_Custom evaluations_: Did not run\n\n_Post-call SMS actions_: Not triggered or sent\n\nIf you consume the post-call webhook, the webhook is still fired and delivered but these fields were empty or missing in the payload for affected calls.\n\n**What was not affected**\n\n• Inbound and outbound call connectivity\n\n• Live conversation with your agents during the call\n\n• Call recordings and audio\n\n• Transcripts\n\n• Post-call webhook delivery itself (the webhook fired as normal)\n\n• Call logs, billing records, and the dashboard\n\n**What you should do**\n\nIf your workflows depend on extracted variables, summaries, or post-call SMS for calls completed between 08:20 and 11:10 UTC on 7 August, treat those records as incomplete. Recordings and transcripts for those calls are intact.\n\n**What we're changing**\n\nWe've completed an internal review of this incident and are making improvements in three areas:\n\n• **Detection** — strengthening monitoring so issues of this type are surfaced and acted on faster.\n\n• **Change controls** — tightening the review and approval process for platform configuration changes.\n\n• **Impact review** — improving how we assess the downstream reach of a change before it is applied.",
"created_at": "2026-08-07T13:57:01Z",
"display_at": "2026-08-07T08:30:00Z",
"id": "01KZE86M0Z5XPWCNTHPZQAMEG7",
"incident_id": "01KZE86M0Z0B0Y602D5H2DBK7V",
"status": "resolved",
"updated_at": "2026-08-07T13:57:01Z"
}
],
"name": "Failures within our post call webhook ",
"page_id": "01KX5ZM3M45N00Z1PM4V3P76WJ",
"resolved_at": "2026-08-07T08:30:00Z",
"status": "resolved",
"updated_at": "2026-08-07T13:57:01Z"
},
{
"created_at": "2026-07-10T12:24:49Z",
"id": "01KX5ZSP9FYMQN36THESDZBNMZ",
"impact": "major",
"incident_updates": [
{
"body": "**Service Incident Report — US Voice Service Disruption**\n\n**Date:** July 6, 2026\n\n**Status:** Resolved\n\n**Summary**\n\nOn July 6, 2026, our US voice service experienced a brief disruption lasting approximately 18 minutes. During this window, calls to and from US-based voice agents failed to connect. The issue has been fully resolved, and all services have returned to normal operation.\n\n**What happened**\n\nDuring planned maintenance on our US infrastructure, a routine cleanup step inadvertently removed a network component that our US telephony service depended on. This severed the connection our call-routing system needed to handle calls, causing them to fail to connect.\n\n**Impact**\n\n• **Duration:** Approximately 18 minutes\n\n• **Region affected:** US only. Our EU and global regions were not affected.\n\n• **Effect:** Inbound and outbound calls to US voice agents did not connect during the affected window. All EU and global traffic was unaffected.\n\n**Resolution**\n\nOur engineering team identified the cause quickly and restored the connection, after which calls recovered immediately.\n\n**Preventing recurrence**\n\nWe take reliability seriously and have already begun work to prevent this from happening again, including:\n\n• Reviewing our infrastructure configuration to remove reliance on any undocumented dependencies\n\n• Introducing additional review and safeguards for changes to production systems\n\n• Adopting a stricter decommissioning checklist that verifies real usage before any shared infrastructure is removed\n\nWe apologize for any disruption this may have caused and appreciate your understanding. If you have any questions or noticed continued issues, please reach out to our support team.",
"created_at": "2026-07-10T12:24:49Z",
"display_at": "2026-07-06T17:21:00Z",
"id": "01KX5ZSP9FSBFAWCYYJXQ1GWRF",
"incident_id": "01KX5ZSP9FYMQN36THESDZBNMZ",
"status": "resolved",
"updated_at": "2026-07-10T12:24:49Z"
},
{
"body": "The issue is now resolved, traffic is flowing again. We will provide an RCA shortly.",
"created_at": "2026-07-10T12:24:49Z",
"display_at": "2026-07-06T15:58:00Z",
"id": "01KX5ZSP9FTQJ3BR0YX19EJFK4",
"incident_id": "01KX5ZSP9FYMQN36THESDZBNMZ",
"status": "resolved",
"updated_at": "2026-07-10T12:24:49Z"
},
{
"body": "We are experiencing degraded performance on our calling infrastructure in the US cluster. This only affects clusters on the US cluster.",
"created_at": "2026-07-10T12:24:49Z",
"display_at": "2026-07-06T15:43:00Z",
"id": "01KX5ZSP9FG07WYV0XBH76PY5V",
"incident_id": "01KX5ZSP9FYMQN36THESDZBNMZ",
"status": "identified",
"updated_at": "2026-07-10T12:24:49Z"
}
],
"name": "US cluster - Degraded performance on end to end calling",
"page_id": "01KX5ZM3M45N00Z1PM4V3P76WJ",
"resolved_at": "2026-07-06T15:58:00Z",
"status": "resolved",
"updated_at": "2026-07-10T12:24:49Z"
},
{
"created_at": "2026-07-10T12:24:49Z",
"id": "01KX5ZSNQSW8DETJQ14HWQYQBK",
"impact": "none",
"incident_updates": [
{
"body": "**June 16, 2026 Synthflow RCA**\n\n****Increased Rate Limiting (HTTP 429\\) in the Synthflow API****\n\n****Summary****\n\nOn June 16, 2026, we experienced a significant increase in overall traffic, which triggered Synthflow’s service-wide rate limiting rules. Users of the Synthflow API would have experienced HTTP 429 Too Many Requests errors. Our service-wide rate limits were set too low for historical purposes. We increased our service-wide rate limits and we are planning further improvements to monitoring and rate limiting.\n\n****Customer impact****\n\nFrom 18:39-20:48 UTC on June 16, 2026, customers experienced intermittent rate limiting when making API requests or using the user interface. Phone calls were unaffected.\n\n****Root cause****\n\nSynthflow’s rate limits have been in place for more than 6 months as-is. When a request is sent to Synthflow, the request routes through Cloudflare, then GCP Cloud Armor, then various internal Synthflow components. Within this pipeline, Cloudflare IPs were treated as the originating traffic IP. As a result, rate limits were applied across the service rather than per IP. Similarly, we relaxed the time windows within our policies to match traffic patterns.\n\nThe combination of limiting by Cloudflare IPs instead of originating IPs and an increase in traffic caused rate limits to trigger for multiple customers during the increase in traffic. The configuration changes during the incident allowed acceptable traffic to return to normal rather than being rate limited.\n\n****Next Steps****\n\nWe apologize for the challenges this service-wide limitation caused for you. We are continuing to work to ensure we have the right limits in place to ensure continued high levels of service availability. We are taking the following next steps to improve our service level protections:\n\n• Measure and install limits that match current traffic patterns.\n\n• Monitor shifts in traffic and how rate limits should be applied on an ongoing basis.\n\n• Verify all changes to rate limiting go through our standard change management process (done).\n\n• Relax time window for applying rate limits to allow for more traffic. (done)\n\n• Update our rate limiting to properly use originating IPs. (done)",
"created_at": "2026-07-10T12:24:49Z",
"display_at": "2026-06-16T20:48:00Z",
"id": "01KX5ZSNQS4BDS4MXS16ZYT986",
"incident_id": "01KX5ZSNQSW8DETJQ14HWQYQBK",
"status": "resolved",
"updated_at": "2026-07-10T12:24:49Z"
}
],
"name": "Increased Rate Limiting (HTTP 429) in the Synthflow API",
"page_id": "01KX5ZM3M45N00Z1PM4V3P76WJ",
"resolved_at": "2026-06-16T20:48:00Z",
"status": "resolved",
"updated_at": "2026-07-10T12:24:49Z"
},
{
"created_at": "2026-07-10T12:24:48Z",
"id": "01KX5ZSN5SY6BM5YQ0T00JTGF5",
"impact": "minor",
"incident_updates": [
{
"body": "Resolved",
"created_at": "2026-07-10T12:24:48Z",
"display_at": "2026-06-11T19:20:00Z",
"id": "01KX5ZSN5SPNY75D4R6PFZ71SD",
"incident_id": "01KX5ZSN5SY6BM5YQ0T00JTGF5",
"status": "resolved",
"updated_at": "2026-07-10T12:24:48Z"
},
{
"body": "Due to a database memory spike, we had limietd degraded performance during this window.",
"created_at": "2026-07-10T12:24:48Z",
"display_at": "2026-06-11T18:40:00Z",
"id": "01KX5ZSN5S76EJWJSV5TD5JS7R",
"incident_id": "01KX5ZSN5SY6BM5YQ0T00JTGF5",
"status": "identified",
"updated_at": "2026-07-10T12:24:48Z"
}
],
"name": "Degraded performance",
"page_id": "01KX5ZM3M45N00Z1PM4V3P76WJ",
"resolved_at": "2026-06-11T19:20:00Z",
"status": "resolved",
"updated_at": "2026-07-10T12:24:48Z"
},
{
"created_at": "2026-07-10T12:24:47Z",
"id": "01KX5ZSMKQH76519BF07E0PV7J",
"impact": "minor",
"incident_updates": [
{
"body": "The issue has been resolved.",
"created_at": "2026-07-10T12:24:48Z",
"display_at": "2026-05-27T17:55:00Z",
"id": "01KX5ZSMKQV0V9ZS2TB4D88VN7",
"incident_id": "01KX5ZSMKQH76519BF07E0PV7J",
"status": "resolved",
"updated_at": "2026-07-10T12:24:48Z"
},
{
"body": "We are seeing an evaluation in failed calls. We are investigating the issue.",
"created_at": "2026-07-10T12:24:48Z",
"display_at": "2026-05-27T17:39:00Z",
"id": "01KX5ZSMKQW78GK0JVDKJQ8HRS",
"incident_id": "01KX5ZSMKQH76519BF07E0PV7J",
"status": "identified",
"updated_at": "2026-07-10T12:24:48Z"
}
],
"name": "We are investigating an issue",
"page_id": "01KX5ZM3M45N00Z1PM4V3P76WJ",
"resolved_at": "2026-05-27T17:55:00Z",
"status": "resolved",
"updated_at": "2026-07-10T12:24:47Z"
},
{
"created_at": "2026-07-10T12:24:47Z",
"id": "01KX5ZSM1T6KTEGWW1F24A8TWF",
"impact": "minor",
"incident_updates": [
{
"body": "This issue has been resolved.",
"created_at": "2026-07-10T12:24:47Z",
"display_at": "2026-05-27T10:29:00Z",
"id": "01KX5ZSM1TKXGCKJ1E4VXS3KYK",
"incident_id": "01KX5ZSM1T6KTEGWW1F24A8TWF",
"status": "resolved",
"updated_at": "2026-07-10T12:24:47Z"
},
{
"body": "This issue only affects customers on the EU cluster.",
"created_at": "2026-07-10T12:24:47Z",
"display_at": "2026-05-27T10:14:00Z",
"id": "01KX5ZSM1T5SZ23SQ1GBCVPPHB",
"incident_id": "01KX5ZSM1T6KTEGWW1F24A8TWF",
"status": "monitoring",
"updated_at": "2026-07-10T12:24:47Z"
},
{
"body": "We are experiencing degraded performance on end to end calling. We are investigating the issue.",
"created_at": "2026-07-10T12:24:47Z",
"display_at": "2026-05-27T10:06:00Z",
"id": "01KX5ZSM1TE4CAA39RV2MFSVKJ",
"incident_id": "01KX5ZSM1T6KTEGWW1F24A8TWF",
"status": "identified",
"updated_at": "2026-07-10T12:24:47Z"
}
],
"name": "[EU region] Degraded performance on calls",
"page_id": "01KX5ZM3M45N00Z1PM4V3P76WJ",
"resolved_at": "2026-05-27T10:29:00Z",
"status": "resolved",
"updated_at": "2026-07-10T12:24:47Z"
},
{
"created_at": "2026-07-10T12:24:46Z",
"id": "01KX5ZSKESKADMYG46XZG849NX",
"impact": "major",
"incident_updates": [
{
"body": "**Voice Agents Not Responding During Phone Calls**\n\n**Summary**\n\n**Impact**: Customers in all regions – Global, US, and EU – experienced unresponsive voice agents. The most common symptom was a silent agent. In some cases, silent agent failures were intermittent.\n\n**Cause**: Our primary inference provider blocked access to models via automated detection of unusual behavior, per the provider. At the time of this writing, the provider has not provided an explanation of the reason for blocking access. When access is blocked, a 403 is returned for any inference call to that provider, leading to our agents failing.\n\n**Steps to resolve**: We removed usage of the primary provider and rerouted inference traffic to another provider, most notably, OpenAI.\n\n**Service Disruption Impact \u0026 Timeline**\n\nBetween approximately 16:48 UTC and 19:39 UTC on Thursday, May 14, 2026, Synthflow experienced a disruption to voice agent functionality across all three production clusters: Global, US, and EU. The trigger was an abrupt block applied to almost all our GPT models by our primary inference provider’s automated abuse-detection system, which caused requests to our inference provider to fail with HTTP 403 Forbidden errors simultaneously across every region. Because the block was applied across models, our region-to-region failover provided little protection, and all customer traffic dependent on the affected models began failing the moment enforcement took effect. The first internal failure reports arrived at 16:48 UTC.\n\nFor customers, the disruption surfaced primarily as:\n\n• Answered calls with silence when a model response was expected (i.e. an agent should respond with voice);\n\n• Calls that terminated unexpectedly mid-conversation;\n\n• A sharp increase in failed or unanswered calls in customer dashboards.\n\nInitial mitigation, which moved most inference traffic to OpenAI, restored service for the majority of agents within approximately 50 minutes of the incident being declared. A subset of customers continued to experience intermittent failures for an additional \\~90 minutes due to a remainder of LLM-dependent traffic still routing to the pre-existing inference provider. A hotfix was applied to reroute the remainder of the traffic.\n\nThe incident was declared resolved at 19:39 UTC once all inference traffic had been migrated to the secondary provider, error rates had returned to baseline across all clusters, and customer-initiated test calls were completed end-to-end successfully.\n\n**Root Cause**\n\nSynthflow did not successfully handle a failure scenario where an inference provider begins erroring across the majority of API requests. During this incident, our primary inference provider blocked almost all inference traffic due to unusual activity. Synthflow is designed with redundancy built into its inference layer for scenarios where our inference provider is unavailable. Synthflow’s architecture includes an inference router to handle such failure scenarios. In certain situations, the inference router did not apply failover mechanisms in real-time.\n\n**Background on Synthflow’s Inference Routing**\n\nLLM API calls required by our voice agents are funneled through our inference router rather than each of our components communicating to inference providers directly. The router maintains a continuously updated pool of model deployments: dozens of endpoints spanning multiple inference providers, multiple geographic regions, and multiple model variants of each generation. For every deployment, it keeps a rolling window of recent latency and error statistics and uses those signals to choose the endpoint expected to give the fastest healthy response for the model the caller has asked for. Deployments that start failing or slowing down are automatically demoted. Deployments that recover are reintroduced. As designed, this architecture provides us with a way to automatically handle poorly performing, or offline inference providers.\n\nThe same mechanism is what gives us cross-provider failover: because the pool contains deployments from more than one inference provider, a localized failure on one of them naturally drains away as its endpoints lose health, and traffic concentrates on whichever provider is currently responding well for the model in question. Under normal failure modes – partial outages, regional brownouts, individual model deployments becoming blocked – this typically resolves itself fast enough that the average customer never sees a failed call.\n\n**Why failover was slower than expected this time**\n\nThis incident exposed some parts of our system that did not route through our router, showing points of failure previously not observed. We initially believed failovers were not happening and took action to remove the failing provider manually. We made a configuration change to remove the unhealthy (primary) provider from the overall pool of available providers. However, after taking this action and monitoring, we observed a number of failing API requests continuing to route to that provider.\n\nAfter further review, we observed some older parts of the codebase still make direct connections for inference. That meant the router's shift to the healthy provider only resolved part of the impact. Two areas of our codebase kept hitting the unhealthy provider directly. Moving them to the secondary provider required a code change and hotfix release to remove direct dependencies on the failing provider.\n\n**Next Steps**\n\nWe understand how an outage such as this can affect you and your business. It’s important we’ve not only resolved this incident, but that we are also taking crucial steps to improve our service to avoid a scenario such as this again. Here are the primary actions we’re taking throughout the coming week:\n\n• Continue working with our primary inference provider to understand why our model deployments were blocked; we will share an amended update here when we have additional information.\n\n• Provide per-customer inference deployments to our enterprise customers in order to reduce the impact of a regional deployment outage.\n\n• Improve our inference router to handle wide-scale issues with a given provider as described in the failure scenario above.\n\n• Finalize the removal of any direct inference communication that does not currently go through our router.",
"created_at": "2026-07-10T12:24:46Z",
"display_at": "2026-05-18T18:47:00Z",
"id": "01KX5ZSKES6X2HE2W064F91YC0",
"incident_id": "01KX5ZSKESKADMYG46XZG849NX",
"status": "resolved",
"updated_at": "2026-07-10T12:24:46Z"
},
{
"body": "We have resolved the issue with our provider and calls have resumed working. We will share more details separately.",
"created_at": "2026-07-10T12:24:46Z",
"display_at": "2026-05-14T19:35:00Z",
"id": "01KX5ZSKESSZQJV0TQJPSC1SFG",
"incident_id": "01KX5ZSKESKADMYG46XZG849NX",
"status": "resolved",
"updated_at": "2026-07-10T12:24:46Z"
},
{
"body": "We have identified the issue was due to a provider failing, we have migrated traffic to another provider. Calls should be working again. We are monitoring the situation.",
"created_at": "2026-07-10T12:24:46Z",
"display_at": "2026-05-14T17:40:00Z",
"id": "01KX5ZSKESTG8432J5MMDSRCD6",
"incident_id": "01KX5ZSKESKADMYG46XZG849NX",
"status": "monitoring",
"updated_at": "2026-07-10T12:24:46Z"
},
{
"body": "We are investigating a spike in calls failing. We will report here as we find more.",
"created_at": "2026-07-10T12:24:46Z",
"display_at": "2026-05-14T17:18:00Z",
"id": "01KX5ZSKESDZ0J6JME58Q34WP6",
"incident_id": "01KX5ZSKESKADMYG46XZG849NX",
"status": "identified",
"updated_at": "2026-07-10T12:24:46Z"
}
],
"name": "Service degradation",
"page_id": "01KX5ZM3M45N00Z1PM4V3P76WJ",
"resolved_at": "2026-05-14T19:35:00Z",
"status": "resolved",
"updated_at": "2026-07-10T12:24:46Z"
},
{
"created_at": "2026-07-15T09:39:11Z",
"id": "01KXJJ9ZR7Q0H4JJ7FKM0NJFXJ",
"impact": "minor",
"incident_updates": [
{
"body": "Resolved.",
"created_at": "2026-07-15T09:39:11Z",
"display_at": "2026-04-30T13:00:00Z",
"id": "01KXJJ9ZR79C7K3RE83VHQDE5R",
"incident_id": "01KXJJ9ZR7Q0H4JJ7FKM0NJFXJ",
"status": "resolved",
"updated_at": "2026-07-15T09:39:11Z"
},
{
"body": "**Summary**\n\nBetween 12:29 UTC and 14:00 UTC, a subset of customers experienced agents that were unresponsive or significantly delayed. The disruption was caused by an upstream incident at Microsoft Azure affecting the Azure OpenAI service in the East US 2 region, which caused authentication failures on requests to certain GPT-4.1 model deployments.\n\n**What happened**\n\nAt approximately 12:29 UTC, a portion of our model traffic to Azure OpenAI's East US 2 region began failing with authentication errors. Our credentials had not changed, and we confirmed our configuration was correct on our side — ruling out any issue within our platform. Further investigation confirmed an active Azure-side incident impacting OpenAI services in East US 2.\n\n**Impact**\n\n• **Duration:** ~1 hour 31 minutes (12:29 UTC – 14:00 UTC)\n\n• **Scope:** A subset of agent traffic dependent on the affected Azure region\n\n• **Symptoms:** Agents not responding, elevated response latency, request timeouts\n\n**Resolution**\n\nThe incident resolved automatically once Microsoft restored service in the East US 2 region. No changes were required on our side, confirming the issue originated upstream at Azure.\n\n**Follow-up actions**\n\nWhile the root cause was external, we are taking steps to reduce the impact of similar upstream provider incidents in the future:\n\n1. Expanding automatic failover coverage so authentication issues from any single provider region are transparently rerouted to healthy regions.\n\n2. Broadening the set of error conditions that trigger immediate failover.\n\n3. Improving monitoring of direct provider incident notifications, as public provider status dashboards did not reflect the outage in a timely manner.\n\nWe apologize for the disruption and appreciate your patience while Azure restored service.",
"created_at": "2026-07-15T09:39:11Z",
"display_at": "2026-04-30T11:39:00Z",
"id": "01KXJJ9ZR7P4710FT90Z9NSAD1",
"incident_id": "01KXJJ9ZR7Q0H4JJ7FKM0NJFXJ",
"status": "identified",
"updated_at": "2026-07-15T09:39:11Z"
}
],
"name": "Degraded agent performance due to upstream Azure OpenAI incident (East US 2)",
"page_id": "01KX5ZM3M45N00Z1PM4V3P76WJ",
"resolved_at": "2026-04-30T13:00:00Z",
"status": "resolved",
"updated_at": "2026-07-15T09:39:11Z"
},
{
"created_at": "2026-07-10T12:24:46Z",
"id": "01KX5ZSJWAY09K4CCRZ93MS482",
"impact": "none",
"incident_updates": [
{
"body": "Resolved.",
"created_at": "2026-07-10T12:24:46Z",
"display_at": "2026-04-30T13:00:00Z",
"id": "01KX5ZSJWAYPDF7T0GXFRYGM04",
"incident_id": "01KX5ZSJWAY09K4CCRZ93MS482",
"status": "resolved",
"updated_at": "2026-07-10T12:24:46Z"
},
{
"body": "**Summary**\n\nBetween 12:29 UTC and 14:00 UTC, a subset of customers experienced agents that were unresponsive or significantly delayed. The disruption was caused by an upstream incident at Microsoft Azure affecting the Azure OpenAI service in the East US 2 region, which caused authentication failures on requests to certain GPT-4.1 model deployments.\n\n**What happened**\n\nAt approximately 12:29 UTC, a portion of our model traffic to Azure OpenAI's East US 2 region began failing with authentication errors. Our credentials had not changed, and we confirmed our configuration was correct on our side — ruling out any issue within our platform. Further investigation confirmed an active Azure-side incident impacting OpenAI services in East US 2.\n\n**Impact**\n\n• **Duration:** ~1 hour 31 minutes (12:29 UTC – 14:00 UTC)\n\n• **Scope:** A subset of agent traffic dependent on the affected Azure region\n\n• **Symptoms:** Agents not responding, elevated response latency, request timeouts\n\n**Resolution**\n\nThe incident resolved automatically once Microsoft restored service in the East US 2 region. No changes were required on our side, confirming the issue originated upstream at Azure.\n\n**Follow-up actions**\n\nWhile the root cause was external, we are taking steps to reduce the impact of similar upstream provider incidents in the future:\n\n1. Expanding automatic failover coverage so authentication issues from any single provider region are transparently rerouted to healthy regions.\n\n2. Broadening the set of error conditions that trigger immediate failover.\n\n3. Improving monitoring of direct provider incident notifications, as public provider status dashboards did not reflect the outage in a timely manner.\n\nWe apologize for the disruption and appreciate your patience while Azure restored service.",
"created_at": "2026-07-10T12:24:46Z",
"display_at": "2026-04-30T11:39:00Z",
"id": "01KX5ZSJWA5WZ779H7HDRT7S7Y",
"incident_id": "01KX5ZSJWAY09K4CCRZ93MS482",
"status": "resolved",
"updated_at": "2026-07-10T12:24:46Z"
}
],
"name": "Degraded agent performance due to upstream Azure OpenAI incident (East US 2)",
"page_id": "01KX5ZM3M45N00Z1PM4V3P76WJ",
"resolved_at": "2026-04-30T11:39:00Z",
"status": "resolved",
"updated_at": "2026-07-10T12:24:46Z"
},
{
"created_at": "2026-07-10T12:24:45Z",
"id": "01KX5ZSJADZ172QAME2DNPPT3M",
"impact": "none",
"incident_updates": [
{
"body": "This incident has been resolved. No user traffic was affected - all requests were handled by our stable service\n\nDuring a routine deployment, an internal configuration change introduced an inconsistency in our traffic routing system, causing an internal reconciliation loop. No user traffic was affected - all requests were served by the stable service throughout. The affected deployment was rolled back and normal operation was restored.",
"created_at": "2026-07-10T12:24:45Z",
"display_at": "2026-02-24T14:02:00Z",
"id": "01KX5ZSJADFNPHZG1YH8B939C1",
"incident_id": "01KX5ZSJADZ172QAME2DNPPT3M",
"status": "resolved",
"updated_at": "2026-07-10T12:24:45Z"
},
{
"body": "We are currently investigating an issue affecting the API service. Our team has identified the root cause and is actively applying a fix. No data loss is involved. Some API requests may experience degraded performance during this time.\n\nETA for resolution: ~15 minutes.\n\nWe will update this page once the issue is resolved.",
"created_at": "2026-07-10T12:24:45Z",
"display_at": "2026-02-24T13:45:00Z",
"id": "01KX5ZSJAD9FQBG210HFHZXQN3",
"incident_id": "01KX5ZSJADZ172QAME2DNPPT3M",
"status": "resolved",
"updated_at": "2026-07-10T12:24:45Z"
}
],
"name": "API Routing Disruption",
"page_id": "01KX5ZM3M45N00Z1PM4V3P76WJ",
"resolved_at": "2026-02-24T13:45:00Z",
"status": "resolved",
"updated_at": "2026-07-10T12:24:45Z"
},
{
"created_at": "2026-07-10T12:24:45Z",
"id": "01KX5ZSHRTC8F1CQSMY7GC2T3V",
"impact": "none",
"incident_updates": [
{
"body": "We are currently experiencing outbound call failures for customers using our RingCentral integration configured with the sip12.ringcentral.com SIP proxy. This only affects customers on that specific configuration. All other Synthflow customers are unaffected.\n\nCalls may fail with a “403 Forbidden” response.\n\nOur team is actively working with RingCentral to resolve the issue as quickly as possible. At this time, the impact is limited to customers routed through the sip12 registrar.\n\nWe will provide further updates as soon as more information becomes available.",
"created_at": "2026-07-10T12:24:45Z",
"display_at": "2026-02-16T11:33:00Z",
"id": "01KX5ZSHRT3V51AG1YTZM78H3B",
"incident_id": "01KX5ZSHRTC8F1CQSMY7GC2T3V",
"status": "resolved",
"updated_at": "2026-07-10T12:24:45Z"
}
],
"name": "Limited issues for RingCentral number",
"page_id": "01KX5ZM3M45N00Z1PM4V3P76WJ",
"resolved_at": "2026-02-16T11:33:00Z",
"status": "resolved",
"updated_at": "2026-07-10T12:24:45Z"
},
{
"created_at": "2026-07-10T12:24:44Z",
"id": "01KX5ZSH77WTHSPC2XJPJ2NE64",
"impact": "none",
"incident_updates": [
{
"body": "We have identified cases where the voice agent prematurely terminates speech output during a response. This is due an external vendor issue we are investigating.",
"created_at": "2026-07-10T12:24:44Z",
"display_at": "2026-01-21T14:55:00Z",
"id": "01KX5ZSH77KYE6XJZVF0V74H1M",
"incident_id": "01KX5ZSH77WTHSPC2XJPJ2NE64",
"status": "resolved",
"updated_at": "2026-07-10T12:24:44Z"
}
],
"name": "Voice quality issue",
"page_id": "01KX5ZM3M45N00Z1PM4V3P76WJ",
"resolved_at": "2026-01-21T14:55:00Z",
"status": "resolved",
"updated_at": "2026-07-10T12:24:44Z"
},
{
"created_at": "2026-07-10T12:24:43Z",
"id": "01KX5ZSGKQN0XKJN1R72AESH7H",
"impact": "minor",
"incident_updates": [
{
"body": "**Incident Summary (RCA)**\n\nEarlier today, some calls experienced media delays due to an issue affecting part of our media infrastructure.\n\n**What Happened**\n\nA configuration issue combined with a network condition caused one of our media services to become unavailable after a restart. Traffic was automatically routed to an alternate, healthy region to maintain call continuity, which resulted in temporary media delays for some calls.\n\n**Resolution**\n\nThe configuration issue was corrected, new media services were brought online, and traffic was safely restored to the primary region. Call media performance has fully recovered.\n\n**Prevention**\n\nWe are adding additional monitoring and alerting to detect similar issues earlier and reduce recovery time in the future.",
"created_at": "2026-07-10T12:24:43Z",
"display_at": "2025-12-19T09:43:00Z",
"id": "01KX5ZSGKQK1FK7Q1MBGSMB3C5",
"incident_id": "01KX5ZSGKQN0XKJN1R72AESH7H",
"status": "resolved",
"updated_at": "2026-07-10T12:24:43Z"
},
{
"body": "The issue is resolved.",
"created_at": "2026-07-10T12:24:43Z",
"display_at": "2025-12-18T19:08:00Z",
"id": "01KX5ZSGKQAW7DBBHFSA1MWP59",
"incident_id": "01KX5ZSGKQN0XKJN1R72AESH7H",
"status": "resolved",
"updated_at": "2026-07-10T12:24:43Z"
},
{
"body": "This incident has restarted, we are still investigating the cause.",
"created_at": "2026-07-10T12:24:43Z",
"display_at": "2025-12-18T18:14:00Z",
"id": "01KX5ZSGKQBMZZYTBAC0XWBB6B",
"incident_id": "01KX5ZSGKQN0XKJN1R72AESH7H",
"status": "monitoring",
"updated_at": "2026-07-10T12:24:43Z"
},
{
"body": "This issue has been resolved. We will post an RCA here shortly.",
"created_at": "2026-07-10T12:24:43Z",
"display_at": "2025-12-18T17:39:00Z",
"id": "01KX5ZSGKQHN5K4H67ZHHDP0YS",
"incident_id": "01KX5ZSGKQN0XKJN1R72AESH7H",
"status": "resolved",
"updated_at": "2026-07-10T12:24:43Z"
},
{
"body": "This issue is only affecting outbound calls. We are applying a fix.",
"created_at": "2026-07-10T12:24:43Z",
"display_at": "2025-12-18T17:22:00Z",
"id": "01KX5ZSGKQRDJX38JDKYDEPWCG",
"incident_id": "01KX5ZSGKQN0XKJN1R72AESH7H",
"status": "monitoring",
"updated_at": "2026-07-10T12:24:43Z"
},
{
"body": "Some calls are failing to connect or are experiencing audio issues. We are investigating the issue.",
"created_at": "2026-07-10T12:24:43Z",
"display_at": "2025-12-18T17:14:00Z",
"id": "01KX5ZSGKQTPBH691ZS5Z3Q06F",
"incident_id": "01KX5ZSGKQN0XKJN1R72AESH7H",
"status": "identified",
"updated_at": "2026-07-10T12:24:43Z"
}
],
"name": "Degraded performance on some calls.",
"page_id": "01KX5ZM3M45N00Z1PM4V3P76WJ",
"resolved_at": "2025-12-18T17:39:00Z",
"status": "resolved",
"updated_at": "2026-07-10T12:24:43Z"
},
{
"created_at": "2026-07-10T12:24:43Z",
"id": "01KX5ZSG0DEM0HR5X65V5V01JB",
"impact": "minor",
"incident_updates": [
{
"body": "The issue has been fully resolved. A small number of requests (approximately 100 calls) failed during the incident window as a result of an upstream service disruption affecting Cloudflare. https://www.cloudflarestatus.com/",
"created_at": "2026-07-10T12:24:43Z",
"display_at": "2025-12-17T11:11:00Z",
"id": "01KX5ZSG0DKTW5MB9429CKCY7G",
"incident_id": "01KX5ZSG0DEM0HR5X65V5V01JB",
"status": "resolved",
"updated_at": "2026-07-10T12:24:43Z"
},
{
"body": "We are investigating report issues of inbound calls ringing out and not connecting to their agents.",
"created_at": "2026-07-10T12:24:43Z",
"display_at": "2025-12-17T10:59:00Z",
"id": "01KX5ZSG0D505FWD2NJW99V2XA",
"incident_id": "01KX5ZSG0DEM0HR5X65V5V01JB",
"status": "identified",
"updated_at": "2026-07-10T12:24:43Z"
}
],
"name": "We are investigating degraded end-to-end calling performance for certain accounts.",
"page_id": "01KX5ZM3M45N00Z1PM4V3P76WJ",
"resolved_at": "2025-12-17T11:11:00Z",
"status": "resolved",
"updated_at": "2026-07-10T12:24:43Z"
},
{
"created_at": "2026-07-10T12:24:42Z",
"id": "01KX5ZSFEA3W37NHFX7PP33STD",
"impact": "minor",
"incident_updates": [
{
"body": "**Incident Summary (RCA)**\n\nOn December 12, 2025, starting at 13:58 CET, we experienced a brief service interruption caused by a DNS management policy change that led to unintended DNS record removals. The issue was quickly identified, rolled back to a safer update-only (upsert) behavior, and all DNS records were restored, with full service recovery by 14:20 CET. We have since strengthened safeguards around DNS policy changes, including clearer policy separation and additional monitoring, to prevent similar incidents in the future.",
"created_at": "2026-07-10T12:24:42Z",
"display_at": "2025-12-12T14:59:00Z",
"id": "01KX5ZSFEA3ETDK3CCNTMNZ3NS",
"incident_id": "01KX5ZSFEA3W37NHFX7PP33STD",
"status": "resolved",
"updated_at": "2026-07-10T12:24:42Z"
},
{
"body": "We have identified the issue and resolved it. We will share an RCA shortly, apologies for the inconvenience.",
"created_at": "2026-07-10T12:24:42Z",
"display_at": "2025-12-12T13:18:00Z",
"id": "01KX5ZSFEAKCQGDT02THF7JS2W",
"incident_id": "01KX5ZSFEA3W37NHFX7PP33STD",
"status": "resolved",
"updated_at": "2026-07-10T12:24:42Z"
},
{
"body": "We are currently seeing an evaluation in call failures. We are investigating the cause.",
"created_at": "2026-07-10T12:24:42Z",
"display_at": "2025-12-12T12:57:00Z",
"id": "01KX5ZSFEAZYBZ42ZHJ3SSYMGC",
"incident_id": "01KX5ZSFEA3W37NHFX7PP33STD",
"status": "identified",
"updated_at": "2026-07-10T12:24:42Z"
}
],
"name": "We are experiencing degraded performance on end to end calling",
"page_id": "01KX5ZM3M45N00Z1PM4V3P76WJ",
"resolved_at": "2025-12-12T13:18:00Z",
"status": "resolved",
"updated_at": "2026-07-10T12:24:42Z"
},
{
"created_at": "2026-07-10T12:24:42Z",
"id": "01KX5ZSETTAN3AM898QC39T0CQ",
"impact": "critical",
"incident_updates": [
{
"body": "Synthflow API recovered.",
"created_at": "2026-07-10T12:24:42Z",
"display_at": "2025-12-08T12:45:18Z",
"id": "01KX5ZSETTX2FWJDAYM6RM1231",
"incident_id": "01KX5ZSETTAN3AM898QC39T0CQ",
"status": "resolved",
"updated_at": "2026-07-10T12:24:42Z"
},
{
"body": "Synthflow API went down.",
"created_at": "2026-07-10T12:24:42Z",
"display_at": "2025-12-08T12:45:00Z",
"id": "01KX5ZSETTNMJHJV28VPGGP1GN",
"incident_id": "01KX5ZSETTAN3AM898QC39T0CQ",
"status": "monitoring",
"updated_at": "2026-07-10T12:24:42Z"
},
{
"body": "Synthflow API recovered.",
"created_at": "2026-07-10T12:24:42Z",
"display_at": "2025-12-08T12:19:43Z",
"id": "01KX5ZSETTMHQAMJ9QN74HF13S",
"incident_id": "01KX5ZSETTAN3AM898QC39T0CQ",
"status": "resolved",
"updated_at": "2026-07-10T12:24:42Z"
},
{
"body": "Synthflow API went down.",
"created_at": "2026-07-10T12:24:42Z",
"display_at": "2025-12-08T12:09:41Z",
"id": "01KX5ZSETT0RS2N15T7D9GH5XA",
"incident_id": "01KX5ZSETTAN3AM898QC39T0CQ",
"status": "identified",
"updated_at": "2026-07-10T12:24:42Z"
}
],
"name": "Synthflow API is down",
"page_id": "01KX5ZM3M45N00Z1PM4V3P76WJ",
"resolved_at": "2025-12-08T12:19:43Z",
"status": "resolved",
"updated_at": "2026-07-10T12:24:42Z"
},
{
"created_at": "2026-07-10T12:24:41Z",
"id": "01KX5ZSE7TN1S21YXDRCJ58PZ8",
"impact": "minor",
"incident_updates": [
{
"body": "The vendor has implemented the fix and we are continuing to monitor the results. https://www.cloudflarestatus.com/incidents/lfrm31y6sw9q",
"created_at": "2026-07-10T12:24:41Z",
"display_at": "2025-12-05T09:21:00Z",
"id": "01KX5ZSE7TYXADEQY2V8RSMQ1B",
"incident_id": "01KX5ZSE7TN1S21YXDRCJ58PZ8",
"status": "resolved",
"updated_at": "2026-07-10T12:24:41Z"
},
{
"body": "We are currently experiencing a service degradation affecting the Synthflow dashboard.\n\nOur vendor, Cloudflare, is reporting an active incident that may impact performance.\n\nYou can track their status here: https://www.cloudflarestatus.com/\n\nThis does not affect our API or end to end calling.\n\nWe’ll provide updates as the situation evolves.",
"created_at": "2026-07-10T12:24:41Z",
"display_at": "2025-12-05T08:56:00Z",
"id": "01KX5ZSE7T4NGJ1P4CM4FQ11SS",
"incident_id": "01KX5ZSE7TN1S21YXDRCJ58PZ8",
"status": "identified",
"updated_at": "2026-07-10T12:24:41Z"
}
],
"name": "Synthflow Dashboard Degradation",
"page_id": "01KX5ZM3M45N00Z1PM4V3P76WJ",
"resolved_at": "2025-12-05T09:21:00Z",
"status": "resolved",
"updated_at": "2026-07-10T12:24:41Z"
},
{
"created_at": "2026-07-10T12:24:40Z",
"id": "01KX5ZSDMENQY77R295N5S4KFP",
"impact": "minor",
"incident_updates": [
{
"body": "**Incident Summary (RCA)**\n\n**1. Summary**\n\nOn 2 December 2025 (16:23–20:10 CET), our dashboard and public API experienced severe degradation affecting all customers. API endpoints and agent loading were intermittently unavailable, and call logs were inaccessible throughout the incident.\n\n---\n\n**2. Root Cause**\n\nA PostgreSQL migration was unintentionally executed through the Cloud SQL managed connection pool due to a misconfigured secret. This created long-running, lock-heavy transactions. At the same time, the “make call” API held PostgreSQL connections open while waiting on slow MongoDB operations caused by a failed index rebuild. These factors exhausted the database connection pool and blocked new requests.\n\n---\n\n**3. Impact \u0026 Resolution**\n\nImpact:\n\n• API endpoints unavailable\n\n• Agent loading unavailable\n\n• Call logs inaccessible, including historical data\n\nResolution:\n\nWe terminated stuck DB sessions, repaired the corrupted MongoDB index, created a clean calls collection, redeployed the BFF with corrected configuration, fixed readiness checks, restored API stability, and later migrated historical call data.\n\n---\n\n**4. Preventive Actions**\n\n• Enforce direct DB connections for all migrations\n\n• Add transaction/connection timeouts and pool saturation alerts\n\n• Refactor internal endpoints to use short-lived transactions\n\n• Move MongoDB index management out of app startup\n\n• Standardize readiness probes to ensure only fully healthy pods receive traffic",
"created_at": "2026-07-10T12:24:40Z",
"display_at": "2025-12-03T15:59:00Z",
"id": "01KX5ZSDMEEW1Y0KBACZF690BK",
"incident_id": "01KX5ZSDMENQY77R295N5S4KFP",
"status": "resolved",
"updated_at": "2026-07-10T12:24:40Z"
},
{
"body": "All issues are now resolved. We apologize for the inconvenience",
"created_at": "2026-07-10T12:24:40Z",
"display_at": "2025-12-02T19:07:00Z",
"id": "01KX5ZSDMEVD5FG3SVV4QZE8T8",
"incident_id": "01KX5ZSDMENQY77R295N5S4KFP",
"status": "resolved",
"updated_at": "2026-07-10T12:24:40Z"
},
{
"body": "Our platform and API are now working correctly.\n\nCall history aren’t showing right now. We’re on it everything is stored safely and there is no loss, the call history visibility will return shortly.",
"created_at": "2026-07-10T12:24:40Z",
"display_at": "2025-12-02T16:32:00Z",
"id": "01KX5ZSDMEPNNS8TW8STB93GH1",
"incident_id": "01KX5ZSDMENQY77R295N5S4KFP",
"status": "monitoring",
"updated_at": "2026-07-10T12:24:40Z"
},
{
"body": "Our dashboard is currently experiencing degraded performance, similarly with our API.\n\nWe are investigating the issue and will have a fix shortly.",
"created_at": "2026-07-10T12:24:40Z",
"display_at": "2025-12-02T15:23:00Z",
"id": "01KX5ZSDME06FTDBE8YMQ0YJTK",
"incident_id": "01KX5ZSDMENQY77R295N5S4KFP",
"status": "identified",
"updated_at": "2026-07-10T12:24:40Z"
}
],
"name": "Degraded performance on our dashboard and API",
"page_id": "01KX5ZM3M45N00Z1PM4V3P76WJ",
"resolved_at": "2025-12-02T19:07:00Z",
"status": "resolved",
"updated_at": "2026-07-10T12:24:40Z"
}
],
"page": {
"id": "01KX5ZM3M45N00Z1PM4V3P76WJ",
"name": "Synthflow AI Status Page",
"updated_at": "2026-11-18T21:12:28Z",
"url": "https://status.synthflow.ai/"
}
}