Incidents reported on status page for CloudZero https://status.cloudzero.com/ https://d1lppblt9t2x15.cloudfront.net/logos/be934dec7a4da6bf6dc0925681bc5bbb.png https://status.cloudzero.com/ en https://status.cloudzero.com/incident/581680 Tue, 27 May 2025 16:27:00 -0000 https://status.cloudzero.com/incident/581680#aa30092a4fa5255da46631ddceacdafa44f971021cefcd73927302fa5f372d7b We have implemented and deployed a workaround, and initial tests indicate that it resolves the service issue. The team continues to monitor the situation. https://status.cloudzero.com/incident/581680 Tue, 27 May 2025 14:40:00 -0000 https://status.cloudzero.com/incident/581680#cce85b9605e9a3cbfbe0f4f0e208ec3663294802f6140ac7a06d3c09a622857e The underlying service provider has identified the issue and pushed a fix. They believe that service should start being restored shortly. We are monitoring our end closely to ensure that service begins to be restored. https://status.cloudzero.com/incident/581680 Tue, 27 May 2025 14:00:00 -0000 https://status.cloudzero.com/incident/581680#6e24cc5c9097e1c0a1c7ede61792581173ce46a41490e27f425324d253498120 We've identified the issue as stemming from one of our underlying service providers. They have updated their status page and are actively working on a resolution. We're beginning to see signs of improvement, with some users regaining access to the CloudZero platform. Our team is closely monitoring the situation and looking at other workarounds. https://status.cloudzero.com/incident/581680 Tue, 27 May 2025 13:35:00 -0000 https://status.cloudzero.com/incident/581680#c6f4d477c36f55a9f63bfdc3f8826982c0fed3a9c1b74bec619f5c2cbc06afab We are currently investigating an issue where some users are unable to log in to the CloudZero platform and are encountering a "403" error. Our engineering team is treating this as a top priority and is actively working to restore full access as quickly as possible. We’ll provide an update as soon as we have more information. https://status.cloudzero.com/incident/572780 Tue, 20 May 2025 12:48:00 -0000 https://status.cloudzero.com/incident/572780#b5feb3cb1a3a7ee6bc618fdc7cf7dbb66650b4755192d67cbb665ec10aed777c Our technology partner has implemented a fix for the recent performance issues. We’re now observing normal performance across the CloudZero platform. https://status.cloudzero.com/ Fri, 16 May 2025 07:52:44 +0000 https://status.cloudzero.com/#a58e647e25cf814e316f904f30828d206784af14cff3fc4154bb26d36a70f069 cloudzero.com recovered https://status.cloudzero.com/ Fri, 16 May 2025 07:52:21 +0000 https://status.cloudzero.com/#a58e647e25cf814e316f904f30828d206784af14cff3fc4154bb26d36a70f069 cloudzero.com went down https://status.cloudzero.com/ Tue, 13 May 2025 05:15:39 +0000 https://status.cloudzero.com/#d3970627f17f967ec7922e3a2091731ce8bbb009e2de264994b81729465be849 cloudzero.com recovered https://status.cloudzero.com/ Tue, 13 May 2025 05:15:09 +0000 https://status.cloudzero.com/#d3970627f17f967ec7922e3a2091731ce8bbb009e2de264994b81729465be849 cloudzero.com went down https://status.cloudzero.com/incident/553361 Fri, 02 May 2025 12:26:00 -0000 https://status.cloudzero.com/incident/553361#c76ebb11ba8b189dc45b179986f56ebf5d8d3f6d548e518b696503cd11b29099 We’re now seeing GCP billing data flow normally across all CloudZero customers. If you continue to experience any issues, please don’t hesitate to reach out to our support team. Thanks for your patience while we monitored this situation. https://status.cloudzero.com/incident/553361 Wed, 30 Apr 2025 19:17:00 -0000 https://status.cloudzero.com/incident/553361#f0a57125cb4d7741ef81304fa2bbb394fbf0d29657032d65b82ba2cd56541047 We’re seeing fresh data starting to load from Google Cloud Platform (GCP), but some delays are still present for a subset of GCP customers. We’re actively monitoring the situation alongside our partners at Google and will share another update once things are fully back to normal. https://status.cloudzero.com/incident/553361 Mon, 28 Apr 2025 21:00:00 -0000 https://status.cloudzero.com/incident/553361#742577e9ff5fe914f48a1f14b8a351a4ef34cf5e1968b0a2625c3aecbb4c0bd1 Google Cloud Platform (GCP) has informed CloudZero and other customers that, beginning on April 25, 2025, they are experiencing delays in delivering cost data for some organizations. We encourage you to contact GCP support directly if you have any questions or concerns. CloudZero will continue to actively monitor the situation and provide updates as necessary. https://status.cloudzero.com/incident/551818 Fri, 25 Apr 2025 16:30:00 -0000 https://status.cloudzero.com/incident/551818#37b0ef6a61357ed6457b93ff9bc6c3201751c15f537703b05afc68184cb2c859 Fix is deployed. https://status.cloudzero.com/incident/540927 Mon, 21 Apr 2025 13:57:00 -0000 https://status.cloudzero.com/incident/540927#90d364f9a509430a1fcc4583f635a2d9fb0a0ad8435c2e296e2fe058d8deccb1 We have seen dashboard performance return to normal through this morning's scale up. The issue has been resolved. https://status.cloudzero.com/incident/540927 Sun, 20 Apr 2025 22:59:00 -0000 https://status.cloudzero.com/incident/540927#6d381a8d72f477ac35e6f8aea9dd293f561f51cf16ec97b8b0707ebc2b9fbc91 We are seeing improved dashboard rendering times across all dashboards following the application of a second patch. We continue to monitor the situation to ensure this issue is resolved. https://status.cloudzero.com/incident/547498 Sat, 19 Apr 2025 00:00:00 +0000 https://status.cloudzero.com/incident/547498#660fbf92a8323c78e4752d95ef8f7580673925989b39cb3547df6f7b1459451c Maintenance completed https://status.cloudzero.com/incident/547498 Fri, 18 Apr 2025 23:00:00 -0000 https://status.cloudzero.com/incident/547498#47958df7d6395284e89e9b019084e9bb7ce25f23f65f44d6f7138117c2a9057c We will be conducting maintenance on our Analytics system related to recent dashboard delays. During the maintenance period the system may be temporarily unavailable. https://status.cloudzero.com/incident/540927 Fri, 18 Apr 2025 22:10:00 -0000 https://status.cloudzero.com/incident/540927#e8503ef14b1602ba3bdbbe808b00226c9b63069557b6060de98f5ac008e23bd5 A patch for the root cause of recent dashboard delays has been created and will be applied during a maintenance window starting at 7PM US ET. We will continue to closely monitor system performance through the weekend and into next week. https://status.cloudzero.com/incident/540927 Tue, 15 Apr 2025 12:50:00 -0000 https://status.cloudzero.com/incident/540927#3ed91d277dbd3a74aab0c12a6cbb171dd31f45d177ae9d75ef79967587e54aa8 A root cause for dashboard delays has been identified. We continue to work with our analytics partner to correct the issue. Impacted dashboards continue to experience delays. https://status.cloudzero.com/ Tue, 15 Apr 2025 05:15:27 +0000 https://status.cloudzero.com/#e780637055bb5e4ab4ffca1fcf886c7c51c02711756dc7b0e317a1c41360bc44 cloudzero.com recovered https://status.cloudzero.com/ Tue, 15 Apr 2025 05:14:56 +0000 https://status.cloudzero.com/#e780637055bb5e4ab4ffca1fcf886c7c51c02711756dc7b0e317a1c41360bc44 cloudzero.com went down https://status.cloudzero.com/incident/540927 Mon, 14 Apr 2025 12:08:00 -0000 https://status.cloudzero.com/incident/540927#35b07a42fc57c354776e3dcce6d1f495e47342f5927c217b367312fef9c9dd1f We applied several updates to counteract elevated dashboard load time. Our metrics show improved performance. Impacted dashboards may continue seeing a delay when loading. We continue to work with our analytics partner on a root cause. https://status.cloudzero.com/incident/540927 Fri, 11 Apr 2025 11:52:00 -0000 https://status.cloudzero.com/incident/540927#04406ce3dc5e329b4c89b5bcdc777daad668c51e300494723632336e7f6138c3 We continue to observe elevated load times for some dashboards and are working with our analytics partner to restore performance. Impacted dashboards may experience a delay of a few seconds when loading. https://status.cloudzero.com/incident/540927 Thu, 10 Apr 2025 13:28:00 -0000 https://status.cloudzero.com/incident/540927#ddcb6bbdc29f6dca0ba119dd7f9694025f837c4230010789164b2f6763ec1c41 We are investigating an issue with analytics dashboard load times. Impacted dashboards may experience a delay of a few seconds when loading. https://status.cloudzero.com/incident/541943 Tue, 08 Apr 2025 21:11:00 -0000 https://status.cloudzero.com/incident/541943#3a5c77487f96d546c3ae30edc3ab653ccd5e2320f0781575ba171003dff68ac1 Fix has been deployed and all impacted customers have been addressed. https://status.cloudzero.com/incident/541943 Tue, 08 Apr 2025 14:24:00 -0000 https://status.cloudzero.com/incident/541943#1d4a095e46eafd92997ca964a66279456fa1b9a60415957ab6d464c6db17aded We have identified a recent change to the platform that caused some inaccurate reporting of cost data related to allocation dimensions for some customers. This change has been rolled back, and ingest is rerunning for impacted customers. https://status.cloudzero.com/incident/540927 Mon, 07 Apr 2025 20:26:00 -0000 https://status.cloudzero.com/incident/540927#ebcd75cc15f65cf64b2e36aae6775e32b02bc48e1c7c98fa4a6763b63926f475 We've identified a cause for this issue. Dashboard query time averages are expected to be within normal ranges going forward. https://status.cloudzero.com/incident/540927 Mon, 07 Apr 2025 14:29:00 -0000 https://status.cloudzero.com/incident/540927#6950c44d118f78c5b732dc1989a463899556276772693c7e56c10f4c5fed0ada Dashboard query averages have returned to normal levels. We continue to monitor the situation and investigate the root cause. https://status.cloudzero.com/incident/540927 Mon, 07 Apr 2025 10:43:00 -0000 https://status.cloudzero.com/incident/540927#1ae83cdddd6970549120d02604246ca3642f25a1675c20c038f6fe0631d93473 We are continuing to investigate the root cause of the dashboard performance issue. We are currently observing an average query overhead between 0.5 and 3 seconds greater than normal. https://status.cloudzero.com/incident/540927 Mon, 07 Apr 2025 01:00:00 -0000 https://status.cloudzero.com/incident/540927#6a6ee5ded1697f6aec7a61f96118af4f70bbe8bf13d03b88381239c4f24d76b8 We are investigating an issue with analytics dashboard load times. Impacted dashboards may experience a delay of a few seconds when loading. https://status.cloudzero.com/ Sat, 29 Mar 2025 03:39:56 +0000 https://status.cloudzero.com/#0f0b265f03801624c88fec0999395f435ec120089a724cb7beeec189a6e0539c cloudzero.com recovered https://status.cloudzero.com/ Sat, 29 Mar 2025 03:38:05 +0000 https://status.cloudzero.com/#0f0b265f03801624c88fec0999395f435ec120089a724cb7beeec189a6e0539c cloudzero.com went down https://status.cloudzero.com/incident/535124 Wed, 26 Mar 2025 23:11:00 -0000 https://status.cloudzero.com/incident/535124#14c63939e7d3a45408f4b8fdeabb6857c733a5bc54c27524d0c0976e700eb0f4 All affected connections have been resumed. They will temporarily show in the "Error" state until they have a successful ingestion, at which time they will return to "Healthy". We expect all connections to return to healthy within 8 hours. https://status.cloudzero.com/incident/535124 Wed, 26 Mar 2025 22:15:00 -0000 https://status.cloudzero.com/incident/535124#36e0456f4bc07a0a0bfdc7a48f380547f249640f5eec780b4fb81f5ea8d704f6 DataDog, Databricks, MongoDB and New Relic billing connections are delayed in retrieving cost data for the prior 24 hours. The root cause has been resolved as of ~5pm EST today. Affected connection were paused automatically by the system, we are currently resuming these connections. We expect all billing data to be caught up by tomorrow morning. https://status.cloudzero.com/incident/534550 Wed, 26 Mar 2025 00:30:00 -0000 https://status.cloudzero.com/incident/534550#101bb344be2ae5f67efd32cd823f66bc17d9f367312e6b41bda5905b58e51c5e Dashboards seem to be loading - we are watching and working on triage with one of our service providers. https://status.cloudzero.com/incident/531004 Wed, 19 Mar 2025 21:13:00 -0000 https://status.cloudzero.com/incident/531004#00b13afca9fcab02a739f9e36c597b1d17159855ce95dd102662b13a18e85f53 We saw periodic failures in Analytics Dashboards to load, depending on how they were referenced. The team has pushed a fix and we are monitoring now. https://status.cloudzero.com/incident/528263 Fri, 14 Mar 2025 14:46:00 -0000 https://status.cloudzero.com/incident/528263#edc19c4653335473e31aae2bad4286db9ca0ecefeae3d4b02b9a0167e885a82d We had a login issue; the issue has been identified and resolved. We are monitoring now that the fix has been pushed. https://status.cloudzero.com/incident/515030 Tue, 18 Feb 2025 21:02:00 -0000 https://status.cloudzero.com/incident/515030#ab83aa9f832998e115cce294b49d050c9d3df382d4d61960bee06a0f5aca3d88 We have identified and rectified the root cause of this issue. All DataDog and New Relic connections that were affected are currently ingesting data, and will be available in the CloudZero platform alongside their respective billing data as soon as our normal nightly billing ingest cycle completes. https://status.cloudzero.com/incident/515030 Tue, 18 Feb 2025 18:15:00 -0000 https://status.cloudzero.com/incident/515030#03b26dbee39d0b3575de822cb07bfe25f071175aaef255ba86e3326e3a186ba2 New Relic and DataDog billing connections are delayed in retrieving cost data since Feb 15th. We understand the root cause and are currently deploying a fix. https://status.cloudzero.com/incident/507679 Wed, 05 Feb 2025 10:54:00 -0000 https://status.cloudzero.com/incident/507679#9e96dac96af63b15580a0ba6a243a5189e9246fd6124f2ef4eacf489106985e5 We have identified and rectified the root cause of this issue. All Azure connections that were affected are currently ingesting data, and will be available in the CloudZero platform alongside their respective billing data as soon as our normal nightly billing ingest cycle completes. https://status.cloudzero.com/incident/507679 Wed, 05 Feb 2025 07:02:00 -0000 https://status.cloudzero.com/incident/507679#4de41f213122525606e093e3aec10c62b59c3dde37739d7d1b7470d82112ec99 Sone Azure billing connections are delayed in retrieving cost data, possibly since Jan 31st. We are actively investigating root cause and will provide an update as soon as possible. https://status.cloudzero.com/incident/492719 Wed, 08 Jan 2025 19:30:00 -0000 https://status.cloudzero.com/incident/492719#83e062879c875f99e50736c8b9cc80fa079076ab3a7f4bc57534bfcad5fadf78 We have identified and rectified the root cause of this issue. All Azure connections are up to date or currently ingesting data, and will be available in the CloudZero platform alongside their respective billing data as soon as our normal nightly billing ingest cycle completes. https://status.cloudzero.com/incident/492719 Wed, 08 Jan 2025 18:22:00 -0000 https://status.cloudzero.com/incident/492719#125f4646f516c6f639167c5c1bae30be7161a6f9bb10797bc9ccde5f8e403176 Azure billing connections are delayed in retrieving cost data for the prior 48 hours. We are actively investigating root cause and will provide an update as soon as possible. https://status.cloudzero.com/incident/467472 Mon, 25 Nov 2024 16:45:00 -0000 https://status.cloudzero.com/incident/467472#5e339dbab397064672242bc59792a40796e184a372d5f3833158b1045e4eb26b We have identified and mitigated the root cause, and have triggered processes to ingest the latest billing data for all impacted billing connections. We expect all billing data to be caught up shortly. https://status.cloudzero.com/incident/467472 Mon, 25 Nov 2024 15:30:00 -0000 https://status.cloudzero.com/incident/467472#20745d4c77479349dfcd55264574ea0e303f450fc0f3798a850b5667051817cb A limited number of billing connections (for cloud service providers Databricks, Datadog, MongoDB, and New Relic) are delayed in retrieving cost data. We are actively investigating root cause and will provide an update as soon as possible. https://status.cloudzero.com/incident/460355 Wed, 13 Nov 2024 05:00:00 -0000 https://status.cloudzero.com/incident/460355#61a2a2cb89d6c597720a95fc790cf2729ffc2b28c9603348f78b88920dee1003 We have identified and rectified the root cause of this issue. All CloudWatch container metrics are up to date for all impacted customers, and will be available in the CloudZero platform alongside their respective billing data as soon as our normal nightly billing ingest cycle completes. https://status.cloudzero.com/incident/460355 Tue, 12 Nov 2024 17:00:00 -0000 https://status.cloudzero.com/incident/460355#dd5751a27d231e0ac58a33bd4ef74927540c4691598d851a4dc390b1fde021c6 We are working on an active issue related to the CloudZero platform's ability to ingest Kubernetes container metrics originating from our CloudWatch agent, for some customers. We are actively investigating and will provide an update at change in status. https://status.cloudzero.com/incident/458263 Sat, 09 Nov 2024 18:00:00 -0000 https://status.cloudzero.com/incident/458263#ebfc07d0aa1a8296cd03d85a1bbee5d63f4be690109f4f421686447b3f3086b0 Datadog has provided a workaround for this behavior for the time being. Please consult your FinOps Account Manager for steps to remediate as needed. We will continue to assist Datadog with troubleshooting in the meantime. https://status.cloudzero.com/incident/458263 Fri, 08 Nov 2024 22:30:00 -0000 https://status.cloudzero.com/incident/458263#0b5e7caa51b89b7c3cdd4fcdb724dc6b5886b9904f2578131e1036e6236ad27c We continue to work with our contacts at Datadog to identify and resolve the root cause of this issue. https://status.cloudzero.com/incident/458263 Fri, 08 Nov 2024 15:00:00 -0000 https://status.cloudzero.com/incident/458263#00982d665923f5553d1b20c9cfcf813bd0f1ad8024089962ff19a482e6c46892 For a small subset of our customers, with Datadog billing connections outside of the Datadog US1 (datadoghq.com) region, we are currently unable to receive billing data. We are working closely with Datadog on this issue. https://status.cloudzero.com/incident/457169 Wed, 06 Nov 2024 21:20:00 -0000 https://status.cloudzero.com/incident/457169#2439f5be34ae42f1c4e0d047df6a0f5c2d044b25f3d8e38701f58dfe713850c2 Our underlying Analytics provider has found a bug that can result in data being slightly delayed. It only impacts a specific backend analytics component, currently only included in the CloudZero standard dashboard "AWS Reservation Catalog" and a few custom dashboards delivered via CloudZero Customer Success (they will reach out if you are impacted). We have opened a bug with the provider and are working on a workaround in parallel. The workaround was deployed last night and we continue to monitor it to ensure it's working. https://status.cloudzero.com/incident/448226 Mon, 21 Oct 2024 13:00:00 -0000 https://status.cloudzero.com/incident/448226#bd39a8ad922f61f03d21fde3d098a194249290fc765864e50a2ebee87974dc43 On October 20th, 2024, it was discovered that GCP Billing Ingest was failing. The team's investigation led us to identify the root cause as a bad code deploy, which was introduced on 10/17/24 15:00 EST. This caused the GCP Billing ingestion feature to fail at startup across all organizations. The team deployed a fix and re-ingested the missing data to restore full functionality. https://status.cloudzero.com/incident/448226 Sun, 20 Oct 2024 16:00:00 -0000 https://status.cloudzero.com/incident/448226#6145feff3d94c6aca0870356977da6047daa2eeebd57df9df78218e0c07100a9 From Thursday 10/17/24 3pm EST until Sunday 10/20/24 12pm EST GCP Billing data was not getting ingested into the platform. https://status.cloudzero.com/incident/434215 Tue, 24 Sep 2024 20:54:00 -0000 https://status.cloudzero.com/incident/434215#5faac8ac8dced30b2b274a1973da4d1b5eb2dd65acf39d59ff29096559bf3c8e The issue has been addressed. Explorer is now loading as expected for all customers. https://status.cloudzero.com/incident/434215 Tue, 24 Sep 2024 20:36:00 -0000 https://status.cloudzero.com/incident/434215#b56afb8de7bb78e222fddeed38819aa91ce7b43b206b9bcecba16adb9b46317f We have had reports of Explorer not loading for some customers. We have identified the issue and are working on a fix. https://status.cloudzero.com/incident/433677 Tue, 24 Sep 2024 09:00:00 -0000 https://status.cloudzero.com/incident/433677#4793179411f90ae53584dbfac28633fff3fc12e9c3957d21890f96958c01dc59 This issue has been resolved for all impacted customers. https://status.cloudzero.com/incident/433677 Mon, 23 Sep 2024 18:30:00 -0000 https://status.cloudzero.com/incident/433677#6b1480a4a65e8c22fa9ce60d8385a2a492145acad530b6e4584172c59d6956c1 We are aware of and working to resolve an issue which is resulting in delayed availability of up-to-date cost data in the CloudZero platform. This is impacting only a portion of our customers, and will be resolved shortly. https://status.cloudzero.com/incident/432722 Sat, 21 Sep 2024 13:06:00 -0000 https://status.cloudzero.com/incident/432722#fda3e64eab90573498d8e3cf57f9c00eca34b90551a763059b7f32560c10b386 Issue has been identified and resolved. https://status.cloudzero.com/incident/432722 Sat, 21 Sep 2024 12:38:00 -0000 https://status.cloudzero.com/incident/432722#2e622681e29ff989bace5657ba5a3afcef5e7c416cc0b161e5d6345775bd821a We have received reports of users unable to login to app.cloudzero.com. We are currently investigating root cause. https://status.cloudzero.com/incident/430618 Wed, 18 Sep 2024 01:00:00 -0000 https://status.cloudzero.com/incident/430618#75f357b848320be6fee8dde05adbd4368fa763860058679a71e6b9b9a87d8bc8 This issue has now been fully resolved. We appreciate your patience as we worked to a full resolution. https://status.cloudzero.com/incident/430618 Tue, 17 Sep 2024 22:30:00 -0000 https://status.cloudzero.com/incident/430618#8cbfa035db98d045cf10e520e69864a1ae162ae159c29f73ea3f72f110c90fd0 We have made Snowflake billing data visible for 75% of impacted customers, and are continuing to ingest this data for the remaining impacted customers. https://status.cloudzero.com/incident/430618 Tue, 17 Sep 2024 19:00:00 -0000 https://status.cloudzero.com/incident/430618#edffea0aaee9fe20bce3f3ed3c10c1fbedb8b606fd738ac7dad03a39a713e2cd We have fixed the root cause of this issue impacting and are actively running ingest processes so that our customers will see their Snowflake billing data in the platform as soon as possible. https://status.cloudzero.com/incident/430618 Tue, 17 Sep 2024 16:30:00 -0000 https://status.cloudzero.com/incident/430618#030dc1e2076f33cddbd40fb13c488ae3d5ee24795810185b9b7d44c9ce23461d We have identified the root cause of this issue and a fix is actively in progress. https://status.cloudzero.com/incident/430618 Tue, 17 Sep 2024 15:55:00 -0000 https://status.cloudzero.com/incident/430618#3bc4b4f999e189dbf626ed976e804d44b0586b78691b2d01b1ea9817916df310 We are aware of and investigating an issue with failed Snowflake billing connections in the CloudZero platform. We are working to fix this problem as quickly as we can, and will share another update shortly. https://status.cloudzero.com/incident/425343 Sun, 15 Sep 2024 02:35:00 -0000 https://status.cloudzero.com/incident/425343#02935f7079e60024196bfa0d63c6426c3cda31c201feabfccf89143a6c97fa76 The backfill of incomplete data has been completed. We appreciate your patience as we worked to resolve this issue. https://status.cloudzero.com/incident/425343 Sat, 14 Sep 2024 11:00:00 -0000 https://status.cloudzero.com/incident/425343#ffd9c24d97c7fa8686d706d713ee908ae2c44bb7bc0f45cce6e48b1e0f32113b As of Saturday morning, we are continuing to actively back-fill Kubernetes container metrics for select date ranges in August. We have all of the necessary container metric data; we are in the process of ingesting it properly so that it can be tied up with the associated cost data. The backfill continues to progress as expected, and we remain confident that this will be completed and in a watching state later today (9/14). https://status.cloudzero.com/incident/425343 Fri, 13 Sep 2024 14:30:00 -0000 https://status.cloudzero.com/incident/425343#801ecda0a8c2ceb0a1c379b165a124981fff98fe98f8e6ca0e56d6a4d8fd11f5 We are continuing to actively back-fill Kubernetes container metrics for select date ranges in August, as we've observed small gaps across 8/16 - 8/22 and 8/28 - 8/31. We have all of the necessary container metric data; we just need to ingest it properly so that it can be tied up with the associated cost data. The backfill continues to progress well, and we remain confident that this will be completed and in a watching state on 9/14. https://status.cloudzero.com/incident/425343 Thu, 12 Sep 2024 15:00:00 -0000 https://status.cloudzero.com/incident/425343#0ef9e088f6b4f8476b9d1cb9f50c5f4f230947b68e14563fa17a449d35205ee7 We continue to actively back-fill Kubernetes container metric select date ranges in August, as we've observed small gaps across 8/16 - 8/22 and 8/28 - 8/31. We have all of the necessary container metric data; we just need to ingest it properly so that it can be tied up with the associated cost data. The backfill is going well, and is expected to be completed and in a watching state on 9/14. https://status.cloudzero.com/incident/425343 Wed, 11 Sep 2024 21:00:00 -0000 https://status.cloudzero.com/incident/425343#ac8c16ad1f1757501c086ab770889328e487bd05c9880048276eb180890012a8 Proper Kubernetes container metrics continue to flow into the CloudZero platform for all customers. We continue to actively back-fill select date ranges in August, as we've observed small gaps across 8/16 - 8/22 and 8/28 - 8/31. We have all of the necessary container metric data; we just need to ingest it properly so that it can be tied up with the associated cost data. In addition, we're looking at an issue impacting a subset of customers where our cost algorithm appears to be incorrectly assigning Idle costs for Kubernetes clusters. https://status.cloudzero.com/incident/425343 Wed, 11 Sep 2024 11:45:00 -0000 https://status.cloudzero.com/incident/425343#104c4426c901ccafda701d78fbb82ba8345e59aedb7c13359bf8f9c871de0d16 Proper Kubernetes container metrics are now flowing in for all CloudZero customers, and these are being used to correctly allocate compute costs. We have observed that we need to selectively back-fill a few date ranges, as we're seeing small gaps across 8/16 - 8/22 and 8/28 - 8/31. We have all of the necessary container metric data; we just need to ingest it properly so that it can be tied up with the associated cost data. https://status.cloudzero.com/incident/425343 Tue, 10 Sep 2024 23:51:00 -0000 https://status.cloudzero.com/incident/425343#1dc4478b122254b806559b3861c72dfadc638bc41d3e63d30e2d41e96e8f7a9d We are continuing to work on fully resolving a known issue when ingesting container usage data for Kubernetes customers, which results in a delay in seeing new Kubernetes clusters or updated container usage statistics. We have identified the root cause, deployed a fix, and are finalizing changes to ensure all cost and container usage data is up to date and available in the CloudZero platform. We expect this issue to be fully resolved by Wed, Sep 11 at 10 AM ET. https://status.cloudzero.com/incident/425343 Mon, 09 Sep 2024 19:54:00 -0000 https://status.cloudzero.com/incident/425343#4400aa72efb7dca95036f1e184b679c767820e5c667f49aaa5c4f1e28f8ca621 Currently, there is a known issue ingesting usage data for Kubernetes customers. This issue has resulted in a delay in seeing new Kubernetes clusters or updated usage statistics. We have identified the root cause, deployed a fix, and are working on backfilling the affected data. https://status.cloudzero.com/incident/409934 Wed, 07 Aug 2024 01:20:00 -0000 https://status.cloudzero.com/incident/409934#704d7e91cb9792932fa7c6f8b010f61930cfc02bc662421b7682c94799a67c3c The rollback is complete, and performance looks better so far. We are monitoring the siutation. https://status.cloudzero.com/incident/409934 Wed, 07 Aug 2024 00:58:00 -0000 https://status.cloudzero.com/incident/409934#074ac81ea8f2583b1b6bd048c2bdfa8cda3d13f0240a6de1e2ef2bfbc48ec58b The third-party provider has identified the issue and is rolling back a change on their end. https://status.cloudzero.com/incident/409934 Tue, 06 Aug 2024 22:05:00 -0000 https://status.cloudzero.com/incident/409934#e28e158976bfbb8782164311de6014a49b17da08a02d92f841b71568fd4987e1 We have identified a issue with a third-party provider, and engaged with them to to continue debugging the issue https://status.cloudzero.com/incident/409934 Tue, 06 Aug 2024 20:56:00 -0000 https://status.cloudzero.com/incident/409934#6cac4c98fe48f7220508f77f844b025df86041a3db37bc7efdb1137ca2d887c0 Some users are experiencing degraded performance on some Analytics dashboards. We are investigating the issue. https://status.cloudzero.com/incident/390801 Fri, 28 Jun 2024 23:26:00 -0000 https://status.cloudzero.com/incident/390801#ed27cfffde3e2ec2f036cb0be14e4fbc79aa0bbc2fff2abab63615c2d497d2f1 The infrastructure provider has found the offending issue and rolled it back. We have been monitoring, and Analytics performance seems good again. https://status.cloudzero.com/incident/390801 Fri, 28 Jun 2024 20:09:00 -0000 https://status.cloudzero.com/incident/390801#379a060e72457223b9b144af0c942038e2d06d257e7260ad24031750f316c273 Our underlying infrastructure provider has found and acknowledged a bug. Their engineering teams are engaged in figuring out a fix or a workaround. Currently, CloudZero Analytics should be functional for most users, but users may still see periodic slowness with queries. That slowness will occasionally spike to 1-2 additional minutes on queries. https://status.cloudzero.com/incident/390801 Fri, 28 Jun 2024 19:31:00 -0000 https://status.cloudzero.com/incident/390801#7ee1ed56bcdb7a360d7beccc81e12b18766c97a8844a03e887c22dd5d13f2958 The underlying infrastructure provider has acknowledged the issue, escalated the issue, and is actively debugging it. https://status.cloudzero.com/incident/390801 Fri, 28 Jun 2024 17:57:00 -0000 https://status.cloudzero.com/incident/390801#5352adbb202a438a7ba4cb0a4e993766a585e7997a3929026fdf3cff386705da We have identified the root cause with one of our infrastructure providers and have escalated the issue with their support. https://status.cloudzero.com/incident/390801 Fri, 28 Jun 2024 15:31:00 -0000 https://status.cloudzero.com/incident/390801#3cbe15b7d8263f3799d386f6009208d2cbc949ccf97e4ed0094d84bdf752243a Some users are experiencing degraded performance on some Analytics dashboards. We are investigating the issue. https://status.cloudzero.com/incident/388714 Mon, 24 Jun 2024 15:03:00 -0000 https://status.cloudzero.com/incident/388714#bdf9aa12463938731f123f73f38a64dab733cbd4ce5721a4e62895b6dd88b389 Some users have reported issues loading and creating Analytics dashboards. We are tracking errors in both the authentication and backend data processing, and the team is working on triage and fixing them. Update: a fix has been pushed, and we are currently monitoring it. Update: the fix has been validated and service is restored. https://status.cloudzero.com/incident/347466 Thu, 28 Mar 2024 16:07:00 -0000 https://status.cloudzero.com/incident/347466#9bfda8f64aa46ddb4da673da9b06f2c1631b03ba3e12e33e609c5a395631e1dc We believe this issue is resolved. We will continue to monitor. https://status.cloudzero.com/incident/347466 Thu, 28 Mar 2024 15:38:00 -0000 https://status.cloudzero.com/incident/347466#194a3fe38e91911d2c4d36170f1c9d94c3705220cbaa9e10830ec72ef2d3f042 We have received reports of some users not being able to log in. Our engineering team is investigating the issue and we will post updates here as we have them. https://status.cloudzero.com/incident/345869 Mon, 25 Mar 2024 16:57:00 -0000 https://status.cloudzero.com/incident/345869#128774a66bca3d73b8dc57dec7bb40262c8ef9405ba0d1fa34d0985ff10fdce9 AWS has reported that the issue has been resolved. Data will be backfilled as it appears and is ingested. AWS reported an outage with their Cost Explorer and Cost and Usage Reporting that caused delays in data drops. They identified the issue and are working towards resolution. You may see a delay in AWS data throughout today. https://status.cloudzero.com/incident/343182 Tue, 19 Mar 2024 22:12:00 -0000 https://status.cloudzero.com/incident/343182#b9faa3bf43ebd32abae223b5e57510585902e3d77e877a3134dfd515c494fe6f This issue has been resolved. All dashboards are visible again. https://status.cloudzero.com/incident/343182 Tue, 19 Mar 2024 22:12:00 -0000 https://status.cloudzero.com/incident/343182#6ccf602ba047ee7e22137c30eee576166580f5ff99eccde95bf606306016d8f1 This has been resolved https://status.cloudzero.com/incident/343182 Tue, 19 Mar 2024 20:45:00 -0000 https://status.cloudzero.com/incident/343182#166b6a0d573c6963498d1d4847d2bd37a7386a9da4803922f992fb8243effb94 We have identified the root cause of the issue and are working on the fix. https://status.cloudzero.com/incident/343182 Tue, 19 Mar 2024 20:13:00 -0000 https://status.cloudzero.com/incident/343182#250e27f8c215b4a11a0c30df369fa27b5a4f0f739e36dc3aa8d75e7e59e5b6f4 Private dashboards are currently not visible in the Analytics page. We are currently investigating the issue. https://status.cloudzero.com/incident/287613 Wed, 15 Nov 2023 15:34:00 -0000 https://status.cloudzero.com/incident/287613#f4f8d4df4253e8c52fd548a2ad0e13fa5e1fe77361a17b6c21fec7bb651196ca Some customers have reported a degraded experience in the Explorer & Analytics. The problem has been identified and we are working on a fix. More information as it becomes available. Issue patched and service has been restored. https://status.cloudzero.com/incident/281993 Fri, 03 Nov 2023 14:55:00 -0000 https://status.cloudzero.com/incident/281993#6fa3b8fd9b0d682ce7d67eb39a42bd663ce375fd22feb8c31f8038ab8a0372ab Starting at 8:00 PM PDT on November 1, AWS reported missing Cost allocation tags data for October 2023 for all cost management products, including Cost Explorer and Cost and Usage Reports (CUR) only for the accounts where cost allocation tags are applied to a resource in the US-EAST-1 Region. AWS is working on the issue, and we are monitoring the situation. Until the situation is resolved, AWS cost data in the CloudZero platform may be inaccurate. https://phd.aws.amazon.com/phd/home?region=us-east-1#/event-log?eventID=arn:aws:health:us-east-1::event/BILLING/AWS_BILLING_OPERATIONAL_ISSUE/AWS_BILLING_OPERATIONAL_ISSUE_8176E_DD1109C04D4&eventTab=details&layout=vertical The provider has resolved the issue and are in the process of backfilling data, as they make the corrected data available, you will see it after your next ingest.