Postmortem -
Read details
Aug 1, 23:29 UTC
Resolved -
All tests have passed. All services are operating nominally and fail back is successful. We are closing this incident.
To ensure service continuity now, after the failback, please ensure that your STA environment is configured according to the KB article: https://supportportal.thalesgroup.com/csm?id=kb_article_protected&sys_id=b7af9a474f1cbfcc102400818110c77e and download the Classic zone Quick Help Guide to set up your primary and disaster recovery secondary sites accordingly.
If you have any questions, please open a case with your support provider or Thales Customer Support as appropriate:
• https://supportportal.thalesgroup.com/csm
• United States: 800-545-6608
• International: +1-410-931-7520
Jul 28, 19:26 UTC
Update -
Authentication and console tests have passed. We are now testing secondary services.
Jul 28, 19:14 UTC
Update -
Fail back to primary has been completed. We are now undergoing testing.
Jul 28, 19:08 UTC
Update -
We are commencing the fail back process. We will update once complete.
Jul 28, 19:00 UTC
Update -
We will be commencing the fail-back to the primary site in 5 minutes.
Jul 28, 18:56 UTC
Update -
We will failback the STA service to the primary site on Thursday, 28 July 2022 at 19.00 UTC.
Our cloud service platform provider's primary site has reached the critical mass that we expected and continues to be functional and stable. Therefore, we’re proceeding with plans to failback the STA service.
When the failback starts, there will be a brief disruption of the STA service for up to a maximum of 15 minutes. We will update this Status Dashboard as the failback progresses.
For customers that have manually configured their secondary site and to ensure service continuity after the failback, please make sure that your STA environment is configured correctly for the primary and disaster recovery (secondary) sites according to the Classic Service Zone’s Quick Help Guide that is available to download from the KB article:
https://supportportal.thalesgroup.com/csm?id=kb_article_protected&sys_id=b7af9a474f1cbfcc102400818110c77e
If you have any questions, please open a case with your support provider or Thales Customer Support as appropriate:
• https://supportportal.thalesgroup.com/csm
• United States: 800-545-6608
• International: +1-410-931-7520
Jul 26, 18:56 UTC
Update -
We continue to monitor our cloud service platform provider's primary site as it is in the process of reaching the critical mass that we expect. It continues to be functional and stable. Once it reaches our expectations, we will provide you with 2 business days notice before moving you back to the primary site.
Jul 25, 15:41 UTC
Update -
We have postponed the failback scheduled for 25 July 2022. We will update this incident 2 business days prior with the new failback date.
In the best interest of our customers, we have decided to postpone the failback because the platform provider’s service has not yet reached the critical mass that we expect.
As a reminder, to ensure service continuity now and after the failback, please ensure that your STA environment is configured according to the KB article: https://supportportal.thalesgroup.com/csm?id=kb_article_protected&sys_id=b7af9a474f1cbfcc102400818110c77e where you can download the Quick Help Guide that explains how to set up your primary and disaster recovery secondary sites accordingly.
Jul 22, 20:26 UTC
Update -
Classic Zone - Incident Update
Background-
Our Cloud Service Platform Provider hosting STA’s primary site experienced a major datacenter infrastructure outage on 19 July amid the record heatwave that caused a region-wide hardware failure. The Thales CloudOps Team identified the issue immediately and successfully failed over the STA service to the disaster recovery site, restoring access to the STA service within several minutes. The STA service has remained fully functional ever since.
Update –
Our Cloud Service Platform Provider reports that their services are now functional after experiencing a major infrastructure outage on 19 July. Out of an abundance of caution, we will continue to monitor the stability over the next several days. If satisfactory, we will failback the STA service to the primary site on 25 July 2022 at 17:00 UTC.
During the failback activity, brief disruption up to an expected maximum of 15 minutes will occur at the beginning of the window.
To ensure service continuity now and after the failback, please ensure that your STA environment is configured according to the KB article: https://supportportal.thalesgroup.com/csm?id=kb_article_protected&sys_id=b7af9a474f1cbfcc102400818110c77e and download the Classic zone Quick Help Guide to set up your primary and disaster recovery secondary sites accordingly.
If you have any questions, please open a case with your support provider or Thales Customer Support as appropriate:
• https://supportportal.thalesgroup.com/csm
• United States: 800-545-6608
• International: +1-410-931-7520
Jul 21, 19:53 UTC
Monitoring -
The failover has been completed and we are monitoring the status.
Jul 19, 16:41 UTC
Update -
Failover has been performed and services are starting up in DR zone.
Jul 19, 16:35 UTC
Identified -
We are performing a failover in order to restore the service.
Jul 19, 16:33 UTC
Investigating -
The SafeNet Trusted Access monitoring system has detected a problem with the Authentication and Console access.
The Thales Incident Response Team is actively investigating this problem and will update this incident with additional information as it becomes available.
Jul 19, 16:28 UTC
