EU - Admin Console Access and SAML/OIDC Authentication Incident
Incident
Report for SafeNet Trusted Access
Postmortem
Resolved
The Problem with the Console Access, API, and SAML Authentication has been resolved and, where necessary, corrective action was taken for the customers affected.
The Thales Incident Response Team is closely monitoring the service stability. This incident is now closed.
The Thales Incident Response Team is closely monitoring the service stability. This incident is now closed.
Update
All testing is complete. All services running as expected on the primary site.
Update
Authentication and Admin Console services have been verified as functioning nominally on the Primary site. Other services are being tested and verified.
Update
We have initiated the failback to the working primary site and will provide updates once completed.
Update
We will be conducting our failback to the primary site in 3 minutes at 03:30 UTC
Update
There is a backend cluster upgrade happening in the EU Service Zone. It should not impact services however you might experience random slowness or timeout. Giving a retry should work.
Update
Please be aware that the current deployed services are performing as expected. The Thales Team are working to fix the cause of the incident that occurred earlier and will test before deploying changes. Once the support team are ready to deploy the proven change, they will provide a least one hour notice before doing so and this will be outside prime operating hours.
Update
Please be aware that the current deployed services are performing as expected. The Thales Team are working to fix the cause of the incident that occurred earlier and will test before deploying changes. Once the support team are ready to deploy the proven change, they will provide a least one hour notice before doing so and this will be outside prime operating hours.
Update
Please be aware that services are performing as expected from our secondary site. The Thales Team continues to investigate the primary site. Once the support team are ready to deploy the primary services again, they will provide a least one hour notice before doing so and this will be outside prime operating hours.
Update
We continue to investigate the issues. All services are working nominally from our secondary site and they continue to do so. We will continue to provide updates as they become available.
Update
We continue to investigate the issues. All services are working nominally from our secondary site and they continue to do so. We will continue to provide updates as they become available.
Update
Please be aware that services are performing as expected from our secondary site. The Thales Team continues to investigate the primary site. Once the support team are ready to deploy the primary services again, they will provide a least one hour notice before doing so and this will be outside prime operating hours.
Update
Please be aware that services are performing as expected from our secondary site. The Thales Team continues to investigate the primary site. Once the support team are ready to deploy the primary services again, they will provide a least one hour notice before doing so and this will be outside prime operating hours.
Update
We continue to investigate the issues. All services are working nominally from our secondary site and they continue to do so. We will continue to provide updates as they become available.
Update
For customers who are manually pointing to the Primary site Radius IP address, Push services may fail. To mitigate this, you may either use manual OTP generation or manually point Radius to our Failover IP address - see below.
Primary RADIUS Server IP: 34.77.242.204:1812 Primary Agent: cloud.eu.safenetid.com:443
Failover RADIUS Server IP: 35.246.223.136:1812 Failover Agent: cloud.eu.safenetid.com:443
Updates will be posted every 3 hours from the initiation of this message, unless new findings emerge in the interim.
For customers who need to configure secondary disaster recovery site please refer to STA EU configuration here: https://supportportal.thalesgroup.com/csm?id=kb_article_protected&sys_id=b7af9a474f1cbfcc102400818110c77e
Primary RADIUS Server IP: 34.77.242.204:1812 Primary Agent: cloud.eu.safenetid.com:443
Failover RADIUS Server IP: 35.246.223.136:1812 Failover Agent: cloud.eu.safenetid.com:443
Updates will be posted every 3 hours from the initiation of this message, unless new findings emerge in the interim.
For customers who need to configure secondary disaster recovery site please refer to STA EU configuration here: https://supportportal.thalesgroup.com/csm?id=kb_article_protected&sys_id=b7af9a474f1cbfcc102400818110c77e
Update
We are currently actively routing traffic through our secondary Disaster Recovery site for STA EU. Operations will continue to operate at the secondary site until the primary site is completely restored. Our commitment to keeping you informed remains steadfast, and we will provide regular updates throughout this process. Updates will be posted every 3 hours from the initiation of this message, unless new findings emerge in the interim.
For customers who need to configure secondary disaster recovery site please refer to STA EU configuration here: https://supportportal.thalesgroup.com/csm?id=kb_article_protected&sys_id=b7af9a474f1cbfcc102400818110c77e
For customers who need to configure secondary disaster recovery site please refer to STA EU configuration here: https://supportportal.thalesgroup.com/csm?id=kb_article_protected&sys_id=b7af9a474f1cbfcc102400818110c77e
Update
We are investigating the issues on our Primary site. All services are working nominally from our Secondary Site and continue to do so. We will continue to provide updates as they become available you should expect the next update within 3 hours.
Update
We are currently investigating the issues on the Primary site. However, All services are working nominally from the Secondary Site. We will continue to provide updates as they become available you should expect the next update within 3 hours.
Update
We are actively directing traffic through our STA EU secondary Disaster Recovery site. Operations will continue from the secondary site until the primary site is fully restored. We are committed to keeping you informed with regular updates as we work towards a resolution. For customers who need to configure a secondary disaster recovery site please refer to STA EU configurations here: https://supportportal.thalesgroup.com/csm?id=kb_article_protected&sys_id=b7af9a474f1cbfcc102400818110c77e
Update
We are actively directing traffic through our STA EU secondary Disaster Recovery site. Operations will continue from the secondary site until the primary site is fully restored. We are committed to keeping you informed with regular updates as we work towards a resolution. For customers who need to configure a secondary disaster recovery site please refer to STA EU configurations here: https://supportportal.thalesgroup.com/csm?id=kb_article_protected&sys_id=b7af9a474f1cbfcc102400818110c77e
Monitoring
We are currently serving the traffic from our secondary Disaster Recover Site, We will continue to run until we fix the issue at the Primary site. We will keep sharing updates as they become available.
Update
All services are verified as stable and available.
Identified
Authentication and Console services have been confirmed as stabilized and available from the DR site.
Update
Testing is now underway to ensure all services are running on the secondary site.
Update
We have failed overall services to the Secondary site. We are currently undergoing testing.
Update
We are still attempting some restorative efforts and will provide updates as they become available.
Update
We are continuing to investigate this issue.
Update
We are continuing to investigate this issue.
Investigating
The SafeNet Trusted Access monitoring system has detected a problem with Console Access, API and SAML Authentication for some customers.
Issues in Admin console may have an impact on your ability to access and manage tokens, users, provisioning, policies, application configuration, and other management operations.
SAML/OIDC Authentication issues can affect users ability to authenticate or access business applications configured with STA.
The Thales Incident Response Team is actively investigating this problem and will update this incident with additional information as it becomes available.
Issues in Admin console may have an impact on your ability to access and manage tokens, users, provisioning, policies, application configuration, and other management operations.
SAML/OIDC Authentication issues can affect users ability to authenticate or access business applications configured with STA.
The Thales Incident Response Team is actively investigating this problem and will update this incident with additional information as it becomes available.
This incident affected: EU Service Zone (Authentication Services, Admin Consoles, End User Services, Push Delivery, Management).
X
STA Service Zone
STA Service Zone
Once logged into their STA console accounts, both STA operators and users with access to the console can easily identify the STA Zone hosting their environment by checking the URL displayed.
- The following URLs refer to the following zone:
- STA Classic Zone: https://sta.safenetid.com/
- STA EU Zone: https://sta.eu.safenetid.com/
- STA US Zone: https://sta.us.safenetid.com/
STA Classic Zone
URL: https://sta.safenetid.com/home/dashboard.
STA Classic Zone Screenshot
STA Classic Zone Screenshot
STA EU Zone
URL: https://sta.eu.safenetid.com/home/dashboard.
STA EU Zone Screenshot
STA EU Zone Screenshot
STA US Zone
URL: https://sta.us.safenetid.com/home/dashboard.
STA US Zone Screenshot
STA US Zone Screenshot
X
Incident Categories
Operational
All systems are operational
Degraded Performance
- Authentication Services ? : One or more methods impacted but works intermittently i.e. RADIUS service
- Admin Console ? : One or both consoles are intermittently accessible and users cannot perform functionalities within console
- End User Services ? : One or more methods impacted and not accessible for users i.e. Self-Service
- Push Delivery ? : One or more methods impacted and not accessible for users i.e. Push OTP
- Management ? : One or more methods impacted and not accessible for users i.e. SOAP Management API
Partial Outage
- Authentication Services ? : One or more methods fails completely i.e. RADIUS
- Admin Console ? : Users are not able to access either one of the consoles
Major Outage
- Authentication Services ? : All the methods are down, all customers are impacted
- Admin Console ? : Users are not able to access both consoles, cannot log in completely
Maintenance
There is a planned upcoming maintenance.
Have questions or need help?
Technical Support Website
Technical Support Website
© 2021 Gemalto. All rights reserved.