SEV2 Processing Failures

Incident Report for ProPay

Postmortem

This event was caused by a nonperforming application node which was disabled, refreshed and re-introduced.

Further analysis indicated the unresponsive services was due to high resource constraints related to a configuration setting for session encryption management.

Action items identified from this event include improvements to

configuration management
environmental unity
resource consumption threshold and alerting tuning
processes for disabling non-performing nodes automatically
traffic throttling solutions

Posted Aug 27, 2020 - 11:35 MDT

Resolved

This incident has been resolved.

Posted Aug 11, 2020 - 09:52 MDT

Monitoring

A fix has been implemented and we are monitoring the results.

Posted Aug 10, 2020 - 18:37 MDT

Update

We identified a system component experiencing slow responsiveness, and pulled it out of the pool. This caused some failures in processing attempts between 18:06 PM (MDT) and 18:26 PM. Since taking steps to address the problem, the failures have stopped. We will continue to monitor the situation to ensure stability for a couple of hours before resolving this incident.

Posted Aug 10, 2020 - 18:37 MDT

Identified

The issue has been identified and a fix is being implemented.

Posted Aug 10, 2020 - 18:32 MDT

Investigating

We are aware of an issue that may impact your ability to process transactions or access our websites. The extent of the issue is not yet known, but our teams are assessing the severity of the issue we will provide an update shortly.

Posted Aug 10, 2020 - 18:19 MDT

This incident affected: Production Processing Solutions (Credit Card and ACH Processing, Websites, Mobile, and Card Readers).