SEV3 ProPay System Intermittent Processing Connection Issue

Incident Report for ProPay

Postmortem

A single application node was underperforming, resulting in timeouts. The affected node was removed and refreshed and reintroduced.

Our teams continue to work on strategies for reducing impact from poorly performing nodes, including recent changes to our authorization connection logic, failover logic and memory management adjustments.

Posted May 18, 2021 - 09:18 MDT

Resolved

The troubled server was pulled and a fix applied by our teams has resolved the issue. Our system issue occurred between 10:00 AM to 10:25 AM MST.

Posted May 03, 2021 - 11:12 MDT

Update

We are continuing to monitor for any further issues.

Posted May 03, 2021 - 11:01 MDT

Monitoring

Although we continue to monitor the event it looks like the fix that was implemented has resolved the issue.

Posted May 03, 2021 - 10:50 MDT

Update

We are continuing to work on a fix for this issue.

Posted May 03, 2021 - 10:19 MDT

Update

We are continuing to work on a fix for this issue.

Posted May 03, 2021 - 10:12 MDT

Update

We are continuing to work on a fix for this issue.

Posted May 03, 2021 - 10:11 MDT

Identified

The issue has been identified and a fix is being implemented.

Posted May 03, 2021 - 10:09 MDT

Investigating

We found that a single server was causing an issue and our teams are wring on a quick resolution to prevent further processing loss.

Posted May 03, 2021 - 09:57 MDT

This incident affected: Production Processing Solutions (Credit Card and ACH Processing, Websites, Mobile, and Card Readers).