SEV1 (High) ProPay System Incident 307 Errors
Incident Report for ProPay
Postmortem

This incident was the result of a major outage at the processor on which ProPay depends for clearing its transactions.

Mitigation efforts and failover measures were attempted by ProPay teams, but all sites of the processor were experiencing issues during the impact window - between 18:20-21:10 MT.

Partners and merchants were receiving timeout or other failure messages. Those transactions did not succeed and ProPay’s system does not retry or re-submit failed transactions in this scenario.

We have limited information on the exact nature of the root cause, but have been informed it was related to a system update at the processor which manifest an initial issue. It then escalated in severity as a result of cascading complications of mitigation actions taken.

Ultimately, once the situation was resolved and processor systems were available again, ProPay transactions resumed normal operations.

Please direct any follow up questions regarding this incident to your relationship manager or the ProPay Client Support Team.

Posted Aug 31, 2021 - 11:00 MDT

Resolved
We have verified that the issue with our upstream processor has now been resolved and processes have been restored.
Posted Aug 21, 2021 - 21:56 MDT
Update
We are continuing to monitor for any further issues.
Posted Aug 21, 2021 - 21:27 MDT
Monitoring
A fix has been implemented and we are monitoring the results.
Posted Aug 21, 2021 - 21:15 MDT
Update
We have identified that there is an issue with our upstream processor. Some failovers options have been attempted but they have been unsuccessful thus far. We will continue to provide updates as we are supplied with them.
Posted Aug 21, 2021 - 20:56 MDT
Update
We are continuing to work on a fix for this issue.
Posted Aug 21, 2021 - 20:17 MDT
Identified
The issue has been identified and a fix is being implemented.
Posted Aug 21, 2021 - 19:49 MDT
Update
We are continuing to investigate this issue.
Posted Aug 21, 2021 - 19:15 MDT
Investigating
We are aware of an issue that is preventing some transactions from being processed.

Our teams are working to resolve the issue as quickly as possible.
We will provide regular updates until the situation is addressed.

clientsupport@propay.com
801-341-5312
Posted Aug 21, 2021 - 18:47 MDT
This incident affected: Production Processing Solutions (Credit Card and ACH Processing).