Document Processing Delay
Incident Report for SPS Commerce
Resolved
We have resolved the issue causing this incident. Performance has returned to acceptable levels. Thank you for your patience while we worked through this issue. We sincerely apologize for the inconvenience.
Posted Nov 26, 2020 - 04:01 CST
Monitoring
Document Processing services have been scaled to necessary capacity to process existing data backlog. All metrics confirms normal processing timing and we monitoring results.
Posted Nov 26, 2020 - 03:43 CST
Update
We still making progress in restoring document processing and work through the backlog of transactions.
We greatly appreciate your patience as we work to get the transactions current.
Posted Nov 26, 2020 - 02:21 CST
Update
We were able to scale our services to increase the rate of document processing and work through the backlog of transactions. Error rates mostly normalized.
We appreciate your patience as we work to get to nominal data processing.
Posted Nov 26, 2020 - 01:24 CST
Update
We continue making progress in restoring document processing and work through the backlog of transactions. Error rates normalize across most of the services.
We greatly appreciate your patience as we work to get the transactions current.
Posted Nov 26, 2020 - 00:16 CST
Update
We continue to see error rates normalize as we work through the backlog of transactions.

We greatly appreciate your patience as we work to get the transactions current.
Posted Nov 25, 2020 - 23:18 CST
Update
We are seeing a decrease in error rates in our system as we continue to work through the backlog of transactions.

Your patience is greatly appreciated as we work to get the transactions current.
Posted Nov 25, 2020 - 22:09 CST
Update
Progress through the backlog on our system remains steady.

We appreciate your patience as we continue to work on getting the transactions current.
Posted Nov 25, 2020 - 21:10 CST
Update
We continue to make good progress through the backlog of transactions on our system.

Your patience as we work through to get current is greatly appreciated.
Posted Nov 25, 2020 - 20:10 CST
Update
We continue to process through the backlog of transactions.

We greatly appreciate your patience as we work to get systems current on processing transactions.
Posted Nov 25, 2020 - 18:58 CST
Update
We are making good progress on the backlog of transactions in our system.

We greatly appreciate your patience as we work to get systems current on processing transactions.
Posted Nov 25, 2020 - 18:06 CST
Update
We are continuing to work on a fix for this issue.
Posted Nov 25, 2020 - 16:57 CST
Update
A number of our services are now available.

We have begun processing through the backlog of transactions and are working diligently to process the transactions we received today as soon as possible.

We appreciate your patience as we work to get systems current on processing transactions.
Posted Nov 25, 2020 - 16:56 CST
Update
We continue to make progress toward resolution.

Thank you for your patience as we restore all services.
Posted Nov 25, 2020 - 15:58 CST
Update
Partial dataflow has been enabled and we continue to work on additional remediation steps.
Posted Nov 25, 2020 - 15:41 CST
Update
We continue to work on resolving this. Thank you again for your patience.
Posted Nov 25, 2020 - 14:37 CST
Update
We continue working toward a resolution.

We greatly appreciate your patience.
Posted Nov 25, 2020 - 13:20 CST
Update
We are still working to resolve this incident.

We greatly appreciate your patience.
Posted Nov 25, 2020 - 12:42 CST
Update
We are still working toward a resolution.

We truly appreciate your patience as we work to resolve this incident.
Posted Nov 25, 2020 - 11:55 CST
Update
We are still actively engaged and working toward a resolution.

We appreciate your patience and understanding.
Posted Nov 25, 2020 - 11:20 CST
Update
We recognize the significant impact this is having and apologize for the delay. We are still working to correct the issue identified earlier. We appreciate your patience while we work to resolve this issue.
Posted Nov 25, 2020 - 10:56 CST
Update
This incident is cascading across multiple services including Fulfillment, Assortment, Commerce VAN, Oracle Transportation Management, and NetSuite.

We appreciate your patience as we work to resolve this incident.
Posted Nov 25, 2020 - 10:20 CST
Update
Fulfillment performance is impacted by this incident as well.
Posted Nov 25, 2020 - 10:04 CST
Update
We are still working to correct the issue identified earlier. We have changed the status to reflect that this is now a major outage.

We appreciate your patience while we work to resolve this issue.
Posted Nov 25, 2020 - 09:53 CST
Update
We are still working to correct the issue identified earlier. We appreciate your patience while we work to resolve this issue.
Posted Nov 25, 2020 - 09:21 CST
Update
We are continuing to work on a fix for this issue.
Posted Nov 25, 2020 - 08:53 CST
Identified
We have identified the cause of this incident and our Technology team is working diligently to correct it. Thanks for your continued patience.
Posted Nov 25, 2020 - 08:24 CST
Investigating
We are investigating an open incident with our document processing infrastructure which is potentially causing delays or failures for transactions processing through the system. No data has been lost and the system should be caught up shortly. We will reconcile all data after we have recovered from the incident to ensure all transactions are processed as expected.
Posted Nov 25, 2020 - 08:23 CST
This incident affected: Integrations (Oracle Transportation Management, NetSuite) and Fulfillment, Document Processing, Assortment, Commerce VAN.