Blog

Last weekend a subset of our customers were affected by a Saturday morning outage on one of the gateways we partner with then later on all customers were affected by an outage by Visa. This post is to fill you in on the details of what happened and what we are doing to try to prevent future occurrences

Payment Gateway Outage Saturday Morning

On Saturday morning customers using the Element payment gateway were unable to process credit cards. This was due to an upgrade Element attempted to perform on their systems that morning. Element quickly discovered the problem and rolled back the upgrade. This happened early enough in the morning that most west coast customers probably did not notice but the parts of the country that already had their day well under way did notice leaving us 49 urgent voicemails with many more non-urgent messages while we were investigating the problem.

Element has recently contacted us directly to let us know what exactly happened and how they will safely perform this upgrade. They also decided to perform the upgrade over the next three Monday mornings when retail traffic is much lighter. I am now confident they will be able to get this done without further interruption and we’ll be standing by watching out during the upgrade.

Visa Outage

Visa decided the weekend wasn’t exciting enough for MerchantOS and had a major outage of their own Sunday afternoon. Fortunately Sunday is one of the lowest retail volume days of the week but customers still noticed and called in. We don’t know too much about what happened here but they did let the press know.

We’re Doing What We Can

These particular events were out of our direct control. Our systems might have been able to handle the Element outage with more grace and we are looking into ways to do that. These kinds of issues are the worst for our customers as they prevent sales and we are doing everything we can to prevent future incident.