The Honest Story Behind Our V3 Rollout & the Bumpy Downtime

On Sunday June 21st, after a long weekend, we rolled out Version 3 (V3) of the Cardtonic App. 

We had been building toward this for months, and we were proud of it. This wasn’t a small update: it was new infrastructure, a rebuilt foundation meant to make the platform faster, more reliable, and ready for everything we wanted to build next.

We planned for a 3-hour maintenance window. We, however, ended up with close to 18 hours of downtime, and the ripple effects from that didn’t fully settle for days.

We owe you the full account of what happened, not just an apology. So, here it is.

Why We Built V3

The Version 2 (V2) of Cardtonic had served us well, but it had limits we could feel every time we tried to ship something new. 

The architecture wasn’t built for the ambitious scale we were growing into, and parts of it were starting to show their age: we started to catch slower processing and more fragile integrations, all pointing to a system that worked, but also increasingly had to maneuver around its own constraints. 

V3 was built to fix that. 

New infrastructure, a cleaner foundation, a platform built for where Cardtonic is headed rather than where it started. The team had been testing, tweaking, and preparing for this migration for a long time, and going into that Sunday, we felt prepared.

But we were not as ready as we believed.

What Went Wrong

The plan was straightforward on paper: migrate from the legacy v2 platform to v3, move wallets, transactions, and user data over, and do it within a tight maintenance window.

What we didn’t fully account for was some unforeseen circumstances around how that migration would behave under certain conditions.

Some of the issues with wallet synchronisation and data consistency between the old and new systems only revealed themselves once actual traffic hit the platform at scale.

The outage extended well past our planned window. 

Wallet funding, core transactions, and KYC verification were affected for a number of users. And once core services were finally restored, we weren’t fully in the clear. 

New issues began surfacing across other parts of the platform as people resumed using it: payment processing, gift card services, Virtual Dollar Card operations, notifications, and some of our admin tools all had problems crop up in the days that followed.

For some of you, this wasn’t an abstract technical hiccup. This became money you couldn’t access when you needed it, or a dollar card you couldn’t fund at the worst possible time. 

We know that. We never lost sight of it, and we’re truly sorry.

How We Responded

Once it became clear this was bigger than a routine deployment hiccup, our Product, Engineering, Growth, Customer Support, and Management teams moved into a coordinated response that didn’t stop for days.

Every issue that came in, whether flagged internally through monitoring or reported by a customer, was logged, assessed for impact, and assigned for resolution. We prioritised based on what affected the most people and the most severely. Wallets and core transactions came first, because that’s where the most immediate pain was. From there, we worked outward: payment flows, KYC, operations, and the rest.

Fixes were tested and verified before they went out, then tracked until we could confirm they’d actually solved the problem, not just patched the symptom. 

It involved a lot of long days, longer nights, and a team that kept showing up to the same hard problem until it stopped being a problem. 

For every hour something stayed broken, we were aware that people were depending on us, and we were letting them down, so we all stayed locked in until the job was done.

What’s Fixed, and What’s Left

As of today, the extended outage itself has been fully resolved; that part of the story is closed, thankfully. 

The majority of the production issues that surfaced afterward have also been identified, fixed, and verified. 

A small number of items are still being actively worked on. If something from this period is still unresolved for you specifically, please reach out to us at support@cardtonic.com and reference your account; we want to close out every last one of these properly, not just declare victory and move on.

We’re also keeping production monitoring in place to ensure stability and catch anything else early if it surfaces.

What We’re Changing and How We’re Compensating You

An incident like this doesn’t just get fixed; it gets learned from, or it should.

We’re changing how we test and deploy major upgrades going forward, specifically to close the gap between what we expect and what actually happens under massive production traffic. That gap is exactly what caught us off guard this time, and it’s the thing we’re most focused on closing, so it doesn’t catch us off guard again.

We also made the decision, as a way of taking real responsibility rather than just saying sorry, to remove every margin entirely for the month of July. 

Every transaction within this window will run at the highest available market rate, with nothing added on our end. It doesn’t undo last week; we know that. But it’s the most concrete thing we could offer, and we wanted to put something behind the apology beyond words.

Where We Are Now

The platform is stable, and the team is still here, still watching, still fixing whatever’s left.

V3 was supposed to be the foundation for everything we build next at Cardtonic, and despite how rough the rollout was, it still is. We just got to that foundation the hard way, through a week that tested the team and tested your patience with us.

We’re not going to dress this up as anything other than what it was: a serious mistake in execution, on something that mattered, that cost you time and trust. What we can tell you is that the people behind Cardtonic took it just as seriously as you did, worked through it without stepping back, and are carrying what we learned into how we build from here.

Thank you for staying with us through it. 

We will never take that for granted.