After a database server failover, one of our six application servers still sometimes tried talking to the wrong database server, leading to one-sixth of ticket orders lead to an error message, so a user often needed to try submitting twice.
However, the disabling was incomplete, which wasn't noticed until 11:24. At 11:26, we fully disabled the faulty server. At 12:31, we deployed a change to our infrastructure that should prevent a similar error from happening again.
We deeply apologize for any inconveniences caused.
At 05:13, our team has first been alerted but first suspected a false alarm since during manual testing, it worked multiple times in a row. At 05:59, our team has been alerted again, identified the problem around 06:02 and disabled the affected server.
Our automatic monitoring has detected an outage. Our team has already been informed and is trying to resolve the issue as quickly as possible. We are sorry for any inconvenience this causes to you.
