This morning, shortly before 8am US Eastern, one of our databases became overloaded and, to use a technical term, crapped out.
Our monitoring apps told us of the situation, we quickly jumped on the apparent problem, and we thought we were done … but there was more to the issue than at first appeared.
One of our downstream web servers that handled tracking became horribly overloaded as a consequence of its being unable to reach that specific database. That in turn caused a failsafe in FeedBlitz to kick in, which had the nasty and unintended consequence of putting up a scary malware message. That’s a design oversight we’ll fix later; right now the message has changed if you’re still getting it.
Right now we’re rerouting traffic to web servers that are OK. Depending on your settings that might take a while to show up. Job #1 is to get the system serving you properly. Then we’ll figure out what caused this cascade of issues and how to prevent them in the future.
Updates will be posted here and on the @FeedBlitz Twitter account.
Update 09:18 – things are looking much better now and click throughs are working properly.
ليست هناك تعليقات:
إرسال تعليق