We've now been down for 45 Minutes. Second time we've been down today. mc.infusionsoft.com is down as well, and I can't reach anyone.

Anyone have any idea on how to reach someone at this hour to find out if they are even aware of the problem?

HELP!!!!

Yes, it's great. I'm sitting here in Australia in the middle of a work day, Infusion goes down.

So I ring Internetional support number and it's disconnected.

Does anyone know how to get these guys out of bed?

Steve

With all due respect, Steve, one of our main providers failed to notify us when you posted here. It was a case of a Friday afternoon + unexpected downtime.

We apologize for the inconvenience. We don't intend to play a 'disappearing act' during these times and truly strive for excellence with our software.

Additionally, we may at times limit network activity to ranges of applications during a system upgrade -- should one be necessary. We aim to make it during minimal usage periods.

~Joseph

With all due respect Joseph, this isn't the first "unexpected" outage, and overall you guys have never done a good job on this front.

In each and every outage that I have had mc.infusionsoft.com has also been down. What's the point of having a "mission critical" site if it also goes down whenever you have a failure?

No one there takes the time to acknowledge outages. On this particular outage I tried mc.infusionsoft, and then as time ticked by I emailed and called every email address and number I could find. Not a single response... not even now after the fact. Just silence.

Outages are a reality of any online business, I get that. What would make them a heck of a lot less stressful is having something or someone at Infusionsoft acknowledge them when they happen.

Finally -- and I've made this request dozens of time before -- how about notifications before upgrades. My guess is that my 50 minute outage was actually planned (even if last minute), as I've seen this pattern before. A brief outage during the day due to an unforeseen problem, then a long outage that night as you address the problem. I get the need to do that - I don't get why you can't send us an email so we can plan accordingly, rather than being pulled away from a family event while trying to figure out what is going on.

The internet does not operate 9-5 M-F, so having 24-7 mission critical support is a requirement, not an option.

For everyone else, a recommendation. Unlike Infusionsoft per Joseph's email, I don't wait for my vendors to notify me of problems. I'd highly recommend setting up monitoring on your infusionsoft instance. I've found that it gives me about a 5 minute headstart on dealing with complaints from our customers and gives me a chance to capture new customers we'd otherwise lose. We use alertra.com for this.

Sean,

I appreciate your immense passion for making sure that not a packet is lost during transactions. Seriously, I commend you, I know it can be incredibly frustrating when a server is down.

The issue was a provider that gives service to our network facility was down and the router did not switch networks as expected. We usually are given such notice and these changes occur without affecting any customers. (Kudos to our Systems team who are able to perform this art without the bloodshed of downtime or lost packets in cyberspace.)

So, where does that leave us?
Clearly, we ought to make sure the MC.Infusionsoft.com is properly segmented from the Infusionsoft main network. In fact, we have that page being hosted at an offsite facility, perhaps the DNS isn't set properly. I'll seek to it, that even if the Infusionsoft network is down, that server is up and running regardless so clients can faithfully let our on-call staff know what's up.

Mind you, this is an emergency stop-gap notification. We also employ numerous SNMP and other "uptime" services across the Web to monitor connectivity to our servers.

Like I said, this was unexpected but was resolved in a relatively short period of time. We're always looking to improve and maintain a solid network uptime, especially as demand increases with customers.

We typically release software every 3 weeks and every 3 months. These releases are divided into "maintenance" and "feature" releases so our clients obtain the most current, reliable and fully-functional versions of our software. While I'm totally with you that more notice is better, we are notorious for pushing release dates because of a "last-minute" change to the code. Sadly, this results in us not being able to hold a specific date to releases.

Now, things have improved with respect to documenting posting such updates to the Fusebox and integrating these notifications into our application. While it may not be advance, maybe we can work into a way to dispatch a specific "pro-user" email that informs users such as you (and I) that a release is being pushed. However, I don't think most users want to receive such notifications three times a month, plus if we push a release by a day or two.

To recap, I'll take a look into ensuring that MC.Infusionsoft.com is operational regardless of network availability. I'll inquire with our Dev team as to being notified when a release is being pushed. (It would be even cooler if we can implement automation into the mix.)

I hope this gives perspective into the matter.

--Joseph

Joseph, just wanted to clarify one thing. When I mentioned notification before upgrades, I was talking about system upgrades, not application. As you mention I may not know exactly when an app upgrade is going to happen, but there is usually at least some communication around it. Most of my problematic outages have been around hardware upgrades for which there was no notice. In a couple of cases in the past it was after a hardware failure earlier in the day which caused a short outage, then some sort of larger fix was put in place that night, but with significant downtime and no warning.

This seemed likely to me Friday as we went down for about 15 minutes midday, and then almost an hour that evening. From your description it sounds like perhaps they were unrelated.

Also one final note. Infusionsoft's overall uptime has been pretty good. It's just really frustrating to have an outage with ZERO ability to communicate with you guys. I'd hope you wouldn't accept that from your vendors, and hopefully someone over there will someday realize it really isn't acceptable to do to your customers.