It was a terrible, horrible, no good, very bad week at ServInt – the worst we’ve had since 10 years ago, when a fiber cut in just the wrong place brought us offline completely for seven hours. That day was one of the most professionally terrifying of my life, but we learned from it and we grew. In the wake of that event we added redundancies far beyond the “industry standard,” we fixed a ton of processes and we quickly regained the faith of our customers. To this day, that date in 2004 was the very last time ServInt’s entire network has gone down.
Every time we experience problems I am determined to make sure we learn from them. This week is a big learning week, because it’s been fraught with some of the biggest problems we’ve seen in nearly a decade. Let me take a few moments to tell you a bit about what challenges this week brought — to show you what went wrong, and what we did right as we resolved them.
This week started with an announcement that the largest kernel level exploit in the history of our VPS and virtual dedicated offerings had been discovered. This exploit could have allowed hackers to access not only our customers’ VPSs but also the machines that they were hosted upon. A fix would require the reboot of literally thousands of servers, while minimizing the impact on our clients’ businesses — always our top priority. Within 48 hours we performed emergency maintenance on nearly every single customer in our datacenter. This meant forcing every single customer to accept at least a little downtime in the pursuit of vital security protections. Some customers did not like this, but if I had to do it all over again I would do it the same way. I am proud of the way ServInt rose to the challenge and protected our customer base from this dangerous exploit in such a swift manner.
I was really hoping that the week would get easier from there — but it didn’t. Last night, one of ServInt’s datacenters experienced one of the strangest, most difficult to explain, and most difficult to solve networking problems we have ever seen.
We build our networks to withstand most anything. We have stayed up through hurricanes, ice storms, and more equipment failures than I can count. We’ve made it through power disruption for extended periods, and other horrendous events that would have taken down providers that aren’t as thorough, many times over. But this one got us good for a while.
On Saturday evening, our network was running smoothly, as it generally has for more than a decade. Suddenly our monitoring system started showing red/green/red/green/etc. The phrase “this is not a drill” had to be used as senior engineers were plucked from their lives and rushed into the datacenter. Our COO was on a plane, I was at dinner, but the engineering fix-it team that really needed to be there was there, immediately. What made this situation unique, and what made it impossible to fix in the normal few minutes, was the fact that the critical equipment that was in the process of failing seemed incapable of making up its mind whether it was healthy or not. Making matters more challenging: high levels of equipment redundancy (normally a very good thing) made it nearly impossible to determine where the problem lay. Our top engineers, without access to reliable diagnostic data, literally had to pull the network apart and put it back together to find the exact piece of hardware that went haywire (in this case a router) that caused everything else to behave erratically. In the meantime, there was simply no information to share with increasingly frustrated customers, and our Tweets and Facebook posts began to sound unnecessarily vague.
In a typical router-failure situation, as soon as the router shows “red/down” on our monitoring system, we post “we had a failed router interrupt traffic impact the network. This is being fixed and we’re routing around it — we’re sorry for the inconvenience.” Those are facts and details, things people can get confidence from. However, with no reliable detail to pass on, our team was left to pass on rather vague updates for quite some time. It was frustrating and made us seem much worse about communication than we actually are.
In the end, last night’s events pointed out some of ServInt’s greatest historical strengths — and some newly discovered weaknesses. We’re still the best in the business at running a reliable, robust network and data center — and, when necessary, finding and fixing complex technical problems. When it comes to customer support and communication through a crisis, however, we need to do better. Having no support/communication failover systems, and forcing ServInt and its customers to rely on Twitter and Facebook to communicate, was totally unacceptable. We will build greater redundancy into our ticketing and communication systems to make sure that never happens again.
Having said that, we can’t promise that technical glitches will never happen again. They are a fact of internet life. What matters most is that we must always — always — learn from these thankfully rare events, and become a better service provider as a result. I promise you we will do so in this case as well. You’ll see the results of this growth as the weeks and months unfold. I am confident you’ll like what you see. Thank you, as always, for your continued faith and trust in us.
There’s a revolution underfoot in the hosting industry, but it’s not what you think. The Cloud has arrived. It has gone mainstream. The revolution I’m talking about is a silent one. It’s a war waged by veteran hosting customers who know the products they want, but see them mothballed or discontinued as they’re forced into one-type-fits-all technologies they don’t need — and by new customers who need powerful hosting but are overwhelmed by the proliferation of complex “solutions” in today’s hosting marketplace.
The Cloud promised to allow massive scalability and utility billing, but for some customers that has translated into complicated deployment and unpredictable costs.
Frankly, I’m tired of seeing those who desire a simpler model of powerful hosting — the tried-and-true dedicated server — marginalized by a hosting industry intent on moving to a unified model of Cloud hosting technology for all users. Read more
For almost 19 years I’ve believed that trust should be the foundation of every action we take here at ServInt. I originally set out to share my belief with small businesses by creating a different kind of hosting company — one that strives to find ways for small businesses to succeed and for entrepreneurs to realize their dreams. I wanted to let them focus on what they do, not on what we do.
Reliability, integrity, transparency, support, and expertise are cornerstones of building such a high level of trust, and these are values all of us at ServInt share. About a year ago I realized that I remained confident in why we do what we do, but despite all the breakthroughs in cloud technology I wasn’t confident in how we do it — not just as a company, but as an industry. Since that day ServInt has been working on a paradigm shift in web hosting, and today I’m proud to announce the result of our efforts.
We’re solving a problem that most people don’t even know exists. The most prolific example is someone with a new VPS or cloud service that complains of unreliable performance. They assume their provider sucks, or the industry sucks. It’s clearly not that customer’s fault and we all know it, but nobody knows what to do about it. Those types of situations make the whole industry look bad, and they should. This problem needed to be solved, and we are disrupting the industry by solving it. Nobody who cares about their business should buy a product they can’t trust, but there haven’t been cloud or VPS products they truly could trust without over-buying or over-engineering – until now. Read more
Over the years, countless businesses have started at ServInt. We are a hub for entrepreneurs looking to turn their dreams into reality. We have talked about Etsy as one of my favorite examples of this, but the list is long. We know what it’s like to start with a dollar and a dream, and we connect with the people who have that passion.
I don’t know if you know this, but ServInt doesn’t generally do a ton of coupons. Building the kind of value we have as an organization takes time and dedication, and we know our value. But it’s the end of the year, and one of my resolutions for 2014 is to find some new companies to help. I know that takes planning, and I want to give you the tools today to make that happen.
Like any other business, ServInt periodically takes stock of the configuration of its products, and makes adjustments to ensure they remain competitive. Unlike most businesses, however, when we make these adjustments, we usually provide them to our existing customers at no extra charge. That’s what we’re doing this time, too. Today, we’re doubling the guaranteed RAM specifications for our Essential, Essential+, Signature, and Ultimate VPS products, and we’re extending that enhancement to our existing customers at no extra cost.
We’re doing this for two basic reasons: one, because we want to keep our current customers happy — and two, because this is a first operational step towards a new and revolutionary platform enhancement that’s on its way soon. Read more
The Cyber Intelligence Sharing and Protection Act is coming. If you haven’t heard as much about CISPA as you did about SOPA and PIPA, you will soon. CISPA needs to change, and we need your help to change it.
CISPA is a ‘cybersecurity’ bill that exists in the U.S. House of Representatives, and it’s only a matter of time before a counterpart appears in the Senate. Last week we explained a bit about the bill and what it does here on the ServInt Source. Prior versions of CISPA were as odious as PIPA and SOPA. The Internet community needs to be vigilant that the next version isn’t as well. CISPA is not the same bill as SOPA and PIPA, but it has the potential to be just as big an affront to your civil liberties.
CISPA confuses access to information with knowledge of that information. Read more
Last week, ServInt put out a joint press release with VC fund Runa Capital announcing a program to offer discounted ServInt hosting and advanced infrastructure consultation to Runa portfolio companies, a kind of grant program for their slate of startups. We announced at Hosting Con and put out a simple joint press release for industry types. The logic was that this partnership between ServInt and Runa Capital is something industry insiders would be interested in, but it would not be on the radar of customers in the hosting industry.
But then I got thinking about just what we are doing with Runa and the wider implications. Runa Capital’s companies include all sorts of tech start-ups who are redefining the possibilities of the Internet. Their portfolio companies include Nginx, Jelastic, and StopTheHacker, and their team’s previous investments include Parallels. These are the companies Christian was talking about last week when he wrote about people changing the world with the Internet. Read more
Today marked the 17th anniversary of when I founded ServInt. 17 is an interesting milestone. Last year, at 16, I marveled at the fact that if the company I had created was a human, it would now be old enough to drive. Now we’re forging further into “adulthood,” and I’m reflecting on what we’ve built and on the friends we’ve made along the way.
I had the unique opportunity to spend this special occasion in Hong Kong, celebrating with our friends and partners from PCCW Global. Every year immediately following the Chinese New Year, PCCW hosts a large kickoff event in Hong Kong. This year they invited us to join them as their guests.
When I started ServInt, the very first bandwidth provider I signed on was a company called CAIS, short for Capitol Area Internet Service. In 2001 they were sold to a company called Ardent Communications, which sold off its networking assets to a company called BtN, or Beyond the Network Access, in 2002. Around 2006 BtN was folded into its parent company, PCCW Global – one of the world’s largest and most powerful telcos. But we still basically have the same account team we started with when I started ServInt 17 years ago. Long-term relationships like this in the Internet industry are rare to say the least! Read more
If you’re a subscriber to ServInt’s Twitter feed, you may have noticed we’ve tweeted nearly 30 separate “PIPA Facts” today. We did this because we wanted to empower all our online friends and colleagues with easy-to-remember information that would make phone calls to their senators as quick and painless as possible.
It’s easy to get scared off when people ask you to upgrade your protests against bad policy from the virtual to the real world. Tweeting and changing your Facebook status seems so much easier than actually picking up the phone and talking to somebody — somebody who you might feel understands the bill in question better than you do.
But that’s where you’re wrong. The fact is, your elected representatives almost always know a lot less than you do about what you do for a living — and they actually want you to help them understand how legislation will affect businesses in their state or district.
Calls to your elected representatives don’t have to take long. A few well-informed points, contained in a few simple sentences, are really what they’re seeking, and they’re what will make the difference. It’s easy.
So here’s what we need you to do — preferably tomorrow, but certainly no later than Monday, Jan. 23: Read more
In the wake of a well-publicized boycott campaign against GoDaddy, hosting providers are racing to try to come up with their stances against SOPA. I am proud that we don’t need to do that. Our stance on SOPA, its sister bill PIPA, and the bill from whence they both came COICA, is well known. We have spent much of the last year not just railing against these bills but trying to do something about them.
The well-intentioned goal of SOPA is to go after piracy, which is noble and very important. But piracy is something that needs to be handled smartly, with a laser-focus. SOPA isn’t a laser, it’s a wrecking ball that if enacted is likely to destroy hard-working legitimate businesses more frequently than it does pirates. SOPA allows people merely accused of ‘contributing to infringement’ to have their business taken from them. Pirates will maintain back-up plans in case they get their resources pulled – it’s legitimate businesses that will suffer most. SOPA will be used for censorship and as an anti-competitive tool. It will stifle innovation, and is one of the most dangerous bills I have seen in my two decades in this industry. Read more