Hurricane Irene and Disaster Preparedness at ServInt

With Hurricane Irene projected to make landfall on the east coast of the United States in just a few days, many customers have been asking about what ServInt is doing to prepare for this potentially severe storm.

During any extraordinary event, ServInt operates under a set of Major Event procedures that have been crafted and refined over the last 16 years. These procedures include augmenting facility staff through overtime and multiple levels of on-call, assignment of task forces to deal with any contingencies that arise, and opening and maintaining lines of communication with our customers and vendors to ensure smooth operation throughout the event.

These procedures are important for making sure we’re always ready to provide our normal level of customer care in abnormal situations.  But the bulk of our preparation for any given event takes place months to years earlier as we design and build our datacenters, choose our network providers and make our staffing decisions.

By choosing data center facilities that are expected to withstand any rationally conceivable event; by designing our network to quickly recover from the loss of any provider; and by making sure the staff we hire are reliable and steady in the face of the unexpected, we are able to work through most natural disasters without missing a beat.

In its 16 years, ServInt has weathered hurricanes, blizzards, tornadoes and—yes—even earthquakes. During Hurricane Isabel in 2003, while most businesses in the Washington DC metro area saw major disruptions, ServInt operated without customer interruption through the entire storm.

Frankly, our biggest concern during any natural disaster is making sure our employees can get to and from our facilities for their shifts. During any major event we roll out the cots and sleeping bags and stock the fridges. While we already run a 24/7 ship, we have to be prepared in the event that on-duty staff cannot make it home from a facility. Waiting for streets to be plowed or trees to be cleared is often our biggest concern.

Any web host’s true value is put to the test when extraordinary events occur, be it the explosion of traffic on a customer website, a hack or DDOS attack, or a natural disaster. These events and the host’s response are what customers remember. So as Hurricane Irene bears down on the eastern United States, rest assured that we are watching the weather carefully, unpacking our rain coats and dry-running all our disaster procedures. Our generators are fueled up and capable of powering your servers for a long time.  Our cupboards are filled to bursting with cans of soup and stew.  Our employees are ready.  Your sites are as safe as any hosting company could possibly make them, and we’ll do everything we can to keep them up and running, no matter what mother nature throws at us.

Photo by NASA Goddard Photo and Video

RAID, Backups and Beyond: The Basics of Data Protection

When you invest in the monthly expense of a hosting service, one of the biggest questions in your mind—other than, will my customers be able to get to my site?—is: How protected is my data?

Managing thousands of servers here at ServInt, we’ve seen it all: from the common problem of a VPS customer who accidentally deletes an important file to a host machine head crash physically etching the disk of the hard drive.

Human error and hardware failures are simply part of doing business as a hosting company. Even the most competent webmaster or programmer can inadvertently overwrite a file. And no piece of hardware created can be guaranteed error-free 100% of the time. So how is a customer supposed to protect him or herself from the inevitable need to replace lost or corrupted data?

There are three important—and distinct—data protection options to consider when researching hosting solutions. ServInt recommends customers utilize all three options whenever possible.

1. RAID. RAID stands for Redundant Array of Independent (or Inexpensive) Disks. Simply put, RAID is a way of replicating data over multiple disks such that if one or more disks in the array fails, no data is lost. The simplest form of RAID is RAID 1, which mirrors of one half of the array onto a second. But many other configurations exist including RAID 0, 2, 3, 4, 5, 6 and any number of combinations of these configurations.

Each RAID configurations takes a slightly different approach to duplicating data across an array including striping, mirroring, and dedicated or distributed parity (a form of checksum used to “rebuild” lost data). Different configurations have benefits and drawbacks regarding efficient use of disk space, write speed, recovery characteristics, etc.

This layer of protection—when available—allows the seamless rebuilding of degraded disk arrays and prevents much of the data loss that would otherwise occur when a disk fails. But while RAID is important data protection, it is not a complete backup solution. RAID is a means of writing data to a drive. If one disk (or more, depending on the size and type of the array) fails, the data remains accessible on the other disks while the failed drive is replaced. But if a file is corrupted or manually deleted, the fanciest RAID array in the world won’t bring it back to life. That’s where backups come in.

2. Backups. All data on a server should be protected by backups daily. Any reputable hosting company will provide a competent backup solution to its customers. Many include off-server solutions that combine the backup needs of many host machines onto large storage servers.

Why off-server? How protected is your data if the backups are stored on the same hardware as the main data? To be fair, in-server second drive backup solutions are fine in many cases, but they do require a technician to remove the drive from the chassis in case of restoration to a new piece of hardware, or replacement of the primary drive before data can be recovered from the secondary drive.  Additionally, on-server backups do not allow for central management of backup data and provide a relatively finite amount of backup capacity as compared to networked backup server solutions.

3. Off-Network/Off-Site Backups. If the only concern in our lives were the reliability of host machine hardware, hosting customers would not need to worry about off-network and offsite backups. But this is just not the case. No matter how robust a hosting company’s data protection measures are, multiple redundancies are built into systems specifically because multiple failures can happen.

Every responsible customer should initiate a program to back his or her data up on machines that are completely independent from the host machine’s hardware and network. For most small customers this might be as simple as archiving data through your control panel and downloading it to a home computer. For larger customers, this may mean seeking out a second solution from their hosting provider. Does your provider offer servers on multiple vlans in independent data centers? If not, it may pay to investigate a second hosting company for your secondary backup solution—or to switch hosts completely.

The needs of customers’ off-network/off-site backup solutions are so varied that it is difficult for hosting companies to provide out-of-the-box solutions. For this reason, it is often left up to the customer to be proactive regarding this last line of defense against data loss.

These three levels of data protection just scratch the surface of what’s available to the customer serious about protecting data. But it’s a baseline. If you can confidently say the data protection solution for your online content includes these three levels of protection, you are well on your way to secure deployment. And you are ahead of much of the competition as well!

 

Photo by Ruben Molnes

ServInt Has Your Golden Ticket! Get Ready for VaultPress on ServInt!

At ServInt, we have thousands of clients who rely on WordPress software to run their business. They’re attracted to its simplicity, its ubiquity, and the myriad of plugins available to scratch nearly any “feature-itch” imaginable. Plus, with our free nightly backups, they have a little extra peace of mind.

Well, that peace of mind just grew 11-fold.

Today, we’re proud to announce that ServInt is one of the exclusive launch partners for VaultPress, the incredible backup and security service developed by Automattic, the creators of WordPress.com. Automattic has granted us access to a “Golden Ticket Machine” enabling us to offer all of our clients who are using WordPress access to the VaultPress beta!

VaultPress is a continuous security and backup system that works in real-time to dynamically backup every post, tag, folder, and font change you can make in WordPress. It even picks up on things that WordPress doesn’t, such as files you’ve added via FTP.

VaultPress security monitoring scans uploads, plugins, and themes and alerts you when it detects potential security issues. VaultPress doesn’t just make one backup, either, it goes to 11. Combine that with ServInt’s backup system, and you have 12 distinct backups (though you lose out on the Spinal Tap reference). This level of redundancy virtually ensures you’ll have all your data, all the time.*

Installation? Easy. VaultPress installs just like any other WordPress plugin.

The cost? ServInt will be extending VaultPress’ beta discount to all interested clients, $15 a month for the Basic Level, $40 a month for the Premium Level.

What are you waiting for? Click here to go to ServInt’s Customer Portal and request a Golden Ticket now!

If you aren’t using WordPress, don’t worry. At ServInt, ALL of our VPS, SuperVPS, and Solo Series clients have access to our incremental backup system which offers daily backups for exactly ZERO dollars. Whether it’s your personal computer at home, the critical documents you have at work, or the website living comfortably on your server, backing up your data is an absolute necessity and at ServInt, we make it easy.

For the thousands of ServInt clients who rely on WordPress to run their business, well, now it’s even easier!

So get your Golden Ticket now!

*Unless there’s a robot uprising or zombie infection. ServInt and VaultPress
can’t really help you there.

What Can Buzz Lightyear Teach Us About Backing Up?

If your data is your livelihood, watch this and learn about the time Toy Story 2 disappeared:

It’s a teaser for a longer story on the Toy Story 2 DVD, which I watched with my son this weekend. It starts:

“When making a film like Toy Story 2, we use a bunch of UNIX and Linux machines. On those kinds of machines there’s a command, RM*, that removes everything on the filesystem as fast as it can.”

“Somebody had run RM* on the drive where all the Toy Story 2 files were kept, and things just started to disappear.”

After the teaser, the story goes on to describe how their backups for the past two months had been corrupt, and that the work on the film from the past two years was just gone. Freaking out, they started putting together a plan to push the film back an entire year to try to keep from having to scrap it altogether.

But Galyn Susman, visual arts director at Pixar, came to the rescue. She had just had a baby, and in order to spend time with her newborn had set up a system she could work on from home. She brought with her a copy of the film. The ONLY copy of the film.

My wife calls this story a lesson about how cool new moms are. I say it’s a reminder that if your data is important you can never have enough backups.

Imagine that for moment. Imagine everything you had worked for, months even years worth of data, literally disappearing in front of you. This happens to countless people everyday and it really underscores the fragility of our data. A few unfortunate keystrokes and Pixar, one of the most respected and lucrative animation studios in the world, nearly lost a Golden Globe winning film to the ether.

At ServInt we back up our clients’ files for free, automatically, on every product we sell on our website. Our backup service is awesome – it is reliable and robust and I think it’s an Industry-leading service. But if anybody tells you that ANY backup is 100% reliable 100% of the time, call them a liar and run away from them immediately.

If your data is critical to your livelihood, you can NEVER have enough backups. Go ask Pixar. Then go watch Toy Story 2 or some other Pixar movie. They are all awesome – and at least one survived to be seen because of backups.

Sometimes Sometimes Redundancy Redundancy Fails

Stuff happens.

Of course, we all know that. Everyone has bad days where everything seems to go wrong and businesses are no different.

In our case, the hosting industry relies on thousands of pieces of ever-changing, ever-evolving hardware that need to be married with new standards and updated software. Problems do happen, even to those who plan carefully while building their infrastructure, but how a company responds when problems occur is far more important than whether problems happen at all.

At ServInt, we have always prided ourselves on being the kind of company that plans extensively. That being said, we are still faced with a serious, and slightly philosophical question:

Why do problems occur and how do we prevent them?

Our industry has always loved to tout how easy it is to be up all the time, immune to problems, and how technology can solve just about anything. Because the industry has been so successful at selling this message, we frequently hear comments like, “Well, if my server has RAID and you have redundant routers, how can anything go wrong?” Many assume, and justifiably so, that if something fails, the redundant resource should take over automatically.

More after the jump…

Read more