IT and Patience

Communicate Solutions' Data Center

Here at Communicate Solutions, we have a distributed "data center".  I use quotes because it's not like a traditional data center where all our servers are housed in a single climate controlled, key-card secured room in South Carolina. Our machines are actually VPS's, located in different parts of the country and hosted by different vendors.  This works well since we don't have to invest in any specific hardware, get to pick and choose the vendor for the application, and get the traditional advantages of the data center (backup power, redundant data, climate control, etc).  The only hard part is backing up the data from all these machines.  We used to do backups via FTP and boy was that a nightmare.  Custom scripts, password management, security issues, and flaky network connections meant headaches and less than ideal results.  What's a geek to do?

Cue Bacula

Bacula is the Open Source, Enterprise ready, Network Backup Tool for Linux, Unix, Mac and Windows.  It is what we implemented and it has been rock solid.   It's got centralized administration, backup, recovery, data verification, volume management, and messaging to name a few features.  The real advantage is the centralized administration of backing up diverse clients.  From one command prompt, I can backup my Windows and Linux machines.  The data gets encrypted over the wire and stored with compression, which keeps backup small.  Volume management means that I can define a retention policy and old backups are cycled out as new backups are taken.

The Catch

If there's any catch with Bacula, it's that the there needs to be a Linux box where the central console runs.  If you don't run Linux anywhere, sorry, you're pretty much S.O.L.  Other than that, there's a bit of a learning curve with the volume management stuff if you're not familiar with the concepts.  The documentation is exquisite though.

So if you're wondering how you're going to backup your data without losing your mind or spending an arm and a leg, definitely take a look at Bacula.  Because of it I can now sleep peacefully :)

P.S.

As Joel says, make sure you're backups are being tested with restores.  Otherwise its all for not!

I had another couple long nights Sunday night and last night.  You see, I moved my several existing websites to a new VPS.  I blogged about my initial move from a physical server to a virtual server here.  The reason I switched this time is because of problems I had with VPSLand (my old VPS vendor).  Basically I was unable to get a connectivity issue resolved for several days.  During that time, several my websites were down for extended periods of time.  Support from VPSLand is only available via their online ticket system and my ticket did not get the attention that I thought it deserved.  It was extremely frustrating to have these issues and not be able to call anyone about them.  It seemed I was always waiting for the "senior admin" to research the issue.

My new VPS vendor is Fluid Hosting.  They are somewhat more expensive than VPSLand but they have online AND phone support and they got pretty good ratings in a WebHostinMagazine.com VPS comparison review.  I guess I'll have to wait and see how the new vendor works out but I'm hoping this is an example of "you get what you pay for".

This move was a little more difficult because the new machine is 64-bit Windows.   I've done plenty of work with 64-bit Windows machines in my day job, so it's nothing new to me, it just added a bit more complexity.  Funny story: I was having a problem with getting one app working.  The error pointed to permission issue but I thought I was smarter than the computer and troubleshooted as if the problem was more of a 64 vs. 32-bit OS issue.  I ran out of time troubleshooting and had to revert back to the old server for this client/website.  After I was done moving my other clients, I spent several more hours troubleshooting the problem with tools like FileMon, Process Explorer, & Dependency Walker.  Finally, I narrowed the problem down to a permission issue.  It ended up that I just fat fingered a new user's password...   That's about ten hours of my life I won't get back!

It was strange though that I couldn't get FileMon to run successfully on the VPS... when I started the program I was getting errors saying "Error loading FILEMON701: This driver has been blocked from loading".  Extensive Googling yielded only a few articles which indicated that the problem was permission related.  That doesn't make sense because I'm an admin on the machine and I can do all the normal admin things.  I also have the appropriate user rights (Debug programs & Load/unload device drivers).  I never got to the bottom of this.  I haven't had this problem on other VPS's.  Let me know if you've seen the same thing or if you know the solution.

Finally, I'm beginning to realize that it's more trouble than its worth to host multiple clients/websites on a single machine.  Because I host multiple clients together, it makes these types of moves very difficult, a caffeine fueled marathon even.  Between the DNS changes, the Bacula backup configuration, complexity of some of the websites, and the windows of time available to do the move, it would have been much easier if I could have moved one website at a time.  Yes, I make a little money by co-hosting the websites but it's not worth the pain when there is an outage; everybody's website is down and I'm giving out refunds.  Besides, if Good to Great (Collins, ISBN 0066620996) has taught me anything, I need to focus on my core business (producing software) rather than pursuing many ends at the same time (hosting).

Lessons learned:

1. When choosing a hosting vendor, ALWAYS make sure they have some sort of phone support.  If not, you'll regret it when you have connectivity problems and there's no one to talk to.
2. If you're in the hosting business but hosting isn't your core competency, get out of the hosting business :)  If you must host, host your clients separately.