IT and Patience

Communicate Solutions' Data Center

Here at Communicate Solutions, we have a distributed "data center".  I use quotes because it's not like a traditional data center where all our servers are housed in a single climate controlled, key-card secured room in South Carolina. Our machines are actually VPS's, located in different parts of the country and hosted by different vendors.  This works well since we don't have to invest in any specific hardware, get to pick and choose the vendor for the application, and get the traditional advantages of the data center (backup power, redundant data, climate control, etc).  The only hard part is backing up the data from all these machines.  We used to do backups via FTP and boy was that a nightmare.  Custom scripts, password management, security issues, and flaky network connections meant headaches and less than ideal results.  What's a geek to do?

Cue Bacula

Bacula is the Open Source, Enterprise ready, Network Backup Tool for Linux, Unix, Mac and Windows.  It is what we implemented and it has been rock solid.   It's got centralized administration, backup, recovery, data verification, volume management, and messaging to name a few features.  The real advantage is the centralized administration of backing up diverse clients.  From one command prompt, I can backup my Windows and Linux machines.  The data gets encrypted over the wire and stored with compression, which keeps backup small.  Volume management means that I can define a retention policy and old backups are cycled out as new backups are taken.

The Catch

If there's any catch with Bacula, it's that the there needs to be a Linux box where the central console runs.  If you don't run Linux anywhere, sorry, you're pretty much S.O.L.  Other than that, there's a bit of a learning curve with the volume management stuff if you're not familiar with the concepts.  The documentation is exquisite though.

So if you're wondering how you're going to backup your data without losing your mind or spending an arm and a leg, definitely take a look at Bacula.  Because of it I can now sleep peacefully :)

P.S.

As Joel says, make sure you're backups are being tested with restores.  Otherwise its all for not!

I had another couple long nights Sunday night and last night.  You see, I moved my several existing websites to a new VPS.  I blogged about my initial move from a physical server to a virtual server here.  The reason I switched this time is because of problems I had with VPSLand (my old VPS vendor).  Basically I was unable to get a connectivity issue resolved for several days.  During that time, several my websites were down for extended periods of time.  Support from VPSLand is only available via their online ticket system and my ticket did not get the attention that I thought it deserved.  It was extremely frustrating to have these issues and not be able to call anyone about them.  It seemed I was always waiting for the "senior admin" to research the issue.

My new VPS vendor is Fluid Hosting.  They are somewhat more expensive than VPSLand but they have online AND phone support and they got pretty good ratings in a WebHostinMagazine.com VPS comparison review.  I guess I'll have to wait and see how the new vendor works out but I'm hoping this is an example of "you get what you pay for".

This move was a little more difficult because the new machine is 64-bit Windows.   I've done plenty of work with 64-bit Windows machines in my day job, so it's nothing new to me, it just added a bit more complexity.  Funny story: I was having a problem with getting one app working.  The error pointed to permission issue but I thought I was smarter than the computer and troubleshooted as if the problem was more of a 64 vs. 32-bit OS issue.  I ran out of time troubleshooting and had to revert back to the old server for this client/website.  After I was done moving my other clients, I spent several more hours troubleshooting the problem with tools like FileMon, Process Explorer, & Dependency Walker.  Finally, I narrowed the problem down to a permission issue.  It ended up that I just fat fingered a new user's password...   That's about ten hours of my life I won't get back!

It was strange though that I couldn't get FileMon to run successfully on the VPS... when I started the program I was getting errors saying "Error loading FILEMON701: This driver has been blocked from loading".  Extensive Googling yielded only a few articles which indicated that the problem was permission related.  That doesn't make sense because I'm an admin on the machine and I can do all the normal admin things.  I also have the appropriate user rights (Debug programs & Load/unload device drivers).  I never got to the bottom of this.  I haven't had this problem on other VPS's.  Let me know if you've seen the same thing or if you know the solution.

Finally, I'm beginning to realize that it's more trouble than its worth to host multiple clients/websites on a single machine.  Because I host multiple clients together, it makes these types of moves very difficult, a caffeine fueled marathon even.  Between the DNS changes, the Bacula backup configuration, complexity of some of the websites, and the windows of time available to do the move, it would have been much easier if I could have moved one website at a time.  Yes, I make a little money by co-hosting the websites but it's not worth the pain when there is an outage; everybody's website is down and I'm giving out refunds.  Besides, if Good to Great (Collins, ISBN 0066620996) has taught me anything, I need to focus on my core business (producing software) rather than pursuing many ends at the same time (hosting).

Lessons learned:

1. When choosing a hosting vendor, ALWAYS make sure they have some sort of phone support.  If not, you'll regret it when you have connectivity problems and there's no one to talk to.
2. If you're in the hosting business but hosting isn't your core competency, get out of the hosting business :)  If you must host, host your clients separately.

Well I think I'm just about caught up on my sleep...

Monday and Tuesday were long days.  Let me give you some background.  This story started months ago when I was just about finished with a website I was working on.  I needed a seperate server where I would be able to house the website.  I knew I wanted a seperate server to host this website because this client needed a solution that involved installing custom applications on the server (think email, which is not only custom but is sort of risky to have on a server which other clients use).   I don't host websites out of my basement (anymore) and the cost of collocating a machine or purchasing a dedicated server is fairly prohibitive.  What the heck was I going to do?

Fortunately I spoke with my good friend Marty who listened to my problem and recommended purchasing a virtual private server (VPS).  A VPS vendor uses virtual machine technology, such as VMWare, to slice up a single, physical server into many, virtual servers.  Apparently there a quite a few vendors which rent/sell VPS instances online, provide full desktop access with administrator permissions via terminal services,  and cost less than $30.00 per month.  Perfect!  I did some research and ended up going with a vendor named VPSLand, based out of Atlanta, GA.  They have a good selection of packages at pretty reasonable prices with guaranteed uptime.

The VPS worked out well for my client.  It worked out so well, I decided I could get rid of my existing dedicated server, which hosted my website among others, and move to a fully virtualized environment.  The dedicated server was hosted by Cari.net, which was excellent, but much more pricey compared to virtual servers.

The move from dedicated to virtual needed to occur during night time since these are live websites.  I started the job Monday afternoon and worked continuously until Tuesday morning at 9:00AM.  It went pretty smoothly for the most part but I did hit a couple snags.  I wasn't able to get everything fully configured until about 12 noon.

Fortunately I have this week off from my day job:)  While it's certainly no record for the number of hours worked consecutively, it was definitely a long night for me but I think it will be worth it.  Me and my clients will be looking forward to much smaller hosting bills!

Update 11/25/2008

See the updated entry about moving my websites to different VPS here.