The greatest task on any job, has to be product tolerance testing.

We are gearing up for a series of tests that will define what sort of SAN we will be using on the VPS Cloud, the actual tests will be performed early next week, for now we are setting up the hardware for it, once we have done it, we will post the results here in the bog.

Here is the info on what we are planning …

Basically, our data backend will be a SAN consisting of a huge number of virtual disk targets. The way this works is as follows:

• Each physical storage server (NAS) breaks down its disk array into smaller logical volumes (using LVM), with 1 volume for each virtual machine

• Each of these volumes is then shared using ATAoE technology across a switched gigabit Ethernet SAN, allowing all the hypervisors to efficiently access any virtual machine’s disks as required

This method allows hypervisors to be fully redundant, allows new storage hosts to be easily added to the SAN, and in the near future will allow full redundancy of storage hosts.

Initially our plan was to use the scenario 1 .. each NAS would consist of 10TB of Raid 5 data, the backups would then reside on separate 10TB (also Raid 5) clusters, if a volume failed, it could be restored into any of the SAN nodes within seconds.

But we then decided we want even greater availability, by RAID-1 the individual NAS themselves, so if a NAS itself fails, a mirror takes over, no need to restore backups (of course, we will still keep the backup nodes, you can never have enough redundancy) .. this will be the case, but not for the January 31st release, as we are still working on the backend code to make this possible.

By keeping the individual nodes at 10TB each, we expected the I/O to be fine, but doubts started to arise lately, and the last thing we want is a 10TB node to become saturated, so a decision was made to do a test, 10TB Raid 5 nodes vs 5TB Raid 10 .. basically, it’s the same config for the nodes, it jut happens Raid 10 halves the available data for the same amount of HD’s than Raid 5 (but you Raid buffs already know that)

Of course, as a tech, I love Raid 10, heck, i’d even use it on the backup nodes if I could .. but the accounting department would be less than impressed `:), no matter how you look at it, Raid 10 doubles the cost of Raid 5, and the more it costs, the more one needs to prove that it’s justifiable …

So come back soon, and we will post our tests results once we have them.

We had done some tests on the VPS Cloud self healing measure by shutting down a Hypervisor’s services, and always, the VPS nodes residing there, would automatically boot up on a different one in the Cloud … always below 40 seconds (yes, 40 seconds)… but we decided that was no fun, it was time to go to the datacenter and pull some cables … yes .. I love my job

So there we are, myself and Paul (head of IT @ UK2), we located the Hypervisor we wanted to test, enabled monitors for the main NIC and the VPS’s IP’s … all systems go, the anticipation of success in the air, our palms sweating, giggling like schoolgirls .. and “click” … we pulled the cable …

We then started counting till the VPS’s where back up elsewhere in the VPS Cloud … 1,2,3 … 40 .. ok, anytime now … 60 … 90 ? .. wait a second … what is wrong here ?

3 minutes later, we looked at each other in horror … the unthinkable happened … the VPS Cloud self healing feature .. one of the cornerstones of our offer .. had failed !!!!

Luckily this is still our beta testing .. but what had happen ? it always worked when we turned the services down, why did it fail when we pulled a NIC cable instead ?, what was the difference ? .. and so it began todays “must know” task.

We started by looking at the logs, nothing strange there, actually, nothing there at all for the past few minutes .. thats when it hit .. there was indeed NOTHING in there, the logs show no downtime had ben detected for a Hypervisor.. we returned to the Admin CP, and sure enough .. it still detected that system as being up .. but how ?

Well, as so often is the case, you look into a few hundred lines of code, until you decide to instead look at the obvious … could the internal monitor daemon failed … and if it failed .. why where we not notified ? .. wait .. what monitors the monitor .. ? ..

Simple schoolboy error, we had all sorts of monitors, bells and whistles .. if anything in the VPS Cloud fails, it gets detected within 5 seconds or less .. a true example of monitoring excellence … but what we forgot was .. what if the monitor fails ?

This brings me back to the post topic, the importance of failure .. there was no difference on our test .. pulling the cable or shutting down the services all lead to the VPS Cloud monitor to kick in and do it’s job … it was just a coincidence that this time the monitor daemon had hang, had the failure not happen, this simply oversight could have cause trouble later on, so yes, failure is good .. as long as it happens during beta testing .

So we rewrote some of the daemon to be more robust (we found what caused it to fail and fixed it) and implemented extra monitoring procedures that now we monitor the monitor too

This emphasizes the importance of beta testing and fault simulation, so often we see companies go live with untested ground breaking products and have a miserable first quarter or two of constant failure and bug fixing, many times driving them to closure .. if not properly tested, it’ the small things that will get you in the end.