Introducing: Server Monitoring

[nubz]

Originally Posted by tekhoi

Is Server Density hosted by VPS.NET? Last night no data was collected for nearly 4 hours (During DDOS problems) but I didn't get any alerts.

are you saying you have this service and were not alerted yesterday that your VPS was unavailable?

[vxnick]

AFAIK, ServerDensity is hosted at Rackspace.

[Steve]

I have my two most important VPSs monitored by server desnisty, and there were no alerts at all. As far as SD was concerned, they were up and running just fine... needless to say, I will no longer be using server density. It's just a cool "toy" (especially on my iphone) but is of no real use to me.

Even the reports/graphs are of no use in troubleshooting all the problems I've been having.

[tekhoi]

nubz, yes thats right - I signed up with this service earlier this week , but didn't get any alerts last night.

I can see from the the dashboard that no data was collected between 19:40 and 23:30 (that period is completely missing) I have an alert set up to be triggered if no data is received for 5 mins, and also alert me when the issue is resolved. I didn't get either alert.

[vxnick]

I'd recommend Pingdom (I'm sure many others would too) for server monitoring.

[BoxedIceDM]

Originally Posted by Steve

I have my two most important VPSs monitored by server desnisty, and there were no alerts at all. As far as SD was concerned, they were up and running just fine... needless to say, I will no longer be using server density. It's just a cool "toy" (especially on my iphone) but is of no real use to me.

Server Density is hosted with Rackspace and we have servers deployed across multiple data centres. These were configured in a "manual" failover setup - whilst we have data replicated, a downtime event would require us to manually switch over to the alternative servers.

We are participating in a new Rackspace disaster recovery service which allows for automatic failover in the event of a data centre going down. Last night we were configuring the initial setup of this, work which was expected to have no service impact. Unfortunately, a problem arose with our database around the same time as the vps.net outage which caused about 20 minutes of downtime for Server Density.

Because this tied in with the vps.net outage, that we were not receiving data was not picked up by the alerting.

Our data collection is split into two data stores. We use an in-memory "real time" cache to store the very latest check data. This is what the alerts are compared against. We also store the data for the graphing, but this is for data points every 5 minutes - we do not store data received every minute.

As a result of our own outage and the vps.net outage, a discrepancy arose with the data in the "real time" cache and the historical data which compounded the issue with the alert detection.

Clearly, this is the worst possible situation - your server(s) are down and you're not getting alerts. So what are we going to do to prevent this from happening again?

1. Our automatical failover is now configured and we will be working with Rackspace to fully test the setup to ensure that our system is redundant.
2. We will be running our alert detection cycles more frequently so we can pick up the alert conditions faster.
3. We will deploy a fix to allow us to correlate the data in the "real time" cache with the historical data so that if a gap arises, we can still trigger alerts.

Originally Posted by vxnick

I'd recommend Pingdom (I'm sure many others would too) for server monitoring.

Server Density works very well with Pingdom. Pingdom will tell you when your website/services are down and you can then use Server Density to look deeper into the server to find out what caused the issue. Our "no data" alerts are not designed to be "service down" alerts as they only tell you when we stop receiving data (or are supposed to). They should be combined with monitoring of specific services using the likes of Pingdom for external monitoring.

Originally Posted by Steve

Even the reports/graphs are of no use in troubleshooting all the problems I've been having.

I'd be interested to find out why our graphs were of no use to you and to find out a bit more about your workflow so we can see how we can improve what we display.

I am sorry for the failure of the service over the last day. If you have any questions you can contact me directly by e-mail on david@boxedice.com and I will continue to monitor this thread.

[Steve]

Originally Posted by BoxedIceDM

I'd be interested to find out why our graphs were of no use to you and to find out a bit more about your workflow so we can see how we can improve what we display.

Sorry for not being clearer: it's not server density's fault, but simply because the problems I've been having are due to VPS.net network problems, servers not rebooting and so on. So the info provided by SD is of no help.