Setting Up Monitoring With Monit

Welcome back to this next part in our series on setting up a high availability solution using the VPS.net public cloud. In our previous post, we finally got all of our components working together to provide a working high availability solution. This meant that all files and databases were synchronized between servers automatically and we had an IP address floating between the servers being used by the Apache web server. You might be wondering why we are still here if we’ve managed to get the system up and running. The reason for this is that if a server fails the other will take over serving the web pages, but you would be unaware that it had done so unless you were manually checking on the servers. So this time we’ll be looking at setting up some automatic notifications for the server status using Monit.
Anyone who has done some reading about heartbeat may have noticed that it is also possible to have heartbeat send email notifications. Unfortunately the system will only mail out when the transition from one server to the other takes place. This means that should the secondary server fail while the primary is serving the website, you would not receive an email notification. The method we are going to use should be more complete, and will notify if one server should go away, but also if the web server stops serving files.
 
To start with, we will install monit itself, a simple task from the default repositories:
sudo apt-get install monit
 
With that installed, we’ll be looking at setting up some basic configuration, which is done in the /etc/monit/monit.rc file:
sudo nano /etc/monit/monitrc
 
Monit has a relatively straightforward configuration format, and the file is heavily commented with examples and explanations. The main elements we want to set here are your mailserver settings and alert settings. Scroll to the bottom of the file and paste in the following lines:
set mailserver localhost
set alert you@yourdomain.com
 
You’ll need to change “you@yourdomain.com” to your own email address. You can also have multiple alert lines to email more than one address with alerts. If you installed iwatch previously, then it will have installed the postfix mail server for you, which means that we can use localhost to send the email alerts for you. If this isn’t installed and you don’t have a mail client on your server, then you can use a remote mail server instead by changing the mailserver line to:
set mailserver mail.yourdomain.com port 587
username “you@yourdomain.com” password “mypassword”
using tlsv1
with timeout 30 seconds
 
Obviously for this to work you’ll need Monit to be logging into a valid email address at the mailserver for your emails to send. You’ll need to change the mailserver address, username and password to match the details you log in with to send email.
With your mailserver and alert destinations set, you can save and exit the file.
Next we need to create the configuration to monitor the servers. Configuration is stored in /etc/monit/conf-available, and we’ll start with a script to monitor that the website is up:
sudo nano /etc/monit/conf-available/website
 
Paste in the following, changing the IP address to your website’s domain name:
check host web-server with address 192.168.0.4
    if failed port 80 protocol http with timeout 10 seconds then alert
Save and exit the file after making your changes. This will get monit to check that it can connect to your website, and will send an alert if it takes longer than 10 seconds to connect.
 
Next, we want to set up a ping test to each server directly to ensure they are online:
sudo nano /etc/monit/conf-available/partner-server
 
Paste in the following code, ensuring that on the first server the IP address used is that of the second server, and on the second that it is the first server’s IP. Change the hostname to match the server being tested – you can use your own names or firstserver/secondserver as you require.
check host firstserver with address 192.168.0.2
    if failed icmp type echo
       count 5 with timeout 5 seconds
       2 times within 3 cycles
    then alert
Save and exit the file. This file tells monit to ping the server, and if it doesn’t respond for 2 out of 3 checks to send an alert.
 
Finally, you need to enable the configuration. This is done by making a link from the files in conf-available to the conf-enabled directory. We’ll also add monitoring for the local MySQL and SSH servers while we are here:
sudo ln -s /etc/monit/conf-available/mysql /etc/monit/conf-enabled/
sudo ln -s /etc/monit/conf-available/openssh-server /etc/monit/conf-enabled/
sudo ln -s /etc/monit/conf-available/website /etc/monit/conf-enabled/
sudo ln -s /etc/monit/conf-available/partner-server /etc/monit/conf-enabled/
 
With those done, you can now restart Monit and the monitoring will start:
sudo systemctl restart monit
 
As with most of our guides in this series, you need to do this on both servers. Next time we’ll be rounding off with some final thoughts on the setup.