Nagios clients behind paranoid firewalls

Acting proactively is one of the key rules to save you from an infrastructure malfunction. However, hosting providers these days are paranoid about firewalls -- unfortunately, monitoring ports do not belong in the list of standard open ports. Whether you have a paranoid provider, a private data center not visible from the outside world, or a home server behind NAT your servers can be monitored using Nagios. In the next few lines we discuss our solution to this problem!

Nagios infrastructure monitoring

Rather than waiting for someone calling you because their server is down due to a disk crash, a broken upgrade or a weird behaviour of the system, having email alerts for such cases can save the day. As numerous organizations do, we use the excellent Nagios free software to monitor servers and get alerts for any system or service misbehaviour. Nagios operates on a classic server-clients model, where clients run a service that listens for connections from the server on port 5666.

We wanted to add another Nagios client, but port 5666 is firewalled on this specific server and there was no way we could overcome the ISP's policy. Another option we considered is using another open port on that server, and have the Nagios server talk to that port -- unfortunately, the ISP's policy does not allow non-standard ports (namely any other than ssh-web) behind that server. Well, the ssh local forwarding technique proved to be the swiss army knife for us!

Setting up the ssh tunnel

We setup a local ssh forwarding tunnel from Nagios server, to the Nagios client machine. Then we configure the server to connect to that port, instead of the client:5666 port.The tunnel can be created by any user, as long as we use non-priviledged ports (> 1024).

We first create a key pair on the server and copy the public key on the client, in .ssh/authorized_keys

user@user:~$ ssh-keygen 
Generating public/private rsa key pair.

Then, we create the tunnel on the server side that installs an ssh tunnel: localhost port 34345 forwards traffic on client port 5666, as our client denies direct access to port 5666.

nohup  ssh -L 34345:localhost:5666 -N -i /home/user/.ssh/nagios_key &

Notice the use of nohup; we want the connection to be kept on the background, and ssh -N which is helpful when forwarding ports, denying the execution of any remote command.

Setting Nagios to use the tunnel

Now on our nagios server we are reay to setup the client, as we have already done with all other servers

define host{
use                     generic-host
alias                   nagios client X
address                 X.Y.Z.A
check_command           check-host-alive
max_check_attempts      5
notification_options    d,u,r
contact_groups          nagios-admins-two

On file commands.cfg, we setup a version of the check_nrpe command that will check for localhost port 34345 (the ssh tunnel) instead of the client's ip at port 5666, that is firewalled. This is the most important thing to remember, after setting up correctly the tunnel. We'll name this command as check_nrpe_firewalled

define command{
command_name check_nrpe_firewalled
command_line $USER1$/check_nrpe -H -p 34345 -t 100 -c $ARG1$

That will do the job. Any service that we need to specify for that client, we will have to prepend the nagios check with check_nrpe_firewalled.

define service{
use                             unweb
contact_groups                  unweb-admins
service_description             Check mail Queue
check_command                   check_nrpe_firewalled!check_mailq

Once we create the host, set the services and install the Nagios client, we restart Nagios server and everything runs like a charm! We have successfully setup Nagios on our firewalled client, using the friendly help of ssh tunnels.

This method also applies to other cases e.g. the client is behind a NAT (home servers, private data centers, etc). In this case however, instead of local ssh forwarding, we would need remote ssh forwarding.

It's also useful for encrypting nagios traffic if you don't want to use the native SSL support Nagios >2.0 provides!