Nagios clients behind paranoid firewalls
Acting proactively is one of the key rules to save you from an infrastructure malfunction. However, hosting providers these days are paranoid about firewalls -- unfortunately, monitoring ports do not belong in the list of standard open ports. Whether you have a paranoid provider, a private data center not visible from the outside world, or a home server behind NAT your servers can be monitored using Nagios. In the next few lines we discuss our solution to this problem!
Nagios infrastructure monitoring
Rather than waiting for someone calling you because their server is down due to a disk crash, a broken upgrade or a weird behaviour of the system, having email alerts for such cases can save the day. As numerous organizations do, we use the excellent Nagios free software to monitor servers and get alerts for any system or service misbehaviour. Nagios operates on a classic server-clients model, where clients run a service that listens for connections from the server on port 5666.
We wanted to add another Nagios client, but port 5666 is firewalled on this specific server and there was no way we could overcome the ISP's policy. Another option we considered is using another open port on that server, and have the Nagios server talk to that port -- unfortunately, the ISP's policy does not allow non-standard ports (namely any other than ssh-web) behind that server. Well, the ssh local forwarding technique proved to be the swiss army knife for us!
Setting up the ssh tunnel
We setup a local ssh forwarding tunnel from Nagios server, to the Nagios client machine. Then we configure the server to connect to that port, instead of the client:5666 port.The tunnel can be created by any user, as long as we use non-priviledged ports (> 1024).
We first create a key pair on the server and copy the public key on the client, in .ssh/authorized_keys
Generating public/private rsa key pair.
Then, we create the tunnel on the server side that installs an ssh tunnel: localhost port 34345 forwards traffic on client port 5666, as our client denies direct access to port 5666.
nohup ssh -L 34345:localhost:5666 nagios_client.net -N -i /home/user/.ssh/nagios_key &
Notice the use of nohup; we want the connection to be kept on the background, and ssh -N which is helpful when forwarding ports, denying the execution of any remote command.
Setting Nagios to use the tunnel
Now on our nagios server we are reay to setup the client, as we have already done with all other servers
alias nagios client X
On file commands.cfg, we setup a version of the check_nrpe command that will check for localhost port 34345 (the ssh tunnel) instead of the client's ip at port 5666, that is firewalled. This is the most important thing to remember, after setting up correctly the tunnel. We'll name this command as check_nrpe_firewalled
command_line $USER1$/check_nrpe -H 127.0.0.1 -p 34345 -t 100 -c $ARG1$
That will do the job. Any service that we need to specify for that client, we will have to prepend the nagios check with check_nrpe_firewalled.
service_description Check mail Queue
Once we create the host, set the services and install the Nagios client, we restart Nagios server and everything runs like a charm! We have successfully setup Nagios on our firewalled client, using the friendly help of ssh tunnels.
This method also applies to other cases e.g. the client is behind a NAT (home servers, private data centers, etc). In this case however, instead of local ssh forwarding, we would need remote ssh forwarding.
It's also useful for encrypting nagios traffic if you don't want to use the native SSL support Nagios >2.0 provides!