Build a badass monitoring server

In many cases, we meet a good system with minor monitoring problems.

Sometimes it's the infrastructure layer, something it's the application layer.. even connectivity.

So what do we need to do? m-o-n-i-t-o-r-!
In this post I will try an cover some tools that can help you do just that!
But first, I would like to thank a good friend and a linux wizard, Mr. Tomer Kom.

First Step
Let's start by installing ES application to store our data:

curl -O https://download.elasticsearch.org/elasticsearch/elasticsearch/elasticsearch-1.0.1.tar.gz
tar zxvf elasticsearch-1.0.1.tar.gz
cd elasticsearch-1.0.1/
./bin/elasticsearch

Ok, cool! we've got ourselves a new DB to store everything.

Now we need it to run as a demon

remove existing startup script

sudo update-rc.d -f elasticsearch remove 

and create a new one and configure it to your server specs

vi /etc/init.d/elasticsearch

#! /bin/sh

### BEGIN INIT INFO

# Provides:          elasticsearch

# Required-Start:    $all

# Required-Stop:    $all

# Default-Start:    2 3 4 5

# Default-Stop:      0 1 6

# Short-Description: Starts elasticsearch

# Description:      Starts elasticsearch using start-stop-daemon

### END INIT INFO

ES_HOME=/opt/elasticsearch

ES_MIN_MEM=256m

ES_MAX_MEM=1g

DAEMON=$ES_HOME/bin/elasticsearch

NAME=elasticsearch

DESC=elasticsearch

PID_FILE=/var/run/$NAME.pid

LOG_DIR=$ES_HOME/logs/

DATA_DIR=$ES_HOME/data/

WORK_DIR=/tmp/$NAME

CONFIG_FILE=$ES_HOME/config/elasticsearch.yml

DAEMON_OPTS="-d -p $PID_FILE -Des.config=$CONFIG_FILE -Des.path.home=$ES_HOME -Des.path.logs=$LOG_DIR -Des.path.data=$DATA_DIR -Des.path.work=$WORK_DIR"

test -x $DAEMON || exit 0

set -e

case "$1" in

start)

    echo -n "Starting $DESC: "

    mkdir -p $LOG_DIR $DATA_DIR $WORK_DIR

    if start-stop-daemon --start --pidfile $PID_FILE --startas $DAEMON -- $DAEMON_OPTS

    then

        echo "started."

    else

        echo "failed."

    fi

    ;;

  stop)

    echo -n "Stopping $DESC: "

    if start-stop-daemon --stop --pidfile $PID_FILE

    then

        echo "stopped."

    else

        echo "failed."

    fi

    ;;

  restart|force-reload)

    ${0} stop

    sleep 0.5

    ${0} start

 ;;

  *)

    N=/etc/init.d/$NAME

    echo "Usage: $N {start|stop|restart|force-reload}" >&2

    exit 1

    ;;

esac

exit 0

AND

add to startup list:

update-rc.d elasticsearch defaults

this is to prevent the instance to accidentally join a cluster, without being configured appropriately. 

you can use the following commands to ensure, that elasticsearch starts when the system is booted and then start up elasticsearch:

sudo update-rc.d elasticsearch defaults 95 10
sudo /etc/init.d/elasticsearch start

keep in ming that you may need to add permissions to this file
chmod u+x /etc/init.d/elasticsearch

Second Step

Install logstash to collect and manage your logs for you.

donaload and extract it:

curl -O https://download.elasticsearch.org/logstash/logstash/logstash-1.4.1.tar.gz

tar zxvf logstash-1.4.1.tar.gz
cd logstash-1.4.1

now create a configuration file that matches our production site:

vi logstash-simple.conf

    input {



     udp { format => "json" port => 9999 type => "sample" }



    }



    output {



     elasticsearch { host => localhost }



  stdout { codec => rubydebug }



}




so now we are getting data from UPD in port 9999 and saving jsons in ES on localhost which we installed earlier.




run logstash as a service

sudo apt-get install upstart




vi /etc/init/logstash_server.conf

#!upstart



description "logstash server"



author      "Rasta"



start on starting networking



stop on shutdown



script



    export HOME="/home/ubuntu"



    echo $$ > /var/run/ShakeCardLogstash.pid



    exec sudo -u ubuntu ~/logstash-1.4.1/bin/logstash -f ~/logstash-1.4.1/logstash-simple.conf >> /var/log/logstash_server.log 2>&1



end script



pre-start script



    # Date format same as (new Date()).toISOString() for consistency



    echo "[`date -u +%Y-%m-%dT%T.%3NZ`] (sys) Starting" >> /var/log/LogServer.log



end script

Manually start/stop

[sudo] /sbin/start logstash_server
[sudo] /sbin/stop logstash_server

One More Thing!

In case you wish to contact the server via IPv4, You'll need to define Ubuntu or the JVM to work BY DEFAULT with IPv4 and not IPv6.We decided to go with the Ubuntu method.
Enter VI to edit interfaces

sudo vi /etc/sysctl.conf

Add the following:

# IPv6 disabled
net.ipv6.conf.all.disable_ipv6 = 1
net.ipv6.conf.default.disable_ipv6 = 1
net.ipv6.conf.lo.disable_ipv6 = 1

restart the sysctl

sudo sysctl -p

Theird Step

install nginx as an http server

sudo vi /etc/apt/sources.list

add these two repo

deb http://nginx.org/packages/ubuntu/ lucid nginx

deb-src http://nginx.org/packages/ubuntu/ lucid nginx

run installation script

sudo -s

nginx=stable # use nginx=development for latest development version

add-apt-repository ppa:nginx/$nginx

apt-get update 

apt-get install nginx

add to startup

sudo update-rc.d nginx defaults

fourth step

Install kibana to view our kickass logs!

wget https://download.elasticsearch.org/kibana/kibana/kibana-3.1.0.tar.gz


tar zxvf kibana-3.1.0.tar.gz

run kibana under nginx server

set nginx config to your needs

an example provided here

vi /etc/nginx/nginx.conf

save and...

reload nginx configuration

cd /etc/nginx

nginx -s reload

Fifth step
install nagios

Based on two great youtube tutorials

Video 1: https://www.youtube.com/watch?v=CSDqMYDF8-k

Video 2: https://www.youtube.com/watch?v=WKZiQoqSSR0

apt-get install nagios3
choose an address and a password

configure apache to work alongside with nginx by changing the port

vi /etc/apache2/ports.conf

# If you just change the port or add more ports here, you will likely also

# have to change the VirtualHost statement in

# /etc/apache2/sites-enabled/000-default.conf




Listen 901




<IfModule ssl_module>

Listen 443

</IfModule>




<IfModule mod_gnutls.c>

Listen 443

</IfModule>






# vim: syntax=apache ts=4 sw=4 sts=4 sr noet

restrat apache:

/etc/init.d/apache2 restart

now you can browes nagios at

*YOUR_SERVER'S_IP*:901/nagios3

username: nagiosadmin

password: *YOUR_INSTALLATION_PASSWORD*

more configuration:

sudo vi /etc/group

add www.data to nagios

sudo chmod g+w /var/lib/nagios3

sudo chmod g+w /var/lib/nagios3/rw

/etc/init.d/apache2 restart

Sixth Step

configure nagios

install nrpe to monitor remote servers

Well that's all.
Hope you enjoyed this post as much as i did.