Search This Blog

Saturday, May 31, 2014

How To: Run a clustered NodeJs app under secured connection

Let me take a few moments to explain the basics and then we'll start:

1. A clustered nodejs app allows you to create several processes (equals to the number of cores you have on your server) that run the same application on the same port.
2. A secured nodejs app allows you to receive and return data on a secure line.

This tutorial should be very short and very practical. Let's start..

1. Requirements

var express = require("express");
var cluster = require('cluster');
var fs = require('fs');
var https = require('https');

2. Initialisations

var sslPort = 443;
var sslOptions = {pfx: fs.readFileSync('personal.pfx'),passphrase: '123456'};


The sslPort variable is kind of fixed, but you know.. I'm trying to give you as much freedom as i can
The ssOptions varible is a JSON object that decalres a .pfx or .pem,
in case of a .pfx file usage like in our example, we need to declare a 'passphrase' property with the private password of the .pfx file.

3. App definition

var app = express();
var numCPUs = require('os').cpus().length;
console.log("Num of cpus up: " + numCPUs);

Now we'll get the number of cpus working for our app.

4. Cluster definition

if (cluster.isMaster) {
// Fork workers.
for (var i = 0; i < numCPUs; i++) {
cluster.fork();
console.log(i + " - fork.");
}

  cluster.on('exit', function(worker, code, signal) {
console.log('worker ' + worker.process.pid + ' died');
});
} else {

// your routes goes here.
app.get(.........


If the instance is a master, it will let other instances to fork it, if not - well.. start the server :)

Note that we didn't close the 'else' section.

5. The tricky part - configure the cluster to listen to a secure port from all the instances

After we finished our routings, we need to let the server to start listening.
For this matter, we will override the server 'listen' method and create our own:

app.listen = function(){
var server = https.createServer(sslOptions, this);
return server.listen.apply(server, arguments);
};


The sslOptions varible is the same one we defined in the "Initialisations" section (2).

6. And last - Listen!

app.listen(sslPort, function(){
logger.module('cluster').debug("Express server listening on port: " + sslPort);
});
} // else cluster


Note that we closed the 'else' section from the Cluster definition part (2).

Sunday, May 25, 2014

Build a badass monitoring server

In many cases, we meet a good system with minor monitoring problems.
Sometimes it's the infrastructure layer, something it's the application layer.. even connectivity.
So what do we need to do? m-o-n-i-t-o-r-!
In this post I will try an cover some tools that can help you do just that!
But first, I would like to thank a good friend and a linux wizard, Mr. Tomer Kom.

First Step
Let's start by installing ES application to store our data:
curl -O https://download.elasticsearch.org/elasticsearch/elasticsearch/elasticsearch-1.0.1.tar.gz
tar zxvf elasticsearch-1.0.1.tar.gz
cd elasticsearch-1.0.1/
./bin/elasticsearch


Ok, cool! we've got ourselves a new DB to store everything.
Now we need it to run as a demon
remove existing startup script
sudo update-rc.d -f elasticsearch remove 
and create a new one and configure it to your server specs
vi /etc/init.d/elasticsearch


#! /bin/sh


### BEGIN INIT INFO


# Provides:          elasticsearch


# Required-Start:    $all


# Required-Stop:    $all


# Default-Start:    2 3 4 5


# Default-Stop:      0 1 6


# Short-Description: Starts elasticsearch


# Description:      Starts elasticsearch using start-stop-daemon


### END INIT INFO


ES_HOME=/opt/elasticsearch


ES_MIN_MEM=256m


ES_MAX_MEM=1g


DAEMON=$ES_HOME/bin/elasticsearch


NAME=elasticsearch


DESC=elasticsearch


PID_FILE=/var/run/$NAME.pid


LOG_DIR=$ES_HOME/logs/


DATA_DIR=$ES_HOME/data/


WORK_DIR=/tmp/$NAME


CONFIG_FILE=$ES_HOME/config/elasticsearch.yml


DAEMON_OPTS="-d -p $PID_FILE -Des.config=$CONFIG_FILE -Des.path.home=$ES_HOME -Des.path.logs=$LOG_DIR -Des.path.data=$DATA_DIR -Des.path.work=$WORK_DIR"


test -x $DAEMON || exit 0

set -e


case "$1" in


start)


    echo -n "Starting $DESC: "


    mkdir -p $LOG_DIR $DATA_DIR $WORK_DIR


    if start-stop-daemon --start --pidfile $PID_FILE --startas $DAEMON -- $DAEMON_OPTS


    then


        echo "started."


    else


        echo "failed."


    fi


    ;;


  stop)


    echo -n "Stopping $DESC: "


    if start-stop-daemon --stop --pidfile $PID_FILE


    then


        echo "stopped."


    else


        echo "failed."


    fi


    ;;


  restart|force-reload)


    ${0} stop


    sleep 0.5


    ${0} start


;;


  *)


    N=/etc/init.d/$NAME


    echo "Usage: $N {start|stop|restart|force-reload}" >&2


    exit 1


    ;;


esac


exit 0

AND
add to startup list:
update-rc.d elasticsearch defaults
this is to prevent the instance to accidentally join a cluster, without being configured appropriately. 
you can use the following commands to ensure, that elasticsearch starts when the system is booted and then start up elasticsearch:
sudo update-rc.d elasticsearch defaults 95 10
sudo
/etc/init.d/elasticsearch start
keep in ming that you may need to add permissions to this file
chmod u+x /etc/init.d/elasticsearch

Second Step
Install logstash to collect and manage your logs for you.
donaload and extract it:

curl -O https://download.elasticsearch.org/logstash/logstash/logstash-1.4.1.tar.gz
tar zxvf logstash-1.4.1.tar.gz
cd logstash-1.4.1

now create a configuration file that matches our production site:
vi logstash-simple.conf


    input {




     udp { format => "json" port => 9999 type => "sample" }




    }




    output {




     elasticsearch { host => localhost }




  stdout { codec => rubydebug }




}



so now we are getting data from UPD in port 9999 and saving jsons in ES on localhost which we installed earlier.

run logstash as a service
sudo apt-get install upstart



vi /etc/init/logstash_server.conf









#!upstart




description "logstash server"




author      "Rasta"




start on starting networking




stop on shutdown




script




    export HOME="/home/ubuntu"




    echo $$ > /var/run/ShakeCardLogstash.pid




    exec sudo -u ubuntu ~/logstash-1.4.1/bin/logstash -f ~/logstash-1.4.1/logstash-simple.conf >> /var/log/logstash_server.log 2>&1




end script




pre-start script




    # Date format same as (new Date()).toISOString() for consistency




    echo "[`date -u +%Y-%m-%dT%T.%3NZ`] (sys) Starting" >> /var/log/LogServer.log




end script


Manually start/stop

[sudo] /sbin/start logstash_server
[sudo] /sbin/stop
logstash_server

One More Thing!

In case you wish to contact the server via IPv4, You'll need to define Ubuntu or the JVM to work BY DEFAULT with IPv4 and not IPv6.We decided to go with the Ubuntu method.
Enter VI to edit interfaces
sudo vi /etc/sysctl.conf
Add the following:
# IPv6 disabled
net.ipv6.conf.all.disable_ipv6 = 1
net.ipv6.conf.default.disable_ipv6 = 1
net.ipv6.conf.lo.disable_ipv6 = 1
restart the sysctl
sudo sysctl -p
Theird Step
install nginx as an http server
sudo vi /etc/apt/sources.list
add these two repo

deb http://nginx.org/packages/ubuntu/ lucid nginx


deb-src http://nginx.org/packages/ubuntu/ lucid nginx


run installation script




sudo -s


nginx=stable # use nginx=development for latest development version


add-apt-repository ppa:nginx/$nginx


apt-get update 


apt-get install nginx



add to startup





sudo update-rc.d nginx defaults







fourth step
Install kibana to view our kickass logs!





wget https://download.elasticsearch.org/kibana/kibana/kibana-3.1.0.tar.gz



tar zxvf kibana-3.1.0.tar.gz









run kibana under nginx server

set nginx config to your needs
an example provided here

vi /etc/nginx/nginx.conf
save and...
reload nginx configuration












cd /etc/nginx


nginx -s reload









Fifth step
install nagios


Based on two great youtube tutorials


Video 1: https://www.youtube.com/watch?v=CSDqMYDF8-k
Video 2: https://www.youtube.com/watch?v=WKZiQoqSSR0

apt-get install nagios3
choose an address and a password

configure apache to work alongside with nginx by changing the port


vi /etc/apache2/ports.conf









# If you just change the port or add more ports here, you will likely also


# have to change the VirtualHost statement in


# /etc/apache2/sites-enabled/000-default.conf





Listen 901





<IfModule ssl_module>


Listen 443


</IfModule>





<IfModule mod_gnutls.c>


Listen 443


</IfModule>







# vim: syntax=apache ts=4 sw=4 sts=4 sr noet









restrat apache:
/etc/init.d/apache2 restart

now you can browes nagios at
*YOUR_SERVER'S_IP*:901/nagios3
username: nagiosadmin
password: *YOUR_INSTALLATION_PASSWORD*

more configuration:
sudo vi /etc/group
add www.data to nagios
sudo chmod g+w /var/lib/nagios3
sudo chmod g+w /var/lib/nagios3/rw
/etc/init.d/apache2 restart

Sixth Step
configure nagios
install nrpe to monitor remote servers

Well that's all.
Hope you enjoyed this post as much as i did.