– Notes

ELK Stack (Elastic, Logstash and Kibana)

Complete stack is open-source, see more.

Logstash
Server to process incoming logs
Elastic
Data storage for all of the logs (ex. Elasticsearch)
Kibana
Web interface for searching and visualizing logs
Filebeat
Filebeat is file harvester, mostly used to fetch logs files and feed them into logstash. A tool to collect logs locally and prepare it for processing elsewhere. Installed on servers to send their logs to Logstash, as a log forwarding agent. Ex. names: Logstash Forwarder, before that: Lumberjack.

How To Use the DigitalOcean ELK Stack ELK : Logstash, ElasticSearch, and Kibana

cd ~upravitelj
git clone https://github.com/cogniteev/oncrawl-elk.git
cd oncrawl*

# port: hhvm is using 9000, so I switched to 9002
sed -i -r 's/9000:80/9002:80/' docker-compose.yml

# start contaniners
docker-compose -p oncrawlelk -f docker-compose.yml up -d

# Ili samo:
# docker-compose up -d
# docker-compose down

# copy logs for save-up.ch
cp /var/www/save-up.ch/log/* ~upravitelj/oncrawl-elk/logs/nginx


# Redirect to Crawl container
#
location / {
    proxy_pass https://127.0.0.1:9002;
    proxy_set_header Host $host;
    proxy_set_header X-Real-IP $remote_addr;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_set_header X-Forwarded-Host $server_name;
}

   location ~ /\. {
        allow all;
    }

http://logs.etaktiker.com/app/kibana#/visualize/edit/Unique-crawled-pages?embed&_g=(time:(from:'2016-03-21T00:00:00.000Z',mode:quick,to:'2016-04-20T23:59:59.999Z'))