ELK Stack (Elastic, Logstash and Kibana)

ELK Stack (Elastic, Logstash and Kibana)

Complete stack is open-source, see more.

  • Logstash
    Server to process incoming logs

  • Elastic
    Data storage for all of the logs (ex. Elasticsearch)

  • Kibana
    Web interface for searching and visualizing logs

  • Filebeat
    Filebeat is file harvester, mostly used to fetch logs files and feed them into logstash. A tool to collect logs locally and prepare it for processing elsewhere. Installed on servers to send their logs to Logstash, as a log forwarding agent. Ex. names: Logstash Forwarder, before that: Lumberjack.

How To Use the DigitalOcean ELK Stack ELK : Logstash, ElasticSearch, and Kibana

cd ~upravitelj
git clone https://github.com/cogniteev/oncrawl-elk.git
cd oncrawl*

# port: hhvm is using 9000, so I switched to 9002
sed -i -r 's/9000:80/9002:80/' docker-compose.yml

# start contaniners
docker-compose -p oncrawlelk -f docker-compose.yml up -d

# Ili samo:
# docker-compose up -d
# docker-compose down

# copy logs for save-up.ch
cp /var/www/save-up.ch/log/* ~upravitelj/oncrawl-elk/logs/nginx


# Redirect to Crawl container
#
location / {
    proxy_pass https://127.0.0.1:9002;
    proxy_set_header Host $host;
    proxy_set_header X-Real-IP $remote_addr;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_set_header X-Forwarded-Host $server_name;
}

   location ~ /\. {
        allow all;
    }

http://logs.etaktiker.com/app/kibana#/visualize/edit/Unique-crawled-pages?embed&_g=(time:(from:'2016-03-21T00:00:00.000Z',mode:quick,to:'2016-04-20T23:59:59.999Z'))

Security?

Securing Elasticsearch using Nginx as a Proxy

Testing logz.io

/var/www/save-up.ch/log/access.log /var/www/save-up.ch/log/error.log

  1. Installed Filebeat as per: Repositories for APT and YUM
  2. Configured Filebeat as per here
  3. Created configuration: /etc/filebeat/filebeat.yml

Shipping Logs to Logz.io with Filebeat - Logz.io

Kibana Recipes: oncrawl-elk/recipes/kibana-config at master · cogniteev/oncrawl-elk · GitHub SEO-Dashboard/kibana-dataseo.json at master · voltek62/SEO-Dashboard · GitHub

CDN Log Forwarding

KeyCDN Log Forwarding

date 01. Jan 0001 | modified 28. May 2021
filename: Task - Log File Management, Install ELK Stack