filebeat+elk日志收集平台搭建流程
1、 整体简介：
模式：单机
平台：Linux - centos - 7
ELK：elasticsearch、logstash、kibana三款开源软件的集合。
FILEBEAT：代替logstash的采集功能，轻量、耗用小。
目前收集的有nginx日志、java日志[单行|多行]。
都是通过在客户端的生成日志配置文件中定义好初步json格式，然后利用filebeat采集到logstash，存储到elasticsearch，最后通过kibana在浏览器页面中展示出来。
elasticsearch、logstash、kibana 如果是root安装，默认会创建（elasticsearch、logstash、kibana）三个单独用户来独立运行。用www用户安装好后默认是以www用户运行。本文演示用的是www。
2、 软件环境安装：
（1）、java环境：
Elasticsearch需要安装Java 8的环境。
如果没有下载jdk1.8的软件包，可以直接安装java-1.8.0-openjdk。
（2）、下载elasticsearch（Linux版本）的二进制包并解压：
$ wget https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-6.4.0.tar.gz
$ unzip elasticsearch-6.4.0.tar.gz
$ mv elasticsearch-6.4.0 elasticsearch
$ vim elasticsearch/config/elasticsearch.yml
path.data: /data/soft/elasticsearch/data/
path.logs: /data/soft/elasticsearch/logs/
$ sudo sysctl -w vm.max_map_count=262144
$ vim /etc/sysctl.conf文件，添加：
vm.max_map_count=262144
$ sysctl -p
$ vim /etc/security/limits.conf文件，末尾添加:
* soft nproc 20536
* hard nproc 20536
* soft nofile 65536
* hard nofile 65536
退出重新登录生效。
将elasticsearch、logstash、kibana、filebeat的bin目录加入到环境变量PATH：
/data/soft/logstash/bin:/data/soft/kibana/bin:/data
/soft/elasticsearch/bin:/data/soft/filebeat
$ nohup elasticsearch -d >/data/soft/elasticsearch/nohup.out
2>&1 &
启动后，Elastic默认在9200端口运行。
$ curl -X GET localhost:9200或curl localhost:9200
{
"name" : "zny0iRv",
"cluster_name" : "elasticsearch",
"cluster_uuid" : "AErImFrFQOaoFPzNSdVmfA",
"version" : {
"number" : "6.4.0",
"build_flavor" : "default",
"build_type" : "tar",
"build_hash" : "595516e",
"build_date" : "2018-08-17T23:18:47.308994Z",
"build_snapshot" : false,
"lucene_version" : "7.4.0",
"minimum_wire_compatibility_version" : "5.6.0",
"minimum_index_compatibility_version" : "5.0.0"
},
"tagline" : "You Know, for Search"
}
（3）、下载logstash（Linux版本）的二进制包并解压：
$ wget https://artifacts.elastic.co/downloads/logstash/logstash-6.4.0.zip
$ unzip logstash-6.4.0.zip logstash-6.4.0
$ mv logstash-6.4.0 logstash
$ cd logstash/config/
$ cat config/logstash.yml|grep -vE '^$|#'
# pipeline 线程数，可优化为 ---> pipeline.workers: CPU内核数（或几倍cpu内核数）
pipeline.workers: 32
# 实际output 时的线程数，可优化为 ---> pipeline.output.workers: 不超过pipeline 线程数
pipeline.output.workers: 32
# 每次发送的事件数
pipeline.batch.size: 8000
# 发送延时
pipeline.batch.delay: 15
# filter设置multiline后，pipline worker会自动将为1，如果使用filebeat，建议在beat中就使用multiline，如果使用logstash作为shipper，建议在input 中设置multiline，不要在filter中设置multiline。
$ vim logstash.conf
input {
beats {
codec => json
port => 5044
#host => “0.0.0.0”
}
}
filter {
mutate {
remove_field => ["@version","[beat][name]","[beat][version]","[beat][hostname]","tags"]
#remove_field => "message"
}
if [nx_upstream_host] != "-" {
mutate {
convert => {"nx_upstream_response_time"=>"float"}
convert => {"nx_upstream_response_length"=>"integer"}
convert => {"nx_upstream_connect_time"=>"float"}
}
} else {
mutate {
remove_field => ["nx_upstream_host","nx_upstream_response_time","nx_upstream_response_length","nx_upstream_status","nx_upstream_connect_time"]
}
}
}
output {
# 8bet-test-srv-4:nginx
if [project] == "8bet-admin" {
elasticsearch {
hosts => ["http://localhost:9200"]
index => "8bet-nginx-admin-%{+YYYY.MM.dd}"
}
} else if [project] == "8bet-h5" {
elasticsearch {
hosts => ["http://localhost:9200"]
index => "8bet-nginx-h5-%{+YYYY.MM.dd}"
}
} else if [project] == "8bet-newadmin" {
elasticsearch {
hosts => ["http://localhost:9200"]
index => "8bet-nginx-newadmin-%{+YYYY.MM.dd}"
}
} else if [project] == "8bet-newh5" {
elasticsearch {
hosts => ["http://localhost:9200"]
index => "8bet-nginx-newh5-%{+YYYY.MM.dd}"
}
} else if [project] == "8bet-nginx-error" {
elasticsearch {
hosts => ["http://localhost:9200"]
index => "8bet-nginx-error-%{+YYYY.MM.dd}"
}
# 8bet-test-srv-5:nginx
} else if [project] == "pay-nginx-admin" {
elasticsearch {
hosts => ["http://localhost:9200"]
index => "pay-nginx-admin-%{+YYYY.MM.dd}"
}
} else if [project] == "pay-nginx-user" {
elasticsearch {
hosts => ["http://localhost:9200"]
index => "pay-nginx-user-%{+YYYY.MM.dd}"
}
} else if [project] == "pay-nginx-api" {
elasticsearch {
hosts => ["http://localhost:9200"]
index => "pay-nginx-api-%{+YYYY.MM.dd}"
}
} else if [project] == "pay-nginx-error" {
elasticsearch {
hosts => ["http://localhost:9200"]
index => "pay-nginx-error-%{+YYYY.MM.dd}"
}
# 8bet-test-srv-4:java
} else if [source] == "/log/billing/info.log" {
elasticsearch {
hosts => ["http://localhost:9200"]
#manage_template => true
#template_overwrite => true
index => "8bet-java-billing-info-%{+YYYY.MM.dd}"
}
} else if [source] == "/log/billing/error.log" {
elasticsearch {
hosts => ["http://localhost:9200"]
#manage_template => true
#template_overwrite => true
index => "8bet-java-billing-error-%{+YYYY.MM.dd}"
}
} else if [source] == "/log/member/info.log" {
elasticsearch {
hosts => ["http://localhost:9200"]
#manage_template => true
#template_overwrite => true
index => "8bet-java-member-info-%{+YYYY.MM.dd}"
}
} else if [source] == "/log/member/error.log" {
elasticsearch {
hosts => ["http://localhost:9200"]
#manage_template => true
#template_overwrite => true
index => "8bet-java-member-error-%{+YYYY.MM.dd}"
}
} else if [source] == "/log/admin/info.log" {
elasticsearch {
hosts => ["http://localhost:9200"]
#manage_template => true
#template_overwrite => true
index => "8bet-java-admin-info-%{+YYYY.MM.dd}"
}
} else if [source] == "/log/admin/error.log" {
elasticsearch {
hosts => ["http://localhost:9200"]
#manage_template => true
#template_overwrite => true
index => "8bet-java-admin-error-%{+YYYY.MM.dd}"
}
} else if [source] == "/log/pay/info.log" {
elasticsearch {
hosts => ["http://localhost:9200"]
#manage_template => true
#template_overwrite => true
index => "8bet-java-pay-info-%{+YYYY.MM.dd}"
}
} else if [source] == "/log/pay/error.log" {
elasticsearch {
hosts => ["http://localhost:9200"]
#manage_template => true
#template_overwrite => true
index => "8bet-java-pay-error-%{+YYYY.MM.dd}"
}
} else if [source] == "/log/discount/info.log" {
elasticsearch {
hosts => ["http://localhost:9200"]
#manage_template => true
#template_overwrite => true
index => "8bet-java-discount-info-%{+YYYY.MM.dd}"
}
} else if [source] == "/log/discount/error.log" {
elasticsearch {
hosts => ["http://localhost:9200"]
#manage_template => true
#template_overwrite => true
index => "8bet-java-discount-error-%{+YYYY.MM.dd}"
}
} else if [source] == "/log/schedule/info.log" {
elasticsearch {
hosts => ["http://localhost:9200"]
#manage_template => true
#template_overwrite => true
index => "8bet-java-schedule-info-%{+YYYY.MM.dd}"
}
} else if [source] == "/log/schedule/error.log" {
elasticsearch {
hosts => ["http://localhost:9200"]
#manage_template => true
#template_overwrite => true
index => "8bet-java-schedule-error-%{+YYYY.MM.dd}"
}
} else if [source] == "/log/security/info.log" {
elasticsearch {
hosts => ["http://localhost:9200"]
#manage_template => true
#template_overwrite => true
index => "8bet-java-security-info-%{+YYYY.MM.dd}"
}
} else if [source] == "/log/security/error.log" {
elasticsearch {
hosts => ["http://localhost:9200"]
#manage_template => true
#template_overwrite => true
index => "8bet-java-security-error-%{+YYYY.MM.dd}"
}
} else if [source] == "/log/caipiao/info.log" {
elasticsearch {
hosts => ["http://localhost:9200"]
#manage_template => true
#template_overwrite => true
index => "8bet-java-caipiao-info-%{+YYYY.MM.dd}"
}
} else if [source] == "/log/caipiao/error.log" {
elasticsearch {
hosts => ["http://localhost:9200"]
#manage_template => true
#template_overwrite => true
index => "8bet-java-caipiao-error-%{+YYYY.MM.dd}"
}
} else if [source] == "/log/cpbilling/info.log" {
elasticsearch {
hosts => ["http://localhost:9200"]
#manage_template => true
#template_overwrite => true
index => "8bet-java-cpbilling-info-%{+YYYY.MM.dd}"
}
} else if [source] == "/log/cpbilling/error.log" {
elasticsearch {
hosts => ["http://localhost:9200"]
#manage_template => true
#template_overwrite => true
index => "8bet-java-cpbilling-error-%{+YYYY.MM.dd}"
}
} else if [source] == "/log/cpmessage/info.log" {
elasticsearch {
hosts => ["http://localhost:9200"]
#manage_template => true
#template_overwrite => true
index => "8bet-java-cpmessage-info-%{+YYYY.MM.dd}"
}
} else if [source] == "/log/cpmessage/error.log" {
elasticsearch {
hosts => ["http://localhost:9200"]
#manage_template => true
#template_overwrite => true
index => "8bet-java-cpmessage-error-%{+YYYY.MM.dd}"
}
} else if [source] == "/log/cpschedule/info.log" {
elasticsearch {
hosts => ["http://localhost:9200"]
#manage_template => true
#template_overwrite => true
index => "8bet-java-cpschedule-info-%{+YYYY.MM.dd}"
}
} else if [source] == "/log/cpschedule/error.log" {
elasticsearch {
hosts => ["http://localhost:9200"]
#manage_template => true
#template_overwrite => true
index => "8bet-java-cpschedule-error-%{+YYYY.MM.dd}"
}
}
stdout {
codec => rubydebug
}
}
#output {
# stdout { codec => rubydebug }
# elasticsearch {
# hosts => ["http://localhost:9200"]
# index => "%{[@metadata][beat]}-%{[@metadata][version]}-%{+YYYY.MM}"
# }
#}
检查logstash配置文件的语法命令：
$ logstash --path.settings /data/soft/logstash/config/ -f /data/soft/logstash/config/logstash.conf -t
$ nohup logstash -f /data/soft/logstash/config/logstash.conf >/data/soft/logstash/out.log 2>&1 &
（4）、下载kibana（Linux版本）的二进制包并解压：
$ wget https://artifacts.elastic.co/downloads/kibana/kibana-6.4.0-linux-x86_64.tar.gz
$ tar -zxf kibana-6.4.0-linux-x86_64.tar.gz
$ mv kibana-6.4.0-linux-x86_64 kibana
$ vim kibana/config/kibana.yml
elasticsearch.url: "http://localhost:9200"
elasticsearch.shardTimeout: 0
pid.file: /data/soft/kibana/kibana.pid
$ nohup /data/soft/kibana/bin/kibana >/data/soft/kibana/out.log 2>&1 &
3、 FILEBEAT（收集端）单独配置：
以一台filebeat采集端做示例：
（1）、nginx日志格式配置：
$ vim /data/soft/nginx/conf/nginx.conf
user www;
worker_processes 4;
worker_rlimit_nofile 20000;
worker_cpu_affinity 00000001 00000010 00000100 00001000;
error_log /log/nginx/error.log error;
pid /log/nginx/nginx.pid;
events {
use epoll;
worker_connections 20000;
}
http {
include mime.types;
default_type application/octet-stream;
sendfile on;
tcp_nodelay on;
tcp_nopush on;
client_body_timeout 10;
client_header_timeout 10;
send_timeout 10;
keepalive_timeout 50;
client_body_buffer_size 4k;
client_header_buffer_size 1k;
client_max_body_size 10m;
large_client_header_buffers 2 1k;
open_file_cache max=65535 inactive=20s;
open_file_cache_valid 30s;
open_file_cache_min_uses 1;
gzip on;
gzip_min_length 1k;
gzip_buffers 8 16k;
gzip_comp_level 3;
gzip_http_version 1.1;
gzip_disable "MSIE [1-6].";
gzip_types text/plain application/x-javascript text/css application/xml application/x-httpd-php image/jpeg image/gif image/png;
gzip_vary on;