安装ELK Stack海量日志分析系统(2)

需要注意的是，此处有三个插件。
input插件指的是logstash从哪里读数据；
filter插件指的是怎么对文本进行过滤；
output插件指的是需要把结果输出到哪里
这三个插件的意思是： logstash从文件中读取数据（input）， 经过内置的COMBINEDAPACHELOG规则匹配之后（filter）， 把结果输出到屏幕（output）

输出结果： [root@bc ~]# logstash -f apachelog.conf { "message" => "192.168.31.242 - - [03/Mar/2017:14:00:41 +0800] \"GET /noindex/css/fonts/Bold/OpenSans-Bold.ttf HTTP/1.1\" 404 238 \"http://192.168.31.201/noindex/css/open-sans.css\" \"Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/56.0.2924.87 Safari/537.36\"", "@version" => "1", "@timestamp" => "2017-03-03T06:00:52.934Z", "path" => "/var/log/httpd/access_log", "host" => "bc.com", "type" => "apachelog", "clientip" => "192.168.31.242", "ident" => "-", "auth" => "-", "timestamp" => "03/Mar/2017:14:00:41 +0800", "verb" => "GET", "request" => "/noindex/css/fonts/Bold/OpenSans-Bold.ttf", "httpversion" => "1.1", "response" => "404", "bytes" => "238", "referrer" => "\"http://192.168.31.201/noindex/css/open-sans.css\"", "agent" => "\"Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/56.0.2924.87 Safari/537.36\"" } 延伸：我们查看COMBINEDAPACHELOG是否可以匹配nginx日志的规则 使用combinedlog匹配nginx规则 { "message" => "192.168.31.242 - - [03/Mar/2017:14:11:01 +0800] \"GET / HTTP/1.1\" 304 0 \"-\" \"Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/56.0.2924.87 Safari/537.36\" \"-\"", "@version" => "1", "@timestamp" => "2017-03-03T06:11:49.150Z", "path" => "/var/log/nginx/access.log", "host" => "bc.com", "type" => "nginx", "clientip" => "192.168.31.242", "ident" => "-", "auth" => "-", "timestamp" => "03/Mar/2017:14:11:01 +0800", "verb" => "GET", "request" => "/", "httpversion" => "1.1", "response" => "304", "bytes" => "0", "referrer" => "\"-\"", "agent" => "\"Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/56.0.2924.87 Safari/537.36\"" }

我们可以看出来，用COMBINEDAPACHELOG也可以匹配nginx日志，但是会漏掉一些东西信息。

从web服务器收集日志，并使用redis作为消息队列 1.node6配置logstash的运行文件 [root@node6 ~]# vim /etc/logstash/conf.d/nginx-out.conf input { file { path => ["/var/log/nginx/access.log"] type => "nginxlog" start_position => "beginning" } } filter { grok { match => { "message" => "%{COMBINEDAPACHELOG} %{QS:x_forwarded_for}" } } } output{ redis { port => "6379" host => ["192.168.31.205"] data_type => "list" key => "logstash-%{type}" }

这里的意思是，从nginx日志读入，使用规则匹配，并输出到redis服务器

2.node5作为消息队列，安装redis [root@node5 ~]# redis-cli 127.0.0.1:6379> LLEN logstash-nginxlog (integer) 19 127.0.0.1:6379> LLEN logstash-nginxlog (integer) 27 127.0.0.1:6379> LINDEX logstash-nginxlog 1 "{\"message\":\"192.168.31.242 - - [03/Mar/2017:20:39:47 +0800] \\\"GET /nginx-logo.png HTTP/1.1\\\" 200 368 \\\"http://192.168.31.203/\\\" \\\"Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/56.0.2924.87 Safari/537.36\\\" \\\"-\\\"\",\"@version\":\"1\",\"@timestamp\":\"2017-03-03T12:41:38.315Z\",\"path\":\"/var/log/nginx/access.log\",\"host\":\"node3.bc.com\",\"type\":\"nginxlog\",\"clientip\":\"192.168.31.242\",\"ident\":\"-\",\"auth\":\"-\",\"timestamp\":\"03/Mar/2017:20:39:47 +0800\",\"verb\":\"GET\",\"request\":\"/nginx-logo.png\",\"httpversion\":\"1.1\",\"response\":\"200\",\"bytes\":\"368\",\"referrer\":\"\\\"http://192.168.31.203/\\\"\",\"agent\":\"\\\"Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/56.0.2924.87 Safari/537.36\\\"\",\"x_forwarded_for\":\"\\\"-\\\"\"}" 3.node4作为logstash Server从redis读取数据 [root@node4 ~]# vim /etc/logstash/conf.d/redis-in.conf input { redis { batch_count => 1 data_type => "list" key => "logstash-nginxlog" host => "192.168.31.205" port => 6379 threads => 5 } } output{ elasticsearch{ hosts => ["192.168.31.201", "192.168.31.202", "192.168.31.203"] } } 3.启动服务 #启动方式也可以使用nohup logstash -f nginxout.conf &来启动 #也可以通过启动脚本来启动。 #但使用启动脚本容易因为权限问题，而导致logstash无法正常运行。 #修改启动脚本的启动用户为root [root@node4 ~]# vim /etc/init.d/logstash LS_USER=root LS_GROUP=root #启动logstash [root@node4 ~]# /etc/init.d/logstash start