ELK实践（二）：收集Nginx日志 (2)

我们先使用Grok Debugger 工具在线调试下，看看写的grok是否正确。我之前没有测试之前启动，发现ES里没有grok里解析出来的字段，后来在命令行看到filebeat的输出（前台运行）：

$ ./beats/filebeat/filebeat -c beats/filebeat/filebeat.test_nginx2.yml { "@timestamp" => 2018-09-24T09:01:19.555Z, "logIndex" => "nginx", "offset" => 6467, "docType" => "nginx-access", "@version" => "1", "input_type" => "log", "beat" => { "name" => "2106567e5bce", "hostname" => "2106567e5bce", "version" => "5.6.2" }, "host" => "2106567e5bce", "source" => "/work/yphp/nginx/logs/hello71.log", "message" => "172.16.10.1 - - [24/Sep/2018:09:01:14 +0000] \"GET /?time=2244 HTTP/1.1\" 200 98087 \"-\" \"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/63.0.3239.132 Safari/537.36\" \"-\" 0.195", "type" => "log", "tags" => [ [0] "ngx", [1] "yujc", [2] "beats_input_codec_plain_applied", [3] "_grokparsefailure" ] }

最后面提示了_grokparsefailure，说明grok部分写的有问题。由于是参考的网上教程，也加上刚接触，不知道怎么配置，filebeat.conf调试了很久才生效。

我们打开Grok Debugger，第一个输入框输入filebeat采集的消息原文message字段里的内容，第二个输入框输入grok表达式：

点击Go按钮即可解析。如果下面的内容是{}说明解析失败，然后可以修改表达式，该工具会自动解析。最终解析结果：

{ "remote_ip": [ [ "172.16.10.1" ] ], "HOSTNAME": [ [ "172.16.10.1" ] ], "IP": [ [ null ] ], "IPV6": [ [ null ] ], "IPV4": [ [ null ] ], "user_name": [ [ "-" ] ], "time": [ [ "24/Sep/2018:08:47:59 +0000" ] ], "MONTHDAY": [ [ "24" ] ], "MONTH": [ [ "Sep" ] ], "YEAR": [ [ "2018" ] ], "TIME": [ [ "08:47:59" ] ], "HOUR": [ [ "08" ] ], "MINUTE": [ [ "47" ] ], "SECOND": [ [ "59" ] ], "INT": [ [ "+0000" ] ], "method": [ [ "GET" ] ], "url": [ [ "/?time=2244" ] ], "http_version": [ [ "1.1" ] ], "BASE10NUM": [ [ "1.1", "200", "98086", "0.002" ] ], "response_code": [ [ "200" ] ], "body_sent": [ [ "98086" ] ], "referrer": [ [ "-" ] ], "agent": [ [ "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/63.0.3239.132 Safari/537.36" ] ], "x_forwarded_for": [ [ "-" ] ], "request_time": [ [ "0.002" ] ] }

然后可以启动logstash了。

测试logstash配置是否通过：

./logstash/bin/logstash -f logstash/config/conf.d/filebeat.conf --config.test_and_exit Config Validation Result: OK. Exiting Logstash # 启动logstash ./logstash/bin/logstash & # 启动filebeat ./beats/filebeat/filebeat -c beats/filebeat/filebeat.test_nginx2.yml

我们再次访问Nginx应用，然后我们查看一条数据：

$ curl :9200/test-nginx2-log-2018.09.24/_search?q=*&size=1&sort=@timestamp:desc { "took": 14, "timed_out": false, "_shards": { "total": 5, "successful": 5, "skipped": 0, "failed": 0 }, "hits": { "total": 3, "max_score": null, "hits": [ { "_index": "test-nginx2-log-2018.09.24", "_type": "log", "_id": "AWYK0to8JzfnbYlB_DRx", "_score": null, "_source": { "response_code": "200", "agent": "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/63.0.3239.132 Safari/537.36", "logIndex": "nginx", "offset": 6875, "method": "GET", "docType": "nginx-access", "user_name": "-", "input_type": "log", "http_version": "1.1", "source": "/work/yphp/nginx/logs/hello71.log", "message": """172.16.10.1 - - [24/Sep/2018:09:04:40 +0000] "GET /?time=2244 HTTP/1.1" 200 98086 "-" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/63.0.3239.132 Safari/537.36" "-" 0.002""", "type": "log", "url": "/?time=2244", "tags": [ "ngx", "yujc", "beats_input_codec_plain_applied" ], "x_forwarded_for": "-", "referrer": "-", "@timestamp": "2018-09-24T09:04:40.404Z", "remote_ip": "172.16.10.1", "request_time": "0.002", "@version": "1", "beat": { "name": "2106567e5bce", "hostname": "2106567e5bce", "version": "5.6.2" }, "host": "2106567e5bce", "body_sent": "98086", "time": "24/Sep/2018:09:04:40 +0000" }, "sort": [ 1537779880404 ] } ] } }

里面就包含了所有我们解析出来的字段。

kibana里查看

打开kibana web地址：:5601，依次打开：Management
-> Kibana -> Index Patterns ,选择Create Index Pattern：

a. Index pattern 输入：test-nginx2-* ；

b. Time Filter field name 选择 @timestamp。

c. 点击Create。

然后打开Discover，选择 filebeat-* 就能看到日志数据了。