ELK实践（一）：基础入门 (3)

由于现在我们还没有安装filebeat，也不打算直接使用logstash收集日志，所以先简单测试下标准输入输出，只要正常就行了：

$ ./logstash/bin/logstash -e 'input { stdin { } } output { stdout {}}'

稍等几秒钟：

Sending Logstash's logs to /usr/local/elk/logstash/logs which is now configured via log4j2.properties [2018-09-24T23:07:35,424][INFO ][logstash.modules.scaffold] Initializing module {:module_name=>"netflow", :directory=>"/usr/local/elk/logstash/modules/netflow/configuration"} [2018-09-24T23:07:35,434][INFO ][logstash.modules.scaffold] Initializing module {:module_name=>"fb_apache", :directory=>"/usr/local/elk/logstash/modules/fb_apache/configuration"} [2018-09-24T23:07:35,657][INFO ][logstash.pipeline ] Starting pipeline {"id"=>"main", "pipeline.workers"=>4, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>5, "pipeline.max_inflight"=>500} [2018-09-24T23:07:35,683][INFO ][logstash.pipeline ] Pipeline main started The stdin plugin is now waiting for input: [2018-09-24T23:07:35,776][INFO ][logstash.agent ] Successfully started Logstash API endpoint {:port=>9600} hello 2018-09-24T15:07:42.760Z 2106567e5bce hello welcome 2018-09-24T15:07:49.501Z 2106567e5bce welcome

我们输入了hello、welcome，终端实时的输出了内容。后面的内容里，我们将会修改 stdin 为为beats，修改 stdout 为ElasticSearch。

如果新增配置，需要后台常驻运行，可以使用下列命令：

/usr/local/elk/logstash/bin/logstash &

查看是否已运行(需要过10s左右，可以多刷几次)：

$ netstat -tulnp | grep 5044 tcp 0 0 0.0.0.0:5044 0.0.0.0:* LISTEN 1040/java 安装 Filebeat

Filebeat不依赖JDK。

二进制安装 wget https://artifacts.elastic.co/downloads/beats/filebeat/filebeat-5.6.2-linux-x86_64.tar.gz

filebeat 8.4M。解压、设置 ：

mkdir /usr/local/elk/beats tar zxvf filebeat-5.6.2-linux-x86_64.tar.gz -C /usr/local/elk/beats/ mv /usr/local/elk/beats/filebeat-5.6.2-linux-x86_64/ /usr/local/elk/beats/filebeat chown -R elk:elk /usr/local/elk/beats

注意：后续对目录/usr/local/elk/beats的增删改操作建议使用上面创建的elk用户，否则还要使用chown命令修改权限。

# 切换用户 su - elk filebeat采集内容发送到ElasticSearch

为了让测试简单，我们手动模拟日志的生成：

echo "test - test2" >> /tmp/test1.log echo "test - test2" >> /tmp/test2.log

生成了2个日志文件test1.log、test2.log，各有一行日志。

新建一个filebeat配置文件：

cd /usr/local/elk vim beats/filebeat/filebeat.test.yml

配置内容：

filebeat.prospectors: - type: log paths: - /tmp/test1.log tags: ["test1"] document_type: test1 - type: log paths: - /tmp/test2.log tags: ["test2"] document_type: test2 output.elasticsearch: hosts: ["127.0.0.1:9200"] index: "test-filebeat"

配置说明：

filebeat.prospectors:

type 日志类型，默认log

input_type 输入类型，默认log

paths 采集的日志，可以使用通配符。支持多个

tags 自定义标签，是个数组。自定义

document_type 自定义字段，用于Logsatsh区分来源，在Logsatsh里用变量type表示

一个-表示一个filebeat.prospector，这里设置了2个。日志发送到elasticsearch，索引index 是test-filebeat。

我们运行 filebeat：

# 修改权限 chmod 600 beats/filebeat/filebeat.test.yml # 指定配置文件前台运行 ./beats/filebeat/filebeat -c beats/filebeat/filebeat.test.yml

我们新开终端查看ES里是否新增了内容：

curl :9200/test-filebeat/_search?q=* { "took": 0, "timed_out": false, "_shards": { "total": 5, "successful": 5, "skipped": 0, "failed": 0 }, "hits": { "total": 2, "max_score": 1, "hits": [ { "_index": "test-filebeat", "_type": "test2", "_id": "AWYL9s4WJzfnbYlB_DSO", "_score": 1, "_source": { "@timestamp": "2018-09-24T14:23:30.652Z", "beat": { "hostname": "2106567e5bce", "name": "2106567e5bce", "version": "5.6.2" }, "input_type": "log", "message": "test - test2", "offset": 13, "source": "/tmp/test2.log", "tags": [ "test2" ], "type": "test2" } }, { "_index": "test-filebeat", "_type": "test1", "_id": "AWYL9s4WJzfnbYlB_DSP", "_score": 1, "_source": { "@timestamp": "2018-09-24T14:23:30.652Z", "beat": { "hostname": "2106567e5bce", "name": "2106567e5bce", "version": "5.6.2" }, "input_type": "log", "message": "test - test2", "offset": 13, "source": "/tmp/test1.log", "tags": [ "test1" ], "type": "test1" } } ] } }

新开命令行追加一行日志：

echo "new msg" >> /tmp/test1.log curl :9200/test-filebeat/_search?q=*&size=1&sort=@timestamp:desc { "took": 0, "timed_out": false, "_shards": { "total": 5, "successful": 5, "skipped": 0, "failed": 0 }, "hits": { "total": 3, "max_score": null, "hits": [ { "_index": "test-filebeat", "_type": "test1", "_id": "AWYL-BjvJzfnbYlB_DSQ", "_score": null, "_source": { "@timestamp": "2018-09-24T14:24:55.654Z", "beat": { "hostname": "2106567e5bce", "name": "2106567e5bce", "version": "5.6.2" }, "input_type": "log", "message": "new msg", "offset": 21, "source": "/tmp/test1.log", "tags": [ "test1" ], "type": "test1" }, "sort": [ 1537799095654 ] } ] } }