ELKstack是Elasticsearch、Logstash、Kibana三个开源软件的组合。目前都在Elastic.co公司名下。
ELK是一套常用的开源日志监控和分析系统，包括一个分布式索引与搜索服务Elasticsearch，一个管理日志和事件的工具logstash，和一个数据可视化服务Kibana
logstash_1.5.3 负责日志的收集，处理和储存
elasticsearch-1.7.2 负责日志检索和分析
kibana-4.1.2-linux-x64.tar.gz 负责日志的可视化
jdk-1.7.0_03 java环境
redis-2.4.14 DB
一、基础环境
1、角色、ip、版本、内核
serverA 10.1.10.185 3.2.0-4-amd64 7.8 java elasticsearch redis kibana logstash(agent indexer)
clientB 10.1.10.117 3.2.0-4-amd64 7.8 java logstash(agent)
2、安装基础包
apt-get -y install curl wget lrzsz axel
二、安装redis server
1、安装包
apt-get -y install redis-server
2、创建redis存储目录
mkdir /opt/redis -p
3、权限
chown redis /opt/redis/ -R
4、配置
1)备份配置
cp /etc/redis/redis.conf /etc/redis/redis.conf.bak
2)修改配置
sed -i 's!^bind.*!bind 10.1.10.185!g' /etc/redis/redis.conf
sed -i 's!^dir.*!dir /opt/redis!g' /etc/redis/redis.conf
5、重启服务
/etc/init.d/redis-server restart
6、查看进程和端口
1)查看进程
ps -ef |grep redis
redis 23193 1 0 16:41 ? 00:00:00 /usr/bin/redis-server /etc/redis/redis.conf
2)查看端口
netstat -tupnl |grep redis
tcp 0 0 10.1.10.185:6379 0.0.0.0:* LISTEN 25188/redis-server
7、检查开机启动(默认设置开机启动了)
ll /etc/rc2.d/ |grep redis
lrwxrwxrwx 1 root root 22 Sep 20 16:41 S02redis-server -> ../init.d/redis-server
三、安装java环境
1、安装包
apt-get -y install openjdk-7-jdk
2、查看版本
java -version
java version "1.7.0_03"
OpenJDK Runtime Environment (IcedTea7 2.1.7) (7u3-2.1.7-1)
OpenJDK 64-Bit Server VM (build 22.0-b10, mixed mode)
四、安装elasticsearch
1、下载elasticsearch
wget https://download.elastic.co/elasticsearch/elasticsearch/elasticsearch-1.7.2.deb
2、安装elasticsearch
dpkg -i elasticsearch-1.7.2.deb
Selecting previously unselected package elasticsearch.
(Reading database ... 30240 files and directories currently installed.)
Unpacking elasticsearch (from elasticsearch-1.7.2.deb) ...
Creating elasticsearch group... OK
Creating elasticsearch user... OK
Setting up elasticsearch (1.7.2) ...
3、配置
1)备份配置
cp /etc/elasticsearch/elasticsearch.yml /etc/elasticsearch/elasticsearch.yml.bak
2)修改配置
echo "network.bind_host: 10.1.10.185" >> /etc/elasticsearch/elasticsearch.yml
4、启动elasticsearch服务
/etc/init.d/elasticsearch start
5、查看进程和端口
1)查看进程
ps -ef |grep java
106 22835 1 63 15:14 ? 00:00:03 /usr/lib/jvm/java-7-openjdk-amd64//bin/java -Xms256m -Xmx1g -Djava.awt.headless=true -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:+HeapDumpOnOutOfMemoryError -XX:+DisableExplicitGC -Dfile.encoding=UTF-8 -Delasticsearch -Des.pidfile=/var/run/elasticsearch/elasticsearch.pid -Des.path.home=/usr/share/elasticsearch -cp :/usr/share/elasticsearch/lib/elasticsearch-1.7.2.jar:/usr/share/elasticsearch/lib/*:/usr/share/elasticsearch/lib/sigar/* -Des.default.config=/etc/elasticsearch/elasticsearch.yml -Des.default.path.home=/usr/share/elasticsearch -Des.default.path.logs=/var/log/elasticsearch -Des.default.path.data=/var/lib/elasticsearch -Des.default.path.work=/tmp/elasticsearch -Des.default.path.conf=/etc/elasticsearch org.elasticsearch.bootstrap.Elasticsearch
2)查看端口
netstat -tupnl |grep java
tcp6 0 0 10.1.10.185:9200 :::* LISTEN 22835/java
tcp6 0 0 10.1.10.185:9300 :::* LISTEN 22835/java
udp6 0 0 :::54328 :::* 22835/java
6、测试
curl -X GET :9200
{
"status" : 200,
"name" : "Ned Leeds",
"cluster_name" : "elasticsearch",
"version" : {
"number" : "1.7.2",
"build_hash" : "e43676b1385b8125d647f593f7202acbd816e8ec",
"build_timestamp" : "2015-09-14T09:49:53Z",
"build_snapshot" : false,
"lucene_version" : "4.10.4"
},
"tagline" : "You Know, for Search"
}
7、添加到开机启动
update-rc.d elasticsearch defaults
update-rc.d: using dependency based boot sequencing
五、安装logstash
1、下载logstash
wget https://download.elastic.co/logstash/logstash/packages/debian/logstash_1.5.3-1_all.deb
2、安装logstash
dpkg -i logstash_1.5.3-1_all.deb
(Reading database ... 30338 files and directories currently installed.)
Unpacking logstash (from logstash_1.5.3-1_all.deb) ...
Setting up logstash (1:1.5.3-1) ...
3、配置(默认没有这个配置文件)
1)配置logstash_agent
cat /etc/logstash/conf.d/logstash_agent.conf
input {
file {
type => "messages"
path => ["/var/log/messages"]
}
file {
type => "elasticsearch"
path => ['/var/log/elasticsearch/elasticsearch.log*']
}
}
output {
redis {
host => "10.1.10.185"
data_type => "list"
key => "logstash:redis"
}
}
2)配置logstash_indexer
cat /etc/logstash/conf.d/logstash_indexer.conf
input {
redis {
host => "10.1.10.185"
data_type => "list"
key => "logstash:redis"
type => "redis-input"
port => "6379"
}
}
output {
elasticsearch {
host => "10.1.10.185"
}
}
4、启动服务
/etc/init.d/logstash start
logstash started.
5、使用jps -mlv或ps -ef来查看下进程
ps -ef|grep logst
logstash 22932 1 16 15:19 pts/0 00:00:01 /usr/bin/java -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -Djava.awt.headless=true -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -Djava.io.tmpdir=/var/lib/logstash -Xmx500m -Xss2048k -Djffi.boot.library.path=/opt/logstash/vendor/jruby/lib/jni -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -Djava.awt.headless=true -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -Djava.io.tmpdir=/var/lib/logstash -Xbootclasspath/a:/opt/logstash/vendor/jruby/lib/jruby.jar -classpath : -Djruby.home=/opt/logstash/vendor/jruby -Djruby.lib=/opt/logstash/vendor/jruby/lib -Djruby.script=jruby -Djruby.shell=/bin/sh org.jruby.Main --1.9 /opt/logstash/lib/bootstrap/environment.rb logstash/runner.rb agent -f /etc/logstash/conf.d -l /var/log/logstash/logstash.log
6、设置开机启动
update-rc.d logstash defaults
update-rc.d: using dependency based boot sequencing
六、安装kibana(前端web)
1、下载
axel -n 10 https://download.elastic.co/kibana/kibana/kibana-4.1.2-linux-x64.tar.gz
2、解压到指定目录
tar zxvf kibana-4.1.2-linux-x64.tar.gz -C /opt
3、创建日志目录
mkdir -p /opt/kibanalog
4、配置
1)备份配置
cp /opt/kibana-4.1.2-linux-x64/config/kibana.yml /opt/kibana-4.1.2-linux-x64/config/kibana.yml.bak
2)修改配置
sed -i 's!^elasticsearch_url: .*!elasticsearch_url: "http://10.1.10.185:9200"!g' /opt/kibana-4.1.2-linux-x64/config/kibana.yml
sed -i 's!^host: .*!host: "10.1.10.185"!g' /opt/kibana-4.1.2-linux-x64/config/kibana.yml
5、启动服务
cd /opt/kibanalog && nohup /opt/kibana-4.1.2-linux-x64/bin/kibana &
6、查看进程和端口
1)查看进程
ps aux |grep kibana
root 22982 5.4 20.1 612576 47716 pts/0 Sl 15:22 0:01 /opt/kibana-4.1.2-linux-x64/bin/../node/bin/node /opt/kibana-4.1.2-linux-x64/bin/../src/bin/kibana.js
2)查看端口
netstat -tupnl|grep 5601
tcp 0 0 10.1.10.185:5601 0.0.0.0:* LISTEN 22982/node
7、在windows上访问:5601