使用 docker 进行 ElasticSearch + Kibana 集群搭建

在Docker容器中运行Elasticsearch Kibana和Cerebro

机器信息

10.160.13.139 10.160.9.162 10.160.11.171 1. 安装dockerdocker compose

安装docker

# 安装 docker 社区版 yum -y install docker-ce docker-ce-cli containerd.io # 启动 Docker systemctl start docker # 查看 docker 版本 docker version # 开机启动 Docker systemctl enable docker

安装docker-compose

sudo curl -L https://github.com/docker/compose/releases/download/1.27.4/docker-compose-`uname -s`-`uname -m` \ > /usr/local/bin/docker-compose # 添加执行权限 sudo chmod +x /usr/local/bin/docker-compose # 验证 docker-compose --version docker-compose version 1.27.4, build 40524192 2. 下载镜像 docker pull elasticsearch:7.10.1 docker pull kibana:7.10.1 3. 将 kibana 上传到私有harbor [@master software]# docker images |grep kibana kibana 7.10.1 3e014820ee3f 3 weeks ago 992MB [@master software]# docker tag 3e014820ee3f :7.10.1 [@master software]# docker login Username: song Password: Login Succeeded [@master software]# docker push :7.10.1 The push refers to repository [] 00cfe116c0b3: Pushed 20889a7593bc: Pushed d1409dea493d: Pushed 3448a6e61306: Pushed f3a4630cb1b0: Pushed 0e3f51adbdfe: Pushed 7bb6eb357f15: Pushed 0263a48f9a3a: Pushed 5acb536ef82f: Pushed 58d51bd5abeb: Pushed e116279745ca: Pushed cc33c1bea858: Pushed 5ec5e9c8cf6f: Pushed 291f6e44771a: Pushed 7.10.1: digest: sha256:1731793b7f3e453c65ebaf92ec0b55f4029310ba8abae9e04753a4680dd8210b size: 3247 [@master software]# 4. 将 elasticsearch 上传到私有harbor [@master software]# docker images |grep elastic elasticsearch 7.10.1 558380375f1a 3 weeks ago 774MB [@master software]# docker tag 558380375f1a :7.10.1 [@master software]# docker push :7.10.1 The push refers to repository [] d6208dd58075: Pushed 9117213e7470: Pushed b662ef6f4fc1: Pushed 3f481e912123: Pushed b456348ecd84: Pushed e2834bada95f: Pushed 8f84abff69b8: Pushed 911490c8ab00: Pushed 291f6e44771a: Mounted from elk/kibana 7.10.1: digest: sha256:e9a1fe65f68b2d2b9583287d1190f67f23af08582eac4d2a8dc342e4219c7306 size: 2202 [@master software]# 5 .操作系统优化

操作系统以及JVM调优主要是针对安装elasticsearch的机器。对于操作系统,需要调整几个内核参数

vim /etc/sysctl.conf # fs.file-max主要是配置系统最大打开文件描述符数,建议修改为655360或者更高 fs.file-max=655360 # vm.max_map_count影响Java线程数量,用于限制一个进程可以拥有的VMA(虚拟内存区域)的大小 # 系统默认是65530,建议修改成262144或者更高。 vm.max_map_count = 262144 sysctl -p sysctl -a

另外,还需要调整进程最大打开文件描述符(nofile)、最大用户进程数(nproc)和最大锁定内存地址空间(memlock),

vim /etc/security/limits.conf * soft nproc 20480 * hard nproc 20480 * soft nofile 65536 * hard nofile 65536 * soft memlock unlimited * hard memlock unlimited # 备注: * 代表Linux所有用户名称(比如es),保存,退出,重新登录才可生效 # 查看 ulimit -a

最后,还需要修改

# /etc/security/limits.d/90-nproc.conf(centos6.x系统) # /etc/security/limits.d/20-nproc.conf(centos7.x系统) vim /etc/security/limits.d/20-nproc.conf * soft nproc 4096 # 修改为: * soft nproc 20480

或者直接删除/etc/security/limits.d/20-nproc.conf文件也行。

6. JVM调优

JVM调优主要是针对elasticsearch的JVM内存资源进行优化,elasticsearch的内存资源配置文件为jvm.options,7.1下载的默认设置是1GB
从ES 6开始,只支持64位的JVM(配置config/jvm.options)

配置的建议

将内存Xms和Xmx设置成一样,避免heap resize时引发停顿

Xmx设置不要超过物理内存的50%,单个节点上,最大内存建议不要超过32G内存

生产环境,JVM必须使用Server模式

关闭JVM Swapping

vim /usr/local/elasticsearch/config/jvm.options -Xms30g -Xmx30g 7. 安装 elasticsearch

你可以先新建一个 elasticsearch 文件夹。然后在这个文件夹中,新建一个 docker-compose.yml 文件。

运行docker-compose,本地构建更高效的开发环境,更直观地了解 Elasticsearch 分布式特性

集成Cerebro,方便查看集群状态

mkdir -p /search/odin/elasticsearch/{data01,data02,data03} mkdir -p /Users/song/study/elasticsearch/plugins/ik # ES_JAVA_OPTS="-Xms64m -Xmx512m" 内存限制,最小64m,最大512m,可根据需要自行调整,除此之外其他的配置不用做修改 cat /search/odin/elasticsearch/docker-compose.yml version: '2.2' services: cerebro: image: lmenezes/cerebro:0.8.3 container_name: cerebro ports: - "9000:9000" command: - -Dhosts.0.host=http://elasticsearch:9200 networks: - elastic kibana: image: kibana:7.10.1 container_name: kibana7 environment: - I18N_LOCALE=zh-CN - XPACK_GRAPH_ENABLED=true - TIMELION_ENABLED=true - XPACK_MONITORING_COLLECTION_ENABLED="true" ports: - "5601:5601" networks: - elastic elasticsearch: image: elasticsearch:7.10.1 container_name: es01 environment: - node.name=es01 - cluster.name=es-docker-cluster - discovery.seed_hosts=es02,es03 - cluster.initial_master_nodes=es01,es02,es03 - bootstrap.memory_lock=true - "ES_JAVA_OPTS=-Xms1024m -Xmx1024m" ulimits: nproc: 65535 memlock: soft: -1 hard: -1 volumes: - data01:/usr/share/elasticsearch/data - /Users/song/study/elasticsearch/plugins/ik:/usr/share/elasticsearch/plugins/ik ports: - 9200:9200 - 9300:9300 networks: - elastic elasticsearch2: image: elasticsearch:7.10.1 container_name: es02 environment: - node.name=es02 - cluster.name=es-docker-cluster - discovery.seed_hosts=es01,es03 - cluster.initial_master_nodes=es01,es02,es03 - bootstrap.memory_lock=true - "ES_JAVA_OPTS=-Xms1024m -Xmx1024m" ulimits: nproc: 65535 memlock: soft: -1 hard: -1 volumes: - data02:/usr/share/elasticsearch/data - /Users/song/study/elasticsearch/plugins/ik:/usr/share/elasticsearch/plugins/ik ports: - 9201:9201 - 9301:9301 networks: - elastic elasticsearch3: image: elasticsearch:7.10.1 container_name: es03 environment: - node.name=es03 - cluster.name=es-docker-cluster - discovery.seed_hosts=es01,es02 - cluster.initial_master_nodes=es01,es02,es03 - bootstrap.memory_lock=true - "ES_JAVA_OPTS=-Xms1024m -Xmx1024m" ulimits: nproc: 65535 memlock: soft: -1 hard: -1 volumes: - data03:/usr/share/elasticsearch/data - /Users/song/study/elasticsearch/plugins/ik:/usr/share/elasticsearch/plugins/ik ports: - 9202:9202 - 9302:9302 networks: - elastic elastichd: image: containerize/elastichd:latest container_name: elasticsearch-hd networks: - elastic ports: - 9800:9800 depends_on: - "elasticsearch" links: - "elasticsearch:demo" volumes: data01: driver: local data02: driver: local data03: driver: local networks: elastic: driver: bridge

运行docker-compose,安装Elasticsearch

启动

screen docker-compose up

访问测试

curl -X GET "localhost:9200/_cat/nodes?v&pretty"

停止容器

docker-compose down

停止容器并且移除数据

docker-compose down -v

内容版权声明:除非注明,否则皆为本站原创文章。

转载注明出处:https://www.heiqu.com/wpzjxf.html