可通过开源 Helm 插件或自定义镜像在 Kubernetes 上部署 Hadoop 集群,主要部署 HDFS、Hive MetaStore 组件。在腾讯云 TKE 中推荐使用 k8s-big-data-suite 大数据应用自动化部署 Hadoop 集群。
k8s-big-data-suite 是我们基于生产经验开发的大数据套件,可支持主流的大数据组件在 Kubernetes 上一键部署。部署之前请先按照要求做集群初始化: # 标识存储节点,至少三个 $ kubectl label node xxx storage=true
部署成功后,连入 TKE 集群查看组件状态:
$ kubectl get po NAME READY STATUS RESTARTS AGE alertmanager-tkbs-prometheus-operator-alertmanager-0 2/2 Running 0 6d23h cert-job-kv5tm 0/1 Completed 0 6d23h elasticsearch-master-0 1/1 Running 0 6d23h elasticsearch-master-1 1/1 Running 0 6d23h flink-operator-controller-manager-9485b8f4c-75zvb 2/2 Running 0 6d23h kudu-master-0 2/2 Running 2034 6d23h kudu-master-1 2/2 Running 0 6d23h kudu-master-2 2/2 Running 0 6d23h kudu-tserver-0 1/1 Running 0 6d23h kudu-tserver-1 1/1 Running 0 6d23h kudu-tserver-2 1/1 Running 0 6d23h prometheus-tkbs-prometheus-operator-prometheus-0 3/3 Running 0 6d23h superset-init-db-g6nz2 0/1 Completed 0 6d23h thrift-jdbcodbc-server-1603699044755-exec-1 1/1 Running 0 6d23h tkbs-admission-5559c4cddf-w7wtf 1/1 Running 0 6d23h tkbs-admission-init-x8sqd 0/1 Completed 0 6d23h tkbs-airflow-scheduler-5d44f5bf66-5hd8k 1/1 Running 2 6d23h tkbs-airflow-web-84579bc4cd-6dftv 1/1 Running 2 6d23h tkbs-client-844559f5d7-r86rb 1/1 Running 6 6d23h tkbs-controllers-6b9b95d768-vr7t5 1/1 Running 0 6d23h tkbs-cp-kafka-0 3/3 Running 2 6d23h tkbs-cp-kafka-1 3/3 Running 2 6d23h tkbs-cp-kafka-2 3/3 Running 2 6d23h tkbs-cp-kafka-connect-657bdff584-g9f2r 2/2 Running 2 6d23h tkbs-cp-schema-registry-84cd7cbdbc-d28jk 2/2 Running 4 6d23h tkbs-grafana-68586d8f97-zbc2m 2/2 Running 0 6d23h tkbs-hadoop-hdfs-dn-6jng4 2/2 Running 0 6d23h tkbs-hadoop-hdfs-dn-rn8z9 2/2 Running 0 6d23h tkbs-hadoop-hdfs-dn-t68zq 2/2 Running 0 6d23h tkbs-hadoop-hdfs-jn-0 2/2 Running 0 6d23h tkbs-hadoop-hdfs-jn-1 2/2 Running 0 6d23h tkbs-hadoop-hdfs-jn-2 2/2 Running 0 6d23h tkbs-hadoop-hdfs-nn-0 2/2 Running 5 6d23h tkbs-hadoop-hdfs-nn-1 2/2 Running 0 6d23h tkbs-hbase-master-0 1/1 Running 3 6d23h tkbs-hbase-master-1 1/1 Running 0 6d23h tkbs-hbase-rs-0 1/1 Running 3 6d23h tkbs-hbase-rs-1 1/1 Running 0 6d23h tkbs-hbase-rs-2 1/1 Running 0 6d23h tkbs-hive-metastore-0 2/2 Running 0 6d23h tkbs-hive-metastore-1 2/2 Running 0 6d23h tkbs-hive-server-8649cb7446-jq426 2/2 Running 1 6d23h tkbs-impala-catalogd-6f46fd97c6-b6j7b 1/1 Running 0 6d23h tkbs-impala-coord-exec-0 1/1 Running 7 6d23h tkbs-impala-coord-exec-1 1/1 Running 7 6d23h tkbs-impala-coord-exec-2 1/1 Running 7 6d23h tkbs-impala-shell-844796695-fgsjt 1/1 Running 0 6d23h tkbs-impala-statestored-798d44765f-ffp82 1/1 Running 0 6d23h tkbs-kibana-7994978d8f-5fbcx 1/1 Running 0 6d23h tkbs-kube-state-metrics-57ff4b79cb-lmsxp 1/1 Running 0 6d23h tkbs-loki-0 1/1 Running 0 6d23h tkbs-mist-d88b8bc67-s8pxx 1/1 Running 0 6d23h tkbs-nginx-ingress-controller-87b7fb9bb-mpgtj 1/1 Running 0 6d23h tkbs-nginx-ingress-default-backend-6857b58896-rgc5c 1/1 Running 0 6d23h tkbs-nginx-proxy-64964c4c79-7xqx6 1/1 Running 6 6d23h tkbs-postgresql-5b9ddc464c-xc5nn 1/1 Running 1 6d23h tkbs-postgresql-ha-pgpool-5cbf85d847-v5dsr 1/1 Running 1 6d23h tkbs-postgresql-ha-postgresql-0 2/2 Running 0 6d23h tkbs-postgresql-ha-postgresql-1 2/2 Running 0 6d23h tkbs-prometheus-node-exporter-bdp9v 1/1 Running 0 6d23h tkbs-prometheus-node-exporter-cdrqr 1/1 Running 0 6d23h tkbs-prometheus-node-exporter-cv767 1/1 Running 0 6d23h tkbs-prometheus-node-exporter-l82wp 1/1 Running 0 6d23h tkbs-prometheus-node-exporter-nb4pk 1/1 Running 0 6d23h tkbs-prometheus-operator-operator-f74dd4f6f-lnscv 2/2 Running 0 6d23h tkbs-promtail-d6r9r 1/1 Running 0 6d23h tkbs-promtail-gd5nz 1/1 Running 0 6d23h tkbs-promtail-l9kjw 1/1 Running 0 6d23h tkbs-promtail-llwvh 1/1 Running 0 6d23h tkbs-promtail-prgt9 1/1 Running 0 6d23h tkbs-scheduler-74f5777c5d-hr88l 1/1 Running 0 6d23h tkbs-spark-history-7d78cf8b56-82xg7 1/1 Running 4 6d23h tkbs-spark-thirftserver-5757f9588d-gdnzz 1/1 Running 4 6d23h tkbs-sparkoperator-f9fc5b8bf-8s4m2 1/1 Running 0 6d23h tkbs-sparkoperator-f9fc5b8bf-m9pjk 1/1 Running 0 6d23h tkbs-sparkoperator-webhook-init-m6fn5 0/1 Completed 0 6d23h tkbs-superset-54d587c867-b99kw 1/1 Running 0 6d23h tkbs-zeppelin-controller-65c454cfb9-m4snp 1/1 Running 0 6d23h tkbs-zookeeper-0 3/3 Running 0 6d23h tkbs-zookeeper-1 3/3 Running 0 6d23h tkbs-zookeeper-2 3/3 Running 0 6d23h 注意