把etcd启动文件拷贝到另外两台节点，修改下配置就可以启动了。
查看集群状态：
由于etcd使用了证书，所以etcd命令需要带上证书：
#查看etcd成员列表
root@k8s-master01:~# etcdctl --key-file /etc/etcd/ssl/etcd-key.pem --cert-file /etc/etcd/ssl/etcd.pem --ca-file /etc/kubernetes/ssl/ca.pem member list
702819a30dfa37b8: name=etcd-host2 peerURLs=https://10.3.1.20:2380 clientURLs=https://10.3.1.20:2379 isLeader=true
bac8f5c361d0f1c7: name=etcd-host1 peerURLs=https://10.3.1.21:2380 clientURLs=https://10.3.1.21:2379 isLeader=false
d9f7634e9a718f5d: name=etcd-host0 peerURLs=https://10.3.1.25:2380 clientURLs=https://10.3.1.25:2379 isLeader=false
#或查看集群是否健康
root@k8s-maste01:~/ssl# etcdctl --key-file /etc/etcd/ssl/etcd-key.pem --cert-file /etc/etcd/ssl/etcd.pem --ca-file /etc/kubernetes/ssl/ca.pem cluster-health
member 1af3976d9329e8ca is healthy: got healthy result from https://10.3.1.20:2379
member 34b6c7df0ad76116 is healthy: got healthy result from https://10.3.1.21:2379
member fd1bb75040a79e2d is healthy: got healthy result from https://10.3.1.25:2379
cluster is healthy
apt-get update
apt-get install \
apt-transport-https \
ca-certificates \
curl \
software-properties-common
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo apt-key add -
apt-key fingerprint 0EBFCD88
add-apt-repository \
"deb [arch=amd64] https://download.docker.com/linux/ubuntu \
$(lsb_release -cs) \
stable"
apt-get update
apt-get install -y docker-ce=17.03.2~ce-0~ubuntu-xenial
安装完Docker后，设置FORWARD规则为ACCEPT
#默认为DROP
iptables -P FORWARD ACCEPT
所有节点都需要安装kubeadm
apt-get update && apt-get install -y apt-transport-https curl
curl -s https://packages.cloud.google.com/apt/doc/apt-key.gpg | apt-key add -
echo 'deb kubernetes-xenial main' >/etc/apt/sources.list.d/kubernetes.list
apt-get update
apt-get install -y kubeadm
#它会自动安装kubeadm、kubectl、kubelet、kubernetes-cni、socat
安装完后，设置kubelet服务开机自启：
systemctl enable kubelet
必须设置Kubelet开机自启动，才能让k8s集群各组件在系统重启后自动运行。
集群初始化接下开始在三台master执行集群初始化。
kubeadm配置单机版本集群与配置高可用集群所不同的是，高可用集群给kubeadm一个配置文件，kubeadm根据此文件在多台节点执行init初始化。
root@k8s-master01:~/kubeadm-config# cat kubeadm-config.yaml
apiVersion: kubeadm.k8s.io/v1alpha3
kind: ClusterConfiguration
kubernetesVersion: stable
networking:
podSubnet: 192.168.0.0/16
apiServerCertSANs:
- k8s-master01
- k8s-master02
- k8s-master03
- 10.3.1.20
- 10.3.1.21
- 10.3.1.25
- 10.3.1.29
- 127.0.0.1
etcd:
external:
endpoints:
- https://10.3.1.20:2379
- https://10.3.1.21:2379
- https://10.3.1.25:2379
caFile: /etc/kubernetes/ssl/ca.pem
certFile: /etc/etcd/ssl/etcd.pem
keyFile: /etc/etcd/ssl/etcd-key.pem
dataDir: /var/lib/etcd
token: 547df0.182e9215291ff27f
tokenTTL: "0"
root@k8s-master01:~/kubeadm-config#
配置解析：
版本v1.12的api版本已提升为kubeadm.k8s.io/v1alpha3，kind已变成ClusterConfiguration。
podSubnet：自定义pod网段。
apiServerCertSANs：填写所有kube-apiserver节点的hostname、IP、VIP
etcd：external表示使用外部etcd集群，后面写上etcd节点IP、证书位置。
如果etcd集群由kubeadm配置，则应该写local，加上自定义的启动参数。
token：可以不指定，使用指令 kubeadm token generate 生成。
#确保swap已关闭
root@k8s-master01:~/kubeadm-config# kubeadm init --config kubeadm-config.yaml
输出如下信息：
#kubernetes v1.12.0开始初始化
[init] using Kubernetes version: v1.12.0
#初始化之前预检
[preflight] running pre-flight checks
[preflight/images] Pulling images required for setting up a Kubernetes cluster
[preflight/images] This might take a minute or two, depending on the speed of your internet connection
#可以在init之前用kubeadm config images pull先拉镜像
[preflight/images] You can also perform this action in beforehand using 'kubeadm config images pull'
#生成kubelet服务的配置
[kubelet] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[preflight] Activating the kubelet service
#生成证书
[certificates] Generated ca certificate and key.
[certificates] Generated apiserver certificate and key.
[certificates] apiserver serving cert is signed for DNS names [k8s-master01 kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local k8s-master01 k8s-master02 k8s-master03] and IPs [10.96.0.1 10.3.1.20 10.3.1.20 10.3.1.21 10.3.1.25 10.3.1.29 127.0.0.1]
[certificates] Generated apiserver-kubelet-client certificate and key.
[certificates] Generated front-proxy-ca certificate and key.
[certificates] Generated front-proxy-client certificate and key.
[certificates] valid certificates and keys now exist in "/etc/kubernetes/pki"
[certificates] Generated sa key and public key.
#生成kubeconfig
[kubeconfig] Wrote KubeConfig file to disk: "/etc/kubernetes/admin.conf"
[kubeconfig] Wrote KubeConfig file to disk: "/etc/kubernetes/kubelet.conf"
[kubeconfig] Wrote KubeConfig file to disk: "/etc/kubernetes/controller-manager.conf"
[kubeconfig] Wrote KubeConfig file to disk: "/etc/kubernetes/scheduler.conf"
#生成要启动Pod清单文件
[controlplane] wrote Static Pod manifest for component kube-apiserver to "/etc/kubernetes/manifests/kube-apiserver.yaml"
[controlplane] wrote Static Pod manifest for component kube-controller-manager to "/etc/kubernetes/manifests/kube-controller-manager.yaml"
[controlplane] wrote Static Pod manifest for component kube-scheduler to "/etc/kubernetes/manifests/kube-scheduler.yaml"
#启动Kubelet服务，读取pod清单文件/etc/kubernetes/manifests
[init] waiting for the kubelet to boot up the control plane as Static Pods from directory "/etc/kubernetes/manifests"
#根据清单文件拉取镜像
[init] this might take a minute or longer if the control plane images have to be pulled
#所有组件启动完成
[apiclient] All control plane components are healthy after 27.014452 seconds
#上传配置kubeadm-config" in the "kube-system"
[uploadconfig] storing the configuration used in ConfigMap "kubeadm-config" in the "kube-system" Namespace
[kubelet] Creating a ConfigMap "kubelet-config-1.12" in namespace kube-system with the configuration for the kubelets in the cluster
#给master添加一个污点的标签taint
[markmaster] Marking the node k8s-master01 as master by adding the label "node-role.kubernetes.io/master=''"
[markmaster] Marking the node k8s-master01 as master by adding the taints [node-role.kubernetes.io/master:NoSchedule]
[patchnode] Uploading the CRI Socket information "/var/run/dockershim.sock" to the Node API object "k8s-master01" as an annotation
#使用的token
[bootstraptoken] using token: w79yp6.erls1tlc4olfikli
[bootstraptoken] configured RBAC rules to allow Node Bootstrap tokens to post CSRs in order for nodes to get long term certificate credentials
[bootstraptoken] configured RBAC rules to allow the csrapprover controller automatically approve CSRs from a Node Bootstrap Token
[bootstraptoken] configured RBAC rules to allow certificate rotation for all node client certificates in the cluster
[bootstraptoken] creating the "cluster-info" ConfigMap in the "kube-public" namespace
#最后安装基础组件kube-dns和kube-proxy daemonset
[addons] Applied essential addon: CoreDNS
[addons] Applied essential addon: kube-proxy
Your Kubernetes master has initialized successfully!
To start using your cluster, you need to run the following as a regular user:
mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config