我们可以通过查看service来查看对应的容器的ip。
[root@A01-R06-I184-22 yaml]# kubectl describe service tensorflow-ps-service Name: tensorflow-ps-service Namespace: default Labels: name=tensorflow-ps,role=service Selector: name=tensorflow-ps Type: ClusterIP IP: 10.254.170.61 Port: <unset> 2222/TCP Endpoints: 4.0.84.3:2222,4.0.84.4:2222 Session Affinity: None No events. [root@A01-R06-I184-22 yaml]# kubectl describe service tensorflow-wk-service Name: tensorflow-wk-service Namespace: default Labels: name=tensorflow-worker,role=service Selector: name=tensorflow-worker Type: ClusterIP IP: 10.254.70.9 Port: <unset> 2222/TCP Endpoints: 4.0.84.5:2222,4.0.84.6:2222 Session Affinity: None No events.这里我使用deep_recommend_system来进行分布式的实验。
在pod中先下载对应的deep_recommend_system的代码。
curl https://codeload.github.com/tobegit3hub/deep_recommend_system/zip/master -o drs.zip unzip drs.zip cd deep_recommend_system-master/distributed/在ps的其中一个容器(4.0.84.3)中执行启动ps服务器的任务：
root@tensorflow-ps-rc-b5d6g:/notebooks/deep_recommend_system-master/distributed# nohup Python cancer_classifier.py --ps_hosts=4.0.84.3:2222,4.0.84.3:2223 --worker_hosts=4.0.84.5:2222,4.0.84.6:2222 --job_name=ps --task_index=0 >log1 & [1] 502 root@tensorflow-ps-rc-b5d6g:/notebooks/deep_recommend_system-master/distributed# nohup: ignoring input and redirecting stderr to stdout root@tensorflow-ps-rc-b5d6g:/notebooks/deep_recommend_system-master/distributed# nohup python cancer_classifier.py --ps_hosts=4.0.84.3:2222,4.0.84.4:2223 --worker_hosts=4.0.84.5:2222,4.0.84.6:2222 --job_name=ps --task_index=1 >log2 & [2] 603 root@tensorflow-ps-rc-b5d6g:/notebooks/deep_recommend_system-master/distributed# nohup: ignoring input and redirecting stderr to stdout这里我尝试使用两个pod分别做ps服务器，但是总是报core dump的错误。官网也有类似错误，未能解决，推测原因可能是复用了某个设备的缘故(两个pod都在同一个宿主机上)。使用一个pod作为两个ps服务器即无问题。
在worker两个容器中分别执行：
root@tensorflow-worker-rc-vznvt:/notebooks/deep_recommend_system-master/distributed# nohup python cancer_classifier.py --ps_hosts=4.0.84.3:2222,4.0.84.3:2223 --worker_hosts=4.0.84.5:2222,4.0.84.6:2222 --job_name=worker --task_index=0 >log & *********************** root@tensorflow-worker-rc-cpnt7:/notebooks/deep_recommend_system-master/distributed# nohup python cancer_classifier.py --ps_hosts=4.0.84.3:2222,4.0.84.3:2223 --worker_hosts=4.0.84.5:2222,4.0.84.6:2222 --job_name=worker --task_index=1 >log &之后在worker服务器上的checkpoint文件夹中可以查看计算模型的中间保存结果。
root@tensorflow-worker-rc-vznvt:/notebooks/deep_recommend_system-master/distributed# ll checkpoint/ total 840 drwxr-xr-x 2 root root 4096 Oct 10 15:45 ./ drwxr-xr-x 3 root root 76 Oct 10 15:18 ../ -rw-r--r-- 1 root root 0 Sep 23 14:27 .gitkeeper -rw-r--r-- 1 root root 270 Oct 10 15:45 checkpoint -rw-r--r-- 1 root root 86469 Oct 10 15:45 events.out.tfevents.1476113854.tensorflow-worker-rc-vznvt -rw-r--r-- 1 root root 248875 Oct 10 15:37 graph.pbtxt -rw-r--r-- 1 root root 2229 Oct 10 15:42 model.ckpt-1172 -rw-r--r-- 1 root root 94464 Oct 10 15:42 model.ckpt-1172.meta -rw-r--r-- 1 root root 2229 Oct 10 15:43 model.ckpt-1422 -rw-r--r-- 1 root root 94464 Oct 10 15:43 model.ckpt-1422.meta -rw-r--r-- 1 root root 2229 Oct 10 15:44 model.ckpt-1670 -rw-r--r-- 1 root root 94464 Oct 10 15:44 model.ckpt-1670.meta -rw-r--r-- 1 root root 2229 Oct 10 15:45 model.ckpt-1921 -rw-r--r-- 1 root root 94464 Oct 10 15:45 model.ckpt-1921.meta -rw-r--r-- 1 root root 2229 Oct 10 15:41 model.ckpt-921 -rw-r--r-- 1 root root 94464 Oct 10 15:41 model.ckpt-921.meta tensorflow gpu支持 tensorflow gpu in docker