I have the following setup:
Image of omg/telperion docker on the docker hub Kubernetovsky cluster (with 4 nodes, each with ~ 50 GB of RAM) and many resources
I followed tutorials to pull images from dockerhub to kubernetes
SERVICE_NAME=telperion DOCKER_SERVER="https://index.docker.io/v1/" DOCKER_USERNAME=username DOCKER_PASSWORD=password DOCKER_EMAIL=" omg@whatever.com " # Create secret kubectl create secret docker-registry dockerhub --docker-server=$DOCKER_SERVER --docker-username=$DOCKER_USERNAME --docker-password=$DOCKER_PASSWORD --docker-email=$DOCKER_EMAIL # Create service yaml echo "apiVersion: v1 \n\ kind: Pod \n\ metadata: \n\ name: ${SERVICE_NAME} \n\ spec: \n\ containers: \n\ - name: ${SERVICE_NAME} \n\ image: omg/${SERVICE_NAME} \n\ imagePullPolicy: Always \n\ command: [ \"echo\",\"done deploying $SERVICE_NAME\" ] \n\ imagePullSecrets: \n\ - name: dockerhub" > $SERVICE_NAME.yaml # Deploy to kubernetes kubectl create -f $SERVICE_NAME.yaml
This causes the pod to go into CrashLoopBackoff
docker run -it -p8080:9546 omg/telperion working fine.
So my question is: Is this debugging ?, if so, how do I debug this?
Some magazines:
kubectl get nodes NAME STATUS AGE VERSION k8s-agent-adb12ed9-0 Ready 22h v1.6.6 k8s-agent-adb12ed9-1 Ready 22h v1.6.6 k8s-agent-adb12ed9-2 Ready 22h v1.6.6 k8s-master-adb12ed9-0 Ready,SchedulingDisabled 22h v1.6.6
.
kubectl get pods NAME READY STATUS RESTARTS AGE telperion 0/1 CrashLoopBackOff 10 28m
.
kubectl describe pod telperion Name: telperion Namespace: default Node: k8s-agent-adb12ed9-2/10.240.0.4 Start Time: Wed, 21 Jun 2017 10:18:23 +0000 Labels: <none> Annotations: <none> Status: Running IP: 10.244.1.4 Controllers: <none> Containers: telperion: Container ID: docker://c2dd021b3d619d1d4e2afafd7a71070e1e43132563fdc370e75008c0b876d567 Image: omg/telperion Image ID: docker-pullable://omg/ telperion@sha256 :c7e3beb0457b33cd2043c62ea7b11ae44a5629a5279a88c086ff4853828a6d96 Port: Command: echo done deploying telperion State: Waiting Reason: CrashLoopBackOff Last State: Terminated Reason: Completed Exit Code: 0 Started: Wed, 21 Jun 2017 10:19:25 +0000 Finished: Wed, 21 Jun 2017 10:19:25 +0000 Ready: False Restart Count: 3 Environment: <none> Mounts: /var/run/secrets/kubernetes.io/serviceaccount from default-token-n7ll0 (ro) Conditions: Type Status Initialized True Ready False PodScheduled True Volumes: default-token-n7ll0: Type: Secret (a volume populated by a Secret) SecretName: default-token-n7ll0 Optional: false QoS Class: BestEffort Node-Selectors: <none> Tolerations: <none> Events: FirstSeen LastSeen Count From SubObjectPath Type Reason Message
Edit 1: Errors reported by kubelet to master:
journalctl -u kubelet
.
Jun 21 10:28:49 k8s-master-ADB12ED9-0 docker[1622]: E0621 10:28:49.798140 1809 fsHandler.go:121] failed to collect filesystem stats - rootDiskErr: du command failed on /var/lib/docker/overlay/5cfff16d670f2df6520360595d7858fb5d16607b6999a88e5dcbc09e1e7ab9ce with output Jun 21 10:28:49 k8s-master-ADB12ED9-0 docker[1622]: , stderr: du: cannot access '/var/lib/docker/overlay/5cfff16d670f2df6520360595d7858fb5d16607b6999a88e5dcbc09e1e7ab9ce/merged/proc/13122/task/13122/fd/4': No such file or directory Jun 21 10:28:49 k8s-master-ADB12ED9-0 docker[1622]: du: cannot access '/var/lib/docker/overlay/5cfff16d670f2df6520360595d7858fb5d16607b6999a88e5dcbc09e1e7ab9ce/merged/proc/13122/task/13122/fdinfo/4': No such file or directory Jun 21 10:28:49 k8s-master-ADB12ED9-0 docker[1622]: du: cannot access '/var/lib/docker/overlay/5cfff16d670f2df6520360595d7858fb5d16607b6999a88e5dcbc09e1e7ab9ce/merged/proc/13122/fd/3': No such file or directory Jun 21 10:28:49 k8s-master-ADB12ED9-0 docker[1622]: du: cannot access '/var/lib/docker/overlay/5cfff16d670f2df6520360595d7858fb5d16607b6999a88e5dcbc09e1e7ab9ce/merged/proc/13122/fdinfo/3': No such file or directory Jun 21 10:28:49 k8s-master-ADB12ED9-0 docker[1622]: - exit status 1, rootInodeErr: <nil>, extraDiskErr: <nil>
Edit 2: more magazines
kubectl logs $SERVICE_NAME -p done deploying telperion