This job view page is being replaced by Spyglass soon. Check out the new job view.
ResultFAILURE
Tests 8 failed / 841 succeeded
Started2020-03-28 17:48
Elapsed15h11m
Revision
Builder365bc658-711c-11ea-a0b0-0e43f1618cf3
links{u'resultstore': {u'url': u'https://source.cloud.google.com/results/invocations/af5e6d27-8902-4ddf-bd16-cc4b1cb4d82f/targets/test'}}
resultstorehttps://source.cloud.google.com/results/invocations/af5e6d27-8902-4ddf-bd16-cc4b1cb4d82f/targets/test
infra-commitee73f7188
job-versionv1.16.9-beta.0.17+bb616cba1cc983
master_os_imagecos-77-12371-175-0
node_os_imagecos-73-11647-163-0
revisionv1.16.9-beta.0.17+bb616cba1cc983

Test Failures


Kubernetes e2e suite [k8s.io] [sig-cloud-provider-gcp] Upgrade [Feature:Upgrade] master upgrade should maintain a functioning cluster [Feature:MasterUpgrade] 17m48s

go run hack/e2e.go -v --test --test_args='--ginkgo.focus=Kubernetes\se2e\ssuite\s\[k8s\.io\]\s\[sig\-cloud\-provider\-gcp\]\sUpgrade\s\[Feature\:Upgrade\]\smaster\supgrade\sshould\smaintain\sa\sfunctioning\scluster\s\[Feature\:MasterUpgrade\]$'
/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/cloud/gcp/cluster_upgrade.go:91
Mar 28 18:02:54.324: Unexpected error:
    <*errors.errorString | 0xc002f5c400>: {
        s: "error running /workspace/kubernetes_skew/cluster/gce/upgrade.sh [-M v1.17.5-beta.0.22+ea3eeb494a528e]; got error exit status 1, stdout \"Fetching the previously installed CoreDNS version\\n\\n***WARNING***\\nUpgrading Kubernetes with this script might result in an upgrade to a new etcd version.\\nSome etcd version upgrades, such as 3.0.x to 3.1.x, DO NOT offer a downgrade path.\\nTo pin the etcd version to your current one (e.g. v3.0.17), set the following variables\\nbefore running this script:\\n\\n# example: pin to etcd v3.0.17\\nexport ETCD_IMAGE=3.0.17\\nexport ETCD_VERSION=3.0.17\\n\\nAlternatively, if you choose to allow an etcd upgrade that doesn't support downgrade,\\nyou might still be able to downgrade Kubernetes by pinning to the newer etcd version.\\nIn all cases, it is strongly recommended to have an etcd backup before upgrading.\\n\\n== Pre-Upgrade Node OS and Kubelet Versions ==\\nname: \\\"bootstrap-e2e-master\\\", osImage: \\\"Container-Optimized OS from Google\\\", kubeletVersion: \\\"v1.16.9-beta.0.17+bb616cba1cc983\\\"\\nname: \\\"bootstrap-e2e-minion-group-9sfv\\\", osImage: \\\"Container-Optimized OS from Google\\\", kubeletVersion: \\\"v1.16.9-beta.0.17+bb616cba1cc983\\\"\\nname: \\\"bootstrap-e2e-minion-group-nt4p\\\", osImage: \\\"Container-Optimized OS from Google\\\", kubeletVersion: \\\"v1.16.9-beta.0.17+bb616cba1cc983\\\"\\nname: \\\"bootstrap-e2e-minion-group-pn0s\\\", osImage: \\\"Container-Optimized OS from Google\\\", kubeletVersion: \\\"v1.16.9-beta.0.17+bb616cba1cc983\\\"\\nFound subnet for region us-west1 in network bootstrap-e2e: bootstrap-e2e\\n== Upgrading master to 'https://storage.googleapis.com/kubernetes-release-dev/ci/v1.17.5-beta.0.22+ea3eeb494a528e/kubernetes-server-linux-amd64.tar.gz'. Do not interrupt, deleting master instance. ==\\n== Upgrading master environment variables. ==\\n== Waiting for new master to respond to API requests ==\\n.........................== Done ==\\nWaiting for CoreDNS to update\\nFetching the latest installed CoreDNS version\\n== Downloading the CoreDNS migration tool ==\\n== Upgrading the CoreDNS ConfigMap ==\\nconfigmap/coredns configured\\n== The CoreDNS Config has been updated ==\\n== Validating cluster post-upgrade ==\\nValidating gce cluster, MULTIZONE=\\nFound 4 node(s).\\nNAME                              STATUS                     ROLES    AGE     VERSION\\nbootstrap-e2e-master              Ready,SchedulingDisabled   <none>   9m20s   v1.17.5-beta.0.22+ea3eeb494a528e\\nbootstrap-e2e-minion-group-9sfv   Ready                      <none>   9m23s   v1.16.9-beta.0.17+bb616cba1cc983\\nbootstrap-e2e-minion-group-nt4p   Ready                      <none>   9m23s   v1.16.9-beta.0.17+bb616cba1cc983\\nbootstrap-e2e-minion-group-pn0s   Ready                      <none>   9m23s   v1.16.9-beta.0.17+bb616cba1cc983\\nValidate output:\\nNAME                 STATUS    MESSAGE             ERROR\\netcd-1               Healthy   {\\\"health\\\":\\\"true\\\"}   \\ncontroller-manager   Healthy   ok                  \\netcd-0               Healthy   {\\\"health\\\":\\\"true\\\"}   \\nscheduler            Healthy   ok                  \\n\\x1b[0;32mCluster validation succeeded\\x1b[0m\\n== Post-Upgrade Node OS and Kubelet Versions ==\\nname: \\\"bootstrap-e2e-master\\\", osImage: \\\"Container-Optimized OS from Google\\\", kubeletVersion: \\\"v1.17.5-beta.0.22+ea3eeb494a528e\\\"\\nname: \\\"bootstrap-e2e-minion-group-9sfv\\\", osImage: \\\"Container-Optimized OS from Google\\\", kubeletVersion: \\\"v1.16.9-beta.0.17+bb616cba1cc983\\\"\\nname: \\\"bootstrap-e2e-minion-group-nt4p\\\", osImage: \\\"Container-Optimized OS from Google\\\", kubeletVersion: \\\"v1.16.9-beta.0.17+bb616cba1cc983\\\"\\nname: \\\"bootstrap-e2e-minion-group-pn0s\\\", osImage: \\\"Container-Optimized OS from Google\\\", kubeletVersion: \\\"v1.16.9-beta.0.17+bb616cba1cc983\\\"\\n\", stderr \"Project: kube-gce-upg-1-4-1-5-upg-mas\\nNetwork Project: kube-gce-upg-1-4-1-5-upg-mas\\nZone: us-west1-b\\nINSTANCE_GROUPS=bootstrap-e2e-minion-group\\nNODE_NAMES=bootstrap-e2e-minion-group-9sfv bootstrap-e2e-minion-group-nt4p bootstrap-e2e-minion-group-pn0s\\nTrying to find master named 'bootstrap-e2e-master'\\nLooking for address 'bootstrap-e2e-master-ip'\\nUsing master: bootstrap-e2e-master (external IP: 34.82.249.35; internal IP: (not set))\\nDeleted [https://www.googleapis.com/compute/v1/projects/kube-gce-upg-1-4-1-5-upg-mas/zones/us-west1-b/instances/bootstrap-e2e-master].\\nWARNING: You have selected a disk size of under [200GB]. This may result in poor I/O performance. For more information, see: https://developers.google.com/compute/docs/disks#performance.\\nCreated [https://www.googleapis.com/compute/v1/projects/kube-gce-upg-1-4-1-5-upg-mas/zones/us-west1-b/instances/bootstrap-e2e-master].\\nWARNING: Some requests generated warnings:\\n - Disk size: '20 GB' is larger than image size: '10 GB'. You might need to resize the root repartition manually if the operating system does not support automatic resizing. See https://cloud.google.com/compute/docs/disks/add-persistent-disk#resize_pd for details.\\n - The resource 'projects/cos-cloud/global/images/cos-77-12371-175-0' is deprecated. A suggested replacement is 'projects/cos-cloud/global/images/cos-77-12371-183-0'.\\n\\nNAME                  ZONE        MACHINE_TYPE   PREEMPTIBLE  INTERNAL_IP  EXTERNAL_IP   STATUS\\nbootstrap-e2e-master  us-west1-b  n1-standard-1               10.138.0.6   34.82.249.35  RUNNING\\nWarning: kubectl apply should be used on resource created by either kubectl create --save-config or kubectl apply\\nProject: kube-gce-upg-1-4-1-5-upg-mas\\nNetwork Project: kube-gce-upg-1-4-1-5-upg-mas\\nZone: us-west1-b\\n/workspace/kubernetes_skew/cluster/gce/upgrade.sh: line 452: download_dir: unbound variable\\n\"",
    }
    error running /workspace/kubernetes_skew/cluster/gce/upgrade.sh [-M v1.17.5-beta.0.22+ea3eeb494a528e]; got error exit status 1, stdout "Fetching the previously installed CoreDNS version\n\n***WARNING***\nUpgrading Kubernetes with this script might result in an upgrade to a new etcd version.\nSome etcd version upgrades, such as 3.0.x to 3.1.x, DO NOT offer a downgrade path.\nTo pin the etcd version to your current one (e.g. v3.0.17), set the following variables\nbefore running this script:\n\n# example: pin to etcd v3.0.17\nexport ETCD_IMAGE=3.0.17\nexport ETCD_VERSION=3.0.17\n\nAlternatively, if you choose to allow an etcd upgrade that doesn't support downgrade,\nyou might still be able to downgrade Kubernetes by pinning to the newer etcd version.\nIn all cases, it is strongly recommended to have an etcd backup before upgrading.\n\n== Pre-Upgrade Node OS and Kubelet Versions ==\nname: \"bootstrap-e2e-master\", osImage: \"Container-Optimized OS from Google\", kubeletVersion: \"v1.16.9-beta.0.17+bb616cba1cc983\"\nname: \"bootstrap-e2e-minion-group-9sfv\", osImage: \"Container-Optimized OS from Google\", kubeletVersion: \"v1.16.9-beta.0.17+bb616cba1cc983\"\nname: \"bootstrap-e2e-minion-group-nt4p\", osImage: \"Container-Optimized OS from Google\", kubeletVersion: \"v1.16.9-beta.0.17+bb616cba1cc983\"\nname: \"bootstrap-e2e-minion-group-pn0s\", osImage: \"Container-Optimized OS from Google\", kubeletVersion: \"v1.16.9-beta.0.17+bb616cba1cc983\"\nFound subnet for region us-west1 in network bootstrap-e2e: bootstrap-e2e\n== Upgrading master to 'https://storage.googleapis.com/kubernetes-release-dev/ci/v1.17.5-beta.0.22+ea3eeb494a528e/kubernetes-server-linux-amd64.tar.gz'. Do not interrupt, deleting master instance. ==\n== Upgrading master environment variables. ==\n== Waiting for new master to respond to API requests ==\n.........................== Done ==\nWaiting for CoreDNS to update\nFetching the latest installed CoreDNS version\n== Downloading the CoreDNS migration tool ==\n== Upgrading the CoreDNS ConfigMap ==\nconfigmap/coredns configured\n== The CoreDNS Config has been updated ==\n== Validating cluster post-upgrade ==\nValidating gce cluster, MULTIZONE=\nFound 4 node(s).\nNAME                              STATUS                     ROLES    AGE     VERSION\nbootstrap-e2e-master              Ready,SchedulingDisabled   <none>   9m20s   v1.17.5-beta.0.22+ea3eeb494a528e\nbootstrap-e2e-minion-group-9sfv   Ready                      <none>   9m23s   v1.16.9-beta.0.17+bb616cba1cc983\nbootstrap-e2e-minion-group-nt4p   Ready                      <none>   9m23s   v1.16.9-beta.0.17+bb616cba1cc983\nbootstrap-e2e-minion-group-pn0s   Ready                      <none>   9m23s   v1.16.9-beta.0.17+bb616cba1cc983\nValidate output:\nNAME                 STATUS    MESSAGE             ERROR\netcd-1               Healthy   {\"health\":\"true\"}   \ncontroller-manager   Healthy   ok                  \netcd-0               Healthy   {\"health\":\"true\"}   \nscheduler            Healthy   ok                  \n\x1b[0;32mCluster validation succeeded\x1b[0m\n== Post-Upgrade Node OS and Kubelet Versions ==\nname: \"bootstrap-e2e-master\", osImage: \"Container-Optimized OS from Google\", kubeletVersion: \"v1.17.5-beta.0.22+ea3eeb494a528e\"\nname: \"bootstrap-e2e-minion-group-9sfv\", osImage: \"Container-Optimized OS from Google\", kubeletVersion: \"v1.16.9-beta.0.17+bb616cba1cc983\"\nname: \"bootstrap-e2e-minion-group-nt4p\", osImage: \"Container-Optimized OS from Google\", kubeletVersion: \"v1.16.9-beta.0.17+bb616cba1cc983\"\nname: \"bootstrap-e2e-minion-group-pn0s\", osImage: \"Container-Optimized OS from Google\", kubeletVersion: \"v1.16.9-beta.0.17+bb616cba1cc983\"\n", stderr "Project: kube-gce-upg-1-4-1-5-upg-mas\nNetwork Project: kube-gce-upg-1-4-1-5-upg-mas\nZone: us-west1-b\nINSTANCE_GROUPS=bootstrap-e2e-minion-group\nNODE_NAMES=bootstrap-e2e-minion-group-9sfv bootstrap-e2e-minion-group-nt4p bootstrap-e2e-minion-group-pn0s\nTrying to find master named 'bootstrap-e2e-master'\nLooking for address 'bootstrap-e2e-master-ip'\nUsing master: bootstrap-e2e-master (external IP: 34.82.249.35; internal IP: (not set))\nDeleted [https://www.googleapis.com/compute/v1/projects/kube-gce-upg-1-4-1-5-upg-mas/zones/us-west1-b/instances/bootstrap-e2e-master].\nWARNING: You have selected a disk size of under [200GB]. This may result in poor I/O performance. For more information, see: https://developers.google.com/compute/docs/disks#performance.\nCreated [https://www.googleapis.com/compute/v1/projects/kube-gce-upg-1-4-1-5-upg-mas/zones/us-west1-b/instances/bootstrap-e2e-master].\nWARNING: Some requests generated warnings:\n - Disk size: '20 GB' is larger than image size: '10 GB'. You might need to resize the root repartition manually if the operating system does not support automatic resizing. See https://cloud.google.com/compute/docs/disks/add-persistent-disk#resize_pd for details.\n - The resource 'projects/cos-cloud/global/images/cos-77-12371-175-0' is deprecated. A suggested replacement is 'projects/cos-cloud/global/images/cos-77-12371-183-0'.\n\nNAME                  ZONE        MACHINE_TYPE   PREEMPTIBLE  INTERNAL_IP  EXTERNAL_IP   STATUS\nbootstrap-e2e-master  us-west1-b  n1-standard-1               10.138.0.6   34.82.249.35  RUNNING\nWarning: kubectl apply should be used on resource created by either kubectl create --save-config or kubectl apply\nProject: kube-gce-upg-1-4-1-5-upg-mas\nNetwork Project: kube-gce-upg-1-4-1-5-upg-mas\nZone: us-west1-b\n/workspace/kubernetes_skew/cluster/gce/upgrade.sh: line 452: download_dir: unbound variable\n"
occurred
/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/cloud/gcp/cluster_upgrade.go:106
				
				Click to see stdout/stderrfrom junit_upgrade01.xml

Filter through log files | View test history on testgrid


Kubernetes e2e suite [k8s.io] [sig-node] NodeProblemDetector [DisabledForLargeClusters] should run without error 2m17s

go run hack/e2e.go -v --test --test_args='--ginkgo.focus=Kubernetes\se2e\ssuite\s\[k8s\.io\]\s\[sig\-node\]\sNodeProblemDetector\s\[DisabledForLargeClusters\]\sshould\srun\swithout\serror$'
/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/node/node_problem_detector.go:58
Mar 29 07:05:56.853: Timed out after 60.001s.
Expected success, but got an error:
    <*errors.errorString | 0xc004e78cb0>: {
        s: "Event KubeletStart does not exist: [{{ } {bootstrap-e2e-minion-group-hwq3.1600b0f6842060ff  default /api/v1/namespaces/default/events/bootstrap-e2e-minion-group-hwq3.1600b0f6842060ff a5724dc6-b1ae-4752-a832-e6045cde2b7f 28081 0 2020-03-29 06:00:41 +0000 UTC <nil> <nil> map[] map[] [] []  []} {Node  bootstrap-e2e-minion-group-hwq3 bootstrap-e2e-minion-group-hwq3   } KubeletStart Started Kubernetes kubelet. {systemd-monitor bootstrap-e2e-minion-group-hwq3} 2020-03-29 06:00:41 +0000 UTC 2020-03-29 06:30:35 +0000 UTC 5 Warning 0001-01-01 00:00:00 +0000 UTC nil  nil  } {{ } {bootstrap-e2e-minion-group-hwq3.1600b1254c309350  default /api/v1/namespaces/default/events/bootstrap-e2e-minion-group-hwq3.1600b1254c309350 5d55ca7b-e88c-46ba-bf4a-507714108707 27952 0 2020-03-29 06:04:02 +0000 UTC <nil> <nil> map[] map[] [] []  []} {Node  bootstrap-e2e-minion-group-hwq3 37747262-3d5d-4e8b-b235-1dd4f0b51f51   } NodeNotReady Node bootstrap-e2e-minion-group-hwq3 status is now: NodeNotReady {node-controller } 2020-03-29 06:04:02 +0000 UTC 2020-03-29 06:28:27 +0000 UTC 2 Normal 0001-01-01 00:00:00 +0000 UTC nil  nil  } {{ } {bootstrap-e2e-minion-group-hwq3.1600b27aea8e93f9  default /api/v1/namespaces/default/events/bootstrap-e2e-minion-group-hwq3.1600b27aea8e93f9 ade541af-6a57-44fc-a659-6df76e84f3e1 27954 0 2020-03-29 06:28:29 +0000 UTC <nil> <nil> map[] map[] [] []  []} {Node  bootstrap-e2e-minion-group-hwq3 bootstrap-e2e-minion-group-hwq3   } Starting Starting kubelet. {kubelet bootstrap-e2e-minion-group-hwq3} 2020-03-29 06:28:29 +0000 UTC 2020-03-29 06:28:29 +0000 UTC 1 Normal 0001-01-01 00:00:00 +0000 UTC nil  nil  } {{ } {bootstrap-e2e-minion-group-hwq3.1600b27af4e6f232  default /api/v1/namespaces/default/events/bootstrap-e2e-minion-group-hwq3.1600b27af4e6f232 4583c656-b5c8-4d98-b735-f57cf9bc1bf1 27958 0 2020-03-29 06:28:30 +0000 UTC <nil> <nil> map[] map[] [] []  []} {Node  bootstrap-e2e-minion-group-hwq3 bootstrap-e2e-minion-group-hwq3   } NodeHasSufficientMemory Node bootstrap-e2e-minion-group-hwq3 status is now: NodeHasSufficientMemory {kubelet bootstrap-e2e-minion-group-hwq3} 2020-03-29 06:28:29 +0000 UTC 2020-03-29 06:28:30 +0000 UTC 2 Normal 0001-01-01 00:00:00 +0000 UTC nil  nil  } {{ } {bootstrap-e2e-minion-group-hwq3.1600b27af4eb8638  default /api/v1/namespaces/default/events/bootstrap-e2e-minion-group-hwq3.1600b27af4eb8638 85eabb44-da97-4eaa-bf29-b19ccfb713b4 27959 0 2020-03-29 06:28:30 +0000 UTC <nil> <nil> map[] map[] [] []  []} {Node  bootstrap-e2e-minion-group-hwq3 bootstrap-e2e-minion-group-hwq3   } NodeHasNoDiskPressure Node bootstrap-e2e-minion-group-hwq3 status is now: NodeHasNoDiskPressure {kubelet bootstrap-e2e-minion-group-hwq3} 2020-03-29 06:28:29 +0000 UTC 2020-03-29 06:28:30 +0000 UTC 2 Normal 0001-01-01 00:00:00 +0000 UTC nil  nil  } {{ } {bootstrap-e2e-minion-group-hwq3.1600b27af4f014f0  default /api/v1/namespaces/default/events/bootstrap-e2e-minion-group-hwq3.1600b27af4f014f0 91f5c81a-a741-448f-8144-65189e179c04 27960 0 2020-03-29 06:28:30 +0000 UTC <nil> <nil> map[] map[] [] []  []} {Node  bootstrap-e2e-minion-group-hwq3 bootstrap-e2e-minion-group-hwq3   } NodeHasSufficientPID Node bootstrap-e2e-minion-group-hwq3 status is now: NodeHasSufficientPID {kubelet bootstrap-e2e-minion-group-hwq3} 2020-03-29 06:28:29 +0000 UTC 2020-03-29 06:28:30 +0000 UTC 2 Normal 0001-01-01 00:00:00 +0000 UTC nil  nil  } {{ } {bootstrap-e2e-minion-group-hwq3.1600b27afb49cf63  default /api/v1/namespaces/default/events/bootstrap-e2e-minion-group-hwq3.1600b27afb49cf63 e4aeb24a-64ae-40b4-b0d5-94e24eab19e3 27961 0 2020-03-29 06:28:30 +0000 UTC <nil> <nil> map[] map[] [] []  []} {Node  bootstrap-e2e-minion-group-hwq3 bootstrap-e2e-minion-group-hwq3   } NodeNotReady Node bootstrap-e2e-minion-group-hwq3 status is now: NodeNotReady {kubelet bootstrap-e2e-minion-group-hwq3} 2020-03-29 06:28:30 +0000 UTC 2020-03-29 06:28:30 +0000 UTC 1 Normal 0001-01-01 00:00:00 +0000 UTC nil  nil  } {{ } {bootstrap-e2e-minion-group-hwq3.1600b27b148d6f59  default /api/v1/namespaces/default/events/bootstrap-e2e-minion-group-hwq3.1600b27b148d6f59 2e137161-abe3-4dbe-92e5-09ab1452e411 27962 0 2020-03-29 06:28:30 +0000 UTC <nil> <nil> map[] map[] [] []  []} {Node  bootstrap-e2e-minion-group-hwq3 bootstrap-e2e-minion-group-hwq3   } NodeAllocatableEnforced Updated Node Allocatable limit across pods {kubelet bootstrap-e2e-minion-group-hwq3} 2020-03-29 06:28:30 +0000 UTC 2020-03-29 06:28:30 +0000 UTC 1 Normal 0001-01-01 00:00:00 +0000 UTC nil  nil  } {{ } {bootstrap-e2e-minion-group-hwq3.1600b27b16028d95  default /api/v1/namespaces/default/events/bootstrap-e2e-minion-group-hwq3.1600b27b16028d95 0702ccdb-9574-4211-b2d8-e252c6ee3899 27963 0 2020-03-29 06:28:30 +0000 UTC <nil> <nil> map[] map[] [] []  []} {Node  bootstrap-e2e-minion-group-hwq3 bootstrap-e2e-minion-group-hwq3   } NodeReady Node bootstrap-e2e-minion-group-hwq3 status is now: NodeReady {kubelet bootstrap-e2e-minion-group-hwq3} 2020-03-29 06:28:30 +0000 UTC 2020-03-29 06:28:30 +0000 UTC 1 Normal 0001-01-01 00:00:00 +0000 UTC nil  nil  } {{ } {bootstrap-e2e-minion-group-hwq3.1600b298382f2316  default /api/v1/namespaces/default/events/bootstrap-e2e-minion-group-hwq3.1600b298382f2316 86f8728c-3166-48e1-810d-7a0246eb99d8 28082 0 2020-03-29 06:30:35 +0000 UTC <nil> <nil> map[] map[] [] []  []} {Node  bootstrap-e2e-minion-group-hwq3 bootstrap-e2e-minion-group-hwq3   } Starting Starting kubelet. {kubelet bootstrap-e2e-minion-group-hwq3} 2020-03-29 06:30:35 +0000 UTC 2020-03-29 06:30:35 +0000 UTC 1 Normal 0001-01-01 00:00:00 +0000 UTC nil  nil  } {{ } {bootstrap-e2e-minion-group-hwq3.1600b29841aaba43  default /api/v1/namespaces/default/events/bootstrap-e2e-minion-group-hwq3.1600b29841aaba43 36365b62-f9bd-4d6e-974c-b1fd3fac6d4b 28083 0 2020-03-29 06:30:35 +0000 UTC <nil> <nil> map[] map[] [] []  []} {Node  bootstrap-e2e-minion-group-hwq3 bootstrap-e2e-minion-group-hwq3   } NodeHasSufficientMemory Node bootstrap-e2e-minion-group-hwq3 status is now: NodeHasSufficientMemory {kubelet bootstrap-e2e-minion-group-hwq3} 2020-03-29 06:30:35 +0000 UTC 2020-03-29 06:30:35 +0000 UTC 1 Normal 0001-01-01 00:00:00 +0000 UTC nil  nil  } {{ } {bootstrap-e2e-minion-group-hwq3.1600b29841aaf543  default /api/v1/namespaces/default/events/bootstrap-e2e-minion-group-hwq3.1600b29841aaf543 6e67e250-0d60-4361-903d-b055a9530ca7 28084 0 2020-03-29 06:30:35 +0000 UTC <nil> <nil> map[] map[] [] []  []} {Node  bootstrap-e2e-minion-group-hwq3 bootstrap-e2e-minion-group-hwq3   } NodeHasNoDiskPressure Node bootstrap-e2e-minion-group-hwq3 status is now: NodeHasNoDiskPressure {kubelet bootstrap-e2e-minion-group-hwq3} 2020-03-29 06:30:35 +0000 UTC 2020-03-29 06:30:35 +0000 UTC 1 Normal 0001-01-01 00:00:00 +0000 UTC nil  nil  } {{ } {bootstrap-e2e-minion-group-hwq3.1600b29841ab1d0f  default /api/v1/namespaces/default/events/bootstrap-e2e-minion-group-hwq3.1600b29841ab1d0f f2f088e0-bbbd-4d37-8d96-73b9c167791f 28085 0 2020-03-29 06:30:35 +0000 UTC <nil> <nil> map[] map[] [] []  []} {Node  bootstrap-e2e-minion-group-hwq3 bootstrap-e2e-minion-group-hwq3   } NodeHasSufficientPID Node bootstrap-e2e-minion-group-hwq3 status is now: NodeHasSufficientPID {kubelet bootstrap-e2e-minion-group-hwq3} 2020-03-29 06:30:35 +0000 UTC 2020-03-29 06:30:35 +0000 UTC 1 Normal 0001-01-01 00:00:00 +0000 UTC nil  nil  } {{ } {bootstrap-e2e-minion-group-hwq3.1600b29844f98f92  default /api/v1/namespaces/default/events/bootstrap-e2e-minion-group-hwq3.1600b29844f98f92 b5918b5c-cbd5-45e7-a772-302f27ff4c98 28086 0 2020-03-29 06:30:35 +0000 UTC <nil> <nil> map[] map[] [] []  []} {Node  bootstrap-e2e-minion-group-hwq3 bootstrap-e2e-minion-group-hwq3   } NodeNotReady Node bootstrap-e2e-minion-group-hwq3 status is now: NodeNotReady {kubelet bootstrap-e2e-minion-group-hwq3} 2020-03-29 06:30:35 +0000 UTC 2020-03-29 06:30:35 +0000 UTC 1 Normal 0001-01-01 00:00:00 +0000 UTC nil  nil  } {{ } {bootstrap-e2e-minion-group-hwq3.1600b298544a1e27  default /api/v1/namespaces/default/events/bootstrap-e2e-minion-group-hwq3.1600b298544a1e27 fbac06e9-816e-413a-b9f2-3c0cb479f5e1 28087 0 2020-03-29 06:30:36 +0000 UTC <nil> <nil> map[] map[] [] []  []} {Node  bootstrap-e2e-minion-group-hwq3 bootstrap-e2e-minion-group-hwq3   } NodeAllocatableEnforced Updated Node Allocatable limit across pods {kubelet bootstrap-e2e-minion-group-hwq3} 2020-03-29 06:30:36 +0000 UTC 2020-03-29 06:30:36 +0000 UTC 1 Normal 0001-01-01 00:00:00 +0000 UTC nil  nil  } {{ } {bootstrap-e2e-minion-group-hwq3.1600b29855d25ca8  default /api/v1/namespaces/default/events/bootstrap-e2e-minion-group-hwq3.1600b29855d25ca8 25e0457a-f3f9-410f-b836-97a293053868 28088 0 2020-03-29 06:30:36 +0000 UTC <nil> <nil> map[] map[] [] []  []} {Node  bootstrap-e2e-minion-group-hwq3 bootstrap-e2e-minion-group-hwq3   } NodeReady Node bootstrap-e2e-minion-group-hwq3 status is now: NodeReady {kubelet bootstrap-e2e-minion-group-hwq3} 2020-03-29 06:30:36 +0000 UTC 2020-03-29 06:30:36 +0000 UTC 1 Normal 0001-01-01 00:00:00 +0000 UTC nil  nil  } {{ } {bootstrap-e2e-minion-group-hwq3.1600b476e0fc4684  default /api/v1/namespaces/default/events/bootstrap-e2e-minion-group-hwq3.1600b476e0fc4684 9eadf897-405d-48e8-8c68-5e1221554883 29228 0 2020-03-29 07:04:51 +0000 UTC <nil> <nil> map[] map[] [] []  []} {Node  bootstrap-e2e-minion-group-hwq3 bootstrap-e2e-minion-group-hwq3   } TaskHung kernel: INFO: task umount.aufs:21568 blocked for more than 120 seconds. {kernel-monitor bootstrap-e2e-minion-group-hwq3} 2020-03-29 07:04:51 +0000 UTC 2020-03-29 07:04:51 +0000 UTC 1 Warning 0001-01-01 00:00:00 +0000 UTC nil  nil  } {{ } {bootstrap-e2e-minion-group-hwq3.1600b476e0fc667d  default /api/v1/namespaces/default/events/bootstrap-e2e-minion-group-hwq3.1600b476e0fc667d ff9ec0b6-b10d-42ce-9f45-1d5e8eefac49 29227 0 2020-03-29 07:04:51 +0000 UTC <nil> <nil> map[] map[] [] []  []} {Node  bootstrap-e2e-minion-group-hwq3 bootstrap-e2e-minion-group-hwq3   } AUFSUmountHung Node condition KernelDeadlock is now: True, reason: AUFSUmountHung {kernel-monitor bootstrap-e2e-minion-group-hwq3} 2020-03-29 07:04:51 +0000 UTC 2020-03-29 07:04:51 +0000 UTC 1 Normal 0001-01-01 00:00:00 +0000 UTC nil  nil  } {{ } {bootstrap-e2e-minion-group-pn0s.1600b47783fb5cc0  default /api/v1/namespaces/default/events/bootstrap-e2e-minion-group-pn0s.1600b47783fb5cc0 95b1f0ae-db5d-42db-a8da-2890a6442575 29230 0 2020-03-29 07:04:54 +0000 UTC <nil> <nil> map[] map[] [] []  []} {Node  bootstrap-e2e-minion-group-pn0s bootstrap-e2e-minion-group-pn0s   } TaskHung kernel: INFO: task umount.aufs:21568 blocked for more than 120 seconds. {kernel-monitor bootstrap-e2e-minion-group-pn0s} 2020-03-29 07:04:54 +0000 UTC 2020-03-29 07:04:54 +0000 UTC 1 Warning 0001-01-01 00:00:00 +0000 UTC nil  nil  } {{ } {bootstrap-e2e-minion-group-pn0s.1600b47783fb852d  default /api/v1/namespaces/default/events/bootstrap-e2e-minion-group-pn0s.1600b47783fb852d 2345c5b6-f1b6-4df9-8458-ce9242a46619 29229 0 2020-03-29 07:04:54 +0000 UTC <nil> <nil> map[] map[] [] []  []} {Node  bootstrap-e2e-minion-group-pn0s bootstrap-e2e-minion-group-pn0s   } AUFSUmountHung Node condition KernelDeadlock is now: True, reason: AUFSUmountHung {kernel-monitor bootstrap-e2e-minion-group-pn0s} 2020-03-29 07:04:54 +0000 UTC 2020-03-29 07:04:54 +0000 UTC 1 Normal 0001-01-01 00:00:00 +0000 UTC nil  nil  } {{ } {bootstrap-e2e-minion-group-zc0r.1600b47811f69b71  default /api/v1/namespaces/default/events/bootstrap-e2e-minion-group-zc0r.1600b47811f69b71 4b230369-9c41-41bc-a2dc-ff9650e7eb28 29232 0 2020-03-29 07:04:56 +0000 UTC <nil> <nil> map[] map[] [] []  []} {Node  bootstrap-e2e-minion-group-zc0r bootstrap-e2e-minion-group-zc0r   } TaskHung kernel: INFO: task umount.aufs:21568 blocked for more than 120 seconds. {kernel-monitor bootstrap-e2e-minion-group-zc0r} 2020-03-29 07:04:56 +0000 UTC 2020-03-29 07:04:56 +0000 UTC 1 Warning 0001-01-01 00:00:00 +0000 UTC nil  nil  } {{ } {bootstrap-e2e-minion-group-zc0r.1600b47811f6b83b  default /api/v1/namespaces/default/events/bootstrap-e2e-minion-group-zc0r.1600b47811f6b83b 5bd2e07e-f9ed-4eed-82d7-74ec5b769a4b 29231 0 2020-03-29 07:04:56 +0000 UTC <nil> <nil> map[] map[] [] []  []} {Node  bootstrap-e2e-minion-group-zc0r bootstrap-e2e-minion-group-zc0r   } AUFSUmountHung Node condition KernelDeadlock is now: True, reason: AUFSUmountHung {kernel-monitor bootstrap-e2e-minion-group-zc0r} 2020-03-29 07:04:56 +0000 UTC 2020-03-29 07:04:56 +0000 UTC 1 Normal 0001-01-01 00:00:00 +0000 UTC nil  nil  }]",
    }
    Event KubeletStart does not exist: [{{ } {bootstrap-e2e-minion-group-hwq3.1600b0f6842060ff  default /api/v1/namespaces/default/events/bootstrap-e2e-minion-group-hwq3.1600b0f6842060ff a5724dc6-b1ae-4752-a832-e6045cde2b7f 28081 0 2020-03-29 06:00:41 +0000 UTC <nil> <nil> map[] map[] [] []  []} {Node  bootstrap-e2e-minion-group-hwq3 bootstrap-e2e-minion-group-hwq3   } KubeletStart Started Kubernetes kubelet. {systemd-monitor bootstrap-e2e-minion-group-hwq3} 2020-03-29 06:00:41 +0000 UTC 2020-03-29 06:30:35 +0000 UTC 5 Warning 0001-01-01 00:00:00 +0000 UTC nil  nil  } {{ } {bootstrap-e2e-minion-group-hwq3.1600b1254c309350  default /api/v1/namespaces/default/events/bootstrap-e2e-minion-group-hwq3.1600b1254c309350 5d55ca7b-e88c-46ba-bf4a-507714108707 27952 0 2020-03-29 06:04:02 +0000 UTC <nil> <nil> map[] map[] [] []  []} {Node  bootstrap-e2e-minion-group-hwq3 37747262-3d5d-4e8b-b235-1dd4f0b51f51   } NodeNotReady Node bootstrap-e2e-minion-group-hwq3 status is now: NodeNotReady {node-controller } 2020-03-29 06:04:02 +0000 UTC 2020-03-29 06:28:27 +0000 UTC 2 Normal 0001-01-01 00:00:00 +0000 UTC nil  nil  } {{ } {bootstrap-e2e-minion-group-hwq3.1600b27aea8e93f9  default /api/v1/namespaces/default/events/bootstrap-e2e-minion-group-hwq3.1600b27aea8e93f9 ade541af-6a57-44fc-a659-6df76e84f3e1 27954 0 2020-03-29 06:28:29 +0000 UTC <nil> <nil> map[] map[] [] []  []} {Node  bootstrap-e2e-minion-group-hwq3 bootstrap-e2e-minion-group-hwq3   } Starting Starting kubelet. {kubelet bootstrap-e2e-minion-group-hwq3} 2020-03-29 06:28:29 +0000 UTC 2020-03-29 06:28:29 +0000 UTC 1 Normal 0001-01-01 00:00:00 +0000 UTC nil  nil  } {{ } {bootstrap-e2e-minion-group-hwq3.1600b27af4e6f232  default /api/v1/namespaces/default/events/bootstrap-e2e-minion-group-hwq3.1600b27af4e6f232 4583c656-b5c8-4d98-b735-f57cf9bc1bf1 27958 0 2020-03-29 06:28:30 +0000 UTC <nil> <nil> map[] map[] [] []  []} {Node  bootstrap-e2e-minion-group-hwq3 bootstrap-e2e-minion-group-hwq3   } NodeHasSufficientMemory Node bootstrap-e2e-minion-group-hwq3 status is now: NodeHasSufficientMemory {kubelet bootstrap-e2e-minion-group-hwq3} 2020-03-29 06:28:29 +0000 UTC 2020-03-29 06:28:30 +0000 UTC 2 Normal 0001-01-01 00:00:00 +0000 UTC nil  nil  } {{ } {bootstrap-e2e-minion-group-hwq3.1600b27af4eb8638  default /api/v1/namespaces/default/events/bootstrap-e2e-minion-group-hwq3.1600b27af4eb8638 85eabb44-da97-4eaa-bf29-b19ccfb713b4 27959 0 2020-03-29 06:28:30 +0000 UTC <nil> <nil> map[] map[] [] []  []} {Node  bootstrap-e2e-minion-group-hwq3 bootstrap-e2e-minion-group-hwq3   } NodeHasNoDiskPressure Node bootstrap-e2e-minion-group-hwq3 status is now: NodeHasNoDiskPressure {kubelet bootstrap-e2e-minion-group-hwq3} 2020-03-29 06:28:29 +0000 UTC 2020-03-29 06:28:30 +0000 UTC 2 Normal 0001-01-01 00:00:00 +0000 UTC nil  nil  } {{ } {bootstrap-e2e-minion-group-hwq3.1600b27af4f014f0  default /api/v1/namespaces/default/events/bootstrap-e2e-minion-group-hwq3.1600b27af4f014f0 91f5c81a-a741-448f-8144-65189e179c04 27960 0 2020-03-29 06:28:30 +0000 UTC <nil> <nil> map[] map[] [] []  []} {Node  bootstrap-e2e-minion-group-hwq3 bootstrap-e2e-minion-group-hwq3   } NodeHasSufficientPID Node bootstrap-e2e-minion-group-hwq3 status is now: NodeHasSufficientPID {kubelet bootstrap-e2e-minion-group-hwq3} 2020-03-29 06:28:29 +0000 UTC 2020-03-29 06:28:30 +0000 UTC 2 Normal 0001-01-01 00:00:00 +0000 UTC nil  nil  } {{ } {bootstrap-e2e-minion-group-hwq3.1600b27afb49cf63  default /api/v1/namespaces/default/events/bootstrap-e2e-minion-group-hwq3.1600b27afb49cf63 e4aeb24a-64ae-40b4-b0d5-94e24eab19e3 27961 0 2020-03-29 06:28:30 +0000 UTC <nil> <nil> map[] map[] [] []  []} {Node  bootstrap-e2e-minion-group-hwq3 bootstrap-e2e-minion-group-hwq3   } NodeNotReady Node bootstrap-e2e-minion-group-hwq3 status is now: NodeNotReady {kubelet bootstrap-e2e-minion-group-hwq3} 2020-03-29 06:28:30 +0000 UTC 2020-03-29 06:28:30 +0000 UTC 1 Normal 0001-01-01 00:00:00 +0000 UTC nil  nil  } {{ } {bootstrap-e2e-minion-group-hwq3.1600b27b148d6f59  default /api/v1/namespaces/default/events/bootstrap-e2e-minion-group-hwq3.1600b27b148d6f59 2e137161-abe3-4dbe-92e5-09ab1452e411 27962 0 2020-03-29 06:28:30 +0000 UTC <nil> <nil> map[] map[] [] []  []} {Node  bootstrap-e2e-minion-group-hwq3 bootstrap-e2e-minion-group-hwq3   } NodeAllocatableEnforced Updated Node Allocatable limit across pods {kubelet bootstrap-e2e-minion-group-hwq3} 2020-03-29 06:28:30 +0000 UTC 2020-03-29 06:28:30 +0000 UTC 1 Normal 0001-01-01 00:00:00 +0000 UTC nil  nil  } {{ } {bootstrap-e2e-minion-group-hwq3.1600b27b16028d95  default /api/v1/namespaces/default/events/bootstrap-e2e-minion-group-hwq3.1600b27b16028d95 0702ccdb-9574-4211-b2d8-e252c6ee3899 27963 0 2020-03-29 06:28:30 +0000 UTC <nil> <nil> map[] map[] [] []  []} {Node  bootstrap-e2e-minion-group-hwq3 bootstrap-e2e-minion-group-hwq3   } NodeReady Node bootstrap-e2e-minion-group-hwq3 status is now: NodeReady {kubelet bootstrap-e2e-minion-group-hwq3} 2020-03-29 06:28:30 +0000 UTC 2020-03-29 06:28:30 +0000 UTC 1 Normal 0001-01-01 00:00:00 +0000 UTC nil  nil  } {{ } {bootstrap-e2e-minion-group-hwq3.1600b298382f2316  default /api/v1/namespaces/default/events/bootstrap-e2e-minion-group-hwq3.1600b298382f2316 86f8728c-3166-48e1-810d-7a0246eb99d8 28082 0 2020-03-29 06:30:35 +0000 UTC <nil> <nil> map[] map[] [] []  []} {Node  bootstrap-e2e-minion-group-hwq3 bootstrap-e2e-minion-group-hwq3   } Starting Starting kubelet. {kubelet bootstrap-e2e-minion-group-hwq3} 2020-03-29 06:30:35 +0000 UTC 2020-03-29 06:30:35 +0000 UTC 1 Normal 0001-01-01 00:00:00 +0000 UTC nil  nil  } {{ } {bootstrap-e2e-minion-group-hwq3.1600b29841aaba43  default /api/v1/namespaces/default/events/bootstrap-e2e-minion-group-hwq3.1600b29841aaba43 36365b62-f9bd-4d6e-974c-b1fd3fac6d4b 28083 0 2020-03-29 06:30:35 +0000 UTC <nil> <nil> map[] map[] [] []  []} {Node  bootstrap-e2e-minion-group-hwq3 bootstrap-e2e-minion-group-hwq3   } NodeHasSufficientMemory Node bootstrap-e2e-minion-group-hwq3 status is now: NodeHasSufficientMemory {kubelet bootstrap-e2e-minion-group-hwq3} 2020-03-29 06:30:35 +0000 UTC 2020-03-29 06:30:35 +0000 UTC 1 Normal 0001-01-01 00:00:00 +0000 UTC nil  nil  } {{ } {bootstrap-e2e-minion-group-hwq3.1600b29841aaf543  default /api/v1/namespaces/default/events/bootstrap-e2e-minion-group-hwq3.1600b29841aaf543 6e67e250-0d60-4361-903d-b055a9530ca7 28084 0 2020-03-29 06:30:35 +0000 UTC <nil> <nil> map[] map[] [] []  []} {Node  bootstrap-e2e-minion-group-hwq3 bootstrap-e2e-minion-group-hwq3   } NodeHasNoDiskPressure Node bootstrap-e2e-minion-group-hwq3 status is now: NodeHasNoDiskPressure {kubelet bootstrap-e2e-minion-group-hwq3} 2020-03-29 06:30:35 +0000 UTC 2020-03-29 06:30:35 +0000 UTC 1 Normal 0001-01-01 00:00:00 +0000 UTC nil  nil  } {{ } {bootstrap-e2e-minion-group-hwq3.1600b29841ab1d0f  default /api/v1/namespaces/default/events/bootstrap-e2e-minion-group-hwq3.1600b29841ab1d0f f2f088e0-bbbd-4d37-8d96-73b9c167791f 28085 0 2020-03-29 06:30:35 +0000 UTC <nil> <nil> map[] map[] [] []  []} {Node  bootstrap-e2e-minion-group-hwq3 bootstrap-e2e-minion-group-hwq3   } NodeHasSufficientPID Node bootstrap-e2e-minion-group-hwq3 status is now: NodeHasSufficientPID {kubelet bootstrap-e2e-minion-group-hwq3} 2020-03-29 06:30:35 +0000 UTC 2020-03-29 06:30:35 +0000 UTC 1 Normal 0001-01-01 00:00:00 +0000 UTC nil  nil  } {{ } {bootstrap-e2e-minion-group-hwq3.1600b29844f98f92  default /api/v1/namespaces/default/events/bootstrap-e2e-minion-group-hwq3.1600b29844f98f92 b5918b5c-cbd5-45e7-a772-302f27ff4c98 28086 0 2020-03-29 06:30:35 +0000 UTC <nil> <nil> map[] map[] [] []  []} {Node  bootstrap-e2e-minion-group-hwq3 bootstrap-e2e-minion-group-hwq3   } NodeNotReady Node bootstrap-e2e-minion-group-hwq3 status is now: NodeNotReady {kubelet bootstrap-e2e-minion-group-hwq3} 2020-03-29 06:30:35 +0000 UTC 2020-03-29 06:30:35 +0000 UTC 1 Normal 0001-01-01 00:00:00 +0000 UTC nil  nil  } {{ } {bootstrap-e2e-minion-group-hwq3.1600b298544a1e27  default /api/v1/namespaces/default/events/bootstrap-e2e-minion-group-hwq3.1600b298544a1e27 fbac06e9-816e-413a-b9f2-3c0cb479f5e1 28087 0 2020-03-29 06:30:36 +0000 UTC <nil> <nil> map[] map[] [] []  []} {Node  bootstrap-e2e-minion-group-hwq3 bootstrap-e2e-minion-group-hwq3   } NodeAllocatableEnforced Updated Node Allocatable limit across pods {kubelet bootstrap-e2e-minion-group-hwq3} 2020-03-29 06:30:36 +0000 UTC 2020-03-29 06:30:36 +0000 UTC 1 Normal 0001-01-01 00:00:00 +0000 UTC nil  nil  } {{ } {bootstrap-e2e-minion-group-hwq3.1600b29855d25ca8  default /api/v1/namespaces/default/events/bootstrap-e2e-minion-group-hwq3.1600b29855d25ca8 25e0457a-f3f9-410f-b836-97a293053868 28088 0 2020-03-29 06:30:36 +0000 UTC <nil> <nil> map[] map[] [] []  []} {Node  bootstrap-e2e-minion-group-hwq3 bootstrap-e2e-minion-group-hwq3   } NodeReady Node bootstrap-e2e-minion-group-hwq3 status is now: NodeReady {kubelet bootstrap-e2e-minion-group-hwq3} 2020-03-29 06:30:36 +0000 UTC 2020-03-29 06:30:36 +0000 UTC 1 Normal 0001-01-01 00:00:00 +0000 UTC nil  nil  } {{ } {bootstrap-e2e-minion-group-hwq3.1600b476e0fc4684  default /api/v1/namespaces/default/events/bootstrap-e2e-minion-group-hwq3.1600b476e0fc4684 9eadf897-405d-48e8-8c68-5e1221554883 29228 0 2020-03-29 07:04:51 +0000 UTC <nil> <nil> map[] map[] [] []  []} {Node  bootstrap-e2e-minion-group-hwq3 bootstrap-e2e-minion-group-hwq3   } TaskHung kernel: INFO: task umount.aufs:21568 blocked for more than 120 seconds. {kernel-monitor bootstrap-e2e-minion-group-hwq3} 2020-03-29 07:04:51 +0000 UTC 2020-03-29 07:04:51 +0000 UTC 1 Warning 0001-01-01 00:00:00 +0000 UTC nil  nil  } {{ } {bootstrap-e2e-minion-group-hwq3.1600b476e0fc667d  default /api/v1/namespaces/default/events/bootstrap-e2e-minion-group-hwq3.1600b476e0fc667d ff9ec0b6-b10d-42ce-9f45-1d5e8eefac49 29227 0 2020-03-29 07:04:51 +0000 UTC <nil> <nil> map[] map[] [] []  []} {Node  bootstrap-e2e-minion-group-hwq3 bootstrap-e2e-minion-group-hwq3   } AUFSUmountHung Node condition KernelDeadlock is now: True, reason: AUFSUmountHung {kernel-monitor bootstrap-e2e-minion-group-hwq3} 2020-03-29 07:04:51 +0000 UTC 2020-03-29 07:04:51 +0000 UTC 1 Normal 0001-01-01 00:00:00 +0000 UTC nil  nil  } {{ } {bootstrap-e2e-minion-group-pn0s.1600b47783fb5cc0  default /api/v1/namespaces/default/events/bootstrap-e2e-minion-group-pn0s.1600b47783fb5cc0 95b1f0ae-db5d-42db-a8da-2890a6442575 29230 0 2020-03-29 07:04:54 +0000 UTC <nil> <nil> map[] map[] [] []  []} {Node  bootstrap-e2e-minion-group-pn0s bootstrap-e2e-minion-group-pn0s   } TaskHung kernel: INFO: task umount.aufs:21568 blocked for more than 120 seconds. {kernel-monitor bootstrap-e2e-minion-group-pn0s} 2020-03-29 07:04:54 +0000 UTC 2020-03-29 07:04:54 +0000 UTC 1 Warning 0001-01-01 00:00:00 +0000 UTC nil  nil  } {{ } {bootstrap-e2e-minion-group-pn0s.1600b47783fb852d  default /api/v1/namespaces/default/events/bootstrap-e2e-minion-group-pn0s.1600b47783fb852d 2345c5b6-f1b6-4df9-8458-ce9242a46619 29229 0 2020-03-29 07:04:54 +0000 UTC <nil> <nil> map[] map[] [] []  []} {Node  bootstrap-e2e-minion-group-pn0s bootstrap-e2e-minion-group-pn0s   } AUFSUmountHung Node condition KernelDeadlock is now: True, reason: AUFSUmountHung {kernel-monitor bootstrap-e2e-minion-group-pn0s} 2020-03-29 07:04:54 +0000 UTC 2020-03-29 07:04:54 +0000 UTC 1 Normal 0001-01-01 00:00:00 +0000 UTC nil  nil  } {{ } {bootstrap-e2e-minion-group-zc0r.1600b47811f69b71  default /api/v1/namespaces/default/events/bootstrap-e2e-minion-group-zc0r.1600b47811f69b71 4b230369-9c41-41bc-a2dc-ff9650e7eb28 29232 0 2020-03-29 07:04:56 +0000 UTC <nil> <nil> map[] map[] [] []  []} {Node  bootstrap-e2e-minion-group-zc0r bootstrap-e2e-minion-group-zc0r   } TaskHung kernel: INFO: task umount.aufs:21568 blocked for more than 120 seconds. {kernel-monitor bootstrap-e2e-minion-group-zc0r} 2020-03-29 07:04:56 +0000 UTC 2020-03-29 07:04:56 +0000 UTC 1 Warning 0001-01-01 00:00:00 +0000 UTC nil  nil  } {{ } {bootstrap-e2e-minion-group-zc0r.1600b47811f6b83b  default /api/v1/namespaces/default/events/bootstrap-e2e-minion-group-zc0r.1600b47811f6b83b 5bd2e07e-f9ed-4eed-82d7-74ec5b769a4b 29231 0 2020-03-29 07:04:56 +0000 UTC <nil> <nil> map[] map[] [] []  []} {Node  bootstrap-e2e-minion-group-zc0r bootstrap-e2e-minion-group-zc0r   } AUFSUmountHung Node condition KernelDeadlock is now: True, reason: AUFSUmountHung {kernel-monitor bootstrap-e2e-minion-group-zc0r} 2020-03-29 07:04:56 +0000 UTC 2020-03-29 07:04:56 +0000 UTC 1 Normal 0001-01-01 00:00:00 +0000 UTC nil  nil  }]
/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/node/node_problem_detector.go:141
				
				Click to see stdout/stderrfrom junit_01.xml

Filter through log files | View test history on testgrid


Kubernetes e2e suite [sig-cluster-lifecycle] Nodes [Disruptive] Resize [Slow] should be able to delete nodes 4m46s

go run hack/e2e.go -v --test --test_args='--ginkgo.focus=Kubernetes\se2e\ssuite\s\[sig\-cluster\-lifecycle\]\sNodes\s\[Disruptive\]\sResize\s\[Slow\]\sshould\sbe\sable\sto\sdelete\snodes$'
/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/lifecycle/resize_nodes.go:112
Mar 28 22:05:11.102: Unexpected error:
    <*errors.errorString | 0xc003ae40a0>: {
        s: "failed to wait for pods responding: pod with UID 32eb23e6-d939-4378-b9a3-51e16a6fbf57 is no longer a member of the replica set.  Must have been restarted for some reason.  Current replica set: &PodList{ListMeta:{/api/v1/namespaces/resize-nodes-1123/pods 62724  <nil>},Items:[]Pod{Pod{ObjectMeta:{my-hostname-delete-node-82mxl my-hostname-delete-node- resize-nodes-1123 /api/v1/namespaces/resize-nodes-1123/pods/my-hostname-delete-node-82mxl ab14664e-040d-4e4b-9e30-9c616ff117d0 62148 0 2020-03-28 22:02:03 +0000 UTC <nil> <nil> map[name:my-hostname-delete-node] map[] [{v1 ReplicationController my-hostname-delete-node 441d8555-3948-4f41-bc4b-110f4c2e6a96 0xc0049540be 0xc0049540bf}] []  []},Spec:PodSpec{Volumes:[]Volume{Volume{Name:default-token-t6xpg,VolumeSource:VolumeSource{HostPath:nil,EmptyDir:nil,GCEPersistentDisk:nil,AWSElasticBlockStore:nil,GitRepo:nil,Secret:&SecretVolumeSource{SecretName:default-token-t6xpg,Items:[]KeyToPath{},DefaultMode:*420,Optional:nil,},NFS:nil,ISCSI:nil,Glusterfs:nil,PersistentVolumeClaim:nil,RBD:nil,FlexVolume:nil,Cinder:nil,CephFS:nil,Flocker:nil,DownwardAPI:nil,FC:nil,AzureFile:nil,ConfigMap:nil,VsphereVolume:nil,Quobyte:nil,AzureDisk:nil,PhotonPersistentDisk:nil,PortworxVolume:nil,ScaleIO:nil,Projected:nil,StorageOS:nil,CSI:nil,},},},Containers:[]Container{Container{Name:my-hostname-delete-node,Image:gcr.io/kubernetes-e2e-test-images/agnhost:2.6,Command:[],Args:[serve-hostname],WorkingDir:,Ports:[]ContainerPort{ContainerPort{Name:,HostPort:0,ContainerPort:9376,Protocol:TCP,HostIP:,},},Env:[]EnvVar{},Resources:ResourceRequirements{Limits:ResourceList{},Requests:ResourceList{},},VolumeMounts:[]VolumeMount{VolumeMount{Name:default-token-t6xpg,ReadOnly:true,MountPath:/var/run/secrets/kubernetes.io/serviceaccount,SubPath:,MountPropagation:nil,SubPathExpr:,},},LivenessProbe:nil,ReadinessProbe:nil,Lifecycle:nil,TerminationMessagePath:/dev/termination-log,ImagePullPolicy:IfNotPresent,SecurityContext:nil,Stdin:false,StdinOnce:false,TTY:false,EnvFrom:[]EnvFromSource{},TerminationMessagePolicy:File,VolumeDevices:[]VolumeDevice{},StartupProbe:nil,},},RestartPolicy:Always,TerminationGracePeriodSeconds:*0,ActiveDeadlineSeconds:nil,DNSPolicy:ClusterFirst,NodeSelector:map[string]string{},ServiceAccountName:default,DeprecatedServiceAccount:default,NodeName:bootstrap-e2e-minion-group-pn0s,HostNetwork:false,HostPID:false,HostIPC:false,SecurityContext:&PodSecurityContext{SELinuxOptions:nil,RunAsUser:nil,RunAsNonRoot:nil,SupplementalGroups:[],FSGroup:nil,RunAsGroup:nil,Sysctls:[]Sysctl{},WindowsOptions:nil,},ImagePullSecrets:[]LocalObjectReference{},Hostname:,Subdomain:,Affinity:nil,SchedulerName:default-scheduler,InitContainers:[]Container{},AutomountServiceAccountToken:nil,Tolerations:[]Toleration{Toleration{Key:node.kubernetes.io/not-ready,Operator:Exists,Value:,Effect:NoExecute,TolerationSeconds:*300,},Toleration{Key:node.kubernetes.io/unreachable,Operator:Exists,Value:,Effect:NoExecute,TolerationSeconds:*300,},},HostAliases:[]HostAlias{},PriorityClassName:,Priority:*0,DNSConfig:nil,ShareProcessNamespace:nil,ReadinessGates:[]PodReadinessGate{},RuntimeClassName:nil,EnableServiceLinks:*true,PreemptionPolicy:nil,Overhead:ResourceList{},TopologySpreadConstraints:[]TopologySpreadConstraint{},EphemeralContainers:[]EphemeralContainer{},},Status:PodStatus{Phase:Running,Conditions:[]PodCondition{PodCondition{Type:Initialized,Status:True,LastProbeTime:0001-01-01 00:00:00 +0000 UTC,LastTransitionTime:2020-03-28 22:02:03 +0000 UTC,Reason:,Message:,},PodCondition{Type:Ready,Status:True,LastProbeTime:0001-01-01 00:00:00 +0000 UTC,LastTransitionTime:2020-03-28 22:02:06 +0000 UTC,Reason:,Message:,},PodCondition{Type:ContainersReady,Status:True,LastProbeTime:0001-01-01 00:00:00 +0000 UTC,LastTransitionTime:2020-03-28 22:02:06 +0000 UTC,Reason:,Message:,},PodCondition{Type:PodScheduled,Status:True,LastProbeTime:0001-01-01 00:00:00 +0000 UTC,LastTransitionTime:2020-03-28 22:02:03 +0000 UTC,Reason:,Message:,},},Message:,Reason:,HostIP:10.138.0.3,PodIP:10.64.0.174,StartTime:2020-03-28 22:02:03 +0000 UTC,ContainerStatuses:[]ContainerStatus{ContainerStatus{Name:my-hostname-delete-node,State:ContainerState{Waiting:nil,Running:&ContainerStateRunning{StartedAt:2020-03-28 22:02:05 +0000 UTC,},Terminated:nil,},LastTerminationState:ContainerState{Waiting:nil,Running:nil,Terminated:nil,},Ready:true,RestartCount:0,Image:gcr.io/k8s-authenticated-test/agnhost:2.6,ImageID:docker-pullable://gcr.io/k8s-authenticated-test/agnhost@sha256:4057a5580c7b59c4fe10d8ab2732c9dec35eea80fd41f7bafc7bd5acc7edf727,ContainerID:docker://b0286a369b10cd1bb2662eb397eeedc305731c5f576649c127e6757b4197dc08,Started:*true,},},QOSClass:BestEffort,InitContainerStatuses:[]ContainerStatus{},NominatedNodeName:,PodIPs:[]PodIP{PodIP{IP:10.64.0.174,},},EphemeralContainerStatuses:[]ContainerStatus{},},},Pod{ObjectMeta:{my-hostname-delete-node-8jzps my-hostname-delete-node- resize-nodes-1123 /api/v1/namespaces/resize-nodes-1123/pods/my-hostname-delete-node-8jzps 8334e35a-bcb9-4139-a321-7c769b115cd2 62143 0 2020-03-28 22:02:03 +0000 UTC <nil> <nil> map[name:my-hostname-delete-node] map[] [{v1 ReplicationController my-hostname-delete-node 441d8555-3948-4f41-bc4b-110f4c2e6a96 0xc004954206 0xc004954207}] []  []},Spec:PodSpec{Volumes:[]Volume{Volume{Name:default-token-t6xpg,VolumeSource:VolumeSource{HostPath:nil,EmptyDir:nil,GCEPersistentDisk:nil,AWSElasticBlockStore:nil,GitRepo:nil,Secret:&SecretVolumeSource{SecretName:default-token-t6xpg,Items:[]KeyToPath{},DefaultMode:*420,Optional:nil,},NFS:nil,ISCSI:nil,Glusterfs:nil,PersistentVolumeClaim:nil,RBD:nil,FlexVolume:nil,Cinder:nil,CephFS:nil,Flocker:nil,DownwardAPI:nil,FC:nil,AzureFile:nil,ConfigMap:nil,VsphereVolume:nil,Quobyte:nil,AzureDisk:nil,PhotonPersistentDisk:nil,PortworxVolume:nil,ScaleIO:nil,Projected:nil,StorageOS:nil,CSI:nil,},},},Containers:[]Container{Container{Name:my-hostname-delete-node,Image:gcr.io/kubernetes-e2e-test-images/agnhost:2.6,Command:[],Args:[serve-hostname],WorkingDir:,Ports:[]ContainerPort{ContainerPort{Name:,HostPort:0,ContainerPort:9376,Protocol:TCP,HostIP:,},},Env:[]EnvVar{},Resources:ResourceRequirements{Limits:ResourceList{},Requests:ResourceList{},},VolumeMounts:[]VolumeMount{VolumeMount{Name:default-token-t6xpg,ReadOnly:true,MountPath:/var/run/secrets/kubernetes.io/serviceaccount,SubPath:,MountPropagation:nil,SubPathExpr:,},},LivenessProbe:nil,ReadinessProbe:nil,Lifecycle:nil,TerminationMessagePath:/dev/termination-log,ImagePullPolicy:IfNotPresent,SecurityContext:nil,Stdin:false,StdinOnce:false,TTY:false,EnvFrom:[]EnvFromSource{},TerminationMessagePolicy:File,VolumeDevices:[]VolumeDevice{},StartupProbe:nil,},},RestartPolicy:Always,TerminationGracePeriodSeconds:*0,ActiveDeadlineSeconds:nil,DNSPolicy:ClusterFirst,NodeSelector:map[string]string{},ServiceAccountName:default,DeprecatedServiceAccount:default,NodeName:bootstrap-e2e-minion-group-nt4p,HostNetwork:false,HostPID:false,HostIPC:false,SecurityContext:&PodSecurityContext{SELinuxOptions:nil,RunAsUser:nil,RunAsNonRoot:nil,SupplementalGroups:[],FSGroup:nil,RunAsGroup:nil,Sysctls:[]Sysctl{},WindowsOptions:nil,},ImagePullSecrets:[]LocalObjectReference{},Hostname:,Subdomain:,Affinity:nil,SchedulerName:default-scheduler,InitContainers:[]Container{},AutomountServiceAccountToken:nil,Tolerations:[]Toleration{Toleration{Key:node.kubernetes.io/not-ready,Operator:Exists,Value:,Effect:NoExecute,TolerationSeconds:*300,},Toleration{Key:node.kubernetes.io/unreachable,Operator:Exists,Value:,Effect:NoExecute,TolerationSeconds:*300,},},HostAliases:[]HostAlias{},PriorityClassName:,Priority:*0,DNSConfig:nil,ShareProcessNamespace:nil,ReadinessGates:[]PodReadinessGate{},RuntimeClassName:nil,EnableServiceLinks:*true,PreemptionPolicy:nil,Overhead:ResourceList{},TopologySpreadConstraints:[]TopologySpreadConstraint{},EphemeralContainers:[]EphemeralContainer{},},Status:PodStatus{Phase:Running,Conditions:[]PodCondition{PodCondition{Type:Initialized,Status:True,LastProbeTime:0001-01-01 00:00:00 +0000 UTC,LastTransitionTime:2020-03-28 22:02:03 +0000 UTC,Reason:,Message:,},PodCondition{Type:Ready,Status:True,LastProbeTime:0001-01-01 00:00:00 +0000 UTC,LastTransitionTime:2020-03-28 22:02:05 +0000 UTC,Reason:,Message:,},PodCondition{Type:ContainersReady,Status:True,LastProbeTime:0001-01-01 00:00:00 +0000 UTC,LastTransitionTime:2020-03-28 22:02:05 +0000 UTC,Reason:,Message:,},PodCondition{Type:PodScheduled,Status:True,LastProbeTime:0001-01-01 00:00:00 +0000 UTC,LastTransitionTime:2020-03-28 22:02:03 +0000 UTC,Reason:,Message:,},},Message:,Reason:,HostIP:10.138.0.4,PodIP:10.64.1.212,StartTime:2020-03-28 22:02:03 +0000 UTC,ContainerStatuses:[]ContainerStatus{ContainerStatus{Name:my-hostname-delete-node,State:ContainerState{Waiting:nil,Running:&ContainerStateRunning{StartedAt:2020-03-28 22:02:04 +0000 UTC,},Terminated:nil,},LastTerminationState:ContainerState{Waiting:nil,Running:nil,Terminated:nil,},Ready:true,RestartCount:0,Image:gcr.io/kubernetes-e2e-test-images/agnhost:2.6,ImageID:docker-pullable://gcr.io/kubernetes-e2e-test-images/agnhost@sha256:4057a5580c7b59c4fe10d8ab2732c9dec35eea80fd41f7bafc7bd5acc7edf727,ContainerID:docker://388be5536b567ad2cf2e54c0c4d884c1fefa745f1fb9cd131e1028c539888302,Started:*true,},},QOSClass:BestEffort,InitContainerStatuses:[]ContainerStatus{},NominatedNodeName:,PodIPs:[]PodIP{PodIP{IP:10.64.1.212,},},EphemeralContainerStatuses:[]ContainerStatus{},},},Pod{ObjectMeta:{my-hostname-delete-node-mv8sr my-hostname-delete-node- resize-nodes-1123 /api/v1/namespaces/resize-nodes-1123/pods/my-hostname-delete-node-mv8sr 2492fe32-70ed-4cc1-a200-ec5296f8c2b8 62640 0 2020-03-28 22:04:40 +0000 UTC <nil> <nil> map[name:my-hostname-delete-node] map[] [{v1 ReplicationController my-hostname-delete-node 441d8555-3948-4f41-bc4b-110f4c2e6a96 0xc004954346 0xc004954347}] []  []},Spec:PodSpec{Volumes:[]Volume{Volume{Name:default-token-t6xpg,VolumeSource:VolumeSource{HostPath:nil,EmptyDir:nil,GCEPersistentDisk:nil,AWSElasticBlockStore:nil,GitRepo:nil,Secret:&SecretVolumeSource{SecretName:default-token-t6xpg,Items:[]KeyToPath{},DefaultMode:*420,Optional:nil,},NFS:nil,ISCSI:nil,Glusterfs:nil,PersistentVolumeClaim:nil,RBD:nil,FlexVolume:nil,Cinder:nil,CephFS:nil,Flocker:nil,DownwardAPI:nil,FC:nil,AzureFile:nil,ConfigMap:nil,VsphereVolume:nil,Quobyte:nil,AzureDisk:nil,PhotonPersistentDisk:nil,PortworxVolume:nil,ScaleIO:nil,Projected:nil,StorageOS:nil,CSI:nil,},},},Containers:[]Container{Container{Name:my-hostname-delete-node,Image:gcr.io/kubernetes-e2e-test-images/agnhost:2.6,Command:[],Args:[serve-hostname],WorkingDir:,Ports:[]ContainerPort{ContainerPort{Name:,HostPort:0,ContainerPort:9376,Protocol:TCP,HostIP:,},},Env:[]EnvVar{},Resources:ResourceRequirements{Limits:ResourceList{},Requests:ResourceList{},},VolumeMounts:[]VolumeMount{VolumeMount{Name:default-token-t6xpg,ReadOnly:true,MountPath:/var/run/secrets/kubernetes.io/serviceaccount,SubPath:,MountPropagation:nil,SubPathExpr:,},},LivenessProbe:nil,ReadinessProbe:nil,Lifecycle:nil,TerminationMessagePath:/dev/termination-log,ImagePullPolicy:IfNotPresent,SecurityContext:nil,Stdin:false,StdinOnce:false,TTY:false,EnvFrom:[]EnvFromSource{},TerminationMessagePolicy:File,VolumeDevices:[]VolumeDevice{},StartupProbe:nil,},},RestartPolicy:Always,TerminationGracePeriodSeconds:*0,ActiveDeadlineSeconds:nil,DNSPolicy:ClusterFirst,NodeSelector:map[string]string{},ServiceAccountName:default,DeprecatedServiceAccount:default,NodeName:bootstrap-e2e-minion-group-pn0s,HostNetwork:false,HostPID:false,HostIPC:false,SecurityContext:&PodSecurityContext{SELinuxOptions:nil,RunAsUser:nil,RunAsNonRoot:nil,SupplementalGroups:[],FSGroup:nil,RunAsGroup:nil,Sysctls:[]Sysctl{},WindowsOptions:nil,},ImagePullSecrets:[]LocalObjectReference{},Hostname:,Subdomain:,Affinity:nil,SchedulerName:default-scheduler,InitContainers:[]Container{},AutomountServiceAccountToken:nil,Tolerations:[]Toleration{Toleration{Key:node.kubernetes.io/not-ready,Operator:Exists,Value:,Effect:NoExecute,TolerationSeconds:*300,},Toleration{Key:node.kubernetes.io/unreachable,Operator:Exists,Value:,Effect:NoExecute,TolerationSeconds:*300,},},HostAliases:[]HostAlias{},PriorityClassName:,Priority:*0,DNSConfig:nil,ShareProcessNamespace:nil,ReadinessGates:[]PodReadinessGate{},RuntimeClassName:nil,EnableServiceLinks:*true,PreemptionPolicy:nil,Overhead:ResourceList{},TopologySpreadConstraints:[]TopologySpreadConstraint{},EphemeralContainers:[]EphemeralContainer{},},Status:PodStatus{Phase:Running,Conditions:[]PodCondition{PodCondition{Type:Initialized,Status:True,LastProbeTime:0001-01-01 00:00:00 +0000 UTC,LastTransitionTime:2020-03-28 22:04:40 +0000 UTC,Reason:,Message:,},PodCondition{Type:Ready,Status:True,LastProbeTime:0001-01-01 00:00:00 +0000 UTC,LastTransitionTime:2020-03-28 22:04:42 +0000 UTC,Reason:,Message:,},PodCondition{Type:ContainersReady,Status:True,LastProbeTime:0001-01-01 00:00:00 +0000 UTC,LastTransitionTime:2020-03-28 22:04:42 +0000 UTC,Reason:,Message:,},PodCondition{Type:PodScheduled,Status:True,LastProbeTime:0001-01-01 00:00:00 +0000 UTC,LastTransitionTime:2020-03-28 22:04:40 +0000 UTC,Reason:,Message:,},},Message:,Reason:,HostIP:10.138.0.3,PodIP:10.64.0.175,StartTime:2020-03-28 22:04:40 +0000 UTC,ContainerStatuses:[]ContainerStatus{ContainerStatus{Name:my-hostname-delete-node,State:ContainerState{Waiting:nil,Running:&ContainerStateRunning{StartedAt:2020-03-28 22:04:41 +0000 UTC,},Terminated:nil,},LastTerminationState:ContainerState{Waiting:nil,Running:nil,Terminated:nil,},Ready:true,RestartCount:0,Image:gcr.io/k8s-authenticated-test/agnhost:2.6,ImageID:docker-pullable://gcr.io/k8s-authenticated-test/agnhost@sha256:4057a5580c7b59c4fe10d8ab2732c9dec35eea80fd41f7bafc7bd5acc7edf727,ContainerID:docker://6855d469a6f9767e0d15b34e35d063f3ab029494788257d8adb50d4f638870fc,Started:*true,},},QOSClass:BestEffort,InitContainerStatuses:[]ContainerStatus{},NominatedNodeName:,PodIPs:[]PodIP{PodIP{IP:10.64.0.175,},},EphemeralContainerStatuses:[]ContainerStatus{},},},},}",
    }
    failed to wait for pods responding: pod with UID 32eb23e6-d939-4378-b9a3-51e16a6fbf57 is no longer a member of the replica set.  Must have been restarted for some reason.  Current replica set: &PodList{ListMeta:{/api/v1/namespaces/resize-nodes-1123/pods 62724  <nil>},Items:[]Pod{Pod{ObjectMeta:{my-hostname-delete-node-82mxl my-hostname-delete-node- resize-nodes-1123 /api/v1/namespaces/resize-nodes-1123/pods/my-hostname-delete-node-82mxl ab14664e-040d-4e4b-9e30-9c616ff117d0 62148 0 2020-03-28 22:02:03 +0000 UTC <nil> <nil> map[name:my-hostname-delete-node] map[] [{v1 ReplicationController my-hostname-delete-node 441d8555-3948-4f41-bc4b-110f4c2e6a96 0xc0049540be 0xc0049540bf}] []  []},Spec:PodSpec{Volumes:[]Volume{Volume{Name:default-token-t6xpg,VolumeSource:VolumeSource{HostPath:nil,EmptyDir:nil,GCEPersistentDisk:nil,AWSElasticBlockStore:nil,GitRepo:nil,Secret:&SecretVolumeSource{SecretName:default-token-t6xpg,Items:[]KeyToPath{},DefaultMode:*420,Optional:nil,},NFS:nil,ISCSI:nil,Glusterfs:nil,PersistentVolumeClaim:nil,RBD:nil,FlexVolume:nil,Cinder:nil,CephFS:nil,Flocker:nil,DownwardAPI:nil,FC:nil,AzureFile:nil,ConfigMap:nil,VsphereVolume:nil,Quobyte:nil,AzureDisk:nil,PhotonPersistentDisk:nil,PortworxVolume:nil,ScaleIO:nil,Projected:nil,StorageOS:nil,CSI:nil,},},},Containers:[]Container{Container{Name:my-hostname-delete-node,Image:gcr.io/kubernetes-e2e-test-images/agnhost:2.6,Command:[],Args:[serve-hostname],WorkingDir:,Ports:[]ContainerPort{ContainerPort{Name:,HostPort:0,ContainerPort:9376,Protocol:TCP,HostIP:,},},Env:[]EnvVar{},Resources:ResourceRequirements{Limits:ResourceList{},Requests:ResourceList{},},VolumeMounts:[]VolumeMount{VolumeMount{Name:default-token-t6xpg,ReadOnly:true,MountPath:/var/run/secrets/kubernetes.io/serviceaccount,SubPath:,MountPropagation:nil,SubPathExpr:,},},LivenessProbe:nil,ReadinessProbe:nil,Lifecycle:nil,TerminationMessagePath:/dev/termination-log,ImagePullPolicy:IfNotPresent,SecurityContext:nil,Stdin:false,StdinOnce:false,TTY:false,EnvFrom:[]EnvFromSource{},TerminationMessagePolicy:File,VolumeDevices:[]VolumeDevice{},StartupProbe:nil,},},RestartPolicy:Always,TerminationGracePeriodSeconds:*0,ActiveDeadlineSeconds:nil,DNSPolicy:ClusterFirst,NodeSelector:map[string]string{},ServiceAccountName:default,DeprecatedServiceAccount:default,NodeName:bootstrap-e2e-minion-group-pn0s,HostNetwork:false,HostPID:false,HostIPC:false,SecurityContext:&PodSecurityContext{SELinuxOptions:nil,RunAsUser:nil,RunAsNonRoot:nil,SupplementalGroups:[],FSGroup:nil,RunAsGroup:nil,Sysctls:[]Sysctl{},WindowsOptions:nil,},ImagePullSecrets:[]LocalObjectReference{},Hostname:,Subdomain:,Affinity:nil,SchedulerName:default-scheduler,InitContainers:[]Container{},AutomountServiceAccountToken:nil,Tolerations:[]Toleration{Toleration{Key:node.kubernetes.io/not-ready,Operator:Exists,Value:,Effect:NoExecute,TolerationSeconds:*300,},Toleration{Key:node.kubernetes.io/unreachable,Operator:Exists,Value:,Effect:NoExecute,TolerationSeconds:*300,},},HostAliases:[]HostAlias{},PriorityClassName:,Priority:*0,DNSConfig:nil,ShareProcessNamespace:nil,ReadinessGates:[]PodReadinessGate{},RuntimeClassName:nil,EnableServiceLinks:*true,PreemptionPolicy:nil,Overhead:ResourceList{},TopologySpreadConstraints:[]TopologySpreadConstraint{},EphemeralContainers:[]EphemeralContainer{},},Status:PodStatus{Phase:Running,Conditions:[]PodCondition{PodCondition{Type:Initialized,Status:True,LastProbeTime:0001-01-01 00:00:00 +0000 UTC,LastTransitionTime:2020-03-28 22:02:03 +0000 UTC,Reason:,Message:,},PodCondition{Type:Ready,Status:True,LastProbeTime:0001-01-01 00:00:00 +0000 UTC,LastTransitionTime:2020-03-28 22:02:06 +0000 UTC,Reason:,Message:,},PodCondition{Type:ContainersReady,Status:True,LastProbeTime:0001-01-01 00:00:00 +0000 UTC,LastTransitionTime:2020-03-28 22:02:06 +0000 UTC,Reason:,Message:,},PodCondition{Type:PodScheduled,Status:True,LastProbeTime:0001-01-01 00:00:00 +0000 UTC,LastTransitionTime:2020-03-28 22:02:03 +0000 UTC,Reason:,Message:,},},Message:,Reason:,HostIP:10.138.0.3,PodIP:10.64.0.174,StartTime:2020-03-28 22:02:03 +0000 UTC,ContainerStatuses:[]ContainerStatus{ContainerStatus{Name:my-hostname-delete-node,State:ContainerState{Waiting:nil,Running:&ContainerStateRunning{StartedAt:2020-03-28 22:02:05 +0000 UTC,},Terminated:nil,},LastTerminationState:ContainerState{Waiting:nil,Running:nil,Terminated:nil,},Ready:true,RestartCount:0,Image:gcr.io/k8s-authenticated-test/agnhost:2.6,ImageID:docker-pullable://gcr.io/k8s-authenticated-test/agnhost@sha256:4057a5580c7b59c4fe10d8ab2732c9dec35eea80fd41f7bafc7bd5acc7edf727,ContainerID:docker://b0286a369b10cd1bb2662eb397eeedc305731c5f576649c127e6757b4197dc08,Started:*true,},},QOSClass:BestEffort,InitContainerStatuses:[]ContainerStatus{},NominatedNodeName:,PodIPs:[]PodIP{PodIP{IP:10.64.0.174,},},EphemeralContainerStatuses:[]ContainerStatus{},},},Pod{ObjectMeta:{my-hostname-delete-node-8jzps my-hostname-delete-node- resize-nodes-1123 /api/v1/namespaces/resize-nodes-1123/pods/my-hostname-delete-node-8jzps 8334e35a-bcb9-4139-a321-7c769b115cd2 62143 0 2020-03-28 22:02:03 +0000 UTC <nil> <nil> map[name:my-hostname-delete-node] map[] [{v1 ReplicationController my-hostname-delete-node 441d8555-3948-4f41-bc4b-110f4c2e6a96 0xc004954206 0xc004954207}] []  []},Spec:PodSpec{Volumes:[]Volume{Volume{Name:default-token-t6xpg,VolumeSource:VolumeSource{HostPath:nil,EmptyDir:nil,GCEPersistentDisk:nil,AWSElasticBlockStore:nil,GitRepo:nil,Secret:&SecretVolumeSource{SecretName:default-token-t6xpg,Items:[]KeyToPath{},DefaultMode:*420,Optional:nil,},NFS:nil,ISCSI:nil,Glusterfs:nil,PersistentVolumeClaim:nil,RBD:nil,FlexVolume:nil,Cinder:nil,CephFS:nil,Flocker:nil,DownwardAPI:nil,FC:nil,AzureFile:nil,ConfigMap:nil,VsphereVolume:nil,Quobyte:nil,AzureDisk:nil,PhotonPersistentDisk:nil,PortworxVolume:nil,ScaleIO:nil,Projected:nil,StorageOS:nil,CSI:nil,},},},Containers:[]Container{Container{Name:my-hostname-delete-node,Image:gcr.io/kubernetes-e2e-test-images/agnhost:2.6,Command:[],Args:[serve-hostname],WorkingDir:,Ports:[]ContainerPort{ContainerPort{Name:,HostPort:0,ContainerPort:9376,Protocol:TCP,HostIP:,},},Env:[]EnvVar{},Resources:ResourceRequirements{Limits:ResourceList{},Requests:ResourceList{},},VolumeMounts:[]VolumeMount{VolumeMount{Name:default-token-t6xpg,ReadOnly:true,MountPath:/var/run/secrets/kubernetes.io/serviceaccount,SubPath:,MountPropagation:nil,SubPathExpr:,},},LivenessProbe:nil,ReadinessProbe:nil,Lifecycle:nil,TerminationMessagePath:/dev/termination-log,ImagePullPolicy:IfNotPresent,SecurityContext:nil,Stdin:false,StdinOnce:false,TTY:false,EnvFrom:[]EnvFromSource{},TerminationMessagePolicy:File,VolumeDevices:[]VolumeDevice{},StartupProbe:nil,},},RestartPolicy:Always,TerminationGracePeriodSeconds:*0,ActiveDeadlineSeconds:nil,DNSPolicy:ClusterFirst,NodeSelector:map[string]string{},ServiceAccountName:default,DeprecatedServiceAccount:default,NodeName:bootstrap-e2e-minion-group-nt4p,HostNetwork:false,HostPID:false,HostIPC:false,SecurityContext:&PodSecurityContext{SELinuxOptions:nil,RunAsUser:nil,RunAsNonRoot:nil,SupplementalGroups:[],FSGroup:nil,RunAsGroup:nil,Sysctls:[]Sysctl{},WindowsOptions:nil,},ImagePullSecrets:[]LocalObjectReference{},Hostname:,Subdomain:,Affinity:nil,SchedulerName:default-scheduler,InitContainers:[]Container{},AutomountServiceAccountToken:nil,Tolerations:[]Toleration{Toleration{Key:node.kubernetes.io/not-ready,Operator:Exists,Value:,Effect:NoExecute,TolerationSeconds:*300,},Toleration{Key:node.kubernetes.io/unreachable,Operator:Exists,Value:,Effect:NoExecute,TolerationSeconds:*300,},},HostAliases:[]HostAlias{},PriorityClassName:,Priority:*0,DNSConfig:nil,ShareProcessNamespace:nil,ReadinessGates:[]PodReadinessGate{},RuntimeClassName:nil,EnableServiceLinks:*true,PreemptionPolicy:nil,Overhead:ResourceList{},TopologySpreadConstraints:[]TopologySpreadConstraint{},EphemeralContainers:[]EphemeralContainer{},},Status:PodStatus{Phase:Running,Conditions:[]PodCondition{PodCondition{Type:Initialized,Status:True,LastProbeTime:0001-01-01 00:00:00 +0000 UTC,LastTransitionTime:2020-03-28 22:02:03 +0000 UTC,Reason:,Message:,},PodCondition{Type:Ready,Status:True,LastProbeTime:0001-01-01 00:00:00 +0000 UTC,LastTransitionTime:2020-03-28 22:02:05 +0000 UTC,Reason:,Message:,},PodCondition{Type:ContainersReady,Status:True,LastProbeTime:0001-01-01 00:00:00 +0000 UTC,LastTransitionTime:2020-03-28 22:02:05 +0000 UTC,Reason:,Message:,},PodCondition{Type:PodScheduled,Status:True,LastProbeTime:0001-01-01 00:00:00 +0000 UTC,LastTransitionTime:2020-03-28 22:02:03 +0000 UTC,Reason:,Message:,},},Message:,Reason:,HostIP:10.138.0.4,PodIP:10.64.1.212,StartTime:2020-03-28 22:02:03 +0000 UTC,ContainerStatuses:[]ContainerStatus{ContainerStatus{Name:my-hostname-delete-node,State:ContainerState{Waiting:nil,Running:&ContainerStateRunning{StartedAt:2020-03-28 22:02:04 +0000 UTC,},Terminated:nil,},LastTerminationState:ContainerState{Waiting:nil,Running:nil,Terminated:nil,},Ready:true,RestartCount:0,Image:gcr.io/kubernetes-e2e-test-images/agnhost:2.6,ImageID:docker-pullable://gcr.io/kubernetes-e2e-test-images/agnhost@sha256:4057a5580c7b59c4fe10d8ab2732c9dec35eea80fd41f7bafc7bd5acc7edf727,ContainerID:docker://388be5536b567ad2cf2e54c0c4d884c1fefa745f1fb9cd131e1028c539888302,Started:*true,},},QOSClass:BestEffort,InitContainerStatuses:[]ContainerStatus{},NominatedNodeName:,PodIPs:[]PodIP{PodIP{IP:10.64.1.212,},},EphemeralContainerStatuses:[]ContainerStatus{},},},Pod{ObjectMeta:{my-hostname-delete-node-mv8sr my-hostname-delete-node- resize-nodes-1123 /api/v1/namespaces/resize-nodes-1123/pods/my-hostname-delete-node-mv8sr 2492fe32-70ed-4cc1-a200-ec5296f8c2b8 62640 0 2020-03-28 22:04:40 +0000 UTC <nil> <nil> map[name:my-hostname-delete-node] map[] [{v1 ReplicationController my-hostname-delete-node 441d8555-3948-4f41-bc4b-110f4c2e6a96 0xc004954346 0xc004954347}] []  []},Spec:PodSpec{Volumes:[]Volume{Volume{Name:default-token-t6xpg,VolumeSource:VolumeSource{HostPath:nil,EmptyDir:nil,GCEPersistentDisk:nil,AWSElasticBlockStore:nil,GitRepo:nil,Secret:&SecretVolumeSource{SecretName:default-token-t6xpg,Items:[]KeyToPath{},DefaultMode:*420,Optional:nil,},NFS:nil,ISCSI:nil,Glusterfs:nil,PersistentVolumeClaim:nil,RBD:nil,FlexVolume:nil,Cinder:nil,CephFS:nil,Flocker:nil,DownwardAPI:nil,FC:nil,AzureFile:nil,ConfigMap:nil,VsphereVolume:nil,Quobyte:nil,AzureDisk:nil,PhotonPersistentDisk:nil,PortworxVolume:nil,ScaleIO:nil,Projected:nil,StorageOS:nil,CSI:nil,},},},Containers:[]Container{Container{Name:my-hostname-delete-node,Image:gcr.io/kubernetes-e2e-test-images/agnhost:2.6,Command:[],Args:[serve-hostname],WorkingDir:,Ports:[]ContainerPort{ContainerPort{Name:,HostPort:0,ContainerPort:9376,Protocol:TCP,HostIP:,},},Env:[]EnvVar{},Resources:ResourceRequirements{Limits:ResourceList{},Requests:ResourceList{},},VolumeMounts:[]VolumeMount{VolumeMount{Name:default-token-t6xpg,ReadOnly:true,MountPath:/var/run/secrets/kubernetes.io/serviceaccount,SubPath:,MountPropagation:nil,SubPathExpr:,},},LivenessProbe:nil,ReadinessProbe:nil,Lifecycle:nil,TerminationMessagePath:/dev/termination-log,ImagePullPolicy:IfNotPresent,SecurityContext:nil,Stdin:false,StdinOnce:false,TTY:false,EnvFrom:[]EnvFromSource{},TerminationMessagePolicy:File,VolumeDevices:[]VolumeDevice{},StartupProbe:nil,},},RestartPolicy:Always,TerminationGracePeriodSeconds:*0,ActiveDeadlineSeconds:nil,DNSPolicy:ClusterFirst,NodeSelector:map[string]string{},ServiceAccountName:default,DeprecatedServiceAccount:default,NodeName:bootstrap-e2e-minion-group-pn0s,HostNetwork:false,HostPID:false,HostIPC:false,SecurityContext:&PodSecurityContext{SELinuxOptions:nil,RunAsUser:nil,RunAsNonRoot:nil,SupplementalGroups:[],FSGroup:nil,RunAsGroup:nil,Sysctls:[]Sysctl{},WindowsOptions:nil,},ImagePullSecrets:[]LocalObjectReference{},Hostname:,Subdomain:,Affinity:nil,SchedulerName:default-scheduler,InitContainers:[]Container{},AutomountServiceAccountToken:nil,Tolerations:[]Toleration{Toleration{Key:node.kubernetes.io/not-ready,Operator:Exists,Value:,Effect:NoExecute,TolerationSeconds:*300,},Toleration{Key:node.kubernetes.io/unreachable,Operator:Exists,Value:,Effect:NoExecute,TolerationSeconds:*300,},},HostAliases:[]HostAlias{},PriorityClassName:,Priority:*0,DNSConfig:nil,ShareProcessNamespace:nil,ReadinessGates:[]PodReadinessGate{},RuntimeClassName:nil,EnableServiceLinks:*true,PreemptionPolicy:nil,Overhead:ResourceList{},TopologySpreadConstraints:[]TopologySpreadConstraint{},EphemeralContainers:[]EphemeralContainer{},},Status:PodStatus{Phase:Running,Conditions:[]PodCondition{PodCondition{Type:Initialized,Status:True,LastProbeTime:0001-01-01 00:00:00 +0000 UTC,LastTransitionTime:2020-03-28 22:04:40 +0000 UTC,Reason:,Message:,},PodCondition{Type:Ready,Status:True,LastProbeTime:0001-01-01 00:00:00 +0000 UTC,LastTransitionTime:2020-03-28 22:04:42 +0000 UTC,Reason:,Message:,},PodCondition{Type:ContainersReady,Status:True,LastProbeTime:0001-01-01 00:00:00 +0000 UTC,LastTransitionTime:2020-03-28 22:04:42 +0000 UTC,Reason:,Message:,},PodCondition{Type:PodScheduled,Status:True,LastProbeTime:0001-01-01 00:00:00 +0000 UTC,LastTransitionTime:2020-03-28 22:04:40 +0000 UTC,Reason:,Message:,},},Message:,Reason:,HostIP:10.138.0.3,PodIP:10.64.0.175,StartTime:2020-03-28 22:04:40 +0000 UTC,ContainerStatuses:[]ContainerStatus{ContainerStatus{Name:my-hostname-delete-node,State:ContainerState{Waiting:nil,Running:&ContainerStateRunning{StartedAt:2020-03-28 22:04:41 +0000 UTC,},Terminated:nil,},LastTerminationState:ContainerState{Waiting:nil,Running:nil,Terminated:nil,},Ready:true,RestartCount:0,Image:gcr.io/k8s-authenticated-test/agnhost:2.6,ImageID:docker-pullable://gcr.io/k8s-authenticated-test/agnhost@sha256:4057a5580c7b59c4fe10d8ab2732c9dec35eea80fd41f7bafc7bd5acc7edf727,ContainerID:docker://6855d469a6f9767e0d15b34e35d063f3ab029494788257d8adb50d4f638870fc,Started:*true,},},QOSClass:BestEffort,InitContainerStatuses:[]ContainerStatus{},NominatedNodeName:,PodIPs:[]PodIP{PodIP{IP:10.64.0.175,},},EphemeralContainerStatuses:[]ContainerStatus{},},},},}
occurred
/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/lifecycle/resize_nodes.go:138