This job view page is being replaced by Spyglass soon. Check out the new job view.
ResultFAILURE
Tests 23 failed / 1180 succeeded
Started2020-10-20 13:09
Elapsed15h2m
Revision
Builder67ed031a-12d5-11eb-a3cf-6ef374375bd7
infra-commit6ad9eb5f3
job-versionv1.17.14-rc.0.7+a70c45e3b1926e
master_os_imagecos-77-12371-175-0
node_os_imagecos-77-12371-175-0
revisionv1.17.14-rc.0.7+a70c45e3b1926e

Test Failures


Cluster upgrade [sig-cloud-provider-gcp] cluster-upgrade 4m30s

go run hack/e2e.go -v --test --test_args='--ginkgo.focus=Cluster\supgrade\s\[sig\-cloud\-provider\-gcp\]\scluster\-upgrade$'
Oct 20 13:25:46.118: Unexpected error:
    <*errors.errorString | 0xc0000a2380>: {
        s: "error running /workspace/kubernetes_skew/cluster/gce/upgrade.sh [-M v1.18.11-rc.0.7+c194df43db09e4]; got error exit status 1, stdout \"Fetching the previously installed CoreDNS version\\n\\n***WARNING***\\nUpgrading Kubernetes with this script might result in an upgrade to a new etcd version.\\nSome etcd version upgrades, such as 3.0.x to 3.1.x, DO NOT offer a downgrade path.\\nTo pin the etcd version to your current one (e.g. v3.0.17), set the following variables\\nbefore running this script:\\n\\n# example: pin to etcd v3.0.17\\nexport ETCD_IMAGE=3.0.17\\nexport ETCD_VERSION=3.0.17\\n\\nAlternatively, if you choose to allow an etcd upgrade that doesn't support downgrade,\\nyou might still be able to downgrade Kubernetes by pinning to the newer etcd version.\\nIn all cases, it is strongly recommended to have an etcd backup before upgrading.\\n\\n== Pre-Upgrade Node OS and Kubelet Versions ==\\nname: \\\"bootstrap-e2e-master\\\", osImage: \\\"Container-Optimized OS from Google\\\", kubeletVersion: \\\"v1.17.14-rc.0.7+a70c45e3b1926e\\\"\\nname: \\\"bootstrap-e2e-minion-group-1571\\\", osImage: \\\"Container-Optimized OS from Google\\\", kubeletVersion: \\\"v1.17.14-rc.0.7+a70c45e3b1926e\\\"\\nname: \\\"bootstrap-e2e-minion-group-nn9d\\\", osImage: \\\"Container-Optimized OS from Google\\\", kubeletVersion: \\\"v1.17.14-rc.0.7+a70c45e3b1926e\\\"\\nname: \\\"bootstrap-e2e-minion-group-qr9r\\\", osImage: \\\"Container-Optimized OS from Google\\\", kubeletVersion: \\\"v1.17.14-rc.0.7+a70c45e3b1926e\\\"\\nFound subnet for region us-west1 in network bootstrap-e2e: bootstrap-e2e\\n== Upgrading master to 'https://storage.googleapis.com/kubernetes-release-dev/ci/v1.18.11-rc.0.7+c194df43db09e4/kubernetes-server-linux-amd64.tar.gz'. Do not interrupt, deleting master instance. ==\\n== Upgrading master environment variables. ==\\n== Waiting for new master to respond to API requests ==\\n......................== Done ==\\nWaiting for CoreDNS to update\\nFetching the latest installed CoreDNS version\\n== Downloading the CoreDNS migration tool ==\\n== Applying the latest default CoreDNS configuration ==\\nserviceaccount/coredns unchanged\\nclusterrole.rbac.authorization.k8s.io/system:coredns unchanged\\nclusterrolebinding.rbac.authorization.k8s.io/system:coredns configured\\nconfigmap/coredns configured\\ndeployment.apps/coredns unchanged\\nservice/kube-dns unchanged\\n== The CoreDNS Config has been updated ==\\n== Validating cluster post-upgrade ==\\nValidating gce cluster, MULTIZONE=\\nFound 4 node(s).\\nNAME                              STATUS                     ROLES    AGE     VERSION\\nbootstrap-e2e-master              Ready,SchedulingDisabled   <none>   9m13s   v1.18.11-rc.0.7+c194df43db09e4\\nbootstrap-e2e-minion-group-1571   Ready                      <none>   9m13s   v1.17.14-rc.0.7+a70c45e3b1926e\\nbootstrap-e2e-minion-group-nn9d   Ready                      <none>   9m13s   v1.17.14-rc.0.7+a70c45e3b1926e\\nbootstrap-e2e-minion-group-qr9r   Ready                      <none>   9m12s   v1.17.14-rc.0.7+a70c45e3b1926e\\nValidate output:\\nNAME                 STATUS    MESSAGE             ERROR\\nscheduler            Healthy   ok                  \\netcd-1               Healthy   {\\\"health\\\":\\\"true\\\"}   \\ncontroller-manager   Healthy   ok                  \\netcd-0               Healthy   {\\\"health\\\":\\\"true\\\"}   \\n\\x1b[0;32mCluster validation succeeded\\x1b[0m\\n== Post-Upgrade Node OS and Kubelet Versions ==\\nname: \\\"bootstrap-e2e-master\\\", osImage: \\\"Container-Optimized OS from Google\\\", kubeletVersion: \\\"v1.18.11-rc.0.7+c194df43db09e4\\\"\\nname: \\\"bootstrap-e2e-minion-group-1571\\\", osImage: \\\"Container-Optimized OS from Google\\\", kubeletVersion: \\\"v1.17.14-rc.0.7+a70c45e3b1926e\\\"\\nname: \\\"bootstrap-e2e-minion-group-nn9d\\\", osImage: \\\"Container-Optimized OS from Google\\\", kubeletVersion: \\\"v1.17.14-rc.0.7+a70c45e3b1926e\\\"\\nname: \\\"bootstrap-e2e-minion-group-qr9r\\\", osImage: \\\"Container-Optimized OS from Google\\\", kubeletVersion: \\\"v1.17.14-rc.0.7+a70c45e3b1926e\\\"\\n\", stderr \"Project: k8s-gce-cvm-1-5-1-6-ctl-skew\\nNetwork Project: k8s-gce-cvm-1-5-1-6-ctl-skew\\nZone: us-west1-b\\nINSTANCE_GROUPS=bootstrap-e2e-minion-group\\nNODE_NAMES=bootstrap-e2e-minion-group-1571 bootstrap-e2e-minion-group-nn9d bootstrap-e2e-minion-group-qr9r\\nTrying to find master named 'bootstrap-e2e-master'\\nLooking for address 'bootstrap-e2e-master-ip'\\nUsing master: bootstrap-e2e-master (external IP: 35.199.184.112; internal IP: (not set))\\nDeleted [https://www.googleapis.com/compute/v1/projects/k8s-gce-cvm-1-5-1-6-ctl-skew/zones/us-west1-b/instances/bootstrap-e2e-master].\\nWARNING: You have selected a disk size of under [200GB]. This may result in poor I/O performance. For more information, see: https://developers.google.com/compute/docs/disks#performance.\\nCreated [https://www.googleapis.com/compute/v1/projects/k8s-gce-cvm-1-5-1-6-ctl-skew/zones/us-west1-b/instances/bootstrap-e2e-master].\\nWARNING: Some requests generated warnings:\\n - Disk size: '20 GB' is larger than image size: '10 GB'. You might need to resize the root repartition manually if the operating system does not support automatic resizing. See https://cloud.google.com/compute/docs/disks/add-persistent-disk#resize_pd for details.\\n - The resource 'projects/cos-cloud/global/images/cos-77-12371-175-0' is deprecated. A suggested replacement is 'projects/cos-cloud/global/images/cos-77-12371-183-0'.\\n\\nNAME                  ZONE        MACHINE_TYPE   PREEMPTIBLE  INTERNAL_IP  EXTERNAL_IP     STATUS\\nbootstrap-e2e-master  us-west1-b  n1-standard-1               10.138.0.6   35.199.184.112  RUNNING\\nWarning: Permanently added 'compute.192763636264163877' (ED25519) to the list of known hosts.\\r\\nWarning: kubectl apply should be used on resource created by either kubectl create --save-config or kubectl apply\\nWarning: kubectl apply should be used on resource created by either kubectl create --save-config or kubectl apply\\nProject: k8s-gce-cvm-1-5-1-6-ctl-skew\\nNetwork Project: k8s-gce-cvm-1-5-1-6-ctl-skew\\nZone: us-west1-b\\n/workspace/kubernetes_skew/cluster/gce/upgrade.sh: line 452: download_dir: unbound variable\\n\"",
    }
    error running /workspace/kubernetes_skew/cluster/gce/upgrade.sh [-M v1.18.11-rc.0.7+c194df43db09e4]; got error exit status 1, stdout "Fetching the previously installed CoreDNS version\n\n***WARNING***\nUpgrading Kubernetes with this script might result in an upgrade to a new etcd version.\nSome etcd version upgrades, such as 3.0.x to 3.1.x, DO NOT offer a downgrade path.\nTo pin the etcd version to your current one (e.g. v3.0.17), set the following variables\nbefore running this script:\n\n# example: pin to etcd v3.0.17\nexport ETCD_IMAGE=3.0.17\nexport ETCD_VERSION=3.0.17\n\nAlternatively, if you choose to allow an etcd upgrade that doesn't support downgrade,\nyou might still be able to downgrade Kubernetes by pinning to the newer etcd version.\nIn all cases, it is strongly recommended to have an etcd backup before upgrading.\n\n== Pre-Upgrade Node OS and Kubelet Versions ==\nname: \"bootstrap-e2e-master\", osImage: \"Container-Optimized OS from Google\", kubeletVersion: \"v1.17.14-rc.0.7+a70c45e3b1926e\"\nname: \"bootstrap-e2e-minion-group-1571\", osImage: \"Container-Optimized OS from Google\", kubeletVersion: \"v1.17.14-rc.0.7+a70c45e3b1926e\"\nname: \"bootstrap-e2e-minion-group-nn9d\", osImage: \"Container-Optimized OS from Google\", kubeletVersion: \"v1.17.14-rc.0.7+a70c45e3b1926e\"\nname: \"bootstrap-e2e-minion-group-qr9r\", osImage: \"Container-Optimized OS from Google\", kubeletVersion: \"v1.17.14-rc.0.7+a70c45e3b1926e\"\nFound subnet for region us-west1 in network bootstrap-e2e: bootstrap-e2e\n== Upgrading master to 'https://storage.googleapis.com/kubernetes-release-dev/ci/v1.18.11-rc.0.7+c194df43db09e4/kubernetes-server-linux-amd64.tar.gz'. Do not interrupt, deleting master instance. ==\n== Upgrading master environment variables. ==\n== Waiting for new master to respond to API requests ==\n......................== Done ==\nWaiting for CoreDNS to update\nFetching the latest installed CoreDNS version\n== Downloading the CoreDNS migration tool ==\n== Applying the latest default CoreDNS configuration ==\nserviceaccount/coredns unchanged\nclusterrole.rbac.authorization.k8s.io/system:coredns unchanged\nclusterrolebinding.rbac.authorization.k8s.io/system:coredns configured\nconfigmap/coredns configured\ndeployment.apps/coredns unchanged\nservice/kube-dns unchanged\n== The CoreDNS Config has been updated ==\n== Validating cluster post-upgrade ==\nValidating gce cluster, MULTIZONE=\nFound 4 node(s).\nNAME                              STATUS                     ROLES    AGE     VERSION\nbootstrap-e2e-master              Ready,SchedulingDisabled   <none>   9m13s   v1.18.11-rc.0.7+c194df43db09e4\nbootstrap-e2e-minion-group-1571   Ready                      <none>   9m13s   v1.17.14-rc.0.7+a70c45e3b1926e\nbootstrap-e2e-minion-group-nn9d   Ready                      <none>   9m13s   v1.17.14-rc.0.7+a70c45e3b1926e\nbootstrap-e2e-minion-group-qr9r   Ready                      <none>   9m12s   v1.17.14-rc.0.7+a70c45e3b1926e\nValidate output:\nNAME                 STATUS    MESSAGE             ERROR\nscheduler            Healthy   ok                  \netcd-1               Healthy   {\"health\":\"true\"}   \ncontroller-manager   Healthy   ok                  \netcd-0               Healthy   {\"health\":\"true\"}   \n\x1b[0;32mCluster validation succeeded\x1b[0m\n== Post-Upgrade Node OS and Kubelet Versions ==\nname: \"bootstrap-e2e-master\", osImage: \"Container-Optimized OS from Google\", kubeletVersion: \"v1.18.11-rc.0.7+c194df43db09e4\"\nname: \"bootstrap-e2e-minion-group-1571\", osImage: \"Container-Optimized OS from Google\", kubeletVersion: \"v1.17.14-rc.0.7+a70c45e3b1926e\"\nname: \"bootstrap-e2e-minion-group-nn9d\", osImage: \"Container-Optimized OS from Google\", kubeletVersion: \"v1.17.14-rc.0.7+a70c45e3b1926e\"\nname: \"bootstrap-e2e-minion-group-qr9r\", osImage: \"Container-Optimized OS from Google\", kubeletVersion: \"v1.17.14-rc.0.7+a70c45e3b1926e\"\n", stderr "Project: k8s-gce-cvm-1-5-1-6-ctl-skew\nNetwork Project: k8s-gce-cvm-1-5-1-6-ctl-skew\nZone: us-west1-b\nINSTANCE_GROUPS=bootstrap-e2e-minion-group\nNODE_NAMES=bootstrap-e2e-minion-group-1571 bootstrap-e2e-minion-group-nn9d bootstrap-e2e-minion-group-qr9r\nTrying to find master named 'bootstrap-e2e-master'\nLooking for address 'bootstrap-e2e-master-ip'\nUsing master: bootstrap-e2e-master (external IP: 35.199.184.112; internal IP: (not set))\nDeleted [https://www.googleapis.com/compute/v1/projects/k8s-gce-cvm-1-5-1-6-ctl-skew/zones/us-west1-b/instances/bootstrap-e2e-master].\nWARNING: You have selected a disk size of under [200GB]. This may result in poor I/O performance. For more information, see: https://developers.google.com/compute/docs/disks#performance.\nCreated [https://www.googleapis.com/compute/v1/projects/k8s-gce-cvm-1-5-1-6-ctl-skew/zones/us-west1-b/instances/bootstrap-e2e-master].\nWARNING: Some requests generated warnings:\n - Disk size: '20 GB' is larger than image size: '10 GB'. You might need to resize the root repartition manually if the operating system does not support automatic resizing. See https://cloud.google.com/compute/docs/disks/add-persistent-disk#resize_pd for details.\n - The resource 'projects/cos-cloud/global/images/cos-77-12371-175-0' is deprecated. A suggested replacement is 'projects/cos-cloud/global/images/cos-77-12371-183-0'.\n\nNAME                  ZONE        MACHINE_TYPE   PREEMPTIBLE  INTERNAL_IP  EXTERNAL_IP     STATUS\nbootstrap-e2e-master  us-west1-b  n1-standard-1               10.138.0.6   35.199.184.112  RUNNING\nWarning: Permanently added 'compute.192763636264163877' (ED25519) to the list of known hosts.\r\nWarning: kubectl apply should be used on resource created by either kubectl create --save-config or kubectl apply\nWarning: kubectl apply should be used on resource created by either kubectl create --save-config or kubectl apply\nProject: k8s-gce-cvm-1-5-1-6-ctl-skew\nNetwork Project: k8s-gce-cvm-1-5-1-6-ctl-skew\nZone: us-west1-b\n/workspace/kubernetes_skew/cluster/gce/upgrade.sh: line 452: download_dir: unbound variable\n"
occurred

k8s.io/kubernetes/test/e2e/cloud/gcp.glob..func2.3.1.1()
	/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/cloud/gcp/cluster_upgrade.go:153 +0x132
k8s.io/kubernetes/test/e2e/chaosmonkey.(*Chaosmonkey).Do(0xc002f7b1e8)
	/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/chaosmonkey/chaosmonkey.go:111 +0x38a
k8s.io/kubernetes/test/e2e/cloud/gcp.runUpgradeSuite(0xc0010e0000, 0x799a240, 0xc, 0xc, 0xc0009a71d0, 0xc000a2cb40, 0x2, 0xc002b85320)
	/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/cloud/gcp/cluster_upgrade.go:485 +0x47a
k8s.io/kubernetes/test/e2e/cloud/gcp.glob..func2.3.1()
	/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/cloud/gcp/cluster_upgrade.go:158 +0x222
k8s.io/kubernetes/test/e2e.RunE2ETests(0xc0001da300)
	_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/e2e.go:125 +0x324
k8s.io/kubernetes/test/e2e.TestE2E(0xc0001da300)
	_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/e2e_test.go:111 +0x2b
testing.tRunner(0xc0001da300, 0x4af5d58)
	/usr/local/go/src/testing/testing.go:909 +0xc9
created by testing.(*T).Run
	/usr/local/go/src/testing/testing.go:960 +0x350
				from junit_upgradeupgrades.xml

Filter through log files | View test history on testgrid


Kubernetes e2e suite [k8s.io] [sig-cloud-provider-gcp] Upgrade [Feature:Upgrade] cluster upgrade should maintain a functioning cluster [Feature:ClusterUpgrade] 16m46s

go run hack/e2e.go -v --test --test_args='--ginkgo.focus=Kubernetes\se2e\ssuite\s\[k8s\.io\]\s\[sig\-cloud\-provider\-gcp\]\sUpgrade\s\[Feature\:Upgrade\]\scluster\supgrade\sshould\smaintain\sa\sfunctioning\scluster\s\[Feature\:ClusterUpgrade\]$'
/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/cloud/gcp/cluster_upgrade.go:142
Oct 20 13:25:46.118: Unexpected error:
    <*errors.errorString | 0xc0000a2380>: {
        s: "error running /workspace/kubernetes_skew/cluster/gce/upgrade.sh [-M v1.18.11-rc.0.7+c194df43db09e4]; got error exit status 1, stdout \"Fetching the previously installed CoreDNS version\\n\\n***WARNING***\\nUpgrading Kubernetes with this script might result in an upgrade to a new etcd version.\\nSome etcd version upgrades, such as 3.0.x to 3.1.x, DO NOT offer a downgrade path.\\nTo pin the etcd version to your current one (e.g. v3.0.17), set the following variables\\nbefore running this script:\\n\\n# example: pin to etcd v3.0.17\\nexport ETCD_IMAGE=3.0.17\\nexport ETCD_VERSION=3.0.17\\n\\nAlternatively, if you choose to allow an etcd upgrade that doesn't support downgrade,\\nyou might still be able to downgrade Kubernetes by pinning to the newer etcd version.\\nIn all cases, it is strongly recommended to have an etcd backup before upgrading.\\n\\n== Pre-Upgrade Node OS and Kubelet Versions ==\\nname: \\\"bootstrap-e2e-master\\\", osImage: \\\"Container-Optimized OS from Google\\\", kubeletVersion: \\\"v1.17.14-rc.0.7+a70c45e3b1926e\\\"\\nname: \\\"bootstrap-e2e-minion-group-1571\\\", osImage: \\\"Container-Optimized OS from Google\\\", kubeletVersion: \\\"v1.17.14-rc.0.7+a70c45e3b1926e\\\"\\nname: \\\"bootstrap-e2e-minion-group-nn9d\\\", osImage: \\\"Container-Optimized OS from Google\\\", kubeletVersion: \\\"v1.17.14-rc.0.7+a70c45e3b1926e\\\"\\nname: \\\"bootstrap-e2e-minion-group-qr9r\\\", osImage: \\\"Container-Optimized OS from Google\\\", kubeletVersion: \\\"v1.17.14-rc.0.7+a70c45e3b1926e\\\"\\nFound subnet for region us-west1 in network bootstrap-e2e: bootstrap-e2e\\n== Upgrading master to 'https://storage.googleapis.com/kubernetes-release-dev/ci/v1.18.11-rc.0.7+c194df43db09e4/kubernetes-server-linux-amd64.tar.gz'. Do not interrupt, deleting master instance. ==\\n== Upgrading master environment variables. ==\\n== Waiting for new master to respond to API requests ==\\n......................== Done ==\\nWaiting for CoreDNS to update\\nFetching the latest installed CoreDNS version\\n== Downloading the CoreDNS migration tool ==\\n== Applying the latest default CoreDNS configuration ==\\nserviceaccount/coredns unchanged\\nclusterrole.rbac.authorization.k8s.io/system:coredns unchanged\\nclusterrolebinding.rbac.authorization.k8s.io/system:coredns configured\\nconfigmap/coredns configured\\ndeployment.apps/coredns unchanged\\nservice/kube-dns unchanged\\n== The CoreDNS Config has been updated ==\\n== Validating cluster post-upgrade ==\\nValidating gce cluster, MULTIZONE=\\nFound 4 node(s).\\nNAME                              STATUS                     ROLES    AGE     VERSION\\nbootstrap-e2e-master              Ready,SchedulingDisabled   <none>   9m13s   v1.18.11-rc.0.7+c194df43db09e4\\nbootstrap-e2e-minion-group-1571   Ready                      <none>   9m13s   v1.17.14-rc.0.7+a70c45e3b1926e\\nbootstrap-e2e-minion-group-nn9d   Ready                      <none>   9m13s   v1.17.14-rc.0.7+a70c45e3b1926e\\nbootstrap-e2e-minion-group-qr9r   Ready                      <none>   9m12s   v1.17.14-rc.0.7+a70c45e3b1926e\\nValidate output:\\nNAME                 STATUS    MESSAGE             ERROR\\nscheduler            Healthy   ok                  \\netcd-1               Healthy   {\\\"health\\\":\\\"true\\\"}   \\ncontroller-manager   Healthy   ok                  \\netcd-0               Healthy   {\\\"health\\\":\\\"true\\\"}   \\n\\x1b[0;32mCluster validation succeeded\\x1b[0m\\n== Post-Upgrade Node OS and Kubelet Versions ==\\nname: \\\"bootstrap-e2e-master\\\", osImage: \\\"Container-Optimized OS from Google\\\", kubeletVersion: \\\"v1.18.11-rc.0.7+c194df43db09e4\\\"\\nname: \\\"bootstrap-e2e-minion-group-1571\\\", osImage: \\\"Container-Optimized OS from Google\\\", kubeletVersion: \\\"v1.17.14-rc.0.7+a70c45e3b1926e\\\"\\nname: \\\"bootstrap-e2e-minion-group-nn9d\\\", osImage: \\\"Container-Optimized OS from Google\\\", kubeletVersion: \\\"v1.17.14-rc.0.7+a70c45e3b1926e\\\"\\nname: \\\"bootstrap-e2e-minion-group-qr9r\\\", osImage: \\\"Container-Optimized OS from Google\\\", kubeletVersion: \\\"v1.17.14-rc.0.7+a70c45e3b1926e\\\"\\n\", stderr \"Project: k8s-gce-cvm-1-5-1-6-ctl-skew\\nNetwork Project: k8s-gce-cvm-1-5-1-6-ctl-skew\\nZone: us-west1-b\\nINSTANCE_GROUPS=bootstrap-e2e-minion-group\\nNODE_NAMES=bootstrap-e2e-minion-group-1571 bootstrap-e2e-minion-group-nn9d bootstrap-e2e-minion-group-qr9r\\nTrying to find master named 'bootstrap-e2e-master'\\nLooking for address 'bootstrap-e2e-master-ip'\\nUsing master: bootstrap-e2e-master (external IP: 35.199.184.112; internal IP: (not set))\\nDeleted [https://www.googleapis.com/compute/v1/projects/k8s-gce-cvm-1-5-1-6-ctl-skew/zones/us-west1-b/instances/bootstrap-e2e-master].\\nWARNING: You have selected a disk size of under [200GB]. This may result in poor I/O performance. For more information, see: https://developers.google.com/compute/docs/disks#performance.\\nCreated [https://www.googleapis.com/compute/v1/projects/k8s-gce-cvm-1-5-1-6-ctl-skew/zones/us-west1-b/instances/bootstrap-e2e-master].\\nWARNING: Some requests generated warnings:\\n - Disk size: '20 GB' is larger than image size: '10 GB'. You might need to resize the root repartition manually if the operating system does not support automatic resizing. See https://cloud.google.com/compute/docs/disks/add-persistent-disk#resize_pd for details.\\n - The resource 'projects/cos-cloud/global/images/cos-77-12371-175-0' is deprecated. A suggested replacement is 'projects/cos-cloud/global/images/cos-77-12371-183-0'.\\n\\nNAME                  ZONE        MACHINE_TYPE   PREEMPTIBLE  INTERNAL_IP  EXTERNAL_IP     STATUS\\nbootstrap-e2e-master  us-west1-b  n1-standard-1               10.138.0.6   35.199.184.112  RUNNING\\nWarning: Permanently added 'compute.192763636264163877' (ED25519) to the list of known hosts.\\r\\nWarning: kubectl apply should be used on resource created by either kubectl create --save-config or kubectl apply\\nWarning: kubectl apply should be used on resource created by either kubectl create --save-config or kubectl apply\\nProject: k8s-gce-cvm-1-5-1-6-ctl-skew\\nNetwork Project: k8s-gce-cvm-1-5-1-6-ctl-skew\\nZone: us-west1-b\\n/workspace/kubernetes_skew/cluster/gce/upgrade.sh: line 452: download_dir: unbound variable\\n\"",
    }
    error running /workspace/kubernetes_skew/cluster/gce/upgrade.sh [-M v1.18.11-rc.0.7+c194df43db09e4]; got error exit status 1, stdout "Fetching the previously installed CoreDNS version\n\n***WARNING***\nUpgrading Kubernetes with this script might result in an upgrade to a new etcd version.\nSome etcd version upgrades, such as 3.0.x to 3.1.x, DO NOT offer a downgrade path.\nTo pin the etcd version to your current one (e.g. v3.0.17), set the following variables\nbefore running this script:\n\n# example: pin to etcd v3.0.17\nexport ETCD_IMAGE=3.0.17\nexport ETCD_VERSION=3.0.17\n\nAlternatively, if you choose to allow an etcd upgrade that doesn't support downgrade,\nyou might still be able to downgrade Kubernetes by pinning to the newer etcd version.\nIn all cases, it is strongly recommended to have an etcd backup before upgrading.\n\n== Pre-Upgrade Node OS and Kubelet Versions ==\nname: \"bootstrap-e2e-master\", osImage: \"Container-Optimized OS from Google\", kubeletVersion: \"v1.17.14-rc.0.7+a70c45e3b1926e\"\nname: \"bootstrap-e2e-minion-group-1571\", osImage: \"Container-Optimized OS from Google\", kubeletVersion: \"v1.17.14-rc.0.7+a70c45e3b1926e\"\nname: \"bootstrap-e2e-minion-group-nn9d\", osImage: \"Container-Optimized OS from Google\", kubeletVersion: \"v1.17.14-rc.0.7+a70c45e3b1926e\"\nname: \"bootstrap-e2e-minion-group-qr9r\", osImage: \"Container-Optimized OS from Google\", kubeletVersion: \"v1.17.14-rc.0.7+a70c45e3b1926e\"\nFound subnet for region us-west1 in network bootstrap-e2e: bootstrap-e2e\n== Upgrading master to 'https://storage.googleapis.com/kubernetes-release-dev/ci/v1.18.11-rc.0.7+c194df43db09e4/kubernetes-server-linux-amd64.tar.gz'. Do not interrupt, deleting master instance. ==\n== Upgrading master environment variables. ==\n== Waiting for new master to respond to API requests ==\n......................== Done ==\nWaiting for CoreDNS to update\nFetching the latest installed CoreDNS version\n== Downloading the CoreDNS migration tool ==\n== Applying the latest default CoreDNS configuration ==\nserviceaccount/coredns unchanged\nclusterrole.rbac.authorization.k8s.io/system:coredns unchanged\nclusterrolebinding.rbac.authorization.k8s.io/system:coredns configured\nconfigmap/coredns configured\ndeployment.apps/coredns unchanged\nservice/kube-dns unchanged\n== The CoreDNS Config has been updated ==\n== Validating cluster post-upgrade ==\nValidating gce cluster, MULTIZONE=\nFound 4 node(s).\nNAME                              STATUS                     ROLES    AGE     VERSION\nbootstrap-e2e-master              Ready,SchedulingDisabled   <none>   9m13s   v1.18.11-rc.0.7+c194df43db09e4\nbootstrap-e2e-minion-group-1571   Ready                      <none>   9m13s   v1.17.14-rc.0.7+a70c45e3b1926e\nbootstrap-e2e-minion-group-nn9d   Ready                      <none>   9m13s   v1.17.14-rc.0.7+a70c45e3b1926e\nbootstrap-e2e-minion-group-qr9r   Ready                      <none>   9m12s   v1.17.14-rc.0.7+a70c45e3b1926e\nValidate output:\nNAME                 STATUS    MESSAGE             ERROR\nscheduler            Healthy   ok                  \netcd-1               Healthy   {\"health\":\"true\"}   \ncontroller-manager   Healthy   ok                  \netcd-0               Healthy   {\"health\":\"true\"}   \n\x1b[0;32mCluster validation succeeded\x1b[0m\n== Post-Upgrade Node OS and Kubelet Versions ==\nname: \"bootstrap-e2e-master\", osImage: \"Container-Optimized OS from Google\", kubeletVersion: \"v1.18.11-rc.0.7+c194df43db09e4\"\nname: \"bootstrap-e2e-minion-group-1571\", osImage: \"Container-Optimized OS from Google\", kubeletVersion: \"v1.17.14-rc.0.7+a70c45e3b1926e\"\nname: \"bootstrap-e2e-minion-group-nn9d\", osImage: \"Container-Optimized OS from Google\", kubeletVersion: \"v1.17.14-rc.0.7+a70c45e3b1926e\"\nname: \"bootstrap-e2e-minion-group-qr9r\", osImage: \"Container-Optimized OS from Google\", kubeletVersion: \"v1.17.14-rc.0.7+a70c45e3b1926e\"\n", stderr "Project: k8s-gce-cvm-1-5-1-6-ctl-skew\nNetwork Project: k8s-gce-cvm-1-5-1-6-ctl-skew\nZone: us-west1-b\nINSTANCE_GROUPS=bootstrap-e2e-minion-group\nNODE_NAMES=bootstrap-e2e-minion-group-1571 bootstrap-e2e-minion-group-nn9d bootstrap-e2e-minion-group-qr9r\nTrying to find master named 'bootstrap-e2e-master'\nLooking for address 'bootstrap-e2e-master-ip'\nUsing master: bootstrap-e2e-master (external IP: 35.199.184.112; internal IP: (not set))\nDeleted [https://www.googleapis.com/compute/v1/projects/k8s-gce-cvm-1-5-1-6-ctl-skew/zones/us-west1-b/instances/bootstrap-e2e-master].\nWARNING: You have selected a disk size of under [200GB]. This may result in poor I/O performance. For more information, see: https://developers.google.com/compute/docs/disks#performance.\nCreated [https://www.googleapis.com/compute/v1/projects/k8s-gce-cvm-1-5-1-6-ctl-skew/zones/us-west1-b/instances/bootstrap-e2e-master].\nWARNING: Some requests generated warnings:\n - Disk size: '20 GB' is larger than image size: '10 GB'. You might need to resize the root repartition manually if the operating system does not support automatic resizing. See https://cloud.google.com/compute/docs/disks/add-persistent-disk#resize_pd for details.\n - The resource 'projects/cos-cloud/global/images/cos-77-12371-175-0' is deprecated. A suggested replacement is 'projects/cos-cloud/global/images/cos-77-12371-183-0'.\n\nNAME                  ZONE        MACHINE_TYPE   PREEMPTIBLE  INTERNAL_IP  EXTERNAL_IP     STATUS\nbootstrap-e2e-master  us-west1-b  n1-standard-1               10.138.0.6   35.199.184.112  RUNNING\nWarning: Permanently added 'compute.192763636264163877' (ED25519) to the list of known hosts.\r\nWarning: kubectl apply should be used on resource created by either kubectl create --save-config or kubectl apply\nWarning: kubectl apply should be used on resource created by either kubectl create --save-config or kubectl apply\nProject: k8s-gce-cvm-1-5-1-6-ctl-skew\nNetwork Project: k8s-gce-cvm-1-5-1-6-ctl-skew\nZone: us-west1-b\n/workspace/kubernetes_skew/cluster/gce/upgrade.sh: line 452: download_dir: unbound variable\n"
occurred
/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/cloud/gcp/cluster_upgrade.go:153
				
				Click to see stdout/stderrfrom junit_upgrade01.xml

Filter through log files | View test history on testgrid


Kubernetes e2e suite [k8s.io] [sig-node] NodeProblemDetector [DisabledForLargeClusters] should run without error 2m10s

go run hack/e2e.go -v --test --test_args='--ginkgo.focus=Kubernetes\se2e\ssuite\s\[k8s\.io\]\s\[sig\-node\]\sNodeProblemDetector\s\[DisabledForLargeClusters\]\sshould\srun\swithout\serror$'
/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/node/node_problem_detector.go:57
Oct 21 03:25:43.450: Timed out after 60.001s.
Expected success, but got an error:
    <*errors.errorString | 0xc0074cae00>: {
        s: "Event KubeletStart does not exist: [{{ } {bootstrap-e2e-minion-group-907m.163fe3ff1dd5745b  default /api/v1/namespaces/default/events/bootstrap-e2e-minion-group-907m.163fe3ff1dd5745b 83a3f15d-66b0-4631-a834-f93d88a0cc3e 43721 0 2020-10-21 03:24:37 +0000 UTC <nil> <nil> map[] map[] [] []  [{node-problem-detector Update v1 2020-10-21 03:24:37 +0000 UTC FieldsV1 &FieldsV1{Raw:*[123 34 102 58 99 111 117 110 116 34 58 123 125 44 34 102 58 102 105 114 115 116 84 105 109 101 115 116 97 109 112 34 58 123 125 44 34 102 58 105 110 118 111 108 118 101 100 79 98 106 101 99 116 34 58 123 34 102 58 107 105 110 100 34 58 123 125 44 34 102 58 110 97 109 101 34 58 123 125 44 34 102 58 117 105 100 34 58 123 125 125 44 34 102 58 108 97 115 116 84 105 109 101 115 116 97 109 112 34 58 123 125 44 34 102 58 109 101 115 115 97 103 101 34 58 123 125 44 34 102 58 114 101 97 115 111 110 34 58 123 125 44 34 102 58 115 111 117 114 99 101 34 58 123 34 102 58 99 111 109 112 111 110 101 110 116 34 58 123 125 44 34 102 58 104 111 115 116 34 58 123 125 125 44 34 102 58 116 121 112 101 34 58 123 125 125],}}]} {Node  bootstrap-e2e-minion-group-907m bootstrap-e2e-minion-group-907m   } TaskHung kernel: INFO: task umount.aufs:21568 blocked for more than 120 seconds. {kernel-monitor bootstrap-e2e-minion-group-907m} 2020-10-21 03:24:37 +0000 UTC 2020-10-21 03:24:37 +0000 UTC 1 Warning 0001-01-01 00:00:00 +0000 UTC nil  nil  } {{ } {bootstrap-e2e-minion-group-907m.163fe3ff1dd59d06  default /api/v1/namespaces/default/events/bootstrap-e2e-minion-group-907m.163fe3ff1dd59d06 1fc978a6-acfc-4ae5-afe9-628577a927a7 43720 0 2020-10-21 03:24:37 +0000 UTC <nil> <nil> map[] map[] [] []  [{node-problem-detector Update v1 2020-10-21 03:24:37 +0000 UTC FieldsV1 &FieldsV1{Raw:*[123 34 102 58 99 111 117 110 116 34 58 123 125 44 34 102 58 102 105 114 115 116 84 105 109 101 115 116 97 109 112 34 58 123 125 44 34 102 58 105 110 118 111 108 118 101 100 79 98 106 101 99 116 34 58 123 34 102 58 107 105 110 100 34 58 123 125 44 34 102 58 110 97 109 101 34 58 123 125 44 34 102 58 117 105 100 34 58 123 125 125 44 34 102 58 108 97 115 116 84 105 109 101 115 116 97 109 112 34 58 123 125 44 34 102 58 109 101 115 115 97 103 101 34 58 123 125 44 34 102 58 114 101 97 115 111 110 34 58 123 125 44 34 102 58 115 111 117 114 99 101 34 58 123 34 102 58 99 111 109 112 111 110 101 110 116 34 58 123 125 44 34 102 58 104 111 115 116 34 58 123 125 125 44 34 102 58 116 121 112 101 34 58 123 125 125],}}]} {Node  bootstrap-e2e-minion-group-907m bootstrap-e2e-minion-group-907m   } AUFSUmountHung Node condition KernelDeadlock is now: True, reason: AUFSUmountHung {kernel-monitor bootstrap-e2e-minion-group-907m} 2020-10-21 03:24:37 +0000 UTC 2020-10-21 03:24:37 +0000 UTC 1 Normal 0001-01-01 00:00:00 +0000 UTC nil  nil  } {{ } {bootstrap-e2e-minion-group-qr9r.163fc881ece1ed8c  default /api/v1/namespaces/default/events/bootstrap-e2e-minion-group-qr9r.163fc881ece1ed8c 29dc0842-bf9b-47e1-8655-75befe190774 41689 0 2020-10-21 02:30:14 +0000 UTC <nil> <nil> map[] map[] [] []  [{node-problem-detector Update v1 2020-10-21 02:47:59 +0000 UTC FieldsV1 &FieldsV1{Raw:*[123 34 102 58 99 111 117 110 116 34 58 123 125 44 34 102 58 102 105 114 115 116 84 105 109 101 115 116 97 109 112 34 58 123 125 44 34 102 58 105 110 118 111 108 118 101 100 79 98 106 101 99 116 34 58 123 34 102 58 107 105 110 100 34 58 123 125 44 34 102 58 110 97 109 101 34 58 123 125 44 34 102 58 117 105 100 34 58 123 125 125 44 34 102 58 108 97 115 116 84 105 109 101 115 116 97 109 112 34 58 123 125 44 34 102 58 109 101 115 115 97 103 101 34 58 123 125 44 34 102 58 114 101 97 115 111 110 34 58 123 125 44 34 102 58 115 111 117 114 99 101 34 58 123 34 102 58 99 111 109 112 111 110 101 110 116 34 58 123 125 44 34 102 58 104 111 115 116 34 58 123 125 125 44 34 102 58 116 121 112 101 34 58 123 125 125],}}]} {Node  bootstrap-e2e-minion-group-qr9r bootstrap-e2e-minion-group-qr9r   } KubeletStart Started Kubernetes kubelet. {systemd-monitor bootstrap-e2e-minion-group-qr9r} 2020-10-20 19:00:52 +0000 UTC 2020-10-21 02:47:59 +0000 UTC 9 Warning 0001-01-01 00:00:00 +0000 UTC nil  nil  } {{ } {bootstrap-e2e-minion-group-qr9r.163fd78884de5b8f  default /api/v1/namespaces/default/events/bootstrap-e2e-minion-group-qr9r.163fd78884de5b8f 01490cc4-837c-4c1b-8720-9568bcb9c48f 41688 0 2020-10-21 02:30:12 +0000 UTC <nil> <nil> map[] map[] [] []  [{kube-controller-manager Update v1 2020-10-21 02:47:57 +0000 UTC FieldsV1 &FieldsV1{Raw:*[123 34 102 58 99 111 117 110 116 34 58 123 125 44 34 102 58 102 105 114 115 116 84 105 109 101 115 116 97 109 112 34 58 123 125 44 34 102 58 105 110 118 111 108 118 101 100 79 98 106 101 99 116 34 58 123 34 102 58 97 112 105 86 101 114 115 105 111 110 34 58 123 125 44 34 102 58 107 105 110 100 34 58 123 125 44 34 102 58 110 97 109 101 34 58 123 125 44 34 102 58 117 105 100 34 58 123 125 125 44 34 102 58 108 97 115 116 84 105 109 101 115 116 97 109 112 34 58 123 125 44 34 102 58 109 101 115 115 97 103 101 34 58 123 125 44 34 102 58 114 101 97 115 111 110 34 58 123 125 44 34 102 58 115 111 117 114 99 101 34 58 123 34 102 58 99 111 109 112 111 110 101 110 116 34 58 123 125 125 44 34 102 58 116 121 112 101 34 58 123 125 125],}}]} {Node  bootstrap-e2e-minion-group-qr9r 9e1627f0-3edc-4412-b273-bf023668f093 v1  } NodeNotReady Node bootstrap-e2e-minion-group-qr9r status is now: NodeNotReady {node-controller } 2020-10-20 23:36:13 +0000 UTC 2020-10-21 02:47:57 +0000 UTC 4 Normal 0001-01-01 00:00:00 +0000 UTC nil  nil  } {{ } {bootstrap-e2e-minion-group-qr9r.163fe107799097b6  default /api/v1/namespaces/default/events/bootstrap-e2e-minion-group-qr9r.163fe107799097b6 f8267cb3-d338-477e-bace-ecd5db36d519 40972 0 2020-10-21 02:30:14 +0000 UTC <nil> <nil> map[] map[] [] []  [{kubelet Update v1 2020-10-21 02:30:14 +0000 UTC FieldsV1 &FieldsV1{Raw:*[123 34 102 58 99 111 117 110 116 34 58 123 125 44 34 102 58 102 105 114 115 116 84 105 109 101 115 116 97 109 112 34 58 123 125 44 34 102 58 105 110 118 111 108 118 101 100 79 98 106 101 99 116 34 58 123 34 102 58 107 105 110 100 34 58 123 125 44 34 102 58 110 97 109 101 34 58 123 125 44 34 102 58 117 105 100 34 58 123 125 125 44 34 102 58 108 97 115 116 84 105 109 101 115 116 97 109 112 34 58 123 125 44 34 102 58 109 101 115 115 97 103 101 34 58 123 125 44 34 102 58 114 101 97 115 111 110 34 58 123 125 44 34 102 58 115 111 117 114 99 101 34 58 123 34 102 58 99 111 109 112 111 110 101 110 116 34 58 123 125 44 34 102 58 104 111 115 116 34 58 123 125 125 44 34 102 58 116 121 112 101 34 58 123 125 125],}}]} {Node  bootstrap-e2e-minion-group-qr9r bootstrap-e2e-minion-group-qr9r   } Starting Starting kubelet. {kubelet bootstrap-e2e-minion-group-qr9r} 2020-10-21 02:30:14 +0000 UTC 2020-10-21 02:30:14 +0000 UTC 1 Normal 0001-01-01 00:00:00 +0000 UTC nil  nil  } {{ } {bootstrap-e2e-minion-group-qr9r.163fe107876103b0  default /api/v1/namespaces/default/events/bootstrap-e2e-minion-group-qr9r.163fe107876103b0 d76d703e-b07b-488b-b0f1-3f386a32a7ad 40976 0 2020-10-21 02:30:14 +0000 UTC <nil> <nil> map[] map[] [] []  [{kubelet Update v1 2020-10-21 02:30:15 +0000 UTC FieldsV1 &FieldsV1{Raw:*[123 34 102 58 99 111 117 110 116 34 58 123 125 44 34 102 58 102 105 114 115 116 84 105 109 101 115 116 97 109 112 34 58 123 125 44 34 102 58 105 110 118 111 108 118 101 100 79 98 106 101 99 116 34 58 123 34 102 58 107 105 110 100 34 58 123 125 44 34 102 58 110 97 109 101 34 58 123 125 44 34 102 58 117 105 100 34 58 123 125 125 44 34 102 58 108 97 115 116 84 105 109 101 115 116 97 109 112 34 58 123 125 44 34 102 58 109 101 115 115 97 103 101 34 58 123 125 44 34 102 58 114 101 97 115 111 110 34 58 123 125 44 34 102 58 115 111 117 114 99 101 34 58 123 34 102 58 99 111 109 112 111 110 101 110 116 34 58 123 125 44 34 102 58 104 111 115 116 34 58 123 125 125 44 34 102 58 116 121 112 101 34 58 123 125 125],}}]} {Node  bootstrap-e2e-minion-group-qr9r bootstrap-e2e-minion-group-qr9r   } NodeHasSufficientMemory Node bootstrap-e2e-minion-group-qr9r status is now: NodeHasSufficientMemory {kubelet bootstrap-e2e-minion-group-qr9r} 2020-10-21 02:30:14 +0000 UTC 2020-10-21 02:30:15 +0000 UTC 2 Normal 0001-01-01 00:00:00 +0000 UTC nil  nil  } {{ } {bootstrap-e2e-minion-group-qr9r.163fe107876146fa  default /api/v1/namespaces/default/events/bootstrap-e2e-minion-group-qr9r.163fe107876146fa e0f40f84-de50-42c1-a0c3-a2b22d0bf7b3 40977 0 2020-10-21 02:30:14 +0000 UTC <nil> <nil> map[] map[] [] []  [{kubelet Update v1 2020-10-21 02:30:15 +0000 UTC FieldsV1 &FieldsV1{Raw:*[123 34 102 58 99 111 117 110 116 34 58 123 125 44 34 102 58 102 105 114 115 116 84 105 109 101 115 116 97 109 112 34 58 123 125 44 34 102 58 105 110 118 111 108 118 101 100 79 98 106 101 99 116 34 58 123 34 102 58 107 105 110 100 34 58 123 125 44 34 102 58 110 97 109 101 34 58 123 125 44 34 102 58 117 105 100 34 58 123 125 125 44 34 102 58 108 97 115 116 84 105 109 101 115 116 97 109 112 34 58 123 125 44 34 102 58 109 101 115 115 97 103 101 34 58 123 125 44 34 102 58 114 101 97 115 111 110 34 58 123 125 44 34 102 58 115 111 117 114 99 101 34 58 123 34 102 58 99 111 109 112 111 110 101 110 116 34 58 123 125 44 34 102 58 104 111 115 116 34 58 123 125 125 44 34 102 58 116 121 112 101 34 58 123 125 125],}}]} {Node  bootstrap-e2e-minion-group-qr9r bootstrap-e2e-minion-group-qr9r   } NodeHasNoDiskPressure Node bootstrap-e2e-minion-group-qr9r status is now: NodeHasNoDiskPressure {kubelet bootstrap-e2e-minion-group-qr9r} 2020-10-21 02:30:14 +0000 UTC 2020-10-21 02:30:15 +0000 UTC 2 Normal 0001-01-01 00:00:00 +0000 UTC nil  nil  } {{ } {bootstrap-e2e-minion-group-qr9r.163fe1078761869b  default /api/v1/namespaces/default/events/bootstrap-e2e-minion-group-qr9r.163fe1078761869b 230a1c6a-284a-44e3-bcdc-d4609a1f8017 40978 0 2020-10-21 02:30:14 +0000 UTC <nil> <nil> map[] map[] [] []  [{kubelet Update v1 2020-10-21 02:30:15 +0000 UTC FieldsV1 &FieldsV1{Raw:*[123 34 102 58 99 111 117 110 116 34 58 123 125 44 34 102 58 102 105 114 115 116 84 105 109 101 115 116 97 109 112 34 58 123 125 44 34 102 58 105 110 118 111 108 118 101 100 79 98 106 101 99 116 34 58 123 34 102 58 107 105 110 100 34 58 123 125 44 34 102 58 110 97 109 101 34 58 123 125 44 34 102 58 117 105 100 34 58 123 125 125 44 34 102 58 108 97 115 116 84 105 109 101 115 116 97 109 112 34 58 123 125 44 34 102 58 109 101 115 115 97 103 101 34 58 123 125 44 34 102 58 114 101 97 115 111 110 34 58 123 125 44 34 102 58 115 111 117 114 99 101 34 58 123 34 102 58 99 111 109 112 111 110 101 110 116 34 58 123 125 44 34 102 58 104 111 115 116 34 58 123 125 125 44 34 102 58 116 121 112 101 34 58 123 125 125],}}]} {Node  bootstrap-e2e-minion-group-qr9r bootstrap-e2e-minion-group-qr9r   } NodeHasSufficientPID Node bootstrap-e2e-minion-group-qr9r status is now: NodeHasSufficientPID {kubelet bootstrap-e2e-minion-group-qr9r} 2020-10-21 02:30:14 +0000 UTC 2020-10-21 02:30:15 +0000 UTC 2 Normal 0001-01-01 00:00:00 +0000 UTC nil  nil  } {{ } {bootstrap-e2e-minion-group-qr9r.163fe107941736cd  default /api/v1/namespaces/default/events/bootstrap-e2e-minion-group-qr9r.163fe107941736cd 79daa5cd-546a-48b1-a5cd-51e7908e3da1 40979 0 2020-10-21 02:30:15 +0000 UTC <nil> <nil> map[] map[] [] []  [{kubelet Update v1 2020-10-21 02:30:15 +0000 UTC FieldsV1 &FieldsV1{Raw:*[123 34 102 58 99 111 117 110 116 34 58 123 125 44 34 102 58 102 105 114 115 116 84 105 109 101 115 116 97 109 112 34 58 123 125 44 34 102 58 105 110 118 111 108 118 101 100 79 98 106 101 99 116 34 58 123 34 102 58 107 105 110 100 34 58 123 125 44 34 102 58 110 97 109 101 34 58 123 125 44 34 102 58 117 105 100 34 58 123 125 125 44 34 102 58 108 97 115 116 84 105 109 101 115 116 97 109 112 34 58 123 125 44 34 102 58 109 101 115 115 97 103 101 34 58 123 125 44 34 102 58 114 101 97 115 111 110 34 58 123 125 44 34 102 58 115 111 117 114 99 101 34 58 123 34 102 58 99 111 109 112 111 110 101 110 116 34 58 123 125 44 34 102 58 104 111 115 116 34 58 123 125 125 44 34 102 58 116 121 112 101 34 58 123 125 125],}}]} {Node  bootstrap-e2e-minion-group-qr9r bootstrap-e2e-minion-group-qr9r   } NodeNotReady Node bootstrap-e2e-minion-group-qr9r status is now: NodeNotReady {kubelet bootstrap-e2e-minion-group-qr9r} 2020-10-21 02:30:15 +0000 UTC 2020-10-21 02:30:15 +0000 UTC 1 Normal 0001-01-01 00:00:00 +0000 UTC nil  nil  } {{ } {bootstrap-e2e-minion-group-qr9r.163fe107a9a19689  default /api/v1/namespaces/default/events/bootstrap-e2e-minion-group-qr9r.163fe107a9a19689 eb600255-cad4-4d2b-bf46-62b9756012c4 40980 0 2020-10-21 02:30:15 +0000 UTC <nil> <nil> map[] map[] [] []  [{kubelet Update v1 2020-10-21 02:30:15 +0000 UTC FieldsV1 &FieldsV1{Raw:*[123 34 102 58 99 111 117 110 116 34 58 123 125 44 34 102 58 102 105 114 115 116 84 105 109 101 115 116 97 109 112 34 58 123 125 44 34 102 58 105 110 118 111 108 118 101 100 79 98 106 101 99 116 34 58 123 34 102 58 107 105 110 100 34 58 123 125 44 34 102 58 110 97 109 101 34 58 123 125 44 34 102 58 117 105 100 34 58 123 125 125 44 34 102 58 108 97 115 116 84 105 109 101 115 116 97 109 112 34 58 123 125 44 34 102 58 109 101 115 115 97 103 101 34 58 123 125 44 34 102 58 114 101 97 115 111 110 34 58 123 125 44 34 102 58 115 111 117 114 99 101 34 58 123 34 102 58 99 111 109 112 111 110 101 110 116 34 58 123 125 44 34 102 58 104 111 115 116 34 58 123 125 125 44 34 102 58 116 121 112 101 34 58 123 125 125],}}]} {Node  bootstrap-e2e-minion-group-qr9r bootstrap-e2e-minion-group-qr9r   } NodeAllocatableEnforced Updated Node Allocatable limit across pods {kubelet bootstrap-e2e-minion-group-qr9r} 2020-10-21 02:30:15 +0000 UTC 2020-10-21 02:30:15 +0000 UTC 1 Normal 0001-01-01 00:00:00 +0000 UTC nil  nil  } {{ } {bootstrap-e2e-minion-group-qr9r.163fe107b063d789  default /api/v1/namespaces/default/events/bootstrap-e2e-minion-group-qr9r.163fe107b063d789 6a5abc79-97f2-49b9-9ee4-8aab289dae2b 40981 0 2020-10-21 02:30:15 +0000 UTC <nil> <nil> map[] map[] [] []  [{kubelet Update v1 2020-10-21 02:30:15 +0000 UTC FieldsV1 &FieldsV1{Raw:*[123 34 102 58 99 111 117 110 116 34 58 123 125 44 34 102 58 102 105 114 115 116 84 105 109 101 115 116 97 109 112 34 58 123 125 44 34 102 58 105 110 118 111 108 118 101 100 79 98 106 101 99 116 34 58 123 34 102 58 107 105 110 100 34 58 123 125 44 34 102 58 110 97 109 101 34 58 123 125 44 34 102 58 117 105 100 34 58 123 125 125 44 34 102 58 108 97 115 116 84 105 109 101 115 116 97 109 112 34 58 123 125 44 34 102 58 109 101 115 115 97 103 101 34 58 123 125 44 34 102 58 114 101 97 115 111 110 34 58 123 125 44 34 102 58 115 111 117 114 99 101 34 58 123 34 102 58 99 111 109 112 111 110 101 110 116 34 58 123 125 44 34 102 58 104 111 115 116 34 58 123 125 125 44 34 102 58 116 121 112 101 34 58 123 125 125],}}]} {Node  bootstrap-e2e-minion-group-qr9r bootstrap-e2e-minion-group-qr9r   } NodeReady Node bootstrap-e2e-minion-group-qr9r status is now: NodeReady {kubelet bootstrap-e2e-minion-group-qr9r} 2020-10-21 02:30:15 +0000 UTC 2020-10-21 02:30:15 +0000 UTC 1 Normal 0001-01-01 00:00:00 +0000 UTC nil  nil  } {{ } {bootstrap-e2e-minion-group-qr9r.163fe1ff9311a32d  default /api/v1/namespaces/default/events/bootstrap-e2e-minion-group-qr9r.163fe1ff9311a32d eb5e65e5-6cfb-4c12-8738-bc6410a14eda 41690 0 2020-10-21 02:48:00 +0000 UTC <nil> <nil> map[] map[] [] []  [{kubelet Update v1 2020-10-21 02:48:00 +0000 UTC FieldsV1 &FieldsV1{Raw:*[123 34 102 58 99 111 117 110 116 34 58 123 125 44 34 102 58 102 105 114 115 116 84 105 109 101 115 116 97 109 112 34 58 123 125 44 34 102 58 105 110 118 111 108 118 101 100 79 98 106 101 99 116 34 58 123 34 102 58 107 105 110 100 34 58 123 125 44 34 102 58 110 97 109 101 34 58 123 125 44 34 102 58 117 105 100 34 58 123 125 125 44 34 102 58 108 97 115 116 84 105 109 101 115 116 97 109 112 34 58 123 125 44 34 102 58 109 101 115 115 97 103 101 34 58 123 125 44 34 102 58 114 101 97 115 111 110 34 58 123 125 44 34 102 58 115 111 117 114 99 101 34 58 123 34 102 58 99 111 109 112 111 110 101 110 116 34 58 123 125 44 34 102 58 104 111 115 116 34 58 123 125 125 44 34 102 58 116 121 112 101 34 58 123 125 125],}}]} {Node  bootstrap-e2e-minion-group-qr9r bootstrap-e2e-minion-group-qr9r   } Starting Starting kubelet. {kubelet bootstrap-e2e-minion-group-qr9r} 2020-10-21 02:48:00 +0000 UTC 2020-10-21 02:48:00 +0000 UTC 1 Normal 0001-01-01 00:00:00 +0000 UTC nil  nil  } {{ } {bootstrap-e2e-minion-group-qr9r.163fe1ffa1ee791c  default /api/v1/namespaces/default/events/bootstrap-e2e-minion-group-qr9r.163fe1ffa1ee791c 775d89fb-44a0-4679-ad52-9eadd9864991 41695 0 2020-10-21 02:48:00 +0000 UTC <nil> <nil> map[] map[] [] []  [{kubelet Update v1 2020-10-21 02:48:00 +0000 UTC FieldsV1 &FieldsV1{Raw:*[123 34 102 58 99 111 117 110 116 34 58 123 125 44 34 102 58 102 105 114 115 116 84 105 109 101 115 116 97 109 112 34 58 123 125 44 34 102 58 105 110 118 111 108 118 101 100 79 98 106 101 99 116 34 58 123 34 102 58 107 105 110 100 34 58 123 125 44 34 102 58 110 97 109 101 34 58 123 125 44 34 102 58 117 105 100 34 58 123 125 125 44 34 102 58 108 97 115 116 84 105 109 101 115 116 97 109 112 34 58 123 125 44 34 102 58 109 101 115 115 97 103 101 34 58 123 125 44 34 102 58 114 101 97 115 111 110 34 58 123 125 44 34 102 58 115 111 117 114 99 101 34 58 123 34 102 58 99 111 109 112 111 110 101 110 116 34 58 123 125 44 34 102 58 104 111 115 116 34 58 123 125 125 44 34 102 58 116 121 112 101 34 58 123 125 125],}}]} {Node  bootstrap-e2e-minion-group-qr9r bootstrap-e2e-minion-group-qr9r   } NodeHasSufficientMemory Node bootstrap-e2e-minion-group-qr9r status is now: NodeHasSufficientMemory {kubelet bootstrap-e2e-minion-group-qr9r} 2020-10-21 02:48:00 +0000 UTC 2020-10-21 02:48:00 +0000 UTC 2 Normal 0001-01-01 00:00:00 +0000 UTC nil  nil  } {{ } {bootstrap-e2e-minion-group-qr9r.163fe1ffa1eec561  default /api/v1/namespaces/default/events/bootstrap-e2e-minion-group-qr9r.163fe1ffa1eec561 02a5755a-f8b3-4e56-a47c-a78c62ba2760 41694 0 2020-10-21 02:48:00 +0000 UTC <nil> <nil> map[] map[] [] []  [{kubelet Update v1 2020-10-21 02:48:00 +0000 UTC FieldsV1 &FieldsV1{Raw:*[123 34 102 58 99 111 117 110 116 34 58 123 125 44 34 102 58 102 105 114 115 116 84 105 109 101 115 116 97 109 112 34 58 123 125 44 34 102 58 105 110 118 111 108 118 101 100 79 98 106 101 99 116 34 58 123 34 102 58 107 105 110 100 34 58 123 125 44 34 102 58 110 97 109 101 34 58 123 125 44 34 102 58 117 105 100 34 58 123 125 125 44 34 102 58 108 97 115 116 84 105 109 101 115 116 97 109 112 34 58 123 125 44 34 102 58 109 101 115 115 97 103 101 34 58 123 125 44 34 102 58 114 101 97 115 111 110 34 58 123 125 44 34 102 58 115 111 117 114 99 101 34 58 123 34 102 58 99 111 109 112 111 110 101 110 116 34 58 123 125 44 34 102 58 104 111 115 116 34 58 123 125 125 44 34 102 58 116 121 112 101 34 58 123 125 125],}}]} {Node  bootstrap-e2e-minion-group-qr9r bootstrap-e2e-minion-group-qr9r   } NodeHasNoDiskPressure Node bootstrap-e2e-minion-group-qr9r status is now: NodeHasNoDiskPressure {kubelet bootstrap-e2e-minion-group-qr9r} 2020-10-21 02:48:00 +0000 UTC 2020-10-21 02:48:00 +0000 UTC 2 Normal 0001-01-01 00:00:00 +0000 UTC nil  nil  } {{ } {bootstrap-e2e-minion-group-qr9r.163fe1ffa1eef9f1  default /api/v1/namespaces/default/events/bootstrap-e2e-minion-group-qr9r.163fe1ffa1eef9f1 ed1e49d8-1fe6-41fc-9bc4-643e6a859c1e 41696 0 2020-10-21 02:48:00 +0000 UTC <nil> <nil> map[] map[] [] []  [{kubelet Update v1 2020-10-21 02:48:00 +0000 UTC FieldsV1 &FieldsV1{Raw:*[123 34 102 58 99 111 117 110 116 34 58 123 125 44 34 102 58 102 105 114 115 116 84 105 109 101 115 116 97 109 112 34 58 123 125 44 34 102 58 105 110 118 111 108 118 101 100 79 98 106 101 99 116 34 58 123 34 102 58 107 105 110 100 34 58 123 125 44 34 102 58 110 97 109 101 34 58 123 125 44 34 102 58 117 105 100 34 58 123 125 125 44 34 102 58 108 97 115 116 84 105 109 101 115 116 97 109 112 34 58 123 125 44 34 102 58 109 101 115 115 97 103 101 34 58 123 125 44 34 102 58 114 101 97 115 111 110 34 58 123 125 44 34 102 58 115 111 117 114 99 101 34 58 123 34 102 58 99 111 109 112 111 110 101 110 116 34 58 123 125 44 34 102 58 104 111 115 116 34 58 123 125 125 44 34 102 58 116 121 112 101 34 58 123 125 125],}}]} {Node  bootstrap-e2e-minion-group-qr9r bootstrap-e2e-minion-group-qr9r   } NodeHasSufficientPID Node bootstrap-e2e-minion-group-qr9r status is now: NodeHasSufficientPID {kubelet bootstrap-e2e-minion-group-qr9r} 2020-10-21 02:48:00 +0000 UTC 2020-10-21 02:48:00 +0000 UTC 2 Normal 0001-01-01 00:00:00 +0000 UTC nil  nil  } {{ } {bootstrap-e2e-minion-group-qr9r.163fe1ffab270870  default /api/v1/namespaces/default/events/bootstrap-e2e-minion-group-qr9r.163fe1ffab270870 0132c140-0e08-4bc4-98f8-28b5ee2d1664 41697 0 2020-10-21 02:48:00 +0000 UTC <nil> <nil> map[] map[] [] []  [{kubelet Update v1 2020-10-21 02:48:00 +0000 UTC FieldsV1 &FieldsV1{Raw:*[123 34 102 58 99 111 117 110 116 34 58 123 125 44 34 102 58 102 105 114 115 116 84 105 109 101 115 116 97 109 112 34 58 123 125 44 34 102 58 105 110 118 111 108 118 101 100 79 98 106 101 99 116 34 58 123 34 102 58 107 105 110 100 34 58 123 125 44 34 102 58 110 97 109 101 34 58 123 125 44 34 102 58 117 105 100 34 58 123 125 125 44 34 102 58 108 97 115 116 84 105 109 101 115 116 97 109 112 34 58 123 125 44 34 102 58 109 101 115 115 97 103 101 34 58 123 125 44 34 102 58 114 101 97 115 111 110 34 58 123 125 44 34 102 58 115 111 117 114 99 101 34 58 123 34 102 58 99 111 109 112 111 110 101 110 116 34 58 123 125 44 34 102 58 104 111 115 116 34 58 123 125 125 44 34 102 58 116 121 112 101 34 58 123 125 125],}}]} {Node  bootstrap-e2e-minion-group-qr9r bootstrap-e2e-minion-group-qr9r   } NodeNotReady Node bootstrap-e2e-minion-group-qr9r status is now: NodeNotReady {kubelet bootstrap-e2e-minion-group-qr9r} 2020-10-21 02:48:00 +0000 UTC 2020-10-21 02:48:00 +0000 UTC 1 Normal 0001-01-01 00:00:00 +0000 UTC nil  nil  } {{ } {bootstrap-e2e-minion-group-qr9r.163fe1ffb7a228f2  default /api/v1/namespaces/default/events/bootstrap-e2e-minion-group-qr9r.163fe1ffb7a228f2 2106ad90-8fb0-44e4-837c-b9892e24fb5e 41698 0 2020-10-21 02:48:00 +0000 UTC <nil> <nil> map[] map[] [] []  [{kubelet Update v1 2020-10-21 02:48:00 +0000 UTC FieldsV1 &FieldsV1{Raw:*[123 34 102 58 99 111 117 110 116 34 58 123 125 44 34 102 58 102 105 114 115 116 84 105 109 101 115 116 97 109 112 34 58 123 125 44 34 102 58 105 110 118 111 108 118 101 100 79 98 106 101 99 116 34 58 123 34 102 58 107 105 110 100 34 58 123 125 44 34 102 58 110 97 109 101 34 58 123 125 44 34 102 58 117 105 100 34 58 123 125 125 44 34 102 58 108 97 115 116 84 105 109 101 115 116 97 109 112 34 58 123 125 44 34 102 58 109 101 115 115 97 103 101 34 58 123 125 44 34 102 58 114 101 97 115 111 110 34 58 123 125 44 34 102 58 115 111 117 114 99 101 34 58 123 34 102 58 99 111 109 112 111 110 101 110 116 34 58 123 125 44 34 102 58 104 111 115 116 34 58 123 125 125 44 34 102 58 116 121 112 101 34 58 123 125 125],}}]} {Node  bootstrap-e2e-minion-group-qr9r bootstrap-e2e-minion-group-qr9r   } NodeAllocatableEnforced Updated Node Allocatable limit across pods {kubelet bootstrap-e2e-minion-group-qr9r} 2020-10-21 02:48:00 +0000 UTC 2020-10-21 02:48:00 +0000 UTC 1 Normal 0001-01-01 00:00:00 +0000 UTC nil  nil  } {{ } {bootstrap-e2e-minion-group-qr9r.163fe1ffc0e3eda9  default /api/v1/namespaces/default/events/bootstrap-e2e-minion-group-qr9r.163fe1ffc0e3eda9 1c300aa2-f3f6-4502-9e10-df9be75bf84b 41699 0 2020-10-21 02:48:00 +0000 UTC <nil> <nil> map[] map[] [] []  [{kubelet Update v1 2020-10-21 02:48:00 +0000 UTC FieldsV1 &FieldsV1{Raw:*[123 34 102 58 99 111 117 110 116 34 58 123 125 44 34 102 58 102 105 114 115 116 84 105 109 101 115 116 97 109 112 34 58 123 125 44 34 102 58 105 110 118 111 108 118 101 100 79 98 106 101 99 116 34 58 123 34 102 58 107 105 110 100 34 58 123 125 44 34 102 58 110 97 109 101 34 58 123 125 44 34 102 58 117 105 100 34 58 123 125 125 44 34 102 58 108 97 115 116 84 105 109 101 115 116 97 109 112 34 58 123 125 44 34 102 58 109 101 115 115 97 103 101 34 58 123 125 44 34 102 58 114 101 97 115 111 110 34 58 123 125 44 34 102 58 115 111 117 114 99 101 34 58 123 34 102 58 99 111 109 112 111 110 101 110 116 34 58 123 125 44 34 102 58 104 111 115 116 34 58 123 125 125 44 34 102 58 116 121 112 101 34 58 123 125 125],}}]} {Node  bootstrap-e2e-minion-group-qr9r bootstrap-e2e-minion-group-qr9r   } NodeReady Node bootstrap-e2e-minion-group-qr9r status is now: NodeReady {kubelet bootstrap-e2e-minion-group-qr9r} 2020-10-21 02:48:00 +0000 UTC 2020-10-21 02:48:00 +0000 UTC 1 Normal 0001-01-01 00:00:00 +0000 UTC nil  nil  } {{ } {bootstrap-e2e-minion-group-qr9r.163fe3ffdbfe3edb  default /api/v1/namespaces/default/events/bootstrap-e2e-minion-group-qr9r.163fe3ffdbfe3edb ec4ddc9b-e1f2-41fb-9626-67cd9a8e3795 43723 0 2020-10-21 03:24:40 +0000 UTC <nil> <nil> map[] map[] [] []  [{node-problem-detector Update v1 2020-10-21 03:24:40 +0000 UTC FieldsV1 &FieldsV1{Raw:*[123 34 102 58 99 111 117 110 116 34 58 123 125 44 34 102 58 102 105 114 115 116 84 105 109 101 115 116 97 109 112 34 58 123 125 44 34 102 58 105 110 118 111 108 118 101 100 79 98 106 101 99 116 34 58 123 34 102 58 107 105 110 100 34 58 123 125 44 34 102 58 110 97 109 101 34 58 123 125 44 34 102 58 117 105 100 34 58 123 125 125 44 34 102 58 108 97 115 116 84 105 109 101 115 116 97 109 112 34 58 123 125 44 34 102 58 109 101 115 115 97 103 101 34 58 123 125 44 34 102 58 114 101 97 115 111 110 34 58 123 125 44 34 102 58 115 111 117 114 99 101 34 58 123 34 102 58 99 111 109 112 111 110 101 110 116 34 58 123 125 44 34 102 58 104 111 115 116 34 58 123 125 125 44 34 102 58 116 121 112 101 34 58 123 125 125],}}]} {Node  bootstrap-e2e-minion-group-qr9r bootstrap-e2e-minion-group-qr9r   } TaskHung kernel: INFO: task umount.aufs:21568 blocked for more than 120 seconds. {kernel-monitor bootstrap-e2e-minion-group-qr9r} 2020-10-21 03:24:40 +0000 UTC 2020-10-21 03:24:40 +0000 UTC 1 Warning 0001-01-01 00:00:00 +0000 UTC nil  nil  } {{ } {bootstrap-e2e-minion-group-qr9r.163fe3ffdbfe6666  default /api/v1/namespaces/default/events/bootstrap-e2e-minion-group-qr9r.163fe3ffdbfe6666 a86f0667-f311-4a1e-b994-e1ca6a7c5607 43722 0 2020-10-21 03:24:40 +0000 UTC <nil> <nil> map[] map[] [] []  [{node-problem-detector Update v1 2020-10-21 03:24:40 +0000 UTC FieldsV1 &FieldsV1{Raw:*[123 34 102 58 99 111 117 110 116 34 58 123 125 44 34 102 58 102 105 114 115 116 84 105 109 101 115 116 97 109 112 34 58 123 125 44 34 102 58 105 110 118 111 108 118 101 100 79 98 106 101 99 116 34 58 123 34 102 58 107 105 110 100 34 58 123 125 44 34 102 58 110 97 109 101 34 58 123 125 44 34 102 58 117 105 100 34 58 123 125 125 44 34 102 58 108 97 115 116 84 105 109 101 115 116 97 109 112 34 58 123 125 44 34 102 58 109 101 115 115 97 103 101 34 58 123 125 44 34 102 58 114 101 97 115 111 110 34 58 123 125 44 34 102 58 115 111 117 114 99 101 34 58 123 34 102 58 99 111 109 112 111 110 101 110 116 34 58 123 125 44 34 102 58 104 111 115 116 34 58 123 125 125 44 34 102 58 116 121 112 101 34 58 123 125 125],}}]} {Node  bootstrap-e2e-minion-group-qr9r bootstrap-e2e-minion-group-qr9r   } AUFSUmountHung Node condition KernelDeadlock is now: True, reason: AUFSUmountHung {kernel-monitor bootstrap-e2e-minion-group-qr9r} 2020-10-21 03:24:40 +0000 UTC 2020-10-21 03:24:40 +0000 UTC 1 Normal 0001-01-01 00:00:00 +0000 UTC nil  nil  } {{ } {bootstrap-e2e-minion-group-v5x2.163fe4008b5c76c6  default /api/v1/namespaces/default/events/bootstrap-e2e-minion-group-v5x2.163fe4008b5c76c6 cd14ae60-1c9e-4dd9-9a48-2d3296e81616 43725 0 2020-10-21 03:24:43 +0000 UTC <nil> <nil> map[] map[] [] []  [{node-problem-detector Update v1 2020-10-21 03:24:43 +0000 UTC FieldsV1 &FieldsV1{Raw:*[123 34 102 58 99 111 117 110 116 34 58 123 125 44 34 102 58 102 105 114 115 116 84 105 109 101 115 116 97 109 112 34 58 123 125 44 34 102 58 105 110 118 111 108 118 101 100 79 98 106 101 99 116 34 58 123 34 102 58 107 105 110 100 34 58 123 125 44 34 102 58 110 97 109 101 34 58 123 125 44 34 102 58 117 105 100 34 58 123 125 125 44 34 102 58 108 97 115 116 84 105 109 101 115 116 97 109 112 34 58 123 125 44 34 102 58 109 101 115 115 97 103 101 34 58 123 125 44 34 102 58 114 101 97 115 111 110 34 58 123 125 44 34 102 58 115 111 117 114 99 101 34 58 123 34 102 58 99 111 109 112 111 110 101 110 116 34 58 123 125 44 34 102 58 104 111 115 116 34 58 123 125 125 44 34 102 58 116 121 112 101 34 58 123 125 125],}}]} {Node  bootstrap-e2e-minion-group-v5x2 bootstrap-e2e-minion-group-v5x2   } TaskHung kernel: INFO: task umount.aufs:21568 blocked for more than 120 seconds. {kernel-monitor bootstrap-e2e-minion-group-v5x2} 2020-10-21 03:24:43 +0000 UTC 2020-10-21 03:24:43 +0000 UTC 1 Warning 0001-01-01 00:00:00 +0000 UTC nil  nil  } {{ } {bootstrap-e2e-minion-group-v5x2.163fe4008b5cba1b  default /api/v1/namespaces/default/events/bootstrap-e2e-minion-group-v5x2.163fe4008b5cba1b 38b6ad0c-e954-40e0-a25d-2b904a117a25 43724 0 2020-10-21 03:24:43 +0000 UTC <nil> <nil> map[] map[] [] []  [{node-problem-detector Update v1 2020-10-21 03:24:43 +0000 UTC FieldsV1 &FieldsV1{Raw:*[123 34 102 58 99 111 117 110 116 34 58 123 125 44 34 102 58 102 105 114 115 116 84 105 109 101 115 116 97 109 112 34 58 123 125 44 34 102 58 105 110 118 111 108 118 101 100 79 98 106 101 99 116 34 58 123 34 102 58 107 105 110 100 34 58 123 125 44 34 102 58 110 97 109 101 34 58 123 125 44 34 102 58 117 105 100 34 58 123 125 125 44 34 102 58 108 97 115 116 84 105 109 101 115 116 97 109 112 34 58 123 125 44 34 102 58 109 101 115 115 97 103 101 34 58 123 125 44 34 102 58 114 101 97 115 111 110 34 58 123 125 44 34 102 58 115 111 117 114 99 101 34 58 123 34 102 58 99 111 109 112 111 110 101 110 116 34 58 123 125 44 34 102 58 104 111 115 116 34 58 123 125 125 44 34 102 58 116 121 112 101 34 58 123 125 125],}}]} {Node  bootstrap-e2e-minion-group-v5x2 bootstrap-e2e-minion-group-v5x2   } AUFSUmountHung Node condition KernelDeadlock is now: True, reason: AUFSUmountHung {kernel-monitor bootstrap-e2e-minion-group-v5x2} 2020-10-21 03:24:43 +0000 UTC 2020-10-21 03:24:43 +0000 UTC 1 Normal 0001-01-01 00:00:00 +0000 UTC nil  nil  }]",
    }
    Event KubeletStart does not exist: [{{ } {bootstrap-e2e-minion-group-907m.163fe3ff1dd5745b  default /api/v1/namespaces/default/events/bootstrap-e2e-minion-group-907m.163fe3ff1dd5745b 83a3f15d-66b0-4631-a834-f93d88a0cc3e 43721 0 2020-10-21 03:24:37 +0000 UTC <nil> <nil> map[] map[] [] []  [{node-problem-detector Update v1 2020-10-21 03:24:37 +0000 UTC FieldsV1 &FieldsV1{Raw:*[123 34 102 58 99 111 117 110 116 34 58 123 125 44 34 102 58 102 105 114 115 116 84 105 109 101 115 116 97 109 112 34 58 123 125 44 34 102 58 105 110 118 111 108 118 101 100 79 98 106 101 99 116 34 58 123 34 102 58 107 105 110 100 34 58 123 125 44 34 102 58 110 97 109 101 34 58 123 125 44 34 102 58 117 105 100 34 58 123 125 125 44 34 102 58 108 97 115 116 84 105 109 101 115 116 97 109 112 34 58 123 125 44 34 102 58 109 101 115 115 97 103 101 34 58 123 125 44 34 102 58 114 101 97 115 111 110 34 58 123 125 44 34 102 58 115 111 117 114 99 101 34 58 123 34 102 58 99 111 109 112 111 110 101 110 116 34 58 123 125 44 34 102 58 104 111 115 116 34 58 123 125 125 44 34 102 58 116 121 112 101 34 58 123 125 125],}}]} {Node  bootstrap-e2e-minion-group-907m bootstrap-e2e-minion-group-907m   } TaskHung kernel: INFO: task umount.aufs:21568 blocked for more than 120 seconds. {kernel-monitor bootstrap-e2e-minion-group-907m} 2020-10-21 03:24:37 +0000 UTC 2020-10-21 03:24:37 +0000 UTC 1 Warning 0001-01-01 00:00:00 +0000 UTC nil  nil  } {{ } {bootstrap-e2e-minion-group-907m.163fe3ff1dd59d06  default /api/v1/namespaces/default/events/bootstrap-e2e-minion-group-907m.163fe3ff1dd59d06 1fc978a6-acfc-4ae5-afe9-628577a927a7 43720 0 2020-10-21 03:24:37 +0000 UTC <nil> <nil> map[] map[] [] []  [{node-problem-detector Update v1 2020-10-21 03:24:37 +0000 UTC FieldsV1 &FieldsV1{Raw:*[123 34 102 58 99 111 117 110 116 34 58 123 125 44 34 102 58 102 105 114 115 116 84 105 109 101 115 116 97 109 112 34 58 123 125 44 34 102 58 105 110 118 111 108 118 101 100 79 98 106 101 99 116 34 58 123 34 102 58 107 105 110 100 34 58 123 125 44 34 102 58 110 97 109 101 34 58 123 125 44 34 102 58 117 105 100 34 58 123 125 125 44 34 102 58 108 97 115 116 84 105 109 101 115 116 97 109 112 34 58 123 125 44 34 102 58 109 101 115 115 97 103 101 34 58 123 125 44 34 102 58 114 101 97 115 111 110 34 58 123 125 44 34 102 58 115 111 117 114 99 101 34 58 123 34 102 58 99 111 109 112 111 110 101 110 116 34 58 123 125 44 34 102 58 104 111 115 116 34 58 123 125 125 44 34 102 58 116 121 112 101 34 58 123 125 125],}}]} {Node  bootstrap-e2e-minion-group-907m bootstrap-e2e-minion-group-907m   } AUFSUmountHung Node condition KernelDeadlock is now: True, reason: AUFSUmountHung {kernel-monitor bootstrap-e2e-minion-group-907m} 2020-10-21 03:24:37 +0000 UTC 2020-10-21 03:24:37 +0000 UTC 1 Normal 0001-01-01 00:00:00 +0000 UTC nil  nil  } {{ } {bootstrap-e2e-minion-group-qr9r.163fc881ece1ed8c  default /api/v1/namespaces/default/events/bootstrap-e2e-minion-group-qr9r.163fc881ece1ed8c 29dc0842-bf9b-47e1-8655-75befe190774 41689 0 2020-10-21 02:30:14 +0000 UTC <nil> <nil> map[] map[] [] []  [{node-problem-detector Update v1 2020-10-21 02:47:59 +0000 UTC FieldsV1 &FieldsV1{Raw:*[123 34 102 58 99 111 117 110 116 34 58 123 125 44 34 102 58 102 105 114 115 116 84 105 109 101 115 116 97 109 112 34 58 123 125 44 34 102 58 105 110 118 111 108 118 101 100 79 98 106 101 99 116 34 58 123 34 102 58 107 105 110 100 34 58 123 125 44 34 102 58 110 97 109 101 34 58 123 125 44 34 102 58 117 105 100 34 58 123 125 125 44 34 102 58 108 97 115 116 84 105 109 101 115 116 97 109 112 34 58 123 125 44 34 102 58 109 101 115 115 97 103 101 34 58 123 125 44 34 102 58 114 101 97 115 111 110 34 58 123 125 44 34 102 58 115 111 117 114 99 101 34 58 123 34 102 58 99 111 109 112 111 110 101 110 116 34 58 123 125 44 34 102 58 104 111 115 116 34 58 123 125 125 44 34 102 58 116 121 112 101 34 58 123 125 125],}}]} {Node  bootstrap-e2e-minion-group-qr9r bootstrap-e2e-minion-group-qr9r   } KubeletStart Started Kubernetes kubelet. {systemd-monitor bootstrap-e2e-minion-group-qr9r} 2020-10-20 19:00:52 +0000 UTC 2020-10-21 02:47:59 +0000 UTC 9 Warning 0001-01-01 00:00:00 +0000 UTC nil  nil  } {{ } {bootstrap-e2e-minion-group-qr9r.163fd78884de5b8f  default /api/v1/namespaces/default/events/bootstrap-e2e-minion-group-qr9r.163fd78884de5b8f 01490cc4-837c-4c1b-8720-9568bcb9c48f 41688 0 2020-10-21 02:30:12 +0000 UTC <nil> <nil> map[] map[] [] []  [{kube-controller-manager Update v1 2020-10-21 02:47:57 +0000 UTC FieldsV1 &FieldsV1{Raw:*[123 34 102 58 99 111 117 110 116 34 58 123 125 44 34 102 58 102 105 114 115 116 84 105 109 101 115 116 97 109 112 34 58 123 125 44 34 102 58 105 110 118 111 108 118 101 100 79 98 106 101 99 116 34 58 123 34 102 58 97 112 105 86 101 114 115 105 111 110 34 58 123 125 44 34 102 58 107 105 110 100 34 58 123 125 44 34 102 58 110 97 109 101 34 58 123 125 44 34 102 58 117 105 100 34 58 123 125 125 44 34 102 58 108 97 115 116 84 105 109 101 115 116 97 109 112 34 58 123 125 44 34 102 58 109 101 115 115 97 103 101 34 58 123 125 44 34 102 58 114 101 97 115 111 110 34 58 123 125 44 34 102 58 115 111 117 114 99 101 34 58 123 34 102 58 99 111 109 112 111 110 101 110 116 34 58 123 125 125 44 34 102 58 116 121 112 101 34 58 123 125 125],}}]} {Node  bootstrap-e2e-minion-group-qr9r 9e1627f0-3edc-4412-b273-bf023668f093 v1  } NodeNotReady Node bootstrap-e2e-minion-group-qr9r status is now: NodeNotReady {node-controller } 2020-10-20 23:36:13 +0000 UTC 2020-10-21 02:47:57 +0000 UTC 4 Normal 0001-01-01 00:00:00 +0000 UTC nil  nil  } {{ } {bootstrap-e2e-minion-group-qr9r.163fe107799097b6  default /api/v1/namespaces/default/events/bootstrap-e2e-minion-group-qr9r.163fe107799097b6 f8267cb3-d338-477e-bace-ecd5db36d519 40972 0 2020-10-21 02:30:14 +0000 UTC <nil> <nil> map[] map[] [] []  [{kubelet Update v1 2020-10-21 02:30:14 +0000 UTC FieldsV1 &FieldsV1{Raw:*[123 34 102 58 99 111 117 110 116 34 58 123 125 44 34 102 58 102 105 114 115 116 84 105 109 101 115 116 97 109 112 34 58 123 125 44 34 102 58 105 110 118 111 108 118 101 100 79 98 106 101 99 116 34 58 123 34 102 58 107 105 110 100 34 58 123 125 44 34 102 58 110 97 109 101 34 58 123 125 44 34 102 58 117 105 100 34 58 123 125 125 44 34 102 58 108 97 115 116 84 105 109 101 115 116 97 109 112 34 58 123 125 44 34 102 58 109 101 115 115 97 103 101 34 58 123 125 44 34 102 58 114 101 97 115 111 110 34 58 123 125 44 34 102 58 115 111 117 114 99 101 34 58 123 34 102 58 99 111 109 112 111 110 101 110 116 34 58 123 125 44 34 102 58 104 111 115 116 34 58 123 125 125 44 34 102 58 116 121 112 101 34 58 123 125 125],}}]} {Node  bootstrap-e2e-minion-group-qr9r bootstrap-e2e-minion-group-qr9r   } Starting Starting kubelet. {kubelet bootstrap-e2e-minion-group-qr9r} 2020-10-21 02:30:14 +0000 UTC 2020-10-21 02:30:14 +0000 UTC 1 Normal 0001-01-01 00:00:00 +0000 UTC nil  nil  } {{ } {bootstrap-e2e-minion-group-qr9r.163fe107876103b0  default /api/v1/namespaces/default/events/bootstrap-e2e-minion-group-qr9r.163fe107876103b0 d76d703e-b07b-488b-b0f1-3f386a32a7ad 40976 0 2020-10-21 02:30:14 +0000 UTC <nil> <nil> map[] map[] [] []  [{kubelet Update v1 2020-10-21 02:30:15 +0000 UTC FieldsV1 &FieldsV1{Raw:*[123 34 102 58 99 111 117 110 116 34 58 123 125 44 34 102 58 102 105 114 115 116 84 105 109 101 115 116 97 109 112 34 58 123 125 44 34 102 58 105 110 118 111 108 118 101 100 79 98 106 101 99 116 34 58 123 34 102 58 107 105 110 100 34 58 123 125 44 34 102 58 110 97 109 101 34 58 123 125 44 34 102 58 117 105 100 34 58 123 125 125 44 34 102 58 108 97 115 116 84 105 109 101 115 116 97 109 112 34 58 123 125 44 34 102 58 109 101 115 115 97 103 101 34 58 123 125 44 34 102 58 114 101 97 115 111 110 34 58 123 125 44 34 102 58 115 111 117 114 99 101 34 58 123 34 102 58 99 111 109 112 111 110 101 110 116 34 58 123 125 44 34 102 58 104 111 115 116 34 58 123 125 125 44 34 102 58 116 121 112 101 34 58 123 125 125],}}]} {Node  bootstrap-e2e-minion-group-qr9r bootstrap-e2e-minion-group-qr9r   } NodeHasSufficientMemory Node bootstrap-e2e-minion-group-qr9r status is now: NodeHasSufficientMemory {kubelet bootstrap-e2e-minion-group-qr9r} 2020-10-21 02:30:14 +0000 UTC 2020-10-21 02:30:15 +0000 UTC 2 Normal 0001-01-01 00:00:00 +0000 UTC nil  nil  } {{ } {bootstrap-e2e-minion-group-qr9r.163fe107876146fa  default /api/v1/namespaces/default/events/bootstrap-e2e-minion-group-qr9r.163fe107876146fa e0f40f84-de50-42c1-a0c3-a2b22d0bf7b3 40977 0 2020-10-21 02:30:14 +0000 UTC <nil> <nil> map[] map[] [] []  [{kubelet Update v1 2020-10-21 02:30:15 +0000 UTC FieldsV1 &FieldsV1{Raw:*[123 34 102 58 99 111 117 110 116 34 58 123 125 44 34 102 58 102 105 114 115 116 84 105 109 101 115 116 97 109 112 34 58 123 125 44 34 102 58 105 110 118 111 108 118 101 100 79 98 106 101 99 116 34 58 123 34 102 58 107 105 110 100 34 58 123 125 44 34 102 58 110 97 109 101 34 58 123 125 44 34 102 58 117 105 100 34 58 123 125 125 44 34 102 58 108 97 115 116 84 105 109 101 115 116 97 109 112 34 58 123 125 44 34 102 58 109 101 115 115 97 103 101 34 58 123 125 44 34 102 58 114 101 97 115 111 110 34 58 123 125 44 34 102 58 115 111 117 114 99 101 34 58 123 34 102 58 99 111 109 112 111 110 101 110 116 34 58 123 125 44 34 102 58 104 111 115 116 34 58 123 125 125 44 34 102 58 116 121 112 101 34 58 123 125 125],}}]} {Node  bootstrap-e2e-minion-group-qr9r bootstrap-e2e-minion-group-qr9r   } NodeHasNoDiskPressure Node bootstrap-e2e-minion-group-qr9r status is now: NodeHasNoDiskPressure {kubelet bootstrap-e2e-minion-group-qr9r} 2020-10-21 02:30:14 +0000 UTC 2020-10-21 02:30:15 +0000 UTC 2 Normal 0001-01-01 00:00:00 +0000 UTC nil  nil  } {{ } {bootstrap-e2e-minion-group-qr9r.163fe1078761869b  default /api/v1/namespaces/default/events/bootstrap-e2e-minion-group-qr9r.163fe1078761869b 230a1c6a-284a-44e3-bcdc-d4609a1f8017 40978 0 2020-10-21 02:30:14 +0000 UTC <nil> <nil> map[] map[] [] []  [{kubelet Update v1 2020-10-21 02:30:15 +0000 UTC FieldsV1 &FieldsV1{Raw:*[123 34 102 58 99 111 117 110 116 34 58 123 125 44 34 102 58 102 105 114 115 116 84 105 109 101 115 116 97 109 112 34 58 123 125 44 34 102 58 105 110 118 111 108 118 101 100 79 98 106 101 99 116 34 58 123 34 102 58 107 105 110 100 34 58 123 125 44 34 102 58 110 97 109 101 34 58 123 125 44 34 102 58 117 105 100 34 58 123 125 125 44 34 102 58 108 97 115 116 84 105 109 101 115 116 97 109 112 34 58 123 125 44 34 102 58 109 101 115 115 97 103 101 34 58 123 125 44 34 102 58 114 101 97 115 111 110 34 58 123 125 44 34 102 58 115 111 117 114 99 101 34 58 123 34 102 58 99 111 109 112 111 110 101 110 116 34 58 123 125 44 34 102 58 104 111 115 116 34 58 123 125 125 44 34 102 58 116 121 112 101 34 58 123 125 125],}}]} {Node  bootstrap-e2e-minion-group-qr9r bootstrap-e2e-minion-group-qr9r   } NodeHasSufficientPID Node bootstrap-e2e-minion-group-qr9r status is now: NodeHasSufficientPID {kubelet bootstrap-e2e-minion-group-qr9r} 2020-10-21 02:30:14 +0000 UTC 2020-10-21 02:30:15 +0000 UTC 2 Normal 0001-01-01 00:00:00 +0000 UTC nil  nil  } {{ } {bootstrap-e2e-minion-group-qr9r.163fe107941736cd  default /api/v1/namespaces/default/events/bootstrap-e2e-minion-group-qr9r.163fe107941736cd 79daa5cd-546a-48b1-a5cd-51e7908e3da1 40979 0 2020-10-21 02:30:15 +0000 UTC <nil> <nil> map[] map[] [] []  [{kubelet Update v1 2020-10-21 02:30:15 +0000 UTC FieldsV1 &FieldsV1{Raw:*[123 34 102 58 99 111 117 110 116 34 58 123 125 44 34 102 58 102 105 114 115 116 84 105 109 101 115 116 97 109 112 34 58 123 125 44 34 102 58 105 110 118 111 108 118 101 100 79 98 106 101 99 116 34 58 123 34 102 58 107 105 110 100 34 58 123 125 44 34 102 58 110 97 109 101 34 58 123 125 44 34 102 58 117 105 100 34 58 123 125 125 44 34 102 58 108 97 115 116 84 105 109 101 115 116 97 109 112 34 58 123 125 44 34 102 58 109 101 115 115 97 103 101 34 58 123 125 44 34 102 58 114 101 97 115 111 110 34 58 123 125 44 34 102 58 115 111 117 114 99 101 34 58 123 34 102 58 99 111 109 112 111 110 101 110 116 34 58 123 125 44 34 102 58 104 111 115 116 34 58 123 125 125 44 34 102 58 116 121 112 101 34 58 123 125 125],}}]} {Node  bootstrap-e2e-minion-group-qr9r bootstrap-e2e-minion-group-qr9r   } NodeNotReady Node bootstrap-e2e-minion-group-qr9r status is now: NodeNotReady {kubelet bootstrap-e2e-minion-group-qr9r} 2020-10-21 02:30:15 +0000 UTC 2020-10-21 02:30:15 +0000 UTC 1 Normal 0001-01-01 00:00:00 +0000 UTC nil  nil  } {{ } {bootstrap-e2e-minion-group-qr9r.163fe107a9a19689  default /api/v1/namespaces/default/events/bootstrap-e2e-minion-group-qr9r.163fe107a9a19689 eb600255-cad4-4d2b-bf46-62b9756012c4 40980 0 2020-10-21 02:30:15 +0000 UTC <nil> <nil> map[] map[] [] []  [{kubelet Update v1 2020-10-21 02:30:15 +0000 UTC FieldsV1 &FieldsV1{Raw:*[123 34 102 58 99 111 117 110 116 34 58 123 125 44 34 102 58 102 105 114 115 116 84 105 109 101 115 116 97 109 112 34 58 123 125 44 34 102 58 105 110 118 111 108 118 101 100 79 98 106 101 99 116 34 58 123 34 102 58 107 105 110 100 34 58 123 125 44 34 102 58 110 97 109 101 34 58 123 125 44 34 102 58 117 105 100 34 58 123 125 125 44 34 102 58 108 97 115 116 84 105 109 101 115 116 97 109 112 34 58 123 125 44 34 102 58 109 101 115 115 97 103 101 34 58 123 125 44 34 102 58 114 101 97 115 111 110 34 58 123 125 44 34 102 58 115 111 117 114 99 101 34 58 123 34 102 58 99 111 109 112 111 110 101 110 116 34 58 123 125 44 34 102 58 104 111 115 116 34 58 123 125 125 44 34 102 58 116 121 112 101 34 58 123 125 125],}}]} {Node  bootstrap-e2e-minion-group-qr9r bootstrap-e2e-minion-group-qr9r   } NodeAllocatableEnforced Updated Node Allocatable limit across pods {kubelet bootstrap-e2e-minion-group-qr9r} 2020-10-21 02:30:15 +0000 UTC 2020-10-21 02:30:15 +0000 UTC 1 Normal 0001-01-01 00:00:00 +0000 UTC nil  nil  } {{ } {bootstrap-e2e-minion-group-qr9r.163fe107b063d789  default /api/v1/namespaces/default/events/bootstrap-e2e-minion-group-qr9r.163fe107b063d789 6a5abc79-97f2-49b9-9ee4-8aab289dae2b 40981 0 2020-10-21 02:30:15 +0000 UTC <nil> <nil> map[] map[] [] []  [{kubelet Update v1 2020-10-21 02:30:15 +0000 UTC FieldsV1 &FieldsV1{Raw:*[123 34 102 58 99 111 117 110 116 34 58 123 125 44 34 102 58 102 105 114 115 116 84 105 109 101 115 116 97 109 112 34 58 123 125 44 34 102 58 105 110 118 111 108 118 101 100 79 98 106 101 99 116 34 58 123 34 102 58 107 105 110 100 34 58 123 125 44 34 102 58 110 97 109 101 34 58 123 125 44 34 102 58 117 105 100 34 58 123 125 125 44 34 102 58 108 97 115 116 84 105 109 101 115 116 97 109 112 34 58 123 125 44 34 102 58 109 101 115 115 97 103 101 34 58 123 125 44 34 102 58 114 101 97 115 111 110 34 58 123 125 44 34 102 58 115 111 117 114 99 101 34 58 123 34 102 58 99 111 109 112 111 110 101 110 116 34 58 123 125 44 34 102 58 104 111 115 116 34 58 123 125 125 44 34 102 58 116 121 112 101 34 58 123 125 125],}}]} {Node  bootstrap-e2e-minion-group-qr9r bootstrap-e2e-minion-group-qr9r   } NodeReady Node bootstrap-e2e-minion-group-qr9r status is now: NodeReady {kubelet bootstrap-e2e-minion-group-qr9r} 2020-10-21 02:30:15 +0000 UTC 2020-10-21 02:30:15 +0000 UTC 1 Normal 0001-01-01 00:00:00 +0000 UTC nil  nil  } {{ } {bootstrap-e2e-minion-group-qr9r.163fe1ff9311a32d  default /api/v1/namespaces/default/events/bootstrap-e2e-minion-group-qr9r.163fe1ff9311a32d eb5e65e5-6cfb-4c12-8738-bc6410a14eda 41690 0 2020-10-21 02:48:00 +0000 UTC <nil> <nil> map[] map[] [] []  [{kubelet Update v1 2020-10-21 02:48:00 +0000 UTC FieldsV1 &FieldsV1{Raw:*[123 34 102 58 99 111 117 110 116 34 58 123 125 44 34 102 58 102 105 114 115 116 84 105 109 101 115 116 97 109 112 34 58 123 125 44 34 102 58 105 110 118 111 108 118 101 100 79 98 106 101 99 116 34 58 123 34 102 58 107 105 110 100 34 58 123 125 44 34 102 58 110 97 109 101 34 58 123 125 44 34 102 58 117 105 100 34 58 123 125 125 44 34 102 58 108 97 115 116 84 105 109 101 115 116 97 109 112 34 58 123 125 44 34 102 58 109 101 115 115 97 103 101 34 58 123 125 44 34 102 58 114 101 97 115 111 110 34 58 123 125 44 34 102 58 115 111 117 114 99 101 34 58 123 34 102 58 99 111 109 112 111 110 101 110 116 34 58 123 125 44 34 102 58 104 111 115 116 34 58 123 125 125 44 34 102 58 116 121 112 101 34 58 123 125 125],}}]} {Node  bootstrap-e2e-minion-group-qr9r bootstrap-e2e-minion-group-qr9r   } Starting Starting kubelet. {kubelet bootstrap-e2e-minion-group-qr9r} 2020-10-21 02:48:00 +0000 UTC 2020-10-21 02:48:00 +0000 UTC 1 Normal 0001-01-01 00:00:00 +0000 UTC nil  nil  } {{ } {bootstrap-e2e-minion-group-qr9r.163fe1ffa1ee791c  default /api/v1/namespaces/default/events/bootstrap-e2e-minion-group-qr9r.163fe1ffa1ee791c 775d89fb-44a0-4679-ad52-9eadd9864991 41695 0 2020-10-21 02:48:00 +0000 UTC <nil> <nil> map[] map[] [] []  [{kubelet Update v1 2020-10-21 02:48:00 +0000 UTC FieldsV1 &FieldsV1{Raw:*[123 34 102 58 99 111 117 110 116 34 58 123 125 44 34 102 58 102 105 114 115 116 84 105 109 101 115 116 97 109 112 34 58 123 125 44 34 102 58 105 110 118 111 108 118 101 100 79 98 106 101 99 116 34 58 123 34 102 58 107 105 110 100 34 58 123 125 44 34 102 58 110 97 109 101 34 58 123 125 44 34 102 58 117 105 100 34 58 123 125 125 44 34 102 58 108 97 115 116 84 105 109 101 115 116 97 109 112 34 58 123 125 44 34 102 58 109 101 115 115 97 103 101 34 58 123 125 44 34 102 58 114 101 97 115 111 110 34 58 123 125 44 34 102 58 115 111 117 114 99 101 34 58 123 34 102 58 99 111 109 112 111 110 101 110 116 34 58 123 125 44 34 102 58 104 111 115 116 34 58 123 125 125 44 34 102 58 116 121 112 101 34 58 123 125 125],}}]} {Node  bootstrap-e2e-minion-group-qr9r bootstrap-e2e-minion-group-qr9r   } NodeHasSufficientMemory Node bootstrap-e2e-minion-group-qr9r status is now: NodeHasSufficientMemory {kubelet bootstrap-e2e-minion-group-qr9r} 2020-10-21 02:48:00 +0000 UTC 2020-10-21 02:48:00 +0000 UTC 2 Normal 0001-01-01 00:00:00 +0000 UTC nil  nil  } {{ } {bootstrap-e2e-minion-group-qr9r.163fe1ffa1eec561  default /api/v1/namespaces/default/events/bootstrap-e2e-minion-group-qr9r.163fe1ffa1eec561 02a5755a-f8b3-4e56-a47c-a78c62ba2760 41694 0 2020-10-21 02:48:00 +0000 UTC <nil> <nil> map[] map[] [] []  [{kubelet Update v1 2020-10-21 02:48:00 +0000 UTC FieldsV1 &FieldsV1{Raw:*[123 34 102 58 99 111 117 110 116 34 58 123 125 44 34 102 58 102 105 114 115 116 84 105 109 101 115 116 97 109 112 34 58 123 125 44 34 102 58 105 110 118 111 108 118 101 100 79 98 106 101 99 116 34 58 123 34 102 58 107 105 110 100 34 58 123 125 44 34 102 58 110 97 109 101 34 58 123 125 44 34 102 58 117 105 100 34 58 123 125 125 44 34 102 58 108 97 115 116 84 105 109 101 115 116 97 109 112 34 58 123 125 44 34 102 58 109 101 115 115 97 103 101 34 58 123 125 44 34 102 58 114 101 97 115 111 110 34 58 123 125 44 34 102 58 115 111 117 114 99 101 34 58 123 34 102 58 99 111 109 112 111 110 101 110 116 34 58 123 125 44 34 102 58 104 111 115 116 34 58 123 125 125 44 34 102 58 116 121 112 101 34 58 123 125 125],}}]} {Node  bootstrap-e2e-minion-group-qr9r bootstrap-e2e-minion-group-qr9r   } NodeHasNoDiskPressure Node bootstrap-e2e-minion-group-qr9r status is now: NodeHasNoDiskPressure {kubelet bootstrap-e2e-minion-group-qr9r} 2020-10-21 02:48:00 +0000 UTC 2020-10-21 02:48:00 +0000 UTC 2 Normal 0001-01-01 00:00:00 +0000 UTC nil  nil  } {{ } {bootstrap-e2e-minion-group-qr9r.163fe1ffa1eef9f1  default /api/v1/namespaces/default/events/bootstrap-e2e-minion-group-qr9r.163fe1ffa1eef9f1 ed1e49d8-1fe6-41fc-9bc4-643e6a859c1e 41696 0 2020-10-21 02:48:00 +0000 UTC <nil> <nil> map[] map[] [] []  [{kubelet Update v1 2020-10-21 02:48:00 +0000 UTC FieldsV1 &FieldsV1{Raw:*[123 34 102 58 99 111 117 110 116 34 58 123 125 44 34 102 58 102 105 114 115 116 84 105 109 101 115 116 97 109 112 34 58 123 125 44 34 102 58 105 110 118 111 108 118 101 100 79 98 106 101 99 116 34 58 123 34 102 58 107 105 110 100 34 58 123 125 44 34 102 58 110 97 109 101 34 58 123 125 44 34 102 58 117 105 100 34 58 123 125 125 44 34 102 58 108 97 115 116 84 105 109 101 115 116 97 109 112 34 58 123 125 44 34 102 58 109 101 115 115 97 103 101 34 58 123 125 44 34 102 58 114 101 97 115 111 110 34 58 123 125 44 34 102 58 115 111 117 114 99 101 34 58 123 34 102 58 99 111 109 112 111 110 101 110 116 34 58 123 125 44 34 102 58 104 111 115 116 34 58 123 125 125 44 34 102 58 116 121 112 101 34 58 123 125 125],}}]} {Node  bootstrap-e2e-minion-group-qr9r bootstrap-e2e-minion-group-qr9r   } NodeHasSufficientPID Node bootstrap-e2e-minion-group-qr9r status is now: NodeHasSufficientPID {kubelet bootstrap-e2e-minion-group-qr9r} 2020-10-21 02:48:00 +0000 UTC 2020-10-21 02:48:00 +0000 UTC 2 Normal 0001-01-01 00:00:00 +0000 UTC nil  nil  } {{ } {bootstrap-e2e-minion-group-qr9r.163fe1ffab270870  default /api/v1/namespaces/default/events/bootstrap-e2e-minion-group-qr9r.163fe1ffab270870 0132c140-0e08-4bc4-98f8-28b5ee2d1664 41697 0 2020-10-21 02:48:00 +0000 UTC <nil> <nil> map[] map[] [] []  [{kubelet Update v1 2020-10-21 02:48:00 +0000 UTC FieldsV1 &FieldsV1{Raw:*[123 34 102 58 99 111 117 110 116 34 58 123 125 44 34 102 58 102 105 114 115 116 84 105 109 101 115 116 97 109 112 34 58 123 125 44 34 102 58 105 110 118 111 108 118 101 100 79 98 106 101 99 116 34 58 123 34 102 58 107 105 110 100 34 58 123 125 44 34 102 58 110 97 109 101 34 58 123 125 44 34 102 58 117 105 100 34 58 123 125 125 44 34 102 58 108 97 115 116 84 105 109 101 115 116 97 109 112 34 58 123 125 44 34 102 58 109 101 115 115 97 103 101 34 58 123 125 44 34 102 58 114 101 97 115 111 110 34 58 123 125 44 34 102 58 115 111 117 114 99 101 34 58 123 34 102 58 99 111 109 112 111 110 101 110 116 34 58 123 125 44 34 102 58 104 111 115 116 34 58 123 125 125 44 34 102 58 116 121 112 101 34 58 123 125 125],}}]} {Node  bootstrap-e2e-minion-group-qr9r bootstrap-e2e-minion-group-qr9r   } NodeNotReady Node bootstrap-e2e-minion-group-qr9r status is now: NodeNotReady {kubelet bootstrap-e2e-minion-group-qr9r} 2020-10-21 02:48:00 +0000 UTC 2020-10-21 02:48:00 +0000 UTC 1 Normal 0001-01-01 00:00:00 +0000 UTC nil  nil  } {{ } {bootstrap-e2e-minion-group-qr9r.163fe1ffb7a228f2  default /api/v1/namespaces/default/events/bootstrap-e2e-minion-group-qr9r.163fe1ffb7a228f2 2106ad90-8fb0-44e4-837c-b9892e24fb5e 41698 0 2020-10-21 02:48:00 +0000 UTC <nil> <nil> map[] map[] [] []  [{kubelet Update v1 2020-10-21 02:48:00 +0000 UTC FieldsV1 &FieldsV1{Raw:*[123 34 102 58 99 111 117 110 116 34 58 123 125 44 34 102 58 102 105 114 115 116 84 105 109 101 115 116 97 109 112 34 58 123 125 44 34 102 58 105 110 118 111 108 118 101 100 79 98 106 101 99 116 34 58 123 34 102 58 107 105 110 100 34 58 123 125 44 34 102 58 110 97 109 101 34 58 123 125 44 34 102 58 117 105 100 34 58 123 125 125 44 34 102 58 108 97 115 116 84 105 109 101 115 116 97 109 112 34 58 123 125 44 34 102 58 109 101 115 115 97 103 101 34 58 123 125 44 34 102 58 114 101 97 115 111 110 34 58 123 125 44 34 102 58 115 111 117 114 99 101 34 58 123 34 102 58 99 111 109 112 111 110 101 110 116 34 58 123 125 44 34 102 58 104 111 115 116 34 58 123 125 125 44 34 102 58 116 121 112 101 34 58 123 125 125],}}]} {Node  bootstrap-e2e-minion-group-qr9r bootstrap-e2e-minion-group-qr9r   } NodeAllocatableEnforced Updated Node Allocatable limit across pods {kubelet bootstrap-e2e-minion-group-qr9r} 2020-10-21 02:48:00 +0000 UTC 2020-10-21 02:48:00 +0000 UTC 1 Normal 0001-01-01 00:00:00 +0000 UTC nil  nil  } {{ } {bootstrap-e2e-minion-group-qr9r.163fe1ffc0e3eda9  default /api/v1/namespaces/default/events/bootstrap-e2e-minion-group-qr9r.163fe1ffc0e3eda9 1c300aa2-f3f6-4502-9e10-df9be75bf84b 41699 0 2020-10-21 02:48:00 +0000 UTC <nil> <nil> map[] map[] [] []  [{kubelet Update v1 2020-10-21 02:48:00 +0000 UTC FieldsV1 &FieldsV1{Raw:*[123 34 102 58 99 111 117 110 116 34 58 123 125 44 34 102 58 102 105 114 115 116 84 105 109 101 115 116 97 109 112 34 58 123 125 44 34 102 58 105 110 118 111 108 118 101 100 79 98 106 101 99 116 34 58 123 34 102 58 107 105 110 100 34 58 123 125 44 34 102 58 110 97 109 101 34 58 123 125 44 34 102 58 117 105 100 34 58 123 125 125 44 34 102 58 108 97 115 116 84 105 109 101 115 116 97 109 112 34 58 123 125 44 34 102 58 109 101 115 115 97 103 101 34 58 123 125 44 34 102 58 114 101 97 115 111 110 34 58 123 125 44 34 102 58 115 111 117 114 99 101 34 58 123 34 102 58 99 111 109 112 111 110 101 110 116 34 58 123 125 44 34 102 58 104 111 115 116 34 58 123 125 125 44 34 102 58 116 121 112 101 34 58 123 125 125],}}]} {Node  bootstrap-e2e-minion-group-qr9r bootstrap-e2e-minion-group-qr9r   } NodeReady Node bootstrap-e2e-minion-group-qr9r status is now: NodeReady {kubelet bootstrap-e2e-minion-group-qr9r} 2020-10-21 02:48:00 +0000 UTC 2020-10-21 02:48:00 +0000 UTC 1 Normal 0001-01-01 00:00:00 +0000 UTC nil  nil  } {{ } {bootstrap-e2e-minion-group-qr9r.163fe3ffdbfe3edb  default /api/v1/namespaces/default/events/bootstrap-e2e-minion-group-qr9r.163fe3ffdbfe3edb ec4ddc9b-e1f2-41fb-9626-67cd9a8e3795 43723 0 2020-10-21 03:24:40 +0000 UTC <nil> <nil> map[] map[] [] []  [{node-problem-detector Update v1 2020-10-21 03:24:40 +0000 UTC FieldsV1 &FieldsV1{Raw:*[123 34 102 58 99 111 117 110 116 34 58 123 125 44 34 102 58 102 105 114 115 116 84 105 109 101 115 116 97 109 112 34 58 123 125 44 34 102 58 105 110 118 111 108 118 101 100 79 98 106 101 99 116 34 58 123 34 102 58 107 105 110 100 34 58 123 125 44 34 102 58 110 97 109 101 34 58 123 125 44 34 102 58 117 105 100 34 58 123 125 125 44 34 102 58 108 97 115 116 84 105 109 101 115 116 97 109 112 34 58 123 125 44 34 102 58 109 101 115 115 97 103 101 34 58 123 125 44 34 102 58 114 101 97 115 111 110 34 58 123 125 44 34 102 58 115 111 117 114 99 101 34 58 123 34 102 58 99 111 109 112 111 110 101 110 116 34 58 123 125 44 34 102 58 104 111 115 116 34 58 123 125 125 44 34 102 58 116 121 112 101 34 58 123 125 125],}}]} {Node  bootstrap-e2e-minion-group-qr9r bootstrap-e2e-minion-group-qr9r   } TaskHung kernel: INFO: task umount.aufs:21568 blocked for more than 120 seconds. {kernel-monitor bootstrap-e2e-minion-group-qr9r} 2020-10-21 03:24:40 +0000 UTC 2020-10-21 03:24:40 +0000 UTC 1 Warning 0001-01-01 00:00:00 +0000 UTC nil  nil  } {{ } {bootstrap-e2e-minion-group-qr9r.163fe3ffdbfe6666  default /api/v1/namespaces/default/events/bootstrap-e2e-minion-group-qr9r.163fe3ffdbfe6666 a86f0667-f311-4a1e-b994-e1ca6a7c5607 43722 0 2020-10-21 03:24:40 +0000 UTC <nil> <nil> map[] map[] [] []  [{node-problem-detector Update v1 2020-10-21 03:24:40 +0000 UTC FieldsV1 &FieldsV1{Raw:*[123 34 102 58 99 111 117 110 116 34 58 123 125 44 34 102 58 102 105 114 115 116 84 105 109 101 115 116 97 109 112 34 58 123 125 44 34 102 58 105 110 118 111 108 118 101 100 79 98 106 101 99 116 34 58 123 34 102 58 107 105 110 100 34 58 123 125 44 34 102 58 110 97 109 101 34 58 123 125 44 34 102 58 117 105 100 34 58 123 125 125 44 34 102 58 108 97 115 116 84 105 109 101 115 116 97 109 112 34 58 123 125 44 34 102 58 109 101 115 115 97 103 101 34 58 123 125 44 34 102 58 114 101 97 115 111 110 34 58 123 125 44 34 102 58 115 111 117 114 99 101 34 58 123 34 102 58 99 111 109 112 111 110 101 110 116 34 58 123 125 44 34 102 58 104 111 115 116 34 58 123 125 125 44 34 102 58 116 121 112 101 34 58 123 125 125],}}]} {Node  bootstrap-e2e-minion-group-qr9r bootstrap-e2e-minion-group-qr9r   } AUFSUmountHung Node condition KernelDeadlock is now: True, reason: AUFSUmountHung {kernel-monitor bootstrap-e2e-minion-group-qr9r} 2020-10-21 03:24:40 +0000 UTC 2020-10-21 03:24:40 +0000 UTC 1 Normal 0001-01-01 00:00:00 +0000 UTC nil  nil  } {{ } {bootstrap-e2e-minion-group-v5x2.163fe4008b5c76c6  default /api/v1/namespaces/default/events/bootstrap-e2e-minion-group-v5x2.163fe4008b5c76c6 cd14ae60-1c9e-4dd9-9a48-2d3296e81616 43725 0 2020-10-21 03:24:43 +0000 UTC <nil> <nil> map[] map[] [] []  [{node-problem-detector Update v1 2020-10-21 03:24:43 +0000 UTC FieldsV1 &FieldsV1{Raw:*[123 34 102 58 99 111 117 110 116 34 58 123 125 44 34 102 58 102 105 114 115 116 84 105 109 101 115 116 97 109 112 34 58 123 125 44 34 102 58 105 110 118 111 108 118 101 100 79 98 106 101 99 116 34 58 123 34 102 58 107 105 110 100 34 58 123 125 44 34 102 58 110 97 109 101 34 58 123 125 44 34 102 58 117 105 100 34 58 123 125 125 44 34 102 58 108 97 115 116 84 105 109 101 115 116 97 109 112 34 58 123 125 44 34 102 58 109 101 115 115 97 103 101 34 58 123 125 44 34 102 58 114 101 97 115 111 110 34 58 123 125 44 34 102 58 115 111 117 114 99 101 34 58 123 34 102 58 99 111 109 112 111 110 101 110 116 34 58 123 125 44 34 102 58 104 111 115 116 34 58 123 125 125 44 34 102 58 116 121 112 101 34 58 123 125 125],}}]} {Node  bootstrap-e2e-minion-group-v5x2 bootstrap-e2e-minion-group-v5x2   } TaskHung kernel: INFO: task umount.aufs:21568 blocked for more than 120 seconds. {kernel-monitor bootstrap-e2e-minion-group-v5x2} 2020-10-21 03:24:43 +0000 UTC 2020-10-21 03:24:43 +0000 UTC 1 Warning 0001-01-01 00:00:00 +0000 UTC nil  nil  } {{ } {bootstrap-e2e-minion-group-v5x2.163fe4008b5cba1b  default /api/v1/namespaces/default/events/bootstrap-e2e-minion-group-v5x2.163fe4008b5cba1b 38b6ad0c-e954-40e0-a25d-2b904a117a25 43724 0 2020-10-21 03:24:43 +0000 UTC <nil> <nil> map[] map[] [] []  [{node-problem-detector Update v1 2020-10-21 03:24:43 +0000 UTC FieldsV1 &FieldsV1{Raw:*[123 34 102 58 99 111 117 110 116 34 58 123 125 44 34 102 58 102 105 114 115 116 84 105 109 101 115 116 97 109 112 34 58 123 125 44 34 102 58 105 110 118 111 108 118 101 100 79 98 106 101 99 116 34 58 123 34 102 58 107 105 110 100 34 58 123 125 44 34 102 58 110 97 109 101 34 58 123 125 44 34 102 58 117 105 100 34 58 123 125 125 44 34 102 58 108 97 115 116 84 105 109 101 115 116 97 109 112 34 58 123 125 44 34 102 58 109 101 115 115 97 103 101 34 58 123 125 44 34 102 58 114 101 97 115 111 110 34 58 123 125 44 34 102 58 115 111 117 114 99 101 34 58 123 34 102 58 99 111 109 112 111 110 101 110 116 34 58 123 125 44 34 102 58 104 111 115 116 34 58 123 125 125 44 34 102 58 116 121 112 101 34 58 123 125 125],}}]} {Node  bootstrap-e2e-minion-group-v5x2 bootstrap-e2e-minion-group-v5x2   } AUFSUmountHung Node condition KernelDeadlock is now: True, reason: AUFSUmountHung {kernel-monitor bootstrap-e2e-minion-group-v5x2} 2020-10-21 03:24:43 +0000 UTC 2020-10-21 03:24:43 +0000 UTC 1 Normal 0001-01-01 00:00:00 +0000 UTC nil  nil  }]
/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/node/node_problem_detector.go:140
				
				Click to see stdout/stderrfrom junit_01.xml

Filter through