This job view page is being replaced by Spyglass soon. Check out the new job view.
ResultFAILURE
Tests 6 failed / 934 succeeded
Started2020-03-25 01:38
Elapsed15h12m
Revision
Builder37b3e42d-6e39-11ea-80f4-d2b8db82001b
links{u'resultstore': {u'url': u'https://source.cloud.google.com/results/invocations/7fe96009-c359-4fae-8fd1-3ffe71527294/targets/test'}}
resultstorehttps://source.cloud.google.com/results/invocations/7fe96009-c359-4fae-8fd1-3ffe71527294/targets/test
infra-commite86e60c9c
job-versionv1.16.9-beta.0.9+3b7cbcf52403ff
master_os_imagecos-77-12371-175-0
node_os_imagecos-73-11647-163-0
revisionv1.16.9-beta.0.9+3b7cbcf52403ff

Test Failures


Kubernetes e2e suite [k8s.io] [sig-cloud-provider-gcp] Upgrade [Feature:Upgrade] master upgrade should maintain a functioning cluster [Feature:MasterUpgrade] 18m35s

go run hack/e2e.go -v --test --test_args='--ginkgo.focus=Kubernetes\se2e\ssuite\s\[k8s\.io\]\s\[sig\-cloud\-provider\-gcp\]\sUpgrade\s\[Feature\:Upgrade\]\smaster\supgrade\sshould\smaintain\sa\sfunctioning\scluster\s\[Feature\:MasterUpgrade\]$'
/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/cloud/gcp/cluster_upgrade.go:91
Mar 25 01:54:21.715: Unexpected error:
    <*errors.errorString | 0xc0021c66e0>: {
        s: "error running /workspace/kubernetes_skew/cluster/gce/upgrade.sh [-M v1.17.5-beta.0.13+222605b0aa03bc]; got error exit status 1, stdout \"Fetching the previously installed CoreDNS version\\n\\n***WARNING***\\nUpgrading Kubernetes with this script might result in an upgrade to a new etcd version.\\nSome etcd version upgrades, such as 3.0.x to 3.1.x, DO NOT offer a downgrade path.\\nTo pin the etcd version to your current one (e.g. v3.0.17), set the following variables\\nbefore running this script:\\n\\n# example: pin to etcd v3.0.17\\nexport ETCD_IMAGE=3.0.17\\nexport ETCD_VERSION=3.0.17\\n\\nAlternatively, if you choose to allow an etcd upgrade that doesn't support downgrade,\\nyou might still be able to downgrade Kubernetes by pinning to the newer etcd version.\\nIn all cases, it is strongly recommended to have an etcd backup before upgrading.\\n\\n== Pre-Upgrade Node OS and Kubelet Versions ==\\nname: \\\"bootstrap-e2e-master\\\", osImage: \\\"Container-Optimized OS from Google\\\", kubeletVersion: \\\"v1.16.9-beta.0.9+3b7cbcf52403ff\\\"\\nname: \\\"bootstrap-e2e-minion-group-8lfn\\\", osImage: \\\"Container-Optimized OS from Google\\\", kubeletVersion: \\\"v1.16.9-beta.0.9+3b7cbcf52403ff\\\"\\nname: \\\"bootstrap-e2e-minion-group-hlrz\\\", osImage: \\\"Container-Optimized OS from Google\\\", kubeletVersion: \\\"v1.16.9-beta.0.9+3b7cbcf52403ff\\\"\\nname: \\\"bootstrap-e2e-minion-group-vgzb\\\", osImage: \\\"Container-Optimized OS from Google\\\", kubeletVersion: \\\"v1.16.9-beta.0.9+3b7cbcf52403ff\\\"\\nFound subnet for region us-west1 in network bootstrap-e2e: bootstrap-e2e\\n== Upgrading master to 'https://storage.googleapis.com/kubernetes-release-dev/ci/v1.17.5-beta.0.13+222605b0aa03bc/kubernetes-server-linux-amd64.tar.gz'. Do not interrupt, deleting master instance. ==\\n== Upgrading master environment variables. ==\\n== Waiting for new master to respond to API requests ==\\n........................== Done ==\\nWaiting for CoreDNS to update\\nFetching the latest installed CoreDNS version\\n== Downloading the CoreDNS migration tool ==\\n== Upgrading the CoreDNS ConfigMap ==\\nconfigmap/coredns configured\\n== The CoreDNS Config has been updated ==\\n== Validating cluster post-upgrade ==\\nValidating gce cluster, MULTIZONE=\\nFound 4 node(s).\\nNAME                              STATUS                     ROLES    AGE     VERSION\\nbootstrap-e2e-master              Ready,SchedulingDisabled   <none>   9m51s   v1.17.5-beta.0.13+222605b0aa03bc\\nbootstrap-e2e-minion-group-8lfn   Ready                      <none>   9m55s   v1.16.9-beta.0.9+3b7cbcf52403ff\\nbootstrap-e2e-minion-group-hlrz   Ready                      <none>   9m55s   v1.16.9-beta.0.9+3b7cbcf52403ff\\nbootstrap-e2e-minion-group-vgzb   Ready                      <none>   9m54s   v1.16.9-beta.0.9+3b7cbcf52403ff\\nValidate output:\\nNAME                 STATUS    MESSAGE             ERROR\\netcd-0               Healthy   {\\\"health\\\":\\\"true\\\"}   \\nscheduler            Healthy   ok                  \\ncontroller-manager   Healthy   ok                  \\netcd-1               Healthy   {\\\"health\\\":\\\"true\\\"}   \\n\\x1b[0;32mCluster validation succeeded\\x1b[0m\\n== Post-Upgrade Node OS and Kubelet Versions ==\\nname: \\\"bootstrap-e2e-master\\\", osImage: \\\"Container-Optimized OS from Google\\\", kubeletVersion: \\\"v1.17.5-beta.0.13+222605b0aa03bc\\\"\\nname: \\\"bootstrap-e2e-minion-group-8lfn\\\", osImage: \\\"Container-Optimized OS from Google\\\", kubeletVersion: \\\"v1.16.9-beta.0.9+3b7cbcf52403ff\\\"\\nname: \\\"bootstrap-e2e-minion-group-hlrz\\\", osImage: \\\"Container-Optimized OS from Google\\\", kubeletVersion: \\\"v1.16.9-beta.0.9+3b7cbcf52403ff\\\"\\nname: \\\"bootstrap-e2e-minion-group-vgzb\\\", osImage: \\\"Container-Optimized OS from Google\\\", kubeletVersion: \\\"v1.16.9-beta.0.9+3b7cbcf52403ff\\\"\\n\", stderr \"Project: k8s-gce-soak-1-5\\nNetwork Project: k8s-gce-soak-1-5\\nZone: us-west1-b\\nINSTANCE_GROUPS=bootstrap-e2e-minion-group\\nNODE_NAMES=bootstrap-e2e-minion-group-8lfn bootstrap-e2e-minion-group-hlrz bootstrap-e2e-minion-group-vgzb\\nTrying to find master named 'bootstrap-e2e-master'\\nLooking for address 'bootstrap-e2e-master-ip'\\nUsing master: bootstrap-e2e-master (external IP: 34.82.199.207; internal IP: (not set))\\nDeleted [https://www.googleapis.com/compute/v1/projects/k8s-gce-soak-1-5/zones/us-west1-b/instances/bootstrap-e2e-master].\\nWARNING: You have selected a disk size of under [200GB]. This may result in poor I/O performance. For more information, see: https://developers.google.com/compute/docs/disks#performance.\\nCreated [https://www.googleapis.com/compute/v1/projects/k8s-gce-soak-1-5/zones/us-west1-b/instances/bootstrap-e2e-master].\\nWARNING: Some requests generated warnings:\\n - Disk size: '20 GB' is larger than image size: '10 GB'. You might need to resize the root repartition manually if the operating system does not support automatic resizing. See https://cloud.google.com/compute/docs/disks/add-persistent-disk#resize_pd for details.\\n - The resource 'projects/cos-cloud/global/images/cos-77-12371-175-0' is deprecated. A suggested replacement is 'projects/cos-cloud/global/images/cos-77-12371-183-0'.\\n\\nNAME                  ZONE        MACHINE_TYPE   PREEMPTIBLE  INTERNAL_IP  EXTERNAL_IP    STATUS\\nbootstrap-e2e-master  us-west1-b  n1-standard-1               10.138.0.6   34.82.199.207  RUNNING\\nWarning: kubectl apply should be used on resource created by either kubectl create --save-config or kubectl apply\\nProject: k8s-gce-soak-1-5\\nNetwork Project: k8s-gce-soak-1-5\\nZone: us-west1-b\\n/workspace/kubernetes_skew/cluster/gce/upgrade.sh: line 452: download_dir: unbound variable\\n\"",
    }
    error running /workspace/kubernetes_skew/cluster/gce/upgrade.sh [-M v1.17.5-beta.0.13+222605b0aa03bc]; got error exit status 1, stdout "Fetching the previously installed CoreDNS version\n\n***WARNING***\nUpgrading Kubernetes with this script might result in an upgrade to a new etcd version.\nSome etcd version upgrades, such as 3.0.x to 3.1.x, DO NOT offer a downgrade path.\nTo pin the etcd version to your current one (e.g. v3.0.17), set the following variables\nbefore running this script:\n\n# example: pin to etcd v3.0.17\nexport ETCD_IMAGE=3.0.17\nexport ETCD_VERSION=3.0.17\n\nAlternatively, if you choose to allow an etcd upgrade that doesn't support downgrade,\nyou might still be able to downgrade Kubernetes by pinning to the newer etcd version.\nIn all cases, it is strongly recommended to have an etcd backup before upgrading.\n\n== Pre-Upgrade Node OS and Kubelet Versions ==\nname: \"bootstrap-e2e-master\", osImage: \"Container-Optimized OS from Google\", kubeletVersion: \"v1.16.9-beta.0.9+3b7cbcf52403ff\"\nname: \"bootstrap-e2e-minion-group-8lfn\", osImage: \"Container-Optimized OS from Google\", kubeletVersion: \"v1.16.9-beta.0.9+3b7cbcf52403ff\"\nname: \"bootstrap-e2e-minion-group-hlrz\", osImage: \"Container-Optimized OS from Google\", kubeletVersion: \"v1.16.9-beta.0.9+3b7cbcf52403ff\"\nname: \"bootstrap-e2e-minion-group-vgzb\", osImage: \"Container-Optimized OS from Google\", kubeletVersion: \"v1.16.9-beta.0.9+3b7cbcf52403ff\"\nFound subnet for region us-west1 in network bootstrap-e2e: bootstrap-e2e\n== Upgrading master to 'https://storage.googleapis.com/kubernetes-release-dev/ci/v1.17.5-beta.0.13+222605b0aa03bc/kubernetes-server-linux-amd64.tar.gz'. Do not interrupt, deleting master instance. ==\n== Upgrading master environment variables. ==\n== Waiting for new master to respond to API requests ==\n........................== Done ==\nWaiting for CoreDNS to update\nFetching the latest installed CoreDNS version\n== Downloading the CoreDNS migration tool ==\n== Upgrading the CoreDNS ConfigMap ==\nconfigmap/coredns configured\n== The CoreDNS Config has been updated ==\n== Validating cluster post-upgrade ==\nValidating gce cluster, MULTIZONE=\nFound 4 node(s).\nNAME                              STATUS                     ROLES    AGE     VERSION\nbootstrap-e2e-master              Ready,SchedulingDisabled   <none>   9m51s   v1.17.5-beta.0.13+222605b0aa03bc\nbootstrap-e2e-minion-group-8lfn   Ready                      <none>   9m55s   v1.16.9-beta.0.9+3b7cbcf52403ff\nbootstrap-e2e-minion-group-hlrz   Ready                      <none>   9m55s   v1.16.9-beta.0.9+3b7cbcf52403ff\nbootstrap-e2e-minion-group-vgzb   Ready                      <none>   9m54s   v1.16.9-beta.0.9+3b7cbcf52403ff\nValidate output:\nNAME                 STATUS    MESSAGE             ERROR\netcd-0               Healthy   {\"health\":\"true\"}   \nscheduler            Healthy   ok                  \ncontroller-manager   Healthy   ok                  \netcd-1               Healthy   {\"health\":\"true\"}   \n\x1b[0;32mCluster validation succeeded\x1b[0m\n== Post-Upgrade Node OS and Kubelet Versions ==\nname: \"bootstrap-e2e-master\", osImage: \"Container-Optimized OS from Google\", kubeletVersion: \"v1.17.5-beta.0.13+222605b0aa03bc\"\nname: \"bootstrap-e2e-minion-group-8lfn\", osImage: \"Container-Optimized OS from Google\", kubeletVersion: \"v1.16.9-beta.0.9+3b7cbcf52403ff\"\nname: \"bootstrap-e2e-minion-group-hlrz\", osImage: \"Container-Optimized OS from Google\", kubeletVersion: \"v1.16.9-beta.0.9+3b7cbcf52403ff\"\nname: \"bootstrap-e2e-minion-group-vgzb\", osImage: \"Container-Optimized OS from Google\", kubeletVersion: \"v1.16.9-beta.0.9+3b7cbcf52403ff\"\n", stderr "Project: k8s-gce-soak-1-5\nNetwork Project: k8s-gce-soak-1-5\nZone: us-west1-b\nINSTANCE_GROUPS=bootstrap-e2e-minion-group\nNODE_NAMES=bootstrap-e2e-minion-group-8lfn bootstrap-e2e-minion-group-hlrz bootstrap-e2e-minion-group-vgzb\nTrying to find master named 'bootstrap-e2e-master'\nLooking for address 'bootstrap-e2e-master-ip'\nUsing master: bootstrap-e2e-master (external IP: 34.82.199.207; internal IP: (not set))\nDeleted [https://www.googleapis.com/compute/v1/projects/k8s-gce-soak-1-5/zones/us-west1-b/instances/bootstrap-e2e-master].\nWARNING: You have selected a disk size of under [200GB]. This may result in poor I/O performance. For more information, see: https://developers.google.com/compute/docs/disks#performance.\nCreated [https://www.googleapis.com/compute/v1/projects/k8s-gce-soak-1-5/zones/us-west1-b/instances/bootstrap-e2e-master].\nWARNING: Some requests generated warnings:\n - Disk size: '20 GB' is larger than image size: '10 GB'. You might need to resize the root repartition manually if the operating system does not support automatic resizing. See https://cloud.google.com/compute/docs/disks/add-persistent-disk#resize_pd for details.\n - The resource 'projects/cos-cloud/global/images/cos-77-12371-175-0' is deprecated. A suggested replacement is 'projects/cos-cloud/global/images/cos-77-12371-183-0'.\n\nNAME                  ZONE        MACHINE_TYPE   PREEMPTIBLE  INTERNAL_IP  EXTERNAL_IP    STATUS\nbootstrap-e2e-master  us-west1-b  n1-standard-1               10.138.0.6   34.82.199.207  RUNNING\nWarning: kubectl apply should be used on resource created by either kubectl create --save-config or kubectl apply\nProject: k8s-gce-soak-1-5\nNetwork Project: k8s-gce-soak-1-5\nZone: us-west1-b\n/workspace/kubernetes_skew/cluster/gce/upgrade.sh: line 452: download_dir: unbound variable\n"
occurred
/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/cloud/gcp/cluster_upgrade.go:106
				
				Click to see stdout/stderrfrom junit_upgrade01.xml

Filter through log files | View test history on testgrid


Kubernetes e2e suite [k8s.io] [sig-node] NodeProblemDetector [DisabledForLargeClusters] should run without error 2m17s

go run hack/e2e.go -v --test --test_args='--ginkgo.focus=Kubernetes\se2e\ssuite\s\[k8s\.io\]\s\[sig\-node\]\sNodeProblemDetector\s\[DisabledForLargeClusters\]\sshould\srun\swithout\serror$'
/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/node/node_problem_detector.go:58
Mar 25 07:16:46.818: Timed out after 60.004s.
Expected success, but got an error:
    <*errors.errorString | 0xc0051d3b30>: {
        s: "Event KubeletStart does not exist: [{{ } {bootstrap-e2e-master.15ff7939402bce17  default /api/v1/namespaces/default/events/bootstrap-e2e-master.15ff7939402bce17 0493ebef-ff5a-441f-b718-e01f66faff08 10779 0 2020-03-25 06:48:00 +0000 UTC <nil> <nil> map[] map[] [] []  []} {Node  bootstrap-e2e-master 8adfaea6-c107-4dff-85c7-fb25478e3bc0   } RegisteredNode Node bootstrap-e2e-master event: Registered Node bootstrap-e2e-master in Controller {node-controller } 2020-03-25 06:48:00 +0000 UTC 2020-03-25 06:48:00 +0000 UTC 1 Normal 0001-01-01 00:00:00 +0000 UTC nil  nil  } {{ } {bootstrap-e2e-minion-group-5sfb.15ff773d1eac999c  default /api/v1/namespaces/default/events/bootstrap-e2e-minion-group-5sfb.15ff773d1eac999c 03979117-da87-405e-9338-aca122dc20fb 10533 0 2020-03-25 06:11:38 +0000 UTC <nil> <nil> map[] map[] [] []  []} {Node  bootstrap-e2e-minion-group-5sfb bootstrap-e2e-minion-group-5sfb   } NodeHasSufficientMemory Node bootstrap-e2e-minion-group-5sfb status is now: NodeHasSufficientMemory {kubelet bootstrap-e2e-minion-group-5sfb} 2020-03-25 06:11:38 +0000 UTC 2020-03-25 06:35:27 +0000 UTC 9 Normal 0001-01-01 00:00:00 +0000 UTC nil  nil  } {{ } {bootstrap-e2e-minion-group-5sfb.15ff773d1eacce82  default /api/v1/namespaces/default/events/bootstrap-e2e-minion-group-5sfb.15ff773d1eacce82 89f85cc8-bc67-46d9-b8bc-6bb669378dbd 10524 0 2020-03-25 06:11:38 +0000 UTC <nil> <nil> map[] map[] [] []  []} {Node  bootstrap-e2e-minion-group-5sfb bootstrap-e2e-minion-group-5sfb   } NodeHasNoDiskPressure Node bootstrap-e2e-minion-group-5sfb status is now: NodeHasNoDiskPressure {kubelet bootstrap-e2e-minion-group-5sfb} 2020-03-25 06:11:38 +0000 UTC 2020-03-25 06:35:27 +0000 UTC 8 Normal 0001-01-01 00:00:00 +0000 UTC nil  nil  } {{ } {bootstrap-e2e-minion-group-5sfb.15ff773d1eacfe77  default /api/v1/namespaces/default/events/bootstrap-e2e-minion-group-5sfb.15ff773d1eacfe77 ff429526-adb5-4a21-9a0a-48c5264ed3f3 10534 0 2020-03-25 06:11:38 +0000 UTC <nil> <nil> map[] map[] [] []  []} {Node  bootstrap-e2e-minion-group-5sfb bootstrap-e2e-minion-group-5sfb   } NodeHasSufficientPID Node bootstrap-e2e-minion-group-5sfb status is now: NodeHasSufficientPID {kubelet bootstrap-e2e-minion-group-5sfb} 2020-03-25 06:11:38 +0000 UTC 2020-03-25 06:36:51 +0000 UTC 10 Normal 0001-01-01 00:00:00 +0000 UTC nil  nil  } {{ } {bootstrap-e2e-minion-group-5sfb.15ff787d7bd3709d  default /api/v1/namespaces/default/events/bootstrap-e2e-minion-group-5sfb.15ff787d7bd3709d 0c64ebfc-c658-448f-b64b-630f0e82aea7 10499 0 2020-03-25 06:34:34 +0000 UTC <nil> <nil> map[] map[] [] []  []} {Node  bootstrap-e2e-minion-group-5sfb df38cbc7-a6c2-4b6e-9f35-b18c75bcc67c   } RemovingNode Node bootstrap-e2e-minion-group-5sfb event: Removing Node bootstrap-e2e-minion-group-5sfb from Controller {node-controller } 2020-03-25 06:34:34 +0000 UTC 2020-03-25 06:34:34 +0000 UTC 1 Normal 0001-01-01 00:00:00 +0000 UTC nil  nil  } {{ } {bootstrap-e2e-minion-group-5sfb.15ff78d3a52bcf31  default /api/v1/namespaces/default/events/bootstrap-e2e-minion-group-5sfb.15ff78d3a52bcf31 c5a08857-7082-4529-9212-cd02f50b2e62 10540 0 2020-03-25 06:40:44 +0000 UTC <nil> <nil> map[] map[] [] []  []} {Node  bootstrap-e2e-minion-group-5sfb e566d472-2c71-4018-97d7-84d7539ec88d   } RegisteredNode Node bootstrap-e2e-minion-group-5sfb event: Registered Node bootstrap-e2e-minion-group-5sfb in Controller {node-controller } 2020-03-25 06:40:44 +0000 UTC 2020-03-25 06:40:44 +0000 UTC 1 Normal 0001-01-01 00:00:00 +0000 UTC nil  nil  } {{ } {bootstrap-e2e-minion-group-5sfb.15ff7939402a58ad  default /api/v1/namespaces/default/events/bootstrap-e2e-minion-group-5sfb.15ff7939402a58ad bf73b372-9ad7-498e-be89-50bb8e214b99 10780 0 2020-03-25 06:48:00 +0000 UTC <nil> <nil> map[] map[] [] []  []} {Node  bootstrap-e2e-minion-group-5sfb e566d472-2c71-4018-97d7-84d7539ec88d   } RegisteredNode Node bootstrap-e2e-minion-group-5sfb event: Registered Node bootstrap-e2e-minion-group-5sfb in Controller {node-controller } 2020-03-25 06:48:00 +0000 UTC 2020-03-25 06:48:00 +0000 UTC 1 Normal 0001-01-01 00:00:00 +0000 UTC nil  nil  } {{ } {bootstrap-e2e-minion-group-5sfb.15ff7abbfb7e7de6  default /api/v1/namespaces/default/events/bootstrap-e2e-minion-group-5sfb.15ff7abbfb7e7de6 83baa4d1-328d-4c95-93dc-e52a2d32eba4 11827 0 2020-03-25 07:15:41 +0000 UTC <nil> <nil> map[] map[] [] []  []} {Node  bootstrap-e2e-minion-group-5sfb bootstrap-e2e-minion-group-5sfb   } TaskHung kernel: INFO: task umount.aufs:21568 blocked for more than 120 seconds. {kernel-monitor bootstrap-e2e-minion-group-5sfb} 2020-03-25 07:15:41 +0000 UTC 2020-03-25 07:15:41 +0000 UTC 1 Warning 0001-01-01 00:00:00 +0000 UTC nil  nil  } {{ } {bootstrap-e2e-minion-group-5sfb.15ff7abbfb7ea091  default /api/v1/namespaces/default/events/bootstrap-e2e-minion-group-5sfb.15ff7abbfb7ea091 2a42af40-45ae-4071-9440-536238a8c72a 11826 0 2020-03-25 07:15:41 +0000 UTC <nil> <nil> map[] map[] [] []  []} {Node  bootstrap-e2e-minion-group-5sfb bootstrap-e2e-minion-group-5sfb   } AUFSUmountHung Node condition KernelDeadlock is now: True, reason: AUFSUmountHung {kernel-monitor bootstrap-e2e-minion-group-5sfb} 2020-03-25 07:15:41 +0000 UTC 2020-03-25 07:15:41 +0000 UTC 1 Normal 0001-01-01 00:00:00 +0000 UTC nil  nil  } {{ } {bootstrap-e2e-minion-group-hlrz.15ff68a8639e9966  default /api/v1/namespaces/default/events/bootstrap-e2e-minion-group-hlrz.15ff68a8639e9966 5802868d-bf90-4e58-99ef-5e7734909f03 11786 0 2020-03-25 07:12:53 +0000 UTC <nil> <nil> map[] map[] [] []  []} {Node  bootstrap-e2e-minion-group-hlrz bootstrap-e2e-minion-group-hlrz   } KubeletStart Started Kubernetes kubelet. {systemd-monitor bootstrap-e2e-minion-group-hlrz} 2020-03-25 01:44:26 +0000 UTC 2020-03-25 07:12:53 +0000 UTC 6 Warning 0001-01-01 00:00:00 +0000 UTC nil  nil  } {{ } {bootstrap-e2e-minion-group-hlrz.15ff7939402b13a4  default /api/v1/namespaces/default/events/bootstrap-e2e-minion-group-hlrz.15ff7939402b13a4 8f9b21ec-2ecf-4a3b-b326-a2891439bdaf 10781 0 2020-03-25 06:48:00 +0000 UTC <nil> <nil> map[] map[] [] []  []} {Node  bootstrap-e2e-minion-group-hlrz b83a42dc-5e58-4566-bfad-90409a5f5c7f   } RegisteredNode Node bootstrap-e2e-minion-group-hlrz event: Registered Node bootstrap-e2e-minion-group-hlrz in Controller {node-controller } 2020-03-25 06:48:00 +0000 UTC 2020-03-25 06:48:00 +0000 UTC 1 Normal 0001-01-01 00:00:00 +0000 UTC nil  nil  } {{ } {bootstrap-e2e-minion-group-hlrz.15ff7a9439f26910  default /api/v1/namespaces/default/events/bootstrap-e2e-minion-group-hlrz.15ff7a9439f26910 36aa17d6-a6c1-41d0-9a24-fd0897e96e4a 11785 0 2020-03-25 07:12:50 +0000 UTC <nil> <nil> map[] map[] [] []  []} {Node  bootstrap-e2e-minion-group-hlrz b83a42dc-5e58-4566-bfad-90409a5f5c7f   } NodeNotReady Node bootstrap-e2e-minion-group-hlrz status is now: NodeNotReady {node-controller } 2020-03-25 07:12:50 +0000 UTC 2020-03-25 07:12:50 +0000 UTC 1 Normal 0001-01-01 00:00:00 +0000 UTC nil  nil  } {{ } {bootstrap-e2e-minion-group-hlrz.15ff7a94ce1b0226  default /api/v1/namespaces/default/events/bootstrap-e2e-minion-group-hlrz.15ff7a94ce1b0226 8ee596c7-7735-428a-9843-6d0d706234ab 11787 0 2020-03-25 07:12:53 +0000 UTC <nil> <nil> map[] map[] [] []  []} {Node  bootstrap-e2e-minion-group-hlrz bootstrap-e2e-minion-group-hlrz   } Starting Starting kubelet. {kubelet bootstrap-e2e-minion-group-hlrz} 2020-03-25 07:12:53 +0000 UTC 2020-03-25 07:12:53 +0000 UTC 1 Normal 0001-01-01 00:00:00 +0000 UTC nil  nil  } {{ } {bootstrap-e2e-minion-group-hlrz.15ff7a94d75e4cda  default /api/v1/namespaces/default/events/bootstrap-e2e-minion-group-hlrz.15ff7a94d75e4cda d85cb1d3-cc13-4207-8b9a-24aa9ce19254 11791 0 2020-03-25 07:12:53 +0000 UTC <nil> <nil> map[] map[] [] []  []} {Node  bootstrap-e2e-minion-group-hlrz bootstrap-e2e-minion-group-hlrz   } NodeHasSufficientMemory Node bootstrap-e2e-minion-group-hlrz status is now: NodeHasSufficientMemory {kubelet bootstrap-e2e-minion-group-hlrz} 2020-03-25 07:12:53 +0000 UTC 2020-03-25 07:12:53 +0000 UTC 2 Normal 0001-01-01 00:00:00 +0000 UTC nil  nil  } {{ } {bootstrap-e2e-minion-group-hlrz.15ff7a94d75e8e59  default /api/v1/namespaces/default/events/bootstrap-e2e-minion-group-hlrz.15ff7a94d75e8e59 06c1411f-5a6c-4b9e-a27f-c09dcb595033 11792 0 2020-03-25 07:12:53 +0000 UTC <nil> <nil> map[] map[] [] []  []} {Node  bootstrap-e2e-minion-group-hlrz bootstrap-e2e-minion-group-hlrz   } NodeHasNoDiskPressure Node bootstrap-e2e-minion-group-hlrz status is now: NodeHasNoDiskPressure {kubelet bootstrap-e2e-minion-group-hlrz} 2020-03-25 07:12:53 +0000 UTC 2020-03-25 07:12:53 +0000 UTC 2 Normal 0001-01-01 00:00:00 +0000 UTC nil  nil  } {{ } {bootstrap-e2e-minion-group-hlrz.15ff7a94d75ebd55  default /api/v1/namespaces/default/events/bootstrap-e2e-minion-group-hlrz.15ff7a94d75ebd55 72911487-cb27-4075-bd12-61ac17f31faa 11793 0 2020-03-25 07:12:53 +0000 UTC <nil> <nil> map[] map[] [] []  []} {Node  bootstrap-e2e-minion-group-hlrz bootstrap-e2e-minion-group-hlrz   } NodeHasSufficientPID Node bootstrap-e2e-minion-group-hlrz status is now: NodeHasSufficientPID {kubelet bootstrap-e2e-minion-group-hlrz} 2020-03-25 07:12:53 +0000 UTC 2020-03-25 07:12:53 +0000 UTC 2 Normal 0001-01-01 00:00:00 +0000 UTC nil  nil  } {{ } {bootstrap-e2e-minion-group-hlrz.15ff7a94d9598ed7  default /api/v1/namespaces/default/events/bootstrap-e2e-minion-group-hlrz.15ff7a94d9598ed7 0d769589-5902-44f2-bc86-9a1f7a1d1918 11794 0 2020-03-25 07:12:53 +0000 UTC <nil> <nil> map[] map[] [] []  []} {Node  bootstrap-e2e-minion-group-hlrz bootstrap-e2e-minion-group-hlrz   } NodeNotReady Node bootstrap-e2e-minion-group-hlrz status is now: NodeNotReady {kubelet bootstrap-e2e-minion-group-hlrz} 2020-03-25 07:12:53 +0000 UTC 2020-03-25 07:12:53 +0000 UTC 1 Normal 0001-01-01 00:00:00 +0000 UTC nil  nil  } {{ } {bootstrap-e2e-minion-group-hlrz.15ff7a94f15ecb83  default /api/v1/namespaces/default/events/bootstrap-e2e-minion-group-hlrz.15ff7a94f15ecb83 0740e5dc-9510-48ce-aaf2-0bce9e8ed74d 11795 0 2020-03-25 07:12:53 +0000 UTC <nil> <nil> map[] map[] [] []  []} {Node  bootstrap-e2e-minion-group-hlrz bootstrap-e2e-minion-group-hlrz   } NodeAllocatableEnforced Updated Node Allocatable limit across pods {kubelet bootstrap-e2e-minion-group-hlrz} 2020-03-25 07:12:53 +0000 UTC 2020-03-25 07:12:53 +0000 UTC 1 Normal 0001-01-01 00:00:00 +0000 UTC nil  nil  } {{ } {bootstrap-e2e-minion-group-hlrz.15ff7a94f4b8a12c  default /api/v1/namespaces/default/events/bootstrap-e2e-minion-group-hlrz.15ff7a94f4b8a12c 9afd4369-629f-4d31-832f-b98242cd1dd3 11796 0 2020-03-25 07:12:54 +0000 UTC <nil> <nil> map[] map[] [] []  []} {Node  bootstrap-e2e-minion-group-hlrz bootstrap-e2e-minion-group-hlrz   } NodeReady Node bootstrap-e2e-minion-group-hlrz status is now: NodeReady {kubelet bootstrap-e2e-minion-group-hlrz} 2020-03-25 07:12:54 +0000 UTC 2020-03-25 07:12:54 +0000 UTC 1 Normal 0001-01-01 00:00:00 +0000 UTC nil  nil  } {{ } {bootstrap-e2e-minion-group-hlrz.15ff7abc94a298c7  default /api/v1/namespaces/default/events/bootstrap-e2e-minion-group-hlrz.15ff7abc94a298c7 cd604241-122b-43cf-b616-a41208ec8d3e 11829 0 2020-03-25 07:15:44 +0000 UTC <nil> <nil> map[] map[] [] []  []} {Node  bootstrap-e2e-minion-group-hlrz bootstrap-e2e-minion-group-hlrz   } TaskHung kernel: INFO: task umount.aufs:21568 blocked for more than 120 seconds. {kernel-monitor bootstrap-e2e-minion-group-hlrz} 2020-03-25 07:15:44 +0000 UTC 2020-03-25 07:15:44 +0000 UTC 1 Warning 0001-01-01 00:00:00 +0000 UTC nil  nil  } {{ } {bootstrap-e2e-minion-group-hlrz.15ff7abc94a2c693  default /api/v1/namespaces/default/events/bootstrap-e2e-minion-group-hlrz.15ff7abc94a2c693 1732c564-7ab4-4c8d-80f7-1744ad59cd79 11828 0 2020-03-25 07:15:44 +0000 UTC <nil> <nil> map[] map[] [] []  []} {Node  bootstrap-e2e-minion-group-hlrz bootstrap-e2e-minion-group-hlrz   } AUFSUmountHung Node condition KernelDeadlock is now: True, reason: AUFSUmountHung {kernel-monitor bootstrap-e2e-minion-group-hlrz} 2020-03-25 07:15:44 +0000 UTC 2020-03-25 07:15:44 +0000 UTC 1 Normal 0001-01-01 00:00:00 +0000 UTC nil  nil  } {{ } {bootstrap-e2e-minion-group-vgzb.15ff7939402b9734  default /api/v1/namespaces/default/events/bootstrap-e2e-minion-group-vgzb.15ff7939402b9734 bb59e095-5afb-4696-84ab-7bc672d8ec68 10782 0 2020-03-25 06:48:00 +0000 UTC <nil> <nil> map[] map[] [] []  []} {Node  bootstrap-e2e-minion-group-vgzb c03e18c6-52fa-463b-8f7d-b26c5bb8acfb   } RegisteredNode Node bootstrap-e2e-minion-group-vgzb event: Registered Node bootstrap-e2e-minion-group-vgzb in Controller {node-controller } 2020-03-25 06:48:00 +0000 UTC 2020-03-25 06:48:00 +0000 UTC 1 Normal 0001-01-01 00:00:00 +0000 UTC nil  nil  } {{ } {bootstrap-e2e-minion-group-vgzb.15ff7abd28bea503  default /api/v1/namespaces/default/events/bootstrap-e2e-minion-group-vgzb.15ff7abd28bea503 882e6db8-8e2e-4543-ad3b-1f6b6b23ba79 11831 0 2020-03-25 07:15:46 +0000 UTC <nil> <nil> map[] map[] [] []  []} {Node  bootstrap-e2e-minion-group-vgzb bootstrap-e2e-minion-group-vgzb   } TaskHung kernel: INFO: task umount.aufs:21568 blocked for more than 120 seconds. {kernel-monitor bootstrap-e2e-minion-group-vgzb} 2020-03-25 07:15:46 +0000 UTC 2020-03-25 07:15:46 +0000 UTC 1 Warning 0001-01-01 00:00:00 +0000 UTC nil  nil  } {{ } {bootstrap-e2e-minion-group-vgzb.15ff7abd28bebc5c  default /api/v1/namespaces/default/events/bootstrap-e2e-minion-group-vgzb.15ff7abd28bebc5c 5733e955-2407-4109-9294-ad41376c96eb 11830 0 2020-03-25 07:15:46 +0000 UTC <nil> <nil> map[] map[] [] []  []} {Node  bootstrap-e2e-minion-group-vgzb bootstrap-e2e-minion-group-vgzb   } AUFSUmountHung Node condition KernelDeadlock is now: True, reason: AUFSUmountHung {kernel-monitor bootstrap-e2e-minion-group-vgzb} 2020-03-25 07:15:46 +0000 UTC 2020-03-25 07:15:46 +0000 UTC 1 Normal 0001-01-01 00:00:00 +0000 UTC nil  nil  }]",
    }
    Event KubeletStart does not exist: [{{ } {bootstrap-e2e-master.15ff7939402bce17  default /api/v1/namespaces/default/events/bootstrap-e2e-master.15ff7939402bce17 0493ebef-ff5a-441f-b718-e01f66faff08 10779 0 2020-03-25 06:48:00 +0000 UTC <nil> <nil> map[] map[] [] []  []} {Node  bootstrap-e2e-master 8adfaea6-c107-4dff-85c7-fb25478e3bc0   } RegisteredNode Node bootstrap-e2e-master event: Registered Node bootstrap-e2e-master in Controller {node-controller } 2020-03-25 06:48:00 +0000 UTC 2020-03-25 06:48:00 +0000 UTC 1 Normal 0001-01-01 00:00:00 +0000 UTC nil  nil  } {{ } {bootstrap-e2e-minion-group-5sfb.15ff773d1eac999c  default /api/v1/namespaces/default/events/bootstrap-e2e-minion-group-5sfb.15ff773d1eac999c 03979117-da87-405e-9338-aca122dc20fb 10533 0 2020-03-25 06:11:38 +0000 UTC <nil> <nil> map[] map[] [] []  []} {Node  bootstrap-e2e-minion-group-5sfb bootstrap-e2e-minion-group-5sfb   } NodeHasSufficientMemory Node bootstrap-e2e-minion-group-5sfb status is now: NodeHasSufficientMemory {kubelet bootstrap-e2e-minion-group-5sfb} 2020-03-25 06:11:38 +0000 UTC 2020-03-25 06:35:27 +0000 UTC 9 Normal 0001-01-01 00:00:00 +0000 UTC nil  nil  } {{ } {bootstrap-e2e-minion-group-5sfb.15ff773d1eacce82  default /api/v1/namespaces/default/events/bootstrap-e2e-minion-group-5sfb.15ff773d1eacce82 89f85cc8-bc67-46d9-b8bc-6bb669378dbd 10524 0 2020-03-25 06:11:38 +0000 UTC <nil> <nil> map[] map[] [] []  []} {Node  bootstrap-e2e-minion-group-5sfb bootstrap-e2e-minion-group-5sfb   } NodeHasNoDiskPressure Node bootstrap-e2e-minion-group-5sfb status is now: NodeHasNoDiskPressure {kubelet bootstrap-e2e-minion-group-5sfb} 2020-03-25 06:11:38 +0000 UTC 2020-03-25 06:35:27 +0000 UTC 8 Normal 0001-01-01 00:00:00 +0000 UTC nil  nil  } {{ } {bootstrap-e2e-minion-group-5sfb.15ff773d1eacfe77  default /api/v1/namespaces/default/events/bootstrap-e2e-minion-group-5sfb.15ff773d1eacfe77 ff429526-adb5-4a21-9a0a-48c5264ed3f3 10534 0 2020-03-25 06:11:38 +0000 UTC <nil> <nil> map[] map[] [] []  []} {Node  bootstrap-e2e-minion-group-5sfb bootstrap-e2e-minion-group-5sfb   } NodeHasSufficientPID Node bootstrap-e2e-minion-group-5sfb status is now: NodeHasSufficientPID {kubelet bootstrap-e2e-minion-group-5sfb} 2020-03-25 06:11:38 +0000 UTC 2020-03-25 06:36:51 +0000 UTC 10 Normal 0001-01-01 00:00:00 +0000 UTC nil  nil  } {{ } {bootstrap-e2e-minion-group-5sfb.15ff787d7bd3709d  default /api/v1/namespaces/default/events/bootstrap-e2e-minion-group-5sfb.15ff787d7bd3709d 0c64ebfc-c658-448f-b64b-630f0e82aea7 10499 0 2020-03-25 06:34:34 +0000 UTC <nil> <nil> map[] map[] [] []  []} {Node  bootstrap-e2e-minion-group-5sfb df38cbc7-a6c2-4b6e-9f35-b18c75bcc67c   } RemovingNode Node bootstrap-e2e-minion-group-5sfb event: Removing Node bootstrap-e2e-minion-group-5sfb from Controller {node-controller } 2020-03-25 06:34:34 +0000 UTC 2020-03-25 06:34:34 +0000 UTC 1 Normal 0001-01-01 00:00:00 +0000 UTC nil  nil  } {{ } {bootstrap-e2e-minion-group-5sfb.15ff78d3a52bcf31  default /api/v1/namespaces/default/events/bootstrap-e2e-minion-group-5sfb.15ff78d3a52bcf31 c5a08857-7082-4529-9212-cd02f50b2e62 10540 0 2020-03-25 06:40:44 +0000 UTC <nil> <nil> map[] map[] [] []  []} {Node  bootstrap-e2e-minion-group-5sfb e566d472-2c71-4018-97d7-84d7539ec88d   } RegisteredNode Node bootstrap-e2e-minion-group-5sfb event: Registered Node bootstrap-e2e-minion-group-5sfb in Controller {node-controller } 2020-03-25 06:40:44 +0000 UTC 2020-03-25 06:40:44 +0000 UTC 1 Normal 0001-01-01 00:00:00 +0000 UTC nil  nil  } {{ } {bootstrap-e2e-minion-group-5sfb.15ff7939402a58ad  default /api/v1/namespaces/default/events/bootstrap-e2e-minion-group-5sfb.15ff7939402a58ad bf73b372-9ad7-498e-be89-50bb8e214b99 10780 0 2020-03-25 06:48:00 +0000 UTC <nil> <nil> map[] map[] [] []  []} {Node  bootstrap-e2e-minion-group-5sfb e566d472-2c71-4018-97d7-84d7539ec88d   } RegisteredNode Node bootstrap-e2e-minion-group-5sfb event: Registered Node bootstrap-e2e-minion-group-5sfb in Controller {node-controller } 2020-03-25 06:48:00 +0000 UTC 2020-03-25 06:48:00 +0000 UTC 1 Normal 0001-01-01 00:00:00 +0000 UTC nil  nil  } {{ } {bootstrap-e2e-minion-group-5sfb.15ff7abbfb7e7de6  default /api/v1/namespaces/default/events/bootstrap-e2e-minion-group-5sfb.15ff7abbfb7e7de6 83baa4d1-328d-4c95-93dc-e52a2d32eba4 11827 0 2020-03-25 07:15:41 +0000 UTC <nil> <nil> map[] map[] [] []  []} {Node  bootstrap-e2e-minion-group-5sfb bootstrap-e2e-minion-group-5sfb   } TaskHung kernel: INFO: task umount.aufs:21568 blocked for more than 120 seconds. {kernel-monitor bootstrap-e2e-minion-group-5sfb} 2020-03-25 07:15:41 +0000 UTC 2020-03-25 07:15:41 +0000 UTC 1 Warning 0001-01-01 00:00:00 +0000 UTC nil  nil  } {{ } {bootstrap-e2e-minion-group-5sfb.15ff7abbfb7ea091  default /api/v1/namespaces/default/events/bootstrap-e2e-minion-group-5sfb.15ff7abbfb7ea091 2a42af40-45ae-4071-9440-536238a8c72a 11826 0 2020-03-25 07:15:41 +0000 UTC <nil> <nil> map[] map[] [] []  []} {Node  bootstrap-e2e-minion-group-5sfb bootstrap-e2e-minion-group-5sfb   } AUFSUmountHung Node condition KernelDeadlock is now: True, reason: AUFSUmountHung {kernel-monitor bootstrap-e2e-minion-group-5sfb} 2020-03-25 07:15:41 +0000 UTC 2020-03-25 07:15:41 +0000 UTC 1 Normal 0001-01-01 00:00:00 +0000 UTC nil  nil  } {{ } {bootstrap-e2e-minion-group-hlrz.15ff68a8639e9966  default /api/v1/namespaces/default/events/bootstrap-e2e-minion-group-hlrz.15ff68a8639e9966 5802868d-bf90-4e58-99ef-5e7734909f03 11786 0 2020-03-25 07:12:53 +0000 UTC <nil> <nil> map[] map[] [] []  []} {Node  bootstrap-e2e-minion-group-hlrz bootstrap-e2e-minion-group-hlrz   } KubeletStart Started Kubernetes kubelet. {systemd-monitor bootstrap-e2e-minion-group-hlrz} 2020-03-25 01:44:26 +0000 UTC 2020-03-25 07:12:53 +0000 UTC 6 Warning 0001-01-01 00:00:00 +0000 UTC nil  nil  } {{ } {bootstrap-e2e-minion-group-hlrz.15ff7939402b13a4  default /api/v1/namespaces/default/events/bootstrap-e2e-minion-group-hlrz.15ff7939402b13a4 8f9b21ec-2ecf-4a3b-b326-a2891439bdaf 10781 0 2020-03-25 06:48:00 +0000 UTC <nil> <nil> map[] map[] [] []  []} {Node  bootstrap-e2e-minion-group-hlrz b83a42dc-5e58-4566-bfad-90409a5f5c7f   } RegisteredNode Node bootstrap-e2e-minion-group-hlrz event: Registered Node bootstrap-e2e-minion-group-hlrz in Controller {node-controller } 2020-03-25 06:48:00 +0000 UTC 2020-03-25 06:48:00 +0000 UTC 1 Normal 0001-01-01 00:00:00 +0000 UTC nil  nil  } {{ } {bootstrap-e2e-minion-group-hlrz.15ff7a9439f26910  default /api/v1/namespaces/default/events/bootstrap-e2e-minion-group-hlrz.15ff7a9439f26910 36aa17d6-a6c1-41d0-9a24-fd0897e96e4a 11785 0 2020-03-25 07:12:50 +0000 UTC <nil> <nil> map[] map[] [] []  []} {Node  bootstrap-e2e-minion-group-hlrz b83a42dc-5e58-4566-bfad-90409a5f5c7f   } NodeNotReady Node bootstrap-e2e-minion-group-hlrz status is now: NodeNotReady {node-controller } 2020-03-25 07:12:50 +0000 UTC 2020-03-25 07:12:50 +0000 UTC 1 Normal 0001-01-01 00:00:00 +0000 UTC nil  nil  } {{ } {bootstrap-e2e-minion-group-hlrz.15ff7a94ce1b0226  default /api/v1/namespaces/default/events/bootstrap-e2e-minion-group-hlrz.15ff7a94ce1b0226 8ee596c7-7735-428a-9843-6d0d706234ab 11787 0 2020-03-25 07:12:53 +0000 UTC <nil> <nil> map[] map[] [] []  []} {Node  bootstrap-e2e-minion-group-hlrz bootstrap-e2e-minion-group-hlrz   } Starting Starting kubelet. {kubelet bootstrap-e2e-minion-group-hlrz} 2020-03-25 07:12:53 +0000 UTC 2020-03-25 07:12:53 +0000 UTC 1 Normal 0001-01-01 00:00:00 +0000 UTC nil  nil  } {{ } {bootstrap-e2e-minion-group-hlrz.15ff7a94d75e4cda  default /api/v1/namespaces/default/events/bootstrap-e2e-minion-group-hlrz.15ff7a94d75e4cda d85cb1d3-cc13-4207-8b9a-24aa9ce19254 11791 0 2020-03-25 07:12:53 +0000 UTC <nil> <nil> map[] map[] [] []  []} {Node  bootstrap-e2e-minion-group-hlrz bootstrap-e2e-minion-group-hlrz   } NodeHasSufficientMemory Node bootstrap-e2e-minion-group-hlrz status is now: NodeHasSufficientMemory {kubelet bootstrap-e2e-minion-group-hlrz} 2020-03-25 07:12:53 +0000 UTC 2020-03-25 07:12:53 +0000 UTC 2 Normal 0001-01-01 00:00:00 +0000 UTC nil  nil  } {{ } {bootstrap-e2e-minion-group-hlrz.15ff7a94d75e8e59  default /api/v1/namespaces/default/events/bootstrap-e2e-minion-group-hlrz.15ff7a94d75e8e59 06c1411f-5a6c-4b9e-a27f-c09dcb595033 11792 0 2020-03-25 07:12:53 +0000 UTC <nil> <nil> map[] map[] [] []  []} {Node  bootstrap-e2e-minion-group-hlrz bootstrap-e2e-minion-group-hlrz   } NodeHasNoDiskPressure Node bootstrap-e2e-minion-group-hlrz status is now: NodeHasNoDiskPressure {kubelet bootstrap-e2e-minion-group-hlrz} 2020-03-25 07:12:53 +0000 UTC 2020-03-25 07:12:53 +0000 UTC 2 Normal 0001-01-01 00:00:00 +0000 UTC nil  nil  } {{ } {bootstrap-e2e-minion-group-hlrz.15ff7a94d75ebd55  default /api/v1/namespaces/default/events/bootstrap-e2e-minion-group-hlrz.15ff7a94d75ebd55 72911487-cb27-4075-bd12-61ac17f31faa 11793 0 2020-03-25 07:12:53 +0000 UTC <nil> <nil> map[] map[] [] []  []} {Node  bootstrap-e2e-minion-group-hlrz bootstrap-e2e-minion-group-hlrz   } NodeHasSufficientPID Node bootstrap-e2e-minion-group-hlrz status is now: NodeHasSufficientPID {kubelet bootstrap-e2e-minion-group-hlrz} 2020-03-25 07:12:53 +0000 UTC 2020-03-25 07:12:53 +0000 UTC 2 Normal 0001-01-01 00:00:00 +0000 UTC nil  nil  } {{ } {bootstrap-e2e-minion-group-hlrz.15ff7a94d9598ed7  default /api/v1/namespaces/default/events/bootstrap-e2e-minion-group-hlrz.15ff7a94d9598ed7 0d769589-5902-44f2-bc86-9a1f7a1d1918 11794 0 2020-03-25 07:12:53 +0000 UTC <nil> <nil> map[] map[] [] []  []} {Node  bootstrap-e2e-minion-group-hlrz bootstrap-e2e-minion-group-hlrz   } NodeNotReady Node bootstrap-e2e-minion-group-hlrz status is now: NodeNotReady {kubelet bootstrap-e2e-minion-group-hlrz} 2020-03-25 07:12:53 +0000 UTC 2020-03-25 07:12:53 +0000 UTC 1 Normal 0001-01-01 00:00:00 +0000 UTC nil  nil  } {{ } {bootstrap-e2e-minion-group-hlrz.15ff7a94f15ecb83  default /api/v1/namespaces/default/events/bootstrap-e2e-minion-group-hlrz.15ff7a94f15ecb83 0740e5dc-9510-48ce-aaf2-0bce9e8ed74d 11795 0 2020-03-25 07:12:53 +0000 UTC <nil> <nil> map[] map[] [] []  []} {Node  bootstrap-e2e-minion-group-hlrz bootstrap-e2e-minion-group-hlrz   } NodeAllocatableEnforced Updated Node Allocatable limit across pods {kubelet bootstrap-e2e-minion-group-hlrz} 2020-03-25 07:12:53 +0000 UTC 2020-03-25 07:12:53 +0000 UTC 1 Normal 0001-01-01 00:00:00 +0000 UTC nil  nil  } {{ } {bootstrap-e2e-minion-group-hlrz.15ff7a94f4b8a12c  default /api/v1/namespaces/default/events/bootstrap-e2e-minion-group-hlrz.15ff7a94f4b8a12c 9afd4369-629f-4d31-832f-b98242cd1dd3 11796 0 2020-03-25 07:12:54 +0000 UTC <nil> <nil> map[] map[] [] []  []} {Node  bootstrap-e2e-minion-group-hlrz bootstrap-e2e-minion-group-hlrz   } NodeReady Node bootstrap-e2e-minion-group-hlrz status is now: NodeReady {kubelet bootstrap-e2e-minion-group-hlrz} 2020-03-25 07:12:54 +0000 UTC 2020-03-25 07:12:54 +0000 UTC 1 Normal 0001-01-01 00:00:00 +0000 UTC nil  nil  } {{ } {bootstrap-e2e-minion-group-hlrz.15ff7abc94a298c7  default /api/v1/namespaces/default/events/bootstrap-e2e-minion-group-hlrz.15ff7abc94a298c7 cd604241-122b-43cf-b616-a41208ec8d3e 11829 0 2020-03-25 07:15:44 +0000 UTC <nil> <nil> map[] map[] [] []  []} {Node  bootstrap-e2e-minion-group-hlrz bootstrap-e2e-minion-group-hlrz   } TaskHung kernel: INFO: task umount.aufs:21568 blocked for more than 120 seconds. {kernel-monitor bootstrap-e2e-minion-group-hlrz} 2020-03-25 07:15:44 +0000 UTC 2020-03-25 07:15:44 +0000 UTC 1 Warning 0001-01-01 00:00:00 +0000 UTC nil  nil  } {{ } {bootstrap-e2e-minion-group-hlrz.15ff7abc94a2c693  default /api/v1/namespaces/default/events/bootstrap-e2e-minion-group-hlrz.15ff7abc94a2c693 1732c564-7ab4-4c8d-80f7-1744ad59cd79 11828 0 2020-03-25 07:15:44 +0000 UTC <nil> <nil> map[] map[] [] []  []} {Node  bootstrap-e2e-minion-group-hlrz bootstrap-e2e-minion-group-hlrz   } AUFSUmountHung Node condition KernelDeadlock is now: True, reason: AUFSUmountHung {kernel-monitor bootstrap-e2e-minion-group-hlrz} 2020-03-25 07:15:44 +0000 UTC 2020-03-25 07:15:44 +0000 UTC 1 Normal 0001-01-01 00:00:00 +0000 UTC nil  nil  } {{ } {bootstrap-e2e-minion-group-vgzb.15ff7939402b9734  default /api/v1/namespaces/default/events/bootstrap-e2e-minion-group-vgzb.15ff7939402b9734 bb59e095-5afb-4696-84ab-7bc672d8ec68 10782 0 2020-03-25 06:48:00 +0000 UTC <nil> <nil> map[] map[] [] []  []} {Node  bootstrap-e2e-minion-group-vgzb c03e18c6-52fa-463b-8f7d-b26c5bb8acfb   } RegisteredNode Node bootstrap-e2e-minion-group-vgzb event: Registered Node bootstrap-e2e-minion-group-vgzb in Controller {node-controller } 2020-03-25 06:48:00 +0000 UTC 2020-03-25 06:48:00 +0000 UTC 1 Normal 0001-01-01 00:00:00 +0000 UTC nil  nil  } {{ } {bootstrap-e2e-minion-group-vgzb.15ff7abd28bea503  default /api/v1/namespaces/default/events/bootstrap-e2e-minion-group-vgzb.15ff7abd28bea503 882e6db8-8e2e-4543-ad3b-1f6b6b23ba79 11831 0 2020-03-25 07:15:46 +0000 UTC <nil> <nil> map[] map[] [] []  []} {Node  bootstrap-e2e-minion-group-vgzb bootstrap-e2e-minion-group-vgzb   } TaskHung kernel: INFO: task umount.aufs:21568 blocked for more than 120 seconds. {kernel-monitor bootstrap-e2e-minion-group-vgzb} 2020-03-25 07:15:46 +0000 UTC 2020-03-25 07:15:46 +0000 UTC 1 Warning 0001-01-01 00:00:00 +0000 UTC nil  nil  } {{ } {bootstrap-e2e-minion-group-vgzb.15ff7abd28bebc5c  default /api/v1/namespaces/default/events/bootstrap-e2e-minion-group-vgzb.15ff7abd28bebc5c 5733e955-2407-4109-9294-ad41376c96eb 11830 0 2020-03-25 07:15:46 +0000 UTC <nil> <nil> map[] map[] [] []  []} {Node  bootstrap-e2e-minion-group-vgzb bootstrap-e2e-minion-group-vgzb   } AUFSUmountHung Node condition KernelDeadlock is now: True, reason: AUFSUmountHung {kernel-monitor bootstrap-e2e-minion-group-vgzb} 2020-03-25 07:15:46 +0000 UTC 2020-03-25 07:15:46 +0000 UTC 1 Normal 0001-01-01 00:00:00 +0000 UTC nil  nil  }]
/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/node/node_problem_detector.go:141
				
				Click to see stdout/stderrfrom junit_01.xml

Filter through log files | View test history on testgrid


Master upgrade [sig-cloud-provider-gcp] master-upgrade 5m20s

go run hack/e2e.go -v --test --test_args='--ginkgo.focus=Master\supgrade\s\[sig\-cloud\-provider\-gcp\]\smaster\-upgrade$'
Mar 25 01:54:21.715: Unexpected error:
    <*errors.errorString | 0xc0021c66e0>: {
        s: "error running /workspace/kubernetes_skew/cluster/gce/upgrade.sh [-M v1.17.5-beta.0.13+222605b0aa03bc]; got error exit status 1, stdout \"Fetching the previously installed CoreDNS version\\n\\n***WARNING***\\nUpgrading Kubernetes with this script might result in an upgrade to a new etcd version.\\nSome etcd version upgrades, such as 3.0.x to 3.1.x, DO NOT offer a downgrade path.\\nTo pin the etcd version to your current one (e.g. v3.0.17), set the following variables\\nbefore running this script:\\n\\n# example: pin to etcd v3.0.17\\nexport ETCD_IMAGE=3.0.17\\nexport ETCD_VERSION=3.0.17\\n\\nAlternatively, if you choose to allow an etcd upgrade that doesn't support downgrade,\\nyou might still be able to downgrade Kubernetes by pinning to the newer etcd version.\\nIn all cases, it is strongly recommended to have an etcd backup before upgrading.\\n\\n== Pre-Upgrade Node OS and Kubelet Versions ==\\nname: \\\"bootstrap-e2e-master\\\", osImage: \\\"Container-Optimized OS from Google\\\", kubeletVersion: \\\"v1.16.9-beta.0.9+3b7cbcf52403ff\\\"\\nname: \\\"bootstrap-e2e-minion-group-8lfn\\\", osImage: \\\"Container-Optimized OS from Google\\\", kubeletVersion: \\\"v1.16.9-beta.0.9+3b7cbcf52403ff\\\"\\nname: \\\"bootstrap-e2e-minion-group-hlrz\\\", osImage: \\\"Container-Optimized OS from Google\\\", kubeletVersion: \\\"v1.16.9-beta.0.9+3b7cbcf52403ff\\\"\\nname: \\\"bootstrap-e2e-minion-group-vgzb\\\", osImage: \\\"Container-Optimized OS from Google\\\", kubeletVersion: \\\"v1.16.9-beta.0.9+3b7cbcf52403ff\\\"\\nFound subnet for region us-west1 in network bootstrap-e2e: bootstrap-e2e\\n== Upgrading master to 'https://storage.googleapis.com/kubernetes-release-dev/ci/v1.17.5-beta.0.13+222605b0aa03bc/kubernetes-server-linux-amd64.tar.gz'. Do not interrupt, deleting master instance. ==\\n== Upgrading master environment variables. ==\\n== Waiting for new master to respond to API requests ==\\n........................== Done ==\\nWaiting for CoreDNS to update\\nFetching the latest installed CoreDNS version\\n== Downloading the CoreDNS migration tool ==\\n== Upgrading the CoreDNS ConfigMap ==\\nconfigmap/coredns configured\\n== The CoreDNS Config has been updated ==\\n== Validating cluster post-upgrade ==\\nValidating gce cluster, MULTIZONE=\\nFound 4 node(s).\\nNAME                              STATUS                     ROLES    AGE     VERSION\\nbootstrap-e2e-master              Ready,SchedulingDisabled   <none>   9m51s   v1.17.5-beta.0.13+222605b0aa03bc\\nbootstrap-e2e-minion-group-8lfn   Ready                      <none>   9m55s   v1.16.9-beta.0.9+3b7cbcf52403ff\\nbootstrap-e2e-minion-group-hlrz   Ready                      <none>   9m55s   v1.16.9-beta.0.9+3b7cbcf52403ff\\nbootstrap-e2e-minion-group-vgzb   Ready                      <none>   9m54s   v1.16.9-beta.0.9+3b7cbcf52403ff\\nValidate output:\\nNAME                 STATUS    MESSAGE             ERROR\\netcd-0               Healthy   {\\\"health\\\":\\\"true\\\"}   \\nscheduler            Healthy   ok                  \\ncontroller-manager   Healthy   ok                  \\netcd-1               Healthy   {\\\"health\\\":\\\"true\\\"}   \\n\\x1b[0;32mCluster validation succeeded\\x1b[0m\\n== Post-Upgrade Node OS and Kubelet Versions ==\\nname: \\\"bootstrap-e2e-master\\\", osImage: \\\"Container-Optimized OS from Google\\\", kubeletVersion: \\\"v1.17.5-beta.0.13+222605b0aa03bc\\\"\\nname: \\\"bootstrap-e2e-minion-group-8lfn\\\", osImage: \\\"Container-Optimized OS from Google\\\", kubeletVersion: \\\"v1.16.9-beta.0.9+3b7cbcf52403ff\\\"\\nname: \\\"bootstrap-e2e-minion-group-hlrz\\\", osImage: \\\"Container-Optimized OS from Google\\\", kubeletVersion: \\\"v1.16.9-beta.0.9+3b7cbcf52403ff\\\"\\nname: \\\"bootstrap-e2e-minion-group-vgzb\\\", osImage: \\\"Container-Optimized OS from Google\\\", kubeletVersion: \\\"v1.16.9-beta.0.9+3b7cbcf52403ff\\\"\\n\", stderr \"Project: k8s-gce-soak-1-5\\nNetwork Project: k8s-gce-soak-1-5\\nZone: us-west1-b\\nINSTANCE_GROUPS=bootstrap-e2e-minion-group\\nNODE_NAMES=bootstrap-e2e-minion-group-8lfn bootstrap-e2e-minion-group-hlrz bootstrap-e2e-minion-group-vgzb\\nTrying to find master named 'bootstrap-e2e-master'\\nLooking for address 'bootstrap-e2e-master-ip'\\nUsing master: bootstrap-e2e-master (external IP: 34.82.199.207; internal IP: (not set))\\nDeleted [https://www.googleapis.com/compute/v1/projects/k8s-gce-soak-1-5/zones/us-west1-b/instances/bootstrap-e2e-master].\\nWARNING: You have selected a disk size of under [200GB]. This may result in poor I/O performance. For more information, see: https://developers.google.com/compute/docs/disks#performance.\\nCreated [https://www.googleapis.com/compute/v1/projects/k8s-gce-soak-1-5/zones/us-west1-b/instances/bootstrap-e2e-master].\\nWARNING: Some requests generated warnings:\\n - Disk size: '20 GB' is larger than image size: '10 GB'. You might need to resize the root repartition manually if the operating system does not support automatic resizing. See https://cloud.google.com/compute/docs/disks/add-persistent-disk#resize_pd for details.\\n - The resource 'projects/cos-cloud/global/images/cos-77-12371-175-0' is deprecated. A suggested replacement is 'projects/cos-cloud/global/images/cos-77-12371-183-0'.\\n\\nNAME                  ZONE        MACHINE_TYPE   PREEMPTIBLE  INTERNAL_IP  EXTERNAL_IP    STATUS\\nbootstrap-e2e-master  us-west1-b  n1-standard-1               10.138.0.6   34.82.199.207  RUNNING\\nWarning: kubectl apply should be used on resource created by either kubectl create --save-config or kubectl apply\\nProject: k8s-gce-soak-1-5\\nNetwork Project: k8s-gce-soak-1-5\\nZone: us-west1-b\\n/workspace/kubernetes_skew/cluster/gce/upgrade.sh: line 452: download_dir: unbound variable\\n\"",
    }
    error running /workspace/kubernetes_skew/cluster/gce/upgrade.sh [-M v1.17.5-beta.0.13+222605b0aa03bc]; got error exit status 1, stdout "Fetching the previously installed CoreDNS version\n\n***WARNING***\nUpgrading Kubernetes with this script might result in an upgrade to a new etcd version.\nSome etcd version upgrades, such as 3.0.x to 3.1.x, DO NOT offer a downgrade path.\nTo pin the etcd version to your current one (e.g. v3.0.17), set the following variables\nbefore running this script:\n\n# example: pin to etcd v3.0.17\nexport ETCD_IMAGE=3.0.17\nexport ETCD_VERSION=3.0.17\n\nAlternatively, if you choose to allow an etcd upgrade that doesn't support downgrade,\nyou might still be able to downgrade Kubernetes by pinning to the newer etcd version.\nIn all cases, it is strongly recommended to have an etcd backup before upgrading.\n\n== Pre-Upgrade Node OS and Kubelet Versions ==\nname: \"bootstrap-e2e-master\", osImage: \"Container-Optimized OS from Google\", kubeletVersion: \"v1.16.9-beta.0.9+3b7cbcf52403ff\"\nname: \"bootstrap-e2e-minion-group-8lfn\", osImage: \"Container-Optimized OS from Google\", kubeletVersion: \"v1.16.9-beta.0.9+3b7cbcf52403ff\"\nname: \"bootstrap-e2e-minion-group-hlrz\", osImage: \"Container-Optimized OS from Google\", kubeletVersion: \"v1.16.9-beta.0.9+3b7cbcf52403ff\"\nname: \"bootstrap-e2e-minion-group-vgzb\", osImage: \"Container-Optimized OS from Google\", kubeletVersion: \"v1.16.9-beta.0.9+3b7cbcf52403ff\"\nFound subnet for region us-west1 in network bootstrap-e2e: bootstrap-e2e\n== Upgrading master to 'https://storage.googleapis.com/kubernetes-release-dev/ci/v1.17.5-beta.0.13+222605b0aa03bc/kubernetes-server-linux-amd64.tar.gz'. Do not interrupt, deleting master instance. ==\n== Upgrading master environment variables. ==\n== Waiting for new master to respond to API requests ==\n........................== Done ==\nWaiting for CoreDNS to update\nFetching the latest installed CoreDNS version\n== Downloading the CoreDNS migration tool ==\n== Upgrading the CoreDNS ConfigMap ==\nconfigmap/coredns configured\n== The CoreDNS Config has been updated ==\n== Validating cluster post-upgrade ==\nValidating gce cluster, MULTIZONE=\nFound 4 node(s).\nNAME                              STATUS                     ROLES    AGE     VERSION\nbootstrap-e2e-master              Ready,SchedulingDisabled   <none>   9m51s   v1.17.5-beta.0.13+222605b0aa03bc\nbootstrap-e2e-minion-group-8lfn   Ready                      <none>   9m55s   v1.16.9-beta.0.9+3b7cbcf52403ff\nbootstrap-e2e-minion-group-hlrz   Ready                      <none>   9m55s   v1.16.9-beta.0.9+3b7cbcf52403ff\nbootstrap-e2e-minion-group-vgzb   Ready                      <none>   9m54s   v1.16.9-beta.0.9+3b7cbcf52403ff\nValidate output:\nNAME                 STATUS    MESSAGE             ERROR\netcd-0               Healthy   {\"health\":\"true\"}   \nscheduler            Healthy   ok                  \ncontroller-manager   Healthy   ok                  \netcd-1               Healthy   {\"health\":\"true\"}   \n\x1b[0;32mCluster validation succeeded\x1b[0m\n== Post-Upgrade Node OS and Kubelet Versions ==\nname: \"bootstrap-e2e-master\", osImage: \"Container-Optimized OS from Google\", kubeletVersion: \"v1.17.5-beta.0.13+222605b0aa03bc\"\nname: \"bootstrap-e2e-minion-group-8lfn\", osImage: \"Container-Optimized OS from Google\", kubeletVersion: \"v1.16.9-beta.0.9+3b7cbcf52403ff\"\nname: \"bootstrap-e2e-minion-group-hlrz\", osImage: \"Container-Optimized OS from Google\", kubeletVersion: \"v1.16.9-beta.0.9+3b7cbcf52403ff\"\nname: \"bootstrap-e2e-minion-group-vgzb\", osImage: \"Container-Optimized OS from Google\", kubeletVersion: \"v1.16.9-beta.0.9+3b7cbcf52403ff\"\n", stderr "Project: k8s-gce-soak-1-5\nNetwork Project: k8s-gce-soak-1-5\nZone: us-west1-b\nINSTANCE_GROUPS=bootstrap-e2e-minion-group\nNODE_NAMES=bootstrap-e2e-minion-group-8lfn bootstrap-e2e-minion-group-hlrz bootstrap-e2e-minion-group-vgzb\nTrying to find master named 'bootstrap-e2e-master'\nLooking for address 'bootstrap-e2e-master-ip'\nUsing master: bootstrap-e2e-master (external IP: 34.82.199.207; internal IP: (not set))\nDeleted [https://www.googleapis.com/compute/v1/projects/k8s-gce-soak-1-5/zones/us-west1-b/instances/bootstrap-e2e-master].\nWARNING: You have selected a disk size of under [200GB]. This may result in poor I/O performance. For more information, see: https://developers.google.com/compute/docs/disks#performance.\nCreated [https://www.googleapis.com/compute/v1/projects/k8s-gce-soak-1-5/zones/us-west1-b/instances/bootstrap-e2e-master].\nWARNING: Some requests generated warnings:\n - Disk size: '20 GB' is larger than image size: '10 GB'. You might need to resize the root repartition manually if the operating system does not support automatic resizing. See https://cloud.google.com/compute/docs/disks/add-persistent-disk#resize_pd for details.\n - The resource 'projects/cos-cloud/global/images/cos-77-12371-175-0' is deprecated. A suggested replacement is 'projects/cos-cloud/global/images/cos-77-12371-183-0'.\n\nNAME                  ZONE        MACHINE_TYPE   PREEMPTIBLE  INTERNAL_IP  EXTERNAL_IP    STATUS\nbootstrap-e2e-master  us-west1-b  n1-standard-1               10.138.0.6   34.82.199.207  RUNNING\nWarning: kubectl apply should be used on resource created by either kubectl create --save-config or kubectl apply\nProject: k8s-gce-soak-1-5\nNetwork Project: k8s-gce-soak-1-5\nZone: us-west1-b\n/workspace/kubernetes_skew/cluster/gce/upgrade.sh: line 452: download_dir: unbound variable\n"
occurred

k8s.io/kubernetes/test/e2e/cloud/gcp.glob..func2.1.1.1()
	/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/cloud/gcp/cluster_upgrade.go:106 +0x12c
k8s.io/kubernetes/test/e2e/chaosmonkey.(*Chaosmonkey).Do(0xc002fbd200)
	/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/chaosmonkey/chaosmonkey.go:111 +0x38a
k8s.io/kubernetes/test/e2e/cloud/gcp.runUpgradeSuite(0xc000b223c0, 0x7757ca0, 0xc, 0xc, 0xc00078d680, 0xc0035251d0, 0x0, 0xc0023bb2c0)
	/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/cloud/gcp/cluster_upgrade.go:479 +0x47a
k8s.io/kubernetes/test/e2e/cloud/gcp.glob..func2.1.1()
	/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/cloud/gcp/cluster_upgrade.go:109 +0x222
k8s.io/kubernetes/test/e2e.RunE2ETests(0xc002f04100)
	_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/e2e.go:110 +0x30a
k8s.io/kubernetes/test/e2e.TestE2E(0xc002f04100)
	_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/e2e_test.go:112 +0x2b
testing.tRunner(0xc002f04100, 0x49f9638)
	/usr/local/go/src/testing/testing.go:909 +0xc9
created by testing.(*T).Run
	/usr/local/go/src/testing/testing.go:960 +0x350
				from junit_upgradeupgrades.xml

Filter through log files | View test history on testgrid


Test 14h34m

error during ./hack/ginkgo-e2e.sh --kubectl-path=../../../../kubernetes_skew/cluster/kubectl.sh --minStartupPods=8 --report-dir=/workspace/_artifacts --disable-log-dump=true (interrupted): exit status 1
				from junit_runner.xml

Filter through log files | View test history on testgrid


Timeout 15h0m

kubetest --timeout triggered
				from junit_runner.xml

Filter through log files | View test history on testgrid


UpgradeTest 19m0s

error during kubetest --test --test_args=--ginkgo.focus=\[Feature:MasterUpgrade\] --upgrade-target=ci/k8s-stable1 --report-dir=/workspace/_artifacts --disable-log-dump=true --report-prefix=upgrade --check-version-skew=false: exit status 1
				from junit_runner.xml

Filter through log files | View test history on testgrid


Show 934 Passed Tests