This job view page is being replaced by Spyglass soon. Check out the new job view.
ResultFAILURE
Tests 7 failed / 925 succeeded
Started2020-03-20 02:53
Elapsed15h15m
Revision
Builderd73fcd68-6a55-11ea-8217-ae90f73d0ce9
links{u'resultstore': {u'url': u'https://source.cloud.google.com/results/invocations/5c11a1f1-b209-40cb-b8de-756ec9e34069/targets/test'}}
resultstorehttps://source.cloud.google.com/results/invocations/5c11a1f1-b209-40cb-b8de-756ec9e34069/targets/test
infra-commitc05cefcf0
job-versionv1.16.9-beta.0.7+5116ee4b159565
master_os_imagecos-77-12371-175-0
node_os_imagecos-73-11647-163-0
revisionv1.16.9-beta.0.7+5116ee4b159565

Test Failures


Kubernetes e2e suite [k8s.io] [sig-cloud-provider-gcp] Upgrade [Feature:Upgrade] master upgrade should maintain a functioning cluster [Feature:MasterUpgrade] 18m21s

go run hack/e2e.go -v --test --test_args='--ginkgo.focus=Kubernetes\se2e\ssuite\s\[k8s\.io\]\s\[sig\-cloud\-provider\-gcp\]\sUpgrade\s\[Feature\:Upgrade\]\smaster\supgrade\sshould\smaintain\sa\sfunctioning\scluster\s\[Feature\:MasterUpgrade\]$'
/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/cloud/gcp/cluster_upgrade.go:91
Mar 20 03:08:53.696: Unexpected error:
    <*errors.errorString | 0xc001b42060>: {
        s: "error running /workspace/kubernetes_skew/cluster/gce/upgrade.sh [-M v1.17.5-beta.0.11+4dfcd1cc87879f]; got error exit status 1, stdout \"Fetching the previously installed CoreDNS version\\n\\n***WARNING***\\nUpgrading Kubernetes with this script might result in an upgrade to a new etcd version.\\nSome etcd version upgrades, such as 3.0.x to 3.1.x, DO NOT offer a downgrade path.\\nTo pin the etcd version to your current one (e.g. v3.0.17), set the following variables\\nbefore running this script:\\n\\n# example: pin to etcd v3.0.17\\nexport ETCD_IMAGE=3.0.17\\nexport ETCD_VERSION=3.0.17\\n\\nAlternatively, if you choose to allow an etcd upgrade that doesn't support downgrade,\\nyou might still be able to downgrade Kubernetes by pinning to the newer etcd version.\\nIn all cases, it is strongly recommended to have an etcd backup before upgrading.\\n\\n== Pre-Upgrade Node OS and Kubelet Versions ==\\nname: \\\"bootstrap-e2e-master\\\", osImage: \\\"Container-Optimized OS from Google\\\", kubeletVersion: \\\"v1.16.9-beta.0.7+5116ee4b159565\\\"\\nname: \\\"bootstrap-e2e-minion-group-0g7j\\\", osImage: \\\"Container-Optimized OS from Google\\\", kubeletVersion: \\\"v1.16.9-beta.0.7+5116ee4b159565\\\"\\nname: \\\"bootstrap-e2e-minion-group-2873\\\", osImage: \\\"Container-Optimized OS from Google\\\", kubeletVersion: \\\"v1.16.9-beta.0.7+5116ee4b159565\\\"\\nname: \\\"bootstrap-e2e-minion-group-v8hd\\\", osImage: \\\"Container-Optimized OS from Google\\\", kubeletVersion: \\\"v1.16.9-beta.0.7+5116ee4b159565\\\"\\nFound subnet for region us-west1 in network bootstrap-e2e: bootstrap-e2e\\n== Upgrading master to 'https://storage.googleapis.com/kubernetes-release-dev/ci/v1.17.5-beta.0.11+4dfcd1cc87879f/kubernetes-server-linux-amd64.tar.gz'. Do not interrupt, deleting master instance. ==\\n== Upgrading master environment variables. ==\\n== Waiting for new master to respond to API requests ==\\n......................== Done ==\\nWaiting for CoreDNS to update\\nFetching the latest installed CoreDNS version\\n== Downloading the CoreDNS migration tool ==\\n== Upgrading the CoreDNS ConfigMap ==\\nconfigmap/coredns configured\\n== The CoreDNS Config has been updated ==\\n== Validating cluster post-upgrade ==\\nValidating gce cluster, MULTIZONE=\\nFound 4 node(s).\\nNAME                              STATUS                     ROLES    AGE     VERSION\\nbootstrap-e2e-master              Ready,SchedulingDisabled   <none>   9m38s   v1.17.5-beta.0.11+4dfcd1cc87879f\\nbootstrap-e2e-minion-group-0g7j   Ready                      <none>   9m32s   v1.16.9-beta.0.7+5116ee4b159565\\nbootstrap-e2e-minion-group-2873   Ready                      <none>   9m41s   v1.16.9-beta.0.7+5116ee4b159565\\nbootstrap-e2e-minion-group-v8hd   Ready                      <none>   9m40s   v1.16.9-beta.0.7+5116ee4b159565\\nValidate output:\\nNAME                 STATUS    MESSAGE             ERROR\\ncontroller-manager   Healthy   ok                  \\netcd-1               Healthy   {\\\"health\\\":\\\"true\\\"}   \\nscheduler            Healthy   ok                  \\netcd-0               Healthy   {\\\"health\\\":\\\"true\\\"}   \\n\\x1b[0;32mCluster validation succeeded\\x1b[0m\\n== Post-Upgrade Node OS and Kubelet Versions ==\\nname: \\\"bootstrap-e2e-master\\\", osImage: \\\"Container-Optimized OS from Google\\\", kubeletVersion: \\\"v1.17.5-beta.0.11+4dfcd1cc87879f\\\"\\nname: \\\"bootstrap-e2e-minion-group-0g7j\\\", osImage: \\\"Container-Optimized OS from Google\\\", kubeletVersion: \\\"v1.16.9-beta.0.7+5116ee4b159565\\\"\\nname: \\\"bootstrap-e2e-minion-group-2873\\\", osImage: \\\"Container-Optimized OS from Google\\\", kubeletVersion: \\\"v1.16.9-beta.0.7+5116ee4b159565\\\"\\nname: \\\"bootstrap-e2e-minion-group-v8hd\\\", osImage: \\\"Container-Optimized OS from Google\\\", kubeletVersion: \\\"v1.16.9-beta.0.7+5116ee4b159565\\\"\\n\", stderr \"Project: kubernetes-gci-ingress-1-3\\nNetwork Project: kubernetes-gci-ingress-1-3\\nZone: us-west1-b\\nINSTANCE_GROUPS=bootstrap-e2e-minion-group\\nNODE_NAMES=bootstrap-e2e-minion-group-0g7j bootstrap-e2e-minion-group-2873 bootstrap-e2e-minion-group-v8hd\\nTrying to find master named 'bootstrap-e2e-master'\\nLooking for address 'bootstrap-e2e-master-ip'\\nUsing master: bootstrap-e2e-master (external IP: 34.83.213.100; internal IP: (not set))\\nDeleted [https://www.googleapis.com/compute/v1/projects/kubernetes-gci-ingress-1-3/zones/us-west1-b/instances/bootstrap-e2e-master].\\nWARNING: You have selected a disk size of under [200GB]. This may result in poor I/O performance. For more information, see: https://developers.google.com/compute/docs/disks#performance.\\nCreated [https://www.googleapis.com/compute/v1/projects/kubernetes-gci-ingress-1-3/zones/us-west1-b/instances/bootstrap-e2e-master].\\nWARNING: Some requests generated warnings:\\n - Disk size: '20 GB' is larger than image size: '10 GB'. You might need to resize the root repartition manually if the operating system does not support automatic resizing. See https://cloud.google.com/compute/docs/disks/add-persistent-disk#resize_pd for details.\\n - The resource 'projects/cos-cloud/global/images/cos-77-12371-175-0' is deprecated. A suggested replacement is 'projects/cos-cloud/global/images/cos-77-12371-183-0'.\\n\\nNAME                  ZONE        MACHINE_TYPE   PREEMPTIBLE  INTERNAL_IP  EXTERNAL_IP    STATUS\\nbootstrap-e2e-master  us-west1-b  n1-standard-1               10.138.0.6   34.83.213.100  RUNNING\\nWarning: kubectl apply should be used on resource created by either kubectl create --save-config or kubectl apply\\nProject: kubernetes-gci-ingress-1-3\\nNetwork Project: kubernetes-gci-ingress-1-3\\nZone: us-west1-b\\n/workspace/kubernetes_skew/cluster/gce/upgrade.sh: line 452: download_dir: unbound variable\\n\"",
    }
    error running /workspace/kubernetes_skew/cluster/gce/upgrade.sh [-M v1.17.5-beta.0.11+4dfcd1cc87879f]; got error exit status 1, stdout "Fetching the previously installed CoreDNS version\n\n***WARNING***\nUpgrading Kubernetes with this script might result in an upgrade to a new etcd version.\nSome etcd version upgrades, such as 3.0.x to 3.1.x, DO NOT offer a downgrade path.\nTo pin the etcd version to your current one (e.g. v3.0.17), set the following variables\nbefore running this script:\n\n# example: pin to etcd v3.0.17\nexport ETCD_IMAGE=3.0.17\nexport ETCD_VERSION=3.0.17\n\nAlternatively, if you choose to allow an etcd upgrade that doesn't support downgrade,\nyou might still be able to downgrade Kubernetes by pinning to the newer etcd version.\nIn all cases, it is strongly recommended to have an etcd backup before upgrading.\n\n== Pre-Upgrade Node OS and Kubelet Versions ==\nname: \"bootstrap-e2e-master\", osImage: \"Container-Optimized OS from Google\", kubeletVersion: \"v1.16.9-beta.0.7+5116ee4b159565\"\nname: \"bootstrap-e2e-minion-group-0g7j\", osImage: \"Container-Optimized OS from Google\", kubeletVersion: \"v1.16.9-beta.0.7+5116ee4b159565\"\nname: \"bootstrap-e2e-minion-group-2873\", osImage: \"Container-Optimized OS from Google\", kubeletVersion: \"v1.16.9-beta.0.7+5116ee4b159565\"\nname: \"bootstrap-e2e-minion-group-v8hd\", osImage: \"Container-Optimized OS from Google\", kubeletVersion: \"v1.16.9-beta.0.7+5116ee4b159565\"\nFound subnet for region us-west1 in network bootstrap-e2e: bootstrap-e2e\n== Upgrading master to 'https://storage.googleapis.com/kubernetes-release-dev/ci/v1.17.5-beta.0.11+4dfcd1cc87879f/kubernetes-server-linux-amd64.tar.gz'. Do not interrupt, deleting master instance. ==\n== Upgrading master environment variables. ==\n== Waiting for new master to respond to API requests ==\n......................== Done ==\nWaiting for CoreDNS to update\nFetching the latest installed CoreDNS version\n== Downloading the CoreDNS migration tool ==\n== Upgrading the CoreDNS ConfigMap ==\nconfigmap/coredns configured\n== The CoreDNS Config has been updated ==\n== Validating cluster post-upgrade ==\nValidating gce cluster, MULTIZONE=\nFound 4 node(s).\nNAME                              STATUS                     ROLES    AGE     VERSION\nbootstrap-e2e-master              Ready,SchedulingDisabled   <none>   9m38s   v1.17.5-beta.0.11+4dfcd1cc87879f\nbootstrap-e2e-minion-group-0g7j   Ready                      <none>   9m32s   v1.16.9-beta.0.7+5116ee4b159565\nbootstrap-e2e-minion-group-2873   Ready                      <none>   9m41s   v1.16.9-beta.0.7+5116ee4b159565\nbootstrap-e2e-minion-group-v8hd   Ready                      <none>   9m40s   v1.16.9-beta.0.7+5116ee4b159565\nValidate output:\nNAME                 STATUS    MESSAGE             ERROR\ncontroller-manager   Healthy   ok                  \netcd-1               Healthy   {\"health\":\"true\"}   \nscheduler            Healthy   ok                  \netcd-0               Healthy   {\"health\":\"true\"}   \n\x1b[0;32mCluster validation succeeded\x1b[0m\n== Post-Upgrade Node OS and Kubelet Versions ==\nname: \"bootstrap-e2e-master\", osImage: \"Container-Optimized OS from Google\", kubeletVersion: \"v1.17.5-beta.0.11+4dfcd1cc87879f\"\nname: \"bootstrap-e2e-minion-group-0g7j\", osImage: \"Container-Optimized OS from Google\", kubeletVersion: \"v1.16.9-beta.0.7+5116ee4b159565\"\nname: \"bootstrap-e2e-minion-group-2873\", osImage: \"Container-Optimized OS from Google\", kubeletVersion: \"v1.16.9-beta.0.7+5116ee4b159565\"\nname: \"bootstrap-e2e-minion-group-v8hd\", osImage: \"Container-Optimized OS from Google\", kubeletVersion: \"v1.16.9-beta.0.7+5116ee4b159565\"\n", stderr "Project: kubernetes-gci-ingress-1-3\nNetwork Project: kubernetes-gci-ingress-1-3\nZone: us-west1-b\nINSTANCE_GROUPS=bootstrap-e2e-minion-group\nNODE_NAMES=bootstrap-e2e-minion-group-0g7j bootstrap-e2e-minion-group-2873 bootstrap-e2e-minion-group-v8hd\nTrying to find master named 'bootstrap-e2e-master'\nLooking for address 'bootstrap-e2e-master-ip'\nUsing master: bootstrap-e2e-master (external IP: 34.83.213.100; internal IP: (not set))\nDeleted [https://www.googleapis.com/compute/v1/projects/kubernetes-gci-ingress-1-3/zones/us-west1-b/instances/bootstrap-e2e-master].\nWARNING: You have selected a disk size of under [200GB]. This may result in poor I/O performance. For more information, see: https://developers.google.com/compute/docs/disks#performance.\nCreated [https://www.googleapis.com/compute/v1/projects/kubernetes-gci-ingress-1-3/zones/us-west1-b/instances/bootstrap-e2e-master].\nWARNING: Some requests generated warnings:\n - Disk size: '20 GB' is larger than image size: '10 GB'. You might need to resize the root repartition manually if the operating system does not support automatic resizing. See https://cloud.google.com/compute/docs/disks/add-persistent-disk#resize_pd for details.\n - The resource 'projects/cos-cloud/global/images/cos-77-12371-175-0' is deprecated. A suggested replacement is 'projects/cos-cloud/global/images/cos-77-12371-183-0'.\n\nNAME                  ZONE        MACHINE_TYPE   PREEMPTIBLE  INTERNAL_IP  EXTERNAL_IP    STATUS\nbootstrap-e2e-master  us-west1-b  n1-standard-1               10.138.0.6   34.83.213.100  RUNNING\nWarning: kubectl apply should be used on resource created by either kubectl create --save-config or kubectl apply\nProject: kubernetes-gci-ingress-1-3\nNetwork Project: kubernetes-gci-ingress-1-3\nZone: us-west1-b\n/workspace/kubernetes_skew/cluster/gce/upgrade.sh: line 452: download_dir: unbound variable\n"
occurred
/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/cloud/gcp/cluster_upgrade.go:106
				
				Click to see stdout/stderrfrom junit_upgrade01.xml

Filter through log files | View test history on testgrid


Kubernetes e2e suite [k8s.io] [sig-node] NodeProblemDetector [DisabledForLargeClusters] should run without error 2m16s

go run hack/e2e.go -v --test --test_args='--ginkgo.focus=Kubernetes\se2e\ssuite\s\[k8s\.io\]\s\[sig\-node\]\sNodeProblemDetector\s\[DisabledForLargeClusters\]\sshould\srun\swithout\serror$'
/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/node/node_problem_detector.go:58
Mar 20 05:49:38.383: Timed out after 60.001s.
Expected success, but got an error:
    <*errors.errorString | 0xc003a6a880>: {
        s: "Event KubeletStart does not exist: [{{ } {bootstrap-e2e-minion-group-0g7j.15fded13dd7b3aeb  default /api/v1/namespaces/default/events/bootstrap-e2e-minion-group-0g7j.15fded13dd7b3aeb 179bbb80-d128-4c4c-9ed7-1dc222e034a4 6437 0 2020-03-20 05:48:33 +0000 UTC <nil> <nil> map[] map[] [] []  []} {Node  bootstrap-e2e-minion-group-0g7j bootstrap-e2e-minion-group-0g7j   } TaskHung kernel: INFO: task umount.aufs:21568 blocked for more than 120 seconds. {kernel-monitor bootstrap-e2e-minion-group-0g7j} 2020-03-20 05:48:33 +0000 UTC 2020-03-20 05:48:33 +0000 UTC 1 Warning 0001-01-01 00:00:00 +0000 UTC nil  nil  } {{ } {bootstrap-e2e-minion-group-0g7j.15fded13dd7b5e6b  default /api/v1/namespaces/default/events/bootstrap-e2e-minion-group-0g7j.15fded13dd7b5e6b 713ff2e9-19e2-47cf-ac06-deeab504a1b4 6436 0 2020-03-20 05:48:33 +0000 UTC <nil> <nil> map[] map[] [] []  []} {Node  bootstrap-e2e-minion-group-0g7j bootstrap-e2e-minion-group-0g7j   } AUFSUmountHung Node condition KernelDeadlock is now: True, reason: AUFSUmountHung {kernel-monitor bootstrap-e2e-minion-group-0g7j} 2020-03-20 05:48:33 +0000 UTC 2020-03-20 05:48:33 +0000 UTC 1 Normal 0001-01-01 00:00:00 +0000 UTC nil  nil  } {{ } {bootstrap-e2e-minion-group-2873.15fded146af2eebc  default /api/v1/namespaces/default/events/bootstrap-e2e-minion-group-2873.15fded146af2eebc 7be70a79-333a-4687-a3ca-1820a313a211 6439 0 2020-03-20 05:48:35 +0000 UTC <nil> <nil> map[] map[] [] []  []} {Node  bootstrap-e2e-minion-group-2873 bootstrap-e2e-minion-group-2873   } TaskHung kernel: INFO: task umount.aufs:21568 blocked for more than 120 seconds. {kernel-monitor bootstrap-e2e-minion-group-2873} 2020-03-20 05:48:35 +0000 UTC 2020-03-20 05:48:35 +0000 UTC 1 Warning 0001-01-01 00:00:00 +0000 UTC nil  nil  } {{ } {bootstrap-e2e-minion-group-2873.15fded146af30c59  default /api/v1/namespaces/default/events/bootstrap-e2e-minion-group-2873.15fded146af30c59 e010e87d-d0d9-491a-87dd-18a313910fc1 6438 0 2020-03-20 05:48:35 +0000 UTC <nil> <nil> map[] map[] [] []  []} {Node  bootstrap-e2e-minion-group-2873 bootstrap-e2e-minion-group-2873   } AUFSUmountHung Node condition KernelDeadlock is now: True, reason: AUFSUmountHung {kernel-monitor bootstrap-e2e-minion-group-2873} 2020-03-20 05:48:35 +0000 UTC 2020-03-20 05:48:35 +0000 UTC 1 Normal 0001-01-01 00:00:00 +0000 UTC nil  nil  } {{ } {bootstrap-e2e-minion-group-v8hd.15fded14fb1ace2f  default /api/v1/namespaces/default/events/bootstrap-e2e-minion-group-v8hd.15fded14fb1ace2f 7081ab95-779e-44b7-b31c-2df8dd60ce43 6441 0 2020-03-20 05:48:38 +0000 UTC <nil> <nil> map[] map[] [] []  []} {Node  bootstrap-e2e-minion-group-v8hd bootstrap-e2e-minion-group-v8hd   } TaskHung kernel: INFO: task umount.aufs:21568 blocked for more than 120 seconds. {kernel-monitor bootstrap-e2e-minion-group-v8hd} 2020-03-20 05:48:38 +0000 UTC 2020-03-20 05:48:38 +0000 UTC 1 Warning 0001-01-01 00:00:00 +0000 UTC nil  nil  } {{ } {bootstrap-e2e-minion-group-v8hd.15fded14fb1aeb2f  default /api/v1/namespaces/default/events/bootstrap-e2e-minion-group-v8hd.15fded14fb1aeb2f 7fec400b-a4a3-4fa0-bc3e-ef128a5f167f 6440 0 2020-03-20 05:48:38 +0000 UTC <nil> <nil> map[] map[] [] []  []} {Node  bootstrap-e2e-minion-group-v8hd bootstrap-e2e-minion-group-v8hd   } AUFSUmountHung Node condition KernelDeadlock is now: True, reason: AUFSUmountHung {kernel-monitor bootstrap-e2e-minion-group-v8hd} 2020-03-20 05:48:38 +0000 UTC 2020-03-20 05:48:38 +0000 UTC 1 Normal 0001-01-01 00:00:00 +0000 UTC nil  nil  }]",
    }
    Event KubeletStart does not exist: [{{ } {bootstrap-e2e-minion-group-0g7j.15fded13dd7b3aeb  default /api/v1/namespaces/default/events/bootstrap-e2e-minion-group-0g7j.15fded13dd7b3aeb 179bbb80-d128-4c4c-9ed7-1dc222e034a4 6437 0 2020-03-20 05:48:33 +0000 UTC <nil> <nil> map[] map[] [] []  []} {Node  bootstrap-e2e-minion-group-0g7j bootstrap-e2e-minion-group-0g7j   } TaskHung kernel: INFO: task umount.aufs:21568 blocked for more than 120 seconds. {kernel-monitor bootstrap-e2e-minion-group-0g7j} 2020-03-20 05:48:33 +0000 UTC 2020-03-20 05:48:33 +0000 UTC 1 Warning 0001-01-01 00:00:00 +0000 UTC nil  nil  } {{ } {bootstrap-e2e-minion-group-0g7j.15fded13dd7b5e6b  default /api/v1/namespaces/default/events/bootstrap-e2e-minion-group-0g7j.15fded13dd7b5e6b 713ff2e9-19e2-47cf-ac06-deeab504a1b4 6436 0 2020-03-20 05:48:33 +0000 UTC <nil> <nil> map[] map[] [] []  []} {Node  bootstrap-e2e-minion-group-0g7j bootstrap-e2e-minion-group-0g7j   } AUFSUmountHung Node condition KernelDeadlock is now: True, reason: AUFSUmountHung {kernel-monitor bootstrap-e2e-minion-group-0g7j} 2020-03-20 05:48:33 +0000 UTC 2020-03-20 05:48:33 +0000 UTC 1 Normal 0001-01-01 00:00:00 +0000 UTC nil  nil  } {{ } {bootstrap-e2e-minion-group-2873.15fded146af2eebc  default /api/v1/namespaces/default/events/bootstrap-e2e-minion-group-2873.15fded146af2eebc 7be70a79-333a-4687-a3ca-1820a313a211 6439 0 2020-03-20 05:48:35 +0000 UTC <nil> <nil> map[] map[] [] []  []} {Node  bootstrap-e2e-minion-group-2873 bootstrap-e2e-minion-group-2873   } TaskHung kernel: INFO: task umount.aufs:21568 blocked for more than 120 seconds. {kernel-monitor bootstrap-e2e-minion-group-2873} 2020-03-20 05:48:35 +0000 UTC 2020-03-20 05:48:35 +0000 UTC 1 Warning 0001-01-01 00:00:00 +0000 UTC nil  nil  } {{ } {bootstrap-e2e-minion-group-2873.15fded146af30c59  default /api/v1/namespaces/default/events/bootstrap-e2e-minion-group-2873.15fded146af30c59 e010e87d-d0d9-491a-87dd-18a313910fc1 6438 0 2020-03-20 05:48:35 +0000 UTC <nil> <nil> map[] map[] [] []  []} {Node  bootstrap-e2e-minion-group-2873 bootstrap-e2e-minion-group-2873   } AUFSUmountHung Node condition KernelDeadlock is now: True, reason: AUFSUmountHung {kernel-monitor bootstrap-e2e-minion-group-2873} 2020-03-20 05:48:35 +0000 UTC 2020-03-20 05:48:35 +0000 UTC 1 Normal 0001-01-01 00:00:00 +0000 UTC nil  nil  } {{ } {bootstrap-e2e-minion-group-v8hd.15fded14fb1ace2f  default /api/v1/namespaces/default/events/bootstrap-e2e-minion-group-v8hd.15fded14fb1ace2f 7081ab95-779e-44b7-b31c-2df8dd60ce43 6441 0 2020-03-20 05:48:38 +0000 UTC <nil> <nil> map[] map[] [] []  []} {Node  bootstrap-e2e-minion-group-v8hd bootstrap-e2e-minion-group-v8hd   } TaskHung kernel: INFO: task umount.aufs:21568 blocked for more than 120 seconds. {kernel-monitor bootstrap-e2e-minion-group-v8hd} 2020-03-20 05:48:38 +0000 UTC 2020-03-20 05:48:38 +0000 UTC 1 Warning 0001-01-01 00:00:00 +0000 UTC nil  nil  } {{ } {bootstrap-e2e-minion-group-v8hd.15fded14fb1aeb2f  default /api/v1/namespaces/default/events/bootstrap-e2e-minion-group-v8hd.15fded14fb1aeb2f 7fec400b-a4a3-4fa0-bc3e-ef128a5f167f 6440 0 2020-03-20 05:48:38 +0000 UTC <nil> <nil> map[] map[] [] []  []} {Node  bootstrap-e2e-minion-group-v8hd bootstrap-e2e-minion-group-v8hd   } AUFSUmountHung Node condition KernelDeadlock is now: True, reason: AUFSUmountHung {kernel-monitor bootstrap-e2e-minion-group-v8hd} 2020-03-20 05:48:38 +0000 UTC 2020-03-20 05:48:38 +0000 UTC 1 Normal 0001-01-01 00:00:00 +0000 UTC nil  nil  }]
/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/node/node_problem_detector.go:141
				
				Click to see stdout/stderrfrom junit_01.xml

Filter through log files | View test history on testgrid


Kubernetes e2e suite [sig-storage] In-tree Volumes [Driver: local][LocalVolumeType: block] [Testpattern: Pre-provisioned PV (default fs)] subPath should unmount if pod is gracefully deleted while kubelet is down [Disruptive][Slow][LinuxOnly] 6m39s

go run hack/e2e.go -v --test --test_args='--ginkgo.focus=Kubernetes\se2e\ssuite\s\[sig\-storage\]\sIn\-tree\sVolumes\s\[Driver\:\slocal\]\[LocalVolumeType\:\sblock\]\s\[Testpattern\:\sPre\-provisioned\sPV\s\(default\sfs\)\]\ssubPath\sshould\sunmount\sif\spod\sis\sgracefully\sdeleted\swhile\skubelet\sis\sdown\s\[Disruptive\]\[Slow\]\[LinuxOnly\]$'
/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/storage/testsuites/subpath.go:328
Mar 20 05:56:28.197: Expected pod to be not found.
Unexpected error:
    <*errors.errorString | 0xc0000d1000>: {
        s: "timed out waiting for the condition",
    }
    timed out waiting for the condition
occurred
/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/storage/utils/utils.go:278
				
				Click to see stdout/stderrfrom junit_01.xml

Find to mentions in log files | View test history on testgrid


Master upgrade [sig-cloud-provider-gcp] master-upgrade 5m23s

go run hack/e2e.go -v --test --test_args='--ginkgo.focus=Master\supgrade\s\[sig\-cloud\-provider\-gcp\]\smaster\-upgrade$'
Mar 20 03:08:53.696: Unexpected error:
    <*errors.errorString | 0xc001b42060>: {
        s: "error running /workspace/kubernetes_skew/cluster/gce/upgrade.sh [-M v1.17.5-beta.0.11+4dfcd1cc87879f]; got error exit status 1, stdout \"Fetching the previously installed CoreDNS version\\n\\n***WARNING***\\nUpgrading Kubernetes with this script might result in an upgrade to a new etcd version.\\nSome etcd version upgrades, such as 3.0.x to 3.1.x, DO NOT offer a downgrade path.\\nTo pin the etcd version to your current one (e.g. v3.0.17), set the following variables\\nbefore running this script:\\n\\n# example: pin to etcd v3.0.17\\nexport ETCD_IMAGE=3.0.17\\nexport ETCD_VERSION=3.0.17\\n\\nAlternatively, if you choose to allow an etcd upgrade that doesn't support downgrade,\\nyou might still be able to downgrade Kubernetes by pinning to the newer etcd version.\\nIn all cases, it is strongly recommended to have an etcd backup before upgrading.\\n\\n== Pre-Upgrade Node OS and Kubelet Versions ==\\nname: \\\"bootstrap-e2e-master\\\", osImage: \\\"Container-Optimized OS from Google\\\", kubeletVersion: \\\"v1.16.9-beta.0.7+5116ee4b159565\\\"\\nname: \\\"bootstrap-e2e-minion-group-0g7j\\\", osImage: \\\"Container-Optimized OS from Google\\\", kubeletVersion: \\\"v1.16.9-beta.0.7+5116ee4b159565\\\"\\nname: \\\"bootstrap-e2e-minion-group-2873\\\", osImage: \\\"Container-Optimized OS from Google\\\", kubeletVersion: \\\"v1.16.9-beta.0.7+5116ee4b159565\\\"\\nname: \\\"bootstrap-e2e-minion-group-v8hd\\\", osImage: \\\"Container-Optimized OS from Google\\\", kubeletVersion: \\\"v1.16.9-beta.0.7+5116ee4b159565\\\"\\nFound subnet for region us-west1 in network bootstrap-e2e: bootstrap-e2e\\n== Upgrading master to 'https://storage.googleapis.com/kubernetes-release-dev/ci/v1.17.5-beta.0.11+4dfcd1cc87879f/kubernetes-server-linux-amd64.tar.gz'. Do not interrupt, deleting master instance. ==\\n== Upgrading master environment variables. ==\\n== Waiting for new master to respond to API requests ==\\n......................== Done ==\\nWaiting for CoreDNS to update\\nFetching the latest installed CoreDNS version\\n== Downloading the CoreDNS migration tool ==\\n== Upgrading the CoreDNS ConfigMap ==\\nconfigmap/coredns configured\\n== The CoreDNS Config has been updated ==\\n== Validating cluster post-upgrade ==\\nValidating gce cluster, MULTIZONE=\\nFound 4 node(s).\\nNAME                              STATUS                     ROLES    AGE     VERSION\\nbootstrap-e2e-master              Ready,SchedulingDisabled   <none>   9m38s   v1.17.5-beta.0.11+4dfcd1cc87879f\\nbootstrap-e2e-minion-group-0g7j   Ready                      <none>   9m32s   v1.16.9-beta.0.7+5116ee4b159565\\nbootstrap-e2e-minion-group-2873   Ready                      <none>   9m41s   v1.16.9-beta.0.7+5116ee4b159565\\nbootstrap-e2e-minion-group-v8hd   Ready                      <none>   9m40s   v1.16.9-beta.0.7+5116ee4b159565\\nValidate output:\\nNAME                 STATUS    MESSAGE             ERROR\\ncontroller-manager   Healthy   ok                  \\netcd-1               Healthy   {\\\"health\\\":\\\"true\\\"}   \\nscheduler            Healthy   ok                  \\netcd-0               Healthy   {\\\"health\\\":\\\"true\\\"}   \\n\\x1b[0;32mCluster validation succeeded\\x1b[0m\\n== Post-Upgrade Node OS and Kubelet Versions ==\\nname: \\\"bootstrap-e2e-master\\\", osImage: \\\"Container-Optimized OS from Google\\\", kubeletVersion: \\\"v1.17.5-beta.0.11+4dfcd1cc87879f\\\"\\nname: \\\"bootstrap-e2e-minion-group-0g7j\\\", osImage: \\\"Container-Optimized OS from Google\\\", kubeletVersion: \\\"v1.16.9-beta.0.7+5116ee4b159565\\\"\\nname: \\\"bootstrap-e2e-minion-group-2873\\\", osImage: \\\"Container-Optimized OS from Google\\\", kubeletVersion: \\\"v1.16.9-beta.0.7+5116ee4b159565\\\"\\nname: \\\"bootstrap-e2e-minion-group-v8hd\\\", osImage: \\\"Container-Optimized OS from Google\\\", kubeletVersion: \\\"v1.16.9-beta.0.7+5116ee4b159565\\\"\\n\", stderr \"Project: kubernetes-gci-ingress-1-3\\nNetwork Project: kubernetes-gci-ingress-1-3\\nZone: us-west1-b\\nINSTANCE_GROUPS=bootstrap-e2e-minion-group\\nNODE_NAMES=bootstrap-e2e-minion-group-0g7j bootstrap-e2e-minion-group-2873 bootstrap-e2e-minion-group-v8hd\\nTrying to find master named 'bootstrap-e2e-master'\\nLooking for address 'bootstrap-e2e-master-ip'\\nUsing master: bootstrap-e2e-master (external IP: 34.83.213.100; internal IP: (not set))\\nDeleted [https://www.googleapis.com/compute/v1/projects/kubernetes-gci-ingress-1-3/zones/us-west1-b/instances/bootstrap-e2e-master].\\nWARNING: You have selected a disk size of under [200GB]. This may result in poor I/O performance. For more information, see: https://developers.google.com/compute/docs/disks#performance.\\nCreated [https://www.googleapis.com/compute/v1/projects/kubernetes-gci-ingress-1-3/zones/us-west1-b/instances/bootstrap-e2e-master].\\nWARNING: Some requests generated warnings:\\n - Disk size: '20 GB' is larger than image size: '10 GB'. You might need to resize the root repartition manually if the operating system does not support automatic resizing. See https://cloud.google.com/compute/docs/disks/add-persistent-disk#resize_pd for details.\\n - The resource 'projects/cos-cloud/global/images/cos-77-12371-175-0' is deprecated. A suggested replacement is 'projects/cos-cloud/global/images/cos-77-12371-183-0'.\\n\\nNAME                  ZONE        MACHINE_TYPE   PREEMPTIBLE  INTERNAL_IP  EXTERNAL_IP    STATUS\\nbootstrap-e2e-master  us-west1-b  n1-standard-1               10.138.0.6   34.83.213.100  RUNNING\\nWarning: kubectl apply should be used on resource created by either kubectl create --save-config or kubectl apply\\nProject: kubernetes-gci-ingress-1-3\\nNetwork Project: kubernetes-gci-ingress-1-3\\nZone: us-west1-b\\n/workspace/kubernetes_skew/cluster/gce/upgrade.sh: line 452: download_dir: unbound variable\\n\"",
    }
    error running /workspace/kubernetes_skew/cluster/gce/upgrade.sh [-M v1.17.5-beta.0.11+4dfcd1cc87879f]; got error exit status 1, stdout "Fetching the previously installed CoreDNS version\n\n***WARNING***\nUpgrading Kubernetes with this script might result in an upgrade to a new etcd version.\nSome etcd version upgrades, such as 3.0.x to 3.1.x, DO NOT offer a downgrade path.\nTo pin the etcd version to your current one (e.g. v3.0.17), set the following variables\nbefore running this script:\n\n# example: pin to etcd v3.0.17\nexport ETCD_IMAGE=3.0.17\nexport ETCD_VERSION=3.0.17\n\nAlternatively, if you choose to allow an etcd upgrade that doesn't support downgrade,\nyou might still be able to downgrade Kubernetes by pinning to the newer etcd version.\nIn all cases, it is strongly recommended to have an etcd backup before upgrading.\n\n== Pre-Upgrade Node OS and Kubelet Versions ==\nname: \"bootstrap-e2e-master\", osImage: \"Container-Optimized OS from Google\", kubeletVersion: \"v1.16.9-beta.0.7+5116ee4b159565\"\nname: \"bootstrap-e2e-minion-group-0g7j\", osImage: \"Container-Optimized OS from Google\", kubeletVersion: \"v1.16.9-beta.0.7+5116ee4b159565\"\nname: \"bootstrap-e2e-minion-group-2873\", osImage: \"Container-Optimized OS from Google\", kubeletVersion: \"v1.16.9-beta.0.7+5116ee4b159565\"\nname: \"bootstrap-e2e-minion-group-v8hd\", osImage: \"Container-Optimized OS from Google\", kubeletVersion: \"v1.16.9-beta.0.7+5116ee4b159565\"\nFound subnet for region us-west1 in network bootstrap-e2e: bootstrap-e2e\n== Upgrading master to 'https://storage.googleapis.com/kubernetes-release-dev/ci/v1.17.5-beta.0.11+4dfcd1cc87879f/kubernetes-server-linux-amd64.tar.gz'. Do not interrupt, deleting master instance. ==\n== Upgrading master environment variables. ==\n== Waiting for new master to respond to API requests ==\n......................== Done ==\nWaiting for CoreDNS to update\nFetching the latest installed CoreDNS version\n== Downloading the CoreDNS migration tool ==\n== Upgrading the CoreDNS ConfigMap ==\nconfigmap/coredns configured\n== The CoreDNS Config has been updated ==\n== Validating cluster post-upgrade ==\nValidating gce cluster, MULTIZONE=\nFound 4 node(s).\nNAME                              STATUS                     ROLES    AGE     VERSION\nbootstrap-e2e-master              Ready,SchedulingDisabled   <none>   9m38s   v1.17.5-beta.0.11+4dfcd1cc87879f\nbootstrap-e2e-minion-group-0g7j   Ready                      <none>   9m32s   v1.16.9-beta.0.7+5116ee4b159565\nbootstrap-e2e-minion-group-2873   Ready                      <none>   9m41s   v1.16.9-beta.0.7+5116ee4b159565\nbootstrap-e2e-minion-group-v8hd   Ready                      <none>   9m40s   v1.16.9-beta.0.7+5116ee4b159565\nValidate output:\nNAME                 STATUS    MESSAGE             ERROR\ncontroller-manager   Healthy   ok                  \netcd-1               Healthy   {\"health\":\"true\"}   \nscheduler            Healthy   ok                  \netcd-0               Healthy   {\"health\":\"true\"}   \n\x1b[0;32mCluster validation succeeded\x1b[0m\n== Post-Upgrade Node OS and Kubelet Versions ==\nname: \"bootstrap-e2e-master\", osImage: \"Container-Optimized OS from Google\", kubeletVersion: \"v1.17.5-beta.0.11+4dfcd1cc87879f\"\nname: \"bootstrap-e2e-minion-group-0g7j\", osImage: \"Container-Optimized OS from Google\", kubeletVersion: \"v1.16.9-beta.0.7+5116ee4b159565\"\nname: \"bootstrap-e2e-minion-group-2873\", osImage: \"Container-Optimized OS from Google\", kubeletVersion: \"v1.16.9-beta.0.7+5116ee4b159565\"\nname: \"bootstrap-e2e-minion-group-v8hd\", osImage: \"Container-Optimized OS from Google\", kubeletVersion: \"v1.16.9-beta.0.7+5116ee4b159565\"\n", stderr "Project: kubernetes-gci-ingress-1-3\nNetwork Project: kubernetes-gci-ingress-1-3\nZone: us-west1-b\nINSTANCE_GROUPS=bootstrap-e2e-minion-group\nNODE_NAMES=bootstrap-e2e-minion-group-0g7j bootstrap-e2e-minion-group-2873 bootstrap-e2e-minion-group-v8hd\nTrying to find master named 'bootstrap-e2e-master'\nLooking for address 'bootstrap-e2e-master-ip'\nUsing master: bootstrap-e2e-master (external IP: 34.83.213.100; internal IP: (not set))\nDeleted [https://www.googleapis.com/compute/v1/projects/kubernetes-gci-ingress-1-3/zones/us-west1-b/instances/bootstrap-e2e-master].\nWARNING: You have selected a disk size of under [200GB]. This may result in poor I/O performance. For more information, see: https://developers.google.com/compute/docs/disks#performance.\nCreated [https://www.googleapis.com/compute/v1/projects/kubernetes-gci-ingress-1-3/zones/us-west1-b/instances/bootstrap-e2e-master].\nWARNING: Some requests generated warnings:\n - Disk size: '20 GB' is larger than image size: '10 GB'. You might need to resize the root repartition manually if the operating system does not support automatic resizing. See https://cloud.google.com/compute/docs/disks/add-persistent-disk#resize_pd for details.\n - The resource 'projects/cos-cloud/global/images/cos-77-12371-175-0' is deprecated. A suggested replacement is 'projects/cos-cloud/global/images/cos-77-12371-183-0'.\n\nNAME                  ZONE        MACHINE_TYPE   PREEMPTIBLE  INTERNAL_IP  EXTERNAL_IP    STATUS\nbootstrap-e2e-master  us-west1-b  n1-standard-1               10.138.0.6   34.83.213.100  RUNNING\nWarning: kubectl apply should be used on resource created by either kubectl create --save-config or kubectl apply\nProject: kubernetes-gci-ingress-1-3\nNetwork Project: kubernetes-gci-ingress-1-3\nZone: us-west1-b\n/workspace/kubernetes_skew/cluster/gce/upgrade.sh: line 452: download_dir: unbound variable\n"
occurred

k8s.io/kubernetes/test/e2e/cloud/gcp.glob..func2.1.1.1()
	/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/cloud/gcp/cluster_upgrade.go:106 +0x12c
k8s.io/kubernetes/test/e2e/chaosmonkey.(*Chaosmonkey).Do(0xc000f61200)
	/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/chaosmonkey/chaosmonkey.go:111 +0x38a
k8s.io/kubernetes/test/e2e/cloud/gcp.runUpgradeSuite(0xc000ca4a00, 0x7757ca0, 0xc, 0xc, 0xc000679410, 0xc0036161e0, 0x0, 0xc0035fca60)
	/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/cloud/gcp/cluster_upgrade.go:479 +0x47a
k8s.io/kubernetes/test/e2e/cloud/gcp.glob..func2.1.1()
	/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/cloud/gcp/cluster_upgrade.go:109 +0x222
k8s.io/kubernetes/test/e2e.RunE2ETests(0xc00190e800)
	_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/e2e.go:110 +0x30a
k8s.io/kubernetes/test/e2e.TestE2E(0xc00190e800)
	_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/e2e_test.go:112 +0x2b
testing.tRunner(0xc00190e800, 0x49f9638)
	/usr/local/go/src/testing/testing.go:909 +0xc9
created by testing.(*T).Run
	/usr/local/go/src/testing/testing.go:960 +0x350
				from junit_upgradeupgrades.xml

Filter through log files | View test history on testgrid


Test 14h34m

error during ./hack/ginkgo-e2e.sh --kubectl-path=../../../../kubernetes_skew/cluster/kubectl.sh --minStartupPods=8 --report-dir=/workspace/_artifacts --disable-log-dump=true (interrupted): exit status 1
				from junit_runner.xml

Filter through log files | View test history on testgrid


Timeout 15h0m

kubetest --timeout triggered
				from junit_runner.xml

Filter through log files | View test history on testgrid


UpgradeTest 18m39s

error during kubetest --test --test_args=--ginkgo.focus=\[Feature:MasterUpgrade\] --upgrade-target=ci/k8s-stable1 --report-dir=/workspace/_artifacts --disable-log-dump=true --report-prefix=upgrade --check-version-skew=false: exit status 1
				from junit_runner.xml

Filter through log files | View test history on testgrid


Show 925 Passed Tests