This job view page is being replaced by Spyglass soon. Check out the new job view.
ResultFAILURE
Tests 7 failed / 244 succeeded
Started2021-10-21 15:08
Elapsed6h22m
Revision
Builderae434d76-3280-11ec-9f2a-5a371f369f05
control_plane_node_os_imagecos-85-13310-1308-1
infra-commitef51bf0ff
job-versionv1.21.6-rc.0.60+0fb71846df9bab
kubetest-version
revisionv1.21.6-rc.0.60+0fb71846df9bab
worker_node_os_imagecos-85-13310-1308-1

Test Failures


Cluster upgrade [sig-cloud-provider-gcp] cluster-upgrade 19m43s

go run hack/e2e.go -v --test --test_args='--ginkgo.focus=Cluster\supgrade\s\[sig\-cloud\-provider\-gcp\]\scluster\-upgrade$'
Oct 21 15:38:19.093: Unexpected error:
    <*errors.errorString | 0xc004670940>: {
        s: "error running /workspace/kubernetes_skew/cluster/gce/upgrade.sh [-N -o v1.23.0-alpha.3.464+2dede1d4d45341]; got error exit status 8, stdout \"Fetching the previously installed CoreDNS version\\n== Pre-Upgrade Node OS and Kubelet Versions ==\\nname: \\\"bootstrap-e2e-master\\\", osImage: \\\"Container-Optimized OS from Google\\\", kubeletVersion: \\\"v1.23.0-alpha.3.464+2dede1d4d45341\\\"\\nname: \\\"bootstrap-e2e-minion-group-1x5k\\\", osImage: \\\"Container-Optimized OS from Google\\\", kubeletVersion: \\\"v1.21.6-rc.0.60+0fb71846df9bab\\\"\\nname: \\\"bootstrap-e2e-minion-group-2gn9\\\", osImage: \\\"Container-Optimized OS from Google\\\", kubeletVersion: \\\"v1.21.6-rc.0.60+0fb71846df9bab\\\"\\nname: \\\"bootstrap-e2e-minion-group-gt7d\\\", osImage: \\\"Container-Optimized OS from Google\\\", kubeletVersion: \\\"v1.21.6-rc.0.60+0fb71846df9bab\\\"\\nFound subnet for region us-west1 in network bootstrap-e2e: bootstrap-e2e\\n== Upgrading node environment variables. ==\\nUsing subnet bootstrap-e2e\\nInstance template name: bootstrap-e2e-minion-template-v1-23-0-alpha-3-464-2dede1d4d4534\\nnode/bootstrap-e2e-minion-group-1x5k cordoned\\nevicting pod apparmor-upgrade-2359/apparmor-loader-2lpwx\\nevicting pod sig-apps-job-upgrade-2553/foo-trffq\\nevicting pod apparmor-upgrade-2359/test-apparmor-bxpkt\\nevicting pod apparmor-upgrade-2359/test-apparmor-lwt65\\nevicting pod kube-system/coredns-755cd654d4-hv4mr\\nevicting pod hpa-upgrade-1278/res-cons-upgrade-bldxc\\nevicting pod sig-apps-statefulset-upgrade-4870/ss-2\\nevicting pod hpa-upgrade-1278/res-cons-upgrade-ctrl-hn6pd\\nevicting pod kube-system/event-exporter-v0.3.4-6c59d5574d-7kmpz\\nevicting pod service-upgrade-3087/service-test-6dwxt\\nevicting pod kube-system/metrics-server-v0.5.1-5999f95767-x6lcg\\npod/test-apparmor-bxpkt evicted\\npod/coredns-755cd654d4-hv4mr evicted\\npod/service-test-6dwxt evicted\\npod/res-cons-upgrade-bldxc evicted\\npod/metrics-server-v0.5.1-5999f95767-x6lcg evicted\\npod/apparmor-loader-2lpwx evicted\\npod/res-cons-upgrade-ctrl-hn6pd evicted\\npod/ss-2 evicted\\npod/event-exporter-v0.3.4-6c59d5574d-7kmpz evicted\\npod/foo-trffq evicted\\npod/test-apparmor-lwt65 evicted\\nnode/bootstrap-e2e-minion-group-1x5k evicted\\n.....................................................................................................................................................Node bootstrap-e2e-minion-group-1x5k recreated.\\nNode bootstrap-e2e-minion-group-1x5k Ready=True\\nnode/bootstrap-e2e-minion-group-1x5k uncordoned\\nnode/bootstrap-e2e-minion-group-2gn9 cordoned\\nevicting pod sig-apps-statefulset-upgrade-4870/ss-0\\nevicting pod kube-system/event-exporter-v0.3.4-6c59d5574d-bwzh4\\nevicting pod kube-system/fluentd-gcp-scaler-55f8dfc997-7r75k\\nevicting pod kube-system/kube-dns-autoscaler-6494c4c647-w6m9v\\nevicting pod kube-system/l7-default-backend-79858d8f86-ctflr\\nevicting pod apparmor-upgrade-2359/apparmor-loader-4v5ln\\nevicting pod kube-system/metrics-server-v0.5.1-5999f95767-qkhrc\\nevicting pod hpa-upgrade-1278/res-cons-upgrade-7bpst\\nevicting pod service-upgrade-3087/service-test-zn5rw\\nevicting pod hpa-upgrade-1278/res-cons-upgrade-7h62f\\nevicting pod kube-system/coredns-755cd654d4-5zvxc\\nevicting pod hpa-upgrade-1278/res-cons-upgrade-mg7kv\\nevicting pod sig-apps-job-upgrade-2553/foo-ft2b6\\npod/ss-0 evicted\\npod/res-cons-upgrade-7h62f evicted\\npod/res-cons-upgrade-7bpst evicted\\npod/res-cons-upgrade-mg7kv evicted\\npod/apparmor-loader-4v5ln evicted\\npod/metrics-server-v0.5.1-5999f95767-qkhrc evicted\\npod/service-test-zn5rw evicted\\npod/kube-dns-autoscaler-6494c4c647-w6m9v evicted\\npod/l7-default-backend-79858d8f86-ctflr evicted\\npod/coredns-755cd654d4-5zvxc evicted\\npod/foo-ft2b6 evicted\\npod/fluentd-gcp-scaler-55f8dfc997-7r75k evicted\\npod/event-exporter-v0.3.4-6c59d5574d-bwzh4 evicted\\nnode/bootstrap-e2e-minion-group-2gn9 evicted\\n....................................................................................................................................................Node bootstrap-e2e-minion-group-2gn9 recreated.\\nNode bootstrap-e2e-minion-group-2gn9 Ready=True\\nnode/bootstrap-e2e-minion-group-2gn9 uncordoned\\nnode/bootstrap-e2e-minion-group-gt7d cordoned\\nevicting pod sig-apps-statefulset-upgrade-4870/ss-2\\nevicting pod hpa-upgrade-1278/res-cons-upgrade-ctrl-299sh\\nevicting pod hpa-upgrade-1278/res-cons-upgrade-74zbg\\nevicting pod hpa-upgrade-1278/res-cons-upgrade-7jkzd\\nevicting pod sig-apps-deployment-upgrade-6798/dp-67f7589d48-kqlqh\\nevicting pod hpa-upgrade-1278/res-cons-upgrade-jqxsm\\nevicting pod kube-system/coredns-755cd654d4-7dtq6\\nevicting pod kube-system/volume-snapshot-controller-0\\nevicting pod service-upgrade-3087/service-test-mth6w\\nevicting pod sig-apps-job-upgrade-2553/foo-ljw2g\\nevicting pod sig-apps-job-upgrade-2553/foo-8fhhm\\nevicting pod sig-apps-replicaset-upgrade-884/rs-vkvj5\\nevicting pod sig-apps-statefulset-upgrade-4870/ss-1\\npod/dp-67f7589d48-kqlqh evicted\\npod/rs-vkvj5 evicted\\npod/res-cons-upgrade-ctrl-299sh evicted\\npod/volume-snapshot-controller-0 evicted\\npod/res-cons-upgrade-7jkzd evicted\\npod/ss-1 evicted\\npod/ss-2 evicted\\npod/res-cons-upgrade-74zbg evicted\\npod/coredns-755cd654d4-7dtq6 evicted\\npod/res-cons-upgrade-jqxsm evicted\\npod/service-test-mth6w evicted\\npod/foo-ljw2g evicted\\npod/foo-8fhhm evicted\\nnode/bootstrap-e2e-minion-group-gt7d evicted\\n..............................................................................................................................................Node bootstrap-e2e-minion-group-gt7d recreated.\\nNode bootstrap-e2e-minion-group-gt7d Ready=True\\nnode/bootstrap-e2e-minion-group-gt7d uncordoned\\n== Waiting for CoreDNS to update ==\\n== CoreDNS ResourceVersion changed ==\\n== Fetching the latest installed CoreDNS version ==\\n== Downloading the CoreDNS migration tool ==\\n\", stderr \"Project: k8s-jkns-gci-gce-kubenet\\nNetwork Project: k8s-jkns-gci-gce-kubenet\\nZone: us-west1-b\\nINSTANCE_GROUPS=bootstrap-e2e-minion-group\\nNODE_NAMES=bootstrap-e2e-minion-group-1x5k bootstrap-e2e-minion-group-2gn9 bootstrap-e2e-minion-group-gt7d\\n== Preparing node upgrade (to v1.23.0-alpha.3.464+2dede1d4d45341). ==\\nAttempt 1 to create bootstrap-e2e-minion-template-v1-23-0-alpha-3-464-2dede1d4d4534\\nWARNING: You have selected a disk size of under [200GB]. This may result in poor I/O performance. For more information, see: https://developers.google.com/compute/docs/disks#performance.\\nCreated [https://www.googleapis.com/compute/v1/projects/k8s-jkns-gci-gce-kubenet/global/instanceTemplates/bootstrap-e2e-minion-template-v1-23-0-alpha-3-464-2dede1d4d4534].\\nNAME                                                             MACHINE_TYPE   PREEMPTIBLE  CREATION_TIMESTAMP\\nbootstrap-e2e-minion-template-v1-23-0-alpha-3-464-2dede1d4d4534  n1-standard-2               2021-10-21T08:26:04.089-07:00\\n== Finished preparing node upgrade (to v1.23.0-alpha.3.464+2dede1d4d45341). ==\\n== Upgrading nodes to v1.23.0-alpha.3.464+2dede1d4d45341 with max parallelism of 1. ==\\n== Draining bootstrap-e2e-minion-group-1x5k. == \\nWARNING: deleting Pods not managed by ReplicationController, ReplicaSet, Job, DaemonSet or StatefulSet: apparmor-upgrade-2359/test-apparmor-lwt65; ignoring DaemonSet-managed Pods: kube-system/fluentd-gcp-v3.2.0-flq2q, kube-system/konnectivity-agent-785d6, kube-system/metadata-proxy-v0.1-sslht, sig-apps-daemonset-upgrade-5708/ds1-szbq8\\nI1021 15:26:29.709909   21961 request.go:665] Waited for 1.04853537s due to client-side throttling, not priority and fairness, request: GET:https://34.145.10.33/api/v1/namespaces/sig-apps-statefulset-upgrade-4870/pods/ss-2\\n== Recreating instance bootstrap-e2e-minion-group-1x5k. ==\\n== Waiting for new node to be added to k8s.  ==\\n== Waiting for bootstrap-e2e-minion-group-1x5k to become ready. ==\\n== Uncordon bootstrap-e2e-minion-group-1x5k. == \\n== Draining bootstrap-e2e-minion-group-2gn9. == \\nWARNING: ignoring DaemonSet-managed Pods: kube-system/fluentd-gcp-v3.2.0-nplcc, kube-system/konnectivity-agent-smnhx, kube-system/metadata-proxy-v0.1-hwgzx, sig-apps-daemonset-upgrade-5708/ds1-jfzcf\\nI1021 15:30:31.136956   30268 request.go:665] Waited for 1.055579883s due to client-side throttling, not priority and fairness, request: GET:https://34.145.10.33/api/v1/namespaces/service-upgrade-3087/pods/service-test-zn5rw\\n== Recreating instance bootstrap-e2e-minion-group-2gn9. ==\\n== Waiting for new node to be added to k8s.  ==\\n== Waiting for bootstrap-e2e-minion-group-2gn9 to become ready. ==\\n== Uncordon bootstrap-e2e-minion-group-2gn9. == \\n== Draining bootstrap-e2e-minion-group-gt7d. == \\nWARNING: ignoring DaemonSet-managed Pods: kube-system/fluentd-gcp-v3.2.0-xt6vd, kube-system/konnectivity-agent-mmhgw, kube-system/metadata-proxy-v0.1-7b5cl, sig-apps-daemonset-upgrade-5708/ds1-mjg5r\\nI1021 15:34:30.706894   38523 request.go:665] Waited for 1.060197998s due to client-side throttling, not priority and fairness, request: GET:https://34.145.10.33/api/v1/namespaces/sig-apps-job-upgrade-2553/pods/foo-8fhhm\\n== Recreating instance bootstrap-e2e-minion-group-gt7d. ==\\n== Waiting for new node to be added to k8s.  ==\\n== Waiting for bootstrap-e2e-minion-group-gt7d to become ready. ==\\n== Uncordon bootstrap-e2e-minion-group-gt7d. == \\n== Deleting old templates in k8s-jkns-gci-gce-kubenet. ==\\nDeleted [https://www.googleapis.com/compute/v1/projects/k8s-jkns-gci-gce-kubenet/global/instanceTemplates/bootstrap-e2e-minion-template].\\n== Finished upgrading nodes to v1.23.0-alpha.3.464+2dede1d4d45341. ==\\n\"",
    }
    error running /workspace/kubernetes_skew/cluster/gce/upgrade.sh [-N -o v1.23.0-alpha.3.464+2dede1d4d45341]; got error exit status 8, stdout "Fetching the previously installed CoreDNS version\n== Pre-Upgrade Node OS and Kubelet Versions ==\nname: \"bootstrap-e2e-master\", osImage: \"Container-Optimized OS from Google\", kubeletVersion: \"v1.23.0-alpha.3.464+2dede1d4d45341\"\nname: \"bootstrap-e2e-minion-group-1x5k\", osImage: \"Container-Optimized OS from Google\", kubeletVersion: \"v1.21.6-rc.0.60+0fb71846df9bab\"\nname: \"bootstrap-e2e-minion-group-2gn9\", osImage: \"Container-Optimized OS from Google\", kubeletVersion: \"v1.21.6-rc.0.60+0fb71846df9bab\"\nname: \"bootstrap-e2e-minion-group-gt7d\", osImage: \"Container-Optimized OS from Google\", kubeletVersion: \"v1.21.6-rc.0.60+0fb71846df9bab\"\nFound subnet for region us-west1 in network bootstrap-e2e: bootstrap-e2e\n== Upgrading node environment variables. ==\nUsing subnet bootstrap-e2e\nInstance template name: bootstrap-e2e-minion-template-v1-23-0-alpha-3-464-2dede1d4d4534\nnode/bootstrap-e2e-minion-group-1x5k cordoned\nevicting pod apparmor-upgrade-2359/apparmor-loader-2lpwx\nevicting pod sig-apps-job-upgrade-2553/foo-trffq\nevicting pod apparmor-upgrade-2359/test-apparmor-bxpkt\nevicting pod apparmor-upgrade-2359/test-apparmor-lwt65\nevicting pod kube-system/coredns-755cd654d4-hv4mr\nevicting pod hpa-upgrade-1278/res-cons-upgrade-bldxc\nevicting pod sig-apps-statefulset-upgrade-4870/ss-2\nevicting pod hpa-upgrade-1278/res-cons-upgrade-ctrl-hn6pd\nevicting pod kube-system/event-exporter-v0.3.4-6c59d5574d-7kmpz\nevicting pod service-upgrade-3087/service-test-6dwxt\nevicting pod kube-system/metrics-server-v0.5.1-5999f95767-x6lcg\npod/test-apparmor-bxpkt evicted\npod/coredns-755cd654d4-hv4mr evicted\npod/service-test-6dwxt evicted\npod/res-cons-upgrade-bldxc evicted\npod/metrics-server-v0.5.1-5999f95767-x6lcg evicted\npod/apparmor-loader-2lpwx evicted\npod/res-cons-upgrade-ctrl-hn6pd evicted\npod/ss-2 evicted\npod/event-exporter-v0.3.4-6c59d5574d-7kmpz evicted\npod/foo-trffq evicted\npod/test-apparmor-lwt65 evicted\nnode/bootstrap-e2e-minion-group-1x5k evicted\n.....................................................................................................................................................Node bootstrap-e2e-minion-group-1x5k recreated.\nNode bootstrap-e2e-minion-group-1x5k Ready=True\nnode/bootstrap-e2e-minion-group-1x5k uncordoned\nnode/bootstrap-e2e-minion-group-2gn9 cordoned\nevicting pod sig-apps-statefulset-upgrade-4870/ss-0\nevicting pod kube-system/event-exporter-v0.3.4-6c59d5574d-bwzh4\nevicting pod kube-system/fluentd-gcp-scaler-55f8dfc997-7r75k\nevicting pod kube-system/kube-dns-autoscaler-6494c4c647-w6m9v\nevicting pod kube-system/l7-default-backend-79858d8f86-ctflr\nevicting pod apparmor-upgrade-2359/apparmor-loader-4v5ln\nevicting pod kube-system/metrics-server-v0.5.1-5999f95767-qkhrc\nevicting pod hpa-upgrade-1278/res-cons-upgrade-7bpst\nevicting pod service-upgrade-3087/service-test-zn5rw\nevicting pod hpa-upgrade-1278/res-cons-upgrade-7h62f\nevicting pod kube-system/coredns-755cd654d4-5zvxc\nevicting pod hpa-upgrade-1278/res-cons-upgrade-mg7kv\nevicting pod sig-apps-job-upgrade-2553/foo-ft2b6\npod/ss-0 evicted\npod/res-cons-upgrade-7h62f evicted\npod/res-cons-upgrade-7bpst evicted\npod/res-cons-upgrade-mg7kv evicted\npod/apparmor-loader-4v5ln evicted\npod/metrics-server-v0.5.1-5999f95767-qkhrc evicted\npod/service-test-zn5rw evicted\npod/kube-dns-autoscaler-6494c4c647-w6m9v evicted\npod/l7-default-backend-79858d8f86-ctflr evicted\npod/coredns-755cd654d4-5zvxc evicted\npod/foo-ft2b6 evicted\npod/fluentd-gcp-scaler-55f8dfc997-7r75k evicted\npod/event-exporter-v0.3.4-6c59d5574d-bwzh4 evicted\nnode/bootstrap-e2e-minion-group-2gn9 evicted\n....................................................................................................................................................Node bootstrap-e2e-minion-group-2gn9 recreated.\nNode bootstrap-e2e-minion-group-2gn9 Ready=True\nnode/bootstrap-e2e-minion-group-2gn9 uncordoned\nnode/bootstrap-e2e-minion-group-gt7d cordoned\nevicting pod sig-apps-statefulset-upgrade-4870/ss-2\nevicting pod hpa-upgrade-1278/res-cons-upgrade-ctrl-299sh\nevicting pod hpa-upgrade-1278/res-cons-upgrade-74zbg\nevicting pod hpa-upgrade-1278/res-cons-upgrade-7jkzd\nevicting pod sig-apps-deployment-upgrade-6798/dp-67f7589d48-kqlqh\nevicting pod hpa-upgrade-1278/res-cons-upgrade-jqxsm\nevicting pod kube-system/coredns-755cd654d4-7dtq6\nevicting pod kube-system/volume-snapshot-controller-0\nevicting pod service-upgrade-3087/service-test-mth6w\nevicting pod sig-apps-job-upgrade-2553/foo-ljw2g\nevicting pod sig-apps-job-upgrade-2553/foo-8fhhm\nevicting pod sig-apps-replicaset-upgrade-884/rs-vkvj5\nevicting pod sig-apps-statefulset-upgrade-4870/ss-1\npod/dp-67f7589d48-kqlqh evicted\npod/rs-vkvj5 evicted\npod/res-cons-upgrade-ctrl-299sh evicted\npod/volume-snapshot-controller-0 evicted\npod/res-cons-upgrade-7jkzd evicted\npod/ss-1 evicted\npod/ss-2 evicted\npod/res-cons-upgrade-74zbg evicted\npod/coredns-755cd654d4-7dtq6 evicted\npod/res-cons-upgrade-jqxsm evicted\npod/service-test-mth6w evicted\npod/foo-ljw2g evicted\npod/foo-8fhhm evicted\nnode/bootstrap-e2e-minion-group-gt7d evicted\n..............................................................................................................................................Node bootstrap-e2e-minion-group-gt7d recreated.\nNode bootstrap-e2e-minion-group-gt7d Ready=True\nnode/bootstrap-e2e-minion-group-gt7d uncordoned\n== Waiting for CoreDNS to update ==\n== CoreDNS ResourceVersion changed ==\n== Fetching the latest installed CoreDNS version ==\n== Downloading the CoreDNS migration tool ==\n", stderr "Project: k8s-jkns-gci-gce-kubenet\nNetwork Project: k8s-jkns-gci-gce-kubenet\nZone: us-west1-b\nINSTANCE_GROUPS=bootstrap-e2e-minion-group\nNODE_NAMES=bootstrap-e2e-minion-group-1x5k bootstrap-e2e-minion-group-2gn9 bootstrap-e2e-minion-group-gt7d\n== Preparing node upgrade (to v1.23.0-alpha.3.464+2dede1d4d45341). ==\nAttempt 1 to create bootstrap-e2e-minion-template-v1-23-0-alpha-3-464-2dede1d4d4534\nWARNING: You have selected a disk size of under [200GB]. This may result in poor I/O performance. For more information, see: https://developers.google.com/compute/docs/disks#performance.\nCreated [https://www.googleapis.com/compute/v1/projects/k8s-jkns-gci-gce-kubenet/global/instanceTemplates/bootstrap-e2e-minion-template-v1-23-0-alpha-3-464-2dede1d4d4534].\nNAME                                                             MACHINE_TYPE   PREEMPTIBLE  CREATION_TIMESTAMP\nbootstrap-e2e-minion-template-v1-23-0-alpha-3-464-2dede1d4d4534  n1-standard-2               2021-10-21T08:26:04.089-07:00\n== Finished preparing node upgrade (to v1.23.0-alpha.3.464+2dede1d4d45341). ==\n== Upgrading nodes to v1.23.0-alpha.3.464+2dede1d4d45341 with max parallelism of 1. ==\n== Draining bootstrap-e2e-minion-group-1x5k. == \nWARNING: deleting Pods not managed by ReplicationController, ReplicaSet, Job, DaemonSet or StatefulSet: apparmor-upgrade-2359/test-apparmor-lwt65; ignoring DaemonSet-managed Pods: kube-system/fluentd-gcp-v3.2.0-flq2q, kube-system/konnectivity-agent-785d6, kube-system/metadata-proxy-v0.1-sslht, sig-apps-daemonset-upgrade-5708/ds1-szbq8\nI1021 15:26:29.709909   21961 request.go:665] Waited for 1.04853537s due to client-side throttling, not priority and fairness, request: GET:https://34.145.10.33/api/v1/namespaces/sig-apps-statefulset-upgrade-4870/pods/ss-2\n== Recreating instance bootstrap-e2e-minion-group-1x5k. ==\n== Waiting for new node to be added to k8s.  ==\n== Waiting for bootstrap-e2e-minion-group-1x5k to become ready. ==\n== Uncordon bootstrap-e2e-minion-group-1x5k. == \n== Draining bootstrap-e2e-minion-group-2gn9. == \nWARNING: ignoring DaemonSet-managed Pods: kube-system/fluentd-gcp-v3.2.0-nplcc, kube-system/konnectivity-agent-smnhx, kube-system/metadata-proxy-v0.1-hwgzx, sig-apps-daemonset-upgrade-5708/ds1-jfzcf\nI1021 15:30:31.136956   30268 request.go:665] Waited for 1.055579883s due to client-side throttling, not priority and fairness, request: GET:https://34.145.10.33/api/v1/namespaces/service-upgrade-3087/pods/service-test-zn5rw\n== Recreating instance bootstrap-e2e-minion-group-2gn9. ==\n== Waiting for new node to be added to k8s.  ==\n== Waiting for bootstrap-e2e-minion-group-2gn9 to become ready. ==\n== Uncordon bootstrap-e2e-minion-group-2gn9. == \n== Draining bootstrap-e2e-minion-group-gt7d. == \nWARNING: ignoring DaemonSet-managed Pods: kube-system/fluentd-gcp-v3.2.0-xt6vd, kube-system/konnectivity-agent-mmhgw, kube-system/metadata-proxy-v0.1-7b5cl, sig-apps-daemonset-upgrade-5708/ds1-mjg5r\nI1021 15:34:30.706894   38523 request.go:665] Waited for 1.060197998s due to client-side throttling, not priority and fairness, request: GET:https://34.145.10.33/api/v1/namespaces/sig-apps-job-upgrade-2553/pods/foo-8fhhm\n== Recreating instance bootstrap-e2e-minion-group-gt7d. ==\n== Waiting for new node to be added to k8s.  ==\n== Waiting for bootstrap-e2e-minion-group-gt7d to become ready. ==\n== Uncordon bootstrap-e2e-minion-group-gt7d. == \n== Deleting old templates in k8s-jkns-gci-gce-kubenet. ==\nDeleted [https://www.googleapis.com/compute/v1/projects/k8s-jkns-gci-gce-kubenet/global/instanceTemplates/bootstrap-e2e-minion-template].\n== Finished upgrading nodes to v1.23.0-alpha.3.464+2dede1d4d45341. ==\n"
occurred

k8s.io/kubernetes/test/e2e/cloud/gcp/common.ClusterUpgradeFunc.func1()
	/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/cloud/gcp/common/upgrade_mechanics.go:55 +0x23c
k8s.io/kubernetes/test/e2e/chaosmonkey.(*Chaosmonkey).Do(0xc004dcd0b8)
	/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/chaosmonkey/chaosmonkey.go:111 +0x35b
k8s.io/kubernetes/test/e2e/upgrades.RunUpgradeSuite(0xc003cf0198, {0x9eacce0, 0xc, 0x0}, 0x0, 0xc003e4c000, 0x2, 0x0)
	/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/upgrades/upgrade_suite.go:149 +0x12e
k8s.io/kubernetes/test/e2e/cloud/gcp.glob..func2.2.1()
	/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/cloud/gcp/cluster_upgrade.go:84 +0x199
k8s.io/kubernetes/test/e2e.RunE2ETests(0x2299bd7)
	_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/e2e.go:131 +0x697
k8s.io/kubernetes/test/e2e.TestE2E(0x2534c4b)
	_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/e2e_test.go:136 +0x19
testing.tRunner(0xc000c6a820, 0x6d369f8)
	/usr/local/go/src/testing/testing.go:1259 +0x102
created by testing.(*T).Run
	/usr/local/go/src/testing/testing.go:1306 +0x35a
				from junit_upgradeupgrades.xml

Find apparmor-upgrade-2359/apparmor-loader-2lpwxnevicting mentions in log files | View test history on testgrid


Cluster upgrade hpa-upgrade 37m44s

go run hack/e2e.go -v --test --test_args='--ginkgo.focus=Cluster\supgrade\shpa\-upgrade$'
Oct 21 15:53:19.422: timeout waiting 15m0s for 1 replicas
Unexpected error:
    <*errors.errorString | 0xc0002be220>: {
        s: "timed out waiting for the condition",
    }
    timed out waiting for the condition
occurred

k8s.io/kubernetes/test/e2e/upgrades/autoscaling.(*HPAUpgradeTest).test(0xa072530)
	/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/upgrades/autoscaling/horizontal_pod_autoscalers.go:88 +0x15b
k8s.io/kubernetes/test/e2e/upgrades/autoscaling.(*HPAUpgradeTest).Test(0xa072530, 0x7364b78, 0xa072530, 0x0)
	/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/upgrades/autoscaling/horizontal_pod_autoscalers.go:71 +0x5b
k8s.io/kubernetes/test/e2e/upgrades.(*chaosMonkeyAdapter).Test(0xc001fdd940, 0xc0015bac48)
	/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/upgrades/upgrade_suite.go:65 +0x2ee
k8s.io/kubernetes/test/e2e/chaosmonkey.(*Chaosmonkey).Do.func1()
	/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/chaosmonkey/chaosmonkey.go:90 +0x6a
created by k8s.io/kubernetes/test/e2e/chaosmonkey.(*Chaosmonkey).Do
	/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/chaosmonkey/chaosmonkey.go:87 +0x8c
				from junit_upgradeupgrades.xml

Filter through log files | View test history on testgrid


Kubernetes e2e suite [sig-cloud-provider-gcp] Upgrade [Feature:Upgrade] cluster upgrade should maintain a functioning cluster [Feature:ClusterUpgrade] 38m12s

go run hack/e2e.go -v --test --test_args='--ginkgo.focus=Kubernetes\se2e\ssuite\s\[sig\-cloud\-provider\-gcp\]\sUpgrade\s\[Feature\:Upgrade\]\scluster\supgrade\sshould\smaintain\sa\sfunctioning\scluster\s\[Feature\:ClusterUpgrade\]$'
/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/cloud/gcp/cluster_upgrade.go:75
Oct 21 15:38:19.093: Unexpected error:
    <*errors.errorString | 0xc004670940>: {
        s: "error running /workspace/kubernetes_skew/cluster/gce/upgrade.sh [-N -o v1.23.0-alpha.3.464+2dede1d4d45341]; got error exit status 8, stdout \"Fetching the previously installed CoreDNS version\\n== Pre-Upgrade Node OS and Kubelet Versions ==\\nname: \\\"bootstrap-e2e-master\\\", osImage: \\\"Container-Optimized OS from Google\\\", kubeletVersion: \\\"v1.23.0-alpha.3.464+2dede1d4d45341\\\"\\nname: \\\"bootstrap-e2e-minion-group-1x5k\\\", osImage: \\\"Container-Optimized OS from Google\\\", kubeletVersion: \\\"v1.21.6-rc.0.60+0fb71846df9bab\\\"\\nname: \\\"bootstrap-e2e-minion-group-2gn9\\\", osImage: \\\"Container-Optimized OS from Google\\\", kubeletVersion: \\\"v1.21.6-rc.0.60+0fb71846df9bab\\\"\\nname: \\\"bootstrap-e2e-minion-group-gt7d\\\", osImage: \\\"Container-Optimized OS from Google\\\", kubeletVersion: \\\"v1.21.6-rc.0.60+0fb71846df9bab\\\"\\nFound subnet for region us-west1 in network bootstrap-e2e: bootstrap-e2e\\n== Upgrading node environment variables. ==\\nUsing subnet bootstrap-e2e\\nInstance template name: bootstrap-e2e-minion-template-v1-23-0-alpha-3-464-2dede1d4d4534\\nnode/bootstrap-e2e-minion-group-1x5k cordoned\\nevicting pod apparmor-upgrade-2359/apparmor-loader-2lpwx\\nevicting pod sig-apps-job-upgrade-2553/foo-trffq\\nevicting pod apparmor-upgrade-2359/test-apparmor-bxpkt\\nevicting pod apparmor-upgrade-2359/test-apparmor-lwt65\\nevicting pod kube-system/coredns-755cd654d4-hv4mr\\nevicting pod hpa-upgrade-1278/res-cons-upgrade-bldxc\\nevicting pod sig-apps-statefulset-upgrade-4870/ss-2\\nevicting pod hpa-upgrade-1278/res-cons-upgrade-ctrl-hn6pd\\nevicting pod kube-system/event-exporter-v0.3.4-6c59d5574d-7kmpz\\nevicting pod service-upgrade-3087/service-test-6dwxt\\nevicting pod kube-system/metrics-server-v0.5.1-5999f95767-x6lcg\\npod/test-apparmor-bxpkt evicted\\npod/coredns-755cd654d4-hv4mr evicted\\npod/service-test-6dwxt evicted\\npod/res-cons-upgrade-bldxc evicted\\npod/metrics-server-v0.5.1-5999f95767-x6lcg evicted\\npod/apparmor-loader-2lpwx evicted\\npod/res-cons-upgrade-ctrl-hn6pd evicted\\npod/ss-2 evicted\\npod/event-exporter-v0.3.4-6c59d5574d-7kmpz evicted\\npod/foo-trffq evicted\\npod/test-apparmor-lwt65 evicted\\nnode/bootstrap-e2e-minion-group-1x5k evicted\\n.....................................................................................................................................................Node bootstrap-e2e-minion-group-1x5k recreated.\\nNode bootstrap-e2e-minion-group-1x5k Ready=True\\nnode/bootstrap-e2e-minion-group-1x5k uncordoned\\nnode/bootstrap-e2e-minion-group-2gn9 cordoned\\nevicting pod sig-apps-statefulset-upgrade-4870/ss-0\\nevicting pod kube-system/event-exporter-v0.3.4-6c59d5574d-bwzh4\\nevicting pod kube-system/fluentd-gcp-scaler-55f8dfc997-7r75k\\nevicting pod kube-system/kube-dns-autoscaler-6494c4c647-w6m9v\\nevicting pod kube-system/l7-default-backend-79858d8f86-ctflr\\nevicting pod apparmor-upgrade-2359/apparmor-loader-4v5ln\\nevicting pod kube-system/metrics-server-v0.5.1-5999f95767-qkhrc\\nevicting pod hpa-upgrade-1278/res-cons-upgrade-7bpst\\nevicting pod service-upgrade-3087/service-test-zn5rw\\nevicting pod hpa-upgrade-1278/res-cons-upgrade-7h62f\\nevicting pod kube-system/coredns-755cd654d4-5zvxc\\nevicting pod hpa-upgrade-1278/res-cons-upgrade-mg7kv\\nevicting pod sig-apps-job-upgrade-2553/foo-ft2b6\\npod/ss-0 evicted\\npod/res-cons-upgrade-7h62f evicted\\npod/res-cons-upgrade-7bpst evicted\\npod/res-cons-upgrade-mg7kv evicted\\npod/apparmor-loader-4v5ln evicted\\npod/metrics-server-v0.5.1-5999f95767-qkhrc evicted\\npod/service-test-zn5rw evicted\\npod/kube-dns-autoscaler-6494c4c647-w6m9v evicted\\npod/l7-default-backend-79858d8f86-ctflr evicted\\npod/coredns-755cd654d4-5zvxc evicted\\npod/foo-ft2b6 evicted\\npod/fluentd-gcp-scaler-55f8dfc997-7r75k evicted\\npod/event-exporter-v0.3.4-6c59d5574d-bwzh4 evicted\\nnode/bootstrap-e2e-minion-group-2gn9 evicted\\n....................................................................................................................................................Node bootstrap-e2e-minion-group-2gn9 recreated.\\nNode bootstrap-e2e-minion-group-2gn9 Ready=True\\nnode/bootstrap-e2e-minion-group-2gn9 uncordoned\\nnode/bootstrap-e2e-minion-group-gt7d cordoned\\nevicting pod sig-apps-statefulset-upgrade-4870/ss-2\\nevicting pod hpa-upgrade-1278/res-cons-upgrade-ctrl-299sh\\nevicting pod hpa-upgrade-1278/res-cons-upgrade-74zbg\\nevicting pod hpa-upgrade-1278/res-cons-upgrade-7jkzd\\nevicting pod sig-apps-deployment-upgrade-6798/dp-67f7589d48-kqlqh\\nevicting pod hpa-upgrade-1278/res-cons-upgrade-jqxsm\\nevicting pod kube-system/coredns-755cd654d4-7dtq6\\nevicting pod kube-system/volume-snapshot-controller-0\\nevicting pod service-upgrade-3087/service-test-mth6w\\nevicting pod sig-apps-job-upgrade-2553/foo-ljw2g\\nevicting pod sig-apps-job-upgrade-2553/foo-8fhhm\\nevicting pod sig-apps-replicaset-upgrade-884/rs-vkvj5\\nevicting pod sig-apps-statefulset-upgrade-4870/ss-1\\npod/dp-67f7589d48-kqlqh evicted\\npod/rs-vkvj5 evicted\\npod/res-cons-upgrade-ctrl-299sh evicted\\npod/volume-snapshot-controller-0 evicted\\npod/res-cons-upgrade-7jkzd evicted\\npod/ss-1 evicted\\npod/ss-2 evicted\\npod/res-cons-upgrade-74zbg evicted\\npod/coredns-755cd654d4-7dtq6 evicted\\npod/res-cons-upgrade-jqxsm evicted\\npod/service-test-mth6w evicted\\npod/foo-ljw2g evicted\\npod/foo-8fhhm evicted\\nnode/bootstrap-e2e-minion-group-gt7d evicted\\n..............................................................................................................................................Node bootstrap-e2e-minion-group-gt7d recreated.\\nNode bootstrap-e2e-minion-group-gt7d Ready=True\\nnode/bootstrap-e2e-minion-group-gt7d uncordoned\\n== Waiting for CoreDNS to update ==\\n== CoreDNS ResourceVersion changed ==\\n== Fetching the latest installed CoreDNS version ==\\n== Downloading the CoreDNS migration tool ==\\n\", stderr \"Project: k8s-jkns-gci-gce-kubenet\\nNetwork Project: k8s-jkns-gci-gce-kubenet\\nZone: us-west1-b\\nINSTANCE_GROUPS=bootstrap-e2e-minion-group\\nNODE_NAMES=bootstrap-e2e-minion-group-1x5k bootstrap-e2e-minion-group-2gn9 bootstrap-e2e-minion-group-gt7d\\n== Preparing node upgrade (to v1.23.0-alpha.3.464+2dede1d4d45341). ==\\nAttempt 1 to create bootstrap-e2e-minion-template-v1-23-0-alpha-3-464-2dede1d4d4534\\nWARNING: You have selected a disk size of under [200GB]. This may result in poor I/O performance. For more information, see: https://developers.google.com/compute/docs/disks#performance.\\nCreated [https://www.googleapis.com/compute/v1/projects/k8s-jkns-gci-gce-kubenet/global/instanceTemplates/bootstrap-e2e-minion-template-v1-23-0-alpha-3-464-2dede1d4d4534].\\nNAME                                                             MACHINE_TYPE   PREEMPTIBLE  CREATION_TIMESTAMP\\nbootstrap-e2e-minion-template-v1-23-0-alpha-3-464-2dede1d4d4534  n1-standard-2               2021-10-21T08:26:04.089-07:00\\n== Finished preparing node upgrade (to v1.23.0-alpha.3.464+2dede1d4d45341). ==\\n== Upgrading nodes to v1.23.0-alpha.3.464+2dede1d4d45341 with max parallelism of 1. ==\\n== Draining bootstrap-e2e-minion-group-1x5k. == \\nWARNING: deleting Pods not managed by ReplicationController, ReplicaSet, Job, DaemonSet or StatefulSet: apparmor-upgrade-2359/test-apparmor-lwt65; ignoring DaemonSet-managed Pods: kube-system/fluentd-gcp-v3.2.0-flq2q, kube-system/konnectivity-agent-785d6, kube-system/metadata-proxy-v0.1-sslht, sig-apps-daemonset-upgrade-5708/ds1-szbq8\\nI1021 15:26:29.709909   21961 request.go:665] Waited for 1.04853537s due to client-side throttling, not priority and fairness, request: GET:https://34.145.10.33/api/v1/namespaces/sig-apps-statefulset-upgrade-4870/pods/ss-2\\n== Recreating instance bootstrap-e2e-minion-group-1x5k. ==\\n== Waiting for new node to be added to k8s.  ==\\n== Waiting for bootstrap-e2e-minion-group-1x5k to become ready. ==\\n== Uncordon bootstrap-e2e-minion-group-1x5k. == \\n== Draining bootstrap-e2e-minion-group-2gn9. == \\nWARNING: ignoring DaemonSet-managed Pods: kube-system/fluentd-gcp-v3.2.0-nplcc, kube-system/konnectivity-agent-smnhx, kube-system/metadata-proxy-v0.1-hwgzx, sig-apps-daemonset-upgrade-5708/ds1-jfzcf\\nI1021 15:30:31.136956   30268 request.go:665] Waited for 1.055579883s due to client-side throttling, not priority and fairness, request: GET:https://34.145.10.33/api/v1/namespaces/service-upgrade-3087/pods/service-test-zn5rw\\n== Recreating instance bootstrap-e2e-minion-group-2gn9. ==\\n== Waiting for new node to be added to k8s.  ==\\n== Waiting for bootstrap-e2e-minion-group-2gn9 to become ready. ==\\n== Uncordon bootstrap-e2e-minion-group-2gn9. == \\n== Draining bootstrap-e2e-minion-group-gt7d. == \\nWARNING: ignoring DaemonSet-managed Pods: kube-system/fluentd-gcp-v3.2.0-xt6vd, kube-system/konnectivity-agent-mmhgw, kube-system/metadata-proxy-v0.1-7b5cl, sig-apps-daemonset-upgrade-5708/ds1-mjg5r\\nI1021 15:34:30.706894   38523 request.go:665] Waited for 1.060197998s due to client-side throttling, not priority and fairness, request: GET:https://34.145.10.33/api/v1/namespaces/sig-apps-job-upgrade-2553/pods/foo-8fhhm\\n== Recreating instance bootstrap-e2e-minion-group-gt7d. ==\\n== Waiting for new node to be added to k8s.  ==\\n== Waiting for bootstrap-e2e-minion-group-gt7d to become ready. ==\\n== Uncordon bootstrap-e2e-minion-group-gt7d. == \\n== Deleting old templates in k8s-jkns-gci-gce-kubenet. ==\\nDeleted [https://www.googleapis.com/compute/v1/projects/k8s-jkns-gci-gce-kubenet/global/instanceTemplates/bootstrap-e2e-minion-template].\\n== Finished upgrading nodes to v1.23.0-alpha.3.464+2dede1d4d45341. ==\\n\"",
    }
    error running /workspace/kubernetes_skew/cluster/gce/upgrade.sh [-N -o v1.23.0-alpha.3.464+2dede1d4d45341]; got error exit status 8, stdout "Fetching the previously installed CoreDNS version\n== Pre-Upgrade Node OS and Kubelet Versions ==\nname: \"bootstrap-e2e-master\", osImage: \"Container-Optimized OS from Google\", kubeletVersion: \"v1.23.0-alpha.3.464+2dede1d4d45341\"\nname: \"bootstrap-e2e-minion-group-1x5k\", osImage: \"Container-Optimized OS from Google\", kubeletVersion: \"v1.21.6-rc.0.60+0fb71846df9bab\"\nname: \"bootstrap-e2e-minion-group-2gn9\", osImage: \"Container-Optimized OS from Google\", kubeletVersion: \"v1.21.6-rc.0.60+0fb71846df9bab\"\nname: \"bootstrap-e2e-minion-group-gt7d\", osImage: \"Container-Optimized OS from Google\", kubeletVersion: \"v1.21.6-rc.0.60+0fb71846df9bab\"\nFound subnet for region us-west1 in network bootstrap-e2e: bootstrap-e2e\n== Upgrading node environment variables. ==\nUsing subnet bootstrap-e2e\nInstance template name: bootstrap-e2e-minion-template-v1-23-0-alpha-3-464-2dede1d4d4534\nnode/bootstrap-e2e-minion-group-1x5k cordoned\nevicting pod apparmor-upgrade-2359/apparmor-loader-2lpwx\nevicting pod sig-apps-job-upgrade-2553/foo-trffq\nevicting pod apparmor-upgrade-2359/test-apparmor-bxpkt\nevicting pod apparmor-upgrade-2359/test-apparmor-lwt65\nevicting pod kube-system/coredns-755cd654d4-hv4mr\nevicting pod hpa-upgrade-1278/res-cons-upgrade-bldxc\nevicting pod sig-apps-statefulset-upgrade-4870/ss-2\nevicting pod hpa-upgrade-1278/res-cons-upgrade-ctrl-hn6pd\nevicting pod kube-system/event-exporter-v0.3.4-6c59d5574d-7kmpz\nevicting pod service-upgrade-3087/service-test-6dwxt\nevicting pod kube-system/metrics-server-v0.5.1-5999f95767-x6lcg\npod/test-apparmor-bxpkt evicted\npod/coredns-755cd654d4-hv4mr evicted\npod/service-test-6dwxt evicted\npod/res-cons-upgrade-bldxc evicted\npod/metrics-server-v0.5.1-5999f95767-x6lcg evicted\npod/apparmor-loader-2lpwx evicted\npod/res-cons-upgrade-ctrl-hn6pd evicted\npod/ss-2 evicted\npod/event-exporter-v0.3.4-6c59d5574d-7kmpz evicted\npod/foo-trffq evicted\npod/test-apparmor-lwt65 evicted\nnode/bootstrap-e2e-minion-group-1x5k evicted\n.....................................................................................................................................................Node bootstrap-e2e-minion-group-1x5k recreated.\nNode bootstrap-e2e-minion-group-1x5k Ready=True\nnode/bootstrap-e2e-minion-group-1x5k uncordoned\nnode/bootstrap-e2e-minion-group-2gn9 cordoned\nevicting pod sig-apps-statefulset-upgrade-4870/ss-0\nevicting pod kube-system/event-exporter-v0.3.4-6c59d5574d-bwzh4\nevicting pod kube-system/fluentd-gcp-scaler-55f8dfc997-7r75k\nevicting pod kube-system/kube-dns-autoscaler-6494c4c647-w6m9v\nevicting pod kube-system/l7-default-backend-79858d8f86-ctflr\nevicting pod apparmor-upgrade-2359/apparmor-loader-4v5ln\nevicting pod kube-system/metrics-server-v0.5.1-5999f95767-qkhrc\nevicting pod hpa-upgrade-1278/res-cons-upgrade-7bpst\nevicting pod service-upgrade-3087/service-test-zn5rw\nevicting pod hpa-upgrade-1278/res-cons-upgrade-7h62f\nevicting pod kube-system/coredns-755cd654d4-5zvxc\nevicting pod hpa-upgrade-1278/res-cons-upgrade-mg7kv\nevicting pod sig-apps-job-upgrade-2553/foo-ft2b6\npod/ss-0 evicted\npod/res-cons-upgrade-7h62f evicted\npod/res-cons-upgrade-7bpst evicted\npod/res-cons-upgrade-mg7kv evicted\npod/apparmor-loader-4v5ln evicted\npod/metrics-server-v0.5.1-5999f95767-qkhrc evicted\npod/service-test-zn5rw evicted\npod/kube-dns-autoscaler-6494c4c647-w6m9v evicted\npod/l7-default-backend-79858d8f86-ctflr evicted\npod/coredns-755cd654d4-5zvxc evicted\npod/foo-ft2b6 evicted\npod/fluentd-gcp-scaler-55f8dfc997-7r75k evicted\npod/event-exporter-v0.3.4-6c59d5574d-bwzh4 evicted\nnode/bootstrap-e2e-minion-group-2gn9 evicted\n....................................................................................................................................................Node bootstrap-e2e-minion-group-2gn9 recreated.\nNode bootstrap-e2e-minion-group-2gn9 Ready=True\nnode/bootstrap-e2e-minion-group-2gn9 uncordoned\nnode/bootstrap-e2e-minion-group-gt7d cordoned\nevicting pod sig-apps-statefulset-upgrade-4870/ss-2\nevicting pod hpa-upgrade-1278/res-cons-upgrade-ctrl-299sh\nevicting pod hpa-upgrade-1278/res-cons-upgrade-74zbg\nevicting pod hpa-upgrade-1278/res-cons-upgrade-7jkzd\nevicting pod sig-apps-deployment-upgrade-6798/dp-67f7589d48-kqlqh\nevicting pod hpa-upgrade-1278/res-cons-upgrade-jqxsm\nevicting pod kube-system/coredns-755cd654d4-7dtq6\nevicting pod kube-system/volume-snapshot-controller-0\nevicting pod service-upgrade-3087/service-test-mth6w\nevicting pod sig-apps-job-upgrade-2553/foo-ljw2g\nevicting pod sig-apps-job-upgrade-2553/foo-8fhhm\nevicting pod sig-apps-replicaset-upgrade-884/rs-vkvj5\nevicting pod sig-apps-statefulset-upgrade-4870/ss-1\npod/dp-67f7589d48-kqlqh evicted\npod/rs-vkvj5 evicted\npod/res-cons-upgrade-ctrl-299sh evicted\npod/volume-snapshot-controller-0 evicted\npod/res-cons-upgrade-7jkzd evicted\npod/ss-1 evicted\npod/ss-2 evicted\npod/res-cons-upgrade-74zbg evicted\npod/coredns-755cd654d4-7dtq6 evicted\npod/res-cons-upgrade-jqxsm evicted\npod/service-test-mth6w evicted\npod/foo-ljw2g evicted\npod/foo-8fhhm evicted\nnode/bootstrap-e2e-minion-group-gt7d evicted\n..............................................................................................................................................Node bootstrap-e2e-minion-group-gt7d recreated.\nNode bootstrap-e2e-minion-group-gt7d Ready=True\nnode/bootstrap-e2e-minion-group-gt7d uncordoned\n== Waiting for CoreDNS to update ==\n== CoreDNS ResourceVersion changed ==\n== Fetching the latest installed CoreDNS version ==\n== Downloading the CoreDNS migration tool ==\n", stderr "Project: k8s-jkns-gci-gce-kubenet\nNetwork Project: k8s-jkns-gci-gce-kubenet\nZone: us-west1-b\nINSTANCE_GROUPS=bootstrap-e2e-minion-group\nNODE_NAMES=bootstrap-e2e-minion-group-1x5k bootstrap-e2e-minion-group-2gn9 bootstrap-e2e-minion-group-gt7d\n== Preparing node upgrade (to v1.23.0-alpha.3.464+2dede1d4d45341). ==\nAttempt 1 to create bootstrap-e2e-minion-template-v1-23-0-alpha-3-464-2dede1d4d4534\nWARNING: You have selected a disk size of under [200GB]. This may result in poor I/O performance. For more information, see: https://developers.google.com/compute/docs/disks#performance.\nCreated [https://www.googleapis.com/compute/v1/projects/k8s-jkns-gci-gce-kubenet/global/instanceTemplates/bootstrap-e2e-minion-template-v1-23-0-alpha-3-464-2dede1d4d4534].\nNAME                                                             MACHINE_TYPE   PREEMPTIBLE  CREATION_TIMESTAMP\nbootstrap-e2e-minion-template-v1-23-0-alpha-3-464-2dede1d4d4534  n1-standard-2               2021-10-21T08:26:04.089-07:00\n== Finished preparing node upgrade (to v1.23.0-alpha.3.464+2dede1d4d45341). ==\n== Upgrading nodes to v1.23.0-alpha.3.464+2dede1d4d45341 with max parallelism of 1. ==\n== Draining bootstrap-e2e-minion-group-1x5k. == \nWARNING: deleting Pods not managed by ReplicationController, ReplicaSet, Job, DaemonSet or StatefulSet: apparmor-upgrade-2359/test-apparmor-lwt65; ignoring DaemonSet-managed Pods: kube-system/fluentd-gcp-v3.2.0-flq2q, kube-system/konnectivity-agent-785d6, kube-system/metadata-proxy-v0.1-sslht, sig-apps-daemonset-upgrade-5708/ds1-szbq8\nI1021 15:26:29.709909   21961 request.go:665] Waited for 1.04853537s due to client-side throttling, not priority and fairness, request: GET:https://34.145.10.33/api/v1/namespaces/sig-apps-statefulset-upgrade-4870/pods/ss-2\n== Recreating instance bootstrap-e2e-minion-group-1x5k. ==\n== Waiting for new node to be added to k8s.  ==\n== Waiting for bootstrap-e2e-minion-group-1x5k to become ready. ==\n== Uncordon bootstrap-e2e-minion-group-1x5k. == \n== Draining bootstrap-e2e-minion-group-2gn9. == \nWARNING: ignoring DaemonSet-managed Pods: kube-system/fluentd-gcp-v3.2.0-nplcc, kube-system/konnectivity-agent-smnhx, kube-system/metadata-proxy-v0.1-hwgzx, sig-apps-daemonset-upgrade-5708/ds1-jfzcf\nI1021 15:30:31.136956   30268 request.go:665] Waited for 1.055579883s due to client-side throttling, not priority and fairness, request: GET:https://34.145.10.33/api/v1/namespaces/service-upgrade-3087/pods/service-test-zn5rw\n== Recreating instance bootstrap-e2e-minion-group-2gn9. ==\n== Waiting for new node to be added to k8s.  ==\n== Waiting for bootstrap-e2e-minion-group-2gn9 to become ready. ==\n== Uncordon bootstrap-e2e-minion-group-2gn9. == \n== Draining bootstrap-e2e-minion-group-gt7d. == \nWARNING: ignoring DaemonSet-managed Pods: kube-system/fluentd-gcp-v3.2.0-xt6vd, kube-system/konnectivity-agent-mmhgw, kube-system/metadata-proxy-v0.1-7b5cl, sig-apps-daemonset-upgrade-5708/ds1-mjg5r\nI1021 15:34:30.706894   38523 request.go:665] Waited for 1.060197998s due to client-side throttling, not priority and fairness, request: GET:https://34.145.10.33/api/v1/namespaces/sig-apps-job-upgrade-2553/pods/foo-8fhhm\n== Recreating instance bootstrap-e2e-minion-group-gt7d. ==\n== Waiting for new node to be added to k8s.  ==\n== Waiting for bootstrap-e2e-minion-group-gt7d to become ready. ==\n== Uncordon bootstrap-e2e-minion-group-gt7d. == \n== Deleting old templates in k8s-jkns-gci-gce-kubenet. ==\nDeleted [https://www.googleapis.com/compute/v1/projects/k8s-jkns-gci-gce-kubenet/global/instanceTemplates/bootstrap-e2e-minion-template].\n== Finished upgrading nodes to v1.23.0-alpha.3.464+2dede1d4d45341. ==\n"
occurred
/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/cloud/gcp/common/upgrade_mechanics.go:55