This job view page is being replaced by Spyglass soon. Check out the new job view.
ResultFAILURE
Tests 5 failed / 186 succeeded
Started2020-01-16 16:43
Elapsed5h11m
Revision
Buildergke-prow-default-pool-cf4891d4-w60c
links{u'resultstore': {u'url': u'https://source.cloud.google.com/results/invocations/aafc698b-c510-45b4-afde-c1df9f12b091/targets/test'}}
pod464d81de-387f-11ea-8f3e-66e48c863062
resultstorehttps://source.cloud.google.com/results/invocations/aafc698b-c510-45b4-afde-c1df9f12b091/targets/test
infra-commit6592ba878
job-versionv1.16.5-beta.1.51+e7f962ba86f4ce
master_os_imagecos-77-12371-89-0
node_os_imagecos-73-11647-163-0
pod464d81de-387f-11ea-8f3e-66e48c863062
revisionv1.16.5-beta.1.51+e7f962ba86f4ce

Test Failures


Kubernetes e2e suite [k8s.io] [sig-cloud-provider-gcp] Upgrade [Feature:Upgrade] master upgrade should maintain a functioning cluster [Feature:MasterUpgrade] 17m10s

go run hack/e2e.go -v --test --test_args='--ginkgo.focus=Kubernetes\se2e\ssuite\s\[k8s\.io\]\s\[sig\-cloud\-provider\-gcp\]\sUpgrade\s\[Feature\:Upgrade\]\smaster\supgrade\sshould\smaintain\sa\sfunctioning\scluster\s\[Feature\:MasterUpgrade\]$'
/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/cloud/gcp/cluster_upgrade.go:91
Jan 16 16:58:31.838: Unexpected error:
    <*errors.errorString | 0xc0022822f0>: {
        s: "error running /workspace/kubernetes_skew/cluster/gce/upgrade.sh [-M v1.17.2-beta.0]; got error exit status 1, stdout \"Fetching the previously installed CoreDNS version\\n\\n***WARNING***\\nUpgrading Kubernetes with this script might result in an upgrade to a new etcd version.\\nSome etcd version upgrades, such as 3.0.x to 3.1.x, DO NOT offer a downgrade path.\\nTo pin the etcd version to your current one (e.g. v3.0.17), set the following variables\\nbefore running this script:\\n\\n# example: pin to etcd v3.0.17\\nexport ETCD_IMAGE=3.0.17\\nexport ETCD_VERSION=3.0.17\\n\\nAlternatively, if you choose to allow an etcd upgrade that doesn't support downgrade,\\nyou might still be able to downgrade Kubernetes by pinning to the newer etcd version.\\nIn all cases, it is strongly recommended to have an etcd backup before upgrading.\\n\\n== Pre-Upgrade Node OS and Kubelet Versions ==\\nname: \\\"bootstrap-e2e-master\\\", osImage: \\\"Container-Optimized OS from Google\\\", kubeletVersion: \\\"v1.16.5-beta.1.51+e7f962ba86f4ce\\\"\\nname: \\\"bootstrap-e2e-minion-group-fbnv\\\", osImage: \\\"Container-Optimized OS from Google\\\", kubeletVersion: \\\"v1.16.5-beta.1.51+e7f962ba86f4ce\\\"\\nname: \\\"bootstrap-e2e-minion-group-z472\\\", osImage: \\\"Container-Optimized OS from Google\\\", kubeletVersion: \\\"v1.16.5-beta.1.51+e7f962ba86f4ce\\\"\\nname: \\\"bootstrap-e2e-minion-group-zx21\\\", osImage: \\\"Container-Optimized OS from Google\\\", kubeletVersion: \\\"v1.16.5-beta.1.51+e7f962ba86f4ce\\\"\\nFound subnet for region us-west1 in network bootstrap-e2e: bootstrap-e2e\\n== Upgrading master to 'https://storage.googleapis.com/kubernetes-release/release/v1.17.2-beta.0/kubernetes-server-linux-amd64.tar.gz'. Do not interrupt, deleting master instance. ==\\n== Upgrading master environment variables. ==\\n== Waiting for new master to respond to API requests ==\\n......................== Done ==\\nWaiting for CoreDNS to update\\nFetching the latest installed CoreDNS version\\n== Downloading the CoreDNS migration tool ==\\n== Upgrading the CoreDNS ConfigMap ==\\nconfigmap/coredns configured\\n== The CoreDNS Config has been updated ==\\n== Validating cluster post-upgrade ==\\nValidating gce cluster, MULTIZONE=\\nFound 4 node(s).\\nNAME                              STATUS                     ROLES    AGE     VERSION\\nbootstrap-e2e-master              Ready,SchedulingDisabled   <none>   9m10s   v1.17.1\\nbootstrap-e2e-minion-group-fbnv   Ready                      <none>   9m13s   v1.16.5-beta.1.51+e7f962ba86f4ce\\nbootstrap-e2e-minion-group-z472   Ready                      <none>   9m13s   v1.16.5-beta.1.51+e7f962ba86f4ce\\nbootstrap-e2e-minion-group-zx21   Ready                      <none>   9m13s   v1.16.5-beta.1.51+e7f962ba86f4ce\\nValidate output:\\nNAME                 STATUS    MESSAGE             ERROR\\nscheduler            Healthy   ok                  \\netcd-0               Healthy   {\\\"health\\\":\\\"true\\\"}   \\netcd-1               Healthy   {\\\"health\\\":\\\"true\\\"}   \\ncontroller-manager   Healthy   ok                  \\n\\x1b[0;32mCluster validation succeeded\\x1b[0m\\n== Post-Upgrade Node OS and Kubelet Versions ==\\nname: \\\"bootstrap-e2e-master\\\", osImage: \\\"Container-Optimized OS from Google\\\", kubeletVersion: \\\"v1.17.1\\\"\\nname: \\\"bootstrap-e2e-minion-group-fbnv\\\", osImage: \\\"Container-Optimized OS from Google\\\", kubeletVersion: \\\"v1.16.5-beta.1.51+e7f962ba86f4ce\\\"\\nname: \\\"bootstrap-e2e-minion-group-z472\\\", osImage: \\\"Container-Optimized OS from Google\\\", kubeletVersion: \\\"v1.16.5-beta.1.51+e7f962ba86f4ce\\\"\\nname: \\\"bootstrap-e2e-minion-group-zx21\\\", osImage: \\\"Container-Optimized OS from Google\\\", kubeletVersion: \\\"v1.16.5-beta.1.51+e7f962ba86f4ce\\\"\\n\", stderr \"Project: kube-gce-upg-1-4-1-5-upg-clu-n\\nNetwork Project: kube-gce-upg-1-4-1-5-upg-clu-n\\nZone: us-west1-b\\nINSTANCE_GROUPS=bootstrap-e2e-minion-group\\nNODE_NAMES=bootstrap-e2e-minion-group-fbnv bootstrap-e2e-minion-group-z472 bootstrap-e2e-minion-group-zx21\\nWARNING: Some requests did not succeed.\\n - Invalid value for field 'zone': 'asia-northeast3-a'. Unknown zone.\\n - Invalid value for field 'zone': 'asia-northeast3-b'. Unknown zone.\\n - Invalid value for field 'zone': 'asia-northeast3-c'. Unknown zone.\\n\\nTrying to find master named 'bootstrap-e2e-master'\\nLooking for address 'bootstrap-e2e-master-ip'\\nUsing master: bootstrap-e2e-master (external IP: 34.83.130.135; internal IP: (not set))\\nDeleted [https://www.googleapis.com/compute/v1/projects/kube-gce-upg-1-4-1-5-upg-clu-n/zones/us-west1-b/instances/bootstrap-e2e-master].\\nWARNING: You have selected a disk size of under [200GB]. This may result in poor I/O performance. For more information, see: https://developers.google.com/compute/docs/disks#performance.\\nCreated [https://www.googleapis.com/compute/v1/projects/kube-gce-upg-1-4-1-5-upg-clu-n/zones/us-west1-b/instances/bootstrap-e2e-master].\\nWARNING: Some requests generated warnings:\\n - Disk size: '20 GB' is larger than image size: '10 GB'. You might need to resize the root repartition manually if the operating system does not support automatic resizing. See https://cloud.google.com/compute/docs/disks/add-persistent-disk#resize_pd for details.\\n - The resource 'projects/cos-cloud/global/images/cos-77-12371-89-0' is deprecated. A suggested replacement is 'projects/cos-cloud/global/images/cos-77-12371-114-0'.\\n\\nNAME                  ZONE        MACHINE_TYPE   PREEMPTIBLE  INTERNAL_IP  EXTERNAL_IP    STATUS\\nbootstrap-e2e-master  us-west1-b  n1-standard-1               10.138.0.6   34.83.130.135  RUNNING\\nWarning: kubectl apply should be used on resource created by either kubectl create --save-config or kubectl apply\\nProject: kube-gce-upg-1-4-1-5-upg-clu-n\\nNetwork Project: kube-gce-upg-1-4-1-5-upg-clu-n\\nZone: us-west1-b\\nWARNING: Some requests did not succeed.\\n - Invalid value for field 'zone': 'asia-northeast3-a'. Unknown zone.\\n - Invalid value for field 'zone': 'asia-northeast3-b'. Unknown zone.\\n - Invalid value for field 'zone': 'asia-northeast3-c'. Unknown zone.\\n\\n/workspace/kubernetes_skew/cluster/gce/upgrade.sh: line 452: download_dir: unbound variable\\n\"",
    }
    error running /workspace/kubernetes_skew/cluster/gce/upgrade.sh [-M v1.17.2-beta.0]; got error exit status 1, stdout "Fetching the previously installed CoreDNS version\n\n***WARNING***\nUpgrading Kubernetes with this script might result in an upgrade to a new etcd version.\nSome etcd version upgrades, such as 3.0.x to 3.1.x, DO NOT offer a downgrade path.\nTo pin the etcd version to your current one (e.g. v3.0.17), set the following variables\nbefore running this script:\n\n# example: pin to etcd v3.0.17\nexport ETCD_IMAGE=3.0.17\nexport ETCD_VERSION=3.0.17\n\nAlternatively, if you choose to allow an etcd upgrade that doesn't support downgrade,\nyou might still be able to downgrade Kubernetes by pinning to the newer etcd version.\nIn all cases, it is strongly recommended to have an etcd backup before upgrading.\n\n== Pre-Upgrade Node OS and Kubelet Versions ==\nname: \"bootstrap-e2e-master\", osImage: \"Container-Optimized OS from Google\", kubeletVersion: \"v1.16.5-beta.1.51+e7f962ba86f4ce\"\nname: \"bootstrap-e2e-minion-group-fbnv\", osImage: \"Container-Optimized OS from Google\", kubeletVersion: \"v1.16.5-beta.1.51+e7f962ba86f4ce\"\nname: \"bootstrap-e2e-minion-group-z472\", osImage: \"Container-Optimized OS from Google\", kubeletVersion: \"v1.16.5-beta.1.51+e7f962ba86f4ce\"\nname: \"bootstrap-e2e-minion-group-zx21\", osImage: \"Container-Optimized OS from Google\", kubeletVersion: \"v1.16.5-beta.1.51+e7f962ba86f4ce\"\nFound subnet for region us-west1 in network bootstrap-e2e: bootstrap-e2e\n== Upgrading master to 'https://storage.googleapis.com/kubernetes-release/release/v1.17.2-beta.0/kubernetes-server-linux-amd64.tar.gz'. Do not interrupt, deleting master instance. ==\n== Upgrading master environment variables. ==\n== Waiting for new master to respond to API requests ==\n......................== Done ==\nWaiting for CoreDNS to update\nFetching the latest installed CoreDNS version\n== Downloading the CoreDNS migration tool ==\n== Upgrading the CoreDNS ConfigMap ==\nconfigmap/coredns configured\n== The CoreDNS Config has been updated ==\n== Validating cluster post-upgrade ==\nValidating gce cluster, MULTIZONE=\nFound 4 node(s).\nNAME                              STATUS                     ROLES    AGE     VERSION\nbootstrap-e2e-master              Ready,SchedulingDisabled   <none>   9m10s   v1.17.1\nbootstrap-e2e-minion-group-fbnv   Ready                      <none>   9m13s   v1.16.5-beta.1.51+e7f962ba86f4ce\nbootstrap-e2e-minion-group-z472   Ready                      <none>   9m13s   v1.16.5-beta.1.51+e7f962ba86f4ce\nbootstrap-e2e-minion-group-zx21   Ready                      <none>   9m13s   v1.16.5-beta.1.51+e7f962ba86f4ce\nValidate output:\nNAME                 STATUS    MESSAGE             ERROR\nscheduler            Healthy   ok                  \netcd-0               Healthy   {\"health\":\"true\"}   \netcd-1               Healthy   {\"health\":\"true\"}   \ncontroller-manager   Healthy   ok                  \n\x1b[0;32mCluster validation succeeded\x1b[0m\n== Post-Upgrade Node OS and Kubelet Versions ==\nname: \"bootstrap-e2e-master\", osImage: \"Container-Optimized OS from Google\", kubeletVersion: \"v1.17.1\"\nname: \"bootstrap-e2e-minion-group-fbnv\", osImage: \"Container-Optimized OS from Google\", kubeletVersion: \"v1.16.5-beta.1.51+e7f962ba86f4ce\"\nname: \"bootstrap-e2e-minion-group-z472\", osImage: \"Container-Optimized OS from Google\", kubeletVersion: \"v1.16.5-beta.1.51+e7f962ba86f4ce\"\nname: \"bootstrap-e2e-minion-group-zx21\", osImage: \"Container-Optimized OS from Google\", kubeletVersion: \"v1.16.5-beta.1.51+e7f962ba86f4ce\"\n", stderr "Project: kube-gce-upg-1-4-1-5-upg-clu-n\nNetwork Project: kube-gce-upg-1-4-1-5-upg-clu-n\nZone: us-west1-b\nINSTANCE_GROUPS=bootstrap-e2e-minion-group\nNODE_NAMES=bootstrap-e2e-minion-group-fbnv bootstrap-e2e-minion-group-z472 bootstrap-e2e-minion-group-zx21\nWARNING: Some requests did not succeed.\n - Invalid value for field 'zone': 'asia-northeast3-a'. Unknown zone.\n - Invalid value for field 'zone': 'asia-northeast3-b'. Unknown zone.\n - Invalid value for field 'zone': 'asia-northeast3-c'. Unknown zone.\n\nTrying to find master named 'bootstrap-e2e-master'\nLooking for address 'bootstrap-e2e-master-ip'\nUsing master: bootstrap-e2e-master (external IP: 34.83.130.135; internal IP: (not set))\nDeleted [https://www.googleapis.com/compute/v1/projects/kube-gce-upg-1-4-1-5-upg-clu-n/zones/us-west1-b/instances/bootstrap-e2e-master].\nWARNING: You have selected a disk size of under [200GB]. This may result in poor I/O performance. For more information, see: https://developers.google.com/compute/docs/disks#performance.\nCreated [https://www.googleapis.com/compute/v1/projects/kube-gce-upg-1-4-1-5-upg-clu-n/zones/us-west1-b/instances/bootstrap-e2e-master].\nWARNING: Some requests generated warnings:\n - Disk size: '20 GB' is larger than image size: '10 GB'. You might need to resize the root repartition manually if the operating system does not support automatic resizing. See https://cloud.google.com/compute/docs/disks/add-persistent-disk#resize_pd for details.\n - The resource 'projects/cos-cloud/global/images/cos-77-12371-89-0' is deprecated. A suggested replacement is 'projects/cos-cloud/global/images/cos-77-12371-114-0'.\n\nNAME                  ZONE        MACHINE_TYPE   PREEMPTIBLE  INTERNAL_IP  EXTERNAL_IP    STATUS\nbootstrap-e2e-master  us-west1-b  n1-standard-1               10.138.0.6   34.83.130.135  RUNNING\nWarning: kubectl apply should be used on resource created by either kubectl create --save-config or kubectl apply\nProject: kube-gce-upg-1-4-1-5-upg-clu-n\nNetwork Project: kube-gce-upg-1-4-1-5-upg-clu-n\nZone: us-west1-b\nWARNING: Some requests did not succeed.\n - Invalid value for field 'zone': 'asia-northeast3-a'. Unknown zone.\n - Invalid value for field 'zone': 'asia-northeast3-b'. Unknown zone.\n - Invalid value for field 'zone': 'asia-northeast3-c'. Unknown zone.\n\n/workspace/kubernetes_skew/cluster/gce/upgrade.sh: line 452: download_dir: unbound variable\n"
occurred
/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/cloud/gcp/cluster_upgrade.go:106
				
				Click to see stdout/stderrfrom junit_upgrade01.xml

Filter through log files | View test history on testgrid


Kubernetes e2e suite [sig-cluster-lifecycle] Nodes [Disruptive] Resize [Slow] should be able to delete nodes 6m41s

go run hack/e2e.go -v --test --test_args='--ginkgo.focus=Kubernetes\se2e\ssuite\s\[sig\-cluster\-lifecycle\]\sNodes\s\[Disruptive\]\sResize\s\[Slow\]\sshould\sbe\sable\sto\sdelete\snodes$'
/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/lifecycle/resize_nodes.go:112
Jan 16 21:23:55.640: Unexpected error:
    <*errors.errorString | 0xc003c356b0>: {
        s: "failed to wait for pods responding: pod with UID 4007a68d-e411-467e-80c1-7b09ebc6e9cd is no longer a member of the replica set.  Must have been restarted for some reason.  Current replica set: &PodList{ListMeta:{/api/v1/namespaces/resize-nodes-5962/pods 69225  <nil>},Items:[]Pod{Pod{ObjectMeta:{my-hostname-delete-node-7ffk2 my-hostname-delete-node- resize-nodes-5962 /api/v1/namespaces/resize-nodes-5962/pods/my-hostname-delete-node-7ffk2 d60bd1bd-e74d-4d84-adad-63ee6157e892 69133 0 2020-01-16 21:23:23 +0000 UTC <nil> <nil> map[name:my-hostname-delete-node] map[] [{v1 ReplicationController my-hostname-delete-node 5844a37c-bdba-4555-adc9-b5b4e7512845 0xc003d499be 0xc003d499bf}] []  []},Spec:PodSpec{Volumes:[]Volume{Volume{Name:default-token-b46sh,VolumeSource:VolumeSource{HostPath:nil,EmptyDir:nil,GCEPersistentDisk:nil,AWSElasticBlockStore:nil,GitRepo:nil,Secret:&SecretVolumeSource{SecretName:default-token-b46sh,Items:[]KeyToPath{},DefaultMode:*420,Optional:nil,},NFS:nil,ISCSI:nil,Glusterfs:nil,PersistentVolumeClaim:nil,RBD:nil,FlexVolume:nil,Cinder:nil,CephFS:nil,Flocker:nil,DownwardAPI:nil,FC:nil,AzureFile:nil,ConfigMap:nil,VsphereVolume:nil,Quobyte:nil,AzureDisk:nil,PhotonPersistentDisk:nil,PortworxVolume:nil,ScaleIO:nil,Projected:nil,StorageOS:nil,CSI:nil,},},},Containers:[]Container{Container{Name:my-hostname-delete-node,Image:gcr.io/kubernetes-e2e-test-images/agnhost:2.6,Command:[],Args:[serve-hostname],WorkingDir:,Ports:[]ContainerPort{ContainerPort{Name:,HostPort:0,ContainerPort:9376,Protocol:TCP,HostIP:,},},Env:[]EnvVar{},Resources:ResourceRequirements{Limits:ResourceList{},Requests:ResourceList{},},VolumeMounts:[]VolumeMount{VolumeMount{Name:default-token-b46sh,ReadOnly:true,MountPath:/var/run/secrets/kubernetes.io/serviceaccount,SubPath:,MountPropagation:nil,SubPathExpr:,},},LivenessProbe:nil,ReadinessProbe:nil,Lifecycle:nil,TerminationMessagePath:/dev/termination-log,ImagePullPolicy:IfNotPresent,SecurityContext:nil,Stdin:false,StdinOnce:false,TTY:false,EnvFrom:[]EnvFromSource{},TerminationMessagePolicy:File,VolumeDevices:[]VolumeDevice{},StartupProbe:nil,},},RestartPolicy:Always,TerminationGracePeriodSeconds:*0,ActiveDeadlineSeconds:nil,DNSPolicy:ClusterFirst,NodeSelector:map[string]string{},ServiceAccountName:default,DeprecatedServiceAccount:default,NodeName:bootstrap-e2e-minion-group-z472,HostNetwork:false,HostPID:false,HostIPC:false,SecurityContext:&PodSecurityContext{SELinuxOptions:nil,RunAsUser:nil,RunAsNonRoot:nil,SupplementalGroups:[],FSGroup:nil,RunAsGroup:nil,Sysctls:[]Sysctl{},WindowsOptions:nil,},ImagePullSecrets:[]LocalObjectReference{},Hostname:,Subdomain:,Affinity:nil,SchedulerName:default-scheduler,InitContainers:[]Container{},AutomountServiceAccountToken:nil,Tolerations:[]Toleration{Toleration{Key:node.kubernetes.io/not-ready,Operator:Exists,Value:,Effect:NoExecute,TolerationSeconds:*300,},Toleration{Key:node.kubernetes.io/unreachable,Operator:Exists,Value:,Effect:NoExecute,TolerationSeconds:*300,},},HostAliases:[]HostAlias{},PriorityClassName:,Priority:*0,DNSConfig:nil,ShareProcessNamespace:nil,ReadinessGates:[]PodReadinessGate{},RuntimeClassName:nil,EnableServiceLinks:*true,PreemptionPolicy:nil,Overhead:ResourceList{},TopologySpreadConstraints:[]TopologySpreadConstraint{},EphemeralContainers:[]EphemeralContainer{},},Status:PodStatus{Phase:Running,Conditions:[]PodCondition{PodCondition{Type:Initialized,Status:True,LastProbeTime:0001-01-01 00:00:00 +0000 UTC,LastTransitionTime:2020-01-16 21:23:23 +0000 UTC,Reason:,Message:,},PodCondition{Type:Ready,Status:True,LastProbeTime:0001-01-01 00:00:00 +0000 UTC,LastTransitionTime:2020-01-16 21:23:25 +0000 UTC,Reason:,Message:,},PodCondition{Type:ContainersReady,Status:True,LastProbeTime:0001-01-01 00:00:00 +0000 UTC,LastTransitionTime:2020-01-16 21:23:25 +0000 UTC,Reason:,Message:,},PodCondition{Type:PodScheduled,Status:True,LastProbeTime:0001-01-01 00:00:00 +0000 UTC,LastTransitionTime:2020-01-16 21:23:23 +0000 UTC,Reason:,Message:,},},Message:,Reason:,HostIP:10.138.0.4,PodIP:10.64.2.246,StartTime:2020-01-16 21:23:23 +0000 UTC,ContainerStatuses:[]ContainerStatus{ContainerStatus{Name:my-hostname-delete-node,State:ContainerState{Waiting:nil,Running:&ContainerStateRunning{StartedAt:2020-01-16 21:23:25 +0000 UTC,},Terminated:nil,},LastTerminationState:ContainerState{Waiting:nil,Running:nil,Terminated:nil,},Ready:true,RestartCount:0,Image:gcr.io/kubernetes-e2e-test-images/agnhost:2.6,ImageID:docker-pullable://gcr.io/kubernetes-e2e-test-images/agnhost@sha256:4057a5580c7b59c4fe10d8ab2732c9dec35eea80fd41f7bafc7bd5acc7edf727,ContainerID:docker://978d0cddc0adef9ebaa07aa76223e275de2ec45c6a6674929d1342e3b8101f56,Started:*true,},},QOSClass:BestEffort,InitContainerStatuses:[]ContainerStatus{},NominatedNodeName:,PodIPs:[]PodIP{PodIP{IP:10.64.2.246,},},EphemeralContainerStatuses:[]ContainerStatus{},},},Pod{ObjectMeta:{my-hostname-delete-node-fggj9 my-hostname-delete-node- resize-nodes-5962 /api/v1/namespaces/resize-nodes-5962/pods/my-hostname-delete-node-fggj9 f3b37e06-5e30-4106-8a7f-62ffd13bcf5b 68304 0 2020-01-16 21:18:52 +0000 UTC <nil> <nil> map[name:my-hostname-delete-node] map[] [{v1 ReplicationController my-hostname-delete-node 5844a37c-bdba-4555-adc9-b5b4e7512845 0xc003d49af6 0xc003d49af7}] []  []},Spec:PodSpec{Volumes:[]Volume{Volume{Name:default-token-b46sh,VolumeSource:VolumeSource{HostPath:nil,EmptyDir:nil,GCEPersistentDisk:nil,AWSElasticBlockStore:nil,GitRepo:nil,Secret:&SecretVolumeSource{SecretName:default-token-b46sh,Items:[]KeyToPath{},DefaultMode:*420,Optional:nil,},NFS:nil,ISCSI:nil,Glusterfs:nil,PersistentVolumeClaim:nil,RBD:nil,FlexVolume:nil,Cinder:nil,CephFS:nil,Flocker:nil,DownwardAPI:nil,FC:nil,AzureFile:nil,ConfigMap:nil,VsphereVolume:nil,Quobyte:nil,AzureDisk:nil,PhotonPersistentDisk:nil,PortworxVolume:nil,ScaleIO:nil,Projected:nil,StorageOS:nil,CSI:nil,},},},Containers:[]Container{Container{Name:my-hostname-delete-node,Image:gcr.io/kubernetes-e2e-test-images/agnhost:2.6,Command:[],Args:[serve-hostname],WorkingDir:,Ports:[]ContainerPort{ContainerPort{Name:,HostPort:0,ContainerPort:9376,Protocol:TCP,HostIP:,},},Env:[]EnvVar{},Resources:ResourceRequirements{Limits:ResourceList{},Requests:ResourceList{},},VolumeMounts:[]VolumeMount{VolumeMount{Name:default-token-b46sh,ReadOnly:true,MountPath:/var/run/secrets/kubernetes.io/serviceaccount,SubPath:,MountPropagation:nil,SubPathExpr:,},},LivenessProbe:nil,ReadinessProbe:nil,Lifecycle:nil,TerminationMessagePath:/dev/termination-log,ImagePullPolicy:IfNotPresent,SecurityContext:nil,Stdin:false,StdinOnce:false,TTY:false,EnvFrom:[]EnvFromSource{},TerminationMessagePolicy:File,VolumeDevices:[]VolumeDevice{},StartupProbe:nil,},},RestartPolicy:Always,TerminationGracePeriodSeconds:*0,ActiveDeadlineSeconds:nil,DNSPolicy:ClusterFirst,NodeSelector:map[string]string{},ServiceAccountName:default,DeprecatedServiceAccount:default,NodeName:bootstrap-e2e-minion-group-zx21,HostNetwork:false,HostPID:false,HostIPC:false,SecurityContext:&PodSecurityContext{SELinuxOptions:nil,RunAsUser:nil,RunAsNonRoot:nil,SupplementalGroups:[],FSGroup:nil,RunAsGroup:nil,Sysctls:[]Sysctl{},WindowsOptions:nil,},ImagePullSecrets:[]LocalObjectReference{},Hostname:,Subdomain:,Affinity:nil,SchedulerName:default-scheduler,InitContainers:[]Container{},AutomountServiceAccountToken:nil,Tolerations:[]Toleration{Toleration{Key:node.kubernetes.io/not-ready,Operator:Exists,Value:,Effect:NoExecute,TolerationSeconds:*300,},Toleration{Key:node.kubernetes.io/unreachable,Operator:Exists,Value:,Effect:NoExecute,TolerationSeconds:*300,},},HostAliases:[]HostAlias{},PriorityClassName:,Priority:*0,DNSConfig:nil,ShareProcessNamespace:nil,ReadinessGates:[]PodReadinessGate{},RuntimeClassName:nil,EnableServiceLinks:*true,PreemptionPolicy:nil,Overhead:ResourceList{},TopologySpreadConstraints:[]TopologySpreadConstraint{},EphemeralContainers:[]EphemeralContainer{},},Status:PodStatus{Phase:Running,Conditions:[]PodCondition{PodCondition{Type:Initialized,Status:True,LastProbeTime:0001-01-01 00:00:00 +0000 UTC,LastTransitionTime:2020-01-16 21:18:52 +0000 UTC,Reason:,Message:,},PodCondition{Type:Ready,Status:True,LastProbeTime:0001-01-01 00:00:00 +0000 UTC,LastTransitionTime:2020-01-16 21:18:54 +0000 UTC,Reason:,Message:,},PodCondition{Type:ContainersReady,Status:True,LastProbeTime:0001-01-01 00:00:00 +0000 UTC,LastTransitionTime:2020-01-16 21:18:54 +0000 UTC,Reason:,Message:,},PodCondition{Type:PodScheduled,Status:True,LastProbeTime:0001-01-01 00:00:00 +0000 UTC,LastTransitionTime:2020-01-16 21:18:52 +0000 UTC,Reason:,Message:,},},Message:,Reason:,HostIP:10.138.0.5,PodIP:10.64.1.219,StartTime:2020-01-16 21:18:52 +0000 UTC,ContainerStatuses:[]ContainerStatus{ContainerStatus{Name:my-hostname-delete-node,State:ContainerState{Waiting:nil,Running:&ContainerStateRunning{StartedAt:2020-01-16 21:18:53 +0000 UTC,},Terminated:nil,},LastTerminationState:ContainerState{Waiting:nil,Running:nil,Terminated:nil,},Ready:true,RestartCount:0,Image:gcr.io/kubernetes-e2e-test-images/agnhost:2.6,ImageID:docker-pullable://gcr.io/kubernetes-e2e-test-images/agnhost@sha256:4057a5580c7b59c4fe10d8ab2732c9dec35eea80fd41f7bafc7bd5acc7edf727,ContainerID:docker://f11888dd31cef81ce4282b9f17ab34aa35f89870cae3ed9b29f7a343a13a54b0,Started:*true,},},QOSClass:BestEffort,InitContainerStatuses:[]ContainerStatus{},NominatedNodeName:,PodIPs:[]PodIP{PodIP{IP:10.64.1.219,},},EphemeralContainerStatuses:[]ContainerStatus{},},},Pod{ObjectMeta:{my-hostname-delete-node-rzj9m my-hostname-delete-node- resize-nodes-5962 /api/v1/namespaces/resize-nodes-5962/pods/my-hostname-delete-node-rzj9m 83a82417-bd87-4230-b370-54bbfa9b990a 68301 0 2020-01-16 21:18:52 +0000 UTC <nil> <nil> map[name:my-hostname-delete-node] map[] [{v1 ReplicationController my-hostname-delete-node 5844a37c-bdba-4555-adc9-b5b4e7512845 0xc003d49c26 0xc003d49c27}] []  []},Spec:PodSpec{Volumes:[]Volume{Volume{Name:default-token-b46sh,VolumeSource:VolumeSource{HostPath:nil,EmptyDir:nil,GCEPersistentDisk:nil,AWSElasticBlockStore:nil,GitRepo:nil,Secret:&SecretVolumeSource{SecretName:default-token-b46sh,Items:[]KeyToPath{},DefaultMode:*420,Optional:nil,},NFS:nil,ISCSI:nil,Glusterfs:nil,PersistentVolumeClaim:nil,RBD:nil,FlexVolume:nil,Cinder:nil,CephFS:nil,Flocker:nil,DownwardAPI:nil,FC:nil,AzureFile:nil,ConfigMap:nil,VsphereVolume:nil,Quobyte:nil,AzureDisk:nil,PhotonPersistentDisk:nil,PortworxVolume:nil,ScaleIO:nil,Projected:nil,StorageOS:nil,CSI:nil,},},},Containers:[]Container{Container{Name:my-hostname-delete-node,Image:gcr.io/kubernetes-e2e-test-images/agnhost:2.6,Command:[],Args:[serve-hostname],WorkingDir:,Ports:[]ContainerPort{ContainerPort{Name:,HostPort:0,ContainerPort:9376,Protocol:TCP,HostIP:,},},Env:[]EnvVar{},Resources:ResourceRequirements{Limits:ResourceList{},Requests:ResourceList{},},VolumeMounts:[]VolumeMount{VolumeMount{Name:default-token-b46sh,ReadOnly:true,MountPath:/var/run/secrets/kubernetes.io/serviceaccount,SubPath:,MountPropagation:nil,SubPathExpr:,},},LivenessProbe:nil,ReadinessProbe:nil,Lifecycle:nil,TerminationMessagePath:/dev/termination-log,ImagePullPolicy:IfNotPresent,SecurityContext:nil,Stdin:false,StdinOnce:false,TTY:false,EnvFrom:[]EnvFromSource{},TerminationMessagePolicy:File,VolumeDevices:[]VolumeDevice{},StartupProbe:nil,},},RestartPolicy:Always,TerminationGracePeriodSeconds:*0,ActiveDeadlineSeconds:nil,DNSPolicy:ClusterFirst,NodeSelector:map[string]string{},ServiceAccountName:default,DeprecatedServiceAccount:default,NodeName:bootstrap-e2e-minion-group-z472,HostNetwork:false,HostPID:false,HostIPC:false,SecurityContext:&PodSecurityContext{SELinuxOptions:nil,RunAsUser:nil,RunAsNonRoot:nil,SupplementalGroups:[],FSGroup:nil,RunAsGroup:nil,Sysctls:[]Sysctl{},WindowsOptions:nil,},ImagePullSecrets:[]LocalObjectReference{},Hostname:,Subdomain:,Affinity:nil,SchedulerName:default-scheduler,InitContainers:[]Container{},AutomountServiceAccountToken:nil,Tolerations:[]Toleration{Toleration{Key:node.kubernetes.io/not-ready,Operator:Exists,Value:,Effect:NoExecute,TolerationSeconds:*300,},Toleration{Key:node.kubernetes.io/unreachable,Operator:Exists,Value:,Effect:NoExecute,TolerationSeconds:*300,},},HostAliases:[]HostAlias{},PriorityClassName:,Priority:*0,DNSConfig:nil,ShareProcessNamespace:nil,ReadinessGates:[]PodReadinessGate{},RuntimeClassName:nil,EnableServiceLinks:*true,PreemptionPolicy:nil,Overhead:ResourceList{},TopologySpreadConstraints:[]TopologySpreadConstraint{},EphemeralContainers:[]EphemeralContainer{},},Status:PodStatus{Phase:Running,Conditions:[]PodCondition{PodCondition{Type:Initialized,Status:True,LastProbeTime:0001-01-01 00:00:00 +0000 UTC,LastTransitionTime:2020-01-16 21:18:52 +0000 UTC,Reason:,Message:,},PodCondition{Type:Ready,Status:True,LastProbeTime:0001-01-01 00:00:00 +0000 UTC,LastTransitionTime:2020-01-16 21:18:54 +0000 UTC,Reason:,Message:,},PodCondition{Type:ContainersReady,Status:True,LastProbeTime:0001-01-01 00:00:00 +0000 UTC,LastTransitionTime:2020-01-16 21:18:54 +0000 UTC,Reason:,Message:,},PodCondition{Type:PodScheduled,Status:True,LastProbeTime:0001-01-01 00:00:00 +0000 UTC,LastTransitionTime:2020-01-16 21:18:52 +0000 UTC,Reason:,Message:,},},Message:,Reason:,HostIP:10.138.0.4,PodIP:10.64.2.245,StartTime:2020-01-16 21:18:52 +0000 UTC,ContainerStatuses:[]ContainerStatus{ContainerStatus{Name:my-hostname-delete-node,State:ContainerState{Waiting:nil,Running:&ContainerStateRunning{StartedAt:2020-01-16 21:18:53 +0000 UTC,},Terminated:nil,},LastTerminationState:ContainerState{Waiting:nil,Running:nil,Terminated:nil,},Ready:true,RestartCount:0,Image:gcr.io/kubernetes-e2e-test-images/agnhost:2.6,ImageID:docker-pullable://gcr.io/kubernetes-e2e-test-images/agnhost@sha256:4057a5580c7b59c4fe10d8ab2732c9dec35eea80fd41f7bafc7bd5acc7edf727,ContainerID:docker://ef937af9ea395a06cc3956869e00ca95ff89acfa19f9a84da27ca2bce4619f10,Started:*true,},},QOSClass:BestEffort,InitContainerStatuses:[]ContainerStatus{},NominatedNodeName:,PodIPs:[]PodIP{PodIP{IP:10.64.2.245,},},EphemeralContainerStatuses:[]ContainerStatus{},},},},}",
    }
    failed to wait for pods responding: pod with UID 4007a68d-e411-467e-80c1-7b09ebc6e9cd is no longer a member of the replica set.  Must have been restarted for some reason.  Current replica set: &PodList{ListMeta:{/api/v1/namespaces/resize-nodes-5962/pods 69225  <nil>},Items:[]Pod{Pod{ObjectMeta:{my-hostname-delete-node-7ffk2 my-hostname-delete-node- resize-nodes-5962 /api/v1/namespaces/resize-nodes-5962/pods/my-hostname-delete-node-7ffk2 d60bd1bd-e74d-4d84-adad-63ee6157e892 69133 0 2020-01-16 21:23:23 +0000 UTC <nil> <nil> map[name:my-hostname-delete-node] map[] [{v1 ReplicationController my-hostname-delete-node 5844a37c-bdba-4555-adc9-b5b4e7512845 0xc003d499be 0xc003d499bf}] []  []},Spec:PodSpec{Volumes:[]Volume{Volume{Name:default-token-b46sh,VolumeSource:VolumeSource{HostPath:nil,EmptyDir:nil,GCEPersistentDisk:nil,AWSElasticBlockStore:nil,GitRepo:nil,Secret:&SecretVolumeSource{SecretName:default-token-b46sh,Items:[]KeyToPath{},DefaultMode:*420,Optional:nil,},NFS:nil,ISCSI:nil,Glusterfs:nil,PersistentVolumeClaim:nil,RBD:nil,FlexVolume:nil,Cinder:nil,CephFS:nil,Flocker:nil,DownwardAPI:nil,FC:nil,AzureFile:nil,ConfigMap:nil,VsphereVolume:nil,Quobyte:nil,AzureDisk:nil,PhotonPersistentDisk:nil,PortworxVolume:nil,ScaleIO:nil,Projected:nil,StorageOS:nil,CSI:nil,},},},Containers:[]Container{Container{Name:my-hostname-delete-node,Image:gcr.io/kubernetes-e2e-test-images/agnhost:2.6,Command:[],Args:[serve-hostname],WorkingDir:,Ports:[]ContainerPort{ContainerPort{Name:,HostPort:0,ContainerPort:9376,Protocol:TCP,HostIP:,},},Env:[]EnvVar{},Resources:ResourceRequirements{Limits:ResourceList{},Requests:ResourceList{},},VolumeMounts:[]VolumeMount{VolumeMount{Name:default-token-b46sh,ReadOnly:true,MountPath:/var/run/secrets/kubernetes.io/serviceaccount,SubPath:,MountPropagation:nil,SubPathExpr:,},},LivenessProbe:nil,ReadinessProbe:nil,Lifecycle:nil,TerminationMessagePath:/dev/termination-log,ImagePullPolicy:IfNotPresent,SecurityContext:nil,Stdin:false,StdinOnce:false,TTY:false,EnvFrom:[]EnvFromSource{},TerminationMessagePolicy:File,VolumeDevices:[]VolumeDevice{},StartupProbe:nil,},},RestartPolicy:Always,TerminationGracePeriodSeconds:*0,ActiveDeadlineSeconds:nil,DNSPolicy:ClusterFirst,NodeSelector:map[string]string{},ServiceAccountName:default,DeprecatedServiceAccount:default,NodeName:bootstrap-e2e-minion-group-z472,HostNetwork:false,HostPID:false,HostIPC:false,SecurityContext:&PodSecurityContext{SELinuxOptions:nil,RunAsUser:nil,RunAsNonRoot:nil,SupplementalGroups:[],FSGroup:nil,RunAsGroup:nil,Sysctls:[]Sysctl{},WindowsOptions:nil,},ImagePullSecrets:[]LocalObjectReference{},Hostname:,Subdomain:,Affinity:nil,SchedulerName:default-scheduler,InitContainers:[]Container{},AutomountServiceAccountToken:nil,Tolerations:[]Toleration{Toleration{Key:node.kubernetes.io/not-ready,Operator:Exists,Value:,Effect:NoExecute,TolerationSeconds:*300,},Toleration{Key:node.kubernetes.io/unreachable,Operator:Exists,Value:,Effect:NoExecute,TolerationSeconds:*300,},},HostAliases:[]HostAlias{},PriorityClassName:,Priority:*0,DNSConfig:nil,ShareProcessNamespace:nil,ReadinessGates:[]PodReadinessGate{},RuntimeClassName:nil,EnableServiceLinks:*true,PreemptionPolicy:nil,Overhead:ResourceList{},TopologySpreadConstraints:[]TopologySpreadConstraint{},EphemeralContainers:[]EphemeralContainer{},},Status:PodStatus{Phase:Running,Conditions:[]PodCondition{PodCondition{Type:Initialized,Status:True,LastProbeTime:0001-01-01 00:00:00 +0000 UTC,LastTransitionTime:2020-01-16 21:23:23 +0000 UTC,Reason:,Message:,},PodCondition{Type:Ready,Status:True,LastProbeTime:0001-01-01 00:00:00 +0000 UTC,LastTransitionTime:2020-01-16 21:23:25 +0000 UTC,Reason:,Message:,},PodCondition{Type:ContainersReady,Status:True,LastProbeTime:0001-01-01 00:00:00 +0000 UTC,LastTransitionTime:2020-01-16 21:23:25 +0000 UTC,Reason:,Message:,},PodCondition{Type:PodScheduled,Status:True,LastProbeTime:0001-01-01 00:00:00 +0000 UTC,LastTransitionTime:2020-01-16 21:23:23 +0000 UTC,Reason:,Message:,},},Message:,Reason:,HostIP:10.138.0.4,PodIP:10.64.2.246,StartTime:2020-01-16 21:23:23 +0000 UTC,ContainerStatuses:[]ContainerStatus{ContainerStatus{Name:my-hostname-delete-node,State:ContainerState{Waiting:nil,Running:&ContainerStateRunning{StartedAt:2020-01-16 21:23:25 +0000 UTC,},Terminated:nil,},LastTerminationState:ContainerState{Waiting:nil,Running:nil,Terminated:nil,},Ready:true,RestartCount:0,Image:gcr.io/kubernetes-e2e-test-images/agnhost:2.6,ImageID:docker-pullable://gcr.io/kubernetes-e2e-test-images/agnhost@sha256:4057a5580c7b59c4fe10d8ab2732c9dec35eea80fd41f7bafc7bd5acc7edf727,ContainerID:docker://978d0cddc0adef9ebaa07aa76223e275de2ec45c6a6674929d1342e3b8101f56,Started:*true,},},QOSClass:BestEffort,InitContainerStatuses:[]ContainerStatus{},NominatedNodeName:,PodIPs:[]PodIP{PodIP{IP:10.64.2.246,},},EphemeralContainerStatuses:[]ContainerStatus{},},},Pod{ObjectMeta:{my-hostname-delete-node-fggj9 my-hostname-delete-node- resize-nodes-5962 /api/v1/namespaces/resize-nodes-5962/pods/my-hostname-delete-node-fggj9 f3b37e06-5e30-4106-8a7f-62ffd13bcf5b 68304 0 2020-01-16 21:18:52 +0000 UTC <nil> <nil> map[name:my-hostname-delete-node] map[] [{v1 ReplicationController my-hostname-delete-node 5844a37c-bdba-4555-adc9-b5b4e7512845 0xc003d49af6 0xc003d49af7}] []  []},Spec:PodSpec{Volumes:[]Volume{Volume{Name:default-token-b46sh,VolumeSource:VolumeSource{HostPath:nil,EmptyDir:nil,GCEPersistentDisk:nil,AWSElasticBlockStore:nil,GitRepo:nil,Secret:&SecretVolumeSource{SecretName:default-token-b46sh,Items:[]KeyToPath{},DefaultMode:*420,Optional:nil,},NFS:nil,ISCSI:nil,Glusterfs:nil,PersistentVolumeClaim:nil,RBD:nil,FlexVolume:nil,Cinder:nil,CephFS:nil,Flocker:nil,DownwardAPI:nil,FC:nil,AzureFile:nil,ConfigMap:nil,VsphereVolume:nil,Quobyte:nil,AzureDisk:nil,PhotonPersistentDisk:nil,PortworxVolume:nil,ScaleIO:nil,Projected:nil,StorageOS:nil,CSI:nil,},},},Containers:[]Container{Container{Name:my-hostname-delete-node,Image:gcr.io/kubernetes-e2e-test-images/agnhost:2.6,Command:[],Args:[serve-hostname],WorkingDir:,Ports:[]ContainerPort{ContainerPort{Name:,HostPort:0,ContainerPort:9376,Protocol:TCP,HostIP:,},},Env:[]EnvVar{},Resources:ResourceRequirements{Limits:ResourceList{},Requests:ResourceList{},},VolumeMounts:[]VolumeMount{VolumeMount{Name:default-token-b46sh,ReadOnly:true,MountPath:/var/run/secrets/kubernetes.io/serviceaccount,SubPath:,MountPropagation:nil,SubPathExpr:,},},LivenessProbe:nil,ReadinessProbe:nil,Lifecycle:nil,TerminationMessagePath:/dev/termination-log,ImagePullPolicy:IfNotPresent,SecurityContext:nil,Stdin:false,StdinOnce:false,TTY:false,EnvFrom:[]EnvFromSource{},TerminationMessagePolicy:File,VolumeDevices:[]VolumeDevice{},StartupProbe:nil,},},RestartPolicy:Always,TerminationGracePeriodSeconds:*0,ActiveDeadlineSeconds:nil,DNSPolicy:ClusterFirst,NodeSelector:map[string]string{},ServiceAccountName:default,DeprecatedServiceAccount:default,NodeName:bootstrap-e2e-minion-group-zx21,HostNetwork:false,HostPID:false,HostIPC:false,SecurityContext:&PodSecurityContext{SELinuxOptions:nil,RunAsUser:nil,RunAsNonRoot:nil,SupplementalGroups:[],FSGroup:nil,RunAsGroup:nil,Sysctls:[]Sysctl{},WindowsOptions:nil,},ImagePullSecrets:[]LocalObjectReference{},Hostname:,Subdomain:,Affinity:nil,SchedulerName:default-scheduler,InitContainers:[]Container{},AutomountServiceAccountToken:nil,Tolerations:[]Toleration{Toleration{Key:node.kubernetes.io/not-ready,Operator:Exists,Value:,Effect:NoExecute,TolerationSeconds:*300,},Toleration{Key:node.kubernetes.io/unreachable,Operator:Exists,Value:,Effect:NoExecute,TolerationSeconds:*300,},},HostAliases:[]HostAlias{},PriorityClassName:,Priority:*0,DNSConfig:nil,ShareProcessNamespace:nil,ReadinessGates:[]PodReadinessGate{},RuntimeClassName:nil,EnableServiceLinks:*true,PreemptionPolicy:nil,Overhead:ResourceList{},TopologySpreadConstraints:[]TopologySpreadConstraint{},EphemeralContainers:[]EphemeralContainer{},},Status:PodStatus{Phase:Running,Conditions:[]PodCondition{PodCondition{Type:Initialized,Status:True,LastProbeTime:0001-01-01 00:00:00 +0000 UTC,LastTransitionTime:2020-01-16 21:18:52 +0000 UTC,Reason:,Message:,},PodCondition{Type:Ready,Status:True,LastProbeTime:0001-01-01 00:00:00 +0000 UTC,LastTransitionTime:2020-01-16 21:18:54 +0000 UTC,Reason:,Message:,},PodCondition{Type:ContainersReady,Status:True,LastProbeTime:0001-01-01 00:00:00 +0000 UTC,LastTransitionTime:2020-01-16 21:18:54 +0000 UTC,Reason:,Message:,},PodCondition{Type:PodScheduled,Status:True,LastProbeTime:0001-01-01 00:00:00 +0000 UTC,LastTransitionTime:2020-01-16 21:18:52 +0000 UTC,Reason:,Message:,},},Message:,Reason:,HostIP:10.138.0.5,PodIP:10.64.1.219,StartTime:2020-01-16 21:18:52 +0000 UTC,ContainerStatuses:[]ContainerStatus{ContainerStatus{Name:my-hostname-delete-node,State:ContainerState{Waiting:nil,Running:&ContainerStateRunning{StartedAt:2020-01-16 21:18:53 +0000 UTC,},Terminated:nil,},LastTerminationState:ContainerState{Waiting:nil,Running:nil,Terminated:nil,},Ready:true,RestartCount:0,Image:gcr.io/kubernetes-e2e-test-images/agnhost:2.6,ImageID:docker-pullable://gcr.io/kubernetes-e2e-test-images/agnhost@sha256:4057a5580c7b59c4fe10d8ab2732c9dec35eea80fd41f7bafc7bd5acc7edf727,ContainerID:docker://f11888dd31cef81ce4282b9f17ab34aa35f89870cae3ed9b29f7a343a13a54b0,Started:*true,},},QOSClass:BestEffort,InitContainerStatuses:[]ContainerStatus{},NominatedNodeName:,PodIPs:[]PodIP{PodIP{IP:10.64.1.219,},},EphemeralContainerStatuses:[]ContainerStatus{},},},Pod{ObjectMeta:{my-hostname-delete-node-rzj9m my-hostname-delete-node- resize-nodes-5962 /api/v1/namespaces/resize-nodes-5962/pods/my-hostname-delete-node-rzj9m 83a82417-bd87-4230-b370-54bbfa9b990a 68301 0 2020-01-16 21:18:52 +0000 UTC <nil> <nil> map[name:my-hostname-delete-node] map[] [{v1 ReplicationController my-hostname-delete-node 5844a37c-bdba-4555-adc9-b5b4e7512845 0xc003d49c26 0xc003d49c27}] []  []},Spec:PodSpec{Volumes:[]Volume{Volume{Name:default-token-b46sh,VolumeSource:VolumeSource{HostPath:nil,EmptyDir:nil,GCEPersistentDisk:nil,AWSElasticBlockStore:nil,GitRepo:nil,Secret:&SecretVolumeSource{SecretName:default-token-b46sh,Items:[]KeyToPath{},DefaultMode:*420,Optional:nil,},NFS:nil,ISCSI:nil,Glusterfs:nil,PersistentVolumeClaim:nil,RBD:nil,FlexVolume:nil,Cinder:nil,CephFS:nil,Flocker:nil,DownwardAPI:nil,FC:nil,AzureFile:nil,ConfigMap:nil,VsphereVolume:nil,Quobyte:nil,AzureDisk:nil,PhotonPersistentDisk:nil,PortworxVolume:nil,ScaleIO:nil,Projected:nil,StorageOS:nil,CSI:nil,},},},Containers:[]Container{Container{Name:my-hostname-delete-node,Image:gcr.io/kubernetes-e2e-test-images/agnhost:2.6,Command:[],Args:[serve-hostname],WorkingDir:,Ports:[]ContainerPort{ContainerPort{Name:,HostPort:0,ContainerPort:9376,Protocol:TCP,HostIP:,},},Env:[]EnvVar{},Resources:ResourceRequirements{Limits:ResourceList{},Requests:ResourceList{},},VolumeMounts:[]VolumeMount{VolumeMount{Name:default-token-b46sh,ReadOnly:true,MountPath:/var/run/secrets/kubernetes.io/serviceaccount,SubPath:,MountPropagation:nil,SubPathExpr:,},},LivenessProbe:nil,ReadinessProbe:nil,Lifecycle:nil,TerminationMessagePath:/dev/termination-log,ImagePullPolicy:IfNotPresent,SecurityContext:nil,Stdin:false,StdinOnce:false,TTY:false,EnvFrom:[]EnvFromSource{},TerminationMessagePolicy:File,VolumeDevices:[]VolumeDevice{},StartupProbe:nil,},},RestartPolicy:Always,TerminationGracePeriodSeconds:*0,ActiveDeadlineSeconds:nil,DNSPolicy:ClusterFirst,NodeSelector:map[string]string{},ServiceAccountName:default,DeprecatedServiceAccount:default,NodeName:bootstrap-e2e-minion-group-z472,HostNetwork:false,HostPID:false,HostIPC:false,SecurityContext:&PodSecurityContext{SELinuxOptions:nil,RunAsUser:nil,RunAsNonRoot:nil,SupplementalGroups:[],FSGroup:nil,RunAsGroup:nil,Sysctls:[]Sysctl{},WindowsOptions:nil,},ImagePullSecrets:[]LocalObjectReference{},Hostname:,Subdomain:,Affinity:nil,SchedulerName:default-scheduler,InitContainers:[]Container{},AutomountServiceAccountToken:nil,Tolerations:[]Toleration{Toleration{Key:node.kubernetes.io/not-ready,Operator:Exists,Value:,Effect:NoExecute,TolerationSeconds:*300,},Toleration{Key:node.kubernetes.io/unreachable,Operator:Exists,Value:,Effect:NoExecute,TolerationSeconds:*300,},},HostAliases:[]HostAlias{},PriorityClassName:,Priority:*0,DNSConfig:nil,ShareProcessNamespace:nil,ReadinessGates:[]PodReadinessGate{},RuntimeClassName:nil,EnableServiceLinks:*true,PreemptionPolicy:nil,Overhead:ResourceList{},TopologySpreadConstraints:[]TopologySpreadConstraint{},EphemeralContainers:[]EphemeralContainer{},},Status:PodStatus{Phase:Running,Conditions:[]PodCondition{PodCondition{Type:Initialized,Status:True,LastProbeTime:0001-01-01 00:00:00 +0000 UTC,LastTransitionTime:2020-01-16 21:18:52 +0000 UTC,Reason:,Message:,},PodCondition{Type:Ready,Status:True,LastProbeTime:0001-01-01 00:00:00 +0000 UTC,LastTransitionTime:2020-01-16 21:18:54 +0000 UTC,Reason:,Message:,},PodCondition{Type:ContainersReady,Status:True,LastProbeTime:0001-01-01 00:00:00 +0000 UTC,LastTransitionTime:2020-01-16 21:18:54 +0000 UTC,Reason:,Message:,},PodCondition{Type:PodScheduled,Status:True,LastProbeTime:0001-01-01 00:00:00 +0000 UTC,LastTransitionTime:2020-01-16 21:18:52 +0000 UTC,Reason:,Message:,},},Message:,Reason:,HostIP:10.138.0.4,PodIP:10.64.2.245,StartTime:2020-01-16 21:18:52 +0000 UTC,ContainerStatuses:[]ContainerStatus{ContainerStatus{Name:my-hostname-delete-node,State:ContainerState{Waiting:nil,Running:&ContainerStateRunning{StartedAt:2020-01-16 21:18:53 +0000 UTC,},Terminated:nil,},LastTerminationState:ContainerState{Waiting:nil,Running:nil,Terminated:nil,},Ready:true,RestartCount:0,Image:gcr.io/kubernetes-e2e-test-images/agnhost:2.6,ImageID:docker-pullable://gcr.io/kubernetes-e2e-test-images/agnhost@sha256:4057a5580c7b59c4fe10d8ab2732c9dec35eea80fd41f7bafc7bd5acc7edf727,ContainerID:docker://ef937af9ea395a06cc3956869e00ca95ff89acfa19f9a84da27ca2bce4619f10,Started:*true,},},QOSClass:BestEffort,InitContainerStatuses:[]ContainerStatus{},NominatedNodeName:,PodIPs:[]PodIP{PodIP{IP:10.64.2.245,},},EphemeralContainerStatuses:[]ContainerStatus{},},},},}
occurred
/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/lifecycle/resize_nodes.go:138
				
				Click to see stdout/stderrfrom junit_01.xml

Find with mentions in log files | View test history on testgrid


Master upgrade [sig-cloud-provider-gcp] master-upgrade 4m55s

go run hack/e2e.go -v --test --test_args='--ginkgo.focus=Master\supgrade\s\[sig\-cloud\-provider\-gcp\]\smaster\-upgrade$'
Jan 16 16:58:31.838: Unexpected error:
    <*errors.errorString | 0xc0022822f0>: {
        s: "error running /workspace/kubernetes_skew/cluster/gce/upgrade.sh [-M v1.17.2-beta.0]; got error exit status 1, stdout \"Fetching the previously installed CoreDNS version\\n\\n***WARNING***\\nUpgrading Kubernetes with this script might result in an upgrade to a new etcd version.\\nSome etcd version upgrades, such as 3.0.x to 3.1.x, DO NOT offer a downgrade path.\\nTo pin the etcd version to your current one (e.g. v3.0.17), set the following variables\\nbefore running this script:\\n\\n# example: pin to etcd v3.0.17\\nexport ETCD_IMAGE=3.0.17\\nexport ETCD_VERSION=3.0.17\\n\\nAlternatively, if you choose to allow an etcd upgrade that doesn't support downgrade,\\nyou might still be able to downgrade Kubernetes by pinning to the newer etcd version.\\nIn all cases, it is strongly recommended to have an etcd backup before upgrading.\\n\\n== Pre-Upgrade Node OS and Kubelet Versions ==\\nname: \\\"bootstrap-e2e-master\\\", osImage: \\\"Container-Optimized OS from Google\\\", kubeletVersion: \\\"v1.16.5-beta.1.51+e7f962ba86f4ce\\\"\\nname: \\\"bootstrap-e2e-minion-group-fbnv\\\", osImage: \\\"Container-Optimized OS from Google\\\", kubeletVersion: \\\"v1.16.5-beta.1.51+e7f962ba86f4ce\\\"\\nname: \\\"bootstrap-e2e-minion-group-z472\\\", osImage: \\\"Container-Optimized OS from Google\\\", kubeletVersion: \\\"v1.16.5-beta.1.51+e7f962ba86f4ce\\\"\\nname: \\\"bootstrap-e2e-minion-group-zx21\\\", osImage: \\\"Container-Optimized OS from Google\\\", kubeletVersion: \\\"v1.16.5-beta.1.51+e7f962ba86f4ce\\\"\\nFound subnet for region us-west1 in network bootstrap-e2e: bootstrap-e2e\\n== Upgrading master to 'https://storage.googleapis.com/kubernetes-release/release/v1.17.2-beta.0/kubernetes-server-linux-amd64.tar.gz'. Do not interrupt, deleting master instance. ==\\n== Upgrading master environment variables. ==\\n== Waiting for new master to respond to API requests ==\\n......................== Done ==\\nWaiting for CoreDNS to update\\nFetching the latest installed CoreDNS version\\n== Downloading the CoreDNS migration tool ==\\n== Upgrading the CoreDNS ConfigMap ==\\nconfigmap/coredns configured\\n== The CoreDNS Config has been updated ==\\n== Validating cluster post-upgrade ==\\nValidating gce cluster, MULTIZONE=\\nFound 4 node(s).\\nNAME                              STATUS                     ROLES    AGE     VERSION\\nbootstrap-e2e-master              Ready,SchedulingDisabled   <none>   9m10s   v1.17.1\\nbootstrap-e2e-minion-group-fbnv   Ready                      <none>   9m13s   v1.16.5-beta.1.51+e7f962ba86f4ce\\nbootstrap-e2e-minion-group-z472   Ready                      <none>   9m13s   v1.16.5-beta.1.51+e7f962ba86f4ce\\nbootstrap-e2e-minion-group-zx21   Ready                      <none>   9m13s   v1.16.5-beta.1.51+e7f962ba86f4ce\\nValidate output:\\nNAME                 STATUS    MESSAGE             ERROR\\nscheduler            Healthy   ok                  \\netcd-0               Healthy   {\\\"health\\\":\\\"true\\\"}   \\netcd-1               Healthy   {\\\"health\\\":\\\"true\\\"}   \\ncontroller-manager   Healthy   ok                  \\n\\x1b[0;32mCluster validation succeeded\\x1b[0m\\n== Post-Upgrade Node OS and Kubelet Versions ==\\nname: \\\"bootstrap-e2e-master\\\", osImage: \\\"Container-Optimized OS from Google\\\", kubeletVersion: \\\"v1.17.1\\\"\\nname: \\\"bootstrap-e2e-minion-group-fbnv\\\", osImage: \\\"Container-Optimized OS from Google\\\", kubeletVersion: \\\"v1.16.5-beta.1.51+e7f962ba86f4ce\\\"\\nname: \\\"bootstrap-e2e-minion-group-z472\\\", osImage: \\\"Container-Optimized OS from Google\\\", kubeletVersion: \\\"v1.16.5-beta.1.51+e7f962ba86f4ce\\\"\\nname: \\\"bootstrap-e2e-minion-group-zx21\\\", osImage: \\\"Container-Optimized OS from Google\\\", kubeletVersion: \\\"v1.16.5-beta.1.51+e7f962ba86f4ce\\\"\\n\", stderr \"Project: kube-gce-upg-1-4-1-5-upg-clu-n\\nNetwork Project: kube-gce-upg-1-4-1-5-upg-clu-n\\nZone: us-west1-b\\nINSTANCE_GROUPS=bootstrap-e2e-minion-group\\nNODE_NAMES=bootstrap-e2e-minion-group-fbnv bootstrap-e2e-minion-group-z472 bootstrap-e2e-minion-group-zx21\\nWARNING: Some requests did not succeed.\\n - Invalid value for field 'zone': 'asia-northeast3-a'. Unknown zone.\\n - Invalid value for field 'zone': 'asia-northeast3-b'. Unknown zone.\\n - Invalid value for field 'zone': 'asia-northeast3-c'. Unknown zone.\\n\\nTrying to find master named 'bootstrap-e2e-master'\\nLooking for address 'bootstrap-e2e-master-ip'\\nUsing master: bootstrap-e2e-master (external IP: 34.83.130.135; internal IP: (not set))\\nDeleted [https://www.googleapis.com/compute/v1/projects/kube-gce-upg-1-4-1-5-upg-clu-n/zones/us-west1-b/instances/bootstrap-e2e-master].\\nWARNING: You have selected a disk size of under [200GB]. This may result in poor I/O performance. For more information, see: https://developers.google.com/compute/docs/disks#performance.\\nCreated [https://www.googleapis.com/compute/v1/projects/kube-gce-upg-1-4-1-5-upg-clu-n/zones/us-west1-b/instances/bootstrap-e2e-master].\\nWARNING: Some requests generated warnings:\\n - Disk size: '20 GB' is larger than image size: '10 GB'. You might need to resize the root repartition manually if the operating system does not support automatic resizing. See https://cloud.google.com/compute/docs/disks/add-persistent-disk#resize_pd for details.\\n - The resource 'projects/cos-cloud/global/images/cos-77-12371-89-0' is deprecated. A suggested replacement is 'projects/cos-cloud/global/images/cos-77-12371-114-0'.\\n\\nNAME                  ZONE        MACHINE_TYPE   PREEMPTIBLE  INTERNAL_IP  EXTERNAL_IP    STATUS\\nbootstrap-e2e-master  us-west1-b  n1-standard-1               10.138.0.6   34.83.130.135  RUNNING\\nWarning: kubectl apply should be used on resource created by either kubectl create --save-config or kubectl apply\\nProject: kube-gce-upg-1-4-1-5-upg-clu-n\\nNetwork Project: kube-gce-upg-1-4-1-5-upg-clu-n\\nZone: us-west1-b\\nWARNING: Some requests did not succeed.\\n - Invalid value for field 'zone': 'asia-northeast3-a'. Unknown zone.\\n - Invalid value for field 'zone': 'asia-northeast3-b'. Unknown zone.\\n - Invalid value for field 'zone': 'asia-northeast3-c'. Unknown zone.\\n\\n/workspace/kubernetes_skew/cluster/gce/upgrade.sh: line 452: download_dir: unbound variable\\n\"",
    }
    error running /workspace/kubernetes_skew/cluster/gce/upgrade.sh [-M v1.17.2-beta.0]; got error exit status 1, stdout "Fetching the previously installed CoreDNS version\n\n***WARNING***\nUpgrading Kubernetes with this script might result in an upgrade to a new etcd version.\nSome etcd version upgrades, such as 3.0.x to 3.1.x, DO NOT offer a downgrade path.\nTo pin the etcd version to your current one (e.g. v3.0.17), set the following variables\nbefore running this script:\n\n# example: pin to etcd v3.0.17\nexport ETCD_IMAGE=3.0.17\nexport ETCD_VERSION=3.0.17\n\nAlternatively, if you choose to allow an etcd upgrade that doesn't support downgrade,\nyou might still be able to downgrade Kubernetes by pinning to the newer etcd version.\nIn all cases, it is strongly recommended to have an etcd backup before upgrading.\n\n== Pre-Upgrade Node OS and Kubelet Versions ==\nname: \"bootstrap-e2e-master\", osImage: \"Container-Optimized OS from Google\", kubeletVersion: \"v1.16.5-beta.1.51+e7f962ba86f4ce\"\nname: \"bootstrap-e2e-minion-group-fbnv\", osImage: \"Container-Optimized OS from Google\", kubeletVersion: \"v1.16.5-beta.1.51+e7f962ba86f4ce\"\nname: \"bootstrap-e2e-minion-group-z472\", osImage: \"Container-Optimized OS from Google\", kubeletVersion: \"v1.16.5-beta.1.51+e7f962ba86f4ce\"\nname: \"bootstrap-e2e-minion-group-zx21\", osImage: \"Container-Optimized OS from Google\", kubeletVersion: \"v1.16.5-beta.1.51+e7f962ba86f4ce\"\nFound subnet for region us-west1 in network bootstrap-e2e: bootstrap-e2e\n== Upgrading master to 'https://storage.googleapis.com/kubernetes-release/release/v1.17.2-beta.0/kubernetes-server-linux-amd64.tar.gz'. Do not interrupt, deleting master instance. ==\n== Upgrading master environment variables. ==\n== Waiting for new master to respond to API requests ==\n......................== Done ==\nWaiting for CoreDNS to update\nFetching the latest installed CoreDNS version\n== Downloading the CoreDNS migration tool ==\n== Upgrading the CoreDNS ConfigMap ==\nconfigmap/coredns configured\n== The CoreDNS Config has been updated ==\n== Validating cluster post-upgrade ==\nValidating gce cluster, MULTIZONE=\nFound 4 node(s).\nNAME                              STATUS                     ROLES    AGE     VERSION\nbootstrap-e2e-master              Ready,SchedulingDisabled   <none>   9m10s   v1.17.1\nbootstrap-e2e-minion-group-fbnv   Ready                      <none>   9m13s   v1.16.5-beta.1.51+e7f962ba86f4ce\nbootstrap-e2e-minion-group-z472   Ready                      <none>   9m13s   v1.16.5-beta.1.51+e7f962ba86f4ce\nbootstrap-e2e-minion-group-zx21   Ready                      <none>   9m13s   v1.16.5-beta.1.51+e7f962ba86f4ce\nValidate output:\nNAME                 STATUS    MESSAGE             ERROR\nscheduler            Healthy   ok                  \netcd-0               Healthy   {\"health\":\"true\"}   \netcd-1               Healthy   {\"health\":\"true\"}   \ncontroller-manager   Healthy   ok                  \n\x1b[0;32mCluster validation succeeded\x1b[0m\n== Post-Upgrade Node OS and Kubelet Versions ==\nname: \"bootstrap-e2e-master\", osImage: \"Container-Optimized OS from Google\", kubeletVersion: \"v1.17.1\"\nname: \"bootstrap-e2e-minion-group-fbnv\", osImage: \"Container-Optimized OS from Google\", kubeletVersion: \"v1.16.5-beta.1.51+e7f962ba86f4ce\"\nname: \"bootstrap-e2e-minion-group-z472\", osImage: \"Container-Optimized OS from Google\", kubeletVersion: \"v1.16.5-beta.1.51+e7f962ba86f4ce\"\nname: \"bootstrap-e2e-minion-group-zx21\", osImage: \"Container-Optimized OS from Google\", kubeletVersion: \"v1.16.5-beta.1.51+e7f962ba86f4ce\"\n", stderr "Project: kube-gce-upg-1-4-1-5-upg-clu-n\nNetwork Project: kube-gce-upg-1-4-1-5-upg-clu-n\nZone: us-west1-b\nINSTANCE_GROUPS=bootstrap-e2e-minion-group\nNODE_NAMES=bootstrap-e2e-minion-group-fbnv bootstrap-e2e-minion-group-z472 bootstrap-e2e-minion-group-zx21\nWARNING: Some requests did not succeed.\n - Invalid value for field 'zone': 'asia-northeast3-a'. Unknown zone.\n - Invalid value for field 'zone': 'asia-northeast3-b'. Unknown zone.\n - Invalid value for field 'zone': 'asia-northeast3-c'. Unknown zone.\n\nTrying to find master named 'bootstrap-e2e-master'\nLooking for address 'bootstrap-e2e-master-ip'\nUsing master: bootstrap-e2e-master (external IP: 34.83.130.135; internal IP: (not set))\nDeleted [https://www.googleapis.com/compute/v1/projects/kube-gce-upg-1-4-1-5-upg-clu-n/zones/us-west1-b/instances/bootstrap-e2e-master].\nWARNING: You have selected a disk size of under [200GB]. This may result in poor I/O performance. For more information, see: https://developers.google.com/compute/docs/disks#performance.\nCreated [https://www.googleapis.com/compute/v1/projects/kube-gce-upg-1-4-1-5-upg-clu-n/zones/us-west1-b/instances/bootstrap-e2e-master].\nWARNING: Some requests generated warnings:\n - Disk size: '20 GB' is larger than image size: '10 GB'. You might need to resize the root repartition manually if the operating system does not support automatic resizing. See https://cloud.google.com/compute/docs/disks/add-persistent-disk#resize_pd for details.\n - The resource 'projects/cos-cloud/global/images/cos-77-12371-89-0' is deprecated. A suggested replacement is 'projects/cos-cloud/global/images/cos-77-12371-114-0'.\n\nNAME                  ZONE        MACHINE_TYPE   PREEMPTIBLE  INTERNAL_IP  EXTERNAL_IP    STATUS\nbootstrap-e2e-master  us-west1-b  n1-standard-1               10.138.0.6   34.83.130.135  RUNNING\nWarning: kubectl apply should be used on resource created by either kubectl create --save-config or kubectl apply\nProject: kube-gce-upg-1-4-1-5-upg-clu-n\nNetwork Project: kube-gce-upg-1-4-1-5-upg-clu-n\nZone: us-west1-b\nWARNING: Some requests did not succeed.\n - Invalid value for field 'zone': 'asia-northeast3-a'. Unknown zone.\n - Invalid value for field 'zone': 'asia-northeast3-b'. Unknown zone.\n - Invalid value for field 'zone': 'asia-northeast3-c'. Unknown zone.\n\n/workspace/kubernetes_skew/cluster/gce/upgrade.sh: line 452: download_dir: unbound variable\n"
occurred

k8s.io/kubernetes/test/e2e/cloud/gcp.glob..func2.1.1.1()
	/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/cloud/gcp/cluster_upgrade.go:106 +0x12c
k8s.io/kubernetes/test/e2e/chaosmonkey.(*Chaosmonkey).Do(0xc0035ed200)
	/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/chaosmonkey/chaosmonkey.go:111 +0x38a
k8s.io/kubernetes/test/e2e/cloud/gcp.runUpgradeSuite(0xc0009a57c0, 0x7bdf120, 0xc, 0xc, 0xc0007c5bf0, 0xc0017daff0, 0x0, 0xc00249fe40)
	/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/cloud/gcp/cluster_upgrade.go:479 +0x47a
k8s.io/kubernetes/test/e2e/cloud/gcp.glob..func2.1.1()
	/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/cloud/gcp/cluster_upgrade.go:109 +0x222
k8s.io/kubernetes/test/e2e.RunE2ETests(0xc0008b8400)
	_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/e2e.go:110 +0x30a
k8s.io/kubernetes/test/e2e.TestE2E(0xc0008b8400)
	_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/e2e_test.go:112 +0x2b
testing.tRunner(0xc0008b8400, 0x4c764b0)
	/usr/local/go/src/testing/testing.go:909 +0xc9
created by testing.(*T).Run
	/usr/local/go/src/testing/testing.go:960 +0x350
				from junit_upgradeupgrades.xml

Filter through log files | View test history on testgrid


Test 4h36m

error during ./hack/ginkgo-e2e.sh --ginkgo.focus=\[Serial\]|\[Disruptive\] --ginkgo.skip=\[Flaky\]|\[Feature:.+\] --kubectl-path=../../../../kubernetes_skew/cluster/kubectl.sh --minStartupPods=8 --report-dir=/workspace/_artifacts --disable-log-dump=true: exit status 1
				from junit_runner.xml

Filter through log files | View test history on testgrid


UpgradeTest 17m25s

error during kubetest --test --test_args=--ginkgo.focus=\[Feature:MasterUpgrade\] --upgrade-target=ci/k8s-beta --report-dir=/workspace/_artifacts --disable-log-dump=true --report-prefix=upgrade --check-version-skew=false: exit status 1
				from junit_runner.xml

Filter through log files | View test history on testgrid


Show 186 Passed Tests

Show 9416 Skipped Tests