PR #105271 pull-kubernetes-e2e-gce-100-performance #1509444298272149504
Recent runs || View in Spyglass
Recent runs || View in Spyglass
| PR | pacoxu: add --system-reserved support for swap |
| Result | ABORTED |
| Tests | 0 failed / 101 succeeded |
| Started | |
| Elapsed | 1h1m |
| Revision | 20135dff21c3c9f513aab8862504ce748f9cd70d |
| Refs |
105271 |
| job-version | v1.24.0-alpha.4.511+b0c22305ae4fa0 |
| kubetest-version | |
| revision | v1.24.0-alpha.4.511+b0c22305ae4fa0 |
ClusterLoaderV2 huge-service overall (testing/huge-service/config.yaml)
ClusterLoaderV2 huge-service: [step: 01] starting measurements
ClusterLoaderV2 huge-service: [step: 01] starting measurements / Measurement[00] - APIResponsivenessPrometheus - APIResponsivenessPrometheus
ClusterLoaderV2 huge-service: [step: 01] starting measurements / Measurement[01] - APIResponsivenessPrometheus - APIResponsivenessPrometheusSimple
ClusterLoaderV2 huge-service: [step: 01] starting measurements / Measurement[02] - TestMetrics - TestMetrics
ClusterLoaderV2 huge-service: [step: 01] starting measurements / Measurement[03] - InClusterNetworkLatency - InClusterNetworkLatency
ClusterLoaderV2 huge-service: [step: 02] Create huge-service
ClusterLoaderV2 huge-service: [step: 03] Creating huge-service measurements
ClusterLoaderV2 huge-service: [step: 03] Creating huge-service measurements / Measurement[00] - WaitForControlledPodsRunning - WaitForHugeServiceDeployments
ClusterLoaderV2 huge-service: [step: 04] Creating huge-service pods
ClusterLoaderV2 huge-service: [step: 05] Waiting for huge-service pods to be created
ClusterLoaderV2 huge-service: [step: 05] Waiting for huge-service pods to be created / Measurement[00] - WaitForControlledPodsRunning - WaitForHugeServiceDeployments
ClusterLoaderV2 huge-service: [step: 06] Deleting huge-service pods
ClusterLoaderV2 huge-service: [step: 07] Waiting for huge-service pods to be deleted
ClusterLoaderV2 huge-service: [step: 07] Waiting for huge-service pods to be deleted / Measurement[00] - WaitForControlledPodsRunning - WaitForHugeServiceDeployments
ClusterLoaderV2 huge-service: [step: 08] Delete huge-service
ClusterLoaderV2 huge-service: [step: 09] gathering measurements
ClusterLoaderV2 huge-service: [step: 09] gathering measurements / Measurement[00] - APIResponsivenessPrometheus - APIResponsivenessPrometheus
ClusterLoaderV2 huge-service: [step: 09] gathering measurements / Measurement[01] - APIResponsivenessPrometheus - APIResponsivenessPrometheusSimple
ClusterLoaderV2 huge-service: [step: 09] gathering measurements / Measurement[02] - TestMetrics - TestMetrics
ClusterLoaderV2 huge-service: [step: 09] gathering measurements / Measurement[03] - InClusterNetworkLatency - InClusterNetworkLatency
ClusterLoaderV2 load overall (testing/load/config.yaml)
ClusterLoaderV2 load: [step: 01] starting measurements
ClusterLoaderV2 load: [step: 01] starting measurements / Measurement[00] - APIResponsivenessPrometheus - APIResponsivenessPrometheus
ClusterLoaderV2 load: [step: 01] starting measurements / Measurement[01] - APIResponsivenessPrometheus - APIResponsivenessPrometheusSimple
ClusterLoaderV2 load: [step: 01] starting measurements / Measurement[02] - PodStartupLatency - CreatePhasePodStartupLatency
ClusterLoaderV2 load: [step: 01] starting measurements / Measurement[03] - InClusterNetworkLatency - InClusterNetworkLatency
ClusterLoaderV2 load: [step: 01] starting measurements / Measurement[04] - SLOMeasurement - SLOMeasurement
ClusterLoaderV2 load: [step: 01] starting measurements / Measurement[05] - NetworkProgrammingLatency - NetworkProgrammingLatency
ClusterLoaderV2 load: [step: 01] starting measurements / Measurement[06] - APIAvailability - APIAvailability
ClusterLoaderV2 load: [step: 01] starting measurements / Measurement[07] - TestMetrics - TestMetrics
ClusterLoaderV2 load: [step: 02] Creating k8s services
ClusterLoaderV2 load: [step: 03] Creating PriorityClass for DaemonSets
ClusterLoaderV2 load: [step: 04] create objects configmaps and secrets
ClusterLoaderV2 load: [step: 05] Starting measurement for 'create objects'
ClusterLoaderV2 load: [step: 05] Starting measurement for 'create objects' / Measurement[00] - WaitForControlledPodsRunning -
ClusterLoaderV2 load: [step: 06] create objects
ClusterLoaderV2 load: [step: 07] Waiting for 'create objects' to be completed
ClusterLoaderV2 load: [step: 07] Waiting for 'create objects' to be completed / Measurement[00] - WaitForControlledPodsRunning -
ClusterLoaderV2 load: [step: 08] Creating scheduler throughput measurements
ClusterLoaderV2 load: [step: 08] Creating scheduler throughput measurements / Measurement[00] - PodStartupLatency - HighThroughputPodStartupLatency
ClusterLoaderV2 load: [step: 08] Creating scheduler throughput measurements / Measurement[01] - WaitForControlledPodsRunning - WaitForSchedulerThroughputDeployments
ClusterLoaderV2 load: [step: 08] Creating scheduler throughput measurements / Measurement[02] - SchedulingThroughput - SchedulingThroughput
ClusterLoaderV2 load: [step: 09] create scheduler throughput pods
ClusterLoaderV2 load: [step: 10] Waiting for scheduler throughput pods to be created
ClusterLoaderV2 load: [step: 10] Waiting for scheduler throughput pods to be created / Measurement[00] - WaitForControlledPodsRunning - WaitForSchedulerThroughputDeployments
ClusterLoaderV2 load: [step: 11] Collecting scheduler throughput measurements
ClusterLoaderV2 load: [step: 11] Collecting scheduler throughput measurements / Measurement[00] - PodStartupLatency - HighThroughputPodStartupLatency
ClusterLoaderV2 load: [step: 11] Collecting scheduler throughput measurements / Measurement[01] - SchedulingThroughput - SchedulingThroughput
ClusterLoaderV2 load: [step: 12] delete scheduler throughput pods
ClusterLoaderV2 load: [step: 13] Waiting for scheduler throughput pods to be deleted
ClusterLoaderV2 load: [step: 13] Waiting for scheduler throughput pods to be deleted / Measurement[00] - WaitForControlledPodsRunning - WaitForSchedulerThroughputDeployments
ClusterLoaderV2 load: [step: 14] Starting latency pod measurements
ClusterLoaderV2 load: [step: 14] Starting latency pod measurements / Measurement[00] - PodStartupLatency - PodStartupLatency
ClusterLoaderV2 load: [step: 14] Starting latency pod measurements / Measurement[01] - WaitForControlledPodsRunning - WaitForRunningLatencyDeployments
ClusterLoaderV2 load: [step: 15] Creating latency pods
ClusterLoaderV2 load: [step: 16] Waiting for latency pods to be running
ClusterLoaderV2 load: [step: 16] Waiting for latency pods to be running / Measurement[00] - WaitForControlledPodsRunning - WaitForRunningLatencyDeployments
ClusterLoaderV2 load: [step: 17] Deleting latency pods
ClusterLoaderV2 load: [step: 18] Waiting for latency pods to be deleted
ClusterLoaderV2 load: [step: 18] Waiting for latency pods to be deleted / Measurement[00] - WaitForControlledPodsRunning - WaitForRunningLatencyDeployments
ClusterLoaderV2 load: [step: 19] Collecting pod startup latency
ClusterLoaderV2 load: [step: 19] Collecting pod startup latency / Measurement[00] - PodStartupLatency - PodStartupLatency
ClusterLoaderV2 load: [step: 20] Starting measurement for 'scale and update objects'
ClusterLoaderV2 load: [step: 20] Starting measurement for 'scale and update objects' / Measurement[00] - WaitForControlledPodsRunning -
ClusterLoaderV2 load: [step: 21] scale and update objects
ClusterLoaderV2 load: [step: 22] Waiting for 'scale and update objects' to be completed
ClusterLoaderV2 load: [step: 22] Waiting for 'scale and update objects' to be completed / Measurement[00] - WaitForControlledPodsRunning -
ClusterLoaderV2 load: [step: 23] Starting measurement for 'delete objects'
ClusterLoaderV2 load: [step: 23] Starting measurement for 'delete objects' / Measurement[00] - WaitForControlledPodsRunning -
ClusterLoaderV2 load: [step: 24] delete objects
ClusterLoaderV2 load: [step: 25] Waiting for 'delete objects' to be completed
ClusterLoaderV2 load: [step: 25] Waiting for 'delete objects' to be completed / Measurement[00] - WaitForControlledPodsRunning -
ClusterLoaderV2 load: [step: 25] Waiting for 'delete objects' to be completed / Measurement[01] - WaitForBoundPVCs - WaitForPVCsToBeDeleted
ClusterLoaderV2 load: [step: 26] delete objects configmaps and secrets
ClusterLoaderV2 load: [step: 27] Deleting PriorityClass for DaemonSets
ClusterLoaderV2 load: [step: 28] Deleting k8s services
ClusterLoaderV2 load: [step: 29] gathering measurements
ClusterLoaderV2 load: [step: 29] gathering measurements / Measurement[00] - APIResponsivenessPrometheus - APIResponsivenessPrometheus
ClusterLoaderV2 load: [step: 29] gathering measurements / Measurement[01] - APIResponsivenessPrometheus - APIResponsivenessPrometheusSimple
ClusterLoaderV2 load: [step: 29] gathering measurements / Measurement[02] - PodStartupLatency - CreatePhasePodStartupLatency
ClusterLoaderV2 load: [step: 29] gathering measurements / Measurement[03] - InClusterNetworkLatency - InClusterNetworkLatency
ClusterLoaderV2 load: [step: 29] gathering measurements / Measurement[04] - SLOMeasurement - SLOMeasurement
ClusterLoaderV2 load: [step: 29] gathering measurements / Measurement[05] - NetworkProgrammingLatency - NetworkProgrammingLatency
ClusterLoaderV2 load: [step: 29] gathering measurements / Measurement[06] - APIAvailability - APIAvailability
ClusterLoaderV2 load: [step: 29] gathering measurements / Measurement[07] - TestMetrics - TestMetrics
kubetest Build
kubetest Check APIReachability
kubetest ClusterLoaderV2
kubetest Deferred TearDown
kubetest DumpClusterLogs
kubetest Extract
kubetest GetDeployer
kubetest Prepare
kubetest Stage
kubetest TearDown
kubetest TearDown Previous
kubetest Timeout
kubetest Up
kubetest list nodes
kubetest test setup
... skipping 717 lines ... Looking for address 'e2e-105271-95a39-master-ip' Looking for address 'e2e-105271-95a39-master-internal-ip' Using master: e2e-105271-95a39-master (external IP: 34.139.46.175; internal IP: 10.40.0.2) Waiting up to 300 seconds for cluster initialization. This will continually check to see if the API for kubernetes is reachable. This may time out if there was some uncaught error during start up. Kubernetes cluster created. Cluster "k8s-infra-e2e-boskos-scale-16_e2e-105271-95a39" set. User "k8s-infra-e2e-boskos-scale-16_e2e-105271-95a39" set. Context "k8s-infra-e2e-boskos-scale-16_e2e-105271-95a39" created. Switched to context "k8s-infra-e2e-boskos-scale-16_e2e-105271-95a39". ... skipping 228 lines ... e2e-105271-95a39-minion-group-xxr7 Ready <none> 65s v1.24.0-alpha.4.511+b0c22305ae4fa0 e2e-105271-95a39-minion-group-zm7r Ready <none> 65s v1.24.0-alpha.4.511+b0c22305ae4fa0 e2e-105271-95a39-minion-heapster Ready <none> 80s v1.24.0-alpha.4.511+b0c22305ae4fa0 Warning: v1 ComponentStatus is deprecated in v1.19+ Validate output: Warning: v1 ComponentStatus is deprecated in v1.19+ NAME STATUS MESSAGE ERROR etcd-1 Healthy {"health":"true","reason":""} etcd-0 Healthy {"health":"true","reason":""} scheduler Healthy ok controller-manager Healthy ok [0;33mCluster validation encountered some problems, but cluster should be in working order[0m ...ignoring non-fatal errors in validate-cluster Done, listing cluster services: [0;32mKubernetes control plane[0m is running at [0;33mhttps://34.139.46.175[0m [0;32mGLBCDefaultBackend[0m is running at [0;33mhttps://34.139.46.175/api/v1/namespaces/kube-system/services/default-http-backend:http/proxy[0m [0;32mCoreDNS[0m is running at [0;33mhttps://34.139.46.175/api/v1/namespaces/kube-system/services/kube-dns:dns/proxy[0m [0;32mMetrics-server[0m is running at [0;33mhttps://34.139.46.175/api/v1/namespaces/kube-system/services/https:metrics-server:/proxy[0m ... skipping 430 lines ... I0331 08:45:15.804594 84938 util.go:93] Not enough active targets (0), expected at least (6), waiting for more to become active... I0331 08:45:30.594322 84938 util.go:93] Not enough active targets (0), expected at least (3), waiting for more to become active... I0331 08:45:30.797131 84938 util.go:93] Not enough active targets (0), expected at least (6), waiting for more to become active... I0331 08:45:45.591439 84938 util.go:93] Not enough active targets (0), expected at least (3), waiting for more to become active... I0331 08:45:45.796963 84938 util.go:93] Not enough active targets (0), expected at least (6), waiting for more to become active... I0331 08:46:00.591711 84938 util.go:93] Not enough active targets (0), expected at least (3), waiting for more to become active... E0331 08:46:00.790696 84938 etcd_metrics.go:147] EtcdMetrics: failed to collect etcd database size I0331 08:46:00.797063 84938 util.go:93] Not enough active targets (0), expected at least (6), waiting for more to become active... I0331 08:46:15.594275 84938 util.go:93] Not enough active targets (0), expected at least (3), waiting for more to become active... I0331 08:46:15.787153 84938 util.go:93] Not enough active targets (0), expected at least (6), waiting for more to become active... I0331 08:46:30.593314 84938 util.go:101] 0/3 targets are ready, example not ready target: {map[container:dns endpoint:metrics instance:10.64.9.3:8080 job:dns namespace:probes pod:dns-56b5668745-kvqk2 service:dns] unknown} I0331 08:46:30.801875 84938 util.go:101] 4/6 targets are ready, example not ready target: {map[container:ping-server endpoint:metrics instance:10.64.23.3:8080 job:ping-server namespace:probes pod:ping-server-7f5864b8c6-mcffd service:ping-server] unknown} I0331 08:46:45.599370 84938 util.go:101] 2/3 targets are ready, example not ready target: {map[container:dns endpoint:metrics instance:10.64.42.3:8080 job:dns namespace:probes pod:dns-56b5668745-qgvwg service:dns] unknown} I0331 08:46:45.801401 84938 util.go:101] 5/6 targets are ready, example not ready target: {map[container:ping-server endpoint:metrics instance:10.64.23.3:8080 job:ping-server namespace:probes pod:ping-server-7f5864b8c6-mcffd service:ping-server] unknown} I0331 08:47:00.593400 84938 util.go:104] All 3 expected targets are ready I0331 08:47:00.593516 84938 prometheus_measurement.go:79] SLOMeasurement has started I0331 08:47:00.794139 84938 util.go:104] All 6 expected targets are ready I0331 08:47:00.794197 84938 simple_test_executor.go:167] Step "[step: 01] starting measurements" ended I0331 08:47:00.794241 84938 simple_test_executor.go:145] Step "[step: 02] Creating k8s services" started E0331 08:47:01.324824 84938 etcd_metrics.go:147] EtcdMetrics: failed to collect etcd database size I0331 08:47:06.638011 84938 simple_test_executor.go:167] Step "[step: 02] Creating k8s services" ended I0331 08:47:06.638048 84938 simple_test_executor.go:145] Step "[step: 03] Creating PriorityClass for DaemonSets" started I0331 08:47:06.672904 84938 simple_test_executor.go:167] Step "[step: 03] Creating PriorityClass for DaemonSets" ended I0331 08:47:06.672943 84938 simple_test_executor.go:145] Step "[step: 04] create objects configmaps and secrets" started I0331 08:47:11.169348 84938 simple_test_executor.go:167] Step "[step: 04] create objects configmaps and secrets" ended I0331 08:47:11.169387 84938 simple_test_executor.go:145] Step "[step: 05] Starting measurement for 'create objects'" started ... skipping 100 lines ... I0331 08:47:59.674251 84938 wait_for_pods.go:101] WaitForControlledPodsRunning: namespace(test-yjc4r7-1), labelSelector(name=small-deployment-170): Pods: 5 out of 5 created, 5 running (5 updated), 0 pending scheduled, 0 not scheduled, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0331 08:47:59.735704 84938 wait_for_pods.go:101] WaitForControlledPodsRunning: namespace(test-yjc4r7-1), labelSelector(name=medium-deployment-13): Pods: 30 out of 30 created, 30 running (30 updated), 0 pending scheduled, 0 not scheduled, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0331 08:47:59.826461 84938 wait_for_pods.go:101] WaitForControlledPodsRunning: namespace(test-yjc4r7-1), labelSelector(name=small-deployment-84): Pods: 5 out of 5 created, 5 running (5 updated), 0 pending scheduled, 0 not scheduled, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0331 08:47:59.932574 84938 wait_for_pods.go:101] WaitForControlledPodsRunning: namespace(test-yjc4r7-1), labelSelector(name=small-deployment-60): Pods: 5 out of 5 created, 5 running (5 updated), 0 pending scheduled, 0 not scheduled, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0331 08:48:00.036997 84938 wait_for_pods.go:101] WaitForControlledPodsRunning: namespace(test-yjc4r7-1), labelSelector(name=small-deployment-218): Pods: 5 out of 5 created, 5 running (5 updated), 0 pending scheduled, 0 not scheduled, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0331 08:48:01.071483 84938 wait_for_pods.go:101] WaitForControlledPodsRunning: namespace(test-yjc4r7-1), labelSelector(name=small-deployment-7): Pods: 5 out of 5 created, 5 running (5 updated), 0 pending scheduled, 0 not scheduled, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady E0331 08:48:01.877397 84938 etcd_metrics.go:147] EtcdMetrics: failed to collect etcd database size I0331 08:48:02.764523 84938 wait_for_pods.go:101] WaitForControlledPodsRunning: namespace(test-yjc4r7-1), labelSelector(name=small-deployment-75): Pods: 5 out of 5 created, 5 running (5 updated), 0 pending scheduled, 0 not scheduled, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0331 08:48:02.795734 84938 wait_for_pods.go:101] WaitForControlledPodsRunning: namespace(test-yjc4r7-1), labelSelector(name=small-deployment-120): Pods: 5 out of 5 created, 5 running (5 updated), 0 pending scheduled, 0 not scheduled, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0331 08:48:03.516749 84938 wait_for_pods.go:101] WaitForControlledPodsRunning: namespace(test-yjc4r7-1), labelSelector(name=small-deployment-68): Pods: 5 out of 5 created, 5 running (5 updated), 0 pending scheduled, 0 not scheduled, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0331 08:48:03.650991 84938 wait_for_pods.go:101] WaitForControlledPodsRunning: namespace(test-yjc4r7-1), labelSelector(name=small-deployment-202): Pods: 5 out of 5 created, 5 running (5 updated), 0 pending scheduled, 0 not scheduled, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0331 08:48:04.047184 84938 wait_for_pods.go:101] WaitForControlledPodsRunning: namespace(test-yjc4r7-1), labelSelector(name=small-deployment-5): Pods: 5 out of 5 created, 5 running (5 updated), 0 pending scheduled, 0 not scheduled, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0331 08:48:04.185824 84938 wait_for_pods.go:101] WaitForControlledPodsRunning: namespace(test-yjc4r7-1), labelSelector(name=small-deployment-240): Pods: 5 out of 5 created, 5 running (5 updated), 0 pending scheduled, 0 not scheduled, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady ... skipping 118 lines ... I0331 08:49:00.547864 84938 wait_for_pods.go:101] WaitForControlledPodsRunning: namespace(test-yjc4r7-1), labelSelector(name=small-deployment-40): Pods: 5 out of 5 created, 5 running (5 updated), 0 pending scheduled, 0 not scheduled, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0331 08:49:00.943927 84938 wait_for_pods.go:101] WaitForControlledPodsRunning: namespace(test-yjc4r7-1), labelSelector(name=medium-deployment-20): Pods: 30 out of 30 created, 30 running (30 updated), 0 pending scheduled, 0 not scheduled, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0331 08:49:01.271974 84938 wait_for_pods.go:101] WaitForControlledPodsRunning: namespace(test-yjc4r7-1), labelSelector(name=small-deployment-85): Pods: 5 out of 5 created, 5 running (5 updated), 0 pending scheduled, 0 not scheduled, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0331 08:49:01.372260 84938 wait_for_pods.go:101] WaitForControlledPodsRunning: namespace(test-yjc4r7-1), labelSelector(name=small-deployment-129): Pods: 5 out of 5 created, 5 running (5 updated), 0 pending scheduled, 0 not scheduled, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0331 08:49:01.458983 84938 wait_for_pods.go:101] WaitForControlledPodsRunning: namespace(test-yjc4r7-1), labelSelector(name=small-deployment-51): Pods: 5 out of 5 created, 5 running (5 updated), 0 pending scheduled, 0 not scheduled, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0331 08:49:02.128905 84938 wait_for_pods.go:101] WaitForControlledPodsRunning: namespace(test-yjc4r7-1), labelSelector(name=small-deployment-13): Pods: 5 out of 5 created, 5 running (5 updated), 0 pending scheduled, 0 not scheduled, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady E0331 08:49:02.428199 84938 etcd_metrics.go:147] EtcdMetrics: failed to collect etcd database size I0331 08:49:03.117974 84938 wait_for_pods.go:101] WaitForControlledPodsRunning: namespace(test-yjc4r7-1), labelSelector(name=small-deployment-198): Pods: 5 out of 5 created, 5 running (5 updated), 0 pending scheduled, 0 not scheduled, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0331 08:49:03.308849 84938 wait_for_pods.go:101] WaitForControlledPodsRunning: namespace(test-yjc4r7-1), labelSelector(name=small-deployment-172): Pods: 5 out of 5 created, 5 running (5 updated), 0 pending scheduled, 0 not scheduled, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0331 08:49:04.273937 84938 wait_for_pods.go:101] WaitForControlledPodsRunning: namespace(test-yjc4r7-1), labelSelector(group=load,name=medium-statefulset-0): Pods: 30 out of 30 created, 0 running (0 updated), 2 pending scheduled, 28 not scheduled, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0331 08:49:04.946263 84938 wait_for_pods.go:101] WaitForControlledPodsRunning: namespace(test-yjc4r7-1), labelSelector(name=small-deployment-128): Pods: 5 out of 5 created, 5 running (5 updated), 0 pending scheduled, 0 not scheduled, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0331 08:49:05.396493 84938 wait_for_pods.go:101] WaitForControlledPodsRunning: namespace(test-yjc4r7-1), labelSelector(name=small-deployment-114): Pods: 5 out of 5 created, 5 running (5 updated), 0 pending scheduled, 0 not scheduled, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0331 08:49:05.433262 84938 wait_for_pods.go:101] WaitForControlledPodsRunning: namespace(test-yjc4r7-1), labelSelector(name=small-deployment-191): Pods: 5 out of 5 created, 5 running (5 updated), 0 pending scheduled, 0 not scheduled, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady ... skipping 131 lines ... I0331 08:50:01.512664 84938 wait_for_pods.go:101] WaitForControlledPodsRunning: namespace(test-yjc4r7-3), labelSelector(name=scheduler-throughput-deployment-0): Pods: 100 out of 100 created, 81 running (81 updated), 19 pending scheduled, 0 not scheduled, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0331 08:50:01.558382 84938 wait_for_pods.go:101] WaitForControlledPodsRunning: namespace(test-yjc4r7-11), labelSelector(name=scheduler-throughput-deployment-0): Pods: 31 out of 100 created, 0 running (0 updated), 31 pending scheduled, 0 not scheduled, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0331 08:50:01.558990 84938 wait_for_pods.go:101] WaitForControlledPodsRunning: namespace(test-yjc4r7-9), labelSelector(name=scheduler-throughput-deployment-0): Pods: 15 out of 100 created, 0 running (0 updated), 15 pending scheduled, 0 not scheduled, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0331 08:50:01.559041 84938 wait_for_pods.go:101] WaitForControlledPodsRunning: namespace(test-yjc4r7-8), labelSelector(name=scheduler-throughput-deployment-0): Pods: 18 out of 100 created, 0 running (0 updated), 17 pending scheduled, 1 not scheduled, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0331 08:50:01.559063 84938 wait_for_pods.go:101] WaitForControlledPodsRunning: namespace(test-yjc4r7-7), labelSelector(name=scheduler-throughput-deployment-0): Pods: 15 out of 100 created, 0 running (0 updated), 15 pending scheduled, 0 not scheduled, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0331 08:50:01.566255 84938 wait_for_pods.go:101] WaitForControlledPodsRunning: namespace(test-yjc4r7-10), labelSelector(name=scheduler-throughput-deployment-0): Pods: 7 out of 100 created, 0 running (0 updated), 7 pending scheduled, 0 not scheduled, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady E0331 08:50:02.983143 84938 etcd_metrics.go:147] EtcdMetrics: failed to collect etcd database size I0331 08:50:06.510152 84938 wait_for_pods.go:101] WaitForControlledPodsRunning: namespace(test-yjc4r7-4), labelSelector(name=scheduler-throughput-deployment-0): Pods: 100 out of 100 created, 100 running (100 updated), 0 pending scheduled, 0 not scheduled, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0331 08:50:06.511293 84938 wait_for_pods.go:101] WaitForControlledPodsRunning: namespace(test-yjc4r7-2), labelSelector(name=scheduler-throughput-deployment-0): Pods: 100 out of 100 created, 100 running (100 updated), 0 pending scheduled, 0 not scheduled, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0331 08:50:06.515989 84938 wait_for_pods.go:101] WaitForControlledPodsRunning: namespace(test-yjc4r7-5), labelSelector(name=scheduler-throughput-deployment-0): Pods: 100 out of 100 created, 100 running (100 updated), 0 pending scheduled, 0 not scheduled, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0331 08:50:06.516233 84938 wait_for_pods.go:101] WaitForControlledPodsRunning: namespace(test-yjc4r7-3), labelSelector(name=scheduler-throughput-deployment-0): Pods: 100 out of 100 created, 100 running (100 updated), 0 pending scheduled, 0 not scheduled, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0331 08:50:06.560726 84938 wait_for_pods.go:101] WaitForControlledPodsRunning: namespace(test-yjc4r7-7), labelSelector(name=scheduler-throughput-deployment-0): Pods: 100 out of 100 created, 54 running (54 updated), 46 pending scheduled, 0 not scheduled, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0331 08:50:06.561350 84938 wait_for_pods.go:101] WaitForControlledPodsRunning: namespace(test-yjc4r7-8), labelSelector(name=scheduler-throughput-deployment-0): Pods: 100 out of 100 created, 79 running (79 updated), 21 pending scheduled, 0 not scheduled, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady ... skipping 42 lines ... I0331 08:50:11.984997 84938 phase_latency.go:141] PodStartupLatency: 0 worst schedule_to_run latencies: [] I0331 08:50:11.985023 84938 phase_latency.go:146] PodStartupLatency: perc50: 0s, perc90: 0s, perc99: 0s I0331 08:50:11.985131 84938 simple_test_executor.go:167] Step "[step: 11] Collecting scheduler throughput measurements" ended I0331 08:50:11.985161 84938 simple_test_executor.go:145] Step "[step: 12] delete scheduler throughput pods" started I0331 08:50:17.075416 84938 wait_for_pods.go:101] WaitForControlledPodsRunning: namespace(test-yjc4r7-2), labelSelector(name=scheduler-throughput-deployment-0): Pods: 0 out of 0 created, 0 running (0 updated), 0 pending scheduled, 0 not scheduled, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0331 08:50:47.076067 84938 wait_for_pods.go:101] WaitForControlledPodsRunning: namespace(test-yjc4r7-3), labelSelector(name=scheduler-throughput-deployment-0): Pods: 0 out of 0 created, 0 running (0 updated), 0 pending scheduled, 0 not scheduled, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady E0331 08:51:03.520953 84938 etcd_metrics.go:147] EtcdMetrics: failed to collect etcd database size I0331 08:51:17.076476 84938 wait_for_pods.go:101] WaitForControlledPodsRunning: namespace(test-yjc4r7-4), labelSelector(name=scheduler-throughput-deployment-0): Pods: 0 out of 0 created, 0 running (0 updated), 0 pending scheduled, 0 not scheduled, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0331 08:51:47.076919 84938 wait_for_pods.go:101] WaitForControlledPodsRunning: namespace(test-yjc4r7-5), labelSelector(name=scheduler-throughput-deployment-0): Pods: 0 out of 0 created, 0 running (0 updated), 0 pending scheduled, 0 not scheduled, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady E0331 08:52:04.070464 84938 etcd_metrics.go:147] EtcdMetrics: failed to collect etcd database size I0331 08:52:17.077136 84938 wait_for_pods.go:101] WaitForControlledPodsRunning: namespace(test-yjc4r7-6), labelSelector(name=scheduler-throughput-deployment-0): Pods: 0 out of 0 created, 0 running (0 updated), 0 pending scheduled, 0 not scheduled, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0331 08:52:47.078913 84938 wait_for_pods.go:101] WaitForControlledPodsRunning: namespace(test-yjc4r7-7), labelSelector(name=scheduler-throughput-deployment-0): Pods: 0 out of 0 created, 0 running (0 updated), 0 pending scheduled, 0 not scheduled, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady E0331 08:53:04.620371 84938 etcd_metrics.go:147] EtcdMetrics: failed to collect etcd database size I0331 08:53:17.078913 84938 wait_for_pods.go:101] WaitForControlledPodsRunning: namespace(test-yjc4r7-8), labelSelector(name=scheduler-throughput-deployment-0): Pods: 0 out of 0 created, 0 running (0 updated), 0 pending scheduled, 0 not scheduled, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0331 08:53:47.078849 84938 wait_for_pods.go:101] WaitForControlledPodsRunning: namespace(test-yjc4r7-9), labelSelector(name=scheduler-throughput-deployment-0): Pods: 0 out of 0 created, 0 running (0 updated), 0 pending scheduled, 0 not scheduled, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady E0331 08:54:05.206044 84938 etcd_metrics.go:147] EtcdMetrics: failed to collect etcd database size I0331 08:54:17.079547 84938 wait_for_pods.go:101] WaitForControlledPodsRunning: namespace(test-yjc4r7-10), labelSelector(name=scheduler-throughput-deployment-0): Pods: 0 out of 0 created, 0 running (0 updated), 0 pending scheduled, 0 not scheduled, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0331 08:54:47.080724 84938 wait_for_pods.go:101] WaitForControlledPodsRunning: namespace(test-yjc4r7-11), labelSelector(name=scheduler-throughput-deployment-0): Pods: 0 out of 0 created, 0 running (0 updated), 0 pending scheduled, 0 not scheduled, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady E0331 08:55:05.760288 84938 etcd_metrics.go:147] EtcdMetrics: failed to collect etcd database size I0331 08:55:11.996143 84938 simple_test_executor.go:167] Step "[step: 12] delete scheduler throughput pods" ended I0331 08:55:11.996193 84938 simple_test_executor.go:145] Step "[step: 13] Waiting for scheduler throughput pods to be deleted" started I0331 08:55:11.996233 84938 wait_for_controlled_pods.go:200] WaitForControlledPodsRunning: waiting for controlled pods measurement... I0331 08:55:17.043409 84938 wait_for_controlled_pods.go:267] WaitForControlledPodsRunning: running 0, deleted 10, timeout: 0, unknown: 0 I0331 08:55:17.043448 84938 wait_for_controlled_pods.go:281] WaitForControlledPodsRunning: 0/0 Deployments are running with all pods I0331 08:55:17.043467 84938 simple_test_executor.go:167] Step "[step: 13] Waiting for scheduler throughput pods to be deleted" ended ... skipping 219 lines ... I0331 08:56:05.157786 84938 wait_for_pods.go:101] WaitForControlledPodsRunning: namespace(test-yjc4r7-1), labelSelector(name=latency-deployment-214): Pods: 1 out of 1 created, 1 running (1 updated), 0 pending scheduled, 0 not scheduled, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0331 08:56:05.357555 84938 wait_for_pods.go:101] WaitForControlledPodsRunning: namespace(test-yjc4r7-1), labelSelector(name=latency-deployment-215): Pods: 1 out of 1 created, 1 running (1 updated), 0 pending scheduled, 0 not scheduled, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0331 08:56:05.558613 84938 wait_for_pods.go:101] WaitForControlledPodsRunning: namespace(test-yjc4r7-1), labelSelector(name=latency-deployment-216): Pods: 1 out of 1 created, 1 running (1 updated), 0 pending scheduled, 0 not scheduled, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0331 08:56:05.757295 84938 wait_for_pods.go:101] WaitForControlledPodsRunning: namespace(test-yjc4r7-1), labelSelector(name=latency-deployment-217): Pods: 1 out of 1 created, 1 running (1 updated), 0 pending scheduled, 0 not scheduled, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0331 08:56:05.958724 84938 wait_for_pods.go:101] WaitForControlledPodsRunning: namespace(test-yjc4r7-1), labelSelector(name=latency-deployment-218): Pods: 1 out of 1 created, 1 running (1 updated), 0 pending scheduled, 0 not scheduled, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0331 08:56:06.159591 84938 wait_for_pods.go:101] WaitForControlledPodsRunning: namespace(test-yjc4r7-1), labelSelector(name=latency-deployment-219): Pods: 1 out of 1 created, 1 running (1 updated), 0 pending scheduled, 0 not scheduled, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady E0331 08:56:06.327022 84938 etcd_metrics.go:147] EtcdMetrics: failed to collect etcd database size I0331 08:56:06.360000 84938 wait_for_pods.go:101] WaitForControlledPodsRunning: namespace(test-yjc4r7-1), labelSelector(name=latency-deployment-220): Pods: 1 out of 1 created, 1 running (1 updated), 0 pending scheduled, 0 not scheduled, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0331 08:56:06.560001 84938 wait_for_pods.go:101] WaitForControlledPodsRunning: namespace(test-yjc4r7-1), labelSelector(name=latency-deployment-221): Pods: 1 out of 1 created, 1 running (1 updated), 0 pending scheduled, 0 not scheduled, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0331 08:56:06.760398 84938 wait_for_pods.go:101] WaitForControlledPodsRunning: namespace(test-yjc4r7-1), labelSelector(name=latency-deployment-222): Pods: 1 out of 1 created, 1 running (1 updated), 0 pending scheduled, 0 not scheduled, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0331 08:56:06.961094 84938 wait_for_pods.go:101] WaitForControlledPodsRunning: namespace(test-yjc4r7-1), labelSelector(name=latency-deployment-223): Pods: 1 out of 1 created, 1 running (1 updated), 0 pending scheduled, 0 not scheduled, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0331 08:56:07.161945 84938 wait_for_pods.go:101] WaitForControlledPodsRunning: namespace(test-yjc4r7-1), labelSelector(name=latency-deployment-224): Pods: 1 out of 1 created, 1 running (1 updated), 0 pending scheduled, 0 not scheduled, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0331 08:56:07.363015 84938 wait_for_pods.go:101] WaitForControlledPodsRunning: namespace(test-yjc4r7-1), labelSelector(name=latency-deployment-225): Pods: 1 out of 1 created, 1 running (1 updated), 0 pending scheduled, 0 not scheduled, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady ... skipping 275 lines ... I0331 08:57:02.129262 84938 wait_for_pods.go:101] WaitForControlledPodsRunning: namespace(test-yjc4r7-1), labelSelector(name=latency-deployment-498): Pods: 1 out of 1 created, 1 running (1 updated), 0 pending scheduled, 0 not scheduled, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0331 08:57:02.329301 84938 wait_for_pods.go:101] WaitForControlledPodsRunning: namespace(test-yjc4r7-1), labelSelector(name=latency-deployment-499): Pods: 1 out of 1 created, 1 running (1 updated), 0 pending scheduled, 0 not scheduled, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0331 08:57:02.756725 84938 wait_for_controlled_pods.go:267] WaitForControlledPodsRunning: running 500, deleted 0, timeout: 0, unknown: 0 I0331 08:57:02.756780 84938 wait_for_controlled_pods.go:281] WaitForControlledPodsRunning: 500/500 Deployments are running with all pods I0331 08:57:02.756815 84938 simple_test_executor.go:167] Step "[step: 16] Waiting for latency pods to be running" ended I0331 08:57:02.756844 84938 simple_test_executor.go:145] Step "[step: 17] Deleting latency pods" started E0331 08:57:06.880040 84938 etcd_metrics.go:147] EtcdMetrics: failed to collect etcd database size I0331 08:57:07.845034 84938 wait_for_pods.go:101] WaitForControlledPodsRunning: namespace(test-yjc4r7-1), labelSelector(name=latency-deployment-0): Pods: 0 out of 0 created, 0 running (0 updated), 0 pending scheduled, 0 not scheduled, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0331 08:57:08.045666 84938 wait_for_pods.go:101] WaitForControlledPodsRunning: namespace(test-yjc4r7-1), labelSelector(name=latency-deployment-1): Pods: 0 out of 0 created, 0 running (0 updated), 0 pending scheduled, 0 not scheduled, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0331 08:57:08.246200 84938 wait_for_pods.go:101] WaitForControlledPodsRunning: namespace(test-yjc4r7-1), labelSelector(name=latency-deployment-2): Pods: 0 out of 0 created, 0 running (0 updated), 0 pending scheduled, 0 not scheduled, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0331 08:57:08.446683 84938 wait_for_pods.go:101] WaitForControlledPodsRunning: namespace(test-yjc4r7-1), labelSelector(name=latency-deployment-3): Pods: 0 out of 0 created, 0 running (0 updated), 0 pending scheduled, 0 not scheduled, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0331 08:57:08.647477 84938 wait_for_pods.go:101] WaitForControlledPodsRunning: namespace(test-yjc4r7-1), labelSelector(name=latency-deployment-4): Pods: 0 out of 0 created, 0 running (0 updated), 0 pending scheduled, 0 not scheduled, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0331 08:57:08.848207 84938 wait_for_pods.go:101] WaitForControlledPodsRunning: namespace(test-yjc4r7-1), labelSelector(name=latency-deployment-5): Pods: 0 out of 0 created, 0 running (0 updated), 0 pending scheduled, 0 not scheduled, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady ... skipping 286 lines ... I0331 08:58:06.420497 84938 wait_for_pods.go:101] WaitForControlledPodsRunning: namespace(test-yjc4r7-1), labelSelector(name=latency-deployment-292): Pods: 0 out of 0 created, 0 running (0 updated), 0 pending scheduled, 0 not scheduled, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0331 08:58:06.620421 84938 wait_for_pods.go:101] WaitForControlledPodsRunning: namespace(test-yjc4r7-1), labelSelector(name=latency-deployment-293): Pods: 0 out of 0 created, 0 running (0 updated), 0 pending scheduled, 0 not scheduled, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0331 08:58:06.821027 84938 wait_for_pods.go:101] WaitForControlledPodsRunning: namespace(test-yjc4r7-1), labelSelector(name=latency-deployment-294): Pods: 0 out of 0 created, 0 running (0 updated), 0 pending scheduled, 0 not scheduled, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0331 08:58:07.023312 84938 wait_for_pods.go:101] WaitForControlledPodsRunning: namespace(test-yjc4r7-1), labelSelector(name=latency-deployment-295): Pods: 0 out of 0 created, 0 running (0 updated), 0 pending scheduled, 0 not scheduled, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0331 08:58:07.221978 84938 wait_for_pods.go:101] WaitForControlledPodsRunning: namespace(test-yjc4r7-1), labelSelector(name=latency-deployment-296): Pods: 0 out of 0 created, 0 running (0 updated), 0 pending scheduled, 0 not scheduled, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0331 08:58:07.423207 84938 wait_for_pods.go:101] WaitForControlledPodsRunning: namespace(test-yjc4r7-1), labelSelector(name=latency-deployment-297): Pods: 0 out of 0 created, 0 running (0 updated), 0 pending scheduled, 0 not scheduled, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady E0331 08:58:07.427916 84938 etcd_metrics.go:147] EtcdMetrics: failed to collect etcd database size I0331 08:58:07.622171 84938 wait_for_pods.go:101] WaitForControlledPodsRunning: namespace(test-yjc4r7-1), labelSelector(name=latency-deployment-298): Pods: 0 out of 0 created, 0 running (0 updated), 0 pending scheduled, 0 not scheduled, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0331 08:58:07.824149 84938 wait_for_pods.go:101] WaitForControlledPodsRunning: namespace(test-yjc4r7-1), labelSelector(name=latency-deployment-299): Pods: 0 out of 0 created, 0 running (0 updated), 0 pending scheduled, 0 not scheduled, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0331 08:58:08.023834 84938 wait_for_pods.go:101] WaitForControlledPodsRunning: namespace(test-yjc4r7-1), labelSelector(name=latency-deployment-300): Pods: 0 out of 0 created, 0 running (0 updated), 0 pending scheduled, 0 not scheduled, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0331 08:58:08.223946 84938 wait_for_pods.go:101] WaitForControlledPodsRunning: namespace(test-yjc4r7-1), labelSelector(name=latency-deployment-301): Pods: 0 out of 0 created, 0 running (0 updated), 0 pending scheduled, 0 not scheduled, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0331 08:58:08.424492 84938 wait_for_pods.go:101] WaitForControlledPodsRunning: namespace(test-yjc4r7-1), labelSelector(name=latency-deployment-302): Pods: 0 out of 0 created, 0 running (0 updated), 0 pending scheduled, 0 not scheduled, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0331 08:58:08.626023 84938 wait_for_pods.go:101] WaitForControlledPodsRunning: namespace(test-yjc4r7-1), labelSelector(name=latency-deployment-303): Pods: 0 out of 0 created, 0 running (0 updated), 0 pending scheduled, 0 not scheduled, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady ... skipping 345 lines ... I0331 08:59:06.995627 84938 wait_for_pods.go:101] WaitForControlledPodsRunning: namespace(test-yjc4r7-1), labelSelector(name=small-deployment-79): Pods: 7 out of 7 created, 7 running (7 updated), 0 pending scheduled, 0 not scheduled, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0331 08:59:07.043774 84938 wait_for_pods.go:101] WaitForControlledPodsRunning: namespace(test-yjc4r7-1), labelSelector(name=small-deployment-199): Pods: 4 out of 4 created, 4 running (4 updated), 0 pending scheduled, 0 not scheduled, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0331 08:59:07.052930 84938 wait_for_pods.go:101] WaitForControlledPodsRunning: namespace(test-yjc4r7-1), labelSelector(name=small-deployment-152): Pods: 7 out of 7 created, 7 running (7 updated), 0 pending scheduled, 0 not scheduled, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0331 08:59:07.180122 84938 wait_for_pods.go:101] WaitForControlledPodsRunning: namespace(test-yjc4r7-1), labelSelector(name=small-deployment-188): Pods: 4 out of 4 created, 4 running (4 updated), 0 pending scheduled, 0 not scheduled, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0331 08:59:07.213304 84938 wait_for_pods.go:101] WaitForControlledPodsRunning: namespace(test-yjc4r7-1), labelSelector(name=small-deployment-265): Pods: 2 out of 2 created, 2 running (2 updated), 0 pending scheduled, 0 not scheduled, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0331 08:59:07.746882 84938 wait_for_pods.go:101] WaitForControlledPodsRunning: namespace(test-yjc4r7-1), labelSelector(name=small-deployment-239): Pods: 3 out of 3 created, 3 running (3 updated), 0 pending scheduled, 0 not scheduled, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady E0331 08:59:07.966020 84938 etcd_metrics.go:147] EtcdMetrics: failed to collect etcd database size I0331 08:59:08.005075 84938 wait_for_pods.go:101] WaitForControlledPodsRunning: namespace(test-yjc4r7-1), labelSelector(name=small-deployment-134): Pods: 6 out of 6 created, 6 running (6 updated), 0 pending scheduled, 0 not scheduled, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0331 08:59:08.059295 84938 wait_for_pods.go:101] WaitForControlledPodsRunning: namespace(test-yjc4r7-1), labelSelector(name=small-deployment-289): Pods: 6 out of 6 created, 6 running (6 updated), 0 pending scheduled, 0 not scheduled, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0331 08:59:08.117393 84938 wait_for_pods.go:101] WaitForControlledPodsRunning: namespace(test-yjc4r7-1), labelSelector(name=small-deployment-138): Pods: 3 out of 3 created, 3 running (3 updated), 0 pending scheduled, 0 not scheduled, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0331 08:59:08.197721 84938 wait_for_pods.go:101] WaitForControlledPodsRunning: namespace(test-yjc4r7-1), labelSelector(name=small-deployment-195): Pods: 4 out of 4 created, 4 running (4 updated), 0 pending scheduled, 0 not scheduled, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0331 08:59:08.470460 84938 wait_for_pods.go:101] WaitForControlledPodsRunning: namespace(test-yjc4r7-1), labelSelector(name=small-deployment-60): Pods: 7 out of 7 created, 7 running (7 updated), 0 pending scheduled, 0 not scheduled, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0331 08:59:08.522296 84938 wait_for_pods.go:101] WaitForControlledPodsRunning: namespace(test-yjc4r7-1), labelSelector(name=medium-deployment-20): Pods: 44 out of 44 created, 42 running (42 updated), 2 pending scheduled, 0 not scheduled, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady ... skipping 225 lines ... I0331 09:00:06.415196 84938 wait_for_pods.go:101] WaitForControlledPodsRunning: namespace(test-yjc4r7-1), labelSelector(name=medium-deployment-15): Pods: 0 out of 0 created, 0 running (0 updated), 0 pending scheduled, 0 not scheduled, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0331 09:00:06.918236 84938 wait_for_pods.go:101] WaitForControlledPodsRunning: namespace(test-yjc4r7-1), labelSelector(name=small-deployment-100): Pods: 0 out of 0 created, 0 running (0 updated), 0 pending scheduled, 0 not scheduled, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0331 09:00:07.056449 84938 wait_for_pods.go:101] WaitForControlledPodsRunning: namespace(test-yjc4r7-1), labelSelector(name=small-deployment-221): Pods: 0 out of 0 created, 0 running (0 updated), 0 pending scheduled, 0 not scheduled, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0331 09:00:07.893578 84938 wait_for_pods.go:101] WaitForControlledPodsRunning: namespace(test-yjc4r7-1), labelSelector(name=small-deployment-48): Pods: 0 out of 0 created, 0 running (0 updated), 0 pending scheduled, 0 not scheduled, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0331 09:00:07.895771 84938 wait_for_pods.go:101] WaitForControlledPodsRunning: namespace(test-yjc4r7-1), labelSelector(name=small-deployment-278): Pods: 0 out of 0 created, 0 running (0 updated), 0 pending scheduled, 0 not scheduled, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0331 09:00:08.230387 84938 wait_for_pods.go:101] WaitForControlledPodsRunning: namespace(test-yjc4r7-1), labelSelector(name=small-deployment-279): Pods: 0 out of 0 created, 0 running (0 updated), 0 pending scheduled, 0 not scheduled, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady E0331 09:00:08.502750 84938 etcd_metrics.go:147] EtcdMetrics: failed to collect etcd database size I0331 09:00:08.807869 84938 wait_for_pods.go:101] WaitForControlledPodsRunning: namespace(test-yjc4r7-1), labelSelector(name=small-deployment-82): Pods: 0 out of 0 created, 0 running (0 updated), 0 pending scheduled, 0 not scheduled, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0331 09:00:09.306413 84938 wait_for_pods.go:101] WaitForControlledPodsRunning: namespace(test-yjc4r7-1), labelSelector(name=small-deployment-178): Pods: 0 out of 0 created, 0 running (0 updated), 0 pending scheduled, 0 not scheduled, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0331 09:00:11.038525 84938 wait_for_pods.go:101] WaitForControlledPodsRunning: namespace(test-yjc4r7-1), labelSelector(name=small-deployment-119): Pods: 0 out of 0 created, 0 running (0 updated), 0 pending scheduled, 0 not scheduled, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0331 09:00:11.318473 84938 wait_for_pods.go:101] WaitForControlledPodsRunning: namespace(test-yjc4r7-1), labelSelector(name=small-deployment-254): Pods: 0 out of 0 created, 0 running (0 updated), 0 pending scheduled, 0 not scheduled, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0331 09:00:11.641238 84938 wait_for_pods.go:101] WaitForControlledPodsRunning: namespace(test-yjc4r7-1), labelSelector(name=small-deployment-268): Pods: 0 out of 0 created, 0 running (0 updated), 0 pending scheduled, 0 not scheduled, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0331 09:00:13.000578 84938 wait_for_pods.go:101] WaitForControlledPodsRunning: namespace(test-yjc4r7-1), labelSelector(name=small-deployment-165): Pods: 0 out of 0 created, 0 running (0 updated), 0 pending scheduled, 0 not scheduled, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady ... skipping 71 lines ... I0331 09:01:02.212675 84938 wait_for_pods.go:101] WaitForControlledPodsRunning: namespace(test-yjc4r7-1), labelSelector(name=small-deployment-24): Pods: 0 out of 0 created, 0 running (0 updated), 0 pending scheduled, 0 not scheduled, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0331 09:01:04.321442 84938 wait_for_pods.go:101] WaitForControlledPodsRunning: namespace(test-yjc4r7-1), labelSelector(name=small-deployment-205): Pods: 0 out of 0 created, 0 running (0 updated), 0 pending scheduled, 0 not scheduled, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0331 09:01:04.374998 84938 wait_for_pods.go:101] WaitForControlledPodsRunning: namespace(test-yjc4r7-1), labelSelector(name=small-deployment-238): Pods: 0 out of 0 created, 0 running (0 updated), 0 pending scheduled, 0 not scheduled, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0331 09:01:05.118379 84938 wait_for_pods.go:101] WaitForControlledPodsRunning: namespace(test-yjc4r7-1), labelSelector(name=small-deployment-128): Pods: 0 out of 0 created, 0 running (0 updated), 0 pending scheduled, 0 not scheduled, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0331 09:01:08.529781 84938 wait_for_pods.go:101] WaitForControlledPodsRunning: namespace(test-yjc4r7-1), labelSelector(name=small-deployment-192): Pods: 0 out of 0 created, 0 running (0 updated), 0 pending scheduled, 0 not scheduled, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0331 09:01:08.960631 84938 wait_for_pods.go:101] WaitForControlledPodsRunning: namespace(test-yjc4r7-1), labelSelector(name=small-deployment-201): Pods: 0 out of 0 created, 0 running (0 updated), 0 pending scheduled, 0 not scheduled, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady E0331 09:01:09.044998 84938 etcd_metrics.go:147] EtcdMetrics: failed to collect etcd database size I0331 09:01:09.259496 84938 wait_for_pods.go:101] WaitForControlledPodsRunning: namespace(test-yjc4r7-1), labelSelector(name=medium-deployment-0): Pods: 0 out of 0 created, 0 running (0 updated), 0 pending scheduled, 0 not scheduled, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0331 09:01:09.569807 84938 wait_for_pods.go:101] WaitForControlledPodsRunning: namespace(test-yjc4r7-1), labelSelector(name=small-deployment-158): Pods: 0 out of 0 created, 0 running (0 updated), 0 pending scheduled, 0 not scheduled, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0331 09:01:10.979581 84938 wait_for_pods.go:101] WaitForControlledPodsRunning: namespace(test-yjc4r7-1), labelSelector(name=small-deployment-80): Pods: 0 out of 0 created, 0 running (0 updated), 0 pending scheduled, 0 not scheduled, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0331 09:01:12.389726 84938 wait_for_pods.go:101] WaitForControlledPodsRunning: namespace(test-yjc4r7-1), labelSelector(name=small-deployment-138): Pods: 0 out of 0 created, 0 running (0 updated), 0 pending scheduled, 0 not scheduled, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0331 09:01:12.974400 84938 wait_for_pods.go:101] WaitForControlledPodsRunning: namespace(test-yjc4r7-1), labelSelector(name=small-deployment-87): Pods: 0 out of 0 created, 0 running (0 updated), 0 pending scheduled, 0 not scheduled, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0331 09:01:13.052849 84938 wait_for_pods.go:101] WaitForControlledPodsRunning: namespace(test-yjc4r7-1), labelSelector(name=small-deployment-274): Pods: 0 out of 0 created, 0 running (0 updated), 0 pending scheduled, 0 not scheduled, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady ... skipping 86 lines ... I0331 09:02:07.728956 84938 wait_for_pods.go:101] WaitForControlledPodsRunning: namespace(test-yjc4r7-1), labelSelector(name=small-deployment-127): Pods: 0 out of 0 created, 0 running (0 updated), 0 pending scheduled, 0 not scheduled, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0331 09:02:08.130850 84938 wait_for_pods.go:101] WaitForControlledPodsRunning: namespace(test-yjc4r7-1), labelSelector(name=small-deployment-120): Pods: 0 out of 0 created, 0 running (0 updated), 0 pending scheduled, 0 not scheduled, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0331 09:02:08.314924 84938 wait_for_pods.go:101] WaitForControlledPodsRunning: namespace(test-yjc4r7-1), labelSelector(name=small-deployment-151): Pods: 0 out of 0 created, 0 running (0 updated), 0 pending scheduled, 0 not scheduled, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0331 09:02:08.465463 84938 wait_for_pods.go:101] WaitForControlledPodsRunning: namespace(test-yjc4r7-1), labelSelector(name=medium-deployment-11): Pods: 0 out of 0 created, 0 running (0 updated), 0 pending scheduled, 0 not scheduled, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0331 09:02:08.940179 84938 wait_for_pods.go:101] WaitForControlledPodsRunning: namespace(test-yjc4r7-1), labelSelector(name=small-deployment-251): Pods: 0 out of 0 created, 0 running (0 updated), 0 pending scheduled, 0 not scheduled, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0331 09:02:09.383177 84938 wait_for_pods.go:101] WaitForControlledPodsRunning: namespace(test-yjc4r7-1), labelSelector(group=load,name=medium-statefulset-0): Pods: 0 out of 0 created, 0 running (0 updated), 0 pending scheduled, 0 not scheduled, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady E0331 09:02:09.589228 84938 etcd_metrics.go:147] EtcdMetrics: failed to collect etcd database size I0331 09:02:10.242265 84938 wait_for_pods.go:101] WaitForControlledPodsRunning: namespace(test-yjc4r7-1), labelSelector(name=small-deployment-200): Pods: 0 out of 0 created, 0 running (0 updated), 0 pending scheduled, 0 not scheduled, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0331 09:02:10.598600 84938 wait_for_pods.go:101] WaitForControlledPodsRunning: namespace(test-yjc4r7-1), labelSelector(name=small-deployment-218): Pods: 0 out of 0 created, 0 running (0 updated), 0 pending scheduled, 0 not scheduled, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0331 09:02:10.815899 84938 wait_for_pods.go:101] WaitForControlledPodsRunning: namespace(test-yjc4r7-1), labelSelector(name=small-deployment-57): Pods: 0 out of 0 created, 0 running (0 updated), 0 pending scheduled, 0 not scheduled, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0331 09:02:12.215923 84938 wait_for_pods.go:101] WaitForControlledPodsRunning: namespace(test-yjc4r7-1), labelSelector(name=small-deployment-129): Pods: 0 out of 0 created, 0 running (0 updated), 0 pending scheduled, 0 not scheduled, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0331 09:02:12.409916 84938 wait_for_pods.go:101] WaitForControlledPodsRunning: namespace(test-yjc4r7-1), labelSelector(name=small-deployment-175): Pods: 0 out of 0 created, 0 running (0 updated), 0 pending scheduled, 0 not scheduled, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0331 09:02:12.923178 84938 wait_for_pods.go:101] WaitForControlledPodsRunning: namespace(test-yjc4r7-1), labelSelector(name=small-deployment-204): Pods: 0 out of 0 created, 0 running (0 updated), 0 pending scheduled, 0 not scheduled, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady ... skipping 74 lines ... I0331 09:03:06.007361 84938 wait_for_pods.go:101] WaitForControlledPodsRunning: namespace(test-yjc4r7-1), labelSelector(name=small-deployment-44): Pods: 0 out of 0 created, 0 running (0 updated), 0 pending scheduled, 0 not scheduled, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0331 09:03:06.567534 84938 wait_for_pods.go:101] WaitForControlledPodsRunning: namespace(test-yjc4r7-1), labelSelector(name=medium-deployment-5): Pods: 0 out of 0 created, 0 running (0 updated), 0 pending scheduled, 0 not scheduled, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0331 09:03:07.778423 84938 wait_for_pods.go:101] WaitForControlledPodsRunning: namespace(test-yjc4r7-1), labelSelector(name=medium-deployment-12): Pods: 0 out of 0 created, 0 running (0 updated), 0 pending scheduled, 0 not scheduled, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0331 09:03:08.497681 84938 wait_for_pods.go:101] WaitForControlledPodsRunning: namespace(test-yjc4r7-1), labelSelector(name=small-deployment-219): Pods: 0 out of 0 created, 0 running (0 updated), 0 pending scheduled, 0 not scheduled, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0331 09:03:09.223668 84938 wait_for_pods.go:101] WaitForControlledPodsRunning: namespace(test-yjc4r7-1), labelSelector(name=small-deployment-97): Pods: 0 out of 0 created, 0 running (0 updated), 0 pending scheduled, 0 not scheduled, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0331 09:03:10.087140 84938 wait_for_pods.go:101] WaitForControlledPodsRunning: namespace(test-yjc4r7-1), labelSelector(name=small-deployment-10): Pods: 0 out of 0 created, 0 running (0 updated), 0 pending scheduled, 0 not scheduled, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady E0331 09:03:10.137304 84938 etcd_metrics.go:147] EtcdMetrics: failed to collect etcd database size I0331 09:03:10.331564 84938 wait_for_pods.go:101] WaitForControlledPodsRunning: namespace(test-yjc4r7-1), labelSelector(name=small-deployment-94): Pods: 0 out of 0 created, 0 running (0 updated), 0 pending scheduled, 0 not scheduled, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0331 09:03:10.779075 84938 wait_for_pods.go:101] WaitForControlledPodsRunning: namespace(test-yjc4r7-1), labelSelector(name=small-deployment-105): Pods: 0 out of 0 created, 0 running (0 updated), 0 pending scheduled, 0 not scheduled, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0331 09:03:10.883606 84938 wait_for_pods.go:101] WaitForControlledPodsRunning: namespace(test-yjc4r7-1), labelSelector(name=small-deployment-103): Pods: 0 out of 0 created, 0 running (0 updated), 0 pending scheduled, 0 not scheduled, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0331 09:03:12.209257 84938 wait_for_pods.go:101] WaitForControlledPodsRunning: namespace(test-yjc4r7-1), labelSelector(name=medium-deployment-3): Pods: 0 out of 0 created, 0 running (0 updated), 0 pending scheduled, 0 not scheduled, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0331 09:03:12.382651 84938 wait_for_pods.go:101] WaitForControlledPodsRunning: namespace(test-yjc4r7-1), labelSelector(name=small-deployment-280): Pods: 0 out of 0 created, 0 running (0 updated), 0 pending scheduled, 0 not scheduled, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0331 09:03:12.472116 84938 wait_for_pods.go:101] WaitForControlledPodsRunning: namespace(test-yjc4r7-1), labelSelector(name=small-deployment-96): Pods: 0 out of 0 created, 0 running (0 updated), 0 pending scheduled, 0 not scheduled, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady ... skipping 269 lines ... I0331 09:04:21.391744 84938 util.go:93] Not enough active targets (1), expected at least (6), waiting for more to become active... I0331 09:04:36.393151 84938 util.go:101] 3/6 targets are ready, example not ready target: {map[container:ping-server endpoint:metrics instance:10.64.19.56:8080 job:ping-server namespace:probes pod:ping-server-7f5864b8c6-nsf58 service:ping-server] unknown} I0331 09:04:51.385354 84938 util.go:101] 5/6 targets are ready, example not ready target: {map[container:ping-server endpoint:metrics instance:10.64.19.56:8080 job:ping-server namespace:probes pod:ping-server-7f5864b8c6-nsf58 service:ping-server] unknown} I0331 09:05:06.383429 84938 util.go:104] All 6 expected targets are ready I0331 09:05:06.383491 84938 simple_test_executor.go:167] Step "[step: 01] starting measurements" ended I0331 09:05:06.383534 84938 simple_test_executor.go:145] Step "[step: 02] Create huge-service" started E0331 09:05:06.420945 84938 etcd_metrics.go:147] EtcdMetrics: failed to collect etcd database size I0331 09:05:06.424616 84938 simple_test_executor.go:167] Step "[step: 02] Create huge-service" ended I0331 09:05:06.424655 84938 simple_test_executor.go:145] Step "[step: 03] Creating huge-service measurements" started I0331 09:05:06.424723 84938 wait_for_controlled_pods.go:175] WaitForControlledPodsRunning: starting wait for controlled pods measurement... I0331 09:05:06.525680 84938 simple_test_executor.go:167] Step "[step: 03] Creating huge-service measurements" ended I0331 09:05:06.525740 84938 simple_test_executor.go:145] Step "[step: 04] Creating huge-service pods" started I0331 09:05:06.567713 84938 simple_test_executor.go:167] Step "[step: 04] Creating huge-service pods" ended ... skipping 202 lines ... Specify --start=60654 in the next get-serial-port-output invocation to get only the new output starting from here. scp: /var/log/cluster-autoscaler.log*: No such file or directory scp: /var/log/fluentd.log*: No such file or directory scp: /var/log/kubelet.cov*: No such file or directory scp: /var/log/cl2-**: No such file or directory scp: /var/log/startupscript.log*: No such file or directory ERROR: (gcloud.compute.scp) [/usr/bin/scp] exited with return code [1]. Dumping logs from nodes to GCS directly at 'gs://sig-scalability-logs/pull-kubernetes-e2e-gce-100-performance/1509444298272149504' using logexporter namespace/logexporter created secret/google-service-account created daemonset.apps/logexporter created Listing marker files (gs://sig-scalability-logs/pull-kubernetes-e2e-gce-100-performance/1509444298272149504/logexported-nodes-registry) for successful nodes... CommandException: One or more URLs matched no objects. ... skipping 3 lines ...