kubernetes-sigs/kueue PR #433 pull-kueue-test-integration-main #1596133600498028544
Recent runs || View in Spyglass
Recent runs || View in Spyglass
| PR | mimowo: Design for all-or-nothing semantics for workload resource assignment |
| Result | ABORTED |
| Tests | 0 failed / 66 succeeded |
| Started | |
| Elapsed | 3m30s |
| Revision | 7b8f79d3a4411c96fd73d1e63cb02e4a978e5c40 |
| Refs |
433 |
Core Controllers Suite [AfterSuite]
Core Controllers Suite [AfterSuite]
Core Controllers Suite [BeforeSuite]
Core Controllers Suite [BeforeSuite]
Core Controllers Suite [It] ClusterQueue controller when Deleting clusterQueues Should be stuck in termination until admitted workloads finished running
Core Controllers Suite [It] ClusterQueue controller when Deleting clusterQueues Should be stuck in termination until admitted workloads finished running
Core Controllers Suite [It] ClusterQueue controller when Deleting clusterQueues Should delete clusterQueues successfully when no admitted workloads are running
Core Controllers Suite [It] ClusterQueue controller when Deleting clusterQueues Should delete clusterQueues successfully when no admitted workloads are running
Core Controllers Suite [It] ClusterQueue controller when Reconciling clusterQueue status Should update status when workloads are assigned and finish
Core Controllers Suite [It] ClusterQueue controller when Reconciling clusterQueue status Should update status when workloads are assigned and finish
Core Controllers Suite [It] Queue controller Should update status when workloads are created
Core Controllers Suite [It] Queue controller Should update status when workloads are created
Core Controllers Suite [It] ResourceFlavor controller when one clusterQueue references resourceFlavors Should delete the resourceFlavor when the corresponding clusterQueue is deleted
Core Controllers Suite [It] ResourceFlavor controller when one clusterQueue references resourceFlavors Should delete the resourceFlavor when the corresponding clusterQueue is deleted
Core Controllers Suite [It] ResourceFlavor controller when one clusterQueue references resourceFlavors Should delete the resourceFlavor when the corresponding clusterQueue no longer uses the resourceFlavor
Core Controllers Suite [It] ResourceFlavor controller when one clusterQueue references resourceFlavors Should delete the resourceFlavor when the corresponding clusterQueue no longer uses the resourceFlavor
Core Controllers Suite [It] Workload controller when Workload with RuntimeClass defined Should accumulate RuntimeClass's overhead
Core Controllers Suite [It] Workload controller when Workload with RuntimeClass defined Should accumulate RuntimeClass's overhead
Core Controllers Suite [It] Workload controller when Workload with non-existent RuntimeClass defined Should not accumulate RuntimeClass's overhead
Core Controllers Suite [It] Workload controller when Workload with non-existent RuntimeClass defined Should not accumulate RuntimeClass's overhead
Core Controllers Suite [It] Workload controller when the clusterqueue doesn't exist Should update status when workloads are created
Core Controllers Suite [It] Workload controller when the clusterqueue doesn't exist Should update status when workloads are created
Core Controllers Suite [It] Workload controller when the queue doesn't exist Should update status when workloads are created
Core Controllers Suite [It] Workload controller when the queue doesn't exist Should update status when workloads are created
Core Controllers Suite [It] Workload controller when the queue is not defined in the workload Should update status when workloads are created
Core Controllers Suite [It] Workload controller when the queue is not defined in the workload Should update status when workloads are created
Core Controllers Suite [It] Workload controller when the workload is admitted Should update the workload's condition
Core Controllers Suite [It] Workload controller when the workload is admitted Should update the workload's condition
Job Controller Suite [It] Job Webhook when With manageJobsWithoutQueueName enabled Should not update unsuspend Job successfully when adding queue name
Job Controller Suite [It] Job Webhook when With manageJobsWithoutQueueName enabled Should not update unsuspend Job successfully when adding queue name
Job Controller Suite [It] Job Webhook when With manageJobsWithoutQueueName enabled Should suspend a Job even no queue name specified
Job Controller Suite [It] Job Webhook when With manageJobsWithoutQueueName enabled Should suspend a Job even no queue name specified
Job Controller Suite [It] Job Webhook when with manageJobsWithoutQueueName disabled should not suspend a Job when no queue name specified
Job Controller Suite [It] Job Webhook when with manageJobsWithoutQueueName disabled should not suspend a Job when no queue name specified
Job Controller Suite [It] Job Webhook when with manageJobsWithoutQueueName disabled should not update unsuspend Job successfully when changing queue name
Job Controller Suite [It] Job Webhook when with manageJobsWithoutQueueName disabled should not update unsuspend Job successfully when changing queue name
Job Controller Suite [It] Job Webhook when with manageJobsWithoutQueueName disabled should suspend a Job when created in unsuspend state
Job Controller Suite [It] Job Webhook when with manageJobsWithoutQueueName disabled should suspend a Job when created in unsuspend state
Job Controller Suite [It] Job controller Should reconcile workload and job for all jobs
Job Controller Suite [It] Job controller Should reconcile workload and job for all jobs
Job Controller Suite [It] Job controller for workloads with no queue set Should reconcile jobs only when queue is set
Job Controller Suite [It] Job controller for workloads with no queue set Should reconcile jobs only when queue is set
Job Controller Suite [It] Job controller interacting with scheduler Should schedule jobs as they fit in their ClusterQueue
Job Controller Suite [It] Job controller interacting with scheduler Should schedule jobs as they fit in their ClusterQueue
Scheduler Suite [AfterSuite]
Scheduler Suite [BeforeSuite]
Scheduler Suite [It] Scheduler when Creating objects out-of-order Should admit workload when creating ResourceFlavor->LocalQueue->Workload->ClusterQueue
Scheduler Suite [It] Scheduler when Creating objects out-of-order Should admit workload when creating Workload->ClusterQueue->LocalQueue->ResourceFlavor
Scheduler Suite [It] Scheduler when Creating objects out-of-order Should admit workload when creating Workload->ResourceFlavor->ClusterQueue->LocalQueue
Scheduler Suite [It] Scheduler when Creating objects out-of-order Should admit workload when creating Workload->ResourceFlavor->LocalQueue->ClusterQueue
Scheduler Suite [It] Scheduler when Deleting clusterQueues Should not admit new created workloads
Scheduler Suite [It] Scheduler when Handling clusterQueue events Should re-enqueue by the update event of ClusterQueue
Scheduler Suite [It] Scheduler when Handling workloads events Should re-enqueue by the delete event of workload belonging to the same ClusterQueue
Scheduler Suite [It] Scheduler when Handling workloads events Should re-enqueue by the delete event of workload belonging to the same Cohort
Scheduler Suite [It] Scheduler when Queueing with StrictFIFO Should schedule workloads by their priority strictly
Scheduler Suite [It] Scheduler when Queueing with StrictFIFO Workloads not matching namespaceSelector should not block others
Scheduler Suite [It] Scheduler when Referencing resourceFlavors in clusterQueue Should be inactive until the flavor is created
Scheduler Suite [It] Scheduler when Scheduling workloads on clusterQueues Should admit two small workloads after a big one finishes
Scheduler Suite [It] Scheduler when Scheduling workloads on clusterQueues Should admit workloads according to their priorities
Scheduler Suite [It] Scheduler when Scheduling workloads on clusterQueues Should admit workloads as they fit in their ClusterQueue
Scheduler Suite [It] Scheduler when Using affinity in resourceFlavors Should admit workloads with affinity to specific flavor
Scheduler Suite [It] Scheduler when Using clusterQueue NamespaceSelector Should admit workloads from the selected namespaces
Scheduler Suite [It] Scheduler when Using cohorts for fair-sharing Should admit workloads using borrowed ClusterQueue
Scheduler Suite [It] Scheduler when Using cohorts for fair-sharing Should schedule workloads borrowing quota from ClusterQueues in the same Cohort
Scheduler Suite [It] Scheduler when Using cohorts for fair-sharing Should start workloads that are under min quota before borrowing
Scheduler Suite [It] Scheduler when Using taints in resourceFlavors Should schedule workloads on tolerated flavors
... skipping 607 lines ... [1mSTEP:[0m Create workload [38;5;243m11/25/22 13:29:28.341[0m 2022-11-25T13:29:28.344334698Z DEBUG controller-runtime.webhook.webhooks admission/http.go:96 received request {"webhook": "/mutate-kueue-x-k8s-io-v1alpha2-workload", "UID": "24b2b095-6e91-40d7-85e2-919677a2ac2c", "kind": "kueue.x-k8s.io/v1alpha2, Kind=Workload", "resource": {"group":"kueue.x-k8s.io","version":"v1alpha2","resource":"workloads"}} 2022-11-25T13:29:28.344722735Z DEBUG controller-runtime.webhook.webhooks admission/http.go:143 wrote response {"webhook": "/mutate-kueue-x-k8s-io-v1alpha2-workload", "code": 200, "reason": "", "UID": "24b2b095-6e91-40d7-85e2-919677a2ac2c", "allowed": true} 2022-11-25T13:29:28.34648607Z DEBUG controller-runtime.webhook.webhooks admission/http.go:96 received request {"webhook": "/validate-kueue-x-k8s-io-v1alpha2-workload", "UID": "81273184-6ad2-4893-9f72-e78a01af1937", "kind": "kueue.x-k8s.io/v1alpha2, Kind=Workload", "resource": {"group":"kueue.x-k8s.io","version":"v1alpha2","resource":"workloads"}} 2022-11-25T13:29:28.346736029Z DEBUG controller-runtime.webhook.webhooks admission/http.go:143 wrote response {"webhook": "/validate-kueue-x-k8s-io-v1alpha2-workload", "code": 200, "reason": "", "UID": "81273184-6ad2-4893-9f72-e78a01af1937", "allowed": true} 2022-11-25T13:29:28.349640901Z LEVEL(-2) workload-reconciler core/workload_controller.go:119 Workload create event {"workload": {"name":"one","namespace":"core-workload-kghgd"}, "queue": "queue", "status": "admitted"} 2022-11-25T13:29:28.349757806Z ERROR workload-reconciler core/workload_controller.go:287 Could not get RuntimeClass {"error": "RuntimeClass.node.k8s.io \"kata\" not found"} sigs.k8s.io/kueue/pkg/controller/core.handlePodOverhead /home/prow/go/src/sigs.k8s.io/kueue/pkg/controller/core/workload_controller.go:287 sigs.k8s.io/kueue/pkg/controller/core.(*WorkloadReconciler).Create /home/prow/go/src/sigs.k8s.io/kueue/pkg/controller/core/workload_controller.go:126 sigs.k8s.io/controller-runtime/pkg/source/internal.EventHandler.OnAdd /home/prow/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.12.3/pkg/source/internal/eventsource.go:57 ... skipping 13 lines ... /home/prow/go/pkg/mod/k8s.io/apimachinery@v0.24.3/pkg/util/wait/wait.go:73 2022-11-25T13:29:28.349923524Z LEVEL(-2) core/workload_controller.go:82 Reconciling Workload {"controller": "workload", "controllerGroup": "kueue.x-k8s.io", "controllerKind": "Workload", "workload": {"name":"one","namespace":"core-workload-kghgd"}, "namespace": "core-workload-kghgd", "name": "one", "reconcileID": "3087f1ad-ef87-480d-8d1b-f381b21e4980", "workload": {"name":"one","namespace":"core-workload-kghgd"}} 2022-11-25T13:29:28.350680065Z LEVEL(-3) localqueue-reconciler core/localqueue_controller.go:144 Got Workload event {"workload": {"name":"one","namespace":"core-workload-kghgd"}} 2022-11-25T13:29:28.350776376Z LEVEL(-2) cluster-queue-reconciler core/clusterqueue_controller.go:201 Got Workload event {"workload": {"name":"one","namespace":"core-workload-kghgd"}} [1mSTEP:[0m Got ClusterQueueStatus [38;5;243m11/25/22 13:29:28.351[0m 2022-11-25T13:29:28.356637176Z LEVEL(-2) workload-reconciler core/workload_controller.go:198 Workload update event {"workload": {"name":"one","namespace":"core-workload-kghgd"}, "queue": "queue", "status": "admitted", "clusterQueue": "clusterqueue"} 2022-11-25T13:29:28.356732021Z ERROR workload-reconciler core/workload_controller.go:287 Could not get RuntimeClass {"error": "RuntimeClass.node.k8s.io \"kata\" not found"} sigs.k8s.io/kueue/pkg/controller/core.handlePodOverhead /home/prow/go/src/sigs.k8s.io/kueue/pkg/controller/core/workload_controller.go:287 sigs.k8s.io/kueue/pkg/controller/core.(*WorkloadReconciler).Update /home/prow/go/src/sigs.k8s.io/kueue/pkg/controller/core/workload_controller.go:202 sigs.k8s.io/controller-runtime/pkg/source/internal.EventHandler.OnUpdate /home/prow/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.12.3/pkg/source/internal/eventsource.go:88 ... skipping 54 lines ... [1mSTEP:[0m Create workload [38;5;243m11/25/22 13:29:28.341[0m 2022-11-25T13:29:28.344334698Z DEBUG controller-runtime.webhook.webhooks admission/http.go:96 received request {"webhook": "/mutate-kueue-x-k8s-io-v1alpha2-workload", "UID": "24b2b095-6e91-40d7-85e2-919677a2ac2c", "kind": "kueue.x-k8s.io/v1alpha2, Kind=Workload", "resource": {"group":"kueue.x-k8s.io","version":"v1alpha2","resource":"workloads"}} 2022-11-25T13:29:28.344722735Z DEBUG controller-runtime.webhook.webhooks admission/http.go:143 wrote response {"webhook": "/mutate-kueue-x-k8s-io-v1alpha2-workload", "code": 200, "reason": "", "UID": "24b2b095-6e91-40d7-85e2-919677a2ac2c", "allowed": true} 2022-11-25T13:29:28.34648607Z DEBUG controller-runtime.webhook.webhooks admission/http.go:96 received request {"webhook": "/validate-kueue-x-k8s-io-v1alpha2-workload", "UID": "81273184-6ad2-4893-9f72-e78a01af1937", "kind": "kueue.x-k8s.io/v1alpha2, Kind=Workload", "resource": {"group":"kueue.x-k8s.io","version":"v1alpha2","resource":"workloads"}} 2022-11-25T13:29:28.346736029Z DEBUG controller-runtime.webhook.webhooks admission/http.go:143 wrote response {"webhook": "/validate-kueue-x-k8s-io-v1alpha2-workload", "code": 200, "reason": "", "UID": "81273184-6ad2-4893-9f72-e78a01af1937", "allowed": true} 2022-11-25T13:29:28.349640901Z LEVEL(-2) workload-reconciler core/workload_controller.go:119 Workload create event {"workload": {"name":"one","namespace":"core-workload-kghgd"}, "queue": "queue", "status": "admitted"} 2022-11-25T13:29:28.349757806Z ERROR workload-reconciler core/workload_controller.go:287 Could not get RuntimeClass {"error": "RuntimeClass.node.k8s.io \"kata\" not found"} sigs.k8s.io/kueue/pkg/controller/core.handlePodOverhead /home/prow/go/src/sigs.k8s.io/kueue/pkg/controller/core/workload_controller.go:287 sigs.k8s.io/kueue/pkg/controller/core.(*WorkloadReconciler).Create /home/prow/go/src/sigs.k8s.io/kueue/pkg/controller/core/workload_controller.go:126 sigs.k8s.io/controller-runtime/pkg/source/internal.EventHandler.OnAdd /home/prow/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.12.3/pkg/source/internal/eventsource.go:57 ... skipping 13 lines ... /home/prow/go/pkg/mod/k8s.io/apimachinery@v0.24.3/pkg/util/wait/wait.go:73 2022-11-25T13:29:28.349923524Z LEVEL(-2) core/workload_controller.go:82 Reconciling Workload {"controller": "workload", "controllerGroup": "kueue.x-k8s.io", "controllerKind": "Workload", "workload": {"name":"one","namespace":"core-workload-kghgd"}, "namespace": "core-workload-kghgd", "name": "one", "reconcileID": "3087f1ad-ef87-480d-8d1b-f381b21e4980", "workload": {"name":"one","namespace":"core-workload-kghgd"}} 2022-11-25T13:29:28.350680065Z LEVEL(-3) localqueue-reconciler core/localqueue_controller.go:144 Got Workload event {"workload": {"name":"one","namespace":"core-workload-kghgd"}} 2022-11-25T13:29:28.350776376Z LEVEL(-2) cluster-queue-reconciler core/clusterqueue_controller.go:201 Got Workload event {"workload": {"name":"one","namespace":"core-workload-kghgd"}} [1mSTEP:[0m Got ClusterQueueStatus [38;5;243m11/25/22 13:29:28.351[0m 2022-11-25T13:29:28.356637176Z LEVEL(-2) workload-reconciler core/workload_controller.go:198 Workload update event {"workload": {"name":"one","namespace":"core-workload-kghgd"}, "queue": "queue", "status": "admitted", "clusterQueue": "clusterqueue"} 2022-11-25T13:29:28.356732021Z ERROR workload-reconciler core/workload_controller.go:287 Could not get RuntimeClass {"error": "RuntimeClass.node.k8s.io \"kata\" not found"} sigs.k8s.io/kueue/pkg/controller/core.handlePodOverhead /home/prow/go/src/sigs.k8s.io/kueue/pkg/controller/core/workload_controller.go:287 sigs.k8s.io/kueue/pkg/controller/core.(*WorkloadReconciler).Update /home/prow/go/src/sigs.k8s.io/kueue/pkg/controller/core/workload_controller.go:202 sigs.k8s.io/controller-runtime/pkg/source/internal.EventHandler.OnUpdate /home/prow/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.12.3/pkg/source/internal/eventsource.go:88 ... skipping 682 lines ... 2022-11-25T13:29:34.363016477Z LEVEL(-2) core/workload_controller.go:82 Reconciling Workload {"controller": "workload", "controllerGroup": "kueue.x-k8s.io", "controllerKind": "Workload", "workload": {"name":"five","namespace":"core-clusterqueue-zmzxm"}, "namespace": "core-clusterqueue-zmzxm", "name": "five", "reconcileID": "769ae432-2377-4960-9afe-92f13ebbfeef", "workload": {"name":"five","namespace":"core-clusterqueue-zmzxm"}} 2022-11-25T13:29:34.363022379Z LEVEL(-3) localqueue-reconciler core/localqueue_controller.go:144 Got Workload event {"workload": {"name":"five","namespace":"core-clusterqueue-zmzxm"}} 2022-11-25T13:29:34.363071197Z LEVEL(-3) localqueue-reconciler core/localqueue_controller.go:144 Got Workload event {"workload": {"name":"five","namespace":"core-clusterqueue-zmzxm"}} 2022-11-25T13:29:34.363095008Z LEVEL(-2) cluster-queue-reconciler core/clusterqueue_controller.go:201 Got Workload event {"workload": {"name":"five","namespace":"core-clusterqueue-zmzxm"}} 2022-11-25T13:29:34.363109707Z LEVEL(-2) cluster-queue-reconciler core/clusterqueue_controller.go:201 Got Workload event {"workload": {"name":"five","namespace":"core-clusterqueue-zmzxm"}} 2022-11-25T13:29:34.369456269Z LEVEL(-2) workload-reconciler core/workload_controller.go:198 Workload update event {"workload": {"name":"five","namespace":"core-clusterqueue-zmzxm"}, "queue": "other", "status": "admitted", "clusterQueue": "other"} 2022-11-25T13:29:34.36953648Z ERROR workload-reconciler core/workload_controller.go:240 Updating workload in cache {"workload": {"name":"five","namespace":"core-clusterqueue-zmzxm"}, "queue": "other", "status": "admitted", "clusterQueue": "other", "error": "old ClusterQueue doesn't exist"} sigs.k8s.io/kueue/pkg/controller/core.(*WorkloadReconciler).Update /home/prow/go/src/sigs.k8s.io/kueue/pkg/controller/core/workload_controller.go:240 sigs.k8s.io/controller-runtime/pkg/source/internal.EventHandler.OnUpdate /home/prow/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.12.3/pkg/source/internal/eventsource.go:88 k8s.io/client-go/tools/cache.(*processorListener).run.func1 /home/prow/go/pkg/mod/k8s.io/client-go@v0.24.3/tools/cache/shared_informer.go:816 ... skipping 47 lines ... 2022-11-25T13:29:35.437285889Z LEVEL(-3) localqueue-reconciler core/localqueue_controller.go:144 Got Workload event {"workload": {"name":"four","namespace":"core-clusterqueue-zmzxm"}} 2022-11-25T13:29:35.437398365Z LEVEL(-3) localqueue-reconciler core/localqueue_controller.go:144 Got Workload event {"workload": {"name":"four","namespace":"core-clusterqueue-zmzxm"}} 2022-11-25T13:29:35.437386308Z LEVEL(-2) core/workload_controller.go:82 Reconciling Workload {"controller": "workload", "controllerGroup": "kueue.x-k8s.io", "controllerKind": "Workload", "workload": {"name":"four","namespace":"core-clusterqueue-zmzxm"}, "namespace": "core-clusterqueue-zmzxm", "name": "four", "reconcileID": "7a7282a4-f08d-4127-8131-15efe3575b30", "workload": {"name":"four","namespace":"core-clusterqueue-zmzxm"}} 2022-11-25T13:29:35.437440913Z LEVEL(-2) cluster-queue-reconciler core/clusterqueue_controller.go:201 Got Workload event {"workload": {"name":"four","namespace":"core-clusterqueue-zmzxm"}} 2022-11-25T13:29:35.437462568Z LEVEL(-2) cluster-queue-reconciler core/clusterqueue_controller.go:201 Got Workload event {"workload": {"name":"four","namespace":"core-clusterqueue-zmzxm"}} 2022-11-25T13:29:35.44683727Z LEVEL(-2) workload-reconciler core/workload_controller.go:198 Workload update event {"workload": {"name":"five","namespace":"core-clusterqueue-zmzxm"}, "queue": "other", "status": "finished", "prevStatus": "admitted", "clusterQueue": "other"} 2022-11-25T13:29:35.446908566Z ERROR workload-reconciler core/workload_controller.go:207 Failed to delete workload from cache {"workload": {"name":"five","namespace":"core-clusterqueue-zmzxm"}, "queue": "other", "status": "finished", "prevStatus": "admitted", "clusterQueue": "other", "error": "cluster queue not found"} sigs.k8s.io/kueue/pkg/controller/core.(*WorkloadReconciler).Update /home/prow/go/src/sigs.k8s.io/kueue/pkg/controller/core/workload_controller.go:207 sigs.k8s.io/controller-runtime/pkg/source/internal.EventHandler.OnUpdate /home/prow/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.12.3/pkg/source/internal/eventsource.go:88 k8s.io/client-go/tools/cache.(*processorListener).run.func1 /home/prow/go/pkg/mod/k8s.io/client-go@v0.24.3/tools/cache/shared_informer.go:816 ... skipping 31 lines ... 2022-11-25T13:29:36.486991166Z DEBUG controller-runtime.webhook.webhooks admission/http.go:96 received request {"webhook": "/validate-kueue-x-k8s-io-v1alpha2-clusterqueue", "UID": "85f64ef6-632d-4267-a1e7-47e1e4bd57bd", "kind": "kueue.x-k8s.io/v1alpha2, Kind=ClusterQueue", "resource": {"group":"kueue.x-k8s.io","version":"v1alpha2","resource":"clusterqueues"}} 2022-11-25T13:29:36.48764843Z DEBUG controller-runtime.webhook.webhooks admission/http.go:143 wrote response {"webhook": "/validate-kueue-x-k8s-io-v1alpha2-clusterqueue", "code": 200, "reason": "", "UID": "85f64ef6-632d-4267-a1e7-47e1e4bd57bd", "allowed": true} 2022-11-25T13:29:36.489796804Z LEVEL(-2) localqueue-reconciler core/localqueue_controller.go:119 LocalQueue delete event {"localQueue": {"name":"queue","namespace":"core-clusterqueue-zmzxm"}} 2022-11-25T13:29:36.493160498Z LEVEL(-2) cluster-queue-reconciler core/clusterqueue_controller.go:165 ClusterQueue delete event {"clusterQueue": {"name":"cluster-queue"}} 2022-11-25T13:29:36.493374897Z LEVEL(-2) resourceflavor-reconciler core/resourceflavor_controller.go:156 Got generic event {"obj": {"name":"cluster-queue"}, "kind": "/, Kind="} 2022-11-25T13:29:36.497006782Z LEVEL(-2) workload-reconciler core/workload_controller.go:149 Workload delete event {"workload": {"name":"five","namespace":"core-clusterqueue-zmzxm"}, "queue": "other", "status": "finished"} 2022-11-25T13:29:36.497070979Z ERROR workload-reconciler core/workload_controller.go:158 Failed to delete workload from cache {"workload": {"name":"five","namespace":"core-clusterqueue-zmzxm"}, "queue": "other", "status": "finished", "error": "cluster queue not found"} sigs.k8s.io/kueue/pkg/controller/core.(*WorkloadReconciler).Delete /home/prow/go/src/sigs.k8s.io/kueue/pkg/controller/core/workload_controller.go:158 sigs.k8s.io/controller-runtime/pkg/source/internal.EventHandler.OnDelete /home/prow/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.12.3/pkg/source/internal/eventsource.go:131 k8s.io/client-go/tools/cache.(*processorListener).run.func1 /home/prow/go/pkg/mod/k8s.io/client-go@v0.24.3/tools/cache/shared_informer.go:820 ... skipping 9 lines ... /home/prow/go/pkg/mod/k8s.io/client-go@v0.24.3/tools/cache/shared_informer.go:812 k8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1 /home/prow/go/pkg/mod/k8s.io/apimachinery@v0.24.3/pkg/util/wait/wait.go:73 2022-11-25T13:29:36.497177148Z LEVEL(-3) localqueue-reconciler core/localqueue_controller.go:144 Got Workload event {"workload": {"name":"five","namespace":"core-clusterqueue-zmzxm"}} 2022-11-25T13:29:36.497252374Z LEVEL(-2) cluster-queue-reconciler core/clusterqueue_controller.go:201 Got Workload event {"workload": {"name":"five","namespace":"core-clusterqueue-zmzxm"}} 2022-11-25T13:29:36.500087817Z LEVEL(-2) workload-reconciler core/workload_controller.go:149 Workload delete event {"workload": {"name":"four","namespace":"core-clusterqueue-zmzxm"}, "queue": "queue", "status": "finished"} 2022-11-25T13:29:36.500155049Z ERROR workload-reconciler core/workload_controller.go:158 Failed to delete workload from cache {"workload": {"name":"four","namespace":"core-clusterqueue-zmzxm"}, "queue": "queue", "status": "finished", "error": "cluster queue not found"} sigs.k8s.io/kueue/pkg/controller/core.(*WorkloadReconciler).Delete /home/prow/go/src/sigs.k8s.io/kueue/pkg/controller/core/workload_controller.go:158 sigs.k8s.io/controller-runtime/pkg/source/internal.EventHandler.OnDelete /home/prow/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.12.3/pkg/source/internal/eventsource.go:131 k8s.io/client-go/tools/cache.(*processorListener).run.func1 /home/prow/go/pkg/mod/k8s.io/client-go@v0.24.3/tools/cache/shared_informer.go:820 ... skipping 9 lines ... /home/prow/go/pkg/mod/k8s.io/client-go@v0.24.3/tools/cache/shared_informer.go:812 k8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1 /home/prow/go/pkg/mod/k8s.io/apimachinery@v0.24.3/pkg/util/wait/wait.go:73 2022-11-25T13:29:36.500253092Z LEVEL(-3) localqueue-reconciler core/localqueue_controller.go:144 Got Workload event {"workload": {"name":"four","namespace":"core-clusterqueue-zmzxm"}} 2022-11-25T13:29:36.500326149Z LEVEL(-2) cluster-queue-reconciler core/clusterqueue_controller.go:201 Got Workload event {"workload": {"name":"four","namespace":"core-clusterqueue-zmzxm"}} 2022-11-25T13:29:36.502373726Z LEVEL(-2) workload-reconciler core/workload_controller.go:149 Workload delete event {"workload": {"name":"one","namespace":"core-clusterqueue-zmzxm"}, "queue": "queue", "status": "finished"} 2022-11-25T13:29:36.502422248Z ERROR workload-reconciler core/workload_controller.go:158 Failed to delete workload from cache {"workload": {"name":"one","namespace":"core-clusterqueue-zmzxm"}, "queue": "queue", "status": "finished", "error": "cluster queue not found"} sigs.k8s.io/kueue/pkg/controller/core.(*WorkloadReconciler).Delete /home/prow/go/src/sigs.k8s.io/kueue/pkg/controller/core/workload_controller.go:158 sigs.k8s.io/controller-runtime/pkg/source/internal.EventHandler.OnDelete /home/prow/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.12.3/pkg/source/internal/eventsource.go:131 k8s.io/client-go/tools/cache.(*processorListener).run.func1 /home/prow/go/pkg/mod/k8s.io/client-go@v0.24.3/tools/cache/shared_informer.go:820 ... skipping 12 lines ... 2022-11-25T13:29:36.502563215Z LEVEL(-3) localqueue-reconciler core/localqueue_controller.go:144 Got Workload event {"workload": {"name":"one","namespace":"core-clusterqueue-zmzxm"}} 2022-11-25T13:29:36.502655372Z LEVEL(-2) cluster-queue-reconciler core/clusterqueue_controller.go:201 Got Workload event {"workload": {"name":"one","namespace":"core-clusterqueue-zmzxm"}} 2022-11-25T13:29:36.505412903Z LEVEL(-2) workload-reconciler core/workload_controller.go:149 Workload delete event {"workload": {"name":"six","namespace":"core-clusterqueue-zmzxm"}, "queue": "queue", "status": "finished"} 2022-11-25T13:29:36.505549631Z LEVEL(-2) cluster-queue-reconciler core/clusterqueue_controller.go:201 Got Workload event {"workload": {"name":"six","namespace":"core-clusterqueue-zmzxm"}} 2022-11-25T13:29:36.505519634Z LEVEL(-3) localqueue-reconciler core/localqueue_controller.go:144 Got Workload event {"workload": {"name":"six","namespace":"core-clusterqueue-zmzxm"}} 2022-11-25T13:29:36.508727549Z LEVEL(-2) workload-reconciler core/workload_controller.go:149 Workload delete event {"workload": {"name":"three","namespace":"core-clusterqueue-zmzxm"}, "queue": "queue", "status": "finished"} 2022-11-25T13:29:36.50877532Z ERROR workload-reconciler core/workload_controller.go:158 Failed to delete workload from cache {"workload": {"name":"three","namespace":"core-clusterqueue-zmzxm"}, "queue": "queue", "status": "finished", "error": "cluster queue not found"} sigs.k8s.io/kueue/pkg/controller/core.(*WorkloadReconciler).Delete /home/prow/go/src/sigs.k8s.io/kueue/pkg/controller/core/workload_controller.go:158 sigs.k8s.io/controller-runtime/pkg/source/internal.EventHandler.OnDelete /home/prow/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.12.3/pkg/source/internal/eventsource.go:131 k8s.io/client-go/tools/cache.(*processorListener).run.func1 /home/prow/go/pkg/mod/k8s.io/client-go@v0.24.3/tools/cache/shared_informer.go:820 ... skipping 9 lines ... /home/prow/go/pkg/mod/k8s.io/client-go@v0.24.3/tools/cache/shared_informer.go:812 k8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1 /home/prow/go/pkg/mod/k8s.io/apimachinery@v0.24.3/pkg/util/wait/wait.go:73 2022-11-25T13:29:36.508883707Z LEVEL(-2) cluster-queue-reconciler core/clusterqueue_controller.go:201 Got Workload event {"workload": {"name":"three","namespace":"core-clusterqueue-zmzxm"}} 2022-11-25T13:29:36.508849012Z LEVEL(-3) localqueue-reconciler core/localqueue_controller.go:144 Got Workload event {"workload": {"name":"three","namespace":"core-clusterqueue-zmzxm"}} 2022-11-25T13:29:36.511893198Z LEVEL(-2) workload-reconciler core/workload_controller.go:149 Workload delete event {"workload": {"name":"two","namespace":"core-clusterqueue-zmzxm"}, "queue": "queue", "status": "finished"} 2022-11-25T13:29:36.511949455Z ERROR workload-reconciler core/workload_controller.go:158 Failed to delete workload from cache {"workload": {"name":"two","namespace":"core-clusterqueue-zmzxm"}, "queue": "queue", "status": "finished", "error": "cluster queue not found"} sigs.k8s.io/kueue/pkg/controller/core.(*WorkloadReconciler).Delete /home/prow/go/src/sigs.k8s.io/kueue/pkg/controller/core/workload_controller.go:158 sigs.k8s.io/controller-runtime/pkg/source/internal.EventHandler.OnDelete /home/prow/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.12.3/pkg/source/internal/eventsource.go:131 k8s.io/client-go/tools/cache.(*processorListener).run.func1 /home/prow/go/pkg/mod/k8s.io/client-go@v0.24.3/tools/cache/shared_informer.go:820 ... skipping 204 lines ... 2022-11-25T13:29:34.363016477Z LEVEL(-2) core/workload_controller.go:82 Reconciling Workload {"controller": "workload", "controllerGroup": "kueue.x-k8s.io", "controllerKind": "Workload", "workload": {"name":"five","namespace":"core-clusterqueue-zmzxm"}, "namespace": "core-clusterqueue-zmzxm", "name": "five", "reconcileID": "769ae432-2377-4960-9afe-92f13ebbfeef", "workload": {"name":"five","namespace":"core-clusterqueue-zmzxm"}} 2022-11-25T13:29:34.363022379Z LEVEL(-3) localqueue-reconciler core/localqueue_controller.go:144 Got Workload event {"workload": {"name":"five","namespace":"core-clusterqueue-zmzxm"}} 2022-11-25T13:29:34.363071197Z LEVEL(-3) localqueue-reconciler core/localqueue_controller.go:144 Got Workload event {"workload": {"name":"five","namespace":"core-clusterqueue-zmzxm"}} 2022-11-25T13:29:34.363095008Z LEVEL(-2) cluster-queue-reconciler core/clusterqueue_controller.go:201 Got Workload event {"workload": {"name":"five","namespace":"core-clusterqueue-zmzxm"}} 2022-11-25T13:29:34.363109707Z LEVEL(-2) cluster-queue-reconciler core/clusterqueue_controller.go:201 Got Workload event {"workload": {"name":"five","namespace":"core-clusterqueue-zmzxm"}} 2022-11-25T13:29:34.369456269Z LEVEL(-2) workload-reconciler core/workload_controller.go:198 Workload update event {"workload": {"name":"five","namespace":"core-clusterqueue-zmzxm"}, "queue": "other", "status": "admitted", "clusterQueue": "other"} 2022-11-25T13:29:34.36953648Z ERROR workload-reconciler core/workload_controller.go:240 Updating workload in cache {"workload": {"name":"five","namespace":"core-clusterqueue-zmzxm"}, "queue": "other", "status": "admitted", "clusterQueue": "other", "error": "old ClusterQueue doesn't exist"} sigs.k8s.io/kueue/pkg/controller/core.(*WorkloadReconciler).Update /home/prow/go/src/sigs.k8s.io/kueue/pkg/controller/core/workload_controller.go:240 sigs.k8s.io/controller-runtime/pkg/source/internal.EventHandler.OnUpdate /home/prow/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.12.3/pkg/source/internal/eventsource.go:88 k8s.io/client-go/tools/cache.(*processorListener).run.func1 /home/prow/go/pkg/mod/k8s.io/client-go@v0.24.3/tools/cache/shared_informer.go:816 ... skipping 47 lines ... 2022-11-25T13:29:35.437285889Z LEVEL(-3) localqueue-reconciler core/localqueue_controller.go:144 Got Workload event {"workload": {"name":"four","namespace":"core-clusterqueue-zmzxm"}} 2022-11-25T13:29:35.437398365Z LEVEL(-3) localqueue-reconciler core/localqueue_controller.go:144 Got Workload event {"workload": {"name":"four","namespace":"core-clusterqueue-zmzxm"}} 2022-11-25T13:29:35.437386308Z LEVEL(-2) core/workload_controller.go:82 Reconciling Workload {"controller": "workload", "controllerGroup": "kueue.x-k8s.io", "controllerKind": "Workload", "workload": {"name":"four","namespace":"core-clusterqueue-zmzxm"}, "namespace": "core-clusterqueue-zmzxm", "name": "four", "reconcileID": "7a7282a4-f08d-4127-8131-15efe3575b30", "workload": {"name":"four","namespace":"core-clusterqueue-zmzxm"}} 2022-11-25T13:29:35.437440913Z LEVEL(-2) cluster-queue-reconciler core/clusterqueue_controller.go:201 Got Workload event {"workload": {"name":"four","namespace":"core-clusterqueue-zmzxm"}} 2022-11-25T13:29:35.437462568Z LEVEL(-2) cluster-queue-reconciler core/clusterqueue_controller.go:201 Got Workload event {"workload": {"name":"four","namespace":"core-clusterqueue-zmzxm"}} 2022-11-25T13:29:35.44683727Z LEVEL(-2) workload-reconciler core/workload_controller.go:198 Workload update event {"workload": {"name":"five","namespace":"core-clusterqueue-zmzxm"}, "queue": "other", "status": "finished", "prevStatus": "admitted", "clusterQueue": "other"} 2022-11-25T13:29:35.446908566Z ERROR workload-reconciler core/workload_controller.go:207 Failed to delete workload from cache {"workload": {"name":"five","namespace":"core-clusterqueue-zmzxm"}, "queue": "other", "status": "finished", "prevStatus": "admitted", "clusterQueue": "other", "error": "cluster queue not found"} sigs.k8s.io/kueue/pkg/controller/core.(*WorkloadReconciler).Update /home/prow/go/src/sigs.k8s.io/kueue/pkg/controller/core/workload_controller.go:207 sigs.k8s.io/controller-runtime/pkg/source/internal.EventHandler.OnUpdate /home/prow/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.12.3/pkg/source/internal/eventsource.go:88 k8s.io/client-go/tools/cache.(*processorListener).run.func1 /home/prow/go/pkg/mod/k8s.io/client-go@v0.24.3/tools/cache/shared_informer.go:816 ... skipping 31 lines ... 2022-11-25T13:29:36.486991166Z DEBUG controller-runtime.webhook.webhooks admission/http.go:96 received request {"webhook": "/validate-kueue-x-k8s-io-v1alpha2-clusterqueue", "UID": "85f64ef6-632d-4267-a1e7-47e1e4bd57bd", "kind": "kueue.x-k8s.io/v1alpha2, Kind=ClusterQueue", "resource": {"group":"kueue.x-k8s.io","version":"v1alpha2","resource":"clusterqueues"}} 2022-11-25T13:29:36.48764843Z DEBUG controller-runtime.webhook.webhooks admission/http.go:143 wrote response {"webhook": "/validate-kueue-x-k8s-io-v1alpha2-clusterqueue", "code": 200, "reason": "", "UID": "85f64ef6-632d-4267-a1e7-47e1e4bd57bd", "allowed": true} 2022-11-25T13:29:36.489796804Z LEVEL(-2) localqueue-reconciler core/localqueue_controller.go:119 LocalQueue delete event {"localQueue": {"name":"queue","namespace":"core-clusterqueue-zmzxm"}} 2022-11-25T13:29:36.493160498Z LEVEL(-2) cluster-queue-reconciler core/clusterqueue_controller.go:165 ClusterQueue delete event {"clusterQueue": {"name":"cluster-queue"}} 2022-11-25T13:29:36.493374897Z LEVEL(-2) resourceflavor-reconciler core/resourceflavor_controller.go:156 Got generic event {"obj": {"name":"cluster-queue"}, "kind": "/, Kind="} 2022-11-25T13:29:36.497006782Z LEVEL(-2) workload-reconciler core/workload_controller.go:149 Workload delete event {"workload": {"name":"five","namespace":"core-clusterqueue-zmzxm"}, "queue": "other", "status": "finished"} 2022-11-25T13:29:36.497070979Z ERROR workload-reconciler core/workload_controller.go:158 Failed to delete workload from cache {"workload": {"name":"five","namespace":"core-clusterqueue-zmzxm"}, "queue": "other", "status": "finished", "error": "cluster queue not found"} sigs.k8s.io/kueue/pkg/controller/core.(*WorkloadReconciler).Delete /home/prow/go/src/sigs.k8s.io/kueue/pkg/controller/core/workload_controller.go:158 sigs.k8s.io/controller-runtime/pkg/source/internal.EventHandler.OnDelete /home/prow/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.12.3/pkg/source/internal/eventsource.go:131 k8s.io/client-go/tools/cache.(*processorListener).run.func1 /home/prow/go/pkg/mod/k8s.io/client-go@v0.24.3/tools/cache/shared_informer.go:820 ... skipping 9 lines ... /home/prow/go/pkg/mod/k8s.io/client-go@v0.24.3/tools/cache/shared_informer.go:812 k8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1 /home/prow/go/pkg/mod/k8s.io/apimachinery@v0.24.3/pkg/util/wait/wait.go:73 2022-11-25T13:29:36.497177148Z LEVEL(-3) localqueue-reconciler core/localqueue_controller.go:144 Got Workload event {"workload": {"name":"five","namespace":"core-clusterqueue-zmzxm"}} 2022-11-25T13:29:36.497252374Z LEVEL(-2) cluster-queue-reconciler core/clusterqueue_controller.go:201 Got Workload event {"workload": {"name":"five","namespace":"core-clusterqueue-zmzxm"}} 2022-11-25T13:29:36.500087817Z LEVEL(-2) workload-reconciler core/workload_controller.go:149 Workload delete event {"workload": {"name":"four","namespace":"core-clusterqueue-zmzxm"}, "queue": "queue", "status": "finished"} 2022-11-25T13:29:36.500155049Z ERROR workload-reconciler core/workload_controller.go:158 Failed to delete workload from cache {"workload": {"name":"four","namespace":"core-clusterqueue-zmzxm"}, "queue": "queue", "status": "finished", "error": "cluster queue not found"} sigs.k8s.io/kueue/pkg/controller/core.(*WorkloadReconciler).Delete /home/prow/go/src/sigs.k8s.io/kueue/pkg/controller/core/workload_controller.go:158 sigs.k8s.io/controller-runtime/pkg/source/internal.EventHandler.OnDelete /home/prow/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.12.3/pkg/source/internal/eventsource.go:131 k8s.io/client-go/tools/cache.(*processorListener).run.func1 /home/prow/go/pkg/mod/k8s.io/client-go@v0.24.3/tools/cache/shared_informer.go:820 ... skipping 9 lines ... /home/prow/go/pkg/mod/k8s.io/client-go@v0.24.3/tools/cache/shared_informer.go:812 k8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1 /home/prow/go/pkg/mod/k8s.io/apimachinery@v0.24.3/pkg/util/wait/wait.go:73 2022-11-25T13:29:36.500253092Z LEVEL(-3) localqueue-reconciler core/localqueue_controller.go:144 Got Workload event {"workload": {"name":"four","namespace":"core-clusterqueue-zmzxm"}} 2022-11-25T13:29:36.500326149Z LEVEL(-2) cluster-queue-reconciler core/clusterqueue_controller.go:201 Got Workload event {"workload": {"name":"four","namespace":"core-clusterqueue-zmzxm"}} 2022-11-25T13:29:36.502373726Z LEVEL(-2) workload-reconciler core/workload_controller.go:149 Workload delete event {"workload": {"name":"one","namespace":"core-clusterqueue-zmzxm"}, "queue": "queue", "status": "finished"} 2022-11-25T13:29:36.502422248Z ERROR workload-reconciler core/workload_controller.go:158 Failed to delete workload from cache {"workload": {"name":"one","namespace":"core-clusterqueue-zmzxm"}, "queue": "queue", "status": "finished", "error": "cluster queue not found"} sigs.k8s.io/kueue/pkg/controller/core.(*WorkloadReconciler).Delete /home/prow/go/src/sigs.k8s.io/kueue/pkg/controller/core/workload_controller.go:158 sigs.k8s.io/controller-runtime/pkg/source/internal.EventHandler.OnDelete /home/prow/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.12.3/pkg/source/internal/eventsource.go:131 k8s.io/client-go/tools/cache.(*processorListener).run.func1 /home/prow/go/pkg/mod/k8s.io/client-go@v0.24.3/tools/cache/shared_informer.go:820 ... skipping 12 lines ... 2022-11-25T13:29:36.502563215Z LEVEL(-3) localqueue-reconciler core/localqueue_controller.go:144 Got Workload event {"workload": {"name":"one","namespace":"core-clusterqueue-zmzxm"}} 2022-11-25T13:29:36.502655372Z LEVEL(-2) cluster-queue-reconciler core/clusterqueue_controller.go:201 Got Workload event {"workload": {"name":"one","namespace":"core-clusterqueue-zmzxm"}} 2022-11-25T13:29:36.505412903Z LEVEL(-2) workload-reconciler core/workload_controller.go:149 Workload delete event {"workload": {"name":"six","namespace":"core-clusterqueue-zmzxm"}, "queue": "queue", "status": "finished"} 2022-11-25T13:29:36.505549631Z LEVEL(-2) cluster-queue-reconciler core/clusterqueue_controller.go:201 Got Workload event {"workload": {"name":"six","namespace":"core-clusterqueue-zmzxm"}} 2022-11-25T13:29:36.505519634Z LEVEL(-3) localqueue-reconciler core/localqueue_controller.go:144 Got Workload event {"workload": {"name":"six","namespace":"core-clusterqueue-zmzxm"}} 2022-11-25T13:29:36.508727549Z LEVEL(-2) workload-reconciler core/workload_controller.go:149 Workload delete event {"workload": {"name":"three","namespace":"core-clusterqueue-zmzxm"}, "queue": "queue", "status": "finished"} 2022-11-25T13:29:36.50877532Z ERROR workload-reconciler core/workload_controller.go:158 Failed to delete workload from cache {"workload": {"name":"three","namespace":"core-clusterqueue-zmzxm"}, "queue": "queue", "status": "finished", "error": "cluster queue not found"} sigs.k8s.io/kueue/pkg/controller/core.(*WorkloadReconciler).Delete /home/prow/go/src/sigs.k8s.io/kueue/pkg/controller/core/workload_controller.go:158 sigs.k8s.io/controller-runtime/pkg/source/internal.EventHandler.OnDelete /home/prow/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.12.3/pkg/source/internal/eventsource.go:131 k8s.io/client-go/tools/cache.(*processorListener).run.func1 /home/prow/go/pkg/mod/k8s.io/client-go@v0.24.3/tools/cache/shared_informer.go:820 ... skipping 9 lines ... /home/prow/go/pkg/mod/k8s.io/client-go@v0.24.3/tools/cache/shared_informer.go:812 k8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1 /home/prow/go/pkg/mod/k8s.io/apimachinery@v0.24.3/pkg/util/wait/wait.go:73 2022-11-25T13:29:36.508883707Z LEVEL(-2) cluster-queue-reconciler core/clusterqueue_controller.go:201 Got Workload event {"workload": {"name":"three","namespace":"core-clusterqueue-zmzxm"}} 2022-11-25T13:29:36.508849012Z LEVEL(-3) localqueue-reconciler core/localqueue_controller.go:144 Got Workload event {"workload": {"name":"three","namespace":"core-clusterqueue-zmzxm"}} 2022-11-25T13:29:36.511893198Z LEVEL(-2) workload-reconciler core/workload_controller.go:149 Workload delete event {"workload": {"name":"two","namespace":"core-clusterqueue-zmzxm"}, "queue": "queue", "status": "finished"} 2022-11-25T13:29:36.511949455Z ERROR workload-reconciler core/workload_controller.go:158 Failed to delete workload from cache {"workload": {"name":"two","namespace":"core-clusterqueue-zmzxm"}, "queue": "queue", "status": "finished", "error": "cluster queue not found"} sigs.k8s.io/kueue/pkg/controller/core.(*WorkloadReconciler).Delete /home/prow/go/src/sigs.k8s.io/kueue/pkg/controller/core/workload_controller.go:158 sigs.k8s.io/controller-runtime/pkg/source/internal.EventHandler.OnDelete /home/prow/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.12.3/pkg/source/internal/eventsource.go:131 k8s.io/client-go/tools/cache.(*processorListener).run.func1 /home/prow/go/pkg/mod/k8s.io/client-go@v0.24.3/tools/cache/shared_informer.go:820 ... skipping 106 lines ... 2022-11-25T13:29:37.834189977Z DEBUG controller-runtime.webhook.webhooks admission/http.go:96 received request {"webhook": "/validate-kueue-x-k8s-io-v1alpha2-clusterqueue", "UID": "9f89547a-4c45-44d5-b9a5-b863022bf753", "kind": "kueue.x-k8s.io/v1alpha2, Kind=ClusterQueue", "resource": {"group":"kueue.x-k8s.io","version":"v1alpha2","resource":"clusterqueues"}} 2022-11-25T13:29:37.834480355Z DEBUG controller-runtime.webhook.webhooks admission/http.go:143 wrote response {"webhook": "/validate-kueue-x-k8s-io-v1alpha2-clusterqueue", "code": 200, "reason": "", "UID": "9f89547a-4c45-44d5-b9a5-b863022bf753", "allowed": true} 2022-11-25T13:29:37.837953133Z LEVEL(-2) cluster-queue-reconciler core/clusterqueue_controller.go:165 ClusterQueue delete event {"clusterQueue": {"name":"foo-cq"}} 2022-11-25T13:29:37.83809517Z LEVEL(-2) resourceflavor-reconciler core/resourceflavor_controller.go:156 Got generic event {"obj": {"name":"foo-cq"}, "kind": "/, Kind="} 2022-11-25T13:29:37.872371966Z LEVEL(-2) localqueue-reconciler core/localqueue_controller.go:119 LocalQueue delete event {"localQueue": {"name":"queue","namespace":"core-clusterqueue-bkmw9"}} 2022-11-25T13:29:37.878902158Z LEVEL(-2) workload-reconciler core/workload_controller.go:149 Workload delete event {"workload": {"name":"workload","namespace":"core-clusterqueue-bkmw9"}, "queue": "queue", "status": "finished"} 2022-11-25T13:29:37.878995262Z ERROR workload-reconciler core/workload_controller.go:158 Failed to delete workload from cache {"workload": {"name":"workload","namespace":"core-clusterqueue-bkmw9"}, "queue": "queue", "status": "finished", "error": "cluster queue not found"} sigs.k8s.io/kueue/pkg/controller/core.(*WorkloadReconciler).Delete /home/prow/go/src/sigs.k8s.io/kueue/pkg/controller/core/workload_controller.go:158 sigs.k8s.io/controller-runtime/pkg/source/internal.EventHandler.OnDelete /home/prow/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.12.3/pkg/source/internal/eventsource.go:131 k8s.io/client-go/tools/cache.(*processorListener).run.func1 /home/prow/go/pkg/mod/k8s.io/client-go@v0.24.3/tools/cache/shared_informer.go:820 ... skipping 62 lines ... 2022-11-25T13:29:37.834189977Z DEBUG controller-runtime.webhook.webhooks admission/http.go:96 received request {"webhook": "/validate-kueue-x-k8s-io-v1alpha2-clusterqueue", "UID": "9f89547a-4c45-44d5-b9a5-b863022bf753", "kind": "kueue.x-k8s.io/v1alpha2, Kind=ClusterQueue", "resource": {"group":"kueue.x-k8s.io","version":"v1alpha2","resource":"clusterqueues"}} 2022-11-25T13:29:37.834480355Z DEBUG controller-runtime.webhook.webhooks admission/http.go:143 wrote response {"webhook": "/validate-kueue-x-k8s-io-v1alpha2-clusterqueue", "code": 200, "reason": "", "UID": "9f89547a-4c45-44d5-b9a5-b863022bf753", "allowed": true} 2022-11-25T13:29:37.837953133Z LEVEL(-2) cluster-queue-reconciler core/clusterqueue_controller.go:165 ClusterQueue delete event {"clusterQueue": {"name":"foo-cq"}} 2022-11-25T13:29:37.83809517Z LEVEL(-2) resourceflavor-reconciler core/resourceflavor_controller.go:156 Got generic event {"obj": {"name":"foo-cq"}, "kind": "/, Kind="} 2022-11-25T13:29:37.872371966Z LEVEL(-2) localqueue-reconciler core/localqueue_controller.go:119 LocalQueue delete event {"localQueue": {"name":"queue","namespace":"core-clusterqueue-bkmw9"}} 2022-11-25T13:29:37.878902158Z LEVEL(-2) workload-reconciler core/workload_controller.go:149 Workload delete event {"workload": {"name":"workload","namespace":"core-clusterqueue-bkmw9"}, "queue": "queue", "status": "finished"} 2022-11-25T13:29:37.878995262Z ERROR workload-reconciler core/workload_controller.go:158 Failed to delete workload from cache {"workload": {"name":"workload","namespace":"core-clusterqueue-bkmw9"}, "queue": "queue", "status": "finished", "error": "cluster queue not found"} sigs.k8s.io/kueue/pkg/controller/core.(*WorkloadReconciler).Delete /home/prow/go/src/sigs.k8s.io/kueue/pkg/controller/core/workload_controller.go:158 sigs.k8s.io/controller-runtime/pkg/source/internal.EventHandler.OnDelete /home/prow/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.12.3/pkg/source/internal/eventsource.go:131 k8s.io/client-go/tools/cache.(*processorListener).run.func1 /home/prow/go/pkg/mod/k8s.io/client-go@v0.24.3/tools/cache/shared_informer.go:820 ... skipping 59 lines ... [38;5;10m[ReportAfterSuite] PASSED [0.004 seconds][0m [ReportAfterSuite] Autogenerated ReportAfterSuite for --junit-report [38;5;243mautogenerated by Ginkgo[0m [38;5;243m------------------------------[0m [38;5;10m[1mRan 12 of 12 Specs in 22.303 seconds[0m [38;5;10m[1mSUCCESS![0m -- [38;5;10m[1m12 Passed[0m | [38;5;9m[1m0 Failed[0m | [38;5;11m[1m0 Pending[0m | [38;5;14m[1m0 Skipped[0m PASS Running Suite: Job Controller Suite - /home/prow/go/src/sigs.k8s.io/kueue/test/integration/controller/job ========================================================================================================= Random Seed: [1m1669382947[0m Will run [1m8[0m of [1m8[0m specs ... skipping 34 lines ... 2022-11-25T13:29:44.754625473Z INFO controller-runtime.webhook logr@v1.2.3/logr.go:261 Serving webhook server {"host": "127.0.0.1", "port": 36919} 2022-11-25T13:29:44.754688713Z INFO controller-runtime.certwatcher logr@v1.2.3/logr.go:261 Starting certificate watcher 2022-11-25T13:29:44.776208414Z DEBUG controller-runtime.webhook.webhooks admission/http.go:96 received request {"webhook": "/mutate-batch-v1-job", "UID": "7c843bb9-5343-4be4-8e41-8a648b605b06", "kind": "batch/v1, Kind=Job", "resource": {"group":"batch","version":"v1","resource":"jobs"}} 2022-11-25T13:29:44.782412316Z DEBUG controller-runtime.webhook.webhooks admission/http.go:143 wrote response {"webhook": "/mutate-batch-v1-job", "code": 200, "reason": "", "UID": "7c843bb9-5343-4be4-8e41-8a648b605b06", "allowed": true} [1mSTEP:[0m tearing down the test environment [38;5;243m11/25/22 13:29:44.895[0m 2022-11-25T13:29:44.896425557Z DEBUG controller-runtime.certwatcher logr@v1.2.3/logr.go:261 certificate event {"event": "\"/tmp/envtest-serving-certs-2742600227/tls.key\": REMOVE"} 2022-11-25T13:29:44.896606735Z ERROR controller-runtime.certwatcher logr@v1.2.3/logr.go:279 error re-watching file {"error": "no such file or directory"} github.com/go-logr/logr.Logger.Error /home/prow/go/pkg/mod/github.com/go-logr/logr@v1.2.3/logr.go:279 sigs.k8s.io/controller-runtime/pkg/certwatcher.(*CertWatcher).handleEvent /home/prow/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.12.3/pkg/certwatcher/certwatcher.go:147 sigs.k8s.io/controller-runtime/pkg/certwatcher.(*CertWatcher).Watch /home/prow/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.12.3/pkg/certwatcher/certwatcher.go:103 2022-11-25T13:29:44.896669036Z ERROR controller-runtime.certwatcher logr@v1.2.3/logr.go:279 error re-reading certificate {"error": "open /tmp/envtest-serving-certs-2742600227/tls.crt: no such file or directory"} github.com/go-logr/logr.Logger.Error /home/prow/go/pkg/mod/github.com/go-logr/logr@v1.2.3/logr.go:279 sigs.k8s.io/controller-runtime/pkg/certwatcher.(*CertWatcher).handleEvent /home/prow/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.12.3/pkg/certwatcher/certwatcher.go:152 sigs.k8s.io/controller-runtime/pkg/certwatcher.(*CertWatcher).Watch /home/prow/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.12.3/pkg/certwatcher/certwatcher.go:103 2022-11-25T13:29:44.896749203Z DEBUG controller-runtime.certwatcher logr@v1.2.3/logr.go:261 certificate event {"event": "\"/tmp/envtest-serving-certs-2742600227/tls.crt\": REMOVE"} 2022-11-25T13:29:44.896817051Z ERROR controller-runtime.certwatcher logr@v1.2.3/logr.go:279 error re-watching file {"error": "no such file or directory"} github.com/go-logr/logr.Logger.Error /home/prow/go/pkg/mod/github.com/go-logr/logr@v1.2.3/logr.go:279 sigs.k8s.io/controller-runtime/pkg/certwatcher.(*CertWatcher).handleEvent /home/prow/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.12.3/pkg/certwatcher/certwatcher.go:147 sigs.k8s.io/controller-runtime/pkg/certwatcher.(*CertWatcher).Watch /home/prow/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.12.3/pkg/certwatcher/certwatcher.go:103 2022-11-25T13:29:44.89684279Z ERROR controller-runtime.certwatcher logr@v1.2.3/logr.go:279 error re-reading certificate {"error": "open /tmp/envtest-serving-certs-2742600227/tls.crt: no such file or directory"} github.com/go-logr/logr.Logger.Error /home/prow/go/pkg/mod/github.com/go-logr/logr@v1.2.3/logr.go:279 sigs.k8s.io/controller-runtime/pkg/certwatcher.(*CertWatcher).handleEvent /home/prow/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.12.3/pkg/certwatcher/certwatcher.go:152 sigs.k8s.io/controller-runtime/pkg/certwatcher.(*CertWatcher).Watch /home/prow/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.12.3/pkg/certwatcher/certwatcher.go:103 2022-11-25T13:29:44.954216759Z INFO logr@v1.2.3/logr.go:261 Stopping and waiting for non leader election runnables ... skipping 4 lines ... 2022-11-25T13:29:44.954507144Z INFO controller/controller.go:227 Starting workers {"controller": "job", "controllerGroup": "batch", "controllerKind": "Job", "worker count": 1} 2022-11-25T13:29:44.954553103Z INFO controller/controller.go:247 Shutdown signal received, waiting for all workers to finish {"controller": "job", "controllerGroup": "batch", "controllerKind": "Job"} 2022-11-25T13:29:44.95457013Z INFO controller/controller.go:249 All workers finished {"controller": "job", "controllerGroup": "batch", "controllerKind": "Job"} 2022-11-25T13:29:44.954587008Z INFO logr@v1.2.3/logr.go:261 Stopping and waiting for caches 2022-11-25T13:29:44.954770568Z INFO logr@v1.2.3/logr.go:261 Stopping and waiting for webhooks 2022-11-25T13:29:44.954857634Z INFO controller-runtime.webhook logr@v1.2.3/logr.go:261 shutting down webhook server 2022-11-25T13:29:44.954850706Z ERROR controller-runtime.source logr@v1.2.3/logr.go:279 failed to get informer from cache {"error": "Timeout: failed waiting for *v1.Job Informer to sync"} github.com/go-logr/logr.Logger.Error /home/prow/go/pkg/mod/github.com/go-logr/logr@v1.2.3/logr.go:279 sigs.k8s.io/controller-runtime/pkg/source.(*Kind).Start.func1.1 /home/prow/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.12.3/pkg/source/source.go:144 k8s.io/apimachinery/pkg/util/wait.runConditionWithCrashProtectionWithContext /home/prow/go/pkg/mod/k8s.io/apimachinery@v0.24.3/pkg/util/wait/wait.go:233 k8s.io/apimachinery/pkg/util/wait.poll ... skipping 46 lines ... 2022-11-25T13:29:44.754625473Z INFO controller-runtime.webhook logr@v1.2.3/logr.go:261 Serving webhook server {"host": "127.0.0.1", "port": 36919} 2022-11-25T13:29:44.754688713Z INFO controller-runtime.certwatcher logr@v1.2.3/logr.go:261 Starting certificate watcher 2022-11-25T13:29:44.776208414Z DEBUG controller-runtime.webhook.webhooks admission/http.go:96 received request {"webhook": "/mutate-batch-v1-job", "UID": "7c843bb9-5343-4be4-8e41-8a648b605b06", "kind": "batch/v1, Kind=Job", "resource": {"group":"batch","version":"v1","resource":"jobs"}} 2022-11-25T13:29:44.782412316Z DEBUG controller-runtime.webhook.webhooks admission/http.go:143 wrote response {"webhook": "/mutate-batch-v1-job", "code": 200, "reason": "", "UID": "7c843bb9-5343-4be4-8e41-8a648b605b06", "allowed": true} [1mSTEP:[0m tearing down the test environment [38;5;243m11/25/22 13:29:44.895[0m 2022-11-25T13:29:44.896425557Z DEBUG controller-runtime.certwatcher logr@v1.2.3/logr.go:261 certificate event {"event": "\"/tmp/envtest-serving-certs-2742600227/tls.key\": REMOVE"} 2022-11-25T13:29:44.896606735Z ERROR controller-runtime.certwatcher logr@v1.2.3/logr.go:279 error re-watching file {"error": "no such file or directory"} github.com/go-logr/logr.Logger.Error /home/prow/go/pkg/mod/github.com/go-logr/logr@v1.2.3/logr.go:279 sigs.k8s.io/controller-runtime/pkg/certwatcher.(*CertWatcher).handleEvent /home/prow/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.12.3/pkg/certwatcher/certwatcher.go:147 sigs.k8s.io/controller-runtime/pkg/certwatcher.(*CertWatcher).Watch /home/prow/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.12.3/pkg/certwatcher/certwatcher.go:103 2022-11-25T13:29:44.896669036Z ERROR controller-runtime.certwatcher logr@v1.2.3/logr.go:279 error re-reading certificate {"error": "open /tmp/envtest-serving-certs-2742600227/tls.crt: no such file or directory"} github.com/go-logr/logr.Logger.Error /home/prow/go/pkg/mod/github.com/go-logr/logr@v1.2.3/logr.go:279 sigs.k8s.io/controller-runtime/pkg/certwatcher.(*CertWatcher).handleEvent /home/prow/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.12.3/pkg/certwatcher/certwatcher.go:152 sigs.k8s.io/controller-runtime/pkg/certwatcher.(*CertWatcher).Watch /home/prow/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.12.3/pkg/certwatcher/certwatcher.go:103 2022-11-25T13:29:44.896749203Z DEBUG controller-runtime.certwatcher logr@v1.2.3/logr.go:261 certificate event {"event": "\"/tmp/envtest-serving-certs-2742600227/tls.crt\": REMOVE"} 2022-11-25T13:29:44.896817051Z ERROR controller-runtime.certwatcher logr@v1.2.3/logr.go:279 error re-watching file {"error": "no such file or directory"} github.com/go-logr/logr.Logger.Error /home/prow/go/pkg/mod/github.com/go-logr/logr@v1.2.3/logr.go:279 sigs.k8s.io/controller-runtime/pkg/certwatcher.(*CertWatcher).handleEvent /home/prow/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.12.3/pkg/certwatcher/certwatcher.go:147 sigs.k8s.io/controller-runtime/pkg/certwatcher.(*CertWatcher).Watch /home/prow/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.12.3/pkg/certwatcher/certwatcher.go:103 2022-11-25T13:29:44.89684279Z ERROR controller-runtime.certwatcher logr@v1.2.3/logr.go:279 error re-reading certificate {"error": "open /tmp/envtest-serving-certs-2742600227/tls.crt: no such file or directory"} github.com/go-logr/logr.Logger.Error /home/prow/go/pkg/mod/github.com/go-logr/logr@v1.2.3/logr.go:279 sigs.k8s.io/controller-runtime/pkg/certwatcher.(*CertWatcher).handleEvent /home/prow/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.12.3/pkg/certwatcher/certwatcher.go:152 sigs.k8s.io/controller-runtime/pkg/certwatcher.(*CertWatcher).Watch /home/prow/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.12.3/pkg/certwatcher/certwatcher.go:103 2022-11-25T13:29:44.954216759Z INFO logr@v1.2.3/logr.go:261 Stopping and waiting for non leader election runnables ... skipping 4 lines ... 2022-11-25T13:29:44.954507144Z INFO controller/controller.go:227 Starting workers {"controller": "job", "controllerGroup": "batch", "controllerKind": "Job", "worker count": 1} 2022-11-25T13:29:44.954553103Z INFO controller/controller.go:247 Shutdown signal received, waiting for all workers to finish {"controller": "job", "controllerGroup": "batch", "controllerKind": "Job"} 2022-11-25T13:29:44.95457013Z INFO controller/controller.go:249 All workers finished {"controller": "job", "controllerGroup": "batch", "controllerKind": "Job"} 2022-11-25T13:29:44.954587008Z INFO logr@v1.2.3/logr.go:261 Stopping and waiting for caches 2022-11-25T13:29:44.954770568Z INFO logr@v1.2.3/logr.go:261 Stopping and waiting for webhooks 2022-11-25T13:29:44.954857634Z INFO controller-runtime.webhook logr@v1.2.3/logr.go:261 shutting down webhook server 2022-11-25T13:29:44.954850706Z ERROR controller-runtime.source logr@v1.2.3/logr.go:279 failed to get informer from cache {"error": "Timeout: failed waiting for *v1.Job Informer to sync"} github.com/go-logr/logr.Logger.Error /home/prow/go/pkg/mod/github.com/go-logr/logr@v1.2.3/logr.go:279 sigs.k8s.io/controller-runtime/pkg/source.(*Kind).Start.func1.1 /home/prow/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.12.3/pkg/source/source.go:144 k8s.io/apimachinery/pkg/util/wait.runConditionWithCrashProtectionWithContext /home/prow/go/pkg/mod/k8s.io/apimachinery@v0.24.3/pkg/util/wait/wait.go:233 k8s.io/apimachinery/pkg/util/wait.poll ... skipping 43 lines ... 2022-11-25T13:29:51.534677511Z DEBUG controller-runtime.webhook.webhooks admission/http.go:96 received request {"webhook": "/mutate-batch-v1-job", "UID": "7fdbca7c-965c-4d25-9232-fcce6c00d925", "kind": "batch/v1, Kind=Job", "resource": {"group":"batch","version":"v1","resource":"jobs"}} 2022-11-25T13:29:51.535288232Z DEBUG controller-runtime.webhook.webhooks admission/http.go:143 wrote response {"webhook": "/mutate-batch-v1-job", "code": 200, "reason": "", "UID": "7fdbca7c-965c-4d25-9232-fcce6c00d925", "allowed": true} 2022-11-25T13:29:51.553955069Z DEBUG controller-runtime.webhook.webhooks admission/http.go:96 received request {"webhook": "/validate-batch-v1-job", "UID": "5f7eea0d-11d2-4223-a57c-6c4a6cc1a7b3", "kind": "batch/v1, Kind=Job", "resource": {"group":"batch","version":"v1","resource":"jobs"}} 2022-11-25T13:29:51.554489779Z DEBUG controller-runtime.webhook.webhooks admission/http.go:143 wrote response {"webhook": "/validate-batch-v1-job", "code": 403, "reason": "job.spec.suspend: Forbidden: suspend should be true when adding the queue name", "UID": "5f7eea0d-11d2-4223-a57c-6c4a6cc1a7b3", "allowed": false} [1mSTEP:[0m tearing down the test environment [38;5;243m11/25/22 13:29:51.643[0m 2022-11-25T13:29:51.644424676Z DEBUG controller-runtime.certwatcher logr@v1.2.3/logr.go:261 certificate event {"event": "\"/tmp/envtest-serving-certs-547358336/tls.key\": REMOVE"} 2022-11-25T13:29:51.644527876Z ERROR controller-runtime.certwatcher logr@v1.2.3/logr.go:279 error re-watching file {"error": "no such file or directory"} github.com/go-logr/logr.Logger.Error /home/prow/go/pkg/mod/github.com/go-logr/logr@v1.2.3/logr.go:279 sigs.k8s.io/controller-runtime/pkg/certwatcher.(*CertWatcher).handleEvent /home/prow/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.12.3/pkg/certwatcher/certwatcher.go:147 sigs.k8s.io/controller-runtime/pkg/certwatcher.(*CertWatcher).Watch /home/prow/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.12.3/pkg/certwatcher/certwatcher.go:103 2022-11-25T13:29:51.644567557Z ERROR controller-runtime.certwatcher logr@v1.2.3/logr.go:279 error re-reading certificate {"error": "open /tmp/envtest-serving-certs-547358336/tls.crt: no such file or directory"} github.com/go-logr/logr.Logger.Error /home/prow/go/pkg/mod/github.com/go-logr/logr@v1.2.3/logr.go:279 sigs.k8s.io/controller-runtime/pkg/certwatcher.(*CertWatcher).handleEvent /home/prow/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.12.3/pkg/certwatcher/certwatcher.go:152 sigs.k8s.io/controller-runtime/pkg/certwatcher.(*CertWatcher).Watch /home/prow/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.12.3/pkg/certwatcher/certwatcher.go:103 2022-11-25T13:29:51.644590581Z DEBUG controller-runtime.certwatcher logr@v1.2.3/logr.go:261 certificate event {"event": "\"/tmp/envtest-serving-certs-547358336/tls.crt\": REMOVE"} 2022-11-25T13:29:51.644614933Z ERROR controller-runtime.certwatcher logr@v1.2.3/logr.go:279 error re-watching file {"error": "no such file or directory"} github.com/go-logr/logr.Logger.Error /home/prow/go/pkg/mod/github.com/go-logr/logr@v1.2.3/logr.go:279 sigs.k8s.io/controller-runtime/pkg/certwatcher.(*CertWatcher).handleEvent /home/prow/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.12.3/pkg/certwatcher/certwatcher.go:147 sigs.k8s.io/controller-runtime/pkg/certwatcher.(*CertWatcher).Watch /home/prow/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.12.3/pkg/certwatcher/certwatcher.go:103 2022-11-25T13:29:51.644633Z ERROR controller-runtime.certwatcher logr@v1.2.3/logr.go:279 error re-reading certificate {"error": "open /tmp/envtest-serving-certs-547358336/tls.crt: no such file or directory"} github.com/go-logr/logr.Logger.Error /home/prow/go/pkg/mod/github.com/go-logr/logr@v1.2.3/logr.go:279 sigs.k8s.io/controller-runtime/pkg/certwatcher.(*CertWatcher).handleEvent /home/prow/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.12.3/pkg/certwatcher/certwatcher.go:152 sigs.k8s.io/controller-runtime/pkg/certwatcher.(*CertWatcher).Watch /home/prow/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.12.3/pkg/certwatcher/certwatcher.go:103 2022-11-25T13:29:51.690866475Z INFO logr@v1.2.3/logr.go:261 Stopping and waiting for non leader election runnables ... skipping 4 lines ... 2022-11-25T13:29:51.691152776Z INFO controller/controller.go:227 Starting workers {"controller": "job", "controllerGroup": "batch", "controllerKind": "Job", "worker count": 1} 2022-11-25T13:29:51.691167847Z INFO controller/controller.go:247 Shutdown signal received, waiting for all workers to finish {"controller": "job", "controllerGroup": "batch", "controllerKind": "Job"} 2022-11-25T13:29:51.691181357Z INFO controller/controller.go:249 All workers finished {"controller": "job", "controllerGroup": "batch", "controllerKind": "Job"} 2022-11-25T13:29:51.691196036Z INFO logr@v1.2.3/logr.go:261 Stopping and waiting for caches 2022-11-25T13:29:51.691350657Z INFO logr@v1.2.3/logr.go:261 Stopping and waiting for webhooks 2022-11-25T13:29:51.691442384Z INFO controller-runtime.webhook logr@v1.2.3/logr.go:261 shutting down webhook server 2022-11-25T13:29:51.691629199Z ERROR controller-runtime.source logr@v1.2.3/logr.go:279 failed to get informer from cache {"error": "Timeout: failed waiting for *v1.Job Informer to sync"} github.com/go-logr/logr.Logger.Error /home/prow/go/pkg/mod/github.com/go-logr/logr@v1.2.3/logr.go:279 sigs.k8s.io/controller-runtime/pkg/source.(*Kind).Start.func1.1 /home/prow/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.12.3/pkg/source/source.go:144 k8s.io/apimachinery/pkg/util/wait.runConditionWithCrashProtectionWithContext /home/prow/go/pkg/mod/k8s.io/apimachinery@v0.24.3/pkg/util/wait/wait.go:233 k8s.io/apimachinery/pkg/util/wait.poll ... skipping 48 lines ... 2022-11-25T13:29:51.534677511Z DEBUG controller-runtime.webhook.webhooks admission/http.go:96 received request {"webhook": "/mutate-batch-v1-job", "UID": "7fdbca7c-965c-4d25-9232-fcce6c00d925", "kind": "batch/v1, Kind=Job", "resource": {"group":"batch","version":"v1","resource":"jobs"}} 2022-11-25T13:29:51.535288232Z DEBUG controller-runtime.webhook.webhooks admission/http.go:143 wrote response {"webhook": "/mutate-batch-v1-job", "code": 200, "reason": "", "UID": "7fdbca7c-965c-4d25-9232-fcce6c00d925", "allowed": true} 2022-11-25T13:29:51.553955069Z DEBUG controller-runtime.webhook.webhooks admission/http.go:96 received request {"webhook": "/validate-batch-v1-job", "UID": "5f7eea0d-11d2-4223-a57c-6c4a6cc1a7b3", "kind": "batch/v1, Kind=Job", "resource": {"group":"batch","version":"v1","resource":"jobs"}} 2022-11-25T13:29:51.554489779Z DEBUG controller-runtime.webhook.webhooks admission/http.go:143 wrote response {"webhook": "/validate-batch-v1-job", "code": 403, "reason": "job.spec.suspend: Forbidden: suspend should be true when adding the queue name", "UID": "5f7eea0d-11d2-4223-a57c-6c4a6cc1a7b3", "allowed": false} [1mSTEP:[0m tearing down the test environment [38;5;243m11/25/22 13:29:51.643[0m 2022-11-25T13:29:51.644424676Z DEBUG controller-runtime.certwatcher logr@v1.2.3/logr.go:261 certificate event {"event": "\"/tmp/envtest-serving-certs-547358336/tls.key\": REMOVE"} 2022-11-25T13:29:51.644527876Z ERROR controller-runtime.certwatcher logr@v1.2.3/logr.go:279 error re-watching file {"error": "no such file or directory"} github.com/go-logr/logr.Logger.Error /home/prow/go/pkg/mod/github.com/go-logr/logr@v1.2.3/logr.go:279 sigs.k8s.io/controller-runtime/pkg/certwatcher.(*CertWatcher).handleEvent /home/prow/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.12.3/pkg/certwatcher/certwatcher.go:147 sigs.k8s.io/controller-runtime/pkg/certwatcher.(*CertWatcher).Watch /home/prow/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.12.3/pkg/certwatcher/certwatcher.go:103 2022-11-25T13:29:51.644567557Z ERROR controller-runtime.certwatcher logr@v1.2.3/logr.go:279 error re-reading certificate {"error": "open /tmp/envtest-serving-certs-547358336/tls.crt: no such file or directory"} github.com/go-logr/logr.Logger.Error /home/prow/go/pkg/mod/github.com/go-logr/logr@v1.2.3/logr.go:279 sigs.k8s.io/controller-runtime/pkg/certwatcher.(*CertWatcher).handleEvent /home/prow/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.12.3/pkg/certwatcher/certwatcher.go:152 sigs.k8s.io/controller-runtime/pkg/certwatcher.(*CertWatcher).Watch /home/prow/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.12.3/pkg/certwatcher/certwatcher.go:103 2022-11-25T13:29:51.644590581Z DEBUG controller-runtime.certwatcher logr@v1.2.3/logr.go:261 certificate event {"event": "\"/tmp/envtest-serving-certs-547358336/tls.crt\": REMOVE"} 2022-11-25T13:29:51.644614933Z ERROR controller-runtime.certwatcher logr@v1.2.3/logr.go:279 error re-watching file {"error": "no such file or directory"} github.com/go-logr/logr.Logger.Error /home/prow/go/pkg/mod/github.com/go-logr/logr@v1.2.3/logr.go:279 sigs.k8s.io/controller-runtime/pkg/certwatcher.(*CertWatcher).handleEvent /home/prow/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.12.3/pkg/certwatcher/certwatcher.go:147 sigs.k8s.io/controller-runtime/pkg/certwatcher.(*CertWatcher).Watch /home/prow/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.12.3/pkg/certwatcher/certwatcher.go:103 2022-11-25T13:29:51.644633Z ERROR controller-runtime.certwatcher logr@v1.2.3/logr.go:279 error re-reading certificate {"error": "open /tmp/envtest-serving-certs-547358336/tls.crt: no such file or directory"} github.com/go-logr/logr.Logger.Error /home/prow/go/pkg/mod/github.com/go-logr/logr@v1.2.3/logr.go:279 sigs.k8s.io/controller-runtime/pkg/certwatcher.(*CertWatcher).handleEvent /home/prow/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.12.3/pkg/certwatcher/certwatcher.go:152 sigs.k8s.io/controller-runtime/pkg/certwatcher.(*CertWatcher).Watch /home/prow/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.12.3/pkg/certwatcher/certwatcher.go:103 2022-11-25T13:29:51.690866475Z INFO logr@v1.2.3/logr.go:261 Stopping and waiting for non leader election runnables ... skipping 4 lines ... 2022-11-25T13:29:51.691152776Z INFO controller/controller.go:227 Starting workers {"controller": "job", "controllerGroup": "batch", "controllerKind": "Job", "worker count": 1} 2022-11-25T13:29:51.691167847Z INFO controller/controller.go:247 Shutdown signal received, waiting for all workers to finish {"controller": "job", "controllerGroup": "batch", "controllerKind": "Job"} 2022-11-25T13:29:51.691181357Z INFO controller/controller.go:249 All workers finished {"controller": "job", "controllerGroup": "batch", "controllerKind": "Job"} 2022-11-25T13:29:51.691196036Z INFO logr@v1.2.3/logr.go:261 Stopping and waiting for caches 2022-11-25T13:29:51.691350657Z INFO logr@v1.2.3/logr.go:261 Stopping and waiting for webhooks 2022-11-25T13:29:51.691442384Z INFO controller-runtime.webhook logr@v1.2.3/logr.go:261 shutting down webhook server 2022-11-25T13:29:51.691629199Z ERROR controller-runtime.source logr@v1.2.3/logr.go:279 failed to get informer from cache {"error": "Timeout: failed waiting for *v1.Job Informer to sync"} github.com/go-logr/logr.Logger.Error /home/prow/go/pkg/mod/github.com/go-logr/logr@v1.2.3/logr.go:279 sigs.k8s.io/controller-runtime/pkg/source.(*Kind).Start.func1.1 /home/prow/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.12.3/pkg/source/source.go:144 k8s.io/apimachinery/pkg/util/wait.runConditionWithCrashProtectionWithContext /home/prow/go/pkg/mod/k8s.io/apimachinery@v0.24.3/pkg/util/wait/wait.go:233 k8s.io/apimachinery/pkg/util/wait.poll ... skipping 41 lines ... 2022-11-25T13:29:58.124900435Z INFO controller-runtime.webhook logr@v1.2.3/logr.go:261 Serving webhook server {"host": "127.0.0.1", "port": 43397} 2022-11-25T13:29:58.124943873Z INFO controller-runtime.certwatcher logr@v1.2.3/logr.go:261 Starting certificate watcher 2022-11-25T13:29:58.147667326Z DEBUG controller-runtime.webhook.webhooks admission/http.go:96 received request {"webhook": "/mutate-batch-v1-job", "UID": "208aec67-4b0d-47a3-b1d7-52be28a62d06", "kind": "batch/v1, Kind=Job", "resource": {"group":"batch","version":"v1","resource":"jobs"}} 2022-11-25T13:29:58.148215913Z DEBUG controller-runtime.webhook.webhooks admission/http.go:143 wrote response {"webhook": "/mutate-batch-v1-job", "code": 200, "reason": "", "UID": "208aec67-4b0d-47a3-b1d7-52be28a62d06", "allowed": true} [1mSTEP:[0m tearing down the test environment [38;5;243m11/25/22 13:29:58.262[0m 2022-11-25T13:29:58.263739953Z DEBUG controller-runtime.certwatcher logr@v1.2.3/logr.go:261 certificate event {"event": "\"/tmp/envtest-serving-certs-3998955363/tls.key\": REMOVE"} 2022-11-25T13:29:58.263848593Z ERROR controller-runtime.certwatcher logr@v1.2.3/logr.go:279 error re-watching file {"error": "no such file or directory"} github.com/go-logr/logr.Logger.Error /home/prow/go/pkg/mod/github.com/go-logr/logr@v1.2.3/logr.go:279 sigs.k8s.io/controller-runtime/pkg/certwatcher.(*CertWatcher).handleEvent /home/prow/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.12.3/pkg/certwatcher/certwatcher.go:147 sigs.k8s.io/controller-runtime/pkg/certwatcher.(*CertWatcher).Watch /home/prow/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.12.3/pkg/certwatcher/certwatcher.go:103 2022-11-25T13:29:58.263892757Z ERROR controller-runtime.certwatcher logr@v1.2.3/logr.go:279 error re-reading certificate {"error": "open /tmp/envtest-serving-certs-3998955363/tls.crt: no such file or directory"} github.com/go-logr/logr.Logger.Error /home/prow/go/pkg/mod/github.com/go-logr/logr@v1.2.3/logr.go:279 sigs.k8s.io/controller-runtime/pkg/certwatcher.(*CertWatcher).handleEvent /home/prow/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.12.3/pkg/certwatcher/certwatcher.go:152 sigs.k8s.io/controller-runtime/pkg/certwatcher.(*CertWatcher).Watch /home/prow/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.12.3/pkg/certwatcher/certwatcher.go:103 2022-11-25T13:29:58.263923033Z DEBUG controller-runtime.certwatcher logr@v1.2.3/logr.go:261 certificate event {"event": "\"/tmp/envtest-serving-certs-3998955363/tls.crt\": REMOVE"} 2022-11-25T13:29:58.263942735Z ERROR controller-runtime.certwatcher logr@v1.2.3/logr.go:279 error re-watching file {"error": "no such file or directory"} github.com/go-logr/logr.Logger.Error /home/prow/go/pkg/mod/github.com/go-logr/logr@v1.2.3/logr.go:279 sigs.k8s.io/controller-runtime/pkg/certwatcher.(*CertWatcher).handleEvent /home/prow/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.12.3/pkg/certwatcher/certwatcher.go:147 sigs.k8s.io/controller-runtime/pkg/certwatcher.(*CertWatcher).Watch /home/prow/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.12.3/pkg/certwatcher/certwatcher.go:103 2022-11-25T13:29:58.263962352Z ERROR controller-runtime.certwatcher logr@v1.2.3/logr.go:279 error re-reading certificate {"error": "open /tmp/envtest-serving-certs-3998955363/tls.crt: no such file or directory"} github.com/go-logr/logr.Logger.Error /home/prow/go/pkg/mod/github.com/go-logr/logr@v1.2.3/logr.go:279 sigs.k8s.io/controller-runtime/pkg/certwatcher.(*CertWatcher).handleEvent /home/prow/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.12.3/pkg/certwatcher/certwatcher.go:152 sigs.k8s.io/controller-runtime/pkg/certwatcher.(*CertWatcher).Watch /home/prow/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.12.3/pkg/certwatcher/certwatcher.go:103 2022-11-25T13:29:58.325398064Z INFO logr@v1.2.3/logr.go:261 Stopping and waiting for non leader election runnables ... skipping 2 lines ... 2022-11-25T13:29:58.325564883Z INFO controller/controller.go:185 Starting EventSource {"controller": "job", "controllerGroup": "batch", "controllerKind": "Job", "source": "kind source: *v1alpha2.Workload"} 2022-11-25T13:29:58.325585825Z INFO controller/controller.go:193 Starting Controller {"controller": "job", "controllerGroup": "batch", "controllerKind": "Job"} 2022-11-25T13:29:58.325603613Z INFO controller/controller.go:227 Starting workers {"controller": "job", "controllerGroup": "batch", "controllerKind": "Job", "worker count": 1} 2022-11-25T13:29:58.325614113Z INFO controller/controller.go:247 Shutdown signal received, waiting for all workers to finish {"controller": "job", "controllerGroup": "batch", "controllerKind": "Job"} 2022-11-25T13:29:58.32579079Z INFO controller/controller.go:249 All workers finished {"controller": "job", "controllerGroup": "batch", "controllerKind": "Job"} 2022-11-25T13:29:58.325865687Z INFO logr@v1.2.3/logr.go:261 Stopping and waiting for caches 2022-11-25T13:29:58.326066473Z ERROR controller-runtime.source logr@v1.2.3/logr.go:279 failed to get informer from cache {"error": "Timeout: failed waiting for *v1.Job Informer to sync"} github.com/go-logr/logr.Logger.Error /home/prow/go/pkg/mod/github.com/go-logr/logr@v1.2.3/logr.go:279 sigs.k8s.io/controller-runtime/pkg/source.(*Kind).Start.func1.1 /home/prow/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.12.3/pkg/source/source.go:144 k8s.io/apimachinery/pkg/util/wait.runConditionWithCrashProtectionWithContext /home/prow/go/pkg/mod/k8s.io/apimachinery@v0.24.3/pkg/util/wait/wait.go:233 k8s.io/apimachinery/pkg/util/wait.poll ... skipping 48 lines ... 2022-11-25T13:29:58.124900435Z INFO controller-runtime.webhook logr@v1.2.3/logr.go:261 Serving webhook server {"host": "127.0.0.1", "port": 43397} 2022-11-25T13:29:58.124943873Z INFO controller-runtime.certwatcher logr@v1.2.3/logr.go:261 Starting certificate watcher 2022-11-25T13:29:58.147667326Z DEBUG controller-runtime.webhook.webhooks admission/http.go:96 received request {"webhook": "/mutate-batch-v1-job", "UID": "208aec67-4b0d-47a3-b1d7-52be28a62d06", "kind": "batch/v1, Kind=Job", "resource": {"group":"batch","version":"v1","resource":"jobs"}} 2022-11-25T13:29:58.148215913Z DEBUG controller-runtime.webhook.webhooks admission/http.go:143 wrote response {"webhook": "/mutate-batch-v1-job", "code": 200, "reason": "", "UID": "208aec67-4b0d-47a3-b1d7-52be28a62d06", "allowed": true} [1mSTEP:[0m tearing down the test environment [38;5;243m11/25/22 13:29:58.262[0m 2022-11-25T13:29:58.263739953Z DEBUG controller-runtime.certwatcher logr@v1.2.3/logr.go:261 certificate event {"event": "\"/tmp/envtest-serving-certs-3998955363/tls.key\": REMOVE"} 2022-11-25T13:29:58.263848593Z ERROR controller-runtime.certwatcher logr@v1.2.3/logr.go:279 error re-watching file {"error": "no such file or directory"} github.com/go-logr/logr.Logger.Error /home/prow/go/pkg/mod/github.com/go-logr/logr@v1.2.3/logr.go:279 sigs.k8s.io/controller-runtime/pkg/certwatcher.(*CertWatcher).handleEvent /home/prow/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.12.3/pkg/certwatcher/certwatcher.go:147 sigs.k8s.io/controller-runtime/pkg/certwatcher.(*CertWatcher).Watch /home/prow/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.12.3/pkg/certwatcher/certwatcher.go:103 2022-11-25T13:29:58.263892757Z ERROR controller-runtime.certwatcher logr@v1.2.3/logr.go:279 error re-reading certificate {"error": "open /tmp/envtest-serving-certs-3998955363/tls.crt: no such file or directory"} github.com/go-logr/logr.Logger.Error /home/prow/go/pkg/mod/github.com/go-logr/logr@v1.2.3/logr.go:279 sigs.k8s.io/controller-runtime/pkg/certwatcher.(*CertWatcher).handleEvent /home/prow/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.12.3/pkg/certwatcher/certwatcher.go:152 sigs.k8s.io/controller-runtime/pkg/certwatcher.(*CertWatcher).Watch /home/prow/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.12.3/pkg/certwatcher/certwatcher.go:103 2022-11-25T13:29:58.263923033Z DEBUG controller-runtime.certwatcher logr@v1.2.3/logr.go:261 certificate event {"event": "\"/tmp/envtest-serving-certs-3998955363/tls.crt\": REMOVE"} 2022-11-25T13:29:58.263942735Z ERROR controller-runtime.certwatcher logr@v1.2.3/logr.go:279 error re-watching file {"error": "no such file or directory"} github.com/go-logr/logr.Logger.Error /home/prow/go/pkg/mod/github.com/go-logr/logr@v1.2.3/logr.go:279 sigs.k8s.io/controller-runtime/pkg/certwatcher.(*CertWatcher).handleEvent /home/prow/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.12.3/pkg/certwatcher/certwatcher.go:147 sigs.k8s.io/controller-runtime/pkg/certwatcher.(*CertWatcher).Watch /home/prow/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.12.3/pkg/certwatcher/certwatcher.go:103 2022-11-25T13:29:58.263962352Z ERROR controller-runtime.certwatcher logr@v1.2.3/logr.go:279 error re-reading certificate {"error": "open /tmp/envtest-serving-certs-3998955363/tls.crt: no such file or directory"} github.com/go-logr/logr.Logger.Error /home/prow/go/pkg/mod/github.com/go-logr/logr@v1.2.3/logr.go:279 sigs.k8s.io/controller-runtime/pkg/certwatcher.(*CertWatcher).handleEvent /home/prow/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.12.3/pkg/certwatcher/certwatcher.go:152 sigs.k8s.io/controller-runtime/pkg/certwatcher.(*CertWatcher).Watch /home/prow/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.12.3/pkg/certwatcher/certwatcher.go:103 2022-11-25T13:29:58.325398064Z INFO logr@v1.2.3/logr.go:261 Stopping and waiting for non leader election runnables ... skipping 2 lines ... 2022-11-25T13:29:58.325564883Z INFO controller/controller.go:185 Starting EventSource {"controller": "job", "controllerGroup": "batch", "controllerKind": "Job", "source": "kind source: *v1alpha2.Workload"} 2022-11-25T13:29:58.325585825Z INFO controller/controller.go:193 Starting Controller {"controller": "job", "controllerGroup": "batch", "controllerKind": "Job"} 2022-11-25T13:29:58.325603613Z INFO controller/controller.go:227 Starting workers {"controller": "job", "controllerGroup": "batch", "controllerKind": "Job", "worker count": 1} 2022-11-25T13:29:58.325614113Z INFO controller/controller.go:247 Shutdown signal received, waiting for all workers to finish {"controller": "job", "controllerGroup": "batch", "controllerKind": "Job"} 2022-11-25T13:29:58.32579079Z INFO controller/controller.go:249 All workers finished {"controller": "job", "controllerGroup": "batch", "controllerKind": "Job"} 2022-11-25T13:29:58.325865687Z INFO logr@v1.2.3/logr.go:261 Stopping and waiting for caches 2022-11-25T13:29:58.326066473Z ERROR controller-runtime.source logr@v1.2.3/logr.go:279 failed to get informer from cache {"error": "Timeout: failed waiting for *v1.Job Informer to sync"} github.com/go-logr/logr.Logger.Error /home/prow/go/pkg/mod/github.com/go-logr/logr@v1.2.3/logr.go:279 sigs.k8s.io/controller-runtime/pkg/source.(*Kind).Start.func1.1 /home/prow/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.12.3/pkg/source/source.go:144 k8s.io/apimachinery/pkg/util/wait.runConditionWithCrashProtectionWithContext /home/prow/go/pkg/mod/k8s.io/apimachinery@v0.24.3/pkg/util/wait/wait.go:233 k8s.io/apimachinery/pkg/util/wait.poll ... skipping 43 lines ... 2022-11-25T13:30:05.271577988Z INFO controller-runtime.webhook logr@v1.2.3/logr.go:261 Serving webhook server {"host": "127.0.0.1", "port": 36775} 2022-11-25T13:30:05.271640097Z INFO controller-runtime.certwatcher logr@v1.2.3/logr.go:261 Starting certificate watcher 2022-11-25T13:30:05.293925448Z DEBUG controller-runtime.webhook.webhooks admission/http.go:96 received request {"webhook": "/mutate-batch-v1-job", "UID": "1ebf39b8-25e5-4329-9b7f-5c8ab5289660", "kind": "batch/v1, Kind=Job", "resource": {"group":"batch","version":"v1","resource":"jobs"}} 2022-11-25T13:30:05.294364804Z DEBUG controller-runtime.webhook.webhooks admission/http.go:143 wrote response {"webhook": "/mutate-batch-v1-job", "code": 200, "reason": "", "UID": "1ebf39b8-25e5-4329-9b7f-5c8ab5289660", "allowed": true} [1mSTEP:[0m tearing down the test environment [38;5;243m11/25/22 13:30:05.418[0m 2022-11-25T13:30:05.41882821Z DEBUG controller-runtime.certwatcher logr@v1.2.3/logr.go:261 certificate event {"event": "\"/tmp/envtest-serving-certs-1513103083/tls.key\": REMOVE"} 2022-11-25T13:30:05.418924012Z ERROR controller-runtime.certwatcher logr@v1.2.3/logr.go:279 error re-watching file {"error": "no such file or directory"} github.com/go-logr/logr.Logger.Error /home/prow/go/pkg/mod/github.com/go-logr/logr@v1.2.3/logr.go:279 sigs.k8s.io/controller-runtime/pkg/certwatcher.(*CertWatcher).handleEvent /home/prow/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.12.3/pkg/certwatcher/certwatcher.go:147 sigs.k8s.io/controller-runtime/pkg/certwatcher.(*CertWatcher).Watch /home/prow/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.12.3/pkg/certwatcher/certwatcher.go:103 2022-11-25T13:30:05.418964153Z ERROR controller-runtime.certwatcher logr@v1.2.3/logr.go:279 error re-reading certificate {"error": "open /tmp/envtest-serving-certs-1513103083/tls.crt: no such file or directory"} github.com/go-logr/logr.Logger.Error /home/prow/go/pkg/mod/github.com/go-logr/logr@v1.2.3/logr.go:279 sigs.k8s.io/controller-runtime/pkg/certwatcher.(*CertWatcher).handleEvent /home/prow/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.12.3/pkg/certwatcher/certwatcher.go:152 sigs.k8s.io/controller-runtime/pkg/certwatcher.(*CertWatcher).Watch /home/prow/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.12.3/pkg/certwatcher/certwatcher.go:103 2022-11-25T13:30:05.418989933Z DEBUG controller-runtime.certwatcher logr@v1.2.3/logr.go:261 certificate event {"event": "\"/tmp/envtest-serving-certs-1513103083/tls.crt\": REMOVE"} 2022-11-25T13:30:05.419006508Z ERROR controller-runtime.certwatcher logr@v1.2.3/logr.go:279 error re-watching file {"error": "no such file or directory"} github.com/go-logr/logr.Logger.Error /home/prow/go/pkg/mod/github.com/go-logr/logr@v1.2.3/logr.go:279 sigs.k8s.io/controller-runtime/pkg/certwatcher.(*CertWatcher).handleEvent /home/prow/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.12.3/pkg/certwatcher/certwatcher.go:147 sigs.k8s.io/controller-runtime/pkg/certwatcher.(*CertWatcher).Watch /home/prow/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.12.3/pkg/certwatcher/certwatcher.go:103 2022-11-25T13:30:05.419024029Z ERROR controller-runtime.certwatcher logr@v1.2.3/logr.go:279 error re-reading certificate {"error": "open /tmp/envtest-serving-certs-1513103083/tls.crt: no such file or directory"} github.com/go-logr/logr.Logger.Error /home/prow/go/pkg/mod/github.com/go-logr/logr@v1.2.3/logr.go:279 sigs.k8s.io/controller-runtime/pkg/certwatcher.(*CertWatcher).handleEvent /home/prow/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.12.3/pkg/certwatcher/certwatcher.go:152 sigs.k8s.io/controller-runtime/pkg/certwatcher.(*CertWatcher).Watch /home/prow/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.12.3/pkg/certwatcher/certwatcher.go:103 2022-11-25T13:30:05.47175787Z INFO logr@v1.2.3/logr.go:261 Stopping and waiting for non leader election runnables ... skipping 3 lines ... 2022-11-25T13:30:05.471988246Z INFO controller/controller.go:193 Starting Controller {"controller": "job", "controllerGroup": "batch", "controllerKind": "Job"} 2022-11-25T13:30:05.471998621Z INFO controller/controller.go:227 Starting workers {"controller": "job", "controllerGroup": "batch", "controllerKind": "Job", "worker count": 1} 2022-11-25T13:30:05.472009781Z INFO controller/controller.go:247 Shutdown signal received, waiting for all workers to finish {"controller": "job", "controllerGroup": "batch", "controllerKind": "Job"} 2022-11-25T13:30:05.472057022Z INFO controller/controller.go:249 All workers finished {"controller": "job", "controllerGroup": "batch", "controllerKind": "Job"} 2022-11-25T13:30:05.472071148Z INFO logr@v1.2.3/logr.go:261 Stopping and waiting for caches 2022-11-25T13:30:05.472213691Z INFO logr@v1.2.3/logr.go:261 Stopping and waiting for webhooks 2022-11-25T13:30:05.472303785Z ERROR controller-runtime.source logr@v1.2.3/logr.go:279 failed to get informer from cache {"error": "Timeout: failed waiting for *v1.Job Informer to sync"} github.com/go-logr/logr.Logger.Error /home/prow/go/pkg/mod/github.com/go-logr/logr@v1.2.3/logr.go:279 sigs.k8s.io/controller-runtime/pkg/source.(*Kind).Start.func1.1 /home/prow/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.12.3/pkg/source/source.go:144 k8s.io/apimachinery/pkg/util/wait.runConditionWithCrashProtectionWithContext /home/prow/go/pkg/mod/k8s.io/apimachinery@v0.24.3/pkg/util/wait/wait.go:233 k8s.io/apimachinery/pkg/util/wait.poll ... skipping 47 lines ... 2022-11-25T13:30:05.271577988Z INFO controller-runtime.webhook logr@v1.2.3/logr.go:261 Serving webhook server {"host": "127.0.0.1", "port": 36775} 2022-11-25T13:30:05.271640097Z INFO controller-runtime.certwatcher logr@v1.2.3/logr.go:261 Starting certificate watcher 2022-11-25T13:30:05.293925448Z DEBUG controller-runtime.webhook.webhooks admission/http.go:96 received request {"webhook": "/mutate-batch-v1-job", "UID": "1ebf39b8-25e5-4329-9b7f-5c8ab5289660", "kind": "batch/v1, Kind=Job", "resource": {"group":"batch","version":"v1","resource":"jobs"}} 2022-11-25T13:30:05.294364804Z DEBUG controller-runtime.webhook.webhooks admission/http.go:143 wrote response {"webhook": "/mutate-batch-v1-job", "code": 200, "reason": "", "UID": "1ebf39b8-25e5-4329-9b7f-5c8ab5289660", "allowed": true} [1mSTEP:[0m tearing down the test environment [38;5;243m11/25/22 13:30:05.418[0m 2022-11-25T13:30:05.41882821Z DEBUG controller-runtime.certwatcher logr@v1.2.3/logr.go:261 certificate event {"event": "\"/tmp/envtest-serving-certs-1513103083/tls.key\": REMOVE"} 2022-11-25T13:30:05.418924012Z ERROR controller-runtime.certwatcher logr@v1.2.3/logr.go:279 error re-watching file {"error": "no such file or directory"} github.com/go-logr/logr.Logger.Error /home/prow/go/pkg/mod/github.com/go-logr/logr@v1.2.3/logr.go:279 sigs.k8s.io/controller-runtime/pkg/certwatcher.(*CertWatcher).handleEvent /home/prow/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.12.3/pkg/certwatcher/certwatcher.go:147 sigs.k8s.io/controller-runtime/pkg/certwatcher.(*CertWatcher).Watch /home/prow/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.12.3/pkg/certwatcher/certwatcher.go:103 2022-11-25T13:30:05.418964153Z ERROR controller-runtime.certwatcher logr@v1.2.3/logr.go:279 error re-reading certificate {"error": "open /tmp/envtest-serving-certs-1513103083/tls.crt: no such file or directory"} github.com/go-logr/logr.Logger.Error /home/prow/go/pkg/mod/github.com/go-logr/logr@v1.2.3/logr.go:279 sigs.k8s.io/controller-runtime/pkg/certwatcher.(*CertWatcher).handleEvent /home/prow/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.12.3/pkg/certwatcher/certwatcher.go:152 sigs.k8s.io/controller-runtime/pkg/certwatcher.(*CertWatcher).Watch /home/prow/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.12.3/pkg/certwatcher/certwatcher.go:103 2022-11-25T13:30:05.418989933Z DEBUG controller-runtime.certwatcher logr@v1.2.3/logr.go:261 certificate event {"event": "\"/tmp/envtest-serving-certs-1513103083/tls.crt\": REMOVE"} 2022-11-25T13:30:05.419006508Z ERROR controller-runtime.certwatcher logr@v1.2.3/logr.go:279 error re-watching file {"error": "no such file or directory"} github.com/go-logr/logr.Logger.Error /home/prow/go/pkg/mod/github.com/go-logr/logr@v1.2.3/logr.go:279 sigs.k8s.io/controller-runtime/pkg/certwatcher.(*CertWatcher).handleEvent /home/prow/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.12.3/pkg/certwatcher/certwatcher.go:147 sigs.k8s.io/controller-runtime/pkg/certwatcher.(*CertWatcher).Watch /home/prow/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.12.3/pkg/certwatcher/certwatcher.go:103 2022-11-25T13:30:05.419024029Z ERROR controller-runtime.certwatcher logr@v1.2.3/logr.go:279 error re-reading certificate {"error": "open /tmp/envtest-serving-certs-1513103083/tls.crt: no such file or directory"} github.com/go-logr/logr.Logger.Error /home/prow/go/pkg/mod/github.com/go-logr/logr@v1.2.3/logr.go:279 sigs.k8s.io/controller-runtime/pkg/certwatcher.(*CertWatcher).handleEvent /home/prow/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.12.3/pkg/certwatcher/certwatcher.go:152 sigs.k8s.io/controller-runtime/pkg/certwatcher.(*CertWatcher).Watch /home/prow/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.12.3/pkg/certwatcher/certwatcher.go:103 2022-11-25T13:30:05.47175787Z INFO logr@v1.2.3/logr.go:261 Stopping and waiting for non leader election runnables ... skipping 3 lines ... 2022-11-25T13:30:05.471988246Z INFO controller/controller.go:193 Starting Controller {"controller": "job", "controllerGroup": "batch", "controllerKind": "Job"} 2022-11-25T13:30:05.471998621Z INFO controller/controller.go:227 Starting workers {"controller": "job", "controllerGroup": "batch", "controllerKind": "Job", "worker count": 1} 2022-11-25T13:30:05.472009781Z INFO controller/controller.go:247 Shutdown signal received, waiting for all workers to finish {"controller": "job", "controllerGroup": "batch", "controllerKind": "Job"} 2022-11-25T13:30:05.472057022Z INFO controller/controller.go:249 All workers finished {"controller": "job", "controllerGroup": "batch", "controllerKind": "Job"} 2022-11-25T13:30:05.472071148Z INFO logr@v1.2.3/logr.go:261 Stopping and waiting for caches 2022-11-25T13:30:05.472213691Z INFO logr@v1.2.3/logr.go:261 Stopping and waiting for webhooks 2022-11-25T13:30:05.472303785Z ERROR controller-runtime.source logr@v1.2.3/logr.go:279 failed to get informer from cache {"error": "Timeout: failed waiting for *v1.Job Informer to sync"} github.com/go-logr/logr.Logger.Error /home/prow/go/pkg/mod/github.com/go-logr/logr@v1.2.3/logr.go:279 sigs.k8s.io/controller-runtime/pkg/source.(*Kind).Start.func1.1 /home/prow/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.12.3/pkg/source/source.go:144 k8s.io/apimachinery/pkg/util/wait.runConditionWithCrashProtectionWithContext /home/prow/go/pkg/mod/k8s.io/apimachinery@v0.24.3/pkg/util/wait/wait.go:233 k8s.io/apimachinery/pkg/util/wait.poll ... skipping 49 lines ... 2022-11-25T13:30:12.543652881Z INFO controller/controller.go:185 Starting EventSource {"controller": "job", "controllerGroup": "batch", "controllerKind": "Job", "source": "kind source: *v1alpha2.Workload"} 2022-11-25T13:30:12.543669322Z INFO controller/controller.go:193 Starting Controller {"controller": "job", "controllerGroup": "batch", "controllerKind": "Job"} [1mSTEP:[0m tearing down the test environment [38;5;243m11/25/22 13:30:12.566[0m 2022-11-25T13:30:12.567484463Z INFO logr@v1.2.3/logr.go:261 Stopping and waiting for non leader election runnables 2022-11-25T13:30:12.56764729Z INFO logr@v1.2.3/logr.go:261 Stopping and waiting for leader election runnables 2022-11-25T13:30:12.567746943Z INFO controller/controller.go:227 Starting workers {"controller": "job", "controllerGroup": "batch", "controllerKind": "Job", "worker count": 1} 2022-11-25T13:30:12.567742944Z ERROR controller-runtime.source logr@v1.2.3/logr.go:279 failed to get informer from cache {"error": "Timeout: failed waiting for *v1.Job Informer to sync"} github.com/go-logr/logr.Logger.Error /home/prow/go/pkg/mod/github.com/go-logr/logr@v1.2.3/logr.go:279 sigs.k8s.io/controller-runtime/pkg/source.(*Kind).Start.func1.1 /home/prow/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.12.3/pkg/source/source.go:144 k8s.io/apimachinery/pkg/util/wait.runConditionWithCrashProtectionWithContext /home/prow/go/pkg/mod/k8s.io/apimachinery@v0.24.3/pkg/util/wait/wait.go:233 k8s.io/apimachinery/pkg/util/wait.poll ... skipping 3 lines ... sigs.k8s.io/controller-runtime/pkg/source.(*Kind).Start.func1 /home/prow/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.12.3/pkg/source/source.go:132 2022-11-25T13:30:12.567781182Z INFO controller/controller.go:247 Shutdown signal received, waiting for all workers to finish {"controller": "job", "controllerGroup": "batch", "controllerKind": "Job"} 2022-11-25T13:30:12.567810843Z DEBUG controller-runtime.certwatcher logr@v1.2.3/logr.go:261 certificate event {"event": "\"/tmp/envtest-serving-certs-231791174/tls.key\": REMOVE"} 2022-11-25T13:30:12.567834444Z INFO controller/controller.go:249 All workers finished {"controller": "job", "controllerGroup": "batch", "controllerKind": "Job"} 2022-11-25T13:30:12.567852183Z INFO logr@v1.2.3/logr.go:261 Stopping and waiting for caches 2022-11-25T13:30:12.567850504Z ERROR controller-runtime.certwatcher logr@v1.2.3/logr.go:279 error re-watching file {"error": "no such file or directory"} github.com/go-logr/logr.Logger.Error /home/prow/go/pkg/mod/github.com/go-logr/logr@v1.2.3/logr.go:279 sigs.k8s.io/controller-runtime/pkg/certwatcher.(*CertWatcher).handleEvent /home/prow/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.12.3/pkg/certwatcher/certwatcher.go:147 sigs.k8s.io/controller-runtime/pkg/certwatcher.(*CertWatcher).Watch /home/prow/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.12.3/pkg/certwatcher/certwatcher.go:103 2022-11-25T13:30:12.567890892Z ERROR controller-runtime.certwatcher logr@v1.2.3/logr.go:279 error re-reading certificate {"error": "open /tmp/envtest-serving-certs-231791174/tls.crt: no such file or directory"} github.com/go-logr/logr.Logger.Error /home/prow/go/pkg/mod/github.com/go-logr/logr@v1.2.3/logr.go:279 sigs.k8s.io/controller-runtime/pkg/certwatcher.(*CertWatcher).handleEvent /home/prow/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.12.3/pkg/certwatcher/certwatcher.go:152 sigs.k8s.io/controller-runtime/pkg/certwatcher.(*CertWatcher).Watch /home/prow/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.12.3/pkg/certwatcher/certwatcher.go:103 2022-11-25T13:30:12.567918534Z DEBUG controller-runtime.certwatcher logr@v1.2.3/logr.go:261 certificate event {"event": "\"/tmp/envtest-serving-certs-231791174/tls.crt\": REMOVE"} 2022-11-25T13:30:12.56794685Z ERROR controller-runtime.certwatcher logr@v1.2.3/logr.go:279 error re-watching file {"error": "no such file or directory"} github.com/go-logr/logr.Logger.Error /home/prow/go/pkg/mod/github.com/go-logr/logr@v1.2.3/logr.go:279 sigs.k8s.io/controller-runtime/pkg/certwatcher.(*CertWatcher).handleEvent /home/prow/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.12.3/pkg/certwatcher/certwatcher.go:147 sigs.k8s.io/controller-runtime/pkg/certwatcher.(*CertWatcher).Watch /home/prow/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.12.3/pkg/certwatcher/certwatcher.go:103 2022-11-25T13:30:12.567971293Z ERROR controller-runtime.certwatcher logr@v1.2.3/logr.go:279 error re-reading certificate {"error": "open /tmp/envtest-serving-certs-231791174/tls.crt: no such file or directory"} github.com/go-logr/logr.Logger.Error /home/prow/go/pkg/mod/github.com/go-logr/logr@v1.2.3/logr.go:279 sigs.k8s.io/controller-runtime/pkg/certwatcher.(*CertWatcher).handleEvent /home/prow/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.12.3/pkg/certwatcher/certwatcher.go:152 sigs.k8s.io/controller-runtime/pkg/certwatcher.(*CertWatcher).Watch /home/prow/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.12.3/pkg/certwatcher/certwatcher.go:103 2022-11-25T13:30:12.567921293Z INFO logr@v1.2.3/logr.go:261 Stopping and waiting for webhooks ... skipping 49 lines ... 2022-11-25T13:30:12.543652881Z INFO controller/controller.go:185 Starting EventSource {"controller": "job", "controllerGroup": "batch", "controllerKind": "Job", "source": "kind source: *v1alpha2.Workload"} 2022-11-25T13:30:12.543669322Z INFO controller/controller.go:193 Starting Controller {"controller": "job", "controllerGroup": "batch", "controllerKind": "Job"} [1mSTEP:[0m tearing down the test environment [38;5;243m11/25/22 13:30:12.566[0m 2022-11-25T13:30:12.567484463Z INFO logr@v1.2.3/logr.go:261 Stopping and waiting for non leader election runnables 2022-11-25T13:30:12.56764729Z INFO logr@v1.2.3/logr.go:261 Stopping and waiting for leader election runnables 2022-11-25T13:30:12.567746943Z INFO controller/controller.go:227 Starting workers {"controller": "job", "controllerGroup": "batch", "controllerKind": "Job", "worker count": 1} 2022-11-25T13:30:12.567742944Z ERROR controller-runtime.source logr@v1.2.3/logr.go:279 failed to get informer from cache {"error": "Timeout: failed waiting for *v1.Job Informer to sync"} github.com/go-logr/logr.Logger.Error /home/prow/go/pkg/mod/github.com/go-logr/logr@v1.2.3/logr.go:279 sigs.k8s.io/controller-runtime/pkg/source.(*Kind).Start.func1.1 /home/prow/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.12.3/pkg/source/source.go:144 k8s.io/apimachinery/pkg/util/wait.runConditionWithCrashProtectionWithContext /home/prow/go/pkg/mod/k8s.io/apimachinery@v0.24.3/pkg/util/wait/wait.go:233 k8s.io/apimachinery/pkg/util/wait.poll ... skipping 3 lines ... sigs.k8s.io/controller-runtime/pkg/source.(*Kind).Start.func1 /home/prow/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.12.3/pkg/source/source.go:132 2022-11-25T13:30:12.567781182Z INFO controller/controller.go:247 Shutdown signal received, waiting for all workers to finish {"controller": "job", "controllerGroup": "batch", "controllerKind": "Job"} 2022-11-25T13:30:12.567810843Z DEBUG controller-runtime.certwatcher logr@v1.2.3/logr.go:261 certificate event {"event": "\"/tmp/envtest-serving-certs-231791174/tls.key\": REMOVE"} 2022-11-25T13:30:12.567834444Z INFO controller/controller.go:249 All workers finished {"controller": "job", "controllerGroup": "batch", "controllerKind": "Job"} 2022-11-25T13:30:12.567852183Z INFO logr@v1.2.3/logr.go:261 Stopping and waiting for caches 2022-11-25T13:30:12.567850504Z ERROR controller-runtime.certwatcher logr@v1.2.3/logr.go:279 error re-watching file {"error": "no such file or directory"} github.com/go-logr/logr.Logger.Error /home/prow/go/pkg/mod/github.com/go-logr/logr@v1.2.3/logr.go:279 sigs.k8s.io/controller-runtime/pkg/certwatcher.(*CertWatcher).handleEvent /home/prow/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.12.3/pkg/certwatcher/certwatcher.go:147 sigs.k8s.io/controller-runtime/pkg/certwatcher.(*CertWatcher).Watch /home/prow/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.12.3/pkg/certwatcher/certwatcher.go:103 2022-11-25T13:30:12.567890892Z ERROR controller-runtime.certwatcher logr@v1.2.3/logr.go:279 error re-reading certificate {"error": "open /tmp/envtest-serving-certs-231791174/tls.crt: no such file or directory"} github.com/go-logr/logr.Logger.Error /home/prow/go/pkg/mod/github.com/go-logr/logr@v1.2.3/logr.go:279 sigs.k8s.io/controller-runtime/pkg/certwatcher.(*CertWatcher).handleEvent /home/prow/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.12.3/pkg/certwatcher/certwatcher.go:152 sigs.k8s.io/controller-runtime/pkg/certwatcher.(*CertWatcher).Watch /home/prow/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.12.3/pkg/certwatcher/certwatcher.go:103 2022-11-25T13:30:12.567918534Z DEBUG controller-runtime.certwatcher logr@v1.2.3/logr.go:261 certificate event {"event": "\"/tmp/envtest-serving-certs-231791174/tls.crt\": REMOVE"} 2022-11-25T13:30:12.56794685Z ERROR controller-runtime.certwatcher logr@v1.2.3/logr.go:279 error re-watching file {"error": "no such file or directory"} github.com/go-logr/logr.Logger.Error /home/prow/go/pkg/mod/github.com/go-logr/logr@v1.2.3/logr.go:279 sigs.k8s.io/controller-runtime/pkg/certwatcher.(*CertWatcher).handleEvent /home/prow/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.12.3/pkg/certwatcher/certwatcher.go:147 sigs.k8s.io/controller-runtime/pkg/certwatcher.(*CertWatcher).Watch /home/prow/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.12.3/pkg/certwatcher/certwatcher.go:103 2022-11-25T13:30:12.567971293Z ERROR controller-runtime.certwatcher logr@v1.2.3/logr.go:279 error re-reading certificate {"error": "open /tmp/envtest-serving-certs-231791174/tls.crt: no such file or directory"} github.com/go-logr/logr.Logger.Error /home/prow/go/pkg/mod/github.com/go-logr/logr@v1.2.3/logr.go:279 sigs.k8s.io/controller-runtime/pkg/certwatcher.(*CertWatcher).handleEvent /home/prow/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.12.3/pkg/certwatcher/certwatcher.go:152 sigs.k8s.io/controller-runtime/pkg/certwatcher.(*CertWatcher).Watch /home/prow/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.12.3/pkg/certwatcher/certwatcher.go:103 2022-11-25T13:30:12.567921293Z INFO logr@v1.2.3/logr.go:261 Stopping and waiting for webhooks ... skipping 44 lines ... 2022-11-25T13:30:20.909485668Z LEVEL(-2) job/job_controller.go:141 Reconciling Job {"controller": "job", "controllerGroup": "batch", "controllerKind": "Job", "job": {"name":"test-job","namespace":"default"}, "namespace": "default", "name": "test-job", "reconcileID": "20821244-a9e8-446c-8bf0-fbf786a290c7", "job": {"name":"test-job","namespace":"default"}} 2022-11-25T13:30:20.909601636Z LEVEL(-2) job/job_controller.go:200 Job changed queues, updating workload {"controller": "job", "controllerGroup": "batch", "controllerKind": "Job", "job": {"name":"test-job","namespace":"default"}, "namespace": "default", "name": "test-job", "reconcileID": "20821244-a9e8-446c-8bf0-fbf786a290c7", "job": {"name":"test-job","namespace":"default"}} 2022-11-25T13:30:20.916465873Z LEVEL(-2) job/job_controller.go:141 Reconciling Job {"controller": "job", "controllerGroup": "batch", "controllerKind": "Job", "job": {"name":"test-job","namespace":"default"}, "namespace": "default", "name": "test-job", "reconcileID": "ab6451f1-5998-4877-ae6e-797321fb05d1", "job": {"name":"test-job","namespace":"default"}} 2022-11-25T13:30:20.916611287Z LEVEL(-3) job/job_controller.go:208 Job is suspended and workload not yet admitted by a clusterQueue, nothing to do {"controller": "job", "controllerGroup": "batch", "controllerKind": "Job", "job": {"name":"test-job","namespace":"default"}, "namespace": "default", "name": "test-job", "reconcileID": "ab6451f1-5998-4877-ae6e-797321fb05d1", "job": {"name":"test-job","namespace":"default"}} [1mSTEP:[0m checking a second non-matching workload is deleted [38;5;243m11/25/22 13:30:21.165[0m 2022-11-25T13:30:21.173160419Z LEVEL(-2) job/job_controller.go:141 Reconciling Job {"controller": "job", "controllerGroup": "batch", "controllerKind": "Job", "job": {"name":"test-job","namespace":"default"}, "namespace": "default", "name": "test-job", "reconcileID": "a5d54306-dec0-4da7-a1f7-85a56e42add4", "job": {"name":"test-job","namespace":"default"}} 2022-11-25T13:30:21.177321094Z ERROR job/job_controller.go:153 Getting existing workloads {"controller": "job", "controllerGroup": "batch", "controllerKind": "Job", "job": {"name":"test-job","namespace":"default"}, "namespace": "default", "name": "test-job", "reconcileID": "a5d54306-dec0-4da7-a1f7-85a56e42add4", "job": {"name":"test-job","namespace":"default"}, "error": "only one workload should exist, found 2"} sigs.k8s.io/kueue/pkg/controller/workload/job.(*JobReconciler).Reconcile /home/prow/go/src/sigs.k8s.io/kueue/pkg/controller/workload/job/job_controller.go:153 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile /home/prow/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.12.3/pkg/internal/controller/controller.go:121 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler /home/prow/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.12.3/pkg/internal/controller/controller.go:320 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem /home/prow/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.12.3/pkg/internal/controller/controller.go:273 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2 /home/prow/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.12.3/pkg/internal/controller/controller.go:234 2022-11-25T13:30:21.177410165Z ERROR controller/controller.go:326 Reconciler error {"controller": "job", "controllerGroup": "batch", "controllerKind": "Job", "job": {"name":"test-job","namespace":"default"}, "namespace": "default", "name": "test-job", "reconcileID": "a5d54306-dec0-4da7-a1f7-85a56e42add4", "error": "only one workload should exist, found 2"} sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler /home/prow/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.12.3/pkg/internal/controller/controller.go:326 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem /home/prow/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.12.3/pkg/internal/controller/controller.go:273 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2 /home/prow/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.12.3/pkg/internal/controller/controller.go:234 ... skipping 9 lines ... 2022-11-25T13:30:24.61939029Z DEBUG events recorder/recorder.go:103 Normal {"object": {"kind":"Job","namespace":"default","name":"test-job","uid":"21fda717-f266-4612-b31c-2ce0e9782311","apiVersion":"batch/v1","resourceVersion":"225"}, "reason": "Started", "message": "Admitted by clusterQueue cluster-queue"} 2022-11-25T13:30:24.619462903Z LEVEL(-3) job/job_controller.go:223 Job running with admitted workload, nothing to do {"controller": "job", "controllerGroup": "batch", "controllerKind": "Job", "job": {"name":"test-job","namespace":"default"}, "namespace": "default", "name": "test-job", "reconcileID": "44338bff-5d75-45f7-bc0a-e3692067ba26", "job": {"name":"test-job","namespace":"default"}} [1mSTEP:[0m checking the job gets suspended when parallelism changes and the added node selectors are removed [38;5;243m11/25/22 13:30:27.772[0m 2022-11-25T13:30:27.777726063Z LEVEL(-2) job/job_controller.go:141 Reconciling Job {"controller": "job", "controllerGroup": "batch", "controllerKind": "Job", "job": {"name":"test-job","namespace":"default"}, "namespace": "default", "name": "test-job", "reconcileID": "2106a62d-762b-445b-9433-b4ff0b9c5c44", "job": {"name":"test-job","namespace":"default"}} 2022-11-25T13:30:27.777827287Z LEVEL(-2) job/job_controller.go:362 job with no matching workload, suspending {"controller": "job", "controllerGroup": "batch", "controllerKind": "Job", "job": {"name":"test-job","namespace":"default"}, "namespace": "default", "name": "test-job", "reconcileID": "2106a62d-762b-445b-9433-b4ff0b9c5c44", "job": {"name":"test-job","namespace":"default"}} 2022-11-25T13:30:27.782136769Z DEBUG events recorder/recorder.go:103 Normal {"object": {"kind":"Job","namespace":"default","name":"test-job","uid":"21fda717-f266-4612-b31c-2ce0e9782311","apiVersion":"batch/v1","resourceVersion":"228"}, "reason": "Stopped", "message": "No matching Workload"} 2022-11-25T13:30:27.791823415Z ERROR job/job_controller.go:153 Getting existing workloads {"controller": "job", "controllerGroup": "batch", "controllerKind": "Job", "job": {"name":"test-job","namespace":"default"}, "namespace": "default", "name": "test-job", "reconcileID": "2106a62d-762b-445b-9433-b4ff0b9c5c44", "job": {"name":"test-job","namespace":"default"}, "error": "no matching workload was found, tried deleting 1 existing workload(s)"} sigs.k8s.io/kueue/pkg/controller/workload/job.(*JobReconciler).Reconcile /home/prow/go/src/sigs.k8s.io/kueue/pkg/controller/workload/job/job_controller.go:153 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile /home/prow/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.12.3/pkg/internal/controller/controller.go:121 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler /home/prow/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.12.3/pkg/internal/controller/controller.go:320 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem /home/prow/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.12.3/pkg/internal/controller/controller.go:273 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2 /home/prow/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.12.3/pkg/internal/controller/controller.go:234 2022-11-25T13:30:27.791931177Z ERROR controller/controller.go:326 Reconciler error {"controller": "job", "controllerGroup": "batch", "controllerKind": "Job", "job": {"name":"test-job","namespace":"default"}, "namespace": "default", "name": "test-job", "reconcileID": "2106a62d-762b-445b-9433-b4ff0b9c5c44", "error": "no matching workload was found, tried deleting 1 existing workload(s)"} sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler /home/prow/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.12.3/pkg/internal/controller/controller.go:326 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem /home/prow/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.12.3/pkg/internal/controller/controller.go:273 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2 /home/prow/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.12.3/pkg/internal/controller/controller.go:234 ... skipping 18 lines ... 2022-11-25T13:30:31.568162632Z INFO controller/controller.go:247 Shutdown signal received, waiting for all workers to finish {"controller": "job", "controllerGroup": "batch", "controllerKind": "Job"} 2022-11-25T13:30:31.568199915Z INFO controller/controller.go:249 All workers finished {"controller": "job", "controllerGroup": "batch", "controllerKind": "Job"} 2022-11-25T13:30:31.568222326Z INFO logr@v1.2.3/logr.go:261 Stopping and waiting for caches 2022-11-25T13:30:31.568365079Z INFO logr@v1.2.3/logr.go:261 Stopping and waiting for webhooks 2022-11-25T13:30:31.568407652Z INFO controller-runtime.webhook logr@v1.2.3/logr.go:261 shutting down webhook server 2022-11-25T13:30:31.568436339Z DEBUG controller-runtime.certwatcher logr@v1.2.3/logr.go:261 certificate event {"event": "\"/tmp/envtest-serving-certs-3105267789/tls.key\": REMOVE"} 2022-11-25T13:30:31.568482307Z ERROR controller-runtime.certwatcher logr@v1.2.3/logr.go:279 error re-watching file {"error": "inotify instance already closed"} github.com/go-logr/logr.Logger.Error /home/prow/go/pkg/mod/github.com/go-logr/logr@v1.2.3/logr.go:279 sigs.k8s.io/controller-runtime/pkg/certwatcher.(*CertWatcher).handleEvent /home/prow/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.12.3/pkg/certwatcher/certwatcher.go:147 sigs.k8s.io/controller-runtime/pkg/certwatcher.(*CertWatcher).Watch /home/prow/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.12.3/pkg/certwatcher/certwatcher.go:103 2022-11-25T13:30:31.568543073Z ERROR controller-runtime.certwatcher logr@v1.2.3/logr.go:279 error re-reading certificate {"error": "open /tmp/envtest-serving-certs-3105267789/tls.crt: no such file or directory"} github.com/go-logr/logr.Logger.Error /home/prow/go/pkg/mod/github.com/go-logr/logr@v1.2.3/logr.go:279 sigs.k8s.io/controller-runtime/pkg/certwatcher.(*CertWatcher).handleEvent /home/prow/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.12.3/pkg/certwatcher/certwatcher.go:152 sigs.k8s.io/controller-runtime/pkg/certwatcher.(*CertWatcher).Watch /home/prow/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.12.3/pkg/certwatcher/certwatcher.go:103 2022-11-25T13:30:31.569696256Z INFO logr@v1.2.3/logr.go:261 Wait completed, proceeding to shutdown the manager ... skipping 45 lines ... 2022-11-25T13:30:20.909485668Z LEVEL(-2) job/job_controller.go:141 Reconciling Job {"controller": "job", "controllerGroup": "batch", "controllerKind": "Job", "job": {"name":"test-job","namespace":"default"}, "namespace": "default", "name": "test-job", "reconcileID": "20821244-a9e8-446c-8bf0-fbf786a290c7", "job": {"name":"test-job","namespace":"default"}} 2022-11-25T13:30:20.909601636Z LEVEL(-2) job/job_controller.go:200 Job changed queues, updating workload {"controller": "job", "controllerGroup": "batch", "controllerKind": "Job", "job": {"name":"test-job","namespace":"default"}, "namespace": "default", "name": "test-job", "reconcileID": "20821244-a9e8-446c-8bf0-fbf786a290c7", "job": {"name":"test-job","namespace":"default"}} 2022-11-25T13:30:20.916465873Z LEVEL(-2) job/job_controller.go:141 Reconciling Job {"controller": "job", "controllerGroup": "batch", "controllerKind": "Job", "job": {"name":"test-job","namespace":"default"}, "namespace": "default", "name": "test-job", "reconcileID": "ab6451f1-5998-4877-ae6e-797321fb05d1", "job": {"name":"test-job","namespace":"default"}} 2022-11-25T13:30:20.916611287Z LEVEL(-3) job/job_controller.go:208 Job is suspended and workload not yet admitted by a clusterQueue, nothing to do {"controller": "job", "controllerGroup": "batch", "controllerKind": "Job", "job": {"name":"test-job","namespace":"default"}, "namespace": "default", "name": "test-job", "reconcileID": "ab6451f1-5998-4877-ae6e-797321fb05d1", "job": {"name":"test-job","namespace":"default"}} [1mSTEP:[0m checking a second non-matching workload is deleted [38;5;243m11/25/22 13:30:21.165[0m 2022-11-25T13:30:21.173160419Z LEVEL(-2) job/job_controller.go:141 Reconciling Job {"controller": "job", "controllerGroup": "batch", "controllerKind": "Job", "job": {"name":"test-job","namespace":"default"}, "namespace": "default", "name": "test-job", "reconcileID": "a5d54306-dec0-4da7-a1f7-85a56e42add4", "job": {"name":"test-job","namespace":"default"}} 2022-11-25T13:30:21.177321094Z ERROR job/job_controller.go:153 Getting existing workloads {"controller": "job", "controllerGroup": "batch", "controllerKind": "Job", "job": {"name":"test-job","namespace":"default"}, "namespace": "default", "name": "test-job", "reconcileID": "a5d54306-dec0-4da7-a1f7-85a56e42add4", "job": {"name":"test-job","namespace":"default"}, "error": "only one workload should exist, found 2"} sigs.k8s.io/kueue/pkg/controller/workload/job.(*JobReconciler).Reconcile /home/prow/go/src/sigs.k8s.io/kueue/pkg/controller/workload/job/job_controller.go:153 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile /home/prow/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.12.3/pkg/internal/controller/controller.go:121 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler /home/prow/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.12.3/pkg/internal/controller/controller.go:320 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem /home/prow/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.12.3/pkg/internal/controller/controller.go:273 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2 /home/prow/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.12.3/pkg/internal/controller/controller.go:234 2022-11-25T13:30:21.177410165Z ERROR controller/controller.go:326 Reconciler error {"controller": "job", "controllerGroup": "batch", "controllerKind": "Job", "job": {"name":"test-job","namespace":"default"}, "namespace": "default", "name": "test-job", "reconcileID": "a5d54306-dec0-4da7-a1f7-85a56e42add4", "error": "only one workload should exist, found 2"} sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler /home/prow/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.12.3/pkg/internal/controller/controller.go:326 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem /home/prow/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.12.3/pkg/internal/controller/controller.go:273 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2 /home/prow/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.12.3/pkg/internal/controller/controller.go:234 ... skipping 9 lines ... 2022-11-25T13:30:24.61939029Z DEBUG events recorder/recorder.go:103 Normal {"object": {"kind":"Job","namespace":"default","name":"test-job","uid":"21fda717-f266-4612-b31c-2ce0e9782311","apiVersion":"batch/v1","resourceVersion":"225"}, "reason": "Started", "message": "Admitted by clusterQueue cluster-queue"} 2022-11-25T13:30:24.619462903Z LEVEL(-3) job/job_controller.go:223 Job running with admitted workload, nothing to do {"controller": "job", "controllerGroup": "batch", "controllerKind": "Job", "job": {"name":"test-job","namespace":"default"}, "namespace": "default", "name": "test-job", "reconcileID": "44338bff-5d75-45f7-bc0a-e3692067ba26", "job": {"name":"test-job","namespace":"default"}} [1mSTEP:[0m checking the job gets suspended when parallelism changes and the added node selectors are removed [38;5;243m11/25/22 13:30:27.772[0m 2022-11-25T13:30:27.777726063Z LEVEL(-2) job/job_controller.go:141 Reconciling Job {"controller": "job", "controllerGroup": "batch", "controllerKind": "Job", "job": {"name":"test-job","namespace":"default"}, "namespace": "default", "name": "test-job", "reconcileID": "2106a62d-762b-445b-9433-b4ff0b9c5c44", "job": {"name":"test-job","namespace":"default"}} 2022-11-25T13:30:27.777827287Z LEVEL(-2) job/job_controller.go:362 job with no matching workload, suspending {"controller": "job", "controllerGroup": "batch", "controllerKind": "Job", "job": {"name":"test-job","namespace":"default"}, "namespace": "default", "name": "test-job", "reconcileID": "2106a62d-762b-445b-9433-b4ff0b9c5c44", "job": {"name":"test-job","namespace":"default"}} 2022-11-25T13:30:27.782136769Z DEBUG events recorder/recorder.go:103 Normal {"object": {"kind":"Job","namespace":"default","name":"test-job","uid":"21fda717-f266-4612-b31c-2ce0e9782311","apiVersion":"batch/v1","resourceVersion":"228"}, "reason": "Stopped", "message": "No matching Workload"} 2022-11-25T13:30:27.791823415Z ERROR job/job_controller.go:153 Getting existing workloads {"controller": "job", "controllerGroup": "batch", "controllerKind": "Job", "job": {"name":"test-job","namespace":"default"}, "namespace": "default", "name": "test-job", "reconcileID": "2106a62d-762b-445b-9433-b4ff0b9c5c44", "job": {"name":"test-job","namespace":"default"}, "error": "no matching workload was found, tried deleting 1 existing workload(s)"} sigs.k8s.io/kueue/pkg/controller/workload/job.(*JobReconciler).Reconcile /home/prow/go/src/sigs.k8s.io/kueue/pkg/controller/workload/job/job_controller.go:153 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile /home/prow/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.12.3/pkg/internal/controller/controller.go:121 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler /home/prow/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.12.3/pkg/internal/controller/controller.go:320 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem /home/prow/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.12.3/pkg/internal/controller/controller.go:273 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2 /home/prow/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.12.3/pkg/internal/controller/controller.go:234 2022-11-25T13:30:27.791931177Z ERROR controller/controller.go:326 Reconciler error {"controller": "job", "controllerGroup": "batch", "controllerKind": "Job", "job": {"name":"test-job","namespace":"default"}, "namespace": "default", "name": "test-job", "reconcileID": "2106a62d-762b-445b-9433-b4ff0b9c5c44", "error": "no matching workload was found, tried deleting 1 existing workload(s)"} sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler /home/prow/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.12.3/pkg/internal/controller/controller.go:326 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem /home/prow/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.12.3/pkg/internal/controller/controller.go:273 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2 /home/prow/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.12.3/pkg/internal/controller/controller.go:234 ... skipping 18 lines ... 2022-11-25T13:30:31.568162632Z INFO controller/controller.go:247 Shutdown signal received, waiting for all workers to finish {"controller": "job", "controllerGroup": "batch", "controllerKind": "Job"} 2022-11-25T13:30:31.568199915Z INFO controller/controller.go:249 All workers finished {"controller": "job", "controllerGroup": "batch", "controllerKind": "Job"} 2022-11-25T13:30:31.568222326Z INFO logr@v1.2.3/logr.go:261 Stopping and waiting for caches 2022-11-25T13:30:31.568365079Z INFO logr@v1.2.3/logr.go:261 Stopping and waiting for webhooks 2022-11-25T13:30:31.568407652Z INFO controller-runtime.webhook logr@v1.2.3/logr.go:261 shutting down webhook server 2022-11-25T13:30:31.568436339Z DEBUG controller-runtime.certwatcher logr@v1.2.3/logr.go:261 certificate event {"event": "\"/tmp/envtest-serving-certs-3105267789/tls.key\": REMOVE"} 2022-11-25T13:30:31.568482307Z ERROR controller-runtime.certwatcher logr@v1.2.3/logr.go:279 error re-watching file {"error": "inotify instance already closed"} github.com/go-logr/logr.Logger.Error /home/prow/go/pkg/mod/github.com/go-logr/logr@v1.2.3/logr.go:279 sigs.k8s.io/controller-runtime/pkg/certwatcher.(*CertWatcher).handleEvent /home/prow/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.12.3/pkg/certwatcher/certwatcher.go:147 sigs.k8s.io/controller-runtime/pkg/certwatcher.(*CertWatcher).Watch /home/prow/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.12.3/pkg/certwatcher/certwatcher.go:103 2022-11-25T13:30:31.568543073Z ERROR controller-runtime.certwatcher logr@v1.2.3/logr.go:279 error re-reading certificate {"error": "open /tmp/envtest-serving-certs-3105267789/tls.crt: no such file or directory"} github.com/go-logr/logr.Logger.Error /home/prow/go/pkg/mod/github.com/go-logr/logr@v1.2.3/logr.go:279 sigs.k8s.io/controller-runtime/pkg/certwatcher.(*CertWatcher).handleEvent /home/prow/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.12.3/pkg/certwatcher/certwatcher.go:152 sigs.k8s.io/controller-runtime/pkg/certwatcher.(*CertWatcher).Watch /home/prow/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.12.3/pkg/certwatcher/certwatcher.go:103 2022-11-25T13:30:31.569696256Z INFO logr@v1.2.3/logr.go:261 Wait completed, proceeding to shutdown the manager ... skipping 69 lines ... 2022-11-25T13:30:40.111341793Z LEVEL(-2) localqueue-reconciler core/localqueue_controller.go:102 LocalQueue create event {"localQueue": {"name":"dev-queue","namespace":"core-zs2wg"}} 2022-11-25T13:30:40.111516392Z LEVEL(-2) core/localqueue_controller.go:75 Reconciling LocalQueue {"controller": "localqueue", "controllerGroup": "kueue.x-k8s.io", "controllerKind": "LocalQueue", "localQueue": {"name":"dev-queue","namespace":"core-zs2wg"}, "namespace": "core-zs2wg", "name": "dev-queue", "reconcileID": "c4c6c7ab-4945-4fc7-921b-d9798ebf16b5", "localQueue": {"name":"dev-queue","namespace":"core-zs2wg"}} [1mSTEP:[0m checking the first prod job starts [38;5;243m11/25/22 13:30:40.111[0m 2022-11-25T13:30:40.115440727Z LEVEL(-2) core/clusterqueue_controller.go:81 Reconciling ClusterQueue {"controller": "clusterqueue", "controllerGroup": "kueue.x-k8s.io", "controllerKind": "ClusterQueue", "clusterQueue": {"name":"prod-cq"}, "namespace": "", "name": "prod-cq", "reconcileID": "8ee772e1-6cbe-465c-9a71-bc6d97bde8bc", "clusterQueue": {"name":"prod-cq"}} 2022-11-25T13:30:40.115623422Z LEVEL(-2) core/clusterqueue_controller.go:81 Reconciling ClusterQueue {"controller": "clusterqueue", "controllerGroup": "kueue.x-k8s.io", "controllerKind": "ClusterQueue", "clusterQueue": {"name":"dev-clusterqueue"}, "namespace": "", "name": "dev-clusterqueue", "reconcileID": "e4898472-6ed8-4826-be1f-a4db13956247", "clusterQueue": {"name":"dev-clusterqueue"}} 2022-11-25T13:30:40.115876687Z LEVEL(-2) cluster-queue-reconciler core/clusterqueue_controller.go:184 ClusterQueue update event {"clusterQueue": {"name":"dev-clusterqueue"}} 2022-11-25T13:30:40.119812137Z ERROR controller/controller.go:326 Reconciler error {"controller": "clusterqueue", "controllerGroup": "kueue.x-k8s.io", "controllerKind": "ClusterQueue", "clusterQueue": {"name":"dev-clusterqueue"}, "namespace": "", "name": "dev-clusterqueue", "reconcileID": "e4898472-6ed8-4826-be1f-a4db13956247", "error": "Operation cannot be fulfilled on clusterqueues.kueue.x-k8s.io \"dev-clusterqueue\": the object has been modified; please apply your changes to the latest version and try again"} sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler /home/prow/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.12.3/pkg/internal/controller/controller.go:326 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem /home/prow/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.12.3/pkg/internal/controller/controller.go:273 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2 /home/prow/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.12.3/pkg/internal/controller/controller.go:234 ... skipping 174 lines ... 2022-11-25T13:30:44.736663363Z LEVEL(-2) resourceflavor-reconciler core/resourceflavor_controller.go:128 ResourceFlavor delete event {"resourceFlavor": {"name":"spot-tainted"}} 2022-11-25T13:30:44.736727567Z LEVEL(-2) resourceflavor-reconciler core/resourceflavor_controller.go:143 ResourceFlavor update event {"resourceFlavor": {"name":"spot-untainted"}} [1mSTEP:[0m tearing down the test environment [38;5;243m11/25/22 13:30:44.736[0m 2022-11-25T13:30:44.73692371Z LEVEL(-2) core/resourceflavor_controller.go:69 Reconciling ResourceFlavor {"controller": "resourceflavor", "controllerGroup": "kueue.x-k8s.io", "controllerKind": "ResourceFlavor", "resourceFlavor": {"name":"spot-untainted"}, "namespace": "", "name": "spot-untainted", "reconcileID": "e7a51687-8f5f-4fe4-959c-5b3d3597d852", "resourceFlavor": {"name":"spot-untainted"}} 2022-11-25T13:30:44.737102153Z INFO logr@v1.2.3/logr.go:261 Stopping and waiting for non leader election runnables 2022-11-25T13:30:44.737161912Z INFO logr@v1.2.3/logr.go:261 Stopping and waiting for leader election runnables 2022-11-25T13:30:44.73729923Z ERROR controller/controller.go:326 Reconciler error {"controller": "resourceflavor", "controllerGroup": "kueue.x-k8s.io", "controllerKind": "ResourceFlavor", "resourceFlavor": {"name":"spot-untainted"}, "namespace": "", "name": "spot-untainted", "reconcileID": "e7a51687-8f5f-4fe4-959c-5b3d3597d852", "error": "Put \"https://127.0.0.1:41791/apis/kueue.x-k8s.io/v1alpha2/resourceflavors/spot-untainted\": context canceled"} sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler /home/prow/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.12.3/pkg/internal/controller/controller.go:326 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem /home/prow/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.12.3/pkg/internal/controller/controller.go:273 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2 /home/prow/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.12.3/pkg/internal/controller/controller.go:234 2022-11-25T13:30:44.737299259Z DEBUG controller-runtime.certwatcher logr@v1.2.3/logr.go:261 certificate event {"event": "\"/tmp/envtest-serving-certs-2180812139/tls.key\": REMOVE"} 2022-11-25T13:30:44.737392144Z INFO controller/controller.go:247 Shutdown signal received, waiting for all workers to finish {"controller": "localqueue", "controllerGroup": "kueue.x-k8s.io", "controllerKind": "LocalQueue"} 2022-11-25T13:30:44.737409122Z INFO controller/controller.go:247 Shutdown signal received, waiting for all workers to finish {"controller": "job", "controllerGroup": "batch", "controllerKind": "Job"} 2022-11-25T13:30:44.737431812Z INFO controller/controller.go:247 Shutdown signal received, waiting for all workers to finish {"controller": "resourceflavor", "controllerGroup": "kueue.x-k8s.io", "controllerKind": "ResourceFlavor"} 2022-11-25T13:30:44.737427966Z ERROR controller-runtime.certwatcher logr@v1.2.3/logr.go:279 error re-watching file {"error": "no such file or directory"} github.com/go-logr/logr.Logger.Error /home/prow/go/pkg/mod/github.com/go-logr/logr@v1.2.3/logr.go:279 sigs.k8s.io/controller-runtime/pkg/certwatcher.(*CertWatcher).handleEvent /home/prow/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.12.3/pkg/certwatcher/certwatcher.go:147 sigs.k8s.io/controller-runtime/pkg/certwatcher.(*CertWatcher).Watch /home/prow/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.12.3/pkg/certwatcher/certwatcher.go:103 2022-11-25T13:30:44.737452772Z INFO controller/controller.go:247 Shutdown signal received, waiting for all workers to finish {"controller": "clusterqueue", "controllerGroup": "kueue.x-k8s.io", "controllerKind": "ClusterQueue"} 2022-11-25T13:30:44.737456204Z INFO controller/controller.go:249 All workers finished {"controller": "localqueue", "controllerGroup": "kueue.x-k8s.io", "controllerKind": "LocalQueue"} 2022-11-25T13:30:44.737462044Z ERROR controller-runtime.certwatcher logr@v1.2.3/logr.go:279 error re-reading certificate {"error": "open /tmp/envtest-serving-certs-2180812139/tls.crt: no such file or directory"} github.com/go-logr/logr.Logger.Error /home/prow/go/pkg/mod/github.com/go-logr/logr@v1.2.3/logr.go:279 sigs.k8s.io/controller-runtime/pkg/certwatcher.(*CertWatcher).handleEvent /home/prow/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.12.3/pkg/certwatcher/certwatcher.go:152 sigs.k8s.io/controller-runtime/pkg/certwatcher.(*CertWatcher).Watch /home/prow/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.12.3/pkg/certwatcher/certwatcher.go:103 2022-11-25T13:30:44.737519981Z DEBUG controller-runtime.certwatcher logr@v1.2.3/logr.go:261 certificate event {"event": "\"/tmp/envtest-serving-certs-2180812139/tls.crt\": REMOVE"} 2022-11-25T13:30:44.737539367Z ERROR controller-runtime.certwatcher logr@v1.2.3/logr.go:279 error re-watching file {"error": "no such file or directory"} github.com/go-logr/logr.Logger.Error /home/prow/go/pkg/mod/github.com/go-logr/logr@v1.2.3/logr.go:279 sigs.k8s.io/controller-runtime/pkg/certwatcher.(*CertWatcher).handleEvent /home/prow/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.12.3/pkg/certwatcher/certwatcher.go:147 sigs.k8s.io/controller-runtime/pkg/certwatcher.(*CertWatcher).Watch /home/prow/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.12.3/pkg/certwatcher/certwatcher.go:103 2022-11-25T13:30:44.737554869Z ERROR controller-runtime.certwatcher logr@v1.2.3/logr.go:279 error re-reading certificate {"error": "open /tmp/envtest-serving-certs-2180812139/tls.crt: no such file or directory"} github.com/go-logr/logr.Logger.Error /home/prow/go/pkg/mod/github.com/go-logr/logr@v1.2.3/logr.go:279 sigs.k8s.io/controller-runtime/pkg/certwatcher.(*CertWatcher).handleEvent /home/prow/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.12.3/pkg/certwatcher/certwatcher.go:152 sigs.k8s.io/controller-runtime/pkg/certwatcher.(*CertWatcher).Watch /home/prow/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.12.3/pkg/certwatcher/certwatcher.go:103 2022-11-25T13:30:44.737470436Z INFO controller/controller.go:249 All workers finished {"controller": "resourceflavor", "controllerGroup": "kueue.x-k8s.io", "controllerKind": "ResourceFlavor"} ... skipping 80 lines ... 2022-11-25T13:30:40.111341793Z LEVEL(-2) localqueue-reconciler core/localqueue_controller.go:102 LocalQueue create event {"localQueue": {"name":"dev-queue","namespace":"core-zs2wg"}} 2022-11-25T13:30:40.111516392Z LEVEL(-2) core/localqueue_controller.go:75 Reconciling LocalQueue {"controller": "localqueue", "controllerGroup": "kueue.x-k8s.io", "controllerKind": "LocalQueue", "localQueue": {"name":"dev-queue","namespace":"core-zs2wg"}, "namespace": "core-zs2wg", "name": "dev-queue", "reconcileID": "c4c6c7ab-4945-4fc7-921b-d9798ebf16b5", "localQueue": {"name":"dev-queue","namespace":"core-zs2wg"}} [1mSTEP:[0m checking the first prod job starts [38;5;243m11/25/22 13:30:40.111[0m 2022-11-25T13:30:40.115440727Z LEVEL(-2) core/clusterqueue_controller.go:81 Reconciling ClusterQueue {"controller": "clusterqueue", "controllerGroup": "kueue.x-k8s.io", "controllerKind": "ClusterQueue", "clusterQueue": {"name":"prod-cq"}, "namespace": "", "name": "prod-cq", "reconcileID": "8ee772e1-6cbe-465c-9a71-bc6d97bde8bc", "clusterQueue": {"name":"prod-cq"}} 2022-11-25T13:30:40.115623422Z LEVEL(-2) core/clusterqueue_controller.go:81 Reconciling ClusterQueue {"controller": "clusterqueue", "controllerGroup": "kueue.x-k8s.io", "controllerKind": "ClusterQueue", "clusterQueue": {"name":"dev-clusterqueue"}, "namespace": "", "name": "dev-clusterqueue", "reconcileID": "e4898472-6ed8-4826-be1f-a4db13956247", "clusterQueue": {"name":"dev-clusterqueue"}} 2022-11-25T13:30:40.115876687Z LEVEL(-2) cluster-queue-reconciler core/clusterqueue_controller.go:184 ClusterQueue update event {"clusterQueue": {"name":"dev-clusterqueue"}} 2022-11-25T13:30:40.119812137Z ERROR controller/controller.go:326 Reconciler error {"controller": "clusterqueue", "controllerGroup": "kueue.x-k8s.io", "controllerKind": "ClusterQueue", "clusterQueue": {"name":"dev-clusterqueue"}, "namespace": "", "name": "dev-clusterqueue", "reconcileID": "e4898472-6ed8-4826-be1f-a4db13956247", "error": "Operation cannot be fulfilled on clusterqueues.kueue.x-k8s.io \"dev-clusterqueue\": the object has been modified; please apply your changes to the latest version and try again"} sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler /home/prow/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.12.3/pkg/internal/controller/controller.go:326 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem /home/prow/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.12.3/pkg/internal/controller/controller.go:273 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2 /home/prow/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.12.3/pkg/internal/controller/controller.go:234 ... skipping 174 lines ... 2022-11-25T13:30:44.736663363Z LEVEL(-2) resourceflavor-reconciler core/resourceflavor_controller.go:128 ResourceFlavor delete event {"resourceFlavor": {"name":"spot-tainted"}} 2022-11-25T13:30:44.736727567Z LEVEL(-2) resourceflavor-reconciler core/resourceflavor_controller.go:143 ResourceFlavor update event {"resourceFlavor": {"name":"spot-untainted"}} [1mSTEP:[0m tearing down the test environment [38;5;243m11/25/22 13:30:44.736[0m 2022-11-25T13:30:44.73692371Z LEVEL(-2) core/resourceflavor_controller.go:69 Reconciling ResourceFlavor {"controller": "resourceflavor", "controllerGroup": "kueue.x-k8s.io", "controllerKind": "ResourceFlavor", "resourceFlavor": {"name":"spot-untainted"}, "namespace": "", "name": "spot-untainted", "reconcileID": "e7a51687-8f5f-4fe4-959c-5b3d3597d852", "resourceFlavor": {"name":"spot-untainted"}} 2022-11-25T13:30:44.737102153Z INFO logr@v1.2.3/logr.go:261 Stopping and waiting for non leader election runnables 2022-11-25T13:30:44.737161912Z INFO logr@v1.2.3/logr.go:261 Stopping and waiting for leader election runnables 2022-11-25T13:30:44.73729923Z ERROR controller/controller.go:326 Reconciler error {"controller": "resourceflavor", "controllerGroup": "kueue.x-k8s.io", "controllerKind": "ResourceFlavor", "resourceFlavor": {"name":"spot-untainted"}, "namespace": "", "name": "spot-untainted", "reconcileID": "e7a51687-8f5f-4fe4-959c-5b3d3597d852", "error": "Put \"https://127.0.0.1:41791/apis/kueue.x-k8s.io/v1alpha2/resourceflavors/spot-untainted\": context canceled"} sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler /home/prow/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.12.3/pkg/internal/controller/controller.go:326 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem /home/prow/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.12.3/pkg/internal/controller/controller.go:273 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2 /home/prow/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.12.3/pkg/internal/controller/controller.go:234 2022-11-25T13:30:44.737299259Z DEBUG controller-runtime.certwatcher logr@v1.2.3/logr.go:261 certificate event {"event": "\"/tmp/envtest-serving-certs-2180812139/tls.key\": REMOVE"} 2022-11-25T13:30:44.737392144Z INFO controller/controller.go:247 Shutdown signal received, waiting for all workers to finish {"controller": "localqueue", "controllerGroup": "kueue.x-k8s.io", "controllerKind": "LocalQueue"} 2022-11-25T13:30:44.737409122Z INFO controller/controller.go:247 Shutdown signal received, waiting for all workers to finish {"controller": "job", "controllerGroup": "batch", "controllerKind": "Job"} 2022-11-25T13:30:44.737431812Z INFO controller/controller.go:247 Shutdown signal received, waiting for all workers to finish {"controller": "resourceflavor", "controllerGroup": "kueue.x-k8s.io", "controllerKind": "ResourceFlavor"} 2022-11-25T13:30:44.737427966Z ERROR controller-runtime.certwatcher logr@v1.2.3/logr.go:279 error re-watching file {"error": "no such file or directory"} github.com/go-logr/logr.Logger.Error /home/prow/go/pkg/mod/github.com/go-logr/logr@v1.2.3/logr.go:279 sigs.k8s.io/controller-runtime/pkg/certwatcher.(*CertWatcher).handleEvent /home/prow/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.12.3/pkg/certwatcher/certwatcher.go:147 sigs.k8s.io/controller-runtime/pkg/certwatcher.(*CertWatcher).Watch /home/prow/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.12.3/pkg/certwatcher/certwatcher.go:103 2022-11-25T13:30:44.737452772Z INFO controller/controller.go:247 Shutdown signal received, waiting for all workers to finish {"controller": "clusterqueue", "controllerGroup": "kueue.x-k8s.io", "controllerKind": "ClusterQueue"} 2022-11-25T13:30:44.737456204Z INFO controller/controller.go:249 All workers finished {"controller": "localqueue", "controllerGroup": "kueue.x-k8s.io", "controllerKind": "LocalQueue"} 2022-11-25T13:30:44.737462044Z ERROR controller-runtime.certwatcher logr@v1.2.3/logr.go:279 error re-reading certificate {"error": "open /tmp/envtest-serving-certs-2180812139/tls.crt: no such file or directory"} github.com/go-logr/logr.Logger.Error /home/prow/go/pkg/mod/github.com/go-logr/logr@v1.2.3/logr.go:279 sigs.k8s.io/controller-runtime/pkg/certwatcher.(*CertWatcher).handleEvent /home/prow/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.12.3/pkg/certwatcher/certwatcher.go:152 sigs.k8s.io/controller-runtime/pkg/certwatcher.(*CertWatcher).Watch /home/prow/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.12.3/pkg/certwatcher/certwatcher.go:103 2022-11-25T13:30:44.737519981Z DEBUG controller-runtime.certwatcher logr@v1.2.3/logr.go:261 certificate event {"event": "\"/tmp/envtest-serving-certs-2180812139/tls.crt\": REMOVE"} 2022-11-25T13:30:44.737539367Z ERROR controller-runtime.certwatcher logr@v1.2.3/logr.go:279 error re-watching file {"error": "no such file or directory"} github.com/go-logr/logr.Logger.Error /home/prow/go/pkg/mod/github.com/go-logr/logr@v1.2.3/logr.go:279 sigs.k8s.io/controller-runtime/pkg/certwatcher.(*CertWatcher).handleEvent /home/prow/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.12.3/pkg/certwatcher/certwatcher.go:147 sigs.k8s.io/controller-runtime/pkg/certwatcher.(*CertWatcher).Watch /home/prow/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.12.3/pkg/certwatcher/certwatcher.go:103 2022-11-25T13:30:44.737554869Z ERROR controller-runtime.certwatcher logr@v1.2.3/logr.go:279 error re-reading certificate {"error": "open /tmp/envtest-serving-certs-2180812139/tls.crt: no such file or directory"} github.com/go-logr/logr.Logger.Error /home/prow/go/pkg/mod/github.com/go-logr/logr@v1.2.3/logr.go:279 sigs.k8s.io/controller-runtime/pkg/certwatcher.(*CertWatcher).handleEvent /home/prow/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.12.3/pkg/certwatcher/certwatcher.go:152 sigs.k8s.io/controller-runtime/pkg/certwatcher.(*CertWatcher).Watch /home/prow/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.12.3/pkg/certwatcher/certwatcher.go:103 2022-11-25T13:30:44.737470436Z INFO controller/controller.go:249 All workers finished {"controller": "resourceflavor", "controllerGroup": "kueue.x-k8s.io", "controllerKind": "ResourceFlavor"} ... skipping 45 lines ... 2022-11-25T13:30:54.097590382Z LEVEL(-2) job/job_controller.go:141 Reconciling Job {"controller": "job", "controllerGroup": "batch", "controllerKind": "Job", "job": {"name":"test-job","namespace":"default"}, "namespace": "default", "name": "test-job", "reconcileID": "9e945d92-e793-408a-a987-fcadd01578d4", "job": {"name":"test-job","namespace":"default"}} 2022-11-25T13:30:54.206326148Z LEVEL(-2) job/job_controller.go:141 Reconciling Job {"controller": "job", "controllerGroup": "batch", "controllerKind": "Job", "job": {"name":"test-job","namespace":"default"}, "namespace": "default", "name": "test-job", "reconcileID": "40f6b02f-f448-4f2a-b8f1-b123af97d5a9", "job": {"name":"test-job","namespace":"default"}} 2022-11-25T13:30:54.206413317Z LEVEL(-3) job/job_controller.go:208 Job is suspended and workload not yet admitted by a clusterQueue, nothing to do {"controller": "job", "controllerGroup": "batch", "controllerKind": "Job", "job": {"name":"test-job","namespace":"default"}, "namespace": "default", "name": "test-job", "reconcileID": "40f6b02f-f448-4f2a-b8f1-b123af97d5a9", "job": {"name":"test-job","namespace":"default"}} 2022-11-25T13:30:54.206622371Z DEBUG events recorder/recorder.go:103 Normal {"object": {"kind":"Job","namespace":"default","name":"test-job","uid":"dd6c4ed9-b7dc-427e-bce9-09d35e2b19ed","apiVersion":"batch/v1","resourceVersion":"213"}, "reason": "CreatedWorkload", "message": "Created Workload: default/test-job"} [1mSTEP:[0m tearing down the test environment [38;5;243m11/25/22 13:30:54.354[0m 2022-11-25T13:30:54.355226025Z DEBUG controller-runtime.certwatcher logr@v1.2.3/logr.go:261 certificate event {"event": "\"/tmp/envtest-serving-certs-892651104/tls.key\": REMOVE"} 2022-11-25T13:30:54.355302768Z ERROR controller-runtime.certwatcher logr@v1.2.3/logr.go:279 error re-watching file {"error": "no such file or directory"} github.com/go-logr/logr.Logger.Error /home/prow/go/pkg/mod/github.com/go-logr/logr@v1.2.3/logr.go:279 sigs.k8s.io/controller-runtime/pkg/certwatcher.(*CertWatcher).handleEvent /home/prow/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.12.3/pkg/certwatcher/certwatcher.go:147 sigs.k8s.io/controller-runtime/pkg/certwatcher.(*CertWatcher).Watch /home/prow/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.12.3/pkg/certwatcher/certwatcher.go:103 2022-11-25T13:30:54.355278504Z INFO logr@v1.2.3/logr.go:261 Stopping and waiting for non leader election runnables 2022-11-25T13:30:54.3553323Z ERROR controller-runtime.certwatcher logr@v1.2.3/logr.go:279 error re-reading certificate {"error": "open /tmp/envtest-serving-certs-892651104/tls.crt: no such file or directory"} github.com/go-logr/logr.Logger.Error /home/prow/go/pkg/mod/github.com/go-logr/logr@v1.2.3/logr.go:279 sigs.k8s.io/controller-runtime/pkg/certwatcher.(*CertWatcher).handleEvent /home/prow/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.12.3/pkg/certwatcher/certwatcher.go:152 sigs.k8s.io/controller-runtime/pkg/certwatcher.(*CertWatcher).Watch /home/prow/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.12.3/pkg/certwatcher/certwatcher.go:103 2022-11-25T13:30:54.355345979Z DEBUG controller-runtime.certwatcher logr@v1.2.3/logr.go:261 certificate event {"event": "\"/tmp/envtest-serving-certs-892651104/tls.crt\": REMOVE"} 2022-11-25T13:30:54.355352303Z INFO logr@v1.2.3/logr.go:261 Stopping and waiting for leader election runnables 2022-11-25T13:30:54.355356929Z ERROR controller-runtime.certwatcher logr@v1.2.3/logr.go:279 error re-watching file {"error": "no such file or directory"} github.com/go-logr/logr.Logger.Error /home/prow/go/pkg/mod/github.com/go-logr/logr@v1.2.3/logr.go:279 sigs.k8s.io/controller-runtime/pkg/certwatcher.(*CertWatcher).handleEvent /home/prow/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.12.3/pkg/certwatcher/certwatcher.go:147 sigs.k8s.io/controller-runtime/pkg/certwatcher.(*CertWatcher).Watch /home/prow/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.12.3/pkg/certwatcher/certwatcher.go:103 2022-11-25T13:30:54.355400095Z ERROR controller-runtime.certwatcher logr@v1.2.3/logr.go:279 error re-reading certificate {"error": "open /tmp/envtest-serving-certs-892651104/tls.crt: no such file or directory"} github.com/go-logr/logr.Logger.Error /home/prow/go/pkg/mod/github.com/go-logr/logr@v1.2.3/logr.go:279 sigs.k8s.io/controller-runtime/pkg/certwatcher.(*CertWatcher).handleEvent /home/prow/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.12.3/pkg/certwatcher/certwatcher.go:152 sigs.k8s.io/controller-runtime/pkg/certwatcher.(*CertWatcher).Watch /home/prow/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.12.3/pkg/certwatcher/certwatcher.go:103 2022-11-25T13:30:54.355401433Z INFO controller/controller.go:247 Shutdown signal received, waiting for all workers to finish {"controller": "job", "controllerGroup": "batch", "controllerKind": "Job"} ... skipping 45 lines ... 2022-11-25T13:30:54.097590382Z LEVEL(-2) job/job_controller.go:141 Reconciling Job {"controller": "job", "controllerGroup": "batch", "controllerKind": "Job", "job": {"name":"test-job","namespace":"default"}, "namespace": "default", "name": "test-job", "reconcileID": "9e945d92-e793-408a-a987-fcadd01578d4", "job": {"name":"test-job","namespace":"default"}} 2022-11-25T13:30:54.206326148Z LEVEL(-2) job/job_controller.go:141 Reconciling Job {"controller": "job", "controllerGroup": "batch", "controllerKind": "Job", "job": {"name":"test-job","namespace":"default"}, "namespace": "default", "name": "test-job", "reconcileID": "40f6b02f-f448-4f2a-b8f1-b123af97d5a9", "job": {"name":"test-job","namespace":"default"}} 2022-11-25T13:30:54.206413317Z LEVEL(-3) job/job_controller.go:208 Job is suspended and workload not yet admitted by a clusterQueue, nothing to do {"controller": "job", "controllerGroup": "batch", "controllerKind": "Job", "job": {"name":"test-job","namespace":"default"}, "namespace": "default", "name": "test-job", "reconcileID": "40f6b02f-f448-4f2a-b8f1-b123af97d5a9", "job": {"name":"test-job","namespace":"default"}} 2022-11-25T13:30:54.206622371Z DEBUG events recorder/recorder.go:103 Normal {"object": {"kind":"Job","namespace":"default","name":"test-job","uid":"dd6c4ed9-b7dc-427e-bce9-09d35e2b19ed","apiVersion":"batch/v1","resourceVersion":"213"}, "reason": "CreatedWorkload", "message": "Created Workload: default/test-job"} [1mSTEP:[0m tearing down the test environment [38;5;243m11/25/22 13:30:54.354[0m 2022-11-25T13:30:54.355226025Z DEBUG controller-runtime.certwatcher logr@v1.2.3/logr.go:261 certificate event {"event": "\"/tmp/envtest-serving-certs-892651104/tls.key\": REMOVE"} 2022-11-25T13:30:54.355302768Z ERROR controller-runtime.certwatcher logr@v1.2.3/logr.go:279 error re-watching file {"error": "no such file or directory"} github.com/go-logr/logr.Logger.Error /home/prow/go/pkg/mod/github.com/go-logr/logr@v1.2.3/logr.go:279 sigs.k8s.io/controller-runtime/pkg/certwatcher.(*CertWatcher).handleEvent /home/prow/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.12.3/pkg/certwatcher/certwatcher.go:147 sigs.k8s.io/controller-runtime/pkg/certwatcher.(*CertWatcher).Watch /home/prow/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.12.3/pkg/certwatcher/certwatcher.go:103 2022-11-25T13:30:54.355278504Z INFO logr@v1.2.3/logr.go:261 Stopping and waiting for non leader election runnables 2022-11-25T13:30:54.3553323Z ERROR controller-runtime.certwatcher logr@v1.2.3/logr.go:279 error re-reading certificate {"error": "open /tmp/envtest-serving-certs-892651104/tls.crt: no such file or directory"} github.com/go-logr/logr.Logger.Error /home/prow/go/pkg/mod/github.com/go-logr/logr@v1.2.3/logr.go:279 sigs.k8s.io/controller-runtime/pkg/certwatcher.(*CertWatcher).handleEvent /home/prow/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.12.3/pkg/certwatcher/certwatcher.go:152 sigs.k8s.io/controller-runtime/pkg/certwatcher.(*CertWatcher).Watch /home/prow/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.12.3/pkg/certwatcher/certwatcher.go:103 2022-11-25T13:30:54.355345979Z DEBUG controller-runtime.certwatcher logr@v1.2.3/logr.go:261 certificate event {"event": "\"/tmp/envtest-serving-certs-892651104/tls.crt\": REMOVE"} 2022-11-25T13:30:54.355352303Z INFO logr@v1.2.3/logr.go:261 Stopping and waiting for leader election runnables 2022-11-25T13:30:54.355356929Z ERROR controller-runtime.certwatcher logr@v1.2.3/logr.go:279 error re-watching file {"error": "no such file or directory"} github.com/go-logr/logr.Logger.Error /home/prow/go/pkg/mod/github.com/go-logr/logr@v1.2.3/logr.go:279 sigs.k8s.io/controller-runtime/pkg/certwatcher.(*CertWatcher).handleEvent /home/prow/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.12.3/pkg/certwatcher/certwatcher.go:147 sigs.k8s.io/controller-runtime/pkg/certwatcher.(*CertWatcher).Watch /home/prow/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.12.3/pkg/certwatcher/certwatcher.go:103 2022-11-25T13:30:54.355400095Z ERROR controller-runtime.certwatcher logr@v1.2.3/logr.go:279 error re-reading certificate {"error": "open /tmp/envtest-serving-certs-892651104/tls.crt: no such file or directory"} github.com/go-logr/logr.Logger.Error /home/prow/go/pkg/mod/github.com/go-logr/logr@v1.2.3/logr.go:279 sigs.k8s.io/controller-runtime/pkg/certwatcher.(*CertWatcher).handleEvent /home/prow/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.12.3/pkg/certwatcher/certwatcher.go:152 sigs.k8s.io/controller-runtime/pkg/certwatcher.(*CertWatcher).Watch /home/prow/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.12.3/pkg/certwatcher/certwatcher.go:103 2022-11-25T13:30:54.355401433Z INFO controller/controller.go:247 Shutdown signal received, waiting for all workers to finish {"controller": "job", "controllerGroup": "batch", "controllerKind": "Job"} ... skipping 10 lines ... [38;5;10m[ReportAfterSuite] PASSED [0.004 seconds][0m [ReportAfterSuite] Autogenerated ReportAfterSuite for --junit-report [38;5;243mautogenerated by Ginkgo[0m [38;5;243m------------------------------[0m [38;5;10m[1mRan 8 of 8 Specs in 76.415 seconds[0m [38;5;10m[1mSUCCESS![0m -- [38;5;10m[1m8 Passed[0m | [38;5;9m[1m0 Failed[0m | [38;5;11m[1m0 Pending[0m | [38;5;14m[1m0 Skipped[0m PASS Running Suite: Scheduler Suite - /home/prow/go/src/sigs.k8s.io/kueue/test/integration/scheduler =============================================================================================== Random Seed: [1m1669382947[0m Will run [1m20[0m of [1m20[0m specs ... skipping 655 lines ... 2022-11-25T13:31:05.600492144Z LEVEL(-2) workload-reconciler core/workload_controller.go:198 Workload update event {"workload": {"name":"wl-high-priority","namespace":"core-4g2kt"}, "queue": "queue", "status": "admitted", "prevStatus": "pending", "clusterQueue": "prod-cq"} 2022-11-25T13:31:05.600682212Z LEVEL(-2) core/workload_controller.go:82 Reconciling Workload {"controller": "workload", "controllerGroup": "kueue.x-k8s.io", "controllerKind": "Workload", "workload": {"name":"wl-high-priority","namespace":"core-4g2kt"}, "namespace": "core-4g2kt", "name": "wl-high-priority", "reconcileID": "1493c4d4-7c1c-4dda-94ce-129af7b41953", "workload": {"name":"wl-high-priority","namespace":"core-4g2kt"}} 2022-11-25T13:31:05.600729238Z LEVEL(-3) localqueue-reconciler core/localqueue_controller.go:144 Got Workload event {"workload": {"name":"wl-high-priority","namespace":"core-4g2kt"}} 2022-11-25T13:31:05.600770595Z LEVEL(-3) localqueue-reconciler core/localqueue_controller.go:144 Got Workload event {"workload": {"name":"wl-high-priority","namespace":"core-4g2kt"}} 2022-11-25T13:31:05.600802544Z LEVEL(-2) cluster-queue-reconciler core/clusterqueue_controller.go:201 Got Workload event {"workload": {"name":"wl-high-priority","namespace":"core-4g2kt"}} 2022-11-25T13:31:05.600823117Z LEVEL(-2) cluster-queue-reconciler core/clusterqueue_controller.go:201 Got Workload event {"workload": {"name":"wl-high-priority","namespace":"core-4g2kt"}} 2022-11-25T13:31:05.603860227Z ERROR scheduler scheduler/scheduler.go:570 Could not update Workload status {"error": "Operation cannot be fulfilled on workloads.kueue.x-k8s.io \"wl-high-priority\": the object has been modified; please apply your changes to the latest version and try again"} sigs.k8s.io/kueue/pkg/scheduler.(*Scheduler).requeueAndUpdate /home/prow/go/src/sigs.k8s.io/kueue/pkg/scheduler/scheduler.go:570 sigs.k8s.io/kueue/pkg/scheduler.(*Scheduler).schedule /home/prow/go/src/sigs.k8s.io/kueue/pkg/scheduler/scheduler.go:148 k8s.io/apimachinery/pkg/util/wait.JitterUntilWithContext.func1 /home/prow/go/pkg/mod/k8s.io/apimachinery@v0.24.3/pkg/util/wait/wait.go:188 ... skipping 176 lines ... 2022-11-25T13:31:05.600492144Z LEVEL(-2) workload-reconciler core/workload_controller.go:198 Workload update event {"workload": {"name":"wl-high-priority","namespace":"core-4g2kt"}, "queue": "queue", "status": "admitted", "prevStatus": "pending", "clusterQueue": "prod-cq"} 2022-11-25T13:31:05.600682212Z LEVEL(-2) core/workload_controller.go:82 Reconciling Workload {"controller": "workload", "controllerGroup": "kueue.x-k8s.io", "controllerKind": "Workload", "workload": {"name":"wl-high-priority","namespace":"core-4g2kt"}, "namespace": "core-4g2kt", "name": "wl-high-priority", "reconcileID": "1493c4d4-7c1c-4dda-94ce-129af7b41953", "workload": {"name":"wl-high-priority","namespace":"core-4g2kt"}} 2022-11-25T13:31:05.600729238Z LEVEL(-3) localqueue-reconciler core/localqueue_controller.go:144 Got Workload event {"workload": {"name":"wl-high-priority","namespace":"core-4g2kt"}} 2022-11-25T13:31:05.600770595Z LEVEL(-3) localqueue-reconciler core/localqueue_controller.go:144 Got Workload event {"workload": {"name":"wl-high-priority","namespace":"core-4g2kt"}} 2022-11-25T13:31:05.600802544Z LEVEL(-2) cluster-queue-reconciler core/clusterqueue_controller.go:201 Got Workload event {"workload": {"name":"wl-high-priority","namespace":"core-4g2kt"}} 2022-11-25T13:31:05.600823117Z LEVEL(-2) cluster-queue-reconciler core/clusterqueue_controller.go:201 Got Workload event {"workload": {"name":"wl-high-priority","namespace":"core-4g2kt"}} 2022-11-25T13:31:05.603860227Z ERROR scheduler scheduler/scheduler.go:570 Could not update Workload status {"error": "Operation cannot be fulfilled on workloads.kueue.x-k8s.io \"wl-high-priority\": the object has been modified; please apply your changes to the latest version and try again"} sigs.k8s.io/kueue/pkg/scheduler.(*Scheduler).requeueAndUpdate /home/prow/go/src/sigs.k8s.io/kueue/pkg/scheduler/scheduler.go:570 sigs.k8s.io/kueue/pkg/scheduler.(*Scheduler).schedule /home/prow/go/src/sigs.k8s.io/kueue/pkg/scheduler/scheduler.go:148 k8s.io/apimachinery/pkg/util/wait.JitterUntilWithContext.func1 /home/prow/go/pkg/mod/k8s.io/apimachinery@v0.24.3/pkg/util/wait/wait.go:188 ... skipping 1312 lines ... 2022-11-25T13:31:12.783136347Z LEVEL(-2) core/clusterqueue_controller.go:81 Reconciling ClusterQueue {"controller": "clusterqueue", "controllerGroup": "kueue.x-k8s.io", "controllerKind": "ClusterQueue", "clusterQueue": {"name":"cluster-queue"}, "namespace": "", "name": "cluster-queue", "reconcileID": "972d0141-5013-4832-9c64-04c987fa1c75", "clusterQueue": {"name":"cluster-queue"}} 2022-11-25T13:31:12.78615213Z DEBUG controller-runtime.webhook.webhooks admission/http.go:96 received request {"webhook": "/validate-kueue-x-k8s-io-v1alpha2-clusterqueue", "UID": "87df45dd-0b42-4eb3-910c-1302bd7bc81c", "kind": "kueue.x-k8s.io/v1alpha2, Kind=ClusterQueue", "resource": {"group":"kueue.x-k8s.io","version":"v1alpha2","resource":"clusterqueues"}} 2022-11-25T13:31:12.786630697Z DEBUG controller-runtime.webhook.webhooks admission/http.go:143 wrote response {"webhook": "/validate-kueue-x-k8s-io-v1alpha2-clusterqueue", "code": 200, "reason": "", "UID": "87df45dd-0b42-4eb3-910c-1302bd7bc81c", "allowed": true} 2022-11-25T13:31:12.790284389Z LEVEL(-2) cluster-queue-reconciler core/clusterqueue_controller.go:165 ClusterQueue delete event {"clusterQueue": {"name":"cluster-queue"}} 2022-11-25T13:31:12.79036756Z LEVEL(-2) resourceflavor-reconciler core/resourceflavor_controller.go:156 Got generic event {"obj": {"name":"cluster-queue"}, "kind": "/, Kind="} 2022-11-25T13:31:12.790443179Z LEVEL(-2) core/resourceflavor_controller.go:69 Reconciling ResourceFlavor {"controller": "resourceflavor", "controllerGroup": "kueue.x-k8s.io", "controllerKind": "ResourceFlavor", "resourceFlavor": {"name":"on-demand"}, "namespace": "", "name": "on-demand", "reconcileID": "b9187cc4-174e-4b02-bca3-306a7be04159", "resourceFlavor": {"name":"on-demand"}} {"component":"entrypoint","file":"k8s.io/test-infra/prow/entrypoint/run.go:168","func":"k8s.io/test-infra/prow/entrypoint.Options.ExecuteProcess","level":"error","msg":"Entrypoint received interrupt: terminated","severity":"error","time":"2022-11-25T13:31:12Z"}