This job view page is being replaced by Spyglass soon. Check out the new job view.
PRmimowo: Design for all-or-nothing semantics for workload resource assignment
ResultABORTED
Tests 0 failed / 36 succeeded
Started2022-11-25 12:57
Elapsed2m53s
Revisiona10513f2ffc835e73a0a7b62de6bbac997e2c687
Refs 433

No Test Failures!


Show 36 Passed Tests

Error lines from build-log.txt

... skipping 645 lines ...
2022-11-25T12:59:14.293499476Z	LEVEL(-2)	core/resourceflavor_controller.go:69	Reconciling ResourceFlavor	{"controller": "resourceflavor", "controllerGroup": "kueue.x-k8s.io", "controllerKind": "ResourceFlavor", "resourceFlavor": {"name":"on-demand"}, "namespace": "", "name": "on-demand", "reconcileID": "012233db-dd57-4634-80d9-d8d7ab6dfb5a", "resourceFlavor": {"name":"on-demand"}}
2022-11-25T12:59:14.295612122Z	DEBUG	controller-runtime.webhook.webhooks	admission/http.go:96	received request	{"webhook": "/validate-kueue-x-k8s-io-v1alpha2-resourceflavor", "UID": "57987954-f210-435b-876c-2f800878c7f0", "kind": "kueue.x-k8s.io/v1alpha2, Kind=ResourceFlavor", "resource": {"group":"kueue.x-k8s.io","version":"v1alpha2","resource":"resourceflavors"}}
2022-11-25T12:59:14.295755604Z	DEBUG	controller-runtime.webhook.webhooks	admission/http.go:143	wrote response	{"webhook": "/validate-kueue-x-k8s-io-v1alpha2-resourceflavor", "code": 200, "reason": "", "UID": "57987954-f210-435b-876c-2f800878c7f0", "allowed": true}
2022-11-25T12:59:14.297785858Z	DEBUG	controller-runtime.webhook.webhooks	admission/http.go:96	received request	{"webhook": "/mutate-kueue-x-k8s-io-v1alpha2-clusterqueue", "UID": "fc204faf-bfeb-433f-8a38-a7e2e33edea9", "kind": "kueue.x-k8s.io/v1alpha2, Kind=ClusterQueue", "resource": {"group":"kueue.x-k8s.io","version":"v1alpha2","resource":"clusterqueues"}}
2022-11-25T12:59:14.298552515Z	DEBUG	controller-runtime.webhook.webhooks	admission/http.go:143	wrote response	{"webhook": "/mutate-kueue-x-k8s-io-v1alpha2-clusterqueue", "code": 200, "reason": "", "UID": "fc204faf-bfeb-433f-8a38-a7e2e33edea9", "allowed": true}
2022-11-25T12:59:14.29980665Z	LEVEL(-2)	resourceflavor-reconciler	core/resourceflavor_controller.go:128	ResourceFlavor delete event	{"resourceFlavor": {"name":"on-demand"}}
2022/11/25 12:59:14 http: TLS handshake error from 127.0.0.1:39086: EOF
2022-11-25T12:59:14.302411085Z	DEBUG	controller-runtime.webhook.webhooks	admission/http.go:96	received request	{"webhook": "/validate-kueue-x-k8s-io-v1alpha2-clusterqueue", "UID": "363f9db4-c08b-48d8-8c13-7f4fbfb6775d", "kind": "kueue.x-k8s.io/v1alpha2, Kind=ClusterQueue", "resource": {"group":"kueue.x-k8s.io","version":"v1alpha2","resource":"clusterqueues"}}
2022-11-25T12:59:14.302773673Z	DEBUG	controller-runtime.webhook.webhooks	admission/http.go:143	wrote response	{"webhook": "/validate-kueue-x-k8s-io-v1alpha2-clusterqueue", "code": 200, "reason": "", "UID": "363f9db4-c08b-48d8-8c13-7f4fbfb6775d", "allowed": true}
2022-11-25T12:59:14.305138617Z	LEVEL(-2)	cluster-queue-reconciler	core/clusterqueue_controller.go:145	ClusterQueue create event	{"clusterQueue": {"name":"clusterqueue"}}
2022-11-25T12:59:14.305349815Z	LEVEL(-2)	core/clusterqueue_controller.go:81	Reconciling ClusterQueue	{"controller": "clusterqueue", "controllerGroup": "kueue.x-k8s.io", "controllerKind": "ClusterQueue", "clusterQueue": {"name":"clusterqueue"}, "namespace": "", "name": "clusterqueue", "reconcileID": "0e1be036-3b3c-4328-973d-95b1774401d2", "clusterQueue": {"name":"clusterqueue"}}
2022-11-25T12:59:14.306964092Z	DEBUG	controller-runtime.webhook.webhooks	admission/http.go:96	received request	{"webhook": "/validate-kueue-x-k8s-io-v1alpha2-localqueue", "UID": "d7b80612-4830-4b59-8781-8b6c26763609", "kind": "kueue.x-k8s.io/v1alpha2, Kind=LocalQueue", "resource": {"group":"kueue.x-k8s.io","version":"v1alpha2","resource":"localqueues"}}
2022-11-25T12:59:14.307137185Z	DEBUG	controller-runtime.webhook.webhooks	admission/http.go:143	wrote response	{"webhook": "/validate-kueue-x-k8s-io-v1alpha2-localqueue", "code": 200, "reason": "", "UID": "d7b80612-4830-4b59-8781-8b6c26763609", "allowed": true}
... skipping 115 lines ...
2022-11-25T12:59:15.901785317Z	LEVEL(-2)	core/localqueue_controller.go:75	Reconciling LocalQueue	{"controller": "localqueue", "controllerGroup": "kueue.x-k8s.io", "controllerKind": "LocalQueue", "localQueue": {"name":"queue","namespace":"core-workload-6sqkv"}, "namespace": "core-workload-6sqkv", "name": "queue", "reconcileID": "0fe717a9-2b21-4ef8-9fb7-eb9a20404e42", "localQueue": {"name":"queue","namespace":"core-workload-6sqkv"}}
2022-11-25T12:59:15.90422918Z	DEBUG	controller-runtime.webhook.webhooks	admission/http.go:96	received request	{"webhook": "/mutate-kueue-x-k8s-io-v1alpha2-workload", "UID": "5a12baae-5e9e-44d7-bcc8-5e29a790fc0d", "kind": "kueue.x-k8s.io/v1alpha2, Kind=Workload", "resource": {"group":"kueue.x-k8s.io","version":"v1alpha2","resource":"workloads"}}
2022-11-25T12:59:15.904627902Z	DEBUG	controller-runtime.webhook.webhooks	admission/http.go:143	wrote response	{"webhook": "/mutate-kueue-x-k8s-io-v1alpha2-workload", "code": 200, "reason": "", "UID": "5a12baae-5e9e-44d7-bcc8-5e29a790fc0d", "allowed": true}
2022-11-25T12:59:15.906359856Z	DEBUG	controller-runtime.webhook.webhooks	admission/http.go:96	received request	{"webhook": "/validate-kueue-x-k8s-io-v1alpha2-workload", "UID": "4eaf69ea-90c4-4a45-8987-67c71b6164aa", "kind": "kueue.x-k8s.io/v1alpha2, Kind=Workload", "resource": {"group":"kueue.x-k8s.io","version":"v1alpha2","resource":"workloads"}}
2022-11-25T12:59:15.906552057Z	DEBUG	controller-runtime.webhook.webhooks	admission/http.go:143	wrote response	{"webhook": "/validate-kueue-x-k8s-io-v1alpha2-workload", "code": 200, "reason": "", "UID": "4eaf69ea-90c4-4a45-8987-67c71b6164aa", "allowed": true}
2022-11-25T12:59:15.908852047Z	LEVEL(-2)	workload-reconciler	core/workload_controller.go:119	Workload create event	{"workload": {"name":"one","namespace":"core-workload-6sqkv"}, "queue": "queue", "status": "admitted"}
2022-11-25T12:59:15.908924064Z	ERROR	workload-reconciler	core/workload_controller.go:287	Could not get RuntimeClass	{"error": "RuntimeClass.node.k8s.io \"kata\" not found"}
sigs.k8s.io/kueue/pkg/controller/core.handlePodOverhead
	/home/prow/go/src/sigs.k8s.io/kueue/pkg/controller/core/workload_controller.go:287
sigs.k8s.io/kueue/pkg/controller/core.(*WorkloadReconciler).Create
	/home/prow/go/src/sigs.k8s.io/kueue/pkg/controller/core/workload_controller.go:126
sigs.k8s.io/controller-runtime/pkg/source/internal.EventHandler.OnAdd
	/home/prow/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.12.3/pkg/source/internal/eventsource.go:57
... skipping 13 lines ...
	/home/prow/go/pkg/mod/k8s.io/apimachinery@v0.24.3/pkg/util/wait/wait.go:73
2022-11-25T12:59:15.909030255Z	LEVEL(-2)	core/workload_controller.go:82	Reconciling Workload	{"controller": "workload", "controllerGroup": "kueue.x-k8s.io", "controllerKind": "Workload", "workload": {"name":"one","namespace":"core-workload-6sqkv"}, "namespace": "core-workload-6sqkv", "name": "one", "reconcileID": "f766024e-236c-47e0-a67d-7abdbee6ec7f", "workload": {"name":"one","namespace":"core-workload-6sqkv"}}
2022-11-25T12:59:15.9092792Z	LEVEL(-3)	localqueue-reconciler	core/localqueue_controller.go:144	Got Workload event	{"workload": {"name":"one","namespace":"core-workload-6sqkv"}}
2022-11-25T12:59:15.909317012Z	LEVEL(-2)	cluster-queue-reconciler	core/clusterqueue_controller.go:201	Got Workload event	{"workload": {"name":"one","namespace":"core-workload-6sqkv"}}
STEP: Got ClusterQueueStatus 11/25/22 12:59:15.909
2022-11-25T12:59:15.915742842Z	LEVEL(-2)	workload-reconciler	core/workload_controller.go:198	Workload update event	{"workload": {"name":"one","namespace":"core-workload-6sqkv"}, "queue": "queue", "status": "admitted", "clusterQueue": "clusterqueue"}
2022-11-25T12:59:15.915817089Z	ERROR	workload-reconciler	core/workload_controller.go:287	Could not get RuntimeClass	{"error": "RuntimeClass.node.k8s.io \"kata\" not found"}
sigs.k8s.io/kueue/pkg/controller/core.handlePodOverhead
	/home/prow/go/src/sigs.k8s.io/kueue/pkg/controller/core/workload_controller.go:287
sigs.k8s.io/kueue/pkg/controller/core.(*WorkloadReconciler).Update
	/home/prow/go/src/sigs.k8s.io/kueue/pkg/controller/core/workload_controller.go:202
sigs.k8s.io/controller-runtime/pkg/source/internal.EventHandler.OnUpdate
	/home/prow/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.12.3/pkg/source/internal/eventsource.go:88
... skipping 54 lines ...
    2022-11-25T12:59:15.901785317Z	LEVEL(-2)	core/localqueue_controller.go:75	Reconciling LocalQueue	{"controller": "localqueue", "controllerGroup": "kueue.x-k8s.io", "controllerKind": "LocalQueue", "localQueue": {"name":"queue","namespace":"core-workload-6sqkv"}, "namespace": "core-workload-6sqkv", "name": "queue", "reconcileID": "0fe717a9-2b21-4ef8-9fb7-eb9a20404e42", "localQueue": {"name":"queue","namespace":"core-workload-6sqkv"}}
    2022-11-25T12:59:15.90422918Z	DEBUG	controller-runtime.webhook.webhooks	admission/http.go:96	received request	{"webhook": "/mutate-kueue-x-k8s-io-v1alpha2-workload", "UID": "5a12baae-5e9e-44d7-bcc8-5e29a790fc0d", "kind": "kueue.x-k8s.io/v1alpha2, Kind=Workload", "resource": {"group":"kueue.x-k8s.io","version":"v1alpha2","resource":"workloads"}}
    2022-11-25T12:59:15.904627902Z	DEBUG	controller-runtime.webhook.webhooks	admission/http.go:143	wrote response	{"webhook": "/mutate-kueue-x-k8s-io-v1alpha2-workload", "code": 200, "reason": "", "UID": "5a12baae-5e9e-44d7-bcc8-5e29a790fc0d", "allowed": true}
    2022-11-25T12:59:15.906359856Z	DEBUG	controller-runtime.webhook.webhooks	admission/http.go:96	received request	{"webhook": "/validate-kueue-x-k8s-io-v1alpha2-workload", "UID": "4eaf69ea-90c4-4a45-8987-67c71b6164aa", "kind": "kueue.x-k8s.io/v1alpha2, Kind=Workload", "resource": {"group":"kueue.x-k8s.io","version":"v1alpha2","resource":"workloads"}}
    2022-11-25T12:59:15.906552057Z	DEBUG	controller-runtime.webhook.webhooks	admission/http.go:143	wrote response	{"webhook": "/validate-kueue-x-k8s-io-v1alpha2-workload", "code": 200, "reason": "", "UID": "4eaf69ea-90c4-4a45-8987-67c71b6164aa", "allowed": true}
    2022-11-25T12:59:15.908852047Z	LEVEL(-2)	workload-reconciler	core/workload_controller.go:119	Workload create event	{"workload": {"name":"one","namespace":"core-workload-6sqkv"}, "queue": "queue", "status": "admitted"}
    2022-11-25T12:59:15.908924064Z	ERROR	workload-reconciler	core/workload_controller.go:287	Could not get RuntimeClass	{"error": "RuntimeClass.node.k8s.io \"kata\" not found"}
    sigs.k8s.io/kueue/pkg/controller/core.handlePodOverhead
    	/home/prow/go/src/sigs.k8s.io/kueue/pkg/controller/core/workload_controller.go:287
    sigs.k8s.io/kueue/pkg/controller/core.(*WorkloadReconciler).Create
    	/home/prow/go/src/sigs.k8s.io/kueue/pkg/controller/core/workload_controller.go:126
    sigs.k8s.io/controller-runtime/pkg/source/internal.EventHandler.OnAdd
    	/home/prow/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.12.3/pkg/source/internal/eventsource.go:57
... skipping 13 lines ...
    	/home/prow/go/pkg/mod/k8s.io/apimachinery@v0.24.3/pkg/util/wait/wait.go:73
    2022-11-25T12:59:15.909030255Z	LEVEL(-2)	core/workload_controller.go:82	Reconciling Workload	{"controller": "workload", "controllerGroup": "kueue.x-k8s.io", "controllerKind": "Workload", "workload": {"name":"one","namespace":"core-workload-6sqkv"}, "namespace": "core-workload-6sqkv", "name": "one", "reconcileID": "f766024e-236c-47e0-a67d-7abdbee6ec7f", "workload": {"name":"one","namespace":"core-workload-6sqkv"}}
    2022-11-25T12:59:15.9092792Z	LEVEL(-3)	localqueue-reconciler	core/localqueue_controller.go:144	Got Workload event	{"workload": {"name":"one","namespace":"core-workload-6sqkv"}}
    2022-11-25T12:59:15.909317012Z	LEVEL(-2)	cluster-queue-reconciler	core/clusterqueue_controller.go:201	Got Workload event	{"workload": {"name":"one","namespace":"core-workload-6sqkv"}}
    STEP: Got ClusterQueueStatus 11/25/22 12:59:15.909
    2022-11-25T12:59:15.915742842Z	LEVEL(-2)	workload-reconciler	core/workload_controller.go:198	Workload update event	{"workload": {"name":"one","namespace":"core-workload-6sqkv"}, "queue": "queue", "status": "admitted", "clusterQueue": "clusterqueue"}
    2022-11-25T12:59:15.915817089Z	ERROR	workload-reconciler	core/workload_controller.go:287	Could not get RuntimeClass	{"error": "RuntimeClass.node.k8s.io \"kata\" not found"}
    sigs.k8s.io/kueue/pkg/controller/core.handlePodOverhead
    	/home/prow/go/src/sigs.k8s.io/kueue/pkg/controller/core/workload_controller.go:287
    sigs.k8s.io/kueue/pkg/controller/core.(*WorkloadReconciler).Update
    	/home/prow/go/src/sigs.k8s.io/kueue/pkg/controller/core/workload_controller.go:202
    sigs.k8s.io/controller-runtime/pkg/source/internal.EventHandler.OnUpdate
    	/home/prow/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.12.3/pkg/source/internal/eventsource.go:88
... skipping 215 lines ...
2022-11-25T12:59:18.108273623Z	LEVEL(-2)	cluster-queue-reconciler	core/clusterqueue_controller.go:201	Got Workload event	{"workload": {"name":"five","namespace":"core-clusterqueue-sx74m"}}
2022-11-25T12:59:18.108375206Z	LEVEL(-2)	cluster-queue-reconciler	core/clusterqueue_controller.go:201	Got Workload event	{"workload": {"name":"five","namespace":"core-clusterqueue-sx74m"}}
2022-11-25T12:59:18.108265238Z	LEVEL(-3)	localqueue-reconciler	core/localqueue_controller.go:144	Got Workload event	{"workload": {"name":"five","namespace":"core-clusterqueue-sx74m"}}
2022-11-25T12:59:18.113656435Z	DEBUG	controller-runtime.webhook.webhooks	admission/http.go:96	received request	{"webhook": "/mutate-kueue-x-k8s-io-v1alpha2-workload", "UID": "db46e8a3-d821-4e77-aa2d-4730ca322622", "kind": "kueue.x-k8s.io/v1alpha2, Kind=Workload", "resource": {"group":"kueue.x-k8s.io","version":"v1alpha2","resource":"workloads"}}
2022-11-25T12:59:18.114246194Z	DEBUG	controller-runtime.webhook.webhooks	admission/http.go:143	wrote response	{"webhook": "/mutate-kueue-x-k8s-io-v1alpha2-workload", "code": 200, "reason": "", "UID": "db46e8a3-d821-4e77-aa2d-4730ca322622", "allowed": true}
2022-11-25T12:59:18.114717129Z	LEVEL(-2)	workload-reconciler	core/workload_controller.go:198	Workload update event	{"workload": {"name":"five","namespace":"core-clusterqueue-sx74m"}, "queue": "other", "status": "admitted", "clusterQueue": "other"}
2022-11-25T12:59:18.114807103Z	ERROR	workload-reconciler	core/workload_controller.go:240	Updating workload in cache	{"workload": {"name":"five","namespace":"core-clusterqueue-sx74m"}, "queue": "other", "status": "admitted", "clusterQueue": "other", "error": "old ClusterQueue doesn't exist"}
sigs.k8s.io/kueue/pkg/controller/core.(*WorkloadReconciler).Update
	/home/prow/go/src/sigs.k8s.io/kueue/pkg/controller/core/workload_controller.go:240
sigs.k8s.io/controller-runtime/pkg/source/internal.EventHandler.OnUpdate
	/home/prow/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.12.3/pkg/source/internal/eventsource.go:88
k8s.io/client-go/tools/cache.(*processorListener).run.func1
	/home/prow/go/pkg/mod/k8s.io/client-go@v0.24.3/tools/cache/shared_informer.go:816
... skipping 45 lines ...
2022-11-25T12:59:19.173909212Z	LEVEL(-3)	localqueue-reconciler	core/localqueue_controller.go:144	Got Workload event	{"workload": {"name":"four","namespace":"core-clusterqueue-sx74m"}}
2022-11-25T12:59:19.173953472Z	LEVEL(-2)	cluster-queue-reconciler	core/clusterqueue_controller.go:201	Got Workload event	{"workload": {"name":"four","namespace":"core-clusterqueue-sx74m"}}
2022-11-25T12:59:19.173974778Z	LEVEL(-2)	cluster-queue-reconciler	core/clusterqueue_controller.go:201	Got Workload event	{"workload": {"name":"four","namespace":"core-clusterqueue-sx74m"}}
2022-11-25T12:59:19.173988209Z	LEVEL(-3)	localqueue-reconciler	core/localqueue_controller.go:144	Got Workload event	{"workload": {"name":"four","namespace":"core-clusterqueue-sx74m"}}
2022-11-25T12:59:19.17393518Z	LEVEL(-2)	core/workload_controller.go:82	Reconciling Workload	{"controller": "workload", "controllerGroup": "kueue.x-k8s.io", "controllerKind": "Workload", "workload": {"name":"four","namespace":"core-clusterqueue-sx74m"}, "namespace": "core-clusterqueue-sx74m", "name": "four", "reconcileID": "260eecb1-fc1e-43e1-9487-1811b15ec54c", "workload": {"name":"four","namespace":"core-clusterqueue-sx74m"}}
2022-11-25T12:59:19.182580734Z	LEVEL(-2)	workload-reconciler	core/workload_controller.go:198	Workload update event	{"workload": {"name":"five","namespace":"core-clusterqueue-sx74m"}, "queue": "other", "status": "finished", "prevStatus": "admitted", "clusterQueue": "other"}
2022-11-25T12:59:19.182653129Z	ERROR	workload-reconciler	core/workload_controller.go:207	Failed to delete workload from cache	{"workload": {"name":"five","namespace":"core-clusterqueue-sx74m"}, "queue": "other", "status": "finished", "prevStatus": "admitted", "clusterQueue": "other", "error": "cluster queue not found"}
sigs.k8s.io/kueue/pkg/controller/core.(*WorkloadReconciler).Update
	/home/prow/go/src/sigs.k8s.io/kueue/pkg/controller/core/workload_controller.go:207
sigs.k8s.io/controller-runtime/pkg/source/internal.EventHandler.OnUpdate
	/home/prow/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.12.3/pkg/source/internal/eventsource.go:88
k8s.io/client-go/tools/cache.(*processorListener).run.func1
	/home/prow/go/pkg/mod/k8s.io/client-go@v0.24.3/tools/cache/shared_informer.go:816
... skipping 31 lines ...
2022-11-25T12:59:20.221005304Z	DEBUG	controller-runtime.webhook.webhooks	admission/http.go:96	received request	{"webhook": "/validate-kueue-x-k8s-io-v1alpha2-clusterqueue", "UID": "b4ff3ba6-52b9-4d68-ab03-db126c6d3ce5", "kind": "kueue.x-k8s.io/v1alpha2, Kind=ClusterQueue", "resource": {"group":"kueue.x-k8s.io","version":"v1alpha2","resource":"clusterqueues"}}
2022-11-25T12:59:20.221101744Z	LEVEL(-2)	localqueue-reconciler	core/localqueue_controller.go:119	LocalQueue delete event	{"localQueue": {"name":"queue","namespace":"core-clusterqueue-sx74m"}}
2022-11-25T12:59:20.221511159Z	DEBUG	controller-runtime.webhook.webhooks	admission/http.go:143	wrote response	{"webhook": "/validate-kueue-x-k8s-io-v1alpha2-clusterqueue", "code": 200, "reason": "", "UID": "b4ff3ba6-52b9-4d68-ab03-db126c6d3ce5", "allowed": true}
2022-11-25T12:59:20.225868629Z	LEVEL(-2)	cluster-queue-reconciler	core/clusterqueue_controller.go:165	ClusterQueue delete event	{"clusterQueue": {"name":"cluster-queue"}}
2022-11-25T12:59:20.226138366Z	LEVEL(-2)	resourceflavor-reconciler	core/resourceflavor_controller.go:156	Got generic event	{"obj": {"name":"cluster-queue"}, "kind": "/, Kind="}
2022-11-25T12:59:20.228516024Z	LEVEL(-2)	workload-reconciler	core/workload_controller.go:149	Workload delete event	{"workload": {"name":"five","namespace":"core-clusterqueue-sx74m"}, "queue": "other", "status": "finished"}
2022-11-25T12:59:20.22857333Z	ERROR	workload-reconciler	core/workload_controller.go:158	Failed to delete workload from cache	{"workload": {"name":"five","namespace":"core-clusterqueue-sx74m"}, "queue": "other", "status": "finished", "error": "cluster queue not found"}
sigs.k8s.io/kueue/pkg/controller/core.(*WorkloadReconciler).Delete
	/home/prow/go/src/sigs.k8s.io/kueue/pkg/controller/core/workload_controller.go:158
sigs.k8s.io/controller-runtime/pkg/source/internal.EventHandler.OnDelete
	/home/prow/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.12.3/pkg/source/internal/eventsource.go:131
k8s.io/client-go/tools/cache.(*processorListener).run.func1
	/home/prow/go/pkg/mod/k8s.io/client-go@v0.24.3/tools/cache/shared_informer.go:820
... skipping 9 lines ...
	/home/prow/go/pkg/mod/k8s.io/client-go@v0.24.3/tools/cache/shared_informer.go:812
k8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1
	/home/prow/go/pkg/mod/k8s.io/apimachinery@v0.24.3/pkg/util/wait/wait.go:73
2022-11-25T12:59:20.228685784Z	LEVEL(-3)	localqueue-reconciler	core/localqueue_controller.go:144	Got Workload event	{"workload": {"name":"five","namespace":"core-clusterqueue-sx74m"}}
2022-11-25T12:59:20.22871646Z	LEVEL(-2)	cluster-queue-reconciler	core/clusterqueue_controller.go:201	Got Workload event	{"workload": {"name":"five","namespace":"core-clusterqueue-sx74m"}}
2022-11-25T12:59:20.230177607Z	LEVEL(-2)	workload-reconciler	core/workload_controller.go:149	Workload delete event	{"workload": {"name":"four","namespace":"core-clusterqueue-sx74m"}, "queue": "queue", "status": "finished"}
2022-11-25T12:59:20.230220697Z	ERROR	workload-reconciler	core/workload_controller.go:158	Failed to delete workload from cache	{"workload": {"name":"four","namespace":"core-clusterqueue-sx74m"}, "queue": "queue", "status": "finished", "error": "cluster queue not found"}
sigs.k8s.io/kueue/pkg/controller/core.(*WorkloadReconciler).Delete
	/home/prow/go/src/sigs.k8s.io/kueue/pkg/controller/core/workload_controller.go:158
sigs.k8s.io/controller-runtime/pkg/source/internal.EventHandler.OnDelete
	/home/prow/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.12.3/pkg/source/internal/eventsource.go:131
k8s.io/client-go/tools/cache.(*processorListener).run.func1
	/home/prow/go/pkg/mod/k8s.io/client-go@v0.24.3/tools/cache/shared_informer.go:820
... skipping 9 lines ...
	/home/prow/go/pkg/mod/k8s.io/client-go@v0.24.3/tools/cache/shared_informer.go:812
k8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1
	/home/prow/go/pkg/mod/k8s.io/apimachinery@v0.24.3/pkg/util/wait/wait.go:73
2022-11-25T12:59:20.230282901Z	LEVEL(-3)	localqueue-reconciler	core/localqueue_controller.go:144	Got Workload event	{"workload": {"name":"four","namespace":"core-clusterqueue-sx74m"}}
2022-11-25T12:59:20.230323558Z	LEVEL(-2)	cluster-queue-reconciler	core/clusterqueue_controller.go:201	Got Workload event	{"workload": {"name":"four","namespace":"core-clusterqueue-sx74m"}}
2022-11-25T12:59:20.232766354Z	LEVEL(-2)	workload-reconciler	core/workload_controller.go:149	Workload delete event	{"workload": {"name":"one","namespace":"core-clusterqueue-sx74m"}, "queue": "queue", "status": "finished"}
2022-11-25T12:59:20.232813635Z	ERROR	workload-reconciler	core/workload_controller.go:158	Failed to delete workload from cache	{"workload": {"name":"one","namespace":"core-clusterqueue-sx74m"}, "queue": "queue", "status": "finished", "error": "cluster queue not found"}
sigs.k8s.io/kueue/pkg/controller/core.(*WorkloadReconciler).Delete
	/home/prow/go/src/sigs.k8s.io/kueue/pkg/controller/core/workload_controller.go:158
sigs.k8s.io/controller-runtime/pkg/source/internal.EventHandler.OnDelete
	/home/prow/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.12.3/pkg/source/internal/eventsource.go:131
k8s.io/client-go/tools/cache.(*processorListener).run.func1
	/home/prow/go/pkg/mod/k8s.io/client-go@v0.24.3/tools/cache/shared_informer.go:820
... skipping 12 lines ...
2022-11-25T12:59:20.232981246Z	LEVEL(-3)	localqueue-reconciler	core/localqueue_controller.go:144	Got Workload event	{"workload": {"name":"one","namespace":"core-clusterqueue-sx74m"}}
2022-11-25T12:59:20.23304536Z	LEVEL(-2)	cluster-queue-reconciler	core/clusterqueue_controller.go:201	Got Workload event	{"workload": {"name":"one","namespace":"core-clusterqueue-sx74m"}}
2022-11-25T12:59:20.23467726Z	LEVEL(-2)	workload-reconciler	core/workload_controller.go:149	Workload delete event	{"workload": {"name":"six","namespace":"core-clusterqueue-sx74m"}, "queue": "queue", "status": "finished"}
2022-11-25T12:59:20.234770511Z	LEVEL(-2)	cluster-queue-reconciler	core/clusterqueue_controller.go:201	Got Workload event	{"workload": {"name":"six","namespace":"core-clusterqueue-sx74m"}}
2022-11-25T12:59:20.234752441Z	LEVEL(-3)	localqueue-reconciler	core/localqueue_controller.go:144	Got Workload event	{"workload": {"name":"six","namespace":"core-clusterqueue-sx74m"}}
2022-11-25T12:59:20.237022952Z	LEVEL(-2)	workload-reconciler	core/workload_controller.go:149	Workload delete event	{"workload": {"name":"three","namespace":"core-clusterqueue-sx74m"}, "queue": "queue", "status": "finished"}
2022-11-25T12:59:20.237064311Z	ERROR	workload-reconciler	core/workload_controller.go:158	Failed to delete workload from cache	{"workload": {"name":"three","namespace":"core-clusterqueue-sx74m"}, "queue": "queue", "status": "finished", "error": "cluster queue not found"}
sigs.k8s.io/kueue/pkg/controller/core.(*WorkloadReconciler).Delete
	/home/prow/go/src/sigs.k8s.io/kueue/pkg/controller/core/workload_controller.go:158
sigs.k8s.io/controller-runtime/pkg/source/internal.EventHandler.OnDelete
	/home/prow/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.12.3/pkg/source/internal/eventsource.go:131
k8s.io/client-go/tools/cache.(*processorListener).run.func1
	/home/prow/go/pkg/mod/k8s.io/client-go@v0.24.3/tools/cache/shared_informer.go:820
... skipping 9 lines ...
	/home/prow/go/pkg/mod/k8s.io/client-go@v0.24.3/tools/cache/shared_informer.go:812
k8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1
	/home/prow/go/pkg/mod/k8s.io/apimachinery@v0.24.3/pkg/util/wait/wait.go:73
2022-11-25T12:59:20.237183872Z	LEVEL(-3)	localqueue-reconciler	core/localqueue_controller.go:144	Got Workload event	{"workload": {"name":"three","namespace":"core-clusterqueue-sx74m"}}
2022-11-25T12:59:20.237233498Z	LEVEL(-2)	cluster-queue-reconciler	core/clusterqueue_controller.go:201	Got Workload event	{"workload": {"name":"three","namespace":"core-clusterqueue-sx74m"}}
2022-11-25T12:59:20.238644289Z	LEVEL(-2)	workload-reconciler	core/workload_controller.go:149	Workload delete event	{"workload": {"name":"two","namespace":"core-clusterqueue-sx74m"}, "queue": "queue", "status": "finished"}
2022-11-25T12:59:20.238681595Z	ERROR	workload-reconciler	core/workload_controller.go:158	Failed to delete workload from cache	{"workload": {"name":"two","namespace":"core-clusterqueue-sx74m"}, "queue": "queue", "status": "finished", "error": "cluster queue not found"}
sigs.k8s.io/kueue/pkg/controller/core.(*WorkloadReconciler).Delete
	/home/prow/go/src/sigs.k8s.io/kueue/pkg/controller/core/workload_controller.go:158
sigs.k8s.io/controller-runtime/pkg/source/internal.EventHandler.OnDelete
	/home/prow/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.12.3/pkg/source/internal/eventsource.go:131
k8s.io/client-go/tools/cache.(*processorListener).run.func1
	/home/prow/go/pkg/mod/k8s.io/client-go@v0.24.3/tools/cache/shared_informer.go:820
... skipping 202 lines ...
    2022-11-25T12:59:18.108273623Z	LEVEL(-2)	cluster-queue-reconciler	core/clusterqueue_controller.go:201	Got Workload event	{"workload": {"name":"five","namespace":"core-clusterqueue-sx74m"}}
    2022-11-25T12:59:18.108375206Z	LEVEL(-2)	cluster-queue-reconciler	core/clusterqueue_controller.go:201	Got Workload event	{"workload": {"name":"five","namespace":"core-clusterqueue-sx74m"}}
    2022-11-25T12:59:18.108265238Z	LEVEL(-3)	localqueue-reconciler	core/localqueue_controller.go:144	Got Workload event	{"workload": {"name":"five","namespace":"core-clusterqueue-sx74m"}}
    2022-11-25T12:59:18.113656435Z	DEBUG	controller-runtime.webhook.webhooks	admission/http.go:96	received request	{"webhook": "/mutate-kueue-x-k8s-io-v1alpha2-workload", "UID": "db46e8a3-d821-4e77-aa2d-4730ca322622", "kind": "kueue.x-k8s.io/v1alpha2, Kind=Workload", "resource": {"group":"kueue.x-k8s.io","version":"v1alpha2","resource":"workloads"}}
    2022-11-25T12:59:18.114246194Z	DEBUG	controller-runtime.webhook.webhooks	admission/http.go:143	wrote response	{"webhook": "/mutate-kueue-x-k8s-io-v1alpha2-workload", "code": 200, "reason": "", "UID": "db46e8a3-d821-4e77-aa2d-4730ca322622", "allowed": true}
    2022-11-25T12:59:18.114717129Z	LEVEL(-2)	workload-reconciler	core/workload_controller.go:198	Workload update event	{"workload": {"name":"five","namespace":"core-clusterqueue-sx74m"}, "queue": "other", "status": "admitted", "clusterQueue": "other"}
    2022-11-25T12:59:18.114807103Z	ERROR	workload-reconciler	core/workload_controller.go:240	Updating workload in cache	{"workload": {"name":"five","namespace":"core-clusterqueue-sx74m"}, "queue": "other", "status": "admitted", "clusterQueue": "other", "error": "old ClusterQueue doesn't exist"}
    sigs.k8s.io/kueue/pkg/controller/core.(*WorkloadReconciler).Update
    	/home/prow/go/src/sigs.k8s.io/kueue/pkg/controller/core/workload_controller.go:240
    sigs.k8s.io/controller-runtime/pkg/source/internal.EventHandler.OnUpdate
    	/home/prow/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.12.3/pkg/source/internal/eventsource.go:88
    k8s.io/client-go/tools/cache.(*processorListener).run.func1
    	/home/prow/go/pkg/mod/k8s.io/client-go@v0.24.3/tools/cache/shared_informer.go:816
... skipping 45 lines ...
    2022-11-25T12:59:19.173909212Z	LEVEL(-3)	localqueue-reconciler	core/localqueue_controller.go:144	Got Workload event	{"workload": {"name":"four","namespace":"core-clusterqueue-sx74m"}}
    2022-11-25T12:59:19.173953472Z	LEVEL(-2)	cluster-queue-reconciler	core/clusterqueue_controller.go:201	Got Workload event	{"workload": {"name":"four","namespace":"core-clusterqueue-sx74m"}}
    2022-11-25T12:59:19.173974778Z	LEVEL(-2)	cluster-queue-reconciler	core/clusterqueue_controller.go:201	Got Workload event	{"workload": {"name":"four","namespace":"core-clusterqueue-sx74m"}}
    2022-11-25T12:59:19.173988209Z	LEVEL(-3)	localqueue-reconciler	core/localqueue_controller.go:144	Got Workload event	{"workload": {"name":"four","namespace":"core-clusterqueue-sx74m"}}
    2022-11-25T12:59:19.17393518Z	LEVEL(-2)	core/workload_controller.go:82	Reconciling Workload	{"controller": "workload", "controllerGroup": "kueue.x-k8s.io", "controllerKind": "Workload", "workload": {"name":"four","namespace":"core-clusterqueue-sx74m"}, "namespace": "core-clusterqueue-sx74m", "name": "four", "reconcileID": "260eecb1-fc1e-43e1-9487-1811b15ec54c", "workload": {"name":"four","namespace":"core-clusterqueue-sx74m"}}
    2022-11-25T12:59:19.182580734Z	LEVEL(-2)	workload-reconciler	core/workload_controller.go:198	Workload update event	{"workload": {"name":"five","namespace":"core-clusterqueue-sx74m"}, "queue": "other", "status": "finished", "prevStatus": "admitted", "clusterQueue": "other"}
    2022-11-25T12:59:19.182653129Z	ERROR	workload-reconciler	core/workload_controller.go:207	Failed to delete workload from cache	{"workload": {"name":"five","namespace":"core-clusterqueue-sx74m"}, "queue": "other", "status": "finished", "prevStatus": "admitted", "clusterQueue": "other", "error": "cluster queue not found"}
    sigs.k8s.io/kueue/pkg/controller/core.(*WorkloadReconciler).Update
    	/home/prow/go/src/sigs.k8s.io/kueue/pkg/controller/core/workload_controller.go:207
    sigs.k8s.io/controller-runtime/pkg/source/internal.EventHandler.OnUpdate
    	/home/prow/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.12.3/pkg/source/internal/eventsource.go:88
    k8s.io/client-go/tools/cache.(*processorListener).run.func1
    	/home/prow/go/pkg/mod/k8s.io/client-go@v0.24.3/tools/cache/shared_informer.go:816
... skipping 31 lines ...
    2022-11-25T12:59:20.221005304Z	DEBUG	controller-runtime.webhook.webhooks	admission/http.go:96	received request	{"webhook": "/validate-kueue-x-k8s-io-v1alpha2-clusterqueue", "UID": "b4ff3ba6-52b9-4d68-ab03-db126c6d3ce5", "kind": "kueue.x-k8s.io/v1alpha2, Kind=ClusterQueue", "resource": {"group":"kueue.x-k8s.io","version":"v1alpha2","resource":"clusterqueues"}}
    2022-11-25T12:59:20.221101744Z	LEVEL(-2)	localqueue-reconciler	core/localqueue_controller.go:119	LocalQueue delete event	{"localQueue": {"name":"queue","namespace":"core-clusterqueue-sx74m"}}
    2022-11-25T12:59:20.221511159Z	DEBUG	controller-runtime.webhook.webhooks	admission/http.go:143	wrote response	{"webhook": "/validate-kueue-x-k8s-io-v1alpha2-clusterqueue", "code": 200, "reason": "", "UID": "b4ff3ba6-52b9-4d68-ab03-db126c6d3ce5", "allowed": true}
    2022-11-25T12:59:20.225868629Z	LEVEL(-2)	cluster-queue-reconciler	core/clusterqueue_controller.go:165	ClusterQueue delete event	{"clusterQueue": {"name":"cluster-queue"}}
    2022-11-25T12:59:20.226138366Z	LEVEL(-2)	resourceflavor-reconciler	core/resourceflavor_controller.go:156	Got generic event	{"obj": {"name":"cluster-queue"}, "kind": "/, Kind="}
    2022-11-25T12:59:20.228516024Z	LEVEL(-2)	workload-reconciler	core/workload_controller.go:149	Workload delete event	{"workload": {"name":"five","namespace":"core-clusterqueue-sx74m"}, "queue": "other", "status": "finished"}
    2022-11-25T12:59:20.22857333Z	ERROR	workload-reconciler	core/workload_controller.go:158	Failed to delete workload from cache	{"workload": {"name":"five","namespace":"core-clusterqueue-sx74m"}, "queue": "other", "status": "finished", "error": "cluster queue not found"}
    sigs.k8s.io/kueue/pkg/controller/core.(*WorkloadReconciler).Delete
    	/home/prow/go/src/sigs.k8s.io/kueue/pkg/controller/core/workload_controller.go:158
    sigs.k8s.io/controller-runtime/pkg/source/internal.EventHandler.OnDelete
    	/home/prow/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.12.3/pkg/source/internal/eventsource.go:131
    k8s.io/client-go/tools/cache.(*processorListener).run.func1
    	/home/prow/go/pkg/mod/k8s.io/client-go@v0.24.3/tools/cache/shared_informer.go:820
... skipping 9 lines ...
    	/home/prow/go/pkg/mod/k8s.io/client-go@v0.24.3/tools/cache/shared_informer.go:812
    k8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1
    	/home/prow/go/pkg/mod/k8s.io/apimachinery@v0.24.3/pkg/util/wait/wait.go:73
    2022-11-25T12:59:20.228685784Z	LEVEL(-3)	localqueue-reconciler	core/localqueue_controller.go:144	Got Workload event	{"workload": {"name":"five","namespace":"core-clusterqueue-sx74m"}}
    2022-11-25T12:59:20.22871646Z	LEVEL(-2)	cluster-queue-reconciler	core/clusterqueue_controller.go:201	Got Workload event	{"workload": {"name":"five","namespace":"core-clusterqueue-sx74m"}}
    2022-11-25T12:59:20.230177607Z	LEVEL(-2)	workload-reconciler	core/workload_controller.go:149	Workload delete event	{"workload": {"name":"four","namespace":"core-clusterqueue-sx74m"}, "queue": "queue", "status": "finished"}
    2022-11-25T12:59:20.230220697Z	ERROR	workload-reconciler	core/workload_controller.go:158	Failed to delete workload from cache	{"workload": {"name":"four","namespace":"core-clusterqueue-sx74m"}, "queue": "queue", "status": "finished", "error": "cluster queue not found"}
    sigs.k8s.io/kueue/pkg/controller/core.(*WorkloadReconciler).Delete
    	/home/prow/go/src/sigs.k8s.io/kueue/pkg/controller/core/workload_controller.go:158
    sigs.k8s.io/controller-runtime/pkg/source/internal.EventHandler.OnDelete
    	/home/prow/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.12.3/pkg/source/internal/eventsource.go:131
    k8s.io/client-go/tools/cache.(*processorListener).run.func1
    	/home/prow/go/pkg/mod/k8s.io/client-go@v0.24.3/tools/cache/shared_informer.go:820
... skipping 9 lines ...
    	/home/prow/go/pkg/mod/k8s.io/client-go@v0.24.3/tools/cache/shared_informer.go:812
    k8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1
    	/home/prow/go/pkg/mod/k8s.io/apimachinery@v0.24.3/pkg/util/wait/wait.go:73
    2022-11-25T12:59:20.230282901Z	LEVEL(-3)	localqueue-reconciler	core/localqueue_controller.go:144	Got Workload event	{"workload": {"name":"four","namespace":"core-clusterqueue-sx74m"}}
    2022-11-25T12:59:20.230323558Z	LEVEL(-2)	cluster-queue-reconciler	core/clusterqueue_controller.go:201	Got Workload event	{"workload": {"name":"four","namespace":"core-clusterqueue-sx74m"}}
    2022-11-25T12:59:20.232766354Z	LEVEL(-2)	workload-reconciler	core/workload_controller.go:149	Workload delete event	{"workload": {"name":"one","namespace":"core-clusterqueue-sx74m"}, "queue": "queue", "status": "finished"}
    2022-11-25T12:59:20.232813635Z	ERROR	workload-reconciler	core/workload_controller.go:158	Failed to delete workload from cache	{"workload": {"name":"one","namespace":"core-clusterqueue-sx74m"}, "queue": "queue", "status": "finished", "error": "cluster queue not found"}
    sigs.k8s.io/kueue/pkg/controller/core.(*WorkloadReconciler).Delete
    	/home/prow/go/src/sigs.k8s.io/kueue/pkg/controller/core/workload_controller.go:158
    sigs.k8s.io/controller-runtime/pkg/source/internal.EventHandler.OnDelete
    	/home/prow/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.12.3/pkg/source/internal/eventsource.go:131
    k8s.io/client-go/tools/cache.(*processorListener).run.func1
    	/home/prow/go/pkg/mod/k8s.io/client-go@v0.24.3/tools/cache/shared_informer.go:820
... skipping 12 lines ...
    2022-11-25T12:59:20.232981246Z	LEVEL(-3)	localqueue-reconciler	core/localqueue_controller.go:144	Got Workload event	{"workload": {"name":"one","namespace":"core-clusterqueue-sx74m"}}
    2022-11-25T12:59:20.23304536Z	LEVEL(-2)	cluster-queue-reconciler	core/clusterqueue_controller.go:201	Got Workload event	{"workload": {"name":"one","namespace":"core-clusterqueue-sx74m"}}
    2022-11-25T12:59:20.23467726Z	LEVEL(-2)	workload-reconciler	core/workload_controller.go:149	Workload delete event	{"workload": {"name":"six","namespace":"core-clusterqueue-sx74m"}, "queue": "queue", "status": "finished"}
    2022-11-25T12:59:20.234770511Z	LEVEL(-2)	cluster-queue-reconciler	core/clusterqueue_controller.go:201	Got Workload event	{"workload": {"name":"six","namespace":"core-clusterqueue-sx74m"}}
    2022-11-25T12:59:20.234752441Z	LEVEL(-3)	localqueue-reconciler	core/localqueue_controller.go:144	Got Workload event	{"workload": {"name":"six","namespace":"core-clusterqueue-sx74m"}}
    2022-11-25T12:59:20.237022952Z	LEVEL(-2)	workload-reconciler	core/workload_controller.go:149	Workload delete event	{"workload": {"name":"three","namespace":"core-clusterqueue-sx74m"}, "queue": "queue", "status": "finished"}
    2022-11-25T12:59:20.237064311Z	ERROR	workload-reconciler	core/workload_controller.go:158	Failed to delete workload from cache	{"workload": {"name":"three","namespace":"core-clusterqueue-sx74m"}, "queue": "queue", "status": "finished", "error": "cluster queue not found"}
    sigs.k8s.io/kueue/pkg/controller/core.(*WorkloadReconciler).Delete
    	/home/prow/go/src/sigs.k8s.io/kueue/pkg/controller/core/workload_controller.go:158
    sigs.k8s.io/controller-runtime/pkg/source/internal.EventHandler.OnDelete
    	/home/prow/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.12.3/pkg/source/internal/eventsource.go:131
    k8s.io/client-go/tools/cache.(*processorListener).run.func1
    	/home/prow/go/pkg/mod/k8s.io/client-go@v0.24.3/tools/cache/shared_informer.go:820
... skipping 9 lines ...
    	/home/prow/go/pkg/mod/k8s.io/client-go@v0.24.3/tools/cache/shared_informer.go:812
    k8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1
    	/home/prow/go/pkg/mod/k8s.io/apimachinery@v0.24.3/pkg/util/wait/wait.go:73
    2022-11-25T12:59:20.237183872Z	LEVEL(-3)	localqueue-reconciler	core/localqueue_controller.go:144	Got Workload event	{"workload": {"name":"three","namespace":"core-clusterqueue-sx74m"}}
    2022-11-25T12:59:20.237233498Z	LEVEL(-2)	cluster-queue-reconciler	core/clusterqueue_controller.go:201	Got Workload event	{"workload": {"name":"three","namespace":"core-clusterqueue-sx74m"}}
    2022-11-25T12:59:20.238644289Z	LEVEL(-2)	workload-reconciler	core/workload_controller.go:149	Workload delete event	{"workload": {"name":"two","namespace":"core-clusterqueue-sx74m"}, "queue": "queue", "status": "finished"}
    2022-11-25T12:59:20.238681595Z	ERROR	workload-reconciler	core/workload_controller.go:158	Failed to delete workload from cache	{"workload": {"name":"two","namespace":"core-clusterqueue-sx74m"}, "queue": "queue", "status": "finished", "error": "cluster queue not found"}
    sigs.k8s.io/kueue/pkg/controller/core.(*WorkloadReconciler).Delete
    	/home/prow/go/src/sigs.k8s.io/kueue/pkg/controller/core/workload_controller.go:158
    sigs.k8s.io/controller-runtime/pkg/source/internal.EventHandler.OnDelete
    	/home/prow/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.12.3/pkg/source/internal/eventsource.go:131
    k8s.io/client-go/tools/cache.(*processorListener).run.func1
    	/home/prow/go/pkg/mod/k8s.io/client-go@v0.24.3/tools/cache/shared_informer.go:820
... skipping 106 lines ...
2022-11-25T12:59:21.809162732Z	DEBUG	controller-runtime.webhook.webhooks	admission/http.go:96	received request	{"webhook": "/validate-kueue-x-k8s-io-v1alpha2-clusterqueue", "UID": "0fd7f6ba-4f9d-4014-9e22-e3ccce4aafd8", "kind": "kueue.x-k8s.io/v1alpha2, Kind=ClusterQueue", "resource": {"group":"kueue.x-k8s.io","version":"v1alpha2","resource":"clusterqueues"}}
2022-11-25T12:59:21.809362873Z	DEBUG	controller-runtime.webhook.webhooks	admission/http.go:143	wrote response	{"webhook": "/validate-kueue-x-k8s-io-v1alpha2-clusterqueue", "code": 200, "reason": "", "UID": "0fd7f6ba-4f9d-4014-9e22-e3ccce4aafd8", "allowed": true}
2022-11-25T12:59:21.81264038Z	LEVEL(-2)	cluster-queue-reconciler	core/clusterqueue_controller.go:165	ClusterQueue delete event	{"clusterQueue": {"name":"foo-cq"}}
2022-11-25T12:59:21.812855242Z	LEVEL(-2)	resourceflavor-reconciler	core/resourceflavor_controller.go:156	Got generic event	{"obj": {"name":"foo-cq"}, "kind": "/, Kind="}
2022-11-25T12:59:21.841264785Z	LEVEL(-2)	localqueue-reconciler	core/localqueue_controller.go:119	LocalQueue delete event	{"localQueue": {"name":"queue","namespace":"core-clusterqueue-s4cb4"}}
2022-11-25T12:59:21.846596563Z	LEVEL(-2)	workload-reconciler	core/workload_controller.go:149	Workload delete event	{"workload": {"name":"workload","namespace":"core-clusterqueue-s4cb4"}, "queue": "queue", "status": "finished"}
2022-11-25T12:59:21.846651961Z	ERROR	workload-reconciler	core/workload_controller.go:158	Failed to delete workload from cache	{"workload": {"name":"workload","namespace":"core-clusterqueue-s4cb4"}, "queue": "queue", "status": "finished", "error": "cluster queue not found"}
sigs.k8s.io/kueue/pkg/controller/core.(*WorkloadReconciler).Delete
	/home/prow/go/src/sigs.k8s.io/kueue/pkg/controller/core/workload_controller.go:158
sigs.k8s.io/controller-runtime/pkg/source/internal.EventHandler.OnDelete
	/home/prow/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.12.3/pkg/source/internal/eventsource.go:131
k8s.io/client-go/tools/cache.(*processorListener).run.func1
	/home/prow/go/pkg/mod/k8s.io/client-go@v0.24.3/tools/cache/shared_informer.go:820
... skipping 62 lines ...
    2022-11-25T12:59:21.809162732Z	DEBUG	controller-runtime.webhook.webhooks	admission/http.go:96	received request	{"webhook": "/validate-kueue-x-k8s-io-v1alpha2-clusterqueue", "UID": "0fd7f6ba-4f9d-4014-9e22-e3ccce4aafd8", "kind": "kueue.x-k8s.io/v1alpha2, Kind=ClusterQueue", "resource": {"group":"kueue.x-k8s.io","version":"v1alpha2","resource":"clusterqueues"}}
    2022-11-25T12:59:21.809362873Z	DEBUG	controller-runtime.webhook.webhooks	admission/http.go:143	wrote response	{"webhook": "/validate-kueue-x-k8s-io-v1alpha2-clusterqueue", "code": 200, "reason": "", "UID": "0fd7f6ba-4f9d-4014-9e22-e3ccce4aafd8", "allowed": true}
    2022-11-25T12:59:21.81264038Z	LEVEL(-2)	cluster-queue-reconciler	core/clusterqueue_controller.go:165	ClusterQueue delete event	{"clusterQueue": {"name":"foo-cq"}}
    2022-11-25T12:59:21.812855242Z	LEVEL(-2)	resourceflavor-reconciler	core/resourceflavor_controller.go:156	Got generic event	{"obj": {"name":"foo-cq"}, "kind": "/, Kind="}
    2022-11-25T12:59:21.841264785Z	LEVEL(-2)	localqueue-reconciler	core/localqueue_controller.go:119	LocalQueue delete event	{"localQueue": {"name":"queue","namespace":"core-clusterqueue-s4cb4"}}
    2022-11-25T12:59:21.846596563Z	LEVEL(-2)	workload-reconciler	core/workload_controller.go:149	Workload delete event	{"workload": {"name":"workload","namespace":"core-clusterqueue-s4cb4"}, "queue": "queue", "status": "finished"}
    2022-11-25T12:59:21.846651961Z	ERROR	workload-reconciler	core/workload_controller.go:158	Failed to delete workload from cache	{"workload": {"name":"workload","namespace":"core-clusterqueue-s4cb4"}, "queue": "queue", "status": "finished", "error": "cluster queue not found"}
    sigs.k8s.io/kueue/pkg/controller/core.(*WorkloadReconciler).Delete
    	/home/prow/go/src/sigs.k8s.io/kueue/pkg/controller/core/workload_controller.go:158
    sigs.k8s.io/controller-runtime/pkg/source/internal.EventHandler.OnDelete
    	/home/prow/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.12.3/pkg/source/internal/eventsource.go:131
    k8s.io/client-go/tools/cache.(*processorListener).run.func1
    	/home/prow/go/pkg/mod/k8s.io/client-go@v0.24.3/tools/cache/shared_informer.go:820
... skipping 317 lines ...
2022-11-25T12:59:25.009226936Z	LEVEL(-2)	core/clusterqueue_controller.go:81	Reconciling ClusterQueue	{"controller": "clusterqueue", "controllerGroup": "kueue.x-k8s.io", "controllerKind": "ClusterQueue", "clusterQueue": {"name":"cluster-queue.queue-controller"}, "namespace": "", "name": "cluster-queue.queue-controller", "reconcileID": "7f770119-e10e-4602-b8e2-4c9fbace3065", "clusterQueue": {"name":"cluster-queue.queue-controller"}}
2022-11-25T12:59:25.009201621Z	INFO	logr@v1.2.3/logr.go:261	Stopping and waiting for non leader election runnables
2022-11-25T12:59:25.009434926Z	INFO	logr@v1.2.3/logr.go:261	Stopping and waiting for leader election runnables
2022-11-25T12:59:25.009417658Z	DEBUG	controller-runtime.certwatcher	logr@v1.2.3/logr.go:261	certificate event	{"event": "\"/tmp/envtest-serving-certs-395016376/tls.key\": REMOVE"}
2022-11-25T12:59:25.009490071Z	INFO	controller/controller.go:247	Shutdown signal received, waiting for all workers to finish	{"controller": "resourceflavor", "controllerGroup": "kueue.x-k8s.io", "controllerKind": "ResourceFlavor"}
2022-11-25T12:59:25.009536054Z	INFO	controller/controller.go:247	Shutdown signal received, waiting for all workers to finish	{"controller": "workload", "controllerGroup": "kueue.x-k8s.io", "controllerKind": "Workload"}
2022-11-25T12:59:25.009534541Z	ERROR	controller-runtime.certwatcher	logr@v1.2.3/logr.go:279	error re-watching file	{"error": "no such file or directory"}
github.com/go-logr/logr.Logger.Error
	/home/prow/go/pkg/mod/github.com/go-logr/logr@v1.2.3/logr.go:279
sigs.k8s.io/controller-runtime/pkg/certwatcher.(*CertWatcher).handleEvent
	/home/prow/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.12.3/pkg/certwatcher/certwatcher.go:147
sigs.k8s.io/controller-runtime/pkg/certwatcher.(*CertWatcher).Watch
	/home/prow/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.12.3/pkg/certwatcher/certwatcher.go:103
2022-11-25T12:59:25.009548349Z	ERROR	controller/controller.go:326	Reconciler error	{"controller": "clusterqueue", "controllerGroup": "kueue.x-k8s.io", "controllerKind": "ClusterQueue", "clusterQueue": {"name":"cluster-queue.queue-controller"}, "namespace": "", "name": "cluster-queue.queue-controller", "reconcileID": "7f770119-e10e-4602-b8e2-4c9fbace3065", "error": "Put \"https://127.0.0.1:44317/apis/kueue.x-k8s.io/v1alpha2/clusterqueues/cluster-queue.queue-controller\": context canceled"}
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
	/home/prow/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.12.3/pkg/internal/controller/controller.go:326
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
	/home/prow/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.12.3/pkg/internal/controller/controller.go:273
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
	/home/prow/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.12.3/pkg/internal/controller/controller.go:234
2022-11-25T12:59:25.009572711Z	INFO	controller/controller.go:249	All workers finished	{"controller": "resourceflavor", "controllerGroup": "kueue.x-k8s.io", "controllerKind": "ResourceFlavor"}
2022-11-25T12:59:25.009583151Z	INFO	controller/controller.go:249	All workers finished	{"controller": "workload", "controllerGroup": "kueue.x-k8s.io", "controllerKind": "Workload"}
2022-11-25T12:59:25.009518944Z	INFO	controller/controller.go:247	Shutdown signal received, waiting for all workers to finish	{"controller": "localqueue", "controllerGroup": "kueue.x-k8s.io", "controllerKind": "LocalQueue"}
2022-11-25T12:59:25.009589076Z	ERROR	controller-runtime.certwatcher	logr@v1.2.3/logr.go:279	error re-reading certificate	{"error": "open /tmp/envtest-serving-certs-395016376/tls.crt: no such file or directory"}
github.com/go-logr/logr.Logger.Error
	/home/prow/go/pkg/mod/github.com/go-logr/logr@v1.2.3/logr.go:279
sigs.k8s.io/controller-runtime/pkg/certwatcher.(*CertWatcher).handleEvent
	/home/prow/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.12.3/pkg/certwatcher/certwatcher.go:152
sigs.k8s.io/controller-runtime/pkg/certwatcher.(*CertWatcher).Watch
	/home/prow/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.12.3/pkg/certwatcher/certwatcher.go:103
2022-11-25T12:59:25.009608949Z	INFO	controller/controller.go:249	All workers finished	{"controller": "localqueue", "controllerGroup": "kueue.x-k8s.io", "controllerKind": "LocalQueue"}
2022-11-25T12:59:25.009609832Z	DEBUG	controller-runtime.certwatcher	logr@v1.2.3/logr.go:261	certificate event	{"event": "\"/tmp/envtest-serving-certs-395016376/tls.crt\": REMOVE"}
2022-11-25T12:59:25.009628731Z	ERROR	controller-runtime.certwatcher	logr@v1.2.3/logr.go:279	error re-watching file	{"error": "no such file or directory"}
github.com/go-logr/logr.Logger.Error
	/home/prow/go/pkg/mod/github.com/go-logr/logr@v1.2.3/logr.go:279
sigs.k8s.io/controller-runtime/pkg/certwatcher.(*CertWatcher).handleEvent
	/home/prow/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.12.3/pkg/certwatcher/certwatcher.go:147
sigs.k8s.io/controller-runtime/pkg/certwatcher.(*CertWatcher).Watch
	/home/prow/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.12.3/pkg/certwatcher/certwatcher.go:103
2022-11-25T12:59:25.009547942Z	INFO	controller/controller.go:247	Shutdown signal received, waiting for all workers to finish	{"controller": "clusterqueue", "controllerGroup": "kueue.x-k8s.io", "controllerKind": "ClusterQueue"}
2022-11-25T12:59:25.009648558Z	ERROR	controller-runtime.certwatcher	logr@v1.2.3/logr.go:279	error re-reading certificate	{"error": "open /tmp/envtest-serving-certs-395016376/tls.crt: no such file or directory"}
github.com/go-logr/logr.Logger.Error
	/home/prow/go/pkg/mod/github.com/go-logr/logr@v1.2.3/logr.go:279
sigs.k8s.io/controller-runtime/pkg/certwatcher.(*CertWatcher).handleEvent
	/home/prow/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.12.3/pkg/certwatcher/certwatcher.go:152
sigs.k8s.io/controller-runtime/pkg/certwatcher.(*CertWatcher).Watch
	/home/prow/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.12.3/pkg/certwatcher/certwatcher.go:103
2022-11-25T12:59:25.009649132Z	INFO	controller/controller.go:249	All workers finished	{"controller": "clusterqueue", "controllerGroup": "kueue.x-k8s.io", "controllerKind": "ClusterQueue"}
... skipping 12 lines ...
    2022-11-25T12:59:25.009226936Z	LEVEL(-2)	core/clusterqueue_controller.go:81	Reconciling ClusterQueue	{"controller": "clusterqueue", "controllerGroup": "kueue.x-k8s.io", "controllerKind": "ClusterQueue", "clusterQueue": {"name":"cluster-queue.queue-controller"}, "namespace": "", "name": "cluster-queue.queue-controller", "reconcileID": "7f770119-e10e-4602-b8e2-4c9fbace3065", "clusterQueue": {"name":"cluster-queue.queue-controller"}}
    2022-11-25T12:59:25.009201621Z	INFO	logr@v1.2.3/logr.go:261	Stopping and waiting for non leader election runnables
    2022-11-25T12:59:25.009434926Z	INFO	logr@v1.2.3/logr.go:261	Stopping and waiting for leader election runnables
    2022-11-25T12:59:25.009417658Z	DEBUG	controller-runtime.certwatcher	logr@v1.2.3/logr.go:261	certificate event	{"event": "\"/tmp/envtest-serving-certs-395016376/tls.key\": REMOVE"}
    2022-11-25T12:59:25.009490071Z	INFO	controller/controller.go:247	Shutdown signal received, waiting for all workers to finish	{"controller": "resourceflavor", "controllerGroup": "kueue.x-k8s.io", "controllerKind": "ResourceFlavor"}
    2022-11-25T12:59:25.009536054Z	INFO	controller/controller.go:247	Shutdown signal received, waiting for all workers to finish	{"controller": "workload", "controllerGroup": "kueue.x-k8s.io", "controllerKind": "Workload"}
    2022-11-25T12:59:25.009534541Z	ERROR	controller-runtime.certwatcher	logr@v1.2.3/logr.go:279	error re-watching file	{"error": "no such file or directory"}
    github.com/go-logr/logr.Logger.Error
    	/home/prow/go/pkg/mod/github.com/go-logr/logr@v1.2.3/logr.go:279
    sigs.k8s.io/controller-runtime/pkg/certwatcher.(*CertWatcher).handleEvent
    	/home/prow/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.12.3/pkg/certwatcher/certwatcher.go:147
    sigs.k8s.io/controller-runtime/pkg/certwatcher.(*CertWatcher).Watch
    	/home/prow/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.12.3/pkg/certwatcher/certwatcher.go:103
    2022-11-25T12:59:25.009548349Z	ERROR	controller/controller.go:326	Reconciler error	{"controller": "clusterqueue", "controllerGroup": "kueue.x-k8s.io", "controllerKind": "ClusterQueue", "clusterQueue": {"name":"cluster-queue.queue-controller"}, "namespace": "", "name": "cluster-queue.queue-controller", "reconcileID": "7f770119-e10e-4602-b8e2-4c9fbace3065", "error": "Put \"https://127.0.0.1:44317/apis/kueue.x-k8s.io/v1alpha2/clusterqueues/cluster-queue.queue-controller\": context canceled"}
    sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
    	/home/prow/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.12.3/pkg/internal/controller/controller.go:326
    sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
    	/home/prow/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.12.3/pkg/internal/controller/controller.go:273
    sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
    	/home/prow/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.12.3/pkg/internal/controller/controller.go:234
    2022-11-25T12:59:25.009572711Z	INFO	controller/controller.go:249	All workers finished	{"controller": "resourceflavor", "controllerGroup": "kueue.x-k8s.io", "controllerKind": "ResourceFlavor"}
    2022-11-25T12:59:25.009583151Z	INFO	controller/controller.go:249	All workers finished	{"controller": "workload", "controllerGroup": "kueue.x-k8s.io", "controllerKind": "Workload"}
    2022-11-25T12:59:25.009518944Z	INFO	controller/controller.go:247	Shutdown signal received, waiting for all workers to finish	{"controller": "localqueue", "controllerGroup": "kueue.x-k8s.io", "controllerKind": "LocalQueue"}
    2022-11-25T12:59:25.009589076Z	ERROR	controller-runtime.certwatcher	logr@v1.2.3/logr.go:279	error re-reading certificate	{"error": "open /tmp/envtest-serving-certs-395016376/tls.crt: no such file or directory"}
    github.com/go-logr/logr.Logger.Error
    	/home/prow/go/pkg/mod/github.com/go-logr/logr@v1.2.3/logr.go:279
    sigs.k8s.io/controller-runtime/pkg/certwatcher.(*CertWatcher).handleEvent
    	/home/prow/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.12.3/pkg/certwatcher/certwatcher.go:152
    sigs.k8s.io/controller-runtime/pkg/certwatcher.(*CertWatcher).Watch
    	/home/prow/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.12.3/pkg/certwatcher/certwatcher.go:103
    2022-11-25T12:59:25.009608949Z	INFO	controller/controller.go:249	All workers finished	{"controller": "localqueue", "controllerGroup": "kueue.x-k8s.io", "controllerKind": "LocalQueue"}
    2022-11-25T12:59:25.009609832Z	DEBUG	controller-runtime.certwatcher	logr@v1.2.3/logr.go:261	certificate event	{"event": "\"/tmp/envtest-serving-certs-395016376/tls.crt\": REMOVE"}
    2022-11-25T12:59:25.009628731Z	ERROR	controller-runtime.certwatcher	logr@v1.2.3/logr.go:279	error re-watching file	{"error": "no such file or directory"}
    github.com/go-logr/logr.Logger.Error
    	/home/prow/go/pkg/mod/github.com/go-logr/logr@v1.2.3/logr.go:279
    sigs.k8s.io/controller-runtime/pkg/certwatcher.(*CertWatcher).handleEvent
    	/home/prow/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.12.3/pkg/certwatcher/certwatcher.go:147
    sigs.k8s.io/controller-runtime/pkg/certwatcher.(*CertWatcher).Watch
    	/home/prow/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.12.3/pkg/certwatcher/certwatcher.go:103
    2022-11-25T12:59:25.009547942Z	INFO	controller/controller.go:247	Shutdown signal received, waiting for all workers to finish	{"controller": "clusterqueue", "controllerGroup": "kueue.x-k8s.io", "controllerKind": "ClusterQueue"}
    2022-11-25T12:59:25.009648558Z	ERROR	controller-runtime.certwatcher	logr@v1.2.3/logr.go:279	error re-reading certificate	{"error": "open /tmp/envtest-serving-certs-395016376/tls.crt: no such file or directory"}
    github.com/go-logr/logr.Logger.Error
    	/home/prow/go/pkg/mod/github.com/go-logr/logr@v1.2.3/logr.go:279
    sigs.k8s.io/controller-runtime/pkg/certwatcher.(*CertWatcher).handleEvent
    	/home/prow/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.12.3/pkg/certwatcher/certwatcher.go:152
    sigs.k8s.io/controller-runtime/pkg/certwatcher.(*CertWatcher).Watch
    	/home/prow/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.12.3/pkg/certwatcher/certwatcher.go:103
    2022-11-25T12:59:25.009649132Z	INFO	controller/controller.go:249	All workers finished	{"controller": "clusterqueue", "controllerGroup": "kueue.x-k8s.io", "controllerKind": "ClusterQueue"}
... skipping 9 lines ...
[ReportAfterSuite] PASSED [0.004 seconds]
[ReportAfterSuite] Autogenerated ReportAfterSuite for --junit-report
autogenerated by Ginkgo
------------------------------

Ran 12 of 12 Specs in 23.189 seconds
SUCCESS! -- 12 Passed | 0 Failed | 0 Pending | 0 Skipped
PASS
Running Suite: Job Controller Suite - /home/prow/go/src/sigs.k8s.io/kueue/test/integration/controller/job
=========================================================================================================
Random Seed: 1669381133

Will run 8 of 8 specs
... skipping 41 lines ...
2022-11-25T12:59:34.407197612Z	LEVEL(-2)	job/job_controller.go:141	Reconciling Job	{"controller": "job", "controllerGroup": "batch", "controllerKind": "Job", "job": {"name":"test-job","namespace":"default"}, "namespace": "default", "name": "test-job", "reconcileID": "76d2c3f7-7098-4b30-a8c5-3fabd9c64b6b", "job": {"name":"test-job","namespace":"default"}}
2022-11-25T12:59:34.407306855Z	LEVEL(-2)	job/job_controller.go:200	Job changed queues, updating workload	{"controller": "job", "controllerGroup": "batch", "controllerKind": "Job", "job": {"name":"test-job","namespace":"default"}, "namespace": "default", "name": "test-job", "reconcileID": "76d2c3f7-7098-4b30-a8c5-3fabd9c64b6b", "job": {"name":"test-job","namespace":"default"}}
2022-11-25T12:59:34.413989468Z	LEVEL(-2)	job/job_controller.go:141	Reconciling Job	{"controller": "job", "controllerGroup": "batch", "controllerKind": "Job", "job": {"name":"test-job","namespace":"default"}, "namespace": "default", "name": "test-job", "reconcileID": "375e91f2-ea9e-468d-9f05-09195979638c", "job": {"name":"test-job","namespace":"default"}}
2022-11-25T12:59:34.414097219Z	LEVEL(-3)	job/job_controller.go:208	Job is suspended and workload not yet admitted by a clusterQueue, nothing to do	{"controller": "job", "controllerGroup": "batch", "controllerKind": "Job", "job": {"name":"test-job","namespace":"default"}, "namespace": "default", "name": "test-job", "reconcileID": "375e91f2-ea9e-468d-9f05-09195979638c", "job": {"name":"test-job","namespace":"default"}}
STEP: checking a second non-matching workload is deleted 11/25/22 12:59:34.664
2022-11-25T12:59:34.67384398Z	LEVEL(-2)	job/job_controller.go:141	Reconciling Job	{"controller": "job", "controllerGroup": "batch", "controllerKind": "Job", "job": {"name":"test-job","namespace":"default"}, "namespace": "default", "name": "test-job", "reconcileID": "37748b64-8be4-42f6-a26f-6fb8ac9f4a58", "job": {"name":"test-job","namespace":"default"}}
2022-11-25T12:59:34.679557942Z	ERROR	job/job_controller.go:153	Getting existing workloads	{"controller": "job", "controllerGroup": "batch", "controllerKind": "Job", "job": {"name":"test-job","namespace":"default"}, "namespace": "default", "name": "test-job", "reconcileID": "37748b64-8be4-42f6-a26f-6fb8ac9f4a58", "job": {"name":"test-job","namespace":"default"}, "error": "only one workload should exist, found 2"}
sigs.k8s.io/kueue/pkg/controller/workload/job.(*JobReconciler).Reconcile
	/home/prow/go/src/sigs.k8s.io/kueue/pkg/controller/workload/job/job_controller.go:153
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile
	/home/prow/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.12.3/pkg/internal/controller/controller.go:121
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
	/home/prow/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.12.3/pkg/internal/controller/controller.go:320
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
	/home/prow/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.12.3/pkg/internal/controller/controller.go:273
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
	/home/prow/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.12.3/pkg/internal/controller/controller.go:234
2022-11-25T12:59:34.679670527Z	ERROR	controller/controller.go:326	Reconciler error	{"controller": "job", "controllerGroup": "batch", "controllerKind": "Job", "job": {"name":"test-job","namespace":"default"}, "namespace": "default", "name": "test-job", "reconcileID": "37748b64-8be4-42f6-a26f-6fb8ac9f4a58", "error": "only one workload should exist, found 2"}
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
	/home/prow/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.12.3/pkg/internal/controller/controller.go:326
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
	/home/prow/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.12.3/pkg/internal/controller/controller.go:273
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
	/home/prow/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.12.3/pkg/internal/controller/controller.go:234
... skipping 9 lines ...
2022-11-25T12:59:38.092835668Z	DEBUG	events	recorder/recorder.go:103	Normal	{"object": {"kind":"Job","namespace":"default","name":"test-job","uid":"aa04ec8e-bab7-4dd8-a615-dcad19f9338a","apiVersion":"batch/v1","resourceVersion":"227"}, "reason": "Started", "message": "Admitted by clusterQueue cluster-queue"}
2022-11-25T12:59:38.092949161Z	LEVEL(-3)	job/job_controller.go:223	Job running with admitted workload, nothing to do	{"controller": "job", "controllerGroup": "batch", "controllerKind": "Job", "job": {"name":"test-job","namespace":"default"}, "namespace": "default", "name": "test-job", "reconcileID": "e601a184-8a21-4de5-8041-6c0e4ba3fcaf", "job": {"name":"test-job","namespace":"default"}}
STEP: checking the job gets suspended when parallelism changes and the added node selectors are removed 11/25/22 12:59:41.246
2022-11-25T12:59:41.252833455Z	LEVEL(-2)	job/job_controller.go:141	Reconciling Job	{"controller": "job", "controllerGroup": "batch", "controllerKind": "Job", "job": {"name":"test-job","namespace":"default"}, "namespace": "default", "name": "test-job", "reconcileID": "699cba3e-49f2-4afe-9fb2-f1d6efb1af9e", "job": {"name":"test-job","namespace":"default"}}
2022-11-25T12:59:41.252943831Z	LEVEL(-2)	job/job_controller.go:362	job with no matching workload, suspending	{"controller": "job", "controllerGroup": "batch", "controllerKind": "Job", "job": {"name":"test-job","namespace":"default"}, "namespace": "default", "name": "test-job", "reconcileID": "699cba3e-49f2-4afe-9fb2-f1d6efb1af9e", "job": {"name":"test-job","namespace":"default"}}
2022-11-25T12:59:41.256781665Z	DEBUG	events	recorder/recorder.go:103	Normal	{"object": {"kind":"Job","namespace":"default","name":"test-job","uid":"aa04ec8e-bab7-4dd8-a615-dcad19f9338a","apiVersion":"batch/v1","resourceVersion":"231"}, "reason": "Stopped", "message": "No matching Workload"}
2022-11-25T12:59:41.265015687Z	ERROR	job/job_controller.go:153	Getting existing workloads	{"controller": "job", "controllerGroup": "batch", "controllerKind": "Job", "job": {"name":"test-job","namespace":"default"}, "namespace": "default", "name": "test-job", "reconcileID": "699cba3e-49f2-4afe-9fb2-f1d6efb1af9e", "job": {"name":"test-job","namespace":"default"}, "error": "no matching workload was found, tried deleting 1 existing workload(s)"}
sigs.k8s.io/kueue/pkg/controller/workload/job.(*JobReconciler).Reconcile
	/home/prow/go/src/sigs.k8s.io/kueue/pkg/controller/workload/job/job_controller.go:153
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile
	/home/prow/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.12.3/pkg/internal/controller/controller.go:121
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
	/home/prow/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.12.3/pkg/internal/controller/controller.go:320
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
	/home/prow/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.12.3/pkg/internal/controller/controller.go:273
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
	/home/prow/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.12.3/pkg/internal/controller/controller.go:234
2022-11-25T12:59:41.26509569Z	ERROR	controller/controller.go:326	Reconciler error	{"controller": "job", "controllerGroup": "batch", "controllerKind": "Job", "job": {"name":"test-job","namespace":"default"}, "namespace": "default", "name": "test-job", "reconcileID": "699cba3e-49f2-4afe-9fb2-f1d6efb1af9e", "error": "no matching workload was found, tried deleting 1 existing workload(s)"}
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
	/home/prow/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.12.3/pkg/internal/controller/controller.go:326
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
	/home/prow/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.12.3/pkg/internal/controller/controller.go:273
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
	/home/prow/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.12.3/pkg/internal/controller/controller.go:234
... skipping 17 lines ...
2022-11-25T12:59:45.037812481Z	INFO	logr@v1.2.3/logr.go:261	Stopping and waiting for leader election runnables
2022-11-25T12:59:45.037839632Z	INFO	controller/controller.go:247	Shutdown signal received, waiting for all workers to finish	{"controller": "job", "controllerGroup": "batch", "controllerKind": "Job"}
2022-11-25T12:59:45.037890756Z	INFO	controller/controller.go:249	All workers finished	{"controller": "job", "controllerGroup": "batch", "controllerKind": "Job"}
2022-11-25T12:59:45.037905573Z	INFO	logr@v1.2.3/logr.go:261	Stopping and waiting for caches
2022-11-25T12:59:45.038107772Z	DEBUG	controller-runtime.certwatcher	logr@v1.2.3/logr.go:261	certificate event	{"event": "\"/tmp/envtest-serving-certs-2730287444/tls.key\": REMOVE"}
2022-11-25T12:59:45.03817182Z	INFO	logr@v1.2.3/logr.go:261	Stopping and waiting for webhooks
2022-11-25T12:59:45.038205039Z	ERROR	controller-runtime.certwatcher	logr@v1.2.3/logr.go:279	error re-watching file	{"error": "no such file or directory"}
github.com/go-logr/logr.Logger.Error
	/home/prow/go/pkg/mod/github.com/go-logr/logr@v1.2.3/logr.go:279
sigs.k8s.io/controller-runtime/pkg/certwatcher.(*CertWatcher).handleEvent
	/home/prow/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.12.3/pkg/certwatcher/certwatcher.go:147
sigs.k8s.io/controller-runtime/pkg/certwatcher.(*CertWatcher).Watch
	/home/prow/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.12.3/pkg/certwatcher/certwatcher.go:103
2022-11-25T12:59:45.038248862Z	ERROR	controller-runtime.certwatcher	logr@v1.2.3/logr.go:279	error re-reading certificate	{"error": "open /tmp/envtest-serving-certs-2730287444/tls.crt: no such file or directory"}
github.com/go-logr/logr.Logger.Error
	/home/prow/go/pkg/mod/github.com/go-logr/logr@v1.2.3/logr.go:279
sigs.k8s.io/controller-runtime/pkg/certwatcher.(*CertWatcher).handleEvent
	/home/prow/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.12.3/pkg/certwatcher/certwatcher.go:152
sigs.k8s.io/controller-runtime/pkg/certwatcher.(*CertWatcher).Watch
	/home/prow/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.12.3/pkg/certwatcher/certwatcher.go:103
2022-11-25T12:59:45.038276783Z	INFO	controller-runtime.webhook	logr@v1.2.3/logr.go:261	shutting down webhook server
... skipping 46 lines ...
    2022-11-25T12:59:34.407197612Z	LEVEL(-2)	job/job_controller.go:141	Reconciling Job	{"controller": "job", "controllerGroup": "batch", "controllerKind": "Job", "job": {"name":"test-job","namespace":"default"}, "namespace": "default", "name": "test-job", "reconcileID": "76d2c3f7-7098-4b30-a8c5-3fabd9c64b6b", "job": {"name":"test-job","namespace":"default"}}
    2022-11-25T12:59:34.407306855Z	LEVEL(-2)	job/job_controller.go:200	Job changed queues, updating workload	{"controller": "job", "controllerGroup": "batch", "controllerKind": "Job", "job": {"name":"test-job","namespace":"default"}, "namespace": "default", "name": "test-job", "reconcileID": "76d2c3f7-7098-4b30-a8c5-3fabd9c64b6b", "job": {"name":"test-job","namespace":"default"}}
    2022-11-25T12:59:34.413989468Z	LEVEL(-2)	job/job_controller.go:141	Reconciling Job	{"controller": "job", "controllerGroup": "batch", "controllerKind": "Job", "job": {"name":"test-job","namespace":"default"}, "namespace": "default", "name": "test-job", "reconcileID": "375e91f2-ea9e-468d-9f05-09195979638c", "job": {"name":"test-job","namespace":"default"}}
    2022-11-25T12:59:34.414097219Z	LEVEL(-3)	job/job_controller.go:208	Job is suspended and workload not yet admitted by a clusterQueue, nothing to do	{"controller": "job", "controllerGroup": "batch", "controllerKind": "Job", "job": {"name":"test-job","namespace":"default"}, "namespace": "default", "name": "test-job", "reconcileID": "375e91f2-ea9e-468d-9f05-09195979638c", "job": {"name":"test-job","namespace":"default"}}
    STEP: checking a second non-matching workload is deleted 11/25/22 12:59:34.664
    2022-11-25T12:59:34.67384398Z	LEVEL(-2)	job/job_controller.go:141	Reconciling Job	{"controller": "job", "controllerGroup": "batch", "controllerKind": "Job", "job": {"name":"test-job","namespace":"default"}, "namespace": "default", "name": "test-job", "reconcileID": "37748b64-8be4-42f6-a26f-6fb8ac9f4a58", "job": {"name":"test-job","namespace":"default"}}
    2022-11-25T12:59:34.679557942Z	ERROR	job/job_controller.go:153	Getting existing workloads	{"controller": "job", "controllerGroup": "batch", "controllerKind": "Job", "job": {"name":"test-job","namespace":"default"}, "namespace": "default", "name": "test-job", "reconcileID": "37748b64-8be4-42f6-a26f-6fb8ac9f4a58", "job": {"name":"test-job","namespace":"default"}, "error": "only one workload should exist, found 2"}
    sigs.k8s.io/kueue/pkg/controller/workload/job.(*JobReconciler).Reconcile
    	/home/prow/go/src/sigs.k8s.io/kueue/pkg/controller/workload/job/job_controller.go:153
    sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile
    	/home/prow/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.12.3/pkg/internal/controller/controller.go:121
    sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
    	/home/prow/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.12.3/pkg/internal/controller/controller.go:320
    sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
    	/home/prow/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.12.3/pkg/internal/controller/controller.go:273
    sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
    	/home/prow/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.12.3/pkg/internal/controller/controller.go:234
    2022-11-25T12:59:34.679670527Z	ERROR	controller/controller.go:326	Reconciler error	{"controller": "job", "controllerGroup": "batch", "controllerKind": "Job", "job": {"name":"test-job","namespace":"default"}, "namespace": "default", "name": "test-job", "reconcileID": "37748b64-8be4-42f6-a26f-6fb8ac9f4a58", "error": "only one workload should exist, found 2"}
    sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
    	/home/prow/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.12.3/pkg/internal/controller/controller.go:326
    sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
    	/home/prow/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.12.3/pkg/internal/controller/controller.go:273
    sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
    	/home/prow/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.12.3/pkg/internal/controller/controller.go:234
... skipping 9 lines ...
    2022-11-25T12:59:38.092835668Z	DEBUG	events	recorder/recorder.go:103	Normal	{"object": {"kind":"Job","namespace":"default","name":"test-job","uid":"aa04ec8e-bab7-4dd8-a615-dcad19f9338a","apiVersion":"batch/v1","resourceVersion":"227"}, "reason": "Started", "message": "Admitted by clusterQueue cluster-queue"}
    2022-11-25T12:59:38.092949161Z	LEVEL(-3)	job/job_controller.go:223	Job running with admitted workload, nothing to do	{"controller": "job", "controllerGroup": "batch", "controllerKind": "Job", "job": {"name":"test-job","namespace":"default"}, "namespace": "default", "name": "test-job", "reconcileID": "e601a184-8a21-4de5-8041-6c0e4ba3fcaf", "job": {"name":"test-job","namespace":"default"}}
    STEP: checking the job gets suspended when parallelism changes and the added node selectors are removed 11/25/22 12:59:41.246
    2022-11-25T12:59:41.252833455Z	LEVEL(-2)	job/job_controller.go:141	Reconciling Job	{"controller": "job", "controllerGroup": "batch", "controllerKind": "Job", "job": {"name":"test-job","namespace":"default"}, "namespace": "default", "name": "test-job", "reconcileID": "699cba3e-49f2-4afe-9fb2-f1d6efb1af9e", "job": {"name":"test-job","namespace":"default"}}
    2022-11-25T12:59:41.252943831Z	LEVEL(-2)	job/job_controller.go:362	job with no matching workload, suspending	{"controller": "job", "controllerGroup": "batch", "controllerKind": "Job", "job": {"name":"test-job","namespace":"default"}, "namespace": "default", "name": "test-job", "reconcileID": "699cba3e-49f2-4afe-9fb2-f1d6efb1af9e", "job": {"name":"test-job","namespace":"default"}}
    2022-11-25T12:59:41.256781665Z	DEBUG	events	recorder/recorder.go:103	Normal	{"object": {"kind":"Job","namespace":"default","name":"test-job","uid":"aa04ec8e-bab7-4dd8-a615-dcad19f9338a","apiVersion":"batch/v1","resourceVersion":"231"}, "reason": "Stopped", "message": "No matching Workload"}
    2022-11-25T12:59:41.265015687Z	ERROR	job/job_controller.go:153	Getting existing workloads	{"controller": "job", "controllerGroup": "batch", "controllerKind": "Job", "job": {"name":"test-job","namespace":"default"}, "namespace": "default", "name": "test-job", "reconcileID": "699cba3e-49f2-4afe-9fb2-f1d6efb1af9e", "job": {"name":"test-job","namespace":"default"}, "error": "no matching workload was found, tried deleting 1 existing workload(s)"}
    sigs.k8s.io/kueue/pkg/controller/workload/job.(*JobReconciler).Reconcile
    	/home/prow/go/src/sigs.k8s.io/kueue/pkg/controller/workload/job/job_controller.go:153
    sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile
    	/home/prow/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.12.3/pkg/internal/controller/controller.go:121
    sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
    	/home/prow/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.12.3/pkg/internal/controller/controller.go:320
    sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
    	/home/prow/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.12.3/pkg/internal/controller/controller.go:273
    sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
    	/home/prow/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.12.3/pkg/internal/controller/controller.go:234
    2022-11-25T12:59:41.26509569Z	ERROR	controller/controller.go:326	Reconciler error	{"controller": "job", "controllerGroup": "batch", "controllerKind": "Job", "job": {"name":"test-job","namespace":"default"}, "namespace": "default", "name": "test-job", "reconcileID": "699cba3e-49f2-4afe-9fb2-f1d6efb1af9e", "error": "no matching workload was found, tried deleting 1 existing workload(s)"}
    sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
    	/home/prow/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.12.3/pkg/internal/controller/controller.go:326
    sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
    	/home/prow/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.12.3/pkg/internal/controller/controller.go:273
    sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
    	/home/prow/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.12.3/pkg/internal/controller/controller.go:234
... skipping 17 lines ...
    2022-11-25T12:59:45.037812481Z	INFO	logr@v1.2.3/logr.go:261	Stopping and waiting for leader election runnables
    2022-11-25T12:59:45.037839632Z	INFO	controller/controller.go:247	Shutdown signal received, waiting for all workers to finish	{"controller": "job", "controllerGroup": "batch", "controllerKind": "Job"}
    2022-11-25T12:59:45.037890756Z	INFO	controller/controller.go:249	All workers finished	{"controller": "job", "controllerGroup": "batch", "controllerKind": "Job"}
    2022-11-25T12:59:45.037905573Z	INFO	logr@v1.2.3/logr.go:261	Stopping and waiting for caches
    2022-11-25T12:59:45.038107772Z	DEBUG	controller-runtime.certwatcher	logr@v1.2.3/logr.go:261	certificate event	{"event": "\"/tmp/envtest-serving-certs-2730287444/tls.key\": REMOVE"}
    2022-11-25T12:59:45.03817182Z	INFO	logr@v1.2.3/logr.go:261	Stopping and waiting for webhooks
    2022-11-25T12:59:45.038205039Z	ERROR	controller-runtime.certwatcher	logr@v1.2.3/logr.go:279	error re-watching file	{"error": "no such file or directory"}
    github.com/go-logr/logr.Logger.Error
    	/home/prow/go/pkg/mod/github.com/go-logr/logr@v1.2.3/logr.go:279
    sigs.k8s.io/controller-runtime/pkg/certwatcher.(*CertWatcher).handleEvent
    	/home/prow/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.12.3/pkg/certwatcher/certwatcher.go:147
    sigs.k8s.io/controller-runtime/pkg/certwatcher.(*CertWatcher).Watch
    	/home/prow/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.12.3/pkg/certwatcher/certwatcher.go:103
    2022-11-25T12:59:45.038248862Z	ERROR	controller-runtime.certwatcher	logr@v1.2.3/logr.go:279	error re-reading certificate	{"error": "open /tmp/envtest-serving-certs-2730287444/tls.crt: no such file or directory"}
    github.com/go-logr/logr.Logger.Error
    	/home/prow/go/pkg/mod/github.com/go-logr/logr@v1.2.3/logr.go:279
    sigs.k8s.io/controller-runtime/pkg/certwatcher.(*CertWatcher).handleEvent
    	/home/prow/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.12.3/pkg/certwatcher/certwatcher.go:152
    sigs.k8s.io/controller-runtime/pkg/certwatcher.(*CertWatcher).Watch
    	/home/prow/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.12.3/pkg/certwatcher/certwatcher.go:103
    2022-11-25T12:59:45.038276783Z	INFO	controller-runtime.webhook	logr@v1.2.3/logr.go:261	shutting down webhook server
... skipping 2 lines ...
------------------------------
Job Webhook when With manageJobsWithoutQueueName enabled
  Should suspend a Job even no queue name specified
  /home/prow/go/src/sigs.k8s.io/kueue/test/integration/controller/job/job_webhook_test.go:59
STEP: bootstrapping test environment 11/25/22 12:59:46.189
2022-11-25T12:59:46.189864729Z	DEBUG	controller-runtime.test-env	logr@v1.2.3/logr.go:261	starting control plane
{"component":"entrypoint","file":"k8s.io/test-infra/prow/entrypoint/run.go:168","func":"k8s.io/test-infra/prow/entrypoint.Options.ExecuteProcess","level":"error","msg":"Entrypoint received interrupt: terminated","severity":"error","time":"2022-11-25T12:59:47Z"}